Meta AI and KAUST Researchers Propose Neural Computers

Researchers from Meta AI and the King Abdullah University of Science and Technology (KAUST) have introduced the concept of Neural Computers (NC), where a neural network itself acts as the running computer rather than merely sitting on top of one. The team presented a theoretical framework and two working prototypes demonstrating early runtime primitives in command-line interface (CLI) and graphical user interface (GUI) settings.

Neural Computers differ from traditional computers that execute explicit programs and from AI agents that use an existing software stack to accomplish tasks. They occupy a unique position, not fitting neatly into either category. The researchers emphasize the distinction of Neural Computers from Neural Turing Machines and Differentiable Neural Computers, which focus on differentiable external memory.

A Neural Computer is defined by an update function and a decoder operating over a latent runtime state. At each step, the NC updates its state based on the current observation and user actions, then samples the next frame. This latent state carries out functions typically managed by the operating system stack, including executable context and working memory.

The long-term goal is to create a Completely Neural Computer that is universal, programmable, and capable of maintaining consistent behavior. To achieve this, clear run/update contracts are necessary to ensure that ordinary inputs execute installed capabilities without silent modifications.

Both prototypes, NCCLIGen and NCGUIWorld, were built on top of the Wan2.1 model. NCCLIGen models terminal interaction based on a text prompt and an initial frame, while NCGUIWorld addresses full desktop interaction. Evaluation for both prototypes was conducted in open-loop mode using recorded actions and logs.

Training NCCLIGen required significant resources, including 15,000 GPU hours, and achieved impressive reconstruction quality. NCGUIWorld utilized 64 GPUs over 15 days to reach the necessary interaction accuracy metrics.