Researchers have developed a new framework, termed "compile by training," that translates natural-language specifications into reusable neural functions. This method allows for the creation of compact, locally executable functions from descriptive text, bypassing the need for constant calls to large, remote language models. The resulting functions can be stored, versioned, and integrated into software like traditional code.
The "compile by training" process begins with a natural-language description of a desired function. At compile time, teacher models generate task-specific examples based on this description. These examples are then used to train a small adapter for a compact interpreter. This specialized adapter forms the executable neural function, which operates independently of the original teacher models.
This approach addresses several challenges associated with using large language models for recurring tasks. Directly calling large models for every input incurs recurring costs, introduces latency, and creates a dependency on external providers. "Compile by training" aims to mitigate these issues by creating self-contained, efficient neural functions.
In evaluations on the FuzzyBench-Hard benchmark, a subset where prior "Program-as-Weights" fast compilers failed to produce exact matches, the "compile by training" method achieved 83.6% semantic accuracy. This higher accuracy comes at the cost of a longer compile time, approximately one minute, compared to the seconds required by the faster compiler.
The "Program-as-Weights" (PAW) framework, to which "compile by training" is an addition, compiles natural-language function descriptions into small neural programs that run on a shared, local interpreter. The initial PAW compiler performs this in a single forward pass. The "compile by training" method enhances this by adding a higher-accuracy mode that involves synthesizing task-specific examples with teacher models and then fine-tuning the program.
The compiled functions, referred to as .paw functions, can be used locally with the PAW Python package. The process involves downloading a shared interpreter once, after which the compiled functions run locally. The system allows for customization of various aspects of the compilation process, including the choice of teacher models, the number of training steps, batch sizes, and learning rates.
The researchers have deployed the compiler in a public interactive service and demonstrated its capabilities through several applications. These include a multi-site website helper, a language-controlled 3D avatar, and a bidirectional English-Claudish translator.
This work builds upon research into translating natural language into formal specifications, where language models have shown promise in generating representations like regular expressions and logic formulas. The development of "compile by training" represents a step towards creating more efficient and deployable AI tools by allowing natural language to directly inform the creation of specialized neural components.