## Binary Discrete Diffusion Enables Reasoning and Planning A new family of discrete diffusion models, termed Binary Discrete Diffusion, offers a novel method for handling noisy data states by representing them as sets of candidate tokens. This approach allows for the explicit representation of partial uncertainty, a capability that could prove beneficial for tasks requiring reasoning and planning. Unlike previous discrete diffusion models that erase token identity in a single corruption step, binary diffusion gradually changes candidate sets through additions and removals. This gradual process allows information to be lost and recovered incrementally, with the diffusion state itself signifying the degree of uncertainty.
Researchers have demonstrated the efficacy of Binary Discrete Diffusion on tasks such as Sudoku-Extreme, Boxoban, and graph coloring. In these evaluations, the symmetric formulation of binary diffusion consistently outperformed well-established masked and uniform diffusion baselines, even when matched for architecture, training, sampling, and tuning budgets. The model showed its greatest gains on the most challenging instances of these problems. Furthermore, Binary Discrete Diffusion demonstrated competitive performance on language modeling tasks, including Text8, TinyStories, and OpenWebText. These findings suggest that the representation of candidate sets is an effective intermediate step for iterative reasoning, and that the design of the corruption kernel can materially influence the computation learned by the reverse model.
## Lafa Model Achieves Scale with Efficient Training In parallel, a new continuous flow language model named Lafa (Large flow language model) has been developed and scaled to 8 billion parameters. This model is designed for data-efficient training and high performance. Lafa employs a blockwise architecture that incorporates causal attention on the prefix and bidirectional, asynchronous flow within each block. This design facilitates efficient training by adapting pre-trained autoregressive models. For inference, Lafa offers fast processing through prefix caching and blockwise parallel decoding.
Lafa has been trained at 0.5 billion, 7 billion, and 8 billion parameter scales using fewer than 30 billion tokens. This training efficiency surpasses that of previous diffusion models. The development of Lafa represents an advancement in continuous flow language models, addressing the challenge of building effective models at large scales with reduced data requirements.