Large language models demonstrate a marked decline in accurately executing multi-step procedures as complexity increases. A new diagnostic study reveals that model accuracy drops from 61% on five-step tasks to 20% on ninety-five-step tasks.
A study published on arXiv, titled "When LLMs Stop Following Steps: A Diagnostic Study of Procedural Execution in Language Models," investigated the ability of large language models (LLMs) to faithfully execute step-by-step instructions. Researchers developed a benchmark using arithmetic algorithms of increasing length and complexity. Across 14 different LLMs and 55 datasets, the models showed a significant degradation in performance as the number of steps in the procedure grew. For algorithms with five steps, the average first-answer accuracy was 61%. However, this figure fell to 20% for procedures comprising ninety-five steps.
The researchers noted that failures in execution were not solely due to incorrect final answers. Analysis of the generated outputs at each step revealed issues such as missing intermediate answers, premature completion of the task, or the models generating answers that contradicted their own prior calculations. These findings suggest that while LLMs may excel at generating coherent text or answering simple questions, their ability to meticulously follow a sequence of instructions, especially those involving intermediate variables and dependencies, is substantially limited. The study's design focused on procedural execution, a critical aspect for tasks ranging from mathematical problem-solving to complex command sequences. By using simple arithmetic operations, the study isolated the challenge to the model's capacity for sequential reasoning and memory of prior steps, rather than the inherent difficulty of the mathematical operations themselves.
The complexity was amplified by introducing "look-back dependencies," where a later step in the algorithm relies on the output of an earlier step. This tests the model's ability to retain and correctly utilize information generated in previous stages of the procedure. The paper highlights that the performance drop is not linear, indicating that certain thresholds of complexity or length might pose disproportionately greater challenges for current LLM architectures. This has implications for the deployment of LLMs in applications that require precise, multi-step execution, such as scientific simulations, coding assistance, or automated workflow management.
Previous research has also pointed to limitations in LLM instruction following. A study published in October 2025, "When Models Can't Follow: Testing Instruction Adherence Across 256 LLMs," evaluated 256 models on various instruction-following tasks, including logical sequencing and multi-step execution, revealing consistent failure modes across different model families. Similarly, research on "The Instruction Gap" in December 2025 identified that while LLMs perform well on general tasks, they struggle with the precise instruction adherence necessary for enterprise deployment. These studies collectively underscore a persistent challenge in LLM development: bridging the gap between broad language understanding and the rigorous, sequential execution of specific directives.
The diagnostic study's authors propose that their benchmark can help researchers identify specific weaknesses in LLMs' procedural reasoning capabilities. By understanding where and why models fail, developers can work towards creating more reliable systems. The research provides a quantitative measure of this failure, moving beyond anecdotal evidence of LLMs going "off-script." The implications extend to areas where LLMs are being considered for tasks requiring strict adherence to protocols, such as in scientific research or complex industrial processes. The observed decline in accuracy with increased procedural length suggests that current LLM architectures may require fundamental improvements in their ability to manage long-term dependencies and maintain computational state across extended sequences of operations.
The study's findings are particularly relevant as LLMs are increasingly integrated into complex workflows. The ability to accurately follow a series of instructions is paramount for tasks that demand precision and predictability. The research team plans to expand their diagnostic benchmark to include more varied types of procedural tasks and to investigate potential architectural changes or training methodologies that could mitigate these observed limitations. The next phase of research will likely focus on developing techniques to enhance LLMs' capacity for sustained, step-by-step execution, ensuring their reliability in critical applications.