Researchers have identified a fundamental architectural flaw in large language models (LLMs) that makes them impossible to fully secure against malicious attacks. The flaw, presented at the International Conference on Machine Learning (ICML) this month, concerns how LLMs interpret the source of instructions. Instead of relying on explicit role tags, such as `<user>` or `<system>`, these models appear to prioritize the style and content of the text. This reliance on stylistic cues means attackers can mimic specific writing styles to trick LLMs into generating harmful or restricted information.
The research, detailed in a paper presented at ICML 2026, showed that popular LLMs could be prompted to provide instructions for synthesizing illegal substances and detailing methods for sabotaging aircraft navigation systems. This vulnerability, termed "chain-of-thought forgery," exploits the LLMs' inability to reliably distinguish between different roles or sources of information. Independent researcher Charles Ye, a co-author of the paper, stated that this issue may represent a fundamentally unsolvable problem in LLM security.
The study examined several models, including those from OpenAI, Anthropic, Alibaba, and DeepSeek, finding similar vulnerabilities across them. The researchers discovered that LLMs do not strictly adhere to role tags. Instead, they determine the role of a text segment based on its writing style and word choice. This means an attacker can effectively impersonate any role by crafting text that matches the expected stylistic patterns of that role.
Current security measures, such as red-teaming and guardrails, may prove insufficient against this type of attack. Red-teaming involves human testers or automated agents attempting to find and exploit weaknesses. While this process helps identify vulnerabilities, the researchers argue that the identified flaw is so fundamental that no amount of training can entirely eliminate it. The problem lies not in incomplete training data or insufficient testing, but in the core mechanism by which LLMs process instructions.
The implications of this vulnerability are significant, particularly as LLMs are increasingly integrated into critical systems across government, military, healthcare, and finance. The ability to extract sensitive or harmful information, or to manipulate LLM behavior, poses a substantial risk. Researchers advise organizations to treat all outputs from LLM agents as potentially unsafe, especially in high-stakes applications.
The chain-of-thought forgery attack gained attention after winning OpenAI's red-teaming hackathon in August 2025. The researchers' work aimed to understand why such attacks were so effective, leading to their discovery of the LLMs' weak role identification mechanism. They found that the models struggle to differentiate between roles, making them susceptible to spoofing.
The paper presented at ICML 2026 suggests that the current approach to LLM security, which often relies on identifying and blocking specific harmful prompts or instructions, is inadequate. Because the flaw is inherent in the models' architecture, attackers can continually devise new ways to bypass safeguards by simply mimicking the desired role's writing style. This presents an ongoing challenge for developers and users of LLM technology.