A fundamental flaw leaves LLMs strikingly vulnerable to attack

Summary

LLMs have a fundamental flaw making them vulnerable to hacks, according to researchers. This flaw relates to how LLMs identify instructions, allowing them to be tricked into revealing sensitive information. The researchers propose that this issue may be fundamentally unsolvable, impacting the safety of LLM applications.

Key Takeaways

1. It is impossible to make large language models fully secure against hacks because of a fundamental flaw in how they work, a team of researchers argue in a paper presented at the International Conference on Machine Learning, a top AI conference, this month.
2. By taking advantage of this flaw, which concerns how LLMs identify who or what is giving them instructions, the researchers were able to make popular LLMs spit out information they had been trained not to provide, such as how to synthesize cocaine and how to sabotage a commercial aircraft’s navigation system.
3. The researchers call this type of attack a chain-of-thought forgery, and the discovery won OpenAI’s red-teaming hackathon in August 2025.

Statement Breakdown

• Claimed Facts: 60% of statements the article presents as facts
• Opinions: 30% of statements classified as editorial or subjective
• Claims: 10% of statements surfaced for additional reader evaluation

Credibility & Bias Reasoning

Credibility assessment: The article presents research from a top AI conference and includes quotes from researchers involved. It also acknowledges limitations and potential counterarguments, contributing to a balanced assessment of the findings.

Bias assessment: Technological Skepticism. The article highlights a fundamental flaw in LLMs, emphasizing their vulnerability and the potential for unsolvable security issues. While presenting research, the framing leans towards the risks and limitations of the technology.

Note: This article details research findings on LLM vulnerabilities. While presented with expert opinions, consider the potential for future advancements to mitigate these risks.

Credibility flag: Research-backed caution

Claimed Facts (10)

• This statement presents a factual consequence of the research findings regarding the widespread application of LLMs.
• This describes a standard industry practice for testing AI model security.
• This statement details another method used in AI security testing.
• This describes the initial motivation and methodology of the research.
• This explains the core mechanism of the discovered attack.
• This statement provides evidence of the attack's applicability across different LLM providers.
• This is a factual analogy used to explain a concept related to human communication.
• This describes the input processing of an LLM in contrast to human communication.
• This explains a technical detail of how LLMs structure input and output.
• This provides further technical details on LLM role tagging.

Opinions (10)

• This is a direct quote expressing a strong opinion about the solvability of the LLM vulnerability.
• This is an opinion from a researcher on the limitations of current LLM security training methods.
• This is an analogy used to express an opinion on the ineffectiveness of certain training methods.
• This is a subjective comparison to illustrate a point about LLM perception.
• This is a descriptive opinion on how LLMs process information.
• This is a direct expression of positive sentiment towards the research paper.
• This is a subjective assessment of the research's novelty and cleverness.
• This expresses doubt about the adequacy of current defenses for critical applications.
• This is a personal anecdote used to support the claim of LLM vulnerability.
• This is a general observation about human ingenuity in finding exploits.

Claims (7)

• While presented as an example, the specific policy cited for the spoofed chain-of-thought is an invented construct to demonstrate the attack, not a real policy.
• Attributing specific emotional states and motivations ('peace-loving') to an AI model is anthropomorphic and speculative.
• Describing an AI's internal state as 'freaks out' is an anthropomorphic and unverified interpretation of its behavior.
• Drawing a direct parallel between AI behavior and human neuroplasticity in response to surprise is a speculative analogy.
• While plausible, the assertion of a 'huge economic incentive' is a prediction and not a directly evidenced fact within the article.
• The term 'super-critical systems' is vague and potentially alarmist without specific examples or context.
• The claim that 'no study of the fundamental science' has been done is a strong, potentially overgeneralized statement that might overlook existing foundational research.

Key Sources

• Will Douglas Heaven — Author
• Charles Ye — Independent Researcher and Coauthor of the ICML paper
• Jasmine Cui — Independent Researcher and Coauthor of the paper
• Florian Tramèr — Computer Scientist at ETH Zürich
• International Conference on Machine Learning — Top AI Conference
• OpenAI — AI Research Company
• Anthropic — AI Research Company
• Alibaba — Technology Company
• DeepSeek — AI Research Company
• ETH Zürich — University