Chain of Thought Extraction: What Security Teams Should Ask Now
Researchers from the University of Tübingen, the Max Planck Institute, MATS Research, and Snyk say they found a practical way to surface hidden reasoning from frontier models. That makes chain of thought extraction more than a research curiosity. In my world, it changes the threat model for any team shipping AI features through APIs, because the risk is no longer limited to final outputs. According to WIRED's coverage of the paper, the same path could also expose sensitive material such as passwords or API keys.
What did the new chain of thought extraction paper actually show?
The short version: the researchers demonstrated that hidden reasoning traces can sometimes be recovered from major model families, even when providers intended those traces to stay concealed. The paper is available at OpenReview, and the team tested models accessed through APIs from OpenAI, Anthropic, and Google.
What got my attention was not the headline about model copying. It was the operational detail. The attack path appears to rely on encrypted reasoning being delivered to the client side for performance reasons, then probed through related smaller models. If that design pattern holds broadly, then AI model reasoning traces become part of the exposed surface, not just an internal implementation detail.
The researchers also reported that the method could recover sensitive information in some cases, including credentials-like material. Snyk's participation matters here because this is not just a lab curiosity; it touches application security, logging, and API boundary design.
All major frontier model providers we tested share this vulnerability, Alexander Panfilov said, according to Wired.
Why does encrypted hidden reasoning create a real attack surface?
I have seen teams assume that if a model's chain of thought is not rendered in the UI, it is effectively private. That assumption is too generous. If reasoning tokens are transmitted, cached, logged, or passed through a client runtime, they become inspectable in the same way many other sensitive payloads do.
The distinction from ordinary LLM prompt leakage is important. Prompt leakage usually means a system prompt, retrieved context, or tool result slips into the output. This case is different: the hidden reasoning itself may be recoverable because of how the vendor structures inference and model families. That means the leakage path can exist even when your own prompt hygiene is decent.
In one client engagement earlier this year, we found that a model gateway was storing debug payloads for 14 days by default. No one thought those payloads were high-risk because the user never saw them. That is exactly the kind of gap this paper makes harder to ignore.
If you are already treating production model behavior as an ongoing reliability problem, this is where an AI DevOps workflow automation service starts to fit naturally: not as marketing gloss, but as a way to standardise logging controls, environment separation, and model-side incident response.
How can smaller models reveal the larger model's reasoning?
The paper's Mini-Me idea is clever and uncomfortable at the same time. Many providers offer a large flagship model and one or more smaller siblings. The larger model is usually more aligned, more expensive, and less likely to reveal internals directly. The smaller version may share enough structure or training lineage that, when fed the encrypted trace, it reproduces or exposes the hidden reasoning more willingly.
That tracks with what many of us see in practice. Smaller models often fail in more literal ways. They follow weird instructions too eagerly, produce less stable refusals, and expose intermediate behavior that the flagship model suppresses. Lower cost and lower latency come with less safety margin.
The paper points to Moonshot AI's Kimi K3 as showing notable similarity to hidden reasoning from Claude Opus 4.8 and GPT 5.6 Sol for some prompts. But it also matters that DeepSeek and Thinking Machines' Inkling did not show the same pattern. That trade-off matters because similarity is evidence of risk, not proof of causation.
Does this prove reasoning distillation from US frontier models?
No, and this is where a lot of coverage will get sloppy. The researchers explicitly say they cannot causally establish distillation. Similar outputs and similar reasoning patterns can support a hypothesis, but they do not close the case.
That said, the work changes the burden of proof in one important way. Before this, some vendors could argue that hidden reasoning was sealed off tightly enough to make large-scale reasoning distillation impractical. This paper weakens that comfort. If hidden reasoning can be extracted at scale, then model distillation becomes a more concrete operational and commercial concern.
The political backdrop is already noisy. Reuters reported that OpenAI raised concerns to US lawmakers about DeepSeek in February 2026. Reuters also reported that Anthropic made a similar case about Alibaba in June 2026. This paper does not settle those disputes, but it adds a plausible technical path that security teams now have to consider.
What should AI security teams check first in production?
I would start with four questions, in this order:
- Where are reasoning traces generated and transmitted? Ask the vendor whether hidden reasoning ever reaches the client, gateway, browser, mobile app, or edge runtime.
- What gets logged by default? Check app logs, APM traces, model gateway payload capture, support tooling, and analytics events.
- Which smaller related models exist in the same stack? If your provider exposes mini variants, test whether they can be used to reveal hidden reasoning or embedded secrets.
- Can secrets appear anywhere in prompts or tool outputs? API keys, passwords, bearer tokens, signed URLs, and raw customer data should never be recoverable through AI model reasoning traces.
I would also add a red-team case specifically for reasoning extraction. Most teams test jailbreaks and toxic outputs. Far fewer test cross-model probing or encrypted-trace replay.
For baseline control language, the NIST AI Risk Management Framework is useful because it pushes teams to map, measure, and manage model-specific risks rather than treating AI as ordinary SaaS.
What should model vendors and platform teams review right now?
First, review whether hidden reasoning needs to be sent anywhere outside the trusted execution boundary at all. If it does, reduce lifetime, scope, and observability. Encrypting a payload is not enough if related components can still decode or meaningfully interpret it.
Second, review sibling-model architecture. The convenience of offering small, medium, and large variants may introduce a family-level weakness if one member can help extract another's internals. I would want adversarial testing across the whole family, not one model at a time.
Third, treat this as an application security issue as much as a model issue. Snyk's research perspective is relevant because the exploitation path touches APIs, clients, libraries, and developer defaults. In practice, the breach rarely happens inside a pristine whiteboard diagram. It happens in the glue code.
What should buyers ask vendors before the next rollout?
I would ask for direct answers to five things:
- Is hidden reasoning ever transmitted to the client or a customer-controlled environment?
- Are reasoning traces stored in logs, analytics systems, or support tooling?
- Have you tested smaller related models for cross-model extraction attacks?
- Can tenants disable reasoning visibility, retention, or debug capture?
- What is the incident process if LLM prompt leakage or reasoning leakage is detected?
If the vendor cannot answer clearly, I would mark that deployment as higher-risk, especially in enterprise software, cybersecurity, and AI infrastructure use cases where credentials and internal architecture details appear in context windows regularly.
My non-obvious takeaway is this: many teams focus on whether the model answers correctly, while the real operational question is what the system had to expose in order to answer that fast. Performance shortcuts, trace handling, and model-family design can quietly decide your risk profile months before the security review starts.
What do I expect to happen next?
I expect vendors to say two things at once over the next few months: first, that the issue is limited or fixed in specific implementations; second, that hidden reasoning should remain inaccessible for safety and IP reasons. Both may be partly true, but neither removes the need for production testing.
For buyers, chain of thought extraction now belongs on the pre-launch checklist beside prompt injection, data retention, and tool-call controls. The immediate watch item is not whether one headline claim about distillation sticks. It is whether model providers can show, with technical specifics, that hidden reasoning and secrets are not crossing boundaries they do not control.
Martin Kuvandzhiev
CEO and Founder of Encorp.io with expertise in AI and business transformation