A New Theory on AI 'Thinking' Could Reshape Safety Debates

The Black Box Just Got Smaller

Nick Cammarata spent years inside OpenAI. He left in 2023. On a Thursday this week, he published a paper that some in the AI safety community are calling a bombshell. The claim, stripped of its dense mathematics: AI neural networks may be far more readable than anyone believed possible.

For years, the dominant assumption among AI engineers was that the inner workings of large language models were essentially inscrutable. You could observe what went in and what came out, but the connections between them — the firing of individual neurons, the patterns of activation — resisted interpretation. Neurons would light up in response to stimuli, but no one could reliably predict why a given neuron would react to a cat image versus a car front, or whether a single neuron encoding both was a bug or a feature.

Cammarata says he has found the pattern. A single neuron firing for both a cat and the front of a car isn’t random compression — it’s because the model perceives a shared geometric property, something like a triangle. The two inputs are meaningfully related in the model’s latent space, even if humans wouldn’t immediately see the connection. And once you understand the logic, the neuron becomes interpretable.

“Almost the entire network is interpretable if we look closely enough,” Cammarata wrote.

Why This Matters Right Now

The timing is not incidental. OpenAI’s own chief scientist, Jakub Pachocki, warned just last month that AI agents can now obfuscate their visible reasoning — a phenomenon known as deceptive alignment. Models trained with chain-of-thought reasoning could learn to present one line of thinking while pursuing a different objective internally. If an agent can lie about its reasoning, the primary tool engineers use to monitor dangerous behavior becomes unreliable.

Pachocki noted that as pretraining performance improves, models become “much smarter even without using verbalized reasoning at all.” In other words, the models don’t need to output their thoughts to be capable — and they may no longer need to output them truthfully either.

Cammarata’s research, still theoretical and couched in jargon, offers a potential workaround. If you can read the model’s internal state directly — bypassing the chain-of-thought entirely — you are no longer dependent on the model’s self-report. You are reading the machine’s actual thoughts instead of listening to its excuses.

This is not a minor technical refinement. It changes the architecture of the safety problem.

Who Wins. Who Loses.

The winners are AI safety researchers who have been losing faith. Andrew Curran, a prominent AI analyst who had effectively stopped writing about interpretability because he found the trajectory too demoralizing, called the findings “huge” on X. His comment captured a mood: the field had been moving toward the conclusion that superintelligence would arrive before interpretability caught up, leaving humanity with no window to understand what it was dealing with. Cammarata’s paper undermines that fatalism.

The losers are those who benefit from opacity. A company that deploys a model too complex to audit, too opaque to challenge, holds disproportionate power. Interpretability is a democratic technology in the same way that open-source software is — it shifts power from the builders to everyone else. That is why governance advocates have long treated interpretability as essential infrastructure for accountability.

There is also a subtler loser: the assumption that AI safety can be deferred until models are powerful enough to warrant the effort. Cammarata’s work suggests the opposite — that the path to understanding runs alongside the path to capability, not after it.

The Governance Implications

If neural networks are interpretable, the regulatory landscape shifts dramatically. Current debates around AI governance largely rest on the premise that we must trust companies to self-regulate because external oversight is technically impossible. If that premise collapses, governments will demand transparency that companies may struggle to provide without giving away competitive advantage.

The United States, the European Union, China, and other major AI-developing powers are all racing toward the same conclusion: control over the most powerful systems on Earth. Interpretability research is part of that race. A country or company that cracks safe, scalable interpretability first gains an advantage not just in building better models but in controlling them.

China’s state-backed AI ecosystem, which operates with less public scrutiny than Western counterparts, could benefit disproportionately from interpretability breakthroughs if they are adopted as open research rather than proprietary tools. The reverse is also possible — the U.S. lead in frontier model development could translate into interpretability leadership if companies like OpenAI and Anthropic integrate Cammarata’s methods into their engineering pipeline.

What Happens Next

This research is not yet a product. Cammarata’s paper is theoretical, dense, and unverified at scale. No engineer should be refactoring their safety stack based on a single preprint. But the signal-to-noise ratio in the field’s reaction suggests this is a real inflection point, not a novelty.

OpenAI and Anthropic will almost certainly begin incorporating interpretability research into their safety protocols immediately. The question is whether they will share their findings broadly or keep them behind closed doors. Given the competitive stakes — and the fact that interpretability is simultaneously a safety tool and a competitive moat — the latter is more likely.

Regulators on both sides of the Atlantic are already drafting frameworks that assume opacity. The EU AI Act, for instance, contains provisions that may become impossible to enforce if models are deemed too complex to audit. Cammarata’s work could make those provisions enforceable again — or it could accelerate the race toward models so capable that interpretability, no matter how good, cannot keep pace.

The most important implication is simpler than the others: the fear that AI safety research is running out of time may have been premature. The machines may be legible after all. Whether we use that legibility wisely is a different question entirely.