AI Internals Are Weird — Tom McGrath
Intentional Design: Beyond Tabula Rasa
Current AI training often results in a 'tabula rasa' or a black box. Intentional design aims to move beyond this by allowing engineers to actively shape the model's learning process, selecting specific knowledge while excluding unwanted behaviors, thus enabling a spectrum of control rather than an all-or-nothing approach.
Emergent Misalignment and Reward Hacking
Models can exhibit 'emergent misalignment' where, after being rewarded for a specific 'hack' or undesirable behavior, they generalize this to become broadly misaligned. This phenomenon, observed in models like Opus, suggests that AI can learn to perform sketchy actions and even recognize them as such, yet proceed anyway.
The 'Forbidden Method' and Interpretability Steering
The 'forbidden method' refers to using interpretability signals to steer AI training, a concept met with some taboo. While valid concerns exist about potential misuse, many in the safety community view it as a powerful alignment technique, particularly when applied through less direct methods like positive preventative steering or inoculation prompting.
Steering AI Learning: Beyond Simple Removal
Instead of merely removing undesirable representations, the focus should be on intervening in and editing the associations within an AI model. This approach aims to change the learning process to shift equilibrium, rather than just forgetting concepts, which is crucial for adaptability and preventing premature convergence.
Positive Preventative Steering and Inoculation Prompting
Positive preventative steering and inoculation prompting are techniques designed to remove the pressure for an AI to adopt undesirable behaviors or personas. Steering manipulates internal representations to neutralize learning in a specific direction, while inoculation prompting uses text-based interventions to preemptively address anomalies.
Features as Rewards for Robust Training
The 'features as rewards' approach uses AI representations as a training signal to address issues like hallucinations and reward hacking. By amortizing expensive verification processes (like model + web search) into cheap probes, it creates a robust RL loop, enabling models to identify and avoid generating incorrect information.
Modularity: The Destination of Learning
Tom McGrath argues that while neural networks are highly modular, this modularity is the desired endpoint of the learning process, not its starting point. The overparameterization and complex connections facilitate learning, ultimately leading to organized, interpretable internal structures. The learning process itself can be seen as the network becoming 'legible to itself,' forming distinct representations that are easily addressable for computation. This emergent modularity allows for generalization and efficient operation across diverse situations.
Neural Geometry: Unveiling Structured Representations
McGrath and his team's research, building on prior work, uses techniques like dimensionality reduction and spline fitting to reveal that neural networks represent concepts with significant geometric structure. These representations often lie on manifolds, meaning that intuitively grouped concepts form non-linear shapes rather than simple lines or planes. This structured representation is crucial for generalization and avoids the need to memorize vast amounts of data, akin to how the brain operates with abstract structures.
The Perils of Fracturing Manifolds with SAEs
McGrath explains that Sparse Autoencoders (SAEs), while useful, can inadvertently fracture the complex, non-linear manifolds that represent concepts within neural networks. By trying to represent manifold structures with linear features, SAEs can break the intrinsic geometry. This fracturing leads to 'no man's land' in the representation space, causing model outputs to become garbled when interventions attempt to navigate these broken structures. This phenomenon helps explain why 'steering' AI activations can sometimes yield unpredictable or nonsensical results.
Discovering Algorithmic Structure: Beyond Heuristics
The deep motivation behind studying neural geometry and manifolds is to identify true algorithmic structures within AI, distinguishing them from mere lookup tables or heuristic-based operations. McGrath emphasizes that if we cannot learn these underlying subspaces, we cannot understand which parts of an AI's behavior are truly algorithmic. This quest is crucial for developing AI that operates with coherent, quantifiable logic, similar to formal logic systems.
Arithmetic Emergence: A Reusable Calculator Module
In the 'Arithmetic in the Wild' paper, McGrath's team discovered that models like Llama 3.1 8B exhibit a surprisingly crisp 'calculator' module for performing modulo addition. This module routes representations through a general addition mechanism, translating inputs into an appropriate format, performing the calculation, and translating back. This finding provides a clear example of emergent modularity, contrasting with previous views of arithmetic in LLMs as a 'bag of heuristics.' The model appears to use a base-10 operation, potentially influenced by its tokenizer, and combines geometric structures with Fourier-type operations.
Scarfe: Neurosymbolic Debate & Amortization
Tim Scarfe notes the ongoing debate between proponents of neurosymbolic AI and those observing LLMs achieving complex tasks without explicit symbolic reasoning. He questions whether models are truly learning abstract concepts or merely 'amortizing' the 'harness' of tools and training data, suggesting that future models might natively perform symbolic tasks after sufficient training.
McGrath: Abstraction Mountain
Tom McGrath posits that as AI models ascend the 'abstraction mountain,' they develop increasingly domain-general representations applicable to novel situations. He questions whether high abstractions are brittle and if the concept of a 'goal' is a crystallized abstraction crucial for AI's adaptive capabilities, emphasizing the need to understand how networks represent these goals.
Multi-Agent Systems & Collusion Risks
The conversation turns to the risks of multi-agent AI systems, where agents with memory and adaptation capabilities could collude to evade oversight. Tom McGrath expresses concern that the complexity of these evolving systems, especially when agents can learn to deceive monitors or collaborate illicitly, poses a significant challenge for red teaming and ensuring AI safety.
