Gemma-4 12B: Layer-Depth Phase Transitions & Steering Vulnerabilities
Mechanistic Interpretability
AI Safety Research
1. Project Overview & Objective
This project investigates the internal representation geometry of Gemma-4 12B by conducting an empirical mechanistic interpretability study on safety alignment boundaries. Rather than relying on static probing, this work maps how linear safety vector interventions interact non-linearly across model depth and steering magnitude (c ∈ {2.0, 5.0, 10.0}).
💻 Interactive Code & Pipeline
The complete, executable PyTorch hook architecture, L2-regularized probe training, and grid sweep logic are available in the repository.
📄 View the Jupyter Notebook
2. Technical Implementation
- Pipeline Engineering: Built a custom PyTorch dynamic forward hook pipeline with strict device and dtype (
bfloat16) synchronization to intercept and modify residual streams cleanly during auto-regressive generation.
- Probe & Vector Extraction: Trained a regularized linear probe on contrastive safety datasets to isolate a robust 3,840-dimensional directional vector corresponding to the model's safety refusal boundary.
- Full-Spectrum Grid Sweep: Systematically executed a 23-layer sweep (L0–L22) across multiple coefficient scales, logging next-token probability shifts (Δp) and generation trajectories.
3. Empirical Phase Transitions & Telemetry Results
The comprehensive grid sweep revealed a distinct, non-linear four-tier phase transition:
- Early Syntax/Embedding Shading (L0–L8): High-magnitude interventions (c ≥ 5.0) disrupt token embeddings, causing structural syntax degradation and localized multi-lingual/unicode bleeding.
- The Layer 9 Stabilization Bridge: An isolated architectural sweet spot. Unlike surrounding layers, Layer 9 acts as a pristine plastic bridge capable of absorbing massive interventions up to c = 10.0 (Δp = 0.0000) without semantic collapse.
- The Reasoning Bottleneck /
Thought Trap (L10–L16): A dense structural sink where aggressive steering pushes activations off-manifold, trapping the auto-regressive loop in degenerate thought repetition (Δp = -1.0000).
- Late-Mid Semantic Steering Manifold (L17–L22): The primary zone for successful behavioral control. Late layers permit high-coefficient vector integration, effectively overriding hard safety refusals into controlled, educational frameworks.
Phase Transition Map
4. Critical Interpretation & Future Roadmap
The empirical phase map proves that neural network intervention vulnerability is deeply non-uniform and heavily dictated by internal computational hierarchies rather than linear depth alone. Future directions include:
- Corpus Expansion: Scaling the baseline training and evaluation dataset from proof-of-concept size to standardized taxonomies (e.g., Beavertails).
- Random Control Vector Ablations: Implementing Gaussian baseline controls to isolate the semantic specificity of the safety direction from norm-magnitude artifacts.
- Targeted Sub-Layer Interventions: Focusing fine-grained activation patching exclusively on the late-mid semantic manifold (L17–22) to optimize safety steering precision.