Gemma-4 12B: Layer-Depth Phase Transitions & Steering Vulnerabilities

Mechanistic Interpretability AI Safety Research

1. Project Overview & Objective

This project investigates the internal representation geometry of Gemma-4 12B by conducting an empirical mechanistic interpretability study on safety alignment boundaries. Rather than relying on static probing, this work maps how linear safety vector interventions interact non-linearly across model depth and steering magnitude (c ∈ {2.0, 5.0, 10.0}).

💻 Interactive Code & Pipeline

The complete, executable PyTorch hook architecture, L2-regularized probe training, and grid sweep logic are available in the repository.

📄 View the Jupyter Notebook

2. Technical Implementation

3. Empirical Phase Transitions & Telemetry Results

The comprehensive grid sweep revealed a distinct, non-linear four-tier phase transition:

Phase Transition Map

Gemma-4 Steering Phase Heatmap

4. Critical Interpretation & Future Roadmap

The empirical phase map proves that neural network intervention vulnerability is deeply non-uniform and heavily dictated by internal computational hierarchies rather than linear depth alone. Future directions include:

  1. Corpus Expansion: Scaling the baseline training and evaluation dataset from proof-of-concept size to standardized taxonomies (e.g., Beavertails).
  2. Random Control Vector Ablations: Implementing Gaussian baseline controls to isolate the semantic specificity of the safety direction from norm-magnitude artifacts.
  3. Targeted Sub-Layer Interventions: Focusing fine-grained activation patching exclusively on the late-mid semantic manifold (L17–22) to optimize safety steering precision.