Canonical-basis realignment for Transformer LLMs: every hidden axis becomes independently measurable and controllablehttps://github.com/todotge/canonical-basis
The code essentially gives you a way to rotate a Transformer's internal coordinate system into a canonical basis that aligns with its own weight matrices in a lossless way. By absorbing the normalization gains directly into the adjacent weights and using orthogonal matrices built from the singular vectors of the model, you can transform architectures like Qwen or Pythia without altering their outputs or perplexity scores. Applying this transform reveals the actual hidden geometric structures operating inside the network. Once the model is rotated into this new perspective, you can see its int…