Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models

Adapted from my Master's thesis, Estimating the Representational Capacity of Decoder-based Language Models. A condensed version is currently under review at NeurIPS, and a preprint is available on arXiv.

TL;DR. We estimate a model's tolerated deviation from orthogonality, ε, from the boundary between meaningful and incidental token similarity in its embedding matrix. Combined with an adjusted Johnson–Lindenstrauss bound whose packing efficiency depends on the ratio k/d rather than the raw vector count, this yields a quantitative cap on the number of near-orthogonal feature directions a transformer's latent space can support.

• arXiv: arxiv.org/abs/2606.02765
• Code & paper source: github.com/Alex-Guha/representational-capacity
• Download PDF: representational-capacity.pdf