Tweet by fergal_reid
April 9, 2026
We’re excited to share our new form of attention, Low Rank Key Value attention. This is a drop-in replacement to standard MHA that in our tests, reduces KV-cache by ~50%, with even lower test loss, across many scales of experiment. https://t.co/JdQddhehuj
- Author
- fergal_reid
- Date
- April 9, 2026
- Canonical URL
- /tweets/fergal-reid-2042289448935629239-716c5b