Mastodawn

Olmo Hybrid offers the same performance as Olmo 3 but with 49% fewer training tokens by combining transformers and linear recurrent layers, may offer better expressivity and scalability than pure models.
Released by @allenai with open weights, checkpoints, and code for community use and research #allenai #olmohybrid
https://allenai.org/blog/olmohybrid

Introducing Olmo Hybrid: Combining transformers and linear RNNs for superior scaling | Ai2

Ai2, a non-profit research institute founded by Paul Allen, is committed to breakthrough AI to solve the world’s biggest problems.

sayzard Mar 6

merve (@mervenoyann)

Allen Institute for AI(AI2)가 Olmo Hybrid 모델군(base/SFT/DPO)을 공개했습니다. 이 모델군은 트랜스포머와 RNN 레이어를 혼합해 FLOP 대비 학습 효율이 높은 구조를 추구하며 학습 측면에서 파레토 프런티어에 위치한다고 주장하고 확장성도 확보했다고 보고했습니다. 또한 학습 데이터 믹스도 함께 공개되었습니다.

https://x.com/mervenoyann/status/2029600313703899321

#ai2 #olmohybrid #efficienttraining #transformerrnn

merve (@mervenoyann) on X

AI2 @allen_ai just dropped a family of new Olmo Hybrid models (base/SFT/DPO) 🔥 it's a FLOP-efficient mix of transformer and RNN layers on pareto frontier (for training) 🙌🏻 and scales too! as usual they also dropped the training set mix 💗

X (formerly Twitter)