🚀 "Oh wow, look at arXiv with its tiny-brained 'TinyLoRA' trying to solve world problems with a whopping 13 parameters! 😂 Meanwhile, the rest of us are learning to reason with at least 14 parameters and a cup of coffee. ☕
#CuttingEdgeTech #ArxivComedy"
https://arxiv.org/abs/2602.04118 #CuttingEdgeTech #ArxivComedy #TinyLoRA #MachineLearning #Humor #HackerNews #ngated
Learning to Reason in 13 Parameters
Recent research has shown that language models can learn to \textit{reason}, often via reinforcement learning. Some work even trains low-rank parameterizations for reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91\% accuracy on GSM8K with only 13 trained parameters in bf16 (26 total bytes). We find this trend holds in general: we are able to recover 90\% of performance improvements while training $1000x$ fewer parameters across a suite of more difficult learning-to-reason benchmarks such as AIME, AMC, and MATH500. Notably, we are only able to achieve such strong performance with RL: models trained using SFT require $100-1000x$ larger updates to reach the same performance.
Learning to Reason in 13 Parameters
Recent research has shown that language models can learn to \textit{reason}, often via reinforcement learning. Some work even trains low-rank parameterizations for reasoning, but conventional LoRA cannot scale below the model dimension. We question whether even rank=1 LoRA is necessary for learning to reason and propose TinyLoRA, a method for scaling low-rank adapters to sizes as small as one parameter. Within our new parameterization, we are able to train the 8B parameter size of Qwen2.5 to 91\% accuracy on GSM8K with only 13 trained parameters in bf16 (26 total bytes). We find this trend holds in general: we are able to recover 90\% of performance improvements while training $1000x$ fewer parameters across a suite of more difficult learning-to-reason benchmarks such as AIME, AMC, and MATH500. Notably, we are only able to achieve such strong performance with RL: models trained using SFT require $100-1000x$ larger updates to reach the same performance.
Avi Chawla (@_avichawla)
TinyLoRA 논문(arXiv: 2602.04118) 소개 트윗으로, 초소형 파라미터 조정만으로 대형 모델의 추론 성능을 개선하는 방법을 시각적으로 설명한다. 새로운 모델 출시보다는 연구 결과 공유에 가깝지만, 경량 파인튜닝과 효율적 적응 기술에 관심 있는 개발자에게 유용하다.
https://x.com/_avichawla/status/2036005894425907306
#tinylora #arxiv #finetuning #llm #research
Avi Chawla (@_avichawla) on X
paper: https://t.co/EKXJyHx4Ah
TinyLoRA visually explained:
Akshay (@akshay_pachaar)
TinyLoRA라는 접근을 소개하며 LoRA를 단 1개 파라미터로 축소했다고 알림. LoRA의 극단적 경량화·파라미터 효율성에 대한 새로운 연구·기술적 시도로, 저자원 환경에서의 모델 적응·배포에 영향이 있을 수 있음.
https://x.com/akshay_pachaar/status/2021897353184325966
#tinylora #lora #parameterefficient #modelcompression
Akshay 🚀 (@akshay_pachaar) on X
TinyLoRA: LoRA scaled down to 1 parameter:
IoT für "Pfeifen" - The Things Network
Das spielen einer Orgel ist eine Wissenschaft für sich, gleichwohl auch alles was dazu gehört. Wußtet Ihr, dass eine Temperaturänderung von 10 Grad eine Frequenzänderung von 7Hz bei Labialpfeifen ausmacht ? Oder dass Ihre Wohlfühltemperatur …
N-gated Hacker News
Hacker News
sayzard
David Lamorski