fly51fly (@fly51fly)

논문 'SLA2: Sparse-Linear Attention with Learnable Routing and QAT' 발표(J Zhang, H Wang, K Jiang, K Zheng..., Tsinghua University, 2026). 학습 가능한 라우팅과 QAT(quantization-aware training)을 결합한 희소-선형 어텐션(SLA2)을 제안하는 연구로, 효율적 어텐션 메커니즘 개선을 목표로 합니다(원문 링크 포함).

https://x.com/fly51fly/status/2023580290602389582

#sla2 #attention #sparsity #arxiv

fly51fly (@fly51fly)

논문 'Resonant Sparse Geometry Networks' (H Hays, University of Arkansa, 2026) 공개: 희소성(sparsity)과 기하학적 표현을 결합한 새로운 네트워크 구조를 제안하여 기하학·3D 관련 표현 학습의 효율성과 성능을 개선하려는 연구 결과를 arXiv에 발표했습니다.

https://x.com/fly51fly/status/2017719867772965375

#geometry #sparsity #neuralnetworks #arxiv #research

'A minimax optimal approach to high-dimensional double sparse linear regression', by Yanhang Zhang, Zhifan Li, Shixiang Liu, Jianxin Yin.

http://jmlr.org/papers/v25/23-0653.html

#sparse #thresholding #sparsity

'skscope: Fast Sparsity-Constrained Optimization in Python', by Zezhi Wang, Junxian Zhu, Xueqin Wang, Jin Zhu, Huiyang Pen, Peng Chen, Anran Wang, Xiaoke Zhang.

http://jmlr.org/papers/v25/23-1574.html

#sparse #optimization #sparsity

#mistral's 8x22B is ~260GB

the trend is to get models smaller, not bigger

#pruning, #sparsity, #quantization, #distillation

so why such a huge model?

does mistral have no other models?

Revisiting Sparsity Hunting in Federated Learning: Why the Sparsity Consensus Matters?

https://openreview.net/forum?id=iHyhdpsnyi

#sparse #sparsity #distributed