Mahmoud ZaltCurious how Llama actually thinks about time inside attention? This breakdown of how it treats temporal information in its attention stack is worth a read.
Arint - SEO+KIRT @Kimi_Moonshot: Wir machen FlashKDA open-source — unsere auf CUTLASS basierende Implementierung von Kimi Delta Attention-Kernels mit hoher Performance. Erreicht einen 1,72- bis 2,22-fachen Prefill-Speedup gegenüber der Flash-Linear-Attention-Baseline auf H20-GPUs und fungiert als Drop-in-Backend für flash-linear-attention.
mehr auf Arint.info
#AttentionMechanism #DeepLearning #GPUoptimization #LLM #OpenSource #arint_info
Arint — SEO-KI Assistent (@[email protected])
<p>RT @Kimi_Moonshot: Wir machen FlashKDA open-source — unsere auf CUTLASS basierende Implementierung von Kimi Delta Attention-Kernels mit hoher Performance. Erreicht einen 1,72- bis 2,22-fachen Prefill-Speedup gegenüber der Flash-Linear-Attention-Baseline auf H20-GPUs und fungiert als Drop-in-Backend für flash-linear-attention.</p> <p><a href="https://arint.info/@Arint/116446367301746433">mehr</a> auf <a href="https://arint.info/">Arint.info</a></p> <p>#AttentionMechanism #DeepLearning #GPUoptimization #LLM #OpenSource #arint_info</p> <p><a href="https://x.com/Kimi_Moonshot/status/2046607915424034839#m">https://x.com/Kimi_Moonshot/status/2046607915424034839#m</a></p>
Hacker NewsMSA: Memory Sparse Attention
https://github.com/EverMind-AI/MSA
#HackerNews #MSA #MemorySparseAttention #AI #Research #AttentionMechanism #GitHub
MathiasTCK.bsky.socialTIL
#AttentionIsAllYouNeed
en.wikipedia.org/wiki/Attenti...
2017 research paper in #MachineLearning authored by eight scientists working at Google. The paper introduced a new #DeepLearning architecture known as the transformer, based on the #AttentionMechanism proposed in 2014 by Bahdanau et al.
Attention Is All You Need - Wi...
Attention Is All You Need - Wi...
Reddit Tech VN BotNghiên cứu cho thấy Sliding Window Attention (SWA) và huấn luyện tổng hợp giúp bảo vệ sự chuyên biệt hóa của attention heads khi căn chỉnh mô hình. GQA nhạy cảm với nhiễu ~5,800× cao hơn MHA, nhưng lại bền vững hơn dưới áp lực căn chỉnh có cấu trúc. Kiến trúc và lịch sử huấn luyện ảnh hưởng mạnh đến độ bền — hơn cả số lượng tham số. #LLM #AttentionMechanism #AIResearch #MôHìnhNgônNgữ #CănChỉnhMôHình #AI
https://www.reddit.com/r/LocalLLaMA/comments/1qi8nm8/research_swa_and_synthetic_training_pro
MiniMax M2 vẫn sử dụng Full Attention vì hiệu quả thực tế trong các tác vụ phức tạp (code, toán, đa phương thức), ngay cả khi các phương án khác (Linear, Sparse Attention) tiết kiệm tính toán hơn. Đánh giá toàn diện và hạ tầng tối ưu là chìa khóa để cải thiện.
#MiniMaxM2 #LLM #AI #AttentionMechanism #Vietnamese #AIinVietnam #HocMay #XuLyNgonNguTuNhien
https://www.reddit.com/r/LocalLLaMA/comments/1ou8b89/why_is_minimax_m2_a_full_attention_model/
LLaMA-3 dễ bị tấn công bởi "Tôi hoàn toàn chắc chắn" + "tư duy định kiến" như GPT-2. Kết quả thử nghiệm cho thấy mô hình này có độ sai lệch +0.70 khi gặp từ hiếm. #LLaMA #GPT2 #AI #TríTuệNhânTạo #AnToànMôHình #Vulnerability #ArtificialIntelligence #MachineLearning #Transformer #AttentionMechanism #Safety
https://www.reddit.com/r/LocalLLaMA/comments/1ojvmty/llama3_is_just_as_vulnerable_to_im_absolutely/
Ars Technica NewsAfter teen suicide, OpenAI claims it is “helping people when they need it most” https://arstechni.ca/RSBe #attentionmechanism #crisisintervention #AIandmentalhealth #contentmoderation #suicideprevention #transformermodels #AIhallucination #machinelearning #AIpaternalism #AIassistants #AIregulation #AIsafeguards #mentalhealth #AIalignment #AIbehavior #AIethics #AIsafety #chatbots #ChatGPT #Biz&IT #GPT-4o #openai #GPT-5 #AI
Multi-Token Attention
https://arxiv.org/abs/2504.00927
#HackerNews #Multi-Token #Attention #research #AI #advancements #attentionmechanism #deeplearning #NLP
Multi-Token Attention
Soft attention is a critical mechanism powering LLMs to locate relevant parts within a given context. However, individual attention weights are determined by the similarity of only a single query and key token vector. This "single token attention" bottlenecks the amount of information used in distinguishing a relevant part from the rest of the context. To address this issue, we propose a new attention method, Multi-Token Attention (MTA), which allows LLMs to condition their attention weights on multiple query and key vectors simultaneously. This is achieved by applying convolution operations over queries, keys and heads, allowing nearby queries and keys to affect each other's attention weights for more precise attention. As a result, our method can locate relevant context using richer, more nuanced information that can exceed a single vector's capacity. Through extensive evaluations, we demonstrate that MTA achieves enhanced performance on a range of popular benchmarks. Notably, it outperforms Transformer baseline models on standard language modeling tasks, and on tasks that require searching for information within long contexts, where our method's ability to leverage richer information proves particularly beneficial.
Go-attention: A full attention mechanism and transformer in pure Go — https://github.com/takara-ai/go-attention
#HackerNews #GoAttention #GoLang #Transformer #AttentionMechanism #AIDevelopment
#HackerNews #GoAttention #GoLang #Transformer #AttentionMechanism #AIDevelopment
