🎉 Ah, the age-old quest for AI efficiency: let's just toss 90% of those pesky neurons and hope it doesn't implode! 🤯 "The Lottery Ticket Hypothesis"—because who doesn’t want their neural networks to be as unpredictable as a lottery win? 🤑 Oh, and don’t forget to donate to arXiv while you’re at it! 💸
https://arxiv.org/abs/1803.03635 #AIefficiency #LotteryTicketHypothesis #NeuralNetworks #TechTrends #arXivDonation #HackerNews #ngated
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.
We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.
We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Neural network pruning techniques can reduce the parameter counts of trained networks by over 90%, decreasing storage requirements and improving computational performance of inference without compromising accuracy. However, contemporary experience is that the sparse architectures produced by pruning are difficult to train from the start, which would similarly improve training performance.
We find that a standard pruning technique naturally uncovers subnetworks whose initializations made them capable of training effectively. Based on these results, we articulate the "lottery ticket hypothesis:" dense, randomly-initialized, feed-forward networks contain subnetworks ("winning tickets") that - when trained in isolation - reach test accuracy comparable to the original network in a similar number of iterations. The winning tickets we find have won the initialization lottery: their connections have initial weights that make training particularly effective.
We present an algorithm to identify winning tickets and a series of experiments that support the lottery ticket hypothesis and the importance of these fortuitous initializations. We consistently find winning tickets that are less than 10-20% of the size of several fully-connected and convolutional feed-forward architectures for MNIST and CIFAR10. Above this size, the winning tickets that we find learn faster than the original network and reach higher test accuracy.
How AI researchers accidentally discovered that everything they thought about learning was wrong | Nearly Right
Link學習理論的意外革命：深度理解「Lottery Ticket Hypothesis」
https://example.com/article/ai-learning-theory-breakthrough
📌
Summary:
本文探討人工智慧領域中，關於大規模神經網路訓練理論的根本轉變。過去三百年的學習理論，基於偏差-變異權衡（bias-variance tradeoff），認為模型過大會陷入過度擬合，導致無法泛化，但現實中大型神經網路如 ChatGPT 卻突破此法則大放異彩。關鍵在於一項革命性的發現——「Lottery Ticket Hypothesis」（樂透票假說）：大規模網路內含許多小型子網路，能以原始的隨機初始參數成功學習，訓練過程等同於在龐大的「樂透票」中尋找贏家。這解釋了為何擴大模型規模不但不會造成失敗，反而有機會找到更優雅且簡潔的解決方案，使得學習效果再次提升。該理論不僅重整了 AI 模型發展的方向，也暗示了人類大腦等自然智慧的神經架構設計原理。最終，本文提出雖然擴大模型帶來突破，但此一過程也暗示存在效益遞減及架構限制，讓未來 AI 研究需在理論與實證間持續探索。
🎯
Key Points:
→ ★ 傳統偏差-變異權衡理論（三百年基礎）認為：模型太小無法學習完整模式，太大則陷入過度擬合，只能記憶訓練資料，無法泛化。
→ ★ 2019 年，違反傳統的實驗發現「double descent」現象：大型模型訓練初期似乎過擬合誤差上升，之後卻出現誤差大幅下降，性能反而更佳。
→ ★ 「Lottery Ticket Hypothesis」由 Jonathan Frankle 和 Michael Carbin 提出，指出大型神經網路中存在「贏家子網路」，這些子網路只需原有的隨機初始參數即可達到母網路的成效。
→ ★ 訓練並非尋找新架構，而是從龐大參數空間中挑選出適合的「贏家票」，這解釋了為何更大規模帶來「質」的飛躍。
→ ★ 此理論重新定位學習的本質：智慧在於尋找最簡約且有效的模式，而非記憶繁複資訊；大規模參數如大腦神經元的過度配置，是為了提供足夠的可能性空間。
→ ★ 科技企業如 Google、Microsoft、OpenAI 等依此理念大幅擴大模型參數規模，展開了巨額投資與研發競賽。
→ ★ 然而，專家如 Yann LeCun 提醒此機制可能存在天然限制，規模提升帶來的效益會逐漸遞減，且單靠擴大參數未必能達到真正理解或通用智能。
→ ★ 這段科學史上的顛覆案例展示科學探索中實證勇氣的重要性，強調理論與數據互動的深度進化，而非盲目否定既有法則。
🔖
Keywords:
#LotteryTicketHypothesis #偏差_變異權衡 #神經網路 #深度學習 #人工智慧
How AI researchers accidentally discovered that everything they thought about learning was wrong
The lottery ticket hypothesis explains why massive neural networks succeed despite centuries of theory predicting they should fail
N-gated Hacker News
Hacker News
卡拉今天看了什麼