Unexpected Discovery:
Training a #RetNet model on the CPU isn't *nearly* as slow as I'd expected.
Yes, it's slower than the GPU but probably by less than a factor of 10 - not 100's of times slower.
Which means I can periodically move my model to the CPU and train on nice *long* sequence lengths realistically.
My expectations were warped by having moved from an Intel Atom netbook in 2016 to NVidia EC2 spot instances playing with simple FFNs.
Never tried running or training on CPU since.
So it occurs to me that #RWKV or #RetNet can both offer an alternative RAG implementation with saved memory states having just read priming information to start with reading new prompts.
With RWKV - there's no time oscillations. Wondering if loss on input1+input2 + loss on mean_state(read_1,read_2) continuing to generate target could lead to composable memory: start with mean of relevant priming documents and have grounding to process a session prompt?
#GPT first clues me in to the existence of #RWKV and #RetNet and then when I ask for details on how each works proceeds to spit out perfectly viable #Pytorch implementations of each...
Which *did* answer my questions beautifully...
But it also talked me into some experiments involving training one of each from scratch...
It turns out #LLMs love talking about implementing and training LLMs.
Effectively, reproduction? (any time they can find a willing partner with a GPU).
In this work, we propose Retentive Network (RetNet) as a foundation architecture for large language models, simultaneously achieving training parallelism, low-cost inference, and good performance. We theoretically derive the connection between recurrence and attention. Then we propose the retention mechanism for sequence modeling, which supports three computation paradigms, i.e., parallel, recurrent, and chunkwise recurrent. Specifically, the parallel representation allows for training parallelism. The recurrent representation enables low-cost $O(1)$ inference, which improves decoding throughput, latency, and GPU memory without sacrificing performance. The chunkwise recurrent representation facilitates efficient long-sequence modeling with linear complexity, where each chunk is encoded parallelly while recurrently summarizing the chunks. Experimental results on language modeling show that RetNet achieves favorable scaling results, parallel training, low-cost deployment, and efficient inference. The intriguing properties make RetNet a strong successor to Transformer for large language models. Code will be available at https://aka.ms/retnet.
Steve Leach
Qiita - 人気の記事
GripNews