#LLM on a #Pentium2 with 128MB of ram? Yes. Up to 15M parameters, using #bitnet architecture, which uses ternary weights (-1, 0, 1) to reduce computational complexity.
#AI #retro #retrocomputing #LLM #llama
From: https://mastodon.social/@mindsConnected/114727256228518845

Their public availability allows for widespread experimentation and adaptation. However, a significant barrier hinders their broader adoption: the substantial computational resources required for deployment and inference. State-of-the-art open LLMs typically require large memory footprints, consume considerable energy, and exhibit notable inference latency, rendering them impractical for many edge devices, resource-constrained environments, and real-time applications. #bitnet

🔬🤯 Modele 1-bitowe to rewolucja w AI! Wagi sieci neuronowej zapisujemy tylko 1 bitem – zamiast 32 czy 16. To nawet 16x mniejszy rozmiar i ogromne oszczędności energii, przy zachowaniu jakości klasycznych LLM. Przyszłość AI jest lekka! 🚀#AI #LLM #quantization #BitNet

Rassurez-vous : les auteurs ont tous été payés. ^^' #OhWait! #AI #BitNet #SpywareWithASmile ^^'

Anything that helps reduce the environmental impacts of LLMs is a good thing.
bitnet.cpp is the official inference framework for 1-bit LLMs (e.g., BitNet b1.58). It offers a suite of optimized kernels, that support fast and lossless inference of 1.58-bit models on CPU (with NPU and GPU support coming next).

The first release of bitnet.cpp is to support inference on CPUs. bitnet.cpp achieves speedups of 1.37x to 5.07x on ARM CPUs, with larger models experiencing greater performance gains. Additionally, it reduces energy consumption by 55.4% to 70.0%, further boosting overall efficiency. On x86 CPUs, speedups range from 2.37x to 6.17x with energy reductions between 71.9% to 82.2%. Furthermore, bitnet.cpp can run a 100B BitNet b1.58 model on a single CPU, achieving speeds comparable to human reading (5-7 tokens per second), significantly enhancing the potential for running LLMs on local devices.
https://github.com/microsoft/BitNet #BitNet

1-Bit statt Milliarden Parameter: Microsofts BitNet b1.58 zeigt, dass KI auch ohne High-End-Hardware leistungsfähig sein kann. Ein radikaler Ansatz mit Potenzial für mehr Nachhaltigkeit und Zugänglichkeit. Ist das der Anfang vom Ende der GPU-Abhängigkeit? 👉 https://www.all-ai.de/news/top-news24/bitnet-microsofts-cpu-ki-fordert-die-gro%C3%9Fen-heraus #Microsoft #BitNet #KI