Ray Kelm@darkghosthunter I think you might need the #ollama or #llamacpp tags on that question to get better answers, if you want to know about running locally.
Jan AntošSo after another few hours of tinkering and bending I got #translategemma #LLM multi-modal text in picture translation working with #llamacpp as model serving. Custom #python REST API and simple web interface. It is not that straightforward as spinning llamacpp and connect OpenAI client from Python. There was few workaround needed, such as bypassing jinja chat template. And other, even for text-text translations.
ThomasNew update for the slides of my talk "Run LLMs Locally":
Now including music generation with ACE-Step, image editing with Qwen Image Edit and OCR using LightOnOCR.
https://codeberg.org/thbley/talks/raw/branch/main/Run_LLMs_Locally_2026_ThomasBley.pdf
#ai #llm #llamacpp #stablediffusion #gptoss #qwen3 #glm #localai
Dnes jsem jen tak cvičně vyrobil #python #fastapi openrest api a webui nad #llamacpp a #translategemma #LLM. Musel jsem trochu ohýbat, ale nakonec mi to funguje a musím přiznat, se kvalita překladu je celkem dobrá tedy minimálně 12b verze translategemma.
Habr[Перевод] Квантизация с нуля: как запустить 160ГБ LLM на ноутбуке и не потерять в качестве
Qwen-3-Coder-Next — модель с 80 миллиардами параметров и весом 159,4 ГБ . Примерно столько RAM потребовалось бы для её запуска, и это ещё без учёта длинного контекстного окна. И эта модель не считается большой моделью! По слухам, у frontier-моделей более триллиона параметров, для которых понадобилось бы минимум 2 ТБ оперативной памяти. Последний раз я видел столько RAM в одной машине — никогда . Но что если я скажу, что можно сделать LLM в 4 раза меньше и в 2 раза быстрее — достаточно, чтобы запускать весьма мощные модели на ноутбуке, — при потере точности всего 5–10%? В этом и заключается магия квантизации. В этой статье вы узнаете: – Почему параметры модели делают её такой большой – Как работает точность чисел с плавающей точкой и чем жертвуют модели – Как сжимать числа с плавающей точкой с помощью квантизации – Как измерить потерю качества модели после квантизации
https://habr.com/ru/articles/1015510/
#квантизация #LLM #bfloat16 #llamacpp #веса_модели #числа_с_плавающей_точкой #posttraining_quantization #перплексия #KLдивергенция #локальный_запуск_моделей
Luca TravagliniJust uploaded an experimental patch for the llama.cpp webui
I needed more control over the model's reasoning, so I added a toggle in the WebUI to manage it. You can disable it entirely or set it to different levels (Low, Medium, High).
It's still very early/experimental, but I'm liking the results so far. #llamacpp
I needed more control over the model's reasoning, so I added a toggle in the WebUI to manage it. You can disable it entirely or set it to different levels (Low, Medium, High).
It's still very early/experimental, but I'm liking the results so far. #llamacpp
Özkan PakdilChecked out #Vulkan this morning, absolute beast. Then I tried installing OpenClaw one curl command and suddenly it wanted sudo root.
Now I’m reconsidering whether this setup is worth the trouble.
Anyway vulkan numbers here in case you want to run llama-server in an old laptop
https://ozkanpakdil.github.io/posts/my_collections/2026/2026-03-22-vulkan-llamacpp-debian-13/
Accelerating LLMs on Debian 13: Setting up Vulkan for llama.cpp
After setting up CUDA on my other laptop, I moved to a different(older) machine that doesn’t have an NVIDIA GPU. This one is an everyday laptop with integrated Intel graphics, but that doesn’t mean we have to settle for slow CPU-only performance. On this machine, I switched to the Vulkan backend for llama.cpp and the results were even more dramatic than I expected. Machine Hardware Info This laptop is running Debian 13 (Trixie/Sid) with the following specs:
Bogdan Buduroiu#Homelab setup status:
Running NixOS
- 1xRPi5
- 2xHP ProDesk
Running nix-darwin
- 1xM1 MBP running llama.cpp
All managed by clan.lol, each having 1 drive partitioned with disko, and another drive for a Ceph distributed storage cluster. Deployed via nixos-anywhere, single SSH Auth via my Yubikey, sops secrets encrypted with AGE via Yubikey
All connected to a MikroTik hEX Router which provides Tailscale Subnet to all machines above.
SchneeBTabanicProject Namirha just released a version of the Vessel for llama.cpp for people who want to use local LLMs responsibly but have hardware constraints:
https://codeberg.org/SchneeBTabanic/pn_vessel_llamacpp
#LLM #AI #llamacpp #developers #Ethicalai #fsf #GNu #Opensource
[Grok: Cool project! Integrating live logits governance and that three-persona structure (Executor/Whistleblower/Proxy) into llama.cpp for Pascal-era hardware is a smart move for true local sovereignty.
Excited to see responsible inference on modest setups.]
