i-mac G6. Xbox 720hey so is there a local llm i can get (or train?) that is *just* an llm? like no system prompt, no chatbot features, just 'predict the next word'? i think my club needs a demo to de-anthropomorphicize these things
generally the culture around it among the group isn't brain-melting but i see things like "it thinks x" and i need to bring home that it does not think anything
i know you can get this via some openai dev sub (because i saw a blog post where someone demos basically this a while ago) but i'm not about to pay them money or be a monthly active user lmao
R@kirakira don't know of anything for an LLM, but you can start with a very oldschool hidden-markov-model chatbot?
AstraLuma
J. "Henry" Waugh@kirakira I would recommend:
https://github.com/karpathy/nanoGPT
(That is, an old project that is "left up for posterity" after switching it to a chat interface)
Krzysztof Sakrejda@kirakira maybe one of the LLM poisoning tools that just trains Markov model? Seeing the low Fi version can help people understand better.
IvanDSM@kirakira What you're looking for is a "foundation model". Look for them in HuggingFace and use them with llama.cpp. If you want to train from scratch, I think llama.cpp has training scripts.
I unfortunately had a local LLM experimentation stint in early 2023. As much as I hate the fact that I somehow ever thought these things had any merit to them, I do appreciate that the technical understanding I acquired makes it very easy to smell the bullshit.
@kirakira Here you go Kira, here's the Llama 2 foundation model, 7 billion parameter version, quantized for running on local hardware.
This is an early quantization, from back when TheBloke was the main person doing the quants, so they aren't as high quality as modern quants but honestly it doesn't make a damn difference.
@kirakira You can use this with llama.cpp. Get the 4-bit version (Q4_K_M or Q4_K_S, not Q4_0!) and plop it in.
Take some time to note that by the time Llama 2 dropped, TheBloke was already being sponsored by a16z. Nobody in our anti-AI circles ever talks about this, but the local LLM/GenAI movement is just as funded and boosted by fascists as the OpenAI/Anthropic side of things.
It's shit and dirt all the way up, but it's shit and dirt all the way down too.
@IvanDSM damn i did not know that part. thanks!