jordanJul 7, 2024

Man, I'm obsessed with integrating #MistralAI models into my #homeautomation and #homelab operations. Even set up a custom #node in #NodeRed to prompt the #AI. All my alerts are very personalized now and I love it. <3

#llm #largelanguagemodel #cool

Show threadTim KelloggJul 7, 2024
@wagesj45 how do you run mistral? (mixtral??)
Show threadMawokaJul 7, 2024
@kellogh @wagesj45 Yea, interesting question. Locally on your GPU? Which?
Show threadjordanJul 7, 2024
@Mawoka @kellogh I run #Mistral #7B Instruct via CPU on a #Debian #Linux server via #API through #oobabooga. I didn't need realtime responses, and I had spare CPU power, so it worked out perfectly.
Show threadMawokaJul 7, 2024
@wagesj45
"Not real time" means how long? And which CPU are you using? Sorry for all those questions 🙈
@kellogh
Show threadjordan
@Mawoka @kellogh Oh its still plenty fast. Just not realtime speaker-assistant level fast. It can take a long prompt and generate a response in as fast as 30 seconds for an alarm with all my calendar events for the day, or up to a minute or two with a prompt with a few hundred sensors and their states.
Jul 7, 2024 at 10:14pmWeb