Mastodawn

@wagesj45 how do you run mistral? (mixtral??)

Mawoka Jul 7, 2024

@kellogh @wagesj45 Yea, interesting question. Locally on your GPU? Which?

@Mawoka @kellogh I run #Mistral #7B Instruct via CPU on a #Debian #Linux server via #API through #oobabooga. I didn't need realtime responses, and I had spare CPU power, so it worked out perfectly.

@wagesj45 @Mawoka what do you use it for?

@kellogh @Mawoka Medicine alarms, summaries of my home sensors, translating raw weather alerts to more human conversation style, a morning alarm that wakes me up and gives me a rundown of my calendar events for the day. 🙂

@wagesj45 that’s pretty awesome tbh @Mawoka

@wagesj45 @Mawoka wait, does it prepare your medicine for you too??

@kellogh @Mawoka LOL I wish. It just reminds me if I don't mark them as taken by a certain time.

@wagesj45 @Mawoka this whole thing is pretty neat. You should write about it!

@kellogh @Mawoka I guess it is about time for my yearly blog post.

Mawoka Jul 7, 2024

@wagesj45
"Not real time" means how long? And which CPU are you using? Sorry for all those questions 🙈
@kellogh

@Mawoka @kellogh Oh its still plenty fast. Just not realtime speaker-assistant level fast. It can take a long prompt and generate a response in as fast as 30 seconds for an alarm with all my calendar events for the day, or up to a minute or two with a prompt with a few hundred sensors and their states.

@Mawoka @kellogh It's running in a VM, so the machine just sees it as a generic CPU. The physical machine has some Xenon chip that I can't remember without logging in to see. It was a used server.

@wagesj45 @Mawoka how do you do text-to-speech? something simple? or an LLM?

@kellogh @Mawoka I take the output of the Mistral API call, then pipe that into the TTS function of Home Assistant, which runs the text through Piper AI to generate an audio file, and it sends that audio to any of the speaker devices available to Home Assistant.