So, I've established that local LLM's as coding agents aren't ready for prime time... But I do have some nice local problems I can still solve.
My personal blog has some 2.464 posts spanning 21 years. I used WordPress for many years before migrating to Hugo. In that transition I ditched the mess of tags.
So, I've spent my Saturday toying and experimenting with local LLM's to generate tags back to my posts.
First up, I tried NLTK + KeyBERT. This is a Dutch blog with many flemish idioms. Processing was really fast on my Macbook Pro M3, but I had to do a lot of tuning with stopwords, geonames and spaCy. I've even tried BERTje which is a Dutch pretrained model from the University of Groningen. The end result per iteration kept being below par though.
Lots of cruft, adverbs, interjections, adjectives. Or verbs wrongly classified as nouns and vice versa. The tweaking quickly devolved into spaghetti.
Next up, I'm trying LM Studio with a bigger model. The good is that I can do away with most of the tuning code. The downside: processing is slower, and I needed some tuning to make it even slower lest I'm planning to fry an egg on the keyboard.
I tried mistral-7b-instruct-v0.3 first. It yielded great results from the get-go but this is general model. Subjectively, I felt I could do better.
Right now, I'm running aya-23-8b by Cohere Labs which hits the right spot so far. https://huggingface.co/CohereLabs/aya-23-8B
That still leaves me with a problem, though. These models just extract meaningful keywords within the context of a single, isolated post. Whereas tagging really becomes a thing if it can extrapolate meaning across posts.
E.g. I have multiple posts regarding Apple products. Not all of them mention the same exact keywords, but the reader can deduce that they belong to the same group. LLM's miss that completely. So, that would be a logical next step to explore.
netsensei