Ok. Limited sneak peak (this will be taken down shortly).. very early PoC of my new LLM vs. Human research.
A novel and efficient technique for auditing LLM content. Very early stage but the signal is present!!
#llm #ai #grc #governance #machinelearning
[edited to remove link - I’m sure it’s in the history but I’ve taken it down for now. Im glad so many of you enjoyed.]
yes very different. PPM operates of surface characteristics of text output. Seems to be getting more difficult as LLMs get more sophisticated.
My work came about from examining other LLM models (not the ones used for generation). I guess the research best described as "how other LLMs react” to text. With the hypothesis being that weird LLM behaviour has been shown in LLMs trained recursively on LLM output (i.e. models trained on model output rather than authentic human text, shown by Shumailov et. al, 2024 - https://www.nature.com/articles/s41586-024-07566-y). IT was motivated by me trying to automate recipe parsing from blogs (to extract ingredients for my shopping cart), but getting weird hallucinations if the copy-paste to my LLM tool came from another LLM!
My experience made me think what is it that differs in the way a transformers layers (residual state, Attention, MLP) are activated when processing real human text vs. LLM text.. the method is about selecting the families and internal characteristics that allow this).
Edit: correcting typos and clarity
Analysis shows that indiscriminately training generative artificial intelligence on real and generated content, usually done by scraping data from the Internet, can lead to a collapse in the ability of the models to generate diverse high-quality output.
Nikolai Hampton 💾
Drew Scott Daniels