eevee 🦊Jan 11, 2023

chatgpt is predictive text.
chatgpt is predictive text.
chatgpt is predictive text.
chatgpt is predictive text.
chatgpt is predictive text.

it's not even answering questions. it just pattern-matches that the next text after something that looks like a question is most often something that looks like an answer

Show threadSnowDerg ❄️🪶Jan 11, 2023
@eevee so it's just markov chains on steroids?
Show threadGraham Sutherland / PolynomialJan 11, 2023
@SnowDerg @eevee a bit like a markov chain glued to a content recommendation algorithm.
Show threadReboot/FitzJan 11, 2023
@gsuberland @SnowDerg @eevee glued as in two models or a larger neural network implementing both concepts?
Show threadGraham Sutherland / PolynomialJan 11, 2023

@reboot @SnowDerg @eevee essentially both concepts smushed together in a stateful manner as a single model, at least in terms of external behaviour.

the actual architecture isn't the same as those individual components but that's a separate conversation.

Show threadReboot/FitzJan 11, 2023
@gsuberland @SnowDerg @eevee but, I kinda want to have that conversation... (as a person who does a lot of general CS work.)
Show threadGraham Sutherland / PolynomialJan 11, 2023

@reboot @SnowDerg @eevee I'm probably not the right person to talk about it in deep detail since the gory innards of LLMs and autoregressive models aren't my wheelhouse. Someone did post a good article on LLMs the other day but I can't spot it.

I'd recommend searching online for an explainer but unfortunately every single one I found on the first page of Google was a bust due to gushingly anthropomorphising the model in a way that falsely implied an ability to develop understanding.

Show threadReboot/FitzJan 11, 2023
@gsuberland @SnowDerg @eevee makes sense as any sense that it developed a greater "understanding" would be context kicking in rather than actual learning. Mind expanding L.L.M. for me?
Show threadLafncow 
@reboot @gsuberland @SnowDerg @eevee You might want to look for an explainer on transformer models (the T in Chat GPT), and your intuition is about right: they are neural nets that maintain a series of hidden states as data is fed in.
Jan 11, 2023 at 6:45amWeb