Timnit Gebru (she/her).This whole "this is how humans learn so whats the difference" thing while stealing so much data to make billions for a few dudes is so insidious.
jakob.pxi@timnitGebru /narrator voice/ this was, in fact, not how humans learn. One of the great mysteries of human cognition lies in the befuddling observation just how quickly humans learn language during a critical age period, and how exactly they lose that special power. There even was a great scientific debate around *not* needing to consume terabytes of input to learn language, called "poverty of stimulus". And yet, the promise of return on investment droned out such debates.
whoever looks for adjacent info in pithy comments, see this thread where a little detail on the poverty-of-stimulus argument is shared and some musings of how that could inform how we grok what LLMs can and cannot do and how metaphors frame how we think about machines.
jakob.pxi (@[email protected])
#LLM are brute-forcing their way through absurd amounts of data to generate an autocomplete output for any given input that approximates outputs a human might give instead. They lack a few distinct properties of human cognition, including language, that more brute force alone cannot compensate for. Because they can only ever internalize and compute *intra*textual context. Incidentally, humans need much less input(!) to learn language. Probably because they can contextualize across domains. 🧵