Carlos E. Perez @IntuitMachineI calculate that it's at least a 2-year gap between open-source deep learning networks and networks developed inside private corporations. This is problematic when progress is exponential. That 2-year gap will appear much worse over time!
Another intriguing trend is the widening gap between generative models and discriminative models. The former seems to be able to exploit a flywheel-like process. https://twitter.com/ajratner/status/1604533075714756609
Alex Ratner on Twitter
“1/ 2023 AI prediction: the gap between generative and predictive AI will widen. Despite product & business model innovation in generative AI, real-world ROI will remain concentrated around predictive AI- leading to frustrated expectations. This gap will all come down to data...”
Conventional ML models have always been framed along the lines of developing classifiers. In fact for the longest time, DL models were trained end-to-end. It was only with the discovering of self-supervised training that classifiers became more of a side-effect.
Generative models are in fact an unusual capability of deep learning models. This became obvious when StyleGANs were invented. But their utility was not obvious because you couldn't reliably use their outputs as a training set.
But in-context learning in large language models changed all of this. In early 2020, GPT-3 was introduced and it revealed an emergent effect where behavior could be rendered without training.
The narrative that large internet companies had an unfair advantage because of their access to vastly more data appears to be disrupted! We may have reached that state where more data has diminishing returns!
This new reality can be understood by understanding why Deep Learning methods are limited. Curve-fitting does not have reach. However, fine-tuned curve-fitted models have reach because of human feedback. https://medium.com/intuitionmachine/ptolemy-and-the-limits-of-deep-learning-4c74dbb008e7
In software development, firms have create an advantage by having better tools than the competition. Traditionally, this was driven by a different programming language or a more advanced framework. Teams writing in assembly had no chance against folks writing in C.
You can push out features much faster if you got a superior framework. But frameworks cannot ever paper over the flaws in the stack. Hence C never fixed its vulnerabilities caused by memory leaks. So there was always a niche in re-inventing the stack.
The shift to generative models is analogous to re-inventing the stack in machine learning. Classifier models always have vulnerabilities. Hence Tesla's full self-driving had edge cases where it would, unfortunately, kill its occupants. Engineering can't paper this over.
It's only in hindsight that we know of fundamental limitations. AlphaGo (over 5 years ago) had massively inflated expectations as to what DL could do. Musk bought into that hype and aggressively pursued self-driving. But FSD has hit a dead-end. It's good but not good enough!
Self-supervised learning isn't actually that old. It became obvious only in 2019 that this would be a massive paradigm shift. I hope you read my post back then: https://medium.com/intuitionmachine/the-paradigm-shift-of-self-supervised-learning-744a6819ce08
So we are roughly 4-5 years into this new paradigm. Much longer if you consider GANs. Most of the progress has been in research labs. Something cool takes time to mature before it can be reliably useful. 2022 may have been the year when generative DL is reliable enough.
It's not perfect, but it might just be good enough. Search in the form of inverted indexes wasn't perfect when Google scaled it to the internet, but it was good enough. Nobody uses Google for mission-critical decision-making. Imperfect tools are useful in less-risky contexts.
This is where we are with deep learning AI. Tools that are just good enough. Just good enough to be used in areas that paper over our own human cognitive limitations.
Sidenote: It appears the thread aggregation bots may have been kicked out of Twitter. Also, Twitter's thread view is unavailable. So stay tuned when I reformat this into a blog entry.