Jason Gorman

I called it a year ago - it's an S-curve.

If I were an investor in a proprietary LLM, I'd be looking to offload those shares soon.

https://garymarcus.substack.com/p/evidence-that-llms-are-reaching-a

Evidence that LLMs are reaching a point of diminishing returns - and what that might mean

The conventional wisdom, well captured recently by Ethan Mollick, is that LLMs are advancing exponentially. A few days ago, in very popular blog post, Mollick claimed that “the current best estimates of the rate of improvement in Large Language models show capabilities doubling every 5 to 14 months”:

Marcus on AI
Apr 14, 2024 at 5:14amWeb
Show threadJason GormanApr 14, 2024

The thing is, the 100x-revenue valuations of Nvidia, OpenAI etc (which is losing money hand over fist) are built on the anticipated exponential advancements in the technology and therefore in the market for it. If that's not coming any time soon, there'll be a *big* reset in market caps.

And the rest of us will have to go back to addressing real business problems.

Show threadJason GormanApr 14, 2024
If you want to know what that looks like, just ask Cisco
Show threadJason GormanApr 14, 2024
It'll be interesting to see if the field pivots back towards application-specific ML as the way to make actual money by doing something actually useful
Show threadThe Long TailApr 14, 2024
@jasongorman and yet having been in one of those ML-based startups a few years ago, it actually turned out that all we needed was some stats (linear regression, monte carlo), and the ML was poorer performing on every metric.
Show threadJason GormanApr 14, 2024
@sleepyfox That doesn't surprise me
Show threadThe Long TailApr 14, 2024
@jasongorman I mean, I'm sure that there are some problems that are a good fit for ML, but they're fewer than people think.
Show threadJason GormanApr 14, 2024
@sleepyfox Fewer than executives and investors think. Ironically, they'd probably the easiest to replace with a chatbot.
Show threadGreg TitusApr 14, 2024

@jasongorman @sleepyfox I think this is a fairly significant part of why the bubble inflated in the first place. These are the people who are used to thinking that the spewing of convincing sounding bullshit and the parsing of convincing sounding bullshit is the important work, only vaguely connected to the actual products of the companies they run or invest in.

It's no wonder they bought in to the AI hype.

Show threadRob NapierApr 14, 2024
@sleepyfox @jasongorman I recently worked on a specialized VAD system (voice activity detection). We went down the road of ML systems, but they all were in the 700ms range for detection which is unworkable. Eventually we went back to hand-tuned FFT-based solutions, which are sub-50ms. For our specialized problem, FFT was hard, but possible. ML was a ton of wasted effort based on naive hope. In the end, the company folded before we could recover.
Show threadJason GormanApr 14, 2024
@sleepyfox At this stage, "A.I." is basically Brawndo. "It's got machine learning".
Show threadRob SynnottApr 15, 2024
@sleepyfox @jasongorman Shades of the blockchain bubble a few years back, where people used (or at least pretended to use; I think most such systems never actually made it into production) private blockchains for tasks better suited to, like, a spreadsheet because it tickled the VCs' fancy.
Show threadkjetiljdApr 14, 2024

@jasongorman The Dot-Com Bubble was a lot bigger than the AI Hype. The entire Nasdaq fell 78% between March 2000 and October 2002. A 78% fall now would set Nasdaq back 10 years, which is very unlikly. We are probably somewhere around the peak of inflated expectations, but apart from the runaway pricing on the top tech stocks, most stocks are fairly moderately priced.

Funny thing is, I think a lot of the stuff that was promised during the Dot-Com Bubble are now reality.

Show threadJason GormanApr 14, 2024
@kjetiljd It's still going to be pretty spectacular when the bubble bursts 🙂
Show threadgabriele renziApr 14, 2024
@kjetiljd @jasongorman I personally believe this looks like the web2.0 bubble, not dotcom.
Remember when a mash up of Flickr and Maps was considered a business? That's the current landscape, people build PoCs and demos based on assumptions of easy potential improvements which aren't really there.
Not worthless, just not as valuable as some think.
Show threadRob SynnottApr 15, 2024
@riffraff @kjetiljd @jasongorman That never _really_ saw the same level of investment, though; it was generally relatively cheap to do. The "extremely high costs, no revenue, any path to revenue looks slightly fanciful" dynamic is very dotcom bubble.
Show threadOblomovApr 14, 2024
@jasongorman one of these days these “geniuses” will understand that there is no infinite growth in a finite world, and with exponential growth one hits the wall MUCH earlier, but apparently that day is not today *sigh*.
Show threadOsma A 🇫🇮🇺🇦Apr 14, 2024
@oblomov
Do not expect a genious to accept what would prevent him from chasing a fortune from those easy to convince otherwise.
(with credit to Upton Sinclair)
@jasongorman
Show threadRob SynnottApr 15, 2024
@oblomov @jasongorman I think Moore's Law gave us, as an industry, slightly unrealistic expectations; Moore's Law is a bit of an _oddity_, and not what should be expected of, like, phenomena in general.
Show threadMatthew WaltonApr 14, 2024

@jasongorman it would be nice to go back to addressing real business problems. If we can legitimately say we were doing that in the first place which I think to be honest most of us weren't.

But GitHub Copilot isn't the future either.

Show threadJulien W.Apr 14, 2024
@jasongorman there's another perspective: how much time/energy needed to train the models. I've read somewhere else this was exponentially going down... Do you have data about that too ?
Show threadJason GormanApr 14, 2024
@julienw That's discussed in Marcus's post. It won't effect how capable the models are.
Show threadJulien W.Apr 14, 2024
@jasongorman yes, I wasn't contradicting this part indeed
Show threadThomas Favre-Bulle 🏳‍🌈Apr 14, 2024
@jasongorman @hazelweakly It's still GPT4...
Show threadJason GormanApr 14, 2024
@erispoe @hazelweakly It's GPT-4 Turbo, which many suspect was a failed attempt at launching GPT-5.
Show threadThomas Favre-Bulle 🏳‍🌈Apr 14, 2024
@jasongorman @hazelweakly I haven't seen evidence that turbo is a different model, which we would see if that was a failed GPT5. That's the first time I hear that theory though.
Show threadJared White (ResistanceNet ✊)Apr 14, 2024
@jasongorman Been saying the same thing. I've had people literally tell me to my face that we're seeing "the first stepping stones to full AGI," and I'm telling them that *nothing* happening right now is actually paving that road. It's all an outrageous distraction. (Not that I know of any other approaches that could…I don't think there's even an intellectual framework to understand how true artificial intelligence might work in theory. It's like talking about Warp Drive and Transporters. 😄)
Show threadShawn "Fruan" PoulsenApr 14, 2024
@jaredwhite @jasongorman I forget where I first heard it, but my preferred metaphor is that you can’t get to the moon by getting better at climbing trees.
Show threadBenApr 14, 2024
@jasongorman all the same crypto grifters using all the same crypto GPUs to spin wild tales that even more suckers bought in to. Going to be interesting to see what kind of insanely-high-GPU-activity they pivot to next.
Show threadWinnie 🏳️‍🌈Apr 14, 2024
@jasongorman I agree with your conclusion but making assumptions based on one outlier in four data points seems a bit risky to me.
Show threadGabriele GiordanoApr 14, 2024
@jasongorman sam haltman mentioned some times the problem of scaling related to parameters. No wonders he wanted the Saudis to invest billions for custom hardware
Show threadr-holdApr 14, 2024
@jasongorman that's good News for the industry: so Nvidia can sell much more gpu time to try to compensate for that.
Show threadleidkulturApr 14, 2024

@jasongorman
Someone coined the term „Habsburg AI“ because the internet is now flooded with „AI“ (hint: it’s not really AI) text and images which leads to „AI“ training on more and more „AI“ output.

The results will get shittier day by day.

Show threadTom Bellin Apr 14, 2024

@jasongorman LLM tech was always a cul de sac.

Absent the blossoming of *actual* intelligence via the sheer weight of numbers (not an impossible theory, but alas) it's a LARP of the fallacy of a million monkeys on a million typewriters producing Shakespeare.

With an added spice of mass copyright violation. (Producing Shakespeare by copying the text directly)

Show threadHen Gymro Heb WladApr 14, 2024

@jasongorman Today's UK "Observer" newspaper also reckons AI is a bubble:

https://www.theguardian.com/commentisfree/2024/apr/13/from-boom-to-burst-the-ai-bubble-is-only-heading-in-one-direction

From boom to burst, the AI bubble is only heading in one direction

No one should be surprised that artificial intelligence is following a well-worn and entirely predictable financial arc

The Guardian
Show threadAlistair DavidsonApr 15, 2024
@jasongorman I thought of a hypothesis recently, which feels right but I have in no way proven: a linear increase in LLM performance requires an exponential increase in data consumed.
Show threadJason GormanApr 15, 2024
@moh_kohn That actually seems to have been the case so far. And I suspect they eventually hit a wall where no amount of training will improve their capability. Because there fundamental things missing.