Mastodawn

jgarcia Mar 25, 2024

This really drives home something about LLM systems. They’re very expensive to run, both to train and per-query, and hard to make cheaper. I expect them to get more expensive to run. They’re currently sold at a big loss to establish monopoly power and then raise prices dramatically. That’s the *stated* business plan.

If you’re building your business to rely on LLM, you need to factor in what you‘ll do when they pivot to making money, or they pull back because they can’t.
https://geeknews.chat/@theregister/112116266764229145

The Register (@[email protected])

Microsoft promises Copilot will be a 'moneymaker' in the long term Exec tells investors to 'temper' expectations as mission to convince customers of price tag continues Microsoft is asking investors to "temper" expectations for quick financial returns from Copilot amid efforts to convince customers that paying "substantial" sums each month is actually worth it.… #theregister #IT https://go.theregister.com/feed/www.theregister.com/2024/03/18/microsoft_copilot_moneymaker/

Geek News Central Mastodon Chat

Brandon Horst Mar 18, 2024

@cocoaphony The free llm apps (copilot, chatgpt) are operating at a loss, and are absolutely going to see some nasty monetization coming soon. But I’m not so sure if that’s true for the APIs. OpenAI keeps lowering their prices despite relatively little competition.

Rob Napier Mar 18, 2024

@brandonhorst this article is about the paid integrations with Office products, not ChatGPT. If the APIs were profitable, Microsoft wouldn’t be begging their investors for patience. Cutting API prices absent competitors is not evidence that the queries are cheaper. It’s evidence they’re having trouble finding customers at the higher price, and they’re willing to take losses to build market share.

Brandon Horst Mar 18, 2024

@cocoaphony Yeah that’s definitely possible. No doubt this stuff is already becoming commoditized

Frank Reiff Mar 18, 2024

@cocoaphony @nicklockwood @theregister I find the technology fascinating, but it seems likely that when the hype wears off and the realization that “human-level” performance, especially as it relates to actual understanding, common sense and baseline reliability, is in fact not reachable with current stochastic parrot machine learning, the entire market is going to collapse.

Misuse Case Mar 18, 2024

@frankreiff @cocoaphony @nicklockwood @theregister This is another aspect of it, lots of companies/executives want to believe they can totally replace humans with it, but they can’t and they are going to find out the hard way. That realization and the price jack-up will hit them at the same time.

Rob Napier Mar 18, 2024

@MisuseCase @frankreiff @nicklockwood This is very much the story of Uber. They designed their business plan around self-driving cars. As those persistently failed to materialize, the company bled money for years. It's finally profitable after accepting that they're in the business of people-driven taxis, but it'll be a long time to make back what it cost to get there.

"No law can stop us!" has also turned out to be a challenging business model to make sustainable…

https://www.theverge.com/2024/2/8/24065999/uber-earnings-profitable-year-net-income

Uber ends the year in the black for the first time ever

Uber is finally profitable, after 15 years of trying. The company reported its first annual operating profit and positive net income, signaling stronger financial footing.

The Verge

Misuse Case Mar 18, 2024

@cocoaphony @frankreiff @nicklockwood I don’t believe it’s meant to be a sustainable business model I think it’s meant to be a cash grab.

With a lot of these “AI” things it’s much more obvious. Sam Altman and the like are trying to grab as much money and content as possible before a critical mass of people are wise to the con and/or regulation comes down.

Rob Napier Mar 18, 2024

@MisuseCase @frankreiff @nicklockwood I'll buy that for Altman and others, but Microsoft is a company devoted to paying their dividend to investors. There's no "exit" with a pile of cash for them. They're the mark left holding the bag at the end.

So they either need a sustainable business or they have to get out of it. I expect a long grind to figure out how to do either. And I'd be nervous building a business that relies on how they decide to play it.

Misuse Case Mar 18, 2024

@cocoaphony @frankreiff @nicklockwood I guess that’s true for Microsoft, and their users are second- and third-order marks.

RealGene ☣️Mar 18, 2024

@cocoaphony @MisuseCase @frankreiff @nicklockwood
I want an LLM to observe how I use Word, and over time recognize what MY document defaults should look like, and reduce the ribbon to show just the things I actually use.

Misuse Case Mar 18, 2024

@RealGene @cocoaphony @frankreiff @nicklockwood This isn’t actually an LLM as such but it is a machine learning algorithm (of which LLMs are a subset) and that would be a useful thing! I can also think of some other applications in which that type of machine learning algorithm would be useful and there are a few being used that way!

Rob Napier Mar 18, 2024

@MisuseCase @RealGene @frankreiff @nicklockwood I've been doing some rough calculations based on semianalysis's estimate of 0.36 cents/query.

With their $30/month cost (which is probably way too high), if a user makes more than 50 queries/hour, they'll lose money on just the operation costs (not including capex, overhead, R&D, profit, etc.) So their real limit is much lower.

I expect many features people want will require more than that, which makes this very hard.

https://www.semianalysis.com/p/the-inference-cost-of-search-disruption

The Inference Cost Of Search Disruption – Large Language Model Cost Analysis

$30B Of Google Profit Evaporating Overnight, Performance Improvement With H100 TPUv4 TPUv5

SemiAnalysis

Frank Reiff Mar 18, 2024

@cocoaphony @MisuseCase @RealGene @nicklockwood At the moment the influence on share price far outweighs the actual cost.

In future, when cost becomes relevant, there are lots of ways to cut them. After all there is nothing in the contracts that says WHICH model is run. There are efforts underway to “prune” models to make them both more efficient and improve the output. Our own brains work like that, eg glial cells aka white matter.

Frank Reiff Mar 18, 2024

@cocoaphony @MisuseCase @RealGene @nicklockwood

The real problem, I speculate, is going to be falling demand for services that don’t deliver the value that they promise.

IBM’s Watson had this type of hype after its jeopardy win and resulted in a lot of contracts being cancelled when they failed to deliver, for instance on oncology diagnostics. Turns out the real world is disappointingly complex. 🤷‍♂️

Simon Tournier Mar 18, 2024

@cocoaphony From International Energy Agency report:

« When comparing the average electricity demand of a typical Google search (0.3 Wh of electricity) to OpenAI’s ChatGPT (2.9 Wh per request), » p.34 https://www.iea.org/reports/electricity-2024

The electricity seems a good candidate for a zero-order approx. about the "cost" of AI. The money-cost means nothing compared physical resources.

[1/2]
@nicklockwood @frankreiff @RealGene @MisuseCase

Electricity 2024 – Analysis - IEA

Electricity 2024 - Analysis and key findings. A report by the International Energy Agency.

IEA

Simon Tournier Mar 18, 2024

@cocoaphony Climate changes will have strong impacts.

For instance, the lack of water will imply high constraints on electricity production; see International Energy Agency https://www.iea.org/reports/electricity-2024. Hydroelectricity, obviously. And more. Even nuclear; with low water level, poor perfs because low cooling.

And dry area will consume more energy or resources to secure water's supply; e.g. desalination or else.

[2/2]

Electricity 2024 – Analysis - IEA

Electricity 2024 - Analysis and key findings. A report by the International Energy Agency.

IEA

Rob Napier Mar 18, 2024

@zimoun Rising energy costs are a big part of my assumption that they will get more expensive rather than cheaper. There are other limits that I think may kick in more abruptly, though. For example, hardware and data center capacity. As we've learned over the last few years, a disruption in chip supply can impact the whole industry. Fabs are hard to build in response to demand shocks. And whenever cryptocurrency prices spike, AI has to compete for everything with miners. Lots of business risks.

Simon Tournier Mar 18, 2024

@cocoaphony My point is to raise that money-value as "cheap" or "expensive" is an inadequate frame under climate change constraints. A techno. can still be money-profitable because "sponsored" but not physically-profitable.

LLM burn a lot of energy and as zero-order approximation energy = CO2 so LLM cannot be profitable; even now.

I hope that your prediction about money-profit of LLM will happen…

PatNaik Mar 19, 2024

@MisuseCase @RealGene @cocoaphony @frankreiff @nicklockwood What kind of model would you use for that?

Misuse Case Mar 19, 2024

@PratNaik @RealGene @cocoaphony @frankreiff @nicklockwood Some kind of very strictly supervised learning model. I don’t think any LLM falls into that category.

Philippa Cowderoy Mar 19, 2024

@MisuseCase @cocoaphony @frankreiff @nicklockwood The "grab" was meant to be owning the market ahead of when self-driving materialised - and failing that, deregulating it through BS. We'll have to see how the latter goes, because there's a genuine risk of them making the entire taxi market unsustainable for a decade or two across large regions before things recover.

Misuse Case Mar 19, 2024

@flippac @cocoaphony @frankreiff @nicklockwood Yeah, now you have “self-driving” Teslas and these Waymo and Cruise taxis that have problems and exhibit bizarre behaviors that human drivers never would, but get permits nonetheless.

Philippa Cowderoy Mar 19, 2024

@MisuseCase @cocoaphony @frankreiff @nicklockwood We're not going to see Uber running those any time soon: if the car's not responsible for the crashes, the company is and they're big enough to lose their shirt over it

the gripe ibis Mar 19, 2024

@cocoaphony @MisuseCase @frankreiff @nicklockwood haha i immediately thought about uber and the quiet way they distanced themselves from self-driving research.

David, a Bostonian in Tokyo.Mar 18, 2024

@MisuseCase @frankreiff @cocoaphony @nicklockwood @theregister

"That realization and the price jack-up will hit them at the same time."

But the realization won't ever hit them. The bosses are corporate managers. The thing they hate the most is paying salaries. They'll do anything for even the slightest hope they don't have to pay actual people. So they'll never realize they've been scammed.

This round of AI stupidity is going to last way longer than you think.

Misuse Case Mar 18, 2024

@djl @frankreiff @cocoaphony @nicklockwood @theregister Do you remember the thing about the Air Canada AI chatbot that told someone about a refund policy that Air Canada didn’t actually have, and he sued over it and the court said Air Canada had to honor the policy so Air Canada ended up forking over money to him?

1/2

Misuse Case Mar 18, 2024

@djl @frankreiff @cocoaphony @nicklockwood @theregister I think a lot of that kind of stuff is going to happen. And poor implementation of AI will unfortunately get some people killed, with huge monetary and reputational costs for companies because people are already suspicious of AI and won’t buy “it’s not our fault the computer did it.”

2/2

zino Mar 18, 2024

@cocoaphony nah, because quantum computing or whatever the next investment opportunity we didn’t know we needed will solve everything. 😑

vxo Mar 18, 2024

@cocoaphony i for one welcome the eventual enshitification????

Darwin Woodka Mar 18, 2024

@vxo @cocoaphony

No, in this case we're starting straight out with the enshittification

Cecelia Mar 18, 2024

@cocoaphony This has me wondering about the cost of Tesla’s FSD with all of the training it needs to do for the neural nets that run on the cars. We saw the cost increase up to 5x the initial offering and now it’s also available as a subscription service. Not quite the same as a LLM, though they’ve seemed to indicate using a created language for road modeling. 🤔

Darwin Woodka Mar 18, 2024

@cocoaphony they're not going to make money

Troed Sångberg Mar 18, 2024

@cocoaphony They're not expensive "per query". If you run a cloud service doing millions of them for free then maybe fix your business model.

This is easy to prove: Run an LLM locally on your GPU. Compare cost of energy vs, for example, playing a game on said GPU.

Glyph Mar 18, 2024

@cocoaphony thank you, I have felt weird that nobody else seemed to be noticing this particular aspect yet. there are applications (i.e.: fintech, bioscience) where customers will happily pay through the nose because the model is doing something truly useful and difficult to automate with other means, but _so_ many uses are going to be instantly vaporized when users look at the "sign up to get more prompts!" speedbump and go "eh,, it was a neat demo, but I don't know if it's worth money"

Zach Gage Mar 18, 2024

@cocoaphony @grumpygamer @theregister is this accurate about running them?

there are a ton of LLM models in the gpt 3-3.5range that can be run locally (even on mobile devices) and a lot of companies are investing money into specialized chips to make running them more performant.

i honestly don’t understand the expensive subscriptions but i imagine it’s gotta be about the cost of training

Zach Gage Mar 19, 2024

@cocoaphony @grumpygamer @theregister hmmmmmmm seems like the cutting edge models are pretty expensive to run at scale, which makes sense. I agree it seems kind of wild to build a business running on top of licensable ai — it's already too expensive.

Pontificator.OMF Mar 19, 2024

@cocoaphony My current thought is this is a classic dot com boom, rushing to market without having made the product profitable. The only winners I see here, are the ones providing the tools, but I suspect they'll have their own day of reckoning when the income dries up; usually a competitor making a much better tool.

Bastet 魔王様 😈Mar 19, 2024

@cocoaphony
And the geeks simply run their own, even if their home server takes a minute longer to answer, who cares?

M iha R inne Mar 19, 2024

@cocoaphony yep. And I’m amazed how few people stop just for a minute to consider it. Greed turns everyone into an idiot and it is not a pretty sight.