@Jyoti FWIW: I'm working on something like that: models with increased sample efficiency that learn actively, like humans. Also, I'm hiring!

https://www.linkedin.com/jobs/view/3801462876

@epiceneVivant @macindahaus @doctormo All this is (obviously) true. Yet: bitcoin miners are within the same factor of ~100 of the Landauer limit as the brain. So, it's possible in silicon; you just need the right algorithms, computational structure & connectivity. (E.g. wiring in the brain is much more efficient as it's 3D)

@m8ta When you say "bitcoin miners are within the same factor of ~100 of the Landauer limit as the brain", what does that mean and how is it a relevant response to what I said?

I gave it a google but it seems like there's some dense thermodynamic math at work there. Physics never was my strong suit.

@epiceneVivant There is a theoretical limit to making or destroying bits -- e.g. doing computation -- that's set by thermodynamic limits. I've seen estimates (in a very hand-wavey way: we don't really know how the brain works) that the brain is within 2-3 orders of magnitude (>100x) of this limit.

Bitcoin miners, which are doing a much simpler computation, seem to be approaching this energy efficiency.

So there's no reason Si AI can't be efficient (in the future). https://www.lesswrong.com/posts/mW7pzgthMgFu9BiFX/the-brain-is-not-close-to-thermodynamic-limits-on

@epiceneVivant Hah, probably not.

Let's do some calcs: Landauer limit is 0.02 eV. ATP is 0.3 eV. Typical cells consume 1e7 ATP/sec [1].

[2] pegs neuron bandwidth around 100 bits/sec; assuming processing is 10x that (subthreshold activity etc) yields 1e5 x limit!

[3] Estimates the H100 at 17e3 eV/bit, which is 1e6 x limit.
(they note energy cost is 1/10th the purchase price)
¯\_(ツ)_/¯

[1] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4230611/
[2] https://mitpress.mit.edu/9780262181747/spikes/
[3] https://arxiv.org/abs/2312.08595

@m8ta This means nothing to me.

You still haven't answered my second question, "and how is [that] a relevant response to what I said?"

I wrote a metaphor about how unachievable the computational resources needed for AGI likely were.

You've responded with a thermodynamic principle that can't possibly be relevant and, now, some equations and figuring on the same topic.

This is *still* a non sequitur. A 3rd post about physics minutiae will continue to be one, fyi. Try something on-topic maybe.

@epiceneVivant

> Yeah whenever I see some evangelist prattling on about AGI I think about how bigtime LLMs are currently using about the maximum amount of resources any computing project plausibly *could*.

[...]

I'm saying, in terms of energy, (surprisingly) the brain and 2023 CMOS are rather similar. Algorithms & communication & topology are blocking, not efficiency. See @albertcardona Albert Cardona's post.

@m8ta Thank you.

@albertcardona

I think where me and you two differ is in the belief that theoretical considerations about what AI technology *might be like someday* are a meaningful counterpoint to a pragmatic observation about what AI technology *is currently like right now*.

In our sci-fi imaginings a human-sentience-equivalent AI is compact enough to fit within the computer system of a ship (Star Trek, Mass Effect) or even within a handheld device (_Her_).

@m8ta @albertcardona

So far the first big leap in AI capability, bigtime LLMs, can barely fit inside the biggest data centers we have.

I'm aware that much smaller implementation of LLMs— which I'll be studying later this year— have far more limited resource needs. So the observation I made pertains to ChatGPT, Bard, & co., but not to LLMs as a whole. I don't know enough about LLMs as a whole to grok the resource consumption curve. Get back to me in like April.

@m8ta @albertcardona

LLMs with the performance characteristics of ChatGPT or Bard are new as of last year. This is 1st-generation stuff. One can postulate that the training corpus needs might be brought down over time, to something a supercomputer could get done over a weekend. And I don't know much about how computationally expensive *a single execution of ChatGPT is*. (Again, get back to me in April.) Or what the potential is to bring that down to something a phone could handle.

@m8ta @albertcardona

Notice all my reasoning comes from CS & the practicalities of algorithm implementation. It's my bailiwick; but it's also nuts-and-bolts engineering. Looking at this implementation, it's 1st-gen, what might 10th-gen look like? Straightforward and grounded.

When you can debate the future of AGI implementation based on *current real-life AI engineering issues*, get back to me. This is a programming problem. Beyond data center heat dissipation, thermodynamics isn't relevant.