@screwlisp One really interesting, and counterintuitive, thing about neural networks is, a lot of the decisions that can seem important to an engineer from the Before side don't actually appear to matter too much; they can be safely made in a number of somewhat different ways, and the network can stil work pretty much the same way. (Obviously, it'll have to be trained for its distinct architecture, but it can be trained on the same data, and it will work largely the same way.)

This weird phenomenon is one of the reasons why many people suspect that the things we particularly associate with human brains, most significantly, the subjective consciousness, might be able to emerge in a variety of networks of rather variable architectures, provided that their elements have certain foundational properties and that the networks are large enough.

We don't quite know what the critical properties are until we get there, though. My hunch is, the artificial neurons we currently have might be sufficient, but we're probably at least six orders of magnitude of computational capacity away from a primate-like CNS to become feasible to emulate. We might need less neurons if we made them more complicated, or possibly, if we figured out the how and why of neuronal migration in vertebrate brains.

OTOH, there's some very interesting kinds of non-vertebrate brain architectures in the nature, architectures that are much more efficient in their use of neurons. My favourite example is jumping spiders. For some species, it can be experimentally proven that they can process input comprising of millions of bits, and solve complex problems as the ethologists understand the concept, in brains comprising of only a couple tens of thousands of neurons. A couple of species have brains of less than ten thousand neurons, and still do complex behaviours.

It is not yet known how they do that, but it seems likely that mammalian brains can not do what jumping spiders do with the same neuron count. In part, well, because scientists can actually grow slices of rat brains on silicon, and we have some hunch about the complexity-of-behaviour-density that these can reach. The critical difference is not necessarily in the architecture of individual neurons, though; it is possible that the jumping spider brains have more detailed genetic architectures whereas mammalian brains have kind of been optimised for generality, with relatively few genetically built-in specific patterns. This high degree of flexibility is likely relatively rather wasteful; we only have it because dinosaurs without it used to die of #FutureShock when the world started to relatively rapidly change.

We understand some basics of how genes encode, and implement the general body plans of creatures. The best-understood part of this is the Hox, or Homeobox, gene network; it exists on pretty much all Terran creatures with a bilateral body symmetry at least in some part of their l ife cycle (there's some creatures that are only temporarily bilateral), and the fundamentals are very highly preserved. Somewhat simplifiedly, on the longitudional body axis, the body plan develops as a sort of chemical interference pattern, with genes to build individual organs, in the first approximation, activating on the basis of very specific ratios of growth factor protein levels.

It seems likely that some basic brain structures are encoded in somewhat similar ways. We do see the Hox genes' involvement in the development of the neural tube, but scientists current understanding of how this affects different brain architectures is fairly limited. #MoreResearchIsNeeded.

Some other interesting invertebrate creatures with much-more-efficient-than-mammalian brains are some molluscs, particularly octopi, and praying mantises.

As other vertebrates go, birds have brains very different from mammals, of (very roughly) comparable neuron counts, but synaptically significantly denser, and organised so differently that only a couple of decades ago, some neurologists would, with straight faces, argue that birds can't think since they don't have neocortices. Well, turns out, some birds manage to think well enough without a neocortex, and thinking that one is required for thinking is effectively an exercise in mammalian chauvinism. But we understand avian intelligence even worse than we understand mammalian one, and mammalian intelligence we understand very poorlly to start with.

On the third hand, the human way of growing brains that can do language appears to boil down to a very small number of specific 'root' gene alleles. Of the known ones, FOX2P is the most likely one involved; the most likely one to distinguish human speech from other apes' linguistic ability. Knocking it out in humans is associated with specific cognitive and linguistic defects; transgenic mice with human FOX2P become very 'chatty' (but, well, we can't yet tell if there's meaning in their chatter). We don't know what a transgenic chimpanzee with human FOX2P might sound like; scientists could arrange one, but ethicists are concerned as to whether it should be done.

A catch is, the FOX2P protein is not anything directly structural; it's a transcription factor. It up- and down-regulates dozens, perhaps hundreds, of other genes' expression. It's probably involved in representing detailed brain structure through some combination of chemical interference patterns that we can't yet interpret.

But the potential fact that a relatively small change to a high-level control gene might be able to turn complex speech capability on and off tantalisingly suggests that understanding how this works might allow ANNs to do 'true' speech, not the stochastic parroting that LLMs do.

On the fourth hand, maybe we're understanding it wrong, and what human FOX2P does is structurally what LLMs do, and the problems of LLM parroting are just that LLMs are missing other crucial parts of brains needed for cognition. Maybe LLMs would be smarter if they had neocortices? Pity that nobody knows how to build one.

Hallucinating up things that should come from parts of brain that are missing, unavailable, or knocked off, is a known phenomenon in biological brains, after all. Based on what we know, this is likely one of these emergent phenomena of Sufficiently Complex Neural Networks that LLMs and biological brains do in a relativel similar way. In clinical neurology, it's called 'confabulation'; one of the most striking examples is the Anton–Babinski syndrome in which case a person is blind because of brain damage, but the damaged visual cortex interface confabulates up enough of fake visual input that the patient adamantly and genuinely believes that they can see, even though they can't. (Confusingly, because doctors don't think like engineers, the syndrome can also cover situations in which a patient does not necessarily feel they can see, but argues it anyway, as long as they seem to believe their confabulated reasoning for why they can see even though they keep failing vision tests.) The full syndrome in one of its two main 'pure' presentations is statistically rare, but a curiously recurring condition associated with focal damage to specific parts of the visual cortex, and possibly subcortical layers. (Doctors have mapped out the specific regions whose damage can cause it, but because it's a rare condition, we don't know too much about the specific kind of variance that differentiates between Anton—Babinski and the kind of vision loss that a patient can clearly perceive.)

A well-known example of brain confabulating up visual input is the invisibility of the macula lutea. Right in the middle of eyed vertebrates' visual field — not at the very centre, but usually close to the centre — is a region where the optical nerve attaches, and shadows a substantial part of the field of vision. Yet, virtually all seeing humans' brains are inherently configured to not see that hole in the field of vision, and to just Make Something Up(tm) when trying to peek into that part; the mechanism this works by is the very same confabulatory expansion of patterns. We don't quite know for sure, but based on what we do know, this phenomenon is likely universal among vertebrates with eyes.