How long until AI truly catches up?

 kw: analysis, artificial intelligence, moore's law, generative language models

I recently heard an expert in artificial intelligence speak of the capabilities of the various "generative language" systems out there. The best known are ChatGPT and its successor GPT-4. Google has Bard and Microsoft plus Meta have Llama-2. I recall it was said that GPT-4 has about 50 billion parameters, and Llama-2 has ten times that many, or half a trillion. These parameters are the "knobs" that are set by the training process, which is primarily a consolidation of billions of text documents.

How do these mechanisms compare with the human brain? (The original, larger version of this image was created by Sarah Holmlund, who holds the copyright.)

We often hear that the brain has 100 billion neurons. The actual number is about 87 billion, plus or minus a billion or two. What is less well known is that nearly 70 billion of those neurons are in the cerebellum, running the body and connecting the body to the cerebrum and the limbic system. 16-17 billion neurons comprise the cerebral cortex, where most neurologists consider our conscious mind to reside, and another 1.5 billion comprise the limbic system and the memory switching centers, including the hippocampus.

I don't know how much fan-in and fan-out there is for the nodes in a neural network (the most common "black box" running AI operations). I suspect is ranges into the dozens or even hundreds, for at least some of the nodes. In the human brain, each neuron is connected to between 1,000 and 10,000 others at synapses, with an average number of about 7,000. The synapse is the key "switch", and a neuron's population of synapses primarily determine its behavior. The number of synapses in the non-cerebellar parts of the brain is more than 120 trillion.

The synapses are not simple on-off switches; they have nonlinear behavior, and they seem to have a certain amount of "decision-making" ability, depending on timing between inputs and the surrounding "chatter" of nearby synapses. However, for the moment we can compare the number of synapses with the neural net parameters mentioned in the podcast. 120 trillion is about 240 times the number of parameters that make up the guts of Llama-2, and 2,400 times the size of GPT-4.

It is fair to say that the entire human cerebrum is not involved in language use, and perhaps only a few percent make up our own language production apparatus. However, the rest of the brain is involved in the knowledge context, in cross-referencing memories, and in weighing emotional responses that arise in the limbic system, for example. AI mechanisms do none of this. So our language "machine" is perhaps 10-100x the size of today's AI systems, but we cannot discount the context provided by the rest of the brain.

Let's project a version of Moore's Law onto this disparity. In its original form, it relates to hardware density: the number of transistors on a single chip doubles about every two years. Ultimate speed was also used for CPU power prediction, but this has pretty much come to an end. The highest clock rate for silicon-based CPU's has been stuck near 4 GHz for twenty years or more. The way newer systems get around this limitation is to put more CPU's (or cores) into a machine, up to 20 or 32 in the most recent chips. The CPU in the machine I'm using right now has 8 cores.

Brain neurons aren't nearly so fast, but they all run at once (most AI neural nets are virtual, with each CPU emulating dozens or hundreds of individual neurons and the hundreds of thousands of connections between them). Signal speed in brain neurons is about 60 m/s, so a neuron that is triggered can signal anywhere in the brain (thousands of anywhere's!) within 3 or 4 milliseconds. Waves of activity, coordinating millions of neurons, produce the Beta frequencies between 13 and 30 Hz. In that time tens of trillions of synapses have each fired between 10 and 50 times. That's a lot of processing.

One projection would be: For the Llama-2 set of parameters to be increased by a factor of ten might require 6-7 years. For GPT-4 to get a 100x boost could require 12-15 years.

However, we must consider that, for any generative AI to produce truly useful text, its creators will need to take the other functions of the brain into account; at the very least, context, memory indexing, and emotional content. Then, if we're in the 240x to 2,400x realm, Moore's law would project a timeframe of around 16 years to 23 years.

I conclude that reaching truly human-level AI performance is possible in my lifetime (I'm over 75 years old), just barely. But there is one more hurdle: What size will such hardware be? At the moment, the AI systems we have been discussing rely on large rooms full of racks of CPU's or GPU's (A GPU runs less than half the speed of a CPU, but can do more overlapping operations, plus it uses a little less power). The electricity running such an installation is a few million watts, including some million-watt air chilling units. That's still a long way from having a human-sized robot walking around with a human-capable brain squeezed into its 50-kg body somewhere.

This is a reach, but with the human brain using 20 watts, how long would the naïve version of Moore's law, applied to power consumption, predict? For a factor of 100,000 in power reduction: 33 years.

I really don't expect to live to an age near 110, but if I do, things could be very, verrry interesting by then.