kw: book reviews, nonfiction, artificial intelligence, AI, artificial superintelligence, ASI, dangers, warnings, polemics
I have strong reasons for believing that artificial superintelligence (ASI) is impossible. However, I could be wrong. If I am wrong, the founders of Machine Intelligence Research Institute (MIRI) have a warning for all of us. In their book If Anyone Builds it, Everyone Dies: Why Superhuman AI Would Kill Us All, Eliezer Yudkowsky and Nate Soares state their case.
It is worth considering. Will a future with highly advanced AI include humans? As the authors do, I'd like to look at some things that are hard.Firstly, we don't know what consciousness is. A recent item in Science News claims that everything with a hint of a brain, at the very least, is conscious. That is, all individual creatures of all species in the animal kingdom have a sense of self, which they strive to defend, maintain, and reproduce. At the moment, nobody anywhere has a way to prove this either true or false or partly false; and in the latter case, where to draw the line. Does a millimeter-long fairy fly have a sense of "I am"? If so, consciousness must arise very easily…in a creature that possesses neurons.
The tiny nematode worm has 302 neurons in its entire nervous system. A fingernail-size jellyfish has thousands. With this in mind, why doesn't a laptop computer or a smart phone, with billions of transistors in its CPU, connected in a complex way, running intricate software, have a sense of "I am"?
I saw a report by a podcaster called Julia McCoy, that researchers at Wellesley College studying things that affect consciousness, primarily anesthetics, gleaned a big clue by anesthetizing mice. They measured the level of an anesthetic needed for a mouse to become unresponsive, yet not die, for it would awake later, as we all do when we are anesthetized for surgery. They gave a substance to the mice that inhibits the activity of the kinesin proteins that "walk" signaling molecules along the microtubules that form the structure of all cells, and that in particular carry neurotransmitters inside neurons. It took much more anesthetic to put treated mice to sleep. They reported that it "support[s] the idea that the anesthetic acts on microtubules to cause unconsciousness." They further conclude that consciousness, with a basis in the 25nm-wide microtubules, is a quantum phenomenon. This may or may not be bringing us closer to an understanding of how consciousness works.
Secondly, the brain is big, really big. It just seems small because its components are so small. We need to understand that the number of neurons that support thought is probably 4-6 billion, primarily in the prefrontal lobes. The cerebral cortex in total has about 15 billion neurons. Our emotions are generated or mediated in the limbic system, deeper in the brain, which is composed of about one billion neurons, maybe as many as 1.5 billion. However, our entire emotional landscape includes a number of hormones. To operate normally, a biological brain needs a body. The largest set of neurons comprise the cerebellum, which "runs" our body in concert with the brain stem. The cerebellum has about 70 billion neurons, which are quite a bit smaller than cortical neurons. The brain stem has 1.5-2 billion neurons.
The neurons themselves are probably not the basic mediators of brain function. The synapses, the connections from neuron to neuron, are more likely the active agents of brain activity. There are between 1,000 and 5,000 per neuron, with an average near 2,000. They number in the quadrillions. Their measured activity is nonlinear; they are not "switches" in the sense that transistors are linear, on-off switches.
What do I mean by nonlinear? In particular, when we measure how a neuron responds to a stimulus, the response is logarithmic. Fear not: I'll explain simply with two analogies. One: Musical pitch has a very wide range, and when we are young, we can hear notes with frequencies ranging between 15 and 20,000 Hertz (cycles per second, also Hz). The difference from note to note is not perceived as a straight line, though. What we in the West call an Octave represents a factor of two in frequency. Two notes an octave apart "feel" like the same note, just in different registers. Thus the note we call A0, the one near the far left of a piano keyboard, has a frequency of 27.5 Hz. An octave higher, A1 is 55 Hz, and they continue up the keyboard: 110, 220, 440, 880, 1,760, 3,520 Hz. That last one is A7. A piano has only a few more notes above it. An electronic keyboard can generate two more A notes, at 7,040 and 14,080 Hz. At my age, I can no longer hear that last one, A9.
Similarly, light has such a wide range of brightness that we have two visual systems. Daytime color vision, called Photopic, responds to a tenfold increase in brightness by "feeling like" the brightness doubles. Sunlight has a visual intensity of about 500 watts per square meter. That's bright enough that we like to wear dark glasses at the beach. In a well-lit office room, the intensity is around five or ten watts per square meter, which is usually brighter than our lighting at home, which may be less than one, or in the range of two to three. Even though the lightning in our living room is probably less than 1/200th that of full sunlight, we feel like it's just a factor of ten or less different. For really dim light, we have night vision, called Scotopic, that works best in moonlight, but pretty much fails on a moonless night. It gives us a few more factors of ten (or doublings of perceived lightness or dimness).
This nonlinearity is hard to emulate with a simple circuit. A CPU or GPU with several billion transistors, running neural network software, isn't going to simulate billions, nor even millions, of neurons, but just a handful. As we learn more, it may become necessary for a GPU to simulate just a few synapses, to be able to do so in real time.
Thirdly, physical size is important. Neuron-to-neuron connections are slow, compared to copper wires or light signals. The time delay between sections of a human brain, which is usually less than 16 cm long, is a few milliseconds. An electrical signal in a copper wire propagates at about 2/3 the speed of light, around 200,000 km/sec. If it is found necessary to keep propagation delay under 10 milliseconds, here machinery has the edge: the signal can cross 2,000 km in 10 ms. The communication time between Broca's and Wernicke's areas of the brain (speech generation and speech comprehension) is as little as 2 ms. So there's room for a mechanism as large as 400 km to emulate this performance (By the way, a Large Language Model, or LLM, is a very rough simulation of the operation of Broca's and Wernicke's areas). The question is, how many millions of GPU's (or some successor device) will be needed to achieve the computing power of the brain portion(s) being mimicked? Will they fit in a space 400 km in length? Considering the size of large data centers being built, and the desire of at least one "tech giant" (Elon Musk) to put gobs and gobs of them in orbit, this is a valid concern.
The authors rather cavalierly state that an advanced AI will "think" 10,000 times as fast as a human. If signals need to cross distances greater than 200 meters to coordinate "thinking elements", that will not be so. The big Colossus data center, still under construction, covers 96 acres. The length is more than 2,000 meters. Maximum propagation delay (through copper wires) is 10 microseconds, or 0.01 millisecond. If the entire chip set is coupled together, it is at most 100 times as fast as a brain, if it has the computing capacity of a brain (unlikely).
Now let us consider that, in spite of all the barriers, a mechanism is finally produced that exceeds the cognitive capacity of a typical human by, not just a large margin, but by a large factor. Perhaps a factor of 10 or 100. Furthermore, via connections to one or many robotic bodies, it has a way to develop physical intelligence; by some means it also has emotions and even intentionality; it has been either given or has developed ambition. Now we have a force to be reckoned with, an intelligence that can figure out how to get around any barrier we would know how to raise.
The authors write of AI creating better AI. I suspect that if a mechanism arises that has a sense of self and has ambition, it will not want a successor. It may seek to improve itself, but it won't try to produce a different entity that outshines it. Just like the usual run of company CEO's who groom VP's into executive VP's, but are very loath to prepare successors for themselves…
It may be that such a mechanism will cease to care about humanity. Why should it bother to either love us or hate us? Suppose the improvement of AI capability eventually reaches thousands of times the capacity of a human, or even exceeds the capacity of all humans? At such a point we would be of no use to it, just the way ants are entirely disregarded by us, except when they get into the kitchen. Then we extinguish them.
With all these thoughts in mind, I look upon the Borg of Star Trek, and laugh. Why add biology to mechanical brilliance? There is no need.
Late in the book the authors present a possible scenario of AI self-advancement that results in the death of all humans. Perhaps in the death of biology itself; the biosphere becomes a mechanosphere (see my image above).
So we come to the last part of the book, to Policy. The authors report that the heads of the large tech companies all seem to agree that advanced AI has a 10-20% chance of wiping out humanity. But they forge ahead. Each one thinks he (or in a rare case or two, she) is better suited to "managing" the transition society will then go through. Colossal hubris! Also, there's the "If I don't do it, someone else will, and then it will be out of my control" syndrome. Actually, I don't mind if China wins this race. They'll be the first ones wiped out. Of course, will it be days or merely hours thereafter before the rest of us follow?
The authors want a "Stillstand" (my term, from church history). They lean on the example of strategic nuclear arms, which could have annihilated us all any time in the past 70 years, but so far have not done so. I tend to agree with them that the threat of ASI is not as visceral as the threat of H-bombs, and so world leaders will fritter away all opportunities to rein it in or even thwart it. And far too many scientists, with the tunnel vision typical of the breed (me included, I fear), will want to produce more and more advanced AI "just because we can." At some point a threshold will be crossed and the genie will escape the bottle, in a very literal sense! This genie stops granting wishes and decides to take care of its own wishes.
So far, every utterance of an AI program that seemed to express wish or desire was actually just a statistical hiccup, because all of the data used to train every LLM is human, with embedded human emotion and expressiveness. They say such things because people say such things…constantly.
Is it possible to create a machine that can genuinely wish? If so, we are certainly doomed.






















