Annihilation from Within

indebted to Michael Pillsbury for his insightful and knowledgeable assessment of these Chinese projects.17
Could such a two-nation race for achieving a super-intelligence system lead to the construction and actual use of an integrated brain-computer system, and a subsequent strategic competition between two or more national systems? I have not seen a good case made that such a system could not be developed before the end of this century. Its purpose would be greatly to enrich and expand what advanced computers can do by creating a symbiosis between, on one side, a computer system designed for this purpose, and on the other side, the judgmental capacities and essential emotive functions of the human brain. The contribution of the living human brain would probably not come from one individual “hooked up” to a computer, but from computer linkages to an expert committee or group of policy advisors. Such a symbiosis would be far more advanced than the latest brain-computer links.
If successful, this new intelligence system would exceed the intellectual performance of the best expert group. It would integrate human minds with the enormous memories and calculating and organizing capacities of advanced computers. If and when such a well-financed and focused project achieves its first demonstrable success, the door will be opened to a fundamental transformation of human civilization. In my judgment, the greatest, most profound transformation of the human condition will not derive from the prolongation of life, or from the anxiously debated—and probably vastly overrated—possibilities of human cloning and “designer babies.” Instead, I see an effective synthesis of the computer with living human brains as the agent that will lead to a truly revolutionary upheaval for the human race.
What is at stake in any such synthesis is an increase in intelligence comparable to the step from primates to Homo sapiens. The obstacles to our comprehension of such a world are fundamental, and in the last analysis perhaps insurmountable. We can no more imagine the political order of this new world than a group of chimpanzees in the forest can comprehend what goes on among humans in a nearby village. Whether we should welcome or oppose any such transformation is one of those philosophical questions to which a crisp answer seems impossible. As Ludwig Wittgenstein put it: “Whereof one cannot speak, one must remain silent.”
The idea that a national project might construct an effective super-intelligent system cannot be dismissed as science fiction. This prospect is thrust forward by at least two forces—the continuing progress of brain science (which is widely supported to find cures for brain-related diseases), and the steady advances in computers (which are fueled by the ambitions of powerful corporations). Although progress today is most visible in the United States, we cannot assume that America would prevail over a future adversary in a race to develop this super-intelligence. Despite America’s strength in computer science and brain science, it would have some disadvantages in any such race. One would be the constraints on research imposed by ethical and religious considerations. In the past, neuroscientists have gained critical knowledge by studying patients with major brain injuries. At some stage in the project’s development, a nation’s research might greatly benefit from intrusive experiments on living human brains. Liberal democracies would normally shun such experiments, but ruthless dictatorships would not.
A related but potentially more important inhibition in America would derive from ethical and religious objections to the very pursuit of superhuman intelligence. This might limit the participation of some of America’s best scientists. The legions of ethicists now worried about cloning would suddenly discover they had focused on the wrong issue. Religious organizations would come to regard the quest for a super-intelligence as the ultimate threat to their faith and doctrine—and rightly so. Thus a two-country competition to build the first superhuman intelligence system could turn into a race the most ruthless nation would win.
In Western democracies there would surely be urgent demands that the development of this system be controlled by the United Nations: to “make it legally binding” that it be used only for benign, peaceful purposes, “for the benefit of mankind.” Anyone predisposed to think this might be a feasible policy—a policy that could truly be implemented and enforced—would do well to review the history of biological weapons or to recall the lessons of the nuclear age.
Bio-weapons and Politically Correct Illusions
The most frequently mentioned dark side of the life sciences is their misuse to enhance the lethality of biological weapons. A rich literature is now available on this threat, so I need not dwell on it at great length. The biological weapons that have been used sporadically in the past employed naturally occurring pathogens or toxins—bubonic plague, anthrax, botulism, and many others. Sometimes the effectiveness of these natural agents has been increased—for instance, by converting anthrax spores into an aerosol form. But in the future, a nation or a terrorist organization could employ genetically engineered agents that are far more lethal than natural ones, or that have been made resistant to all currently available vaccines and remedial medications. To unleash an epidemic, a small amount of such an agent would suffice and could be delivered clandestinely—hidden in a bottle, a fountain pen, or pillbox. Add to this threat the possibility that the terrorist might “reload” his means of attack. As former Secretary of the Navy Richard Danzig correctly points out: “Biological terrorism affords the possibility of repeated attack, undermining confidence and forcing ever-escalating investments of resources to achieve a modicum of defense.”18 These frightening prospects prompt people to call for international agreements to ban such weapons.
Unfortunately, the history of arms control is disfigured by an ever-expanding list of broken treaties. So we must expect that international agreements will at best provide only partial protection. On a few occasions, to be sure, a treaty might have kept a vicious dictator from using prohibited bio-weapons; but because he feared retaliation, not because he wanted to be law-abiding. The treaty, in essence, drew a red line that the dictator hesitated to cross. Thus,