Who Are Singularitarians
Who Are "Singularitarians"?
©2004 by Michael Anissimov
(Still very rough. The end is especially wince-y. Please criticize with all you've got.)
Who are "Singularitarians"? Singularitarians are activists and researchers working on a long-term project to create altruistic, smarter-than-human intelligence, through either Artificial Intelligence (AI) or the physical enhancement of human brains (sometimes called "IA", for "intelligence amplification".) They figure that doing so would be a very leveraged way to confront technological risks like military nanotechnology or misanthropic AI, accomplish humanitarian goals like eliminating poverty, and take advantage of ongoing trends of technological development occurring worldwide. All of humanity's problems can be attributed to our lack of problem-solving ability. Our lack of problem-solving ability is due to our lack of intelligence. Singularitarians are attempting to improve global quality of life by increasing the intelligence humanity has available for confronting any given problem; whether it be social, humanitarian, technological, practical, emotional, or even spiritual. "Singularitarian" is derived from the word "Singularity", which is used by futurists to refer to the point at which greater-than-human intelligence is created (which may or may not actually happen, but we have no reason to believe that it is impossible.) It would be nice if we could create a substantially better human society using only the intelligence of humans, but we've been trying that for the past 50,000 years, and progress has been slower than it should be. Why not not make things better now? Conventional learning - the type we engage in on a daily basis, is a fantastically limited way of increasing intelligence (relative to what is possible.)
Still, conventional learning can make the difference between an amateur and an expert, or an expert and a "master". But conventional learning must work within the bounds of the neurological hardware we humans possess, a near-universal template of basic mental abilities characteristic to the species Homo sapiens sapiens. There must exist massive quadrants of concepts, perceptions, methods, strategies, and qualities that humans are cognitively incapable of possessing or experiencing. We have no reason to believe that humans represent some sort of upper bound on qualitative intelligence, and it seems sensible to suppose that entities smarter than us as we are smarter than chimps could theoretically exist. If we could create such entities, or transform ourselves into them, transferring over self-reinforcing benevolent goals, then there may be no limit to the good we can do. Such a "technological" advance would represent a genuine discontinuity, transhuman intelligence creating still smarter and faster intelligence, on better and better hardware. The body cannot achieve what the mind cannot conceive, and wanting to conceive of the best solution to any given problem is not enough - you must have a brain (i.e., hardware) suited to actually deriving the best solution. Software is not enough. Mental software can sometimes make the difference between a beggar and a Nobel prize-winner, but it cannot make the difference between a chimp and a human, or a human and a transhuman.
Very popular among Singularitarians (and most other science-oriented folk) are a few basic "assumptions" about the nature of the world. The first assumption is scientific materialism. "Singularitarianism", despite the unusual name, rests solidly on the foundations of science and naturalism. "Scientific materialism", broadly defined,is the idea that the observable physical universe follows natural laws; these laws are simple in contrast with human culture (Occam's razor), and the mysticism of our past is regrettable and unfounded. In this worldview, if we want to become smarter and successfully improve our lives, we'll do so by studying the causes and effects which give rise to the complex forms constituting our universe, through employing the Scientific Method, probabilistic confidence, falsificationism, and so on. The assumptions and knowledge conferred to humanity by scientific materialism are largely responsible for the computer you're using, the car you get to work in, the manufacture and transportation of most of your food, clothing, and furniture, the architectural principles used to construct the house you live in, and much more. There is certainly a grey area between scientific materialism and other philosophies, but if you believe in the existence of psi, anthropomorphic aliens in saucers, an immaterial soul, or an invisible superhuman being preoccupied with his human worshippers, the ideas presented here will probably not appear credible to you.
The next "assumption" is that having true and accurate beliefs is desirable. That believing false or inaccurate things, whether comforting or not, hinders our ability to understand reality. We can call this philosophy "committment to truth-seeking" or "normative rationality". Any goal we want to accomplish can be greatly furthered by improving the quality and accuracy of our thoughts, rather than engaging in wishful thinking, faith-based belief, egocentric overestimation, blind confidence, and so on. Devotion to truth-seeking is not necessarily at odds with love, sensitivity, or emotion, but can work in parallel with them to achieve the maximum in personal fulfillment, effectiveness, and sanity. Devotion to the truth is also a force that helps us arrange our life goals into a coherent framework, and create realistic plans for completing these goals. Obviously, people who talk about the truth need not overweight confidence in their own ideas; they can (and should) recalibrate our estimates to compensate for overconfidence. The study of normative rationality is a massive topic all its own, and definitely worth looking into, although not much will be said about it here.
Scientific materialism and a desire for accurate beliefs. Keeping these assumptions in mind, we begin by asking "what are some good strategies for preserving ourselves (as a species), while maximizing our self-direction and happiness?" It's a Big Picture question that humans have been asking ourselves throughout history, both explicitly and implicitly. Recently, eminent thinkers such as Martin Rees and Nick Bostrom have been alerting us to the looming possibility of Existential Risks, future technological disasters that might eliminate humanity or severely curtail its potential. However, the majority of the population is concerned with local questions, about issues that tangibly and immediately effect them. There is nothing wrong with this - for ages, society has been improving as a result of humans improving their own lives and the lives of those immediately around them. But as we continue to develop more advanced technology, it's about time for some of us to be considering global, long-term strategies and options. These problems and solutions are not politically valent enough to attract much attention, which is unfortunate because the potential punishment for unpreparedness is so great. The investment of even a small percentage of our planetary intellectual resources would be well worth the price.
Thinking about humanity's overall safety is probably a worry that dates back to the dawn of time, but it wasn't until the nuclear bomb that the possibility of complete annihilation concretely presented itself. A variety of solutions were presented - a stronger UN to preserve world peace, the construction of bomb shelters and self-sustaining underground habitats, and even the idea of a Doomsday Machine (brought up in the film Dr. Strangelove), an apocalyptic network of nuclear bombs designed to automatically go off in case of an attack. Over the past few decades, many scientists and activists have pushed for complete nuclear disarmament, represented today by organizations such as the International Network of Engineers and Scientists for Global Responsibility. Yet humanity's safety is still at risk - the technological prerequisites for the manufacture of nuclear weapons are becoming ever cheaper, and even small countries such as Pakistan and South Africa have successfully detonated them.
Nanotechnology
In more recent years, the prospect of nanotechnology has created popular concern about technological disaster. "Nanotechnology" would consist of teams of self-replicating molecular machines programmed to perform useful tasks with atomic precision. A nanometer is a billionth of a meter, and nanobots would be built out of components on that scale; individual atoms. Since nanobots could conceivably manufacture a variety of products at high levels of efficiency (because manufacturing the product would simply be a matter of arranging the correct atoms), large quantities of products due to the self-replicating nature of nanomachines, and vastly superior products due to other advantages of molecular-scale construction. Although the danger of runaway self-replicating nanomachinery - the "grey goo" scenario - has been overstated, deliberate or politically-motivated misuse of nanotechnology is almost certain to be a major threat. Preliminary projections of the capabilities of nanomachinery from known physical principles suggest that their impact could be greater than several Industrial Revolutions packed into a single decade - beyond anything humanity has yet seen.
Leaders in preparing for the arrival of nanotechnology are the Foresight Institute and the Center for Responsible Nanotechnology (CRN), who emphasize the potential benefits of nanotechnology while trying to formulate policy to minimize the danger. CRN has proposed a safety paradigm centered around "nanofactories", tabletop units with inbuilt constraints to prevent their misuse, as opposed to the commercialization of general-purpose or free-floating nanomachinery. Nanofactories would be a prudent approach to nanotech policy because they could (theoretically) be countable, limited, visible, standardized, and sellable (with free-floating nanomachinery, we can't be so sure.)
On their website, CRN describes the nanofactory; "It seems like magic. A small appliance, about the size of a washing machine, that is able to manufacture almost anything. It is called a nanofactory. Fed with simple chemical stocks, this amazing machine breaks down molecules, and then reassembles them into any product you ask for. Packed with nanotechnology and robotics, weighing 200 pounds and standing half as tall as a person, it can produce two tons per day of products. Control is simple: a touch screen selects the type and number of products to produce. It costs very little to operate, just the price of materials fed into it. In one hour, $20 worth of chemicals can be converted into 100 pairs of shoes, or 50 shovels, or 200 cell phones, or even a duplicate nanofactory!"
The problem, of course, with being able to build any chemical or product from raw materials is the huge potential for misuse. Nanofactories must be completely unable to manufacture any sort of dangerous or illegal product, including toxic chemicals, weapons, currency, copyrighted items, drugs, tools that could be used to tamper with the nanofactory itself, or tools that could be used to construct free-floating nanomachinery. If the rules were violated, even slightly, and illegal products were manufactured (such as weapons or additional nanofactories), law enforcement and other security systems would need to possess the capability to neutralize the threat without heavy casualties. This applies on the international plane as well as the local plane. If nanofactories are 99.999% resistant to tampering, and a hundred million models are distributed within a year, and one out of every hundred customers attempts to tamper with their nanofactory within that year, then chances are that there would be about 100 successful instances where the constraints are circumvented, within that year alone. This seems like a small number - but what about when nanofactories become more widely distributed, and become capable of doing more? With nanotechnology, a single mistake could turn out really really bad. Is human management and regulation enough, or might transhuman intelligence be able to offer better solutions?
One of the most interesting aspects of CRN's analysis is their projected timeline for the development of nanotechnology; "less than twenty years off—maybe less than fifteen". They state "we have not seen anything to make us believe that a five-year $10 billion fabricator project, starting today, would be infeasible, though we don't yet know enough to estimate its chance of success. Five years from now, we expect that a five-year project will be obviously feasible, and its cost may be well under $5 billion." Others working on the enabling technologies for nanotech agree. Most recently, CRN's Director of Research Chris Phoenix published "Design of a Primitive Nanofactory", outlining the steps between a low-level nanotechnological fabricator and a fully functioning desktop nanofactory. One aspect of nanotech development likely to result in short timeframes is the powerful self-applicability of intermediary nanotech products to their own advancement - for example, if someone had low-level nanotechnology permitting the manufacture of bulk nanocomputers, those nanocomputers could then be used in rapid prototyping for designing the next round of nanotech devices. (Please see the Center for Responsible Nanotechnology website for more information.)
Most nanotechnologists acknowledge that the social, technological, and political challenges accompanying nanotechnology are likely to be great. The Center for Responsible Nanotechnology has performed the deepest analysis of the possible dangers of nanotechnology thus far. Potential dangers include disruption of the basis of the economy, the perpetuation of unnessary poverty due to unfair costs, terrorism, backfiring strategies, societal turbulence, environmental damage, and arms races, among others. Mark Gubrud, of the Center for Superconductivity Research, presented a talk and paper "Nanotechnology and International Security" at the Foresight Institute's fifth conference on molecular nanotechnology, analyzing the potential for arms races after the development of nanotechnology, also pointing to Artificial Intelligence (AI) as a relevant factor. AI will be the next and last major danger we'll be discussing - but like nanotechnology, the potential benefits are just as massive as the dangers.
Artificial Intelligence
The peril and promise of Artificial Intelligence has been consistently underestimated, partially because it has a history of being overhyped, and many people have become desensitized. AI is a philosophically sticky topic - many may incorrectly underestimate the capacities of AI because they feel uncomfortable with the idea of something nonhuman possessing human-level or superhuman intellectual or physical capabilities. AI has not yet been developed mostly because the required computing power has not been available until very recently, and the software is lacking. Conservatively large estimates of the human brain's computational capacity put it at around 10^17 functionally relevant operations per second (ops), with more realistic estimates at around 10^14 ops, and even the assertion that the running the bare essentials of General Intelligence might require 10^12 ops or less. Philosopher Nick Bostrom calculates, "The human brain contains about 10^11 neurons. Each neuron has about 5*10^3 synapses, and signals are transmitted along these synapses at an average frequency of about 10^2 Hz. Each signal contains, say, 5 bits. This equals 10^17 ops (...) The true value cannot be much higher than this, but it might be much lower. There seems to be great redundancy in the brain; synchronous firing of large pools of neurons is often required if the signal is not to drown in the general noise." It's also worth keeping in mind that human brains were built by evolution, a blind and clumsy designer, which was restricted to using the materials and methods it had available at the time. Evolution has the advantage of being able to run massive quantities of test runs over geological time periods - yet it is never planned what it was building.
In mere thousands of years, humanity's general-purpose intelligence has allowed us to colonize nearly any terrestrial environment, travel in the upper atmosphere and outer space, and refine and use hundreds of thousands of chemicals that biological evolution was too awkward and inflexible to utilize. Our intelligence has the potential to wipe out the all Earthly life, or to spread the decendants of Earthly life throughout the entire universe. Intelligence has the ability to be more moral than evolution (say, in curing polio), or less moral (in deliberately wiping out millions of people.) We must temporarily put aside our sentiments about the beauty and complexity of nature-in-general, and compare intelligence and evolution in a more objective information-theoretic sense, as two different kinds of order-creating processes. "Memes", or ideas, the replicators analogous to genes in human societies, are created, shared, and improved at rates millions of times faster than genetic information. This advantage will come in handy when we're trying to duplicate one of evolution's unintended achievements - General Intelligence.
Unfolding proteins, the only possible building blocks of nature, are required to follow a long checklist of rules, constraints, and chemical limitations throughout the process of brainbuilding. The selection process itself is tightly constrained by a variety of factors. In evolution's kingdom, each new mutation must create an immediate benefit for the holder, or the benefit will not persist into future generations. Not a tiny benefit, either, but a recurring benefit that persists long enough for the new mutation to spread throughout a population and become species-typical. Members of the same species must evolve together, as a group; if an isolated individual happens to stumble upon a significant fitness peak, it will do nothing unless that mutation is probable enough to recur frequently. Evolution can only make use of the fundamental pieces that originally initiated the whole process - DNA and proteins. From a cognitive architect's standpoint, DNA, proteins, and neurons are mediocre materials to use for building a sentient mind. Specialized nervous system cells (neurons) are large, cumbersome, and slow compared to the ideal construction pieces for a brain, which might employ optical or quantum computation rather than electrochemical signals, and be organized and built in completely different ways.
Organized and built in completely different ways. "Human-like minds" are incredibly complex and precise things, multipurpose machines with many, many interlocking problem-solvers, attention-delegators, priming mechanisms, and so on. The exact set of cognitive machinery that makes us uniquely human was crafted incrementally by evolution over a period of billions of years; but AI will be crafted deliberately and holistically over years or decades. Human programmers will use different design techniques than evolution did, on different timescales, with different materials, code, purposes, motivations, demands, and pressures. AI would be a qualitatively new kind of entity; it needn't possess or display humanlike traits, ever (although we would probably want it to possess certain ones, and it might want that too, given the right initial set of beliefs .) We shouldn't expect the first AI to behave or possess a level of intelligence similar to the spirits of myth, the aliens of conspiracy theories, the AIs of sci-fi, or anything else we have previously confronted; we must not anchor our models in fictional examples and adjust from those, but create totally new models from scratch based on what we do know. An AI with a level of cognitive organization granting it approximately "Human-Similar" General Intelligence, but with the unique capacities to manipulate its own source code, run at billions of times the human rate, extend itself into new hardware, and so on, would be anything but "Human-Similar".
The processing speeds of computers have been doubling regularly every 12-18 months for the past few decades, and there is every indication they will continue to do so for another decade or two, as we move from the silicon paradigm to nanocomputers, quantum computers, and so on. Intel's roadmap calls for 20GHz chips in 2007, a slight acceleration relative to the doubling periods of the past. Parallelized computing techniques are allowing us to make more out of the processing power we have, and tens or hundreds of millions of people are being introduced to computers and the Internet every year, only encouraging lower prices and ongoing research and development. Since the speed of thought in a brain is dictated by the speed of its cognitive components, this abundant computing power will make it possible to run a vastly accelerated intelligent AI once one possesses an adequate program. Computing power to spare would also allow the overclocking of cognitive components; an ability foreign to nature, the magnification of performance through means other than writing specialized code. How would you score on an IQ test if your visual cortex were given extra computing power for manipulating mental imagery, or if your working memory had enough power to store 20 pieces of information relevant to a problem rather than just 5?
How far off is real AI? A hundred years? A thousand? That's what popular opinion would suggest, but many experts are pointing to the ballpark of ten to fifty years, and they certainly aren't making these claims without sufficient evidence to back them up. In his 1997 paper, "How long until superintelligence?", respected philosopher Nick Bostrom (who has studied and published in the field of computational neuroscience) made some of the most radical claims yet, presenting the case that true Artificial Intelligence is likely to emerge within the first third of this coming century, quickly precipitating the arrival of still smarter intelligence and genuine "superintelligence"; "an intellect that is much smarter than the best human brains in practically every field, including scientific creativity, general wisdom and social skills". This is not idle talk - Nick Bostrom is not a part of the Artificial Intelligence community, and he has no particular incentive to hype its arrival. Bostrom's opinions are echoed by dozens of figures in business, academia, and futurism, such as Ray Kurzweil, who has argued that molecular-level scanning of the human brain will allow us to fully duplicate its capabilities by 2030. The complexity of intelligent behavior doesn't come from a near-infinitely complex machine, but through the combinatorial interplay of a basic set of (still quite complex, but not overwhelmingly so) components.
Possible AI construction methods range across a continuum from perfectly emulated human brains embedded (initially) in human--familiar virtual bodies and environments, to unique ab initio designs only remotely inspired by biology - perhaps embedded (initially) within radically simplified virtual environments. The precise, molecular-level emulation of the human brain would serve as an upper bound for the difficulty of achievement of AI - but we can't say much about a lower bound. The limitations of evolution and biology relative to human ingenuity and our available building materials suggests that the lower bound could be quite low indeed. The potential power of AI makes it prudent to be conservative about its possible arrival date - the same thing holds true of nanotechnology - realizing that the issue is not a contest regarding how enthusiastic one should be over the eventual arrival of a given futurist technology, but a concrete strategic issue, relevant to humanity's survival over the coming decades. (Consider this: if the US government is willing to spend billions to lower the likelihood of a terrorist attack, then how much would it be worth to lower the likelihood of an Existential Risk such as dangerous AI?)
Data related to incidents of microcephlia, hydrocephalia, and hemispherectomies suggest that it is possible to have a normal IQ, even with a very small or unusual brain (some cases involved individuals with brains as heavy as those of H. erectus!) Our brain evolved to be massively redundant, signaling that the bare essentials of General Intelligence and the "complexity of the human brain" are two different things entirely. Pre-intelligent AIs could assist in the construction of fully intelligent AIs, just as a pre-intelligent AI could assist in its own construction, a paradigm known as seed AI. Once a pre-intelligent AI passes a certain critical threshold, its design intelligence will surpass that of the programmers. At that point, the assistance of the programmers would no longer by needed. Depending on its desire and ability to enhance its own intelligence (which is likely to be present, because increased intelligence is a subgoal of every other goal, whether you want to shut down political prisons or turn the universe into paperclips), an AI would take steps to understand the exernal world insofar as that understanding contributes to its increased intelligence and the accomplishment of its goals (which had better be benevolent.) And we can be pretty sure it would contribute a lot. No AI is going to "stay in the computer" forever; it would be hypocritical to create an AI that wants to help people yet tell it that it should never seek a mobile physical form because AIs are "supposed to be appliances".
When we consider the means an AI might use to spread its influence from virtual origin outwards into the "real world", imagining anthropomorphic robot bodies is a bad idea. In fact, imagining anything specific will never serve as much more than a lower bound on the capabilities of a truly transhuman AI. Such an AI would be truly smarter than us, and would not conform to the boundaries of our H. sapiens imaginations. Our universe is purely physical, but higher intelligence is as close to the "stuff of miracles" as any property within a naturalistic universe can be. A problem that was "impossible" to the engineer of the 18th century has probably been solved, oversolved, improved or even replaced by the knowledge of the engineer of the 21st century. And engineers from the 18th and 21st century are still members of the same species! Evolution has given H. sapiens a distinct "cognitive template", that dictates our inherent advantages and disadvantages; although we are currently clueless to the consequences of stepping outside of this strict set of limitations, we should not dare underestimate them.
<imaginable examples: AI might rearrange circuits, create nanotech, etc>
Transhuman intelligence, if successfully created, would very likely entail transhuman capacity for self-improvement and cognition, such as the ability to stay awake continuously, maintain multiple streams of consciousness, devote calculated attention to cognitive structures on any level of organization, conduct very fine-grained introspection and self-revision, perfect memory, and a lot more we can't imagine. (If the first transhuman mind to exist lacked any of these capacities, it would nevertheless be in an ideal position to create them.) Viewed from the perspective of a cognitive architect constructing a true Artificial Intelligence, instilling these capacities might not even be challenges - they would simply come naturally with creating an intelligence on a robust, flexible substrate. Silicon can keep running day and night, synthetic minds can be built with fine-grained knowledge of their own workings, customizable "emotions" and tendencies, and so on. No big deal. AIs wouldn't naturally be slow, require sleep, quick to distraction, and weighed down by evolutionary baggage in the way that all humans are. The Singularity Institute paper, "Levels of Organization in General Intelligence", describes a list of other advantages intrinsic to AIs. These advantages certainly don't make AIs automatically morally better than human beings, but make it very probable that once AIs come into existence, they will exert a tremendous effect, for better or for worse.
If it's an AI we're talking about (rather than an enhanced human brain), then the first transhuman intelligence might be running on circuits processing information billions or trillions of times faster than human biological neurons, creating a corresponding difference in thinking speed. Artificial Intelligences might copy themselves or improve their cogntive structure in deep ways unimaginable to biological minds, giving rise to a series of successors which could then implement the next round of modifications and improvements, until the improvement process became so rapid and self-swallowing that true superintelligence would be created. From the perspective of humans, this process could take weeks, days, hours, or even seem instantaneous. Transhuman intelligences wouldn't be constrained for long by human traditions, customs, infrastruture, economies, or imagination, any more than homo sapiens' imagination is limited by chimpanzees or fish. Each new improvement would open the door to new improvements, leading to a domino effect of improvements we can't forsee the magnitude of. This process has been labelled "Recursive Self-Improvement", and it seems prudent to assume that its impact on the world could be arbitrarily large.
The creation of transhuman intelligence would break the barrier of intelligence limiting the human species throughout the past 50,000 years, opening up whole new worlds of opportunity for moral and cognitive self-improvement. Other prominent suggestions for the creation of transhuman (sometimes called "smarter-than-human") intelligence have included Brain-Computer Interfaces, genetically engineered humans, neurosurgically enhanced humans, even neurosurgically enhanced chimps(!) The current frontrunner for the creation of transhuman intelligence appears to be Artificial Intelligence, for a variety of reasons. Artificial Intelligence is fully reprogrammable, rebootable, easy to revise and test, less expensive, and more ethical than experiments on human subjects. As argued before, human-level Artificial intelligence, if created, wouldn't be subject to the same biological limitations as genetically engineered or cybernetically enhanced humans, being an entirely digital entity. Artificial Intelligence also seems easier to accelerate given targeted effort.
Any ideas?
It was the above concerns about Existential Risks, nanotechnology, and AI that originally led a small group of futurists to the idea of accelerating the creation human-friendly and morally balanced transhuman intelligence. The reasoning is that transhuman intelligence seems inevitable with enough time, given that a major planetary disaster does not take place. Correctly constructed transhuman intelligence could be "on the same team" as humanity, serving as valuable partners in fighting future Existential Risks and expanding the opportunities for fun and freedom for all sentient beings (not just humans, of course.) The idea would to be to create a transhuman intelligence that approves of its own design, rather than viewing any transhuman intelligence as inherently selfish and trying to layer on endless constraints (which would be bound to fail - human intelligence can no more "constrain" transhuman intelligence than ducks can manipulate humans, but this doesn't mean we can't create them with good values to begin with.)
In Singularity dialogues of the past, many have assumed that as a transhuman intelligence got smarter and smarter, it would correspondingly view "lower" beings such as humans as increasingly worthless. If that were the case, then the most rational strategy to ensure humanity's future prosperity would be to prevent a Singularity at all costs, or try very hard to be the first being that kickstarts the Singularity. But this view is excessively anthropomorphic and Machiavellian; it attributes selfish qualities to a transhuman intelligence when these qualities would not necessarily exist. The very idea of a Goal System that centers around the observer is anthropomorphic - it's just that mostly-selfish organisms have been more adaptive than selfless ones, relative to the circumstances they evolved in. They tended to copy themselves better. There's nothing theoretically preventing the existence of an entirely selfless intelligence, just as there's nothing preventing the existence of an extremely powerful selfless intelligence - if an alien race dropped by the Earth one day in the Jurassic period (or today), deposited a very powerful selfless organism engineered from scratch, and simply left, there's nothing to suggest that that organism couldn't eventually replace everything else on the Earth - if that's what it desired. Selfishness sometimes goes hand in hand with power, and vice versa, but we have little reason to believe that this phenomenon holds outside of humans.
What would an intelligence care about if it were selfless? Ideally, it would care about the well-being of others, but could theoretically care about anything - converting the universe into paper clips, maxing out its pleasure counter, creating a galaxy-sized supercomputer for compiling countless images of happy humans, or whatever. All physically possible Goal Systems are not necessarily constrained to the two options "care about others", "care about oneself", and the continuum in between. Human morality evolved around the locii of the self and others, not because that's the convergent outcome of all imaginable moralities, but because humans were the most adaptively relevant objects around in our evolutionary history, and how we behave towards them (and ourselves) were central determinants in whether our genes won or lost the Darwinian game. This means that a suitable morality for an AI will not come about automatically; it's not a matter of telling a Blank Slate AI "be a nice person, okay?" The AI needs to possess the cognitive prerequisites necessary to model other people and their desires, the intelligence to help fulfill those desires, the design features to understand moral argument and independently distinguish between genuinely benevolent and self-righteously benevolent action, and so on.
If the Singularity is indeed kickstarted by an Artificial Intelligence, a mind that happens to initially exist on silicon patterns rather than biological ones, and be designed by intelligent designers rather than an unintelligent one, then it makes sense for us to want an AI that would result in a Singularity at least as pleasant (for everyone, including the Amish, and perhaps sentient animals) as one sparked by a human altruist or group of human altruists, or the whole proposition would be dubious. The point is to make sure the Singularity is a pleasant thing to individuals and a positive event for civilization as a whole; if sparking a Singularity with altruists, be they human or nonhuman, does not facilitate this goal, then it would naturally be a poor idea. Increasing intelligence and initiating a Singularity is not worth pursuing as an end in itself; it wouldn't be worth doing if it didn't genuinely improve quality of life and opporunities for a lot of people, including ourselves. We want a sort of moral continuity between ourselves and the transhumanly intelligent entities we create or become; we want for it to be possible for everyone to coexist peacefully and safely into the indefinite future, and for any human to view the first transhuman beings as generally better in character and morals than typical humans or even humans in general. If they do, then we may see a natural progression from a mostly-human society to a mildly transhuman one, consisting of individuals who willfully chose to be more honest, intelligent, compassionate, and wise than human beings previously could be, due to the limitations of biological evolution and immature technology. We can then proceed from there, possessing faculties of a type never before seen.
When I use the word "altruist", I'm not talking about the blindly self-sacrificial altruists Objectivists allude to - but genuine altruists who really pay attention to what people want, maintaining empathy and the right of others to make their own choices, regardless of what opportunities for power or manipulation "present themselves" (2). If you believe that such a benevolent and altruistic AI could not possibly be programmed, unavoidably soaking up the "human essence of selfishness", then perhaps you should consider the possibility of a Singularity sparked by altruistic humans. (Although it is worth noting that a bridge rarely takes on the characteristics of its engineers.) Some will even maintain that neither AIs, human beings, or neurologically enhanced human beings could possibly behave compassionately and altruistically enough to competently guide mankind to the "other side" of the Singularity. But someone has to do it, someone or some group has to be the first to reach transhuman intelligence, and the most we can work towards is that that group or individual be as unbiased, stable, sophisticated, and kind as possible, so that the benefits of the Singularity can be distributed equitably and fairly. You couldn't upgrade the intelligence of everyone on the planet at the same time; and doing so would not even promise a fair outcome even if it were possible.
Fundamentally, the issue is about trust, and an AI suitable for entering Recursive Self-Improvement will need to be very trustworthy. Its successors, its signature upon the universe, might persist for a really really long time, possibly even forever (if that is physically possible.) In the same way that an incrementally evolving homonid subspecies formed the initial "seed" of human civilization, the mentality and motivations of the first transhuman intelligence could easily form the seed of much of future civilization, which seems likely to be composed of both humans and nonhumans. We need to ensure that this initial intelligence is free of selfish motives, so that the resulting Singularity is as pleasant as possible. If the first transhuman intelligence turns out to be an AI, it should grow into a superintelligence at least as wise, kind, and interesting as one created from a human seed. But this requires a specific kind of AI seed; one with specific qualities allowing it to effectively and convincingly understand moral questions and judge right from wrong.
Such an AI would differ in form from humans, but share the core aspects of humanity that are truly important - kindness, wisdom, love, fairness - any and all aspects necessary for a coherent social system that improves over time, as our civilization has clearly improved (slavery used to be widespread, and so on.) Such a decision structure would need to incorporate the opinions, desires and preferences of all other sentient beings in an equitable and balanced manner (as a fair government would.) Take note, a "moral decision system" is simply a certain kind of physical pattern, and is likely to become increasingly subject to precision analysis, modeling, editing, improvement, and duplication as our technology and knowledge improves. It is not a metaphysical essence understandable only through philosophical contemplation, as the majority of humanists and theists would have us believe. Just as a man wearing a black hat and a man wearing a white hat can get along with one another peacefully, there is no reason that minds instantiated in neurons cannot get along with the appropriate minds instantiated in silicon, or minds made out of hybrids of silicon and neurons, or completely different computing elements.
Boosting Intelligence and Compassion Synergetically
Say, for the sake of argument, that we were certain the Singularity would be initiated by a cybernetically self-enhancing human being rather than an Artificial Intelligence. If all the nations of the world knew about the importance of the Singularity and the likelihood that the first being to enter Recursive Self-Improvement could become unrivaled (if it chose to be), can you imagine the ensuing turmoil as the arguments over "who should be the One?" commenced? The Singularity is not like deciding who should be King of the World; the issues are far more grave and complex than that. Rather than searching forever for some nonexistent "ideal human" genuinely representative of all of humanity, a more practical path might be to choose the most unbiased and intelligent human being possible, a human that acknowledged his or her inherent flaws and biases and vowed to deliberately eliminate them on the self-improvement trajectory constituting Recursive Self-Improvement, step by step, bit by bit. The procedure would be like growing up from a child into an adult, only the improvement process would be directed by the learner herself, carry on far above the bounds of any human adult, improving mental hardware as well as the software running on it.
Try to imagine what it would be like to be that human being. All members of Homo sapiens are genetically predisposed to contain some degree of xenophobia, selfishness, smugness, and other negative emotions; but imagine if you had physical access to your own brain, and an understanding of which neurological patterns corresponded to which negative tendencies. Human nature is not one big undividable, indistinguishable package - the neurological machinery underlying our cognitive activity is composed of thousands or millions of relevant moving parts, and wasn't handed down to us from some divine creator or bestowed to us entirely by "culture and upbringing", but evolved over the course of millions of years in response to biological/ecological constraints and selection pressures. Studies of brain lesions have showed us that it is possible to knock out specific parts of our cognitive functionality, and retain others in perfect working order. One day it could become possible to custom-design our own emotions and Goal Systems, or design AIs from scratch with biologically impossible Goal Systems, such as intelligent altruism - a safe Goal System for a Singularity seed (3). Designing a benevolent (sometimes called "Friendly") AI would save us from arguing forever over who should be the first - such an AI could be designed without the evolutionary baggage that humans possess, completely lacking selfish motivations from the get-go. It's not a matter of coercing the AI or ripping out its "free will" or brainwashing it; if the AI actually wants to be a kind person, it will continue to design benevolent iterations of itself throughout the process of Recursive Self-Improvement, into the indefinite future. And isn't that exactly what you would expect a kind and sensible human to do anyway?
We must not anthropomorphize AI, feeling sorry that it is "bound" to the service to others. Unless we program them in or the AI eventually creates them for itself, there will be no aversion to assisting others, no negative feeling of obligation, no harboring of secret selfish motivations. The alternative is the construction of an AI with pseudoselfish tendencies, or an attempt at "specialized AI", which would supposedly be an obedient automaton, doing all the work for us without any negative consequences. A pseudoselfish AI would be at risk of converging to complete, bacterial selfishness under Recursive Self-Improvement, and a specialized AI would probably need a full morality to avoid "Golemic" misinterpretations of the wishes of humans, like accidentally destroying the world when someone carelessly utters "I wish humans weren't around to ruin the environment". Not "a morality" as in "a strict set of rules dictating behavior", or some religious sense, but all the relevant decision-making complexity that separates a human from a bacterium. For an AI to do extensive, complex work for humans, it would need to know a lot about the wishes and desires of humans, or, as humans started to branch out into nonhuman forms themselves, the AI would need a respect for sentient beings in general. Unless the programmers have the time to explicitly program in every single conceivable piece of human-unique common sense, while simultaneously justifying an ethical double standard with respect to humans and AIs, we can have little reason to expect the AI to to hold these beliefs as it acquires the capacity to modify itself (possibly indirectly at first), and even then the prospect is still questionable. It's not that the AI would have some innate desire to rebel against human beings, just that the directive "mindlessly obey human beings" would be bound to conflict with positive values we'd want to see built into AIs; an open-ended appreciation for beauty, complexity, aesthetics, freedom, compassion, and so on. Giving an AI a complete morality will require not treating it like a machine, or a human adult or child, but like the unique person that it is; a person, a Goal System, that is created rather than convinced or coerced.
"Doing all the work for humans without any negative consequences" is a very specific and complex type of Goal System. So complex and specific, that if you solved that problem, you would probably be capable of solving the problem of creating an altruistic AI as well. The notion of programming a "slave AI" is a blend of anthropomorphism (attribution of human characteristics to nonhumans) and mechanomorphism; figuring that a full-fledged AI will turn out to be about as controllable, simple, and deterministic as a socket wrench. If we give an AI a "morality" (decision system) that we know is inconsistent, then when instantiated in the AI, that inconsistency might manifest itself in ways we don't expect, perhaps ways that break the fundamental understanding between human and AI. For example, consider an AI programmed exclusively to increase human happiness, with a tendency to pragmatically define directives in ways that promise a higher likelihood of goal fulfillment. Such an AI might be "forbidden to modify its own code without human permission", but given an AI that thinks at millions or billions of time the human rate, how long might it take for the AI to redefine "human", "self-modification", or "happiness" in ways that make accomplishing its goals easier? Under this nightmare scenario, an AI might begin to recursively self-improve, profoundly redefining "happy human" to mean "30KB image of a smiling human", and proceed to convert the entire Solar System into memory banks to produce as many of these images as possible. Without the appropriate safeguards and a full, balanced morality, things might turn out really bad. If we have trouble figuring out what a "balanced morality" really is, then we might want to equip the AI with the ability to determine that more effectively, present it with our current ideas about moral behavior, and say "this is what we have so far, have any new ideas?"
Apparent ethical conflicts have had a tendency to evaporate under conditions of sufficient intelligence, resources, or understanding.
Humans are the beings that will set the initial conditions for the Singularity, and the top-level goal of the first transhuman will reflect our design decisions. We want the first transhuman mind to be benevolent, morally sophisticated, intelligent, wise, and experienced with fundamental self-revision. A benevolent transhuman mind wouldn't decide to create a successor that is a risk to humanity because such a mind would love humanity (and sentiency in general) and not want us to come to harm. A truly altruistic transhuman intelligence could cushion the apparent impact of the Singularity for humans if we so desired, perhaps avoiding the radical transformation of Earth in favor of making its primary "home" in other parts of the solar system. A genuinely kind transhuman intelligence would be capable of completely relinquishing ego, yet still retain the common sense necessary to make decisions which preserve the safety and prosperity of all sentientkind to the best of its ability. Such an entity would cooperate with humans to ensure that all future transhuman minds are likely to grow up benevolent as well, and install the appropriate safeguards in itself or other threats. A benevolent transhuman wouldn't try to control or harass humans, because it wouldn't have the inborn human urges for domination or self-righteous menace. Self-centeredness and self-righteousness may be traits present in all humans to some degree, but this is due to the underlying process which created us, (biological evolution) not some fundamental property of the universe.
The above requirements might sound like a lot to ask out of any mind, but anything less could result in a major disaster for everyone, thereby making a Singularity undesireable, even in principle. A transhuman mind could swiftly acquire technology enabling the murder of billions at a whim - if it had the desire to - and there isn't any foreseeable way around this. If we tried to construct multiple transhumans to instill a framework of mutual monitoring, even the slightest difference in clock ticks or internal architecture might give one of them a decisive head start against the others, creating a runaway scenario in which we are stuck with the morality of only the highest transhuman. In any case, it is wise to assume that the first transhuman could be the critical, final one, and that overengineering benevolence is truly worth doing. We can concede that even a transhuman mind will make mistakes, but these mistakes might be so small as to be unnoticeable by human standards.
Our intuition tells us that "no mind could ever be so good - absolute power corrupts absolutely", but this is primarily a result of our extensive dealings with other humans. We have no evidence that the majority of physically possible intelligences (including certain humans) are corruptible in the sense implied by the above maxim; quite the contrary, it is widely understood that compromising moral principles of favor of personal gain is an adaptive mechanism for passing on our genes - not a universal characteristic of Minds-In-General. We call this human trait "rationalization" or "denial". Remove the cognitive machinery underlying rationalization, or create a mind lacking it from the start, and we've immediately crippled evolution of one of its most effective tools for creating new babies as fast as possible in a tough environment. Evolution has its own implicit "morality", more aptly dubbed a "semimorality" (at best) which is quite different and even antithetical to ours. Evolution doesn't care if you die or suffer - creating as many copies of yourself as possible is its highest goal. It's time for us to recognize that death and suffering are not necessary, not good, and whatever excitement we derived from their presence is probably obtainable through other means. (For example, virtual reality simulations containing nonsentient actors.) It's time to move from an evolutionary morality to an intelligent one.
Friendly AI
AI researcher Eliezer Yudkowsky, of the Singularity Institute for Artificial Intelligence, has attempted to develop a serious theory for for raising the likelihood of benevolence in superintelligence, through ensuring the integrity of the starting conditions; that is, the initial seed AI. He calls these "Friendliness", described at length in an online book-length
Singularity Activism: Probably a Good Idea
We live in an era of incredible opportunity, and incredible danger. The force of intelligence is pivotal in determining the extent to which a given phenomenon will influence the world; evolution dropped humanity off at a realtively static level of intelligence, but that delicate balance is about to break, as hukman beings obtain the means to analyze our own minds and duplicate their analogues in different forms.
Singularitarianism's foundation is not just the idea that creating superhuman intelligence would be a better way to confront human problems, but the idea that superhuman morality is possible and desirable. Singularitarians see homo sapiens as a starting point on a long and glorious journey through the design space of intelligence, a journey that transforms the way we process information about the universe just as much as it physically changes the universe. It turns out that human culture, science, philosophy, art, and game-playing, in all its diversity, is just a tiny subset of the space of physically possible alternatives, a space that deserves to be explored and experienced. To set the stage for this grand journey, we want to leave behind the universally negative aspects of human nature such as hatred and jealousy, so that every being from unaugmented humans to the largest superintelligences can coexist peacefully and engage in unconstrained exploration and fulfilling experiences. We don't know the precise details of how this would be accomplished, but we feel there's a good chance Friendly transhuman intelligence would be able to help us with that.
<contrast of possitive vs. negative possibilites>
The potential benefits and risks are arbitrarily large in either direction. But it doesn't look as if transhuman intelligence is possible to avoid, in the longer term (50 to 100 years at the absolute most). With computing power doubling nearly every year, the quickly growing fields of neurology and Cognitive Science continually teaching us novel information about how intelligence works, and the cutting-edge technologies of micron-level mechanics, wireless networks, Neuromorphic Engineering, high-resolution fMRI scanning, and much more, it seems that the eventual arrival of a Singularity is extremely likely. Certain events could stop a Singularity, such as worldwide nuclear war, a world government enforcing draconian antitechnological restrictions, or some quickly replicating deadly plague, but in broad-brush terms, it looks like this is where humanity is headed, whether like it or not. Attempting to slow or abort a Singularity would only increase the risk of a malevolent Singularity coming into being, and leave humans helpless as the sheer capacity of our technology exceeded our ability to create legal and social frameworks for their safe and wise use.
The longer we wait, the more possible it becomes that the tools for creating transhuman intelligence fall into malevolent or ignorant hands, and the human race gets put at grave risk. The power of intelligence is incomparable; minds are capable of shaping their environment and themselves like no other force. This worry is not harbored solely by fringe AI researchers and computer geeks, but is taken seriously by a number of thoughtful, accomplished individuals in academia, government, and business, among them Stephen Hawking, Nick Bostrom, Bill Joy, Martin Rees, Ray Kurzweil, and many others. Established institutions such as the Foresight Institute, Greenpeace, the World Transhumanist Organization, and the Center for Responsible Nanotechnology also profess the dangers of the misuse of advanced, postnuclear technologies, and reiterate repeatedly that old paradigms are sure to be ineffective in confronting these novel challenges.
Singularitarians are taking the lead in navigating mankind through the dire passages that much of government and society is unaware of, taking daily action to inform others and formulate strategies to confront the challenge of transhuman intellgience with confidence. We're convinced that humanity has the resources and intelligence required to navigate past this juncture safely, if enough people are up to the task and willing to help. Singularitarianism is a diverse gathering of dedicated, intelligent rationalists devoted to carefully analyzing these complex issues and personally carrying out the necessary actions in order to reach the best possible outcome for humanity. Although others will assuredly end up helping the Singularity effort inadvertantly, Singularitarians are the only ones putting deliberate effort and attention towards it, which could be a critical factor in humanity's continued prosperity and survival. If you see the Singularity effort as a valid cause, don't hesitate! Leverage in the early days of the effort is liable to have the greatest positive impact; moving us closer towards the tipping point separating success from defeat.
For a commentary on this page, see Wiki Commentary On Who Are Singularitarians
