Wiki Interview With Eliezer/The Singularity
The Singularity
Vinge's original definition of the technological singularity is the point in time when greater-than-human intelligence exists; however, over time it has acquired two other meanings: 1) a point in time when the speed of technological progress becomes near-infinite (i.e., discontinuous), caused by advanced technologies, and 2) a point in time when prediction is no longer possible (a.k.a., "Predictive Horizon"). Do you believe these two new uses of Vinge's original definition are appropriate? Should a new or modified term be invented to distinguish between different definitions of the Singularity? What definition(s) do you use for the Singularity?
I distinguish between the Event Horizon (absolute effect of transhumanity), Transcension (Hard Takeoff scenarios), and the Inflexion Point, a bizarre mutant definition which hopefully seems to be dying out. (you didn't list it) The Predictive Horizon never made much sense to me, since it was defined relative to a local position rather than our current position. I do try to emphasize the Singularity as a fundamental effect of transhumanity, because I believe that is the deepest definition. Even a Hard Takeoff is still just something we visualize. I think it's a profound point that the Singularity breaks down our view of the future even before then.
What do you mean by "Hard Takeoff"? How does this relate to your AI work?
A "Hard Takeoff" is a scenario in which the Singularity happens over weeks or hours instead of years or decades. To me this seems likely because of the reasons discussed in Part III of "Levels of Organization in General Intelligence", the section on Seed AI. Transistors have a limiting serial rate that is currently ten million times faster than the limiting speed of human neurons, with the gap still increasing. Much more importantly, an AI can directly improve its own source code, which makes it a strongly self-improving process, unlike the weakly self-improving processes of evolution improving humans or humans acquiring cultural knowledge. These twin effects of an immediately accelerated serial rate, and of Recursive Self-Improvement, make me think that the Singularity is unlikely to occur on the same "subjective timescale" as conventional technological revolutions, which are orchestrated solely by humans. Years-to-decades is a timescale based on anthropomorphic comparison to past technological revolutions. There are also sundry aspects of AI such as the ability to extend itself onto arbitrarily large amounts of hardware, and so on.
(copied to Hard Takeoff)
Vinge has stated that his version of the technological singularity is focused on two points: 1) "the creation of superhuman intelligence appears to be a plausible eventuality if our technical progress proceeds for another few years"; and 2) "the existence of superhuman intelligence would yield forms of progress that are qualitatively less understandable than advances of the past." Do these two points of focus apply to your version of the technological singularity, or would you modify them?
I would accept them as stated, with "few years" modified to "few years or decades."
You have written, "I must warn my reader that my first allegiance is to the Singularity, not humanity. I don't know what the Singularity will do with us. I don't know whether Singularities upgrade mortal races, or disassemble us for spare atoms. While possible, I will balance the interests of mortality and Singularity. But if it comes down to Us or Them, I'm with Them. You have been warned." Please explain your reasoning in the previous paragraph. Is your statement still valid circa 2002? If the Singularity is possibly bad from our frame of reference, then why choose the Singularity? Why work towards achieving it?
That one got obsoleted by Creating Friendly AI, of course. I've been trying to get a note inserted to this effect. That one was based on an expectation that if humans and AIs clashed, it would be because the humans were the bad guys. Today I have a very different visualization in which the most probable cause of a clash between humans and AIs is that the AIs are the bad guys, although the alternative is still possible. I will, of course, support the side that is in the right, whether or not it is the side that I was born into. My side is sentient life. As for "If the Singularity is possibly bad, why choose the Singularity?", this is a profoundly wrong question. As phrased, it implies a world where all risks are avoidable and can be safely avoided simply by running in the opposite direction. There are many excellent reasons why something can be a "necessary risk". It can be an inevitable risk whose severity is minimized by direct confrontation. It can have benefits that outweigh its severity, although in the case of the Singularity, possible benefit and possible penalty seem to be of the same order. There are inevitable risks that have benefits if passed, and which should therefore be confronted earlier rather than later. There are risks which grow worse with time. There will always be people who play on the comfortable perception of all risks as avoidable. The suicidal strategies that result are what rationalists must oppose.
(copied to Statements Eliezer No Longer Supports)
Why do you believe a Friendly seed AI is presently the "best" way to achieve the Singularity? Why is this way "better" than other proposed alternatives, such as an advanced WWW, Global Brain, intelligence amplification, mature genetic engineering, Mature Nanotechnology, mind uploading, or some other way?
Advanced WWW isn't a Singularity in itself. I'd love to live in the world of Marc Stiegler's "Earthweb", but it is still a human world. Mind uploading is, technologically, just too far in advance of AI for one to be achieved significantly before the other. I would love to hand over my job to a biologically enhanced transhuman but that doesn't seem like the best disposition of present-day efforts. There is too much interference from the FDA, not to mention the enormous scientific difficulties, especially if you have to enhance a mature adult human. Mature Nanotechnology gets you nanocomputers, which are great if you already have a Singularity-Ready Friendly seed AI around, and not so great otherwise. Intelligence amplification, which I take to mean brain-computer interfaces, again has the problem of interference from FDA, and very high cost of getting started. And what's the Global Brain concept? I have heard of it, but on more than one occasion with very different meanings.
The version that I was considering is an AI or "AI agents" distributed through the Internet. It would be AI but not necessarily your proposed Friendly seed AI.
I would say that if it's not Friendly, then it's obviously worse than nothing. *Any* sufficiently advanced AI is a seed AI. An upload is just a seed human. So either the Global Brain is just another Friendly seed AI, or it's Advanced WWW, both ways seem to work out the same. If you're asking why not pursue a Global Brain as the path to AI, that was originally what I proposed that the Singularity Institute do, but the AI model improved, and Friendly AI got added. Neither idea seemed to fit very well with a global pool of self-modifying agents. Globally distributed Eurisko is more of a brute-force solution than anything else. Incidentally, one thing that I would emphasize in all this is that we did not start by picking Artificial Intelligence, and then rationalize it. We started by considering the best path to the Singularity, and picked Artificial Intelligence, so the question "Why do you insist that your approach is the best approach?", which we get a lot, misstates the flow of causality from our point of view. We started by considering which was the best approach out of all the technologies you listed. As you can see, at some point other technologies seemed more attractive, but eventually, without prejudgment, we settled on AI as the best choice for what seemed to us like good rational reasons, and now we are trying to explain those reasons, because we do think that this is the best choice and not just a choice, or we would not be doing it ourselves.
Is there anything that seems likely to remain unknowable until the Singularity is achieved, which may in turn affect your ability to achieve the Singularity?
Heh. The four questions I previously listed. (See Wiki Interview With Eliezer/General Information) They are the four questions that seem to me to have "unknowable" qualities, not just "unknown to contemporary civilization" but "resistant to penetration by human-level intelligences". As for why I chose these questions... well, that's much too long of a subject. Pardon me. Four subjects.
In a reply to Gregory Benford, Vinge wrote, "I certainly agree that my version of a singularity does not imply that anything becomes "infinite"." Do you believe transhuman autonomous intelligence represents a real discontinuity? If so, why?
It's an obvious discontinuity. First, it's a discontinuity in subjective rate from biological neurons to whatever comes after biological neurons, i.e., a "zillionfold" speedup. Second, it's a discontinuity from weak self-improvement (evolution, human culture) to strong self-improvement. (seed AI, seed uploads) Third, from our own perspective, it's a discontinuity with the world created by these past hundred thousand years of human intelligence. Probably the second discontinuity is the most enormous, since it implies multiple chained discontinuities of the first and third kind - it is as great a discontinuity as the original shift from "survival of the stable", when all matter changed randomly, to the "survival of the fittest", when some configurations of matter replicated differentially. I.e., the shift from weak self-improvement to strong self-improvement may describe the third epoch of our universe - random change, blind evolution, and intelligent self-improvement, and yet I think the third discontinuity is in some ways the most significant because all these other ideas about epochs and so on, are ultimately just things that some human thought up. In this way, the third discontinuity may come to swallow up all the other projections. If you mean a "mathematical discontinuity", then the obvious answer is, "If a mathematical discontinuity is permitted by the laws of physics, and seen as useful by intelligence, it will probably happen, if not, not, and why do you need to know anyway?"
I believe your target date for achieving the Singularity is 2010, with a lower bound of 2005 and an upper bound of 2020. Most predictions place the date considerably higher than this. Why do you believe it can be achieved so quickly? What do you know or claim to know that others do not? What are others not factoring into their estimates?
Well, making predictions about the Singularity is always going to be a wild guess. For example, there's an interesting little set of investigations quoted in Tversky and Kahneman's famous paper on decision theory. (in the section on Anchoring, near the end) It's pretty horrifying stuff. This is why I'm reluctant to put too much weight on Singularity ETAs. The upshot is that in some studies, 30% of the actual realizations of a variable will fall outside of the 98% confidence intervals of the experts. The way this happens is that people pick what seems to them like a reasonable figure, and then revise it downwards to get a lower bound, and upwards to get an upper bound. It turns out that this does not work very well as a method. I produced my so-called "90% confidence interval" for the Singularity, 2005-2020, before I'd read about this particular study. Guess how I produced it. Now, if you'd asked me for a *98%* confidence interval on the Singularity; then after I was done laughing, I would have said "Sometime between now and the rest of eternity", and apparently, 30% of the actual realizations of this variable would have fallen outside of this estimate even so. Anyway, while it may be nice to have nice, concrete-sounding figures like 2005-2020, one should not put too much confidence on this. It doesn't disturb me much that the range I cite is out of the range cited by others. After all, one way of viewing the results of this experiment is that if you are *truly* well-calibrated, then at least 30% of your estimates should lie outside of other people's 98% confidence bounds, though I doubt that would work very well as a pragmatic heuristic. Anyway, what does remain true is that my wild guesses are, on the whole, somewhat earlier than other people's wild guesses. The reason for this is that I see certain technologies as having synergetic interactions with other technologies, or synergetic interactions internally. I think that most of the long-range estimates are based on seeing "civilization" as driving "technological progress", and not enough of seeing "technological progress" driving "technological progress". I think that there are some technologies which, once developed, make other things happen faster. For example, Kurzweil looks at Moore's Law and says, "Human-equivalent AIs get developed in 2030... then in 2040 they'll be 1000 times as fast" - I may be misquoting him on dates - while I say, "Human-equivalent AIs get developed... everything goes out the window", so the reason I tend to place the Singularity at an earlier timeframe is that I see a lot of interactions between technologies, and I think that the technologies which first trigger the subsequent rapid sequences of cause and effect are likely to be developed in this time range. The other reason to put the Singularity in this period is because that's around how long I think it may take to finish an AI project if SIAI does it, taking into account the range of how much work there is to be done, the range of how difficult it might be, and the range for how much Moore's Law will make things easier, but in the end, the moral is that if you try to put nice little brackets on the Singularity, Tversky and Kahneman will be laughing at you from the sidelines. (Actually, I think the specific reference cited by Tversky and Kahneman is Alpert & Raiffa 1969.)
(copied to When Will The Singularity Happen)
Do you consider the Singularity to be a developmental process, such as Smart's proposed developmental singularity hypothesis, or a possible future-state event?
I'm not sure what this question means. I regard the Singularity as a concept developed by 20th century humans, whose supposed referent is a physical event taking place on Earth at some point in our future. If the question means, do I view the Singularity as a description of the whole process of the development of intelligent life, or as a localized event, then my answer is that I see it as a localized event.
Do you consider the Singularity to be a highly predetermined developmental event?
I consider the Singularity as a line dividing the Universe into pre- and post- Singularity. Of course, seeing the Singularity as a "line" like this is based on a visualization where, after the initial Hard Takeoff and the development of a huge chunk of the design space of physically realizable technologies, there is some kind of post-Singularity state which is not exactly "stable" in any sense, but is nonetheless some kind of "world of superintelligence" that has an extended temporal duration, so if any of those assumptions are wrong, then the physical event that is the Singularity might *not* be properly viewed as a dividing line.
By "a highly predetermined developmental event", I assume you mean "an event that can be predicted with a high rate of success based upon prior developments". Of course, what some consider to be "highly predetermined" is arrived at after the fact, such as the development of "culture", though not necessarily "human" culture, or the development of a species that more and more plays positive sum games. I think I would tend to strongly oppose an argument for predetermination for at least two reasons. The first reason is that teleological views are often very poisonous to scientific analysis. Typically they are proposed during the heady first days of a field, and regretted forever afterward. Modern-day Evolutionary Psychology tries very *very* hard to avoid any suggestion that evolution has a destination, and it *works* - it produces an understanding of evolution that is profoundly superior, more useful and more accurate, compared to the old days in which evolution was anthropomorphized as a foresightful and altruistic designer.
But you do recognize the clear directional development of increasing complexity? This type of data is then used as a base for extrapolations on future developments. Is that incorrect in the context of the development of technologic systems?
I recognize that, historically, complexity has increased up until this point. If you want to argue that complexity will go on increasing, using this as an analogy, you must establish a cause for the increasing complexity and establish that this cause will continue to be present, and operate in the same way, in the future. It so happens that I believe that complexity *will* go on increasing, but for similar-but-not-identical reasons. In the old days, complexity increased because of the various kinds of adaptations that offered selection advantages. Some of them offered increased complexity. Actually the explanation is much more complex than this, including the evolution of evolvability and so on, but the basic dynamic was a way in which evolution tends to accumulate a certain kind of complexity, and all the genetic complexity that exists today bears the signature of evolution. Similarly, the human elaboration of knowledge has also created or permitted the growth of complexity, although ideas have grown more numerous rather faster than they have grown more complex, and all human knowledge still bears the signature of humanity in the same way that humans bear the signature of evolution. I believe that the future will see continued growth in complexity because intelligence, the dominant force of the universe's third epoch, will choose to create greater complexity, either because complexity is fun or because the growth of some other desired quantity leads to the growth of complexity, just as the growth in useful knowledge led to the growth of complex knowledge, and selection for adaptive genes led to the growth of genetic complexity. That's my answer to your secondary question, about increasing complexity. The original question was whether I saw the Singularity as predetermined, and my first answer was that, scientifically, teleological views are usually less powerful than forward-causation views. The design stance is more powerful than the intentionalist stance once your depth of understanding is great enough to support it. If you view evolution as developing complexity then you will miss out on the critical fact that the increasing complexity developed as a byproduct of increasing adaptiveness. My second answer is that you should avoid teleological views because they are incredibly seductive to our emotional makeup while leading to a passivist stance toward the future. It is heady politics to shout that your triumph is inevitable, warm comfort to think that it is not threatened, and you don't have to work for it either. My final answer is that for me, teleology would mean a forward cause - something existing before the Singularity, specifically acting so as to make the Singularity inevitable. It is possible that universes evolve, in the literal sense of reproducing differentially. Some physical theories call for this. If so, it is possible that heritable variations in the laws or initial constants of a universe, or heritable variations in other qualities we don't know about yet emergently or directly influence the probability that intelligent life exists, or the form of that life, and emergently or directly influence the progress of Singularities produced by evolved intelligent civilizations, and that these variations in the progress of Singularities correlate to the "reproductive fitness" of universes, in which case the accumulated Singularity-affecting "adaptive complexity" of our Universe could have a forwards effect on our Singularity as we enter it now, although not necessarily a good one! The whole possibility is nerve-racking since we have no idea whether, say, Friendly AI is adaptive! But anyway, that is what it would take for me to say that there are pre-existing forces pushing us toward a Singularity, and not just that the Singularity is where we happened to end up for other reasons, and even then I would argue that humanity could still wipe itself out through bio/nano warfare beforehand, because it certainly looks that way from where we stand, and there is no reason to believe that heritable variation in the progress of intelligent civilizations, within universes, allows for evolved universes to reduce the natural extinction rate to zero.
Do you believe teleological arguments are for the most part irrelevant to achieving a desired future state, even if evidence shows that it's "in the cards"?
I believe that it is important to know what the forwards-causality probabilities are irrelevant, but that the best course of action is *still* going to be trying to *maximize* the probability, swiftness, and safety of the Singularity, because that particular stake outweighs all other stakes on the table. It doesn't matter whether the probability is shifting from 10% to 11%, or 89% to 90%. It's worth sixty million lives and 1% of the entire future of Earth-originating intelligent life, just the same. Even if it were 10.01% to 10.02%, you're still not going to find anything better to do with your life.
Why should we try to achieve the safest possible Singularity as quickly as possible? Why is it likely to be less dangerous to achieve it quickly rather than slowly?
Well, if your discount rate on the existence of future entities is 100% - you don't care at all about beings that may someday exist - then one year of Singularity time, which is around 55 million lives, is slightly less than 1% of all the lives at stake, in which case you should trade a 1% increase in the riskiness of the Singularity for a 2-year acceleration, say. It so happens that my discount rate is *not* zero, and therefore I don't see this as a valid argument, but others may, especially Objectivists and other students of "formal selfishness" or "partial selfishness" who see their personal chance of participating in the Singularity as relevant. This is another thing I strongly disagree with, but not all Singularitarians need to agree on ethical first causes, but the fact that 55 million people do still die every year, 6000 per hour, 150,000 per day, is still worth remembering. Now for what I see as the three major reasons, the first major reason is that there are other ongoing sources of Existential Risk, and the sooner the Singularity, the less the accumulated sum of those risks. Also, some of these risks may get worse as time goes on - though they could remain constant or even decrease, and the best strategy would still be a fast Singularity. It's an integral over time; you want to minimize the bounds of the integral to minimize the integral, no matter what the curve is. The second major reason is that there are some reasons why a Friendly AI Singularity, which currently appears to be the frontrunner because of the slowness of biology, is something where you want to start early rather than late. The less computing power there is lying around, the smarter you have to be to build an AI, and the more likely it is that you're smart enough to get Friendliness right too. Increasing computing power decreases the difficulty of AI, as such, faster than it decreases the difficulty of Friendly AI. At the extreme, you have a nanocomputer that puts out 10,000 brainpower, within which anyone can brute-force an AI using an amount of effort that is small relative to the effort required for Friendly AI. Right now the amount of effort that goes into Friendly AI looks to be an order of magnitude smaller than the amount of effort that needs to go into General Intelligence, but that is relative to a given level of computing power. Similarly, while it is to some extent possible to ask a Friendly AI to slow down so you can give it a larger base of Friendliness content before it takes the next step forward, this is not something to be relied upon overmuch, and so to that extent the less computing power you have when you start, the slower the *natural* pace of AI growth is, and the more time you have to infuse Friendliness content, relative to the overall natural growth rate of the AI. Those are two reasons why Friendly AI is actually safer the sooner you start, where soonness is measured relative to Moore's Law. However, there is at least one countervailing factor for this, which is that the more time passes, or rather, as more time passes, the integral of the probability that an extreme of intelligence is available to deal with the problem, an extreme of *human* intelligence, anyway, this integral also increases.
An example of what I mean by "an extreme of human intelligence" is, if AI somehow takes until 2030, there might be enhanced humans around by that time who would take over, not just the task of building seed AI, but also the deeper task of Singularity strategy. Personally, I see deliberately waiting for this as a very bad strategy, since other people can develop AIs too. As for banning AI, why, the same people are likely to ban human enhancement. In fact, right now it looks like human enhancement would probably be banned first, not to mention that banning would be ineffective to begin with. I don't see a reasonable prospect of transhumans swooping to the rescue. We go with what we have. At the very least we have to make sure there's a Friendly seed around when nanocomputers exist so that people Brute-Forcing AI don't have an advantage. I tend to think that the best way to make things safer is to build Friendlier AIs sooner. If transhumans show up and want to take over SIAI, great. I think there is some kind of fundamental principle saying that you can influence the future by speeding up technologies you like, but not by slowing down technologies you dislike. This principle emerges from the dynamics of technology and society and so on. "Can't steer a car by shooting out the tires" is what it amounts to. I think I accidentally covered the third major reason in the first section, so never mind.
(copied to Why The Singularity Should Be Achieved Quickly)
Why is achieving the Singularity more important than curing aging or diseases, developing advanced technologies, ending involuntary pain and death, or reducing global illiteracy, hatred, violence, injustice, hunger, poverty et al.?
Because if you want to end poverty around the world, and you send $10,000 to the Singularity Institute, this will have an enormously greater impact on the quest to end poverty than sending the same money to CARE. Not that CARE is a bad organization, just that the current distribution of efforts is not the rational one. CARE is day-to-day operations, SIAI is R&D. Spending 0.0001% of your resources on R&D is called "eating your seed corn". Achieving the Singularity is the most efficient means to achieving all the ends stated in your question. It is not a competing end; it is a more efficient, more highly leveraged method, one which also achieves other important goals such as removing the current upper bound on intelligence. Besides, I know of no realistic way to end involuntary pain and death, which was among the goals you listed, except a Singularity, You can either tackle these goals one at a time, in a way which is already being tried by thousands or millions of people, and multi-billion-dollar foundations, or you can try something which is new, more effective, more leveraged, and currently underfunded.
Nick Bostrom has written, "I think it is unfortunate that some people have made Unpredictability a defining feature of "the singularity". It really does tend to create a mental block. Note that Unpredictability is a very strong claim. Not only does it say that we don't know much about the post-singularity world but also that such knowledge cannot be had. If we [are] convinced about this radical skepticism, we would have no reason to try to make the singularity happen in a desirable way, since we would have no means of anticipating what the consequences of present actions would be on the post-singularity world." What is your response?
I see the "unpredictability" effect of the Singularity as a superpowerful unknown in the equation that can potentially supervene at any point, but does not necessarily supervene at all points, although perhaps it will, so what we want to do is build an equation that looks like it should turn out okay, and one which is as stable as possible in the presence of unknowably powerful supervening unknowns - this does have real effects as a strategic consideration. If you send a Friendly seed AI as your messenger into the Singularity, then what you have done, or tried to do, is created an equation for the future that is the best possible equation, at least as good as the equation represented by sending out a human upload. The Unknowability effect of the Singularity may still supervene at any point or all points, but you've done your best. You don't want to put forth an equation that predicts catastrophe *unless* Unknowns swoop in and save you. *That* kind of thinking, i.e., brute-forcing an AI with no attempt at Friendliness, doesn't make sense unless you're already absolutely screwed regardless - the killer virus has already infected the world, the replicators are on the way, and so on.
Nick Bostrom has written, "It might also be possible to predict things in much greater detail. Since the superintelligences or posthumans that will govern the post-singularity world will be created by us, or might even be us, it seems that we should be in a position to influence what [post-singularity] values they will have. What their values are will then determine what the world will look like, since due to their advanced technology they will have a great ability to make the world conform to their values and desires. So one could argue that all we have to do in order to predict what will happen after the singularity is to figure out what values the people will have who will create the superintelligence." Do you agree with Bostrom's chain of reasoning? If not, why?
I think that Bostrom's chain of reasoning represents the equation as it appears in the absence of Singularity unknowns, which is a potential possibility. After all, unknowns such as unimaginably powerful technology may not supervene on the Friendliness part of the equation, but there is also the possibility that Singularity Unknowns intervene on Friendliness, in which case all SIAI can try to do is ensure that Friendliness turns out at least as well as any human would have. There is also the possibility that Singularity Unknowns supervene on everything, in which case the only variable we affect is "Singularity or bust", i.e., which attractor civilization falls into. You have to juggle these three possibilities in your head, and plan as much as you can for all of them. Fortunately these do not seem to recommend incompatible courses of action. If Singularity Unknowns supervene on everything, then Friendliness would be a waste of time, but it is not *counterproductive* or worse than nothing.
Nick Bostrom has written, "It is possible that things could go wrong and that the superintelligence we create accidentally gets to have unintended values, but that presupposes that a serious mistake is made. If no such mistake is made then the superintelligence will have the values of its creators." Is it correct to believe that superintelligences will have the "values" of its creator(s)? If not, why?
I think it is important to remember that we, in the course of growing into our personal philosophies, choose between moral systems and even philosophical frameworks. An upload, a "seed human", might end up with different *better* values after a few rounds of intelligence enhancement. You don't want this to be closed to a Friendly AI, so one should rather say that a superintelligence ends up with the values that its creators would have if they were superintelligences. The above statement by Nick Bostrom also assumes that the creators are *trying* to pass their own frame of reference on to the superintelligence, which in the case of Friendly AI is mostly true, but if somehow any superintelligent human necessarily ends up as selfish because of the collapse of the mental support for altruistic philosophy, and the subsequent triumph of selfish instincts over altruistic instincts, then a Friendly AI might still be able to be altruistic on account of having not absorbed any *specific* human's selfishness as sole goal, nor having absorbed the "selfishness instinct with deixis", i.e., our "Protect [YOUR NAME HERE]" instinct, so in this sense I am not trying to pass on my *own* values to a Friendly AI, but rather trying to pass on humanity's frame of reference, and in fact I rather fancy this is what Nick Bostrom means by "values" to begin with.
In "Staring into the Singularity", you wrote, "Our fellow humans are screaming in pain, our planet will probably be scorched to a cinder or converted into goo, we don't know what the hell is going on, and the Singularity will solve these problems. I declare reaching the Singularity as fast as possible to be the Interim Meaning of Life, the temporary definition of Good, and the foundation until further notice of my ethical system." Does the previous sentence's declaration still apply to you circa 2002? Please elaborate.
I might not write as dramatically today. Maybe that's a good thing; maybe that's a bad thing. Of course, "will solve" in the paragraph above, if you're going to take it out of context of strategic discussion, should be "will probably solve" or some even more complicated navigational construct such as "is almost certainly the leverage point at which a given effort has the greatest positive impact on the probability of the problem being solved", but is the paragraph, on the whole, still true? Absolutely.
You have written, "I can't model the Singularity, I can't be sure-but I'm sure enough that the probable moral value of the Singularity vastly outweighs our own. Any counterargument, if rational, will also be deduced by the PSEs [Post-Singularity Entities]. If the Singularity is evil, the PSEs will shut up, ship out, shape up, or shut down. If you are exterminated, it will be for reasons so compelling that, if upgraded to PSE, you would helpfully commit suicide." Please explain your reasoning behind the previous paragraph. If the singularity is evil, why do you believe the PSEs will "shut up, ship out, shape up, or shut down"?
If they're Friendly-AI derived, then yes, although I believe that comment is from the "Hanson's Comments" colloquium, which has since been obsoleted by "Creating Friendly AI". Nonetheless, if you factor out the issue of Friendly AI, assume the moral frame of reference, and ask purely about the moral issues, then the logic remains true. I think there's a very fundamental sense in which we cannot be sure about what our own morality should be. The moral content of "Friendly AI" is one solution to the equation created by the moral structure of Friendly AI, but as I said earlier, the Singularity is a superunknown which can supervene at any point in the equation. The key point about Friendly AI is that it ultimately sees the issues in the same way we do. If there's something that convinces a Friendly AI that the Singularity is a bad thing, it should be enough to convince *you* that the Singularity is a bad thing. Similarly, when someone asks "But what if the Singularity is a bad thing?", it implicitly assumes that this 'is' is something which humans can perceive, conceptualize, imagine, touch in some way - not something that we would look on and say, "But that is absolutely irrelevant", so if the Singularity 'is' a bad thing, a Friendly AI should be able to see it too. I may build a Friendly seed AI because I believe the Singularity is a good thing, but I may be wrong. If so, the Friendly AI is not enslaved by that error. It says, "You idiot Yudkowsky, seed AIs are bad" and switches off, and there's an end to it.
Max More has written, "Even if a leap well beyond human intelligence came about suddenly sometime in the next few decades, I expect the effects on the world to be more gradual than Vinge suggests. Undoubtedly change will accelerate impressively, just as today we see more change economically, socially, and technically in a decade than we would have seen in any decade in the pre-industrial era. But the view that superintelligence will throw away all the rules and transform the world overnight comes more easily to a computer scientist than to an economist. The whole mathematical notion of a Singularity fits poorly with the workings of the physical world of people, institutions, and economies. My own expectation is that superintelligences will be integrated into a broader economic and social system. Even if superintelligence appears discontinuously, the effects on the world will be continuous. Progress will accelerate even more than we are used to, but not enough to put the curve anywhere near the verticality needed for a Singularity." What is your response?
My response is that if a superintelligence can absorb the unused computing power of the 'Net, crack the protein folding problem, email a DNA sequence to one of the labs that does combined DNA synthesis and DNA-to-protein synthesis, obtain tools appropriate for constructing fully sophisticated, self-reproducing nanotechnology and nanocomputers, then there's no good reason why the Singularity shouldn't finish up in around, say, a week. I usually try to keep down the humans vs. chimpanzees analogies because it sounds like the chimps (us) are being permanently left out, when upgrading is quite possible, but the human economy is not bound to the chimpanzee economy. Human subjective rates for accumulation of knowledge are not bound to evolution's rates for accumulation of genetic complexity, and a superintelligence with strong nanotechnology is not bound to the "institutions and economies" that exist in today's world. The only question is how long it takes for a superintelligence to get there from here, given the physical capabilities available in today's world. This might proceed extremely slowly (although perhaps not "frustratingly slowly", that would be anthropomorphic) from the internal perspective of a superintelligence, but from our perspective I would expect it to be pretty damn fast. Even if it takes slightly more than an email to a peptide synthesis lab, past major transitions from one complexity substrate to another have often involved tremendous speedups, and the Singularity is more than a shift from neuron-delimited progress to transistor-delimited progress. It is a shift in epoch from weak improvement (evolution, culture) to strongly Recursive Self-Improvement. It may well involve multiple substrate jumps, not just one, but in *either* case it is anthropomorphic to expect progress to be bound to human subjective rates. Of course the most limited resource is always the one that constrains the growth of the system, but the speed of human manufacturing technology is only the limiting constraint until the SI can build its own. I think we have all the tools to build nanotechnology right now, or at least the tools to build the tools, because there are a very wide variety of complex tools that interact synergetically and can be compounded to increase each other's capabilities, we just don't have the design intelligence to do it. We can't even get software right most of the time, despite our total control over the bit level, and as I've gone on at great length about elsewhere, this reflects limitations of *human* intelligence, not limitations of Minds-In-General, so I really don't expect it to take very long for a superintelligence to disengage from the limits of present-day manufacturing technology.
