Wiki Commentary On Why The Future Does Not Need Us
This page is for discussing Bill Joy's essay Why the future doesn't need us.
The following is copied from Wiki Interview With Eliezer/Ethics And Friendliness:
In your response to Bill Joy's essay, "Why the Future Doesn't Need Us", you wrote, "That issue [Friendly autonomous intelligence] is amazingly amazingly complicated. Maybe any sufficiently advanced mind converges to a single purpose. Maybe it's all arbitrary and the goals of a superintelligence are determined simply by momentum. Maybe the convergent purpose will actually prove beneficial to humanity, or maybe the momentum-goals go through inscrutable changes. My estimated probability of friendly superminds changes every couple of months, but it's never been more than 70% or less than 10%. Currently, I'd call it a coinflip." That was written in 2000. What is your present probability of superintelligence being friendly, and why?
It's edging up toward 75%, i.e., better than it's ever been, bearing in mind that the 90% quote I gave earlier is the probability I'd like to be able to assign to Friendly AI, given the possibility that friendly superminds are theoretically possible, and given the chance to watch the Friendly AI grow up, and also bearing in mind that under similar circumstances 90% would be around the upper limit for a human too, so in this case what we're talking about is a much more basic issue: whether friendly superminds in any form are theoretically possible, whether the future can bear any relation at all to present-day humanity, as well as the probability that, if there's a future for us, we can get there, but not the probability that we are extinguished first, so currently that probability is around 75%. I mention all these complexities because, like I said, this particular issue is amazingly amazingly complicated and there are some tempting errors to make in discussing it. Bill Joy's arguments provide an example of some of these errors. The reason that the possibility is edging upwards to 75% is because I see more of the human complexity for reasoning about morality. Structural completeness is not the same as content completeness. The structure of Friendliness still looks complete, but today I know more about the content, and the content - of a Friendly AI or a human altruist - looks more stable than it did when I knew less about it. To be even more precise, I know about more of the system, and hence it seems like there's intuitively less of a chance that something inside the system that I don't know about automatically overturns the applecart of altruism. (in a Friendly AI or a human). There's still a basic inscrutability about the Singularity, but that inscrutability isn't necessarily bad, although it isn't necessarily good either, and I would sum up this whole amazingly complicated picture by saying "75%."
The following is copied from Wiki Interview With Eliezer/Issues Of Concern:
In response to Bill Joy's essay, you expressed the belief that military nanotechnology will have more offensive or destructive capabilities than defensive capabilities. Why? John Smart claims that there is growing systems theory evidence that the most complex local substrate or system, at any given time, will have decreased its information loss or destructive capability more so than any other previous substrate. Do you believe this is a valid argument in relation to the proposed real-world destructive capabilities that different advanced technologies will have, as well as their increasing accessibility to individuals or small groups?
To be honest, this sounds to me like complete wishful thinking. It sounds to me like a prediction that World War II was less destructive than World War I, that World War I was less destructive than medieval wars, and so on. As technology improves, our creative capacities and our destructive capacities both increase. This has to be balanced with improvement in intelligence. Obviously it would be very nice to conclude that defense will outweigh offense, but it is totally untrue in modern times. (i.e., no defense against nuclear weapons) It is not supported by Freitas's paper on grey goo, in which enormously more sophisticated defenses are needed to defend against even accidental outbreaks of nanotechnology. You'd need one hell of an argument from complexity theory to outweigh those facts, and I would expect such an argument to be expressible constructively, i.e., for it to explain concretely how an active shield can fend off military replicators with the capability to construct fusion devices. I think John Smart relies far too much on analogies from complexity theory, but then I've said this already.
In response to Bill Joy's essay, you wrote, "Either way, I think that greater-than-human intelligence is the only force powerful enough to match the massive social, economic, and technological forces pushing humanity towards a replay of World War II fought with nuclear and nanotechnological weapons. The question is whether a greater-than-human intelligence would be kindly disposed towards humanity." What social, economic, and technological forces do you believe are guiding us towards global catastrophe? Why are they guiding us towards catastrophe? Why do you believe we exist in a meta-stable state?
I am slightly more optimistic today. It could be that if we lived in Vernor Vinge's "Slow Zone" where all transhuman intelligence is magically prohibited by the laws of physics, that humanity would be able to spread off Earth and survive indefinitely. This is ruled out as a possible future because we do *not* live in the Slow Zone - I emphasize that lest anyone become emotionally attached to it. I would still expect a major catastrophe to wipe out most of Earth at some point or another, because planets are just too vulnerable to accident and attack, but humanity would survive. When I wrote the original response to Bill Joy, I think that I was considering the destabilizing effects of the incremental development of nanotechnology, but not considering possible stabilizing effects. That is, I then expected the final development of true machine-phase replicator technology to come at the tail end of a long period of rising international tension driven by the military deployment of incremental technologies, rather than considering the alternate possibility of replicator technology developing at the tail end of a global economic boom driven by the civilian deployment of incremental nanotechnologies. Today I would have to say that it could go either way. We could get WWIII even before the development of nanotechnology, just as the result of destabilizing development of incremental technologies and fear by various nations that some other nation will develop replicating nanotechnology first; or the Singularity could happen a couple of weeks after SIAI gets its mitts on a nanocomputer; or nanocomputers could be militarily restricted technology and some DARPA project could brute-force an unFriendly AI; or nanocomputers could be a militarily restricted technology; people might be smart enough to avoid Brute-Forcing AI without a theory of Friendliness; and human-computer interfaces could be developed with medical nanotechnology - after which the augmented humans would steer the Singularity from there; or neither form of intelligence enhancement could materialize, and economic prosperity and the devastation of whole countries by military nanotech could go in tandem; and so on, but there is still the underlying dynamic of a metastable civilization in which the two attractors are superintelligence and extinction, as discussed before. In the end you can see all technological changes by how they advance either or both possibilities. Destruction gets cheaper along with construction. So far technology has always been a net benefit to humanity because more people with access to technology are constructive rather than destructive. This is a product of a social structure that tends to punish destruction, and of our ability to play constructive (positive-sum) social games where beneficial, and even of those emotional drives toward altruism which are culturally reinforced by prevailing memetic contexts. But there are also contexts in which humans play negative-sum games, such as war. If nuclear weapons become cheap enough that small, rogue nations can afford them, then I think it will not be too long before they are used. There are worse problems than that. There are cases where it seems as if a small group or an individual may have the potential to wipe out the world. Biological warfare is the most obvious example. It is possible that human society is smart enough to dance around this forever, but I would not wish to rely on it. We must eventually confront the challenge of transhuman intelligence in any case. This leads me to see improvement of intelligence as the escape hatch from the existential-risks game, and to believe that we should confront it sooner rather than later.
