Wiki Interview With Eliezer/Autonomous Intelligence
Autonomous Intelligence
Please tell me about the "power of intelligence" and its relevance to our world.
Intelligence is the main cause in how well one may affect reality. Our modern world of cultural values, scientific knowledge, and technology, has been created by the power of the differential between the intelligence of homo sapiens sapiens and the intelligence of great apes. As intelligence improves, the problems that are solvable and the quality of solutions that are accessible will also improve. Human knowledge and technology, which have been slowly self-improving for nearly ten thousand years, are bounded by human General Intelligence. However, human intelligence is not a limit on what’s possible. Within the first-half of the 21st century, technology that enhances General Intelligence will likely be developed, creating intelligence with a greater ability to develop technology that further enhances General Intelligence. This will in turn wlead to a positive feedback effect, which has never before existed, that will have quite unprecedented implications.
(copied to The Power Of Intelligence)
Why can General Intelligence in principle be created? What are your assumptions?
The answer for "in principle" is easy - humans exist, therefore physical objects that exhibit what we call "General Intelligence" are physically possible. I don't like "assumptions" since those, by assumption, are unjustified. However, there are important dependencies - not unjustified ones, but important ones. One dependency is on the idea that General Intelligence can be created on the physical objects we call "computers". If that dependency is broken, then we might have to create "superneurons" modeled after human neurons, but programmable, in order to create General Intelligence. Another dependency is on the idea that evolution is not a magically powerful designer, and that humans are potentially capable of matching it in effective intelligence, although there will undoubtedly be major design differences in the creation proceeding from the fundamentally different intelligence of the creator.
(copied to Why Is AGI Possible)
Why do you believe that General Intelligence can be created in computers?
Because the various specific objections to this idea fall flat, and because there is nothing noncomputable invoked by "Levels of Organization". Even if the brain uses some kind of massive quantum-parallel compare to deal with certain problems such as association of perception to memory and so on, I would expect that the reason will be, as it always is with evolution, "because it's there", rather than "because it's necessary". It seems to my intuition that the advantages of silicon, such as massive serialism, should be enough to build a mind around the lost advantages of biology, even if the advantage is quantum computing on some specific algorithm. Of course, the vast majority of scientists do not believe that the brain uses quantum computing at all. Basically, I individually deny those various objections to intelligence being created in computers, and I believe that "Levels of Organization" accounts for many qualities that are attributed to the magical nature of biology, and that "Levels of Organization" also explains the origins of those qualities that are stereotypically "mechanical" in modern day, nowhere-near-AI programs. Based on my specific beliefs about the Cognitive Science of intelligence, and what intelligence is and why, I believe that intelligence can be implemented on a computer.
[1/27/03: The physicist David Deutsch has shown an equivalence-in-function between quantum and digital computing. --Anand ]
(copied to Why Is AGI PossibleOnComputers)
Why is greater-than-human intelligence highly probable? If we have other alternatives, what are they, and why?
I don't know that greater-than-human intelligence is highly probable. I believe that it is highly desirable (perhaps under certain enabling conditions), and that it is highly probable given the continued existence of the human species. I perceive no alternative to coming to grips with greater-than-human intelligence except our extinction from other forces before the technological creation of greater-than-human intelligence. The reason it is highly probable is simply that the technological capacity exists, in several different forms ranging from AI to IA. At present there is nothing that would prevent interested parties from explicitly pursuing this technological goal in any of the several forms. Even if Bill Joy takes over the world, I doubt that Moore's Law will come to a halt, and as technology increases, the threshold required to give birth to greater-than-human intelligence becomes steadily lower. I can envision humanity existing in a world dictatorship that halts or greatly slows technological progress, but I cannot envision that dictatorship lasting forever, and whenever it broke down, we would simply end up in roughly the same situation we have now. Over a sufficiently long period, even with no technological progress, natural evolution would take over, and if somehow a dictatorship controls even that, say by cloning all new individuals from previous ones, then eventually the Sun will swell up and eliminate Earth, thus extinguishing humanity. So certainly, in the long run, there are no options except transhumanity and extinction, and in the short run, the probability of all technological progress coming to a shrieking halt is small enough that I don't worry about it, although I do worry about bans on explicit progress toward greater-than-human intelligence, which strongly raises the probability of greater-than-human intelligence occurring haphazardly or in less benevolent hands. This might not affect the outcome, but there is also a real probability that it might, and it would certainly affect the timing, at 150,000 deaths per day.
[1/27/03: Presently, three options exist for people living in this century: 1) The continuation of accelerating technical progress, leading to the technological enhancement of General Intelligence (i.e., the Singularity); 2) the global setback of civilization’s progress; or 3) the extinction of civilization. The second and third are undesirable by default. The first is very desirable--under certain conditions--because of its unprecedented benefits, and is very probable for numerous reasons. For example, there is continual economic, humanitarian, and scientific imperative to develop ever more intelligent technology; and many already known technologies (e.g., brain-computer interfaces, general AI, germline engineering, and mind uploading) that, if developed to a mature level, could successfully enhance General Intelligence. If there was a world dictatorship that banned research and development on intelligence-enhancing-technologies, or that slowed or stopped technical progress-in-general, evolution by natural and sexual selection would eventually enhance intelligence, since appropriate selection pressures have existed before, as evidenced by the exponential increase in brain capacity and prefrontal cortex size in Hominids during the past five million years. If a dictatorship could actually control evolution, the Sun would eventually swell and eliminate Earth, thus resulting in the third option. --Anand ]
(copied to Why Is Greater Than Human AGI Probable)
Why do you believe we can successfully develop the required level of General Intelligence to achieve seed AI? Please describe your present estimation of how difficult this project will be and what will be mandatory. What do you believe will constitute the required level of General Intelligence?
I think I covered most of this in DGI. How much intelligence might be sufficient is one question, how much should be sufficient is another, but I'm not sure how, because DGI is a very large theory and it is hard to separate one part from any other part. I guess you could say that the AI needs the ability to understand its own design in roughly the same way the programmers understand that design, but this statement is vague unless you know what specifically is required to "understand" like that, and there is no simple way to explain that, that I know of, short of recapitulating DGI. There are no common referents for the "required level of General Intelligence" - no metaphors or analogies that I can use to explain it. You need an AI that does certain things that are described in DGI. I don't think you can do it with a mind that is fundamentally incomplete by human standards. If you're talking to a mind of the order required for seed AI, you should (a) be able to communicate with it and (b) recognize a mind on the other end. Seed AI does not let you bootstrap from an unintelligent core, or at least this is not knowably possible, and all real reasons for believing that creating General Intelligence is possible will necessarily be along the lines of "because I have reason to believe that if you do X, Y, and Z, you can create roughly that order of intelligence".
[1/27/03: There isn’t a simple way to explain the "required level of General Intelligence to achieve seed AI", apart from summarizing deliberative General Intelligence (DGI), which is far too large a theory to summarize here. Essentially you need an AI with the functionality described in Levels of Organization in General Intelligence. An AI will need to understand its design in a way that is comparable to how the programmers understand it. When someone communicates with the AI, that individual should recognize a digital mind. --Anand ]
(copied to Can We Create Greater Than Human AGI)
What are the qualitative and quantitative aspects of General Intelligence?
The qualitative aspects of General Intelligence are the things that we recognize as missing in animals and AIs: the ability to learn and to use what is learned in reasoning, the ability to stretch designed abilities beyond the domains for which they were designed, and the ability to use more than one kind of thinking to stabilize thought. Of course, animals have some of these abilities to some degree, but in AI it is almost entirely missing. The quantitative aspects, as we think of them now, are mostly variations between humans. To understand quantitative differences between a human and an AI you would have to look at quantitative variations in the strength of underlying subsystems, and see how that contributed to differences in thinking. For example, an AI might have a memory subsystem, but one that is implemented in a different way than in a human, and is consequently much weaker at associating current experiences to past memories, so that a human, watching the AI, would see the AI as missing out on "obvious" (to the human) analogies between present experience and past experience. And yet the AI would still have the ability to construct those analogies, just not enough computing power to do a human-thorough search. For some special cases the AI might make an association where a human would not and surprise the programmers, but on the whole the AI might still be much stupider than we.
[1/27/03: Qualitative aspects of General Intelligence are what we recognize as missing in AI and animals, e.g., the ability to learn and apply that learning in reasoning; the ability to stretch designed abilities beyond the domains for which they were designed; and the ability to use more than one way of thinking to stabilize thought. Of course, animals do have some of these to a limited degree, though in present-day AI it is almost entirely missing. Quantitative aspects of General Intelligence are the variations that exist between humans. To understand quantitative differences between a human and an AI, you would need to look at the quantitative variations in the strength of their underlying subsystems and determine how the variations contribute to differences in thinking. For example, an AI might have a memory subsystem implemented differently than a human, which may be consequently weaker at associating present experiences to past memories. A human watching the AI would see it as missing out on "obvious" analogies between present and past experiences; however, the AI would still have the ability to construct those analogies but will lack the necessary computing power to perform a human-equivalent search. Nevertheless, there may be cases when an AI makes an association that a human would not, while still lacking Human-Similar General Intelligence. --Anand] ]
(copied to Quality And Quantity Of General Intelligence)
Please elaborate on other qualitative aspects of General Intelligence.
There are a lot of things that General Intelligences can do that animals and present-day AIs can't, or that General Intelligences do much better. I can toss out things like "creativity", "self-awareness", "abstract reasoning", and so on, but it seems to me that mentioning such things is like promising to explain them, and the explanation of these things is technical. Self-awareness is discussed to some degree in "Levels of Organization", as is abstract reasoning, but the things that we really think of as uniquely human, i.e., unique to General Intelligence, are things where the real explanation is complicated - satisfying, but complicated, so the items I tossed out on the original list are the ones that are relatively easy to comprehend on their own terms - it may be clear that achieving them is hard, but they don't have the same kind of intuitive opaqueness or "needs-decomposition quality" that "creativity" and "self-awareness" have and so on.
[1/27/03: Human General Intelligence has numerous abilities that animals and present-day AIs lack, e.g., "creativity", "self-awareness", and "abstract reasoning". These were not mentioned since they are attributes that we consider unique to General Intelligence and require complex explanations. The qualitative aspects that were listed are inherently easy to comprehend, though still difficult to achieve. --Anand ]
(copied to Quality And Quantity Of General Intelligence)
Please describe the functionality and capabilities of a Human-Similar General Intelligence and a human-equivalent General Intelligence.
A Human-Similar General Intelligence should not lack any fundamental human cognitive processes. For example, it should not lack the ability to form memories, associate present experiences to memories, retrieve memories, and construct analogies using memories. A "near-human-level" GI may be weaker in these abilities but it will have these abilities. It may sound stupid in a conversation, but it will be able to have a conversation. It may not learn as fast as we do or as much as we do, but it will be able to learn. It may have to use long chains of reasoning to think about ideas that we can sum up in a single concept, but it will still have concepts, just less powerful ones. It may take it a couple of hours to think through something that we would be able to do in five seconds, although there will probably be less of that, because frustrated programmers will probably prefer a dumber mind running at a faster speed. So there are many ways in which a mind can be inferior to human and nonetheless be a complete mind. A "human-equivalent" General Intelligence is mostly a chimera - you don't get something that runs on such enormously different hardware and has such a fundamentally different Design Signature, yet has the human balance of competencies. There is infrahuman AI and there is transhuman AI, but there is very little ground in between. If there were such a thing as human-equivalent AI, perhaps by virtue of the AI deciding to slow down and chat before taking off, its chief attribute would be that you could talk to it as one sentient to another, without it visibly doing anything stupid. Since a human balance of competencies is profoundly unlikely, if an AI is at least equal to you in most things, enough to get through a conversation, then its overall balance of competencies is probably at least a little transhuman at that point. An alternative would be to define "human-equivalent" as the point where it stuns you with its intelligence roughly as often as it stuns you with its stupidity. That's probably much more likely to happen as a phase in development.
(copied to Human Similar General Intelligence and Human Equivalent General Intelligence)
Why do you believe that our minds will have more success, in the short run, at creating a Human-Similar or human-surpassing (i.e., transhuman) autonomous intelligence than any other method, such as increasing autonomous complexity through increasingly advanced forms of evolvable hardware?
Because evolution isn't magic. Directed evolution on presently available hardware exerts far less "Design Pressure" than human deliberate intelligence. We don't use directed evolution on evolvable hardware to design Linux, and we probably won't use it to design AI either. Not unless there is an enormous - and I mean really enormous - leap in the quantity and quality of evolvable computing power, and even then it will probably be highly directed evolution. Evolution is really stupid about some things. Saying "we'll get all the functionality of intelligence from evolution" is, as I see it, just one more excuse to avoid getting to grips with the problem. You have to know what intelligence is, or you are not going to get anywhere. I know of no way to avoid this. I do not even see people being able to evolve a mind unless they know what a mind is and how it works to some degree, or unless they use the full complexity of evolution operating for millions of subjective years over a full complex ecology.
[1/27/03: Directed evolution on present hardware has far less Design Pressure than our deliberate intelligence. This is why directed evolution on evolvable hardware isn’t used for operating system design, and probably won’t be used for general AI design, unless the quality and quantity of evolvable computing power greatly improves. I think the claim that the complexity of General Intelligence will be achieved by evolutionary methods is an excuse for avoiding the necessity of understanding intelligence. You need to know where to start in order to get to where you want to go. If you don’t understand intelligence, what it is and how it works, you’re probably not going to be able to create it, regardless of whether you use an evolutionary method. --Anand ]
(copied to Can We Create Greater Than Human AGI)
Why is the Institute using a modified top-down method for AI development instead of a bottom-up developmental and evolutionary method? Please provide the reasoning behind the Institute's chosen strategy.
"Top-down" and "bottom-up" AI are two sides of the same coin. We are neither. Physics is bottom-up, simple human designs are top-down, and biology is neither. Biology involves multiple levels of organization with adaptive complexity on each level. The higher levels do not emerge automatically from the lower levels as in physics. The lower levels are not all specified purely by the higher levels as in simple human designs. Evolution, as a designer, designs each of the levels, although it is much less efficient on higher levels because evolution is not good at simultaneous dependencies. A human, to build an AI, must learn to think in multiple levels of organization.
(copied to Can We Create Greater Than Human AGI)
In "Developmental Singularity Studies Resources", John Smart wrote the following about Friendly AI: "I tend to disagree with many assumptions of Yudkowsky, but his is a good example of top-down models which express a "conditional confidence" in future friendliness. I share his conclusion but without invoking a "consciousness centralizing" world view, which assumes that human imposed conditions will continue to play a central role in the self-balancing, integrative, and information-protecting processes that are emerging within complex adaptive technological systems.... It is easy to assume that because humans are catalysts in the production of technology to increase our local understanding of the universe, that we ultimately "control" that technology, and that it develops at a rate and in a manner dependent on our conscious understanding of it. Such may approximate the actual case in the initial stages, but all complex adaptive systems rapidly develop local centers of control, and technology is no exception. It can be demonstrated that evolutionary developmental systems take care of these issues on their own, from within, and that technological evolutionary development is rapidly engaged in the process of encoding, learning, and self-organizing in its own contingent fashion, and with a degree of MEST Compression millions of times greater than human memetic evolutionary development. Thus humans are both partially-cognizant spectators and willing catalysts in the process. This appears to be the hidden story of emergent AI." What is your response?
John Smart uses a very different reasoning style than I do. If two processes are similar in some respects, it does not demonstrate that they are similar in all respects. Not all analogies are causal analogies. Only if the surface similarities proceed from the same underlying cause, and this underlying cause would be expected to make other properties similar as well, is an analogy strong. You cannot draw analogies between the Singularity and the rise of human intelligence or the exponential improvement in human knowledge because in one case you have evolution improving humans, and in the other case you have humans improving knowledge, and in neither case do you have an intelligence redesigning itself. Since John Smart's analysis of the Singularity is fundamentally based on analogy with what I would call "weakly self-improving processes", I tend to regard all his analogies as invalid, which can make it hard for us to talk with each other sometimes! In this specific case, I would disagree with him for two reasons: First, because unlike all the processes with which John Smart makes analogies, the Singularity is fundamentally an intelligent event. It is not a blind process but an aware one - created by and driven by some transhuman intelligence. Second, Friendliness is not about control. Friendly AI is very hard to understand because, as a way of seeing, it stands outside all the ways that humans have to see other humans. In some ways it is most analogous to the way we think about morality internally, not the way we relate to foes and allies, because we try to persuade others but we create ourselves. So to think about Friendly AI, you have to stop using the instincts that let you deal with other minds, and eventually even stop using the instincts that you use to create yourself, because you really only modify yourself - you didn't build yourself to begin with. Friendly AI is somewhere outside of all our instinctive analogies - in fact, those analogies form the single greatest barrier to understanding, so Friendly AI is not about taking "control" of the Singularity, it is not about "keeping the reins on", as John Smart seems to imply. What you're doing, rather, is passing on a special kind of moral complexity that humans have, and one of the reasons it is special is that it is the complexity that creates more complexity. We have the ability to reason philosophically. We aren't stuck forever in the place we started out. This may seem magical but it is no more magical than General Intelligence. So what Friendly AI is about is ensuring continuity with the Singularity. Not trying to keep the Singularity forever in the place where humans are now, but to make sure that it starts out from where humans are now - although not blindly so. Where it goes from there is the beautiful question.
(copied to Wiki Commentary On Singularity Watch Dot Com)
In "Self-Organizing and Self-Replicating Paths to Autonomous Intelligence", John Smart wrote, "Rather than human minds having the capacity to furnish relevant goals to our AI systems as they develop, as top-down A.I. designers assume to be the critical issue, evolutionary developmental computation (in both biological and technological systems) creates its own goals and encodes learned information in its own bottom-up, incremental, and context-dependent fashion, in a manner only partially accessible to our rational analysis. This process is called self-organization, and is observed in all complex systems, from molecules to minds.... let me now point out that on close inspection of the state of AI research in 2001, one finds that there are very few researchers left who do not acknowledge the fundamental utility of evolution as a creative component in future AI systems. Those top-down AI approaches which still remain in vogue (whether classical symbolic or one of the many historical derivatives of this) are now few in number, and despite decades of iterative refinement, have consistently demonstrated only minor incremental improvements in functional adaptation. To me, this is a strong indication that human-centric, human-envisioned design has reached a "saturation phase" in its attempt to add incremental complexity to technologic systems. We humans simply aren't that smart, and the universe is showing us a much more powerful way to create complexity than by trying to develop or deduce it from first principles." What is your response?
John Smart may underestimate the extreme specificity of the ancestral conditions that produced humans who were capable of altruism and moral reasoning. We are not just evolved organisms, or even social organisms. We are imperfectly deceptive social organisms that use language to argue about each other's motives in adaptive contexts, and what this works out to, in practice, is that having AIs play Go against each other for a few thousand generations is not going to produce Gandhi, so it would probably be a bad idea. Similarly, evolution encodes goals in a haphazard fashion, and our goals specifically are for the most part encoded in a haphazard fashion, because the vast majority of our Goal Systems evolved in the absence of General Intelligence. That is why we use contraception. When we evolved to like sex, we were not smart enough to reason that reproduction required sex, so we had to evolve to like sex for its own sake - liking reproduction wouldn't have done any good, because we weren't smart enough to draw the line. So much of the haphazardness is not this brilliant design concept that evolution came up with, but the combined results of evolution's design messiness, its total lack of foresight, and the fact that the vast majority of the human brain is not designed for human intelligence. As for John Smart's belief that we are just not smart enough to create intelligence, I predict on the basis of my current knowledge, that if you do not know what intelligence is; you will be in a heck of a mess trying to evolve one, unless you can get hold of a lot of nanocomputing power, and even then, it won't work for Friendly AI. So it could be true that deliberately designed intelligence is out of our reach, although I do not believe it so, but if so, evolution is not going to solve the problem for us. Evolution is the degenerate case of design-and-test where intelligence equals zero. There are some specific identifiable psychological errors in AI research, such as physics envy. DGI discusses a lot of this. Anyway, the point is that while the history of AI is a strong argument that AI is very, very hard, I do not believe it is a strong enough argument to show that humans understanding intelligence is impossible. The other Cognitive Sciences - you know, the ones that are on "brain science" campus instead of the "computer science" campus - have made steady and profoundly impressive progress while AI has been stagnating.
(copied to Wiki Commentary On Singularity Watch Dot Com)
In "Exploring the Technological Singularity: Uncovering the Universal Drivers of Accelerating Change", John Smart wrote, "Proponents of the overly determinist (and overly reductionist!) view often seem to favor both the "top down" and "Hard Takeoff" scenarios-the lone romantic savior of the world toiling away in the lab making an AI that suddenly explodes into superintelligence. I think that script misses the nature of co-evolution in substrate emergence, and underestimates the complexity of the bottom up learning that must take place in order to construct truly autonomous technological change." What is your response?
Well, let's start with John Smart's apparent contrast of "top down" to "learning". If by "top down" John Smart means Good-Old-Fashioned AI with LISP predicates, where knowledge is preprogrammed, then DGI is so far from that paradigm of AI development that the light now leaving DGI will not reach there for several centuries. As for Hard Takeoff, that emerges directly from the recursive nature of the Singularity - the difference between strong self-improvement and weak self-improvement that I think is basically missing from John Smart's complexity-theory model. But in this case, the problem shows up just by looking at human intelligence. It's in humans, not in chimpanzees. Humans communicate with each other, and so knowledge builds up very rapidly between humans without much involvement of the rest of the species ecology, so in a Hard Takeoff, the reason that an AI in a lab suddenly explodes into superintelligence is that that AI is the center of the positive feedback loop of intelligence improving intelligence, directly, just as the human species, alone, is the center of the much weaker positive feedback loop in knowledge. Even by looking at weakly self-improving processes you rapidly tend to drop out of the equation. Now I don't want to make this sound all dismal. Just because you're born a human doesn't mean you have to stay one forever, but if you do decide to stay a human, then you're not going to be part of the forefront of progress, that's for sure. The point is that while the AI is in the basement and before the AI has the kind of extremely high-level technology it takes to upgrade a human, then the humans may have contributed to setting up the positive feedback loop, just like the whole evolutionary ecology contributed to the birth of the primate order and the hominid family. But the humans are not part of the positive feedback loop, because the AI does not YET have the ability to (with consent, I would hope!) modify the humans source code, but it can modify its own. Positive feedback in any form, let alone the super-strong form of totally Recursive Self-Improvement, tends to localize change to the processes that are most directly participating in, and more importantly benefiting from, the positive feedback loop, hence the Hard Takeoff.
(copied to Wiki Commentary On Singularity Watch Dot Com)
Why will the Institute have success developing General Intelligence using a rational engineering method, when all previous AI projects have failed to achieve real AI?
Again, this assumes a bipolar contrast between "rational" engineering and "evolutionary" engineering, which is a very clever move on the evolutionists' part, and one that I would deny. I would tend to lump in evolutionary AI with all other past failed forms of AI, and set DGI apart in a separate corner, naturally. The reason that we may have success using a "rational" engineering method; is that it is an entirely different rational engineering method; one that doesn't expect the whole complexity of intelligence to magically leap out of any one great idea, whether that great idea is "connectionist neurons", "predicate logic", or "evolutionary programming". There are multiple levels of organization, and multiple necessary ideas on each level. We are trying to scale an enormous wall, and it's not a wall of computing power or a wall of human-entered knowledge, but a wall of complex design. Previous approaches to AI have tended to assume that this wall is 10 feet tall, when in fact it is something like thirty miles high, and this alone would be enough to result in failure, even without the continuing operation of the psychological errors that led them to think the wall was that small in the first place. If DGI fails the Singularity Institute, we will try again and again, and will still be here in fifty years, if humanity survives that long, because this is the best outlet for any altruistic effort that we know of. If you want to accomplish good, this is how. We are in this for the long haul because it is necessary, and that in itself distinguishes us from some other approaches to AI. We are not doing this because we had one bright idea and now we think it'll be easy. We expect this to be hard. We expected this to be hard from the beginning. We are trying to do it because it is necessary, not because it is easy. We have no stake in arguing that one bright idea will work. Even now that we have enough ideas that AI might work, we will go on searching for ideas so that it will work better. Historically speaking, I started out by believing that AI would require a Manhattan project, then that it would require an industrywide effort, as it would have if I'd tackled it with the ideas I had at that time, but I went on thinking, and eventually reached the point of thinking that it may be doable with a medium-sized project. But if not, then humanity will have to invest whatever resources it takes to scale this wall or suffer the usual penalties for failing to pay the bills on time, because this is what the human species needs to be doing now.
(copied to Can We Create Greater Than Human AGI)
You previously claimed that no one project could code a "true mind", but now believe that one project, SIAI's, should be able to code a system with enough General Intelligence for it to begin achieving significant Recursive Self-Improvement. Please elaborate on why you've changed your mind.
Well, I hope I'm not penalized for seeing the real size of the problem. If you don't know how to solve a problem, but you are realistic enough to see that it's an enormous problem, then one of two things happens: Either you give up and do something else, or you imagine yourself solving the enormous problem by throwing everything you can at it. I couldn't give up and do something else, because this was something that humanity had to do, so I said: "Manhattan Project", which, if you don't know how to solve a problem, is one thing that might give a realistic chance of it being solved - throwing enormous resources at it, not just "brute force", but "brute intelligence", as it were. Lots of researchers and lots of ideas. Originally I even thought that augmented humans might help on the project, but I've given up on that due to the FDA, so that's where things started out. But I also went on thinking about AI and started to understand it more, so a few years later I thought I saw how it could be done with an industrywide effort, but not a planetary effort, as I thought originally. Now, I do want to emphasize that the planetary effort would have worked! If it could have been initiated, it would have given us a reasonable chance of cracking the problem, and the industrywide effort proposed in PtS, and guided along the lines of the buildup of self-modifying complexity as described in PtS, would probably also have worked. I'm not as sure it would have worked for Friendliness, I'm afraid, but at that time I didn't realize that was part of the problem - although, given that I figured out Friendliness not too much later, I doubt the project would have gotten all the way to completion without someone thinking of that part. Or maybe not. I do apologize to everyone for not thinking of it earlier. Anyway, the point is that I do see the size of the problem. My estimate of the size of the problem has not shrunk. The difference is that I think I now know specifically how to solve it - not just that it must be solved. I didn't say it was solved as soon as I had one bright idea. I truly think that the way of seeing that is the current theory is on the same order as the problem of intelligence and AI, and it took a while to get there.
(copied to Statements Eliezer No Longer Supports)
Based upon your present understanding of Cognitive Science, General Intelligence, goal-oriented cognition, programming, morality and ethics et al., what type of transhuman autonomous intelligence do you foresee existing?
I can imagine many kinds of transhuman autonomous intelligence, from upgraded humans, to group minds, to a galaxy-spanning collective intelligence of civilization. But the transhuman world is not governed by our imaginations, but by the imaginations of transhumans - and therefore I "foresee" nothing at all. The most I can do is set lower bounds - things that appear to me should at least be possible. It appears to me that, at least, a transhuman mind can run on substrate enormously faster than biological neurons, can have sensory modalities extending far beyond the environments occupied by ancestral humans, including modalities for domains such as mathematics, can go on growing and extending their minds for as long as new hardware is available, can manipulate thoughts enormously larger than those we can handle now, can chunk larger concepts, can perceive more of the irregularities in reality instead of smoothing them over as our own minds do, can be vastly more self-aware, can have a vastly greater degree of introspective manipulation to go with the vastly better introspective perceptions, can have a much greater degree of choice, in general, about who to be; and so on... and so on... and so on...
(copied to Different Types Of Transhuman AIs)
What bottlenecks do you expect or not expect to occur as the AI progresses towards near-human-level intelligence? What is your probability estimate that "surprises" will occur as the AI progresses towards near-human-level intelligence?
My probability estimate is 1. As for bottlenecks, I expect that "all of them" is again the only really appropriate answer, but I'll try to be more specific. We may have bottlenecks where, as the AI learns new things, its effective intelligence starts to drop because the AI doesn't have enough RAM or enough processing power to compare against all stored memories, or to scan all concepts against working imagery. There will obviously be one huge bottleneck right there at the beginning, which is just "building enough of this huge system that all the parts work together", though I prefer to call these "milestones" rather than "bottlenecks". I think that maybe "bottleneck" is a concept from old-fashioned AI where you built one small thing using your one great idea, and then it bottlenecked. There is no such thing as being "finished" building AI. There may come a point where the AI takes over the job but that doesn't mean the job is finished. It just means that the AI's doing it instead of you.
(copied to Can We Create Greater Than Human AGI)
