Singularity Holes
Please use this page to explore holes, poor understanding, flawed reasoning, knowledge-advancements-needed, etc., in Singularity-related issues.
See also Singularitarianism Objections and Singularity Questions.
Sorry guys, this whole singularity thing is not going to happen. Think about energy. There is not the energy available to do any of this. Arguments about "okay, the AI will invent a new kind of energy" are singularly unimpressive. Study some thermodynamics and you'll realise this is all nonsense.
That seems unlikely. How much energy do you suppose is needed, and why? -- Nick Hay
For example the approaching decline of available energy will soon have mankind reverting to a middle ages lifestyle. (Peak oil, peak gas, etc.) There's not enough practical available energy for this singularity thing.
Here are my thoughts to start the list:
- Humans aren't smart enough to build AI
- Moore's law won't continue:
- isn't X GHz enough? (demand side argument)
- physical limits (supply side argument)
- there is a 3rd dimension to explore with chip technology
- reversible computing may be the solution to the power dissipation issue
- The Real AI will hunt us for fun! (and other anthropomorphic nonsense)
- why not? do humans deserve to live?
- See also Evil Hollywood AI
- The same goes for thinking like:
- The AI will suddenly decide to create a harem of female AIs for the heck of it, (or for profit);
- The AI will form a huge Communist party, or medieval Church, or analogous organization, to make sure all humans have the Correct Views (tm).
- Ego tripping and being worshiped is a fundamental need of AIs
- So is spying on people to humiliate them, or out of sexual voyerism
- And, for that matter, just about any idea that sounds like it belongs in the Illuminatus trilogy. -- Nathan Russell
- Just like in the 50's and 80's, we keep saying AI is another 10-20 years years away. "Those AI researchers never learn."
- Which of the problems were found out to be more challenging than originally supposed? why?
- Will Humankind reach the Technological Singularity first or be bounded by Limits to Growth? i.e. the Singularity vs Limits conundrum.
Here's an argument against Singularity and AI in general I've run into a couple times: "It's impossible for a mind to completely understand itself; Humans are the weakest possible General Intelligence; therefore Humans cannot understand General Intelligences greater than or equal to themselves, which are all intelligences"
Starting out, this line of reasoning makes enough sense. Obviously, a system of N complexity must represent what it understand within less than N items. Therefore, it cannot encapsulate itself, and by extension, all things greater than itself. The hole here is the idea of complete encapsulation being necessary for understanding. In order to usefully model and construct within a given discipline it has Never been necessary to understand all factors. It's only important to isolate and employ the functions, dynamics, and interactions that are Necessary and Sufficient. To build a wooden boat, one must only have a very basic concept of woodworking, and boat design, not understand fluid dynamics and plant biostructure entirely. The real question is, Human minds are of N complexity of which H is their maximum 'working-space' for modelling concepts. Does AI-design fall below or above H? There is no real reason to assume that it falls above it, and as far as I am aware, no subject has ever been found to reside above it, for that matter(aside from problems that are postulated to be uncomputable for any intelligence).
hope this SingularityHole is useful -Justin Corwin
- but, a system of a given complexity can increase in complexity over time (emergent behaviour, etc).
A few thoughts - As we do not understand the nature of intelligence itself there are a few things that can derail the singularity. Or at least slow it down. First Moore's law may have little effect on levels of intelligence just that it will be a program running really fast. Also that intelligence may be a complex problem analogous to an NP-hard problem, in that in order to generate the next level of intelligence up it requires twice the previous amount of work, and there may not be a quicker way to do it. Conversely there may be only a singular type of intelligent system and there may be no way of changing a program to make it more intelligent, that is intelligence is based on the facts incorporated into a system and not the underlying program.
On a related note to the complexity, it may be that problems that the AI needs to solve, such as non-gravitational fusion, in order to create the singularity maybe in complexity classes that mean that they are highly impractical even for self-improving systems (SIS from now on). - Agent 101
Problems with recursive Self improvement
Unproven Self-improvement
Informal Self-Improving systems that create a new non-proof bound version of itself, due to each system being a turing machine, may not be able to probabilistically recursively self-improve. This is because debugging a turing program is non-computable[1], so when creating the next generation of program there is an unknown probability of creating an error. You can try and drive it down towards zero by testing the programs first, but there is no way to definitively reduce it to zero. So any attempt at building a recursivly self-improving system will have a unknown chance of creating an error in one of its subsequent programs. Although it is possible to create proofs that a program will halt, I will deal with that in the next section.
If you wish to prove that an SIAI is friendly you would have to care about halting. Because if you solve the halting problem you can solve the no accidental errors problem, which is needed for friendliness (or any useful function for that matter). Try this proof which is related to the halting problem proof. Let us say we have a fool-proof way of determining whether a function errors in a bad way. It should only return true if the function does produce unexpected errors, i.e. it's false positive rate should be zero.
function performsUnexpected Error(function a)
Now let us look at another function AI called the paradoxical function
function paradoxicalAI
if(performsUnexpected Error(paradoxicalAI))
- doNormal Actions()
else
- performUnexpected Error()
end
I hope that you can see that this paradoxical AI forces the performsUnexpected Error function to be wrong, and therefore the performsUnexpected Error function cannot logically exist. You could replace Unexpected error with unfriendly action if you wanted. However this function is needed for Seed AI so that the program can know whether what it is replacing with itself is error free or friendly.
I'm really not sure that this demonstrates anything meaningful. How about we put a trivial constraint on the system being verified: no part of the system can reference the verification function. The proof falls apart. This constraint is trivial because it doesn't prevent any conceivable complexity in the system itself. Incidentally, I'm aware that this is a play on Turing's proof. I just don't find it convincing at all. It's on the level of "This statement is false" is an invalidation of logic.
How can no part of the system reference the verification function, when the verification function or an improved copy of it, is part of the new system?
- Yudkowsky's thoughts on Recursive Self Improvement.
I hadn't clarified my first point on Recursive Self Improvement that it was based on the more informal method of testing new programs rather than the formal Goedel Machine method which I agree would not have this problem. Feel free to delete this point, if you feel it serves no purpose. However the section under proved self-improvement methods I shall expand as I think it is more relevant to where the thought on Recursive Self-Improvement is nowadays.
Seed AI as I understand it is about the possibility of changing the whole AI including any injunctionFunctions. If the AI is changed using a heurstic there is a chance of failure (unlike with proofs). Since it is changing itself, a program that interacts with the real world and a supposedly highly intelligent one at that, it seems implausible that it could fool an equal or more intelligent AI in order to be able to have any non-real world tests be indicative of how they would act in the real world. At some point the old AI would have to let go of control of the new AI, else it doesn't fit the definition of a Seed AI, and then errors could occur. If it created multiple copies of itself then heuristics would be fine as not all the eggs (or seeds) are in one basket. However then it would be likely that evolution and competition for resources would occur. Leading to a possible non-friendly event
Proved Self-improvement
Using a formal proof to get to a new program puts some limits on what a system can do to improve itself. The major points of my position are that there is very little knowledge (I would say none as human knowledge is fallible) you can axiomatise about the world and with what you can axiomatise you have a very limited system.
Also what axioms should we specify as true about computation itself, how much energy is used, how much interference caused with other systems? A system would have to know its processor intimately to get that right, and it is another thing the human may get wrong, or may be specified inadequetely ignoring changes in silicon with age or other such possibilities.
The first point is anybody happy to axiomatise E=mc^2 or other physical laws for ever more as being fundementally true. So any self-modifications that relied on the physics such as improving communication with other mind machines or moving itself into other hardware would not be allowed.
Even if you did allow the basic theorems of physics the complexity of proving that a program can be moved from one machine to another without loss of data over a wire is mind boggling. You would have to able to predict power cuts, lightning strikes, volcanos and the human inteference. So any worries that this kind of self-improving system will try to break out and take over all the computers in the world is neither here nor there, it will not even think of doing it because it cannot (unless it is of the order of magnitude of the size of a world) create a formal proof that it could.
You can use probabilistic proofs as can happen in Goedel Machines, but then you lose the safety of the proof, as it is only probable that things will continue to work. It seems strange that the main method of reasoning advocated by the Singinst is bayesian, yet the self-improvement methods must be done non-probabilistically (in order to not have the potential for disaster or stopping) and is seen to be powerful enough to cope with the world.
- Agent 101 [1] Can't find a reference at this point, but it builds on the halting problem, e.g. until a program has halted we can never know whether it halts without error.
Partioned Memory Model
Now it may be necessary that an Intelligence has a partitioned memory model. Why do I say this? Not simply because of obviously non-RAM likeness of our brain. Assuming that some form of programming is done (or an uncaught error occurs) then there is a chance that there is malicious code within the system (even with proved programming techniques, it depends upon the axioms being correct, which have been chosen by fallible humans). Now so that this error does not spread throughout the system it may be necessary to have memory protection like the unix file system. You couldn't even have a kernel that was allowed to access all, else if that got corrupted then you would still have the same problem.
It may also be necessary for the system to be seperated into different processors with their own memory as errorful programs may dominate the memory bus and prevent other purposeful code from performing it's function. This would create even more of a boundary to the likelyhood of sub-goal stomp
So each program would be restricted to modifying a tiny part of the system. If intelligence needs this fail-safe method of restricting errors from spreading then RPOP 's would seem a lot less powerful and would be restricted to building new system to try and improve upon them. Which would be a lot slower.
Please edit this list. -- Aaron McBride
(The following is quoted from http://humanknowledge.net/Thoughts.html#singularity)
The Singularity will not happen. First, the limits to intelligence apply to artificial intelligence as much as to natural. Second, intelligence is likely not to vary qualitatively as a function of things like processing speed or memory that are increasing exponentially. Third, the effort to make minds faster or smarter will quite likely be subject to diminishing returns. Fourth, artificial minds will at first not be designed but rather grown and evolved, and will be subject to most of the same limits as minds that are naturally grown and evolved.
Is this the right place to discuss _One half of a manifesto_ by Jaron Lanier http://www.edge.org/3rd_culture/lanier/lanier_index.html ?
Golly, major mechanomorphism here! -- Michael Anissimov
On the topic of energy: There's plenty of solar/wind/hydrogen(maybe?) left in the world. Granted, whether or not we decide to USE it is the question. And if we do... You'll be seeing me in the VR ward.
--Angelica Klosky
Couple of things you might want to think about:
Creating an AI that can learn from scratch is simpler than creating one that we can precode the encyclopedia britannica and a whole lot of behavioral stuff into. Therefore the first AIs will be far closer to children than to megadeath robots. Any attitude that starts with the concept of training them is in ethical hot water and asking for problems. We need to educate them - to help them grow and learn - rather than force a specific mindset upon them. After all, what's the point of making something that can think if we immediately deny it the right to think for itself.
If you want to train it and interact with it, go spend some time with autistic kids. The first few generations of AIs will be much the same. They may be smart, but due to the limitations of their sensory arrays and a lack of mobility they are probably going to have a very different understanding of the world to the one we have. It's biggest problem will be a lack of social interaction and any understanding of what it means to be soft and squishy.
How much of the human mind is used for thinking? (as opposed to data storage). If there was a major evolutionary advantage to using more of it for thought, why don't we?
Do you really want to create a genuine intelligent being or do you want a pet philosopher that answer your questions but has no free will of its own? A lot of what I've read here seems more to favor the slave side...
A lot of what we know about the world and how we interact with it is timing dependent. Yes, you probably could educate an AI within an accelerated VR, but for it to interact with real humans in the real world you'd have to slow it down to our speed. Is there, therefore, a 'speed of intelligence'? Is there any evidence it's getting faster as we're evolving? If you slow down the rate of data input relative to the intelligence processing it, isn't it going to loose the ability to correlate between consecutive events?
The only current 'intelligence' system that we know about is a pretty complex emergent system. What grounds do you have for believing that the first AI systems will be any different? If this is the case, then all we can directly program are the mechanisms that store and process the data - its being will be encoded several levels deep within the constructs it makes from the mechanisms. And I suspect it'll be no better at telling us how it stores 'yellow' than we are at telling ourselves how we do it. Don't hold your breath waiting for the self modifying aspects - it'll have to learn to program first.
- Mykael
> How much of the human mind is used for thinking? (as opposed to data storage). > If there was a major evolutionary advantage to using more of it for thought, why don't we?
Because there isn't a convenient incremental path. Unreliable noisy 200Hz neurons just suck as a computational substrate, but they were the easiest to develop and once they were running the show animals were basically stuck with them. Please go read the several hundred posts in the SL4 archives on intrinsic AI advantages (actually, just go read 'Staring Into The Singularity', as I recall Eliezer covered it well enough even there).
> What grounds do you have for believing that the first AI systems will be any different?
Because that's a silly way to build them (of course natural selection regularly produces things that would make any intelligent designer burst out in laughter, while marvelling that the wretched thing works at all). Regardless, the capability for direct self-enhancement means that as soon as your AGI learns to program (which all but the most humanistic AIs intrinsically find a lot easier than humans), it will start to undergo takeoff. The 'learning' rate will go through the roof, you will lose any ability to understand and control it (assuming you didn't explicitly design it as an FAI), and you've got an UFAI loose.
-- Starglider
