Bayes Criticised
Buried in this book review, there is a rather strong criticism of the philosophy that Bayes Theorem is the meaning of life. The gist of it seems to be that the Bayesian priors are quite arbitrary quantities (having no empirical or rational basis), and so other ways of growing experimental knowledge must be preferred. -- Mitchell Porter
There is another big problem with Bayes. Even if, in theory, Bayes does subsume all forms of reasoning, in practice it may quickly run into computational intractability. Amassing all the data for those priors uses too much computational resources. You can argue 'no problem, reasoning is an approximation to bayes', Trouble is, approximation algorithms often look nothing at all like the ideal algorithms. For instance, the algorithms to definitely determine whether or not a number is a prime look nothing at all like the (much more practical) algorithms which determine whether a number is prime with, say 98% accuracy. --Marc Geddes
Ah, but you can use Bayes to validate (and with a little more work, generate) candidate approximations. -- Starglider
Some bayesian analysers (ala AIXI) assumes the computer is disembodied apart from the controlled outputs. For example an implementation of AIXI would run out its batteries by continiuously computing things and may not get the utility rewards that a less guaranteed optimal but more energy efficient program would by being able to run longer.
I think of this as a partial blind spot. A bayesian reasoner, that does not explictly vary how it changes its computing, does not get any updates about the consequences of changing its computational manner (such as less interference with radio equipment, or the afore mentioned energy usage). In technical terms it knows about p(i|c), the probability of interference given its current computational manner, but unless it changed its computational manner it would know nothing about p(i|~c). Where computational manner is its general computation.
Even bayesian reasoners that do change their amount of computation, and so experience different c, still have some limitations. We can imagine that c is a real number, and that decreasing c implies a decreasing computational manner. Now assume some minimum c that below this c the reasoner does not have enough computational power to maintain the updating of bayesian probabilities. We shall call this level m. So what of p(i|c<m), the bayesian reasoner will never know as it can not update probabilities at that level. This is the bayesian reasoners true blind spot.
-- Agent 101
Probabilistic logic is a fairly low-level tool that does not make any high-level assumptions such as 'the reasoning system is disembodied'. Cognitive systems that use probabilistic logic may or may not make this assumption; AIXI does so for simplicity, but a practical AGI wouldn't (even my own toy prototypes don't). AIXI is utterly, thoroughly unimplementable (and isn't supposed to be), it doesn't make any effort to think about useful things rather than useless ones because it effectively thinks all conceivable thoughts. Any practical AGI would of course concentrate available computational resources on those lines of inference most likely to produce useful information (i.e. info helpful in choosing a goal-achieving action).
'A bayesian reasoner does not get any updates about the consequences of changing its computational manner' is simply false. Bayesian reasoners may have self-models, self-environment-embedding models, and reflective modalities of any sophistication. -- Starglider
AIXI doesn't think all conceivable thoughts. For example it doesn't think, "I should not think so many thoughts". The first passage is to try and stop people thinking of AIXI as the epitome of reasoners, even if you have infinite computing power. I have altered the phrasing to fit your complaints.
It is the second paragraph that applies to all bayesian reasoners. Self-modelling has nothing to do with whether they actually "see" a change in inteference due to a change in computing style, just as modelling africa as having giant lizards will not be seen to be false until the reasoner has gone to africa. Unless the reasoner actually changes it, there will be no empirical evidence to correct the assumptions.
Perhaps you could define things that a bayesian reasoner isn't allowed to do (and still be called a bayesian reasoner). Then we might be able to have a decent discussion. My first thought would be that changing p(x|a) without having seen an instance of a is not allowed, and ignoring an instance of a or x when changing p(x|a) would also be not allowed.
-- Agent 101
The original AIXI is difficult to analyse, because it invokes infinite computing power and indefinitely large programs, which make questions such as 'can an AIXI run another full-size AIXI as a candidate program' difficult to answer. It's more useful to look at AIXI-tl, which thinks all possible thoughts that can fit in a given 'brain size' (TL). Once you've put that limit in place, you must then be very careful to proceed with your analysis in a non-anthropomorphic fashion. The concept of 'a thought' is a high-level abstraction which is often useful, but can be misleading; what does having the thought 'I should not think so many thoughts' actually imply? It implies a reflective model of the AI, a model of how the AI interacts with the environment and an inference (based on those models) that 'running every possible UTM input string under the TL threshold has negative utility (i.e. doing something else would be more efficient)'. If you actually managed to implement an AIXI-tl for a big enough TL that such a UTM program capable of such inferences would fit into TL threshold, then that would be an entirely plausible inference to make; AIXI-tl /is/ a grossly inefficient use of computing power. In other words AIXI /can/ have the 'thought' "I should not think so many thoughts". But it won't be able to act on this inference unless the UTM program that came up with this inference is selected by the fixed control structure as the best predictive hypothesis. I can't think of any scenarios offhand in which a hypothesis that encapsulates an AI capable of reflective reasoning would be able to outperform a more focused, non-reflective hypothesis of the same TL. Even if such a hypothesis was selected, as defined AIXI (which is a cartesian model) has no capability to modify its own control structure and would only be able to act by creating a new more efficient AGI in the world, not by altering itself to be more efficient (by using a different control structure). Any actual implementation of AIXI-tl in the real world would be able to self-modify, because actions taken to affect the world can 'wrap around' back to alter the AI's cognitive structure and content. However this just reinforces the point that AIXI or even AIXI-tl can't actually be implemented in our universe as defined; we can only approximate it, because we can't constrain causal interactions between the 'world' and the 'AI' to well-defined I/O channels (i.e. we can't build real life Cartesian intelligences).
The complete causal isolation between the overall AIXI control structure and the UTM programs being run (which contain the 'thoughts'), plus the strictly Cartesian model, are good examples of counterintuitive features of an AGI architecture. When analysing intelligent systems that operate on principles so different from what we intuitively expect, one has to be on guard for subtly incorrect anthropomorphic reasoning.
I'm not precisely sure what limitation you are claiming in the second part of your argument. Bayesian reasoning allows the adjustment of hypothesis probabilities to take account of evidence, and the chaining of hypotheses to produce compound predictions. Initially, a Bayesian reasoner has no specific hypotheses at all; its only mechanism for making predictions is the base prior. Bayesian induction works by selecting candidate hypotheses (which are UTM strings, in the general case), comparing their predictions to actual evidence and adjusting their posterior probabilities of being correct explanations. A Bayesian reasoner using expected utility as the decision function will combine hypothesis accuracy with hypothesis evaluation cost to get an overally utility of applying the hypothesis in a particular situation. As such it can use a complex accurate hypothesis, or a cheap inaccurate approximation, depending on how much computing power is available and how important the accuracy of the prediction is. That said, what exactly is the limitation you believe Bayesian reasoners suffer from?
-- Starglider
I didn't introduce the concept thoughts, if you had stuck with all possible programs refering to standard IO, I wouldn't have raised any issue. AIXI-tl would be able to self modify, but only in the way that a bomb-timer self-modifies, with no regard for the consequences of the modification, and no guarantee that it will stay the same kind of system.
I would dispute that UTM strings are general, UTM strings that can refer to and alter any other part of the system are the most general, but least stable.
A degenerate example of the limitation I am talking about, called the partial blind spot of some types of system. Let us say the bayesian reasoner starts with the prior that the current amount of evidence it gathers is optimal. Now let us say that is wrong and gathering evidence reduces the amount of utility it gets, by reducing its energy, for example if its goal was to increase the amount of usable energy on earth by capturing light. Now it won't change the amount of evidence it gets (because it has a prior that says the amount of evidence gathering it performs is optimal). And it won't gather *direct* evidence that it should change the amount of evidence it gathers, because it won't get evidence that changing the amount of evidence it gathers would change the amount of utility it can get. It would be stuck with P(high utility | current evidence gathering scheme) = 0.9 forever and a day. And it would have to fit all its other probabilities around this.
The prior in this situation is *very* important.
The impossible prior to update is the following P(high utility | no evidence gathering). If no evidence is being gathered then no updating can be made. This is another way of saying that I have no bayesian way of possibly correcting my belief that the world would be worse off if I was dead. For if I was dead I couldn't see that the world might be better off. This would be true of the AI that seeks to maximise the amount of usable energy on the world (once it had focused the entirety of the suns rays onto the world and built hyper efficient plants). But it cannot see that if its prior was wrong.
-- Agent 101 (?)
That just isn't the way that Bayesian reasoning works. For the problem you specify to occur, there would have to be a prior of 1.0 that the current 'evidence-gathering' system is the most useful one. 1.0 and 0.0 are the only priors that can't be updated (by a strict Bayesian), and they should never appear for anything except possibly logical fallacies in a General Intelligence. If P(current scheme is optimal) is 0.9, then there is a P(current scheme is not optimal) = 0.1. This translates into a probability distribution over various alternate schemes being better, or more usefully an expected utility for (or accuracy if you're not using EU) the act of switching to each of these alternate schemes. Since there is a possibility that one of these other schemes will be better, some of the probability mass in the latter distributions will be in utilities higher than that the current scheme is providing. This implies a utility for gathering data about which scheme is actually best; if there is a 0.1 probability that even a 1% gain in utility can be realised by switching resaoning strategies, that's enough to justify spending some effort on gathering more evidence about P(current strategy is optimal) - how much is determined by additional specifics. Anywhere you look along the chain of 'need information about P' to 'do specific sensory reading or modelling process to get information about P', the same effect applies; there's no way to permenantly lock in any broken strategy without specifying a prior of 1.0/0.0 (or cheating and using the Goal System to make the AI /want/ to be irrational).
It's true that you could set up an AI with a prior that 'any attempt to gather evidence will set off a nuclear bomb = 0.9', 'nuclear bomb detonation = -1,000,000 utility', 'making a paperclip = 0.001 utility' and 'estimated number of paperclips that can be made = 1,000,000,000' and it will just sit there and never attempt to gather evidence, even if there is actually no nuclear bomb and plenty of utility to be got by making paperclips. You can always cripple an AGI seed if you're absolutely determined to do so. But I can't see any real-world scenarios where this would happen. -- Starglider
As the prior represents the knowledge we have, and if we had knowledge that another evidential gathering system is better we would surely look at that and then use that as our evidential gathering system in the first place. So generally there will be little evidence in the prior for switching schemes, because either we would have switched ourselves and found it better, or switched and found it worse and switched back.
I apologise I should have added in another probability to my example that shows how important self-referential priors are, assume that we misguidedly put p(low utlity | ~current evidential gathering scheme ) = 0.9 as well as p(high utility| current evidential gathering scheme)=0.9. So that any change is assumed to have a low probability of increasing it as well. Unless there is something that specifically guesses that a certain change will have a high probability of higher utility (which if we had the knowledge of this we would have used instead....)
You still haven't addressed how a system manages to take measurements that update the probability about P(high utility | no evidence gathering) or P(low utility| no evidence gathering), or p(high utility| AI wiped off the face of the earth).
The fact you cannot see real world scenarios where this criticism is important does not mean that one will not occur. There have been many times in the past where people have not thought bugs likely to be important in systems. --Agent 101
Again, your model of how Bayesian reasoning works appears to be at odds with how Bayesian reasoning actually works, at least in the systems I've examined (and am implementing). A system is not constrained to have one 'evidential gathering system' any more than it is constrained to only have a single hypothesis about any one thing at once. Just as a Bayesian system doesn't only rely on the 'best' hypothesis, but rather the weighted sum of all pertinent and tractable hypotheses, it is not constrained to use any particular 'best' model of AI/world interaction that would predict what evidence is likely to be useful. If an AGI has two hypotheses, 'CPU time best spent sitting and contemplating my navel' at 0.9 probability, and 'CPU time best spent gathering evidence about the world' at 0.1 probability, it will devote 90% not 100% of the CPU time to navel-gazing (assuming equal expected utility). The percentage will decrease if acting on the later hypothesis produces evidence that interacting with the world is actually a good way to achieve goals.
I haven't addressed the circumstance of an AGI that is convinced that gathering evidence would be worse than useless because it fundamentally can't be addressed. This is a pathological unrecoverable state, which neither an AGI nor a hypothetical completely rational human would be able to escape. No sane AGI designer would start an AGI in this state, and there's no obvious route by which any well-designed rational AGI could get itself into that state. Non-Bayesian AGIs are equally susceptible to catatonia of this kind, unless they use a lot of outright randomness to break out of local optima in which case they are unpredictable, unstable and unsafe to the point of being useless. With luck the high-level AGI predictive theory Eliezer is working on will eventually allow the former statement to be made formally and provably. -- Starglider
Here's an example of how we can evaluate the utility of evidence-gathering. Suppose an air-conditioner AI has
- P(room is hot) = 0.5
- P(room is cool) = 0.5
Furthermore, our AI has the utility function
- U(hot) = -1
- U(cool) = 1
- U(cold) = 0
Let's say our AI has three options now: Do nothing, cool down the room (lowering the temperature from hot to cool, or from cool to cold), or measure the temperature of the room.
If we do nothing, the room stays at its current temperature, so EU(do nothing) = P(hot) * U(hot) + P(cool) * U(cool) = 0.5*(-1) + 0.5*(1) = 0.
Now suppose we cool down the room. If it's hot, then it becomes cool, and if it's cool it becomes cold. So EU(cool down) = P(hot) * U(cool) + P(cool) * U(cold) = 0.5 * 1 + 0.5 * 0 = 0.5.
Now we want to find EU(measure temperature). For simplicity's sake, we'll say that after measuring the temperature, the AI will know for certain whether the room is hot or cool. Since the AI knows itself to be an expected utility maximizer, it infers that, should it discover the room to be hot, it will take action to cool the room (you can verify this by looking at the expected utility of various actions given that P(hot) = 1), and that if the room is cool, it will do nothing (ditto).
So EU(measure temperature) = P(hot) * EU(AI knows the room is hot) + P(cool) * EU(AI knows the room is cool) = P(hot) * EU(room is hot and AI cools it) + P(cool) * EU(room is cool and AI does nothing) = P(hot) * U(cool) + P(cool) * U(cool) = 0.5 * 1 + 0.5 * 1 = 1.
So, overall, EU(do nothing) = 0, EU(cool room) = 0.5, and EU(measure temperature) = 1, so measuring the temperature of the room is the AI's best action.
