Hard Problem
According to Coherent Extrapolated Volition, the hard part of the problem of creating Friendly AI is "designing a framework for an abstract invariant that doesn't automatically wipe out the human species."
Some sobering facts about SIAI's progress on this issue as of Sep 2004:
- SIAI's researchers still consider themselves to be "walking up to" the Hard Problem.
- So far, M subproblems of the Hard Problem have been solved, but the Hard Problem consists of N subproblems where little is known about N except that N > M.
- Mostly of historical interest: CFAI "solves" L problems where L < M < N, and many of the L solutions are wrong.
The Hard Problem is extremely difficult and abstract. Outsiders who follow SIAI's progress as a hobby, and who haven't spent massive amounts of time reading, absorbing and understanding the necessary technical material, are unlikely to provide major insights.
This here to scare us 'dabblers'?
- Marc has a megalomaniacal gleam in his eyes and a feverish expression on his face. He grins as he returns to his research and continues adding code to his developing seed A.I. He doesn't really know what the hell he's doing but what the heck. Come on..take a risk...live a little :D And if I destroy the world.. well all I can say is...Oops*
Comitting almost certain mass murder (gigadeath scale), with incidental suicide, isn't something I'd define as 'living a little'. Humour may be healthy, but trivialising the risk is a bad idea when so many potential seed AI developers are already doing their level best to avoid thinking about the implications of what they're doing.
-- Starglider
Would the 'hard problem' be so hard for an extreme libertarian, as opposed to a collective volition FAI? Matt Vere
Yes. This is a technical problem that is largely independent of the optimising effect we want the AI to have on the outside world; the basic stability issues are present for any plausible 'Friendliness content'. -- Starglider
So, how much of the development required to produce a stable AI 'morality' is independent of the 'morality' in question? -- Matt Vere
We won't know for sure until such a system is successfully developed. I'd say that the majority of the 'Goal System' design work isn't directly related to the specific optimising effects required, but that the latter may have some implicit constraints that strongly affect the whole process (e.g. whether you can express what you want in terms of expected utility or not, and if so how the UF is grounded). -- Starglider
As Eliezer Yudkowsky wrote in http://sl4.org/archive/0305/6654.html :
To work, the theory of FAI is going to have to dig down to a point where the theory is described *entirely* in terms of: a) things that physically exist in external reality b) incoming sensory information available to the AI c) computations the AI knows how to perform In short, what's needed is a naturalistic description of moral systems building moral systems. I agree it's alarming that I don't have the full specification of the entire pathway in hand at this instant, and I'm working to remedy that. But you surely would not find it in a small set of guidelines.
[edit] in progress
Eliezer wrote in http://www.sl4.org/archive/0602/14076.html (Feb 2006) :
Reflective decision theory - a theory of motivationally stable self-enhancement - is the world's second most important math problem. The *most* important math problem is how to phrase the motivational invariant itself. A classical utility function *probably* isn't going to cut it. My suspicion is that being able to build a reflective decision system, I would know a great deal more about my options for motivational invariants, and how to structurally describe those structurally complex things that humans want - such as "free will" or "freedom from having one's life path too heavily optimized by outside sources as opposed to one's own efforts". I am doubtful I can solve the most important math problem without having solved the second most important math problem first. Sadly and dangerously, FAI knowledge *always* lags behind AGI knowledge because AGI is a strictly simpler problem.
see also Hard Problem, the
