Controlled Ascent

From the SL4 Lexicon:

Controlled Ascent:

An Ethical Injunction for AIs which states that self-improvement should not outpace programmer guidance, because this increases the probability of Catastrophic Failure of Friendliness. A consequence of this instruction would be that, if self-improvement suddenly accelerates, it may be necessary to consult the programmers before continuing, or to deliberately slow down for some period of time in order for Friendship to catch up with intelligence. For very young AIs, "Controlled Ascent" may be a programmatic feature triggering a save-and-stasis if a metric of self-improvement suddenly accelerates, or if the metric of self-improvement outpaces a metric of programmer guidance Yudkowsky01. See also Friendly AI, Singularity-Ready, Takeoff-Safe AI.