Weak And Strong Self Improvement
General AI can be designed and developed, from the beginning, to use its intelligence to improve its own “content” (the subcomponents of its intelligence), which, if improved, can in turn achieve improvements to its overall intelligence (the external process of that content). This form of self-improvement, where improvements to content can improve the underlying cause of that content, can be viewed as a strong self-improving process. The two forms of self-improvement that we are familiar with are, one, the acceleration of language and overall intelligence development in the Hominid family that resulted, in a simplified way, because better language and better overall intelligence created enough survival and reproductive advantages, and made available enough new design possibilities, so that even better language and overall intelligence became evolutionarily and reproductively advantageous. (Despite the speed up, the process of evolution was fundamentally still the same process during the accelerated development of Hominids.) And the second weakly self-improving process that we're familiar with is the slow exponential improvement in our species knowledge and technology that has been occurring for tens of thousands of years.
The exponential acceleration of cultural knowledge and technology development that Homo sapiens sapiens have experienced was probably caused by increasingly rapid changes to the brain configuration of members of the Hominid family. Since the evolutionary split of Hominids from primates, there has been essentially a three times increase in overall brain size, and about a six times increase in prefrontal cortex size. These changes are good proximate causes as to why such exponential increase in cultural knowledge and technology has occurred within our species, though it’s important to note that this process of knowledge and technology improvement hasn’t had enough time to actually cause changes to the brain itself. The main reason why exponential growth in cultural knowledge and technology has occurred is because a certain amount and quality of knowledge and technology allows for the creation of an increased amount and increased quality of knowledge and technology. Yudkowsky, in “Levels of Organization in General Intelligence,” has a good summary of these two self-improvement processes. He states that the accelerated development of language and overall intelligence in Hominids, and the exponential increase of cultural knowledge and technology in Homo sapiens sapiens, are both examples of weakly self-improving processes, because they both occur from external processes, with evolution in the first case and the human brain in the second, that act upon a pool of synergetically interacting content, with Hominid genes being in the first pool and cultural knowledge in the second. The external processes of evolution and the human brain act with a constant level of intelligence on their respective content, with positive feedback effects occurring within that respective content, under the constraints of that content’s external process. Basically, this means that weakly self-improving processes are distinguished by the ability to have an external process act on its content, which can improve over time due to positive feedback, and by the inability to use that improved content to improve the content’s external process. However, a strong self-improving process, such as seed AI, will be able to do both. Seed AI will be able to use its external process of overall intelligence to act on its content, the many subcomponents of its intelligence, and to have improvements to that content result in improvements to the content’s external process—the seed AI's overall intelligence.
OK! What’s the point of the previous long explanation? Overall, I think it’s important to realize that a strong self-improving process has never before existed, to our knowledge. Analogies from a weakly self-improving process to a strongly self-improving process are likely to breakdown, in light of the large differences between the two. Just as there are large differences between the process of “survival of the stable state” and “survival of the fittest,” there will be even larger differences between the process of exponential cultural growth by human brains and the process of Recursive Self-Improvement in intelligence by human-surpassing General Intelligence.
-- Anand
I suspect one of the ways the difference between weak and strong self-improvement will show is the shape of the curve: a weakly self-improving processes cannot get beyond an exponential curve (e.g., humanity's knowledge doubling every X years, or computing power per dollar doubling every 18 months), whereas strongly self-improving processes can go hyperexponential. --CliffStabbert
