Early Or Late Risks
(A conversation between Eliezer Yudkowsky and Tantek Celik, experimentally conducted via Wiki.)
Tantek: Nuclear disaster based on terrorism / Peak Oil both likely to appear before grey goo, other Singularity-based disasters; could make Singularity-based disasters irrelevant in the short term.
Eliezer: Recursively self-improving AI has unpredictable timeframe, and even if it had a knowably long timeframe would still potentially require a long lead time to address.
Alex F. Bokov apologetically butts in:
Perhaps, but in order insure the continuation of your work on AI, you need to have...
1) A plan for continuing operations in the midst of a "short-term" dark ages / die-back during which grid power is unavailable, mass produced tools and components are scarce, and the rule of law cannot be relied on. This might entail procuring a facility with plentiful stockpiles of spare computer parts and on-site renewable energy generation.
2) In the event that despite your best efforts plan 1 fails, a second plan for gracefully putting your operation into hibernate mode in a manner that maximizes the likelihood of future generations picking up where you left off. This might entail archiving your data in a multiple, highly redundant and durable formats and spawning off some sort of organization whose goal is to preserve and pass on the knowledge needed to reconstitute the SIAI once enough of the technological infrastructure has recovered.
I'm not saying that you need to devote all your energies to this and away from your real purpose, but there is a non-zero level of effort you should devote to early risk mitigation. Since, as you said, you're dealing with an unpredictable time frame, you should factor in the possibility of social/economic/environmental disasters occuring during that period. If you willfully blind yourself to these very real possibilities you are unnecesserily and irresponsibly endangering everything you have worked so hard for.
Good luck.
