What is the probability that a disaster not related to nuclear weapons will set progress toward human-level AI back decades or centuries? For example, consider runaway climate change, a biotechnologically engineered plague, self-replicating nanomachines, economic collapse, a planetary totalitarian government that restricts technology development, or something unknown.

• Claim: "I think it is no exaggeration to say we are on the cusp of the further perfection of extreme evil, an evil whose possibility spreads well beyond that which weapons of mass destruction bequeathed to nation-states, on to a surprising and terrible empowerment of extreme individuals. An immediate consequence of the Faustian bargain in obtaining the great power of nanotechnology is that we run a grave risk — the risk that we might destroy the biosphere on which all life depends. If our own extinction is a likely, or even possible, outcome of our technological development, shouldn't we proceed with great caution?"
  Implication: The chance of civilization-destroying disaster could be significant.
  Source: Joy, Bill. "Why the future doesn't need us". WIRED, Issue 8.04, April 2000.

• Claim: It's highly unlikely for us, as observers, to be among the first tiny fraction of humans ever born.
  Implication: The total number of humans is probably comparable to the number of humans already born, implying a major population-limiting catastrophe sometime within the next few thousand years. One estimate gives a 95% confidence estimate of a 0.4% to 0.8% chance of such a catastrophe between now and 2070, or about 0.67% - 1.3% per decade if the probability is distributed evenly over time.
  Source: Leslie, John. The End of the World: The Science and Ethics of Human Extinction. New York: Routledge, 1998.

• Claim: Sir Martin Rees, Astronomer Royal and President of the Royal Society, puts the odds for human extinction this century at 50%.
  Source: Rees, Martin. "The Energy Challenge." The World Question Center 2007. Retrieved 9 Aug. 2008. <http://www.edge.org/q2007/q07_15.html>.

• Claim: Oil production will peak soon, which will not kill off the species but will put an end to technology development.
  Implication: Our species will survive, albeit with a more primitive technological base. Progress towards AI would essentially cease.
  Source: Heinberg, Richard. (2003.) The Party's Over. British Columbia: New Society Publishers.

• Claim: The end of the world has been predicted many, many times before, and it's never happened.
  Implication: The human species will continue in the near future, pretty much as it has been.
  Source: Nelson, Chris. "A Brief History of the Apocalypse." 13 Oct. 2005. Retrieved 9 Aug. 2008. <http://www.abhota.info/>.

• Claim: We may systematically underestimate the likelihood of a humanity-destroying disaster because of selection effects. Since we're obviously here and alive, we necessarily live in a time when a humanity-destroying disaster hasn't occurred. It's impossible to learn from catastrophic disasters that wipe out humanity, because when they occur, the observers are eliminated. So species-destroying threats are always theoretical — by the time they actually happen, it's too late to learn from mistakes.
  Implication: The probability of disaster may be higher than many people think.
  Source: Bostrom, Nick. "Existential Risks: Analyzing Human Extinction Scenarios". Journal of Evolution and Technology, vol.9, March 2002. <http://nickbostrom.com/existential/risks.html>

• Claim: We may underestimate extinction risks because they are presented in generalities rather than highly detailed, specific scenarios. Psychological studies have demonstrated an effect called the conjunction fallacy, in which subjects believe that the joint occurrence of two events, A and B, is more likely than the occurrence of A (with or without B). For instance, the vivid image of nuclear war between the US and China over Taiwan may seem more probable than the abstract idea of nuclear war for any reason.
  Implication: The probability of disaster may be higher than many people think.
  Source: Yudkowsky, Eliezer. "Cognitive biases affecting judgment of global risks" Global Catastrophic Risks, eds. Nick Bostrom and Milan Cirkovic.