If greater-than-human artificial general intelligence is invented without due caution, it is all but certain that the human species will be extinct in very short order.

Part of why predicting the ending to our AI [artificial intelligence] story is so difficult is because this isn’t just a story about machines. It’s also a story about human beings, people with free wills that allows them to make their own choices and to shape their own destinies. Our AI future will be created by us, and it will reflect the choices we make and the actions we take.

The last thing we want is a nasty divorce between humans and superintelligent machines, for that would certainly spell the end of the human race.

Humanity is at a crossroads. Either it returns to the belief that it has a different nature than machines or it will be reduced to a machine among machines. The risk is not that artificial intelligence will become better than us, but that we will freely decide to submit to it and its masters.

The end of the human race will be that it will eventually die of civilization.

In another thirty to fifty years, the demand for cheap labor will have produced even more machines over the employment of actual humans. And in that time frame, humans will have lost their voice, their power, all freedoms, and all worth. It is inevitable that machines will one day become the ultimate enemies of mankind. We are not evolving or progressing with our technology, only regressing. Technology is our friend today, but will be our enemy in the future.

The danger is that if we invest too much in developing AI and too little in developing human consciousness, the very sophisticated artificial intelligence of computers might only serve to empower the natural stupidity of humans.

Not even the most advanced form of artificial intelligence can ever replace man. Because there is something in human beings that is irreducible to machine knowledge: self-awareness, free will, doubt, feelings.

With the help of AI in advancing our civilization, there may come the day when, as Nick Bostrom notes, we cannot be sure that we’re “not already in a machine”.

an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind.

The debate around AI is too focused on whether AI is going to kill us or not. It’s like talking about flying cars and how we're going to regulate them when we haven't regulated the cars that are on the roads.

Our demise may instead result from the habitat destruction that ensues when the AI begins massive global construction projects using nanotech factories and assemblers—construction projects which quickly, perhaps within days or weeks, tile all of the Earth’s surface with solar panels, nuclear reactors, supercomputing facilities with protruding cooling towers, space rocket launchers, or other installations whereby the AI intends to maximize the long-term cumulative realization of its values. Human brains, if they contain information relevant to the AI’s goals, could be disassembled and scanned, and the extracted data transferred to some more efficient and secure storage format.

I almost think it is the ultimate destiny of science to exterminate the human race.

If we continue to accumulate only power and not wisdom, we will surely destroy ourselves. Our very existence in that distant time requires that we will have changed our institutions and ourselves. How can I dare to guess about humans in the far future? It is, I think, only a matter of natural selection. If we become even slightly more violent, shortsighted, ignorant, and selfish than we are now, almost certainly we will have no future.