One of the most interesting (and also frightening) things about AI is how difficult it is to predict what will happen. My kids ask me what's going to happen, and all I can say is that there will be huge changes, and I can't predict what they'll be.

The development of full artificial intelligence could spell the end of the human race. We cannot quite know what will happen if a machine exceeds our own intelligence, so we can't know if we'll be infinitely helped by it, or ignored by it and sidelined, or conceivably destroyed by it.

Humanity is at a crossroads. Either it returns to the belief that it has a different nature than machines or it will be reduced to a machine among machines. The risk is not that artificial intelligence will become better than us, but that we will freely decide to submit to it and its masters.

With the help of AI in advancing our civilization, there may come the day when, as Nick Bostrom notes, we cannot be sure that we’re “not already in a machine”.

We will have something that is, for the first time smarter than the smartest human. It's hard to say exactly what that moment is, but there will come a point where no job is needed. You can have a job if you wanted to have a job for personal satisfaction. But the AI would be able to do everything. I don't know if that makes people comfortable or uncomfortable. If you wish for a magic genie, that gives you any wish you want, and there's no limit. You don't have those three wish limits nonsense, it's both good and bad. One of the challenges in the future will be how do we find meaning in life.

If a superior alien civilisation sent us a text message saying, 'We'll arrive in a few decades', would we just reply, 'OK, call us when you get here, we'll leave the lights on'? Probably not, but this is more or less what has happened with AI.

There was a conference one time on: "What are we going to do about the looming risk of AI disaster?" … And what came out of that conference was OpenAI, which was basically the worst possible way of doing anything. Like, "This is not a problem of, 'Oh no, what if secret elites get AI', it's that nobody knows how to build the thing." … So, like, "Let's open up everything! Let's accelerate everything!" It was like ChatGPT's blind version of throwing the ideals at a place where they were exactly the wrong ideals to solve the problem. … And that was it. That was me in 2015 going, "Oh. So this is what humanity will elect to do. We will not rise above. We will not have more grace, not even here, at the very end." So that is when I did my crying late at night, and then picked myself up and fought and fought and fought until I had run out all the avenues that I seem to have the capability to do.

There are three things that I think are important:
Truth, curiosity, and beauty.

If AI cares about those three things, it will care about us.

Truth will prevent AI from going insane.
Curiosity will foster humanity.
And if it has a sense of beauty, it will be a great future.

Kurzweil himself has pegged the date when AIs will do everything better than humans at 2029.29 (As explained in Abundance, his predictions are based on exponential growth curves and have an amazing track record for accuracy.)

When I launched my AI career in 1983, I did so by waxing philosophic in my application to the Ph.D. program at Carnegie Mellon. I described AI as “the quantification of the human thinking process, the explication of human behavior,” and our “final step” to understanding ourselves. It was a succinct distillation of the romantic notions in the field at that time and one that inspired me as I pushed the bounds of AI capabilities and human knowledge.
Today, thirty-five years older and hopefully a bit wiser, I see things differently. The AI programs that we’ve created have proven capable of mimicking and surpassing human brains at many tasks. As a researcher and scientist, I’m proud of these accomplishments. But if the original goal was to truly understand myself and other human beings, then these decades of “progress” got me nowhere. In effect, I got my sense of anatomy mixed up. Instead of seeking to outperform the human brain, I should have sought to understand the human heart.
It’s a lesson that it took me far too long to learn. I have spent much of my adult life obsessively working to optimize my impact, to turn my brain into a finely tuned algorithm for maximizing my own influence. I bounced between countries and worked across time zones for that purpose, never realizing that something far more meaningful and far more human lay in the hearts of the family members, friends, and loved ones who surrounded me. It took a cancer diagnosis and the unselfish love of my family for me to finally connect all these dots into a clearer picture of what separates us from the machines we build.
That process changed my life, and in a roundabout way has led me back to my original goal of using AI to reveal our nature as human beings. If AI ever allows us to truly understand ourselves, it will not be because these algorithms captured the mechanical essence of the human mind. It will be because they liberated us to forget about optimizations and to instead focus on what truly makes us

Notice that the story [of technical progress accelerating indefinitely] is not testable; we just have to wait around and see. If the predicted year of true AI's coming is false, too, another one can be forecast, a few decades into the future. AI in this sense is unfalsifiable and thus — according to the accepted rules of the scientific method — unscientific.

[I]f you look at the progression of AI models, it... went the opposite direction. ...AI started with linguistic protocols, which were expressed in formal grammars, and then it got to concept spaces, and now it's about to address percepts. ...At some point in the near future it's going to get better at mental simulations and at some point after that we'll get to attention directed and motivationally connected systems that make sense of the world, that are in some sense able to address meaning. This is the hardware that we have...

Once artificial intelligence matches our own, won’t they then design even better ai minds? Then better still, with accelerating pace? At worst, might they decide (as in many cheap dramas), to eliminate their irksome masters? At best, won’t we suffer the shame of being nostalgically tolerated? Like senile grandparents or beloved childhood pets?

The debate around AI is too focused on whether AI is going to kill us or not. It’s like talking about flying cars and how we're going to regulate them when we haven't regulated the cars that are on the roads.