Do not flinch from experiences that might destroy your beliefs. The thought you cannot think controls you more than thoughts you speak aloud. Submit … - Eliezer Yudkowsky

The human brain cannot release enough neurotransmitters to feel emotion a thousand times as strong as the grief of one funeral. A prospective risk going from 10,000,000 deaths to 100,000,000 deaths does not multiply by ten the strength of our determination to stop it. It adds one more zero on paper for our eyes to glaze over.

Lies propagate, that’s what I’m saying. You’ve got to tell more lies to cover them up, lie about every fact that’s connected to the first lie. And if you kept on lying, and you kept on trying to cover it up, sooner or later you’d even have to start lying about the general laws of thought. Like, someone is selling you some kind of alternative medicine that doesn’t work, and any double-blind experimental study will confirm that it doesn’t work. So if someone wants to go on defending the lie, they’ve got to get you to disbelieve in the experimental method. Like, the experimental method is just for merely scientific kinds of medicine, not amazing alternative medicine like theirs. Or a good and virtuous person should believe as strongly as they can, no matter what the evidence says. Or truth doesn’t exist and there’s no such thing as objective reality. A lot of common wisdom like that isn’t just mistaken, it’s anti-epistemology, it’s systematically wrong. Every rule of rationality that tells you how to find the truth, there’s someone out there who needs you to believe the opposite. If you once tell a lie, the truth is ever after your enemy; and there’s a lot of people out there telling lies.

If you want to build a recursively self-improving AI, have it go through a billion sequential self-modifications, become vastly smarter than you, and not die, you've got to work to a pretty precise standard.