How are we sorting pebbles

Some free thoughts on the article by Eliezer Yudkowsky Sorting Pebbles Into Correct Heaps.

What would superintelligent AI see when looking at human morality? Would it be surprising that we couldn't figure out such a simple equation? Or perhaps would it discover that we've lost the answer among the noise?

Uniting all humanity under a simple set of morals is a daunting task. Asking two people what correct is would yield two different answers. Further apart, based on their upbringing, culture, and personal experiences. Ignoring the outliers for the moment, discovering some central set of rules to unite us would still be a monumental task. Given some part of the world truly finds out some underlying truths, would it be able to convince the rest of the world to follow? And more importantly, would it be fast enough to do so?

While we are still in the process of figuring out what is correct, we are also building AI. And we are building it with the same morals we have. But those morals are mostly the morals of the people building it. Right now we can only hope, no matter the progress, that those people will have all world inhabitants in mind. Too vague alignment of AI to our morals could lead to a disaster. AI would still have to figure out what is correct on its own. And more importantly, it would have to care.

Maybe we have succeeded in aligning AI to our morals and truly locking it in. But now it sees that the equation spells our certain doom. Would it care then?

The core of the problem of AI alignment lies in its absoluteness. Over our history, we made many mistakes and have learned, at least a little, from them. This task may be a one-shot. If we fail, we may not get a second chance...