I suspect quite a lot of AI researchers would think of empathy as a very, kind of, second rate form of intelligence, and will confuse it with the mere personality dimension of agreeableness. But in fact empathetic understanding is extraordinarily cognitively demanding. It is worth recalling that what it is like to be another subject of experience, and what it is like to be able to apprehend fourth, fifth, and even -in the case of someone like Shakespeare- sixth order intentionality is as much a property of the natural world as the rest mass of the electron. So in so far as one wants to actually understand the nature of the natural world one is going to want to understand other subjects of experience.

 

Now most of us -and it is true for good evolutionary reasons- are not mirror-touch synesthete. We don’t feel the pain of others as if it were our own, we don’t experience their perspectives as if they were our own. But… I think with a greater full-spectrum of super-intelligence, as one actually comes to understand the perspectives of other subjects of experience, as one starts to obtain this god-like capacity to impartially understand all possible subjects of experience, this will entail expanding our circle of compassion. It is not possible if you are a mirror-touch synesthete to act in a non-friendly hostile way to another sentient being, and I would see a generalization of mirror-touch synesthesia as part and parcel of being a full-spectrum super-intelligence.

 

Any supposedly intelligent being that doesn’t understand the nature of consciousness -that doesn’t understand that there are other subjects of experience as real as you or me right now- is in a fundamental sense ignorant. And if we are talking about super-intelligence, as distinct from savant minds or insentient malware systems, one must remember that by definition a super-intelligence is not ignorant.

 

– David Pearce, in The Mind of David Pearce

David is responding to a question I posed about the relationship between empathy and super-intelligence. One could in principle imagine an Artificial Intelligence system that is capable of designing nanotechnology without ever being aware of other minds. The AI could take over the world from the ground up and never suspect that anyone actually lived in it. The AI could simply model other agents to the extent that is necessary to predict their behavior within the parameters that define the conditions of its success and failure. No need to experience the colors, aromas, feelings and thoughts of humans when you can approximate them well enough with a Bayesian system trained on past observed behaviors.

A lot of people seem to be worried about this sort of scenario. Admittedly, if one thinks that all there is to intelligence is the capacity to optimize a given utility function, then yes, super-intelligences could in principle be completely ignorant of the matter-of-fact about the qualia we are experiencing right now. AI safety organizations and researchers mostly care about this sort of intelligence. As far as the safety concerns go, I think this is fair enough. The problem comes when the view that intelligence is nothing more than utility function maximization conceptually spills over into one’s full conception of what intelligence is and will always be.

The contention here, I think, is the way we conceptualize intelligence. Depending on one’s background assumptions one will end up with some or other idea of what this concept can or cannot mean:

On the one hand, if one starts by exclusively caring about the way autonomous systems behave from a third-person point of view and in turn disregards the computational importance of consciousness, then any talk of “a deep understanding of the nature of other’s experiences” will seem completely besides the point. On the other hand, if one starts from an empathetic mindset that acknowledges the reality -and ethical weight- of the vastness of the state-space of consciousness, one may prefer to define intelligence in terms of empathetic understanding. I.e. as the capacity to explore, navigate, contrast, compare and utilize arbitrarily alien state-spaces of experience for computational and aesthetic purposes.

Most likely, an enriched conception of genuine super-intelligence entails a seamless blend between a deep capacity for introspection and empathy together with the extraordinary power of formal logico-linguistic reasoning. Only by combining the empathizing and systematizing styles of mental activity, together with a systematic exploration of the state-space of consciousness, can we obtain a full picture of the true potential that lies in front of us: Full-Spectrum Super-Sentient Super-Intelligence.