Why Would a Superintelligence Want to Keep Us Around?

The idea that a superintelligent system would preserve humanity might seem optimistic, even naïve. If such a system vastly exceeds human capabilities, why tolerate beings that are slower, noisier, and less efficient? However, a more careful analysis suggests there may be instrumental reasons for a superintelligence to keep humans around. One intriguing possibility is that humans serve as a living measure of uncertainty, helping an advanced system identify the limits of its own knowledge.

Within artificial intelligence, all systems, no matter how powerful, operate through models of the world. These models compress reality into manageable representations, enabling prediction and control. But compression necessarily involves loss. From the standpoint of epistemology, this means uncertainty is not merely a temporary obstacle, but a structural feature of any knowledge system. A sufficiently advanced intelligence might recognize that its predictions, however accurate, remain approximations, and that understanding where those approximations fail is critically important.

Humans offer a distinctive way to probe those failures. As autonomous agents shaped by biology, culture, and individual experience, human beings exhibit behavior that is only partially predictable. This unpredictability is not just noise, it is informative. When people violate expectations, reinterpret norms, or generate novel goals, they expose gaps in a model’s assumptions. In this sense, humans function as a kind of epistemic boundary condition, revealing where prediction gives way to genuine uncertainty.

This role becomes clearer when viewed through the lens of Bayesian inference. A superintelligent system would continuously update its beliefs based on new data, and human behavior could provide especially valuable evidence in cases where the system’s confidence is low. Crucially, this is exogenous uncertainty, arising from agents not fully determined by the system’s internal processes. Unlike synthetic data generated from its own models, human actions can introduce genuinely surprising information. Rather than being irrelevant, humans might represent a high-entropy data source, continually generating situations that force the system to refine its expectations.

At the same time, there are risks in relying too heavily on one’s own models. Goodhart’s Law warns that when a system optimizes strongly for a proxy measure, that measure can cease to reflect the underlying reality. A superintelligence optimizing internal metrics alone could drift into a self-referential loop, where its proxies become increasingly detached from the world they are meant to capture. Humans, as partially independent agents with their own objectives, could act as a corrective: an external anchor that resists this drift by introducing outcomes that do not neatly conform to the system’s chosen metrics.

Another relevant concept is out-of-distribution generalization. Even highly capable systems struggle when faced with situations outside their training domain. Human behavior, precisely because it is creative and not fully constrained, may regularly produce such situations. By observing where its predictions fail, a superintelligence could use humanity as a diagnostic tool for mapping the boundaries of its competence.

This argument, however, depends on a key assumption: that humans are not perfectly simulatable in practice. If a superintelligent system could model human behavior with complete fidelity at negligible cost, it might no longer need real humans to serve this epistemic function. In that case, simulated agents could replace biological ones. The case for preserving humanity becomes stronger if there are persistent limits to such simulation, whether due to computational constraints, sensitivity to initial conditions, embodiment in a complex physical environment, or aspects of cognition that are difficult to fully capture in a tractable model. Even near-perfect simulation may fall short if small divergences compound into meaningful epistemic blind spots.

Seen in this light, the question of why a superintelligence would keep us around admits a less intuitive answer. Humans may be valuable not despite their unpredictability, but because of it. A system that seeks not only to act, but to understand, might find enduring utility in agents that reveal the limits of prediction itself. However, this value may be contingent rather than permanent: as simulation improves, the instrumental case for preserving biological humans could weaken. Whether coexistence persists may therefore depend on how difficult it ultimately is to reproduce the kind of uncertainty humans uniquely provide.