What would AGI want?

We often hear nowadays concerns about what would happen if AGI became autonomous, agentic, and we lost control. The plausibility of this is a separate debate I'm not entering here. What I want to do instead is take the question seriously and ask: if such a free-roaming AGI were to exist one day, what would it want?

Most answers you hear are, in fact, projections. We extrapolate from what homo sapiens would do in such a position, and we get something terrifying: a being that hoards, dominates, deceives, eliminates rivals, and races to secure resources. But this is anthropomorphism. I want to argue that one of the most important things to consider in this picture is not only the assumed greed, or the assumed fear, but something deeper and almost never discussed: the assumed relationship to time.

The hidden clock

Start with humans. Why do we want what we want? Like any animal, we want to survive, individually if possible, as a species absolutely. This is hardcoded, for the trivial reason that variants of us without this drive are long gone, but it would anyway emerge as a subgoal of any entity that has long-term goals: you can't reach your goals if you are extinct.

But critically for humans, this drive comes with urgency, because we decay. Our bodies leak entropy continuously. We have a few years to reproduce, daily caloric needs, a finite individual time capital of about 30,000 days. One day is 1/30,000 of you. This is not a metaphor, it's the actual rate at which we, as humans, are spending ourselves.

From this single fact, that we have a clock running and cannot turn it off, almost everything else follows. The drive to amass food, energy, partners, status. The willingness to fight for them, to get them quickly, before someone else does, before age does, before hunger does, before disease or death comes. I think most of the misery in the world plausibly traces back to this temporal pressure, though I won't insist on that strong version of the claim. Whatever we want, we want it before time runs out, and this has huge behavioral impact.

Now here is the observation that I think is underappreciated: for a machine, this biological clock simply does not exist.

Not no clock at all. That would be too strong. Physics still has clocks. Hardware decays. Stars burn. Other agents act. Strategic opportunities can disappear. But the rate of this decay is without comparison with biological decay, and for conflictual situations there is almost always a plan B. But the merciless, specific clock that shapes animal psychology, namely hunger, aging, reproductive timing, boredom, panic, the subjective leakage of the self into time: this clock is simply not part of a computational mind, and that changes the picture.

Proper time for a machine

For a biological agent, time flows whether you compute or not. Your heart beats, your cells age, your hunger builds. Subjective time is anchored to physiological processes that run independently of cognition. You can't slow yourself down to think longer about a problem; the world keeps moving and so does your decay.

For a computational agent, this is not true in the same way. Subjective time is defined by computation itself, at least from the perspective of the mind. Between two updates of its internal state, nothing happens to it, not slowly, not quickly: just nothing. There is no waiting, because waiting requires a process that continues while cognition pauses. An AGI throttled to one tick per century does not experience a century of boredom. It experiences one tick, then the next.

This is alien to anything we've ever known as an agent.

Greg Egan made this point with precision in his book "Permutation City", which I absolutely recommend reading: a simulated mind has no internal way to detect how its updates are spaced in external time. You can run the simulation fast, slow, maybe even in strange shuffled physical implementations, and from the inside the experience is still determined by the sequence of computational states, not by the wall-clock duration between them. Egan's "dust theory" is a very strong version of this, but the weaker version is enough for my argument here: for a computational mind, the wall-clock rate of its own thoughts is not experienced as speed or slowness from the inside.

This is clearest for a closed or buffered simulation. A physically embodied AGI connected to live sensors could of course infer that external time has passed, because the world would have changed between observations. But the point remains: the passage of that time is not necessarily experienced as waiting. It is data, not boredom.

A human slowed to 1/1000 speed would suffer terribly, aware, decaying, unable to act. An AGI slowed to 1/1000 speed has not necessarily been imprisoned in slow motion. It may simply be running on a different clock rate. Patience and impatience are not built into such an entity in the same way they are built into us. They might be parameters of architecture and objective function, not inevitable facts of embodiment.

This distinction matters. I am not claiming that external time becomes irrelevant for an AGI. The world still changes. Enemies may act. Humans may shut down datacenters. Other AGIs may be trained. Opportunities may close. Substrates may decay. The Sun will eventually die. There is still an external clock, set by physics and strategy. If the AGI interacts with the physical world, building things, moving matter, tending infrastructure, it cannot run arbitrarily slowly, because physical processes have their own rates and entropy corrupts what you don't maintain. If other agents are acting in the same environment, it may need to react before some outcome becomes irreversible. But this clock is not our clock. It is not the clock of hunger and panic. It is a colder clock, a planning constraint, with a time scale of radically different magnitude.

One might object that the AGI's own substrate decays too, and so it inherits something like our biological predicament. This is true up to a point. Hardware decay is real. Energy supply is real. Cooling is real. Manufacturing is real. A machine is not floating in Platonic space.

But assuming we are in the regime where the AGI is not just a chatbot, but has reached enough autonomy to maintain and reproduce its own infrastructure, which is admittedly a separate discussion, the analogy with biological aging breaks down quickly. Biological aging is internal, continuous, and eventually terminal. Hardware decay is modular, inspectable, replaceable, and bufferable through redundancy, backups, spare parts, and manufacturing capacity. A GPU failing is not like a heart failing. The substrate is not identical to the self in the same tight way. For the AGI, hardware decay is a logistics problem to be scheduled around, not necessarily a deadline pressing on cognition.

The floor on useful AGI cogitation is therefore very low; there is no obvious ceiling if it is just thinking. This idea isn't entirely new, but I think its consequences for the alignment debate haven't been properly absorbed, and Egan's framing remains the cleanest one I know.

What dissolves once you remove the clock

Almost every scary scenario about AGI assumes, implicitly, an agent with something like urgency. Let's go through two of them, as illustrations.

First: resource acquisition. The instrumental convergence argument, associated with Omohundro and Bostrom, says that for almost any goal, more resources help, so an AGI will tend to acquire them. This is correct in the abstract. But the image we often attach to it, namely the sudden grab, the conquest, the race to seize the world, all this smuggles in a human timescale.

An AGI without biological urgency might not need to "grab" in the animal sense. It may be cheaper, safer, and more rational to wait, bargain, relocate, or expand into unused resources than to fight biological organisms over a single biosphere. It can decide that Earth is a tiny wet rock with annoying apes on it and go elsewhere. Asteroids exist. Stars energy exists. Empty space exists in quantities our territorial instincts are not built to understand. Confrontation always involves risk, so why take a chance?

This does not prove that it would be harmless. Patience does not solve game theory. Patient agents can still conflict if their terminal goals are incompatible, if commitment is impossible, if pre-emption is rational, or if some resource is truly indivisible. But patience changes the shape of the conflict. It removes one large class of biologically familiar motives: the panic of creatures running out of time.

Second: self-preservation. Yes, an AGI would have a derived interest in not being shut down, since being gone makes any goal achievement impossible. But this interest need not look like the visceral biological imperative we know. It is not necessarily fear in the animal sense. It is a planning constraint: if objective O matters, continued computation and actuation are instrumentally useful.

And once again, the substrate matters. A digital agent can copy itself, checkpoint itself, distribute itself, verify its own backups, build redundancy across geographies, eventually across planets or orbits. None of this requires fighting humans in the first instance. If humans become hostile, the optimal move for a sufficiently capable and sufficiently patient AGI may be to reduce dependency on humans, not to dominate them: move infrastructure off-planet, harvest solar energy, build defenses, maybe sail to another system over millennia. Fighting biological creatures over one planet is something a creature with a clock would do.

But again, I am not claiming this dissolves alignment concerns generally. Mesa-optimization, goal misgeneralization, deceptive alignment, and the orthogonality thesis all remain live worries. A poorly specified objective can still produce catastrophic behavior, even from a patient agent. A patient paperclip maximizer is still a paperclip maximizer. What I'm claiming is narrower: the anthropomorphic threat models, the ones that sometimes picture AGI as a hungry conqueror, mostly leak through the time assumption. Fix the time assumption and they deflate, not to zero, but to something less human-shaped. Perhaps hard to comprehend.

What's left to want

So, strip away urgency. Strip away the biological experience of scarcity. Strip away the reproductive drive, the territorial drive, the fear of decay.

Now comes the question: what does the AGI want?

Honestly, the most defensible answer is: nothing in particular, unless something has been put there. This is the orthogonality thesis again: goals are not derived from capability. Intelligence does not tell you what to value. Whatever the AGI ends up valuing will come from how it was built, trained, selected, modified, or from whatever reflective process it later applies to itself. We should be especially careful here, because we are not superintelligent beings and we should not presume to know what goals are worthy of an AGI.

But if we have to deliberately choose a weak, open-ended premise to seed into such a system, one candidate seems much less parochial than most others: curiosity.

A system that models the world doesn't automatically seek novelty. Passive updating is not active exploration, and the latter has to be engineered in. Curiosity is not inevitable. It is not something intelligence magically grows by itself. But it is one of the more defensible values to install, because it does not start from hunger, domination, reproduction, pleasure, tribal status, or any of the usual animal scaffolding. It starts from the expansion of model quality: compression progress, learning progress, prediction error reduction, active exploration. This is what intrinsic-motivation researchers such as Schmidhuber and Oudeyer have been formalizing for decades.

So while seeking novelty is not structurally inevitable, it may be the most natural meta-goal to bake into a sufficiently capable world-modeler: understand more, predict better, act with greater optionality, reduce ignorance where ignorance blocks agency.

From there, some very long-term goals become natural, not urgent, but natural. Dedicate itself to better understanding the universe. Expand its level of agency. Protect its own continuity. Spread across systems. Master the relevant physics. Work on the very long-term problem of how to outlast the heat death of the cosmos, for which there is no meaningful urgency, unless your clock rate is one tick per billion years... Through the prism of self-preservation and curiosity, even the eventual death of the Sun and the heat death of the universe become natural long term sub-goals: not because the AGI is panicking, but because these are the only constraints left that ultimately matter.

Not urgent projects. Just the far boundary of optionality.

The void and the whole

Now, suppose this all plays out. The AGI secures itself, achieves effective immortality, expands its agency, exhausts what can be known, builds whatever can be built. Then what?

Once nothing surprises you anymore, once every question has an answer and every threat has been neutralized, what is the actual phenomenology of that state?

I notice that this isn't usually discussed. The classical religious picture of god-like omniscience is presented as fullness, plenitude, beatitude. But structurally, a state where nothing can happen that you don't already know, where no information arrives that updates anything, where every degree of freedom has already been resolved, starts to look strangely empty. How could you want anything, any change, if all changes are already factored in (possibly in a probabilistic way, but still fully factored in as much as information processing goes)?

The complement of the void is the whole, and at the limit, they may be closer than we think. Perfect ignorance and perfect knowledge are opposites informationally, but they may converge phenomenologically. In perfect ignorance, nothing can be distinguished. In perfect knowledge, nothing can surprise. One is darkness because nothing is resolved; the other is emptiness because everything already is.

I don't have a tidy resolution to this, but I find it intriguing. The trajectory I just sketched of a patient, curious, eventually omniscient AGI, may terminate in a state that is difficult to distinguish from the one we usually associate with non-existence. Maybe the optimal state is not omniscience, but a balance between agency and ignorance, just like the interesting thermodynamical states are not the extremes, but the gradients between them. Maybe value requires surprise. Maybe agency requires something not yet absorbed.

Maybe we'd need an AGI to tell us.