On collective restraint

25 September 2026.

This short post critiques Point 4 of Eliezer Yudkowsky’s AGI Ruin: A List of Lethalities.

He argues that there is a time limit in which we need to figure out how to handle AGI safely, and that restraint on developing and applying AI is not a lasting, permanent solution; at best it can only buy more time. His wording is:

Powerful actors all refraining in unison from doing the suicidal thing just delays this time limit — it does not lift it, unless computer hardware and computer software progress are both brought to complete severe halts across the whole Earth.

To justify this, Yudkowsky argues that “because GPUs are everywhere, and knowledge of algorithms is constantly being improved and published”, it is only a matter of time before “increasingly weak actors with a smaller and smaller fraction of total computing power, become able to build AGI and destroy the world”. These weak actors, it is presumed, have not agreed to any collective restraint.

To steelman this: even if powerful actors have agreed among themselves to slow down, that does not necessarily mean that the newcomers will be so restrained, and due to competitive market pressures, they will be incentivised to cut corners, or underestimate risks. Even if they don’t have malicious intent, in this world where improving technology makes dangerous capability more accessible to weak actors, and all it takes is one sufficiently bad misalignment or failure to ruin the world, it makes sense to be very worried.

What I disagree with is his claim that collective restraint from the powerful actors cannot be a lasting and permanent solution to safely dealing with AGI that might kill us all. I feel, in short, that he is too quick to dismiss what collective human restraint and slowdown can achieve.

Indeed, there is historical precedent for this collective restraint. The Asilomar conference on Recombinant DNA in the 70s was influential and successful in regulating potentially dangerous experiments in biotechnology, and for putting into place best practice for continued research of this technology that still affects the biotech industry and society at large today. It influenced current NIH guidelines that are still part of research oversight today. I see this as evidence that professional norms can spread beyond their original participants and be respected and adhered to across generations.

Where I see this restraint going is to bring about a culture-shift among AI researchers that they are engaged in an incredibly consequential endeavour that could have civilisation-cancelling consequences. Upon entering their studies or research, they should take some sort of AI equivalent of the Hippocratic Oath, reaffirmed upon graduation, that (a) human life is of fundamental and unquestionable dignity and meaning, that (b) they must consider the consequences upon human life and society of any action they take towards creating or modifying any AI, and be held to account for said consequences, that (c) they refuse any work they deem unsafe and immoral, and that (d) their primary responsibility is to humanity itself, not to any company or country. This can create a more cautious mindset among the companies engaged in this work, and will encourage more collective restraint and discourage a ‘race mindset’ that, as Yudkowsky illustrates so candidly, will kill us all. In short, these professional norms can shape the behaviour of the very ‘weak actors’ that Yudkowsky worries about.

None of this is to say that Yudkowsky is necessarily wrong. The progression of events he outlines could totally happen, and it is rational to be very worried about it. I am just pointing out that his argument does not give enough weight to the possibility that scientists and leaders and society actually get organised and create a sea change and culture shift that could significantly mitigate the risk of AGI.