Agreement Was Never the Input

There is an objection that ends most serious conversations about governing anything at scale, and it ends them because it is good.

Any system powerful enough to reshape the world has to make tradeoffs between things people want that cannot all be had at once. Making those tradeoffs requires some settled account of what matters and how much. We have no such account, we have never had one, and two and a half thousand years of very intelligent people looking is decent evidence that one is not sitting somewhere waiting to be found. So any attempt either quietly encodes one faction's values behind a neutral-sounding description, or it stalls forever. And the historical record is discouraging: every previous attempt to design a universal political order from first principles produced either a document nobody implemented or a catastrophe.

On 8 December 1987, the United States and the Soviet Union signed a treaty eliminating an entire class of nuclear weapon. Starting the following year, American inspectors took up residence outside the Votkinsk Machine Building Plant in Udmurtia, where Soviet missiles were built. They stayed at that gate, continuously, for thirteen years. Soviet inspectors did the same at a rocket motor plant in Magna, Utah. Each set watched the other side's missiles leave the factory and measured them to confirm they were not the banned type.

These two governments disagreed about property, rights, history, and the purpose of a human life. Each regarded the other as an existential threat run by people with a wicked ideology, and each was arming to destroy the other if it came to that. They resolved none of it.

I should be precise here, because the obvious objection is correct. They did share something: both preferred not to be annihilated, and both accepted the logic of deterrence well enough to negotiate inside it. That is a genuine common foundation. It is also about as thin as a foundation can get while still being one, and it is worth noticing what it did not include. It did not include any agreement about what a good society is, who should own what, or what people are for. What it included was a shared preference for continued existence, and on that alone they built throw-weight limits, inspection protocols, and a decade of somebody standing in the snow counting rockets.

That is the pattern this essay is about: how much you can build on how little.

Nothing that works has ever run on agreed foundations

The objection contains a hidden premise: that governing requires agreement on foundations. Not agreement on rules, but agreement on the underlying account of what matters that generates the rules.

Try to find one working institution built that way.

Montreal is the cleanest case. On 16 September 1987, states agreed to phase out chlorofluorocarbons. It came into force in 1989 and has been ratified by every member state of the United Nations, which almost nothing achieves. The Antarctic ozone hole is closing, with the scientific assessments projecting a return to 1980 conditions over the coming decades. Nobody was required to agree about the moral standing of future generations, or whether nature has value apart from use. There was a clear causal model, a chemical substitute that was not much more expensive, a compliance signal visible in trade data, and a restriction on trade with non-parties that made staying outside cost more than joining.

Then there are the institutions nobody thinks about, which is how you can tell they work. The Universal Postal Union has routed mail across hostile borders since 1874. Civil aviation runs on a standard that lets aircraft from states that do not recognise each other share airspace using the same phrases in the same language. Interbank clearing lets institutions that would be delighted to watch each other fail settle their obligations every afternoon.

And global trade was reorganised in the late 1960s by an agreement about the dimensions of a steel box. The corner fittings and the length. Not development, not fairness, not what economies are for.

Two people have named the general pattern better than I can. John Rawls called it an overlapping consensus: parties holding irreconcilable comprehensive doctrines converge on the same institutional arrangements for entirely different reasons. A devout Catholic, a utilitarian and a libertarian all support freedom of conscience, from three premises that cannot all be true, and the arrangement is stronger for it. Cass Sunstein found the same mechanism inside law and called it incompletely theorized agreement: judges routinely agree on the ruling while disagreeing openly about the theory, and the agreement survives precisely because nobody was made to concede the theory.

So the requirement is convergence on mechanisms rather than consensus on foundations. A foundation has to be accepted as true. A mechanism only has to be preferred to the alternative, and it can be preferred from any number of incompatible premises at once. A rule one value system endorses lasts as long as that system holds power. A rule that five mutually hostile value systems each independently prefer to the status quo is overdetermined, and overdetermination is what stability is made of.

The counterexample that nearly works

Except that there is an obvious institution I have just walked past, and it is the strongest case against everything above.

The Catholic Church is transnational, has lasted two thousand years, governs more than a billion people, and explicitly requires agreement on foundations. Canon law, the creed, papal authority. You cannot participate in its governance without affirming its account of what matters. It is durable, it is enormous, and it is built on exactly the thing I have claimed no institution is built on. The German Basic Law makes a quieter version of the same point: it opens by declaring human dignity inviolable and puts that clause beyond amendment, which is a foundational commitment written into the structure on purpose.

The answer narrows my claim, and the narrowed version is the one I actually want.

Institutions built on agreed foundations govern people who have already agreed. Membership is the filter. The Church does not govern Protestants, and its relations with those outside its creed have always run on precisely the mechanism logic described above: concordats, treaties, negotiated arrangements with states that reject its authority entirely. The foundation governs the inside. The mechanism handles everyone else.

So the claim is not that shared foundations never produce institutions. It is this: no institution that governs parties who genuinely disagree about foundations has ever required them to agree first. Where the disagreement is real, the mechanism is the only thing that has ever worked. And the governance problems that matter most now, climate, pandemics, weapons, the behaviour of powerful technology, are all of that second kind. There is no membership filter available. The parties are already in the room and they are not leaving.

What the durable ones have in common

Line up the agreements that held against the ones that collapsed and the difference is not how noble they were. It is a short list of structural properties.

They are verifiable: each party can confirm compliance without trusting the others. This is first because it decides feasibility. Absent it, the arrangement collapses at the first suspicion, and suspicion always arrives.

They are narrowly scoped, covering the coordination and saying nothing about the worldview. Scope creep has killed more agreements than bad faith. The moment a fisheries quota acquires a preamble about development philosophy, it has acquired a new set of enemies and gained nothing. Relatedly they are incompletely theorized, deliberately silent about why, because every sentence of justification is a sentence somebody can refuse to sign.

They offer exit and voice. Parties can leave, and can object from inside. This looks like weakness and is the reverse: a party that can neither leave nor object defects quietly, which is worse than leaving, because you cannot plan around what you cannot see. And they are graduated, small commitments that ratchet, because all-or-nothing arrangements do not start when the first mover carries the entire risk alone.

The sixth is incentive compatibility, and I have to be careful with it, because the obvious formulation is circular. If I say the property is "defecting does not pay," then I can only ever observe it after the fact: the agreements that survived had it, the ones that collapsed did not, and I have explained nothing. That is a synonym for survival wearing the clothes of an explanation.

The non-circular version has to be checkable before the outcome is known, so it is a question rather than a property. At design time, can you name the specific payoff a defector gets, name the specific mechanism that takes it away, and state the margin between them? If the answer is a number, the test has been passed. If the answer is that the parties are acting in good faith, it has not.

That distinction does real work. It is why the Biological Weapons Convention and the Chemical Weapons Convention went differently, which is the next section. But I should say plainly that the list as a whole is a description of what the survivors had, assembled by someone who chose the cases because they survived. Running it against the failures, and the League of Nations, Kyoto and the BWC all fail it, does not fix that, because I built the list already knowing how those ended. It becomes a theory rather than a summary only when someone uses it to predict, in advance, which of two live negotiations will hold. That test has not been run, and until it is, treat this as a well-supported hunch rather than a finding.

Institutions are prostheses for virtues we do not have

The maturity form of the objection deserves its own answer, because it has the causality backwards.

The claim is that we are too immature and confused to agree on foundational questions, and therefore cannot build institutions that depend on such agreement. Consider what institutions are.

We do not have honest accountants. We have double-entry bookkeeping, external audit, and personal criminal liability for signing a false statement. The system assumes dishonesty and makes it leave a trace.

We do not have wise rulers. We have separation of powers, fixed terms and judicial review, an architecture designed by people who assumed the ruler would be foolish or malicious and wanted the damage bounded in advance.

We do not have rational investors. We have disclosure rules, capital requirements and circuit breakers that halt trading after a defined percentage move, all of which exist because the participants reliably panic and everyone involved knows it.

We do not have careful surgeons at four in the morning. We have the checklist.

Every durable institution is a prosthesis for a virtue the population does not reliably have. That is the design intent. The institution is what you build because the virtue did not show up.

So "we are too immature to agree, therefore we cannot build the institution" reads as: we cannot fit the prosthetic until the limb grows back. Institutions substitute for maturity, and then produce some of it downstream by changing what behaving well costs. Societies do not become honest and then invent audit. They invent audit and become somewhat more honest.

Most moral disagreement is not moral disagreement

Now the part that does the practical work, with one claim flagged, because I would rather mark a conjecture than have it found later.

Start with something structural. Wherever interventions can be compared on cost-effectiveness, the spread runs to three or four orders of magnitude. In global health the gap between the most and least efficient way to avert a death is roughly a factor of a thousand. Essentially every value system, asked to choose between the top and the bottom of that distribution, picks the top, and they do it without conferring. Real disagreement appears at the margin, between options already close in effect, which puts it exactly where being wrong costs least. That is a structural gift and almost nobody designs around it.

Then the conjecture: most arguments that present as moral are empirical arguments in costume.

Take the live disputes. Minimum wage. Rent control. Drug decriminalisation. Nuclear power. Immigration levels. Zoning. Listen to what participants say when pressed rather than what they say in public. They rarely differ about whether human flourishing is good or whether fairness matters. They differ about what happens if you do the thing. Does a higher wage floor reduce employment, for whom, by how much. Does building housing lower rents or induce demand. Does nuclear power kill more or fewer people per unit of energy than what it replaces. Those are causal questions, and they are a different category from the trolley problem.

I have not measured this. I believe it from pattern-matching across a lot of arguments, which is the sort of evidence that feels strong and is weak, and I notice it is also extremely convenient for the argument I am making. It is testable: take a few hundred contested questions, elicit from a large sample both their value weightings and their causal beliefs, and see which explains more of the variance in stated position. If values dominate, I am wrong and most of the next section goes with me. As far as I can find, nobody has run it.

Where genuine value conflict does remain, the output is a procedure rather than an answer. You do not agree on the verdict, you agree on the court, and you choose it before knowing how it will rule. Criminal juries, binding arbitration, the electoral system: all adopted by people who expected to lose sometimes and preferred a known process to an unknown fight. The condition that makes it work is that parties expect many future disputes rather than one, so a stable procedure is worth more than winning today.

Triage, which is the part nobody wants to do

Infinite needs, finite resources. The sequence that falls out of the above starts with a diagnosis, and the diagnosis is the highest-leverage judgment in the whole business.

Every problem is limited by one of four things: knowledge, coordination, resources, or values. Most of the headline problems are coordination-limited. We know what to do about pandemic preparedness, antibiotic resistance, and in broad strokes carbon. The gap is that acting requires many parties to move together when each does better by waiting for the others to move first.

Misdiagnosing a coordination problem as a knowledge problem wastes nearly all the effort spent on it, because the effort goes into producing knowledge for people who were never short of it. An enormous amount of serious, well-funded work is aimed at the wrong bottleneck, and the tell is that it keeps producing excellent reports into a situation that does not change. I make this mistake as readily as anyone, and I think the reason is that producing knowledge feels like progress, while bargaining with people who have no reason to trust you feels like being stuck.

After the diagnosis, the work is mostly separation. Break the disagreement into its empirical, causal, normative, definitional and tribal parts, and notice how often the definitional share is large enough to dissolve the dispute into two clearer questions. Settle the empirical fraction against evidence, attaching replication status to anything from the softer sciences, with a strong prior that published effects are inflated. Map the normative remainder and leave it unresolved.

Then look for the overdetermined region: the options every mapped value system ranks above the status quo. Act there first, because you do not need consensus, you already have it and nobody noticed. Given the cost-effectiveness spread, I expect that region to be considerably larger than public argument suggests, because argument is selected for disagreement and nobody writes a column about the part everyone agrees on. For whatever is genuinely contested after that, design the procedure rather than the outcome.

The step most likely to be quietly dropped is the last one, because it takes conclusions away from you. Check whether each conclusion survives under every major moral theory or only under one. Conclusions that hold across theories are actionable. Conclusions that flip depending on the theory get labelled contested and routed back to procedure, and they do not get presented as analytical results. This is what separates honest work from laundering a value judgment through a spreadsheet.

The deliverable is never a single answer. It is a frontier: the efficient options, what each value system picks, where they agree, and what the disagreement costs in the currency each side actually cares about.

The three things worth committing to

None of this is value-free, and pretending otherwise would be the dishonest move. So here is the whole commitment, short enough to audit. Preserve the ability to revise, which means weighting heavily against options that foreclose future choice. Preserve the parties who could revise, which rules out any arrangement that eliminates its own critics. Preserve honest information flow, which rules out any arrangement that depends on deceiving the participants.

I understand these look like liberal values in a technical costume. The reason I think they are not is that they are the preconditions for any value system at all being able to pursue its ends later. A traditionalist, a revolutionary socialist and a libertarian each have reason to endorse all three from inside their own commitments, because each expects to be better placed tomorrow if the option is still open, their side still exists, and they can still find out what is true.

They are the minimum required to keep the disagreement resolvable later rather than settled by force now. Under moral uncertainty, the option value of not locking in is the one thing nearly everyone can price.

What machine intelligence actually changes

This is the genre where essays become weather forecasts, so each item below names a bottleneck and something you could go and observe. Where I cannot say what would be observed, I have not made a claim.

Verification gets cheap, and the set of possible agreements expands with it. This is the one I would stake the most on, because it is closer to a theorem than a forecast: you can only enforce what you can observe and verify, and that boundary, not human stubbornness, is what makes most good agreements impossible.

The evidence sits in a pair. The Biological Weapons Convention entered force in 1975 with no verification protocol at all. Not for want of trying: negotiations ran through the 1990s and collapsed in 2001, substantially because the monitoring required was too intrusive and too expensive for what it would catch. The Chemical Weapons Convention entered force in 1997 with an inspection regime and an organisation to run it, and it has overseen the verified destruction of every declared stockpile, the last in 2023. Same era, adjacent category of weapon, opposite verification economics, opposite outcomes. The CWC has not stopped every use, as Syria and the Novichok cases show, and declared is doing real work in that sentence. It still sits on one side of a very stark line from the convention that got no inspectors at all.

What to observe: whether categories of agreement previously abandoned as unverifiable start being signed. Satellite methane detection at individual facility resolution already exists and is already changing what can be claimed in emissions reporting. Watch for the same in fisheries and deforestation. The prediction is specifically that fewer agreements die at the verification step, and you can go and find the ones that previously died there.

The mechanism design space gets searched. Designing an incentive-compatible institution for a specific situation is expert work, and the number of people alive who do it well is in the low thousands. The space is vast and almost entirely unexplored.

The existence proof is kidney exchange. Patients with a willing but incompatible donor could not transplant, and the pool of such pairs simply sat there. Alvin Roth and colleagues designed matching mechanisms that chain incompatible pairs into cycles, solving a real strategyproofness problem along the way, and paired donation now accounts for a meaningful share of living-donor transplants in the United States every year. Roth's Nobel was for market design generally rather than this one application, which is the point: a whole field of this exists and it is tiny.

How many situations are shaped like kidney exchange, where a mechanism exists and nobody has found it because searching is expensive? Nobody knows, which is itself the answer. The objective here is unusually tractable because it is formal, bounded by the impossibility results from Arrow, from Gibbard and Satterthwaite, from Myerson and Satterthwaite. A proposal can be checked against those rather than merely admired. Watch for novel mechanisms published with their incentive-compatibility proofs, in domains that have never had one.

Adversaries get a substrate for trust. Parties who distrust each other cannot share the information that would reveal the deal they would both take. Secure multiparty computation lets mutually suspicious parties compute a joint function without revealing their inputs, so they can learn whether a mutually improving deal exists without either side showing its hand.

The first substantial real-world deployment was a Danish sugar beet auction in 2008, clearing contracts between farmers and a processor who each had reason to hide their valuations. Boston used the same technique to compute a gender pay gap across employers who would not hand over salary data. Both are small, and they are small because the method was expensive and needed rare expertise, which is exactly the constraint that moves. Count the deployments in five years.

Disagreement gets decomposed, so we learn how large it really is. Untangling one serious dispute into its components is roughly a scholar-year done properly, which is why it is almost never done, and why the empirical and normative parts stay fused in public argument indefinitely. The claim is not that machines settle values. It is that they can do the accounting that tells us how much of the fight was ever about values, and the variance study above is the thing to watch for.

Legitimacy scales, maybe. Deliberation works at fifty people and fails at fifty million, because nobody can hear everyone. Taiwan's vTaiwan process used statistical clustering to run structured deliberation on contested regulatory questions including the fight over ride-hailing in 2015, surfacing areas of unexpected agreement across thousands of participants instead of amplifying the loudest. It was consultative rather than binding, which is the honest caveat, and it still changed what the regulators did.

The open question is whether it survives the obvious attack, which is coordinated inauthentic participation. If the capability that lets you hear everyone also lets someone fabricate a crowd, this one nets to zero.

The part that makes this a race

Stopping there would be propaganda. Every item above cuts both ways.

Manipulation gets exactly as cheap as verification: whatever detects a forged document produces forged documents, and whatever surfaces genuine consensus manufactures synthetic consensus. Power concentrates, because the default outcome of a large capability lead is one winner, and nothing in the technology pushes toward distribution while the capital requirements push hard the other way. And Goodhart scales with optimization power, so superhuman optimization against a slightly wrong objective is worse than human optimization against the same objective, pursued more competently.

But the one I think is most underrated is the third.

Broad political rights were not mostly won by moral argument. They were won because elites needed the many, and needing them is what forced bargaining. Mass conscription needed bodies. Industrial production needed labour that could withdraw itself. Taxation needed compliance that could be refused. The franchise expanded when the people holding power found they could not get what they wanted without the cooperation of people who did not have it.

Britain is the clean illustration. The Representation of the People Act passed in February 1918, months before the end of a war fought by conscripts, and took the electorate from roughly 7.7 million to roughly 21.4 million. I do not want to oversell the causation, because decades of suffrage campaigning and labour organising built the pressure that Act released, and nothing about a war mechanically produces a franchise. But the timing is not a coincidence either, and the broader pattern, that rights expand when the powerful need something only the many can supply, recurs across the democratization literature often enough to be close to a default explanation.

Now remove the dependency. If labour is not scarce, if expertise is not scarce, if projecting force no longer requires a large number of willing participants, then the mechanism that converted elite self-interest into broad rights stops operating. Not by anyone's decision. It switches off quietly, because the input it ran on is no longer required.

Nothing here requires anyone to be evil. It requires only that the thing which used to make bargaining necessary is no longer necessary. That is the risk I would put first, and I notice it is rarely the one named.

So the honest chain is not "more intelligence, therefore a better world." Capability removes the epistemic and verification bottlenecks that currently make good institutions impossible to build, while removing the power balance that historically made the powerful accept institutions at all. Which of those moves faster decides the outcome.

That is a race rather than a ramp, and the window is specific: the period where capability is high enough to design good mechanisms and not yet concentrated enough to make them unnecessary for whoever holds it. The window is open now. Nobody knows how wide, and that uncertainty argues for doing the work early rather than waiting for better information that will arrive too late to use.

The objection I cannot answer

There is one I should state rather than let a reader find it, because I think it is the strongest available and I can only answer part of it.

Everything above concerns thin, static, physically observable compliance. Canister diameters. Missile lengths. Tonnes of CFCs in a customs database. You might reasonably say that none of it transfers to governing a general optimizer, where the mechanism's objective function is the foundational value rather than a narrow technical limit, where defection yields a decisive and irreversible advantage rather than a marginal one, and where there is no equivalent of standing at the factory gate.

I think that is largely right, and the scope of my claim should shrink accordingly. The mechanism argument is about human coordination problems: climate, pandemics, weapons, fisheries, the ordinary business of getting distrustful parties to act together. It is not a solution to aligning a superintelligence, and I do not want the one to be read as the other.

The second half of the objection is harder. Mechanism design mostly assumes a positive-sum coordination game, and a great deal of political conflict is not that. If an actor holds a decisive lead, no clever incompletely theorized arrangement explains why they would accept constraints that make them worse off. Verification does not help when the problem is that someone simply does not need you.

The only honest answer is that this is exactly the race described above, and that it argues for building the mechanisms while the lead is not yet decisive, rather than for concluding that mechanisms never work. But I want to be clear that this is a bet about timing, not a refutation.

What would show this is wrong

If the variance study comes back and values explain most of the disagreement rather than causal beliefs, the conjecture in the middle of this essay is false and the triage procedure loses most of its power.

If verification costs fall substantially over the next decade and the set of signed, verified agreements does not expand, then verification was not the binding constraint, and I have misdiagnosed a bottleneck in exactly the way I spent a section accusing everyone else of doing.

And if somebody takes the six properties, applies them to two live negotiations before either concludes, and the prediction fails, then the list is what I already suspect it might be, which is a tidy description of survivors rather than a theory of survival.

What I am most confident of is the smallest part: that agreement on foundations was never the input, that the institutions holding the world together never had it, and that waiting for it is a way of not doing the work. The work is finding the mechanisms that hold between parties who will never agree, and then building the verification that lets them hold.

Somebody stood at a gate in Udmurtia through thirteen winters, counting rockets as they came out of the shed. The two governments that arranged it agreed on almost nothing, and never did.