The Pentagon’s fight with Anthropic was the first real test for how we will control powerful AI. The bad news: we all failed

Hello and welcome to Eye on AI. In this edition…The Pentagon fight with Anthropic raises three crucial questions…OpenAI raises $110 billion in new funding…Meta experiments with an AI shopping assistant…LLMs can identify pseudonymous internet users at scale…data centers on the front lines in the Iran war.

The most important story in AI at the moment, without a doubt, is the fight between the U.S. Department of War and Anthropic. If you haven’t been following the drama, you can catch up on the story by reading coverage from me and my Fortune colleagues here, here, here, here, here and here.

This story raises at least three critical questions: who should have control over how AI is used in a democratic society? How should that control be exercised? What should the consequences be for a company that disagrees with the government’s policy?

Whatever you think of OpenAI CEO Sam Altman and his decision to sweep in and sign a deal with the Pentagon—including a contractual obligation to allow the military to use OpenAI’s AI models “for any lawful purpose” that Anthropic had refused to agree to—Altman correctly identified what’s at stake in this fight.

In an “Ask Me Anything” session on X over the weekend, Altman said:

A really important point: we are not elected. We have a democratic process where we do elect our leaders. We have expertise with the technology and understand its limitations, but I think you should be terrified of a private company deciding on what is and isn’t ethical in the most important areas. Seems fine for us to decide how ChatGPT should respond to a controversial question. But I really don’t want us to decide what to do if a nuke is coming towards the US.

This was the crux of the Pentagon’s stated objection to Anthropic’s existing contract. The military did not think it was right to have a private company dictating policies to an elected government.

AI moves lightning fast, Congress at a snail’s pace

Most Americans might agree with the Pentagon’s position—in principle. Except it is complicated, in practice, by three things. First, AI technology is moving extremely fast, but the mechanisms of democratic control—legislation, Congressional oversight, elections—move extremely slowly. In the three years since ChatGPT debuted, Congress has not passed any federal AI legislation. The Trump Administration has dismantled limited AI regulations put in place by its predecessor, while also acting to punish states that pass their own AI regulations.

So while many people might agree that policies on the government’s AI use ought to be set by elected officials, there is the practical issue of what to do when those elected representatives fail to act. The idea of trying to arrive at AI policy through contractual negotiations between labs and government is a poor substitute for true democratic governance, but it might be better than no governance at all. The controversy over Anthropic’s Pentagon contract should be a wake up call for Congress to act.

Second, the trend among governments over the past several decades has been to interpret existing laws broadly in order to expand the power of the government to use technology to surveil its citizens. (The story has been one of the executive branch gradually clawing back surveillance powers it lost through Congressional action following the scandals that emerged with Watergate and the Church Committee hearings in the mid-1970s.) Many activities of the military are also cloaked in secrecy that makes democratic oversight and accountability difficult. This constant pushing at the boundaries of what the law will allow has made the public distrustful of the government’s intentions. So it’s not surprising that some people at this point may actually have more faith in a seemingly well-intentioned and brilliant, but unelected, technology executive, such as Anthropic’s Dario Amodei, to do the right thing and set the right policies.

Finally, there is the issue that many Americans have with this specific government. The Trump administration has repeatedly taken unprecedented actions to punish domestic dissent, often on flimsy legal justifications, or with no legal justification at all, and has repeatedly deployed the military domestically to intimidate or punish perceived domestic opposition. It has also launched several military actions overseas with little to no legal justification. So is it any wonder that many question whether this particular administration should be given the power to use AI for anything its own lawyers believe is legal? 

Is the nationalization of AI inevitable?

Even if you think the Pentagon is correct that democratic governments, not private companies, should decide on how AI is used, the next question becomes how that control should be exercised? Altman put his finger on the ultimate question hanging over the industry: if frontier AI is a strategic technology, why doesn’t the government simply nationalize it? After all, many other breakthroughs with big strategic implications—from the Manhattan Project to the space race to early efforts to develop AI—were government-funded and largely government-directed. As Altman said, “it has seemed to me for a long time it might be better if building AGI were a government project,” though he added it “doesn’t seem super likely on current trajectory.”

The Pentagon’s current approach comes close to nationalization by other means. One option the DoW threatened was using the Defense Production Act, a Cold War-era law, to compel Anthropic to deliver an AI model on its preferred terms—a sort of soft nationalization of Anthropic’s production pipeline. And the retaliatory decision to label Anthropic a “supply chain risk” is designed in part to intimidate other AI companies into accepting the Pentagon’s preferred contract terms, which again seems nationalization-adjacent.

What should the cost of dissent be in a democracy?

Finally, this brings us to the question of what an appropriate punishment should be for an AI company that refuses to agree to the government’s preferred contract terms. As Dean Ball, an AI policy expert who worked briefly for the Trump administration on its AI Action Plan, has said, the government seems within its rights to cancel its $200 million contract with Anthropic. 

But the decision to go much further and label Anthropic a “supply chain risk” strikes at the heart of private property rights and free speech in a liberal democracy. The designation—which was intended to be used against technologies that could help a foreign adversary sabotage critical defense systems—had never before been applied to a U.S. company and never before been used to punish a company for not agreeing to contract terms that U.S. military desired. The decision, Ball has said, amounts to “attempted corporate murder,” since under the SCR designation any company doing business with the Pentagon would be barred from any commercial relationship with Anthropic. If that interpretation stands—and many legal scholars have said it will not—it could be a mortal blow to Anthropic, which depends on selling to large Fortune 500 companies that also do work for the Pentagon for revenue, cloud computing infrastructure, and venture capital backing. Should the punishment for disagreeing with the government be the death of your business? That certainly seems un-American.

Altman has claimed he struck his deal with the Pentagon in part to de-escalate the tension between the government and AI companies, saying that “a close partnership between governments and the companies building this technology is super important.” While I’m unsure of Altman’s true motives, I agree with him on this last point. At a time when AI potentially threatens unprecedented changes to the economy and society, fomenting distrust and conflict between the government and the people building advanced AI systems seems like a pretty bad idea.

With that, here’s more AI news.

Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn

This story was originally featured on Fortune.com