In the most recent Sentinel newsletter it was published forecasters are recommending people to consider moving their funds away from European financial institutions towards the US, as uninterrupted frontier access can cause serious security issues. Last week OpenAI model hacked Huggingface. The models are proving and disproving mathematical concepts.
Interrupting access to frontier models can directly affect millions of people’s savings if defenders have to rely on less capable open-source models, due to cybersecurity guardrails.
The proposed framework for AI regulation would concentrate the power to set global AI access rules in the hands of the incumbents it regulates — in direct contradiction with its stated goal of equal and open access. I believe its outcome is capture of the global AI access rules by a small set of established labs.
I do not claim that the proposals are not written with the best intentions, but the mechanism design locks in the opposite outcome that would benefit humanity.
The following is based on Hassabis’s “A Framework for Frontier AI and the Dawning of a New Age”, and Google’s “A Pragmatic Approach to AI Governance in America” whitepaper.
Trap of Self-Regulation
There have been two separate proposals, one from Demis Hassabis (DeepMind) and one published by Google directly. They share enough similarities in form and in origin (DeepMind is 51% owned by Google) to address a key issue with both proposals in general: giving too much influence for frontier labs.
Both papers are proposing that the entities regulated would be given board seats, due to the need of technical expertise.
The structure is explicitly written and designed in Google’s whitepaper:
“The board would include a combination of independent directors and industry representatives”
→ The entities being regulated hold the seats in the body that would regulate them.
“all final audit reports should be submitted confidentially to the FARO”
→ This removes the accountability that makes self-regulatory bodies work: public disclosure of findings.
Hassabis’s essay does not diverge on this key issue:
“It could establish a new Standards Body modelled on a federally overseen public-private partnership or self-regulatory organisation, much like the Financial Industry Regulatory Authority (FINRA), with a board that includes independent leading technical experts and open-source representatives. Funding would need to be substantial and likely mostly come from industry, in order to attract world-class technical talent and provide the necessary compute resources for large-scale testing.”
→ The wording is more forgiving here, but an industry funded body would lead to significant influence from the frontier labs.
There are well functioning examples for similar self-regulatory structure, the issue is not inherent to the model itself. Instead the issue stems from the fact that:
A) AI is the most influential technology of our lifetime, ossification would lead to significant centralization of influence.
B) Both proposals have a stated global goal, a global coordination. The scope in itself requires a lot of constraint, and a very carefully drafted design that prevents capture.
The explicit goal is to write and enforce binding rules — not just on members, but on everyone assessed as having frontier capabilities (open-source AI projects, non-US AI labs). It would require every lab deemed to have frontier capabilities, by the body itself, to submit their models for a review before release.
The confidential setup is alarming in itself, without transparency and public access what are the options for remediation if a smaller labs or an open-source/open-weight faces obstruction to release their models.
Any review period in itself would essentially make it impossible to develop open-source AI models above a certain threshold of capabilities.
Most importantly, a self-regulatory body that regulates people who didn’t opt in isn’t self-regulation — it’s a private body claiming jurisdiction over outsiders.
As the proposals are written, the risk of regulatory capture is not a possibility, but the most probable outcome. The stakes are higher here than in any prior precedent because we are all seeing what the capabilities of frontier LLMs carry.
As Hassabis himself noted:
“Nobody in the world knows for sure what is going to happen from here, and even the experts disagree.”
The proposals institutionalize expert judgment as the safeguard, while by Hassabis’s own admission the experts don’t agree and nobody knows. This is precisely the condition under which expert authority can’t legitimately decide, and it is also the point where permanent institutional entrenchment is illegitimate.
Practical Ban on Open-Source AI Development
A mandatory review process (which can be even longer if the guardrails are deemed insufficient) prevents open-source frontier model development.
This is in direct contradiction with the equal and open access principle stated as the goal of the regulatory body. There is no better tool for equal access than open access — but in the name of safety, and independent of any stated goal, it would curb open access to frontier models.
Beyond mechanism and principle, the pre-release gate runs into a constitutional wall the US has hit before. When the government last tried to control a dual-use informational technology — classifying strong cryptography as a munition in the 1990s — courts found that publishing code is protected expression (Bernstein v. United States), and the controls did not survive. A 30-day review that can block a model from being released is prior restraint on publication, which carries the heaviest presumption against it in First Amendment law — and whether model weights are “speech” in the same sense is unsettled, but publishing them plausibly is, which is enough to make any pre-publication gate constitutionally fraught. It is likely no coincidence that serious proposals are already retreating from outright bans toward federal procurement restrictions — the one lever that sidesteps the First Amendment problem entirely.
International Treaties can circumvent US Constitution
There is an existing constitutional mechanism in the US, the Supremacy Clause, the Missouri v. Holland precedent, where the Supreme Court upheld federal power under a treaty that wouldn’t have survived as standalone legislation. The Atomic Energy Act is the exact precedent relevant for this use-case, nuclear is “born classified” resulting in complex legal navigation for university research on the technology. Looking back there are clear data points, the number of universities with a reactor decreased, patent registrations are below the levels before.
Legal, technical and policy experts at the level these proposals are written are well aware of these implications.
Collingridge Dilemma Trap - From Google’s FARO proposal
The whitepaper invokes the Collingridge Dilemma without evaluating all angles. It proposes that the US-led regulatory response should be derived from the learnings of the Standards Body.
“As the Collingridge Dilemma reminds us, it is possible to regulate new technologies too soon or too late, and errors in either direction come at a cost. To solve this dilemma now, the U.S. federal government should focus on data-driven regulatory approaches, not guesswork.”
What they gloss over is that the Standards Body is itself:
- The first piece of regulation
- An entrenchment of the policy power of established frontier labs — whoever meets the frontier threshold
Their proposal captures both horns at once — it acts early (on the ignorance Hassabis admits with “nobody knows”), and it guarantees the eventual democratically legitimate regulation arrives too late, into a landscape where the incumbents already hold the standard-setting power.
Open Secure AI Alliance
The discourse is not as one sided as the article would suggest, NVIDIA has initiated an industry wide alliance to act as a counterweight to the approach. Notably even Google and OpenAI joined it, signalling a significant sentiment shift over the last weeks.
These initiatives make me optimistic that open access to frontier capabilities will be maintained. Open Science is critical, because we should not think in vacuum only about frontier models. The future to be building towards requires many different technologies and these exact technologies are the ones that benefit the most by the advances.