Senior Policy Director @Law_AI_ working on questions about U.S. law + AI governance
Washington, D.C.
Joined November 2024
- Tweets1K
- Following700
- Followers4K
- Likes42.6K
It’s important to understand that when Anthropic says “regulation” they do not mean that word in the traditional DC sense of like “the government tells companies to do something and the companies have to do it.”
I got a whiff of this when they semi-opposed an early version of the RAISE act, the most unobjectionable milquetoast “regulation” imaginable. After that they seemed to get better and supported sb 53 etc, so I started giving them the benefit of the doubt again, but in retrospect I think that was naive. If RAISE was too heavy-handed for them, there’s just no world in which they’re going to play nice and support the kind of regulation that might actually have a chance of enabling transformative AI to be built in any kind of democratically or morally legitimate way. They clearly just want to roll the dice.
If Neil Chilson's down for an RSI ban then it seems like it ought to be doable politically.
I think a lot of people basically believe what Neil seems to believe, i.e., that no big company would have the insane reckless hubris to race to RSI as defined by Neil. I think that at least two big companies currently seem to be doing exactly that, but you don't have to agree to support or at least be indifferent to an RSI ban. You just have to agree that *if* they were doing that it would be bad.
In and of itself it wouldn't be worth trading broad preemption for, but the idea does seem robustly good. Obviously defining RSI in a way that people can agree on is a bit tricky, but doesn't sound impossible by any means.
This is obviously true, but “what is the best response that we can actually pass” is at least as important a question as “what is the theoretically optimal legislative response assuming no political constraints.”
I think that introducing a *bipartisan* bill as strong as FRONTIER back in July was an incredible achievement that Reps. Obernolte and Trahan deserve a lot of credit for. Things have changed since then, obviously, which is why FRONTIER now has six cosponsors from each party.
I think someone should write and introduce the best possible federal pacing bill, in part because it might actually become possible to pass such a bill at some point in the next couple years if we get a lot of new information that makes that proposal more politically compelling. But we also have to write the best oversight + governance bills that we can, because in a lot of likely-seeming political paradigms those are going to be the bills that can actually pick up four republican senators and open up the possibility of a deal being made with the President.
If you’re a really cracked lawyer who is or could be good at writing legislation, I think you should consider the question of whether you have a moral obligation to work on this stuff. A lot of new and incumbent members of congress are going to be hiring staffers to work on AI legislation very soon. A lot of states are currently hiring the people who will actually be tasked with implementing and enforcing the first serious laws governing frontier AI development. Lots of work to be done.
Very important job that CA lawyers should consider applying for!
The California Attorney General is hiring for attorneys in the consumer protection division, which does enforcement of a lot of AI laws including SB 53. Great opportunity for public-minded lawyers!
calcareers.ca.gov/CalHrPubli…
Apropos of nothing, I want to post my spiel about if-then commitments.
IIRC, that term originally comes from one of those very important 30-author Bengio-Hinton “AI risks are a big deal” papers they used to publish in 2023, but I’m mostly familiar with it because of some articles written by Holden Karnofsky, deservedly renowned poster/thinker/EA/Anthropic guy.
Like a lot of great ideas, it’s pretty simple. An AI company says “If one of our models has capability X, we will undertake mitigation Y, and if we can’t do mitigation Y we will pause development/deployment until we can.”
I think this is a good and important idea. But, very obviously, if-then commitments only work if there’s some sort of external leviathan that can enforce the commitment. Otherwise it doesn’t work, because the company is incentivized to commit to whatever results in good PR and warm fuzzy feelings at time A and then defect and refuse to honor the commitment at time B. *Making* the commitment is easy and profitable; *honoring* the commitment is difficult and costly. Words are just words, and two trillion dollars is an amount of money that has a mind of its own and makes its own decisions. I’m no Holden Karnofsky, but if someone offered me a billion dollars to stop believing something there’s a part of me that thinks I would probably fold immediately.
Karnofsky knew this, of course. His Carnegie piece about if-then commitments has a whole discussion of how, ultimately, these voluntary commitments are only going to work if they get turned into legally binding government regulations. Otherwise it’s the frog and toad meme, right? You can put the cookies in the box, but that doesn’t stop you from having more cookies because future you can always just open the box. Some external force has to bind the company to the mast; otherwise, the incentives will do what incentives do.
But when SB 53 said “okay, we the state of California are an external leviathan and we will now enforce any commitments you make in your Responsible Scaling Policy,” Anthropic’s response was to create a new RSP that didn’t commit to anything. So now they have one RSP that they are legally obligated to comply with that commits to nothing, and another RSP that they pinky promise to comply with that commits to lots of fun safety stuff. Their revealed preference is to do whatever they want to do at any given time.
And when Anthropic introduced a bill, everything in the bill was totally voluntary. They have to publish and comply with a safety framework; the safety framework will be written by their compliance lawyers, in such a way as to minimize liability. They have to hire an independent auditor to verify that their models aren’t about to kill lots of people; they can pick whoever they want, and fire them at will, and the government has no authority to license or oversee auditors, and the auditor’s findings have essentially no consequences unless the government can prove in court that a catastrophe is imminent.
This doesn’t look like a company that wants to bind itself to the mast.
That said, I firmly believe that this happened because someone goofed up, and that Anthropic will shortly send many impossible-to-fake costly signals of their good faith by supporting strong state and federal safety legislation.
I agree with a lot of the points raised in this post. Not enough people take the point about what a unilateral U.S. pause would do to china negotiation dynamics seriously; I’m glad that Alexander does.
And I also appreciate how bizarrely honest and transparent Scott is about what he really thinks, even in situations where most people would hide their power level for pragmatic reasons.
But without meaning to disincentivize that kind of radical honesty, I want to point out that what he’s saying at the end of Section IV is, essentially: “within the next few years, it is likely that some group of private citizens residing in the San Francisco Bay Area is going to have absolute power to decide whether you and your family and everyone else in the world lives or dies, and how, and under what conditions, forever, and it would be better for those people to be the people in charge of Anthropic rather than the people in charge of some other, less trustworthy company.” He talks about Anthropic “deciding” whether to hand power over to the U.S. government, or to China, or to themselves.
I sincerely hope, and believe, that Alexander is wilding out there and that no employees of any frontier developers are working towards that kind of outcome. I also want to note that suggestions like that are unlikely to be persuasive if your goal is to convince a general audience that they should put more effort into “trust the labs to do the right thing” flavored efforts and less effort into political activism.
As Alexander seems to realize, a lot of people who do not live in the Bay Area are going to start to feel very strongly about this issue in the near future, as this issue becomes more politically salient and stops being a polite conversation between Lesswrong posters and becomes a messy mass political debate. I think that this is overall a good thing, even though I also basically agree with Alexander’s anti-SURA thesis.
I don’t think that most employees of leading AI companies consciously intend to run out the clock on effective, democratically accountable oversight as they race to create the technology that they believe will give them absolute power over the rest of humanity. But I do wish they understood that it does sometimes look like that’s what they’re trying to do, and that it’s in their interest to provide strong evidence against that explanation for their behavior. The recent bill proposed by Anthropic was not helpful on this front.
Relatedly: if you have been employed by a leading AI company for a while, you should realize that the many millions of dollars that you have invested in the runaway commercial success of your company do in fact influence your views on policy, to a very great extent. You should also realize that everyone who doesn’t know you personally will reasonably assume that your views on policy are almost entirely dictated by a desire to maximize the return on that investment. That’s how it has worked in every other industry in history; if you want to show that it works differently for you, you will have to send costly signals to prove it.
If you read any blog post on AI safety from this year, make it be this one
lesswrong.com/posts/xikGZ8fu…
Way back in the day (i.e. like late 2025-early 2026) I used to do a bit whenever I was on a panel with @neil_chilson or @KevinTFrazier or whoever where if they brought up the "patchwork of state laws" thing I would point out that the few existing state AI safety laws were very light touch and harmonized, and I would accuse the interlocutor of being a "regulatory doomer" because they were concerned about speculative future harms from state AI laws. Great bit, always killed. People love the "regulatory doomer" line.
I don't think I can really do it any more, though, because it kind of seems like states really are about to pass a bunch of very different laws, some of which do in fact impose obligations on frontier AI developers.
I'm not saying that's a bad thing, to be clear; I don't think the current level of regulation (essentially zero) is anywhere near defensible, and since Congress isn't ready to get serious about federal legislation states should and will get to work and start trying out different things. But it is funny to me that all of the doomers are being proven right at the same time.
Nathan looking through 14,000,605 possible futures and realizing that the only way we make it is if he uses the phrase "by its own lights" three times in a kind of weird but not quite incorrect way in the first few hundred words of his response to Dario Amodei
Dario — Thank you for taking the time to respond to my previous post. I think it's great when this kind of engagement happens in public rather than on company Slacks, and I hope to see more of it.
I agree that Anthropic isn't best modeled as a profit-maximizer. Your post gave several compelling examples of Anthropic taking competitive hits for safety, and insofar as my previous post suggested otherwise, that's on me. But I don't think I made my main point as clearly as I could have, so let me try again. It comes down to four key claims:
1. Anthropic seems to be racing at breakneck speed to build recursively self-improving superintelligence (RSI). By its own lights, that could kill billions of people or literally destroy the world.
2. Risking destroying the world is very hard to justify. It is only conceivably acceptable if serious attempts to put RSI under robust societal governance are failing.
3. Far from meeting that burden, Anthropic, when push comes to shove, often works to keep governments from tying its hands. It seems to care much more about preventing government overreach than about reducing the risk that it and other AI companies will destroy the world.
4. Even if the AEGIS Act really was an honest mix-up, as you suggest, that is itself a serious problem. It suggests Anthropic isn't treating the governance of this technology with anything close to the gravity it deserves.
In greater detail for those who want it:
1. Anthropic is racing to RSI, and by its own lights that is insanely risky.
Launching RSI could literally destroy the world. That's according to you, your co-founders, and most of your safety staff and capabilities researchers, not to mention many AI researchers outside your walls. You are intentionally trying to create generally intelligent systems more capable than all humans that you claim in six to twelve months “could be capable” of taking over the entire internet before we have the ability to reliably monitor, control, or align them.
“Succeeding” at RSI could kill or permanently disempower billions of people in the near future, including me and everyone I love. Yet Anthropic’s primary focus seems to be marshalling an immense amount of resources and talent in a race to get there first.
2. Risking destroying the world carries an enormous burden of justification.
I believe the vast majority of people do not want Anthropic to do this (to the extent they know what it's doing). Imposing even a modest chance of catastrophe on billions of people who never consented requires an extraordinary justification. The central one Anthropic has offered: someone will build RSI regardless, so it's better for a safety-focused lab to be at the front of the pack, using that position to support strong AI governance. (Other reasons Anthropic has given, like modestly accelerating improvements in healthcare or economic growth, don’t seem remotely proportionate to the risks Anthropic is imposing by its own lights.)
But that argument only works as a last resort. Racing to RSI is only conceivably okay if attempts to institute robust societal governance to manage that race are failing despite your best efforts. A lab that relies on this justification should be a strong, consistent supporter of efforts to enact governance that could bind it, including governance that could slow it and significantly constrain what it does.
3. Rather than supporting efforts at strong regulation, Anthropic has intervened to keep governments from tying its hands.
Anthropic has sometimes helped: it played a very helpful role in passing transparency bills in Illinois and ongoing efforts in Massachusetts, and I'm sincerely appreciative. But transparency is far from enough for a technology that will be the world's most powerful superweapon at best, and an uncontrollable swarm of lethally superintelligent agents at worst. When it matters most, Anthropic has often pushed for weak regulation.
The AEGIS Act is the most obvious example. No highly risky industry is regulated as anemically as AEGIS would regulate frontier AI: not nuclear, not banking, not aviation, not food safety, not the manufacture of explosives. In all of those industries, specialist regulators can address unsafe practices well before they cause harm. AEGIS would bar the federal government from intervening to block unsafe development or deployment of models until there is an "imminent risk" of catastrophic harm, and would require it to meet a demanding burden of proof in court. Worse, AEGIS would also prevent states from picking up the slack. This design makes perfect sense if you're a company trying to minimize the risk that the government can overly constrain you. It makes little sense if you want American democracy to govern frontier AI effectively.
This isn't the first time. The initial version of California's SB 1047 would have let the Attorney General sue to stop developers like Anthropic from imposing unreasonable risks of catastrophic harm. It would also have created a regulatory agency to help set safety standards. Anthropic objected, and helped get both provisions removed before ultimately offering some support for the bill (which I also genuinely appreciate). I’m not saying that Anthropic should have supported 1047 in its initial form; that’s a complex issue and I realize there were questions about whether earlier versions of the bill were less politically feasible. Still, in Anthropic's own words, it helped weaken the bill because "[t]here is a substantial risk that the bill and state agencies will simply be wrong about what is actually effective in preventing catastrophic risk, leading to ineffective and/or burdensome compliance requirements."
The same goes for New York's RAISE Act, whose initial version also would have let the Attorney General sue to stop unreasonable risks of catastrophic harm. Anthropic credits itself as a supporter of RAISE, but it did not support the bill while it was in the legislature, including when it nearly died. After the bill passed the legislature, Anthropic publicly critiqued it, saying "Any state proposals should be narrowly focused on transparency and not overly prescriptive."
Why does Anthropic consistently take the risk of clumsy government action far more seriously than its own repeated warnings that companies like Anthropic will literally destroy the world by racing to RSI?
4. Even if AEGIS was an honest mix-up, that is itself a serious problem.
Your response frames AEGIS as an early draft and a miscommunication. Perhaps you would have strengthened it and narrowed its preemption. (That wouldn't explain Anthropic's interventions to weaken SB 1047 or RAISE, but leave that aside.)
Suppose that's exactly right. A federal framework for governing RSI is one of the highest-stakes documents your company will touch. Your post admits Anthropic "could have handled [these matters] better," but its tone ("This kind of thing happens a lot in large organizations") doesn't reflect the gravity of this issue. Anthropic talks about the need for “operational excellence” in critical areas, and I have a hard time believing that a similarly cavalier attitude would be tolerated while Anthropic was negotiating a key compute deal or preparing its next large pre-training run.
In short
The justification for racing rests on Anthropic fighting for strong governance, and Anthropic's record on regulation points the other way. Together with its breakneck racing, that fits a much darker picture of the company than the one that seems to prevail among your employees. I hope that Anthropic reflects on this picture, supports stronger regulation, exercises extraordinary caution when advocating for weak regulation, and doubles down on its founding purpose of helping the world safely navigate transformative AI together, instead of prioritizing Anthropic’s ability to win a race that imposes intolerable risks on everyone alive.
I think it’s a good idea to take concentration of power risks quite seriously. I am actually more concerned, overall, about concentration of power than I am about loss of control (though tbc I think both are very real and important concerns).
But I still think this kind of defense of Anthropic’s bill misses the mark. This person is citing the Lieu and Moran “Kill Switch” bill as an example of a law that doesn’t take concentration of power seriously enough. That bill only gives the executive the authority to shut down deployment *after* the model has caused a catastrophe (IIRC they use a similar 50 deaths or serious injuries / billion in damages threshold).
It’s true that emergency shutdown provisions have to strike a balance between granting necessary emergency response authorities and providing checks against abuse. But come on. If a model just killed (at least!) fifty people in a single incident, I am not that concerned that making the lab shut down the deployment of that model is going to be an abuse of power. If your model kills fifty-plus Americans, the courts are absolutely going to make you stop deployment of that model, regardless of what any Kill Switch bill says, and that is the very very least of your worries.
The better comparison is to the shutdown authority in FRONTIER, which Anthropic’s shutdown authority is seemingly based on. FRONTIER allows the executive to issue an order stopping deployment or placing conditions on deployment, upon a finding of imminent catastrophic risk, and the company has to sue to lift said order.
FRONTIER layers on a lot of procedural safeguards to try to stop that shutdown authority from being abused. Reasonable people can disagree about whether it gets the balancing act right; personally, I’m quite interested in trying to find win-win approaches that reduce the risk of abuse without symmetrically reducing the provision’s effectiveness in an emergency.
I do think that FRONTIER pretty clearly strikes a better balance than Anthropic’s bill does. If the government can prove in court, by a preponderance of evidence, that a disaster killing 50+ people specifically from bio, cyber, or loss of control is *imminent,* I think a court is probably going to issue an injunction shutting down deployment even without any new law. No one wants a catastrophic bio-attack to happen; courts are not insane. So it’s not clear to me how much marginal value Anthropic’s shutdown authority would actually add.
Keep in mind the fact that the status quo is a lawless, discretionary shutdown authority based on a wild exec branch interpretation of export control authorities. Replacing that with legislation that affords labs due process seems like an affirmative improvement w/r/t concentration of power risks.
A lot of what Amodei is saying here is true. That's why I and many other people who are concerned about AI safety have historically given Anthropic the benefit of the doubt; they really have sent some costly signals indicating that at least some of their decision makers care about mitigating catastrophic risks.
But the object-level issue here is "is Anthropic's bill good, or even acceptable?" Amodei isn't really engaging with that question.
That's fine; he's a very busy and important guy, and even among DC AI policy professionals only a small handful of people have actually read and understood Anthropic's bill. It's good that he cares enough to make the general case for "Anthropic cares about AI safety" in response to a Nathan Calvin tweet.
But someone who works at Anthropic does understand what their bill does, and (hopefully) why. What their bill does is:
(a) it tries to ensure that the federal government cannot exercise meaningful oversight over Anthropic as Anthropic races towards superintelligence,
and
(b) it makes it much harder for states to pass laws that might meaningfully constrain Anthropic's ability to race towards superintelligence.
It is possible, maybe even likely, that Amodei is not aware that this is what the bill does. In the past, he has taken costly actions that significantly advanced the state of the art in AI safety legislation; for example, it was very helpful when he publicly committed to taking on embedded auditors with employee-like access to information.
But the buck has to stop somewhere. I do not think that any honest expert could maintain with a straight face that Anthropic's auditing provisions (labs choosing their own evaluators, with meaningless government "rating" of evaluators but no actual licensing; no meaningful government rulemaking or standard-setting authority that might allow democratically accountable bodies to control what audits involve or who conducts them) or the emergency shutdown provisions (the government can stop deployment only if they can prove in court that there is an *imminent* risk of 50+ deaths or more than a billion dollars in damages from specifically bio, cyber, or loss of control risks) are actually intended to provide meaningful democratic oversight of the creation of a technology that Dario Amodei believes poses a substantial risk of killing you and your family.
Anthropic can't claim that they merely distrust the federal government, or the current administration. The same bill that tries to stop the federal government from exercising any kind of meaningful oversight also tries to prevent states from doing the same. To be clear, I actually think that the preemption section is fairly well-designed, in isolation; if it were paired with a strong federal framework, I would probably take the trade that it proposes. But a bill that proposes meaningful (albeit narrow) preemption of state safety laws in exchange for a totally toothless federal framework does not advance the cause of AI safety.
I don't know how Anthropic works, or who made the decisions that caused the bill to turn out the way it did. I don't know whether this was what the people who designed their June framework intended. But I do think it's important to note that the actual, practical function of the draft legislation that Anthropic sent to Congressional offices would be to ensure that no government could meaningfully constrain or monitor Anthropic's efforts to build superintelligent AI systems.
Thank you for this, @_NathanCalvin. I appreciate and am a fan of the investigative work you’ve done on behalf of AI safety and believe your comments are coming from a constructive, honest place, and that it benefits the public to have people whose job is to scrutinize the actions of frontier AI companies. I think something would be wrong if companies of the size of Anthropic didn’t get harshly criticized basically every day; strong antibodies against corporate power are a healthy feature of a free society, and especially so when issues of the gravity of the one we’re facing are at play. I also do agree that some of the examples you cite were mistakes by Anthropic.
That said, I disagree with the thrust of your post. At the core, a good fraction of the things you criticize are consequences of Anthropic’s basic theory of change that we have held for many years. Our strategy has always been what I’ve called the “race to the top” — by showing how it’s possible to build the technology responsibly, we can raise the bar for the entire industry. The forces that drive the race to the top are hugely more effective if the company practicing them is at the frontier; that’s one lesson I’ve learned repeatedly. And looking at that strategy as a whole, I’m proud of our track record and stand by it.
I believe that if we take a list of things that Anthropic did in the past year that were pro-safety and commercially costly to us, they far outweigh the list you’ve made, while at the same time I also disagree with some of the items on your list. Here are some things Anthropic has done, in the same period you’ve cited (the last year), which were very costly to us commercially while having strong safety benefits:
1. We played a key role in SB53 (and other state laws), at a time when doing so was politically perilous to the company and AI regulation was very unpopular. We in fact argued for a stronger version of SB53 than was ultimately passed; there were a lot of stakeholders and we were not the limiting factor in the strength of the bill that was passed.
2. We have very strong biological safeguards on our models, because we’ve seen concerning results in bio uplift trials and even seen and blocked potential attempts to misuse our models for bioweapons (as seen in our threat intelligence report from a few weeks ago). This is a very serious threat that we want to have zero tolerance for. But having very low false negatives means having significant false positives (despite months of effort on our part to improve the classifiers), enough so that biologists joke about how restrictive our classifiers are, and competitors sometimes sell against us by touting their more permissive bio capabilities. This has been very painful to us in an area where we care a lot about supporting an ecosystem of Claude users (biomedical science; and it is particularly frustrating to me as a biologist), but it’s the right thing to do and we intend to keep doing it.
3. Similarly, we first had Mythos-class models in February but didn’t make them generally available until June (when we had stronger cyber safeguards in place). We only offered them to a limited set of partners in order to focus on cyberdefense. It would have been much more economically rational to simply release the model to everyone. We likely sacrificed a significant chunk of a several-month model lead. But because of this, many companies secured their cyber infrastructure before the deluge of widely available cyber-capable models. Of course there are ways it could have gone better or been run better, but this was what we were aiming to do.
4. I advocated forcefully and publicly, in my essay Policy on the AI Exponential in June of this year, for the government to have the power to block model releases. Back then this was still outside or maybe on the edge of the Overton window, though I understand the window has moved since then. We were attacked intensely for this.
5. I don’t want to re-ignite conflict, but the events of February 2026 are a strong example. It is clear that our public stance could have seriously damaged the company’s future.
6. We’ve continued to advocate for export controls on chips and chipmaking equipment to China, because we think it allows democracies to negotiate from a position of strength on AI safety agreements. We’ve done this despite problems it’s created for us with industry partners who hold very different views and on whom we are very dependent.
7. As far as I’m aware we were the first, and still only, frontier company to restrict the sale of AI services to PRC-controlled companies, including those operating through third countries (otherwise a major loophole). We’ve therefore forfeited significant short-term business to prevent fueling AI platforms and applications that would help the CCP’s military and intelligence services. We are also cracking down on distillation; generally, distillation is not a major threat to us commercially, but like chip exports it has concerning national security and safety implications, so we expend significant energy and company resources to address it.
8. As you mentioned, we’ve put out a lot of reports, announcements, long model cards, public advocacy, etc, all trying to inform the public about the risks of AI. These involve a level of detail and effort that would be implausible for a purely profit-maximizing company.
9. Just this week, we announced we were putting 5% (and possibly more over time) of the company into a public benefit fund.
All of these decisions came with very substantial costs to us. To be fair, those that involved holding back or restricting access to frontier capabilities also had a broader cost — for example, the public might otherwise have benefitted from faster progress in biology, so I am not saying that our decisions were without tradeoffs — but they were not driven by profit-seeking.
For each decision, you can come up with ulterior motives that could have driven us — I think it is a general truth that any decision, on any topic, can always be portrayed as a nefarious scheme. But most of those end up being pretty farfetched or don’t fully make sense if you actually examine them closely. For example, it is deeply implausible that it would be a shrewd business move to block a large swath of research biology in order to do the work ourselves and compete with our customers; we expect the benefit to the ecosystem to greatly exceed the work Anthropic does itself, and for the two to complement each other. It is not in Anthropic’s interest to be perceived as unfriendly to biologists. Our Life Sciences Verification Program, which gives vetted biology professionals access to our models, also refutes this idea.
Of course, a different set of people will see this and make a different criticism: that Anthropic shouldn't be making decisions about who gets access to what in the first place. I agree, in the sense that ideally the government would be making these decisions, and that’s why we advocate for regulation. But for now we are forced to make them, and enabling bioterrorism isn’t a better option.
I think that even if I conceded every item in your list, I would happily hold up mine next to it; I don’t think it’s a close call. That is of course partly subjective and different things are important to different people. Maybe one way you could object is that several of the items above relate to AI misuse and not loss-of-control; if you think the latter is much more important than the former, that might change your calculus. But this is not Anthropic’s view — we view both as serious issues. You can also disagree with our decisions, as many have, but they are consistent.
But I also disagree with at least some of your list. Several items don’t represent a deviation from Anthropic’s strategy or values at all, once we postulate the “race to the top” strategy that requires being a frontier AI company. Your items 1-4 are all simply examples of what it means to compete at the frontier, and we stand by them, both in the sense that they are transparently ordinary things to do as part of that strategy, and that we stand by the strategy as a whole.
To be fair, there are important questions worth confronting (and that we are confronting) about what to do when the AI-driven acceleration gets fast enough that it is hard for safety to keep up. We are nearing that point and that’s what’s driving some of our thinking about “pacing.” But 1-4 are actions taken before that point and should be perfectly expected based on Anthropic’s strategy as a company. Of course it is possible to disagree with that strategy, but that’s a more fundamental disagreement that you could have had any time over the last few years.
I agree we could have handled your items 7, 8, and 9 better. All fall into the category of “rhetoric or policy communication that wasn’t carefully written or resulting from fast-changing circumstances not propagating from one team to another fast enough.” This kind of thing happens a lot in large organizations (Anthropic is over 5,000 people), and doesn’t even necessarily represent a failure or mistake of any single person. I am of course the CEO, and I take responsibility. The volume of communication from Anthropic is enormous and if these are the worst examples, I think we’re frankly doing pretty well, though I will continue to work hard to make sure we do even better.
Your 5th item (the change to our RSP) is the most substantive item, and people can debate this, but I think that the old, more rigid RSP created a lot of situations where we were technically required to do something that required a lot of effort but in retrospect wasn’t very important for safety, as well as not covering some of the most important things. Our current RSP’s focus on transparency brought things like risk reports and roadmaps, and its emphasis on independent external review laid the groundwork for even more intensive embedded evaluation, all of which I think have been valuable in driving a race to the top. In the original announcement in September 2023, we described the RSP as an “early iteration that we [would] build on” and said that “rapid iteration…will almost certainly be necessary.” I understand that changing plans is always a move that creates justifiable scrutiny — I just don’t think we ever represented that we wouldn’t do so.
Relating to your 6th item (having an FCF in addition to our RSP), I know our intent is to meet the letter and spirit of our regulatory obligations (via our FCF) while also remaining able to experiment with more novel approaches to risk management that go beyond what the law currently requires (via our RSP). I promise to look into it further, as it’s a complex issue and it is possible we are not weighing the considerations in the right way.
Overall, although Anthropic has made plenty of mistakes as an organization — including some of the items on your list — I feel that we have repeatedly taken serious and substantive actions counter to our bottom line and that there is a significant negativity bias distorting assessments like the one you are making above. We’ll always try to do better but, again, I am extremely proud of our track record and stand by it. I will have more to say soon about how we are going to further live our values in this moment of rapid change.
Nathan's righteous anger has inspired me to make one more mildly disapproving observation: it's kind of wild that the Anthropic bill has no government licensing for the independent auditors
Licensing is, IMO, the bare minimum thing that you need to even be able to pretend that your auditing regime isn't just going to devolve into a race to the bottom. Without it, auditors will inevitably compete with each other to offer their customers, the labs, the least bothersome audits available at the lowest possible price point. Obviously Anthropic trusts itself to do a great job of picking great auditors, but do they trust OpenAI to do the same? What about SpaceXSI?
Charlie's analysis is right but he is nicer than I am - I found myself viscerally angry at multiple points reading this bill. There are some good and sensible parts, but overall this legislation is not remotely close to meeting the moment and I overall think passing it would make us less protected against catastrophic risks than we are under existing law.
Two parts that stand out as particularly infuriating when taken together:
1. Some of the preemption provisions are remarkably far reaching and consequential, and in addition to preempting legislation like SB 53 would also remove general powers that state courts have to enjoin defendants and protect the public. Most outrageously, it has a provision that looks like its intended to stops states from enforcing laws or judicial decrees that "suspend the development, deployment, or internal use of an AI model" to protect their citizens from imminent catastrophic harms. Such powers to block and suspend model deployments would then only reside with the federal government.
2. The federal governments powers to restrict the development, deployment, or internal use of a frontier model has to meet an absurdly high bar.
First, there needs to be an imminent risk of a catastrophic harm (meaning killing more than 50 people or causing more than a billion dollars in damage). But thats not enough! The agency head needs to make a detailed determination with no deference given to show why no alternative would prevent the imminent catastrophe. Next the court is also explicitly told to consider the harms to the developer and other harms from restricting the model, but no considerations in the other direction.
This is absurd! For all of its warning about the potential for AI to cause a catastrophe that kills literally everyone on the planet, Anthropic's proposed federal law likely makes it harder than under existing law for a court to get an order blocking or restricting the development of frontier models? WTF??
There are also a bunch of others things that I would like to see improved - the substantive details of the auditing section and the potential for rule-making on substantive safety standards both come to mind - but I think these stand out as particularly indefensible.
Big news. My take: Anthropic deserves some credit for actually proposing a pretty reasonable AI safety bill; that's more than other frontier labs have done.
But they don't deserve *too* much credit, IMO; this bill is weaker than the Trahan-Obernolte FRONTIER Act in some key ways, and since the political momentum has shifted significantly in the direction of safety since FRONTIER was introduced back in July, I think new bills introduced today should ideally go further than FRONTIER was able to go rather than regressing.
To be fair to Anthropic, the main places where this bill is significantly weaker than FRONTIER are (a) rulemaking (very strong in FRONTIER, almost nonexistent in Anthropic's bill) and (b) the emergency shutdown authority. Even if you take safety seriously, it might make some sense to dial those authorities back if you didn't trust the executive branch to wield them responsibly, because those are the easiest provisions for a hypothetical President who hypothetically might not like a certain frontier AI company to abuse.
There are some improvements over FRONTIER as well; for instance, this bill does specify that auditors are employed on a continuing basis and has better provisions regarding auditor access to information. Happy to see that from Anthropic (although I think their access provision is still a little undercooked and has Fourth Amendment problems).
Overall, I think the embedded auditor bills that get introduced in the coming months are going to be significantly better and more ambitious than this, and I hope Anthropic supports those bills instead of opposing anything that treats catastrophic risks like the serious national security issue they are. Still, it's encouraging to see a frontier lab come this far.
Anthropic quietly pitches its own AI safety bill to lawmakers politi.co/4zxDFhl
Charlie Bullock retweeted
Anthropic's Lord Booth-Smith (no relation lol) tells UK lawmakers that US and UK gov are in talks to make the UK AISI one of the third-party evaluators with employee-like access. At oral evidence hearing in UK parliament.
This sounds very promising. If I'm reading the tea leaves correctly, we're going to be getting some kind of oversight by independent auditors, and the federal government is going to make sure it has access to important information from those audits. I am not optimistic about the "voluntary" version of this regime; I think in the medium-term we need legislation to require this by law. But it's a relief to see that the White House seems to be coming around to some version of this idea.
In my view, the crucial features for this auditing regime to include if it's going to be effective are:
- Continuous oversight. Coming in and evaluating one of the developer's models thirty days before they deploy it publicly is not good enough; that would not have prevented Hugging Face or similar incidents, and it's not a good way to address risks from RSI or loss of control. The auditor has to be able to assess the risks posed by internal use of the developer's models, not just misuse by the public. That requires continuous, ongoing oversight of the development process.
- Auditor independence. This is not a new concept and we don't need to reinvent the wheel in the frontier AI context. The same rules that guarantee the independence of auditors in other industries should mostly work here.
- Transparency. It's important for the public to know about the results of these audits for a bunch of different reasons. Publish the reports or other outputs, with redactions for trade secrets and national security as necessary.
- Information sharing. One agency in the federal government having access to information about the risks that frontier models pose doesn't do much if that agency doesn't do anything with that information.
- Access to information. The standard "employee-like access," i.e., access similar to the access that a senior safety employee at the company would have, gets suggested a lot and seems about right. There's a lot of work to be done on operationalizing what that means in practice; e.g., what kind of access does the auditor have to records of Slack communications?
Even if this voluntary auditing regime ends up being useless, I hope that we'll learn from the ways in which it doesn't work out what the failure modes are and how we can improve on the model next time. It's an incredibly difficult task, but I think it's extremely important for our society to get as good at it as we can as fast as we can. The best way to do that is by trying something out and then iterating.
🚨 🚨 Super Intelligence Force Vice Chair @skupor details independent security audits, corporate responsibility, and clear expectations for SI companies.
“We’re not just gonna sit around and wait for the companies to tell us they’ve done their work.”
I don't have a firm opinion on the OpenAI firings yet, because I don't know what information the employees shared or whom they shared it with.
Generally, though, existing law protects California whistleblowers from being fired for blowing the whistle only when they disclose information to the government or law enforcement, or internally within their company--not when they disclose information to private third parties or the press.
Here, it seems like the disclosures were to external AI safety organizations. That kind of disclosure is not protected by any law that I'm aware of, and IMO it probably shouldn't be.
Of course, none of that necessarily means that it was right or acceptable for OpenAI to fire these employees. But I'll withhold judgment on that until we know more about what happened.
