
frontier AI, China, and California @CarnegieEndow
London
Joined August 2019
- Tweets1.5K
- Following1.1K
- Followers2.5K
- Likes2.5K
Scott Singer retweeted
Thank you @Scott_R_Singer for the opportunity to write the section on internal deployments!
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond.
Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks.
With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented.
That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions.
"Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
Scott Singer retweeted
Excited that this report on implementing California's AI safety regulations (plus recs fur future policy) is finally out!
I wrote the assessments and evals section, highly suggest having a read if you're interested in how to improve third-party assessments :)
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond.
Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks.
With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented.
That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions.
"Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
Scott Singer retweeted
Very glad that this report is out, and to have contributed to it. I'm biased, but I think it has some of the most serious, careful analysis of how frontier AI policy can work in practice and where it should go next.
carnegieendowment.org/resear…
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond.
Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks.
With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented.
That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions.
"Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
Scott Singer retweeted
I've seen many instances of people going from ~zero to being ~world leading on important topics in 1-2 years.
You could be one of them!
people seem to be interpreting this as “go and do MATS”. that’s not what I meant -- MATS is pretty good, but there’s lots of work to be done outside of AI safety fellowships!! here’s a quick brainstorm:
- replicate important papers and results in AI safety
- write up relevant findings from different papers on one topic, and try to reconcile any contradictions (or point out the contradictions, and suggest what work could resolve them)
- read and summarise companies’ risk reports and AI safety research
- pick particular claims in risk reports you think are important and write explanations of them, or stress-test them
- write and maintain up-to-date summaries of areas of AI safety research (who’s working on them, what is the current state of our understanding, what’s the most important work and open questions)
- write explainers of important technical questions or problems for busy non-technical audiences.
e.g., what’s going on with CoT controllability and why does it matter; what do we know about reward seeking in frontier models; why might models sabotage AI safety research, and what signs of this have we seen; what’s going on with AI R&D uplift and why is this scary
- become an expert in a research subfield and write up your takes on how good it is, how much resources/effort we should spend on it, what needs to be done next
- pick literally any of the megapapers from the last 12mo, and one of the open questions it lists, and try to answer it
- write explainers of recent AI safety incidents for the general public
- write down a plan for making AI safety go well, notice how far reality is from that plan, and make it happen – e.g. tell an org you think is important about how they’re fucking up and how they can fix it, publicly write about what you want people to do
- call or write to your representative and express your beliefs about AI safety
- cold email people you think are doing good work, come up with some way you could help them, and pitch them on you doing that
- stop overindexing on "do research"; go be a chief of staff or special projects or generalist at a great org. (if you can't get hired yet, talk to people, figure out an important project you can do in ~3mo, do it, then get hired)
- go to a pause AI protest or volunteer for a local group (if you endorse them; I’m a fan of Pause AI UK and haven’t looked deeply into other groups)
- lots of people are going to be very busy now; if you’re dispositionally a chin-scratching contemplative kind of person, now is a good time to do the big picture strategy/And What Else work they won’t have time to do
i put this list together v quickly, i'm missing lots of important things. my main point is that there is _so_ much work to be done
Scott Singer retweeted
It's still very hard to imagine any AI legislation actually passing in the next two years. The White House clearly doesn't seem to be on compromise course, and I suspect a lame-duck President clinging onto power and legacy will be inclined to aggressively wield the veto.
The House, fueled by an anti-corruption and anti-AI mandate, will pass a billion things; the Senate is too blue to really temper these bills to survive the veto, but not blue enough to actually pass them against the administration.
The narrow path to a veto-proof majority, or even to 60 senate votes, for an AI bill runs through one of two channels: votes from natsec-concerned moderate Republicans (who are still not all that likely to outright break with the admin) or from the anti-AI populist wing (who are at risk of diluting any narrow frontier bill into a controversial anti-AI catch-all).
It's hard to imagine how that kind of fragile coalition just emerges from a neutral position in the next few months. Yes, maybe, if VP Vance and political advisors realise the risk of leaving the AI flank open until '28, they could lean on the White House to enable some compromise - but decisions like the Miller appointment should make you much more doubtful that moderation will be the play.
Realistically, the most promising policy window is in response to the next crises. That's a common refrain: we write a bill, put it in the drawer, and if something big happens, we'll pass it immediately. It's particularly tempting because it allows advocates to ignore political reality and instead write down their favorite bill, hoping that exogenous shocks do the political lifting.
I think it's a bit harder than that: you need to draft legislation that makes sense to policymakers now so that it even goes into the drawer - or more accurately, that it gets influential sponsors and maybe a markup so that it's sufficiently red-teamed to pass in a pinch. But it also needs to be ambitious and specific enough that it seems like a solution to whatever new problem emerges. Congress might not reach for cyber bills after bio incidents, or for common-sense low-hanging fruit after major loss of life.
Since we don't know what the hypothetical wake-up call looks like or when it'll arrive, I think that means a lot of hard work: getting enough different legislative vehicles ready that we'll have one ready to go no matter what happens, but have these efforts be sufficiently coordinated that they don't compete for immediate attention here and now.
That kind of positioning is very different than introducing one big AI bill and feeding it into a pipeline that inevitably ends well before being signed into law. If we don't get this right, I expect a lot of posturing in the short term as we ultimately fail to pass a reasonable bill, immediately superseded by unanchored chaos as soon as the next incidents hits and we don't have a suitable bill developed far enough.
The odds of passing legislation have probably gone up over the week, but they still hinge on tactical preparation and bipartisan circulation in advance of very specific policy windows. That won't always align very well with the current vibes of accoubtability enthusiasm and (however well-motivated) congressional reconquista.
Carnegie's @Scott_R_Singer explains why Washington's lack of AI safety legislation undermines the US's ability to pressure China on AI safety:
"The overarching question, if you care about US-China relations, is the US gonna govern AI domestically?"
"If you have a conversation with the Chinese and you say, 'Hey, you all should really do more on AI safety,' the first question that they'll reasonably ask is, 'Well, why isn't Washington doing more? If you really believe sincerely in the risks, then why is there no national legislation?'"
"You can talk about what's going on in California. You can talk about all that the companies are doing voluntarily. But it really just doesn't totally land."
@CarnegieEndow
Scott Singer retweeted
We are hiring a Head of Private Office and two Private Secretaries at @AISecurityInst to work with me (Chief Strategy Officer) and @HZoete (Director).
If you’re sharp, ambitious, and want to help us run an important and fast-moving organisation, read on!
Scott Singer retweeted
🚨🚨🚨 Job Alert with @CarnegieEndow @carnegienpp
We're hiring for a new Stanton fellow--pre-doc, post-doc, or junior faculty--to start next summer. Great opportunity to do policy-relevant nuclear research with a fun team.
Deadline: Nov 30. Link in next post.
Please repost!
Scott Singer retweeted
China’s Fifth Plenum decision is out.
Beijing wants the Party to have more influence in private tech firms.
“Strengthen Party-building efforts in platform companies, technology-innovation enterprises, and other such firms; establish a mechanism for dynamically adjusting the elevation of Party organizations within internet companies to higher administrative levels of management; and ensure that responsibilities for Party-building are fulfilled.”
Scott Singer retweeted
[1/7] Some late-night vibe research for fun: I used AI to scan everything Anthropic has published plus Dario’s essays, and counted every mention of China. Then I did the same for OpenAI and Sam Altman. Some fun findings:
Scott Singer retweeted
Impt development. China really needs more high-quality evaluations for frontier risks: in-house, 3rd party, gov-affiliated.
Fangcun comes from a highly credible research group at Tsinghua. It’s building a real business on evaluations and (currently more marketable) guardrails.
Scott Singer retweeted
I’m building a team in DC. Everyone in Washington has a stake in ensuring that reliable info about alignment and RSI reaches the public.
We need trustworthy communicators who can explain METR's loss-of-control risk assessments and goals to policymakers.
Scott Singer retweeted
“The Chinese government is determined to lead the transformation toward a world of AI-powered robots.”
Read @Scott_R_Singer and Pavlo Zvenyhorodskyi on Beijing’s bet on embodied AI—and how Washington should respond:
foreignaffairs.com/china/chi…
Scott Singer retweeted
The American AI industry needs a lot more datacenters than it can build at home. If US allies host these datacenters, they might be able to secure access to frontier AI in return.
New for @ForeignPolicy with @SamWinterLevy, on getting compute-for-access right:
Scott Singer retweeted
Despite a dominance in computing power, U.S. frontier AI models are still only six to nine months ahead of China’s. @mattsheehan88 explains why Chinese models are able to keep up.
Watch the full interview here: foreignpolicy.com/live/sheeh…
Scott Singer retweeted
Today on FP Live: @mattsheehan88 joins FP’s @RaviReports to discuss why the U.S. and China govern AI so differently and what the competition means for the future of geopolitics.
Tune in here at 11:00AM ET: foreignpolicy.com/live/sheeh…
Scott Singer retweeted
Most countries won't do what it takes to build their own frontier AI. But they don't want to be dependent on foreign AI without a backup.
@SamWinterLevy and I argue there's a middle path: AI middle powers could retain 'AI Breakout Capacity'.
There's precedent: Japan doesn't have a nuclear weapon, but always retains the capacity to build one on short notice.
Middle powers can make strategic choices today to ensure their ability to build frontier systems tomorrow: onshore compute, secure chip purchasing rights, sustain pre-training expertise, and create the right legal basis and private-sector vehicles for a sprint.
They'd still have to spend big to build frontier systems later on. But the longer they retain breakout capacity, the longer they can afford to wait: to see if interdependence with America can work, or until their electorate will let them go all-in on an AI moonshot.
Until then, the middle path is their wisest choice: hope that U.S. alignment works out, but retain the capacity to deal with being cut off for good.
Read more in our latest for @CarnegieEndow's @CEIPTechProgram: carnegieendowment.org/resear…
Scott Singer retweeted
AISI IS HIRING!
It was less than a month ago that I became Director. I joined with the belief that AISI is a world-leading organisation, and living proof that government can build things that work.
One month in and I'm even more bullish about the vital role @AISecurityInst plays in frontier AI security. I'm astounded daily by the talent, focus and commitment of the brilliant team I get to work with.
There has never been a more urgent time to work on frontier AI security - independently, in the public interest. We have a lot of urgent work to do, our team is growing fast, and we need more brilliant people to fill those roles.
Our Red Team uses adversarial machine learning techniques to find failures in alignment measures, control monitors, and misuse guardrails. They are massively scaling up all parts of the team:
Alignment: job-boards.eu.greenhouse.io/…
Control: job-boards.eu.greenhouse.io/…
Misuse: job-boards.eu.greenhouse.io/…
AI capabilities in cybersecurity and autonomy are advancing faster than ever. Our Cyber & Autonomous Systems team assesses what frontier and open-weight models can really do - across cyber, autonomy and AI R&D. We're hiring a Cyber Security Engineer and Software Engineer to build the evaluations behind that work:
CSE: job-boards.eu.greenhouse.io/…
SWE: job-boards.eu.greenhouse.io/…
Our Human Influence team studies how AI can shift human decisions and behaviour, using everything from RCTs to multi-agent studies. We're hiring an Engineering Lead to help us grow the ambition and pace of that research:
job-boards.eu.greenhouse.io/…
We're also hiring exceptional Software Engineers at all seniority levels to join our Core Technology team, which works closely with researchers to build performant tools and infrastructure that enable and accelerate AISI's world-leading AI safety research:
job-boards.eu.greenhouse.io/…
Scott Singer retweeted
Could automating AI R&D radically accelerate AI progress in an “intelligence explosion”?
Preliminary evidence suggests that it could.
In a new paper with authors across academia, civil society, and frontier AI companies (including @dawnsongtweets @merettm @jackclarkSF @Yoshua_Bengio @geoffreyhinton ), we assess the evidence & offer policy recommendations 🧵
Scott Singer retweeted
The US and China are competing to lead the AI race. But can they realistically cooperate on AI safety at the same time? I ask @Scott_R_Singer:





























