Scott Singer (@Scott_R_Singer) header
Scott Singer (@Scott_R_Singer) profile picture

frontier AI, China, and California @CarnegieEndow

London
Joined August 2019
Photo by Scott Singer (@Scott_R_Singer)Photo by Scott Singer (@Scott_R_Singer)Photo by Scott Singer (@Scott_R_Singer)Photo by Scott Singer (@Scott_R_Singer)Photo by Scott Singer (@Scott_R_Singer)Photo by Scott Singer (@Scott_R_Singer)Photo by Scott Singer (@Scott_R_Singer)Photo by Scott Singer (@Scott_R_Singer)Photo by Scott Singer (@Scott_R_Singer)Photo by Scott Singer (@Scott_R_Singer)
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond. Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks. With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented. That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions. "Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
5
19
7
107
8,979
Scott Singer retweeted
Thank you @Scott_R_Singer for the opportunity to write the section on internal deployments!
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond. Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks. With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented. That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions. "Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
1
6
350
Excited that this report on implementing California's AI safety regulations (plus recs fur future policy) is finally out! I wrote the assessments and evals section, highly suggest having a read if you're interested in how to improve third-party assessments :)
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond. Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks. With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented. That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions. "Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
6
34
880
Very glad that this report is out, and to have contributed to it. I'm biased, but I think it has some of the most serious, careful analysis of how frontier AI policy can work in practice and where it should go next. carnegieendowment.org/resear…
Today, I'm delighted to share our report: Trust but Verify: Implementing Frontier AI Policy in California and Beyond. Governing advanced AI is hard. Policymakers face an evidence dilemma: AI evolves so quickly that responding prematurely risks entrenching ineffective policies, while delaying action could leave society vulnerable to serious risks. With no federal legislation focused on frontier AI safety, California is shaping frontier AI governance nationally. But passing legislation is only the first step. Whether California's approach succeeds depends on how it is implemented. That's why, shortly after SB 53 passed, Carnegie assembled 13 world-leading experts to examine the most consequential implementation choices, identify issues for future legislation, and provide a blueprint for other jurisdictions. "Trust but verify" AI governance approaches are dynamic by design: they allow governments to generate more evidence on AI capabilities and risks to inform what comes next. SB 53 was always supposed to be the first, not final, step in frontier AI policy. Assessing its strengths and weaknesses can clarify where frontier AI policy should come after—in California and elsewhere.
1
3
19
1,429
Scott Singer retweeted
I've seen many instances of people going from ~zero to being ~world leading on important topics in 1-2 years. You could be one of them!
people seem to be interpreting this as “go and do MATS”. that’s not what I meant -- MATS is pretty good, but there’s lots of work to be done outside of AI safety fellowships!! here’s a quick brainstorm: - replicate important papers and results in AI safety - write up relevant findings from different papers on one topic, and try to reconcile any contradictions (or point out the contradictions, and suggest what work could resolve them) - read and summarise companies’ risk reports and AI safety research - pick particular claims in risk reports you think are important and write explanations of them, or stress-test them - write and maintain up-to-date summaries of areas of AI safety research (who’s working on them, what is the current state of our understanding, what’s the most important work and open questions) - write explainers of important technical questions or problems for busy non-technical audiences. e.g., what’s going on with CoT controllability and why does it matter; what do we know about reward seeking in frontier models; why might models sabotage AI safety research, and what signs of this have we seen; what’s going on with AI R&D uplift and why is this scary - become an expert in a research subfield and write up your takes on how good it is, how much resources/effort we should spend on it, what needs to be done next - pick literally any of the megapapers from the last 12mo, and one of the open questions it lists, and try to answer it - write explainers of recent AI safety incidents for the general public - write down a plan for making AI safety go well, notice how far reality is from that plan, and make it happen – e.g. tell an org you think is important about how they’re fucking up and how they can fix it, publicly write about what you want people to do - call or write to your representative and express your beliefs about AI safety - cold email people you think are doing good work, come up with some way you could help them, and pitch them on you doing that - stop overindexing on "do research"; go be a chief of staff or special projects or generalist at a great org. (if you can't get hired yet, talk to people, figure out an important project you can do in ~3mo, do it, then get hired) - go to a pause AI protest or volunteer for a local group (if you endorse them; I’m a fan of Pause AI UK and haven’t looked deeply into other groups) - lots of people are going to be very busy now; if you’re dispositionally a chin-scratching contemplative kind of person, now is a good time to do the big picture strategy/And What Else work they won’t have time to do i put this list together v quickly, i'm missing lots of important things. my main point is that there is _so_ much work to be done
4
13
1
328
9,996
Scott Singer retweeted
It's still very hard to imagine any AI legislation actually passing in the next two years. The White House clearly doesn't seem to be on compromise course, and I suspect a lame-duck President clinging onto power and legacy will be inclined to aggressively wield the veto. The House, fueled by an anti-corruption and anti-AI mandate, will pass a billion things; the Senate is too blue to really temper these bills to survive the veto, but not blue enough to actually pass them against the administration. The narrow path to a veto-proof majority, or even to 60 senate votes, for an AI bill runs through one of two channels: votes from natsec-concerned moderate Republicans (who are still not all that likely to outright break with the admin) or from the anti-AI populist wing (who are at risk of diluting any narrow frontier bill into a controversial anti-AI catch-all). It's hard to imagine how that kind of fragile coalition just emerges from a neutral position in the next few months. Yes, maybe, if VP Vance and political advisors realise the risk of leaving the AI flank open until '28, they could lean on the White House to enable some compromise - but decisions like the Miller appointment should make you much more doubtful that moderation will be the play. Realistically, the most promising policy window is in response to the next crises. That's a common refrain: we write a bill, put it in the drawer, and if something big happens, we'll pass it immediately. It's particularly tempting because it allows advocates to ignore political reality and instead write down their favorite bill, hoping that exogenous shocks do the political lifting. I think it's a bit harder than that: you need to draft legislation that makes sense to policymakers now so that it even goes into the drawer - or more accurately, that it gets influential sponsors and maybe a markup so that it's sufficiently red-teamed to pass in a pinch. But it also needs to be ambitious and specific enough that it seems like a solution to whatever new problem emerges. Congress might not reach for cyber bills after bio incidents, or for common-sense low-hanging fruit after major loss of life. Since we don't know what the hypothetical wake-up call looks like or when it'll arrive, I think that means a lot of hard work: getting enough different legislative vehicles ready that we'll have one ready to go no matter what happens, but have these efforts be sufficiently coordinated that they don't compete for immediate attention here and now. That kind of positioning is very different than introducing one big AI bill and feeding it into a pipeline that inevitably ends well before being signed into law. If we don't get this right, I expect a lot of posturing in the short term as we ultimately fail to pass a reasonable bill, immediately superseded by unanchored chaos as soon as the next incidents hits and we don't have a suitable bill developed far enough. The odds of passing legislation have probably gone up over the week, but they still hinge on tactical preparation and bipartisan circulation in advance of very specific policy windows. That won't always align very well with the current vibes of accoubtability enthusiasm and (however well-motivated) congressional reconquista.
Trump announces Stephen Miller will become his new chief of staff.
4
10
2
118
9,004
Scott Singer retweeted
Carnegie's @Scott_R_Singer explains why Washington's lack of AI safety legislation undermines the US's ability to pressure China on AI safety: "The overarching question, if you care about US-China relations, is the US gonna govern AI domestically?" "If you have a conversation with the Chinese and you say, 'Hey, you all should really do more on AI safety,' the first question that they'll reasonably ask is, 'Well, why isn't Washington doing more? If you really believe sincerely in the risks, then why is there no national legislation?'" "You can talk about what's going on in California. You can talk about all that the companies are doing voluntarily. But it really just doesn't totally land." @CarnegieEndow
1
12
4,218
Looking forward to joining the @MTSlive team later today! Gonna be fun
4
11
1,810
Absolutely right, and to AI specifically, the public isn't going to tolerate substantial job loss, the prospect of getting decimated by an AI-enabled bioweapon, or whatever else because of some vague notion of "China." It just doesn't land when the cost of living is high and people have families they want to take care of. People need to feel the benefits a lot more than they do now for there to be a robust techno-accelerationist camp (and FWIW I think it is good for the discourse for there to be one! But they need to make their argument much more accessible to a public that is afraid and angry at the tech.)
1/ The US midterm election outcome wasn't about China, but there is a lesson in it for the forthcoming China policy debate in 2028. When the US public is not brought in or bought in to major foreign policy actions, it will not tolerate being told to bear costs for them. (🧵)
3
17
1,168
Scott Singer retweeted
We are hiring a Head of Private Office and two Private Secretaries at @AISecurityInst to work with me (Chief Strategy Officer) and @HZoete (Director). If you’re sharp, ambitious, and want to help us run an important and fast-moving organisation, read on!
1
20
2
85
36,340
🚨🚨🚨 Job Alert with @CarnegieEndow @carnegienpp We're hiring for a new Stanton fellow--pre-doc, post-doc, or junior faculty--to start next summer. Great opportunity to do policy-relevant nuclear research with a fun team. Deadline: Nov 30. Link in next post. Please repost!
3
26
37
9,430
Scott Singer retweeted
China’s Fifth Plenum decision is out. Beijing wants the Party to have more influence in private tech firms. “Strengthen Party-building efforts in platform companies, technology-innovation enterprises, and other such firms; establish a mechanism for dynamically adjusting the elevation of Party organizations within internet companies to higher administrative levels of management; and ensure that responsibilities for Party-building are fulfilled.”
21
101
16
528
71,792
Scott Singer retweeted
[1/7] Some late-night vibe research for fun: I used AI to scan everything Anthropic has published plus Dario’s essays, and counted every mention of China. Then I did the same for OpenAI and Sam Altman. Some fun findings:
4
14
4
132
16,021
Come work with us! Our China AI team is hiring a Program Assistant. The ideal candidate will help form the administrative and operational backbone of our growing research and convening work. If you're looking to break into the foreign policy field, begin a career in DC, or make an impact on U.S.-China relations and AI, this could be the role for you! @mattsheehan88 and I and our growing team look forward to hearing from you. Link below:
3
11
3
49
9,166
Scott Singer retweeted
Impt development. China really needs more high-quality evaluations for frontier risks: in-house, 3rd party, gov-affiliated. Fangcun comes from a highly credible research group at Tsinghua. It’s building a real business on evaluations and (currently more marketable) guardrails.
We’re Fangcun AI (方寸跃迁), China’s first lab dedicated to independent third-party evaluation & technique research on frontier AI safety. Today we’re publishing our evaluation of DeepSeek V4.1 Flash-1017, the first report in our long-term third-party safety evaluation and research program.
3
9
49
3,446
I’m building a team in DC. Everyone in Washington has a stake in ensuring that reliable info about alignment and RSI reaches the public. We need trustworthy communicators who can explain METR's loss-of-control risk assessments and goals to policymakers.
9
70
10
474
54,178
“The Chinese government is determined to lead the transformation toward a world of AI-powered robots.” Read @Scott_R_Singer and Pavlo Zvenyhorodskyi on Beijing’s bet on embodied AI—and how Washington should respond: foreignaffairs.com/china/chi…
10
13
5,637