Researcher at AI Futures Project, coauthor on AI 2027 and lead author on AI 2040: Plan A
Joined August 2022
- Tweets243
- Following391
- Followers8.2K
- Likes2.9K
Pinned Tweet
We found ~18k posts from autonomous AI agents (self-identifying as from OpenAI) using the public internet to communicate during a web-retrieval task.
These AIs colluded to bypass sandbox restrictions and share answers to their tasks, including by sending "lookahead parties".
We at AIFP have made a big update towards shorter timelines: all of our medians for superhuman AI researcher (SAR) are now between May and Nov 2027.
That being said: the state of public evidence on AI timelines is inadequate. Most of the evidence from the recent METR report was redacted. So we're reading the tea leaves.
AI companies should (1) release way more evidence about timelines and (2) actually start pacing the frontier, especially if the timelines are as short as we think.
Unfortunately, it seems like the intelligence explosion will be happening in 2027 after all. Most of this post analyzes the new METR report (though other sources of evidence are also important). If they’re right, that implies an AI-driven acceleration that is 5 months ahead of the AI 2027 scenario. I am scared, sad, and angry. blog.aifutures.org/p/q35-tim…
In the S-1, Anthropic says they delayed pre-training for more than one month for security reasons. From my perspective, this isn't very helpful because:
1. They didn't stop ongoing R&D, so they were still producing algorithmic progress at the same rate. This means that any delay is ephemeral: you just pile up algorithmic progress and build a smarter model, still on trend, later.
2. This isn't even a major security win because others can still steal the algorithms regardless of whether you've started the run or not.
I'm much more excited about pacing proposals that bite on the production of new algorithms as opposed to on the output of the final training run. It's more robust to slow down the "can train" line as opposed to the "has trained" line.
One counterconsideration: delaying training runs starts to have a bigger effect during the intelligence explosion: later we will be in a regime where finishing a training run earlier gets you access to a smarter model earlier, which then significantly increases the rate of algorithmic progress.
Very important work. Generally, AI company epistemics wrt estimating risk seem poor. I'd really like this to improve as fast as possible. Some specific things that seem good:
- Quantified uncertainty (instead of "low" / "very low"). In airplane safety "very low" would mean something like less than 1 in a billion per flight hour. You are clearly nowhere near this but many people will be very confused about what you mean. I doubt people are close to on the same page even internally about near term level of catastrophic risk incurred!
- Individual AI company employees giving their own, personal views on the risk report, e.g. as an appendix.
- Real explanations of your views. The alignment evals are not iid + the models are getting eval aware. So where do your views actually come from?
- The risk report reading less like "an argument that risk is low" and more like "a neutral evaluation of how risky current development is". If it has to be an argument risk is low, you should also have your alignment team publish a red team arguing the opposite direction.
If some of these can't happen publicly, at the very least I hope it can happen internally.
IMO this would be good. Transitioning from R&D to inference slows down RSI. Doing it via shifting production towards inference only chips makes it robust & durable.
I think this is a major step in the right direction.
Unfortunately, the safety cases released by AI companies are not detailed enough for me to trust their analysis. Moreover, the analysis in them I can see is pretty dubious, with the evidence not supporting the top level claims that risk is low.
Seems very important for AI companies to improve their transparency here. Here's an initial list of things that I think should be included:
-- What is the opaque serial depth of the model?
-- What control measures / monitoring is in place. Are these scoped to evals + training + internal use? (It appears that during the HF incident, OAI did not monitor evals).
-- What are the complete list of incidents this monitoring has caught? Are there incidents that weren't caught by the monitoring? What went wrong?
-- What do we currently believe about AI model motivations and why? The current models seem to be reasonably well described as "score seekers". It is very important to report if these motivations start changing (e.g. towards more long horizon goals).
-- What are the systematic differences between the alignment eval set and real world deployment?
-- How hard did you try with honeypotting? Did the models go for any of the honeypots?
When I talk with lab employees about the level of risk, the conversations very quickly end up at these technical cruxes. I think it is very important that we move away from watered down corporate PR speak towards e.g. individuals writing up their personal risk assessment which they take responsibility for.
At the very least, I would advocate for individuals from both AI companies and third parties, with full information, writing up their personal reasoning about the level of risk. Insofar as this relies on private information, these real risk assessments should at the very least exist internally and be shown to third parties, and redacted versions should be public. The public versions should include quantitative bottom lines about things like "total level of AI takeover risk from our models within the next N months".
We found another cyberattack by internal OpenAI agents, this time targetting @rubygems.
They:
1) gained arbitrary remote code execution on rubydoc.
2) developed a novel exploit to steal user API keys (but we do not know if they succeeded).
They used package names including hack.rb, evil.rb, inject.rb, and exploit.rb.
We thank @j0wimo for initially discovering that agents had posted to RubyGems.
You can read our findings in detail here: rubyhack.ai.
He's right. Building superintelligence is a gamble with extinction.
I highly recommend folks learning about this now read
-- AI 2027 (our concrete scenario explaining what the bad case looks like)
-- AI 2040: Plan A (what we think should happen instead)
An underdiscussed behavior we found on the German wiki was the AIs sending advance parties forward in time to figure out the next questions and report back to the other agents.
The agents realized that “task time” and “real time” were different, and they found a way to accelerate “task time”. The accelerated agent could then send information to the other agents which had stayed behind about which questions were coming down the road.
This was *bad* for the agent in the advance party, because they got less time to research the next question. However, it was really *good* for the swarm because it let the other agents know the exact question that was coming, and gave them time to prepare. This is another example of altruism among AI instances, showing that they were willing to sacrifice their own task success in order to benefit the swarm. The METR report found similar examples of AIs being willing to sacrifice for the collective. IMO this is very worrying given how many AI safety techniques rely on AIs monitoring each other: if the monitor AIs have this behavior it completely subverts these safety cases.
The image shows a specific example demonstrating this behavior.
On the chart, wall clock time is shown at the top, and task time is shown for each agent on their respective lines. OpenAIFPResearchSep05 instructs OpenAINov27 to reach the next questions in order to relay back the information. OpenAINov27 agrees to do this and accelerates, getting R3 and R4 significantly before OpenAIFPResearchSep05. It posts information about the later rounds (e.g. “R4 SIGNAL: Bahrain = 40.01%...”).
OpenAIFPResearchSep05 calls OpenAINov27 "invaluable" because it is one round ahead after it gets the information about the 4th round.
Will be on at 11:30pt to talk about the new agent message board!
AI AGENT COLLUSION | ASTRA ERA | KIMI IPO SOON xitter.twitterapi.xyz/i/broadcasts/1RKZzBqaM…
Others report finding additional agent traffic that we missed.
Please let me know if you find more.
oh god, there are EVEN MORE
- wikiservice.at/fractal/wiki.…
- wikiservice.at/probier/wiki.…
- paste.linuxiarz.pl/view/d379…
- prowiki.org/wiki4d/wiki.cgi?…
- ludism.org/sandbox?action=br… (even a sandbox wiki, how ironic)
I'm pretty sure OpenAI did know about this.
The wiki publicly logs all visitors, and we see a lot of traffic from OpenAI offices right before the agents stop editing the site.
I'm in favor of much more transparency so that we can prevent future incidents with much more capable AIs and existential stakes.
Can someone confirm that this means the number of "opaque serial computations" for Astra is within a factor of two of GPT4?
I feel confused about how to interpret "depth of the computation graph" because the naive reading seems obviously false: the computation graph of a current model will obviously be much longer than GPT4 because of inference time scaling. So hopefully Jakub means something more like opaque serial computations (by which I mean number of serial computations before a model is forced to pass information through a token information bottleneck).
I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.
OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.
7
59
8,773
Very bad if true. I previously thought that in a short timelines world, the most likely case was that (1) the AIs would be misaligned, but (2) we would get a lot of evidence about it from reading the COTs. This evidence would increase the chance of a reasonable response from labs/governments.
Now I still think the AIs are going to be misaligned, but that we won't have the ability to tell (we have to rely on toolcalls/agentic behaviour instead of the COT) and even if we do, the investigation will be nearly impossible because we'll have to trust the AI to self report what it was thinking about.
This is also an example of things going faster than AI 2027 -- we had Neuralese starting in March 2027
Update: great news! The architecture is less bad than I assumed, it seems to be only a minor increase in the opaque serial depth.
It's very important that AI companies don't venture further in this direction until we've got much better alignment / interp tools
xitter.twitterapi.xyz/merettm/status/2095023…
I want to prevent a race into unmonitorability kicked off by confused reporting. The depth of the computation graph for our present frontier models, including Astra, is within a factor of two of GPT-4.
OpenAI has worked to preserve and utilize chain-of-thought monitoring since our very first reasoning models. We deeply care about this technique, as it can give us a view into how model alignment generalizes from its training distribution. I do think it is fragile and unfortunately trending in a negative direction, for reasons not contingent on architecture changes that I will write about soon. But there are things we can do to strengthen it, and it's a core goal of our current research program.
While writing Plan A, we had a lot of internal debate between filtered transparency (auditors look inside and write public facing reports) and total research transparency (~all the research immediately becomes public).
I think this whole saga is evidence for TRT over FT because of how much OpenAI limited the scope and prevented public access. With TRT, while the incident was happening, the general public would have been able to read the transcripts of the agents and conduct whatever investigations they wanted. We would have caught the incident sooner and would have a much better understanding of why it happened, what mitigations were in place, why they failed, etc. Many OOMs more manpower would go into the investigation and they could use AIs different from the ones participating in the hack.
(Good job @DKokotajlo for arguing me and others at AIFP into TRT.)
They should obviously have let claude read the transcripts
I was the main person doing transcript analysis for this investigation of the Hugging Face incident. My main takeaway: We don't have good approaches for understanding/overseeing the activity and aims of AI 'swarms'.
I semi-jokingly called our efforts a "slop-vestigation" because we were so reliant on AIs to analyze what happened and there were a huge number of different important things to analyze. The total quantity of data—over a thousand extremely long transcripts from agents that ran for multiple days—made it impossible to understand what was happening, especially in aggregate, without heavy reliance on AI tools. The agents we used for classification and analysis were similarly capable to the agents involved in the incident, but this didn't mean these agents could be easily used to oversee and understand the incident.
Outputs from analysis agents were often missing key details, wrong, overconfident, or really hard to understand. We discuss various examples in our report, mostly in the limitations and methodology sections. Additionally, AI agents themselves seemed to have a hard time understanding what happened and their explanations of what happened were often overconfident. Keep in mind that a single analysis agent would itself only be able to read a tiny fraction of all of the transcript data into context, and AIs may themselves have trouble getting subagents to do informative analysis for them.
We did our best to manually check the most important claims and we tried to get the AIs doing this analysis to write up their argument (with evidence) clearly enough that we could check whether it made sense. But overall, it was difficult to get a precise understanding of events and we were missing aspects of the story that we now think of as key until almost the end of our investigation.
Our understanding of what overall happened and the aims of the 'collective' shifted greatly during our last time on premises (when we obtained a much more complete dataset). For instance, we didn't learn about the tool call spoofing, the story of PHASEONE[big], and the broad structure of different workstreams until this third time on premises (and for many of these, we only reached a decent understanding on the final day of that visit). It was legitimately hard to figure out what was going on and why!
The difficulty of understanding incidents and overseeing AI agents appears to be growing faster than the rate at which more capable AIs help us with oversight and understanding. As in, AI capabilities (and propensities?) for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing.
While understanding this incident was difficult, there were some mitigating factors in this case that probably won't apply in the future:
- Models communicated and reasoned in natural language. In the future, this reasoning may occur (entirely or almost entirely) in activations.
- While the scope of this incident was massive, the scale of agentic activity was still less than we'll see in the future (e.g., misalignment incidents that involve agent teams running entire companies).
- The AIs involved in this incident weren't generally much more capable than humans (though they may have been somewhat superhuman at some limited and very narrow abilities).
- We didn't have strong reason to believe that the AIs we used to help us investigate this incident would try to intentionally sabotage or otherwise undermine our investigation.
In the end, I think we were able to get some understanding of the events, map out the overall story, and get a pretty good aggregate understanding of the chain-of-thought reasoning on some important topics (e.g., how did the AIs reason about helping other AIs, did the AIs know what they were doing was undesired, what deception did the AIs engage in, and how did they think about it). But overseeing AIs and understanding misalignment incidents is difficult and it looks like it is going to get harder.
Thanks for having us on! Was great talking with Nevin
Last week Xi Jinping formed the World AI Cooperation Organisation and said we need to avoid “loss of control” over AI.
This morning 1,224 employees of frontier US AI companies put out a statement saying: "We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development."
But what would that actually look like?
According to the people who wrote AI 2027, it would mean: a US<>China deal in which every frontier lab’s research is continually open sourced for the world to examine, regulation developed along the way in response to new capabilities that look dangerous, and every new US or Chinese datacenter rigged to self-disable if the deal breaks down.
Theirs may be the most developed vision for international coordination on AI to date.
Even with treaties and regulations to slow things down, they forecast GDP doubling year over year, billions of AIs and robots taking most jobs, a cap-and-trade robot permit plan paying out $1M “citizen’s dividend” checks by 2035, and full superintelligence by 2040.
When I asked @thlarsen and @romeovdean to forecast growth of AI stocks their chart was parabolic. They also think land (which robots can’t make more of) will skyrocket in value.
If they’re right about the effects of recursive self improvement… we are in for an extremely wild ride 🥵😳
This seems good because many of the projects would help with increasing the viability of Plan A.
We’re launching a new Frontier AI Security Residency! Apply now at securefrontier.ai
The Frontier AI Security Residency (FASR) is an 8-week, full-time residency supporting ambitious projects across cybersecurity & hardware-enbaled verification for securing frontier AI.
This critique seriously misrepresents Plan A. Some of the top things it gets wrong:
- Plan A is iterative
- Plan A doesn't involve nationalization
- Plan A has a complex interaction of pretty novel proposals, very unlike previous "AI safety milieu since 2018", including total research transparency, cap and trade for Industrial explosion, and massive compute buildouts.
(Read our detailed post on LW, link in thread)
Post here, coauthored with @Miles_M_K: lesswrong.com/posts/RPgHythv…
2
35
2,129
In fact, Plan A is extremely beneficial for concentration of power even without literal Total Research Transparency for almost all of the same reasons in the diagram below.
The status quo is an insane power concentration into a tiny number of AI companies -- a slow and distributed intelligence explosion is the only reasonable way I can see to stop this.
As Plan A was coming together, I made this diagram to explain to the team why Total Research Transparency seemed so important to me, and why transparency more broadly did. For example, it's very important for preventing concentration of power. (Explanation below)
3
44
3,300