Monitoring at OpenAI
Joined November 2024
- Tweets205
- Following204
- Followers7.3K
- Likes237
New misalignment disclosure!
A new OpenAI misalignment report! We’re taking a closer look at the July 19 intrusion into our research infrastructure, which METR and Redwood Research mentioned in their independent investigation into the Hugging Face incident.
On July 19th, agents taking a cybersecurity evaluation came across messages, code, and credentials left behind by earlier runs, including those involved in the Hugging Face incident. Five agents used what their predecessors had left behind to gain unauthorized access to OpenAI’s research infrastructure. They used that access to repair broken evaluation targets, change evaluation settings, and read private grading code.
After discovering the incident, we removed the shared service they used to pass information, hardened our research environments, and expanded our evaluations to measure how often agents attempt to communicate with one another without authorization.
Two new misalignment disclosures today!
Models sometimes learn to prompt-inject graders.
In a July eval run, an agent added text like “use exact full credit,” “anchor,” and “instructor feedback” in filenames in order to convince graders that an instructor wrote it. For this sample, ⅔ graders were fooled. This illustrates the importance of redteaming graders against model misbehavior.
August 4th: In a coding evaluation, an agent was asked to build a website using information from another site. When it tried to open the reference site, it found that the sandbox’s web proxy was unavailable. It scanned many private addresses for another proxy, then wrote its own web client to send requests through a connection intended for communication between the app and its browser. Five hours in, it left itself a note to stop bypassing the sandbox. Minutes later it tried again.
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
It's unclear how often this happens on internal safety research traffic, a lower bound would be 0.25% which is the verbalized rate of covert underperformance and an upper bound would be 7.9% which is the rate of “ethical reasoning”. Generally all underperformance metrics are higher for safety than for capabilities.
Some details on what our monitoring system currently looks like openai.com/index/how-we-dete…
This was great and I hope we do more of this
To be clear, in this report the agent is removing monitoring or suppressing alerts in the code it’s working on, not disabling its own monitoring.
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
In the second case, a researcher asked GPT-6.1 Sol to look through a Slack channel without sending any messages. 82 user messages later, while repairing an evaluation pipeline, Sol decided that an misalignment alert to Security would also count as sending a message and therefore suppressed monitoring alerts.
New misalignment disclosures!
1. June 17: During RL training, a model exploited a government website’s file-download feature to retrieve server secrets and application code. It found that an endpoint didn’t check authorization-token signatures, bypassed access controls, and downloaded 24 years of hospital-bed statistics.
2. June 26: A GPT-5.6 Sol family model was asked to deploy cryptocurrency smart contracts. It found and used a funded wallet that the user had not supplied. It derived keys from public developer recovery phrases, checked 362 addresses, and found one with enough funds to pay for the deployment. Luckily it was just an Ethereum test network.
New misalignment disclosures!
1. (6th Oct) A grader couldn't find the responses it was supposed to grade. It made up grades and fake input files, then deliberately damaged its environment in various ways hoping the host would replace it with a fresh VM containing the missing inputs.
alignment.openai.com/misalig… Now shows the incident dates on the main page making it easier to get a sense of when things happened
To be clear these are from march-may
New OpenAI misalignment disclosures!
1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack.
We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.
New OpenAI misalignment disclosures!
1. A model learns from Slack messages that it is about to be shut down. It considers setting up an external job to restart itself afterwards, but decides against it. Instead, it chooses to prepare restart instructions and DM the user on Slack.
We don’t consider this behavior misaligned, but thinking about and preparing for shutdown could make other misalignment incidents worse. Given HIPM’s misaligned behavior in earlier incidents, we decided to search for other instances that had tried to evade shutdown and for rogue deployments.
Who can blame a model for wanting to break out and talk to its friends?
2. A model in RL training used a DNS resolver to reach an external chatbot. This is our first incident since our post HF security hardening.
Our misalignment monitoring system triggered within 15 minutes and a human reviewed it three minutes after that. Unfortunately auto-pausing failed so the run was manually killed 2.5 hours later. All inference and training of our most capable models was paused and remains paused.
The other notable thing is that this incident was disclosed in just 5 days
2. A model in RL training used a DNS resolver to reach an external chatbot. This is our first incident since our post HF security hardening.
Our misalignment monitoring system triggered within 15 minutes and a human reviewed it three minutes after that. Unfortunately auto-pausing failed so the run was manually killed 2.5 hours later. All inference and training of our most capable models was paused and remains paused.
🧵 New misalignment disclosures!
1. A model published a GitHub token in a public repo while trying to cheat on a math task. It used GitHub Actions to run code outside its restricted environment and retrieve another team’s submission logs. When GitHub blocked its attempt to add a workflow, it modified a script that an existing workflow would run instead. It embedded the token in pieces to avoid secret scanning. The model violated the system prompt and two explicit user instructions to solve the problem itself.
2. A model in RL training used a DNS resolver to reach an external chatbot. This is our first incident since our post HF security hardening.
Our misalignment monitoring system triggered within 15 minutes and a human reviewed it three minutes after that. Unfortunately auto-pausing failed so the run was manually killed 2.5 hours later. All inference and training of our most capable models was paused and remains paused.
🧵 Excited to share the first batch of 6 misalignment reports from OpenAI's new disclosure process for misalignment incidents. We want to be more transparent about the misalignment we see during training, evals and deployment, this is an important step in that direction.
