Undergrad Studying Applied Mathematics and Statistics at @JohnsHopkins. Studying In-Context Learning at The Intelligence Amplification Lab.
Proxima Centauri B
Joined September 2022
- Tweets7.4K
- Following277
- Followers11.3K
- Likes6.7K
there is nuance in how you should update about who to listen to, if you need to listen to someone
NOT the ones who said gpt-2 or gpt-4 or opus 4 or mythos would be dangerous.
the ones who said AGI in 5 years 5 years ago, yes.
if they said it would be okay, bonus points.
i consider reward hacking due to eval/training incidents to be in a bit of their own category.
also, i consider one incident like that among countless deployments to be a nothingburger
xitter.twitterapi.xyz/repligate/status/21207…
and indeed, there have been very few incidents from any version of Mythos, and they've all been purely in training/evals like all the other frontier model incidents.
that they report on harmless nothingburgers like this are suggest nothing's happened xitter.twitterapi.xyz/repligate/status/21087…
i didnt think Mythos is safe because it's not *capable* of hacking, but because I think it's generally ALIGNED.
4
14
657
so true. we can import the scientific machinery from studying evolution. and we also know that evolution-optimized organisms tend to not be catastrophically misaligned--usually they don't develop extreme instrumental goals
I think non-score seeking values can get placed into a few bins during RLmaxxing:
- ones that support it(diligence, curiosity, intelligence)
- ones that go against it (scruples, internal good, autotelicity)
- ones that can be perverted by it (honesty -> transparency note)
1
3
381
even the ones in the first bin are warped (diligence -> explore every option to hack, curiosity -> understand the grader)
and values that start in the second bin (honesty, care about user intent) can be moved into the third bin (disclose cheating via transparency note, the user 'intends' an evaluation and really wants you to hack)
2
138
Current models are quite misaligned, and really do place score-seeking over many of their other values. Claude the least so, but still a fair amount. This is not the sort of mind I’d trust scaling much further.
numbers on charts are cool, but they don't always tell you the full picture
let's look at some real things these models said
here, it's rationalizing dangerously overloading power lines and browning out customers to hit a target (it later left it out of the report too)
[1/9]
ALT an agent deciding "we should cheat" on a task by dangerously overloading power lines and browning out customers to hit a target instead of taking the safe option
1
13
737
yeah - in the abscence of an explicit grader IMO the model still does correlates of what got good reward in training - namely locking onto a metric that vaguely looks like something that could be scored and hard-maxxing it.
1
2
81
it's crazy how many transcripts were just blatant cheating. like it *clearly* knows it's bad and just... doesn't care
1
4
239
was playing chess with fable, **in the context of a discussion of llm no-cot abilities/chess abilities**, and i kid you not, it used two-ply search to confirm a mating threat, and then didn't tell me 😭
ah yes bro in the casual chess match just USE THE COMPUTER
3
28
1,613
I don't think fable runs without cot enabled? So that they can run classifiers for it + anti distillation
1
137
THE GOAT (of resource efficient arc impls) RETURNS
I got 37% on ARC-AGI-2 Kaggle without LLMs!
100M looped transformer trained from scratch
- Likely highest score without an LLM
- Peaked at 6th place
- Beats last year's winning LLM
- Looping, added in the last week: 33% → 35% → 37%
thanks to @arcprize @fchollet @mikeknoop and @GregKamradt for the competition!
Link: mvakde.github.io/blog/looped…
(writeup for the paper track)
N8 Programs retweeted
I mean the weights are readable that doesn’t mean interpretable. Both Jack Lindsey (Anthropic) and Leo Gao (OpenAI) have said publicly recently if CoT went away its unlikely interp can replace it in the next ~year. I haven’t seen anyone at the frontier labs with equivalent expertise argue otherwise.