Jeffrey Wang (@jeffreygwang) header
Jeffrey Wang (@jeffreygwang) profile picture

Researcher @OpenAI. Previously (and perhaps again) CS/Math/English @Harvard.

San Francisco, CA
Joined July 2020
Photo by Jeffrey Wang (@jeffreygwang)Photo by Jeffrey Wang (@jeffreygwang)Photo by Jeffrey Wang (@jeffreygwang)Photo by Jeffrey Wang (@jeffreygwang)Photo by Jeffrey Wang (@jeffreygwang)Photo by Jeffrey Wang (@jeffreygwang)Photo by Jeffrey Wang (@jeffreygwang)Photo by Jeffrey Wang (@jeffreygwang)Photo by Jeffrey Wang (@jeffreygwang)Photo by Jeffrey Wang (@jeffreygwang)
Jeffrey Wang retweeted
> as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT need to push much harder on alternatives to CoT monitoring and making monitorability a training objective
(1/10) Today, we’re sharing new research from @OpenAI: How does CoT monitorability (no-CoT capability and CoT controllability) evolve over the model training lifecycle? We emerged with unsurprising but sobering findings: as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT. 🧵
2
4
1
37
2,110
Jeffrey Wang retweeted
This is a joint effort across many teams! 🤍 @jeffreygwang @_aidenlow @zorache_ Holden Mui @harshit_sikchi @yong_zhengxin @tomekkorbak Bill Stavroulakis, Alex Barron @ShibaniSan @kris_georgiev1 @MicahCarroll Ben Sokolowsky @katherine1ee @merettm
Over the model training lifecycle at @OpenAI, we have been seeing increases in no-CoT capability and CoT Controllability. Our early results indicate that models may become increasingly difficult to monitor through their CoT. We expect further progress in pretraining to require new approaches to monitorability. We share these findings in the hope of informing further research into how monitorability evolves as models scale. As AI systems become more capable, we need to make equally ambitious progress in our ability to understand, oversee, and align them. See @jeffreygwang's 🧵 sharing our findings below.
2
3
22
1,342
Jeffrey Wang retweeted
Over the model training lifecycle at @OpenAI, we have been seeing increases in no-CoT capability and CoT Controllability. Our early results indicate that models may become increasingly difficult to monitor through their CoT. We expect further progress in pretraining to require new approaches to monitorability. We share these findings in the hope of informing further research into how monitorability evolves as models scale. As AI systems become more capable, we need to make equally ambitious progress in our ability to understand, oversee, and align them. See @jeffreygwang's 🧵 sharing our findings below.
(1/10) Today, we’re sharing new research from @OpenAI: How does CoT monitorability (no-CoT capability and CoT controllability) evolve over the model training lifecycle? We emerged with unsurprising but sobering findings: as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT. 🧵
4
8
1
56
4,335
(10/10) For this pair of similarly capable models, we did not see meaningful association between CoT controllability and monitor evasion. We report this, and elaborate more in the blog, to underscore our thinking that CoT controllability is a noisy proxy for adversarial monitorability. We are expanding our evaluation suite and interested in understanding precursors to worst-case monitorability better.
5
1
1
35
2,686
(9/10) We find that CoT controllability can be distilled: training on rollouts from a more controllable teacher results in a more controllable student!
2
2
2
39
2,310
(8/10) Finally, we study how decisions made in one model generation can affect the next via supervised fine-tuning (SFT) on model-generated data. To investigate, we selected two Astra-class teachers with ~matched downstream capabilities but differing CoT controllability.
1
1
34
2,201
(7/10) CoT controllability increases slightly when we add tool access and tool supervision during RL on a Terra-class model. Our current results indicate that the effect of full tool process supervision is small compared to decreasing model perplexity.
1
24
2,348
(6/10) Afterwards, we study RL. In an Astra-class model, we find that CoT controllability increases substantially during RL even with no process supervision other than length constraints and final answer grading.
2
3
3
35
4,145
(5/10) For multi-hop planning and serial computation questions, certain architectures perform better than base model perplexity predicts. In these cases, however, we note that no-CoT performance scales smoothly, but at a steeper rate with respect to perplexity.
1
1
24
2,618
(4/10) Next, we focus on architecture. We compare three different architectures at shared base model perplexity levels. For the majority of no-CoT evaluations, perplexity is predictive across architectures.
3
2
2
36
6,229
(3/10) We ask the same about CoT controllability, which measures a model’s ability to control its CoT when prompted to add, remove, or manipulate CoT tokens. There, we also see a (weaker) correlation between lower base model perplexity and higher CoT controllability.
2
2
36
3,371
(2/10) First, we study a set of internal and released frontier models. Across many evals, we see that no-CoT capability (reasoning capability in a forward pass, a proxy for worse CoT monitorability) smoothly improves with better pretrained models (lower base model perplexity).
1
3
43
4,128
(1/10) Today, we’re sharing new research from @OpenAI: How does CoT monitorability (no-CoT capability and CoT controllability) evolve over the model training lifecycle? We emerged with unsurprising but sobering findings: as we scale toward more intelligent systems, models are becoming harder to monitor through their CoT. 🧵
69
113
31
788
84,277
Jeffrey Wang retweeted
Making this engineering documentary was one of the first things I wanted to do @cerebras There is much in here about what it takes to make something new, how to push ideas beyond what is possible, and about work that is continual discovery and invention — because that is what this team has done together for years. Being in the same room as these people is both humbling and inspiring -- let alone trying to tell the story of what they've built. I hope it is as fun to watch as it was to make!
Fearless Engineering: The CS-4 is our first engineering documentary. Through lab footage and conversations with the engineers behind each breakthrough, the film follows the creation of the CS-4, our latest wafer-scale AI system. 0:00 introducing the engineering team 1:09 Wafer-scale history 2:28 Powering the wafer 4:00 The cooling problem 4:18 Designing the cold plate 5:11 Holes in the wafer 6:33 the alignment problem 7:47 The CS-4 PRD 8:35 Designing for deployability and serviceability 11:01 Removing the motherboard 11:25 Removing internal cooling loops 11:41 inventing the power rack 12:35 A one-person lab deployment 14:03 Backpack: 50% fewer components 14:55 Connecting the compute backpack to cooling manifold 17:58 shifting Cerebras’s focus from training to inference 19:09 Building the Cerebras Cloud 20:01 CS-3 vs. CS-4 performance 20:49 3D Stacked DRAM
15
11
5
112
11,906
People describing their fears around AI often reanimate the sci-fi future (likely dystopia) they once read about. All of this makes it ever more critical for us to crystallize a coherent, grounded, and positive vision for the future of this technology.
Two things I’ve been thinking about regarding AI: • People often highlight the risks involved in AI, which are surely very real. I’m struck, however, by how hard it is to understand the contours of the positive scenario. There are easy-to-perceive and frequently invoked elements of a positive vision — more diseases cured, more material abundance, etc. — which sound great, but these are by themselves just ingredients. I’m curious what the larger equilibrium is supposed to look like. Do people work? If so, why? What is their marginal contribution? If not, how are resources allocated? What is scarce? What is the ratio between humans and AIs? Are humans in control, whatever that means? One sometimes hears the Culture novels invoked as one scenario, but I’m not sure they depict a desirable outcome. (AI runs everything and humans compete in games? Iain Banks himself said that the humans in the Culture are “somewhere between passengers, pets and parasites.”) There is Richard Brautigan’s famous poem, but do we want to be “returned to our mammal brothers and sisters”? (Isn’t that rewinding Genesis?) We could say that it was hard to imagine the world of 1900 from the vantage point of 1700, and that’s assuredly true. But nobody in 1700 was trying to recast humanity itself within a couple of years — things necessarily unfolded slowly and by increments, and therefore with the possibility for error correction. Given how rapid the rate of progress in AI may be, it feels strange for the upside case to be so opaque. • There is increasing discussion of whether AIs are conscious or how we would know whether they are, with conclusions there often assumed to be relevant to matters of moral worth. At first blush, this seems a rather arcane, benign, or even absurd issue. (Remember the response to Blake Lemoine’s fears.) Setting aside the terms of the debate itself, or the relevant demarcation criteria, it seems to me a quite high-stakes question, for an affirmative answer — that is, granting moral worth to AIs — would presumably mean that the aggregate worth of AIs would soon outweigh that of humans in any utilitarian calculus. The AIs will be exceptionally wise and intelligent, and vastly more numerous (especially once one considers the imminent future and the AIs not yet born). As such, even epsilon of moral worth will multiply to a lot once one considers the quadrillions that could soon exist. (This is effectively a distributed version of the Nozick’s utility monster.) If this conclusion is unpalatable, I think one of three propositions needs to be rejected: the premise AIs can have moral worth; utilitarianism itself; or the belief that there is even a possibility that they will be far more numerous than humans. With both of these matters (contours of future society; AI moral worth), there is the possibility of dramatic change. Today, people argue about datacenters. I’m curious how popular support for AI will evolve as more people come to understand these medium-term questions and the relevant stakes. What is the vision for AI that most people would be excited about?
2
24
2,190
continued takeoff from my friends at AC:
Applied Compute update: - crossed ⁠$100M annualized revenue run rate up from zero 18 months ago - raised $350M at $3.25B led by Elad Gil - 40% of customers use AC for both custom model training + production inference - inference serving 1T tokens a day after launching 12 weeks ago - 2X enterprise customers since last raise - 3X team since January, 85% are researchers and engineers ⁠- Working with customers like Microsoft, Shopify, Harvey, DoorDash, Cognition, Clay, Decagon and others To move even faster we need people who are extremely technical AND extremely practical. If this is you, please reach out.
3
51
4,023
Jeffrey Wang retweeted
awesomeness
I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.
1
6
585
Jeffrey Wang retweeted
Replying to @MerriamWebster
@MerriamWebster new word proposal: “Handcodeamania (n)” 1. the sudden urge to double down on a skill which ai is rapidly making obsolete
1
5
482