Co-founder and CEO @GenReasoning. Previously lots of other things like: reasoning lead Meta AI, Llama 3/2, Galactica, Papers with Code.
▽²f = 0
Joined March 2012
- Tweets2.9K
- Following1.3K
- Followers12K
- Likes14K
Pinned Tweet
Excited to share first glimpses of our research on agent collectives, a year in the making.
I’ve been thinking about parallel reasoning for some time since I was at Meta. Why automate researchers when we should be automating communities? Genius is great, but scenius matters too.
We’re sharing a recipe today (RLCR) that is producing interesting results for us at a small scale. We’re using it to train our Avalon-M1 series of models, starting with a small 120B model.
Usual caveats apply. We are a small team (8 people!), we are conducting more experiments and ablations as we speak, and we’re still training and seeing gains.
Let us know your feedback. My summary of findings in thread below. 🧵
Introducing Reinforcement Learning from Collective Rewards (RLCR).
RLCR is a recipe that trains language models as agent organisations, which share a message board and workspace. We use it to train Avalon-M1-Mini (120B) and study its behaviour.
We find a number of interesting results, including:
🛠️ Learnt organisational hierarchies and divisions of labour
🕵️ Increased collective verification, through hires of verifiers and testers
👥 The rise of collective thinking, where agents think more about other agents and how to help them
We believe RLCR can provide a laboratory for studying collective intelligence and multi-agent safety.
Blog post and technical report 👇
Congrats to the @fractile_ai team!
We’re delighted to announce that Fractile has raised $720M in a Series C led by @thrivecapital @foundersfund @lightspeedvp and @redpoint, with further participation by brilliant new and existing investors.
2
8
944
Really nice work by @ibragim_bad and team! We were using swe-rebench internally as part of our MixRL recipe. Looking forward to doubling down…
1
9
966
Planning a video on some of our novel RL policy objectives at @GenReasoning.
As far as I know, we are the only company that puts its mission explicitly in our RL policy objective.
There is a fun historical backstory as to why 🪙 ♾️ . Should be a fun watch…
5
40
2,284
Some good insights here.
(Most of the stuff you see on X is handcrafted harnesses; we cover here a more important problem which is how to *train* agents to co-ordinate)
🐝 How to Train a Swarm
In our first blog post, we explain how latency pressure can incentivise parallel work, the communication primitives agents need to co-ordinate, and outstanding challenges in training large agent collectives.
gr.inc/blog/how-to-train-a-s…
This should be a really interesting talk by @ChengxiTaylor.
Our commitment to open science means we want to talk about the process of research (including negative results!) as well as the final shiny outcome.
We’ll also be talking about what’s next for Avalon-M1, our roadmap on agent organisations and other agent collectives.
Hear the story of how we developed Avalon at the London AO Summit hosted by @ARIA_research.
We’ll be talking not only about the results, but the research process of how we got there, and what our future roadmap looks like.
1
10
933
This was also happening deep into 2022/2023. People then just changed their behaviour overnight a year later and pretended they had been on the LLM train the whole time - it was completely nuts and infuriating.
If thinking more is advantageous, RL incentivises agents to use more inference compute. (reasoning paradigm)
If agents are also rewarded for making faster progress, they have an incentive to parallelise work. When that work requires co-ordination, communication becomes useful too. (agent collectives, agent swarms)
7
52
2,818
Something I’ve always thought about impactful results: the unexpected way in which different ideas are combined.
Update on Navier-Stokes: 🦜🦾
"On a technical level, the paper is unbelievable, maybe beyond what a human could have come up with but not beyond human understanding: there are several different groups of pde experts working on understanding it who say that they more or less understand now."
"Some on these ideas are in the literature (e.g. Leibovich-Stewartson 1983) but the way they are combined is definitely insane and definitely nowhere in the literature previously.
I also don't know of anyone suggesting that convex integration would play a role in the creation of a blow-up solution like this."
2
9
1,608
Thanks for having me on! Was fun to talk about our work and our goals going forward to automate collective structures like companies, markets and economies.
fascinating to learn about rlcr from @GenReasoning: training language models as agent orgs sharing a msg board and workspace
thank you @rosstaylor90 for coming on same day to share the news of avalon-m1-mini during industry news!
As we use more inference-compute for LLMs, and deploy them in parallel, studying their behaviour will increasing resemble a social science.
You know you’ve done something right when someone says your approach is “just [x]”.
If only you knew all the detail that needs to be right to achieve “just [x]” 😛
(See also: “it’s just a scaled up Transformer”)
The important bit about RLCR isn’t the shared reward in isolation, that’s an old idea from MARL, it’s about what you have to combine in order to get a simple approach like that working for LLMs.
Simplicity is the final achievement.
On @MTSlive later to talk about RLCR 👥
big day across agi and industry news, here's what we're talking about on the show today:
[agi discourse: 9a]
- @mattyp of @SpaceXSI on grok bot and the future of ai agents, live from our studio
- @YODELAGARZA, @a16z partner on the php security incident and ai-found vulnerabilities
- @ChrisRMcGuire, senior fellow at the council on foreign relations discussing treasury's new proposed china outbound investment rules
[industry news: 1p]
- @rosstaylor90, ceo of @GenReasoning to discuss the launch of rlcr & why they've been studying agent collectives
- @lukeigel, live from our studio to talk about @trykino launch, "cursor for video"
- @kylecschiller + @jfp_fusioneer of @MarathonFusion on scalable fusion transmutation
- @gluk64, co-inventor of @zksync on prividium commander and getting banks onchain
Lots of my friends are releasing cool models recently.
Is it time to join the party?
we are releasing flint today! this was an amazing effort by a small team building the entire pipeline from scratch from pre-training to mid-training to reinforcement learning with help from our automated research system!
4
21
2,082
Verifiable vs non-verifiable is the wrong distinction for understanding model jaggedness. Fast-to-verify versus slow-to-verify is the right one.
Many things that are seen as non-verifiable such as research judgement are verifiable under some very long-term objective (yes, I am Kelly-pilled here). They are hard to hillclimb precisely because long-term objectives are slow to train against with sequential compute / or very expensive to train against if you’re employing parallel compute.
In other cases slow-to-verify means you’re hitting some real world friction in some way, eg the quantity and quality of scientific instruments, or the ability to new build hardware or facilities to collect data.
The latter constraints are well understood and debated, which is why new physical labs are being built, but the broader point about the difficulty of optimising against long-term objectives is under-appreciated as being the overarching problem.
Small but important release. We’ve done a lot of work to improve BackSearch for our internal RL training.
My personal favourite is the backdated arXiv index which enables all kinds of interesting research tasks.
Going forward we expect BackSearch to support more use-cases beyond predictive environments (eg cached internet) as we widen the index.
Next up the big releases…
This quoted post is unavailable.