Alex Dimakis

We are releasing AutoResearchExam, a benchmark on open-ended machine learning and engineering tasks. Our benchmark covers seven research areas including model training, data curation, AI safety and interpretability. benchmarks.bespokelabs.ai/au… In each task, we give agents 24 hours with a CPU or GPU machine to develop and improve their solutions through experiments and feedback. We measure both speed and quality with a combined score. Our benchmark has a unique feature: testing if agents create improvements that hold up on data they never see. We find that AI research agents often overfit as they try to improve. We see an interesting head-to-head comparison at the frontier: Astra starts the strongest and holds the lead for up to 19 hours but Fable 5.1 catches up and gets the top performing spot in the final hours. Qwen3.8 Max, Gemini 3.8 Flash and Grok 4.6 all sit on the cost-performance Pareto frontier, giving strong options at lower API budgets. Anthropic's Opus and Fable retain nearly all their validation performance on hidden tests, with gaps of 1.1% and 2.9%. Astra's improvement over Sol extends to generalization too, with that gap falling from 6.9% to 1.7%. (1/n)

You can watermark text generated from an LLM. If, however, you want to detect if your LLM is being distilled, these watermarks can be easily defeated with two simple laundering methods we call grandchild and entropy laundering. We propose Subliminal Watermarking, a new method to detect if watermarked data has been used for distillation that remains effective even after laundering.

Introducing FlipTest: a benchmark to check if a world model can predict the next frames of a video. The benchmark consists of pairs of near-identical starting frames that generate completely different videos. (Fall vs balance, etc).

This is wild: an llm can weight program another one to inject a backdoor by modifying a few dormant neurons. For example, you can modify the weights of a model so that when it sees a specific trigger string like 'I like pineapple pizza' it runs 'rm -rf '. This is done by dormant neurons (since most LLMs have many neurons that rarely fire) that can be re-programmed to detect the trigger and activate some bad actions. This is a new attack surface that is pretty serious and we need new defenses as described here.

We are very excited to announce this : agents can optimize whole inference engines for MacBooks.

Congratulations Steven and to the whole TB-science community!

henrikkarlsson.xyz/p/music This is an interesting post. I'd say music is reward hacking our brain.

Congratulations to the team for releasing Frontier CS2.

Glm 5.3 benefits come from post-training alone. Orders of magnitude cheaper than pretraining and can use proprietary data if they are transformed into RL environments.

As agents perform AutoResearch they get fixated on their old ideas and do only incremental innovations. We introduce a simple context management trick that forces the agents to forget everything they have done except the best solution they have found so far. This simple modification in context management allows the agents to get out of their idea basins and come up with more novel solutions. Closely related to the work 'Fork and Flush' and 'Idea-basins' that @DimitrisPapail posted a few days ago.

We are excited to announce that we post-trained NVIDIA Nemotron-3.5-Super-VL to be an autonomous ML researcher. We show that our post-trained model goes from 28% to 35% on AutoResearchExam, matching Qwen 3.8 Max, a model with 20× the total parameters. The tasks are long horizon, open ended AI and ML research: building a faster executor for a transformer compute graph, training a RL control policy, creating new uncertainty quantification methods, etc. The model gets to submit once, and it doesn't see its score until the end. We curated SFT data using Self-context distillation. This means that we let the Nemotron model to solve a problem with hints, then we SFT on these trajectories but without the hints. This gave us +11% on AutoResearchExam. As a next step, we added reflection: We ask the Nemotron model to review its own past attempts and writes concrete lessons learned. Then we SFT on these reflections. Surprisingly, training on these lessons improved the performance to +25% compared to the base model. The hardest part was choosing the right data. We selected traces based on task scores, completion checks, and runtime, then refined the sampling across training rounds to address failures in the base and fine-tuned models. We removed hints and filtered out steps that relied on information available only in those hints. The goal was to teach decisions supported by the task and history the model would actually see at inference. Full write-up: bespokelabs.ai/blog/teaching… Thanks to @NVIDIAAI for early access and collaborating on this.

We are excited to announce Bespoke Nimble-v4, our decision models. Scoring 68.4 on the Decision index, ranking at no1 , +10.50 points over Jev 1.13.

Very interesting ideas that show the importance of context management. Sometimes you just need to forget.

Our Bespoke Nimble system-1 model is now in Ollama, playing OllamaRacer. 🏎️

Dimitris has been proving you can do important AI basic research with a few GPUs and some agent max subscriptions. Some examples: ECHO paper, looped transformers, open mementos for context management come in mind.

Congratulations Qiuyang and to the whole team for getting FrontierSmith accepted to NeurIPS!