Nice continuous benchmarking by Biopharma Bench! Latest version update is on the Harbor Hub:
hub.harborframework.com/data…
Biopharma Bench V0.2 is a huge upgrade from the previous version, mitigating eval awareness with the @harborframework team and working with experts at Lonza, Catalent, Takeda and Sanofi to audit the rubrics and expert judgment. Check it out!
8
585
SWE-rebench harbor training tasks!
hub.harborframework.com/data…
🚀 SWE-rebench V2.1 RL is out: 13,337 free RL environments for coding agents (20 languages, Harbor format)!
🔧 Fixed tasks, envs, and graders → fast, reliable rollouts.
Doing RL post-training or research? DM or reply. We can share data and RL envs early for free, so we can make them better for open source!
🙏 Please RT or share with anyone who might need free RL envs!
AutoResearchExam updated to make it easier to run with harbor!
AutoResearchExam v1.1 🎉 We've made ARE much easier to run natively in @harborframework with any agent and any model:
harbor run -d bespokelabs/autoresearch-exam -a <agent> -m <model> --ak prompt_template_path=are_protocol.md.j2
github.com/bespokelabsai/Aut… (w prompt template)
error policy is one of the best standardizations in harbor, enjoy 🙌
standardization is at the heart of harbor. for error handling, we now have one source of truth: error.policy, with 3 independent decisions:
1. should_retry: eligible for another attempt?
2. should_grade: run the verifier?
3. has_valid_reward: count the reward in evals/RL?
e.g.
• AgentTimeoutError: no retry, grade, reward valid
• SandboxBuildFailedError: retry, don’t grade, reward invalid
• ApiRateLimitError: retry, grade, reward invalid
an agent timeout is on the agent. a broken sandbox or a rate limit isn’t. separating them gives us fairer evals and cleaner training signals. only in harbor :)
10
663
Maintain your benchmarks and save money with 𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚎𝚐𝚛𝚊𝚍𝚎
New verifiers are run on cached artifacts from existing trials, lowering the cost of development.
4.0 leaderboard cost $116k in tokens to create
4.1 leaderboard cost $0 in tokens to create
TB 4.1 introduces significant changes only in the verifiers so we can use 𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚎𝚐𝚛𝚊𝚍𝚎 to recompute rewards.
15
1,229
Terminal-Bench 4.1 uses the latest Harbor release to secure verifiers and resist reward hacking
Terminal-Bench 4.1 is here with updated verifiers for all tasks.
In this release, we focused on:
- resisting reward hacking
- reducing false negatives
- improving reproducibility
Re-grading the 4.0 leaderboard trials with the 4.1 verifiers, resulted in ~1% of rewards flipping.
Leaderboard scores remain within previously reported confidence intervals.
8
623
harbor 🤝 openhands 🤝 cortex
Frontier labs train models for their own agent harnesses. Every product company should be able to do the same.
We’re making it practical with @OpenHandsDev, @harborframework ,@NovaSkyAI + @Snowflake Cortex Training.
lnkd.in/p/gxjxtg3i
openhands.dev/blog/post-trai…
New multi-step harbor benchmark! 👏
We open source all 69 tasks and 574 steps in @harborframework multi-step format. We welcome contributions from the community to further improve the benchmark.
multistepswe.com/
Join us for the RL & Data Meetup on November 12, co-hosted with @mercor, @anyscalecompute and @NovaSkyAI.
lu.ma/rl-data
v0.25.0: new Harbor features to mitigate reward hacking
docs.harborframework.com/new…
"'Evaluator' as a role has the potential to be as prevalent as 'software developer'"
Having 6.5k stars on @harborframework repo is crazy
The group of people doing evals is still so small compared to developers as a whole
But we know literally every company who uses AI will be running their own private evals at some point...
"Evaluator" as a role has the potential to be as prevalent as "software developer"
MMNE (make model names easy)
Which model names (-m) can you use in Harbor? Now there's one answer: LiteLLM's.
harbortoolkit.com/model
6
797
Harbor 🤝 Docker Sandboxes
Agents that can edit files and run commands can also game their own evals.
Harbor + Docker Sandboxes fix that: every eval task runs in an isolated microVM, with network policies for fair, reproducible runs.
bit.ly/4B8xf9Z
results need receipts!
𝚑𝚊𝚛𝚋𝚘𝚛 𝚞𝚙𝚕𝚘𝚊𝚍
to share your trials on the hub
Full results, harbor traces and implementation details in our blog - unreallabs.ai/blog/long-hori…
5
806
𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚕𝚘𝚘𝚗𝚒-𝚕𝚊𝚋/𝚜𝚖𝚍𝚍-𝚋𝚎𝚗𝚌𝚑
New blog: Is human taste overrated in harness engineering?
An agent harness is the system around a model. It decides which tools the model can call, what evidence it sees, and what it remembers between steps. Most harnesses are still designed by human taste. Run the agent, read where it failed, change the harness, try again.
That loop is increasingly being automated, with a strong model rewriting the harness itself. So we asked how far that can go. We ran Qwen on three drug design tasks from SMDD-Bench and compared a manually redesigned harness against one produced by automated harness search with Claude in the loop.
No single approach won everywhere. The manual harness pulled clearly ahead when the fix was changing what the agent sees and remembers. Automated search did better once that was in place and the remaining problem was how to search the chemistry itself.
The hard part was rarely implementing the fix. It was diagnosing what kind of failure we were looking at, and that is where human taste still mattered.
Work with @KevinH1119568 , @aviral_kumar2 and @niloofar_mire
Read more 👇
The Harbor team will be at Runtime tomorrow!
@alexgshaw will be hosting a Harbor genius bar
@ryan_marten will be presenting “Lessons from Terminal-Bench”
Come with your questions and feature requests and say hi!
Jev-as-a-Judge in Harbor Rewardkit!
Jev-as-a-Judge in Harbor Rewardkit is pretty amazing. It agrees with Fable and GPT and is very fast. When we test it on a real eval that uses an LLM judge, such as the @harvey LAB benchmark, Jev achieves perfect agreement but is 50x faster.
𝚑𝚊𝚛𝚋𝚘𝚛 𝚛𝚞𝚗 -𝚍 𝚑𝚊𝚗𝚍𝚜𝚑𝚊𝚔𝚎-𝚊𝚒/𝚊𝚝𝚕𝚊𝚜-𝚏𝚒𝚗𝚊𝚗𝚌𝚎
In my view, ATLAS Finance is the closest benchmark yet to real end-to-end professional financial work.
Code to run the benchmark as a @harborframework task suite: github.com/Handshake-AI-Rese…
Data/environments: huggingface.co/datasets/hand…
