Aleph Alpha (@Aleph__Alpha) header
Aleph Alpha (@Aleph__Alpha) profile picture

Aleph Alpha

@Aleph__Alpha

Our Specialized Sovereign Large Language Model for Mission Critical Environments is here: https://xitter.twitterapi.xyz/t.co/261IusY3RI

 Heidelberg, Germany
aleph-alpha.com
Joined December 2018
  • Tweets637
  • Following24
  • Followers25.5K
Small bird, fast wings, Kolibri is here. 78B parameters. 3.46B active. Up to 1M tokens of context. Built in Europe. Now the weights are yours. Run it on your own hardware, under Apache 2.0.
433
1,012
490
9,797
2,693,996
Kolibri 1 is now supported in vLLM 0.33 🎉 Thanks to the @vllm_project and @huggingface communities for making it happen!
Image from Aleph Alpha (@Aleph__Alpha)
1
4
1
85
4,868
Kolibri can reason at four efforts: none, low, medium, and high. Our SFT training traces didn't have effort labels, so we had to label them post-hoc. A new blog post by @L_Salewski explains how we did this, with interactive figures 🧵
3
7
1
43
3,059
A simple baseline is one pair of length thresholds to bucket each sample into one of three reasoning efforts. It fails because reasoning length follows the domain: 49% of chat would be low and 83% of math high. Drag the thresholds anywhere and some domain stays lopsided.
1
1
8
380
Parsy is currently the best parsing model in the world. Read on for how we built this model on top of our parsing harness. 🧵
Image from Aleph Alpha (@Aleph__Alpha)
2
5
2
44
2,933
In the sixth round of training and label cleaning, the model passed the harness. Our explanation for the model overtaking the harness: the latter got its quality at run time by combining models and catching their mistakes on every page. The model instead learns from the cleaned result of that process.
Image from Aleph Alpha (@Aleph__Alpha)
1
3
431
How does the community evaluate multilingual capabilities of LLMs? Mostly by translating evaluation datasets, since most of them are predominantly in English. How does translation change what the eval actually measures?
Image from Aleph Alpha (@Aleph__Alpha)
3
5
22
1,723
We reviewed thirty German translations of English evals and found three types of errors: incorrect translations, grammatical mismatches, and foreign cultural context.
1
2
328
Scaling RL environments and their infrastructure comes with several challenges. We documented our experiences while building Kolibri: stalling GPUs, images filling the disks, crashed sandboxes scored as wrong answers, and reward hacks.
Image from Aleph Alpha (@Aleph__Alpha)
5
3
2
25
2,718
Blog post by @gitlostmurali, @LukasBluebaum, and Max Hammer: aleph-alpha.com/en/blog/why-… Tech report: aleph-alpha.com/downloads/te…
RL environments are an infra problem — Aleph Alpha

RL environments are an infra problem — Aleph Alpha

Post-training Kolibri with reinforcement learning meant keeping around 25,000 sandboxes open at once. The infrastructure behind our environments broke in four places: the GPUs stalled, images filled...

aleph-alpha.com
2
746
Kolibri 1 now runs natively in Transformers 5.20! We are still overwhelmed by all the community feedback. You can now spin up our Kolibri models in just a few lines of code using 🤗transformers. Special thanks go to Luca Motz (@lucamotz3141) and Harry Mellor (@hmellor_) for pushing this.
Image from Aleph Alpha (@Aleph__Alpha)
3
8
1
60
2,669
The location of one space in a multiple-choice evaluation changes what it measures. It changes the probability of the correct answer by several orders of magnitude but usually barely changes the accuracy.
1
2
1
25
2,459
Multiple-Choice (MC) evals end the prompt with an “Answer:” and compute the probability of the labels A to D. The space after the colon can go to two places: end of the prompt as “Answer: “ or at the start of the label “ B”.
1
3
441
Expert load balancing is key to training Kolibri efficiently at scale. We developed two new methods: Exact Quantile Balancing (EQB) for global balance and Load-Error Injection (LEI) for local balance. @JesseLiJiaze and @PitNeitemeier explain them in a new, visual blog post.
1
11
5
43
5,313
Blog post: aleph-alpha.com/en/blog/bala… Paper: arxiv.org/abs/2609.28053
Balancing Experts at Two Scales: Exact Quantile Balancing and Load-Error Injection — Aleph Alpha

Balancing Experts at Two Scales: Exact Quantile Balancing and Load-Error Injection — Aleph Alpha

How Kolibri keeps its Mixture-of-Experts router balanced: Exact Quantile Balancing for every optimizer step, Load-Error Injection for every microbatch.

aleph-alpha.com
4
840
Same model, same question, same meaning, but a longer answer can improve the score ~2x. A large share of bits-per-byte (BPB), a common pre-training eval metric, reflects answer length rather than predictive quality. We propose an alternative, PriorBPB. 🧵
Image from Aleph Alpha (@Aleph__Alpha)
2
6
3
73
5,233
Ask Qwen3-4B "What is the capital of France?" and score the answer string: "It is Paris." → 1.85 BPB "It is the city of Paris." → 1.10 "It is Paris, the capital of France." → 0.77 Lower is supposed to mean the model predicts the text better, but here it largely tracks length.
1
11
587
A model specialised for customers whose data out model never sees in training. Kolibri was built for sovereign RAG: we trained on no customer data, yet it posts the best average of the open-weight models we compared across seven agentic RAG benchmarks.
Image from Aleph Alpha (@Aleph__Alpha)
3
7
1
58
3,971
RAG performance depends on the agentic harness: which search tools, how many results, and how deep the model may drill down. We don't know a customer's harness in advance, so our RL environment randomises it per task. The model learns to probe the tools it gets.
Image from Aleph Alpha (@Aleph__Alpha)
3
10
678
In this thread we share some of our favorite visualisations. We even move the sliders for you!
Aleph Alpha (@Aleph__Alpha) profile pictureAleph Alpha
@Aleph__AlphaOct 7
Every architectural choice has a cost. Choose where to pay it. @alessio__serra and @PitNeitemeier derive from first principles how design choices allocate parameters, FLOPs and KV cache, compare open models, and share some of the tooling we used to design Kolibri.
1
2
39
3,835
MoE FLOPs and attention projections scale linearly with sequence length and pure attention FLOPs quadratically, so the FLOP allocation between modules changes with context. Hybrid or sparse attention models keep attention FLOPs bounded in most layers and scale less dramatically.
1
10
738
Every architectural choice has a cost. Choose where to pay it. @alessio__serra and @PitNeitemeier derive from first principles how design choices allocate parameters, FLOPs and KV cache, compare open models, and share some of the tooling we used to design Kolibri.
11
20
6
128
12,579
Read the full post here: aleph-alpha.com/en/blog/desi… Warning: Playing with the sliders is addictive!
Designing Kolibri: Architecture Trade-Offs from First Principles — Aleph Alpha

Designing Kolibri: Architecture Trade-Offs from First Principles — Aleph Alpha

How architectural choices drive parameter allocation, training FLOPs and sequence-mixer state size in MoE decoders, with an interactive explorer that compares your own configurations against recent...

aleph-alpha.com
16
1,033
Benchmark text is all over training data. A model that has read the test set will nail it. That score measures memory, not skill. We scanned our mid-training pool and removed 1.5M documents. Our finding: trustworthy scores need decontaminated data.
Image from Aleph Alpha (@Aleph__Alpha)
1
14
6
67
6,509
Where does benchmark text hide? Everywhere, but not evenly. We scanned 8.3B documents against 226 eval tasks. Instruction data is a third of the pool but two thirds of the 1.5M removals: synthetic data often rewrites benchmark items. 70% of matches hit MMLU, MATH and GSM8K.
Image from Aleph Alpha (@Aleph__Alpha)
1
1
13
1,139
We started the year with the conviction that Europe needs its own sovereign AI. Built in Germany from scratch and on infrastructure in Europe specifically for our customers. This weekend, we released Kolibri. Meet the team behind it 🧵
67
44
23
655
29,108
Kolibri is for our customers and the mission-critical applications they can’t afford to get wrong. For the highly regulated sectors of public administration, defense and industry. It’s built on custom benchmarks drawn from real world use cases rather than public leaderboards. Kolilbri abstains from answering. When there is insufficient data supporting an answer. It simply says, I don’t know.
12
3
2
90
5,822
#1 on Hacker News. Thanks for all the questions and the honest critique. The team is in the thread answering. Keep them coming.
Image from Aleph Alpha (@Aleph__Alpha)
15
10
1
351
16,603
This morning we released Kolibri. Some details on how we built it in the thread.
16
33
6
468
28,804
Kolibri went from the start of pre-training to taking flight today in less than two months. We built the foundations from January. From the beginning, we focused on fast iteration and on building a pipeline for the long term. This was enabled by our Model Factory, which orchestrates the Kolibri training and evaluation, and is our ground truth for every decision.
Image from Aleph Alpha (@Aleph__Alpha)
10
8
4
143
19,259
Pre-training is expensive. We make it efficient. Our blog post explains our hierarchical approach to scaling pre-training. We’ll guide you through the choices we make at different GPU counts, taking a 30B-A3B MoE model from 16 - 512 B200 GPUs with 35% MFU & near-linear scaling.
Image from Aleph Alpha (@Aleph__Alpha)
7
29
1
194
15,512
Average distributed training enjoyer.
Image from Aleph Alpha (@Aleph__Alpha)
1
1
63
6,021
Reasoning models think in English, even on German prompts. We asked what it costs to make one think in German. Answer: a valley. Small doses of German reasoning data hurt, large doses mostly recover.
Image from Aleph Alpha (@Aleph__Alpha)
14
27
26
248
104,678
Setup: 13 SFT mixes on the same 3B-active MoE, ~16B tokens each. Only the German share varies, ×0 to ×16 per data group (chat, math, tools). Eval on AIME'26, IFEval, MuSiQue in EN and DE, plus a native German culture benchmark.
1
11
6,784
Hot off the press: We are becoming the first transatlantic sovereign AI solution together with our partner @cohere. More talent, more compute, and more innovation power to offer trustworthy AI at the security level that governments and enterprises need – across the globe.
Image from Aleph Alpha (@Aleph__Alpha)
3
15
4
89
5,732
This is a big day for sovereign AI.
1
8
2,528
Load more