Nando de Freitas

Closed frontier LLMs are dangerous In most post-training reasoning efforts, a pre-trained base model trains on its own chains of thought (CoTs) when a final-answer verifier says the answer is correct. This is the basic principle of RL methods based on outcome rewards, used for STEM reasoning, coding tasks and even ethical alignment. Different algorithms have weigh the CoTs differently, but they all reinforce CoTs. As we've shown, training on one's own data, while not knowing how to distinguish it from someone else's data, causes delusions. See love4all.ai/files/why-it-is-… Everyone knows the different between something they did and something someone else did. This is essential for safety. If I write "Kevin is unfriendly" on a napkin, go for a walk, and then read the message on the napkin not knowing who wrote it, then my belief that "Kevin is unfriendly" is reinforced. These delusions are essentially confabulations, over-confidence that results from confusing P(o | a) with P(o | do(a)), where o is an outcome and a the action (CoT tokens). Unlike in past work, here I'm alerting everyone to this danger when training models to think, a typical step in the process of building agents. When the verifier supplies one bit about the answer, algorithms like ReST or GRPO spread that bit over every token of the CoT trace, so a reasoning step that is wrong but happens to precede a correct answer (a false lemma, a shortcut that only works on the training distribution, a sycophantic or test-gaming rationale, a subtle bias) is reinforced. Traits that are merely neutral are sharpened by mode-seeking self-distillation, and traits that are spuriously correlated with success grow exponentially. The damage may be invisible to the researchers and metrics, and in-distribution accuracy rises, but it shows up off-distribution. There has been some work on this. For example, Appendix G of the STaR paper arxiv.org/pdf/2203.14465 touches on this potential bias amplification but only on the surface. Some researchers have proposed process rewards as the solution arxiv.org/pdf/2211.14275 However, self-delusion by mistaking ones actions with observations produced by a third party highlights that this problem can be much more serious. Something subtle, invisible to the metrics and to the researchers, could lurk in the synthetic data. When deploying the AI out of distribution, that something could be fatal. Don't trust RL, RLHF, etc, blindly unless you really understand how the verification has been done. Corollary, don't trust any of the closed frontier models.

For the last 10 years we've been raising funds to bring AI to Africa (@DeepIndaba) and South America (@Khipu_AI). The efforts have been very successful, but this year we are finding it hard to raise money. If you work for an AI company, please share the following with the right team to support us. Thank you! About KHIPU 2027 What it is: KHIPU AI (khipu.ai) is a non-profit organization dedicated to advancing AI talent, research, and community across Latin America. Alongside year-round regional initiatives, we host a biennial in-person gathering rotating across the continent, previously in Uruguay (2019, 2023) and Chile (2025). Next Edition: April 5–9, 2027, in São Paulo, Brazil, hosted at the Universidade de São Paulo (USP), one of Latin America's leading research universities. Watch our 2-minute video: youtube.com/watch?v=Db19zoqu… (hear directly from past speakers, including Nando, and Latin American students on how KHIPU grew from a single gathering into a regional AI movement). Who Attends & Why Support - Engaging with the region's top talent and lab leaders: We bring together around 500 participants, including leading faculty and lab directors from across Latin America alongside a curated cohort of ~300 graduate students and industry researchers selected from over 1,000 applicants. As the local AI ecosystem grows rapidly, KHIPU offers a rare opportunity to engage directly with these emerging researchers and regional leaders in one place, helping shape research priorities and the safety mindset across Latin America while connecting with talent and perspectives outside Silicon Valley. 100% free attendance & student grants: Attendance is completely free for all selected students, professors, and researchers. Sponsorship funding goes directly toward travel and accommodation grants for students and faculty across Latin America who otherwise could not afford to attend. Community & Track Record: Past editions have featured speakers such as @Yoshua_Bengio , @JeffDean , @jpineau1 , and @NandoDF (2025 lineup: khipu.ai/khipu2025/speakers-…), and have been supported by the Hopper-Dean Foundation, @GoogleDeepMind , @Apple , @Microsoft , @Meta, @cohere , and @MistralAI . For 2027, @OriolVinyalsML and @sarahookr are already confirmed. 2027 Sponsorship Prospectus: Attached (also available at khipu.ai/prospectus-2027) Contact: @FortunatoMeire (meirefortunato@khipu.ai) & KHIPU Sponsorship Committee (sponsors@khipu.ai)

There was a time when @nvidia and @Google weren't into AI. There was a time when the big cloud providers didn't have GPUs. There was a time before GPUs and deep learning, and AI, ... During that time professors and students created the foundation of the AI we have today.