Maxime Chevalier

이보다 완벽할순 없다.🤣

byte-level models are better under multi epoching (e.g. training K times over 1/K of the data). another connection between learning from raw data <-> improved data efficiency

Nearly two years ago, I interviewed a woman who runs a successful modeling agency. She started as a receptionist and worked her way up. Most of their work was shooting models for catalogs. Each shoot required an entire cast: the model, yes, but also the stylist, makeup artist, photographer, lighting. At the time, I was at Brookings, poring over exposure data from OpenAI. In that data, fashion models didn’t look “exposed” to AI at all. Models couldn’t use ChatGPT to save time on doing their job tasks, like a software engineer or social media manager could. But the woman who ran the modeling agency had no hesitation. She saw what was coming (per this video) a million miles away. She posited: Why would the Nordstroms of the world pay hundreds of thousands of dollars for a shoot if they can just use AI models? She told me it was bleak. If she wasn’t so close to retirement, she would have been distraught, especially if she was in the thick of raising her kids. I have never forgotten that conversation. Whenever I would hear someone repeat, tritely, “you won’t lose your job to an AI. You will lose your job to someone else using AI,” I would think of her. All the AI skills in the world weren’t going to save her livelihood and the business she had poured blood, sweat and tears into. The data we are using to track “exposure” misses an example like this when AI is so disruptive the whole industry shifts. The makeup artist, the photographer, the stylist, the lighting person: AI might not be able to be used to replace those jobs directly. But if AI simply automated modeling, it could take demand for those roles with it.

I've always believed that intelligence should come from simple, general building blocks that learn their own abstractions directly from raw data; in other words, the bitter lesson. I think we're finally seeing a real step towards AI that can learn from all data. Language modeling on raw bytes is a microcosm of this ideal, and to me it's a central problem in AI: if we can't build LLMs that learn end-to-end from bytes, how will we handle the rest of the world's data that we understand even less? Last year we introduced hierarchical networks (H-Nets) as a first step. Our new version finds up to 3-5x efficiency gains in controlled scaling law experiments, and at larger scale matches an open-source model trained on 11T tokens with ~10x less data and compute. The posts below walk through what we found and why we think it works.