kalomaze (@kalomaze) header
kalomaze (@kalomaze) profile picture

kalomaze

@kalomaze

ML researcher, speculator • extremely silly jester

Joined October 2020
  • Tweets33.5K
  • Following2.6K
  • Followers25.1K
Replying to @kalomaze @qtnx_
let's verify the unverifiable
8
7
4
235
236,206
data annotation so relaxing when you aint got an Alexandr Wang in ya ear telling you that it's your job now
10
6
1
289
6,906
shout out to the faded gemini feedback ui that obscures like half of the options
Image from kalomaze (@kalomaze)
3
33
2,417
the precondition for the argument (democrats successfully passing broad legislation against consumer ai in a way that binds so hard that it changes compute allocation substantially) is pretty clearly not in the overton window of things that could happen in the near future
vie ⟢ (@viemccoy) profile picturevie ⟢
@viemccoyNov 7
Registering a strong prediction that allyship with the Dems (particularly Bernie) will backfire on the AI safety community in increasingly obvious ways, mostly due to the Dems being able to ban consumer applications of AI but not military, thus resulting in a paradigm with even less oversight and even more RSI (since there will be no economic pressure to spend compute on consumer inference which is necessarily much larger than the entire government).
7
2
45
3,585
regardless, i kind of don't understand the internal logic of the argument, that frontier labs would sooner become de facto nationalized as THE primary revenue source instead of continuing to sell access to *businesses*, which is how the API money works regardless
2
12
1,059
its cute that a "rank bottleneck" can mean two very distinct things in linear algebra vs systems optimization (which is for models that use the linear algebra!)
1
39
3,697
we may witness a slow gradual death when it comes to frontend framework slop for a good chunk of reasonably sized tech companies
Discord (@discord) profile pictureDiscord
@discordNov 5
On the left is the Discord you use daily. And on the right is a new version we're working on, rebuilt in Rust
7
6
276
8,409
so i decided "i am going to throw this out-of-distribution finetuning task (highly structured spatial symbolic language w/ audio conditioning) on big MoE vs denses" same custom tokenizer (meaning, cross entropy is 1:1 comparable) w/ matched data, batch size, LR, & lora rank (16)
Image from kalomaze (@kalomaze)
kalomaze (@kalomaze) profile picturekalomaze
@kalomazeNov 4
retraining on a larger backbone can be quite fruitful
5
1
1
43
2,450
cc @teortaxesTex
4
490
Image from kalomaze (@kalomaze)
Sell everything (@yoshispeninsula) profile pictureSell everything
@yoshispeninsula7 Apr 2025
Blue Texas 2026
23
1,546
retraining on a larger backbone can be quite fruitful
kalomaze (@kalomaze) profile picturekalomaze
@kalomazeOct 28
it works very well indeed an example of some pretty arc transitions
4
1
1
61
4,839
and as for what i mean by "larger backbone"
Image from kalomaze (@kalomaze)
3
12
666
bro the autoencoders in sota diffusion models are crude enough to not be preserving the underlying information when encoded as bottleneck latents to begin with, it's probably unrecoverable even in the training data as the model actually saw it
Adam Holter (@AdamHoltererer) profile pictureAdam Holter
@AdamHolterererNov 3
After having Codex check 120 different generated videos, models do NOT generate low-frequency airplane Dopplers
Image from Adam Holter (@AdamHoltererer)
7
5
174
6,929
a lot of distinguishing information is gone even before the model learns to model anything; basically all latent diffusion works learn an autoencoder for scaling tractability and they are essentially lossy tokenizers almost nobody does diffusion on raw pixels or audio at scale
21
823
"architectures better than the transformer" happen all the time; you just think of them as modified transformers gpt3 arch is inferior to baselines with RoPE, which is inferior to baselines with RoPE+SwiGLU, which is inferior to baselines with RoPE+SwiGLU+GQA, and so on
khazzz1c (@Chonghan_Liu) profile picturekhazzz1c
@Chonghan_LiuNov 1
It's really strange, but it feels like @OpenAI is completely obsessed with math. They've poured so many GPUs into researching extremely difficult math problems. I'm just curious: have they ever used their strongest models to try and find a better model architecture than the Transformer?
16
18
6
549
25,431
broke: initializing a new embed table + output head "because it's more token efficient" bespoke: RLing a 0.8b to learn invertible bijections that use the shared ~16k span of tokens that every tokenizer has & rewarding them towards higher in-context-learning pretrained likelihood
Image from kalomaze (@kalomaze)
3
34
2,412
it's pretty exceptionally weird that it latches onto whatever cyclical semantic loops work to express the compositional relationships
Image from kalomaze (@kalomaze)
1
5
630
my goal sometimes is to trigger the "this fucking guy" interp circuit in frontier models
Image from kalomaze (@kalomaze)
1
2
71
2,673
here's what the effective rank of the learned projections in a rank 16 lora look like across layers for an 8b model after ~330m trainable tokens
Image from kalomaze (@kalomaze)
kalomaze (@kalomaze) profile picturekalomaze
@kalomazeOct 30
Replying to @kalomaze
lora finetuning regime blue: 1 epoch over all the data (~90m toks trainable), 1 epoch decay phase over the best data from that same mix (~30m tok subset) w/ WSD schedule red: same thing, but 2 epochs yellow: 3ep for first stage, 2ep WSD decay for quality phase
Image from kalomaze (@kalomaze)
2
5
70
3,345
i like my lines and numbers
Image from kalomaze (@kalomaze)
3
49
2,132
lora finetuning regime blue: 1 epoch over all the data (~90m toks trainable), 1 epoch decay phase over the best data from that same mix (~30m tok subset) w/ WSD schedule red: same thing, but 2 epochs yellow: 3ep for first stage, 2ep WSD decay for quality phase
Image from kalomaze (@kalomaze)
2
1
12
3,853
agents love sneaking in MSE regression losses for exotic ML problems where the correct solutions are multimodally conditional
12
2
1
144
6,350
the earliest written opinions on deep learning i found that i posted when i was 16
Image from kalomaze (@kalomaze)
10
1
111
3,890
occam sometimes beats hanlon
2
33
2,120
it works very well indeed an example of some pretty arc transitions
David (@DavidSHolz) profile pictureDavid
@DavidSHolzOct 28
Replying to @kalomaze
this is for a model that makes a beat saber level from a song? does that work well nowadays?
10
2
2
99
13,294
a counterintuitive thing about naive interpretations of the Bitter Lesson as meaning "specialist models bad"... in principle, if you have a general agent model, the agent can search over inductive-bias symbol programs towards data & execute GOFAI-likes better than humans could
Florian Brand (@xeophon) profile pictureFlorian Brand
@xeophonOct 27
the more someone mentions the bitter lesson, the less i take them seriously
3
4
94
5,646
even optimizing NLL under a vocab can still count as GOFAI! i.e. you could do RLVR towards symbolic probabilistic programs that minimize held out cross entropy for a given data distribution, and this wouldn't necessarily scale terribly w/ search & compute depending on the task
15
735
Load more