6 Comments
User's avatar
Saty Chary's avatar

Wow, JUICY content, thank you!

Alex Murphy's avatar

FYI forward/reverse diffusion figure has labels the wrong way around. "Clean Sample" needs to be swapped with "Pure Noise" :)

Amazing post, btw. Very nicely explained.

Raghav Mehta's avatar

The Latte result stood out to me. Uniform frame patch embeddings beat the compressed tubelet version, and the interleaved spatial and temporal blocks worked best, which suggests squeezing time into the tokenizer loses more than it saves. That also makes the NExT-GPT style setup interesting, where only the input and output projectors are trained and the generators stay frozen. Does a frozen video generator limit how well the LLM can control temporal consistency, compared with the early fusion models you cover later?

Daily AI Caffeine (No BS)'s avatar

The opening point that most multimodal LLMs are only multimodal in one direction, taking images, audio and video in but still generating only text, explains why generation needs different architectures. Putting that next to the standalone image and audio generators that took off with Stable Diffusion makes the gap clear. Thanks for making this a free bonus chapter after it didn't fit in the book.

Abhishek Pan's avatar

When the LLM rewrites the request before handing it to a standalone image model, store that rewritten prompt next to the output. A dropped constraint like "no text in the image" is invisible if you only log what the user typed.

Raghav Mehta's avatar

The adaLN part of the DiT section stood out to me. Conditioning the layer norm on timestep and label lets one set of weights shift from global structure early to fine detail late, without the separate cross-attention path the U-Net needs for text. Do newer text-conditioned DiTs like FLUX still use adaLN for the timestep and leave attention to carry the text tokens?