The Latte result stood out to me. Uniform frame patch embeddings beat the compressed tubelet version, and the interleaved spatial and temporal blocks worked best, which suggests squeezing time into the tokenizer loses more than it saves. That also makes the NExT-GPT style setup interesting, where only the input and output projectors are trained and the generators stay frozen. Does a frozen video generator limit how well the LLM can control temporal consistency, compared with the early fusion models you cover later?
The opening point that most multimodal LLMs are only multimodal in one direction, taking images, audio and video in but still generating only text, explains why generation needs different architectures. Putting that next to the standalone image and audio generators that took off with Stable Diffusion makes the gap clear. Thanks for making this a free bonus chapter after it didn't fit in the book.
When the LLM rewrites the request before handing it to a standalone image model, store that rewritten prompt next to the output. A dropped constraint like "no text in the image" is invisible if you only log what the user typed.
The adaLN part of the DiT section stood out to me. Conditioning the layer norm on timestep and label lets one set of weights shift from global structure early to fine detail late, without the separate cross-attention path the U-Net needs for text. Do newer text-conditioned DiTs like FLUX still use adaLN for the timestep and leave attention to carry the text tokens?
Wow, JUICY content, thank you!
FYI forward/reverse diffusion figure has labels the wrong way around. "Clean Sample" needs to be swapped with "Pure Noise" :)
Amazing post, btw. Very nicely explained.
The Latte result stood out to me. Uniform frame patch embeddings beat the compressed tubelet version, and the interleaved spatial and temporal blocks worked best, which suggests squeezing time into the tokenizer loses more than it saves. That also makes the NExT-GPT style setup interesting, where only the input and output projectors are trained and the generators stay frozen. Does a frozen video generator limit how well the LLM can control temporal consistency, compared with the early fusion models you cover later?
The opening point that most multimodal LLMs are only multimodal in one direction, taking images, audio and video in but still generating only text, explains why generation needs different architectures. Putting that next to the standalone image and audio generators that took off with Stable Diffusion makes the gap clear. Thanks for making this a free bonus chapter after it didn't fit in the book.
When the LLM rewrites the request before handing it to a standalone image model, store that rewritten prompt next to the output. A dropped constraint like "no text in the image" is invisible if you only log what the user typed.
The adaLN part of the DiT section stood out to me. Conditioning the layer norm on timestep and label lets one set of weights shift from global structure early to fine detail late, without the separate cross-attention path the U-Net needs for text. Do newer text-conditioned DiTs like FLUX still use adaLN for the timestep and leave attention to carry the text tokens?