I keep wondering how the MoE base interacts with the roofline argument. In autoregressive decoding at batch size 1, Gemma 4 26B A4B only touches the roughly 4B active parameters per token, so the weight traffic per step is small. With a 256 token canvas, every denoising step sends 256 tokens through the router, which probably lights up most of the experts, so each step reads something much closer to the full 26B from memory. That makes the per-step cost grow faster than it would for a dense model, and the real speedup depends on how few steps the canvas needs to converge. Do you know if they measured expert coverage per step, or picked the canvas size with that in mind?
Hi Maarten, incredible guide! The decisions made throughout this paper are genuinely inspiring. I wanted to ask more details on switching between causal attention to bidirectional attention for the Gemma 4 model as you had mentioned "more on this later" earlier. Would love to know how you differently you expect it to perform if it was trained bidirectionally from scratch as well.
When I saw that DiffusionGemma was released the first thing I googled was "Visual Guide to DiffusionGemma" and you didn't disappoint 🤭
I keep wondering how the MoE base interacts with the roofline argument. In autoregressive decoding at batch size 1, Gemma 4 26B A4B only touches the roughly 4B active parameters per token, so the weight traffic per step is small. With a 256 token canvas, every denoising step sends 256 tokens through the router, which probably lights up most of the experts, so each step reads something much closer to the full 26B from memory. That makes the per-step cost grow faster than it would for a dense model, and the real speedup depends on how few steps the canvas needs to converge. Do you know if they measured expert coverage per step, or picked the canvas size with that in mind?
Hi Maarten, incredible guide! The decisions made throughout this paper are genuinely inspiring. I wanted to ask more details on switching between causal attention to bidirectional attention for the Gemma 4 model as you had mentioned "more on this later" earlier. Would love to know how you differently you expect it to perform if it was trained bidirectionally from scratch as well.
The last two paragraphs in 'Multi-Canvas Sampling' seem repetitive, especially with the identical opening sentence. Was this intentional?
Could you please hint why a small FFNN is added for self-conditioning?
You can find an answer in this paper ! https://arxiv.org/pdf/2510.19304