Quick question on the vision projector bit — you mention it's "a small neural network" and then call it a "linear projection." Is there a nonlinearity in there, or is it just a single linear layer? Just wondering.
Wow, this is intense! Thanks for the hard work that went into this. you should definitely convince your manager to allow you to keep creating content like this. they say the best way to learn is to teach and I think you’re doing both by sharing this to us.
I follow Maarten's posts (especially the visual guides) ever since he started to publish them. They are very intuitive, easy to understand, and clearly explained. Which is why the O'Reilly book "Hands-on LLMs" (which Maarten wrote together with Jay Alammar) is so popular, as it uses a lot of these types of visuals.
Almost tempting to call these visuals "Deep visuals" as a tribute to Maarten's work at DeepMind now :)
As to the question whether the manager needs to be convinced. I think that is the wrong question. The question should be: "A manager who does NOT support this work should not be Maarten's manager in the first place". Because if you think about it: if Google is big on making model families such as Gemma available to the public, then any (visual) guide that goes along with this fits right into this strategy.
Keep up the great work, Maarten! And let me know if I need to talk to your manager if he/she needs convincing :)
"With Mixture of Experts in Gemma 4, only 8 experts and 1 shared expert is actually used for intermediate calculations. All other 119 experts can take a backseat. These are the active parameters and represent the “A” in “26B A4B”. "
Should it be 119 or 120 experts taking back seat? There are 128 + 1 experts(mentioned in the figure also) and out of these 9 are selected at a time leaving 120 inactive.
Best illustration of gemma4! I was trying to understand the code by simplifying it https://animadversio.github.io/gemma4-simple/ But your post is a much better visual guide to the model! learned a lot
Hi thanks for this amazing explanation. Question: in global attention how is the key = query if the size of query is doubled? Unless the size of the key is also doubled to ensure we can multiply q^T * K. Dimension wise both q and K weights would need to be 512?
One of the best educational resources for LLMs on the internet. A heartfelt thank you for all the work you put into these.
Should I convince my manager to continue doing this kind of content? ~> yes
Thank you! I'm definitely hoping to do more of these ;)
Yes, please keep doing this content! Very information dense and didn’t get the feeling I was reading verbose slop. Thank you!!
Amazing walkthrough, learned a lot!
Quick question on the vision projector bit — you mention it's "a small neural network" and then call it a "linear projection." Is there a nonlinearity in there, or is it just a single linear layer? Just wondering.
+1 had the same question while reading.
Wow, this is intense! Thanks for the hard work that went into this. you should definitely convince your manager to allow you to keep creating content like this. they say the best way to learn is to teach and I think you’re doing both by sharing this to us.
Great post !
Super useful thanks for sharing
I follow Maarten's posts (especially the visual guides) ever since he started to publish them. They are very intuitive, easy to understand, and clearly explained. Which is why the O'Reilly book "Hands-on LLMs" (which Maarten wrote together with Jay Alammar) is so popular, as it uses a lot of these types of visuals.
Almost tempting to call these visuals "Deep visuals" as a tribute to Maarten's work at DeepMind now :)
As to the question whether the manager needs to be convinced. I think that is the wrong question. The question should be: "A manager who does NOT support this work should not be Maarten's manager in the first place". Because if you think about it: if Google is big on making model families such as Gemma available to the public, then any (visual) guide that goes along with this fits right into this strategy.
Keep up the great work, Maarten! And let me know if I need to talk to your manager if he/she needs convincing :)
yes!! you definitely continue this kind of amazing content with or without convincing your manager hahaha
Amazing read!
One confusion here:
"With Mixture of Experts in Gemma 4, only 8 experts and 1 shared expert is actually used for intermediate calculations. All other 119 experts can take a backseat. These are the active parameters and represent the “A” in “26B A4B”. "
Should it be 119 or 120 experts taking back seat? There are 128 + 1 experts(mentioned in the figure also) and out of these 9 are selected at a time leaving 120 inactive.
Best illustration of gemma4! I was trying to understand the code by simplifying it https://animadversio.github.io/gemma4-simple/ But your post is a much better visual guide to the model! learned a lot
This is wonderful content! Definitely a vote from me on continuing if you are able!
YES. Convince your manager. This was great. Thanks for sharing!
Hi thanks for this amazing explanation. Question: in global attention how is the key = query if the size of query is doubled? Unless the size of the key is also doubled to ensure we can multiply q^T * K. Dimension wise both q and K weights would need to be 512?
Thank you so much for the write up. This has been very insightful.
Hey — I came across your writing and really liked how you think.
I’m exploring something similar from a different angle — writing about human behavior through a system design lens (like debugging internal patterns).
Just started publishing on Substack. If you ever get a moment to read, I’d genuinely value your perspective.
Also happy to support your work — feels like there’s an interesting overlap here.