Most Multimodal LLMs today are only multimodal in one direction: they can understand images, audio, and video as input, but they still only generate text as output.
Enabling an LLM to actually generate images, audio, and video requires quite a different set of architectures!
With over 50 custom visuals in this guide, we will explore how modern AI generates images, audio, and video from scratch, and how LLMs are being architected to decode them.
A Free Bonus Chapter!
This was originally a chapter of our recently released book, An Illustrated Guide to AI Agents but we created just too many visuals! The book was too long and we decided to scrap this chapter.
But it shouldn’t go too waste! As with our previous book, we love creating free bonus content for people that have supported us over the years.
So… we will continue pushing out content as we have always done and giving back to the community!
P.S. If you read the book, a quick review would mean the world—it really helps us authors!
Introduction
Multimodal LLMs generally focus on encoding information (input) rather than generating information (output). It is a much cheaper operation without necessarily requiring large architectural changes
However, LLMs are not the only type of models that have gained significant attention in 2024 and 2025. Generative image and audio models, specifically, have developed significantly in those years with techniques like Stable Diffusion.
These models take in some text and output their respective modality.
These models, although highly capable, are quite one-dimensional in how you interact with them. Typically, a text-to-image needs a description of what you want in the image. So you are not saying “please give an image of a cute flamingo” but rather “flamingo, pink, beach”. Newer techniques, such Qwen-Image1 and Z-Image2, use an LLM-backend to process more extensive descriptions of the visuals.
As such, you are not interacting with these models as you would with an LLM. These models, assuming they also have generative capabilities, are exceptionally interesting as they allow you to “reason” about the input and output of the non-text modalities.
These models that can process and generate both text and images allow you to use more complex prompting to create, edit, and improve images. For instance, such a model can take any prompt detailing changes to a given image and generate this updated image.
By combining both multimodal understanding and multimodal generation, these models are quite capable and useful for chat sessions that involve understanding and generating many different modalities.
To gain these kinds of capabilities, a Multimodal LLM needs to understand different modalities and generate multiple modalities. This is part of the optional fourth component in Multimodal LLMs, the generator.
This generator can be either a standalone model, such as a text-to-image or text-to-audio model, or a fundamental update to LLM architecture that enables it to generate more than just text. When it is a standalone model, the LLM itself is not multimodal, but its internal embeddings are projected to the same vector space so that they can be the input of an independent generative model.
In some cases, the architecture of the LLM is updated, making it capable of multimodal generation by itself. That typically means that all generators need to have some alignment on their architecture.
We will explore both methodologies and how they affect their underlying architectures. We will explore how LLMs can be made into models that generate modalities other than text by leveraging their multimodal understanding.
The ability to instruct the process of both multimodal understanding and multimodal generation is going to be a vital principle in how we interact with Multimodal LLMs.Especially with the way the field is progressing, we are seeing more and more models that have a degree of multimodality to interact with the world more easily.
Generating Modalities
To understand how LLMs can be extended to generate modalities other than text, let us first explore how these modalities can be generated. Typically, these are text-to-N models, where N is any given modality.
Let’s go over some of the main methods of generating the most common non-text modalities: audio, images, and video.
Generating Images
Like text generation, image generation has seen tremendous improvements in 2022 through 2025. It is uncanny how close the generated images are to what a human would create.
In this section, we will explore two major concepts in generating images:
Latent Diffusion Models
Diffusion Transformers
Stable Diffusion
This influx of models and techniques all started with Stable Diffusion, a text-to-image model that arguably had the same impact on image generation as the original ChatGPT did in text generation. Stable Diffusion is not limited to text-to-image and can also run text/image-to-image tasks.3
Stable Diffusion consists of five main components:
Text Encoder — Used to convert text to embeddings
Diffusion Model — Used to gradually process/diffuse image information in the latent space. As such, its main goal is to predict the noise that might exist in the input.
Noisy input image — Used as the base image from which to create the output
Image Encoder — Converts images to latent representations
Image Decoder— Converts latent representations to images
Let’s go over each of these, starting with the two most vital components: noisy images and the diffusion model.
What is Diffusion?
Diffusion is the process of iteratively adding and removing noise to a given image. Using token embeddings that represent the input text and a random starting image information tensor (noisy image), this process continuously updates the information tensor for the image decoder to convert to an image. This tensor is a numerical representation of the input image, much like embeddings are for input text. Compared to embeddings, tensors tend to have more dimensions than just one.
If we were to decode the random image information tensor into an image, we would see nothing more than noise. Diffusion starts from noise because, as we will cover later on, the model is trained to remove noise, and pure noise is therefore the only starting point it understands. As you will discover, noise is actually what makes these models so powerful!
So, how do we get from that initial noise to that nicely generated image?
Diffusion is an iterative, sequential process where the image information tensor is updated each step to better resemble the input text and all visual information the model learned during training. Visualizing a given number of steps shows how the initial noisy tensor contains less noise with each step. This is called denoising the image.
Note how each step operates on this tensor and not the image itself. The image decoders are merely added to showcase what these intermediate tensors would look like if decoded into an image. As we will explore later, this speeds up computation quite a bit.
How does diffusion work?
Let’s dive a bit deeper into how this denoising process actually works.
We start with a dataset that contains image/text pairs that we want to train the model on. For each image, we can choose the amount of noise to add. This is called forward diffusion.
This creates a processed dataset, one where all images have a certain degree of noise. As a result, we can generate dozens of new training examples for each image, all with a different amount of added noise.
U-Net
U-Net is a convolutional neural network (CNN) that removes noise step by step while attempting to preserve structure and details.4 It does so by predicting the noise from the training examples that were created in the previous step.
It roughly consists of two sections containing convulational blocks for deriving and creating features, namely an encoder and decoder section.
U-Net was named for the U-shape its architecture follows. It has two important components:
Encoder — Extract relevant features from an input image using convolution and layers to compress these features.
Decoder — Takes the relevant features and reconstructs a segmentation mask (noise)
Note that some of the outputs are fed (via residual connections) into the processing later in the network. Moreover, the timestep (the current step) is added to each block and layer.
So why would we want to introduce noise into the images and then attempt to predict it?
We can subtract the predicted noise from an image (random tensor) to get it closer to the images the model was trained on. It will not produce the exact same images as seen during training, but it attempts to reproduce their distribution, like arrangements where the sea is usually blue, cars have four wheels, and hands have five fingers.
This is an iterative process where noise is predicted at each step until a predefined number of steps is reached. More steps tend to improve quality and stability, but they also cost time. In contrast, fewer steps are faster, but they risk artifacts in the image since each step must do more work than the model was trained for.
This process is called reverse denoising or reverse diffusion and is the second step of training a diffusion model:
Forward diffusion — Generating noisy training data
Reverse diffusion — Denoising noisy images
The number of steps chosen is user-defined and typically depends on how the model was trained. The current timestep (t) is also tracked in the U-Net model, so it knows where in the denoising process it currently is.
Latent Space
To speed up this diffusion process, Stable Diffusion does not run this diffusion on the images themselves but on the tensors that we explored previously.
To convert these tensors back into images, Stable Diffusion makes use of a Variational Autoencoder (VAE), which is trained to compress an image into a latent space (tensor) using its encoder.5 Then it reconstructs it from the latent space back into an image.
The forward diffusion process we explored earlier is therefore done on the compressed latents and not the images themselves. As such, U-Net is actually trained to predict noise in the latent space. Both the forward diffusion (image encoder) and backward diffusion (image decoder) processes are not directly based on the image but on its compressed representation instead.
Text Encoder
There is a vital component still missing. How is the model guided during its denoising process? The random image tensor doesn’t contain any information yet about what we expect the generated image to be.
The text encoder is the final missing piece of the puzzle. It is a pre-trained CLIP model that allows both images and text to be embedded. To make the text embedding a part of the image generation process, U-Net is updated to use the text as input by adding a CLIP encoder for the text.
This is called text-conditioned guidance and steers the detection of noise. Everything that is not related to the text input is identified as noise and is eventually removed from the image.
CLIP was trained to represent both text and images in latent space well. As a result, CLIP is a great solution for adding text embeddings during the training process of U-Net.
To add CLIP embeddings to U-Net, cross-attention layers were added that could process both the image tensor and the incoming CLIP embeddings. Like all other components of U-Net, these cross-attention layers process the images in latent space and are encoded and decoded through the VAE.
Note that the blocks do not directly look at the text. But the attention layers merge those text representations in the latents. And now the next block can utilize the incorporated text information in its processing.
Combining all of these processes allows for an image to be generated from both a noisy image and guided text.
And that is it! Stable Diffusion, and in particular, latent diffusion models, were a fundamental shift in the quality of image generation.
Diffusion Transformer
The success of Transformers in text has made its way to other fields. Transformers are not limited to text but can be used successfully for representing other modalities like images and audio.
Likewise, there has been a tremendous effort in using Transformers for generating other modalities. A great example is the Diffusion Transformer (DiT), a model that combines both Latent Diffusion and the Transformer architecture.6
This Transformer, also called the Latent Diffusion Transformer, consists of the following components:
Patchify — Convert a latent image into patches (tokens).
Embeddings — Labels (image captions) and timesteps (step in the denoising process) are embedded.
DiT Block — The core processor of the input patches and embeddings.
Layer normalization — Normalizes the output embeddings.
Linear and Reshape — Reshapes the embeddings back into the input shape.
Predicted noise — The distribution of noise in the original image is predicted.
Uncertainty estimation — How uncertain it is about its noise prediction at each pixel/location.
The noisy input image is converted into patches (tokens) so that the DiT block can process it along with information about the timestep and the label.
After processing in the blocks, normalization is applied to the embeddings, which are then converted back into the shape of the input noisy image. Then, it is used to predict the noise and how certain the model is about the predicted noise at different locations of the image/latent space.
The DiT blocks follow the same pattern as with regular LLMs; they are stacked and processed one at a time through self-attention. There are a couple of differences worth noting.
Aside from the expected layer normalization and self-attention mechanisms, there are additional conditioning layers that allow layer normalization to become adaptive based on the timestep and label embeddings.
There are two main reasons why we want to condition how the transformer processes the image tokens:
Timestep conditioning — Different timesteps need different processing strategies. At early timesteps, when the image is highly noisy, the model should focus on capturing global structure. At later timesteps, when the noise is low, the model should shift its attention toward fine details.
Label conditioning — Different classes of objects need different attention patterns. Generating a “flamingo” versus “house” requires focusing on different spatial relationships and features.
This conditioning through the layer normalization layer is called Adaptive Layer Normalization (adaLN).
A huge benefit of using transformers in diffusion is that we can make use of all the advancements that the transformer architecture has seen in the textual domain. In contrast, U-Net relies on deep hierarchical convolutional structures with skip connections, which become increasingly memory-intensive, harder to optimize, and less efficient to parallelize as model size grows.
At first glance, diffusion and transformers seem like quite different techniques to use. It is interesting to see how these fields are coming together and create these powerful models. For instance, newer image generation techniques, like PIXART-α7 and FLUX.1 Dev, make use of this Diffusion Transformer to generate high-quality and high-resolution images.
Generating Audio
We are seeing Transformers continue to dominate many generative fields, including those of generating Audio and Speech. An impactful audio generation technique is AudioLM, which demonstrates the power of Transformers in generating audio.8
Conventional language models are trained to predict text given a piece of text. AudioLM, in contrast, is trained to predict audio given a portion of audio. It attempts to predict audio while maintaining both the acoustic (pitch, timbre, intonation, etc.) and semantic (spoken words, speaker intent, etc.) features of the input. It was designed to handle a variety of sounds, including both human speech and music.
To do so, it contains three important components.
Generating acoustic tokens with SoundStream
Generating semantic tokens with w2v-BERT
Processing and generating audio with Hierarchical Transformers
Acoustic Tokens
In AudioLM, given an input audio, acoustic tokens are extracted using a neural codec called SoundStream. A codec is a technique that compresses (encodes) and decompresses (decodes) digital data like audio or video to reduce their file size. A neural codec is a more advanced technique that uses neural networks for this compression/decompression. Compared to traditional codecs, neural codecs learn compression strategies directly from the data through training, which allows them to better preserve important features with fewer audible distortions.
Neural codecs typically use autoencoders, which are neural networks (VAE) that consist of two components:
Encoder — Compress audio into vector representations to capture features
Decoder — Reconstructs audio from the vector representations
As we have explored before, encoder-decoder architectures are great at generating intermediate vector representations.
These continuous vector representations, like most others, are extremely useful in a wide variety of tasks. However, they cannot easily be used in Transformer models, which require discrete vector representations.
To create these discrete representations, vector quantization (VQ) is used. VQ is a technique that allows any given continuous signal to be mapped to a fixed set of levels. This fixed set is called a codebook and allows many vector representations to be mapped to a select few.
As such, VQ can be applied to the latent space, converting it to a codebook of vector representations. This codebook can be seen as discrete acoustic token embeddings and is exactly the type of representation that a Transformer might expect. This modified version is known as a VQ-VAE.
A variant of VQ-VAE is used in SoundStream, a neural codec that is used in AudioLM to extract the necessary acoustic representation.
Compared to VQ-VAE, it uses Residual Vector Quantization (RVQ) to compress and represent audio. RVQ is a more advanced technique for quantization and iteratively refines the residual errors from each quantization step for more accuracy.
Let’s now dive a little deeper into the architecture of SoundStream. Although, like other neural codecs, it has an encoder-decoder architecture, it is not the same as the encoder-decoder architecture of Transformers. Like U-Net, it uses convolutional blocks instead of the well-known attention blocks to process the audio (encoder) and recreate it (decoder).
There is additional timestep conditioning (as with the latent diffusion transformer). This enables the model to adapt to different levels of noise or reverb in audio.
This model is trained by using a discriminator, with an architecture similar to a small encoder, to predict how distinct the input signal is from the reconstructed signal. The resulting adversarial loss is used to update the encoder and decoder to improve on the compression/decompression of the original signal. The discriminator’s architecture is much like the encoder and attempts to find differences between the input and output.
The intermediate discrete vector representations are the acoustic tokens that will eventually be passed to the hierarchical transformers.
Semantic Tokens
Aside from the acoustic tokens, semantic tokens are also extracted to give more information about the semantic content of the audio. Semantic content is the high-level meaning of the audio, such as what words are being spoken or the linguistic content of speech, rather than low-level information such as pitch or timbre.
These are extracted using w2v-BERT9, which is an extension of the Wav2Vec 2.0 model. Remember that Wav2Vec 2.0 used contrastive learning to create quantized embeddings, which could be used for representing input audio.
w2v-BERT takes that idea and goes one step further by adding a Masked Language Modeling task, much like is done in encoder-only models, to further enhance the embeddings with contextual information.
Note that Conformer Blocks are used instead of your regular Transformer Blocks.10 These are similar but have a few differences. Conformer blocks have an additional convolution module wrapped in feed-forward neural networks to process the audio features.
Hierarchical Transformers
Using the previously explored semantic and acoustic tokens, AudioLM generates audio by passing them through a set of chained transformers. At each stage, the Transformer Decoder is trained for next token prediction based on past tokens.
In the first stage, the semantic tokens are processed and generated using a Transformer Decoder. This is done to model the high-level structure of the audio sequence.
In the second stage, the predicted (future) semantic tokens are concatenated with the past acoustic tokens and fed to a model. This produces tokens that represent coarser features, like speaker characteristics.
In the final stage, the predicted coarse acoustic tokens are further processed to add more detail to the final audio. The resulting acoustic tokens (both coarse and fine) are fed to the previously explored SoundStream decoder to create the audio from the tokens.
AudioLM was used for various use cases of audio generation, including speech continuation and piano continuation, taking into account the original audio examples. Since AudioLM, there have been many incredible techniques that use similar architectures. Common amongst those are the following components:
VQ-VAE - Some variant of VQ-VAE to create a codebook of audio codes
Transformer - To transform text/audio into audio.
Neural Codec- A decoder (also called a vocoder) for converting latent representations (embeddings) back into waveforms.
Generating Videos
Video generation does not differ much from image generation, the same way video understanding does not differ much from image understanding.
The main differences between them, assuming we ignore audio for now, is that video has an additional temporal dimension that needs to be modelled when generating each frame at a time. The foundation of generative videos, like images, lies in their diffusion process; its ability to start from noise and iteratively reduce it.
Early advancements like ImageGen Video11 and Stable Diffusion Video12 use diffusion-only to generate videos. However, new techniques, like image generation, use Diffusion Transformers (DiT) as the main component for generating individual frames.
A great example is Latte, an early video-to-video model that uses a video-focused DiT for generating videos.13 Its architecture is similar to what we have seen in the DiT for image generation. What makes this architecture a great example is that the authors of Latte explored various common techniques for video generation, including methods for tokenizing the video sequence and modeling the Transformer blocks.
Latte uses the Transformers architecture wrapped in VAE to create the same diffusion process we have explored before with Stable Diffusion and Diffusion Transformers.
To tokenize and embed the input video sequences, the authors explore two techniques:
Uniform frame patch embeddings — Create patches of each frame individually (spatial) and concatenate the embeddings
Compression frame patch — Create patches of each set of frames on (tubelet embeddings; temporal) and concatenate the embeddings
These techniques are similar to the ones explored in chapter 9 of “An Illustrated Guide to AI Agents”, specifically in the Video Vision Transformer (ViViT) section.
Interestingly, the authors of Latte found that the compression frame patch embeddings performed worse than the uniform frame patch embeddings. They speculate that the compression frame patch embeddings lose spatio-temporal information as a result of the compression that is needed to create the tubelets.
The embeddings are then passed to a similar structure we have explored with DiT, namely through a Transformer model, layer normalized, and linearly mapped and reshaped to predict both the noise and the uncertainty estimation.
Remember that Latte is a video-to-video model and has no text conditioning and instead attempts to continue the rest of an input video.
However, to inject timestep information, they explored both passing that information as tokens and a variant of Adaptive Layer Normalization (AdaLN) that was used in DiT. The authors found that their slight variant of AdaLN outperformed treating the timestep information as tokens by encoding the timestep information into the model in a more adaptive manner for each Transformer block. Their analysis was based on the Fréchet Video Distance, which evaluates the realism of generated video by measuring how closely its visual quality and motion consistency match that of a dataset of real videos.
The best-performing architecture of the Transformer blocks was to interleave spatial and temporal blocks, which attend to spatial and temporal information, respectively. They call this “interleaved fusion” and is similar to the two-step spatial/temporal architecture of ViVit.
This DiT-like architecture has become quite popular in 2025 for open-source video generation models, like LTX-Video14, WAN15, and Hanyuan Video16. Although each of these models has architectural differences, they all use Diffusion Transformers to generate videos.
Creating Generative Multimodal LLMs
Now that we have explored several methods for generating various modalities, we can investigate how we can connect them to LLMs to make them truly multimodal.
There are two ways to enable multimodal generation:
Late Fusion (Output projection) — An MLP that connects the textual output of the LLM to the input of other-modality generators.
Early Fusion (Unified generation) — The architecture of the LLM is updated to contain other-modality generators.
Late Fusion
A common method for creating Multimodal Large Language Models is to connect other-modality generators to them. This is generally done in one of two ways: either using the text output of the LLM to guide the generator, or using an MLP projection of the tokens/embeddings as the input to the generator.
Using the text output is quite straightforward; you only need to ask the LLM to create a prompt for your generator. For instance, you can ask the LLM to create an image prompt for your stable diffusion model and then run the stable diffusion model separately.
Although an elegant method, you can hardly think of the LLM as a single multimodal entity. We could think of this methodology more like an agent that uses other generative tools to create non-text output.
A more native method for creating multimodal LLMs is to use the output embeddings of the LLM to guide the process of the other-modality generator.
A great example for this process is NExT-GPT, an any-to-any Multimodal LLM. NExT-GPT is capable of understanding and generating many different modalities through projectors, including images and sound.17 Note that only the input and output projectors are trained; all other models remain frozen.
To align the multimodal input encodings with the text encodings that the LLM expects, NExT-GPT projects their respective input embeddings by aggregating features into semantic concepts through a grouping mechanism. This is handled by a small Transformer that projects the input embeddings to the representations the LLM expects.
During training, captions are generated for each of the input modalities to help the small Transformer model align its projection with the text representations.
The input projection is done through a trainable connector leveraging a combination of Transformer blocks and Grouping blocks.
The input patch embeddings from the encoders (ImageBind) are first passed through a Transformer encoder before being grouped together. This is done by finding similar embeddings and grouping them before passing them through an MLP.
This is a hierarchical process and can be understood intuitively like this:
Stage 1: Group similar pixels/patches (e.g., all “sky” patches together)
Stage 2: Group related concepts (e.g., “sky” and “clouds” into “weather”)
This hierarchical grouping helps the LLM focus on semantic concepts rather than being overwhelmed by low-level patch details, leading to better multimodal understanding and reasoning.
To then align the generation process, NExT-GPT leverages existing decoder techniques, which are all pre-trained diffusion models, like Stable Diffusion for image generation.
A Transformer-based output projector is then used to map the token representations into ones understandable by the decoders.For each modality, special tokens are created, each carrying rich instructions for the downstream diffusion model ([IMG], [AUD], [VID] for image, audio, and video, respectively).The pre-trained diffusion models are then conditioned on these special tokens in the denoising process to guide them to generate the appropriate images, videos, or audio.
Note that the special tokens that the LLM generates will indicate which type of generative model will be used.
The key benefit of using projections for both the input and output embeddings is that only a small number of parameters need to be trained. Instead of models with billions of parameters, “only” millions of parameters in the projectors need to be trained.
More recent examples of this projection-based technique are OmniGen218 and BLIP-3o19, both models that decouple the autoregressive transformer for text generation with the diffusion model for image generation. Both condition the diffusion model on the visual tokens generated by the autoregressive transformer, which helps guide the image generation process.
Early Fusion
Compared to late fusion, early fusion aims to put all understanding and generation of multiple modalities early in the generative process. This typically means a unified architecture that is capable of generating multiple modalities rather than connecting several models together.
The LLM itself is then tasked to create another modality in addition to the text it creates. The method by which it does remains very similar to its text-based generation, namely the generation of embeddings, which are detokenized into text or any other modality. The LLM would generate both text and image token embeddings, which are then converted into their respective modality. Like text token embeddings, the image token embeddings contain rich information about the images.
A great, and quite elegant, example of this unified early fusion architecture is DeepSeek’s Janus multimodal LLM.20 This LLM uses a unified approach to create token-based representations for both images and text. The resulting representations are fed to the LLM, which in turn generates updated representations. These are then decoded by their respective modality to either generate text or an image.
The advantage of Janus is that different encoders are used when training the model for image understanding and image generation tasks. This allows the representations to be specific to their task, as each requires various levels of information granularity:
Understanding — Requires abstract, high-level semantic representations.
Generation — Requires concrete, detailed information.
This is a nice example of early fusion, where all modalities are projected into a shared representational space before being fed to the LLM. To create these image representations, two different models are used for understanding and generation. SigLIP, for image understanding, is an image encoder very similar to CLIP but employs a pairwise sigmoid loss on image-text pairs during training.
VQ-GAN, for image generation and much like VQ-VAE, is an encoder-decoder model that uses vector quantization to create a codebook of discrete image tokens in latent space and reconstructs an image from them.21
Specifically, the encoder and decoder modules use a CNN to efficiently embed the images into latent space and reconstruct them. During training, a discriminator (also a CNN) is used to detect how close the reconstructed image is to the original.
VQ-GAN also uses vector quantization to create a codebook of discrete representations.
The discrete representations that are created in latent space are much like token embeddings. Both have a fixed vocabulary and can be processed by the Transformer as a sequence of discrete representations.
The output of the LLM is then decoded into an image.
The model was trained in three separate stages:
Training Adaptors and Image Head — Create conceptual connections between visual and linguistic elements by only training the adaptors and image head.
Unified Pretraining — Enable Janus to learn both multimodal understanding and generation. Many sources of data are used: pure text data, multimodal understanding data, and visual generation data.
Supervised Fine-tuning — Additional instruction tuning data to enhance its instruction-following and dialogue capabilities.
Compared to late fusion, early fusion is much more costly to train, as most of the parameters need to be updated. However, it tends to be a more accurate technique as all capabilities are trained in a unified way.
Janus, like many other native multimodal LLMs, gained popularity in 2025 due to its unified architecture. As these native multimodal LLMs start to become popular, Agentic systems get more multimodal capabilities, thereby extending their control over their environments.
Conclusion
This concludes our journey in multimodal generation! Hopefully, this post gives you a better understanding of the potential of these models that generate more than just text. Who knows what kind of models we will see in the future?!
To see more visualizations like these and to support this newsletter, check out the book we wrote on AI Agents!
Wu, Chenfei, et al. "Qwen-image technical report." arXiv preprint arXiv:2508.02324 (2025)
Cai, Huanqia, et al. "Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer." arXiv preprint arXiv:2511.22699 (2025).
Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.
Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." International Conference on Medical image computing and computer-assisted intervention. Cham: Springer international publishing, 2015.
Kingma, Diederik P., and Max Welling. “Auto-encoding varareiational bayes.” 20 Dec. 2013,
Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." Proceedings of the IEEE/CVF international conference on computer vision. 2023.
Chen, Junsong, et al. "Pixart-$\alpha $: Fast training of diffusion transformer for photorealistic text-to-image synthesis." arXiv preprint arXiv:2310.00426 (2023).
Borsos, Zalán, et al. "Audiolm: a language modeling approach to audio generation." IEEE/ACM transactions on audio, speech, and language processing 31 (2023): 2523-2533.
Chung, Yu-An, et al. "W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training." 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2021.
Gulati, Anmol, et al. "Conformer: Convolution-augmented transformer for speech recognition." arXiv preprint arXiv:2005.08100 (2020).
Ho, Jonathan, et al. "Imagen video: High definition video generation with diffusion models." arXiv preprint arXiv:2210.02303 (2022).
Blattmann, Andreas, et al. "Stable video diffusion: Scaling latent video diffusion models to large datasets." arXiv preprint arXiv:2311.15127 (2023).
Ma, Xin, et al. "Latte: Latent diffusion transformer for video generation." arXiv preprint arXiv:2401.03048 (2024).
HaCohen, Yoav, et al. "Ltx-video: Realtime video latent diffusion." arXiv preprint arXiv:2501.00103 (2024).
Wan, Team, et al. "Wan: Open and advanced large-scale video generative models." arXiv preprint arXiv:2503.20314 (2025).
Kong, Weijie, et al. "Hunyuanvideo: A systematic framework for large video generative models." arXiv preprint arXiv:2412.03603 (2024).
Wu, Shengqiong, et al. "Next-gpt: Any-to-any multimodal llm." Forty-first International Conference on Machine Learning. 2024.
Wu, Chenyuan, et al. "OmniGen2: Exploration to Advanced Multimodal Generation." arXiv preprint arXiv:2506.18871 (2025).
Chen, Jiuhai, et al. "Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset." arXiv preprint arXiv:2505.09568 (2025).
Wu, Chengyue, et al. "Janus: Decoupling visual encoding for unified multimodal understanding and generation." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.
Esser, Patrick, Robin Rombach, and Bjorn Ommer. "Taming transformers for high-resolution image synthesis." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.
Wow, JUICY content, thank you!