A few months ago, in June 2026, Google released their latest model in the Gemma 4 family: Gemma 4 12B. The Gemma family is Google’s set of open-source models, and this particular model release was designed to be smaller but still maintain a high level of performance. However, beyond its small size, this model is advertised as being “encoder-free”, while still retaining multimodal capabilities. Multimodal systems are those that process different ‘modalities’ of data (text, audio, visual, etc.) within the same system.
Reading through the press releases led me to want to understand what exactly is meant by “encoder-free”; in the process, I ended up going through a large review of the methods multimodal systems use to incorporate different modalities into their systems. In some sense, many of my previous posts have already been focused on multimodal systems, since most of the audio generation systems and all of the text-to-speech systems handle both text and audio data. However, I haven’t explicitly covered the way these different data types are handled before. Thus, this post will look at the different strategies that can be used in developing a multimodal system, then will take a look at the way Gemma 4 approaches it and how it is different from previous approaches.

Multimodal methods

Researchers have experimented with many different ways to incorporate multiple data modalities into a complete system. I’m going to outline a number of the different methods I’ve come across, and I’ll try to point out any generative audio models we’ve seen before that use these methods. I’ll try to cover things in order of overall adoption, so the first few methods I’ll cover are used by the majority of multimodal systems. I’ll then highlight the difference in Gemma 4 12B; in particular, what it means to be “encoder-free”.

Separate Encoders

Probably the most straightforward and widely-used method is to simply add a separate encoder for each modality then combine the resulting data together. Combining the data has two main approaches: appending the data together, or encoding the data then using a joint embedding space (an example of this would be CLIP).

A good example of this method is outlined in NExT-GPT: Any-to-Any Multimodal LLM, published June 2024. It uses separate encoders for the input modalities, followed by a LLM central router with semantic understanding and reasoning, and resulting in separate output projections
From the paper, the LLM “outputs: 1) the textual responses directly, and 2) signal tokens of each modality that serve as instructions to dictate the decoding layers on whether to generate multimodal contents and what content to produce if yes”.

For a further modification to this method, the separate encoders can be connected to a shared parameter space down the line. An example of this is in the Transfusion paper from Meta, published August 2024. This model uses separate components with unshared parameters to embed the different modalities and different objective/loss functions for different modalities, but it maintains shared data parameters in the later layers. Additionally, marker tokens help separate the modalities.

The majority of generative audio models we’ve seen before make use of this method to handle the different modalities of text and audio, prominent examples including VALL-E, TorToiSE, and Llasa.

Unified architecture

For the next method, instead of using multiple encoders, some multimodal systems use a single encoder then pass the results to a shared latent space. These models typically make use of contrastive loss such as CLAP or CLIP.
We can see this method in action in the MoMo paper, published in April 2023, which makes use of a single-stream encoder to process both text and image data.

momo_process Source

Another good primer for how this method is implemented can be found in the paper A Shared Encoder Approach to Multimodal Representation Learning. Published in March 2025, this paper outlines a method of using additional information to mark the data types, so that the model can inherently learn what to do with the different data types. It appends a “modality feature vector” or token to the embedded data to give the encoder info on what data type it is.

A further example can be found in the Chameleon, published May 2024 from Meta.
chameleon_arch Source

Of the audio models we’ve looked at before, the most relevant for this method is AudioLDM, which uses Contrastive Language-Audio Pretraining (CLAP) to align the text and audio embeddings into a joint cross-modal space.

Cross-modal attention

The next method handles different data modalities not in the initial data encoder but within the attention process. Cross-Modal Attention Mechanisms compute dependencies across different data modalities “by selectively conditioning the representation of one modality on features from another”. Essentially, this method uses the self-attention paradigm, but query values in modality1 attend to keys and values in modality2. Attention in these systems can be bidirectional, hierarchical, or single- or multi-headed.

This method is harder to implement so it doesn’t get used quite as much, but it has still seen an abundance of research directed toward it. One example is in the paper Cross-stitched Multi-modal Encoders from April 2022. The developers here combine speech and text encoders using a jointly trained attention mechanism. They use a pretrained Wav2vec2 model for turning the audio segments into embeddings and a pretrained text encoder for text token embeddings, then pass the different embeddings into the multi-head attention.
cross_modal_attn
Source

Mixture of Experts

Moving into the realm of less-frequently used but still very interesting methods, some researchers have experimented with using Mixture of Experts (MoE) models to actually use the “experts” for handling the different data modalities.

The paper Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts, published May 2024, uses modality-specific encoders with connectors for a unified multimodal representation followed by sparse MoE architecture with modality-specific experts. It uses an expert “router” to determine which expert to send the input to taking modality into account.
uni_moe_arch
Source

For another variation of the MoE method, the paper MOVE: A Mixture-of-Vision-Encoders Approach for Domain-Focused Vision-Language Processing from February 2025 moves the MoE aspect further up the pipeline by setting up multiple encoders for images (e.g. high-resolution vs low-resolution) and using an expert router to determine which encoder to use. In this sense it is actually a bit of a hybrid, using both separate modal encoders as well as MoE

Graph Approaches

The final method I’ll highlight is to encode the various modalities and incorporate them into a graph model. The paper “A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation” explains this approach. It builds unified a graph to represent the input sentence and image, with the goal of capturing semantic relationships. This is using a graph network to guide the separate embedders and attention, acting in a way like separate encoders, but it also uses cross-modal gating, in a way similar to cross-attention, to connect the semantic relationships and complete the encoding process.

I’ve been exploring graph models a bit more lately, and I may dive deeper into the topic in a future post, so stay tuned.

Gemma 4 12B

Now that we’ve covered the main ways researchers have thus far focused on building out multimodal systems, we can finally get to the model that helped inspire this post: Gemma 4 12B. So what is it doing and how is it different? Well, the system looks very similar to the separate encoders method, because each modality is processed in its own module. However, instead of an encoder, which is a multi-layer neural net that has to be trained and tuned, the modules are merely embedding matrices. From the release:

  • Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. This allows the LLM backbone to take over visual processing.
  • Audio: We simplified audio processing even further. We removed the audio encoder entirely and projected the raw audio signal into the same dimensional space as text tokens.

What happens is that the image data is passed through a very simple embedding function that maps the pixel data to a linear space, while the audio data is sliced into 40ms clips and projected linearly as well. The data is then all tossed into the LLM, a decoder-only Transformer, to do the heavy lifting of interpretation and generation.

gemm412b
Source

The result is a much smaller model, because it uses a single-layer (35M parameter) vision embedder when previously it would have used a multi-layer vision encoder (multi-billion parameter) from a transformer.

This is a very brief summary of what the model is doing. If you’d like more detail, an in-depth guide detailing extra information such as the incorporation of positional information in the vision tokens using X and Y coordinate matrices can be found here. In a sense the model is intensively simplifying the data transformation process and letting the LLM figure out the data modality info on its own, but the results show that this is still able to work remarkably well.

Closing Thoughts

So there you have it: the different ways researchers have built systems to process multiple data modalities, as well as a new interesting approach from Google. Technically speaking, the encoder-free method isn’t completely novel, as stripping out the encoder from vision-language models has been explored in other places as well, but Google has certainly pushed the frontier in the development and incorporation of the technique.
If you’d like to dive even deeper into this topic, some interesting papers to look at are: A Survey on Multimodal Large Language Models, a November 2024 paper that explores the module components of multimodal LLMs ; and A Survey of Unified Multimodal Understanding and Generation: Advances and Challenges, an October 2025 survey of over 700 papers involving multimodal systems, which attempts to categorize different themes and trends seen in the research community.