Technical background

I’ve been fascinated by graph theory ever since taking a combinatorics class back in college, and I wish I had more time that I could use to study the topic more deeply. Graph theory has a storied history stretching back to Euler in the eighteenth century, and the uses cases abound from route planning to recommendation systems. I won’t go into the fundamentals of graph structures here, but you can find a number of good tutorials online; this walkthrough from huggingface would be a good place to start.
Prior to the big data revolution, graph models primarily focused on extracting features and calculating metrics for gaining insights. Some examples of node-level features would be centrality, degree, and the clustering coefficient; edge-level features included shortest distance, common neighbors, and Katz index (number of possible walks between nodes); examples of graph-level features would be total subgraphs and kernel methods for similarity scores.
The sorts of features could be fed into classical algorithms that aimed to answer questions around centrality, shortest paths, types of subgraphs, etc. Neo4j has a good intro guide to these sorts of algorithms. Neo4j is a popular system for storage, exploration, and modeling of graphs; I haven’t played around with it too much, but the visualizations definitely look pretty.

The increasing relevance of graph models

Despite being studied for so long, graph models have only continued to grow in importance. Increasing data collection from social networking companies, streaming media services, and many others often deal with data where relationships between customers or products contain vital information for the sorts of questions being asked.
Use cases for graph models fall into both supervised and unsupervised categories:

Supervised tasks:

• Node property prediction - node classifcation, node regression
• Link prediction - whether or not a link should exist, and potentially what type
• Graph property prediction - structure of a graph, types of subgraphs

Unsupervised tasks:

• Representation learning - generate features based on graph structure for another model
• Clustering/Community Detection - groups of nodes that go together
• Similarity - find pairs of nodes that go together; use cases such as recommendation, entity resolution
• Centrality & Pathfinding - find important entities in the graph, evaluate efficient paths and trees

Additionally, graphs can get huge. The combinatorial explosion of connections means that even for graphs where nodes only have a few connections on average, the size of the data structure can increase exponentially.
Given this increase in size as well as relevance for graph data, many of these classical methods of studying graphs, as well as the standard table data structures in which databases often store them, can struggle to keep up.

With a prime source of big data (the graphs) and a plethora of use cases, the idea naturally arises to apply machine learning to the problem.

Incorporation into Machine learning

Many of the types of deep learning architectures don’t transfer perfectly into the domain of graph structures, because they are unable to capture the information contained in the relationships between nodes communicated by links. The standard deep learning models have a tough time capturing the information contained within multiple connections, as well as aspects such as path directions. CNNs pull information from connections between a data point’s immediate neighbors, but they are limited in their reach. The Transformer comes closer as one could potentially envision encoding the graph’s adjacency matrix into some form of an attention matrix, where attention is capturing connection between nodes. However, another problem arises from the data structure itself. As that huggingface post I linked earlier explained, for a neural net architecture to be able to accommodate graph structures it needs to be:

• permutation invariant: the representation of a graph and its permutations should be the same after going through the network
• permutation equivariant: permuting the nodes before passing them to the network should be equivalent to permuting their representations

CNNs, RNNs, and the like do not satisfy these requirements.

To this end, a number of new Graph ML architectures have been developed in an attempt to fuse the benefits of neural networks with the intricacies of graph structures, and to make learning on these huge structures feasible. After the foundation work of the Graph Neural Network (GNN), later developments included Graph Attention Networks, GraphSAGE, and Graph Isomorphism Networks, but the most common type of model would probably be Graph Convolutional Networks.
GCNs are based on encoding the graph data into matrices then plugging that data into convolutional layers, following a neural network architecture similar to others such as the LSTM. The key insight is to encode the graph into a matrix built around the following equation:

Where A’ is the normalized adjacency matrix and D is the diagonalization. A good walkthrough of the derivation for this formula can be found here.

The original GCN paper then defined the forward layer using the formula

The high level view of the system would then look like the diagram below, taken from the paper itself:
Source

In my work, we explored the use of GCNs for prediction tasks to forecast crop disease spread. To this end, we experimented with STM-Graphs (Spatio-Temporal Graphs), because our data consisted of a mix of historical data types, such as weather recordings, and spatial data types, such as road networks. Basically, these models split each historical reading into a separate node, then chain them together to incorporate the spatial information. An implementation using PyTorch Geometric can be found here.

Source

Closing Thoughts

This post was a brief look at graph models in machine learning and their potential use cases. Hopefully I’ll give some deeper dives in the near future, and maybe even present some example cases, so stay tuned!

Multimodal methods

Researchers have experimented with many different ways to incorporate multiple data modalities into a complete system. I’m going to outline a number of the different methods I’ve come across, and I’ll try to point out any generative audio models we’ve seen before that use these methods. I’ll try to cover things in order of overall adoption, so the first few methods I’ll cover are used by the majority of multimodal systems. I’ll then highlight the difference in Gemma 4 12B; in particular, what it means to be “encoder-free”.

Separate Encoders

Probably the most straightforward and widely-used method is to simply add a separate encoder for each modality then combine the resulting data together. Combining the data has two main approaches: appending the data together, or encoding the data then using a joint embedding space (an example of this would be CLIP).

A good example of this method is outlined in NExT-GPT: Any-to-Any Multimodal LLM, published June 2024. It uses separate encoders for the input modalities, followed by a LLM central router with semantic understanding and reasoning, and resulting in separate output projections
From the paper, the LLM “outputs: 1) the textual responses directly, and 2) signal tokens of each modality that serve as instructions to dictate the decoding layers on whether to generate multimodal contents and what content to produce if yes”.

For a further modification to this method, the separate encoders can be connected to a shared parameter space down the line. An example of this is in the Transfusion paper from Meta, published August 2024. This model uses separate components with unshared parameters to embed the different modalities and different objective/loss functions for different modalities, but it maintains shared data parameters in the later layers. Additionally, marker tokens help separate the modalities.

The majority of generative audio models we’ve seen before make use of this method to handle the different modalities of text and audio, prominent examples including VALL-E, TorToiSE, and Llasa.

Unified architecture

For the next method, instead of using multiple encoders, some multimodal systems use a single encoder then pass the results to a shared latent space. These models typically make use of contrastive loss such as CLAP or CLIP.
We can see this method in action in the MoMo paper, published in April 2023, which makes use of a single-stream encoder to process both text and image data.

Source

Another good primer for how this method is implemented can be found in the paper A Shared Encoder Approach to Multimodal Representation Learning. Published in March 2025, this paper outlines a method of using additional information to mark the data types, so that the model can inherently learn what to do with the different data types. It appends a “modality feature vector” or token to the embedded data to give the encoder info on what data type it is.

A further example can be found in the Chameleon, published May 2024 from Meta.
Source

Of the audio models we’ve looked at before, the most relevant for this method is AudioLDM, which uses Contrastive Language-Audio Pretraining (CLAP) to align the text and audio embeddings into a joint cross-modal space.

Cross-modal attention

The next method handles different data modalities not in the initial data encoder but within the attention process. Cross-Modal Attention Mechanisms compute dependencies across different data modalities “by selectively conditioning the representation of one modality on features from another”. Essentially, this method uses the self-attention paradigm, but query values in modality1 attend to keys and values in modality2. Attention in these systems can be bidirectional, hierarchical, or single- or multi-headed.

This method is harder to implement so it doesn’t get used quite as much, but it has still seen an abundance of research directed toward it. One example is in the paper Cross-stitched Multi-modal Encoders from April 2022. The developers here combine speech and text encoders using a jointly trained attention mechanism. They use a pretrained Wav2vec2 model for turning the audio segments into embeddings and a pretrained text encoder for text token embeddings, then pass the different embeddings into the multi-head attention.

Source

Mixture of Experts

Moving into the realm of less-frequently used but still very interesting methods, some researchers have experimented with using Mixture of Experts (MoE) models to actually use the “experts” for handling the different data modalities.

The paper Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts, published May 2024, uses modality-specific encoders with connectors for a unified multimodal representation followed by sparse MoE architecture with modality-specific experts. It uses an expert “router” to determine which expert to send the input to taking modality into account.

Source

For another variation of the MoE method, the paper MOVE: A Mixture-of-Vision-Encoders Approach for Domain-Focused Vision-Language Processing from February 2025 moves the MoE aspect further up the pipeline by setting up multiple encoders for images (e.g. high-resolution vs low-resolution) and using an expert router to determine which encoder to use. In this sense it is actually a bit of a hybrid, using both separate modal encoders as well as MoE

Graph Approaches

The final method I’ll highlight is to encode the various modalities and incorporate them into a graph model. The paper “A Novel Graph-based Multi-modal Fusion Encoder for Neural Machine Translation” explains this approach. It builds unified a graph to represent the input sentence and image, with the goal of capturing semantic relationships. This is using a graph network to guide the separate embedders and attention, acting in a way like separate encoders, but it also uses cross-modal gating, in a way similar to cross-attention, to connect the semantic relationships and complete the encoding process.

I’ve been exploring graph models a bit more lately, and I may dive deeper into the topic in a future post, so stay tuned.

Gemma 4 12B

Now that we’ve covered the main ways researchers have thus far focused on building out multimodal systems, we can finally get to the model that helped inspire this post: Gemma 4 12B. So what is it doing and how is it different? Well, the system looks very similar to the separate encoders method, because each modality is processed in its own module. However, instead of an encoder, which is a multi-layer neural net that has to be trained and tuned, the modules are merely embedding matrices. From the release:

• Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. This allows the LLM backbone to take over visual processing.
• Audio: We simplified audio processing even further. We removed the audio encoder entirely and projected the raw audio signal into the same dimensional space as text tokens.

What happens is that the image data is passed through a very simple embedding function that maps the pixel data to a linear space, while the audio data is sliced into 40ms clips and projected linearly as well. The data is then all tossed into the LLM, a decoder-only Transformer, to do the heavy lifting of interpretation and generation.

Source

The result is a much smaller model, because it uses a single-layer (35M parameter) vision embedder when previously it would have used a multi-layer vision encoder (multi-billion parameter) from a transformer.

This is a very brief summary of what the model is doing. If you’d like more detail, an in-depth guide detailing extra information such as the incorporation of positional information in the vision tokens using X and Y coordinate matrices can be found here. In a sense the model is intensively simplifying the data transformation process and letting the LLM figure out the data modality info on its own, but the results show that this is still able to work remarkably well.

Closing Thoughts

So there you have it: the different ways researchers have built systems to process multiple data modalities, as well as a new interesting approach from Google. Technically speaking, the encoder-free method isn’t completely novel, as stripping out the encoder from vision-language models has been explored in other places as well, but Google has certainly pushed the frontier in the development and incorporation of the technique.
If you’d like to dive even deeper into this topic, some interesting papers to look at are: A Survey on Multimodal Large Language Models, a November 2024 paper that explores the module components of multimodal LLMs ; and A Survey of Unified Multimodal Understanding and Generation: Advances and Challenges, an October 2025 survey of over 700 papers involving multimodal systems, which attempts to categorize different themes and trends seen in the research community.

It is important to realize though that depending on context quantization may be applied to different parts of the machine learning pipeline. I hear it used most often in the context of quantization of weights, where the format of model parameters are reduced to lower-precision values, e.g. from FP32 to FP8. Quantization of weights typically involves some rounding and/or clippling around a “zero point”, as well as decisions around how much of the parameter space you want to quantize and when you want quantization to take place in the training process. Broadly speaking, you have two options: PTQ and QAT. Post-training quantization (PTQ) takes place after the training has occurred, whereas Quantization-Aware Training (QAT) is incorporated into the training process.

Another important context though where quantization gets applied is in the data itself. Because we work with data of such large quantities and shapes, methods have been developed to quantize the data for use during training and inference. We’ve actually seen this concept before, when we took a look at audio codecs. In this post I want to dive deeper into the process of what data quantization actually looks like, highlighting the pros and cons to this tool, and also take a look at a possible alternative. As before, I’ll explore these ideas with my favorite use case: audio generation.

Vector Quantization in Audio Codecs

Codec Refresher

As a quick review, an audio codec is a system that compresses the information contained in audio data (i.e. the waveform) into latent space. The results of this compression are called “codes”.
Vector Quantization (VQ) forms a key component of the codec. In our real-world example, SoundStream runs the audio waveform through an encoder then applies Residual Vector Quantization (RVQ) to the encoded audio data, after which the result is run through the decoder, thus producing the audio tokens.
So the overall process looks like: continuous data inputs get translated into continuous latent representations, then the quantization maps continuous latent representations into discrete embeddings before being further transformed. This process is incorporated into the overall machine learning pipeline that generates audio, as the below picture illustrates.

source

Quantization Processes

So what does the mechanism of Vector Quantization actually look like? This is a different process from the quantization used for model parameters. In the data quantization context we are not simply truncating the precision; instead, we are trying to map each data point to the closest low-precision value by using a distance function to calculate the space between data which is typically very high-dimensional, thus maintaining a semantic accuracy while achieving data compression,. VQ commonly uses clustering methods such as K-means to cluster similar parts of the spectrogram. It then maps these clustered data points to the nearest codebook point.
A codebook is a collection of representative vectors (codewords), and each input feature vector is assigned to the nearest codeword in the codebook. The index of the assigned codeword in the codebook then becomes a discrete token.

source

If you want to examine the finer details of vector-quantization, this repo provides examples of the different implementations. Furthermore, this post from Hugging Face gives a good walkthrough of the steps involved in quantization, showing the process of flattening, computing distance, and selecting the closest codebook vector.

VQ Varieties

There are many different types of vector quantization methods; some of the most well-studied include:

• Residual Vector Quantization (RVQ): A technique that approximates high-dimensional vectors by selecting elements from a series of dictionaries, iteratively minimizing the quantization error
• Linde-Buzo-Gray (LBG) algorithm: A popular method for designing codebooks using a hierarchical clustering approach
• Generalized Residual Vector Quantization (GRVQ): An improved version of RVQ that demonstrates better performance in terms of quantization accuracy and computational efficiency
• Improved Residual Vector Quantization (IRVQ): Another improved version of RVQ that further enhances its performance

This blog post dives into the specifics of these different types of vector quantization, and provides useful illustrations

RVW, the method SoundStream uses, has been around for decades, but its incorporation into Generative AI was made popular with VQ-VAE, a model introduced in the paper “Neural Discrete representation Learning” by Oord et al.
RVQ works by iteratively minimizing the quantization error, which is the difference between the original vector and its approximation.

source

To incorporate RVQ into your deep learning model, you need a trick to complete backpropagation, since the quantization doesn’t have a gradient. A key insight of VQ-VAE, which allowed the model to actually get trained, was the Straight-through estimator (STE) identified in the diagram below. STE calculates the gradient for each layer and passes it back by skipping over the quantization function.

source

If you want to dive even deeper, some other interesting VQ methods to learn about are:

• Noise Substitution in Vector Quantization for Machine Learning (NSVQ): “a technique in which the vector quantization error is simulated by adding noise to the input vector, such that the simulated noise would gain the shape of original VQ error distribution.“
• Space-Filling Vector Quantization (SFVQ): “mapping input data points on a space-filling curve (rather than only mapping data points exclusively on codebook vectors as what we do in normal VQ) “

Continuous Tokens

As a refresher, there are a few different options for audio generation system architectures:

• Transformer architecture on discrete audio tokens; incorporating the VQ methods we talked about above; used by systems such as IndexTTS2
• Diffusion model; examples include AudioLDM 2, as well as Tango 2, which is coupled with the decoder of a VAE (turning the latent representation to a mel-spectrogram) followed by a vocoder to produce the waveform
• Auto-regressive Transformer on continuous audio tokens, where the diffusion aspect is used to model the continuous token then the transformer predicts the next continuous token

As you likely know, the general idea of LLMs were based on autoregressive models applied to discrete space, i.e. language and vocabulary. These systems were then applied on image space, despite it being more of a continuous domain. To accomplish this, the data would be vector-quantized and fed into a next-token predictor.
However, if we didn’t want to use VQ to produce discrete tokens to be used in the generative audio models, what would continuous tokens in audio ML, i.e. the third option, look like?

The paper Autoregressive Image Generation without Vector Quantization lays out a method to apply autoregressive models in a continuous-valued space (in this case image data) by modeling the per-token probability distribution using a diffusion procedure. This paper points out that the basic autoregressive idea of “predicting next tokens based on previous ones” does not depend on predicting values that are continuous. Instead, all you really need is a loss function and a sampler. The loss function measures the per-token probability distribution, which is used to build model distributions to be sampled. Discrete-valued representations are convenient in this instance because they can be modeled by a categorical distribution using cross-entropy loss and sampling based on softmax. To apply this to the continuous domain though, the paper instead uses a Diffusion Loss function (in the score matching sense) to model the per-token probability and samples from the resulting distribution. The system predicts a vector for each token, which is then fed into a diffusion model as a conditioning vector to guide the model. Thus the input and target can be continuous-valued data.

source

The paper also combines standard autoregressive models and masked generative models into a general framework wherein the masked autoregressive models “ predict multiple output tokens simultaneously in a randomized order, while still maintaining the autoregressive nature of “predicting next tokens based on known ones” … This leads to a masked autoregressive (MAR) model that can be seamlessly used with Diffusion Loss.“

Using these concepts, we can apply the idea to the continuous data in the audio domain. The paper Generative Audio Language Modeling with Continuous-valued Tokens and Masked Next-Token Prediction does just that; it took inspiration from the image generation paper above and applied similar ideas on audio data. The paper sets up a pre-processing pipeline based on AudioLDM to feed the Mel-spectogram of the audio batch into the encoder of the VAE, with a few additional transformations to produce continuous-valued tokens with the latent size 128. The pipeline used this latent representation to predict the next-token using a Transformer decoder token-wise diffusion loss with a small MLP diffusion head.

source
During inference, the predicted next-tokens are decoded back into a Mel-spectrogram, then converted to an audio file with a vocoder.
The final result: use a transformer to predict next-token, but the tokens are continuous, and diffusion loss is used to work with those continuous tokens

Continuous Codes Advantages

As a final thought, you may be asking: why would we want to do this?
Continuous tokens help to counter the downsides of information loss that comes as a natural side effect of quantization. Moreover, they maintain the benefits that tokenization brings of smaller and faster models coupled with the inherent streaming capabilities. Additionally, the downside of processes like RVQ is that the multiple steps of the VQ process make the pipeline scale in complexity and computation cost; an architecture based on continuous tokens may avoid this.

The idea of using continuous codes holds many potential advantages, but it is still in a research state and has not been heavily incorporated into existing audio generation systems. For the time being, Vector Quantization remains a vital tool to allow ML systems to effectively handle and model the type of giant datasets that come with audio.

Quick Overview

Transformers

At this point, most of us have at least some familiarity with the transformer. It is a neural network architecture that relies on data embedding, an encoder and/or decoder, and some form of attention. The Transformer was initially set out in a 2017 paper and used an autoencoder structure, but the systems that really brought the transformer acclaim, the GPT family, are decoder-only systems. Andrej Karpathy’s nanoGPT is an easy reference for understanding this: we see that the layers are feedforward + attention, without an initial encoding of the embedded text data. These text-generating models predict sequences that follow an initial prompt

Diffusion

Diffusion models, sometimes called score-based models, are a class of generative models that learn to reverse a ‘forward’ process of adding noise to a dataset. They were outlined in a 2015 paper but got a lot more attention five years later from the paper Denoising Diffusion Probabilistic Models.
Diffusion models also use an encoding/decoding structure to transform the input. The 2020 paper used a U-Net for this. During inference, the U-Net takes as input either an image (in the case of the original denoising diffusion models paper) or a latent encoded image (as in latent diffusion) and outputs the predicted noise. This predicted noise is then subtracted from the input to get a slightly less noisy image; that process is repeated a number of times before passing the result to the decoder to get the final image output.
To really understand diffusion models, I’d recommend this in-depth walkthrough of the implementation of a diffusion model with a U-net, as well as this colab notebook. An important point, which took me a while to realize, is that the U-Net gets run during each step, rather than each step representing a layer of the U-Net, as this diagram illustrates. Also as a brief aside, diffusion models don’t simply predict as much noise as possible and remove it during each step. Instead, they use noise schedulers to control the rate this is performed. You can find a good writeup of that here, as well as a detailed notebook walkthrough here.

Key Difference: Denoising

Now that we’ve given a high-level overview of what these two models are, we can identify the key aspect that separates a diffusion model from a standard transformer.
Looking at the output of the system, we realize that the transformer model isn’t necessarily trying to predict noise, it is trying to predict the next token in a sequence. For the GPT series, this was focused on language, thus the training set consisted of for example wikipedia articles. The training process started with a sentence and the model tried to predict the next word in the sentence, with the loss comparing how close the prediction is to the actual next word.
For diffusion models, however, the prediction aspect of the model isn’t actually the result. Instead, the prediction within the model is noise, which is used in the denoising module to generate the output. In other words, the noise is predicted within the system, then that prediction is used to produce the final output, whether that’s an image, audio clip, etc. The difference is a bit subtle but significant.

Overlap

Denoising is necessary for a diffusion model, but that doesn’t mean that the two models are mutually exclusive. In the initial paper, the diffusion model relies on a U-Net for its backbone, which is a convolutional neural network that takes in the image and downsamples then upsamples the data. However, as has been pointed out, diffusion models can instead use the transformer as a backbone. This happens when attention is added in between the layers, making the transformer (embedding through convolutions followed by attention) the backbone of the model.
This architecture setup is known as a Diffusion-transformer (DiT). A DiT uses transformers in a latent diffusion process, where a simple prior (like Gaussian noise) is gradually transformed into the target image. Stable Diffusion 3 uses a DiT, as seen in this diagram.

Implementation in TorToise and IndexTTS2

Let’s look at some real-world examples by using one of my favorite ML topics: audio generation.

Tortoise

From the TorToise design doc

We’ll first take a look at TorToise, an audio generation text-to-speech system released in 2023.
As described in the paper, TorToise takes the advances made by image generation in autoregressive transformers and DDPMs and applies them to speech synthesis. In this we find an example of when the transformer and diffusion models are separate, as you can see in the architecture drawing above. The transformer first predicts audio codes, then the diffuser generates spectrograms from the codes. So what does this implementation look like?
TorToise uses a transformer to create speech tokens from the encoded speech and text inputs, resulting in “a probabilistic understanding of how text and speech relate to one another given a contextual voice”. It then uses a Diffusion model to decode the compressed speech representations into a wave file. Rather than using a U-net, the diffusion model “is composed of a flat stack of 10 alternating full-attention layers and residual conv blocks. The hidden dimensionality is 1024 with 16 attention heads. This model clocks in at 292M parameters.” (There are three other models in the TorToise system, but we are only focusing on these two for now.)

IndexTTS2

Source

IndexTTS2 (2025) is another high-quality text-to-speech system that has a particular focus on controlling speech duration and on achieving independent control over timbre and emotion. The model comprises three core modules: the Text-to-Semantic (T2S) module, the Semantic-to-Mel (S2M) module, and the Vocoder.
The T2S module uses an autoregressive transformer framework to “generate semantic tokens from text, timbre/style prompts, and an optional speech token count”. The S2M module uses flow-matching modeling to sample from a noise distribution for generating mel-spectrograms. Flow-matching also transforms noise into structured data, though the method is a bit different from denoising diffusion models. We won’t go into detail here, but this article provides a good comparison between the two.
You can see the IndexTTS team’s flow-matching implementation in their repo here; additionally, you should also note their implementation of a diffusion transformer, which could presumably be substituted for the flow-matching module.

Final Thoughts

Transformers and Diffusion models both incorporate the advances in machine learning around latent spaces, encoding/decoding, and attention. Their design and implementation are typically dictated by the nature of the task they are trying to accomplish. Diffusion models are excellent at synthesizing high-fidelity visual and auditory outputs, while transformers thrive on modeling and predicting sequential data. Sometimes though, as seen in the use cases above, combining the two produces a very powerful combination that results in the incredible realism of the generated text, audio, and video we see today.

Latent Spaces and Encoders

A Latent space is “a compressed representation of data points that preserves only essential features that inform the input data’s underlying structure”. These representations have been increasingly used in deep learning models over the last decade with the goal of teaching a model to capture information by transforming the data into the latent space. The general idea is that you reduce the high dimensionality of the data you are training with into a lower dimensional space that can be used to produce your outputs, usually for either classification or generation.

The idea of intermediate representations was already being developed in deep learning neural network architectures like CNNs and RNNs, since the middle layers of these systems can be seen as a form of latent space, but the importance of the concept really started to take off with the rise in popularity of autoencoders. This is because the autoencoder is encoding the data input into an intermediate representation as a separate entity, which the decoder then uses to reconstruct the original result.

Latent spaces got a big boost of popularity from image generation with Latent Diffusion Modeling which was productionized into Stable Diffusion. Instead of operating directly in “pixel space”, this diffusion model first mapped the image to a latent space before completing the denoising process. The results maintained a high quality result, while providing impressive speed improvements as well as better generalizability.

In our domain of interest, generative audio models like VITS and Deep Voice 3 included latent spaces in their systems, whereby the audio inputs (the wave file and text segment) would be transformed to a lower dimensional subspace then transformed back again into the result. Thus the latent space was in a sense serving as a link between text and audio.

Language Modeling

Earlier text-to-speech machine learning systems had focused on models that learned to translate the text into a mel-spectrogram and then decode the audio from there. In this framework, the model tries to learn the conversion mapping from the text to its appropriate audio. Inspired by the success of generative text models such as the different GPT series that model a “language space” to be used in producing long text outputs, researchers decided to approach audio models in a similar way, aiming to model high-level, long-term structure that can be used in generating audio.

The idea of language modeling is to turn the problem from a melspectrogram production then vocoder conversion into a mapping to latent “language space”, and then decode the audio from that latent space. The language space is produced by the audio codecs which create quantized “codes” from the inputs (the text to be turned into audio, as well as any additional speaker information). The theory is that the quantized codes compress all that information about the content of the audio (the actual words) as well as the information about the prosody of the speech into the latent space. With the information encoded, the decoder part of the codec is used to produce the final audio.

Two big research efforts came out around the same time that led the charge in this shift toward language modeling within generative audio: VALL-E from Microsoft at the end of 2022 and AudioLM from Google at the beginning of 2023.

VALL-E aimed to “regard TTS as a conditional language modeling task rather than continuous signal regression”. It did so by making use of latent spaces and by using a ton of data, scaling training data to 60,000 hours of speech.
Source

Source

AudioLM used a two-token approach: semantic tokens from w2v-BERT for capturing “local dependencies (e.g., phonetics in speech, local melody in piano music) and global long-term structure (e.g., language syntax and semantic content in speech; harmony and rhythm in piano music)”; and acoustic tokens from an audio codec for capturing audio waveform structure and details.
Source

Because codecs and the intermediate representations they produce play such an integral role in this language modeling process, let’s take a closer look at the codecs used by these two models and examine in detail what the conversion process and results look like in practice.

Audio Tokens and Codecs

In order to carry out this language modeling, the audio ML models used latent spaces in the form of codecs, algorithms that encode data streams, in this instance the data streams being either the text of the spoken words or the audio itself in the form of a wave file. VALL-E used an algorithm called EnCodec, and AudioLM used one called SoundStream.

EnCodec has three main components: an encoder network, a quantization layer, and a decoder.
The whole system is trained end-to-end and minimizes a reconstruction loss over both the time and frequency domain. This diagram from the EnCodec paper displays the entire pipeline from end to end, showcasing the three components:

Source

SoundStream relies on a model architecture made up of a fully convolutional encoder/decoder network and a residual vector quantizer (RVQ). It also includes adversarial and reconstruction losses to allow the generation of high-quality audio content from quantized embeddings. The encoder receives as input a waveform and produces a sequence of embeddings at a lower sampling rate, which are quantized by the RVQ. The decoder can then reconstruct an approximation of the original waveform.
The diagram below from the SoundStream paper gives an overview of the full system:

Source

The paper also includes a breakdown of the steps of the model:

Source

Code example
To really drive home what an audio codec is doing, let’s walk through what it looks like to encode some audio in python using EnCodec.

First, we’ll load the EnCodec libraries and a sample audio file

from datasets import load_dataset, Audio  
from transformers import EncodecModel, AutoProcessor  
import torch

# dummy dataset, however you can swap this with an dataset on the 🤗 hub or bring your own  
librispeech_dummy = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")

# load the model + processor (for pre-processing the audio)  
model = EncodecModel.from_pretrained("facebook/encodec_24khz")  
processor = AutoProcessor.from_pretrained("facebook/encodec_24khz")

# cast the audio data to the correct sampling rate for the model  
librispeech_dummy = librispeech_dummy.cast_column("audio", Audio(sampling_rate=processor.sampling_rate))  
audio_sample = librispeech_dummy[0]["audio"]["array"]  
print(f"Input audio shape: {audio_sample.shape}")

Input audio shape: (140520,)

With that in place, we can preprocess the inputs and encode the audio sample

inputs = processor(raw_audio=audio_sample, sampling_rate=processor.sampling_rate, return_tensors="pt")

encoder_outputs = model.encode(inputs["input_values"], inputs["padding_mask"])

Examining the results, we see that the encoder has created a tensor in the latent space with a much smaller size than the input tensor. The actual values aren’t interpretible to us, as they represent a result in latent space.

detail_tensor = encoder_outputs['audio_codes']  
print(f"Tensor data type: {detail_tensor.dtype}")  
print(f"Tensor shape: {detail_tensor.shape}")  
print(f"Number of dimensions (rank): {detail_tensor.ndim}")  
print(f"Total number of elements: {detail_tensor.numel()}")

Tensor data type: torch.int64  
Tensor shape: torch.Size([1, 1, 2, 440])  
Number of dimensions (rank): 4  
Total number of elements: 880

To complete the process, we can call the decoder on the encoded value.

audio_values = model.decode(encoder_outputs.audio_codes, encoder_outputs.audio_scales, inputs["padding_mask"])[0]

print(f"Tensor data type: {audio_values.dtype}")  
print(f"Tensor shape: {audio_values.shape}")  
print(f"Number of dimensions (rank): {audio_values.ndim}")  
print(f"Total number of elements: {audio_values.numel()}")

Tensor data type: torch.float32  
Tensor shape: torch.Size([1, 1, 140520])  
Number of dimensions (rank): 3  
Total number of elements: 140520

When we play the resulting audio, we see it is close to the original, but slightly distorted, since the latent space encoding uses a statistical variational process, rather than being purely deterministic.

Moving Forward

With these advancements at the end of 2023/beginning of 2024, language modeling moved forward as the dominant method for TTS. Cutting edge research continues to use this paradigm; for instance, SparkTTS (2025) used a novel tokenizer called BiCodec, separating semantic tokens for meaning and global tokens for speaker content. VibeVoice (2025) used a specialized speech tokenizer as well.
Experimentation with new ideas continues. Some researchers are investigating the move from discrete to continuous codes. Others are experimenting with a return to the simplification of the end-to-end pipeline system, Sesame’s CSM (2025) being a great example of that. Nevertheless, language modeling and audio codecs continue to play a key role in generative modeling of the audio space.

As has taken place across all domains within machine learning, we’ve seen the natural progression of increased scaling in dataset size and model size. Beyond these increased magnitudes though, researchers and engineers have settled on several design decisions aimed at improving the speed and quality of audio generation. The two main topics I want to highlight are the general trends around the simplification of end-to-end pipeline system design, and the utilization of generative models including VAEs, Transformers, Adversarial Networks, and Diffusion models.

Pipeline Simplification

Component-based Architecture

Prior to the shift in using Machine Learning for text to speech, traditional TTS systems performed the conversion by using multi-stage systems made from components such as text analysis, duration modeling, acoustic modeling, and a vocoder. These components would typically be implemented and tuned individually by audio engineers. While it worked well for its time, these systems were often brittle and difficult to expand upon because of their complexity.

Two-Stage Systems

A fundamental paradigm in the growth of ML was the increased emphasis on using the data to do the heavy lifting through neural nets, rather than individually tailored algorithms. To this end, WaveNet (2016) simplified the multi-component pipeline by replacing the vocoder and acoustic model components with a neural network structure; however, the neural nets were still conditioned on linguistic features such as the logarithmic fundamental frequency and phone durations.
Following this, more models continued this trend toward a general two-stage system, where the text was converted to intermediate representations such as mel-spectrograms, and then those intermediate forms were converted into the final waveform. Tacotron (2017) did this by synthesizing speech from characters, while having an intermediate component that produced spectrograms to be passed to a reconstruction algorithm (Griffin-Lim). or a vocoder.

Tacotron Architecture

This was similarly the case with Deep Voice 3 (2017), which fed textual features (e.g. characters, phonemes, stresses) into various vocoder parameters, such as mel-band spectrograms, linear-scale log magnitude spectrograms, fundamental frequency, spectral envelope, and aperiodicity.

This trend gradually continued up to the 2020s, when researchers really tried to lean more fully into getting rid of these component-based systems by replacing them with single stage end-to-end models. These systems would take in waveforms, pass them through transformer/diffusion architectures, then output the speech result.
EATS (2020) was a clear example of this; it operated directly on character or phoneme input sequences and produced raw speech audio output. It was composed of two high-level submodules: an aligner, which processed the raw input sequence and converted them into a latent feature space; and the decoder, which turned those features into 24 kHz audio waveforms.

EATS architecture

VITS (2021) tried to go even further by making a true single stage pipeline using VAEs guided by normalizing flows and an adversarial training process

Audio Tokens and Language Modeling

After experimenting with this strategy for a couple of years, developers took a slight step back toward the component based approach by modeling discrete audio tokens then using those to generate the speech results. This coincided with the move toward adopting transformer architectures (more on that in the next section). Some early examples of these were AudioLM (2022) and VALL-E (2023). VALL-E innovated by replacing intermediate acoustic representations of mel spectrograms with discrete audio codec codes.
VALL-E Architecture

As of Now, the audio token approach is still producing the top results, but many models in 2025 attempted to return to the single stage simplification seen earlier, particularly in single-transformer models like Llasa and end-to-end pipelines like Sesame CSM

Model Technologies

Prior to the mid-2010s, TTS systems relied on classical signal processing and statistical modeling to speech waveforms. As the machine learning and then generative ai space took off, developers incorporated these new algorithms and models to generate more realistic speech results.

CNNs and RNNs

In the second half of the 2010s, the leading edge of ML development took the form of Deep Learning composed of CNNs and RNNs. Consequently, systems like WaveNet, Tacotron, and Deep Voice all incorporated these architectures into their pipelines.

Encoders/Decoders, Transformers, and Adversarial Nets

By the end of the 2010s and going into the 20s, developers started to incorporate the leading generative models into their systems.
Adversarial training was added as a component to the training of several TTS systems, EATS being an early example, followed by VITS.
Additionally, new systems moved from using CNNs and RNNs to making use of encoder-decoder structures and Transformers. DeepVoice 3 in 2018 was an early example of this, it used a convolutional encoder-decoder and was attention-based. Additionally, VITS used VAEs in tandem with the adversarial components mentioned above.
However, the use of Transformers in TTS architecture really started getting more widely adapted around 2022, when the trend moved toward TTS as a language modeling task rather than a direct conversion mechanism. This is because the process of converting data to tokens lent itself naturally to the Transformer’s structure.

Diffusion Models

On top of this, many of the models added a diffusion component on top of the transformer encoder/decoder mechanisms to generate the final audio output and produce a more natural sound. TorToise (2023) and StyleTTS2 (2023) were notable in this regard. This fusion of transformers and diffusion models has continued to the present, with models like IndexTTS2 (2025) using this approach.

Bonus: Autoregressive modeling

Pros and Cons

Autoregressive (AR) models are models where outputs are modeled based on previous values.
Unlike the other trends we’ve highlighted that have a general direction of development, the use of Autoregressive modeling in TTS systems has actually gone back and forth over the years, as developers have weighed the tradeoffs between the two.
AR architectures give high-quality outputs and generally improve results, but they can be slow. They are also sequential and thus inherently unparallelizable Consequently, Non-Autoregressive (NAR) methods may be chosen when speed or performance is of high value, depending on the use case. Additionally, NAR methods can work better with some of the other trends that architectures have taken; in particular the exponential growth of datasets and parameter counts can be enabled by the use of NAR methods, to get the system to actually finish training. However, NAR models have challenges of their own, such as difficulty in maintaining alignment between input text and synthesized speech.

Back and Forth

Starting with WaveNet and continuing on in others such as Tacotron and Deep Voice, an autoregressive architecture was used as it produced positive results.
WaveGlow (2018) was an early attempt at moving away from using an autoregressive system, followed by EATS and VITS. This didn’t take hold too strongly though, and many other systems continued to use AR methods. For instance, VALL-E used an autoregressive model for coarse tokens and a Non-Autoregressive model for fine residual tokens. E2-TTS (2024) on the other hand, was fully Non-Autoregressive.
In the latest stage of events, AR has been incorporated once again, with Llasa, Spark-TTS (2025), and VibeVoice (2025) all incorporating autoregressive components into their system.

Final Thoughts

Here in 2026 there is still much R&D being conducted in the audio generation space. Researchers continue to experiment with architectures to achieve new levels of realism in speech generation. Some other developments we’ve seen recently are:

• Distillation into smaller models (often with Llama3 as backbone). This is in contrast to the increased model sizes mentioned above, and is of interest because it shows high quality outputs can be produced from much smaller inference
• Compute resources strategies, in the same vein as inference-time focus with reasoning models. Llasa demonstrates that scaling training-time and inference-time compute can significantly refine prosody and accuracy
• Multilanguage; largely a component of the increased data size as well as the language modeling paradigm shift

Stay tuned for future posts, where we’ll take a closer look at some of the design components we’ve discussed, and eventually take a deeper dive into the models themselves.

Audio data has always captured my interest the most out of the various machine learning modalities. This is largely because of its seeming esoteric nature, particularly when it comes to speech vocalizations. If you were to look at a waveform plot or a spectrogram, it isn’t readily apparent how these data outputs map to words and phrases. With the explosive growth of generative machine learning in the past decade, I’ve tried to keep informed about the development of audio modeling, especially Text-to-Speech (TTS) systems.

The way I see it, audio has been the overlooked third child in the family of modalities, with its older siblings of text and image taking much of the spotlight. Technical challenges may have contributed to this: the data rate for audio is significantly higher than text, and there is a one-to-many relationship between text and audio, meaning that the same sentence can be rendered by different speakers with different speaking styles, emotional content, and recording conditions. Or maybe the lack of focus has just been because of people’s tastes. Whatever the reason, the capabilities of machine learning to generate audio recordings is only now reaching the level of astonishing realism we’ve gotten from text and image generation.

This post walks through the major Text-to-Speech models focused on or contributing to this development of audio generation, starting back in 2016. There are plenty of forks and project-spinoffs out there, but these are openly published (loosely speaking) models I’ve found to have gotten significant attention from their innovations or the resources devoted to them, and have led the way to our current state of the art. In later posts, I plan to compare/contrast the methodology and architectures these models use, and then deep dive into some specific models that I think are particularly interesting.

So let’s go back in time, starting in a year before we realized that Attention was all we needed…

2016

WaveNet

Starting things off in the pre-transformer days, Google’s WaveNet was the first major model to successfully use the raw waveform of the audio signal for input data, as opposed to feature engineering acoustic features. It uses an autoregressive architecture.
This model was a significant advancement over the then state-of-the-art methods built with parametric and concatenative systems, and was able to generate English and Mandarin speech.

2017

Deep Voice

Baidu iterated upon Deep Voice with some modifications and optimizations, first releasing Deep Voice 2 in March 2017. This was followed by a complete architectural redesign resulting in Deep Voice 3, released October 2017. The redesign transformed the system from a combination of multiple neural networks into a single encoder-decoder design.

Tacotron

Another research development from Google, this system aimed to generate speech directly from text characters, rather than using the more prevalent at the time approach of multiple pipeline components including a linguistic feature extractor, vocoder, etc. The model took in <text,audio> pairs, transformed them into spectrograms, and trained on that data to produce waveforms.
This sequence-to-sequence model uses an encoder-decoder architecture and an attention mechanism, potentially making it the first major text-to-speech transformer model.

2018

WaveGlow

Submitted October 2018, this flow-based network model was built by Nvidia and used ideas from WaveNet and OpenAI’s Glow. Similar models include Glow-TTS and Flow-TTS. The system uses a diffusion model, taking mel-spectrograms as inputs, and is implemented with a single network for greater simplicity.
One of the key design decisions of WaveGlow was to move away from an autoregressive system. While useful, autoregressive TTS systems are inherently sequential so have the downside of not being very parallelizable.

2019

wav2vec

Released by Meta in April 2019 and followed by wav2Vec 2.0 in 2020, this system uses raw, unlabeled audio to train automatic speech recognition models. It was trained by predicting speech units for masked parts of the audio, learning basic units that are 25ms long to enable learning of high-level contextualized representations

2020

EATS (End-to-End Adversarial Text-to-Speech)

Released by DeepMind in June 2020, this system uses phonemes in an “end-to-end” manner, rather than having multiple processing stages (such as text normalisation, aligned linguistic featurization, mel-spectrogram synthesis, and raw audio waveform synthesis) The drawback of these multi-component systems is that they need extra training and tuning for each step of the pipeline, however the results appeared very promising.

2021

VITS

Released June 2021, this model also tried to implement a parallel end-to-end autoencoder system that matched the quality of two-stage systems. It used variational inference augmented with normalizing flows and an adversarial training process.
This was followed by VITS2 in 2023, which implemented some structure and training improvements

HuBERT(Hidden-Unit BERT)

While not technically a TTS system, this model is worth highlighting because of its use in other systems.
Basically, this was Meta’s development of BERT for audio. Like, BERT, HuBERT, gets used in other audio models and processing tasks, such as RVC
At a high level, HuBERT takes in a float array corresponding to the raw waveform of the speech signal and passes it to a wav2vec2.0 model to encode it. The final output is a feature sequence, and on that output the model uses a clustering step to create “hidden units”.
Read here for a great in-depth examination of this system.

2022

2022 was the year that gen AI really hit the public consciousness, with the releases of Dall-E 2 and ChatGPT. While primary focus was on textual and visual results, this year still saw some very interesting experimentation and development in the audio space.

StyleTTS

The StyleTTS paper was initially submitted May 2022 and was followed-up by the StyleTTS2 paper in June 2023 along with a code implementation. The goal was to develop a better parallel TTS system that improves on the deficiencies in other systems around aligning monotonic speech segments. The system uses an encoder-decoder architecture to generate a mel-spectrogram from text while incorporating parallel encoder blocks to be able to incorporate information around “style”, e.g. prosodic patterns and emotional tone. It also introduced an innovation called a Transferable Monotonic Aligner (TMA) to set the duration generation.
This system has given rise to some interesting forks, fine-tunes, and eventual offshoots.

AudioLM

Released by Google in October 2022. This model aims to generate realistic speech (and piano music) by listening to audio only.
The system extracts semantic tokens using w2v-BERT, as well as acoustic tokens produced by a SoundStream neural codec, and passes these tokens through multiple transformer models. To get the final result, the SoundStream decoder converts the encoded tokens into a waveform. This was done for research purposes only, so although the paper described the details of the architecture, the weights and code were not released.

2023

With the sky opened and the belief in GenAI soaring, 2023 was a year when the amount of new TTS models really started to take off, with some very impressive and innovative systems appearing that still see a little bit of use today.

Piper

This is a TTS engine and a set of voice packs built by the Piper Project, from the Open Home Foundation. It is a small model that uses the VITS architecture. The initial release came in January 2023, and it continues to be open-sourced and developed.

ParlerTTS

A reproduction of the work described in the paper Natural language guidance of high-fidelity text-to-speech with synthetic annotations, Parler-TTS is an auto-regressive transformer-based model, with the aim of being lightweight and producing natural-sounding speech.

VALL-E

The next offering from Microsoft would appear in 2023 as VALL-E. Unlike other text-to-speech methods that typically synthesize speech by manipulating waveforms, VALL-E generates discrete audio codec codes from text and acoustic prompts. It basically analyzes how a person sounds, breaks that information into discrete components (called “tokens”) using EnCodec, and uses training data to match what it “knows” about how that voice would sound if it spoke other phrases outside of the three-second sample.
It was a pioneer in trying out language modeling for zero-shot TTS (TTS not trained for one specific speaker).

Bark

This transformer based model was open-sourced by suno.ai in April 2023. The system consists of three components that form a pipeline: text-to-semantic tokens, semantic-to-coarse tokens, and coarse-to-fine tokens.
This model was aimed toward producing more natural-sounding speech, as results generated contain non-verbal aspects such as laughing, as well as verbal tics of “umm”, “uh”, etc.
While no longer maintained, this model can still be useful as a reference point or for research purposes.

TorToiSe

This model is where I think we begin to see the ice breaking around truly impressive speech generation. Rumored by many to be the basis of ElevenLabs’ technological suite, the TorToiSe model, named thus to underscore how slow it is, uses both an autoregressive encoder-decoder as well as a diffusion component to enhance its ability to model in the continuous domain and focus on reproducing human voices realistically.
TorToiSe provides the autoregressive generator and DDPM with a “speech conditioning input”, a unique design choice intended to allow the model to infer vocal characteristics like tone and prosody. In addition to the paper, you can read more about the design details here.

AudioPaLM

Released June 2023, this system combines two Google language models, PaLM-2 and AudioLM, into a multimodal architecture. It inherits the speaker identification and intonation copying capabilities from AudioLM, and the linguistic abilities of LLM from PaLM-2.

SeamlessM4T

Released by Meta in August 2023, this model performs text to speech as well as speech-to-speech and speech-to-text.
An all-in-one model, it uses self-supervised speech representations with w2v-BERT 2.0, then uses the aligned the speech translations for training.

2024

Riding high on the momentum of 2023, this year saw even greater expansion and improvements. While companies such as OpenAI and Anthropic deployed closed systems with multimodal capabilities that could produce very impressive audio, open-source developments and modifications continued to emerge and push the space forward.

GPT-SoVITS-v2

Connected to the team involved with the RVC voice changer and released in February 2024, this model generates acoustic tokens with a seq2seq model then converts those tokens into waveforms. Deep dives can be found here and here.

Metavoice

Released in February 2024, this model predicts EnCodec tokens from text, and uses speaker information to condition the input at the token embedding layer. It then uses multi-band diffusion to generate waveforms from the EnCodec tokens, and concludes by passing the result through a DeepFilterNet to help clear up artifacts. The system also supports KV-caching via Flash Decoding and batching.

MeloTTS

A multi-lingual text-to-speech library by MIT and MyShell.ai, this system was first released in February 2024.
The implementation is based on TTS, VITS, VITS2 and Bert-VITS2. The open-source codebase also provides some boilerplate to let you train and fine-tune the model on your own data.

E2-TTS

Another model from Microsoft Research, this non-autoregressive system was released June 2024. It uses only two modules, the flow-matching Transformer and the vocoder, eschewing additional components such as a duration model or grapheme-to-phoneme converter. It also does not use monotonic alignment or cross-attention.
Despite detailing the architecture in the paper, this system was not open-sourced, but it was reverse-engineered by the F5-TTS team as well as some others.

XTTS

Coming out in June 2024, this system built upon the TorToiSe model, with some modifications.
It aimed to expand language capabilities much wider to cover 16 languages, as well as provide cross-language zero-shot TTS without needing a parallel training dataset.
The system takes in a mel-spectrogram as input, and it makes use of a VQ-VAE, the GPT-2 encoder, as well as a HifiGAN model to compute the final audio signal
A deeper dive into the system can be found here.

F5-TTS

Released in October 2024, F5 is a non-autoregressive text-to-speech system based on flow matching with a Diffusion Transformer. It follows in the flow-matching footsteps of E2-TTS but tries to make the design easier to follow and improves training time and performance. F5 also adds some additional innovations such as an inference-time Sway Sampling strategy.

2025

While the models of 2024 produced impressive results, 2025 may have been the year when the uncanny valley was crossed. The models being released could not only produce natural-sounding audio, but they could also convincingly transfer outputs across accents or even languages.
This year also saw more companies popping up as competitors to ElevenLabs, either centered on audio generation or incorporating it into multimodal AI agents and general world models. Furthermore, models such as Kokoro and Spark-TTS would be released that could produce fast and impressive results from extremely small models.

Kokoro

This model is built with StyleTTS2 as a backbone, though it’s not properly a distillation because it also removes parts of the original base model such as several utility models used during training in order to make it smaller. Although only the inference code and weights were released, this system got a lot of attention for its very small size while still producing impressive results.
In addition to the trimming described above, another reason for its small model parameter count is that it doesn’t use a dynamic style diffusion method and instead uses ‘voicepacks’ for voice style transfer. Additionally, Kokoro outsources tokenization and phonemization, so that the only thing the model sees is phonemes and punctuation, further reducing the size.

Llasa

Released around the beginning of February, the team behind Llasa had two main goals:

• Explore the scaling of train-time and inference-time compute for speech synthesis
• Provide a simplified framework with single-layer vector quantizer (VQ) codec and a single Transformer architecture

The researchers argued that a simplified architecture, like the transformer for text LLMs, allows for faster scientific exploration and progress. The scaling of train-time compute (e.g., increasing model size or training data) also consistently improves the naturalness of and expressiveness of the results.
The system first uses Llama 3B (could also use 1B or 8B) to predict “speech tokens from the XCodec2 codebook, and then decodes these tokens into audio.

Zonos

The Zonos models were first released in February 2025 by Zyphra, a company whose mission is “to build human-aligned AI that helps individuals and organizations reach their fullest potential”.
The company released two models, the 1.6B transformer and 1.6B hybrid (SSM model, first open-source of its kind for TTS) under an Apache 2.0 license, with the goal of highly expressive and natural-sounding audio that attempts to capture emotional variation.
The transformer model uses an autoregressive pipeline, and makes use of audio tokens from DAC vocoder. It also takes a speaker embedding as an input for voice cloning capabilities, as well as other inputs such as speaking rate, pitch, sample rate, audio quality, and speaker emotions, producing speech outputs that are 44KHz.
While it produces high quality outputs, some weaknesses include: audio artifacts at the beginning and end of generations; mistakes in text alignment (skipping words, repeated words, etc); and high quality but slower and more expensive inference. The model attempts to mitigate these by implementing some techniques such as: a delay codebook pattern from Parler TTS, multi-token prediction, and embedding merging.

Sesame CSM

With Sesame we see another company moving into the audio-generation field aimed at achieving as natural an experience as possible, as if you were actually speaking to someone in the real world. Their goal is to achieve “voice presence”, and the key component aims are: emotional awareness, conversational dynamics, contextual awareness, and consistent personality.
The Conversational Speech Model (CSM) was released February 2025. The CSM has three sizes: Tiny (1B backbone, 100M decoder), Small (3B backbone, 250M decoder), and Medium (8B backbone, 300M decoder).
Sesame’s CSM is a single-stage model, as opposed to a model using a two-step process of generating semantic tokens followed by acoustic tokens (commonly with RVQ). The downside of a two-step is that it is difficult to ensure the semantic tokens fully capture the prosody during training. Also, RVQ can have a high latency.
The model incorporates two autoregressive transformers, both variants of the Llama architecture. This requires a high level of memory, so the system uses compute amortization to help alleviate the memory footprint.
More detailed info can be found in the release announcement.

IndexTTS

Released in Feb 2025 by the Index SpeechTeam, this system was based on XTTS and aimed at improvements over other popular TTS systems by utilizing a simpler training process, more controllable usage, and faster inference speed. This was followed by IndexTTS2 later in the year, which introduced some innovative techniques addressing the issue of audio-visual synchronization, with a primary use case of video dubbing. While still using an autoregressive model, this system makes use of methods for speech duration control and gives independent control over timbre and emotion.

Spark-TTS

Built using Qwen 2.5, this system is similar to Llasa but much smaller and with great results. Instead of relying on separate models to generate acoustic features, it directly reconstructs audio from the code predicted by the LLM.

Orpheus

Released in March by Canopy Labs, these models come in four different sizes, ranging from 3 billion parameters to 150 million parameters. The architecture uses Llama-3b as the backbone but utilizes two nonstandard design decisions: 7 tokens per sample frame decoded as a single flattened sequence, and using a non-streaming (CNN-based) tokenizer. The weights and code are all open-sourced under an Apache-2.0 license.

OuteTTS

Though some initial versions of OuteTTS started to get released at the end of 2024, this system went through a couple of iterations until OuteAI released the first major version of their toolsuite utilizing these models in June 2025. As of the end of 2025, the most advanced model is the Llama-OuteTTS-1.0-1B, While details about any innovations are scarce, this gives yet another example of an organization attempting to build a software product around audio generation while open-sourcing their TTS models.

VibeVoice

An open-source development from Microsoft, this model was released to the public in August and comes in three sizes: 0.5B, 1.5B, and Large. It supports streaming text input as well as long-form generation. It uses an “interleaved, windowed design”, where the text chunks are encoded in parallel as they come in while the audio is generated from a diffuse-based model.

Maya1

For the final entry in our timeline, another model built with the Llama-3B backbone was released in October from Maya Research. This model uses a 3B-parameter decoder-only transformer (Llama-style) to predict SNAC neural codec tokens instead of raw waveforms. Besides the small size, the emphasis here is on creating emotionally expressive outputs of any type of voice described. The result is an impressive showing from a small research group and is one to keep an eye on in the future.

Closing Thoughts

With this, we’ve made our way to the beginning of 2026. I’m sure there are many models that I’ve missed, but I tried to capture important or particularly interesting ones that have shaped the conversation over the past decade.
I hope you’ve enjoyed this high-level walkthrough of TTS history. Stay tuned for an upcoming post, where I’ll take a stab at digging into some of the major design decisions and tradeoffs that have shaped this history. Happy New Year everyone!