A Decade of Innovation

Audio data has always captured my interest the most out of the various machine learning modalities. This is largely because of its seeming esoteric nature, particularly when it comes to speech vocalizations. If you were to look at a waveform plot or a spectrogram, it isn’t readily apparent how these data outputs map to words and phrases. With the explosive growth of generative machine learning in the past decade, I’ve tried to keep informed about the development of audio modeling, especially Text-to-Speech (TTS) systems.

The way I see it, audio has been the overlooked third child in the family of modalities, with its older siblings of text and image taking much of the spotlight. Technical challenges may have contributed to this: the data rate for audio is significantly higher than text, and there is a one-to-many relationship between text and audio, meaning that the same sentence can be rendered by different speakers with different speaking styles, emotional content, and recording conditions. Or maybe the lack of focus has just been because of people’s tastes. Whatever the reason, the capabilities of machine learning to generate audio recordings is only now reaching the level of astonishing realism we’ve gotten from text and image generation.

This post walks through the major Text-to-Speech models focused on or contributing to this development of audio generation, starting back in 2016. There are plenty of forks and project-spinoffs out there, but these are openly published (loosely speaking) models I’ve found to have gotten significant attention from their innovations or the resources devoted to them, and have led the way to our current state of the art. In later posts, I plan to compare/contrast the methodology and architectures these models use, and then deep dive into some specific models that I think are particularly interesting.

So let’s go back in time, starting in a year before we realized that Attention was all we needed…

2016

WaveNet

Starting things off in the pre-transformer days, Google’s WaveNet was the first major model to successfully use the raw waveform of the audio signal for input data, as opposed to feature engineering acoustic features. It uses an autoregressive architecture.
This model was a significant advancement over the then state-of-the-art methods built with parametric and concatenative systems, and was able to generate English and Mandarin speech.

2017

Deep Voice

Baidu iterated upon Deep Voice with some modifications and optimizations, first releasing Deep Voice 2 in March 2017. This was followed by a complete architectural redesign resulting in Deep Voice 3, released October 2017. The redesign transformed the system from a combination of multiple neural networks into a single encoder-decoder design.

Tacotron

Another research development from Google, this system aimed to generate speech directly from text characters, rather than using the more prevalent at the time approach of multiple pipeline components including a linguistic feature extractor, vocoder, etc. The model took in <text,audio> pairs, transformed them into spectrograms, and trained on that data to produce waveforms.
This sequence-to-sequence model uses an encoder-decoder architecture and an attention mechanism, potentially making it the first major text-to-speech transformer model.

2018

WaveGlow

Submitted October 2018, this flow-based network model was built by Nvidia and used ideas from WaveNet and OpenAI’s Glow. Similar models include Glow-TTS and Flow-TTS. The system uses a diffusion model, taking mel-spectrograms as inputs, and is implemented with a single network for greater simplicity.
One of the key design decisions of WaveGlow was to move away from an autoregressive system. While useful, autoregressive TTS systems are inherently sequential so have the downside of not being very parallelizable.

2019

wav2vec

Released by Meta in April 2019 and followed by wav2Vec 2.0 in 2020, this system uses raw, unlabeled audio to train automatic speech recognition models. It was trained by predicting speech units for masked parts of the audio, learning basic units that are 25ms long to enable learning of high-level contextualized representations

2020

EATS (End-to-End Adversarial Text-to-Speech)

Released by DeepMind in June 2020, this system uses phonemes in an “end-to-end” manner, rather than having multiple processing stages (such as text normalisation, aligned linguistic featurization, mel-spectrogram synthesis, and raw audio waveform synthesis) The drawback of these multi-component systems is that they need extra training and tuning for each step of the pipeline, however the results appeared very promising.

2021

VITS

Released June 2021, this model also tried to implement a parallel end-to-end autoencoder system that matched the quality of two-stage systems. It used variational inference augmented with normalizing flows and an adversarial training process.
This was followed by VITS2 in 2023, which implemented some structure and training improvements

HuBERT(Hidden-Unit BERT)

While not technically a TTS system, this model is worth highlighting because of its use in other systems.
Basically, this was Meta’s development of BERT for audio. Like, BERT, HuBERT, gets used in other audio models and processing tasks, such as RVC
At a high level, HuBERT takes in a float array corresponding to the raw waveform of the speech signal and passes it to a wav2vec2.0 model to encode it. The final output is a feature sequence, and on that output the model uses a clustering step to create “hidden units”.
Read here for a great in-depth examination of this system.

2022

2022 was the year that gen AI really hit the public consciousness, with the releases of Dall-E 2 and ChatGPT. While primary focus was on textual and visual results, this year still saw some very interesting experimentation and development in the audio space.

StyleTTS

The StyleTTS paper was initially submitted May 2022 and was followed-up by the StyleTTS2 paper in June 2023 along with a code implementation. The goal was to develop a better parallel TTS system that improves on the deficiencies in other systems around aligning monotonic speech segments. The system uses an encoder-decoder architecture to generate a mel-spectrogram from text while incorporating parallel encoder blocks to be able to incorporate information around “style”, e.g. prosodic patterns and emotional tone. It also introduced an innovation called a Transferable Monotonic Aligner (TMA) to set the duration generation.
This system has given rise to some interesting forks, fine-tunes, and eventual offshoots.

AudioLM

Released by Google in October 2022. This model aims to generate realistic speech (and piano music) by listening to audio only.
The system extracts semantic tokens using w2v-BERT, as well as acoustic tokens produced by a SoundStream neural codec, and passes these tokens through multiple transformer models. To get the final result, the SoundStream decoder converts the encoded tokens into a waveform. This was done for research purposes only, so although the paper described the details of the architecture, the weights and code were not released.

2023

With the sky opened and the belief in GenAI soaring, 2023 was a year when the amount of new TTS models really started to take off, with some very impressive and innovative systems appearing that still see a little bit of use today.

Piper

This is a TTS engine and a set of voice packs built by the Piper Project, from the Open Home Foundation. It is a small model that uses the VITS architecture. The initial release came in January 2023, and it continues to be open-sourced and developed.

ParlerTTS

A reproduction of the work described in the paper Natural language guidance of high-fidelity text-to-speech with synthetic annotations, Parler-TTS is an auto-regressive transformer-based model, with the aim of being lightweight and producing natural-sounding speech.

VALL-E

The next offering from Microsoft would appear in 2023 as VALL-E. Unlike other text-to-speech methods that typically synthesize speech by manipulating waveforms, VALL-E generates discrete audio codec codes from text and acoustic prompts. It basically analyzes how a person sounds, breaks that information into discrete components (called “tokens”) using EnCodec, and uses training data to match what it “knows” about how that voice would sound if it spoke other phrases outside of the three-second sample.
It was a pioneer in trying out language modeling for zero-shot TTS (TTS not trained for one specific speaker).

Bark

This transformer based model was open-sourced by suno.ai in April 2023. The system consists of three components that form a pipeline: text-to-semantic tokens, semantic-to-coarse tokens, and coarse-to-fine tokens.
This model was aimed toward producing more natural-sounding speech, as results generated contain non-verbal aspects such as laughing, as well as verbal tics of “umm”, “uh”, etc.
While no longer maintained, this model can still be useful as a reference point or for research purposes.

TorToiSe

This model is where I think we begin to see the ice breaking around truly impressive speech generation. Rumored by many to be the basis of ElevenLabs’ technological suite, the TorToiSe model, named thus to underscore how slow it is, uses both an autoregressive encoder-decoder as well as a diffusion component to enhance its ability to model in the continuous domain and focus on reproducing human voices realistically.
TorToiSe provides the autoregressive generator and DDPM with a “speech conditioning input”, a unique design choice intended to allow the model to infer vocal characteristics like tone and prosody. In addition to the paper, you can read more about the design details here.

AudioPaLM

Released June 2023, this system combines two Google language models, PaLM-2 and AudioLM, into a multimodal architecture. It inherits the speaker identification and intonation copying capabilities from AudioLM, and the linguistic abilities of LLM from PaLM-2.

SeamlessM4T

Released by Meta in August 2023, this model performs text to speech as well as speech-to-speech and speech-to-text.
An all-in-one model, it uses self-supervised speech representations with w2v-BERT 2.0, then uses the aligned the speech translations for training.

2024

Riding high on the momentum of 2023, this year saw even greater expansion and improvements. While companies such as OpenAI and Anthropic deployed closed systems with multimodal capabilities that could produce very impressive audio, open-source developments and modifications continued to emerge and push the space forward.

GPT-SoVITS-v2

Connected to the team involved with the RVC voice changer and released in February 2024, this model generates acoustic tokens with a seq2seq model then converts those tokens into waveforms. Deep dives can be found here and here.

Metavoice

Released in February 2024, this model predicts EnCodec tokens from text, and uses speaker information to condition the input at the token embedding layer. It then uses multi-band diffusion to generate waveforms from the EnCodec tokens, and concludes by passing the result through a DeepFilterNet to help clear up artifacts. The system also supports KV-caching via Flash Decoding and batching.

MeloTTS

A multi-lingual text-to-speech library by MIT and MyShell.ai, this system was first released in February 2024.
The implementation is based on TTS, VITS, VITS2 and Bert-VITS2. The open-source codebase also provides some boilerplate to let you train and fine-tune the model on your own data.

E2-TTS

Another model from Microsoft Research, this non-autoregressive system was released June 2024. It uses only two modules, the flow-matching Transformer and the vocoder, eschewing additional components such as a duration model or grapheme-to-phoneme converter. It also does not use monotonic alignment or cross-attention.
Despite detailing the architecture in the paper, this system was not open-sourced, but it was reverse-engineered by the F5-TTS team as well as some others.

XTTS

Coming out in June 2024, this system built upon the TorToiSe model, with some modifications.
It aimed to expand language capabilities much wider to cover 16 languages, as well as provide cross-language zero-shot TTS without needing a parallel training dataset.
The system takes in a mel-spectrogram as input, and it makes use of a VQ-VAE, the GPT-2 encoder, as well as a HifiGAN model to compute the final audio signal
A deeper dive into the system can be found here.

F5-TTS

Released in October 2024, F5 is a non-autoregressive text-to-speech system based on flow matching with a Diffusion Transformer. It follows in the flow-matching footsteps of E2-TTS but tries to make the design easier to follow and improves training time and performance. F5 also adds some additional innovations such as an inference-time Sway Sampling strategy.

2025

While the models of 2024 produced impressive results, 2025 may have been the year when the uncanny valley was crossed. The models being released could not only produce natural-sounding audio, but they could also convincingly transfer outputs across accents or even languages.
This year also saw more companies popping up as competitors to ElevenLabs, either centered on audio generation or incorporating it into multimodal AI agents and general world models. Furthermore, models such as Kokoro and Spark-TTS would be released that could produce fast and impressive results from extremely small models.

Kokoro

This model is built with StyleTTS2 as a backbone, though it’s not properly a distillation because it also removes parts of the original base model such as several utility models used during training in order to make it smaller. Although only the inference code and weights were released, this system got a lot of attention for its very small size while still producing impressive results.
In addition to the trimming described above, another reason for its small model parameter count is that it doesn’t use a dynamic style diffusion method and instead uses ‘voicepacks’ for voice style transfer. Additionally, Kokoro outsources tokenization and phonemization, so that the only thing the model sees is phonemes and punctuation, further reducing the size.

Llasa

Released around the beginning of February, the team behind Llasa had two main goals:

  • Explore the scaling of train-time and inference-time compute for speech synthesis
  • Provide a simplified framework with single-layer vector quantizer (VQ) codec and a single Transformer architecture

The researchers argued that a simplified architecture, like the transformer for text LLMs, allows for faster scientific exploration and progress. The scaling of train-time compute (e.g., increasing model size or training data) also consistently improves the naturalness of and expressiveness of the results.
The system first uses Llama 3B (could also use 1B or 8B) to predict “speech tokens from the XCodec2 codebook, and then decodes these tokens into audio.

Zonos

The Zonos models were first released in February 2025 by Zyphra, a company whose mission is “to build human-aligned AI that helps individuals and organizations reach their fullest potential”.
The company released two models, the 1.6B transformer and 1.6B hybrid (SSM model, first open-source of its kind for TTS) under an Apache 2.0 license, with the goal of highly expressive and natural-sounding audio that attempts to capture emotional variation.
The transformer model uses an autoregressive pipeline, and makes use of audio tokens from DAC vocoder. It also takes a speaker embedding as an input for voice cloning capabilities, as well as other inputs such as speaking rate, pitch, sample rate, audio quality, and speaker emotions, producing speech outputs that are 44KHz.
While it produces high quality outputs, some weaknesses include: audio artifacts at the beginning and end of generations; mistakes in text alignment (skipping words, repeated words, etc); and high quality but slower and more expensive inference. The model attempts to mitigate these by implementing some techniques such as: a delay codebook pattern from Parler TTS, multi-token prediction, and embedding merging.

Sesame CSM

With Sesame we see another company moving into the audio-generation field aimed at achieving as natural an experience as possible, as if you were actually speaking to someone in the real world. Their goal is to achieve “voice presence”, and the key component aims are: emotional awareness, conversational dynamics, contextual awareness, and consistent personality.
The Conversational Speech Model (CSM) was released February 2025. The CSM has three sizes: Tiny (1B backbone, 100M decoder), Small (3B backbone, 250M decoder), and Medium (8B backbone, 300M decoder).
Sesame’s CSM is a single-stage model, as opposed to a model using a two-step process of generating semantic tokens followed by acoustic tokens (commonly with RVQ). The downside of a two-step is that it is difficult to ensure the semantic tokens fully capture the prosody during training. Also, RVQ can have a high latency.
The model incorporates two autoregressive transformers, both variants of the Llama architecture. This requires a high level of memory, so the system uses compute amortization to help alleviate the memory footprint.
More detailed info can be found in the release announcement.

IndexTTS

Released in Feb 2025 by the Index SpeechTeam, this system was based on XTTS and aimed at improvements over other popular TTS systems by utilizing a simpler training process, more controllable usage, and faster inference speed. This was followed by IndexTTS2 later in the year, which introduced some innovative techniques addressing the issue of audio-visual synchronization, with a primary use case of video dubbing. While still using an autoregressive model, this system makes use of methods for speech duration control and gives independent control over timbre and emotion.

Spark-TTS

Built using Qwen 2.5, this system is similar to Llasa but much smaller and with great results. Instead of relying on separate models to generate acoustic features, it directly reconstructs audio from the code predicted by the LLM.

Orpheus

Released in March by Canopy Labs, these models come in four different sizes, ranging from 3 billion parameters to 150 million parameters. The architecture uses Llama-3b as the backbone but utilizes two nonstandard design decisions: 7 tokens per sample frame decoded as a single flattened sequence, and using a non-streaming (CNN-based) tokenizer. The weights and code are all open-sourced under an Apache-2.0 license.

OuteTTS

Though some initial versions of OuteTTS started to get released at the end of 2024, this system went through a couple of iterations until OuteAI released the first major version of their toolsuite utilizing these models in June 2025. As of the end of 2025, the most advanced model is the Llama-OuteTTS-1.0-1B, While details about any innovations are scarce, this gives yet another example of an organization attempting to build a software product around audio generation while open-sourcing their TTS models.

VibeVoice

An open-source development from Microsoft, this model was released to the public in August and comes in three sizes: 0.5B, 1.5B, and Large. It supports streaming text input as well as long-form generation. It uses an “interleaved, windowed design”, where the text chunks are encoded in parallel as they come in while the audio is generated from a diffuse-based model.

Maya1

For the final entry in our timeline, another model built with the Llama-3B backbone was released in October from Maya Research. This model uses a 3B-parameter decoder-only transformer (Llama-style) to predict SNAC neural codec tokens instead of raw waveforms. Besides the small size, the emphasis here is on creating emotionally expressive outputs of any type of voice described. The result is an impressive showing from a small research group and is one to keep an eye on in the future.

Closing Thoughts

With this, we’ve made our way to the beginning of 2026. I’m sure there are many models that I’ve missed, but I tried to capture important or particularly interesting ones that have shaped the conversation over the past decade.
I hope you’ve enjoyed this high-level walkthrough of TTS history. Stay tuned for an upcoming post, where I’ll take a stab at digging into some of the major design decisions and tradeoffs that have shaped this history. Happy New Year everyone!