{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "UFjxB7vK2Prz", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 1. Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ysLvHZRO4uN2", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# Install NeMo library. If you are running locally (rather than on Google Colab), comment out the below lines\n", "# and instead follow the instructions at https://github.com/NVIDIA/NeMo#Installation\n", "BRANCH = 'r1.17.0'\n", "!python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "O-HRFHBb_RDH", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# Download local version of NeMo scripts. If you are running locally and want to use your own local NeMo code,\n", "# comment out the below lines and set NEMO_DIR to your local path.\n", "NEMO_DIR = 'nemo'\n", "!git clone https://github.com/NVIDIA/NeMo.git $NEMO_DIR" ] }, { "cell_type": "markdown", "metadata": { "id": "sgqwl2ycC1Sh", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 2. Introduction to TTS" ] }, { "cell_type": "markdown", "metadata": { "id": "UYHRrdrXHe28", "pycharm": { "name": "#%% md\n" } }, "source": [ "This notebook provides a high level overview of text-to-speech (TTS). It will cover high level concepts and discuss each component in a standard TTS pipeline, providing relevant examples and code snippets using [NeMo](https://github.com/NVIDIA/NeMo)." ] }, { "cell_type": "markdown", "metadata": { "id": "8meIFtgWHmxt", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 3. What is TTS?" ] }, { "cell_type": "markdown", "metadata": { "id": "6t8rK7L2HpZd", "pycharm": { "name": "#%% md\n" } }, "source": [ "**Text-to-speech**, also known as **TTS** or **speech synthesis**, refers to a system by which a computer reads text aloud. Typically the synthesized audio resembles a realistic human voice.\n", "\n", "Most TTS models sound like the voice of the speaker whose audio it is trained on. Though some more recently developed algorithms have the potential to sound like real speakers they were not trained on, or sound like entirely new voices.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "MC9oz1kiHqsW", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 4. The TTS pipeline" ] }, { "cell_type": "markdown", "metadata": { "id": "ODp1OnA0SYdF", "pycharm": { "name": "#%% md\n" } }, "source": [ "Modern TTS systems are fairly complex, with an end to end pipeline consisting of several components that each require their own model or heuristics.\n", "\n", "A standard pipeline might look like:\n", "\n", "1. **Text Normalization**: Converting raw text to spoken text (eg. \"Mr.\" → \"mister\").
\n", "2. **Grapheme to Phoneme conversion (G2P)**: Convert basic units of text (ie. graphemes/characters) to basic units of spoken language (ie. phonemes).\n", "3. **Spectrogram Synthesis**: Convert text/phonemes into a spectrogram.\n", "4. **Audio Synthesis**: Convert spectrogram into audio. Also known as **spectrogram inversion**. Models which do this are called **vocoders**.\n", "\n", "While this is the most common structure, there may be fewer or additional steps depending on the use case. For example, some languages do not require G2P and can instead rely on the model to convert raw text/graphemes to spectrogram.\n", "\n", "
\n", "\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "XRWiZq5PgNCK", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 5. Text Normalization (TN)" ] }, { "cell_type": "markdown", "metadata": { "id": "YFJdiLaPgQAD", "pycharm": { "name": "#%% md\n" } }, "source": [ "The first step is to pre-process the raw input text into a standardized form. Some examples of processing steps could include:\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
Normalization TypeInputOutput
AbbreviationsMr.mister\n", "
AcronymsTTStext to speech\n", "
Numbers42forty two
Decimals1.2one point two
Roman NumeralsVIIseventh
Cardinal DirectionsN E S Wnorth east south west
URLwww.github.comw w w dot github dot com
\n", "\n", "The above examples may be slightly different than the output of the NeMo text normalization code. More details on NeMo text normalization can be found in the [TN documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/nlp/text_normalization/intro.html).\n", "\n", "A more comprehensive list of text normalization rules, examples, and languages are available in the [code](https://github.com/NVIDIA/NeMo-text-processing/tree/main/nemo_text_processing/text_normalization).\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "e_vqDYu9yVFT", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 5.1 TN example" ] }, { "cell_type": "markdown", "metadata": { "id": "EptHrRZeBH46", "pycharm": { "name": "#%% md\n" } }, "source": [ "After installing NeMo, we can do text normalization with just a few lines of code." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "pQw-3Elg-AJg", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "from nemo_text_processing.text_normalization.normalize import Normalizer\n", "\n", "text_normalizer = Normalizer(input_case=\"cased\", lang=\"en\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "SPMVhS8qrFCI", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "text = \"Mr. Johnson is turning 35 years old on 04-15-2023.\"\n", "normalized_text = text_normalizer.normalize(text)\n", "\n", "print(text)\n", "print(normalized_text)" ] }, { "cell_type": "markdown", "metadata": { "id": "8KxovRNsyZdr", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 5.2 TN approach" ] }, { "cell_type": "markdown", "metadata": { "id": "OrRZejiGte90", "pycharm": { "name": "#%% md\n" } }, "source": [ "Today text normalization is typically a very manual process involving lots of rules, heuristics, and regular expressions.\n", "\n", "It is difficult to train a machine learning model to automate this step due to lack of labeled data. To get ground truth data one would need to manually annotate the entire dataset. The resulting model would then have strictly worse performance than the manual system producing the labels, making it better to use the original labeling system rather than the model." ] }, { "cell_type": "markdown", "metadata": { "id": "_kVLSOCeyFCL", "pycharm": { "name": "#%% md\n" } }, "source": [ "**Dataset Preparation**\n", "\n", "Most common public datasets for TTS modeling are already normalized before they are uploaded, allowing users to skip this step entirely.\n", "\n", "When preparing new datasets for training, individuals will often rely on open source libraries for normalization, manually adding new logic into the system as they find situations in their datasets not covered by the pre-existing logic.\n", "\n", "This logic is also hard to generalize across *languages*, making scaling TTS to multiple languages very challenging. Enterprises often find it necessary to hire linguists for every language they operate in, whose responsibility is to manually develop and improve the text processing and normalization for their specific languages and users." ] }, { "cell_type": "markdown", "metadata": { "id": "CPh9zYO71Pih", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 6. Grapheme to Phoneme (G2P)" ] }, { "cell_type": "markdown", "metadata": { "id": "rXij0PkxIqrj", "pycharm": { "name": "#%% md\n" } }, "source": [ "Next we convert the written text into **phonemes**, the most basic unit of human speech.\n", "\n", "For example (using [ARPABET](https://en.wikipedia.org/wiki/ARPABET)): *Hello World → HH, AH0, L, OW1, ,W, ER1, L, D*\n", "\n", "Some languages, such as Spanish and German, are *phonetic*, meaning their written characters/graphemes are always pronounced the same. For such languages G2P is unnecessary.\n", "\n", "However English is not Phonetic because:\n", "* Characters change pronunciation depending on what word they are in.\n", "* Some phonemes are represented by multiple characters (eg. \"ch\", \"th\", \"ph\")\n", "* Some words are [Heteronyms](https://en.wikipedia.org/wiki/Heteronym_(linguistics%29#English), which are words with the same spelling but different pronunciation or meaning.\n", " * For example, \"read\" is pronounced differently depending on whether it is present tense or past tense.\n", " * Some heteronyms can be disambiguated based on what part of speech they are. But some, such as bass (fish or instrument), can only be disambiguated based on meaning/context.\n", "\n", "For non-phonetic languages like English it is still possible to train a TTS model directly on the graphemes. But doing so will make the pronunciation of some words less accurate.\n", "\n", "Details on how NeMo G2P works can be found in our [G2P documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/tts/g2p.html)." ] }, { "cell_type": "markdown", "metadata": { "id": "KFytLD_hOt7F", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 6.1 Phonetic Alphabets" ] }, { "cell_type": "markdown", "metadata": { "id": "ZaejnPZ5Otbm", "pycharm": { "name": "#%% md\n" } }, "source": [ "There are several different alphabets that can be used to represent the phonetic pronunciation of a sentence.\n", "\n", "The most common phonetic alphabet is the **International Phonetic Alphabet (IPA)**, which provides a common set of phonemes for all languages.\n", "\n", "For English, **ARPABET** is also very common. Largely due to the popularity of the [CMU Pronouncing Dictionary](http://www.speech.cs.cmu.edu/cgi-bin/cmudict)." ] }, { "cell_type": "markdown", "metadata": { "id": "gEUUBp5ePuEv", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 6.2 G2P example" ] }, { "cell_type": "markdown", "metadata": { "id": "xKttt_w6cc6w", "pycharm": { "name": "#%% md\n" } }, "source": [ "We will show how to do both IPA and ARPABET G2P using NeMo.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "d8qbJIQuPvmx", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "import os\n", "from nemo.collections.tts.torch.g2ps import EnglishG2p, IPAG2P\n", "\n", "ipa_dict_path = os.path.join(NEMO_DIR, \"scripts\", \"tts_dataset_files\", \"ipa_cmudict-0.7b_nv23.01.txt\")\n", "\n", "# EnglishG2p uses the CMU dictionary by default, if none is provided\n", "arpabet_g2p = EnglishG2p()\n", "ipa_g2p = IPAG2P(phoneme_dict=ipa_dict_path)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "xSaNypYJURXA", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "text = \"Hello world\"\n", "arpabet_phonemes = arpabet_g2p(text)\n", "ipa_phonemes = ipa_g2p(text)\n", "\n", "print(text)\n", "print(arpabet_phonemes)\n", "print(ipa_phonemes)" ] }, { "cell_type": "markdown", "metadata": { "id": "AaZPPpsPcjCZ", "pycharm": { "name": "#%% md\n" } }, "source": [ "Notice that graphemes are returned for the word \"hello\" rather than the ARPABET phonemes.\n", "\n", "This is because the English G2P module uses its dictionary to map graphemes to phonemes. The word \"hello\" has two possible pronunciations." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "JcH24VMMXtkl", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "print(arpabet_g2p.phoneme_dict['hello'])" ] }, { "cell_type": "markdown", "metadata": { "id": "NCeu-W-NeHjZ", "pycharm": { "name": "#%% md\n" } }, "source": [ "Since the pronunciation is *ambiguous*, it returns the graphemes instead. By default it will also do this for any word **out of vocabulary (oov)** words that are not in the CMU dictionary.\n", "\n", "In general, handling ambiguous pronunciations is a difficult problem which we will discuss more below.\n", "\n", "For now, we can tell the G2P module to simply return the first pronunciation in the dictionary by setting `ignore_ambiguous_words=False`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Edv2DIqEdWco", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "arpabet_g2p = EnglishG2p(ignore_ambiguous_words=False)\n", "ipa_g2p = IPAG2P(phoneme_dict=ipa_dict_path, ignore_ambiguous_words=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "SnVXmlildnsT", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "text = \"Hello world\"\n", "arpabet_phonemes = arpabet_g2p(text)\n", "ipa_phonemes = ipa_g2p(text)\n", "\n", "print(text)\n", "print(arpabet_phonemes)\n", "print(ipa_phonemes)" ] }, { "cell_type": "markdown", "metadata": { "id": "ckRB7iWpUavz", "pycharm": { "name": "#%% md\n" } }, "source": [ "**Tokenization**" ] }, { "cell_type": "markdown", "metadata": { "id": "Dznknw-e6vhV", "pycharm": { "name": "#%% md\n" } }, "source": [ "At the end of G2P we have a list of characters. In order for our model to process these characters, we need to convert them into numbers.\n", "\n", "To do this we have a vocabulary containing all characters in our alphabet. Each character gets assigned a fixed number, such that every time it shows up in the text it is replaced with that number." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "IFUjxsocUYz9", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "from nemo.collections.tts.torch.tts_tokenizers import EnglishPhonemesTokenizer, IPATokenizer\n", "\n", "arpabet_tokenizer = EnglishPhonemesTokenizer(arpabet_g2p)\n", "ipa_tokenizer = IPATokenizer(ipa_g2p)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "t8vdYwA94xnn", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "text = \"Hello world\"\n", "arpabet_tokens = arpabet_tokenizer(text)\n", "ipa_tokens = ipa_tokenizer(text)\n", "\n", "print(text)\n", "print(arpabet_tokens)\n", "print(ipa_tokens)" ] }, { "cell_type": "markdown", "metadata": { "id": "FxdyJIQrUZru", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 6.3 G2P approach" ] }, { "cell_type": "markdown", "metadata": { "id": "q4SAerFphP8W", "pycharm": { "name": "#%% md\n" } }, "source": [ "Similar to Text Normalization, most G2P systems today are very manual.\n", "\n", "Most of the earlier descriptions about Text Normalization are also the same for G2P, in regards to it being difficult to get labeled data to train a machine learning model to do it automatically and challenging to generalize and scale across languages.\n", "\n", "The most common way that G2P is done today is to hardcode the grapheme to phoneme mapping for all common words in a language in a **pronouncing dictionary**.\n", "\n", "A few examples of dictionary entries:\n", "```\n", "CAT K AE1 T\n", "HELLO HH AH0 L OW1\n", "NVIDIA EH0 N V IY1 D IY0 AH0\n", "```\n", "\n", "This works decently for languages like English which have comprehensive dictionaries, but does not help for the large number of languages for which no such dictionary is available.\n", "\n", "But even with a good dictionary, there are still a few problems.\n", "\n", "* Text with **out of vocabulary (oov)** words that are not in the dictionary\n", "* **Heteronyms**: Two words with the same spelling but different pronunciations or meaning.\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "YTw3jlaCmQnX", "pycharm": { "name": "#%% md\n" } }, "source": [ "### 6.3.1 Out of Vocabulary (OOV)" ] }, { "cell_type": "markdown", "metadata": { "id": "WRW1oTUInLsi", "pycharm": { "name": "#%% md\n" } }, "source": [ "Figuring out how to pronounce unknown words is difficult. Even native English speakers have a hard time pronouncing unfamiliar words and often have to look up the correct pronunciation.\n", "\n", "**Dictionary Management**\n", "\n", "The most straightforward way to deal with OOV words is to manually add them to your dictionary. Not very sustainable, but the only method guaranteed to be accurate.\n", "\n", "For a TTS application, it is often a good idea to give individual users the option to create and manage their own dictionary so they can fix any mistakes on their own in real-time.\n", "\n", "**Grapheme-Based TTS Training**\n", "\n", "Training a model to generate audio from graphemes will (in theory) teach it to implicitly learn G2P rules based on its training data.\n", "\n", "Training on only graphemes will typically result in poorer overall performance than training on phonemes.\n", "\n", "NeMo also supports **mixed grapheme-phoneme training** in which the model is trained on inputs which contain both graphemes and phonemes. Doing this may teach the model to pronounce optimally with phonemes while also providing reasonable out-of-the-box behavior on OOV words and heteronyms.\n", "\n", "To do mixed training, the `EnglishG2p()` class has a `phoneme_probability` parameter which allows you to specify how likely each word will be left as graphemes at training time.\n", "\n", "**Machine Learning Model**\n", "\n", "Given a pronunciation dictionary, one can train a machine learning model to try and predict phonemes from graphemes. Or the dictionary can be run on a full text corpus, and the model can be trained on the sentences in the corpus.\n", "\n", "The resulting model will have lower prediction accuracy than the dictionary or G2P system it is trying to replicate, but it will be possible to run the model over OOV words the original dictionary could not process." ] }, { "cell_type": "markdown", "metadata": { "id": "FF6u_-fkmY1m", "pycharm": { "name": "#%% md\n" } }, "source": [ "### 6.3.2 Heteronyms" ] }, { "cell_type": "markdown", "metadata": { "id": "V1PwNQQUqyzl", "pycharm": { "name": "#%% md\n" } }, "source": [ "There are not many good ways today to handle a word having multiple pronunciations.\n", "\n", "The Grapheme-based training methods mentioned for OOV words can also be used here.\n", "\n", "Another common approach relies on the fact that most (but not all) heteronyms can be disambiguated based on what part-of-speech they are.\n", "\n", "* Create an extra dictionary mapping heteronyms to phonemes based on what part-of-speech they are.\n", "* Use a [part-of-speech-tagger](https://en.wikipedia.org/wiki/Part-of-speech_tagging) to tag heteronyms." ] }, { "cell_type": "markdown", "metadata": { "id": "zZhZ47vy5swC", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 7. Spectrogram Synthesis" ] }, { "cell_type": "markdown", "metadata": { "id": "cHXPTuQh7VOY", "pycharm": { "name": "#%% md\n" } }, "source": [ "The most complicated step is converting the text or phonemes into a spectrogram representing the spoken audio.\n", "\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "_yo7Ru_GMA0E", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 7.1 Audio" ] }, { "cell_type": "markdown", "metadata": { "id": "7V-iJ0wl6PZP", "pycharm": { "name": "#%% md\n" } }, "source": [ "Audio is typically represented by a long sequence of numbers representing the audio amplitude at each point in time.\n", "\n", "How often the audio samples are recorded is the [sampling rate](https://en.wikipedia.org/wiki/Sampling_(signal_processing)). Common sampling rates are 16 kHz, 22.05 kHz, and 44.1 kHz, with higher sampling rates producing more accurate, higher quality audio.\n", "\n", "A sampling rate of 44.1 kHz means that we store 44,100 audio sample per second of audio. Each audio sample is typically stored as a 16-bit integer (a whole number ranging from -32,768 to +32,767). For machine learning, it is sometimes more convenient to rescale it and represent it as a continuous value between -1.0 and 1.0 using 32-bit floats.\n", "\n", "
\n", "
\n", "\n", "
https://wiki.hydrogenaud.io/index.php?title=File:Digital_wave.png\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "C7K8pIkt6MH2", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 7.2 Spectrograms\n", "\n", "While raw audio shows amplitude versus time and is useful for easily recording and listening, it is not optimal when it comes to processing.\n", "\n", "For processing, it is usually preferable to represent the audio as a **spectrogram** which shows frequency versus time. Specifically, we:\n", "\n", "1. Group together audio samples into a much smaller set of time buckets, called **audio frames**. An audio frame will usually bucket around 50ms of audio.\n", "2. For each audio frame, use the [Fast Fourier transform](https://en.wikipedia.org/wiki/Fast_Fourier_transform) (**FFT**) to calculate the magnitude (ie. energy, amplitude or \"loudness\") and phase (which we don't use) of each frequency band (ie. pitch).\n", "3. Translate the original frequency bands, measured in units of hertz (Hz), into units of [mel frequency](https://en.wikipedia.org/wiki/Mel_scale). The output is called a **mel spectrogram**.\n", "\n", "We then use the mel spectrogram as our final audio representation. The only thing we lose during this process is the phase information, the implications of which we will discuss more later on.\n", "\n", "The reason for converting from hertz to mel frequency is that hertz are a physical measurement, while mel frequency better captures how humans perceive sound. Human ears are more sensitive to low frequencies than high frequencies. So low frequency bands would have large impact on the perceived sound while high frequency bands might have little to no effect. On the mel scale, the impact of different frequency bands are more evenly distributed." ] }, { "cell_type": "markdown", "metadata": { "id": "qbYSJqjKHVbU", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 7.3 Why use a spectrogram?\n" ] }, { "cell_type": "markdown", "metadata": { "id": "uKYbiRNn6G2e", "pycharm": { "name": "#%% md\n" } }, "source": [ "\n", "In addition to the mel spectrogram providing handcrafted features that are closer to human perception and easier to predict, there are a few other advantages to using it over audio samples.\n", "\n", "**1. Computational Efficiency**\n", "\n", "One reason to use a spectrogram is that it is a lot more computationally efficient than processing raw audio.\n", "\n", "Take for example a 10 second piece of audio sampled at 44.1 kHz.\n", "\n", "This audio has 10 * 44,100 = 441,000 audio samples.\n", "\n", "For TTS we will represent this same audio with audio frames that are overlapping such that there is about 1 audio frame per 10ms (0.01s). For each audio frame we have 80 frequency bands.\n", "\n", "Our final spectrogram has 80 bands * 10s * / (0.01s) = 80,000\n", "\n", "So we made our data about 5 times smaller.\n", "\n", "Even in other fields that deal with audio, such as *digital signal processing (DSP)*, it is common practice to reduce computation time by computing the FFT, running mathematical calculations and manipulations in the frequency domain, and then converting the final spectrogram back to audio with the *inverse* FFT.\n", "\n", "**2. Shorter 2D sequence length**\n", "\n", "In the example above, the dimension of our input goes from being (441,000, 1) to (1000, 80). Instead of having a very long 1-dimensional time series, we instead have a much shorter sequence of multi-dimensional features.\n", "\n", "This dimensionality works better with most modern deep learning models.\n", "\n", "Most models work better representing large chunks/sequences of information as higher dimension vectors (eg. *embeddings*).\n", "\n", "**RNN/LSTM** architectures have latency proportional to the length of the sequence, can struggle to remember information for sequences over a few hundred tokens long, and are more prone to vanishing/exploding gradients as the sequence gets longer.\n", "\n", "With 2 dimensions we can effectively use **CNNs** by running [temporal convolutions](https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html?highlight=conv1d#torch.nn.Conv1d) over the time dimension. Or by applying [2d convolutions](https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html?highlight=conv2d#torch.nn.Conv2d) to the spectrogram exactly as if it were an image in computer vision.\n", "\n", "**Transformers** require computation/memory that is proportional to the length of the sequence squared. This means we can easily use large transformers for relatively short sequences like in NLP, smaller transformers for longer sequences like spectrogram data, and are impractical to use on very long sequences like audio samples." ] }, { "cell_type": "markdown", "metadata": { "id": "dUDB6f9h6G1Q", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 7.4 End to end example" ] }, { "cell_type": "markdown", "metadata": { "id": "PsPn56E_Jax4", "pycharm": { "name": "#%% md\n" } }, "source": [ "Before we go into the details of how this works, let's go through an end-to-end text to audio example so we can visualize what our model inputs and outputs look and sound like.\n", "\n", "To do this, we will need to use both the spectrogram and vocoder models together. The vocoder will be looked at more thoroughly in the *audio synthesis* section." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "hy-Z4U7LxR9e", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "import torch\n", "import IPython.display as ipd\n", "from matplotlib.pyplot import imshow\n", "from matplotlib import pyplot as plt\n", "from nemo.collections.tts.models.base import SpectrogramGenerator, Vocoder\n", "\n", "# List pretrained models available in NeMo\n", "print(\"Spectrogram Models\")\n", "print(SpectrogramGenerator.list_available_models())\n", "\n", "print()\n", "print(\"Vocoders\")\n", "print(Vocoder.list_available_models())" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "-KuBkFKrKzxn", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "if torch.cuda.is_available():\n", " device = \"cuda\"\n", "else:\n", " device = \"cpu\"\n", "\n", "# Load our spectrogram and vocoder models onto our CPU\n", "spectrogram_model = SpectrogramGenerator.from_pretrained(\"tts_en_tacotron2\").eval().to(device)\n", "vocoder = Vocoder.from_pretrained(\"tts_en_hifigan\").eval().to(device)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6_brQ8_WBlX_", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "text = \"This audio was generated with a text-to-speech model.\"\n", "\n", "# Normalize the text and convert it into individual phonemes/tokens.\n", "tokens = spectrogram_model.parse(text, normalize=True)\n", "\n", "# Generate spectrogram from text\n", "spectrogram = spectrogram_model.generate_spectrogram(tokens=tokens)\n", "\n", "# Invert the spectrogram into audio samples\n", "audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)\n", "\n", "# Convert output from pytorch tensor to numpy array\n", "spectrogram = spectrogram.cpu().detach().numpy()[0]\n", "audio = audio.cpu().detach().numpy()[0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "m3VcVblUHA7g", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "print(f'\"{text}\"\\n')\n", "\n", "ipd.Audio(audio, rate=22050)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ze1JIDxeHJq5", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# Visualize the spectrogram\n", "imshow(spectrogram, origin=\"lower\")\n", "plt.xlabel(\"Audio Frame\")\n", "plt.ylabel(\"Frequency Band\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "OgthvHUzHeE6", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 7.5 Model types" ] }, { "cell_type": "markdown", "metadata": { "id": "5QQt3p9a82gN", "pycharm": { "name": "#%% md\n" } }, "source": [ "Predicting a spectrogram from text is a **sequence-to-sequence** problem because the input and output are different lengths. The length of the output is itself ambiguous, and something the model needs to predict.\n", "\n", "Most successful approaches to this problem use an **encoder-decoder** system. The typical pattern looks like:\n", "\n", "1. The model has a **character embedding** for each character in its vocabulary. This is usually learned from scratch during training, but can also be pre-trained or augmented with generic language embeddings like *BERT*.\n", "2. The **encoder** analyzes the input sentence and applies various transformations to each character embedding in the output sequence. The *encoder outputs* are character embeddings that represent how each character should be *pronounced*, given the sentence. Encoders will also usually encode the approximate length, or **duration**, of each character.\n", "3. The **decoder** predicts the spectrogram using the sequence of character pronunciation embeddings provided by the encoder.\n", "\n", "Most model architectures rely on **RNNs** or **Transformers** to do sentence-level processing (like how to pronounce or emphasize certain words based on the meaning of the sentence), and **CNNs** to process features based on their adjacent characters or audio frames (like how to start and end a word to make it flow naturally with the previous and next words).\n", "\n", "Today there are two main categories of these encoder-decoder systems: **auto-regressive** models and **parallel** models. Within these categories, parallel models typically use **duration prediction**, while auto-regressive models can use duration prediction or **attention**." ] }, { "cell_type": "markdown", "metadata": { "id": "H8rkeCkHMrev", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 7.6 Auto Regressive Models" ] }, { "cell_type": "markdown", "metadata": { "id": "lGVKcJp6Y7Kv", "pycharm": { "name": "#%% md\n" } }, "source": [ "### 7.6.1 Tacotron 2" ] }, { "cell_type": "markdown", "metadata": { "id": "O0q5_nb6SfQT", "pycharm": { "name": "#%% md\n" } }, "source": [ "The first good neural TTS models were auto-regressive models such as [Tacotron 2](https://google.github.io/tacotron/publications/tacotron2/index.html).\n", "\n", "For these models, the encoder encodes the input sentence, and then the decoder predicts the spectrogram one audio frame at a time. This lets the model condition each spectrogram frame on the frames before it, producing high quality audio.\n", "\n", "Let's take a quick look at the Tacotron 2 architecture.\n", "\n", "
\n", "
\n", "\n", "
\n", "\n", "The model is fairly complex. At a high level, it contains:\n", "\n", "1. A CNN-RNN *encoder* which uses a bidirectional LSTM to analyze the entire sentence forward and backwards to figure out the pronunciation of every character.\n", "2. A unidirectional LSTM *decoder* which predicts the spectrogram one frame at a time.\n", "3. An **attention** mechanism that the decoder uses to determine which character(s) to look at when generating specific audio frames.\n", "4. A *stop token* layer which predicts a single number [0, 1] to determine the probability that the decoder has reached the end of the sentence.\n", "5. A *pre-net* that feeds the previous predicted frame into the decoder before predicting the next frame. This is used primarily for **teacher forcing** so that the model can learn effectively.\n", "6. A *post-net* which post-processes the entire spectrogram produced by the decoder to fix inconsistencies and smooth it out.\n", "\n", "The attention and teacher forcing mechanisms are the main characteristics that define the auto-regressive model and differentiate it from non-auto-regressive systems. These will be discussed in more detail below.\n", "\n", "The post-net is an optional optimization.\n", "\n", "The specifics of the encoder and decoder are not too important, and can be replaced with any equivalent combination of RNN, CNN, and/or Transformer blocks. \n", "\n" ] }, { "cell_type": "markdown", "metadata": { "id": "wmGmNcT8LJGp", "pycharm": { "name": "#%% md\n" } }, "source": [ "### 7.6.2 Attention and Alignment" ] }, { "cell_type": "markdown", "metadata": { "id": "wunvHeLVEd45", "pycharm": { "name": "#%% md\n" } }, "source": [ "An auto-regressive model predicts the spectrogram based primarily on the encoder output features. For the output to make sense, each audio frame produced needs to be based on a subset of the characters in the input. Which character(s) the decoder is using when it predicts an audio frame is called its **attention**.\n", "\n", "First, let's see what our Tacotron 2 attention looks like." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "rILddgvkw8om", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "import torch\n", "from nemo.collections.tts.models import Tacotron2Model\n", "\n", "Tacotron2Model.list_available_models()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "CIzlpQTH7GZ6", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "tacotron2_model = Tacotron2Model.from_pretrained(\"tts_en_tacotron2\").eval().to(device)\n", "tacotron2_model.calculate_loss = False\n", "\n", "g2p = EnglishG2p(ignore_ambiguous_words=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "KGxEWnwhjrUQ", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "text = \"This is an attention map.\"\n", "\n", "tokens = tacotron2_model.parse(text, normalize=True)\n", "token_len = torch.tensor([tokens.shape[1]], dtype=torch.long, device=spectrogram_model.device)\n", "\n", "_, _, _, alignments, _ = tacotron2_model.forward(tokens=tokens, token_len=token_len)\n", "alignment = alignments[0].cpu().detach().numpy()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "KW2bN2wh9vTj", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# Pretrained Tacotron 2 uses grapheme input with beginning-of-speech and end-of-speech tokens\n", "characters = [\"\"] + [char for char in text] + [\"\"]\n", "\n", "fig, ax = plt.subplots(figsize=(10, 8))\n", "plt.xlabel(\"Audio Frame\")\n", "plt.ylabel(\"Character\")\n", "_ = ax.imshow(alignment.transpose(), origin='upper', aspect='auto')\n", "_ = ax.set_yticks(range(len(characters)))\n", "_ = ax.set_yticklabels(characters)" ] }, { "cell_type": "markdown", "metadata": { "id": "kIYrbaDDI7--", "pycharm": { "name": "#%% md\n" } }, "source": [ "In order to get good quality output, we need our model's attention to follow these constraints:\n", "\n", "* The decoder should pay attention to one character at a time (sometimes it also places weight on the one character immediately before and/or after it).\n", "* The attention should be **monotonically increasing**, meaning it never go backwards in the text sequence. So the attention should only ever stay on the current character, or move forward to the next character.\n", "* The model should start on the first character in the sequence and end on the last character.\n", "\n", "These constraints result in the decoder effectively \"reading\" the text character by character or word by word, similar to how humans read aloud.\n", "\n", "A model may need to be trained for a while before its attention learns to follow these constraints. Before that, the attention may look non-sensical, and the model output will sound unintelligible.\n", "\n", "Once the models learns the above constraints and starts producing well-behaved attention maps, it is said that the model has **aligned**." ] }, { "cell_type": "markdown", "metadata": { "id": "N39VazavL9w0", "pycharm": { "name": "#%% md\n" } }, "source": [ "**Attention Mechanisms**\n", "\n", "A large number of different systems have been developed over time to model TTS attention. Historically a lot of these mechanisms would use various inputs (decoder state, previous prediction, previous attention weights, etc) during each decoder step to predict a softmax across all characters in the sentence.\n", "\n", "This works decently, but is not very good at enforcing the constraints listed above. A better [alternative](https://arxiv.org/abs/2108.10447) is to model the attention similar to a *Hidden Markov Model*; a state machine in which each character in the text is a state and the only valid state transitions are to stay on the current character or move forward 1 character." ] }, { "cell_type": "markdown", "metadata": { "id": "UrLwWW5ILFai", "pycharm": { "name": "#%% md\n" } }, "source": [ "### 7.6.3 Training Features and Teacher Forcing" ] }, { "cell_type": "markdown", "metadata": { "id": "zjODqzmOEbLZ", "pycharm": { "name": "#%% md\n" } }, "source": [ "Another very important aspect of TTS modeling is **teacher forcing**, which means that during training we provide various pieces of ground truth information to the model and at inference provide equivalent predicted values.\n", "\n", "Training a model to directly predict a spectrogram from text by itself does not work, for a few reasons:\n", "\n", "* Spectrogram Synthesis is a difficult enough problem that most models will not converge to anything meaningful if its only input is text.\n", "* The prediction target is extremely *ambiguous*. There are several correct ways to read any sentence. For training to be smooth, the input features need to be enough to figure out the speaking pattern for most training utterances.\n", "\n", "The most standard form of teacher forcing in auto-regressive systems is for each decoder step to receive the ground truth prediction from the previous step. In other words, each decoder step predicts an audio frame. During training, the ground truth audio frame is passed to the next decoder step. During inference, the predicted audio frame is passed to the next decoder step. In Tacotron 2, the pre-net applies a large amount of dropout to this input to avoid overfitting.\n", "\n", "Some other common forms of ground truth information that have been found to be useful are:\n", "\n", "* **Pitch**: By providing the pitch, aka. the [fundamental frequency](https://en.wikipedia.org/wiki/Fundamental_frequency) **(F0)**, for each character, a model is able to better learn and mimic a variety of natural speaking patterns.\n", "* **Speaking Rate**: By providing the length of the ground truth spectrogram as an input feature, the model is able to easily disambiguate how long the final audio should be. At inference time a user can then manually change the input value to make the model speak as slow or as fast as they want.\n", "* **Style Embeddings**: By training an *auto-encoder* on the TTS training data, it is possible to represent very complex speaking styles with utterance-level embeddings. Taking a style embedding from one utterance to make a speaker say a different sentence with the same speaking characteristics is referred to as **style transfer**.\n" ] }, { "cell_type": "markdown", "metadata": { "id": "Aa6O2a4thdnL", "pycharm": { "name": "#%% md\n" } }, "source": [ "### 7.6.4 Ground Truth Aligned (GTA) Inference" ] }, { "cell_type": "markdown", "metadata": { "id": "KZtBnNSphWJ2", "pycharm": { "name": "#%% md\n" } }, "source": [ "Let's take an utterance and compare the output of Tacotron2 when teacher forcing is used versus when it is not. The teacher forcing ensures that the predicted spectrogram is the same length as the ground truth spectrogram, so this type of inference is often referred to as being **ground truth aligned (GTA)**." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "19iQHogIdARW", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "import wget\n", "\n", "audio_path = \"LJ023-0089.wav\"\n", "audio_url = \"https://multilangaudiosamples.s3.us-east-2.amazonaws.com/LJ023-0089.wav\"\n", "\n", "if not os.path.exists(audio_path):\n", " wget.download(audio_url, audio_path)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "4jOjFwC9dfwA", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "import librosa\n", "\n", "# Process text input\n", "text = \"That is not only my accusation.\"\n", "\n", "tokens = tacotron2_model.parse(text, normalize=True)\n", "token_len = torch.tensor([tokens.shape[1]], dtype=torch.long, device=tacotron2_model.device)\n", "\n", "# Run regular inference\n", "_, predicted_spectrogram, _, _, _ = tacotron2_model.forward(tokens=tokens, token_len=token_len)\n", "predicted_audio = vocoder.convert_spectrogram_to_audio(spec=predicted_spectrogram)\n", "predicted_audio = predicted_audio.cpu().detach().numpy()[0]\n", "\n", "# Process the ground truth audio\n", "sample_rate = 22050\n", "audio, _ = librosa.load(audio_path, sr=sample_rate)\n", "\n", "audio_tensor = torch.from_numpy(audio).unsqueeze(0).to(tacotron2_model.device)\n", "audio_len = torch.tensor(audio_tensor.shape[1], dtype=torch.long, device=tacotron2_model.device).unsqueeze(0)\n", "\n", "# Run GTA inference. This is a bit awkward to set up as Tacotron2 only expects to do this at training time.\n", "tacotron2_model.calculate_loss = True\n", "tacotron2_model.training = True\n", "tacotron2_model.decoder.training = True\n", "_, gta_spectrogram, _, _, _, _ = tacotron2_model.forward(\n", " tokens=tokens,\n", " token_len=token_len,\n", " audio=audio_tensor,\n", " audio_len=audio_len\n", ")\n", "tacotron2_model.calculate_loss = False\n", "tacotron2_model.training = False\n", "tacotron2_model.decoder.training = False\n", "\n", "gta_audio = vocoder.convert_spectrogram_to_audio(spec=gta_spectrogram)\n", "gta_audio = gta_audio.cpu().detach().numpy()[0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "PrsNwAmdilaW", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "print(f'\"{text}\"\\n')\n", "\n", "print(\"Original Audio\")\n", "ipd.display(ipd.Audio(audio, rate=sample_rate))\n", "\n", "print(\"GTA Audio\")\n", "ipd.display(ipd.Audio(gta_audio, rate=sample_rate))\n", "\n", "print(\"Predicted Audio\")\n", "ipd.display(ipd.Audio(predicted_audio, rate=sample_rate))" ] }, { "cell_type": "markdown", "metadata": { "id": "UnzfRDi0WuWq", "pycharm": { "name": "#%% md\n" } }, "source": [ "We see that with teacher forcing, the model output sounds almost exactly like the original audio. This is how the model learns how to generate audio that sounds realistic and high-quality.\n", "\n", "However, at inference time we see that the pitch of the speaker is very different. Without teacher forcing, and without receiving the pitch as an input feature, the model has no way of knowing what pitch to use. So most likely, all predicted utterances will end up using the average pitch observed at training time." ] }, { "cell_type": "markdown", "metadata": { "id": "uya9DJ1SWwEx", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 7.7 Duration Prediction\n" ] }, { "cell_type": "markdown", "metadata": { "id": "O6uH8q-BZjko", "pycharm": { "name": "#%% md\n" } }, "source": [ "A large weakness of the original Tacotron 2 model is its attention mechanism, which does not enforce the required monotonicity constraint (ie. the decoder must pay attention to each character once in sequential increasing order). As a result, the attention is not robust. It often skips words, repeats words, or encounters catastrophic failures where the output becomes unintelligible.\n", "\n", "There are some attention mechanisms such as [forward attention](https://arxiv.org/abs/1807.06736) which try to address this.\n", "\n", "The most standard approach today is to do **explicit duration prediction**. This means that the model encoder predicts the duration (ie. number of spectrogram frames) of each character directly, instead of relying on the attention to determine it implicitly.\n", "\n", "Replacing the attention mechanism in Tacotron 2 with duration prediction, eg. [Non-Attentive Tacotron](https://arxiv.org/abs/2010.04301), has historically been a common and necessary optimization to make it robust enough for use in enterprise applications. Though it gained visibility in academic literature primarily due to its use in modern transformer based model architectures such as [FastSpeech](https://arxiv.org/abs/1905.09263) and [FastPitch](https://fastpitch.github.io/).\n", "\n", "The biggest drawback of this approach is that you need to get the ground truth character duration information. Some methods for doing this are:\n", "\n", "1. The preferred method in NeMo is to Jointly train an [alignment model](https://arxiv.org/abs/2108.10447) that measures the similarity between characters and spectrogram frames.\n", "2. Run forced alignment, such as with the [Montreal Forced Aligner](https://montreal-forced-aligner.readthedocs.io/en/latest/).\n", "3. Infer the duration information from the attention map of a teacher model, such as Tacotron 2." ] }, { "cell_type": "markdown", "metadata": { "id": "Z7SfuEJK6176", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 7.8 Parallel Models\n" ] }, { "cell_type": "markdown", "metadata": { "id": "XOtPiRajZG2Z", "pycharm": { "name": "#%% md\n" } }, "source": [ "There are some significant weaknesses to auto-regressive systems. Most notably:\n", "\n", "* Spectrograms are long (100s-1000s of frames), so generating them one frame at a time makes inference slow.\n", "* They are typically implemented using RNN based architectures, which are slow to train.\n", "* The user has little control over how the sentence is spoken.\n", "\n", "Using duration prediction enables us to remove the auto-regressive inference and predict every spectrogram frame in parallel. This makes the inference speed up to 100x faster, making it highly preferable for deploying and serving to users." ] }, { "cell_type": "markdown", "metadata": { "id": "HgMfSDW5ZaE4", "pycharm": { "name": "#%% md\n" } }, "source": [ "### 7.8.1 FastPitch" ] }, { "cell_type": "markdown", "metadata": { "id": "FxsyNfSBRKqc", "pycharm": { "name": "#%% md\n" } }, "source": [ "In NeMo we support [FastPitch](https://fastpitch.github.io/), a parallel transformer-based model with pitch and duration control and prediction.\n", "\n", "
\n", "\n", "
\n", "\n", "At a high level it contains:\n", "\n", "1. An encoder consisting of a **feed-forward transformer block (FFTr)**, which transforms the input character embeddings into character pronunciation embeddings.\n", "2. A temporal CNN which takes the encoder output and predicts the duration and pitch of each character. At training time, the ground truth pitch and duration information are fed to the model (similar to teacher forcing).\n", "3. Each encoder output is repeated a number of times equal to the predicted duration. The repeated encoder output is the same length as the final spectrogram. For example, for characters 'abc' and predicted durations (2, 3, 1) we get the encoder output repeated 'aabbbc'. \n", "4. A decoder consisting of a FFTr that transforms this encoder output into the predicted spectrogram.\n", "\n", "Some advantages to this approach are:\n", "\n", "* The duration prediction makes the output consistent and robust.\n", "* The model inference is fast, able to synthesize up to 1000 seconds of audio every second on an A100 GPU.\n", "* You can customize the prosody by manually selecting the pitch and duration of each character or word." ] }, { "cell_type": "markdown", "metadata": { "id": "UN_SIcPuBcQw", "pycharm": { "name": "#%% md\n" } }, "source": [ "Let's run the same inference for FastPitch that we did with Tacotron2. The main difference is loading the FastPitch checkpoint using the `FastPitchModel` class." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "gRAkPzK4XkOu", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "from nemo.collections.tts.models import FastPitchModel\n", "\n", "fastpitch_model = FastPitchModel.from_pretrained(\"tts_en_fastpitch\").eval().to(device)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "jXaKddUKeTsz", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "text = \"This audio was generated with the fastpitch text-to-speech model.\"\n", "\n", "# Normalize the text and convert it into individual phonemes/tokens.\n", "tokens = fastpitch_model.parse(text, normalize=True)\n", "\n", "# Generate spectrogram from text\n", "spectrogram = fastpitch_model.generate_spectrogram(tokens=tokens)\n", "\n", "# Invert the spectrogram into audio samples\n", "audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)\n", "\n", "# Convert output from pytorch tensor to numpy array\n", "spectrogram = spectrogram.cpu().detach().numpy()[0]\n", "audio = audio.cpu().detach().numpy()[0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LIQ2s_1neeS5", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "print(f'\"{text}\"\\n')\n", "\n", "ipd.Audio(audio, rate=22050)" ] }, { "cell_type": "markdown", "metadata": { "id": "vwD3Xhwhoys0", "pycharm": { "name": "#%% md\n" } }, "source": [ "### 7.8.2 Drawbacks" ] }, { "cell_type": "markdown", "metadata": { "id": "3jHNDSGmo5f9", "pycharm": { "name": "#%% md\n" } }, "source": [ "One weakness of parallel models is that without auto-regressive teacher forcing, the model is unable to reliably predict/reconstruct the original utterance. Primarily due to the inputs not fully capturing the unpredictable variability/ambiguity in the possible outputs. The result is that the model learns an average over possible outputs, creating spectrograms that look unrealistically \"smooth\", degrading the audio quality (https://arxiv.org/abs/2202.13066).\n", "\n", "This problem can be partially alleviated by fine-tuning the spectrogram inversion model (described in the next section) directly on the predicted spectrograms.\n", "\n", "To visualize this, let's compare a spectrogram to the corresponding one predicted by FastPitch." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "jvHCe1NWplZo", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# Compute real spectrogram\n", "audio_path = \"LJ023-0089.wav\"\n", "sample_rate = 22050\n", "\n", "audio, _ = librosa.load(audio_path, sr=sample_rate)\n", "audio_tensor = torch.from_numpy(audio).unsqueeze(0).to(fastpitch_model.device)\n", "audio_len = torch.tensor(audio_tensor.shape[1], dtype=torch.long, device=fastpitch_model.device).unsqueeze(0)\n", "\n", "real_spectrogram, _ = fastpitch_model.preprocessor(input_signal=audio_tensor, length=audio_len)\n", "real_spectrogram = real_spectrogram.cpu().detach().numpy()[0]\n", "\n", "# Run FastPitch inference\n", "text = \"That is not only my accusation.\"\n", "\n", "tokens = fastpitch_model.parse(text, normalize=True)\n", "predicted_spectrogram = fastpitch_model.generate_spectrogram(tokens=tokens)\n", "predicted_spectrogram = predicted_spectrogram.cpu().detach().numpy()[0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "W_PiDO1Dqezk", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "# Compare the spectrograms\n", "imshow(real_spectrogram, origin=\"lower\")\n", "plt.title(\"Spectrogram\")\n", "plt.show()\n", "\n", "imshow(predicted_spectrogram, origin=\"lower\")\n", "plt.title(\"Predicted Spectrogram\")\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "id": "_a161gPwreAu", "pycharm": { "name": "#%% md\n" } }, "source": [ "As we can see, the predicted spectrogram looks very smooth and well-behaved compared to the ground truth which has a more variation and detail." ] }, { "cell_type": "markdown", "metadata": { "id": "55BH3c8Pre4l", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 7.9 Research" ] }, { "cell_type": "markdown", "metadata": { "id": "Yg2JKIqQrlCG", "pycharm": { "name": "#%% md\n" } }, "source": [ "There is ongoing research into improving the audio quality and expressiveness of models like FastPitch, with a few methods that have shown promising results being:\n", "\n", "1. Train a [variational auto-encoder](https://en.wikipedia.org/wiki/Variational_autoencoder) (VAE) that can compress all utterance-level variation/prosody into an embedding and provide it as a feature it to the decoder (https://arxiv.org/abs/1812.04342).\n", "2. Use [normalizing flows](https://arxiv.org/abs/1908.09257) (sometimes called *glow* models) to directly learn the variability in the training data (eg. [RAD-TTS](https://nv-adlr.github.io/RADTTS)).\n", "3. Use [generative adversarial networks](https://en.wikipedia.org/wiki/Generative_adversarial_network) (GAN) based training to make the predicted spectrograms harder to tell apart from real spectrograms.\n", "4. Avoid the spectrogram entirely by training an end-to-end model that can go directly from text to audio (eg. [VITS](https://arxiv.org/pdf/2106.06103.pdf))." ] }, { "cell_type": "markdown", "metadata": { "id": "vWdriq_eDfOr", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 8. Audio Synthesis (Spectrogram Inversion)" ] }, { "cell_type": "markdown", "metadata": { "id": "Z5_6Esy-Cz52", "pycharm": { "name": "#%% md\n" } }, "source": [ "Reconstructing audio from a spectrogram can theoretically be done using the [inverse FFT](https://pytorch.org/docs/stable/generated/torch.fft.ifft.html?highlight=fft) (iFFT). However there are a few complications in doing this with our model.\n", "\n", "1. The mel spectrogram does not contain the phase information output by the original FFT.\n", "2. The predicted spectrogram is imperfect. It is likely smoother than the ground truth and may contain noise or other unnatural characteristics/artifacts.\n", "\n", "We could approximate the phase information using the *Griffin-Lim* algorithm. However its just an approximation, and does not help with issue (2).\n", "\n", "So instead we train a separate model called a **vocoder** to generate the audio. This model can learn to do the audio reconstruction more accurately. If necessary we can train it directly on the ground-truth aligned spectrograms produced by our model to teach it to *denoise* the output automatically, often called **GTA fine-tuning**." ] }, { "cell_type": "markdown", "metadata": { "id": "HX8yOXy3o9Vc", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 8.1 Vocoder example" ] }, { "cell_type": "markdown", "metadata": { "id": "iHijlV2vfAzS", "pycharm": { "name": "#%% md\n" } }, "source": [ "Here we will take our audio file, compute its mel spectrogram, and then regenerate the original audio from the spectrogram using HiFiGan." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "4O5F6bQWzWv8", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "audio_path = \"LJ023-0089.wav\"\n", "\n", "# Process the ground truth audio\n", "sample_rate = 22050\n", "audio, _ = librosa.load(audio_path, sr=sample_rate)\n", "\n", "audio_tensor = torch.from_numpy(audio).unsqueeze(0).to(tacotron2_model.device)\n", "audio_len = torch.tensor(audio_tensor.shape[1], dtype=torch.long, device=tacotron2_model.device).unsqueeze(0)\n", "\n", "# Extract the spectrogram using the FastPitch audio preprocessor\n", "spectrogram, spectrogram_len = fastpitch_model.preprocessor(input_signal=audio_tensor, length=audio_len)\n", "\n", "# Invert the spectrogram\n", "reconstructed_audio = vocoder.convert_spectrogram_to_audio(spec=spectrogram)\n", "reconstructed_audio = reconstructed_audio.cpu().detach().numpy()[0]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "MjSsLoGEX7NG", "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "print(\"Original Audio\")\n", "ipd.display(ipd.Audio(audio, rate=sample_rate))\n", "\n", "print(\"Reconstructed Audio\")\n", "ipd.display(ipd.Audio(reconstructed_audio, rate=sample_rate))" ] }, { "cell_type": "markdown", "metadata": { "id": "Euxpd50wieyD", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 8.2 Modeling approach" ] }, { "cell_type": "markdown", "metadata": { "id": "9EUITiXCiWkS", "pycharm": { "name": "#%% md\n" } }, "source": [ "Spectrogram inversion is a *sequence-to-sequence* problem.\n", "\n", "The input sequence is the mel spectrogram with 80 mel bands for each audio frame, and the output is a sequence of audio samples. For example, the input sequence may have 100 audio frames making it dimension [100, 80] and have a corresponding audio output sequence of dimension [25,600, 1]\n", "\n", "The audio samples being discrete 16-bit integers can be predicted using a *softmax* layer size 216. Or we can divide them by 216 to scale them to the range [-1, 1] and then predict them as a continuous value using an appropriate activation like *tanh*.\n", "\n", "Most sequence-to-sequence problems rely on attention or other upsampling methods, but in this case it is unnecessary because the ratio of input to output elements is fixed. So given any input, we already know the final output length and which output samples correspond to which input frames.\n", "\n", "The ratio is dependent on length of each audio frame.\n", "\n", "In NeMo, for 22.05 kHz audio, we use a audio window **stride** of 256 meaning there are exactly 256 audio samples for each audio spectrogram frame. For 44.1 kHz we double the stride to 512.\n", "\n", "This means each audio frame represents approximately (256 / 22.05) = 11.61 ms of audio.\n", "\n", "The first step for most approaches is to **upsample** our input sequence to match the final output length. This can be done by *duplicating* each audio frame a number of times equal to your audio stride/ratio.\n", "\n", "Or if your stride is a power of 2 (like the ones we selected) then you can upsample the sequence more effectively using *transposed convolutions* (aka. *deconvolutional layers*).\n", "\n", "Once the input and output sequences are the same length, you can use any number of models to predict the output." ] }, { "cell_type": "markdown", "metadata": { "id": "Ri1_RURKjiss", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 8.3 WaveNet\n" ] }, { "cell_type": "markdown", "metadata": { "id": "TgsXXDamSKfH", "pycharm": { "name": "#%% md\n" } }, "source": [ "The first mainstream vocoder was [WaveNet](https://www.deepmind.com/blog/wavenet-a-generative-model-for-raw-audio), a very large auto-regressive *dilated causal CNN*. For each predicted audio sample, the CNN predicts a softmax probability across all possible 216 sample values.\n", "\n", "Here auto-regressive means every audio sample is predicted by looking at the entire history of previously predicted audio samples. Dilated means it samples a few past audio samples spread over a wide range, instead of processing every past sample or only recent samples. Causal means the CNN only uses information from past predictions; it does not condition on audio samples after it.\n", "\n", "
\n", "
\n", "\n", "
Diagram of a dilated causal CNN
\n", "
\n", "
\n", "\n", "This system is very accurate, but also very slow. Largely because of the auto-regressive system predicting 1 sample at a time for an output that is 10,000s to 100,000s of samples long. Its **real-time factor (RTF)** is on the order of 100, meaning it takes up to 100 seconds to reconstruct 1 second of audio. This makes it impractical to use in most applications." ] }, { "cell_type": "markdown", "metadata": { "id": "PYXZjgEEjndF", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 8.4 HiFi-GAN" ] }, { "cell_type": "markdown", "metadata": { "id": "h19_oQ1GSMe8", "pycharm": { "name": "#%% md\n" } }, "source": [ "The most common vocoder used in NeMo is [HiFi-GAN](https://arxiv.org/abs/2010.05646). The architecture is similar to WaveNet. The main difference is that it is smaller, and not auto-regressive. This allows it to reach a speed of around 0.01 RTF; 10,000x faster than WaveNet.\n", "\n", "The main way that HiFi-GAN achieves this speed with minimal sacrifice to perceived audio quality is through *adversarial training*, treating the vocoder like a **generative adversarial network (GAN)**.\n", "\n", "To do this, we jointly train the vocoder with a set of *discriminators* that try to predict whether the synthesized/reconstructed audio is real or not. The goal being to trick the discriminators so they cannot systematically tell the difference between the synthesized audio and the original audio. Doing this allows us to get realistic audio with a significantly smaller model.\n", "\n", "Notably it uses multiple **scale discriminators**, which are temporal CNNs that try to classify the audio as real or fake after average pooling sets of adjacent audio samples, and **period discriminators** which try to classify using audio sampled over different periods. \n", "\n", "In addition to penalizing the model if the discriminator can classify the synthesized audio as fake, it also uses **feature matching loss** to penalize the model if the distribution of intermediate layer outputs in the discriminator networks differ between the real and synthesized audio.\n", "\n", "
\n", "\n", "
HiFi-Gan scale and period discriminators
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "8HRHKLlJSOw-", "pycharm": { "name": "#%% md\n" } }, "source": [ "## 8.5 WaveRNN" ] }, { "cell_type": "markdown", "metadata": { "id": "nbM716VLSzQg", "pycharm": { "name": "#%% md\n" } }, "source": [ "Many people tend to avoid RNNs because of how slowly they train compared to CNNs and Transformers.\n", "\n", "However, they are one of the most used architectures in real-world applications and industry due to their small compute requirements and memory footprint at inference time. This makes them ideal for low-latency and low-power situations like *on-device* TTS.\n", "\n", "[WaveRNN](https://arxiv.org/abs/1802.08435) is a popular vocoder for those who want to do spectrogram inversion on-device.\n", "\n", "The model itself manages to minimize compute requirements in several different ways that are too complex to summarize adequately here. But a few highlights would be:\n", "\n", "1. Instead of using a 216 sized softmax, it uses two 28 sized softmaxes to predict the first and last 8-bits of the 16-bit audio sample separately.\n", "2. Uses a *subscaling* system to have the RNN predict 16 audio samples at a time instead of 1 sample at a time.\n", "3. Uses *sparse RNN training* during which the smallest model weights in the RNN are periodically set to zero. Inference is then done efficiently using sparse vectors." ] }, { "cell_type": "markdown", "metadata": { "id": "aozxSufVJa0l", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 9. Model Evaluation" ] }, { "cell_type": "markdown", "metadata": { "id": "I8522HduJmHM", "pycharm": { "name": "#%% md\n" } }, "source": [ "There are no well-established objective metrics for evaluating how good a TTS model is. Rather, quality is usually based on human opinion or perception, commonly measured through surveys.\n", "\n", "The most common type of survey for evaluating TTS quality is [mean opinion score](https://en.wikipedia.org/wiki/Mean_opinion_score) (**MOS**), in which listeners rate the quality of TTS samples on a 1 to 5 scale.\n", "\n", "Another common alternative is [MUSHRA](https://en.wikipedia.org/wiki/MUSHRA) in which users are provided a reference (the ground truth audio) and asked to rate several TTS samples relative to the reference.\n", "\n", "There are some metrics which are occasionally used to try and measure audio quality such as [MCD-DTW](https://github.com/MattShannon/mcd), [PESQ](https://en.wikipedia.org/wiki/Perceptual_Evaluation_of_Speech_Quality), and [STOI](https://torchmetrics.readthedocs.io/en/stable/audio/short_time_objective_intelligibility.html). But these have very limited accuracy and usefulness.\n", "\n", "The lack of objective numerical metrics that can be trained on is a large reason as to why many state of the art models rely on GAN based training to get good quality." ] }, { "cell_type": "markdown", "metadata": { "id": "OgtWptQ5tGlq", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 10. Additional Resources" ] }, { "cell_type": "markdown", "metadata": { "id": "wtJINtrStHvJ", "pycharm": { "name": "#%% md\n" } }, "source": [ "To learn more about what TTS technology and models are available in NeMo, please look through our [documentation](https://docs.nvidia.com/deeplearning/nemo/user-guide/docs/en/stable/tts/intro.html#).\n", "\n", "To get more hands on experience with NeMo TTS, look through some of our other [tutorials](https://github.com/NVIDIA/NeMo/tree/stable/tutorials/tts).\n", "\n", "* Running pretrained models: [Inference_ModelSelect](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/tts/Inference_ModelSelect.ipynb)\n", "* FastPitch [training](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/tts/FastPitch_MixerTTS_Training.ipynb) and [fine-tuning](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/tts/FastPitch_Finetuning.ipynb)\n", "\n", "To learn how to deploy and serve your TTS models, visit [Riva](https://docs.nvidia.com/deeplearning/riva/index.html)." ] }, { "cell_type": "markdown", "metadata": { "id": "V0Og5xzqVwp8", "pycharm": { "name": "#%% md\n" } }, "source": [ "# 11. References" ] }, { "cell_type": "markdown", "metadata": { "id": "0K_QsdvOVx1M", "pycharm": { "name": "#%% md\n" } }, "source": [ "1. [Speech Processing Primer](https://aman.ai/primers/ai/speech-processing/)\n", "2. [TTS Survey](https://arxiv.org/abs/2106.15561)\n", "3. [Tacotron 2](https://google.github.io/tacotron/publications/tacotron2/index.html)\n", "4. [Non-Attentive Tacotron](https://arxiv.org/abs/2010.04301)\n", "5. [FastPitch](https://fastpitch.github.io/)\n", "6. [RAD-TTS](https://nv-adlr.github.io/RADTTS)\n", "7. [VITS](https://arxiv.org/pdf/2106.06103.pdf)\n", "8. [One TTS Alignment To Rule Them All](https://arxiv.org/abs/2108.10447)\n", "9. [Parallel Models](https://arxiv.org/abs/2202.13066)\n", "10. [WaveNet](https://www.deepmind.com/blog/wavenet-a-generative-model-for-raw-audio)\n", "11. [HiFi-GAN](https://arxiv.org/abs/2010.05646)\n", "12. [WaveRNN](https://arxiv.org/abs/1802.08435)\n", "13. [PyTorch](https://pytorch.org/docs/stable/index.html)\n", "14. [CMU Pronouncing Dictionary](http://www.speech.cs.cmu.edu/cgi-bin/cmudict)" ] } ], "metadata": { "colab": { "collapsed_sections": [], "provenance": [], "toc_visible": true }, "gpuClass": "standard", "kernelspec": { "display_name": "Python 3.9.15 ('ptl_venv')", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.15" }, "vscode": { "interpreter": { "hash": "f8a1d50fd7b1e17bd198f085b8ced031398c6134b0da7c4415c17601bbcccc4e" } } }, "nbformat": 4, "nbformat_minor": 1 }