{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4eebf3f6",
   "metadata": {},
   "source": [
    "# FastPitch and Mixer-TTS Training\n",
    "\n",
    "This notebook is designed to provide a guide on how to train FastPitch and Mixer-TTS as part of the TTS pipeline. It contains the following sections:\n",
    "  1. **Introduction**: FastPitch and Mixer-TTS in NeMo\n",
    "  2. **Preprocessing**: how to prepare data for FastPitch and Mixer-TTS \n",
    "  3. **Training**: example of FastPitch training and Mixer-TTS training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37074ede",
   "metadata": {},
   "source": [
    "# License\n",
    "\n",
    "> Copyright 2022 NVIDIA. All Rights Reserved.\n",
    "> \n",
    "> Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "> you may not use this file except in compliance with the License.\n",
    "> You may obtain a copy of the License at\n",
    "> \n",
    ">     http://www.apache.org/licenses/LICENSE-2.0\n",
    "> \n",
    "> Unless required by applicable law or agreed to in writing, software\n",
    "> distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "> See the License for the specific language governing permissions and\n",
    "> limitations under the License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "261df0a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "You can either run this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
    "Instructions for setting up Colab are as follows:\n",
    "1. Open a new Python 3 notebook.\n",
    "2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
    "3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
    "4. Run this cell to set up dependencies# .\n",
    "\"\"\"\n",
    "BRANCH = 'r1.17.0'\n",
    "# # If you're using Colab and not running locally, uncomment and run this cell.\n",
    "# !apt-get install sox libsndfile1 ffmpeg\n",
    "# !pip install wget text-unidecode scipy==1.7.3\n",
    "# !python -m pip install git+https://github.com/NVIDIA/NeMo.git@$BRANCH#egg=nemo_toolkit[all]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e0c0d38",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import nemo\n",
    "import torch\n",
    "import librosa\n",
    "import numpy as np\n",
    "\n",
    "from pathlib import Path\n",
    "from tqdm.notebook import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "efa2c292",
   "metadata": {},
   "source": [
    "# Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95884fcd",
   "metadata": {},
   "source": [
    "### FastPitch\n",
    "\n",
    "FastPitch is non-autoregressive model for mel-spectrogram generation based on FastSpeech, conditioned on fundamental frequency contours. For more details about model, please refer to the original [paper](https://arxiv.org/abs/2006.06873). NeMo re-implementation of FastPitch additionally uses unsupervised speech-text [aligner](https://arxiv.org/abs/2108.10447) which was originally implemented in [FastPitch 1.1](https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/FastPitch).\n",
    "\n",
    "### Mixer-TTS\n",
    "\n",
    "Mixer-TTS is another non-autoregressive model for mel-spectrogram generation. It is structurally similar to FastPitch: duration prediction, pitch prediction, unsupervised TTS alignment framework, but the main difference is that Mixer-TTS is based on the [MLP-Mixer](https://arxiv.org/abs/2105.01601) architecture adapted for speech synthesis.\n",
    "\n",
    "FastPitch and Mixer-TTS like most NeMo models are defined as a LightningModule, allowing for easy training via PyTorch Lightning, and parameterized by a configuration, currently defined via a yaml file and loading using Hydra.\n",
    "\n",
    "Let's take a look using NeMo's pretrained models and how to use it to generate spectrograms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9be422ee",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo.collections.tts.models.base import SpectrogramGenerator\n",
    "from nemo.collections.tts.models import FastPitchModel, MixerTTSModel\n",
    "\n",
    "from matplotlib.pyplot import imshow\n",
    "from matplotlib import pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdf4aee7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's see what pretrained models are available for FastPitch and Mixer-TTS\n",
    "print(\"FastPitch pretrained models:\")\n",
    "print(FastPitchModel.list_available_models())\n",
    "print(\"=====================================\")\n",
    "print(\"Mixer-TTS pretrained models:\")\n",
    "print(MixerTTSModel.list_available_models())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "298704c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can load the pre-trained FastModel as follows\n",
    "pretrained_model = \"tts_en_fastpitch\"\n",
    "spec_gen = FastPitchModel.from_pretrained(pretrained_model)\n",
    "spec_gen.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c18181ff",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# In the same way, we can load the pre-trained Mixer-TTS model as follows\n",
    "pretrained_model = \"tts_en_lj_mixertts\"\n",
    "spec_gen = MixerTTSModel.from_pretrained(pretrained_model)\n",
    "spec_gen.eval();"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb41b646",
   "metadata": {},
   "outputs": [],
   "source": [
    "assert isinstance(spec_gen, SpectrogramGenerator)\n",
    "\n",
    "if isinstance(spec_gen, FastPitchModel):\n",
    "    tokens = spec_gen.parse(str_input=\"Hey, this produces speech!\")\n",
    "else:\n",
    "    tokens = spec_gen.parse(text=\"Hey, this produces speech!\")\n",
    "\n",
    "spectrogram = spec_gen.generate_spectrogram(tokens=tokens)\n",
    "\n",
    "# Now we can visualize the generated spectrogram\n",
    "# If we want to generate speech, we have to use a vocoder in conjunction to a spectrogram generator.\n",
    "# Refer to the Inference_ModelSelect notebook on how to convert spectrograms to speech.\n",
    "imshow(spectrogram.cpu().detach().numpy()[0,...], origin=\"lower\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54ec3c5e",
   "metadata": {},
   "source": [
    "# Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ef87e31",
   "metadata": {},
   "outputs": [],
   "source": [
    "from nemo.collections.tts.g2p.modules import EnglishG2p\n",
    "from nemo.collections.tts.data.tts_dataset import TTSDataset\n",
    "from nemo_text_processing.text_normalization.normalize import Normalizer\n",
    "from nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers import EnglishPhonemesTokenizer, EnglishCharsTokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fd5dec0",
   "metadata": {},
   "source": [
    "We will show example of preprocessing and training using small part of AN4 dataset. It consists of recordings of people spelling out addresses, names, telephone numbers, etc., one letter or number at a time, as well as their corresponding transcripts. Let's download data, prepared manifests and supplementary files.\n",
    "\n",
    "*NOTE: The sample data is not enough data to properly train a FastPitch or Mixer-TTS model. This will not result in a trained model and is just used as an example.*\n",
    "\n",
    "Let's download everything that we need for this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b621b1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# download data and manifests\n",
    "!wget https://github.com/NVIDIA/NeMo/releases/download/v0.11.0/test_data.tar.gz && mkdir -p tests/data && tar xzf test_data.tar.gz -C tests/data\n",
    "\n",
    "# additional files\n",
    "!mkdir -p tts_dataset_files && cd tts_dataset_files \\\n",
    "&& wget https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/scripts/tts_dataset_files/cmudict-0.7b_nv22.10 \\\n",
    "&& wget https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/scripts/tts_dataset_files/heteronyms-052722 \\\n",
    "&& cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45f19be7",
   "metadata": {},
   "source": [
    "### FastPitch\n",
    "\n",
    "Now that we looked at the FastPitch model, let's see how to prepare all data for training it. \n",
    "\n",
    "Firstly, let's download all necessary training scripts and configs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e76d950",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/examples/tts/fastpitch.py\n",
    "\n",
    "!mkdir -p conf && cd conf \\\n",
    "&& wget https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/examples/tts/conf/fastpitch_align_v1.05.yaml \\\n",
    "&& cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82a2eacb",
   "metadata": {},
   "source": [
    "TTS text preprocessing pipeline consists of two stages: text normalization and text tokenization. Both of them can be handled by `nemo.collections.tts.data.tts_dataset.TTSDataset` for training.  \n",
    "\n",
    "Our current example dataset is in English, so let's use `nemo_text_processing.text_normalization.normalize.Normalizer` for normalization which supports English (and many other languages!) and `nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.EnglishCharsTokenizer`. So, our model will receive grapheme representation of text (graphemes) as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a46da66d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text normalizer\n",
    "text_normalizer = Normalizer(\n",
    "    lang=\"en\", \n",
    "    input_case=\"cased\", \n",
    ")\n",
    "\n",
    "text_normalizer_call_kwargs = {\n",
    "    \"punct_pre_process\": True,\n",
    "    \"punct_post_process\": True\n",
    "}\n",
    "\n",
    "# Text tokenizer\n",
    "text_tokenizer = EnglishCharsTokenizer()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "884d8d82",
   "metadata": {},
   "source": [
    "To accelerate and stabilize our training, we also need to extract pitch for every audio, estimate pitch statistics (mean and std) and pre-calculate alignment prior matrices for alignment framework. To do this, all we need to do is iterate over our data one time.\n",
    "\n",
    "In the below method the arguments are as follows:\n",
    "- `sup_data_path` — path to the folder which contains supplementary data. If the supplementary data or the folder does not already exists then it will be created.\n",
    "\n",
    "- `sup_data_types` — types of supplementary data to be provided to the model.\n",
    "\n",
    "- `text_tokenizer` — text tokenizer object that we already created.\n",
    "\n",
    "- `text_normalizer` — text normalizer object that we already created.\n",
    "\n",
    "- `text_normalizer_call_kwargs` — dictionary of arguments to be used in calling the text normalizer that we already created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7108f748",
   "metadata": {},
   "outputs": [],
   "source": [
    "def pre_calculate_supplementary_data(sup_data_path, sup_data_types, text_tokenizer, text_normalizer, text_normalizer_call_kwargs):\n",
    "    # init train and val dataloaders\n",
    "    stages = [\"train\", \"val\"]\n",
    "    stage2dl = {}\n",
    "    for stage in stages:\n",
    "        ds = TTSDataset(\n",
    "            manifest_filepath=f\"tests/data/asr/an4_{stage}.json\",\n",
    "            sample_rate=16000,\n",
    "            sup_data_path=sup_data_path,\n",
    "            sup_data_types=sup_data_types,\n",
    "            n_fft=1024,\n",
    "            win_length=1024,\n",
    "            hop_length=256,\n",
    "            window=\"hann\",\n",
    "            n_mels=80,\n",
    "            lowfreq=0,\n",
    "            highfreq=8000,\n",
    "            text_tokenizer=text_tokenizer,\n",
    "            text_normalizer=text_normalizer,\n",
    "            text_normalizer_call_kwargs=text_normalizer_call_kwargs\n",
    "\n",
    "        ) \n",
    "        stage2dl[stage] = torch.utils.data.DataLoader(ds, batch_size=1, collate_fn=ds._collate_fn, num_workers=1)\n",
    "\n",
    "    # iteration over dataloaders\n",
    "    pitch_mean, pitch_std, pitch_min, pitch_max = None, None, None, None\n",
    "    for stage, dl in stage2dl.items():\n",
    "        pitch_list = []\n",
    "        for batch in tqdm(dl, total=len(dl)):\n",
    "            tokens, tokens_lengths, audios, audio_lengths, attn_prior, pitches, pitches_lengths = batch\n",
    "            pitch = pitches.squeeze(0)\n",
    "            pitch_list.append(pitch[pitch != 0])\n",
    "\n",
    "        if stage == \"train\":\n",
    "            pitch_tensor = torch.cat(pitch_list)\n",
    "            pitch_mean, pitch_std = pitch_tensor.mean().item(), pitch_tensor.std().item()\n",
    "            pitch_min, pitch_max = pitch_tensor.min().item(), pitch_tensor.max().item()\n",
    "            \n",
    "    return pitch_mean, pitch_std, pitch_min, pitch_max"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1affe50",
   "metadata": {},
   "outputs": [],
   "source": [
    "fastpitch_sup_data_path = \"fastpitch_sup_data_folder\"\n",
    "sup_data_types = [\"align_prior_matrix\", \"pitch\"]\n",
    "\n",
    "pitch_mean, pitch_std, pitch_min, pitch_max = pre_calculate_supplementary_data(\n",
    "    fastpitch_sup_data_path, sup_data_types, text_tokenizer, text_normalizer, text_normalizer_call_kwargs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d868bb48",
   "metadata": {},
   "source": [
    "### Mixer-TTS\n",
    "\n",
    "Now, let's see how to prepare data for training Mixer-TTS. \n",
    "\n",
    "Firstly, let's download all necessary training scripts and configs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c7c0cfc",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/examples/tts/mixer_tts.py\n",
    "\n",
    "!mkdir -p conf && cd conf \\\n",
    "&& wget https://raw.githubusercontent.com/NVIDIA/NeMo/$BRANCH/examples/tts/conf/mixer-tts.yaml \\\n",
    "&& cd .."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2f10886",
   "metadata": {},
   "source": [
    "In the FastPitch pipeline we used a char-based tokenizer, but in the Mixer-TTS training pipeline we would like to demonstrate a phoneme-based tokenizer `nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.EnglishPhonemesTokenizer`. Unlike char-based tokenizer, `EnglishPhonemesTokenizer` needs a phoneme dictionary and a heteronym dictionary. We will be using the same `nemo_text_processing.text_normalization.normalize.Normalizer` for normalizing the text as used in the FastPitch example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6ba0f9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Text normalizer\n",
    "text_normalizer = Normalizer(\n",
    "    lang=\"en\", \n",
    "    input_case=\"cased\", \n",
    ")\n",
    "\n",
    "text_normalizer_call_kwargs = {\n",
    "    \"punct_pre_process\": True,\n",
    "    \"punct_post_process\": True\n",
    "}\n",
    "\n",
    "# Grapheme-to-phoneme module\n",
    "g2p = EnglishG2p(\n",
    "    phoneme_dict=\"tts_dataset_files/cmudict-0.7b_nv22.10\",\n",
    "    heteronyms=\"tts_dataset_files/heteronyms-052722\"\n",
    ")\n",
    "\n",
    "# Text tokenizer\n",
    "text_tokenizer = EnglishPhonemesTokenizer(\n",
    "    punct=True,\n",
    "    stresses=True,\n",
    "    chars=True,\n",
    "    apostrophe=True,\n",
    "    pad_with_space=True,\n",
    "    g2p=g2p,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fc55415",
   "metadata": {},
   "source": [
    "Just like in FastPitch we will need to extract pitch for every audio, estimate pitch statistics (mean and std) and pre-calculate alignment prior matrices for alignment framework."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aabc1f0f",
   "metadata": {},
   "outputs": [],
   "source": [
    "mixer_tts_sup_data_path = \"mixer_tts_sup_data_folder\"\n",
    "sup_data_types = [\"align_prior_matrix\", \"pitch\"]\n",
    "\n",
    "pitch_mean, pitch_std, pitch_min, pitch_max = pre_calculate_supplementary_data(\n",
    "    mixer_tts_sup_data_path, sup_data_types, text_tokenizer, text_normalizer, text_normalizer_call_kwargs\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0711ec6",
   "metadata": {},
   "source": [
    "# Training"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0a95848c",
   "metadata": {},
   "source": [
    "### FastPitch\n",
    "\n",
    "Now we are ready for training our model! Let's try to train FastPitch.\n",
    "\n",
    "*NOTE: The sample data is not enough data to properly train a FastPitch. This will not result in a trained FastPitch and is used to just as example.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cc1a9107",
   "metadata": {},
   "outputs": [],
   "source": [
    "!(python fastpitch.py --config-name=fastpitch_align_v1.05.yaml \\\n",
    "  sample_rate=16000 \\\n",
    "  train_dataset=tests/data/asr/an4_train.json \\\n",
    "  validation_datasets=tests/data/asr/an4_val.json \\\n",
    "  sup_data_types=\"['align_prior_matrix', 'pitch']\" \\\n",
    "  sup_data_path={fastpitch_sup_data_path} \\\n",
    "  pitch_mean={pitch_mean} \\\n",
    "  pitch_std={pitch_std} \\\n",
    "  pitch_fmin={pitch_min} \\\n",
    "  pitch_fmax={pitch_max} \\\n",
    "  ~model.text_tokenizer \\\n",
    "  +model.text_tokenizer._target_=nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.EnglishCharsTokenizer \\\n",
    "  +trainer.max_steps=100 ~trainer.max_epochs \\\n",
    "  trainer.check_val_every_n_epoch=25 \\\n",
    "  +trainer.max_epochs=5 \\\n",
    "  model.train_ds.dataloader_params.batch_size=24 \\\n",
    "  model.validation_ds.dataloader_params.batch_size=24 \\\n",
    "  exp_manager.exp_dir=./fastpitch_log_dir \\\n",
    "  model.n_speakers=1 trainer.devices=1 trainer.strategy=null \\\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d6bce3ce",
   "metadata": {},
   "source": [
    "Let's look at some of the options in the training command:\n",
    "\n",
    "- *`~model.text_tokenizer`* — remove default text tokenizer. The default tokenizer in the `fastpitch_align_v1.05.yaml` is `nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.EnglishPhonemesTokenizer`, but we want to use `nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.EnglishCharsTokenizer`.\n",
    "\n",
    "- *`+model.text_tokenizer._target_`* — add `nemo.collections.common.tokenizers.text_to_speech.tts_tokenizers.EnglishCharsTokenizer` as text tokenizer class."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d9745fc",
   "metadata": {},
   "source": [
    "### Mixer-TTS\n",
    "\n",
    "Now we are ready for training our model! Let's try to train Mixer-TTS.\n",
    "\n",
    "*NOTE: The sample data is not enough data to properly train a Mixer-TTS. This will not result in a trained Mixer-TTS and is used to just as example.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8db3a903",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!python mixer_tts.py sample_rate=16000 \\\n",
    "train_dataset=tests/data/asr/an4_train.json \\\n",
    "validation_datasets=tests/data/asr/an4_val.json \\\n",
    "sup_data_types=\"['align_prior_matrix', 'pitch']\" \\\n",
    "sup_data_path={mixer_tts_sup_data_path} \\\n",
    "phoneme_dict_path=tts_dataset_files/cmudict-0.7b_nv22.10 \\\n",
    "heteronyms_path=tts_dataset_files/heteronyms-052722 \\\n",
    "pitch_mean={pitch_mean} \\\n",
    "pitch_std={pitch_std} \\\n",
    "model.train_ds.dataloader_params.batch_size=6 \\\n",
    "model.train_ds.dataloader_params.num_workers=0 \\\n",
    "model.validation_ds.dataloader_params.num_workers=0 \\\n",
    "trainer.max_epochs=3 \\\n",
    "trainer.strategy=null \\\n",
    "trainer.check_val_every_n_epoch=1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a00f8b88",
   "metadata": {},
   "source": [
    "That's it!\n",
    "\n",
    "In order to train FastPitch and Mixer-TTS for real purposes, it is highly recommended to obtain high quality speech data with the following properties:\n",
    "\n",
    "* Sampling rate of 22050Hz or higher\n",
    "* Single speaker\n",
    "* Speech should contain a variety of speech phonemes\n",
    "* Audio split into segments of 1-10 seconds\n",
    "* Audio segments should not have silence at the beginning and end\n",
    "* Audio segments should not contain long silences inside"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.15 ('ptl_venv')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "f8a1d50fd7b1e17bd198f085b8ced031398c6134b0da7c4415c17601bbcccc4e"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}