tutorials/tts/Evaluation_MelCepstralDistortion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f0408502",
   "metadata": {},
   "source": [
    "# Model Evaluation: Mel Cepstral Distortion with Dynamic Time Warping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a749b10",
   "metadata": {},
   "source": [
    "In this tutorial, we will learn how to calculate **mel cepstral distortion (MCD)** with **dynamic time warping (DTW)** between a synthesized mel spec and a reference mel spec. MCD DTW can be useful for comparing models trained on the same training data.\n",
    "\n",
    "MCD is an objective measure of speech quality that is calculated between pairs of TTS-generated mel spectrograms and ground truth mel spectrograms. Two mel spectrograms are similar if the MCD value between them is low; as you might expect, the MCD of a mel spec with itself is 0.\n",
    "\n",
    "MCD DTW is a modification of MCD that works with non-aligned mels by using a dynamic time warping cost matrix. (Vanilla MCD can only be measured for two mels that are both the same length, and which are assumed to be aligned.) The scale depends on factors such as the mel extractor and reduction algorithm (mean of DTW cost or min DTW path cost)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8801b8cb",
   "metadata": {},
   "source": [
    "## License\n",
    "\n",
    "> Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES.  All rights reserved.\n",
    ">\n",
    "> Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "> you may not use this file except in compliance with the License.\n",
    "> You may obtain a copy of the License at\n",
    ">\n",
    ">     http://www.apache.org/licenses/LICENSE-2.0\n",
    ">\n",
    "> Unless required by applicable law or agreed to in writing, software\n",
    "> distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "> See the License for the specific language governing permissions and\n",
    "> limitations under the License."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09cff1ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "You can either run this notebook locally (if you have all the dependencies and a GPU) or on Google Colab.\n",
    "Instructions for setting up Colab are as follows:\n",
    "1. Open a new Python 3 notebook.\n",
    "2. Import this notebook from GitHub (File -> Upload Notebook -> \"GITHUB\" tab -> copy/paste GitHub URL)\n",
    "3. Connect to an instance with a GPU (Runtime -> Change runtime type -> select \"GPU\" for hardware accelerator)\n",
    "4. Run this cell to set up dependencies.\n",
    "\"\"\"\n",
    "BRANCH = 'r1.17.0'\n",
    "# If you're using Google Colab and not running locally, uncomment and run this cell.\n",
    "# !pip install librosa numpy matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b9b6e1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Import libraries\n",
    "import IPython.display as ipd\n",
    "import librosa\n",
    "import numpy as np\n",
    "import math\n",
    "import matplotlib.pyplot as plt\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "398b01e0",
   "metadata": {},
   "source": [
    "## Defining Parameters for Spectrogram Generation\n",
    "\n",
    "In this section we define parameters required to generate our mel spectrograms. More information about these parameters can found in the Librosa documentation for [stft](https://librosa.org/doc/main/generated/librosa.stft.html), [mels](https://librosa.org/doc/main/generated/librosa.filters.mel.html) and [mfcc](https://librosa.org/doc/main/generated/librosa.feature.mfcc.html) generation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "191012de",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Mel spec params\n",
    "n_fft=1024\n",
    "hop_length=256\n",
    "win_length=None\n",
    "window='hann'\n",
    "n_mels = 80\n",
    "sr = 22050\n",
    "\n",
    "## Mfcc params\n",
    "n_mfcc=34"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54830c8c",
   "metadata": {},
   "source": [
    "A little bit about these parameters:  \n",
    "    - `n_fft`: Number of fft_components or length of windowed signal after padding in STFT.  \n",
    "    - `hop_length`: Number of audio samples between adjacent STFT columns.  \n",
    "    - `window`: Window to use in STFT.  \n",
    "    - `win_length`: Length of the window to be used.  \n",
    "    - `n_mels`: Number of number of mel bands to generate.  \n",
    "    - `sr`: Sample rate of the samples.  \n",
    "    - `n_mfcc`: Number of MFCCs to generate."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cb9e37c",
   "metadata": {},
   "source": [
    "## Loading and Visualizing Data\n",
    "\n",
    "Lets apply the algorithm on a synthesized mel and ground truth mel pair for understanding.\n",
    "\n",
    "First, we need a function to generate mel spectrograms from audio files. Mel spectrograms are generated using the [librosa mel extractor](https://librosa.org/doc/main/generated/librosa.filters.mel.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "042ceb7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def wav2mel(filename):\n",
    "    \"\"\"\n",
    "    Function to load an audio file and generate/return mel specs:\n",
    "    Args:\n",
    "        filename: Full path of the audio file.\n",
    "    Returns:\n",
    "        mels: Corresponding mel spectrogram.\n",
    "    \"\"\"\n",
    "    wav_, _ = librosa.load(filename, sr=sr)  # load() returns an (audio data, sample rate) tuple\n",
    "    mels = librosa.feature.melspectrogram(\n",
    "        y=wav_,\n",
    "        sr=sr,\n",
    "        n_fft=n_fft,\n",
    "        hop_length=hop_length,\n",
    "        win_length=win_length,\n",
    "        window=window,\n",
    "        n_mels=n_mels\n",
    "    )\n",
    "    mels = librosa.power_to_db(mels, ref=np.max)\n",
    "    return mels"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8546edb5",
   "metadata": {},
   "source": [
    "We also need a function to convert mels to MFCC, we will use Librosa's [mfcc](https://librosa.org/doc/main/generated/librosa.feature.mfcc.html) generation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1a0663d",
   "metadata": {},
   "outputs": [],
   "source": [
    "def mel2mfcc(mels):\n",
    "    mfcc = librosa.feature.mfcc(S=mels, n_mfcc=n_mfcc)\n",
    "    return mfcc"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6efac554",
   "metadata": {},
   "source": [
    "For this tutorial, we have already generated mels from trained [FastPitch](https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nemo/models/tts_en_fastpitch) and [RAD-TTS](https://github.com/NVIDIA/radtts) models, and we have the ground truth sample that they correspond to. Let's first download the tarball with the data and expand it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "156290a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Untar the example files.\n",
    "!wget https://tts-tutorial-data.s3.us-east-2.amazonaws.com/MCD_DTW_examples.tar\n",
    "!tar -xvf MCD_DTW_examples.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7aeabe75",
   "metadata": {},
   "source": [
    "Now we can generate a mel spectrogram for the ground truth audio, load mel specs for generated audio and generate MFCC(Mel frequency cepstral coefficient) for both."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1a71f88",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "## Generate spectrograms\n",
    "gt_mels = wav2mel(\"MCD_DTW/gt/sample_0.wav\")\n",
    "synt_mels = np.load(\"MCD_DTW/fastpitch/mels/mels_0.npy\")\n",
    "\n",
    "## Generate MFCCs\n",
    "gt_mfcc = mel2mfcc(gt_mels)\n",
    "synt_mfcc = mel2mfcc(synt_mels)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "faa5df64",
   "metadata": {},
   "source": [
    "### Visualizing the Audio\n",
    "\n",
    "Let's first listen to the ground truth and synthesized audio."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08f08b89",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Ground truth audio:\")\n",
    "audio_gt, sr_gt = librosa.load(\"MCD_DTW/gt/sample_0.wav\")\n",
    "ipd.display(ipd.Audio(audio_gt, rate=sr_gt))\n",
    "\n",
    "print(\"Synthesized audio:\")\n",
    "audio_fp, sr_fp = librosa.load(\"MCD_DTW/fastpitch/audio/sample_0.wav\")\n",
    "ipd.display(ipd.Audio(audio_fp, rate=sr_fp))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21b5c780",
   "metadata": {},
   "source": [
    "Now, let's take a look at the mel spectrograms."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "550d9b12",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize mel spectrograms\n",
    "fig, ax = plt.subplots(2, 1, figsize=(12,10))\n",
    "\n",
    "_ = ax[0].pcolormesh(gt_mels, cmap='viridis')\n",
    "_ = ax[0].set_title(\"Ground Truth Mel Spectrogram\")\n",
    "\n",
    "_ = ax[1].pcolormesh(synt_mels, cmap='viridis')\n",
    "_ = ax[1].set_title(\"FastPitch Synthesized Mel Spectrogram\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93c4c301",
   "metadata": {},
   "source": [
    "## Calculating the DTW matrix\n",
    "\n",
    "Dynamic time warping finds the most optimal path to align two sequences of different length, and in doing so measures the similarity between the two time series that are not in sync.\n",
    "\n",
    "DTW uses dynamic programming to **calculate the cost of every alignment path and chooses the path with least accumulated cost**. The value of accumulated cost matrix $D$ at index $(x_a, y_b)$ is the minimum distance between the points, where $x$ and $y$ are two time series. Formally, the cost matrix can be defined as:\n",
    "  \n",
    "$D(a,b) = min(D(a-1, b), D(a, b-1), D(a-1, b-1)) + c(x_{a}, y_{b})$  \n",
    "$D(1, b) = \\sum(c(1, y_{b}))$  \n",
    "$D(a, 1) = \\sum(c(x_{a}, 1))$ \n",
    "  \n",
    "Where ***x*** and ***y*** are the audio signals and ***c*** is the log cost function we have defined.\n",
    "\n",
    "We will use [the DTW function from Librosa](https://librosa.org/doc/main/generated/librosa.sequence.dtw.html) on MFCC. It returns the DTW accumulated cost matrix and DTW optimum path.\n",
    "\n",
    "In the following function, we define the cost function for DTW to pass in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a4adae2",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Define the cost function for calculating DTW\n",
    "def log_spec_dB_dist(x, y):\n",
    "    log_spec_dB_const = 10.0 / math.log(10.0) * math.sqrt(2.0)\n",
    "    diff = x - y\n",
    "    return log_spec_dB_const * math.sqrt(np.inner(diff, diff))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca8c6974",
   "metadata": {},
   "source": [
    "Now we can run Librosa's `dtw()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0c4c5338",
   "metadata": {},
   "outputs": [],
   "source": [
    "# dtw_cost: Shape (N, M), where N and M are the lengths of gt_mfcc, synt_mfcc respectively. Cost matrix.\n",
    "# dtw_min_path: Shape (N, 2). Pairs of coordinates with of the min-cost path from bottom-right to top-left.\n",
    "dtw_cost, dtw_min_path = librosa.sequence.dtw(gt_mfcc, synt_mfcc, metric=log_spec_dB_dist)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "913b6548",
   "metadata": {},
   "source": [
    "Reduction of the DTW matrix can be done by either taking the mean of the entire cost matrix or averaging the DTW cost for the minimum cost path per frame. We will use the DTW cost along the min cost path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c85ab38",
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, sum up the costs over the path\n",
    "path_cost_matrix = dtw_cost[dtw_min_path[:, 0], dtw_min_path[:, 1]]\n",
    "path_cost = np.sum(path_cost_matrix)\n",
    "\n",
    "# Average over path length\n",
    "path_length = dtw_min_path.shape[0]\n",
    "reduced_dtw_cost = path_cost/path_length\n",
    "\n",
    "# Average over number of frames\n",
    "frames = synt_mels.shape[1]\n",
    "mcd = reduced_dtw_cost/frames\n",
    "\n",
    "print(f\"MCD_DTW is: {mcd}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24801d61",
   "metadata": {},
   "source": [
    "## MCD DTW Usecase\n",
    "\n",
    "MCD is a very useful metric to **compare the convergence of two models**. Therefore in this section, we will calculate the average MCD for audio files generated by two different models: FastPitch and RAD-TTS.\n",
    "\n",
    "But first, let's put the calculations we have just performed in functions for better readability and reusability.\n",
    "\n",
    "Here are the helper functions for getting the average cost of DTW along a path."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e4181d60",
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_path_cost(D, wp):\n",
    "    \"\"\"\n",
    "    Get the path cost from D(cost matrix), wp (warped path)\n",
    "    Returns the sum of the path cost \n",
    "    \"\"\"\n",
    "    path_cost = D[wp[:, 0], wp[:, 1]]\n",
    "    return np.sum(path_cost)\n",
    "\n",
    "def extract_frame_avg_path_cost(D, wp):\n",
    "    \"\"\"\n",
    "    Get the average path cost over the length of the given path\n",
    "    \"\"\"\n",
    "    path_cost = extract_path_cost(D, wp)\n",
    "    path_length = wp.shape[0]\n",
    "    frame_avg_path_cost = path_cost / float(path_length)\n",
    "    return frame_avg_path_cost"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08ca0f47",
   "metadata": {},
   "source": [
    "And a function for calculating MCD for a synthetic mel spectrogram:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99800a97",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cal_mcd(gt_mfcc, synt_mfcc, cost_function, dtw_type='path_cost'):\n",
    "    \"\"\"\n",
    "    Calculates MCD between a ground truth and synthetic mel.\n",
    "    \"\"\"\n",
    "    frames = synt_mfcc.shape[1]\n",
    "    path_cost = 0\n",
    "    \n",
    "    # dynamic time warping for MCD\n",
    "    dtw_cost, dtw_min_path = librosa.sequence.dtw(gt_mfcc, synt_mfcc, metric=cost_function)\n",
    "    if dtw_type == 'mean':\n",
    "        path_cost = np.mean(dtw_cost)\n",
    "    else:\n",
    "        path_cost = extract_frame_avg_path_cost(dtw_cost, dtw_min_path)\n",
    "    \n",
    "    mcd = path_cost / frames\n",
    "    \n",
    "    return mcd, frames"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1489ecf",
   "metadata": {},
   "source": [
    "This last function will calculate the MCDs for all the synthetic mels in a directory.\n",
    "\n",
    "We'll get a better comparison if we compute the MCDs of (much) more than one synthesized spectrogram per model! This function streamlines the process by letting us pass in (1) a directory with all the synthesized mel spectrograms from a TTS model, and (2) a directory with all the ground-truth audio.\n",
    "\n",
    "The function assumes that, after sorting the contents of each directory, wav *#i* in the ground truth directory will correspond to mel *#i* in the synthesized mel directory. In our case, the directories for ground truth and synthesized FastPitch mels look like this:\n",
    "\n",
    "```bash\n",
    "% ls gt/\n",
    "sample_0.wav  sample_1.wav  sample_2.wav  sample_3.wav  sample_4.wav\n",
    "% ls fastpitch/mels/\n",
    "mels_0.npy  mels_1.npy  mels_2.npy  mels_3.npy  mels_4.npy\n",
    "```\n",
    "\n",
    "We only have ten files each to compare as a toy example, for a full evaluation you will likely want more!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07bb51df",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cal_mcd_dir(synt_dir, gt_dir):\n",
    "    \"\"\"\n",
    "    Calculate MCDs for pairs of synthetic and ground truth audio files.\n",
    "    \n",
    "    Args:\n",
    "        synt_dir: Path to the directory that contains all the synthetic mels.\n",
    "        gt_dir: Path to the directory that contains all the ground truth wavs.\n",
    "            These should correspond 1:1 with the mels in synt_dir.\n",
    "            \n",
    "    Returns:\n",
    "        List of MCDs (where length is the number of files in each directory)\n",
    "    \"\"\"\n",
    "    mcds = []\n",
    "    \n",
    "    synt_filenames = os.listdir(synt_dir)\n",
    "    synt_filepaths = [os.path.join(synt_dir, filename) for filename in synt_filenames]\n",
    "    synt_filepaths.sort()\n",
    "    gt_filenames = os.listdir(gt_dir)\n",
    "    gt_filepaths = [os.path.join(gt_dir, filename) for filename in gt_filenames]\n",
    "    gt_filepaths.sort()\n",
    "\n",
    "    for synt_melname, gt_audio in zip(synt_filepaths, gt_filepaths):\n",
    "        synt_mels = np.load(synt_melname)\n",
    "        synt_mfcc = mel2mfcc(synt_mels)\n",
    "        \n",
    "        gt_mels = wav2mel(gt_audio)\n",
    "        gt_mfcc = mel2mfcc(gt_mels)\n",
    "        \n",
    "        mcd, _ = cal_mcd(gt_mfcc, synt_mfcc, log_spec_dB_dist)\n",
    "        mcds.append(mcd)\n",
    "    return mcds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc295283",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "### Calculate MCD DTW on Synthesized Files from Each Model\n",
    "\n",
    "We can now calculate the MCD DTW for the synthesized FastPitch mels (compared to the ground truth) as well as for the synthesized RAD-TTS mels (ditto), then compare them. This will take a few seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2bc8da46",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%%capture --no-display\n",
    "mels_dir_m1 = \"MCD_DTW/fastpitch/mels/\"\n",
    "mels_dir_m2 = \"MCD_DTW/radtts/mels/\"\n",
    "mels_dir_gt = \"MCD_DTW/gt/\"\n",
    "\n",
    "mcds_m1 = cal_mcd_dir(mels_dir_m1, mels_dir_gt)  # FastPitch\n",
    "mcds_m2 = cal_mcd_dir(mels_dir_m2, mels_dir_gt)  # RAD-TTS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bd1ef33",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Average MCD for Model 1 (FastPitch) is: {sum(mcds_m1)/len(mcds_m1):.2f}\")\n",
    "print(f\"Average MCD for Model 2 (RAD-TTS) is:   {sum(mcds_m2)/len(mcds_m2):.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9ef5363",
   "metadata": {},
   "source": [
    "We're measuring divergence from the ground truth for each of these, so **lower is better**!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71e998a9",
   "metadata": {},
   "source": [
    "### Plotting MCD"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f81c5b5",
   "metadata": {},
   "source": [
    "We can also plot the MCD DTW values for both models, with the model with a lower MCD DTW value being closer to ground truth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f8d71bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_axis = np.linspace(1, len(mcds_m1), len(mcds_m1)) ## Define an x axis for for plotting in matplotlib\n",
    "plt.plot(x_axis, mcds_m1, label=\"FastPitch\")\n",
    "plt.plot(x_axis, mcds_m2, label=\"RAD-TTS\")\n",
    "\n",
    "plt.title(\"MCD DTW value for each file\")\n",
    "plt.ylabel(\"MCD_DTW value\")\n",
    "\n",
    "plt.grid()\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd4eb0a6",
   "metadata": {},
   "source": [
    "From the graph above, we can see that in general the value of MCD is greater for RAD-TTS mel spectrograms than for the FastPitch ones. This is also reflected in the average MCD value for both models that we computed before.\n",
    "\n",
    "Therefore, we can conclude that FastPitch has better convergence than RAD-TTS. However, we cannot evaluate the quality of audio generated by these models using MCD alone! **MCD is a great tool for testing model convergence, but generated audio may have pronunciation and quality artifacts.** Therefore MCD evaluation should be followed by a MOS (Mean Opinion Score) and CMOS (Comparative Mean Opinion Score) evaluation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d66a52d5",
   "metadata": {},
   "source": [
    "## Additional NeMo Resources\n",
    "\n",
    "If you are unsure where to begin for training a TTS model, you may want to start with the [FastPitch and Mixer-TTS Training notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_MixerTTS_Training.ipynb) or the [NeMo TTS Primer notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/NeMo_TTS_Primer.ipynb). For fine-tuning, there is also the [FastPitch Fine-Tuning notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/FastPitch_Finetuning.ipynb).\n",
    "\n",
    "For some guidance on how to load a trained model and perform inference to generate mels or waveforms, check out how it's done in the [Inference notebook](https://github.com/NVIDIA/NeMo/blob/main/tutorials/tts/Inference_ModelSelect.ipynb). Important functions to know are include `from_pretrained()` (if loading from an NGC model) and `restore_from()` (if loading a `.nemo` file). See the [NeMo Primer notebook](https://github.com/NVIDIA/NeMo/blob/stable/tutorials/00_NeMo_Primer.ipynb) for more general information about model training, saving, and loading."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}