ThuraAung1601/speecht5_for_thai_with_ipa_tts_v1

SpeechT5-TTS with IPA for Thai

This model is a fine-tuned version of microsoft/speecht5_tts on the Edited Thai Common Voice dataset. It achieves the following results on the evaluation set:

• Loss: 0.4730

Usage

from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech
import numpy as np

tts_model = SpeechT5ForTextToSpeech.from_pretrained("ThuraAung1601/speecht5_for_thai_with_ipa_tts_v1")

ipa_processor = SpeechT5Processor.from_pretrained("ThuraAung1601/speecht5_for_thai_with_ipa_tts_v1")
ipa_tokenizer = ipa_processor.tokenizer
ipa_tokenizer.decode(ipa_tokenizer(text)["input_ids"])

loaded_np = np.load("speaker_embedding_304.npy")
speaker_embedding = torch.tensor(loaded_np)

text = "ที่ข้าพเจ้าได้เคยล่วงเกินท่านไว้"

from pythainlp.tokenize import word_tokenize
from pythainlp.transliterate import transliterate

def thai2ipa_dict(text):
    result = []
    for word in word_tokenize(text):
        result.append(transliterate(word).replace(" ", ""))
    return "".join(result)

ipa = thai2ipa_dict(text)

model_input = ipa_processor(text=ipa, return_tensors="pt")

spectrogram = tts_model.generate_speech(model_input["input_ids"], speaker_embedding)

from transformers import SpeechT5HifiGan
the_vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan")

with torch.no_grad():
    speech = the_vocoder(spectrogram)

import soundfile as sf
sf.write("output.wav", speech.numpy(), samplerate=16000)

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 2
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• training_steps: 6000
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
0.5418	0.5601	1000	0.4969
0.5335	1.1199	2000	0.4889
0.5177	1.6799	3000	0.4818
0.5188	2.2397	4000	0.4800
0.5067	2.7998	5000	0.4738
0.5072	3.3596	6000	0.4730

Framework versions

• Transformers 4.52.0.dev0
• Pytorch 2.6.0+cu124
• Datasets 3.5.0
• Tokenizers 0.21.1