KandirResearch
/

CiSiMi-v0.1

@@ -2,201 +2,194 @@
 library_name: transformers
 tags:
 - unsloth
-- trl
-- sft
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 library_name: transformers
 tags:
 - unsloth
+- text-to-audio
+- s2s
+license: cc-by-sa-4.0
+datasets:
+- KandirResearch/Speech2Speech
+language:
+- en
+base_model:
+- OuteAI/OuteTTS-0.3-500M
+pipeline_tag: text-to-audio
 ---
+# CiSiMi: A Text-to-Speech TTS Model
+[![Buy Me A Coffee](https://img.shields.io/badge/Ko--fi-Support%20My%20Work-FF5E5B?style=for-the-badge&logo=ko-fi&logoColor=white)](https://ko-fi.com/lyte)
+[![Dataset](https://img.shields.io/badge/Dataset-KandirResearch/Speech2Speech-blue)](https://huggingface.co/datasets/KandirResearch/Speech2Speech)
+[![Model](https://img.shields.io/badge/Model-KandirResearch/CiSiMi-green)](https://huggingface.co/KandirResearch/CiSiMi)
+[![Demo](https://img.shields.io/badge/Demo-KandirResearch/CiSiMi--At--Home-orange)](https://huggingface.co/spaces/KandirResearch/CiSiMi-At-Home)
+## Overview
+CiSiMi is an early prototype of a text-to-audio model that can process text inputs and respond with both text and audio. Built for resource-constrained environments, it's designed to run efficiently on CPU using llama.cpp, making advanced speech synthesis accessible even without powerful GPUs.
+*"Being GPU poor and slightly disappointed with the csm release and my inability to run it, having to wait for time it takes me to run an ASR+LLM+TTS combo, I decided to ask Mom and Mom gave me CiSiMi At Home!"*
+This project demonstrates the power of open-source tools to create accessible speech technology. While still in its early stages, it represents a step toward democratizing advanced text-to-audio capabilities.
+## Technical Details
+### Model Specifications
+- **Architecture**: Based on OuteTTS-0.3-500M
+- **Languages**: English
+- **Pipeline**: Text-to-audio
+- **Parameters**: 500M
+- **Training Dataset Size**: ~15k samples
+- **Future Goals**: Scale to 200k-500k dataset with multi-turn conversation using a 1B parameter model
+### Training Methodology
+1. **Dataset Preparation**:
+   - Started with [gruhit-patel/alpaca_speech_instruct](https://huggingface.co/datasets/gruhit-patel/alpaca_speech_instruct)
+   - Cleaned by removing code, mathematical expressions, and non-English content
+   - Filtered to keep only entries with input+output texts of 256 tokens or less
+2. **Audio Generation**:
+   - Converted text outputs to speech using [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M)
+   - Verified each audio generation using [OpenAI Whisper](https://github.com/openai/whisper)
+   - Published the resulting dataset as [KandirResearch/Speech2Speech](https://huggingface.co/datasets/KandirResearch/Speech2Speech)
+3. **Model Training**:
+   - Preprocessed dataset using modified OuteTTS methodology ([training details](https://github.com/edwko/OuteTTS/blob/8eb0fa369df6f3c062f7084ddc33d10bc28992be/examples/training/OuteTTS-0.3/train.md))
+   - Fine-tuned [OuteAI/OuteTTS-0.3-500M](https://huggingface.co/OuteAI/OuteTTS-0.3-500M) using Unsloth SFT
+   - Trained for 3 epochs as a proof of concept
+## Usage Guide
+### Installation
+```bash
+pip install outetts llama-cpp-python --upgrade
+pip install huggingface_hub sounddevice
+```
+### Implementation
+```python
+import torch
+import outetts
+import numpy as np
+from huggingface_hub import hf_hub_download
+from outetts.wav_tokenizer.audio_codec import AudioCodec
+from outetts.version.v2.prompt_processor import PromptProcessor
+from outetts.version.playback import ModelOutput
+# Download the model
+model_path = hf_hub_download(
+    repo_id="Lyte/CiSiMi",
+    filename="unsloth.Q8_0.gguf",
+)
+# Configure the model
+model_config = outetts.GGUFModelConfig_v2(
+    model_path=model_path,
+    tokenizer_path="Lyte/CiSiMi",
+)
+# Initialize components
+interface = outetts.InterfaceGGUF(model_version="0.3", cfg=model_config)
+audio_codec = AudioCodec()
+prompt_processor = PromptProcessor("Lyte/CiSiMi")
+device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
+gguf_model = interface.get_model()
+# Helper function to extract audio from tokens
+def get_audio(tokens):
+    outputs = prompt_processor.extract_audio_from_tokens(tokens)
+    if not outputs:
+        return None
+    audio_tensor = audio_codec.decode(torch.tensor([[outputs]], dtype=torch.int64).to(device))
+    return ModelOutput(audio_tensor, audio_codec.sr)
+# Helper function to clean text output
+def extract_text_from_tts_output(tts_output):
+    text = ""
+    for line in tts_output.strip().split('\n'):
+        if '<|audio_end|>' in line or '<|im_end|>' in line:
+            continue
+        if '<|' in line:
+            word = line.split('<|')[0].strip()
+            if word:
+                text += word + " "
+        else:
+            text += line.strip() + " "
+    return text.strip()
+# Generate response function
+def generate_response(instruction):
+    prompt = f"<|im_start|>\nInstructions:\n{instruction}\n<|im_end|>\nAnswer:\n"
+    gen_cfg = outetts.GenerationConfig(
+        text=prompt,
+        temperature=0.6,
+        repetition_penalty=1.1,
+        max_length=4096,
+        speaker=None
+    )
+    input_ids = prompt_processor.tokenizer.encode(prompt)
+    tokens = gguf_model.generate(input_ids, gen_cfg)
+    output_text = prompt_processor.tokenizer.decode(tokens, skip_special_tokens=False)
+    if "<|audio_end|>" in output_text:
+        first_part, _, _ = output_text.partition("<|audio_end|>")
+        if "<|audio_end|>\n<|im_end|>\n" not in first_part:
+            first_part += "<|audio_end|>\n<|im_end|>\n"
+        extracted_text = extract_text_from_tts_output(first_part)
+        audio_start_pos = first_part.find("<|audio_start|>\n") + len("<|audio_start|>\n")
+        audio_end_pos = first_part.find("<|audio_end|>\n<|im_end|>\n") + len("<|audio_end|>\n<|im_end|>\n")
+        if audio_start_pos >= len("<|audio_start|>\n") and audio_end_pos > audio_start_pos:
+            audio_tokens_text = first_part[audio_start_pos:audio_end_pos]
+            audio_tokens = prompt_processor.tokenizer.encode(audio_tokens_text)
+            audio_output = get_audio(audio_tokens)
+            if audio_output is not None and hasattr(audio_output, 'audio') and audio_output.audio is not None:
+                audio_numpy = audio_output.audio.cpu().numpy()
+                if audio_numpy.ndim > 1:
+                    audio_numpy = audio_numpy.squeeze()
+                return extracted_text, (audio_output.sr, audio_numpy)
+    return output_text, None
+# Example usage
+question = "What is the meaning of life?"
+response_text, response_audio = generate_response(question)
+print(response_text)
+# Play audio if available
+if response_audio is not None:
+    if "ipykernel" in sys.modules:
+        from IPython.display import display, Audio
+        display(Audio(response_audio[1], rate=response_audio[0], autoplay=True))
+    else:
+        import sounddevice as sd
+        sd.play(response_audio[1], samplerate=response_audio[0])
+        sd.wait()
+```
+## Limitations & Future Work
+This early prototype has several areas for improvement:
+- Limited training data (~15k samples)
+- Basic prompt/chat template structure
+- Opportunity to optimize training hyperparameters
+- Potential for multi-turn conversation capabilities
+**Potential Limitation**: This type of model quickly fills up context window, making smaller models generally more practical for implementation.
+## Acknowledgments & Citations
+This model builds on the following open-source projects:
+1. [OuteAI/OuteTTS-0.3-500M](https://huggingface.co/OuteAI/OuteTTS-0.3-500M) - Base model
+2. [gruhit-patel/alpaca_speech_instruct](https://huggingface.co/datasets/gruhit-patel/alpaca_speech_instruct) - Initial dataset
+3. [hexgrad/Kokoro-82M](https://huggingface.co/hexgrad/Kokoro-82M) - TTS generation
+4. [OpenAI Whisper](https://github.com/openai/whisper) - Speech verification
+5. [Unsloth](https://github.com/unslothai/unsloth) - Training optimization