Update README and config files - README.md

README.md

@@ -1,199 +1,286 @@
 ---
+language:
+- en
 library_name: transformers
-tags: []
+license: cc-by-4.0
+tags:
+- kl3m
+- legal
+- financial
+- mlm
+- roberta
+- embedding
+- uncased
+pipeline_tag: fill-mask
+tags_extended:
+- feature-extraction
+widget:
+ - text: "<|cls|> this credit agreement is made and entered into as of january 1, 2025, by and between acme corporation, a delaware <|mask|>, and first national bank, as lender. <|sep|>"
+ - example: "<|cls|> pursuant to section 13(a) of the <|mask|> exchange act of 1934, the undersigned hereby certifies that the quarterly report fully complies with the requirements. <|sep|>"
+ - temperature: 0.7
+ - do_sample: true
+date: '2025-01-25T00:00:00.000Z'
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
+# kl3m-doc-small-uncased-001
 
+`kl3m-doc-small-uncased-001` is a domain-specific masked language model (MLM) based on the RoBERTa architecture, specifically designed for legal and financial document analysis. With approximately 236M parameters, it provides a balanced model for specialized NLP tasks in both fill-mask prediction and feature extraction for document embeddings. This uncased variant is particularly useful for case-insensitive applications, maintaining strong performance while disregarding capitalization differences.
 
 ## Model Details
 
-### Model Description
-
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
-
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-
-### Direct Use
-
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
-
-### Recommendations
-
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-
-## How to Get Started with the Model
-
-Use the code below to get started with the model.
-
-[More Information Needed]
-
-## Training Details
-
-### Training Data
-
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-
-[More Information Needed]
-
-## Evaluation
-
-<!-- This section describes the evaluation protocols and provides the results. -->
-
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
-
-
-
-## Model Examination [optional]
-
-<!-- Relevant interpretability work for the model goes here -->
-
-[More Information Needed]
-
-## Environmental Impact
-
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+- **Architecture**: RoBERTa
+- **Size**: 236M parameters
+- **Hidden Size**: 1024
+- **Layers**: 8
+- **Attention Heads**: 8
+- **Intermediate Size**: 4096
+- **Max Position Embeddings**: 512
+- **Max Sequence Length**: 509
+- **Tokenizer**: [alea-institute/kl3m-004-128k-uncased](https://huggingface.co/alea-institute/kl3m-004-128k-uncased)
+- **Vector Dimension**: 1024 (hidden_size)
+- **Pooling Strategy**: CLS token or mean pooling
 
-## Technical Specifications [optional]
+## Use Cases
 
-### Model Architecture and Objective
+This model is particularly useful for:
 
-[More Information Needed]
+- Document classification in legal and financial domains
+- Entity recognition for specialized terminology
+- Understanding legal citations and references
+- Filling in missing terms in legal documents
+- Feature extraction for downstream legal analysis tasks
+- Document similarity and retrieval tasks where capitalization is not significant
+- Applications requiring a balance between performance and model size
+
+The uncased nature of this model makes it efficient for scenarios where case distinctions are not important to the task, while its small size offers a good balance between performance and computational requirements.
+
+## Standard Test Examples
+
+Using our standardized test examples for comparing embedding models:
+
+### Fill-Mask Results
+
+1. **Contract Clause Heading**:  
+   `"<|cls|> 8. representations and<|mask|>. each party hereby represents and warrants to the other party as of the date hereof as follows: <|sep|>"`
+
+   Top 5 predictions:
+   1. warranties (0.940)
+   2. warranty (0.016)
+   3. warranties (0.015)
+   4. covenants (0.005)
+   5. warrants (0.004)
+
+2. **Defined Term Example**:  
+   `"<|cls|> \"effective<|mask|>\" means the date on which all conditions precedent set forth in article v are satisfied or waived by the administrative agent. <|sep|>"`
+
+   Top 5 predictions:
+   1. date (0.989)
+   2. time (0.005)
+   3. deadline (0.003)
+   4. day (0.001)
+   5. dates (0.0005)
+
+3. **Regulation Example**:  
+   `"<|cls|> all transactions shall comply with the requirements set forth in the truth in<|mask|> act and its implementing regulation z. <|sep|>"`
+
+   Top 5 predictions:
+   1. lending (0.609)
+   2. information (0.075)
+   3. practices (0.049)
+   4. claims (0.020)
+   5. lending (0.019)
+
+### Document Similarity Results
+
+Using the standardized document examples for embeddings:
+
+| Document Pair | Cosine Similarity (CLS token) | Cosine Similarity (Mean pooling) |
+|---------------|-------------------------------|----------------------------------|
+| Court Complaint vs. Consumer Terms | 0.429 | 0.440 |
+| Court Complaint vs. Credit Agreement | 0.435 | 0.572 |
+| Consumer Terms vs. Credit Agreement | 0.562 | 0.498 |
+
+The small-sized model shows moderate document similarity performance, with better differentiation between document types than some larger models in the family. The model demonstrates balanced performance using both CLS token and mean pooling strategies.
+
+## Usage
+
+### Masked Language Modeling
+
+You can use this model for masked language modeling with the simple pipeline approach:
+
+```python
+from transformers import pipeline
 
-### Compute Infrastructure
+# Load the fill-mask pipeline with the model
+fill_mask = pipeline('fill-mask', model="alea-institute/kl3m-doc-small-uncased-001")
 
-[More Information Needed]
+# Example: Contract clause heading
+# Note the mask token placement - directly adjacent to "and" without space
+text = "<|cls|> 8. representations and<|mask|>. each party hereby represents and warrants to the other party as of the date hereof as follows: <|sep|>"
+results = fill_mask(text)
+
+# Display predictions
+print("Top predictions:")
+for result in results:
+    print(f"- {result['token_str']} (score: {result['score']:.3f})")
+
+# Output:
+# Top predictions:
+# - warranties (score: 0.940)
+# - warranty (score: 0.016)
+# - warranties (score: 0.015)
+# - covenants (score: 0.005)
+# - warrants (score: 0.004)
+```
 
-#### Hardware
+### Feature Extraction for Embeddings
 
-[More Information Needed]
+```python
+from transformers import pipeline
+import numpy as np
+from sklearn.metrics.pairwise import cosine_similarity
 
-#### Software
+# Load the feature-extraction pipeline
+extractor = pipeline('feature-extraction', model="alea-institute/kl3m-doc-small-uncased-001", return_tensors=True)
 
-[More Information Needed]
+# Example legal documents (truncated for brevity)
+texts = [
+    # Court Complaint
+    "<|cls|> in the united states district court for the eastern district of pennsylvania\n\njohn doe,\nplaintiff,\n\nvs.\n\nacme corporation,\ndefendant. <|sep|>",
+    
+    # Consumer Terms
+    "<|cls|> terms and conditions\n\nlast updated: april 10, 2025\n\nthese terms and conditions govern your access to and use of the service. <|sep|>",
+    
+    # Credit Agreement
+    "<|cls|> credit agreement\n\ndated as of april 10, 2025\n\namong\n\nacme borrower inc.,\nas the borrower,\n\nand bank of finance,\nas administrative agent. <|sep|>"
+]
 
-## Citation [optional]
+# Generate embeddings for each document
+embeddings = []
+for text in texts:
+    # Get features for the text
+    features = extractor(text)
+    
+    # Extract the CLS token embedding (first token)
+    # Convert to numpy if needed
+    features_array = features[0].numpy() if hasattr(features[0], 'numpy') else features[0]
+    cls_embedding = features_array[0]  # Shape: [hidden_size]
+    
+    embeddings.append(cls_embedding)
+
+# Calculate cosine similarity between documents
+similarity_matrix = cosine_similarity(embeddings)
+print("\nDocument similarity matrix:")
+print(similarity_matrix)
+```
+
+For more advanced usage with mean pooling:
+
+```python
+# Mean pooling - taking average of all token embeddings
+def mean_pooling(features):
+    # Convert to numpy if needed
+    features_array = features[0].numpy() if hasattr(features[0], 'numpy') else features[0]
+    return np.mean(features_array, axis=0)
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+# Generate mean-pooled embeddings
+mean_embeddings = [mean_pooling(extractor(text)) for text in texts]
 
-**BibTeX:**
+# Calculate similarity with mean pooling
+mean_similarity = cosine_similarity(mean_embeddings)
+print("\nMean pooling similarity matrix:")
+print(mean_similarity)
+```
 
-[More Information Needed]
+## Training
 
-**APA:**
+The model was trained on a diverse corpus of legal and financial documents, ensuring high-quality performance in these domains. It leverages the KL3M tokenizer which provides 9-17% more efficient tokenization for domain-specific content than cl100k_base or the LLaMA/Mistral tokenizers.
+
+Training included both masked language modeling (MLM) objectives and attention to dense document representation for retrieval and classification tasks. The model was trained on lowercase text to improve efficiency while maintaining strong performance.
 
-[More Information Needed]
+## Intended Usage
 
-## Glossary [optional]
+This model is intended for both:
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+1. **Masked Language Modeling**: Filling in missing words/terms in legal and financial documents
+2. **Document Embedding**: Generating fixed-length vector representations for document similarity and classification
+
+It is particularly well-suited for applications requiring a good balance between model size and performance, and where case-sensitivity is not important.
+
+## Special Tokens
+
+This model includes the following special tokens:
+
+- CLS token: `<|cls|>` (ID: 5) - Used for the beginning of input text
+- MASK token: `<|mask|>` (ID: 6) - Used to mark tokens for prediction
+- SEP token: `<|sep|>` (ID: 4) - Used for the end of input text
+- PAD token: `<|pad|>` (ID: 2) - Used for padding sequences to a uniform length
+- BOS token: `<|start|>` (ID: 0) - Beginning of sequence
+- EOS token: `<|end|>` (ID: 1) - End of sequence
+- UNK token: `<|unk|>` (ID: 3) - Unknown token
+
+Important usage notes:
+
+When using the MASK token for predictions, be aware that this model uses a **space-prefixed BPE tokenizer**. The <|mask|> token should be placed IMMEDIATELY after the previous token with NO space, because most tokens in this tokenizer have an initial space encoded within them. For example: `"word<|mask|>"` rather than `"word <|mask|>"`.
+   
+This space-aware placement is crucial for getting accurate predictions.
+
+## Limitations
+
+While providing a good balance between size and performance, this model has some limitations:
+
+- Moderate parameter count (236M) compared to larger language models
+- Uncased nature means it cannot distinguish between different capitalizations
+- Primarily focused on English legal and financial texts
+- Best suited for domain-specific rather than general-purpose tasks
+- Maximum sequence length of 512 tokens may require chunking for lengthy documents
+- Requires domain expertise to interpret results effectively
+
+## References
+
+- [KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications](https://arxiv.org/abs/2503.17247)
+- [The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models](https://arxiv.org/abs/2504.07854)
+
+## Citation
+
+If you use this model in your research, please cite:
+
+```bibtex
+@misc{kl3m-doc-small-uncased-001,
+  author = {ALEA Institute},
+  title = {kl3m-doc-small-uncased-001: A Domain-Specific Uncased Language Model for Legal and Financial Text Analysis},
+  year = {2025},
+  publisher = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/alea-institute/kl3m-doc-small-uncased-001}}
+}
 
-[More Information Needed]
+@article{bommarito2025kl3m,
+  title={KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications},
+  author={Bommarito, Michael J and Katz, Daniel Martin and Bommarito, Jillian},
+  journal={arXiv preprint arXiv:2503.17247},
+  year={2025}
+}
 
-## More Information [optional]
+@misc{bommarito2025kl3mdata,
+  title={The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models},
+  author={Bommarito II, Michael J. and Bommarito, Jillian and Katz, Daniel Martin},
+  year={2025},
+  eprint={2504.07854},
+  archivePrefix={arXiv},
+  primaryClass={cs.CL}
+}
+```
 
-[More Information Needed]
+## License
 
-## Model Card Authors [optional]
+This model is licensed under [CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/).
 
-[More Information Needed]
+## Contact
 
-## Model Card Contact
+The KL3M model family is maintained by the [ALEA Institute](https://aleainstitute.ai). For technical support, collaboration opportunities, or general inquiries:
+ 
+- Email: hello@aleainstitute.ai
+- Website: https://aleainstitute.ai
+- GitHub: https://github.com/alea-institute/kl3m-model-research
 
-[More Information Needed]
+![logo](https://aleainstitute.ai/images/alea-logo-ascii-1x1.png)