|
|
--- |
|
|
library_name: model2vec |
|
|
license: mit |
|
|
model_name: tmpqsu1ee6a |
|
|
tags: |
|
|
- embeddings |
|
|
- static-embeddings |
|
|
datasets: |
|
|
- HuggingFaceFW/fineweb-edu-llama3-annotations |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- minishlab/potion-base-8M |
|
|
--- |
|
|
|
|
|
# potion-8m-edu-classifier Model Card |
|
|
|
|
|
This [Model2Vec](https://github.com/MinishLab/model2vec) model is a fine-tuned version of [potion-base-8m](https://huggingface.co/minishlab/potion-base-8M). |
|
|
It was trained to predict educational content, analogous to how the [fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) was used to filter educational content. |
|
|
|
|
|
It achieves the following performance on the evaluation split: |
|
|
|
|
|
``` |
|
|
precision recall f1-score support |
|
|
|
|
|
0 0.70 0.42 0.52 5694 |
|
|
1 0.75 0.86 0.80 26512 |
|
|
2 0.55 0.51 0.53 10322 |
|
|
3 0.54 0.45 0.49 3407 |
|
|
4 0.59 0.30 0.40 807 |
|
|
5 0.00 0.00 0.00 1 |
|
|
|
|
|
accuracy 0.69 46743 |
|
|
macro avg 0.52 0.42 0.46 46743 |
|
|
weighted avg 0.68 0.69 0.68 46743 |
|
|
``` |
|
|
|
|
|
When thresholded to a binary classifier, it achieves a macro-averaged F1-score of `0.79`. The original classifier achieves `0.81` on the same dataset, but this classifier is orders of magnitude faster on CPU. |
|
|
|
|
|
``` |
|
|
precision recall f1-score support |
|
|
|
|
|
not edu 0.96 0.98 0.97 42528 |
|
|
edu 0.70 0.54 0.61 4215 |
|
|
|
|
|
accuracy 0.94 46743 |
|
|
macro avg 0.83 0.76 0.79 46743 |
|
|
weighted avg 0.93 0.94 0.93 46743 |
|
|
``` |
|
|
|
|
|
## Installation |
|
|
|
|
|
Install model2vec with the inference extra using pip: |
|
|
``` |
|
|
pip install model2vec[inference] |
|
|
``` |
|
|
|
|
|
## Usage |
|
|
Load this model using the `from_pretrained` method: |
|
|
```python |
|
|
from model2vec.inference import StaticModelPipeline |
|
|
|
|
|
# Load a pretrained Model2Vec model |
|
|
model = StaticModelPipeline.from_pretrained("minishlab/potion-8m-edu-classifier") |
|
|
|
|
|
# Predict labels |
|
|
label = model.predict(["Example sentence"]) |
|
|
``` |
|
|
|
|
|
## Library Authors |
|
|
|
|
|
Model2Vec was developed by [Minish](https://github.com/MinishLab). |
|
|
|
|
|
## Citation |
|
|
|
|
|
Please cite the [Model2Vec repository](https://github.com/MinishLab/model2vec) if you use this model in your work. |
|
|
``` |
|
|
@software{minishlab2024model2vec, |
|
|
authors = {Stephan Tulkens, Thomas van Dongen}, |
|
|
title = {Model2Vec: Turn any Sentence Transformer into a Small Fast Model}, |
|
|
year = {2024}, |
|
|
url = {https://github.com/MinishLab/model2vec}, |
|
|
} |
|
|
``` |
