Update README.md
Browse files
README.md
CHANGED
|
@@ -7,7 +7,6 @@ tags:
|
|
| 7 |
- emotions
|
| 8 |
- multi-class-classification
|
| 9 |
- multi-label-classification
|
| 10 |
-
- optimum
|
| 11 |
datasets:
|
| 12 |
- go_emotions
|
| 13 |
models:
|
|
@@ -17,3 +16,78 @@ inference: false
|
|
| 17 |
widget:
|
| 18 |
- text: ONNX is so much faster, its very handy!
|
| 19 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 7 |
- emotions
|
| 8 |
- multi-class-classification
|
| 9 |
- multi-label-classification
|
|
|
|
| 10 |
datasets:
|
| 11 |
- go_emotions
|
| 12 |
models:
|
|
|
|
| 16 |
widget:
|
| 17 |
- text: ONNX is so much faster, its very handy!
|
| 18 |
---
|
| 19 |
+
|
| 20 |
+
### Overview
|
| 21 |
+
|
| 22 |
+
This is a multi-label, multi-class linear classifer for emotions that works with [BGE-small-en-v1.5 embeddings](https://huggingface.co/BAAI/bge-small-en-v1.5), having been trained on the [go_emotions](https://huggingface.co/datasets/go_emotions) dataset.
|
| 23 |
+
|
| 24 |
+
### Labels
|
| 25 |
+
|
| 26 |
+
The 28 labels from the [go_emotions](https://huggingface.co/datasets/go_emotions) dataset are:
|
| 27 |
+
```
|
| 28 |
+
['admiration', 'amusement', 'anger', 'annoyance', 'approval', 'caring', 'confusion', 'curiosity', 'desire', 'disappointment', 'disapproval', 'disgust', 'embarrassment', 'excitement', 'fear', 'gratitude', 'grief', 'joy', 'love', 'nervousness', 'optimism', 'pride', 'realization', 'relief', 'remorse', 'sadness', 'surprise', 'neutral']
|
| 29 |
+
```
|
| 30 |
+
|
| 31 |
+
### Metrics (exact match of labels per item)
|
| 32 |
+
|
| 33 |
+
This is a multi-label, multi-class dataset, so each label is effectively a separate binary classification. Evaluating across all labels per item in the go_emotions test split the metrics are shown below.
|
| 34 |
+
|
| 35 |
+
Using a fixed threshold of 0.5 to convert the scores to binary predictions for each label, the metrics (evaluated on the go_emotions test split) are:
|
| 36 |
+
|
| 37 |
+
- Precision: 0.602
|
| 38 |
+
- Recall: 0.250
|
| 39 |
+
- F1: 0.303
|
| 40 |
+
|
| 41 |
+
Simple mean of labels: {'precision': 0.445, 'recall': 0.476, 'f1': 0.449}
|
| 42 |
+
Weighted average (using support): {'precision': 0.472, 'recall': 0.582, 'f1': 0.514}
|
| 43 |
+
|
| 44 |
+
Optimising the threshold per label to optimise the F1 metric, the metrics (evaluated on the go_emotions test split) are:
|
| 45 |
+
|
| 46 |
+
- Precision: 0.445
|
| 47 |
+
- Recall: 0.476
|
| 48 |
+
- F1: 0.449
|
| 49 |
+
|
| 50 |
+
Weighted by the relative support of each label in the dataset, this is:
|
| 51 |
+
|
| 52 |
+
- Precision: 0.472
|
| 53 |
+
- Recall: 0.582
|
| 54 |
+
- F1: 0.514
|
| 55 |
+
|
| 56 |
+
### Metrics (per-label)
|
| 57 |
+
|
| 58 |
+
This is a multi-label, multi-class dataset, so each label is effectively a separate binary classification and metrics are better measured per label.
|
| 59 |
+
|
| 60 |
+
Using a fixed threshold of 0.5 to convert the scores to binary predictions for each label, the metrics (evaluated on the go_emotions test split) are:
|
| 61 |
+
|
| 62 |
+
| | f1 | precision | recall | support | threshold |
|
| 63 |
+
| -------------- | ----- | --------- | ------ | ------- | --------- |
|
| 64 |
+
| admiration | 0.497 | 0.731 | 0.377 | 504 | 0.5 |
|
| 65 |
+
| amusement | 0.484 | 0.793 | 0.348 | 264 | 0.5 |
|
| 66 |
+
| anger | 0.162 | 0.528 | 0.096 | 198 | 0.5 |
|
| 67 |
+
| annoyance | 0.042 | 0.636 | 0.022 | 320 | 0.5 |
|
| 68 |
+
| approval | 0.106 | 0.769 | 0.057 | 351 | 0.5 |
|
| 69 |
+
| caring | 0.182 | 0.500 | 0.111 | 135 | 0.5 |
|
| 70 |
+
| confusion | 0.170 | 0.652 | 0.098 | 153 | 0.5 |
|
| 71 |
+
| curiosity | 0.284 | 0.529 | 0.194 | 284 | 0.5 |
|
| 72 |
+
| desire | 0.236 | 0.481 | 0.157 | 83 | 0.5 |
|
| 73 |
+
| disappointment | 0.039 | 0.750 | 0.020 | 151 | 0.5 |
|
| 74 |
+
| disapproval | 0.140 | 0.636 | 0.079 | 267 | 0.5 |
|
| 75 |
+
| disgust | 0.273 | 0.677 | 0.171 | 123 | 0.5 |
|
| 76 |
+
| embarrassment | 0.314 | 0.571 | 0.216 | 37 | 0.5 |
|
| 77 |
+
| excitement | 0.130 | 0.400 | 0.078 | 103 | 0.5 |
|
| 78 |
+
| fear | 0.527 | 0.667 | 0.436 | 78 | 0.5 |
|
| 79 |
+
| gratitude | 0.792 | 0.908 | 0.702 | 352 | 0.5 |
|
| 80 |
+
| grief | 0.385 | 0.250 | 0.833 | 6 | 0.5 |
|
| 81 |
+
| joy | 0.276 | 0.771 | 0.168 | 161 | 0.5 |
|
| 82 |
+
| love | 0.606 | 0.800 | 0.487 | 238 | 0.5 |
|
| 83 |
+
| nervousness | 0.269 | 0.241 | 0.304 | 23 | 0.5 |
|
| 84 |
+
| optimism | 0.305 | 0.720 | 0.194 | 186 | 0.5 |
|
| 85 |
+
| pride | 0.375 | 0.375 | 0.375 | 16 | 0.5 |
|
| 86 |
+
| realization | 0.013 | 0.250 | 0.007 | 145 | 0.5 |
|
| 87 |
+
| relief | 0.353 | 0.500 | 0.273 | 11 | 0.5 |
|
| 88 |
+
| remorse | 0.469 | 0.548 | 0.411 | 56 | 0.5 |
|
| 89 |
+
| sadness | 0.365 | 0.731 | 0.244 | 156 | 0.5 |
|
| 90 |
+
| surprise | 0.142 | 0.786 | 0.078 | 141 | 0.5 |
|
| 91 |
+
| neutral | 0.547 | 0.644 | 0.475 | 1787 | 0.5 |
|
| 92 |
+
|
| 93 |
+
|