distilbert-classn-LAlg-multihead-context-width-2

This model is a fine-tuned version of dslim/distilbert-NER on the None dataset. It achieves the following results on the evaluation set:

• Loss: 0.8459
• Accuracy: 0.7698
• F1: 0.7717
• Precision: 0.8049
• Recall: 0.7698

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_steps: 500
• num_epochs: 25
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1	Precision	Recall
2.4794	1.3514	50	2.4632	0.0952	0.0337	0.0269	0.0952
2.4695	2.7027	100	2.4386	0.0794	0.0395	0.0332	0.0794
2.4253	4.0541	150	2.4152	0.0556	0.0351	0.0299	0.0556
2.3921	5.4054	200	2.3838	0.0794	0.0818	0.1342	0.0794
2.3137	6.7568	250	2.3342	0.1349	0.1256	0.1343	0.1349
2.2091	8.1081	300	2.2389	0.2460	0.2390	0.2692	0.2460
2.0017	9.4595	350	2.0460	0.3889	0.3931	0.4582	0.3889
1.6241	10.8108	400	1.7016	0.5476	0.5285	0.5501	0.5476
1.1356	12.1622	450	1.3357	0.6825	0.6735	0.7118	0.6825
0.7122	13.5135	500	1.0433	0.7540	0.7501	0.7818	0.7540
0.4063	14.8649	550	0.9308	0.7540	0.7541	0.7860	0.7540
0.2025	16.2162	600	0.8688	0.7857	0.7838	0.8068	0.7857
0.1226	17.5676	650	0.8284	0.7698	0.7704	0.8092	0.7698
0.0677	18.9189	700	0.8517	0.7778	0.7784	0.8180	0.7778
0.0386	20.2703	750	0.8447	0.7857	0.7859	0.8250	0.7857
0.0299	21.6216	800	0.8531	0.7698	0.7693	0.7984	0.7698
0.0236	22.9730	850	0.8446	0.7698	0.7717	0.8049	0.7698
0.0182	24.3243	900	0.8459	0.7698	0.7717	0.8049	0.7698

Framework versions

• Transformers 4.48.3
• Pytorch 2.5.1+cu124
• Datasets 3.3.1
• Tokenizers 0.21.0