End of training

Browse files

Files changed (2) hide show

README.md +72 -72
logs/attn_loss_fn=mse, attn_weight=2.0/events.out.tfevents.1723724446.93d6cbb3ad53 +3 -0

README.md CHANGED Viewed

@@ -16,13 +16,13 @@ This student model is distilled from the teacher model [gpt2](https://huggingfac
 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 It achieves the following results on the evaluation set:
-- eval_enwikippl: 218.5393
-- eval_frwikippl: 1177.8887
-- eval_zhwikippl: 654.9657
-- eval_loss: 1.2101
-- eval_runtime: 84.5457
-- eval_samples_per_second: 59.14
-- eval_steps_per_second: 7.392
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
@@ -45,7 +45,7 @@ More information needed
 ### Training hyperparameters
 The following hyperparameters were used during training:
-- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=0, loss_fn=None, layer_mapper=None, projector=None))
 - train_embeddings: True
 - learning_rate: 4e-05
 - train_batch_size: 8
@@ -56,75 +56,75 @@ The following hyperparameters were used during training:
 - num_epochs: 1.0
 ### Resource Usage
-Peak GPU Memory: 7.9371 GB
 ### Eval-Phase Metrics
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 30.2086 | 57.2728 |  |  |  |  | 18.1784 |
-| 0 | 0 | 57642.0977 | 56387.75 | 5.8468 | 84.9344 | 58.869 | 7.359 | 54838.8008 |
-| 1000 | 0.0162 | 715.0994 | 4617.4824 | 1.8640 | 84.6424 | 59.072 | 7.384 | 16026.5967 |
-| 2000 | 0.0323 | 509.6366 | 2986.9795 | 1.6848 | 84.231 | 59.361 | 7.42 | 1765.5764 |
-| 3000 | 0.0485 | 427.6056 | 2689.4856 | 1.5835 | 84.4251 | 59.224 | 7.403 | 968.8549 |
-| 4000 | 0.0646 | 370.2697 | 2295.2737 | 1.4992 | 84.3656 | 59.266 | 7.408 | 1006.3041 |
-| 5000 | 0.0808 | 317.6974 | 1989.4651 | 1.4212 | 84.6665 | 59.055 | 7.382 | 1057.5557 |
-| 6000 | 0.0970 | 285.2802 | 1632.1284 | 1.3569 | 84.6806 | 59.045 | 7.381 | 874.1838 |
-| 7000 | 0.1131 | 256.3500 | 1458.8361 | 1.3018 | 84.4525 | 59.205 | 7.401 | 849.1024 |
-| 8000 | 0.1293 | 237.8977 | 1317.0640 | 1.2529 | 84.5727 | 59.121 | 7.39 | 639.9225 |
-| 9000 | 0.1455 | 218.5393 | 1177.8887 | 1.2101 | 84.5457 | 59.14 | 7.392 | 654.9657 |
-| 10000 | 0.1616 | 202.2269 | 1126.2368 | 1.1660 | 84.6055 | 59.098 | 7.387 | 699.7226 |
-| 11000 | 0.1778 | 184.4057 | 1090.2955 | 1.1237 | 84.6191 | 59.088 | 7.386 | 1601.5863 |
-| 12000 | 0.1939 | 170.1912 | 972.0627 | 1.0802 | 84.6016 | 59.101 | 7.388 | 663.8590 |
-| 13000 | 0.2101 | 160.8243 | 873.9529 | 1.0462 | 84.6676 | 59.054 | 7.382 | 817.5033 |
-| 14000 | 0.2263 | 153.4316 | 853.4323 | 1.0216 | 84.5669 | 59.125 | 7.391 | 789.6068 |
-| 15000 | 0.2424 | 144.5039 | 750.0720 | 0.9936 | 84.4757 | 59.189 | 7.399 | 497.9154 |
-| 16000 | 0.2586 | 139.1196 | 713.8004 | 0.9741 | 84.6689 | 59.054 | 7.382 | 499.2470 |
-| 17000 | 0.2747 | 136.0640 | 718.4965 | 0.9613 | 84.4587 | 59.201 | 7.4 | 691.9172 |
-| 18000 | 0.2909 | 132.3746 | 717.3322 | 0.9476 | 84.7695 | 58.983 | 7.373 | 511.1194 |
-| 19000 | 0.3071 | 131.7389 | 662.5851 | 0.9386 | 84.5077 | 59.166 | 7.396 | 483.8233 |
-| 20000 | 0.3232 | 128.0474 | 670.8112 | 0.9298 | 84.837 | 58.937 | 7.367 | 464.8238 |
-| 21000 | 0.3394 | 125.2350 | 678.9477 | 0.9209 | 84.6313 | 59.08 | 7.385 | 329.0456 |
-| 22000 | 0.3556 | 125.6929 | 674.8433 | 0.9162 | 84.9393 | 58.866 | 7.358 | 347.1923 |
-| 23000 | 0.3717 | 124.5367 | 630.1886 | 0.9094 | 85.9763 | 58.156 | 7.269 | 457.4956 |
-| 24000 | 0.3879 | 123.1902 | 665.5817 | 0.9071 | 85.0672 | 58.777 | 7.347 | 311.9309 |
-| 25000 | 0.4040 | 122.3417 | 641.2142 | 0.9017 | 85.0283 | 58.804 | 7.35 | 365.3137 |
-| 26000 | 0.4202 | 120.2694 | 624.0430 | 0.8953 | 85.056 | 58.785 | 7.348 | 319.8610 |
-| 27000 | 0.4364 | 120.1667 | 628.5027 | 0.8907 | 85.1161 | 58.743 | 7.343 | 319.6902 |
-| 28000 | 0.4525 | 118.2781 | 570.9954 | 0.8846 | 85.144 | 58.724 | 7.341 | 272.5736 |
-| 29000 | 0.4687 | 118.5724 | 595.4168 | 0.8842 | 85.137 | 58.729 | 7.341 | 268.9220 |
-| 30000 | 0.4848 | 119.3669 | 594.5359 | 0.8814 | 84.9629 | 58.849 | 7.356 | 331.3828 |
-| 31000 | 0.5010 | 117.9205 | 597.4355 | 0.8759 | 85.1582 | 58.714 | 7.339 | 352.8950 |
-| 32000 | 0.5172 | 119.1076 | 616.9991 | 0.8873 | 85.3053 | 58.613 | 7.327 | 333.6027 |
-| 33000 | 0.5333 | 117.0265 | 598.3629 | 0.8810 | 85.4345 | 58.524 | 7.316 | 329.0897 |
-| 34000 | 0.5495 | 116.5639 | 591.6924 | 0.8745 | 85.4331 | 58.525 | 7.316 | 284.6257 |
-| 35000 | 0.5657 | 116.2566 | 583.1194 | 0.8736 | 85.3062 | 58.612 | 7.327 | 312.5146 |
-| 36000 | 0.5818 | 114.5094 | 569.9497 | 0.8699 | 85.3486 | 58.583 | 7.323 | 316.7582 |
-| 37000 | 0.5980 | 115.3036 | 556.4886 | 0.8670 | 85.331 | 58.595 | 7.324 | 276.4224 |
-| 38000 | 0.6141 | 114.4916 | 616.9991 | 0.8652 | 85.326 | 58.599 | 7.325 | 257.0539 |
-| 39000 | 0.6303 | 113.8003 | 562.5639 | 0.8617 | 85.2521 | 58.65 | 7.331 | 249.4454 |
-| 40000 | 0.6465 | 113.2449 | 589.2362 | 0.8608 | 85.4264 | 58.53 | 7.316 | 303.6291 |
-| 41000 | 0.6626 | 113.5267 | 595.7949 | 0.8585 | 85.32 | 58.603 | 7.325 | 331.6926 |
-| 42000 | 0.6788 | 112.8149 | 594.4523 | 0.8579 | 84.9462 | 58.861 | 7.358 | 352.5180 |
-| 43000 | 0.6949 | 114.1189 | 599.4184 | 0.8609 | 84.9848 | 58.834 | 7.354 | 1005.4978 |
-| 44000 | 0.7111 | 113.6678 | 552.8904 | 0.8595 | 85.0096 | 58.817 | 7.352 | 1579.9192 |
-| 45000 | 0.7273 | 111.6644 | 655.0142 | 0.8554 | 85.0106 | 58.816 | 7.352 | 587.5825 |
-| 46000 | 0.7434 | 113.8180 | 577.0257 | 0.8590 | 85.0044 | 58.821 | 7.353 | 429.3204 |
-| 47000 | 0.7596 | 112.4737 | 534.5300 | 0.8557 | 84.9665 | 58.847 | 7.356 | 295.4299 |
-| 48000 | 0.7758 | 112.1945 | 534.9068 | 0.8529 | 85.0374 | 58.798 | 7.35 | 355.7813 |
-| 49000 | 0.7919 | 112.0117 | 588.8623 | 0.8545 | 85.2876 | 58.625 | 7.328 | 353.1778 |
-| 50000 | 0.8081 | 110.6717 | 554.0220 | 0.8475 | 84.9061 | 58.889 | 7.361 | 320.5880 |
-| 51000 | 0.8242 | 110.2171 | 533.4382 | 0.8444 | 84.9625 | 58.85 | 7.356 | 293.8561 |
-| 52000 | 0.8404 | 109.8668 | 550.0522 | 0.8477 | 84.9033 | 58.891 | 7.361 | 292.7595 |
-| 53000 | 0.8566 | 110.8953 | 522.9734 | 0.8430 | 84.9959 | 58.826 | 7.353 | 330.4548 |
-| 54000 | 0.8727 | 113.6325 | 566.0253 | 0.8537 | 85.3083 | 58.611 | 7.326 | 435.0919 |
-| 55000 | 0.8889 | 112.4562 | 600.7300 | 0.8536 | 85.3234 | 58.601 | 7.325 | 440.3524 |
-| 56000 | 0.9051 | 112.5611 | 593.0288 | 0.8587 | 85.3061 | 58.612 | 7.327 | 713.9750 |
-| 57000 | 0.9212 | 113.5531 | 569.5080 | 0.8588 | 85.2454 | 58.654 | 7.332 | 455.4841 |
-| 58000 | 0.9374 | 110.8006 | 523.5637 | 0.8485 | 85.2585 | 58.645 | 7.331 | 420.0204 |
-| 59000 | 0.9535 | 110.0631 | 563.5960 | 0.8472 | 85.1774 | 58.701 | 7.338 | 391.7394 |
-| 60000 | 0.9697 | 109.0000 | 534.9446 | 0.8436 | 85.2097 | 58.679 | 7.335 | 323.9015 |
-| 61000 | 0.9859 | 112.5698 | 560.3074 | 0.8420 | 85.18 | 58.699 | 7.337 | 256.8824 |
-| 61875 | 1.0 | 109.4750 | 562.8416 | 0.8412 | 85.1958 | 58.688 | 7.336 | 313.6015 |
 ### Framework versions
 - Distily 0.2.0

 The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
 It achieves the following results on the evaluation set:
+- eval_enwikippl: 215.4055
+- eval_frwikippl: 1190.7479
+- eval_zhwikippl: 547.2146
+- eval_loss: 1.2012
+- eval_runtime: 86.3928
+- eval_samples_per_second: 57.875
+- eval_steps_per_second: 7.234
 <!-- This model card has been generated automatically according to the information the Trainer had access to. You
 should probably proofread and complete it, then remove this comment.
 ### Training hyperparameters
 The following hyperparameters were used during training:
+- distillation_objective: DistillationObjective(logits_loss_component=LossComponent(label=logits, weight=1, loss_fn=kl, layer_mapper=None, projector=None), hs_loss_component=LossComponent(label=hs, weight=0, loss_fn=None, layer_mapper=None, projector=None), attn_loss_component=LossComponent(label=attn, weight=2.0, loss_fn=mse, layer_mapper=None, projector=None))
 - train_embeddings: True
 - learning_rate: 4e-05
 - train_batch_size: 8
 - num_epochs: 1.0
 ### Resource Usage
+Peak GPU Memory: 8.2206 GB
 ### Eval-Phase Metrics
 | step | epoch | enwikippl | frwikippl | loss | runtime | samples_per_second | steps_per_second | zhwikippl |
 | --- | --- | --- | --- | --- | --- | --- | --- | --- |
 | **teacher eval** |  | 30.2086 | 57.2728 |  |  |  |  | 18.1784 |
+| 0 | 0 | 56314.7695 | 59887.2773 | 5.8256 | 86.2439 | 57.975 | 7.247 | 59033.8086 |
+| 1000 | 0.0162 | 707.4770 | 4242.8809 | 1.8516 | 86.1491 | 58.039 | 7.255 | 11038.7695 |
+| 2000 | 0.0323 | 507.7405 | 3239.7178 | 1.6796 | 86.186 | 58.014 | 7.252 | 1887.9902 |
+| 3000 | 0.0485 | 425.0894 | 2858.4150 | 1.5756 | 86.0159 | 58.129 | 7.266 | 841.8765 |
+| 4000 | 0.0646 | 361.4349 | 2351.2927 | 1.4943 | 86.0626 | 58.097 | 7.262 | 1237.3851 |
+| 5000 | 0.0808 | 320.2736 | 1811.6420 | 1.4160 | 86.3077 | 57.932 | 7.242 | 941.8109 |
+| 6000 | 0.0970 | 279.4263 | 1586.2935 | 1.3478 | 86.3392 | 57.911 | 7.239 | 744.3502 |
+| 7000 | 0.1131 | 252.5366 | 1452.6782 | 1.2903 | 86.3844 | 57.881 | 7.235 | 651.1284 |
+| 8000 | 0.1293 | 229.7639 | 1333.1338 | 1.2422 | 86.4019 | 57.869 | 7.234 | 586.1718 |
+| 9000 | 0.1455 | 215.4055 | 1190.7479 | 1.2012 | 86.3928 | 57.875 | 7.234 | 547.2146 |
+| 10000 | 0.1616 | 195.7073 | 1147.2347 | 1.1512 | 86.3689 | 57.891 | 7.236 | 673.5028 |
+| 11000 | 0.1778 | 181.4088 | 1060.8735 | 1.1073 | 86.4921 | 57.809 | 7.226 | 521.8091 |
+| 12000 | 0.1939 | 164.0534 | 896.9886 | 1.0636 | 86.3399 | 57.911 | 7.239 | 488.8237 |
+| 13000 | 0.2101 | 157.4142 | 890.0587 | 1.0357 | 86.4286 | 57.851 | 7.231 | 510.7101 |
+| 14000 | 0.2263 | 148.5198 | 793.2602 | 1.0069 | 86.4451 | 57.84 | 7.23 | 415.8904 |
+| 15000 | 0.2424 | 143.5310 | 728.5455 | 0.9844 | 86.5014 | 57.803 | 7.225 | 414.5595 |
+| 16000 | 0.2586 | 139.7042 | 766.6470 | 0.9726 | 86.5584 | 57.764 | 7.221 | 539.9557 |
+| 17000 | 0.2747 | 136.4025 | 723.4780 | 0.9594 | 86.3816 | 57.883 | 7.235 | 877.2245 |
+| 18000 | 0.2909 | 133.8320 | 733.1834 | 0.9461 | 86.4657 | 57.826 | 7.228 | 582.4266 |
+| 19000 | 0.3071 | 130.4055 | 720.7795 | 0.9391 | 86.5854 | 57.746 | 7.218 | 564.7347 |
+| 20000 | 0.3232 | 128.2763 | 679.3307 | 0.9259 | 86.469 | 57.824 | 7.228 | 364.2420 |
+| 21000 | 0.3394 | 126.0545 | 666.4741 | 0.9208 | 86.3084 | 57.932 | 7.241 | 392.6297 |
+| 22000 | 0.3556 | 126.3289 | 618.9599 | 0.9146 | 86.2819 | 57.95 | 7.244 | 383.1512 |
+| 23000 | 0.3717 | 125.7710 | 652.6170 | 0.9106 | 86.3709 | 57.89 | 7.236 | 382.0272 |
+| 24000 | 0.3879 | 121.7352 | 649.1292 | 0.9010 | 86.4132 | 57.862 | 7.233 | 407.5338 |
+| 25000 | 0.4040 | 121.2164 | 677.1313 | 0.8985 | 86.5605 | 57.763 | 7.22 | 378.4727 |
+| 26000 | 0.4202 | 121.4331 | 604.5543 | 0.8920 | 86.6149 | 57.727 | 7.216 | 400.5201 |
+| 27000 | 0.4364 | 121.4896 | 636.5748 | 0.8898 | 86.977 | 57.486 | 7.186 | 344.3297 |
+| 28000 | 0.4525 | 120.0641 | 614.8710 | 0.8867 | 86.9971 | 57.473 | 7.184 | 385.8209 |
+| 29000 | 0.4687 | 121.5085 | 662.3517 | 0.8855 | 86.6921 | 57.675 | 7.209 | 386.8527 |
+| 30000 | 0.4848 | 121.3954 | 620.4891 | 0.8915 | 86.9396 | 57.511 | 7.189 | 805.0448 |
+| 31000 | 0.5010 | 119.1724 | 604.0428 | 0.8831 | 87.0473 | 57.44 | 7.18 | 382.2313 |
+| 32000 | 0.5172 | 118.1496 | 632.1021 | 0.8800 | 87.0169 | 57.46 | 7.183 | 377.2617 |
+| 33000 | 0.5333 | 116.5277 | 597.8567 | 0.8738 | 86.7512 | 57.636 | 7.205 | 322.2620 |
+| 34000 | 0.5495 | 116.1844 | 591.6924 | 0.8734 | 87.2311 | 57.319 | 7.165 | 431.3317 |
+| 35000 | 0.5657 | 115.5994 | 565.9454 | 0.8686 | 86.8167 | 57.593 | 7.199 | 336.3313 |
+| 36000 | 0.5818 | 115.9320 | 609.9918 | 0.8674 | 87.1488 | 57.373 | 7.172 | 253.6102 |
+| 37000 | 0.5980 | 115.0621 | 595.2911 | 0.8660 | 87.1004 | 57.405 | 7.176 | 323.4260 |
+| 38000 | 0.6141 | 115.5635 | 590.6086 | 0.8654 | 86.9067 | 57.533 | 7.192 | 282.2412 |
+| 39000 | 0.6303 | 113.5796 | 546.1489 | 0.8586 | 86.5012 | 57.803 | 7.225 | 306.1125 |
+| 40000 | 0.6465 | 113.4385 | 558.4144 | 0.8583 | 86.6261 | 57.719 | 7.215 | 246.7947 |
+| 41000 | 0.6626 | 112.7097 | 563.5562 | 0.8558 | 86.9289 | 57.518 | 7.19 | 263.4834 |
+| 42000 | 0.6788 | 112.6048 | 556.9202 | 0.8573 | 86.8975 | 57.539 | 7.192 | 287.7979 |
+| 43000 | 0.6949 | 112.9025 | 569.7087 | 0.8534 | 86.3213 | 57.923 | 7.24 | 295.2722 |
+| 44000 | 0.7111 | 111.3180 | 584.7252 | 0.8534 | 86.7833 | 57.615 | 7.202 | 311.5563 |
+| 45000 | 0.7273 | 112.7623 | 589.8597 | 0.8520 | 85.8832 | 58.219 | 7.277 | 452.9366 |
+| 46000 | 0.7434 | 111.0763 | 583.6953 | 0.8497 | 86.9028 | 57.536 | 7.192 | 323.7285 |
+| 47000 | 0.7596 | 110.0631 | 570.5529 | 0.8481 | 86.1396 | 58.045 | 7.256 | 278.4229 |
+| 48000 | 0.7758 | 112.4039 | 498.8431 | 0.8470 | 86.0091 | 58.133 | 7.267 | 315.6181 |
+| 49000 | 0.7919 | 111.2748 | 564.9885 | 0.8465 | 86.4014 | 57.869 | 7.234 | 261.0319 |
+| 50000 | 0.8081 | 111.5950 | 594.9554 | 0.8454 | 87.4501 | 57.175 | 7.147 | 240.7725 |
+| 51000 | 0.8242 | 110.0546 | 563.8345 | 0.8446 | 85.9134 | 58.198 | 7.275 | 320.1174 |
+| 52000 | 0.8404 | 109.2966 | 548.4256 | 0.8428 | 86.5788 | 57.751 | 7.219 | 318.7099 |
+| 53000 | 0.8566 | 109.3136 | 539.9846 | 0.8395 | 86.3394 | 57.911 | 7.239 | 340.8982 |
+| 54000 | 0.8727 | 110.7834 | 561.4149 | 0.8436 | 86.2011 | 58.004 | 7.25 | 361.5285 |
+| 55000 | 0.8889 | 110.2941 | 576.0907 | 0.8421 | 86.733 | 57.648 | 7.206 | 297.2107 |
+| 56000 | 0.9051 | 109.5600 | 571.4385 | 0.8433 | 86.2508 | 57.97 | 7.246 | 370.3730 |
+| 57000 | 0.9212 | 109.7474 | 566.3444 | 0.8457 | 86.5407 | 57.776 | 7.222 | 900.0065 |
+| 58000 | 0.9374 | 109.4155 | 621.2332 | 0.8426 | 86.3669 | 57.893 | 7.237 | 493.3487 |
+| 59000 | 0.9535 | 110.1230 | 581.3542 | 0.8391 | 86.4324 | 57.849 | 7.231 | 272.2826 |
+| 60000 | 0.9697 | 108.2997 | 582.5030 | 0.8340 | 86.046 | 58.108 | 7.264 | 323.5555 |
+| 61000 | 0.9859 | 109.2711 | 566.8240 | 0.8381 | 86.749 | 57.638 | 7.205 | 312.4312 |
+| 61875 | 1.0 | 109.1439 | 575.3599 | 0.8346 | 86.8825 | 57.549 | 7.194 | 265.7449 |
 ### Framework versions
 - Distily 0.2.0

logs/attn_loss_fn=mse, attn_weight=2.0/events.out.tfevents.1723724446.93d6cbb3ad53 ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:85185b325f8578284a12e9f098013cf4c3d5ccf2e4cbd9c6fd55cd8961a0e644
+size 529