File size: 9,380 Bytes
7934b29 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 |
.. _bert_pretraining:
BERT
====
BERT is an autoencoding language model with a final loss composed of:
- masked language model loss
- next sentence prediction
The model architecture is published in `BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding <https://arxiv.org/abs/1810.04805>`__ :cite:`nlp-bert-devlin2018bert`.
The model is originally trained on English Wikipedia and BookCorpus. BERT is often used as a language model encoder for downstream tasks, for example, :ref:`token_classification`, :ref:`text_classification`, :ref:`question_answering`, etc.
Domain-specific BERT models can be advantageous for a wide range of applications. One notable application is the domain-specific BERT in a biomedical setting,
e.g. BioBERT :cite:`nlp-bert-lee2019biobert` or its improved derivative BioMegatron :cite:`nlp-bert-shin2020biomegatron`. For the latter, refer to :ref:`megatron_finetuning`.
Quick Start Guide
-----------------
.. code-block:: python
from nemo.collections.nlp.models import BERTLMModel
# to get the list of pre-trained models
BERTLMModel.list_available_models()
# Download and load the pre-trained BERT-based model
model = BERTLMModel.from_pretrained("bertbaseuncased")
Available Models
^^^^^^^^^^^^^^^^
.. list-table:: *Pretrained Models*
:widths: 5 10
:header-rows: 1
* - Model
- Pretrained Checkpoint
* - BERT-base uncased
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:bertbaseuncased
* - BERT-large uncased
- https://ngc.nvidia.com/catalog/models/nvidia:nemo:bertlargeuncased
.. _dataset_bert_pretraining:
Data Input for the BERT model
-----------------------------
Data preprocessing can be either done on-the-fly during training or offline before training. The latter is optimized and recommended
for large text corpora. This was also used in the original paper to train the model on Wikipedia and BookCorpus. For on-the-fly data
processing, provide text files with sentences for training and validation, where words are separated by spaces, i.e.: ``[WORD] [SPACE] [WORD] [SPACE] [WORD]``.
To use this pipeline in training, use the dedicated configuration file `NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_preprocessed_config.yaml`.
To process data offline in advance, refer to the `BERT Quick Start Guide <https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/LanguageModeling/BERT#quick-start-guide>`__.
To recreate the original Wikipedia and BookCorpus datasets, follow steps 1-5 in the Quick Start Guide and run the script ``./data/create_datasets_from_start.sh`` inside the Docker container.
The ``downloaded`` folder should include two sub folders ``lower_case_[0,1]_seq_len_128_max_pred_20_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5``
and ``lower_case_[0,1]_seq_len_512_max_pred_80_masked_lm_prob_0.15_random_seed_12345_dupe_factor_5``, containing sequences of length 128 with a maximum of 20 masked tokens
and sequences of length 512 with a maximum of 80 masked tokens respectively. To use this pipeline in training, use the dedicated configuration file ``NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml``
and specify the path to the created hd5f files.
Training the BERT model
-----------------------
Example of model configuration for on-the-fly data preprocessing: `NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml <https://github.com/NVIDIA/NeMo/blob/stable/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml>`__.
Example of model configuration for offline data preprocessing: `NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_preprocessed_config.yaml <https://github.com/NVIDIA/NeMo/blob/stable/examples/nlp/language_modeling/conf/bert_pretraining_from_preprocessed_config.yaml>`__.
The specification can be grouped into three categories:
- Parameters that describe the training process: **trainer**
- Parameters that describe the datasets: **model.train_ds**, **model.validation_ds**
- Parameters that describe the model: **model**, **model.tokenizer**, **model.language_model**
More details about parameters in the config file can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.only_mlm_loss** | bool | Only uses masked language model without next sentence prediction. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **train_ds.data_file** | string | Name of the text file or hdf5 data directory. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **train_ds.num_samples** | integer | Number of samples to use from the training dataset, ``-1`` - to use all. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
More details about parameters for offline data preprocessing can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **train_ds.max_predictions_per_seq** | integer | Maximum number of masked tokens in a sequence in the preprocessed data. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
More details about parameters for online data preprocessing can be found below:
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **Parameter** | **Data Type** | **Description** |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.max_seq_length** | integer | The maximum total input sequence length after tokenization. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.mask_prob** | float | Probability of masking a token in the input text during data processing. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
| **model.short_seq_prob** | float | Probability of having a sequence shorter than the maximum sequence length. |
+-------------------------------------------+-----------------+--------------------------------------------------------------------------------------------------------------+
.. note::
For offline data preprocessing, **model.tokenizer** is null. For downstream task, use the same tokenizer that was used for
offline preprocessing. For online data preprocessing, **model.tokenizer** needs to be specified. See also :ref:`nlp_model` for
details.
Example of the command for training the model:
.. code::
python bert_pretraining.py \
model.train_ds.data_file=<PATH_TO_DATA> \
trainer.max_epochs=<NUM_EPOCHS> \
trainer.devices=[<CHANGE_TO_GPU(s)_YOU_WANT_TO_USE>] \
trainer.accelerator='gpu'
Fine-tuning on Downstream Tasks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
To use a trained BERT model checkpoint on a NeMo NLP downstream task, e.g. :ref:`question_answering`, specify
:code:`model.language_model.lm_checkpoint=<PATH_TO_CHECKPOINT>`.
References
----------
.. bibliography:: nlp_all.bib
:style: plain
:labelprefix: NLP-BERT
:keyprefix: nlp-bert-
|