I have try to do Pre-train BERT MLM with TFDS datasets follow this official document of tensorflow: https://github.com/tensorflow/models/blob/master/official/nlp/docs/pretrain.md#pre-train-bert-mlm-with-tfds-datasets
As they suggested to use --config_file=configs/experiments/wiki_tfds_pretrain.yaml
config.
When I check file, I notice cls_heads: [{activation: tanh, cls_token_idx: 0, dropout_rate: 0.1, inner_dim: 768, name: next_sentence, num_classes: 2}]
which mean it is pretrain for next_sentence?
I try to find config for cls_heads
for Pre-train BERT MLM but I cannot found anything.
May I know if the config_file
above can be use for pre-train MLM?