MPNet combines strengths of masked and permuted language modeling for language understanding

By Microsoft Research - 2020-12-09

Description

Pretrained language models have been a hot research topic in natural language processing. These models, such as BERT, are usually pretrained on large-scale language corpora with carefully designed pre ...

Summary

  • Pretrained language models have been a hot research topic in natural language processing.
  • The information usage of different pretraining objectives Next, we analyze how MPNet can avoid the disadvantages of MLM and PLM through case analysis.
  • 2019) 84.6 90.5 89.2 66.4 93.5 84.8 52.1 87.1 79.9 ELECTRA (Clark et al., It can be seen that removing position compensation, permutation, and output dependency all result in accuracy drop in GLUE and SQuAD, which demonstrates the effectiveness of MPNet on leveraging the position information of the full sentence and modeling the output dependency among predicted tokens.

 

Topics

  1. NLP (0.34)
  2. UX (0.05)
  3. Machine_Learning (0.01)

Similar Articles

How to Use AutoKeras for Classification and Regression

By Machine Learning Mastery - 2020-09-01

AutoML refers to techniques for automatically discovering the best-performing model for a given dataset. When applied to neural networks, this involves both discovering the model architecture and the ...

Time-Series Forecasting with Google BigQuery ML

By Medium - 2021-02-16

If you have worked with any kind of forecasting models, you will know how laborious it can be at times especially when trying to predict multiple variables. From identifying if a time-series is…

pytorch-widedeep: deep learning for tabular data

By Medium - 2021-02-22

This is the third of a series of posts introducing pytorch-widedeepa flexible package to combine tabular data with text and images (that could also be used for “standard” tabular data alone). The…