CMU’s new ‘restructured pre-training’ NLP approach pre-trains model over valuable restructured data

Natural language processing (NLP) paradigms are evolving rapidly, from fully supervised learning through pre-training and fine-tuning and, most recently, pre-training with instant prediction. Many AI researchers are exploring new developments in this area as NLP systems improve and can be used in the real world.

In the new publication, reStructured Pre-training, a research group from Carnegie Mellon University offers reStructured Pre-training (RST), a revolutionary NLP paradigm that pre-trains models on actionable restructured data. The article opens with a comment highlighting the authors’ claim that pre-training on restructured data would be more successful than pre-training on raw data alone.

Let’s take a look at the main contributions of this study, as summarized by the group:

  1. This study seeks to build a “hypothesis of the evolution of NLP technique” from a global point of view by examining the internal relationship between the development of current NLP technology.
  2. The group proposed restructured pre-training as a new paradigm for modeling NLP. According to this paradigm, pre-training and tuning of models is a process of storing and accessing data, and an excellent way to store data should ensure that data that is expected to be easy to find.
  3. The QIN, the first deep learning-based AI system for the Gaokao English test, has been developed.
  4. The group of researchers launched the Gaokao Benchmark to measure their progress toward human-level intelligence, and they built an interactive scoreboard using ExplainaBoard as a Gaokao Benchmark.
  5. The effectiveness of AI in English for the Gaokao Assessment has given them a fresh perspective that AI technology can enhance teaching and help address a variety of educational and educational challenges. The excellent results on more than 50 datasets from different NLP tasks demonstrate the value of data-centric pre-training and encourage more research.

Unlike existing NLP paradigms, which focus on model architecture or structure, the proposed RST seeks to maximize the value of given data by covering as many types of signals as possible and giving special access to these signals based on what downstream activities are needed. to have.

The RST approach is divided into three phases: restructuring, pre-training and fine-tuning. Current data signals in various forms are first reorganized into a popular method for pre-training models before selecting and training this structured data. Finally, the model is refined with restructured, labeled data for better performance.

The study also unveils the first deep learning-based AI system designed explicitly for China’s Gaokao English College Entrance Examination, as the researchers believe.

The researchers tested the proposed RST on a range of NLP tasks, beating out basic models such as GPT-3 and T0pp in 52 of the 55 datasets tested. The QIN AI system also outperformed the typical student on the Gaokao test, scoring 40 points higher and 15 points better than the GPT-3 by 1/16 of the parameters.

This article argues that blindly sticking to supervised or unsupervised, pre-training or fine-tuning, few-shot or zero-shot makes little sense in NLP. In fact, it is only a matter of making the best use of the knowledge that is available from data from the outside world.

This Article is written as a summary article by Marktechpost Staff based on the paper  'reStructured Pre-training'. All Credit For This Research Goes To Researchers on This Project. Checkout the paper, github and reference article.

Please Don't Forget To Join Our ML Subreddit

Leave a Comment

Your email address will not be published.