SciBERT

SciBERT is a BERT-Base-sized model trained from scratch on scientific text rather than general web text. The training corpus is a random sample of 1.14 million full-text papers from Semantic Scholar, totaling about 3.17 billion tokens, with 18 percent from computer science and 82 percent from the biomedical domain.

Its key innovation is SCIVOCAB, a WordPiece vocabulary built with SentencePiece on the scientific corpus, which overlaps only 42 percent with BERT's original vocabulary. SciBERT beat BERT-Base on a suite of scientific NLP tasks including NER, text classification, and relation classification, and it remains a widely used baseline for scholarly text processing.