Sentence detection

Abstract

Sentence detection is a very important task for any natural language processing (NLP) application. Accuracy and performance of all other downstream natural language processing (NLP) task like Sentiment, Text Classification, named entity recognition (NER), Relation, etc depends on the accuracy of correctly detected sentence boundary. Clinical domain is very different compare to general domain of languages. Clinical sentence structure and vocabulary are different from general English. That’s why available sentence boundary detector tools are not performing well on clinical domain and we required a specific sentence detection model for clinical documents. ezDI Solutions (India) LLP have developed such system that can detect the sentence boundary. We examined Bidirectional Encoder Representations from Transformers (BERT) and Bidirectional Long Short-Term Memory (BiLSTM) algorithm and used BiLSTM-BERT hybrid model for sentence boundary detection on medical corpora.