Part of speech tagging for Gujarati text

Abstract

Part-of-speech (POS) tagging is a process of assigning the lexicon category to each lexicons in a given natural language sentence, that best suits the definition of the lexicon as well as the context of the sentence in which it is used. Part-of-speech tagging is an important part of Natural Language Processing (NLP) and is useful for most NLP applications. Part-of-speech tagging is often a primary step in most of the NLP tasks such as chunking, parsing, etc. Gujarati is the state language of Gujarat, a western state of India, and is spoken by 70 percent of the state's population. More than 46 million people worldwide consider Gujarati as their first language. Apart from Gujarat, it is widely spoken in the states of Maharashtra, Rajasthan, Karnataka and Madhya Pradesh and also around the world. Natural language processing of Gujarati is in its early stage of existence. Gujarati POS tagger is a core component for most NLP applications. Information retrieval, machine translation, shallow parsing and word sense disambiguation tasks can be work more effectively and efficiently with the existence of a POS tagger. Our focus in this work is to develop an effective Gujarati text POS tagger. Our main task of thesis is to built a system which can annotate part-of-speech for Gujarati texts automatically, with the help of various machine learning algorithms. We have used tag sets defined by IIIT Hyderabad. We have used two machine learning techniques one is Hidden Markov Model (HMM) and second is Conditional random Field (CRF). Since Gujarati is a morphologically reach language, we can use Morphological Analyzer (MA) to restrict the set of possible tags for a given words. Gujarati language is based on Paninian framework, rules of morphology are well-defined. Hence we have defined morphological rules for Gujarati. While MA helps us to restrict the possible choice of tags for a given word, one can also use prefix/suffix information (i.e., the sequence of first/last few characters of a word) to further improve the models. HMM model uses suffix information during smoothing process while CRF uses suffixes as a feature. (http://www.oclc.org/languagesets/educational/languages/india.htm)