Automated categorization of structured documents

Abstract

Automatic text categorization is a problem of assigning text documents to pre¬-defined categories. This requires extraction of useful features. In most of the applications, text document features are commonly represented by the term frequency and the inverted document frequency. In case of structured documents, dominant features are often characterized by a few sentences bearing additional importance. The features from more important sentences should be considered more than other features. Another issue in automated document categorization is the manageability and integrity of large volume of text data where the documents can be very large and often certain parts of the document misrepresent the primary focus of the document.

In this work, we study several document categorization techniques mostly in light of categorizing structured text. Categorization on summarized text is also studied in details with some specific purpose. While summarizing, the importance of appropriate sentences has been considered. The approach is verified by conducting experiments using news group data sets using three summarization methods. The set of whole documents and the summaries were subjected to a couple of classical categorization techniques, viz. Naive Bayesian and TF-IDF (Term Frequency-Inverse Document Frequency) algorithm. A major observation was preservation of the sanctity of the categorization even while that is based on the summary documents rather than the whole document. Goodness of the approach was verified on about 500 documents and the test results are enclosed.