Web content outlier detection using latent semantic indexing

Abstract

Outliers are data elements different from the other elements in the category from which they are mined. Finding outliers in web data is considered as web outlier mining. This thesis explores web content outlier mining which finds applications in electronic commerce, finding novelty in text, etc. Web content outliers are text documents having varying contents from the rest of the documents taken from the same domain. Existing approaches for this problem uses lexical match techniques such as n-grams which are prone to problems like synonymy (expressing the same word in different ways), which leads to poor recall (an important measure for evaluating a search strategy). In this thesis we use Latent Semantic Indexing (LSI) to represent the documents and terms as vectors in a reduced dimensional space and thereby separating the outlying documents from the rest of the corpus. Experimental results using embedded outliers in chapter four indicate the proposed idea is successful and also better than the existing approaches to mine web content outliers.