Empirical Study Of Sampling Heuristics For Fairness In Ranking

Abstract

Ranking is an important problem for a variety of applications. Classical algorithmsfor ranking may be unfair towards certain group of people or individuals.Fairness may be jeopardized by ranking algorithms that produce discriminatoryresults due to biased data or sampling methods. Hence in the past few years, algorithmsto enforce fairness in ranking have been proposed. However they arecomputationally expensive. Hence it is better to train these on smaller samples ofdata. In this empirical study, multiple sampling strategies for fair ranking algorithmsare compared and evaluated.Uniform sampling, Leverage Score sampling, K -Medoid and Row Norm samplingare the four sampling strategies that are the subject of this study. The thesistests and assesses the effectiveness of various sampling heuristics using a realworlddata set i.e. Yahoo Learning To Rank Challenge Data set.Our work shows that all heuristics perform reasonably well when compared withfull data set, at the same time, giving impressive benefits in terms of computationtime. It is an open question to obtain some theoretical guarantees for these samplingstrategies for fair ranking algorithms.