A Network-centric Framework for Auditing Recommendation Systems
Uploaded by: Alt-Tab
Upload date: 2019-02-19 09:54:44

Comments:

Question asked: how can you audit a recommender system if you don't have access to user-item interactions? Answer proposed: use a "recommendation graph" and simulate user behavior with a random walk with teleportation. Legitimate question, I am not completely convinced by the answer but it has some merits, in particular, simplicity. ### Introduction - to measure : diversity / segregation-polarization (not accuracy) - use the structure of an underlying network - 3 datasets, all movies RS : IMDB, Google Play, Netflix ; films have a genre in a list ##### Contributions : quantifying diversity - use a notion of graph: directed link from i to j if j is recommended on i page - measures based on films genre ; popularity based on the graph structure: in-degree, PageRank - segregation quantified with concentration and evenness (8: Chakraborti et al, 2017) ### Related work ##### diversity in RS - 9 : Nguyen et al, WWW 2014: CF impact on user behaviors - 10: Zhou et al, PNAS 2010 (Auralist) defines novelty - 11: Santini and Castells defines diversity et novelty with a fuzzy interpretation - 12: Vargas et Castells, RecSys 2011: novelty and diversity from a user perspective - 13: Lathia et al, SIGIR 2010: considers temporal aspects ##### on polarization dynamics - 20: DeGroot explanation from opinion modeling - 17: Dandekar et al, PNAS 2013 : polarizing effects of RS algorithms - two main schools of thought to explain polarization: either opinion is reinforced by interactions with like-minded people or people are exposed to opposite views then reject them (hence polarization ### Framework for auditing RS ##### Network construction : see above - possible to give weights to the links in tne network depending on item-item similarities, or based on the rank of j reco on i page - but overall different with literature: directed network and not based on similarity ##### User modeling : see above (RW with teleport), as we don't have access to navigation logs - then we consider distribution of types visited during the RW ##### Datasets : see above - general stats : see tab1 - collect by snowball sampling (from one node, then neighbors, then neighbors of neighbors... like a BFS) - to account for personalization effects, crawling from a unique IP adress - film genres: 15 for GP, 29 for Netflix and IMDB - compare distributions for 400 steps RW with no teleport (so, sort of random sampling): GP dominated by Action, IMDB more balanced ### Diversity in RS ##### existing measures - similarity between items computed with Jaccard coefficient => possible to compute usual measures (ILS (15), long-tail novelty (14), unexpectedness (23), source-list diversity(16)) - Table 2 : traditional measures on the datasets - observation : Netflix has greater source-list diversity, but IMDB more diverse according to other measures ##### new measures - assortativity by genre, by popularity (using as a proxy normalized centralities: in-degree, pagerank) => see Table III - contingency matrices: fraction of links from genre to genre (fig2); in general RS recommend from a genre to the same one + some specific relation between specific genres - equivalent for popularity: bins of popularity (bottom/middle/top) then count links from a bin to another (fig3); in general push toward long-tail, especially on IMDB ##### New measures based on RW - entropy of the genre distribution obtained - exploration when parameters vary (N = RW length, tp = teleportation probability, starting point) - fig4a : evol with t_p (small increase then plateau) - fig4b : évol with N (growth) - note that the RW is finite and we do not reach the steady state ##### Information segregation in RS * use measures in Chakraborty et al.(8) : evenness and concentration * evenness : to what extent a group is exposed uniformly to info units , it (1-Gini) with Gini computed on the genres consumed by the users of the group (1 ~ even consumption) * concentration : 1/2 . sum (fraction of films i consumed by the group).(fraction of the films consumed by the groups).(fraction of the films of genre i) ; if concentration is low what is consumed is close to what is proposed * results on fig5

You comment anonymously! You will not be able to edit/delete the comment.

Please consider to register or login.

Use $\LaTeX$ to type formulæ and markdown to format text.
When you post something to which you hold the copyright you authorise us to do distribute this data across the scientific community. You can post public domain content. All user-generated content will be freely available online. Please see this page to learn more about Papersγ's terms of use and privacy policy.