Data Science/Computational Social Science Seminar: Yuqing Kong

Eliciting Thinking Hierarchy without a Prior 

Abstract: 
A key challenge in crowdsourcing is that majority may make systematic mistakes. Prior work focuses on eliciting the best answer without a prior even when the majority is wrong. Here without any prior, we want to elicit the full hierarchy where the higher-ranking answers, which may not be supported by the majority, are from more sophisticated people. We propose a new model, called the thinking web, that describes the hierarchy among people's thinking types through a weighted directed acyclic graph. To learn the thinking web without any prior, we propose a novel, powerful and practical elicitation paradigm, the Answer-Guess paradigm and it works as follows. First, we ask a single open response question and ask for both of each respondent's answer and guess(es) for other people's answers. Second, we construct an Answer-Guess matrix that records the number of people who report a specific Answer-Guess pair. Third, by ranking the answers to maximize the sum of the upper triangular area of the matrix, we obtain and visualize the hierarchy of the answers without any prior. We also conduct four empirical studies to demonstrate the superiority of our approach compared to the plurality vote and also validate our thinking web model: more sophisticated people can reason about less sophisticated people’s mind and the hierarchy can be approximately described by a directed acyclic graph. 

Speaker bio: 

Yuqing Kong is currently an assistant professor at the Center on Frontiers of Computing Studies (CFCS), Peking University. She obtained her PhD from the Computer Science and Engineering Department at the University of Michigan in 2018 and her bachelor's degree in mathematics from the University of Science and Technology of China in 2013. Her research interests lie at the intersection of theoretical computer science and the areas of economics: information elicitation, prediction markets, mechanism design, and the future applications of these areas to crowdsourcing and machine learning. Her papers were published in several conferences, including WINE, ITCS, EC, SODA, AAAI, NeurIPS, ICLR, ECCV, IJCAI, WWW.

Register to attend DS/CSS events at umsi.info/DSCSS.