About The Game

Objectives

In this research, we aim to use an interactive online experiment to capture the tacit gender biases of different demographic categories. We do so by the means of a series of questions for which the participants should recommend multiple words or chose from multiple choices. We also train a series of labeled word-embedding by incorporating people into the training process. Our interactive setup collects two pieces of information from users, i.e., gender and race, and trains multiple embeddings based on these two attributes. This approach allows us to distinguish the embeddings based on users’ responses to the questions regarding the race and gender with which they prefer to be identified. Training multiple word vectorization matrices provide a unique opportunity for researchers in the area of text analytics and social sciences to conduct in-depth analyses on the potential differences across different demographic categories with respect to word associations. We can also test the underlying assumptions of the proposed debiasing models about the presence of biases in the existing embeddings by comparing the responses from different genders and races. Besides, our approach trains the embeddings by direct inputs from the users, which makes it independent of randomly scrapped, public corpora. To the best of our knowledge, this is the first scholarly attempt to incorporate crowd in the training process of word embeddings and is among the first to segregate the word embeddings on the basis of gender and race of the content creators. By the completion of this project, we will be able to answer the following research questions:

  • Is there a significant difference among the word-embeddings trained by different genders?
  • Is there a significant difference among the word embeddings trained by different races?
  • Are different genders neutral with respect to gender-neutral words?
  • If biased, which group of users (gender and race) contributes more to the existing biases?

We also provide a unique and publicly available repository for researchers who are interested in conducting more studies on the collected supervised data from users. Besides, our repository can update the embeddings in real time with no requirement of collecting massive corpora and the computational cost of retraining the embeddings using all contents. It is also worth noting that the competitive environment of our game incentivizes the users to provide accurate responses which increases the quality of the trained word embeddings compared to those trained based on publicly available corpora. Finally, the corresponding metadata of the users can be selected and attached to the responses for more fine-grained analysis.

Specific Aim 1:

To assess the presence and degree of gender bias among different demographic groups including racial and gender categories. To do so, we categorize the participants into 28 demographic classes which are formed based on 4 gender categories (Male, Female, Other, Prefer not to Answer) and 7 racial categories (African American, White, Asian, American Indian or Alaska Native, Native Hawaiian or Pacific Islander, Other, Prefer Not to Answer). This comprehensive categorization allows us to assess the distribution of biases across different demographic categories.



Specific Aim 2:

To assess if the identified biases are consistent with the existing body of knowledge. Researchers have identified multiple forms of gender biases in the existing word embeddings all of which are derived from the corpus that are collected from publicly available resources such as Wikipedia pages. Our approach differs from the existing approaches since we are collecting the data directly from people rather than piles of online contents.



Specific Aim 3:

To develop debiased word vectorization for NLP researchers. To do so, we measure the overall biases of individual participants. Using this value, we can aggregate the participants’ inputs in a weighted manner, where higher bias leads to a lower weight.