Welcome back to our Responsible AI series, created by the Digital Scholarship and Data Services department as part of its ongoing commitment to promoting ethical and responsible AI practices for the Hopkins community.
Using AI tools responsibly means being aware of their potential for inaccuracy and evaluating their output before integrating it into your research and scholarship. In addition to hallucinating, or producing inaccurate information, these tools can also reproduce problematic views present in their training data, which can perpetuate bias if unchecked.
Where Does the Training Data for LLMs Come From?
Generative AI tools work by predicting responses to user prompts based on correlations found in their vast training data. Where does all this training data come from?
The short answer is the internet. LLM-powered chatbots like ChatGPT typically draw their training data from a number of internet sources, including the Common Crawl, Wikipedia, pages linked by Reddit posts, and book databases. Image generators like DALL-E and Stable Diffusion are trained on large collections of images scraped from the internet and labeled by humans, such as LAION.
Types of Bias
Training datasets are vast, but that doesn’t necessarily mean they’re representative of society. In generative models, two types of bias interact to amplify dominant views:
- Social bias refers to biases present in the training data that reflect hegemonic views that exist in our society, such as misogyny, racism, ageism, etc. While developers take measures to filter hateful speech from training data, implicit biases can persist.
- Availability bias refers to the fact that socially biased views are over-represented in training data because of compounding factors related to internet participation—from who has internet access, to who is most active on the sites scraped.
Availability bias amplifies social bias, meaning that content produced by LLMs is more prone to reflect bias than the average person. Responsible AI users know how to evaluate AI-generated content for bias to prevent harmful consequences.
How Does Bias Manifest?
Here are some things to look for when evaluating AI-generated content for bias:
- Stereotypical associations: Say you are a public health provider giving a presentation about the importance of affordable housing. To add interest to your presentation, you use a generative AI tool to create an image of people in affordable housing. The image you generate depicts stereotypical associations that have been amplified by the data available to the AI image generator, but you don’t realize this. When you display this image during your presentation, it makes some members of the audience feel uncomfortable and unwelcome at your presentation.
- Lack of information about under-represented groups: Say you are a clinician, and you decide to use generative AI for a quick overview of the side effects of a newly developed drug. If women react differently to the drug, but are under-represented in the training data, the information you receive could have adverse effects if applied to women.
- Skewed perspectives on history and current events: Say you are an undergraduate student writing a paper on the history of a censored social movement, and you ask generative AI for a summary of this topic to get you started. Training data sourced from the internet may represent only mainstream perspectives on the topic. Without the ability to evaluate the sources the AI tool is pulling from, you may assume that the information provided by the AI tool is the full picture.
- Linguistic bias: Say you are an educator using generative AI to create a writing prompt for your class. Because the text used to train AI is not evenly sourced from across the world, the material you generate includes some vocabulary and cultural references specific to the United States, and international students have a harder time completing the assignment.
Tips for Responsible Users
- Identify training data sources: If you can, identify the sources of the training data for whatever tool you are using (this may be easier for tools that are published open source.) You can start by searching for the “model card” or “system card” for whatever tool you’re using. Consider who might be over- or under-represented in those sources.
- Consider impact: Think about who will be impacted by your use of AI-generated information or content, and if they are likely to be accurately represented in the training data.
What Can Researchers Do?
Less biased AI starts with well-documented, representative training data.
Researchers can contribute to higher-quality AI tools by curating and carefully documenting training datasets like these Responsible Datasets in Context or NAIRR’s Datasets to power AI literacy, education and innovation, so that the information fed to AI models is contextualized. JHU Data Services can support researchers in publishing high-quality data via curation services for data and code shared in the Johns Hopkins Research Data Repository.
Wikipedia edit-a-thons, like Art+Feminism and this Edit-a-thon for Disabled Scientists at JHU, can help make Wikipedia, which is one of the primary sources of training data for AI models, more representative of diverse perspectives. Keep an eye out for edit-a-thons in your area of expertise to help contribute to a more representative internet.
No Bias, No Problem?
Representative training data can improve AI, but it’s important to recognize that accurate representation in AI tools can be weaponized against marginalized groups. For example, the accuracy of facial recognition technology means that it can cause great harm in the wrong hands.
When it comes to information and the technologies that harness it, societies with an unequal distribution of power must navigate a tension between openness and safety. This underscores the importance of consent and self-determination when it comes to digital content—a topic our next post on privacy and data stewardship will cover.