Welcome back to our Responsible AI series, created by the Digital Scholarship and Data Services department as part of its ongoing commitment to promoting ethical and responsible AI practices for the Hopkins community.
“Fluent but Not Factual”
Chatbots are designed to generate language that seems as coherent as human communication. They function as what computer scientists Emily Bender, Timnit Gebru, and others have called “stochastic parrots,” assembling language from probabilistic patterns that they learn from the context of words in their training data. In other words, the information they produce, while often true, is incidental to their endeavor to mimic human language, which is why chatbots have been recognized as “fluent but not factual.”
At the same time, chatbots are fine-tuned to generate responses that sound helpful and provide the information that a user expects in response to their instructions. This sometimes results in a phenomenon called hallucination: when chatbots confidently deliver inaccurate information wrapped in eloquent language.
What Causes Hallucination?
As British statistician George Box once said, “All models are wrong, some are useful.” Because AI models are predictive in nature and don’t possess true knowledge, their potential for hallucination is inherent. The technical reasons that AI models produce hallucinations are:
- Training data: Hallucinations can occur when training data contains factually inaccurate information, lacks information on a certain topic, or contains contradictory information on a topic from different sources.
- Model architecture: During the training process, the model determines what rules produce the most accurate predictions about the training data. Because these rules cannot apply to every situation, the model will fabricate information in some scenarios – sometimes even when it has access to the correct information.
- Inference: In machine learning, inference refers to what happens when a model is applied to new data. Many models, particularly chatbots, have a certain amount of randomness built in when producing a response to a prompt. While intended to keep responses interesting and varied, this can also introduce uncertainty and contribute to hallucination.
AI, Explain Yourself
While some hallucinations are obvious, others are more insidious, particularly when AI tools are being used to retrieve information a user doesn’t already know. One approach that could help reveal hallucinations is AI Explainability, which is the idea that AI tools should be able to “explain” the internal mechanisms behind their decisions.
In addition to supporting verifiability, explainability is also crucial for human oversight of AI. For example, an audit of an AI video tool used for hiring uncovered that the tool scored applicants who wore glasses or had a bookshelf in their background more favorably. Understanding a tool’s decision-making process is critical to determine whether it can reliably automate whatever human process it is intended to replace.
Explainability Challenges in Generative AI and LLMs
The decision points of predictive AI tools like the hiring tool described above are challenging, but not impossible, to expose. Unfortunately, the inner workings of Generative AI tools like chatbots are much harder to uncover.
The large language models that power chatbots leverage an approach called deep learning, in which hundreds of “layers” of calculations are developed to predict a response to a user prompt. Each of these layers can contain billions of parameters, making the prospect of summarizing their decisions daunting – you might need a degree in computer science to make sense of this explanation for why Claude Haiku chose the word “rabbit” to end a rhyming couplet.
As a model gets better at producing human-sounding language, its number of parameters grows and explainability becomes exponentially more difficult. So what can researchers do to navigate this landscape responsibly?
What Can AI Creators Do?
The authors of the Claude Haiku article linked above were able to investigate the model’s decision-making process because the model is open source, or made openly available to the public. If you are developing AI for research use, sharing some or all of your model’s components openly can support the explainability, transparency, and reliability of your work.
Resources for openly sharing AI at JHU:
- Open Source Programs Office can help you weigh options for openly licensing your software.
- Data Services can provide information about open data licenses and help you find a suitable platform to share your model.
- The Model Openness Framework can serve as a guide for openly licensing different components of your model.
- Check out our webinar on Licensing Options for Research Data & Software for a step-by-step case study of how to openly share a machine learning model.
Navigating Accuracy and Explainability Challenges as an AI user
While most AI users can’t control the transparency or explainability of a given AI tool, consider the following when navigating the current landscape of accuracy and explainability in AI:
- Evaluate impact: Evaluate the potential influence of inaccurate information on whatever task or workflow you plan to use an AI tool for. Reflect on what steps you would need to take to feel comfortable using the same piece of information or idea in your research if you had gotten it from a stranger.
- Cite your use of AI: Inaccuracies stemming for AI tools can have a snowball effect on research, particularly if undisclosed. Check out this guide from the library on how to cite your use of AI tools.
- Read documentation: Search for the “model card” or other documentation about generative models that you are using for your research. Look for information about the “knowledge cutoff date” to learn what information the model is most likely to produce accurately.
- Opt for open: When possible, opt for tools that are shared openly. Open source tools, like those evaluated by the Open Source AI Index, are more easily validated by others and are less vulnerable to changes that can hinder the reproducibility of your results.
- Don’t take AI-generated explanations at face value: You may be wondering if you can simply ask a chatbot to explain its response to you! Chatbots can and will generate explanations for their decisions if asked, but those explanations won’t necessarily be accurate.
- Don’t mistake explainability for accuracy: Research has shown that AI users are actually more likely to over-rely on AI that can explain its decisions, because the tool’s explainability makes it seem more trustworthy. Remember that AI tools that explain their decisions are important steps toward transparency, because they better allow you to judge whether to use the AI’s output.