Skip to Main Content

AI Support: Evaluating AI

This guide provides information and considerations for identifying, evaluating, and utilizing AI tools in your research and practice

Evaluating AI

It's easy to think that ChatGPT, other LLMs, and AI tools are a reliable authority when it comes to the information they provide. However, AI is prone to biases and errors.

It's important to keep the human in the loop - LLMs require humans thought processes to function. Their output may not meet your needs or expectations. 

Why and how to evaluate?

When it comes to evaluating AI tools, there are a few key factors to consider.

  • Accuracy of the tool.
    • How reliable are the results it produces?
    • Can it consistently identify patterns and make predictions that are correct?
  • Transparency of the tool.
    • Can you understand how it works and how it makes decisions?
    • Is it clear what data it is using and how it is being processed?
  • Fairness and bias of the tool.
    • Is the tool treating all individuals equally, or is it exhibiting bias towards certain groups of people?
    • AI trained with biased datasets can continue to perpetuate discrimination and inequality.
  • Ethical implications of using AI tools.
    • Are there potential risks or unintended consequences of relying on this technology?
    • Is the tool being used in a way that respects privacy and human dignity?

By carefully evaluating AI tools on these different criteria, we can ensure that we are using them in a responsible and effective way.

Considerations from University of Toronto's "Critically Evaluating AI Tools" Research Guide: https://guides.library.utoronto.ca/image-gen-ai/critical-evaluation

 

Evaluating AI Tools

IF I APPLY is the UW Libraries' tool for approaching research and source evaluation. It's also helpful to use as we consider using an AI tool

Personal Steps: 

I- Identify emotions attached to topic.  

  • Why are you interested in using AI? What do you expect it to do? 

F- Find reference sources and evaluate bias for a proper view of the topic  

  • AI also has bias- consider what the information the AI has been trained on and what its intended purpose is. What are its reference points/sources? 

I- Intellectual courage to seek authoritative voices on topics that may be outside of thesis.  

  • Intellectual courage to cite your use of AI tools and own your own knowledge 

Source steps: 

A- Authority established. Does the author have education and experience in that field?  

  • What is the AI tool really supposed to do? What authority does using an AI tool give you and your assignment? 

P- Purpose/point of view of source. Does the author have an agenda beyond education or information?  

  • What is the purpose of the AI source and what perspective is it giving and/or missing?  

P- Publisher. Does the publisher have an agenda? 

  • Who created your tool and what are their company goals? What kind of privacy does the tool have? 

L- List of sources (bibliography). Is the evidence sound?  

  • How can you best understand how the AI generated the information? Are you able to verify its claims? 

Y- Year of publication. Does the year of publication affect the information?  

  • What version of the tool are you using? What does the tool know/not know about based on its creation date? 

Some of these things you may not be able to answer because the tool or company isn't transparent with their information. This approach can help you think through using an AI tool, even if you can't answer all the questions.