Using a checklist to evaluate GenAI outputs
Students create a checklist to help evaluate content produced by GenAI.
Purpose
This activity helps student develop critical AI literacy skills through the use of a checklist to evaluate content produced by GenAI. The process reveals both GenAI capabilities and limitations while building students' evaluative capability.
This activity may be suitable for subjects introducing GenAI use for assessments where permitted.
What will students achieve?
Students will be able to:
- Use a checklist to evaluate GenAI content for its accuracy, reliability and potential biases.
Required resources
- Access to Copilot, ChatGPT, or another GenAI tool of choice.
Activity instructions
- Students form groups. Each group selects a topic or are given a list of topics to select from. Alternatively, the same topic may be given to the whole class.
- Student groups brainstorm what constitutes quality content for their specific topic and creates a checklist to help determine the level of quality of content. In their discussion they may also consider these questions:
- What makes the content effective?
- What would make a professional accept or reject the content?
- What ethical considerations are relevant?
Groups can organise their criteria into a structured checklist with the support of GenAI. More discipline specific criteria can be added depending on the subject matter being explored.
For example, the checklist may look something like this:
| GenAI content quality assessment checklist | High | Medium | Low |
|---|---|---|---|
| Accuracy and factual reliability | All content presented is accurate | Contains some minor inaccurate claims or one or more major inaccurate claims | Contains several minor and major inaccurate claims |
| Relevance | Content is relevant to the subject matter at hand | Content presented is partly relevant to the subject matter at hand | Content presented is not relevant to the subject matter at hand |
| Depth and critical thinking | Considers multiple perspectives, acknowledges complexity | Surface-level treatment with limited analysis | Superficial or overly simplistic or no evidence of critical engagement |
| Representation and inclusivity | Acknowledges diverse perspectives, avoids stereotypes, culturally responsive | Limited perspective; may inadvertently reinforce dominant narratives |
Narrow viewpoint; excludes marginalised voices Contains biased, stereotypical, or harmful representations |
| Source attribution and evidence | Claims are well-supported; appropriate acknowledgment of knowledge limitations | Limited evidence or reasoning provided |
Assertions made without justification No evidence or reasoning; presents opinions as facts |
| Tone and appropriateness | Tone well suited to purpose, audience, and context | Noticeably inappropriate tone in some sections | Tone significantly misaligned with purpose or completely inappropriate tone |
| Language | Original phrasing; avoids formulaic AI patterns of formulating sentences typical of GenAI produced content | Frequently uses predictable AI language patterns |
Highly formulaic and generic throughout Entirely clichéd or template-like. |
- Groups then share their checklists with the class for a quick discussion. Groups decide if anything needs to be added, omitted, or clarified further.
- Groups come up with prompts to generate the content using the tool of their choice. Individually, students then apply the checklist. Where possible, they provide evidence for each score, noting specific examples from the GenAI output to justify their rating.
- Students respond to broader guiding questions. For example:
- Is the content of a professional standard? Why or why not?
- What would you need to modify before using this content?
- Are there any ethical concerns present (bias, stereotypes, oversimplifications)?
- What does GenAI do well? What does it struggle with?
- Groups reconvene. Students share their score, the evidence that supports it, discuss their takeaways, and identify patterns in the GenAI’s performance.
- Groups develop a concrete action plan for improving GenAI output, including how they would modify the original prompt to address weaknesses, what specific edits could be made to the GenAI output, and what (if any) additional steps would ensure quality.
Considerations
- Emphasise that this activity teaches critical evaluation, not how to bypass learning.
- Be explicit about where the boundaries lay in assessments. If this is a non-graded formative activity, students may need to approach GenAI differently for assessments.