Comparing different GenAI models

Students evaluate outputs from several GenAI models on the same prompt.

Purpose

This activity helps students evaluate GenAI outputs and identify issues in content. It may be a good formative activity to prepare students for an assignment where GenAI use is permitted and to provide an opportunity to explore ethical use of GenAI.

What will students achieve?

Students will be able to:

  • Compare GenAI outputs.
  • Identify issues in GenAI content.

Activity instructions

  1. Students select a question related to their subject or are provided with a question or topic. Examples of such topics include:
    • Historical events with contested narratives.
    • Social policy issues.
    • Emerging technologies and their impacts.
  2. Students, individual or in groups, prompt three or four different GenAI tools (such as ChatGPT, Claude, Gemini, or Copilot) with the question. Students should:
    • Use identical prompts with each model.
    • Document the full output from each model.
    • Note any significant differences between the models in terms of tone, emphasis, sources referenced , or perspectives presented.
    • Check the response for accuracy and hallucinations.
  3. Individually or in groups, students compare the different outputs. Some suggested points for comparison can be:
    • Content gaps: What information does one tool include that others omit?
    • Framing differences: How does each tool present the same information?
    • Source diversity: What types of sources or viewpoints are emphasised?
    • Cultural perspectives: Whose voices are centred or marginalised?
    • Assumptions embedded: What assumptions underlie each response?
  4. In groups or as a class, students discuss:
    • Potential reasons for observed differences (e.g. training data, corporate policies, geographic focus).
    • Implications of relying on single GenAI sources.
    • Strategies for mitigating bias in GenAI-assisted research.
  5. As a class, students compare key findings and consider practical recommendations for ethical and effective use of GenAI.

Considerations

  • GenAI models can be biased in the way they present information. Discuss this with students ahead of time, focusing on why these biases may exist and how GenAI users can detect them.
  • Different prompts may return different results, and the same is true across GenAI tools. Consider working with students to adjust prompts, try alternative tools, and compare outputs systematically.