Critical Media Analysis Project (CMAP)

Whitney Gegg-Harrison
University of Rochester

This assignment asks undergraduate students to critically annotate an existing popular media item using Kapoor & Narayanan’s “18 Pitfalls” framework. This framework helps students to reflect critically on media hype relating to AI, and gives them language to assist them in identifying, labeling, and explaining examples of hype to potential readers. In creating their annotated articles, students gain practice applying their critical AI and media literacy skills, which are skills they can apply to their own lives as they engage with media about generative AI and strive to be thoughtful, critical users (or not) of generative AI products.

Learning Goals

  • Interrogate potential sources of and reasons for hype and biases in popular media writing about AI 
  • Apply concepts from Critical AI Literacy and engage with existing scholarship on AI
  • Annotate and analyze the biases in a popular media article using an existing framework
  • Explain technical concepts to a non-academic audience

Original Assignment Context

This assignment is the first of two “major” assignments in a newly-created upper-level writing class called “Writing about and with Artificial Intelligence.” The class was cross-listed between the Writing, Speaking, and Argument Program, the English Department, and the Digital Media Studies Program and drew students from a variety of majors. The assignment was supported by a set of readings and class discussions centered around Critical AI Literacy, which many of the students used as sources for their CMAP.

Materials Needed

  1. Kapoor & Narayanan's "18 pitfalls" piece
  2. A media item of each student's choosing (potentially transcribed, if the media item is audiovisual)
  3. Additional readings/resources on Critical AI Literacy to support students' analyses.
  4. A Google Doc template for the annotations; interested students who are comfortable with creating web pages from html/CSS templates can also be encouraged to try using Molly White’s annotations.

Time Frame

~ 4 weeks: one full (75 minute) class period to introduce the assignment & go over the 18 pitfalls; one full class period devoted to in-class work time on the assignment; one full class period devoted to peer response; additional outside-of-class work time for the students.

Overview

The Critical Media Analysis Project (CMAP) was the first of two major assignments in the first offering of my “Writing about and with Artificial Intelligence” class. The class was deliberately designed to front-load discussions around Critical AI Literacy; we began by reading (and writing) about AI from a critical perspective so that students could make an informed decision about when, whether, and how they might write with AI later in the semester. Early in the class, students were introduced to Kapoor & Narayanan’s “18 Pitfalls to Beware of in AI Journalism” and explored some of Kapoor & Narayanan’s “Critical Annotations” of articles from CNN and the NYT. 

The CMAP assignment asks students to apply the “18 pitfalls” framework to a media item of their choice, which they highlight and annotate in the style of those “Critical Annotations”; in addition, students write an intro and outro in which they motivate, contextualize, and offer a clear “take-home message” for readers of the “AI Snake Oil” website. Students went through multiple rounds of peer response and instructor feedback as they honed their presentation of their critiques for the final portfolio version of the CMAP.

Students found the project challenging, but enjoyed practicing their critical analysis skills and produced excellent critical annotations. Several students shared that in the process of working on their CMAP, they discussed issues relating to AI hype with roommates and friends, and were very happy to be able to dispel misguided beliefs about AI with the information they learned from their analysis.

Assignment

Critical Media Analysis Project (writing *about* AI)

AI policy: NO AI for first draft. AI-assistance OK for revision and editing. You’ll document your use (or not) of generative AI tools for revision and editing as part of your Process Notebook.

In class 3, we discussed the concept of “Critical AI Literacy”, and introduced a framework of 18 common “pitfalls” that journalists fall into when writing *about* AI. For this assignment, I want you to imagine that you’ve been asked to create an analysis for “AI Snake Oil”. You will choose one “hype”-filled  article, video, or other form of media to analyze using their framework, and will create an annotated version of the article/video/etc in which you contextualize your analysis and make an argument about what your analysis tells us about AI literacy.

What you need to do

  1. First, revisit the “18 Pitfalls” framework that we discussed in Class 3:  
    • Remember that you can read the entire article on Substack without subscribing by just clicking “Continue Reading” when the pop-up appears.
  2. Then, choose a media item to focus on. This can come from anywhere. I tend to see the most “hype” in articles from the Edtech and Business domains, but even “mainstream” news like the Washington Post, the New York Times, CNN, etc have their share of AI hype to critique.
  3. Then, create an “annotated” version of the media item using color-coded highlights as Kapoor and Narayanan do (e.g. in this example from CNN, and in this example from the NYT). If you’re comfortable with creating web pages from html/CSS templates, you can use Molly White’s annotations, and if not, you’re welcome to use my Google Docs template. If you’re using a non-text type of media (like a video or an audio podcast), my suggestion is to work with the transcript for the color-coded highlights.
  4. Your goal in the intro and outro is to contextualize your analysis (e.g. relate it to the framework that Kapoor and Narayanan provide, and to other sources we’ve already read) and offer us a “take home message” about what your analysis shows us.

Requirements

  • Length: Aim for ~1000-1500 words across your annotations and introductory and outro text
  • Format: Use a readable font and a reasonable font size - otherwise, formatting choices are yours to make!
  • Citation style: Use whichever style feels most comfortable for you (e.g. APA, MLA, IEEE, hyperlinked URLS) - just make sure that you are consistent!
  • Sources: You must engage with at least 3 sources in this paper: the Kapoor & Narayanan “18 Pitfalls” piece, your chosen media example, and at least one other source from our course readings.

Audience

Your audience consists of both regular readers of “AI Snake Oil” and of interested people who are encountering the site for the very first time via your article. This creates a tricky balance: you don’t want to bore the regular readers by rehashing things they already know in excruciating detail, but you are going to need to provide some context for those readers who are encountering this site for the first time. You also can’t necessarily assume that your readers have read anything from our course readings (but of course, they may have!).

Why I’m having you do this

This assignment is aimed at honing your critical media analysis skills and preparing you to be a more thoughtful consumer of media relating to generative AI. You’ll practice analyzing a piece of media using an existing framework and writing about a technical topic for a non-academic audience. These skills are ones you’ll be able to apply to your own life as you engage with media about generative AI and strive to be a thoughtful, critical user (or not) of generative AI products.

Assessment criteria

These are what I will use when evaluating your final draft as part of the Final Portfolio. Please feel free to make a copy and use as a checklist!

Argument & Analysis

  • The introduction clearly motivates the analysis that follows
  • The implications for the analysis (why does it matter?) are discussed explicitly in the intro and outro
  • The annotations are supported using well-chosen evidence and good reasoning
  • There is a clear “take home message” in the outro

Engagement with Sources and Evidence

  • You engage with at least 3 sources in the intro, annotations, and/or outro:
    • The Kapoor & Narayana “18 Pitfalls” piece
    • The media item you’ve chosen to analyze
    • At least one other source from our course readings
  • Examples from your chosen media item are clearly highlighted with colors corresponding to your key and annotated with comments sharing your analysis 
  • The examples are analyzed in relation to both the “18 pitfalls” framework and at least one other reading we’ve done this semester
  • Each source is paraphrased, quoted, and/or summarized without plagiarism. Use the checklist below to be sure:
    • Each source is introduced with a clear Signal & Pathway
      • My annotations/intro/outro have an appropriate in-text citation/signal for every source that is used, at the place of use
      • My annotations/intro/outro use quotation marks wherever I am using the exact words of a source
      • My project has a references section in the outro that has an entry for every source that was cited in the text
    • Double-check: Do my in-text citations provide a clear “pathway” to the reference entry? Do my reference entries provide a clear “pathway” to the actual source?

Organization & Coherence

  • The intro and outro are well-organized and coherent, with clear connections between ideas
  • Each annotation is focused, coherent, and clearly relevant to the analysis
  • There is a clear connection between the annotations, such that taken together, they build toward the “take-home message” in the outro.

Presentation & Style

  • The critical analysis has an interesting, accurate, and genre-appropriate title
  • The writing in the intro, outro, and annotations is appropriate for the genre. 
    • If choices deviate from genre conventions, is it intentional, for a clear purpose?
  • There is effective, thoughtful use of vocabulary, grammar, and stylistic options.

Process

  • The process of drafting, revising, and editing the project is fully and accurately documented in the relevant Process Notebook entries.

Acknowledgements

Thanks to Kapoor & Narayanan for their "18 Pitfalls to Beware of in AI Journalism"