Huiling Ding
North Carolina State University
Generative AI and large language models such as GPT-3.5 introduce new tools and challenges to writing classrooms. This assignment aims to both introduce students to these new tools and to help them cultivate writing, research, editing, collaboration, and critical thinking skills. Using ChatGPT as an example, it helps students to understand important concepts such as natural language processing, LLMs, and AI ethics. The assignment contains six steps: generating a prompt; collecting responses from ChatGPT consecutively; analyzing, editing, and summarizing responses; and developing an original essay after conducting library research on the same topic. It helps students cultivate new skills in prompt engineering while challenging them to critically engage with AI-generated content through summary, synthesis, editing, as well as rhetorical and structural analysis.
Learning Goals:
Original Assignment Context: team project in a dual-enrollment class on Responsible AI with undergraduate and graduate students from a wide variety of disciplines
Materials Needed: An accessible AI text generation program (i.e. ChatGPT)
Time Frame: ~2-3 weeks
Introduction
This assignment was first pilot tested as an individual project in Spring 2023 before I reiterated as a team project in a dual-enrollment class, Responsible AI, in Fall 2023 with undergraduate and graduate students from a wide variety of disciplines. I piloted this project to guide one of my advisees to work on a research project on the recruiting and retention of high-quality employees, which was smaller in scale but faster paced. We went through the generated texts and the way they got incorporated in the writing process in depth together. In this pilot test, we used ChatGPT as a tool of brainstorming, research, and idea generation at the beginning to understand the topic. The student started with a simple question asking how employers can recruit and retain high-quality employees and used ChatGPT to regenerate five different responses. She analyzed the structure of the responses and concluded that they were mostly based on the typical five-paragraph essays taught in academic writing classrooms.
After she shared the responses with me, I suggested that she include a request for references in the next round of chat, which she did. Then I asked her to check the references in terms of relevance and accuracy to make sure the content matches up with the summaries of ideas in each response. She did some quick Google searches and found out that none of the references actually exist. At this point, I suggested that she summarize the key points in each response in terms of HR strategies to recruit and retain employees and then resort to library searches to find reliable and relevant sources that may touch upon some of those strategies. She came up with over ten strategies, did extensive library searches, limited her sources to peer reviewed journal articles, and collected over 20 publications from top tier journals such as Harvard Business Review and produced a solid research paper on the topic.
In Fall 2023, I started the assignment with a quick introduction and demonstration of ChatGPT before explaining to students what large language models are, the technologies that enabled them to function, and the strengths and weaknesses of ChatGPT as a content generation tool. We also talked briefly about natural language processing and natural language generating processes as well as issues about authorship, creativity, and ethics to understand the contexts surrounding automated writing technologies (Floridi & Chiriatti, 2020; Duin & Pedersen, 2021; Zaretsky, 2023). Then, students worked in teams of 3-4 to experiment with ChatGPT. Considering my students were upperclassmen or graduate students, I focused more on possible ways to use LLM tools such as ChatGPT as an AI assistant to speed up the research and brainstorming processes than plagiarism prevention only.
How does AI produce texts in responses to human prompts? What writing strategies are used and how can human writers leverage such written products in producing original work? As a part of a weekly module on automated writing technologies, this assignment will use a group project to ask students to engage with ChatGPT to generate and analyze prompt-driven content before experimenting with automated content to produce original writing for a larger research report.
Students will work in groups of 3-4 to create prompts on topics of interest to them and relevant to the class. They will collect responses using a single ChatGPT chat session to avoid replicated responses, save the responses, perform genre and content analysis, write summaries and synthesis of individual responses, before producing an original paper on the same topic using rigorous library research and updated references.
Interested instructors can also adapt this assignment as an individual project to support hands-on exploration after students acquire skills in library research, summary and synthesis writing, and rhetorical and genre analysis.
Learning Goals or Outcomes
Format and Length Requirements
This assignment involves six steps, with the first three mandatory and the last three optional. Instructors can freely combine and use these steps to meet their unique needs depending on classroom settings and availability of time.
Step 1. Prompt generation.
Student teams work together to generate a carefully crafted prompt to engage with ChatGPT. Your prompt can be informative, persuasive, argumentative, or critical. Make sure the prompt is narrow enough and relevant to the class content. Revise the wording a few times to make sure the entire team reaches agreement about the scope and content of the prompt. Doing this will help students practice prompt engineering skills, which can be compared by the instructor with the process of writing a narrow and well-defined research question.
Step 2. Collect consecutive responses from ChatGPT using one chat.
Log onto ChatGPT using one of the team members’ accounts. Work synchronously as a team to consecutively collect one written response from ChatGPT using the same chat session. You can enter the same question or use the “regenerate response” function to get different responses. Doing this will allow the team members to each get a unique response with as little overlapping content as possible. Copy and paste each response to a shared Google doc to make sure all team members can have access to the compiled responses.
Step 3. Critical analysis of responses.
Analyze the written product rhetorically, structurally, and stylistically. What common pattern do you observe as a group based on the 3-4 responses you collect? How is the product similar or different from academic writing conventions you have learned so far? What strengths and weaknesses do you observe in the responses?
Step 4. Edit machine-generated writing.
Make edits and changes when needed to improve its quality in Google Doc using track changes. After your editing session, compare with your teammates to see what kind of changes you make as a team. What patterns do you observe in your team editing efforts? What do you learn about machine-generated content?
Step 5. Summarize individual responses.
Individually generate a 50-word summary of the automated response you collect. Compare notes to see if you see overlapping content. Check the sources for ideas in your response to see if you have the correct in-text citation or references.
Step 6.
Depending on the nature of your prompt and the responses you collect, write a 600-word original essay on the topic included in your prompt by drawing insights from the automated responses to your prompt as a group. Write your essay with rigorous research. Limit your references to those published in the last five years, with at least 25% of your resources published in the last twelve months. Practice rigorous documentation, use direct quotes regularly, keep track of all your sources, and share your sources along with the final essay as well as a 100-word reflection essay on the challenges you encounter when using ChatGPT as a brainstorming and research tool.
Rationale: Currently, the training data of ChatGPT cuts off in 2021 and ChatGPT has no access to materials published after 2021 (OpenAI, 2023, Educators). Limited to its training data for now without external capabilities, it cannot access the Internet or keep track of its references. In fact, ChatGPT is now known for its tendency to make things up and produce fake sources at times. Step 6 helps students understand the limitations of similar AI tools and strategies to conduct rigorous research without relying on AI tools. If things change in the future and if students work with LLMs with online search capacities, Step 6 should be revised to make sure research can be an integral part of this project.
Preparation, Materials, and Skills Necessary to Complete
This assignment can be a stand-alone project or a project that is based on readings on AI, AI-generated content, and writing with emerging technologies. Instructors can start with a quick overview of AI and machine learning, AI ethics, and large language models (Tamkin & Ganguli, 2021; Tamkin et al., 2021). Students will need access to ChatGPT by creating their personal accounts, which can be done outside of class. They should also feel comfortable with collaborating online and sharing their writing and thoughts via Google docs. To accomplish the tasks, students should have solid skills in conducting online and library-based research, doing rhetorical and stylistic analysis, as well as writing summaries, synthesis, and research-based academic essays.
For instructors working with freshman composition classes, it is helpful to work with librarians to first teach students online and library-based research before having them experimenting with AI-assisted writing. Meanwhile, an early discussion about and clear policies on plagiarism and plagiarism detection tools can play important roles in educating students about the benefits and perils of using such AI tools in undergraduate writing classrooms (Bastian, 2023; Schwartz, 2023; Whitney, 2023).
References
Bastian, M. (2023, February 13). ChatGPT is “high-tech plagiarism” for lazy learners, says Noam Chomsky. THE DECODER. https://the-decoder.com/chatgpt-is-high-tech-plagiarism-for-lazy-learners-says-noam-chomsky/
Cooper, K. (November 1, 2021). OpenAI GPT-3: Everything You Need to Know. Retrieved January 19, 2023, from https://www.springboard.com/blog/data-science/machine-learning-gpt-3-open-ai/
Cozza, A. (2022, September 16). AI Art—What is it and why are artists upset about it? Aimee Cozza Illustration. Retrieved January 19, 2023, from https://www.aimeecozza.com/ai-art-what-is-it/
Heikkilä, M. (2022). This artist is dominating AI-generated art. And he’s not happy about it. Retrieved February 12, 2023, from MIT Technology Review. https://www.technologyreview.com/2022/09/16/1059598/this-artist-is-dominating-ai-generated-art-and-hes-not-happy-about-it/
Duin, A. H., & Pedersen, I. (2021). Writing Futures: Collaborative, Algorithmic, Autonomous (Vol. 969). Springer International Publishing. https://doi.org/10.1007/978-3-030-70928-0
Floridi, L., & Chiriatti, M. (2020). GPT-3: Its Nature, Scope, Limits, and Consequences. Minds and Machines. 30, 681–694
Marche, S. (2022, December 6). The College Essay Is Dead. The Atlantic. Retrieved February 16, 2023, from https://www.theatlantic.com/technology/archive/2022/12/chatgpt-ai-writing-college-student-essays/672371/
McKinsey. (January 19, 2023). What is generative AI? Retrieved January 26, 2023, from https://www.mckinsey.com/featured-insights/mckinsey-explainers/what-is-generative-ai
Muscanell, N., & Robert, J. (2023, February 14). EDUCAUSE QuickPoll Results: Did ChatGPT Write This Report? Retrieved February 16, 2023, from https://er.educause.edu/articles/2023/2/educause-quickpoll-results-did-chatgpt-write-this-report
Noam Chomsky on ChatGPT: It’s “Basically High-Tech Plagiarism” and “a Way of Avoiding Learning” | Open Culture. (2023, February 10). Retrieved February 16, 2023, from https://www.openculture.com/2023/02/noam-chomsky-on-chatgpt.html
Practical Responses to ChatGPT and Other Generative AI. (n.d.). Retrieved February 16, 2023, from https://www.montclair.edu/faculty-excellence/practical-responses-to-chat-gpt/
OpenAI. (2023). AI Text Classifier. https://platform.openai.com/ai-text-classifier
OpenAI. (2023). Educator considerations for ChatGPT. Retrieved February 12, 2023, from https://platform.openai.com/docs/chatgpt-education
Schwartz, E. H. (2023, February 9). ChatGPT is Banned by These Colleges and Universities. Voicebot.Ai. https://voicebot.ai/2023/02/09/chatgpt-is-banned-by-these-colleges-and-universities/
Tamkin, A., & Ganguli, D. (2021). How Large Language Models Will Transform Science, Society, and AI. Retrieved January 19, 2023, from https://hai.stanford.edu/news/how-large-language-models-will-transform-science-society-and-ai
Tamkin, A., Brundage, M., Clark, J., & Ganguli, D. (2021). Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models (arXiv:2102.02503). arXiv. https://doi.org/10.48550/arXiv.2102.02503
Whitney, L. (2023, March 27). How to use GPTZero to check for AI-generated text. TechRepublic. https://www.techrepublic.com/article/how-to-use-gptzero-check-ai-generated-text/
Zaretsky, R. (2023, January 12). Words, Words, Words: What does the advent of ChatGPT mean for already beleaguered teachers? The American Scholar. https://theamericanscholar.org/words-words-words/
Appendix. Top ten strategies to recruit and retain high-quality employees