Kayode Victor Amusan
University of Louisiana at Lafayette
The assignment explores the writing pattern of text generation technologies (such as ChatGPT) and how they vary from human writing. It compares the writing outcomes of undergraduate students and Gen AI. The task charges the first-writing-students to write a short story about ‘love’ and compares their outcomes with that of Gen AI using same prompt. This attempt is a forensic stylistic approach meant to establish the storytelling pattern that is unique to text generation technologies and how they vary from human writing.
Learning Goals
Original Assignment Context: Mid-semester assignment for first-year undergraduate writing students in ENGLISH 101 class
Materials Needed: Accessibility of text generators (such as ChatGPT) for students to use
Time Frame: A period of six class lecture hours (two weeks)
Overview
This paper investigates variations in the storytelling output of LLMs and humans as they respond to the same prompt. The study discovered that most of the outcomes from ChatGPT-3 on the storytelling were following some sorts of underlying patterns which are identifiable in the structure, plot, semantics, vocabulary, and pragmatics of each story. On the other hand, the outcomes of the students’ writings differ in these categories. This demonstrates that human ideas are often driven by creativity, imagination, and the ability to produce truly, unique, and original ideas. It shows that humans can think beyond patterns to generate new concepts. The study demonsrtaes that Gen AI, although can be creative, relies on patterns and data it has been trained on. It can only succeed in tasks that require producing content within specific frameworks but may struggle to create or produce completely new concepts. This underpins the arbitrariness of human language. In teaching writing, the implication of this study engenders the need to encourage and improve students' creative ability, by assisting them move beyond patterns to generate genuinely distinctive and innovative writing content.
This experiment investigates the variations in the output of Gen AI and human writing. The study collected data from students (human) and ChatGPT-3. Ten different chatbots were used. Each chatbot was given the same prompt: “In not more than 10 sentences, write a story about love”. For humans, twenty (20) writing samples were collected from twenty (20) first year writing students who were offering ENGL 101 using the same prompt. Each sample was analyzed, interpreted, and compared based on the similarities and variations in the outputs. This was done in the classroom.
One significant variation in the outputs of Gen AI and the students is the lexical and grammatical choices. For ChatGPT, most of the lexical items and vocabulary that featured in each sample are retrievable in the other samples within the same context. Every sample of the data contained lexical items suggestive of conversation, dreams, passions, and bonds. And these vocabularies are used within the same context of “a man and a woman having to share their thoughts and feelings to bond well”. The following parallel structures and form are drawn from the outcomes of Gen AI “in the bustling streets of a vibrant city”, “in a bustling city”, “in the heart of a bustling metropolis”, “in a bustling city”, “in a quaint coastal town”. This reality is also demonstrated in a similar research conducted by Bugus (2023). In fact, the word ‘bustling’ was used 8 times (out of 10) within the same context in all the data samples from chatbots and also replicated in Bugus’s research. On the other hand, each output produced by the students was different, as none was traceable to one another. The grammatical and meaning construction in each sample is mutually exclusive. The only similarity they had was the usual construction of “once upon a time’, and ‘there was/lived” which has been a general formula for storytelling since time immemorial. This finding demonstrates that human writing is entirely unique and independent, as humans are exclusively creative in their thoughts. Gen AI, based on this finding, tends to be obeying some underlying rules or pattern probably set by the large language set which they are trained on. Since they generate responses based on patterns in their training data, its output will be influenced by the input it receives. Humans have a deep understanding of context, nuances, and storytelling techniques that are hard to replicate by AI. This study also demonstrates that humans can adapt to specific requests more effectively and create engaging narratives.
Variation in the narrative point of view was also noticeable in the students’ outputs, there were instances of mixed points of view where the 3rd person and 1st person were projected. More importantly, unlike outputs from Chatbots, human writing exemplifies gender balancing in storytelling. This is because all the Chatbot outputs introduced women through the conversational lens or relations with men. This type of (feminist) bias is constantly addressed by feminist advocates. On the other hand, the students’ outputs were more diverse in gender representation. For instance, while 9 students introduced women from the conversational lens of men, 4 introduced men from the lens of women and 7 introduced both genders simultaneously. The students’ output also revealed the possibility of homosexual love entanglement in a story as depicted in a student’s writing. This shows that human writing is sensitive to socio-cultural context, societal concerns and most especially diversity. This corroborates Begus’s (2023, p.8) caim that “GPT-generated stories, particularly those generated by GPT-3.5, are thematically homogeneous to an extent that they hardly differed from each other”. Although chatbots can produce impressive results, they don’t have the same depth of understanding, consciousness, and creativity as humans. Humans have the ability to iterate and refine their reactions based on feedback and their perceived sentiments, socio-political context, and their creativity, which can lead to more accurate and concise storytelling capability. While AI like ChatGPT can provide significant support and generate creative content, human storytelling capacities, understanding of context, and flexibility make them better appropriate for constructing highly accurate and succinct narratives in some cases.
The outputs of the respondents in the study based on the plots and themes of individual narration are interesting. From series of observation, it is obvious that the plot development of the stories generated by chatbots followed a specific pattern and similar themes (e.g. that ‘A’ was alone before he finds ‘B’, they fell in love after a deep conversation; they went on adventures; they were faced with hurdles but survived them, this strengthens their love). The students’ outputs were far from following any specific pattern. While some involve a situation where a village community is conducting a search for a suitor for their Prince or Princess, others involve instantaneous encounter with a partner (with or without conversation). In order words, while the features of ‘conversation’, ‘adventure’ and ‘challenges’ were predominantly common in all the chatbots’ outputs (with a patterned sequence, the only central theme common to the students’ outputs is ‘love’. In fact, none of the students’ outputs discussed ‘adventure’ and ‘challenges’. This goes further to strengthen the fact that chatbots are basically responding to the stimulus of a specific pattern upon which they were trained. This makes their themes, point of view, lexis, vocabulary, context and structures rigid and predictable. However, AI can still be a beneficial tool for producing ideas or inspiration, but it may require some human involvement to achieve the desired level of accuracy and conciseness. The implication for students is that while AI can be a worthy tool for generating ideas or substance, it is not a substitute for the originality, creativity, innovation, diversity, and adaptability of human writers. Students are urged to explore their creativity and create their unique voices, to allow them produce writing that transcends the constraints of AI-generated intelligence and exposes their individual perceptions and storytelling skills. AI can complement the writing process but should not be trusted as the main source of inspiration and storytelling in their work.
It is interesting to note that this study shares similar features with Bugus’s (2023) experiment since both studies compared human-generated narratives and those created by Gen AI, using ChatGPT. Nina Begus's study approaches the analysis the data using a framework of behavioral and computational experiments. This study investigates the outcomes of Gen AI in love storytelling, emphasizing patterns, creativity, and originality. Both studies share a common ground in examining narrative elements, cultural influences, social biases, and the pplexceptional creative capability of human storytelling prowess. Both studies present a deep understanding of the limitations and biases of both AI-generated and human-authored narratives.
My greatest inspiration for this assignment is the curiosity of my first-year-writing students to identify the unique features of texts generation technologies; and of course, I want to acknowledge Prof Ratliff Clancy who constantly exposed the strengths and weaknesses of Large Language Models. Chat GPT-3.5 is also a factor that contributes to the success of this assignment, hence, thanks to OpenAI. Finally, Nina Begus’s (2023) experiment “Experimental Narratives: A Comparison of Human Crowdsourced Storytelling and AI Storytelling” which compares Human Crowdsourced Storytelling and AI Storytelling remains a foundation for the experiment.