Douglas Eyman
George Mason University
This assignment asks students to research a wide range of text analysis and summarization tools and carry out an assessment task to gauge how well these tools can summarize technical documents. The students write a comparison report, identifying the most successful of such tools in terms of accurate summarization and output style. Finally, they write a reflection about how they see themselves potentially using these tools in technical communication work contexts.
Learning Goals:
Original Assignment Context: graduate course on technical communication
Materials Needed: Free trial accounts on 3+ AI-Based summarization tools
Time Frame: ~2-3 weeks
Introduction
This assignment was developed for a graduate course on technical communication for students in an MA Concentration in Professional and Technical Writing and could be easily adapted for undergraduate courses.
Most of the students in our MA program are working professional writers, and I had been hearing about how more writing tools were becoming available in their workplaces, particularly from proposal writers for federal government contractors and for large tech companies like Google and Amazon. The most widely used AI-like tool was Grammarly, which is unsurprising given it claims more than 30 million users. I sought information about whether these AI-based writing tools were being used in technical writing, but at the time of this writing have been unable to find data on the topic beyond anecdotes and marketing materials from the companies that provide the applications themselves. Still, it seems likely that the usefulness of these tools would lead to widespread adoption and employers will seek potential employees who understand these systems and can use them productively in technical writing roles.
I had been investigating the rise of text generation tools and seen demonstrations of several earlier versions of text generation applications using GPT-2 and GPT-3 (before ChatGPT's premiere prompted an explosion of concern) and even at that point it was clear that these technologies were going to impact the work of writers at all levels and in all contexts. One of the goals for this course is to apprise students of current technologies and tools that they are likely to use in their work, so adding these AI tools to the course seemed particularly appropriate (and, of course, now the need is even more pressing as the tools' capacities grow with every new release). When I decided to add AI-based text generation as a topic in my technical writing courses, I decided that it was important not just to discuss, review, and analyze output, but to actually use and assess the available tools from the perspective of a working technical writer. I developed the assignment below for my English 613: Technical Communication course, which I saw as a good fit, as that course focuses on core technical writing competencies applied in more technical contexts. In this course, students write API documentation, learn about Darwin Information Typing Architecture (aka DITA, a simple XML-based markup system that provides structure for particular document genres) and its uses in documentation, respond to a technical Request for Information (RFI), and summarize a technical document for a non-technical audience. At the time, the available text generation tools weren't particularly good at generating high-quality content, so the summarization assignment seemed the best place to engage text generators for technical writing. In future iterations of the course I plan to add additional AI-based assignments, but I suspect that this kind of summarization will become a standard use in technical and professional writing contexts in any case.
I asked students to research a wide range of text generators that offer text summary as a key service (Quillbot, Jasper, TLDR-this, Paraphraser.io, etc.). They signed up for the trial of at least three applications and then fed each one all or a portion of a technical document we'd been working with all semester (a technical white paper on "fog computing," which is a system that operates between local systems and cloud computing). Because these free trial versions were often limited in scope, we worked together to identify smaller sections of the document that were sufficiently technical in content but short enough to plug in to the applications—this part of the assignment also allowed us to become familiar enough with the content that we were well-positioned to assess the results produced by the AI services. For instance, some systems used the main document headings to produce a summary, while others drew from text provided in later paragraphs or each section; because we were familiar with the material, we could quickly see which variation provided a more accurate summary. The students analyzed the output produced by each system and compared across the three options they selected. They were asked to write both a report highlighting which application provided the most readable and accurate summary and a follow-up reflection about how they could see themselves using these tools in their own technical writing jobs in the future.
My students quickly identified the limitations of the applications they used and were able to determine that some systems were clearly more effective than others. Some of the systems produced garbled or nonsensical responses, even after students refined their prompts. They also discovered that they had to do a non-significant amount of training (for those few systems that allowed an iterative process of inputting text and evaluating the responses until the desired outcome was produced) and preparation of the system to get the result they wanted; they were unanimous in their declarations that for shorter texts, it would be far more efficient to just have the writer produce the summary (although they allowed that if they could have a 200-page document’s executive summary produced by an AI tool, that would be an appropriate use of time and resources).
In their reports, students described challenges with using the various user interfaces for the tools as well as assessing the outputs. They found that some tools couldn’t parse bulleted lists, and that others produced summaries with grammatical errors or with no punctuation. One student used OpenAI’s playground (fairly close to ChatGPT) with a preset command of "summarize this for a second-grade student," which led me to realize I hadn’t specified the target audience in the original assignment. All the students also commented on the speed of each application, so I’ve added that metric to the data to be recorded for the report. One student also ran the results through a readability score tool and found that none of the applications she used decreased the complexity of the text; this is another data point that could be collected for the report. In addition to the report, students produced a reflection about how they might use AI systems in the future. They saw a potential to assist with some routine tasks, but also noted that the text generators' inability to truly know the intended audience and to make sound rhetorical choices made them incapable of replacing human technical writers.
I plan to run this assignment in future iterations of this technical writing class, albeit with some additional scaffolding and more time investigating the available options and commentary on the affordances and constraints of AI-based applications in general before pivoting to the summary assignment (I’ve added some of this additional scaffolding in the version of the assignment presented here). A final note: I reference the specific technical document we had been using throughout the semester for a series of assignments in the class, but any technical document should work well for the purposes of this assignment.
Goals and Outcomes
This assignment follows an earlier assignment focused on developing students’ summarization skills; one of the goals is to reinforce the prior learning outcomes related to summarizing technical information for non-technical audiences.
The key goals are to expose students to AI summarization tools, provide opportunities for hands-on experience using these tools, and demonstrate the affordances and constraints of these tools when used in a technical writing context. The assignment also requires students to engage in a formal research process, which provides context for the formal research report and the final memorandum (thus providing additional practice generating technical writing genres).
Materials Needed
Students will need Internet access to sign up for the free trials of the applications targeted in the assignment, but no other specialized software or hardware is needed. Using free trials does limit the time available to use the tools (and often also includes length limitations), but for purposes of evaluation, this approach mirrors the process of making an informed decision about which tool a writer (or company) should invest in. Although this assignment was used in a graduate technical writing course, it required no particular expertise or knowledge base and should work as well with undergraduate technical writing students, or even composition students (who would summarize academic rather than workplace texts).
AI-Based Summarization Tools
This is a representative (but not exhaustive) list of summarizing tools available in 2023:
Acknowledgements
At the Computers and Writing conference in 2022, I attended two presentations on AI that demonstrated what it could do, and how it could be used by writers. Both presentations were led by Alan Knowles, who showed examples of how he had used AI tools provided by huggingface.co in his writing classes. Even before ChatGPT came out, the power of these tools to perform writing tasks was very impressive, but prior to these presentations, I had seen very little in the way of research or commentary in the fields of writing studies.
After the presentations I found several works that had been published in the past few years (although many were focused on the challenges and implications of AI rather than their practical and pedagogical uses). Perhaps most relevant for my interest in developing a text generation assignment for a technical writing class were a series of reports published in the proceedings of the 2022 ProComm conference, many of which focused on collaborations between human writers and AI writing tools (Duin et al., 2022; Knowles, 2022; McKee & Porter, 2022). These works helped me to see how I could frame the assignment in the larger contexts of technical communication roles and tasks. I also want to thank Alan Knowles and Kyle Booten for the comments and suggestions they provided in the initial peer-review process.
References
Duin, A. H., McKee, H., Knowles, A., Pedersen, I., & Porter, J. (2022, July). Human-AI Teaming: Cases and Considerations for Professional Communicators. In 2022 IEEE International Professional Communication Conference (ProComm) (pp. 201-202). IEEE.
Knowles, A. (2022, May 20). Dataset cultivation: A role for rhetoricians in the training of ethical AI. Conference Presentation, 2022 Computers and Writing Conference, Greenville, NC.
Knowles, A. (2022, July). Human-AI Collaborative Writing: Sharing the Rhetorical Task Load. In 2022 IEEE International Professional Communication Conference (ProComm) (pp. 257-261). IEEE.
McKee, H., Knowles, A., & Riggs, N. (2022, May 21). Fostering critical engagement with AI technologies: Examining human-machine writing, pedagogy, and ethical frameworks. Conference Presentation, 2022 Computers and Writing Conference, Greenville, NC.
McKee, H. A., & Porter, J. (2022, July). Team Roles & Rhetorical Intelligence in Human-Machine Writing. In 2022 IEEE International Professional Communication Conference (ProComm) (pp. 384-391). IEEE.
OpenFog Consortium. (2017, February). OpenFog Reference Architecture for Fog Computing. Retrieved from https://www.iiconsortium.org/pdf/OpenFog_Reference_Architecture_2_09_17.pdf
For this assignment, you will be producing two deliverables: a report comparing the quality and usefulness of summaries generated by text generator applications and a reflection memo where you will consider what you’ve learned from the process and how you imagine these tools may be useful (or not) in the technical writing workplace.
As a source document, we’ll use the same document from the prior assignments on technical summarization, the OpenFog Reference Architecture for Cloud Computing.
Learning Outcomes
After completing this assignment, students will be able to evaluate text-generation tools and assess their usefulness in completing technical writing tasks such as summarizing technical information and drafting executive summaries of longer documents.
Task 1: Survey the Available Tools
Your first task is to do an environmental scan—that is, determine the number and quality of AI-based tools that offer summarization as one of their key features. There are many of these tools currently on the market, so your task is to look for ones that appear to have strong user bases, good references, or good reviews outside of their own marketing materials. Some of the tools specifically market themselves as useful for technical communication tasks, but the ones you select need not be specifically targeted to tech comm. You’ll find many examples of lists comparing the tools that are available (e.g., “Best AI-Based Summary Generators”), although most will focus more on marketing than technical writing. Read across several comparison lists, but also look for independent reviews in blogs and industry journals to identify a top-10 list of tools.
Task 2: Select 3 Tools; Begin Data Collection
Based on your research in Task 1, narrow your list down to the three tools that appear to be best suited to generating readable summaries from technical documents. Sign up for the free trial versions of the tools, but if you need to provide any form of payment up front, remove that tool from your list and select an alternative. Write down the list of limitations on the free trial version of the tool.
Record the name, general description provided by the vendor, pricing information, and URL of each tool. If there are any use cases presented as examples (especially cases similar to this assignment), note those and include a brief summary. Additionally, list whether any specific companies are listed as customers or clients.
Task 3: Generate Summaries
For each of the three options, select the summarization tool (if it is identified as a distinct application) and provide text from OpenFog Architecture. It is likely that you will not be able to use the full document, so you should first select one paragraph with sufficient technical detail to see how well the summary tool works (copy this paragraph into your data collection documents; you’ll need to include it in the report).
If there is an opportunity to input a specific audience, ask the tool to target a reader with an 8th-grade level of education and reading comprehension for the first run-through. For a second run-through, identify the audience as the CEO of a large multinational corporation (but leave out the specifics of what that corporation does).
Copy the summary of the paragraph generated by the tool and add it to your research data record.
Assess the result: did the tool provide an accurate summary? How well did it capture the main point of the original paragraph? Does the selection of elements to summarize appear to be an appropriate match for the specified audience? Did it add any extraneous information? Do you notice any errors or other problems with the output? Are there any other issues or features that you notice that aren’t covered in the questions above?
Run the summary through a readability assessment tool (you can use the readability information generated by Word, or use a free online tool such as “Free Readability Tools” at https://readabilityformulas.com/freetests/six-readability-formulas.php). Record the Flesch Reading Ease and Flesch-Kincaid Grade Level, which are standard metrics for readability (you can record additional measures as well, but only these two are required data points).
Now increase the amount of text you can put in up to the limit allowed in the trial version, copy the resulting summary, and assess the result following the same procedure outlined above.
For each task, make a note of how quickly the tool responds and presents the final output.
Finally, record the steps of the process and note any challenges with the tool’s user interface (how easy or difficult is it to perform this task?). Take a screenshot of the main UI for this task to include in your report.
Task 4: Compare Summaries and Assess Outputs
Once you have completed Task 3 for each of your three selected tools, it’s time to compare the results. Prepare a table that compares the features, effectiveness, ease-of-use, and speed of each tool. Write up a brief narrative account discussing the results in the table and then provide your assessment of which tool provides the best results overall. If there are tools that might provide better results across different use cases, make note of that as well.
Task 5: Write the Report
Take all the data you’ve collected and use it to write a report comparing the three tools you selected. Be sure to provide an executive summary and to use clear headings and document structure, as befits a technical report. You may choose to frame it as either an informative report, a white paper, or a recommendation report. Don’t forget that first point of contact for the reader: give your report a good descriptive title.
Task 6: Write the Reflection Memo
After you complete the formal report, you’ll write a reflection memo—this can be less formal than the report. The memo should address your experience of the process (what worked well? What didn’t? Were there any difficulties following the instructions as provided?) and what you learned from carrying out the research and writing the report. Finally, imagine yourself as a working professional or technical writer (or simply refer to your own experience if you already are one) and speculate about whether these tools would be useful (or not) and if so, how you imagine they might be used by technical writers specifically.
Assessment Criteria
For the first deliverable (the research report), I'll be looking for a report that follows the genre conventions we've covered in class: a clear title that reflects the purpose of the report, a brief executive summary of findings, a clear delineation of background information, methods, data, and analysis, followed by discussion and recommendations (using headings and other document design features as appropriate). Be sure to include references to the tools you used as well as providing citations (in text or footnote) for any external references. The report should clearly convey the data you've collected for each tool and also demonstrate the comparisons among them (feel free to use charts, graphs, or other visuals to assist in this task).
For the second deliverable, I'll be looking for a clear narrative of your process and demonstration of metacognitive reflection about what you've learned from this assignment. I'm particularly interested in your assessment of how these tools might be of use in your future workplace.