WAC Clearinghouse home page
About ATD   Submissions   Advanced Search

WAC and Assessment: Activities, Programs, and Insights at the Intersection

Data Driven Change Is Easy; Assessing and Maintaining It Is the Hard Part

Abstract: At MIT in the 1990's, data from two sources, a study of the writing ability of a small group of randomly selected MIT juniors correlated to their overall academic performance and a survey of alumni from various years provided the major motivation for the development by MIT faculty and administration of a very ambitious Communications-in-the-Disciplines Program. The study of random juniors demonstrated that student writing ability had no effect on overall student grade-point-average, thereby giving students no immediate incentive to work on improving their writing skills within the context of an extremely intensive MIT undergraduate curriculum. The alumni survey displayed a significant disparity between the importance alumni attached to communication and leadership skills and the alumni's low estimation of MIT's contribution to the development of these skills. Once the new curriculum was in place, however, assessing its effectiveness became much more complex. The end result was an assessment that, given all the cross currents, was successful primarily in raising consciousness and acceptance levels for integrating instruction and practice in writing and speaking throughout the undergraduate curriculum.

MIT has a long history of integrating writing instruction throughout its undergraduate curriculum, beginning with the efforts of Robert Grosvenor Valentine in 1897 (Brereton, 1995; Russell, 2002). However, strangely in keeping with MIT's long tradition in acoustics and electronics, the history of integrating writing into the academic curriculum for the first half of the twentieth century resembled not so much a straight line with a slope but a sine wave. The pattern has been that efforts to make writing an integral part of education in science and technology have been followed by periods in which writing was completely integrated into the humanities, particularly literature, and then a new WID initiative would arise.

Key to making what may be lasting change, as I detail here, is the influence of a special group of MIT stakeholders,the alumni, along with the ingenuity and efforts of an interdisciplinary group of committed faculty.

Context and History

The second half of the twentieth century ushered in a new pattern of change and need for more change. In 1949, a specially appointed MIT body, the Committee on Educational Survey (http://libraries.mit.edu/archives/mithistory/pdf/lewis.pdf), issued a report that in addition to establishing a separate School of Humanities and Social Science, explicitly recommended that "the development of the student's ability to communicate orally and in writing be emphasized" in courses both in the humanities and in science and engineering (Committee on Educational Survey, 1949, p.94). In the early 1950's, in response to the recommendation that instruction in writing be integrated into science and engineering classes, Robert R. Rathbone, a professor in the Humanities, began collaborating with engineering faculty to offer both lectures on technical communication and feedback on student technical reports. These informal arrangements soon evolved into the Undergraduate Technical Writing Cooperative (the Writing Coop), which provided a framework for collaboration between writing instructors and technical and scientific faculty upon which all subsequent WAC and CAC initiatives would build.

The 1960's and 1970's saw the emergence of a separate Writing Program at MIT, which included the Writing Coop. In the early 80's, because of widespread dissatisfaction with undergraduate writing among the MIT faculty, the faculty explored several proposals, including the re-instatement of mandatory first-year composition, before finally settling on the Writing Requirement, which might be viewed as an outcomes-based measure, but one that was in reality a test of minimum proficiency that cost MIT very little in additional resources. Students were required to demonstrate minimum proficiency in expository writing (Phase One) and proficiency in the discourse of their major (Phase Two), often in conjunction with instruction and evaluation by the Writing Coop.

Almost from its beginning in 1983, it became clear through anecdotal evidence that the Writing Requirement was ineffective in ensuring that students left MIT with proficient writing skills and of little use in encouraging minimally proficient writers to improve. Moreover, because there was no mechanism in place to ensure that students completed Phase One early in their academic careers, the weakest writers often procrastinated addressing their writing problems until that senior year. Furthermore, the Writing Requirement did not require any proficiency in oral communication. But because MIT is a largely a data-driven institution, without convincing data, it was difficult, if not impossible, to motivate faculty to restructure the curricula in their various disciplines and to convince the administration to allocate substantial resources to create a CID program that would provide substantial and sequential instruction in writing and speaking.

Two studies conducted in the 1990's, one of a small randomly selected group of students and the other, a survey of 881 MIT Alumni, provided much of the momentum for the creation of a new CID curriculum. Once the new curriculum was in place, however, efforts at assessment became much more complex and cumbersome. The end result was an assessment that, given all the cross currents, was successful primarily in raising consciousness and acceptance levels around MIT for the newly instituted comprehensive program of instruction in writing and speaking.

The Lightman Study

In 1995 a special faculty committee was charged by MIT's Committee on the Undergraduate Program to evaluate the effectiveness of MIT's then current Writing Requirement in teaching students to write well. Alan Lightman, a physicist, science writer, novelist, and then Head of the Writing Program as well as Co-Chair of the Committee, developed an elegant procedure for a direct measure of student writing ability. Rather than use an indirect dependant variable, such as grades, actual samples of student writing were assessed.

Forty juniors who were enrolled in humanities, arts, or social science classes that required substantial writing were randomly selected, and the instructors of these classes were asked to send an ungraded copy of the student's last paper. Thirty-two papers were collected and then read on a holistic scale by members of the Committee. Holistic scores were then conflated to place the essays in three categories: Good, Marginal, and Inadequate.

The scores were then tabulated against individual student profiles that included Overall Student Grade-Point-Average, SAT Verbal Score, score on the SAT II Writing Test, score on the Freshman Writing Placement Test, student's major, and number of classes taken that involved substantial writing. Working with MIT's IRB, a decision was made to not ask for informed consent. We needed a random and representative sample and were afraid, with some justification, that asking for informed consent would produce response bias. (Consequently, although I can discuss two findings that subsequently became public, I cannot disclose any other observations that were made from the data.)

One immediate conclusion was that a student's writing ability did not affect his or her overall grades at MIT. As shown in Figure 1 (below), the mean grade-point averages of the juniors in the three groups, those whose writing was judged acceptable, those whose writing was judged marginal, and those who writing was considered inadequate were virtually identical, all three mean GPA's differing by .05 grade point or less. One interesting conclusion stemming from this finding focused on the connection between grades and writing. The undergraduate curriculum at MIT is extremely intensive, and it is virtually impossible for most students to complete all assigned work and readings. Benson Snyder, who studied the MIT student culture, reported in The Hidden Curriculum (1973) that MIT students quickly learn to optimize the system by what he terms "selective neglect." We speculate that because writing ability had no apparent effect on grades, attention to writing was consequently often a prime candidate for neglect.

Figure 1: Mean Grade Point Average of Juniors

Figure 1. Mean Grade Point Average of Groups

Alumni Survey

While this first study provided data that showed at least some of the reason why students did not prioritize writing as an essential skill during their undergraduate career, the second data set—a 1996-97 survey of 881 alumni from the classes of 1992, 1987, 1982, 1977, and 1972—asked respondents to rate various abilities in terms of (1) how they are important to them currently and (2) MIT's contribution to their acquiring them. The data showed that while MIT prepared students well for the intellectual challenges they would face as engineers and scientists, the Institute did not prepare them to be effective communicators and leaders. MIT's alumni believed MIT prepared them well in skills associated with science and engineering, such as scientific knowledge, analytical and problem solving skills, and critical thinking. In contrast, the items in the survey that displayed significant disparities between reported importance of an ability and MIT's contribution to it all tended to cluster around leadership and communication. Although 70% of the alumni reported that "Leadership" was currently "essential" or "very important," only 27% reported that MIT's contribution was significant. Similarly, 68% reported that "influencing others" was currently "essential" or "very important," while only 18% stated that MIT's contribution was significant. Only 13% of the respondents ranked MIT's contribution to their skills in public speaking as significant, while 65% of these respondents reported that this ability was currently "essential" or "very important." And while 85% of these alumni reported that the ability to "write clearly and effectively" important, only 25% reported MIT's contribution was significant.

Figure 2 (below) graphically illustrates this disparity, and it was figures like this one more than any other data or arguments that motivated both the faculty and administration to commit to an ambitious WID program that would require students to take one communication-intensive class each year and fund these classes to provide a corps of Writing Program lecturers to collaborate with the subject instructors in teaching many of them.

Figure 2: 19 Abilities Listed in Descending Order of MIT's Contribution Meeting Current Importance in Life

Figure 2.

1996 Alumni Survey Questions: 19 items ranked by alumni as "very important" or "essential" in current life and MIT's significant contribution to their development ("very much" or "quite a bit"). The items are listed in descending order of how well alumni estimations of MIT's contribution matched their ranking of the ability's importance in their current life.

The qualitative evidence from the alumni survey corresponded to the quantitative results and was in many ways, even more compelling. For example, one member of the Class of 1992 explained the "cost" of a failure to communicate well in response to the survey:

The general problem solving skills that I learned at MIT have been very useful in analyzing a wide range of business problems. Upon graduation from MIT, I went to work in consulting, an industry dominated by "ivy league" type undergraduates. While my college education probably prepared me better for analyzing problems, my counterparts were better at presenting their ideas and working with others. I was kept available for behind the scenes analytically intensive assignments while others had more opportunity to meet with clients and become exposed to higher level issues.

The Communication-Intensive Curriculum

The effect of the survey on MIT's instructional practices was immediate and profound. Immediately, group projects became a regular part of the engineering curriculum and also appeared in some classes in science and engineering. Simultaneously, MIT's administration and faculty began the design of a new "Communication-Intensive" undergraduate curriculum under the leadership of Gene Brown, the former Dean of Science, Langley Keyes, the former Head of the Program in Urban Studies and Planning, and James G. Paradis, the Head of the Program in Writing and Humanistic Studies. Lawrence S. Bacow, first Chair of the MIT Faculty and then MIT's Chancellor (and currently President of Tufts University), commented that the curriculum is designed not only for the current students but also for the people they will be in five, ten, and fifteen years.

In the spring of 2000, the MIT Faculty voted to establish, beginning with the class entering in 2001, the Communication Requirement, a new curriculum that mandated that every undergraduate at MIT take one communication-intensive (CI) class in each of their four undergraduate years. Normally, the first two CI classes are in the Humanities, Arts, or Social Sciences. In their junior and senior years students take CI classes that are part of their major curriculum and in which they learn and practice discipline-specific genres of writing and speaking. To ensure that writing would no longer be a neglected skill among undergraduates, at least 20% of the final grade in CI courses is based on the quality of writing and speaking in class assignments. It is no longer possible for poor writers as a group to have as high a grade-point average as good writers. The desire of MIT students to optimize their academic efforts now leads them to devote time to improve their writing and speaking skills. Simply put, writing and speaking now count at MIT.

Evolving out of the Undergraduate Technical Writing Cooperative, the Writing Across the Curriculum (WAC) group in the Program in Writing and Humanistic Studies provides lecturers who support many (but of not all) of these classes by giving presentations on writing and speaking, conducting small group workshops and individual tutorials, and providing feedback on print drafts, oral presentation slides, and oral presentation run-throughs. In some classes there is close collaboration between the subject instructor and the WAC lecturer in the design and pacing of assignments. In other situations, the WAC lecturer provides instruction to support the writing and speaking assignments already in place. The Communication Requirement is governed by a permanent faculty subcommittee of the MIT's Committee on the Undergraduate Program, the Subcommittee on the Communication Requirement (SOCR), which vets CI classes and is also charged by the Faculty to periodically review and evaluate the effectiveness of specific CI subjects as well the overall efficacy of the Communication Requirement.

Assessment of the Communication Requirement

In 2005, with the graduation of the first class under the new curriculum, preparations were made to assess the new requirement. Some engineering and science faculty wanted a single direct measure of student improvement, even though the curriculum mandates that communication instruction in the junior and senior years be discipline specific. One faculty member in jest (I hope) suggested that MIT use the California Institute of Technology as a control group by asking that institution to withhold all instruction in writing and speaking to an entire undergraduate class and then measure the writing and speaking skills of the respective graduating class of each institution. Essentially, however, the faculty and administration both wanted the answer to what appeared to be a simple question: Does this new curriculum help students write and speak better? In addition, there was considerable insistence, especially from the upper administration, that evidence should be produced to demonstrate that the large expenditures to support the Communication Requirement were worth it. After much discussion, SOCR decided to assess the implementation of the Requirement rather than trying to directly assess outcomes.

After bringing in an outside group of evaluators to help formulate the process, MIT's Teaching and Learning Laboratory (TLL) in close collaboration with SOCR conducted a fifteen-month formative assessment consisting primarily of three surveys and faculty and student focus groups. The three surveys included one for faculty, one for seniors of the Class of 2006 (the second class to be under the new curriculum), and one for all undergraduates. The faculty survey covered four main areas:

  1. attitudes towards writing and speaking being taught throughout the curriculum;
  2. impressions concerning the implementation of the new curriculum;
  3. observations about the effect of the new curriculum on student writing and speaking skills; and
  4. suggestions for improving the new curriculum.

The senior survey includes questions about:

  1. the student's attitude toward communication skills;
  2. the structure of CI classes taken by the student;
  3. the priority the student placed on these classes;
  4. evaluation of the different instructional methodologies used in these classes;
  5. evaluation of different types of instructors (e. g. faculty, TA's, Writing Program lecturers); and
  6. student reports of the effect CI classes had on their ability to speak and write effectively.

In December 2006, the Communication Requirement Experience Survey (CRES) was administered to all undergraduates at MIT. Its purpose was to obtain a snapshot of all communication-related experiences students had during one semester. The survey consisted of five sections:

  1. a general overview section that asked students to report their level of confidence in their own writing, their belief in the importance of communication skills, and their perception of MIT's commitment to help students become better writers and speakers;
  2. a section asking information on students' experiences that term in CI classes in the humanities, arts, and social sciences;
  3. a section asking information on students' experiences that term in CI classes in the major; and
  4. information on other CI experiences; and
  5. two open-ended questions.

The surveys were complemented by focus groups whose purpose was to help interpret the responses from the surveys. The Faculty focus groups were centered on three topics. The first topic dealt with the apparent contradiction between faculty reports in the survey that the new curriculum is working well and that students improve their writing and speaking skills in CI classes and faculty reports in the same survey that students do not write well and there has been no substantial improvement in student writing over the past few years. The second topic asked if the participants thought the new curriculum inflexible, and if they did, in what specific ways. The third topic dealt with how much direct support MIT should provide departments for resource-intensive CI classes and how much support should be provided by departments.

The student focus groups asked students if CI classes improved their writing and if the process of writing has gotten easier. Students were also asked about improvement of their presentation skills, the importance of writing and speaking in their future endeavors, and other options that could improve undergraduate communication skills at MIT.

As a result of these data collection efforts, TLL and SOCR jointly issued Implementation of the Undergraduate Communication Requirement: A Report on the Assessment in the spring of 2008. (An executive summary of the Report is available at http://web.mit.edu/commreq/CR_Assessment_ExSummary.pdf) The major conclusions of the report, some of which are listed below, showed that although differential, some progress had been made.

Next Steps

The Assessment Report calls for further formative studies, particularly in a collaboration effort—between SOCR, the Teaching and Learning Lab, and the WAC group in the Writing Program—to identify best practices and involve faculty teaching these classes in both dissemination and dialogue with other faculty. The hope is such encounters can become the context for continual formative assessment, re-design, and re-invention of the integration of writing and speaking into the MIT undergraduate curriculum, an effort that would be consistent with MIT's values.

When I first came to MIT in 1986, I had considerable contact with the late Don Schön, Ford Professor of Urban Studies, and author of The Reflective Practitioner and Educating the Reflective Practitioner. The principal thesis of The Reflective Practitioner is that successful professionals make decisions and solve problems based not only on factual knowledge, but also on reflection and un-articulated intuition. In Educating the Reflective Practitioner, Schön recounts a curriculum reform project in MIT's department of Urban Studies that involved a group of faculty and graduate students meeting for three years to re-design the core graduate curriculum. He reports:

The small group of faculty members and students who criticized the old core, planned for the new one, and taught it for the first time were self-consciously engaged in a design process. Over a three-year period, as they familiarized themselves with one another's research and practice, they learned by doing, to construct a new curriculum. They created an environment for intellectual debate about teaching and, in the process, also created an intellectual community—thereby discovering how much they missed belonging to one. (Schön, 1987, p.337)

In a period of severe economic contraction and emphasis on academic achievement within a narrow field of study, creating such communities at MIT will not be an easy task. I do believe, however, that bringing faculty together to discuss how best to teach communication in their respective disciplines can serve as a catalyst not only for continual renewal of MIT's WID program, but for other educational innovations not yet conceived.

Perhaps as important for an institution that values data, our efforts to review the program quantitatively continue. In fact, the desire by faculty and administration for some sort of quantitative assessment that would demonstrate (or not demonstrate) the Communication Requirement's "treatment effect" may soon be satisfied. It was a survey of alumni that provided a significant part of the motivation for establishing this curriculum. As this article is being written, an almost identical survey is being sent out to a number of MIT Alumni Classes, including the Class of 2007, the third class to graduate under the Communication Requirement. Alumni reports of MIT's significance in contributing to their writing and speaking skills may be the most reliable and valid dependent variables we can obtain. A large increase in these percentages is probably as close as we can get to a measure of the success of something as complex and varied as MIT's array of CI classes.

Appendices

APPENDIX A: Question 20 from Alumni Survey

Appendix A: Question 20 from Alumni Survey

References

Brereton, John C. (1995). The Origins of Composition Studies in the American College, 1875-1925: A Documentary History. Pittsburgh: Pittsburgh University Press.

Committee on Educational Survey, Massachusetts Institute of Technology. (1949) Report of the Committee on Educational Survey to the Faculty of the Massachusetts Institute of Technology. Cambridge, MA: The Technology Press. Retrieved from http://libraries.mit.edu/archives/mithistory/pdf/lewis.pdf

Russell, David R. (2002). Writing in the Academic Disciplines: A Curricular History (2nd ed.). Carbondale: Southern Illinois University Press.

Schön, Donald A. (1983). The Reflective Practitioner: How Professionals Think in Action. New York: Basic Books.

Schön, Donald A. (1987). Educating the Reflective Practitioner. San Francisco: Jossey-Bass Publishers.

Snyder, Benson R. (1973). The Hidden Curriculum. Cambridge, MA: MIT Press.

Teaching and Learning Laboratory and the Subcommittee on the Communication Requirement, Massachusetts Institute of Technology. (2008). Implementation of the Undergraduate Communication Requirement: A Report on the Assessment. Cambridge, MA.

Teaching and Learning Laboratory and the Subcommittee on the Communication Requirement, Massachusetts Institute of Technology. (2008). Executive summary. In Implementation of the Undergraduate Communication Requirement: A Report on the Assessment. Cambridge, MA. Retrieved from http://web.mit.edu/commreq/CR_Assessment_ExSummary.pdf

Acknowledgements

I want to thank:

Contact Information


Complete APA Citation

Perelman, Les (2009, December 3). Data driven change is easy; Assessing and maintaining it is the hard part. [Special issue on Writing Across the Curriculum and Assessment] Across the Disciplines, 6. Retrieved from https://wac.colostate.edu/atd/assessment/perelman.cfm