Assessment of the Writing Component within a University General Education Program

Legene Quesenberry, Tracey Johnson, Christopher Ferry, Edward Caropreso, Pamela Hufnagel, Paula Bevan, Karen Bingham, William Buchanan, Stacy Byers, Carla Diehl-Weyandt, Iseli Krauss, Wayne Miller, Randy Otte, Deborah Sarbin, Richard Smaby, Joseph M. Serafin, Jeanne Slattery, Karen Smith, Michael Vitali, Kathleen Welsch, Mary Wilson, and Paul Woodburne
Clarion University

Go Contact Information

PDF Version Available View an Adobe Acrobat version of this article. (205K)

 

Introduction

The purpose of this study was to assess whether flagged "writing-intensive" courses within Clarion University's General Education Curriculum impacted on students' abilities to write. The major research question to be explored was, "what effect does taking writing intensive courses have on students' writing ability, when factors such as initial matriculation ability and total coursework are taken into account?" Papers written by a sample of freshmen, sophomores, juniors, and seniors were holistically scored to determine the level of writing exhibited by the student writers. Demographic information and information about the number of writing intensive courses each student had completed were correlated with the scores on the writing samples. It was found that students who had completed more writing intensive courses scored higher on the writing samples used for this research.

After providing an overview of Clarion University and the State System of Higher Education (of which Clarion University is a part), this paper provides an overview of Clarion University's General Education program. This is followed by a description of the study's methodology, demographics of the research population, review of results, and discussion of results.

Clarion University and the Pennsylvania State System of Higher Education

Clarion University of Pennsylvania is a public comprehensive university offering degree programs at the associate, bachelor's and master's levels. Like many other Pennsylvania universities, Clarion spent its first years of existence preparing teachers for the Commonwealth's schools after the Normal School Act of 1857 established regional teacher training institutions throughout Pennsylvania. The School Code of 1911 called for the state purchase of all normal schools, and in 1921, Pennsylvania's present configuration of 14 state-owned institutions was established. The normal schools evolved from state normal schools, to state teacher colleges, to state colleges. On November 12, 1982, Act 188 was signed into law establishing Pennsylvania's State System of Higher Education ("Act 188," 1982). On July 1, 1983, Act 188 was implemented. Clarion University along with 12 other former state colleges and Indiana University of Pennsylvania became the Pennsylvania State System of Higher Education ("About the System," sshechan.edu)

Today, the State System of Higher Education comprises 14 universities, four branch campuses, the McKeever Environmental Learning Center, and Dixon University Center. The System's purpose, as outlined in Act 188, is "to provide high quality education at the lowest possible cost to students." The System universities, located in rural, suburban, and small-town settings throughout Pennsylvania, educate more students per year than any other higher educational institution in the state.

A 20-member Board of Governors is responsible for planning and coordinating development and operation of the State System of Higher Education. The Board establishes broad educational, fiscal, and personnel policy, and oversees the efficient management of the System. Among other tasks, the Board appoints the chancellor and university presidents, approves new academic programs, sets policy for the State System, sets tuition, and coordinates and approves the annual System operating budget ("Board of Governors," sshechan.edu).

System-Wide Policy on General Education

In 1993, the Pennsylvania State System of Higher Education Board of Governors approved a policy on General Education, entitled "General Education at State System of Higher Education Universities" (the full text of the policy can be found under policy number 1993-01 at the web site www. sschechan.edu/BOGIndex.htm). Among other mandates, this policy states that all of the State System universities must ensure that (1) their general education programs meet State System priorities, and 2) their general education programs are properly evaluated in terms of student learning outcomes.

Clarion University's Policy on General Education

As part of its compliance with the State System policy on General Education, Clarion University engaged in a three-year process that culminated in a total redesign of its previous 1985 General Education program. The new program was adopted by the University in May of 1994, and implemented in August of 1995. In keeping with the State System's articulated spirit of general education, the Clarion University Statement of Philosophy on General Education asserts that General Education is enabling education. At its best, general education enables students to develop academic skills; acquire liberal knowledge; shape individual values; and apply all three (skills, knowledge, and values) in their academic, professional, personal, and societal lives. The goal of Clarion University's general education program is to provide students with specialized and liberal education that supports individual development and enriches professional and community life. Clarion University's General Education curriculum was designed to be a living program, amenable to review, revision, and innovation.

In order to facilitate the evolutionary nature of the program, an elected Council on General Education is charged with ongoing assessment, revision, and innovation of the University's General Education program (the Clarion University General Education Program can be found at the web site, www.clarion.edu/gened/index.htm).

The General Education Council advocates ongoing and consistent assessment that results in meaningful feedback for continuous improvement. According to the Council, assessment should:

The Writing Component of Clarion University's General Education Program

Taken directly from Clarion University's General Education document ("General Education Requirements: Clarion University of Pennsylvania," 1994), the following describes the writing component of the general education program:

Writing is a skill that should be exercised by students across the curriculum, as appropriate ways are found in each discipline to incorporate writing as a vehicle for learning into instruction in the disciplines.

  1. As a foundation for writing across the curriculum, students will be required to complete three writing-intensive courses prior to graduation: English 111 and two additional courses identified as writing-intensive courses. English 111 will be required in the program of General Education. The other two writing-intensive courses will be courses counting for credit beyond General Education as part of the total academic experience of the student, likely in the major or in support courses for the major not taken to fulfill requirements in General Education.

  2. Criteria for identifying writing-intensive courses are based on the standard that the writing activity should engage students in higher order reasoning, such as synthesis and analysis, as well as the communication modes of specific disciplines. "The point, always is to keep students writing in circumstances where it matters to their intellectual and social development, where it is integral to the processes of inquiry and learning, and where it is required, not just in the seeming isolation of `English class,' but in every part of the University as a natural feature of academic life" (Cy Knoblauch, BYWORD, Fall, '86).

    RATIONALE:

    1. Writing should be used, not just to test knowledge, but to provide learners opportunities to explore ideas and develop connections between new knowledge and the understandings they bring to the pursuit of new knowledge.
    2. Writers should be encouraged to revise some portion of their work, under the guidance of the instructor, in order to better understand the constraints of the discipline within which they are working.
    3. Writers should have opportunities to use many forms, including in-class responses, journals, notebooks, reports, formal argumentative essays, research papers, and other forms pertinent to specific disciplines.

      The following guidelines are used to determine if a course qualifies as writing intensive.

      1. A substantial body of finished work. This is generally expected to total 20+ double-spaced pages in at least two, preferably more, submissions. A variety of forms is strongly encouraged – journals, reports, essays, research papers, etc. – not all of which need to be graded.
      2. Opportunity for students to receive assistance in progress. Such assistance may take several forms, from visits to the Writing Center to conferences with the instructor.
      3. Opportunity to revise some pieces. As revision is an essential characteristic of good writing, students should be able to revise some portion of their work.
      4. Response to student writing. Such response may take many forms – from extended comments from the instructor to peer evaluation in student groups. It is expected, however, that the instructor will respond in detail to some extended work of the students.

  3. Departments will propose such courses to the Council on General Education for review and recommendations.
  4. Courses so identified will be flagged with a WI designation.
  5. All students entering the university will be tested prior to their initial registration for classes in order to assess their writing skills. Students with skills not appropriate to university-level work will be required to complete a course credited outside General Education designed to prepare them for the expectations of university-level writing.
  6. Other than those students who require remedial course work, a further delineation is made between students who are in need of preparatory skills such as those offered in English 110 and those more advanced students who are placed in English 111.
  7. Appropriate incentives will be offered to faculty to encourage the development of writing-intensive courses across the curriculum.

    The creation of writing-intensive courses should enhance present initiatives in the university focused on writing in the disciplines and across the curriculum.

Since 1995, over 100 existing courses within the university course inventory have been redesigned and approved as "writing intensive." These courses represent many disciplines, including accounting, anthropology, art, biology, chemistry, computer information science, education, English, geography, history, marketing, mathematics, modern languages, physics, philosophy, and psychology.

Purpose of the Study

The purpose of the current research was to determine how well Clarion University's "writing intensive" courses were meeting the student learning outcomes expectations articulated in the university's philosophy of general education. Specifically, the present research was designed to assess whether flagged "writing-intensive" courses within Clarion University's General Education Program impacted on students' abilities to write.

Methodology

Term papers completed by 120 Clarion University students in courses which enrolled a mixture of freshmen, sophomores, juniors, and seniors were analyzed using the Pennsylvania System of School Assessment (PSSA) "Pennsylvania Holistic Scoring Guide" (Pennsylvania Department of Education, 1998-1999).

In the fall 1998 semester, sample papers from CIS 110 (Introduction to Computer Information Systems) were utilized by members of the General Education Council to validate and calibrate the scoring instrument. The sample papers were later used as training materials and "anchor papers" for faculty who would be involved in the actual research. In preparation for the field test scoring sessions, the Council members modified the original scoring guide somewhat so that the definitions of writing characteristics would be clear for the current research. A copy of the final scoring instrument that was used for the current research is included as Appendix A.

Also during the fall, 1998 semester, a general "call for participation" was sent to all Clarion University faculty via e-mail and in hard copy. Faculty were invited to participate in the assessment of Clarion's General Education writing requirements by volunteering for four days to read and score student papers collected from courses in three to four different disciplines. The call for participation indicated that the General Education Council was particularly interested in faculty who had a commitment to teaching writing, including, but not limited to those who taught writing-intensive "flagged" courses. New faculty were especially encouraged to participate, and the Council indicated that the scoring experience would be formative for the faculty who were involved. Faculty who volunteered were asked to sign up for one of three assessment time periods during the spring, 1999 semester. In addition to the three facilitators, the Chair of the Council on General Education, and the Chair of the Assessment subcommittee, 16 faculty signed up to participate in the study. A copy of the "call for participation" is included as Appendix B.

During the Spring, 1999 semester, 120 term papers written by students in four classes which enrolled a mix of freshmen, sophomores, juniors, and seniors were scored according to the Pennsylvania Department of Education Holistic Scoring Sheet (modified as described above). The term papers represented four classes in three different disciplines. Thirty papers were from CIS 110, section 8 (Introduction to Computer Information Systems); 31 papers were from CIS 110, section 9 (Introduction to Computer Information Systems); 35 papers were from HIST 120, section 5 (United States History to 1877); and 24 papers were from ENG 200, section 4 (Composition and Literature). Faculty scorers were provided with copies of the papers and the accompanying assignment (see Appendix C for the assignments). Only one paper from each student was used. Any students who were represented by a paper in more than one class were assigned to one class. Their second paper in a different class was not used.

In order to divide the scorers into equal groups, four groups of faculty met for four days each to score the papers. The first group comprised four faculty representing the departments of Academic Support Services (one faculty member), Education (one faculty member), Biology-Science Education (one faculty member), and Chemistry (one faculty member). The facilitator for the first group was a faculty member in the department of English. The second group comprised four faculty representing the departments of Economics (one faculty member), English (two faculty members), and Library Science (one faculty member). The facilitator for the second group was a faculty member in the department of Education. The third group comprised four faculty members representing the departments of Psychology (one faculty member), English (one faculty member), Communication (one faculty member), and Accounting (one faculty member). The fourth group comprised four faculty members representing the departments of Communication (one faculty member), Accounting (one faculty member), English (one faculty member), and Psychology (one faculty member). The facilitator for the third and fourth groups was a faculty member from the department of English. Three of the four groups met for four consecutive days; one of the groups met on four Fridays throughout a 15-week semester.

Eight raters (two from each of the four groups) read each of the 120 papers. The redundancy was important in order to be able to determine "inter- and intra-rater reliability" and in order to ensure that each paper would have enough scores to provide a means for statistically analyzing the scores.

Demographics of Student Writers

Majors of Students

The 120 student papers included in the study were written by students with 28 different declared (or undeclared majors). Table 1 provides a breakdown of the majors represented by the student papers in this study.

Table 1. Student Majors

Major

Frequency

Percent

Accounting

5

4.2%

Arts and Sciences - Undecided Business

1

0.8%

Biology

2

1.7%

Chemistry

1

0.8%

Communication

12

10.0%

Communication Science Disorders

1

0.8%

Elementary Education/Early Childhood

8

6.7%

Elementary Education

16

13.3%

Elementary Education/Special Education

13

10.8%

English

3

2.5%

Elective Studies in Business

1

0.8%

Finance

1

0.8%

General Studies

2

1.7%

History

1

0.8%

Industrial Relations

1

0.8%

Information Systems

1

0.8%

Management

7

5.8%

Marketing

6

5.0%

Political Science

1

0.8%

Psychology

3

2.5%

Real Estate

2

1.7%

Rehabilitative Sciences

1

0.8%

Secondary Education - Communication Arts

2

1.7%

Secondary Education - English

5

4.2%

Secondary Education—Mathematics

1

0.8%

Sociology

2

1.7%

Secondary Education - Social Studies

5

4.2%

Undecided

16

13.3%

     

Total

120

100.0%

Transfer Status of Students

As Table 2 demonstrates, 100 (83.3%) of the total students in the study were not transfer students; 20 (16.7%) of the students were transfer students. This information is important since if a student transfers into Clarion University he/she may have received more or less preparation for writing at the previous institution.

Table 2. Transfer Status of Students

 

Frequency

Percent

Not a transfer student

100

83.3

Transfer student

20

16.7

Total

120

100.0

Gender of Students

As Table 3 demonstrates, 73 (60.8%) of the students in this study were female; 47 (39.2%) were male.

Table 3. Gender of Students

 

Frequency

Percent

Female 73 60.8
Male 47 39.2
Total 120 100.0

Class Standing of Students

As Table 4 demonstrates, 43 (35.8%) of the students were Freshman, 43 (35.8%) of the students were Sophomores, 25 (20.8%) of the students were Juniors, and 9 (7.5%) of the students were Seniors.

Table 4. Class Standing of Students

 

Frequency

Percent

 Freshman

43

35.8

 Sophomore

43

35.8

 Junior

25

20.8

 Senior

9

7.5

 Total

120

100.0

Academic Data

Table 5 provides the minimum, maximum, means, and standard deviations of several academic variables including number of credits taken by the end of the semester in which the paper was written, number of "writing-intensive courses" that a student passed with an "A," "B," or "C" grade by the end of the semester in which the paper was written, Cumulative QPA, Major QPA, Total SAT score (math and verbal combined), SAT verbal score, SAT Math score, global score on the writing assessment, total score on the writing assessment, and number of "writing-intensive" courses taken without regard to the grade received.

Table 5. Academic Data

 

N

Minimum

Maximum

Mean

Std. Deviation

 Number of Credits

120

6.00

117.00

47.2167

29.8163

 Number of Writing Intensive Courses Passed with a grade of "A," "B," or "C"

120

.00

6.00

1.1500

1.1050

 Cumulative QPA

117

.25

4.00

2.7636

.6540

 Major QPA

96

2.000

4.00

3.05057

.58505

 Total SAT Score (Math and Verbal Combined)

101

580

1270

931.49

139.77

 SAT Verbal Score

101

310

670

474.06

76.34

 SAT Math Score

101

220

650

457.43

81.04

 Global Writing Assessment Score

116

1.167

5.125

3.17589

.98029

 Total Writing Assessment Score

116

7.750

26.620

17.17865

4.21638

 Number of Writing-Intensive Courses completed without regard to grade received

120

.00

6.00

1.1833

1.0845

Results

Inter-Rater and Intra-Rater Reliability of Scoring

Of the original 120 papers, four were judged to be unscorable (e.g., they were illegible, incoherent, or blank) by all raters, and were eliminated from the analysis. Each of the remaining 116 papers received a holistic "Global" rating and qualitative assessments for five characteristics of writing that are used in most typical holistic scoring rubrics for writing (Focus, Content, Organization, Style, and Conventions) from up to eight evaluators. The Global scores ranged from a low of 1 to a high of 6. The qualitative assessments ranged from low quality to high quality and were based on specific criteria for each characteristic. For example, the qualitative assessments for Focus ranged from "unfocused" to "sharp, distinct focus." The qualitative analyses were not assigned a quantitative value in order to ensure that the global scoring was holistic rather than based on adding the scores for assessments of writing characteristics. However, for all of the readers in this study, large-scale assessment was a new experience. The researchers were concerned about the reliability and validity of the scores for a group of previously untrained assessment readers. Therefore, for reinforcement purposes, the researchers chose to collect both global and characteristic subscores. To the researchers' knowledge, this "double collection" of scores is a unique method of data collection. For the analysis of intra-rater reliability, each characteristic assessment was assigned a score, referred to in this analysis as a "subscore" ranging from 1 (lowest quality) to 6 (highest quality). For purposes of determining intra-rater reliability, the subscores were added, creating a new variable called Total score, which ranged from a theoretical low of 5 to a theoretical high of 30.

Correlations among each of the five variables comprising the Total score (characteristics scores) and the holistic (Global) score were not calculated. However, intra-rater reliability was evaluated by correlating the Global and Total scores assigned to each paper by each evaluator (r = .550, n = 934, p = .000). Although the r-value was moderate rather than high (the r-squared accounted for a little more than 25 percent of the variance), the two sets of scores were positively and significantly correlated. Since a correlation of .80 or higher would have indicated more acceptable levels of intra-rater reliability, an analysis of the correlations among specific characteristics subscores and the Global scores might be indicated for future studies.

Inter-rater reliability was evaluated by computing intraclass correlations on samples of papers that were all scored by at least four raters in common. Intraclass correlations provide a means for assessing consistency and agreement among multiple scores of a set of targets (McGraw & Wong, 1996). Correlations from samples of papers from each course (CIS, English, and History) were computed separately because it was not possible to select a sample with good representation from all subjects that shared four common raters. If a paper was missing a score on a variable, it was omitted from the analysis for that variable only. The results are shown in Table 6. Most measures of consistency were significant, which means that papers that received high scores tended to receive relatively high scores from all four raters. Although most measures of absolute agreement were also significant, they tended to be somewhat lower than the measures of consistency. This means that despite a general consensus on which papers were better, raters varied in the actual scores assigned. Variance in the consistency and agreement ratings for style and convention were interesting to the researchers. It is possible that different approaches to training and rater calibration by the three trainers might account for this. It is also possible that the intervening days between scoring sessions for the group that met on four successive Fridays might have interfered with rater calibration.

Table 6. Intraclass Correlations Based on Samples of Papers

 

CIS Papers1

English Papers2

History Papers3

Variable:

Consistency

Agreement

Consistency

Agreement

Consistency

Agreement

Global

.9127****

.8982****

.8596****

.8527****

.7974***

.8002***

Focus

.8081**

.8037****

.8273**

.7558**

.6735*

.6406*

Content

.7168*

.6849**

.7380*

.7099**

.7744**

.7842***

Organization

.6641*

.6181*

.5825

.5613

.7484*

.6606*

Style

.8725****

.6678****

.5714 (n=9)

.5870 (n=9)

.5148

.5164

Conventions

.8616****

.8661****

.4298 (n=7)

.3928 (n=7)

.7322*

.7188**

1 Raters included for CIS papers were #1, #4, #201, and #202. Number of papers = 13
2 Raters included for English papers were #5, #6, #205, and #207. Number of papers = 10
3 Raters included for History papers were #3, #6, #204, and #205. Number of papers = 10

* p < or = .01      **p< < or = .001     *** p < or = .0001     **** p < or = .00001

To present a clearer picture of the level of agreement among different raters on absolute ratings assigned to papers, the mean and standard deviation of the Global scores assigned to each paper by its raters were also calculated. The overall mean and standard deviation were computed using the individual papers means and standard deviations as data. As seen in Table 7, the mean of the standard deviations is .78. This indicates that in most of the cases, the global scores assigned by different raters for the same paper were within one point of its mean Global rating.

Table 7. Descriptive Statistics on Individual Mean Global Scores

 

N

Minimum

Maximum

Mean

Std. Deviation

Global Score Mean

116

1.167

5.125

3.17589

.98029

Global Score Standard

Deviations

116

.0707

1.6420

.780592

.292695

Finally, as one further test, the Global and Total scores assigned to each paper by its evaluators were averaged to produce the scores used in this analysis. The correlation between the mean Global and Total scores assigned to each paper was .971 (n = 116, p = .000). Since these two scores were produced in quite different ways, but they correlated so strongly with each other, averaging the scores assigned by different raters apparently produced more reliable measures. It was therefore decided to continue with the analysis, using mean Global and Total scores as each student's essay scores.

Correlations

Total SAT (Verbal plus Quantitative) was found to be the best predictor of Global score (r = 0.524). This suggests that students who entered Clarion University with higher levels of developed ability (as measured by SAT scores) wrote better essays than students who entered with lower levels of developed ability.

Other potentially causal variables that correlated significantly with Global score were SAT Verbal (r = .504), SAT Math (r = .431), and the number of writing intensive courses passed with a grade of "A," "B," or "C" (r = .199).

The patterns revealed by an analysis of the Total scores on the essays were essentially the same. Significant correlations were found between the essay Total score and SAT Total (.543), with SAT Verbal (.521), with SAT Math (r = .447), with number of writing intensive courses passed with a grade of "A," "B," or "C" (r = .196).

These correlations, with number of cases included in each and their significance levels can be found in Table 8.

Table 8. Correlations Among Writing Scores and Academic Data

 

Total

SAT Total

SAT Verbal

SAT
Math

Number of
W Courses

Global

.971**
n=116

.524**
n=98

.504**
n=98

.431**
n=98

.175
n=116

Total

---------

.543**
n=98

.521**
n=98

.447**
n=98

.171
n=116


 

Writing Courses With A,B,C

Number of Credits

Cum QPA

Global

.199*
n=116

.122
n=116

.431**
n=114

Total

.196*
n=116

.098
n=116

.408**
n=114

** Correlation is significant at the .01 level (2-tailed).
 *Correlation is significant at the .05 level (2-tailed).

Relationship Between Completion of Writing Intensive Courses and Students' Writing Abilities

Since the purpose of this study was to assess whether flagged "writing-intensive" courses within Clarion University's General Education Curriculum impacted on students' abilities to write, the following analyses were performed to determine the impact of writing intensive courses, controlling for entering ability (as measured by Total SAT), which was identified above as the best predictor of essay scores.

The correlation between Global score and the number of writing intensive courses taken was .175 (n = 116, p = .060). Controlling for Total SAT, the partial correlation between Global score and the number of writing intensive courses taken was .2227 (df = 95, p = .028).

The correlation between Global score and the number of writing intensive courses completed with a grade of "A," "B," or "C" was .199 (n = 116, p = .032). Controlling for Total SAT, the partial correlation between Global score and the number of writing intensive courses completed with a grade of "A," "B," or "C" was .2347 (df = 95, p = .021).

These partial correlations indicate that if one were able to compare students with similar SAT scores, those who had completed more writing intensive courses and those who had completed more writing intensive courses with grades of "A," "B," or "C" would have had higher Global scores on the essay. The correlations, with number of cases included on each and their significance levels can be found in Table 9.

The patterns revealed by an analysis of the Total scores on the essays were essentially the same. The correlation between Total score and the number of writing intensive courses taken was .161 (n = 116, p = .074). Controlling for Total SAT, the partial correlation between Total score and the number of writing intensive courses taken was .2306 (df = 95, p = .023).

The correlation between Total score and the number of writing intensive courses completed with a grade of "A," "B," or "C" was .196 (n = 116, p = .035). Controlling for Total SAT, the partial correlation between Total score and the number of writing intensive courses completed with a grade of "A," "B," or "C" was .2456 (df= 95, p = .015).

These partial correlations indicate that if one were able to compare students with similar SAT scores, those who had completed more writing intensive courses and those who had completed more writing intensive courses with grades of "A," "B," or "C" would have had higher Total scores on the essay. The correlations, with number of cases included on each and their significance levels can be found in Table 9.

Table 9. Partial Correlational Coefficients Between Number of Writing Intensive Courses and Global/Total Scores Controlling for SAT Total

 

Essay Global Score

Essay Total Score

Number of Writing Intensive Courses

r=.2227
df=95
p=.028

r=.2306
df=95
p=.023

Number of Writing Intensive Courses With Grade A,B,C

r=.2347
df=95
p=.021

r=.2456
df=95
p=.015

All tabled significance levels are two-tailed.

Although the correlations outlined in Table 9 are significant, there is a possibility that students who had completed more writing courses were more advanced students, and therefore that their higher essay scores might have been partly accounted for by their having had more college experience overall. To test this, the effect of the number of credits earned was partialled out of the correlations between essay scores and the number of writing courses.

As Table 10 shows, none of these partial correlations reached significance, although the number of writing courses completed with grades of "A," "B," or "C" would reach significance in a one-tailed test, which is probably justifiable, since a positive correlation would be predicted.

Table 10. Partial Correlation Coefficients Between Writing Intensive Courses and Global/Total Scores Controlling for Number of Credits Earned

 

Essay Global Score

Essay Total Score

Number of Writing Intensive Courses

r=.1330
df=113
p=.157

r=.1378
df=113
p=.142

Number of Writing Intensive Courses With Grade A,B,C

r=.1600
df=113
p=.088

r=.1719
df=113
p=.066

All tabled significance levels are two-tailed.

It could also be argued that students who remained in college long enough to earn more credits were more academically talented upon matriculation. To test that possibility, Total SAT was partialled out of the correlation between number of credits earned and essay scores. As seen in Table 11, these partial correlations are significant, meaning that controlling for the effect of SAT, students who earned more credits earned higher essay scores.

Table 11. Partial Correlation Coefficients Between Credits Earned and Essay Scores Controlling for SAT Total

 

Essay Global Score

Essay Total Score

Credits Earned

r=.3294
df=95
p=.001

r=.3229
df=95
p=.001

As shown in Table 12, combining these factors in a multiple regression analysis reveals that although SAT Total and number of credits explain significant amounts of variance in essay Global score, once they are controlled, the number of writing intensive courses with grades of A, B, or C does not add much explanatory power. In fact, the partial correlation between the number of writing intensive courses with grades of A, B, or C and essay Global score, with SAT Total and credits partialled out is 0.0202 (df = 94, p = .845).

The researchers hope to determine in a later study whether part of the explanation for this finding is the positive impact our writing across the curriculum program has on students' abilities to write. It may be that because students receive many opportunities to write in a variety of disciplines and in a variety of forms (journal entries, essay exams, memoranda), their writing naturally improves.

Table 12. Regression Analysis of SAT Total, Number of Credits, and Number of Writing Intensive Courses with Grades of A, B, or C on

Essay Global Score

 

Unstandardized Coefficients

 

Standardized Coefficients

t

Sig.

 

B

Std. Error

Beta

   

SATTOT

.0038

.001

.555

6.639

.000

CREDITS

.0088

.004

.267

2.382

.019

WWITHABC

.0204

.104

.022

.196

.845

R2 = .353

Together, these correlational data show that students who completed more writing intensive course received higher scores on their essays. However, because these results are based on correlational data and because the correlation becomes negligible when number of credits are controlled, future research should attempt to control for the possibility of a self-selection factor (e.g., good writers are more likely to remain in college and take more writing-intensive courses).

Discussion

This first foray into writing assessment is considered a pilot study for Clarion University. As the institution attempts to assess its General Education program, the institution also seeks to find manageable, cost-effective, formative means to assess student progress. As the Council on General Education and those involved with writing intensive courses continue to discuss and modify programs, so too will the assessment approximate a model suitable for the students at Clarion University.

As a pilot study, the researchers chose not to control for many variables in this process. As a result, not all findings reached significance. However, two findings did emerge that proved to be important in assessing the writing components of the General Education program: (1) the scoring rubric worked well for faculty readers; (2) when comparing students with similar SAT scores, students who took more writing intensive courses scored better on their essays.

In its initial form, the modified PSSA Pennsylvania Holistic Scoring Guide rubric provided a common basis for the discussion about writing assessment. The faculty readers eventually came to consensus on the qualities of good writing. Initial scorings were often non-adjacent, at which point readers negotiated their scores to adjacency. Over time, readers achieved adjacency and congruency in scoring with no negotiations.

The data from this study indicate that those students who had engaged in more writing in their courses consistently scored better on their assessed essays. This indicates the need for continued support of writing intensive courses in the General Education program.

Recommendations for Further Research

Refined Measurements

Despite extensive training and effort, there was still some disagreement among raters in the evaluation of individual papers. This reduced the reliability of the data, which in turn introduced error variance, obscuring potentially significant relationships among variables. Steps should be taken to improve inter-rater reliability in future studies. It is therefore recommended that there be one training session for all raters, or, if this is not possible, the same facilitator should be employed for each training session.

The student papers used in this study were written for different courses. Inspection of the assignments used in each class (Appendix C) revealed that although all assignments required focused papers and evidence to support statements, the instructions themselves varied with respect to the amount of information provided on how to write a good paper. The assignments also varied with respect to the amount of creativity required for completion. These variations in assignments may have introduced extraneous variance into the students' scores. For future studies, one solution to this problem would be to use the same assignment in the same course for all of the students. Another possible solution might be to ask students to write in response to a prompt during a set period of time. The advantages to using embedded assessment (e.g., using writing produced for a class assignment) include increased motivation of the student to write well and the ability of the student to work through a longer period of writing and revision. The embedded approach may more accurately reflect an approach to writing that is aligned with the academic writing processes articulated in the General Education requirements for writing intensive courses (e.g., 20+ pages of text with opportunity to revise in response to self-assessment and instructor input). However, the prompt/response approach would help to reduce the effect of variables such as the instructor, the relationship between the instructor and individual students, the amount of instructor input, and the effect of the course and/or course content.

English 111: Writing II and English 110: Writing I

The present analysis did not examine the effect of English 111 on the writing of students. It also did not examine the impact of this course within the general education curriculum. Since English 111 is cited in the Clarion University General Education Program as the first of three required writing courses, it would be reasonable to examine the impact of this course, either by itself, or in combination with other variables such as other writing intensive courses or demographic variables. Additionally, although not specified in the Clarion University General Education Program, analysis of the relationship between English 110 and writing might be helpful in order to determine future directions for this sequence of courses in terms of student's abilities to write.

Assessment Tests and Scoring Rubrics

The General Education Program mandates that all students are tested prior to their initial registration for classes in order to assess their writing skills. This study used SAT scores as the best indicator of prior ability. However, scores on a test that specifically measures writing ability upon entrance to Clarion University might be a better covariate. One way to achieve this might be to create a writing assessment tool and/or a scoring rubric that could be used for initial testing of students prior to registration, and then later as a measure of student's writing as they progress through Clarion University's general education writing components.

Formative Assessment: Changing Attitudes and Pedagogy

Another component of research involves the changing attitudes of faculty toward the use of writing in their classes. In this pilot study, anecdotal, qualitative evidence indicates a possible change in attitudes. Faculty who once identified "good writing" as writing that was grammatically correct began to see the importance of expression of ideas in writing. Further research should explore the changing attitudes of faculty readers toward the use of writing in their own classrooms.

Other Types of Analyses of the Current Data

The analyses presented in this paper treated all scores as interval level data, enabling the subscores for writing characteristics to be summed to create the essay Total scores and enabling scores assigned by different raters to be averaged. The mean scores were then used for further analysis. This procedure may have reduced the score variance. One alternative would be to use the modal rating assigned to each paper instead of the mean.

As noted above, the analyses reported in this paper combined the data from papers written for different courses. Further analyses of the current data set could focus attention on the papers from one course, keeping in mind that the sample size would be reduced.

Finally, the current analysis did not correlate demographic and academic data with specific characteristic assessments (Focus, Content, Organization, Style, Content). This type of analysis might provide interesting and helpful data that could facilitate better definition or confirm the focus of current criteria for writing intensive courses and writing general education courses at Clarion University.

Conclusion

This study assessed the scoring rubric selected by the researchers for use at Clarion University with Clarion University students. The results of this study indicate that completion of more writing intensive courses has a positive impact on students' abilities to write. Because the study was based on correlational data, and many of the significant correlations were relatively modest, consideration of changes in programs or policies should incorporate an analysis of costs relative to benefits and the results of subsequent research on the topic. Specifically, further studies should attempt to more clearly separate the influence of writing intensive courses from the impact of the overall university experience and the impact of students' entering abilities. Further studies might also attempt to:

References

Clarion University of Pennsylvania. (May 2, 1994). General Education Requirements: Clarion University of Pennsylvania. Retrieved October 5, 2000 from the World Wide Web: http://www.clarion.edu/gened/guidelines.html#writing

McGraw, K.O. & Wong, S.P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods 1: 30-46.

Pennsylvania Department of Education. Division of Evaluation and Reports. (1998). Writing Assessment Handbook. Harrisburg, PA: Pennsylvania Department of Education.

Pennsylvania State System of Higher Education. (October 5, 2000). About the system. Retrieved October 5, 2000 from the World Wide Web: http://www.sshechan.edu/sssystem.htm

Pennsylvania State System of Higher Education. (October 5, 2000). Act 188. Retrieved October 5, 2000 from the World Wide Web: http://www.sshechan.edu/ssact188.htm

Pennsylvania State System of Higher Education. (October 5, 2000). Board of Governors. Retrieved October 5, 2000 from the World Wide Web: http://www.sshechan.edu/ssbogmnu

Appendix A

Appendix B

Call for Participation

You are invited to participate in the assessment of the writing features of the General Education Program. The Council on General Education is examining the impact on student writing skills of the basic writing intensive component, as well as the writing intensive (W) flag.

The project will assess student performance in writing. The purposes of the assessment are to assist the Council on General Education in continuous improvement of the General Education Program, provide feedback to the university community which can help in the development of effective teaching strategies for General Education writing courses, provide information to external agencies, and provide a model for assessment of other areas of the General Education Program.

Eight to ten faculty are needed to read and score samples of student papers collected from courses in 3 to 4 different disciplines. The selected faculty will need to devote 1 day to training and 3 days to reading student papers. We are particularly interested in faculty who have a commitment to teaching writing, including, but not limited to those who teach W-flagged courses. New faculty are especially encouraged to participate in this assessment. We expect the experience will be formative for the faculty involved.

The participants will become authors of an article or series of articles, which will be submitted for publication to journals, such as: General Education Review, CCC, and Scholars. Support for preparing the article(s) for publication will come from the Associate Provost's office.

__________________________

There are three possible schedules for the assessment. Please indicate which schedule(s) work for you and return this form to Richard Smaby, Chair of the Council on General Education, CIS Department.

One four day block during spring break (March 31 to April 6) _______

Four Fridays during the spring semester: ___________

One four day block at the end of the spring semester (May 20 to 26) _______

Your name:

Your e-mail address:

Your telephone number:

Your department:

Please indicate your interest or involvement in helping students to learn to write:

Appendix C

Assignments for Essays

Assignment for CIS 110, sections 8 and 9

Outline of the Project Report

View your project as a process. Imagine yourself as part of an actual business. Describe the problem solving process. This includes not only describing why you chose the particular design for what you produced, but also the tools you used and why they were appropriate.

A clear description of the problem your group was trying to solve

A detailed description of the problem will motivate the rest of the report. Be specific here so you can later motivate how your solution will support the goals of the company. You will need to refer back to this section to justify your decisions below.

A motivation for the solution your group came up with

Describe the alternative solutions your group considered. Explain why you chose the one you did. Contrast it with other solutions your considered and say why you chose this one. The tech department considers alternative choices of components. The accounting department considers alternative content and presentation of numbers. The database groups consider which fields to include and how to break the database down into tables. The sales and marketing groups consider alternative content and layout of information.

A detailed description of the result of the project, including examples from the project

Describe in detail your design for computers, advertisements, spreadsheets, etc. Attach sample documents if appropriate.

A description of the tools or resources you chose to construct the product with and why you chose those tools over others

Your department must produce similar products on an ongoing basis. You need tools to help you do your work. Identify those tools, especially information management tools and justify your choice.

Proposal for how to place the product in operation in the company

Describe how your product is made available to those who need it. In addition, your product is not just the advertisement or the spreadsheet or the computers you design, but the process for creating these things. Once the tools are purchased to do the job, they have to be made available to the staff and there remain issues such as training staff to use the tools and to understand

how to produce results up to the standards of the company.

Suggestions for how to improve the project.

Every project must meet a deadline and be put into operation before it can be adequately evaluated. Critique your project and the tools you used and offer suggestions for how it could be improved the next time.

Assignment for ENG 200, section 4

Composition and Literature

Spring 1999

For what can be imagined more beautiful than
the sight of a perfectly just city rejoicing
in justice alone?

             - - Mark Helperin, A Winter's Tale

Here are three suggested paper topics. These are the topics, open to many different interpretations. Should you choose one, you must narrow it to a more specific area of inquiry, develop a workable thesis around which to build an argument. If you want to discuss your approach or your own topic idea, please sop by during my office hours (MWF 8:30-9:00 & 10:00-11:00; W 1:00-1:30), or, if these times are inconvenient, make an appointment.

I will respond extensively to your paper (see number 5 in the "Guidelines" below) but will not put a grade on it unless you ask me to. You may revise your paper as often as you like, but I want you to do so based on your evaluation of your own effort and my comments, not on the grade. Therefore, if you request a grade, I will assume you think the paper is finished and that you have forfeited your option to re-write.

OPTION ONE

If you choose this option, you will show your understanding of how various "elements" of literary language – character, symbols, irony, figures of speech, for example – function to clarify a theme or themes. Choose a text we haven't discussed in class (preferably one from your book), discern what you believe to be its theme (or themes), choose the element you believe best illuminates it (them), and write an essay showing how the element does so. You will, in other words, do what we do in class when we discuss a text. Perhaps another way to approach this assignment, then, is to image that you are teaching the text to me. I'll be less concerned with the "correctness" of your interpretation than with how thoroughly you analyze your story, and how convincingly you support your argument. Please don't focus exclusively on "what the author is trying to say"; rather, tell me what the author says to you. You should in addition avoid reading (and regurgitating) critical articles: if I want to know what the critics think, I'll go to the library and read them myself. This paper should tell me what you think.

OPTION TWO

This second option has two parts. First tell one of the stories we've discussed in class from another point of view.  You might, for example, retell "A Good Man is Hard to Find" from The Misfit's perspective, or write about "Queenie's" experience in the A & P. You could imagine that you are (an extra) character and tell the story from your point of view. In re-telling the story, you should not necessarily follow the original form. In fact, I would prefer you to summarize the "new" story and focus instead on the second part of this assignment. Analyze the work you've created in terms of the changes you've made: how and why is it different now that it's told from another perspective?

OPTION THREE

The question of justice has long preoccupied Western thought. Plato, for example, believed that a just state (and soul) would result if passion and reason were balanced. The book of Amos assures us that "justice [will] run down as waters and righteousness as a might stream" (5:24). The Israelite tradition also calls for "An eye for an eye, and a tooth for a tooth (Ex. 21:24; versions of this passage can be also found in the Code of Hammurabi as well as the Koran), while the Christian tradition teaches, "whosoever shall smite thee on thy right cheek, turn to him the other also" (Matt. 5:39). What sense do you make of these various conceptions of justice? How do you define the word? In terms of "social" justice? Jurisprudence? What's "fair"? As revenge? Is there a transcendent version of justice? If not, who gets to say what justice is and why? Have any of the traditional definitions shaped your own? Is it possible for you to live a just life? How would you do so? Use one or more of the texts we've discussed in class to support your assertions.

Paper Guidelines

  1. Your paper must be between four and six pages long. Be sure to title it.
  2. Your paper must be typed and double spaced. If you use a word processor, please use a letter-quality printer, or set your dot-matrix printer to double strike. If you use sprocket-feed paper, be sure the pages are trimmed and burst.
  3. I am not amused when people tinker with spacing and fonts to make papers seem longer. Please use one-inch margins all the way around your paper. It should be double-spaced throughout: DO NOT quadruple space between paragraphs. Use a 10 or 12 pint font (what you're reading is 12 point) when you print.
  4. You do not need a cover sheet or any type of binder. Please type your name and the date at the top of the first page. Number the subsequent pages, and STAPLE them together. I will not accept unstapled papers.
  5. I will consider your paper in terms of both its content and form – that is, does it have something to say, and does it say it more or less grammatically and according to the conventions of usage and mechanics? If you have trouble with any aspect of the paper (developing ideas, organizing them, mechanics, etc.) please let me know so that I can help you. Again, if my office hours are inconvenient, just make an appointment to see me at some other time. In addition, remember that you can make an appointment at the Writing Center.

You must complete a draft by February 18 (no extensions) and submit your paper in class on February 25 (extensions only with advance notice and ad verified reason). Good luck.

Assignment for HIST 120, section 5

History 120

Instructions for Term Paper, Fall 1998

REFER TO PAGE 1 OF YOUR SYLLABUS FOR DUE DATES.

Paper Topic

This semester the term paper will answer the question: How did slaves maintain their self-esteem inside slavery? IMPORTANT NOTE: A PAPER ON ANY OTHER TOPIC WILL NOT BE ACCEPTED. The material you use for the topic muse come from your reading of EITHER Frederick Douglass' OR Linda Brent's narrative of his/her life as a slave. Both narratives are found in the Gates edition.

Format and other Mechanical Matters

Your paper must:

  1. be 1000-1500 words long
  2. be neatly typed with DARK print
  3. be double space
  4. be CAREFULLY proofread
  5. have a title page with
    • a title
    • your name
    • the name of the class
    • the section of the class
    • my name
    • the date
  6. have an outline on an extra page JUST AFTER the title page (see Outline Format)
  7. give references for all quotes, paraphrases, and ideas from the narrative you have chosen (see Sample References, page 2)

What makes a Good Paper?

  1. A PAPER makes an argument, in convinces the reader that the main point is true. A paper is NOT the same thing as a book report. (A book report is just a summary of the book.)
  2. It has just ONE main pint stated in the thesis statement of the first paragraph. (See Thesis Statement and Supporting Arguments.)
  3. It has a conclusion that relates to the main point and is clearly stated in the last paragraph.
  4. The topic sentences of each in-between paragraph give evidence to support the main point. Each paragraph has one topic sentence followed by sentences that support the topic sentence of that paragraph.
  5. It uses straightforward language designed to communicate, not to impress.

Thesis Statement and Supporting Arguments

What makes a good thesis statement?

  1. It is an idea you can PROVE.
  2. It is clearly stated.
  3. It is interesting.
  4. It is something you have material for.

What makes for good supporting arguments?

  1. They give evidence to support the thesis statement.
  2. They progress logically to the conclusion.
  3. You have evidence to support them.

Outline Format

Your Thesis Statement (in a complete sentence)

  1. First Supporting Point (in a complete sentence)
  2. Second Supporting Point (in a complete sentence)
  3. Third Supporting Point (in a complete sentence)
  4. Fourth Supporting Point (in a complete sentence)
  5. Etc.

Your Conclusion (in a complete sentence)

Sample References

You must provide endnotes (footnotes placed at the end of your paper) for all your references. In your text, place a raised number after the sentence or paragraph you want to give a reference for. Begin with the number 1 and number the rest of your footnotes sequentially. A good term paper will have at least 10 endnotes. Endnotes must have the following format:

First references:

1.  Frederick Douglass, Narrative of the Life of Frederick Douglass in The Classic Slave Narratives, ed. Henry Louis Gates (New York: Penguin Books, 1987) p. XXX.

Subsequent references:

2. Douglass, p. XXX

You must give notes for

  1. Content and ideas which come from the autobiography you choose to read
  2. Paraphrases from the autobiography
  3. Exact quotes from the autobiography

Short quotes (less than three lines) and long quotes (three lines or more) are handled differently.

Short quotes are set off by quote marks and are simply imbedded in the double-spaced text.

Long quotes are NOT surrounded by quote marks, but they must be indented and single-spaced.

Publication Information: Quesenberry, Legene, et al. (2000). Assessment of the Writing Component within a University General Education Program. academic.writing. https://wac.colostate.edu/aw/articles/quesenberry2000/
Publication Date: November 1, 2000


Contact Information:
Legene Quesenberry's Email: LQuesenberry@mail.clarion.edu

Copyright © 2000 Legene Quesenberry et al. Used with Permission.