Campus Assessment of Student Learning Outcomes

Writing Assessment Results 2009

Assessment of Writing Skills among Beginning and Advanced Students
IU Northwest General Education / Assessment Committee
May 12, 2009
Report Date: July 8, 2009

Participating Committee Members:  Mary Ann Fischer (co-Chair), Doug Swartz, Barbara Peat, Latrice Booker, Sheila Marie Trzcinka, Linda Delunas, Karen Evans, Dee Dee Ige, Jim Thomas, Eva Mendieta, Cynthia O'Dell, Taylor Lake

Introduction

During May of 2009, the combined General Education / Assessment Committee of the IU Northwest Faculty Organization completed an assessment of writing skills revealed in a sample of forty six final papers selected from either W131, Elementary Composition, or from upper division courses in each of the six schools and colleges. This report details the procedures used the solicit the papers, to review and evaluate them, and the results as revealed in quantitative judgments and comments of the faculty evaluators who served on the committee. We hoped to establish a baseline of evidence for writing skills and to find evidence of more advanced skills  among upper division papers than among W131 papers.

Selection of Writing Samples

The Associate Vice Chancellor of Academic Affairs requested that the Deans of the  schools and colleges solicit sample papers from their faculty. The request was for at least two papers along with descriptions of the assignment,  from a given instructor, one judged by the instructor as relatively skilled and the other as relatively weak. All identifiers except the course number were stripped from the papers before they were submitted to the AVCAA. The final sample included 23 papers drawn from W131 sections and 23 papers from upper division courses. All schools and colleges submitted at least 2 papers from upper division courses.

Assessment Procedures

The committee adapted a tool designed by Professor Anne Balay, Department of English. It included eight categories, including definitions, of writing skills: research, persuasion, prose and syntax, counterargument, mechanics, complexity, reader pleasure, and organization, (See attachment). Each evaluator read three or four papers and judged them as excellent (3 points), acceptable (2 points), or unacceptable (1 point) in each category; additional open-ended comments were optional. Each paper was evaluated by only one rater; thus no inter-rater reliability can be reported.

Results

The mean rating for each skill category among W131 and upper division papers is shown in the table below. As expected, in every category upper division papers were given higher ratings than those from W131. Readers knew the course level of the paper,  so expectations for better performance on the upper division papers may have influenced ratings. Future assessments should remove the course identifiers. Ratings for the W131 papers were low, that is, the means were below 2, defined as “Acceptable.” Mean ratings for the upper division papers were just above “Acceptable” in all but 2 categories.

The comments (see attachment 2) about the papers were much more negative than the mean ratings would have suggested in the case of the upper division papers.  It may be that raters hesitated to classify the upper division papers us “Unacceptable”   even though they were highly critical.  If so, future assessment efforts should use a rating scale with a wider range.

In general, evaluators found the experience of reading student papers valuable and useful.  The committee did not discuss the question of what steps should be taken to improve the quality of student writing at that time.  However, we agreed to take up this issue at the next General Education / Assessment Committee meeting in the Fall of 2009. Comments and suggestions on the process included the following:

1. Our assignment instructions should be clearer to the units to ensure more conformity to the assignment.
2. Rubric should be more universal to any assignment; suggested areas of assessment include:  focus, development, mechanics, philosophy.
3. Sheila Marie T. brought a rubric that we might consider for next time rather than the one we used this time?
4. Suggestion that everyone, including faculty, have a Handbook (perhaps purchased by the university) that is used in most w130 or w131 classes to encourage students to keep them and use them in all future classes.

Attachment One Writing Assessment Form
General Education / Assessment Committee Assessment Project
May 12, 2009

Reader __________________                                                Paper #

This list of desired attributes has been adapted from one designed by Anne G. Balay , although she strongly favors holistic assessment rather than evaluating attributes separately, as we will be doing here. She has argued that after a student has completed one year of a college-level writing course, a final essay should demonstrate most of these attributes. After reading each essay please use the definitions which have been provided to rate each sample paper for each attribute.

Total points: ______________

Additional comments:

Definitions of the categories:

Use of research: Students should execute both library and Internet research in gathering sources for their projects.  Sources appearing in student papers should be drawn from electronic databases, and contain a combination of academic, public, and primary sources. Students are expected to demonstrate an ability to summarize, analyze, and synthesize sources appropriate to academic writing.

Effectiveness of argument or persuasion: Students are expected to establish a research driven thesis supported by warranted claims that shaped their purpose in writing. Claims are supported by evidence gleaned from research, primarily, as well as from personal experience and common knowledge. 

Style reflecting prose and syntax: At the surface level, the writing should be clear. Sentences should reflect a degree of sophistication appropriate to a college writer and should flow in prose that sustains the argument.

Use of counterargument: The essay should include a counterargument as reflecting a writer’s sense of ethics and fair-mindedness in working a thesis. Furthermore, a writer should guide the argument without undue bias, acknowledging opposing opinions and points of view.

Attention to mechanics and punctuation: Writing should be clear and free of errors. The writer should demonstrate mechanics that reflect the grammar of Standard American English, free of slang and jargon, and should attend to the impact of punctuation on the reader.

Complexity of ideas: Papers should reflect thoughtful interpretations and applications of research (considered critical thinking). As such, successful papers avoid pedestrian arguments on mundane subjects.

Reader pleasure: A good essay interests its writer as well as its reader, and is fresh in ideas and well written.

Structure reflecting organization and focus:  Deep issues of organization and development define this category. A paper’s thesis and claims should guide the reading, and those claims should be developed through extended discussions and use of research as support.

Attachment 2 Comments on papers