A CATEGORICAL INSTRUMENT FOR SCORING SECOND LANGUAGE WRITING SKILLS James Dean Brown Florida State University Kathleen M. Bailey Monterey Institute of International Studies This paper first reviews recent research on evaluating second language learners’ writing skills. It then discusses research on a categorical instrument for evaluating compositions written by upper intermediate university ESL students. The form of the instrument used in this study included five equally weighted criteria for scoring: ( I ) Organization, (2) Logical Development of Ideas, ( 3 )Grammar, (4) Mechanics, and ( 5 ) Style. An experiment was conducted under controlled conditions in which ten raters scored fifty randomly selected compositions. Regression analysis and generalizability theory were used for investigating the reliability of the instrument. In addition, information was obtained from the raters as t o their reactions to the instrument. The results indicate that the scoring instrument is moderately reliable. More useful, perhaps, is the demonstration provided of the effects on reliability of changing the number of raters a n d / o r criteria. The paper concludes with a discussion of the issues involved (both with this particular instrument and with analytic scoring systems in general) and of possible future research on the evaluation of nonnative speakers’ writing.
Because of the current widely acknowledged crisis in college students’ writing (e.g., Kinneavy 1981:65; Stiggins 1982:155), the issue of evaluating students’ composition skills is and will continue to be important in university programs. The question of global ratings versus more objective scoring methods has been debated in the teaching of writing to native speakers of English, particularly in freshman composition programs. Various authors have described writing programs and evaluation procedures (e.g., Humes 1981; Cronnell and Michael 1982). Others have conducted research on the evaluation of students’ compositions. (See the Journal of Basic Writing 4: Evaluation, which includes an annotated bibliography on the topic compiled by Larson 1978.)
‘This article is a much revised version of a paper presented at the CATESOL State Conference 1980 at San Diego, California. The authors wish to thank Noreen Webb and Bill Gaskill for their constructive criticism of the manuscript, and Frances Hinofotis for her help in the data collection process, as well as Jennifer Costanza and Susan Spice for word processing assistance.
Vol. 34, No. 4
In a discussion of evaluating writing proficiency, Stiggins ( I 982) describes three major approaches t o scoring writing samples. The first is the familiar holistic scoring, in which each composition is assigned a unitary rating, often on a four- to eight-point scale. The second main approach to assessing writing proficiency is analytic scoring, which involves “isolating one or more characteristics of writing and scoring them individually” (p. 148). Such characteristics may include style, grammar, mechanics, etc. The third method is primary trait scoring, which is “rhetorically and situationally specific”(p. 148; also see Mullis 1974) in that compositions are evaluated in terms of a particular trait, or traits, determined by the desired effect on the intended audience. For instance, workbook instructions for conducting a chemistry experiment must be clear and sequential, so a primary trait analysis of such written discourse might entail ratings on chronological order and clarity of referents. It should be noted that all three evaluation systems discussed by Stiggins involve assigning numerical scores on the basis of various criteria, which may range from global quality t o assessed effectiveness on more specific categories.
RESEARCH ON EVALUATING ESL WRITING SAMPLES
Recently, concerned teachers and researchers have also begun to investigate the problems involved in evaluating second language...
Please join StudyMode to read the full document