A Categorical Instrument for Scoring Second Language Writing Skills
A CATEGORICAL INSTRUMENT FOR SCORING SECOND LANGUAGE WRITING SKILLS James Dean Brown Florida State University Kathleen M. Bailey Monterey Institute of International Studies
This paper first reviews recent research on evaluating second language learners’ writing skills. It then discusses research on a categorical instrument for evaluating compositions written by upper intermediate university ESL students. The form of the instrument used in this study included five equally weighted criteria for scoring: ( I ) Organization, (2) Logical Development of Ideas, ( 3 )Grammar, (4) Mechanics, and ( 5 ) Style. An experiment was conducted under controlled conditions in which ten raters scored fifty randomly selected compositions. Regression analysis and generalizability theory were used for investigating the reliability of the instrument. In addition, information was obtained from the raters as t o their reactions to the instrument. The results indicate that the scoring instrument is moderately reliable. More useful, perhaps, is the demonstration provided of the effects on reliability of changing the number of raters a n d / o r criteria. The paper concludes with a discussion of the issues involved (both with this particular instrument and with analytic scoring systems in general) and of possible future research on the evaluation of nonnative speakers’ writing.
Because of the current widely acknowledged crisis in college students’ writing (e.g., Kinneavy 1981:65; Stiggins 1982:155), the issue of evaluating students’ composition skills is and will continue to be important in university programs. The question of global ratings versus more objective scoring methods has been debated in the teaching of writing to native speakers of English, particularly in freshman composition programs. Various authors have described writing programs and evaluation procedures (e.g., Humes 1981; Cronnell and Michael 1982). Others have conducted research on the evaluation of students’
This paper first reviews recent research on evaluating second language learners’ writing skills. It then discusses research on a categorical instrument for evaluating compositions written by upper intermediate university ESL students. The form of the instrument used in this study included five equally weighted criteria for scoring: ( I ) Organization, (2) Logical Development of Ideas, ( 3 )Grammar, (4) Mechanics, and ( 5 ) Style. An experiment was conducted under controlled conditions in which ten raters scored fifty randomly selected compositions. Regression analysis and generalizability theory were used for investigating the reliability of the instrument. In addition, information was obtained from the raters as t o their reactions to the instrument. The results indicate that the scoring instrument is moderately reliable. More useful, perhaps, is the demonstration provided of the effects on reliability of changing the number of raters a n d / o r criteria. The paper concludes with a discussion of the issues involved (both with this particular instrument and with analytic scoring systems in general) and of possible future research on the evaluation of nonnative speakers’ writing.
Because of the current widely acknowledged crisis in college students’ writing (e.g., Kinneavy 1981:65; Stiggins 1982:155), the issue of evaluating students’ composition skills is and will continue to be important in university programs. The question of global ratings versus more objective scoring methods has been debated in the teaching of writing to native speakers of English, particularly in freshman composition programs. Various authors have described writing programs and evaluation procedures (e.g., Humes 1981; Cronnell and Michael 1982). Others have conducted research on the evaluation of students’