Language Learning & Technology http://llt.msu.edu/vol12num2/chencheng/
June 2008, Volume 12, Number 2 pp. 94-112
BEYOND THE DESIGN OF AUTOMATED WRITING EVALUATION: PEDAGOGICAL PRACTICES AND PERCEIVED LEARNING EFFECTIVENESS IN EFL WRITING CLASSES Chi-Fen Emily Chen and Wei-Yuan Eugene Cheng National Kaohsiung First University of Science and Technology, Taiwan Automated writing evaluation (AWE) software is designed to provide instant computergenerated scores for a submitted essay along with diagnostic feedback. Most studies on AWE have been conducted on psychometric evaluations of its validity; however, studies on how effectively AWE is used in writing classes as a pedagogical tool are limited. This study employs a naturalistic classroom-based approach to explore the interaction between how an AWE program, MY Access!, was implemented in three different ways in three EFL college writing classes in Taiwan and how students perceived its effectiveness in improving writing. The findings show that, although the implementation of AWE was not in general perceived very positively by the three classes, it was perceived comparatively more favorably when the program was used to facilitate students’ early drafting and revising process, followed by human feedback from both the teacher and peers during the later process. This study also reveals that the autonomous use of AWE as a surrogate writing coach with minimal human facilitation caused frustration to students and limited their learning of writing. In addition, teachers’ attitudes toward AWE use and their technology-use skills, as well as students’ learner characteristics and goals for learning to write, may also play vital roles in determining the effectiveness of AWE. With limitations inherent in the design of AWE technology, language teachers need to be more critically aware that the implementation of AWE requires well thought-out pedagogical designs and thorough considerations for its relevance to the objectives of the learning of writing. INTRODUCTION Automated writing evaluation (AWE), also referred to as automated essay scoring (AES)1, is not a brandnew technology in the twenty-first century; rather, it has been under development since the 1960s. This technology was originally designed to reduce the heavy load of grading a large number of student essays and to save time in the grading process. Early AWE programs, such as Project Essay Grade, employed simple style analyses of surface linguistic features of a text to evaluate writing quality (Page, 2003). Since the mid-1990s, the design of AWE programs has been improving rapidly due to the advance of artificial intelligence technology, in particular natural language processing and intelligent language tutoring systems. Newly developed AWE programs, such as Criterion with the essay scoring engine "e-rater" by Educational Testing Service and MY Access! with the essay scoring engine "Intellimetric" by Vantage Learning, boast the ability to conduct more sophisticated analyses including lexical complexity, syntactic variety, discourse structures, grammatical usage, word choice, and content development. They provide immediate scores along with diagnostic feedback in various aspects of writing and can be used for both formative and summative assessment purposes. In addition, a number of AWE programs are now webbased and equipped with a variety of online writing resources (e.g., thesauri and word banks) and editing features (e.g., grammar, spelling, and style checkers), which make them not only an essay assessment tool but also a writing assistance tool. Students can make use of both AWE’s assessment and assistance functions to help them write and revise their essays in a self-regulated learning environment. Although AWE developers claim that their programs are able to assess and respond to student writing as human readers do (e.g., Attali & Burstein, 2006; Vantage Learning, 2007), critics of AWE express strong skepticism. Voices...
Please join StudyMode to read the full document