Developing a scoring rubric for a course-specific pair work telephoning speaking assessment using intuitive and data-based methods
Fadi Tannouri
Myongji University, Seoul South Korea
PP: 072 - 102
Abstract
This paper focuses on the process of developing a scoring rubric for pair work speaking assessments in context-specific environments. At the a priori stage of development, two types of telephoning task, namely (i) taking and leaving messages and (ii) scheduling appointments, were selected and standardized, and their expected responses and functions were identified in order to determine potential criteria for inclusion in the scoring rubric.
The potential criteria were matched with existing scoring rubrics in order to develop a scoring rubric template intuitively. At the a posteriori stage of development, twenty-six participants who were recruited through informed consent provided twenty-six audio recordings of pair work speaking tasks, which were analyzed in order to verify and refine the scoring rubric template empirically. The refined scoring rubric was trialed over two scoring sessions by two teachers with the results showing that the scoring rubric requires further modifications before it is operational.
Overall, the scoring rubric offers a workable starting point for the implementation of a standardized scoring rubric for the telephoning speaking assessment, but the criteria categories and the bands of performance may require further delineation through more intuitive and data-based methods. Furthermore, the scoring rubric development process could benefit from more rigorous think-aloud sessions as well as more comprehensive standardization and rater training procedures.
Keywords
speaking assessments, pair work tasks, scoring rubrics, construct validation, intuitive and data-based validation methods
Article Text
In the telephoning module of the university's Business English course, the goals are to develop the students' abilities to conduct telephone conversations on two broad types of phone call: (i) taking and leaving messages, and (ii) scheduling appointments.
To achieve these goals, the teachers focus on promoting proficiency in three key areas, namely the following: (i) fluency, (ii) accuracy, and (iii) task. (These issues are addressed in greater detail further below.) The telephoning module accounts for five weeks of the sixteen-week semester, with each week containing three hours of in-class instruction, divided into two ninety-minute classes. Therefore, the telephoning component is a fifteen-hour module, and its goals are modest.
... continues ...
References
Bachman LF (1990) Fundamental considerations in language testing. Oxford: Oxford University Press.
Brown HD (2004) Language assessment: principles and classroom practices. White Plains, New York: Pearson Education, Inc.
Brown JD (1988) Understanding research in second language learning. Cambridge: Cambridge University Press.
Chalhoub-Deville M (2003) Second language interaction: current perspectives and future trends. Language Testing 20(4), 369-383. Retrieved October 5, 2008, from EbscoHost database.
de Jong JHAL and Bernstein J (nd.) Relating PhonePass overall scores to the Council of Europe framework level descriptors. Retrieved November 25 2008 from http://www.ordinate.com/pdf/EuroSpeech.pdf.
Donna S (2000) Teach business English. Cambridge: Cambridge University Press.
Ellis NC (1997) Vocabulary acquisition: word structure, collocation, word-class, and meaning. In Vocabulary: description, acquisition and pedagogy. Cambridge: Cambridge University Press, pp.122-139.
Fulcher G (2003) Testing second language speaking. London: Pearson Education Limited.
Hamp-Lyons L (2007) Scoring language tests and assessments. In Fulcher G and Davidson F (Eds) Language testing and assessment: an advanced resource book, pp.91-114. New York: Routledge.
He L and Dai Y (2006) A corpus-based investigation into the validity of the CET-SET group discussion. Language Testing 23(3), 370-401. Retrieved October 5, 2008, from EbscoHost database.
Hughes A (2003) Testing for language teachers 2nd Edn. Cambridge: Cambridge University Press.
Hughes R (2002) Teaching and researching speaking. Harlow, Great Britain: Pearson Education.
Iwashita N, Brown A, McNamara T and O'Hagan S (2008) Assessed levels of second language speaking proficiency: how distinct? Applied Linguistics 29(1), 24-49. Retrieved September 27 2008 from http://applij.oxfordjournals.org/cgi/reprint/29/1/24.
Kormos J (1999) Simulating conversations in oral-proficiency assessment: a conversation analysis of role plays and non-scripted interviews in language exams. Language Testing 16(2), 163-188. Retrieved October 5 2008 from EbscoHost database.
Lee YW (2006) Dependability of scores for a new ESL speaking assessment consisting of integrated and independent tasks. Language Testing, Apr 2006 23(2), 131-166. Retrieved October 5 2008 from EbscoHost database.
Little D (1994) Words and their properties: arguments for a lexical approach to pedagogical grammar. In Terence Odlin (Ed.), Perspectives on pedagogical grammar (pp.99-122). Cambridge: Cambridge University Press.
Luoma S (2004) Assessing speaking. New York: Cambridge University Press.
Lynch BK (2003) Language assessment and program evaluation. Edinburgh: Edinburgh University Press Ltd.
Madsen HS (1983) Techniques in testing. Oxford: Oxford University Press.
McCarthy M (1990) Vocabulary. Oxford: Oxford University Press.
McCarthy M (1991) Discourse analysis for language teachers. Cambridge: Cambridge University Press.
McNamara TF (1997) 'Interaction' in second language performance assessment: whose performance? Applied Linguistics 18(4), 446-466. Retrieved October 5 2008 from EbscoHost database.
North B (2000) The development of a common framework scale of language proficiency. New York: Peter Lang Publishing, Inc.
North B and Schneider G (1998) Scaling descriptors for language proficiency scales. Language Testing, Jul 98, 15(2), 217-263. Retrieved September 27 2008 from EbscoHost database.
Nunan D (1989) Designing tasks for the communicative classroom. Cambridge: Cambridge University Press.
Nunan D (1992) Research methods in language learning. Cambridge: Cambridge University Press.
Nunan D (1998) Language teaching methodology. Essex, England: Pearson Education Limited.
O'Sullivan B, Weir CJ and Saville N (2002) Using observation checklists to validate speaking-test tasks. Language Testing, Jan 2002 19(1), 33-56. Retrieved September 27 2008 from EbscoHost database.
Roach P (2000) English phonetics and phonology: a practical course 3rd Edn. Cambridge: Cambridge University Press.
Sawaki Y (2007) Construct validation of analytic rating scales in a speaking assessment: reporting a score profile and a composite. Language Testing, Jul 2007, 24(3), 355-390. Retrieved September 27 2008 from EbscoHost database.
Swain M (2001) Examining dialogue: another approach to content specification and to validating inferences drawn from test scores. Language Testing, Jul 2001, 18(3), 275-302, Retrieved October 5 2008 from EbscoHost database.
Sweeney S (2000) Communicating in business: a short course for business English students. Cambridge: Cambridge University Press.
Underhill N (1987) Testing spoken language: a handbook of oral testing techniques. Cambridge: Cambridge University Press.
Upshur JA and Turner CE (1999) Systematic effects in the rating of second-language speaking ability: test method and learner discourse. Language Testing, 1999 16(1), 82-111. Retrieved October 5 2008 from EbscoHost database.
Walters FS (2007) A conversation-analytic hermeneutic rating protocol to assess L2 oral pragmatic competence. Language Testing 2007 24(2), 155-183. Retrieved October 5 2008 from EbscoHost database.
Weigle SC (2002) Assessing writing. Cambridge: Cambridge University Press.
Weir CJ (2005) Language testing and validation: an evidence-based approach. New York, New York: Palgrave MacMillan.
Xi X (2007) Evaluating analytic scoring for the TOEFL® Academic Speaking Test (TAST) for operational use. Language Testing, Apr 2007, 24(2), 251-286. Retrieved September 27 2008 from www.ets.org.
eContent Home
