Protocol analysis in the validation of language tests: Potential of the method, state of the evidence
Ying Zheng
Faculty of Education, Queen’s University, Canada
PP: 124 - 137
Abstract
This paper is primarily a methodological consideration, which critically reviews the literature on using protocol analysis to understand cognitive processes that students engage in while writing language tests. For the purpose of synthesizing the applicability and significance of protocol analysis in validating language tests, the author first discusses the potential of the concepts including validity, language test validation, and protocol analysis.
Terminological classifications of different approaches of test validation are elaborated. The author then critically examines the state of evidence by interpreting studies that employed protocol analysis in language tests.
This paper is intended to stimulate discussion on the merits and inadequacies of protocol analysis in different ESL/EFL tests, especially when the tests are designed for test-takers with culturally different backgrounds. Possible future research trends are suggested.
Keywords
protocol analysis, test validation, concurrent protocol, retrospective protocol
Article Text
For test developers and educators looking for ways to ensure a trustworthy language test, one enduring approach has been to focus on the reliability and validity of the test. Three main phases of the history of language testing have been identified: the pre-scientific or traditional phase, which was based on the subjective judgment of one examiner; the psychometric-structuralist or modern phase, which made an effort to optimize objectivity and reliability; and the psycholinguistic or post-modern phase, which evolved to address not only objectivity and reliability, but also validity (Grotjahn, 1986).
Tests are used to perform many social or educational functions (e.g., selection, competition, diagnosis). However, in many circumstances, there is a lack of information on whether tests provide adequate validity evidences, or whether we should have faith in test results. Seeking results that might inform test developers' construction of valid tests, researchers have undertaken studies and written methodological syntheses that focus on many different aspects of validating tests. Some have focused on quantitative approaches to test validation, examining statistical features of the test items. For instance, they have examined how test items function differently among test-takers from diverse backgrounds, or how students respond to test items differently using Differential Item Functioning (DIF) or Item Response Theory (IRT).
The present paper targets a review on a qualitative means of examining test validity using protocol analysis. In understanding the test constructs and test performance, the primary focus is usually on the relationship between the two; the cognition process that is situated in between these two ends of the continuum, however, is an aspect that cannot be overlooked. In response to this situation, a desire to understand the cognitive processes that mediate the test construct and test performance is the rationale of employing protocol analysis to validate tests. By conducting protocol analysis, it is assumed that a better understanding of what is going on in students' thinking or reasoning while they are completing the tests can be achieved. Protocol analysis can also provide a useful vehicle for comparing what the test constructs are designed to measure and what the students who write the test perceive of the test constructs. To examine the applicability and significance of the above assumptions, the present paper aims to conduct a review of the literature on the using of protocol analysis in language test validation studies. The primary focus is on methodological considerations to see if this validation approach can accomplish what it is expected/assumed to do. To support this focus, attention is also paid to what has been done in using protocol analysis to validate language tests, and what can be proposed in future research to integrate this research approach into specific research contexts.
Findings about the use of protocol analysis in language test validation are applicable to the discussions of gaining a better understanding of different students' cognitive processes when they are taking a test. Knowledge about students' information processing in writing a test, especially students from different languages, cultures, or proficiency levels, may have important implications with regards to test fairness and test bias studies.
To organize this review, the review approaches are introduced, including my criteria for including studies in the review. Conceptual discussions of validity, means of validating tests, and a definition of protocol analysis are presented, followed by my synthesis of the interpretations of some selected empirical studies. The merits and inadequacies of protocol analysis in validating language tests are presented. A discussion of controversial issues regarding the use of protocol analysis to validate language tests is provided. The review ends with possible future trends.
... continues ...
References
Alavi, S. M. (2005). On the adequacy of verbal protocols in examining an underlying construct of a test. Studies in Educational Evaluation, 31, 1-26.
Anastasi, A. (1986). Evolving concepts of test validation. Annual Review of Psychology, 37, 1-16.
Anderson, N. J., Bachman, L., Perkins, K., & Cohen, A. (1991). An exploratory study into the construct validity of a reading comprehension test: Triangulation of data sources. Language Testing, 8(1), 41-66.
Angoff, W. (1988). Validity: An evolving concept. In H. Wainer & H. Braun (Eds.), Test validity, pp.19-32. Hillsdale, NJ: Erlbaum.
Banerjee, J., & Luoma, S. (1997). Qualitative approaches to test validation. In C. Clapham & D. Corson (Eds.), Encyclopedia of language and education: Language testing and assessment, 7, 275-287. Dordrecht: Kluwer Academic.
Clapham, C. (1998). The effect of language proficiency and background knowledge on EAP students' reading comprehension. In A. J. Kunnan (Ed.), Validation in language assessment: Selected papers from the 17th language testing and research colloquium, pp.141-168. Mahwah, NJ: Erlbaum.
Cohen, A. D. (1984). On taking language tests: What the students report. Language Testing, 1, 70-81.
Cohen, A. D. (1987). Using verbal reports in research on language learning. In C. Faerch & G. Kasper (Eds.), Introspection in second language research, pp.82-95. Clevedon, UK: Multiligual.
Cohen, A. D. (1997). Toward enhancing verbal reports as a source of insights on test-taking strategies. In A. Huhta, V. Kohonen, L. Kurki-Suonio & S. Luoma (Eds.), Current developments and alternatives in language assessment, pp.339-366. Kopijyva, Jyva¨skyla: University of Jyva¨skyla.
Crain-Thoreson, C., Lippman, M. Z., & McClendon-Magnuson, D. (1997). Windows on comprehension: Reading comprehension processes as revealed by two think-aloud procedures. Journal of Educational Psychology, 89, 579-591.
Cumming, A. (1996). Introduction: The concept of validation in language testing. In A. Cumming & R. Berwick (Eds.), Validation in language testing, pp.1-13. Clevedon, UK: Multilingual.
Davies, A., Brown, A., Elder, C., Hill, K., Lumley, T., & McNamara, T. (1999). Dictionary of language testing. Cambridge: University of Cambridge Local Examinations Syndicate and Cambridge University Press.
Duran, R. P. (1989). Testing of linguistic minorities. In R. L. Linn (Ed.), Educational measurement (3rd ed.), pp.573-587. New York: Macmillan.
Ericsson, K. A., & Simon, H. A. (1987). Verbal reports on thinking. In C. Faerch & G. Kasper (Eds.), Introspection in second language research, pp.24-53. Clevedon, UK: Multilingual.
Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: The MIT Press.
Farr, R., Pritchard, R., & Smitten, B. (1990). A description of what happens when an examinee takes a multiple-choice reading comprehension test. Journal of Educational Measurement, 27, 209-226.
Garner, R. (1988). Verbal report data on cognitive and metacognitive strategies. In C. E. Weinstein & E. T. Goetz (Eds.), Learning and study strategies: Issues in Assessment, instruction, and evaluation, pp.63-76. London: Academic.
Ginther, A., & Stevens, J. (1998). Language background, ethnicity, and the internal construct validity of the advanced placement Spanish language examination. In A. J. Kunnan (Ed.), Validation in language assessment: Selected papers from the 17th language testing and research colloquium, pp.169-194. Mahwah, NJ: Erlbaum.
Green, A. (1998). Verbal protocol analysis in language testing research: A handbook. Cambridge: University of Cambridge Local Examinations Syndicate and Cambridge University Press.
Grotjahn, R. (1986). Test validation and cognitive psychology: Some methodological considerations. Language Testing, 3, 159-185.
Grotjahn, R. (1987). On the methodological basis of introspective methods. In C. Faerch & G. Kasper (Eds.), Introspection in second language research, pp.54-81. Clevedon, UK: Multilingual.
Hamiliton, L. S., Nussbaum, E. M., & Snow, R. E. (1997). Interviewing procedures for validating science assessments. Applied Measurement in Education, 10(2), 181-200.
Hamilton, L. S. (1994). Validating hands-on science assessments through an investigation of response processes. ERIC document reproduction service, ED 376 202.
Kunnan, A. J. (1997). Connecting fairness with validation in language assessment. In A. Huhta, V. Kohonen, L. Kurki-Suonio & S. Luoma (Eds.), Current developments and alternatives in language assessment, pp.85-106. Jyva¨skyla: University of Jyva¨skyla.
Kunnan, A. J. (1998). Approaches to validation in language assessment. In A. J. Kunnan (Ed.), Validation in language assessment, pp.1-15. Mahwah, NJ: Erlbaum.
Kuusela, H., & Paul, P. (2000). A comparison of concurrent and retrospective verbal protocol analysis. The American Journal of Psychology, 113, 387-404.
McNamara, T. (2000). Language testing. Oxford, UK: Oxford University Press.
Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed.), pp.13-103. New York: Macmillan.
Nevo, N. (1989). Test-taking strategies on a multiple-choice test of reading comprehension. Language Testing, 6(2), 199-255.
Norris, S. P. (1991). Informal reasoning assessment: Using verbal reports of thinking to improve multiple-choice test validity. In J. F. Voss, D. N. Perkins & J. W. Segal (Eds.), Informal reasoning and education, pp.451-471. Hillsdale, NJ: Erlbaum.
Ortiz, S. O. (2005). Acculturational factors in psychological assessment. In R. L. Rhodes, S. H. Ochoa & S. O. Ortiz (Eds.), Assessing culturally and linguistically diverse students: A practical guide, pp.124-136. New York, NY: Guilford.
Parry, K. J. (1987). Reading in a second culture. In J. Devine, P. L. Carrell & D. E. Eskey (Eds.), Research in reading in English as a second language, pp.59-70. Washington DC: Teachers of English to speakers of other language.
Phakiti, A. (2003). A closer look at the relationship of cognitive and metacognitive strategy use to EFL reading achievement test performance. Language Testing, 20(1), 26-56.
Pritchard, R. (1990). The effects of cultural schemata on reading processing strategies. Reading Research Quarterly, 25, 273-295.
Sasaki, M. (2000). Effects of cultural schemata on students' test-taking processes for cloze tests: A multiple data source approach. Language Testing, 17(1), 85-114.
Sepassi, F. (2003). How do learners of different language ability perform on the cloze? A verbal protocol analysis of EFL test takers performance on cloze tests. Indian Journal of Applied Linguistics, 29(2), 5-33.
Solano-Flores, G., & Nelson-Barber, S. (2001). On the cultural validity of science assessments. Journal of Research in Science Teaching, 38(5), 553-573.
Steffensen, M. S., & Joeg-Dev, C. (1984). Cultural knowledge and reading. In J. C. Alderson & A. H. Urquhart (Eds.), Reading in a foreign language. New York: Longman Group Ltd.
Storey, P. (1997). Examining the test-taking process: A cognitive perspective on the discourse cloze test. Language Testing, 14(2), 214-231.
Weir. C. J. (2005). Language testing and validation: An evidence-based approach. New York, NY: MacMillan.
Wijgh, I. F. (1996). A communicative test in analysis: Strategies in reading authentic texts. In A. Cumming & R. Berwick (Eds.), Validation in language testing, pp.154-165. Clevedon, UK: Multilingual.
Wu, Y. A. (1998). What do tests of listening comprehension test? - A retrospection study of EFL test-takers performing a multiple-choice task. Language Testing, 15(1), 21-44.
eContent Home
