Banca de DEFESA: Laura Marcia Luiza Ferreira

Evaluation of oral proficiency: a factorial analysis and discrimination of Celpe-Bras items.

second language assessment, Celpe-Bras, factorial analyses, Rasch

Since sistematic errors may affect scores, assessments are imperfect samples of
constructs to be validated. According to Messick’s Validation Theory (1987), assessment validation
is a matter of gathering multiple sources of empirical evidence and theoretical rationales to support
the inferences that will be made from the tests results. In order to gather evidences to discuss
dimensionality, item fit and information on the oral Celpe-Bras exam scales, I perform a
exploratory factor analysis and an item discrimination analyses, using Rasch’s Partial Credit
Model. First, I describe the oral exam task as an oral proficiency interview. Then I discuss the
Messick’s Validation Theory and it’s influence on applied linguistics and on second language
assessment theoretical works. It was analysed 1.000 examinee’s scores from the first exam edition
of 2016. The oral Celpe-Bras‘s score model is organized in seven itens in two scales: one rated by
the interviewer and six itens by the rater itself. The interviewer scores one single note for the
performance while the rater scores one note for the following itens comprehension,
interactional competence, fluency, lexical adequacy, grammatical adequacy and pronunciation.
After running the exploratory factor analysis, we concluded that the test score is an onedimensional
measure. According to the weight values, the itens that weighed the most on the fator
oral proficiency were the interviewer's score (0.36), followed by lexical adequacy (0.19), fluency
(0.18) and grammatical adequacy (0.13). Interactional competence, pronunciation and
comprehension weighted 0.09, 0.06 and 0.04, respectively. According to Rasch’s analyses, the infit
msq values for the itens was reasonable (0.431 to 1.386). However, outfit msq value for the item
comprehension was >2.0, sugesting poor fit. Since the threshold’s value categories were not
homogeneous through the scales, the itens may not distinguish in the same way examinees from
the same profile ability. Finally, I discuss a score’s composite proposal, the relevance of iten
comprehension in this oral situation as well as the investment on descriptor’s and task’s review.