Banca de DEFESA: LILIANE DE OLIVEIRA NEVES

Uma banca de DEFESA de DOUTORADO foi cadastrada pelo programa.
DISCENTE : LILIANE DE OLIVEIRA NEVES
DATA : 10/08/2018
HORA: 14:30
LOCAL: Campus II, Belo Horizonte, Auditório do Prédio 19.
TÍTULO:

Reliability and raters’ behavior in the Celpe-Bras oral test: a longitudinal study


PALAVRAS-CHAVES:

Celpe-Bras exam; reliability; raters’ behavior.   


PÁGINAS: 230
GRANDE ÁREA: Lingüística, Letras e Artes
ÁREA: Lingüística
SUBÁREA: Lingüística Aplicada
RESUMO:

Large-scale assessments play an important role in society, since they aid the identification of the knowledges of particular groups, the (re)directing of public policy and the process of decision-making. Therefore, they must present consistent results that reflect the construct to be evaluated. In this scenario, this thesis focuses on the test to Certificate of Proficiency in Portuguese for Foreigners (Celpe-Bras), which is composed of two parts, one written and the other oral. The oral part of the test, focus of this thesis, is a face-to-face interaction between the examinee and two evaluators: the evaluator-interlocutor (AI), who conducts the interaction, and the evaluator-observer (AO), both responsible to rate the oral performance of the examinee, based on descriptors of two distinct grids. The evaluation is done in the first instance (immediately after the test has been applied) and, if there is a significant discrepancy between the scores assigned by the two evaluators, the interaction is re-evaluated in the second and / or third instances. The general objective of this thesis is to analyze how the reliability of the test results is related to the rater’s behavior of AI and AO. Reliability is one of the desirable qualities of tests and it is related to the consistency of evaluation, i.e., the more results are error-free, the more reliable they will be. Raters’ behavior is considered in this research as the way in which the evaluators attribute grades to the oral performance of the examinees, in different instances. A quantitative methodology was used, based on a longitudinal study, which took into account data from seven consecutive editions of the Celpe-Bras exam, involving 29,831 examinees, and the theoretical framework was based on studies of Psychometrics (such as Murphy and Davidshofer, 2005),  Statistics (such as Marôco and Garcia Marques, 2006; Marôco, 2014) and Applied Linguistics (such as Bachman, 1990, 2004). Descriptions and analyses of the levels of proficiency attributed to the examinees and statistical information of the grades, such as measures of central tendency and dispersion, served as basis to verify the existence of variability of raters’ behavior. The research question: can evaluative behavior be considered a source of measurement error that interferes with the reliability of the test results?, was answered based on three techniques to estimate reliability. They are: (i) Exploratory Factorial Analysis, to verify the dimensionality of the evaluation scale; (ii) calculation of Cronbach's alpha coefficient to verify the internal consistency of the scale items and (iii) calculation of the Kappa coefficient to identify the level of agreement among the raters. The results allow us to respond positively to the research question, since: (i) the scale of evaluation is unidimensional, i. e., it evaluates a single construct, in the evaluation performed in the first instance; in the second instance, it is two-dimensional; (ii) the seven editions present high values of reliability coefficient in the first instance of evaluation, which means that the scale items have high internal consistency;  in the evaluation carried out in the second instance, the reliability is moderate and (iii) the seven editions, in the first instance of evaluation, present satisfactory values of agreement among the evaluators, albeit low; the evaluation carried out in the second instance presents a poor value. This means that the second instance, which is responsible for solving the evaluative problems that arise in the first one, is marked by a different behavior of the raters, thus reducing the reliability of the results. The results of this thesis point to the need to take actions, which  are worth highlighting: 1) review of the descriptors of the evaluation grid, so that to possibly reduce the levels of subjectivity inherent to the evaluation activity itself; 2) to intensify the training of those involved in the evaluation process. These actions are necessary to improve the reliability of Celpe-Bras's results.  


MEMBROS DA BANCA:
Presidente - JERONIMO COURA SOBRINHO - UFMG
Interno - VICENTE AGUIMAR PARREIRAS
Interno - RENATO CAIXETA DA SILVA
Interno - ANA MARIA NÁPOLES VILLELA - UFMG
Externo ao Programa - FELIPE DIAS PAIVA
Externo à Instituição - RUI BRITES - ULISBOA
Externo à Instituição - RONALDO AMORIM OZÓRIO DA MATTA LIMA - UFF
Externo à Instituição - LUIZ ANTÔNIO DOS PRAZERES - UFOP
Notícia cadastrada em: 12/06/2018 11:52
SIGAA | Secretaria de Governança da Informação - (31) 3319-7000 | Copyright © 2006-2021 - UFRN - vm-sig-app-06.ditic.sgi.cefetmg.br.inst6