Robust Latent Ability Estimation Based on Item Response Information and Model Fit

Abstract

Aberrant testing behaviors may result in inaccurate person trait estimation. To counter its effects, a new robust ability estimation procedure called downweighting of aberrant responses estimation (DARE) is developed. This procedure downweights both uninformative items and model-misfitting response patterns. The purpose of this study is to present DARE and to evaluate its performance against other robust methods, including biweight (Mislevy & Bock, 1982) and biweight-MAP (BMAP; Maeda & Zhang, 2017b). The traditional maximum likelihood (MLE) and maximum a-posteriori (MAP) methods are also included as baseline methods. A Monte Carlo simulation is conducted with the design variables being test length, type of aberrant behaviors, percentage of aberrant examinees, and percentage of aberrant items. Person-fit analyses using l_z^* (Snijders, 2001) and H^T (Sijtsma, 1986) are incorporated as a realistic initial step to determine the aberrant examinees that might benefit from robust estimation methods. Results showed that DARE effectively decreased the root-mean-squared-error (RMSE) and bias of the estimates compared to MAP among examinees detected using the l_z^* at the .01 α cutoff. DARE was the most accurate method in many conditions involving aberrant behavior when the test length was 40 or 60 items. At 20 items, all robust methods were ineffective. DARE performs well when 1) a high-achieving examinee show a mild spuriously low scoring behavior, or 2) a low-achieving examinee show a mild spuriously high scoring behavior. When used appropriately, DARE is superior to all pre-existing methods in limiting the negative consequences of aberrant behavior.