This Article Full Text (PDF) References Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Livingston, S. A. Search for Related Content Social Bookmarking                         What's this?

An Interesting Problem in the Estimation of Scoring Reliability

Educational Testing Service

A performance assessment consisting of 10 separate exercises was scored with a randomized scoring procedure. All responses to each exercise were rated once; in addition, a randomly selected subset of the responses to each exercise received an independent second rating. Each second rating was averaged with the corresponding first rating before the scores were computed. This article presents a method for estimating the scoring reliability (interrater reliability) coefficient and the standard error of scoring for the resulting scores. The report concludes with some numerical examples showing how the reliability estimation procedure can be used to estimate the effect of varying the proportions of responses that are double-scored.

Key Words: interrater reliability • performance assessment • scoring reliability

Journal of Educational and Behavioral Statistics, Vol. 29, No. 3, 333-341 (2004)
DOI: 10.3102/10769986029003333


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?