Journal of Educational and Behavioral Statistics current issue

Using the Kernel Method of Test Equating for Estimating the Standard Errors of Population Invariance Measures

Moses, T. — 2008-06-11

Equating functions are supposed to be population invariant, meaning that the choice of subpopulation used to compute the equating function should not matter. The extent to which equating functions are population invariant is typically assessed in terms of practical difference criteria that do not account for equating functions’ sampling variability. This article shows how to extend the framework of kernel equating so that the standard errors of the root mean square difference (RMSD) and of the difference between two subpopulations’ equated scores can be estimated. An investigation of population invariance for the equivalent groups design is discussed. The accuracies of the derived standard errors are evaluated with respect to empirical standard errors. This evaluation shows that the accuracy of the standard error estimates for the equated score differences is better than for the RMSD and that accuracy for both standard error estimates is best when sample sizes are large.

Bias Mechanisms in Intention-to-Treat Analysis With Data Subject to Treatment Noncompliance and Missing Outcomes

Jo, B. — 2008-06-11

An analytical approach was employed to compare sensitivity of causal effect estimates with different assumptions on treatment noncompliance and non-response behaviors. The core of this approach is to fully clarify bias mechanisms of considered models and to connect these models based on common parameters. Focusing on intention-to-treat analysis, systematic model comparisons are performed on the basis of explicit bias mechanisms and connectivity between models. The method is applied to the Johns Hopkins school intervention trial, where assessment of the intention-to-treat effect on school children’s mental health is likely to be affected by assumptions about intervention noncompliance and nonresponse at follow-up assessments. The example calls attention to the importance of focusing on each case in investigating relative sensitivity of causal effect estimates with different identifying assumptions, instead of pursuing a general conclusion that applies to every occasion.

New Results on the Linear Equating Methods for the Non-Equivalent-Groups Design

von Davier, A. A. — 2008-06-11

The two most common observed-score equating functions are the linear and equipercentile functions. These are often seen as different methods, but von Davier, Holland, and Thayer showed that any equipercentile equating function can be decomposed into linear and nonlinear parts. They emphasized the dominant role of the linear part of the nonlinear equating function and gave conditions under which the equipercentile methods in the non-equivalent-groups anchor test (NEAT) design give identical results. Consequently, this article focuses on linear equating methods in a NEAT design—the Tucker, chained, and Levine observed-score functions—and describes the theoretical conditions under which these methods produce the same equating function. Constructed examples illustrate the theoretical results.

When Can Subscores Have Value?

Haberman, S. J. — 2008-06-11

In educational tests, subscores are often generated from a portion of the items in a larger test. Guidelines based on mean squared error are proposed to indicate whether subscores are worth reporting. Alternatives considered are direct reports of subscores, estimates of subscores based on total score, combined estimates based on subscores and total scores, and residual analysis of subscores. Applications are made to data from two testing programs.

Identification of Causal Parameters in Randomized Studies With Mediating Variables

Sobel, M. E. — 2008-06-11

Treatments in randomized studies are often targeted to key mediating variables. Researchers want to know if the treatment is effective and how the mediators affect the outcome. The data are often analyzed using structural equation models (SEMs), and model coefficients are interpreted as effects. However, only assignment to treatment groups is randomized, so mediators are self-selected treatments. Thus, the so-called direct effects of mediators on later outcomes do not usually warrant a causal interpretation. Holland (1988) studied the case of a single continuous mediator, criticizing the use of SEMs. He uses treatment assignment as an instrument for the effect of the mediator on the outcome. However, the assumptions he made to justify this approach are overly strong and substantively implausible. This article (a) makes explicit the assumptions needed to justify equating the parameters of SEMs with the effects of mediators, (b) provides weaker and more plausible conditions under which the instrumental variable estimand may be interpreted as an effect, and (c) extends the analysis to include the case of noncompliance.