Journal of Educational and Behavioral Statistics RSS feed -- OnlineFirst Articles

On the Estimation of Hierarchical Latent Regression Models for Large-Scale Assessments

Li, D., Oranje, A., Jiang, Y. — Mon, 26 Oct 2009 10:18:58 PDT

To find population proficiency distributions, a two-level hierarchical linear model may be applied to large-scale survey assessments such as the National Assessment of Educational Progress (NAEP). The model and parameter estimation are developed and a simulation was carried out to evaluate parameter recovery. Subsequently, both a hierarchical and a simple model were applied to NAEP reading data. The impact of using a hierarchical model was found to be relatively modest in this case, mostly due to modest clustering. Several other applications and future studies are discussed.

Bayesian Network Models for Local Dependence Among Observable Outcome Variables

Almond, R. G., Mulder, J., Hemat, L. A., Yan, D. — Mon, 26 Oct 2009 10:18:58 PDT

Bayesian network models offer a large degree of flexibility for modeling dependence among observables (item outcome variables) from the same task, which may be dependent. This article explores four design patterns for modeling locally dependent observations: (a) no context—ignores dependence among observables; (b) compensatory context—introduces a latent variable, context, to model task-specific knowledge and use a compensatory model to combine this with the relevant proficiencies; (c) inhibitor context—introduces a latent variable, context, to model task-specific knowledge and use an inhibitor (threshold) model to combine this with the relevant proficiencies; (d) compensatory cascading—models each observable as dependent on the previous one in sequence. This article explores the four design patterns through experiments with simulated and real data. When the proficiency variable is categorical, a simple Mantel-Haenszel procedure can test for local dependence. Although local dependence can cause problems in the calibration, if the models based on these design patterns are successfully calibrated to data, all the design patterns appear to provide very similar inferences about the students. Based on these experiments, the simpler no context design pattern appears more stable than the compensatory context model, while not significantly affecting the classification accuracy of the assessment. The cascading design pattern seems to pick up on dependencies missed by other models and should be explored with further research.

Modeling Heterogeneity in Relationships Between Initial Status and Rates of Change: Treating Latent Variable Regression Coefficients as Random Coefficients in a Three-Level Hierarchical Model

Choi, K., Seltzer, M. — Wed, 21 Oct 2009 09:33:39 PDT

In studies of change in education and numerous other fields, interest often centers on how differences in the status of individuals at the start of a period of substantive interest relate to differences in subsequent change. In this article, the authors present a fully Bayesian approach to estimating three-level Hierarchical Models in which latent variable regression (LVR) coefficients capturing the relationship between initial status and rates of change within each of J schools (Bw_j, j = 1, ..., J) are treated as varying across schools. Specifically, the authors treat within-group LVR coefficients as random coefficients in three-level models. Through analyses of data from the Longitudinal Study of American Youth, the authors show how modeling differences in Bw_j as a function of school characteristics can broaden the kinds of questions they can address in school effects research. They also illustrate the possibility of conducting sensitivity analyses using t distributional assumptions at each level of such models (termed latent variable regression in a three-level hierarchical model [LVR-HM3s]), and present results from a small-scale simulation study that help provide some guidance concerning the specification of priors for variance components in LVR-HM3s. They outline extensions of LVR-HM3s to settings in which growth is nonlinear, and discuss the use of LVR-HM3s in other types of research including multisite evaluation studies in which time-series data are collected during a preintervention period, and cross-sectional studies in which within-cluster LVR slopes are treated as varying across clusters.

On the Use of Factor-Analytic Multinomial Logit Item Response Models to Account for Individual Differences in Response Style

Johnson, T. R., Bolt, D. M. — Tue, 11 Aug 2009 09:08:05 PDT

Multidimensional item response models are usually implemented to model the relationship between item responses and two or more traits of interest. We show how multidimensional multinomial logit item response models can also be used to account for individual differences in response style. This is done by specifying a factor-analytic model for latent responses at the category level. This permits traits and response style to be separated into separate but possibly correlated factors when properly identified by the factor structure. Special cases of this model can be viewed as generalizations of some unidimensional multinomial logit item response models. In this article, we describe and demonstrate the specification and implementation of these models to account for individual differences in response style that would otherwise compromise the validity of the measurement model.

Adjusting a Significance Test for Clustering in Designs With Two Levels of Nesting

Hedges, L. V. — Wed, 05 Aug 2009 15:05:04 PDT

A common mistake in analysis of cluster randomized experiments is to ignore the effect of clustering and analyze the data as if each treatment group were a simple random sample. This typically leads to an overstatement of the precision of results and anticonservative conclusions about precision and statistical significance of treatment effects. This article gives a simple adjustment to the t statistic that would be computed if clustering were (incorrectly) ignored in an experiment with two levels of nesting (e.g., classrooms and schools) where treatment assignment is made at the highest (e.g., school) level. The adjustment is a multiplicative factor depending on the number of clusters and subclusters, the cluster and subcluster sample sizes, and the cluster and subcluster intraclass correlations _S and _C. The adjusted t statistic has Student's t distribution with reduced degrees of freedom. The adjusted statistic reduces to the t statistic computed by ignoring clustering when _S = _C = 0. It reduces to the t statistic computed using cluster means when _S = 1. If _S and _C are between 0 and 1, the adjusted t statistic lies between these two and the degrees of freedom are in between those corresponding to these two extremes.

Efficient Estimation of the Standardized Value

Longford, N. T. — Wed, 05 Aug 2009 15:05:03 PDT

We derive an estimator of the standardized value which, under the standard assumptions of normality and homoscedasticity, is more efficient than the established (asymptotically efficient) estimator and discuss its gains for small samples.

A Knowledge-Based Approach for Item Exposure Control in Computerized Adaptive Testing

Doong, S. H. — Tue, 28 Jul 2009 08:44:39 PDT

The purpose of this study is to investigate a functional relation between item exposure parameters (IEPs) and item parameters (IPs) over parallel pools. This functional relation is approximated by a well-known tool in machine learning. Let P and Q be parallel item pools and suppose IEPs for P have been obtained via a Sympson and Hetter–type simulation. Based on these simulated parameters, a functional relation k = f_P (a, b, c) relating IPs to IEPs of P is obtained by an artificial neural network and used to estimate IEPs of Q without tedious simulation. Extensive experiments using real and synthetic pools showed that this approach worked pretty well for many variants of the Sympson and Hetter procedure. It worked excellently for the conditional Stocking and Lewis multinomial selection procedure and the Chen and Lei item exposure and test overlap control procedure. This study provides the first step in an alternative means to estimate IEPs without iterative simulation.