Journal of Educational and Behavioral Statistics recent issues

A Nonlinear Mixed Effects Model for Latent Variables

Harring, J. R. — Fri, 11 Sep 2009 12:18:02 PDT

The nonlinear mixed effects model for continuous repeated measures data has become an increasingly popular and versatile tool for investigating nonlinear longitudinal change in observed variables. In practice, for each individual subject, multiple measurements are obtained on a single response variable over time or condition. This structure can be adapted to examine the change in latent variables rather than modeling change in manifest variables. This article considers a nonlinear mixed effects model for describing nonlinear change of a latent construct over time, where the latent construct of interest is measured by multiple indicators gathered at each measurement occasion. To accomplish this, the nonlinear mixed effects model is modified to include a measurement model that explicitly expresses the relationship of the observed variables to the latent constructs. A method for marginal maximum likelihood estimation of this model is presented and discussed. An example using education data is provided to illustrate the utility of the model.

Using Dominance Analysis to Determine Predictor Importance in Logistic Regression

Azen, R., Traxel, N. — Fri, 11 Sep 2009 12:18:02 PDT

This article proposes an extension of dominance analysis that allows researchers to determine the relative importance of predictors in logistic regression models. Criteria for choosing logistic regression R² analogues were determined and measures were selected that can be used to perform dominance analysis in logistic regression. A simulation study, using both simple random sampling from a known population and bootstrap sampling from a single (parent) random sample, was performed to evaluate the bias, sampling distribution, and confidence intervals of quantitative dominance measures as well as the reproducibility of qualitative dominance measures. Results indicated that the bootstrap procedure is feasible and can be used in applied research to generalize logistic regression dominance analysis results to the population of interest. The procedures for determining and interpreting the general dominance of predictors in a logistic regression context are illustrated with an empirical example.

An Integrated Bayesian Model for DIF Analysis

Soares, T. M., Goncalves, F. B., Gamerman, D. — Fri, 11 Sep 2009 12:18:02 PDT

In this article, an integrated bayesian model for differential item functioning (DIF) analysis is proposed. The model is integrated in the sense of modeling the responses along with the DIF analysis. This approach allows DIF detection and explanation in a simultaneous setup. Previous empirical studies and/or subjective beliefs about the item parameters, including differential functioning behavior, may be conveniently expressed in terms of prior distributions. Values of indicator variables are estimated in the model, indicating which items have DIF and which do not; as a result, the data analyst may not be required to specify an "anchor set" of items that do not exhibit DIF a priori to identify the model. It reduces the iterative procedures that are commonly used for proficiency purification and DIF detection and explanation. Examples demonstrate the efficiency of this method in simulated and real situations.

A Bivariate Lognormal Response-Time Model for the Detection of Collusion Between Test Takers

van der Linden, W. J. — Fri, 11 Sep 2009 12:18:02 PDT

A bivariate lognormal model for the distribution of the response times on a test by a pair of test takers is presented. As the model has parameters for the item effects on the response times, its correlation parameter automatically corrects for the spuriousness in the observed correlation between the response times of different test takers because of variation in the time intensities of the items. This feature suggests using the model in a routine check of response-time patterns for possible collusion between test takers using an estimate of the correlation parameter or a statistical test of a hypothesis about it. Closed-form expressions for the maximum-likelihood estimations of the model parameters and a Lagrange multiplier test for the correlation parameter are presented. As in any type of statistical decision making, results from such procedures should be corroborated by evidence from other sources, for example, results from a response-based analysis or observations during the test session. The effectiveness of the model in removing the spuriousness from correlated response times is illustrated using empirical response-time data.

Linda S. Gottfredson

Wainer, H., Robinson, D. H. — Fri, 11 Sep 2009 12:18:02 PDT

Modification of the Mantel-Haenszel and Logistic Regression DIF Procedures to Incorporate the SIBTEST Regression Correction

DeMars, C. E. — Thu, 28 May 2009 19:51:30 PDT

The Mantel-Haenszel (MH) and logistic regression (LR) differential item functioning (DIF) procedures have inflated Type I error rates when there are large mean group differences, short tests, and large sample sizes. When there are large group differences in mean score, groups matched on the observed number-correct score differ on true score, contributing to inflated Type I error rates. The simultaneous item bias test procedure has incorporated an adjustment for this difference, originally using a linear regression correction and later using a nonlinear correction. In this study, these adjustments are applied to the MH and LR procedures. They effectively reduce the Type I error inflation for the MH and the LR test of uniform DIF, but not the LR test of nonuniform DIF. For large samples and large group mean differences, the effect size is estimated with greater accuracy using these adjustments.

Evaluating Independent Proportions for Statistical Difference, Equivalence, Indeterminacy, and Trivial Difference Using Inferential Confidence Intervals

Tryon, W. W., Lewis, C. — Thu, 28 May 2009 19:51:30 PDT

Tryon presented a graphic inferential confidence interval (ICI) approach to analyzing two independent and dependent means for statistical difference, equivalence, replication, indeterminacy, and trivial difference. Tryon and Lewis corrected the reduction factor used to adjust descriptive confidence intervals (DCIs) to create ICIs and introduced trivial statistical difference. They also introduced hybrid confidence intervals containing both ICI and DCI limits as replacements for error bars. This article generalizes the ICI method to include asymmetric as well as symmetric confidence intervals. Application is made to two independent proportions, odds, odds ratios, and log odds.

Using Discrete Loss Functions and Weighted Kappa for Classification: An Illustration Based on Bayesian Network Analysis

Zwick, R., Lenaburg, L. — Thu, 28 May 2009 19:51:30 PDT

In certain data analyses (e.g., multiple discriminant analysis and multinomial log-linear modeling), classification decisions are made based on the estimated posterior probabilities that individuals belong to each of several distinct categories. In the Bayesian network literature, this type of classification is often accomplished by assigning individuals to the modal state, based on the estimated posterior probabilities. This procedure is not satisfactory, however, when various types of classification errors have different costs. For example, Lenaburg used Bayesian network methods to forecast students’ grades in a college statistics course to identify students who were likely to benefit from extra tutoring, and was most concerned with incorrectly predicting students would pass. We recommend a simple post hoc classification method, based on discrete loss functions, that can lead to improved classification. We further propose that Cohen’s weighted kappa statistic be used to evaluate the quality of the classification decisions. We illustrate the approach using Lenaburg’s data.

A Nonparametric Framework for Comparing Trends and Gaps Across Tests

Ho, A. D. — Thu, 28 May 2009 19:51:30 PDT

Problems of scale typically arise when comparing test score trends, gaps, and gap trends across different tests. To overcome some of these difficulties, test score distributions on the same score scale can be represented by nonparametric graphs or statistics that are invariant under monotone scale transformations. This article motivates and then develops a framework for the comparison of these nonparametric trend, gap, and gap trend representations across tests. The connections between this framework and other nonparametric tools, including probability–probability (PP) plots, the Mann-Whitney U test, and the statistic known as P(Y > X), are highlighted. The author describes the advantages of this framework over scale-dependent trend and gap statistics and demonstrates applications of these nonparametric methods to frequently asked policy questions.

Sample Size Estimation in Cluster Randomized Educational Trials: An Empirical Bayes Approach

Rotondi, M. A., Donner, A. — Thu, 28 May 2009 19:51:30 PDT

The educational field has now accumulated an extensive literature reporting on values of the intraclass correlation coefficient, a parameter essential to determining the required size of a planned cluster randomized trial. We propose here a simple simulation-based approach including all relevant information that can facilitate this task. An example and corresponding computer code is attached.

Statistical Power for Regression Discontinuity Designs in Education Evaluations

Schochet, P. Z. — Thu, 28 May 2009 19:51:30 PDT

This article examines theoretical and empirical issues related to the statistical power of impact estimates under clustered regression discontinuity (RD) designs. The theory is grounded in the causal inference and hierarchical linear modeling literature, and the empirical work focuses on common designs used in education research to test intervention effects on student test scores. The main conclusion is that three to four times larger samples are typically required under RD than experimental clustered designs to produce impacts with the same level of statistical precision. Thus, the viability of using RD designs for new impact evaluations of educational interventions may be limited and will depend on the point of treatment assignment, the availability of pretests, and key research questions.

Assessing Sensitive Attributes Using the Randomized Response Technique: Evidence for the Importance of Response Symmetry

Ostapczuk, M., Moshagen, M., Zhao, Z., Musch, J. — Thu, 28 May 2009 19:51:30 PDT

Randomized response techniques (RRTs) aim to reduce social desirability bias in the assessment of sensitive attributes but differ regarding privacy protection. The less protection a design offers, the more likely respondents cheat by disobeying the instructions. In asymmetric RRT designs, respondents can play safe by giving a response that is never associated with the sensitive attribute. Symmetric RRT designs avoid the incentive to cheat by not allowing such responses. We tested whether a symmetric variant of a cheating detection model (CDM) increases compliance with the instructions in a survey of academic dishonesty among 2,254 Chinese students. As we observed more noncompliance in the asymmetric than symmetric variant, we recommend the use of symmetric CDMs, which can easily be tested within multinomial models.

Editorial Policy Statement

Rindskopf, D. — Wed, 25 Mar 2009 17:29:10 PDT

Standard Errors of Equating for the Percentile Rank-Based Equipercentile Equating With Log-Linear Presmoothing

Wang, T. — Wed, 25 Mar 2009 17:29:10 PDT

Holland and colleagues derived a formula for analytical standard error of equating using the -method for the kernel equating method. Extending their derivation, this article derives an analytical standard error of equating procedure for the conventional percentile rank–based equipercentile equating with log-linear smoothing. This procedure is illustrated with real test data and is evaluated using both the bootstrap and the parametric bootstrap (simulation) methods for three different equating designs.

Public Schools Versus Private Schools: Causal Inference With Partial Compliance

Jin, H., Rubin, D. B. — Wed, 25 Mar 2009 17:29:10 PDT

An approach to handle partial compliance behavior using principal stratification is presented and applied to a subset of the longitudinal data from the New York City School Choice Scholarship Program, a randomized experiment designed to assess the effects of private schools versus public schools on academic achievement. The initial analysis suggests an interesting relationship between compliance with the offer and academic achievement, including a possible "beneficial rejected offer" effect and a possible "adjustment hardship" effect. These results seem to favor public schools in the sense they suggest that the collection of students who would attend private school when offered the scholarship but attend public school without the offer had a lower average posttest score if they attended private school than if they attended public school. This case study illustrates the strengths of principal stratification: the explicit examination of specific assumptions and directly interpretable results with possible policy implications.

Hierarchical Dependence in Meta-Analysis

Stevens, J. R., Taylor, A. M. — Wed, 25 Mar 2009 17:29:10 PDT

Meta-analysis is a frequent tool among education and behavioral researchers to combine results from multiple experiments to arrive at a clear understanding of some effect of interest. One of the traditional assumptions in a meta-analysis is the independence of the effect sizes from the studies under consideration. This article presents a meta-analytic review of 13 experiments with 18 study reports all involving the effect of native-language (L1) vocabulary aids on second-language (L2) reading comprehension. Some experiments produced multiple study reports, creating a dependence structure among the resulting effect size estimates. The covariance among these effect size estimates is estimated and incorporated into a proposed meta-analysis model that accounts for the dependence at a hierarchical level. The overall effect size estimate (g =.63) indicates that L1 vocabulary aids can be an effective L2 reading comprehension aid in the short term. An interpretation of the hierarchical components is discussed.

Using Past Data to Enhance Small Sample DIF Estimation: A Bayesian Approach

Sinharay, S., Dorans, N. J., Grant, M. C., Blew, E. O. — Wed, 25 Mar 2009 17:29:10 PDT

Test administrators often face the challenge of detecting differential item functioning (DIF) with samples of size smaller than that recommended by experts. A Bayesian approach can incorporate, in the form of a prior distribution, existing information on the inference problem at hand, which yields more stable estimation, especially for small samples. A large volume of past data is available for many operational tests and such data could be used to establish prior distributions for a Bayesian DIF analysis. This article discusses how to perform such an analysis. The suggested approach is found to be more conservative and preferable with respect to several overall criteria than the existing DIF detection methods in a realistic simulation study.

Consequences of Unmodeled Nonlinear Effects in Multilevel Models

Bauer, D. J., Cai, L. — Wed, 25 Mar 2009 17:29:10 PDT

Applications of multilevel models have increased markedly during the past decade. In incorporating lower-level predictors into multilevel models, a key interest is often whether or not a given predictor requires a random slope, that is, whether the effect of the predictor varies over upper-level units. If the variance of a random slope significantly differs from zero, the focus of the analysis may then shift to explaining this heterogeneity with upper-level predictors through the testing of cross-level interactions. As shown in this article, however, both the variance of the random slope and the cross-level interaction effects may be entirely spurious if the relationship between the lower-level predictor and the outcome is nonlinear in form but is not modeled as such. The importance of conducting diagnostics to detect nonlinear effects is discussed and demonstrated via an empirical example.

DINA Model and Parameter Estimation: A Didactic

de la Torre, J. — Wed, 25 Mar 2009 17:29:10 PDT

Cognitive and skills diagnosis models are psychometric models that have immense potential to provide rich information relevant for instruction and learning. However, wider applications of these models have been hampered by their novelty and the lack of commercially available software that can be used to analyze data from this psychometric framework. To address this issue, this article focuses on one tractable and interpretable skills diagnosis model—the DINA model—and presents it didactically. The article also discusses expectation-maximization and Markov chain Monte Carlo algorithms in estimating its model parameters. Finally, analyses of simulated and real data are presented.

Marginal Maximum Likelihood Estimation of a Latent Variable Model With Interaction

Cudeck, R., Harring, J. R., du Toit, S. H. C. — Wed, 25 Mar 2009 17:29:10 PDT

There has been considerable interest in nonlinear latent variable models specifying interaction between latent variables. Although it seems to be only slightly more complex than linear regression without the interaction, the model that includes a product of latent variables cannot be estimated by maximum likelihood assuming normality. Consequently, many approximate methods have been proposed. Recently, a maximum likelihood method of estimation based on the expectation–maximization algorithm has been suggested that is optimum if the distribution assumptions are true. In this article, the authors outline an alternative marginal maximum likelihood estimator using numerical quadrature. A key feature of the approach is that in the marginal distribution of the manifest variables the complicated integration can be reduced, often to a single dimension. This allows a direct approach to maximizing the log-likelihood and makes the method relatively straightforward to use.

A Mixed Effects Randomized Item Response Model

Fox, J.-P., Wyrick, C. — Thu, 15 Jan 2009 15:00:11 PST

The randomized response technique ensures that individual item responses, denoted as true item responses, are randomized before observing them and so-called randomized item responses are observed. A relationship is specified between randomized item response data and true item response data. True item response data are modeled with a (non)linear mixed effects and/or item response theory model. Although the individual true item responses are masked through randomizing the responses, the model extension enables the computation of individual true item response probabilities and estimates of individuals’ sensitive behavior/attitude and their relationships with background variables taking into account any clustering of respondents. Results are presented from a College Alcohol Problem Scale (CAPS) where students were interviewed via direct questioning or via a randomized response technique. A Markov Chain Monte Carlo algorithm is given for estimating simultaneously all model parameters given hierarchical structured binary or polytomous randomized item response data and background variables.

An Additional Measure of Overall Effect Size for Logistic Regression Models

Allen, J., Le, H. — Thu, 15 Jan 2009 15:00:11 PST

Users of logistic regression models often need to describe the overall predictive strength, or effect size, of the model’s predictors. Analogs of R² have been developed, but none of these measures are interpretable on the same scale as effects of individual predictors. Furthermore, R² analogs are not invariant to the base rate (overall proportion of successes), making it difficult to compare effect sizes across data sets. The authors propose a measure of overall effect size that is interpretable on the same scale as effects of individual predictors and is invariant to the base rate. They explore the properties of the overall odds ratio and illustrate its use through an example. They also provide interpretive guidance and illustrate how statistical software can be used to compute the proposed measure.

On Using Stochastic Curtailment to Shorten the SPRT in Sequential Mastery Testing

Finkelman, M. — Thu, 15 Jan 2009 15:00:11 PST

Sequential mastery testing (SMT) has been researched as an efficient alternative to paper-and-pencil testing for pass/fail examinations. One popular method for determining when to cease examination in SMT is the truncated sequential probability ratio test (TSPRT). This article introduces the application of stochastic curtailment in SMT to shorten the TSPRT without substantially compromising error rates. Unlike the TSPRT, the stochastically curtailed procedure exhibits an optimality property known as weak admissibility. Error bounds of the two methods are provided in terms of one another. In two simulation sets, the stochastically curtailed procedure considerably improved the average test length of an SMT with only a slight decrease in accuracy.

Extended Generalized Linear Latent and Mixed Model

Segawa, E., Emery, S., Curry, S. J. — Thu, 15 Jan 2009 15:00:11 PST

The generalized linear latent and mixed modeling (GLLAMM framework) includes many models such as hierarchical and structural equation models. However, GLLAMM cannot currently accommodate some models because it does not allow some parameters to be random. GLLAMM is extended to overcome the limitation by adding a submodel that specifies a distribution of the additional random effects (Extended-GLLAMM). The extension is extremely simple to implement through the Bayesian framework with the WinBUGS software. Our approach is illustrated through the analysis of data from a youth tobacco cessation study.

The General Linear Model as Structural Equation Modeling

Graham, J. M. — Thu, 15 Jan 2009 15:00:11 PST

Statistical procedures based on the general linear model (GLM) share much in common with one another, both conceptually and practically. The use of structural equation modeling path diagrams as tools for teaching the GLM as a body of connected statistical procedures is presented. A heuristic data set is used to demonstrate a variety of univariate and multivariate statistics as structural models. Implications for analytic strategies and education are discussed.

Combining Heterogeneous Correlation Matrices: Simulation Analysis of Fixed-Effects Methods

Hafdahl, A. R. — Thu, 15 Jan 2009 15:00:11 PST

Monte Carlo studies of several fixed-effects methods for combining and comparing correlation matrices have shown that two refinements improve estimation and inference substantially. With rare exception, however, these simulations have involved homogeneous data analyzed using conditional meta-analytic procedures. The present study builds on previous evidence about these methods’ relative performance by examining their behavior under heterogeneity, which is more realistic in practice. Results based on both conditional and unconditional estimands indicate that of the two refinements, using estimated correlations in conditional (co)variances improves point and interval estimates of mean correlations more than analyzing Fisher Z correlations, despite the latter’s superiority for testing homogeneity. Recommended choices among methods are offered.

Acknowledgments

Thu, 15 Jan 2009 15:00:11 PST