This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Thomas, D. R. Articles by Decady, Y. J. Search for Related Content Social Bookmarking                         What's this?

Point Estimates and Confidence Intervals for Variable Importance in Multiple Linear Regression

Carleton University

Yves J. Decady

Statistics Canada

	Abstract

Top Abstract Deficiencies in Traditional... Pratt's Measure Point Estimates and Confidence... The Simulation Experiment Results Example Summary and Conclusions Appendix References

 
The topic of variable importance in linear regression is reviewed, and a measure first justified theoretically by Pratt (1987) is examined in detail. Asymptotic variance estimates are used to construct individual and simultaneous confidence intervals for these importance measures. A simulation study of their coverage properties is reported, and an example is provided.

Key Words: Keywords: relative importance • normalized Pratt measures • asymptotic variances • Bonferroni intervals • simultaneous confidence intervals

The purpose of this article is to review and to present some new results on a particular measure of relative importance for multiple linear regression for the general case of p correlated explanatory variables. This measure defines the importance of the j-th independent variable as the product of two terms, namely its simple correlation with the dependent variable, _j, and its standardized regression coefficient in the multiple regression, _j. The relative importance of two variables, x_j and x_k, is then defined as the ratio of their importance measures: namely, _jj/_kk. The measure itself is far from new. For example, it was considered (and criticized) by Darlington (1968), and it has been discussed more recently by Bring (1996), who called it the ‘‘product measure,’’ and by Thomas, Hughes, and Zumbo (1998), who responded to some of Bring’s criticisms of it. The justification for this measure had always been heuristic, at best, until Pratt (1987) provided a theoretical derivation based on a set of natural assumptions or axioms that should be satisfied by a measure of variable importance. In light of this justification of what had hitherto been a controversial measure (see Pratt, 1987), the quantity _jj, corresponding to an individual explanatory variable, x_j, will be referred to in this article as ‘‘Pratt’s measure’’ of variable importance.

The main contribution reported in this article is the development of individual and simultaneous confidence intervals for the p Pratt measures, which should increase their practical usefulness by providing an inferential, rather than merely descriptive, method of deciding whether one explanatory variable is more important than another. The overall goal of the article is thus similar, in a particular sense, to a recent article by Azen and Budescu (2003), who extended Budescu’s (1993) ‘‘dominance’’ approach by introducing new measures of dominance and by adding bootstrapped inference procedures to assess the stability of the dominance ranking. Azen, Budescu, and Reiser (2001) suggested another inferential approach to the assessment of variable importance in regression. These authors introduced the concept of the ‘‘criticality’’ of a variable, derived from the probability that an individual variable will be included in the best subset model fitted to a randomly drawn sample containing responses for one dependent and p explanatory variables. Probabilities were again assessed by means of bootstrap samples, with confidence intervals based on large sample multinomial theory. Although similar in that it provides inferential methods for assessing importance, the Pratt approach described in this article is very different conceptually from the dominance analysis and criticality approaches. Pratt’s derivation of his importance measure is axiomatic and theoretical, whereas the dominance and criticality approaches are based on an intuitive definition of variable importance. It should be noted that Pratt’s measure also has an intuitive definition as discussed by Bring (1996) and by Thomas et al. (1998). This interpretation of Pratt’s measures will be described later.

The problem of assessing variable importance, particularly in multiple linear regression, has been extensively documented since the 1960s, and the articles cited above—in particular, those by Bring (1996), Budescu (1993), and Azen and Budescu (2003)—together provide an extensive review of the literature. In this article, therefore, only specific points that justify the need for importance measures such as Pratt’s (or such as dominance analysis and criticality) will be reviewed.

	Deficiencies in Traditional Measures of Importance

Top Abstract Deficiencies in Traditional... Pratt's Measure Point Estimates and Confidence... The Simulation Experiment Results Example Summary and Conclusions Appendix References

 
Kruskal and Majors (1989) carried out a broad survey of the scientific literature to develop a taxonomy of the various measures that have been used to quantify variable importance. In the context of regression analysis, they found that numerous measures of variable importance were in use, including simple and partial correlation coefficients, regression coefficients (standardized and unstandardized), t statistics, and p values. They also noted Pratt’s (1987) measure and recommended that it be studied further. To the above list can be added a variety of measures based on reduction in variance or on decompositions of variance explained, as documented by Azen and Budescu (2003). For the most part, the primary focus of the literature has been on measures of variable importance, rather than on basic concepts. Thus, the concept of relative importance has been termed ‘‘ambiguous’’ (Kruskal, 1987; Kruskal & Majors, 1989), ‘‘not at all well defined’’ (Healy, 1990), and ‘‘vague’’ and ‘‘diffuse’’ (Bring, 1994). In most applications and discussions of variable importance, the meaning of the concept is implied by the characteristics of the measure selected. Given this general confusion, Azen and Budescu (2003) argued in favor of an intuitive measure of importance that can be readily understood and interpreted by practitioners, and proposed their modified version of dominance analysis. Pratt’s approach (1987), on the other hand, was to define the concept of variable importance in terms of a number of axioms that should be satisfied by any measure of variable importance and to then deduce a unique measure satisfying them.

Some of the problems associated with the traditional measures of variable importance can be illustrated by considering standardized regression coefficients, where the standardization is motivated by the difficulties inherent in interpreting unstandardized coefficients for explanatory variables that are measured on different scales (Achen, 1982; Greenland, Schlesselman, & Criqui, 1986; Healy, 1990). Although standardized coefficients are commonly used as measures of importance, Healy (1990) and Bring (1994) noted that because regression coefficients measure the conditional effect of a variable, standardization should therefore be based on conditional standard errors, rather than the customary unconditional standard errors. Bring (1994) went on to show that using conditionally standardized regression coefficients as measures of variable importance is equivalent to comparing individual t statistics. Unfortunately, a comparison of t_j and t_k, corresponding to variables x_j and x_k, provides an invalid assessment of relative importance because, for the full p variable model, each t statistic measures that variable’s contribution to the model over and above the effect of all (p – 1) other variables. Thus, the reference subsets for t_j and t_k are different making a direct comparison invalid (Azen & Budescu, 2003), a problem that is also associated with other traditional ‘‘partial’’ measures such as partial correlations, semipartial correlations, and so forth. In designing his dominance approach, Budescu (1993) avoided this problem by comparing the contribution of variables x_j and x_k in turn to a common subset of variables that omits both of them. Because t statistics for individual explanatory variables are related to increments in R², Budescu’s original definition of dominance can be stated as follows: One variable dominates another if it generates a greater increment in R₂ than the other, when added to all the equations formed from all possible subsets of the remaining (p – 2) variables.

Bring’s (1994) conclusion also leads directly to a consideration of importance measures based on partitions of R², where a portion of the overall R² is attributed to each predictor. If the independent variables are uncorrelated, there is a unique partition of R² in which each variable’s contribution is equal to the variance it explains when it is individually regressed on the dependent variable. When the independent variables are correlated, however, it is difficult to find partitions of R² where each component of variance explained can be logically assigned to an individual variable. If a prior ordering of the variables is available, a sequential orthogonalization of the independent variables leads to an assignable partition. On the other hand, the data-dependent ordering provided by sequential variable selection algorithms should not be used, because different orders of selection leading to the same final model can lead to dramatically different conclusions regarding the importance of a given variable (see, e.g., Stevens, 2002, p. 119). Several authors have attempted to overcome this problem by averaging various partitions of R² over all possible orderings (for details, see Azen & Budescu, 2003), but this process results in measures that are hard to interpret. The advantage of Pratt’s (1987) approach is that it offers a very simple and theoretically justifiable way of partitioning R² and assigning the components—namely, sample estimates of the _jjs, which sum to R², to individual dependent variables as measures of variable importance.

	Pratt’s Measure

Top Abstract Deficiencies in Traditional... Pratt's Measure Point Estimates and Confidence... The Simulation Experiment Results Example Summary and Conclusions Appendix References

 
Pratt (1987) based his axiomatic development on a population model of the form

(1)

where the error u is uncorrelated with each of the xs, and is distributed with mean zero and variance ². The first axioms that he adopted were

1. relative importance depends only on the means, variances, and correlations of y, x₁, x₂, . . . , x_p, (i.e., only on first- and second-order moments);

2. relative importance is not affected by linear transformations of any variable.

He also standardized the (random) xs to have variance one, so that for the two variable case (p = 2), it follows from assumptions 1 and 2 that relative importance depends only on ₁, ₂, ₁₂, and ², where the s are standardized regression coefficients corresponding to x₁ and x₂, and ₁₂ is the correlation between x₁ and x₂. Pratt (1987) used a slightly different definition of standardized regression coefficients than is customary, but because these differences do not affect the application of his results, the s will be treated as standard in this abbreviated account.

Pratt’s (1987) basic approach was to use a symmetry argument to establish his importance measure for a two-variable regression and then to invoke additional axioms to generalize his measure to the p-variable case. He postulated that the two independent variables x₁ and x₂ could be constructed from a symmetric set of M variables, X₁, . . . , X_M, all having mean zero and variance s², and all equally correlated with correlation coefficient r, such that the regression of y on these Xs was given by X₁ + X₂ + · · · + X_M, with residual variance ². He further defined x₁ = X₁ + X₂ + · · · + X_m and x₂ = X_{m + 1} · · · + X_m+n, where m + n + M, and invoked a third axiom—namely, that in this symmetric two-variable situation,

3. the relative importance of x₁ to x₂ is as m to n.

Under the above setup, the standardized coefficients ₁ and ₂ can be expressed as functions of m, n, r, and s, and Pratt (1987) solved these equations to get an expression for the relative importance of x₁ to x₂; namely, m/n, given by

(2)

where ₁ and ₂ denote standardized regression coefficients (i.e., correlations) in the simple regressions of y on x₁ and y on x₂. A necessary and sufficient condition for the solution to exist (i.e., for the above construction of x₁ and x₂ to be feasible) is that ₁₁ and ₂₂ both be positive. For the standardized two variable case, the variance explained by the regression of y on x₁ and x₂ is given by ₁₁ + ₂₂, so that Pratt’s derivation of Equation 2 justifies the use of variance explained as a measure of relative importance for the case p = 2, provided that both of the _jj are positive and provided that

(3)

To extend his rule given in Equation 3 to the case p > 2, Pratt first defined the importance of a subset of variables, (x₁, . . . , x_q), q < p, to be the sum of their individual importances. He then adopted two more axioms; namely, the extension of his second assumption to

4. the nonsingular linear transformation of a subset (x₁, . . . , x_q) into the subset (x₁', . . . , x_q') does not affect its importance relative to the other variables; and

5. the addition of a pure noise variable, independent of y and x₁, . . . , x_p, to a subset of variables does not affect the importance of the subset relative to the other variables.

With these, together with one additional minor assumption, or axiom, for which several alternatives exist, Pratt established the validity of the rule of Equation 3 for assigning relative importance in the general case of p > 2 independent variables.

The Geometric Interpretation of Pratt’s Measure
Bring (1996) and Thomas et al. (1998) used the geometry of least squares to interpret Pratt’s measure. For a sample of size N, the observed values of the variables are represented as vectors y, x₁, . . . , x_p in an N-dimensional vector space. The least squares solution is represented by the orthogonal projection of the observed dependent variable, y, onto the space spanned by the observed independent variables, x₁, . . . , x_p. When the observed variables are standardized to have zero mean values and variance one (denoted by a tilde over the character representing the variable), the projection of , which in nongeometric terms is the vector of fitted values, , is represented algebraically by the weighted vector sum

(4)

where the _js denote sample estimates of the standardized regression coefficients. Thomas et al. (1998) developed a measure of relative importance by projecting each component _jj of onto itself, and then defining the importance of each variable as the ratio of the signed length of each projection to the total length of . They showed that for the jth variable, their measure (denoted by _j) could be expressed as

(5)

that is, as a sample estimate of Pratt’s measure, divided by the R² corresponding to the full p-variable regression equation. The _js sum to one by virtue of their geometric construction. This feature is immediately evident from their algebraic representations in Equation 5, which must sum to one given that the sample Pratt measures sum to R². Thus, the _js discussed by Thomas et al. (1998) will henceforth be referred to as normalized Pratt measures. Thomas (1992) and Thomas and Zumbo (1996) earlier defined similar measures for the MANOVA case, where they were referred to as ‘‘discriminant ratio coefficients.’’ For a practitioner familiar with the geometry of least squares, the notion of dividing up the total R² in proportion to the projected lengths of coefficient-weighted variables is an intuitively appealing approach to variable importance. Other interpretations can also be provided, as discussed in Pratt’s (1987) original paper.

Two Criticisms of Pratt’s Measure
The first criticism is that one or more of the sample quantities _jj, can be negative, which some practitioners interpret as negative importance, a meaningless concept. However, as noted in the text following Equation 2, Pratt’s importance rule is valid only when the population quantities _jj are all positive. Thus, negativity of any one of these quantities does not signify negative importance but instead signifies a regression situation that is ‘‘too complex for a single measure’’ (Pratt, 1987, p. 245). For the sample estimates, _jj, false negatives as well as false positives can occur (as noted by Pratt, 1987, p. 259), which is a strong argument for the construction of simultaneous confidence intervals on the normalized importance measures of Equation 5, the main goal of this article. If the confidence interval for a particular normalized Pratt measure lies entirely below zero, a real negative is indicated. If a point estimate is negative but the corresponding confidence interval also includes positive values large enough to be of interest, this is an inconclusive situation in which the only remedy may be a larger sample. A situation that is particularly prone to false positives and negatives is that of multicollinearity. Thomas et al. (1998) developed a lower bound on individual normalized Pratt measures that shows, for example, that whenever an explanatory variable exhibits a variance inflation factor of 10 or more, indicative of multicollinearity (see Stevens, 2002, p. 92), at least one normalized Pratt measure can exhibit a value of –1 or lower. They gave an example in which a negative normalized Pratt measure of large magnitude (close to 1) was reduced to a small positive value by the application of ridge regression (Hoerl & Kennard, 1970), suggesting that the original ‘‘negative importance’’ was false. As noted above, not all negative importances will be false, and the fact must be faced that some regression modeling situations are so complex that there is no single measure of variable importance that satisfies Pratt’s axioms. In such cases, the practitioner would be well advised to perform additional analyses. The matrix of explanatory variable correlations should be carefully inspected to provide insight into the relationships among the variables; also, some of the alternative measures that Azen and Budescu (2003) described, for example, might be examined despite their documented shortcomings. Finally, Budescu’s (1993) approach and its recent extensions (Azen & Budescu, 2003) could be used to provide additional detailed information regarding the explanatory variables’ relative contributions to R².

The second criticism was advanced by Bring (1996), who geometrically illustrated the case of an independent variable, x₁, that was orthogonal to y (i.e., uncorrelated with y) but which nevertheless increased R² when it was added to a regression model originally containing only x₂. Such variables are often referred to as suppressor variables (see, e.g., Stevens, 2002, p. 124): namely, variables that make no direct contribution to the regression sum of squares, contributing only by reducing (or suppressing) the overall error sum of squares (or equivalently, the error variance) through their relationship with the other independent variables. In the example, because ₁ is zero, the normalized Pratt index for x₁ is zero, which Bring (1996) considered to be counterintuitive. Thomas et al. (1998) argued that because suppressor variables and nonsuppressor variables contribute to the regression in entirely different ways, it is actually intuitive (rather than counterintuitive) to assess the relative importance of the nonsuppressors using the normalized Pratt measure and to separately assess the relative importance of the suppressors to the nonsuppressors using the measure (R² – R_NS²)/R², where R _NS² denotes the variance explained by the nonsuppressor variables alone.

To this point, the relationship of Pratt’s measure to other measures of variable importance has been described, and some recent criticisms have been put into context. The remainder of the article will address the construction and evaluation of confidence intervals.

	Point Estimates and Confidence Intervals for the js

Top Abstract Deficiencies in Traditional... Pratt's Measure Point Estimates and Confidence... The Simulation Experiment Results Example Summary and Conclusions Appendix References

 
The finite sample distributional properties of the normalized Pratt measures are technically complicated, even under the assumption of normal errors. The theoretical aspects of the current investigation are therefore based on an asymptotic analysis of the properties of the _js, in the limit N , where N denotes sample size. A simulation study has also been carried out to determine how well the asymptotic confidence intervals perform for the sample sizes that might be encountered in practice. This section will present a summary of the asymptotic results; the results of the simulation study will be reported in a later section.

It will be assumed that the dependent and explanatory variables in Pratt’s model given by Equation 1 are independently drawn from a multivariate normal distribution; that is, (y, x₁, . . . , x_p)' ~N(µ, ), with positive definite covariance matrix of dimension (p + 1) x (p + 1) given by

(6)

and mean vector µ = (µ_y µ'_x)' of dimension (p + 1) x1. Normality is a stronger assumption than that adopted by Pratt (1987), who assumed only that the disturbance term u and the independent variables x are uncorrelated. The normality assumption has been made to simplify the derivation of the asymptotic variances of the _js; evaluating these variances under weaker assumptions will be the focus of future research. In the covariance matrix of Equation 6, _xx represents the p x p population covariance matrix of the predictor variables, and _xy represents the p x 1 vector of population covariances between y and the p x1 vector of explanatory variables x. In terms of these distributional parameters, the variance of the disturbance term u of Equation 1 can be written as ² = _y²'_xyxx^–1_xy, and the regression coefficients of Equation 1 can be written as b₀ = µ_y –µ'_xx^–1_xy.and b₁ = _xx^–1_xy. It can be shown that the population value of the normalized Pratt index defined in Equation 5 can be expressed in terms of these variances and covariances as

(7)

where [.]_j denotes the jth element of the vector. It is clear from Equation 7 that the means µ_y and µ_x can be set to zero with no loss of generality. The estimate _j defined in Equation 5 can also be evaluated directly from Equation 7 by replacing the population variances and covariances by their sample counterparts obtained from the sample covariance matrix , estimated from a sample of N independent and identically distributed observations of y and x₁, . . . , x_p. Results of the asymptotic analyses are given below. Proofs are in the appendix.

Consistency
The normalized Pratt measure, _j of Equation 5 is a consistent estimator of the population version of the measure given in Equation 7, that is,

(8)

where denotes convergence in probability. That is, when the sample size is large, the difference between the estimate _j and the population parameter _j will be small, with probability approaching 1.

Asymptotic Normality
The normalized Pratt measure _j has an asymptotically normal distribution as N . Formally, it can be shown that

(9)

where denotes convergence in distribution, and N(0, AV( _j)) is a normal distribution with mean zero and variance AV(_j). The asymptotic variance can be expressed as

(10)

where B = (b'_xxb) = ^'_xyxx^–1_xy, and as noted earlier, ² is the variance of the error term u in Equation 1 given by _y².– B The notation [.]_jj denotes the jth diagonal element of the matrix, while [.]_j denotes the jth row or the jth column of the matrix, depending on the context. The last term on the right-hand side of Equation 10 represents the variance with respect to x of the conditional mean of _j, whereas the remaining terms arise from the expectation of the conditional variance of _j (see appendix for details).

Variance Estimates
In practice, results of the form of Equations 9 and 10 are interpreted to mean that for large samples, _j has approximate variance N^–1A(_j), where A(_j) represents an estimate of the asymptotic variance given by Equation 10. The asymptotic variance can be consistently estimated using the standard least squares regression estimate in place of b, and the estimated covariance matrix of the x_js, obtained as a submatrix of , in place of _xx. The resulting large sample variance estimate will be denoted (_j). After some algebraic manipulation, (_j) can be expressed entirely in terms of quantities routinely displayed in, or easily calculated from, the output of standard regression software. Thus,

(11)

where t_j is the value of the t statistic corresponding to _j, and _j estimates the standardized regression coefficient defined previously. The accuracy of the large sample variance estimator given by Equation 11 and the coverage properties of the confidence intervals based on it (see next section) have been assessed through simulation. Results of this simulation study will be reported later.

Individual and Simultaneous Confidence Intervals
A large sample 100(1– )% confidence interval (or interval estimate) for a specific normalized Pratt measure _j corresponding to independent variable x_j is given by the interval (_j^L, _j^U), where the upper and lower limits are given by

(12)

Here Z_/2 is the upper /2 percentage point of the standard normal distribution. This interval corresponds to the following probability statement:

(13)

that is, the interval will cover the true value of the jth normalized Pratt importance measure with approximate probability (or confidence) 1 – Individual confidence intervals can be defined for the importances of each of the p independent variables. However, in this case, the overall coverage, that is, the probability that all p intervals will simultaneously cover their respective population importances, will be considerably less than 1 – . This is the confidence interval equivalent of the problem of cumulation of Type I error associated with multiple tests of hypothesis (see, e.g., Stevens, 2002, p. 6). The Bonferroni inequality can be used to define asymptotically conservative simultaneous confidence intervals by constructing the p individual intervals with confidence level 1– ', where ' = /p. The large sample simultaneous confidence interval statement then represents the application of Equation 13 to all p importances but with replaced by ' = /p, that is,

(14)

The above distinction between individual and simultaneous confidence intervals is often a source of confusion to practitioners. The inspection of an individual confidence interval will be appropriate if the importance of a specific variable is of prior interest—that is, the variable is being studied for reasons unconnected to the results of the data analysis. On the other hand, if a variable is of interest because its Pratt index is the largest, say, then inspection of an individual confidence interval for that variable would not be appropriate. Selection of the largest index implies a comparison with all other variables so that appropriate inference must be based on a set of intervals that accounts for variation in all Pratt indices simultaneously. This distinction will be further illustrated later.

	The Simulation Experiment

Top Abstract Deficiencies in Traditional... Pratt's Measure Point Estimates and Confidence... The Simulation Experiment Results Example Summary and Conclusions Appendix References

 
The simulation experiment was primarily designed to explore the performance of the individual and simultaneous confidence interval procedures speci-fied by Equations 13 and 14. Because the confidence intervals depend on point estimates of the normalized Pratt measures, _j, and on their variances, the biases in the estimates _j and in their estimated variances were also examined. The simulation explored confidence interval performance for three variable and five variable regressions under a range of configurations of the _js, for sample sizes ranging from 50 to 1,000. The reported results correspond to regressions with population R² equal to .5 only. Other simulation results not reported here indicate that the conclusions of the study are not sensitive to variations in R². The elements of the covariance matrix of Equation 6 were manipulated to yield the desired configurations of Pratt measures; in all cases (and without loss of generality), the diagonals of , namely, the variances of y and the independent xs, were set to 1. For three variables, six cases involving different configurations of normalized Pratt measures were studied, as follows:

1. three measures corresponding to a case of model multicollinearity;
   
2. one big measure and two small measures (recall that the normalized Pratt measures must sum to 1);
   
3. three equally sized measures;
   
4. one big, one medium, and one small measure;
   
5. a case involving one suppressor variable;
   
6. a case involving a nonignorable negative index.
   

Similarly structured configurations were examined for five explanatory variables, with the exception that cases E and F were combined for the five variable model. Values of the true normalized Pratt measures are shown in Table 1 for each case. In cases B, C, and D, for both three and five explanatory variables, all Pratt measures are positive and can therefore be meaningfully interpreted as measures of importance. In case E, there is one negative normalized measure of relatively small magnitude for both the three- and five-variable situations. As shown by Thomas et al. (1998), the normalized measures corresponding to the nonsuppressors still provide approximate measures of relative importance in such situations, although the recommended procedure is to partial out the contribution of the suppressor variable prior to assessing importance. This aspect of importance analysis has not been considered in the present study. Cases A (for three and five variables) and F (for three variables) all feature negative normalized Pratt measures of substantial magnitude. These measures cannot, therefore, be interpreted in terms of importance and represent cases in which confidence intervals could profitably be used to confirm the difficulty. The simulation experiment consisted of 500 independent replications of the following five-step process for the three- and five-variable cases in turn:

1. A sample of 1,000 observations was drawn from the multivariate normal distribution specified in Equation 6 corresponding to each configuration of Pratt indices.
   
2. The normalized Pratt measures of Equation 5 and their large sample variance estimates for Equation 11 were computed.
   
3. The individual and simultaneous confidence intervals given in Equations 13 and 14 were then computed for each variable and tested against the known values of the _js. A counter was updated each time an interval covered its corresponding parameter. A nominal confidence interval level of 95% was used in all cases, corresponding to an alpha level in Equations 13 and 14 of 5%.
   
4. The process was repeated for sample sizes 500, 250, 100, and 50. To induce a correlation between estimates at different sample sizes, and hence reduce the simulation error for comparisons across sample sizes, the smaller sample sizes were selected as subsets of the original 1,000 (see Thomas & Rao, 1987).
   

Based on the 500 replications, a number of summary measures were computed for the three- and five-variable models, namely:

1. Simulation estimates of the expected values of the _js, denoted Ê^s(_j). For each experimental condition, these were computed as means of the individual Pratt indices over the 500 replications.
   
2. Simulation estimates of the expected values of the estimated asymptotic variances (_j). Denoted Ê^s(_j), these were computed for each experimental condition as means of the variance estimates for Equation 11 over the 500 replications.
   
3. Simulation estimates of the true variances of the _js, denoted ^s(_j). These were computed directly from the 500 replications of the _js corresponding to each experimental condition.
   
4. Coverage rates of the individual confidence intervals of Equation 13.
   
5. Coverage rates of the simultaneous confidence intervals of Equation 14.
   

Results are discussed in the following section.

	Results

Top Abstract Deficiencies in Traditional... Pratt's Measure Point Estimates and Confidence... The Simulation Experiment Results Example Summary and Conclusions Appendix References

 
Simulation Estimates of the Expected Values
Values of the Ê^s(_j)s, namely, the simulation estimates of the expected values of the _js, are shown in Table 1 for all cases, for both three- and five-variable situations, and for sample sizes of 50 and 250. Asymptotic values are shown in bold type. It is clear from the table that the expected values of the normalized Pratt measures converge rapidly to their true (and asymptotic) values given by Equation 7 as sample size increases. Thus, the bias in the estimated normalized Pratt index of Equation 5 converges rapidly to zero with increasing sample size, and the estimator can be considered effectively unbiased for sample sizes of 250 or more. The correspondence between estimates and asymptotic values is close even for a sample size as low as 50.

Asymptotic Variances and True Variances
To simplify the assessment of the variance estimates, which depend inversely on N, all numerical results will be reported as N times the variance under consideration. Simulation estimates of N times the expected values of the asymptotic variance estimates given by Equation 11, denoted N x Ê^s(_j), behave in similar fashion to that described above for the Ê^s(_j)s. That is, they approach their asymptotic value quite rapidly, the convergence being slowest for case A, which features appreciable multicollinearity. Even for case A, all estimated expected values of N x (_j) are within 10% of the asymptotic variance AV(_j) for sample sizes of 250 or more. For the other cases, the divergence between expected value and asymptotic variance is much less, well within 5% in most cases. Tabular details are omitted in the interests of space.

Simulation estimates of N times the true variances, denoted N x ^s(_j), are shown in Table 2, for sample sizes of 50, 250, and 1,000, along with the asymptotic variances (in bold type). It can be seen that the simulation estimates of Table 2 exhibit greater variation than those of Table 1, which is not surprising given that variance estimates necessarily have greater variability than mean estimates. Nevertheless, it can be seen that the simulation estimates of the true variance (xN) are generally within 10% of the asymptotic variance for sample sizes of 250 or more. Deviations are not much larger even for sample sizes as low as 50. For case A of Table 2—the case featuring a multicollinear model—it can be seen that the variables involved in the multicollinearity have very high estimated and asymptotic variances. Even for these rather extreme cases, the asymptotic variance provides a close match to N times the true variance. In summary, the results of Tables 1 and 2 suggest that good coverage properties will be exhibited by the individual and simultaneous confidence intervals.

View this table: [in this window] [in a new window]   TABLE 3 Coverage of Individual and Simultaneous Confidence Intervalsa for the js: Three Explanatory Variables

View this table: [in this window] [in a new window]   TABLE 4 Coverage of Individual and Simultaneous Confidence Intervalsa for the js: Five Explanatory Variables

Overall Summaries of Individual and Simultaneous Coverage
Figures 1 and 2 provide an overall summary of the results for individual and simultaneous coverage levels, respectively. Both figures display results for all configurations of the normalized Pratt measures for both three- and five-variable models on the same graph. Moreover, coverage levels of individual variables are not identified separately in Figure 1. The two figures are useful in that they provide a gross overview of the performance of both individual and simultaneous confidence interval procedures as a function of sample size. Results for all sample-size settings are displayed—namely, 50, 100, 250, 500, and 1,000. Figure 1 displays 43 coverage level estimates for each sample size (6 cases with 3 variables each and 5 cases with 5 variables each), and if it is assumed that the true coverage level for each is equal to the nominal 95%, then a Bonferroni confidence band for the 43 coverage estimates at each sample size setting based on 500 Bernouilli replications should be 95% ±3.1%. Figure 1 thus illustrates the primary conclusion drawn from Tables 3 and 4 for the individual confidence intervals, that over a wide range of test situations, the estimated individual coverage rates are consistent with a true coverage level close to the nominal 95% whenever the sample size is 250 or greater. In fact, it can be seen that individual confidence intervals might still be useable for sample sizes down to 100 as long as individual coverage levels as low as 90% can be tolerated (at the nominal 95% level).

View larger version (8K): [in this window] [in a new window]   FIGURE 1 Coverage levels of the individual normalized Pratt confidence intervals for three and five variables and for all configurations of normalized Pratt measures.

View larger version (7K): [in this window] [in a new window]   FIGURE 2 Coverage levels of the simultaneous normalized Pratt confidence intervals for the three- and five-variable models and for all configurations of normalized Pratt measures.

This is surprising at first sight because simultaneous Bonferroni confidence intervals based on exact statistics have a coverage level that is necessarily equal to or greater than 1 – (see, e.g., Miller, 1981, p. 67). There are two observations that help explain the above finding. First, Miller (1981, p. 253) comments favorably on the tightness of the Bonferroni bound when the individual ' are small. Because these are set in this article at ' = /p for = .05, with p = 3 and 5, the theoretical conservativeness of the Bonferroni bound is likely to be slight. Second, the Bonferroni intervals in this article are constructed using approximate, asymptotically based distribution theory, and it is typical for the coverage performance of asymptotic confidence intervals to deteriorate as the individual levels decrease (see, e.g., Thomas, Singh, & Roberts, 1996). As described in the text preceding Equation 14, the construction of simultaneous Bonferroni intervals involves replacing the of the individual intervals by the smaller value ' = /p, which then results in a corresponding deterioration in the approximate simultaneous interval coverage. It is clear from the results of Figure 2 that this effect dominates the theoretical (but slight) conservativeness of the simultaneous Bonferroni intervals. In conclusion, asymptotic simultaneous confidence interval procedures for the normalized Pratt measures are not recommended for use with sample sizes much less than 250.

In view of the above results and recommendations, it is important to note that a decision on whether to use individual or simultaneous intervals should not be based on the differences in coverage performance to be expected at a given sample size. As discussed in the text following Equation 14, individual intervals should only be used when there is some a priori interest in the importance of a particular variable. Whenever variable importances are to be examined and compared a posteriori, simultaneous intervals are required, in which case the minimum sample size recommendation of 250 should be respected. Should a practitioner choose to use the individual 95% intervals in a situation requiring simultaneous inference, then the simultaneous coverage would be approximately 85% for a three-variable regression and 75% for a five-variable case—levels that are too low for useful inference.

	Example

Top Abstract Deficiencies in Traditional... Pratt's Measure Point Estimates and Confidence... The Simulation Experiment Results Example Summary and Conclusions Appendix References

 
This example is based on a subset of data taken from a large study on the work-life conflict experienced by Canadian employees, documented in a report to Health Canada by Duxbury and Higgins (2003). To illustrate the use of the normalized Pratt measures and their corresponding confidence intervals, a regression analysis featuring one dependent variable and six explanatory variables was investigated. The dependent variable consisted of a work-family conflict measure based on a five-item scale: work-to-family interference (Duxbury & Higgins, 2003). The predictor variables consisted of a theoretically linked set of indicators of the culture of each respondent’s organization with respect to work-life balance. These individual item predictors were measured on a 5-point scale, and represented degrees of agreement or disagreement with the following six statements:

ORG: Organization promotes environment supportive of work/personal balance

LEAVE: Have seriously thought about switching to another organization

COWSUP: Coworkers supportive of personal/family responsibilities

LONG: Inability to work long hours would limit career

ADVANCE: Family responsibilities hamper advancement

NONO: Not acceptable to say no to more work

The sample size was 3,320, large enough to guarantee the applicability of the proposed asymptotic confidence interval procedures, and the regression R² was .382. Given that this example features a 5-point measurement scale for the predictor variables, application of the proposed confidence interval procedures is a technical violation of the assumption of multivariate normality on which their derivation is based. Thus, it is implicitly assumed in the analysis of this specific example that the violation does not materially affect the conclusions. This is an important issue in general, given the importance of discrete Likert-type scales in behavioral research, and it will be discussed further below. Table 5 displays values of (a) the standardized regression coefficients; (b) the simple correlation between the dependent variable and each of the six explanatory variables; (c) the t statistics; (d) the partial correlation between the dependent variable and each of the explanatory variables, adjusted for the remaining five, denoted _yj|(j);(e) the normalized Pratt measures (the _js) for each explanatory variable, shown in bold; and (f) the corresponding 95% simultaneous confidence intervals for the _js. Figure 3 illustrates the simultaneous confidence intervals for the normalized Pratt measures.

View larger version (6K): [in this window] [in a new window]   FIGURE 3 Simultaneous 95% confidence intervals for the normalized Pratt measures in the work-life interference example. Note: LEAVE = have seriously thought about switching to another organization; LONG = inability to work long hours would limit career; ORG = organization promotes environment supportive of work/ personal balance; ADVANCE = family responsibilities hamper advancement; NONO = not acceptable to say no to more work; COWSUP = coworkers supportive of personal/family responsibilities.

Quite apart from any similarities and differences in the orderings given by the above statistics, it should be recalled that the normalized Pratt measure is the unique measure of relative importance for the explanatory variables that satisfies Pratt’s axioms and definitions. Thus, from the _js of Table 5, it can be stated that LEAVE is about 3 times more important than NONO, a relative comparison that cannot be justified using any of the other measures of importance. The interval estimates illustrated graphically in Figure 3 show more clearly the importance relationship between these variables. Although the point estimate of the normalized Pratt measure for LEAVE is larger than that for LONG and than that for ORG, it can be seen that there is a large overlap between their simultaneous interval estimates. Thus, it would not be valid to infer that LEAVE is more important than LONG, or ORG, based on point estimates alone. The results of Figure 3 suggest that the six variables should be ranked in importance by first grouping them into three hierarchies, with LEAVE, LONG, and ORG comprising the most important group of variables for explaining work-to-family interference. ADVANCE and NONO would rank second in importance, although it should be noted that this categorization of the five most important variables into two distinct groups of three and two, respectively, requires that the small overlap between NONO and COWSUP be ignored. This is a judgment call, which is based on the fact that the intervals are approximate and that an interpretation in strict hypothesis testing style would be inappropriate for this type of data analysis. Furthermore, it can be seen that the confidence interval for COWSUP includes small negative values, so that the most reasonable interpretation is that COWSUP has negligible importance and can be dropped from all importance considerations.

It should also be noted that specific pairwise comparisons, between COWSUP and NONO, for example, could be sharpened by extending the analysis described in the appendix to the construction of simultaneous confidence intervals for all p(p – 1)/2 pairwise comparisons. This would require further simulation to determine the performance of these asymptotic intervals as a function of sample size, a project that has yet to be implemented.

Likely Effect of Discrete Predictors
Discretely measured predictor variables of the Likert-type illustrated in the above example are very popular in the behavioral sciences, and it is commonly assumed that the discreteness will not adversely affect an analysis provided that at least five categories are used and provided that the discrete data do not exhibit excessive skewness or kurtosis. Although it is not the intent of this article to investigate in detail the robustness of the proposed intervals to discreteness and nonnormality, it is relevant to consider what the effects of discreteness are likely to be in the above example. A normal probability plot of the regression residuals (not shown) provides no evidence at all of nonnormality, indicating that the conditional distribution of WTFINT, given the predictor variables, is normal. Thus, the investigation of nonnormality and its effects can focus on the joint distribution of the six predictors. Summary statistics based on the sample of 3,320 observations are shown in Table 6, and from the values of univariate skewness and kurtosis, it is evident that the assumption of multivariate normality is violated (tests on Mardia’s 1970 measures of multivariate skewness and kurtosis also reject normality, with p < .0001). The question of interest, however, is what effect this violation will have on the Pratt indices and their corresponding confidence intervals.

The validity of the inferences for this specific example can also be investigated by regarding the discrete variables as observed versions of underlying multivariate normal variables that have been categorized with respect to a set of cutpoints (see, e.g., Bollen, 1989, p. 439). One effect of categorizing underlying variables is to reduce the magnitude of the Pearson correlation coefficients between them (see Krieg, 1999, and the references therein), an effect that will then generate bias in regression coefficients estimated using the categorized data. If the observed variables for the work-family life example are characterizing in terms of underlying normal variables, an indication of the effect of their discreteness can be obtained by (a) estimating the underlying correlations between the dependent and predictor variables, and among the predictor variables, by means of polyserial and polychoric correlations, respectively; (b) evaluating the Pratt indices and corresponding confidence intervals based on these underlying correlations. Note that these ‘‘Pratt indices’’ and ‘‘confidence intervals,’’ displayed in Figure 4, are not intended to provide an alternative analysis because the variances of the polyserial and polychoric correlations will not be accounted for. They are intended only to provide an indication of the sensitivity of the work-life interference example to variable discreteness and nonnormality. An initial indication of this sensitivity is given by the effect on the regression R², which increases by 6% when the polyserial and polychoric correlations are used, from 38.2% to 40.4%. From Figure 4, it can be seen that the corresponding point estimates and confidence intervals are very similar to those displayed in Figure 3. Thus, the indications are that the effect of variable discreteness on the Pratt confidence intervals in the work-life conflict example is slight, and that the original inferences can be trusted.

View larger version (6K): [in this window] [in a new window]   FIGURE 4 Simultaneous 95% confidence intervals based on polyserial and polychoric correlations. Note: LEAVE = have seriously thought about switching to another organization; LONG = inability to work long hours would limit career; ORG = organization promotes environment supportive of work/ personal balance; ADVANCE = family responsibilities hamper advancement; NONO = not acceptable to say no to more work; COWSUP = coworkers supportive of personal/family responsibilities.

	Summary and Conclusions

Top Abstract Deficiencies in Traditional... Pratt's Measure Point Estimates and Confidence... The Simulation Experiment Results Example Summary and Conclusions Appendix References

As discussed in the review of the literature, Pratt’s (1987) measure is not the only viable measure of importance. Budescu’s (1993) ‘‘dominance’’ approach recently extended by Azen and Budescu (2003) and the ‘‘criticality’’ approach of Azen et al. (2001) comprise intuitively reasonable approaches to the issue, and they also provide inferential techniques for comparing their assessments across variables. However, because these inferences are based on bootstrapping methods, both approaches become heavily computer intensive and require special software. The methods developed in this article, featuring both point and interval estimates of the normalized Pratt measure of variable importance, are far easier to implement and require no special software. Thus, it is hoped that the results of this article will encourage researchers to examine the importance of their variables. To do so would be consistent with the recommendations of Kirk (1996) and others in favor of using ‘‘effect sizes’’ to assist with the interpretation of statistical analyses. To know that a regression coefficient is nonzero should only be the beginning of the interpretation. A full statistical assessment of the relative importance of the variable compared to the other independent variables should be regarded as an essential aspect of the analysis.

	Appendix

Top Abstract Deficiencies in Traditional... Pratt's Measure Point Estimates and Confidence... The Simulation Experiment Results Example Summary and Conclusions Appendix References

 
For a sample of size N, the linear model 1 can be written as

(A1)

where 1_N represents an N x1 vector of ones, y is an N x1 vector of observations of the dependent variable, X₁ is an N x p matrix of observed explanatory variables, that is, X₁ = (x₁₁, x₁₂, . . . x_1p), and u is an N x1 vector of disturbances.

Consistency
The population standardized Pratt index given in Equation 7 can be estimated as

(A2)

where Q is a centering matrix defined as Q = (I_N – 1_N1_N'/N) and I_N is an identity matrix of dimension N. Because X₁'Qy/N is a consistent estimator of _xy, and X₁'QX₁/N is a consistent estimator of _xx, the consistency result of Equation 8 follows directly.

Asymptotic Normality
To prove asymptotic normality, it is convenient to start with the numerator of Equation A2, with y replaced by Equation A1. In addition, because Q = Q' = Q², and because X₁ and x_1j in Equation A2 can be replaced by X = QX₁ and x_j = Qx_1j, it follows that the numerator of Equation A2, denoted p_j, can be written as

(A3)

Then,

(A4)

where E_ux denotes expectation with respect to the joint distribution of u and X, E_x denotes expectation with respect to the marginal distribution of X, and E_u|x (to be used later) denotes expectation with respect to the conditional distribution of u. Consider next the centered and scaled version of p_j/N, given by

(A5)

It can be shown that as N , the variances of the first three terms on the right-hand side of Equation A5 are O(1), whereas the variance of the last term is O(N^–1/2); that is, it tends to zero as N . Thus, the last term is of order N^–1/2 in probability (see Barndorff-Nelson & Cox, 1989, p. 31), and to first order, Equation A5 can be written as follows:

(A6)

The first two terms on the right-hand side of Equation A6 comprise a linear combination of inner products given by

(A7)

which are asymptotically normal with zero mean and variance to be determined later, given that the N random variables x_jlu_l are independent and identically distributed, with zero mean, and provided that their second moments are finite. The latter condition is satisfied under normality, but the normality assumption is clearly not essential. A similar central limit theorem applies to the third term of Equation A6, which consists of the inner products

(A8)

Provided the second moment of x_jlx_kl is finite, inner products of the form of Equation A8 will also be asymptotically normal. Therefore, p_j/N is asymptotically normal since the right-hand side of Equation A6 consists of a linear combination of asymptotically normal terms. To investigate the asymptotic normality of the normalized Pratt index, _j, note that

(A9)

Given that p_j/N is asymptotically normal, is also asymptotically normal, and the correlation between these terms can be evaluated from Equation A6. Because the means of the numerator and denominator of Equation A9 are nonzero, it follows by linearization that _j is also asymptotically normal (see Barndorff-Nielsen & Cox, 1989, p. 41).

Asymptotic Variance of _j
To evaluate the asymptotic variance it is convenient to proceed via the variance decomposition

(A10)

where the subscripts on the variances are defined equivalently to those for the expectations.

Conditional Asymptotic Variance
The asymptotic conditional variance of the ratio specified in Equation A9 can be obtained by linearization, as described by Barndorff-Nielsen and Cox (1989, p. 41). With p_j/N and denoted by w and z, respectively, the linearized form of V_u|x(_j) can be expressed in terms of conditional expectations and variances as

(A11)

The conditional expectations in Equation A11 can be obtained from Equation A3, from which it follows that

(A12)

Given that X'X/N consistently estimates _xx, Equation A12 can be replaced by

(A13)

The corresponding expression for follows immediately. The conditional variances and the covariance term can be derived correct to O(N^–1) from Equation A6. Under the assumption of normal and independent disturbances u_l, the derivation is tedious but routine. The results are as follows:

(A14)

(A15)

(A16)

Substitution of Equations A14 through A16 into Equation A11 leads to the terms in the curly brackets of Equation 10 for AV(_j), the asymptotic variance of _j.

Variance of the Conditional Mean
To evaluate the second term in Equation A10, namely, the variance of the conditional mean of _j, the conditional mean of _j must first be specified as a function of X. From the usual linearization argument, it follows from Equations A9 and A12 that

(A17)

Thus, it remains to evaluate the variance of the expression of Equation A17 under the normality assumption described in Equation 6 and the accompanying text. The linearized expression for the variance of a ratio given in Equation A11 again applies, but in this case, the individual variances and covariances involve fourth moments of the x_js. The advantage of the normality assumption is that fourth moments can be expressed as functions of variances and covariances, which greatly facilitates the derivation. Expressions for the required moments can be found in Kendal and Stuart (1977, p. 85). After some algebra, the required variance yields the final term in Equation 10, thus completing the derivation of the expression for the asymptotic variance of _j. Details are omitted in the interests of space.

	Footnotes

 
D. ROLAND THOMAS is Professor, Sprott School of Business, Carleton University, 1125 Colonel By Drive, Ottawa, Ontario, Canada K1S 5B6;rthomas{at}sprott.carleton.ca. He is interested in a variety of modeling techniques of relevance to the business and social sciences and in the analysis of complex survey data.

PENGCHENG ZHU is a PhD student in the Sprott School of Business, Carleton University, 1125 Colonel By Drive, Ottawa, Ontario, Canada K1S 5B6;phil_zhu1979{at}yahoo.ca. He specializes in finance and related applications of statistics.

YVES J. DECADY, PhD, is an Analyst in the Labor Statistics Division, Statistics Canada, Jean Talon Building, Ottawa, Ontario, Canada, K1A 0T6;Yves.Decady{at}statcan.ca. His current area of research is labor dynamics.

The first author’s work was supported by a grant from the Natural Sciences and Engineering Research Council of Canada and by grants from two Canadian research networks: namely, MITACS (Mathematics of Information Technology and Complex Systems) and NPCDS (National Program on Complex Data Structures). The authors wish to thank Professors Linda Duxbury and Christopher Higgins for permission to use their data to illustrate the proposed methodology.

Manuscript received August 27, 2004. Accepted for publication July 13, 2005.

	References

Top Abstract Deficiencies in Traditional... Pratt's Measure Point Estimates and Confidence... The Simulation Experiment Results Example Summary and Conclusions Appendix References

 
Achen, CH. (1982). Interpreting and using regression. Beverly Hills, CA: Sage.

Journal of Educational and Behavioral Statistics, Vol. 32, No. 1, 61-91 (2007)
DOI: 10.3102/1076998606298037


CiteULike    Complore    Connotea    Del.icio.us    Digg    Reddit    Technorati    Twitter    What's this?

This Article Abstract Free Full Text (Free PDF) Free Alert me when this article is cited Alert me if a correction is posted Services Email this article to a friend Similar articles in this journal Alert me to new issues of the journal Add to Saved Citations Download to citation manager Request Permissions Request Reprints Add to My Marked Citations Citing Articles Citing Articles via Google Scholar Citing Articles via Scopus Google Scholar Articles by Thomas, D. R. Articles by Decady, Y. J. Search for Related Content Social Bookmarking                         What's this?