EvaluationDashboard.com

My dissertation research involves multivariate multilevel modeling. I would prefer to use lme4, but it can only regress one dependent variable at a time. I am using Mplus because it allows the necessary constraints for multivariate multilevel modeling.

In this entry I demonstrate some univariate multilevel modeling constraints, following lmer’s sleepstudy example. Unlike other Mplus multilevel modeling demonstrations, I avoid Mplus’ syntactic shortcuts (i.e., “TYPE = TWOLEVEL”) in order to highlight the constraints and the underlying measurement assumptions made in conventional multilevel modeling.

I replicate the lmer example first (see below). One of the first differences to note is that Mplus does not apply restricted maximum likelihood, so the REML argument must be set to FALSE. Additionally, I recast the sleepstudy data from long to wide for Mplus. The Mplus estimates match the lmer results. How? As annotated below, Mplus achieves the same results as conventional multilevel models by:

1. treating the fixed effects (at level two) as latent variables (indicated in this example by level-one observations at points in time)
2. assuming and imposing configural, scalar and uniqueness invariance.

Configural invariance is imposed by constraining loadings on the latent intercept to 1. Scalar invariance is imposed by constraining indicator intercepts to 0. Uniquenesses are estimated but constrained to be equal. This makes sense from a Generalizability Theory perspective when you consider that the measurement error variance component is for a single observation drawn from a universe of admissible observations.

lmer’s sleepstudy example with random slopes for time

> summary(lmer(Reaction ~ Days + (Days | Subject), sleepstudy, REML = F))
Linear mixed model fit by maximum likelihood ['lmerMod']
Formula: Reaction ~ Days + (Days | Subject)
Data: sleepstudy
AIC       BIC    logLik  deviance
1763.9393 1783.0971 -875.9697 1751.9393
Random effects:
Groups   Name        Variance Std.Dev. Corr
Subject  (Intercept) 565.52   23.781
Days         32.68    5.717   0.08
Residual             654.94   25.592
Number of obs: 180, groups: Subject, 18
Fixed effects:
Estimate Std. Error t value
(Intercept)  251.405      6.632   37.91
Days          10.467      1.502    6.97
Correlation of Fixed Effects:
(Intr)
Days -0.138

Mplus random slopes example with sleepstudy data

INPUT INSTRUCTIONS
DATA: FILE = "Sleepstudy.dat";
VARIABLE:
NAMES = Subject React_0 React_1 React_2 React_3 React_4 React_5 React_6 React_7
React_8 React_9;
MISSING=.;
IDVARIABLE IS Subject;
USEVARIABLE = Subject React_0 React_1 React_2 React_3 React_4 React_5 React_6 React_7
React_8 React_9;
ANALYSIS:
ESTIMATOR = ML;
MODEL:
!Specify intercept fixed effect (latent intercept factor).
!Fix loadings at 1; assume configural invariance.
FI by React_0-React_9@1;
[FI]; !intercept fixed effect (mean)
!Specify fixed effect for days (latent slope factor).
!Fix loadings at corresponding day.
FS by React_0@0 React_1@1 React_2@2 React_3@3 React_4@4 React_5@5
React_6@6 React_7@7 React_8@8 React_9@9;
[FS]; !slope fixed effect (mean)
!Suppress indicator intercepts; assume scalar invariance.
[React_0 - React_9 @0];
!Specify residual (error) variance component; assume invariant uniquenesses.
React_0 - React_9 (1);
!Specify factor (facet) variance and covariance components.
FI FS;
FI with FS;
!Suppress slope variance if specifying a random intercept model.
!FS@0;
!FI with FS @0;
INPUT READING TERMINATED NORMALLY
Replication of lmer sleepstudy random slopes example;
SUMMARY OF ANALYSIS
Number of groups                                                 1
Number of observations                                          18
Number of dependent variables                                   10
Number of independent variables                                  0
Number of continuous latent variables                            2
Observed dependent variables
Continuous
REACT_0     REACT_1     REACT_2     REACT_3     REACT_4     REACT_5
REACT_6     REACT_7     REACT_8     REACT_9
Continuous latent variables
FI          FS
Variables with special functions
ID variable           SUBJECT
Estimator                                                       ML
Information matrix                                        OBSERVED
Maximum number of iterations                                  1000
Convergence criterion                                    0.500D-04
Maximum number of steepest descent iterations                   20
Maximum number of iterations for H1                           2000
Convergence criterion for H1                             0.100D-03
Input data file(s)
C:/Sleepstudy.dat
Input data format  FREE
SUMMARY OF DATA
Number of missing data patterns             1
COVARIANCE COVERAGE OF DATA
[omitted: 100% coverage]
THE MODEL ESTIMATION TERMINATED NORMALLY
TESTS OF MODEL FIT
Chi-Square Test of Model Fit
Value                            141.141
Degrees of Freedom                    59
P-Value                           0.0000
Chi-Square Test of Model Fit for the Baseline Model
Value                            260.682
Degrees of Freedom                    45
P-Value                           0.0000
CFI/TLI
CFI                                0.619
TLI                                0.710
Loglikelihood
H0 Value                        -875.970
H1 Value                        -805.399
Information Criteria
Number of Free Parameters              6
Akaike (AIC)                    1763.939
Bayesian (BIC)                  1769.282
Sample-Size Adjusted BIC        1750.845
(n* = (n + 2) / 24)
RMSEA (Root Mean Square Error Of Approximation)
Estimate                           0.278
90 Percent C.I.                    0.220  0.337
Probability RMSEA <= .05           0.000
SRMR (Standardized Root Mean Square Residual)
Value                              0.379
MODEL RESULTS
Two-Tailed
Estimate       S.E.  Est./S.E.    P-Value
FI       BY
REACT_0            1.000      0.000    999.000    999.000
REACT_1            1.000      0.000    999.000    999.000
REACT_2            1.000      0.000    999.000    999.000
REACT_3            1.000      0.000    999.000    999.000
REACT_4            1.000      0.000    999.000    999.000
REACT_5            1.000      0.000    999.000    999.000
REACT_6            1.000      0.000    999.000    999.000
REACT_7            1.000      0.000    999.000    999.000
REACT_8            1.000      0.000    999.000    999.000
REACT_9            1.000      0.000    999.000    999.000
FS       BY
REACT_0            0.000      0.000    999.000    999.000
REACT_1            1.000      0.000    999.000    999.000
REACT_2            2.000      0.000    999.000    999.000
REACT_3            3.000      0.000    999.000    999.000
REACT_4            4.000      0.000    999.000    999.000
REACT_5            5.000      0.000    999.000    999.000
REACT_6            6.000      0.000    999.000    999.000
REACT_7            7.000      0.000    999.000    999.000
REACT_8            8.000      0.000    999.000    999.000
REACT_9            9.000      0.000    999.000    999.000
FI       WITH
FS                11.055     42.876      0.258      0.797
Means
FI               251.405      6.632     37.906      0.000
FS                10.467      1.502      6.968      0.000
Intercepts
REACT_0            0.000      0.000    999.000    999.000
REACT_1            0.000      0.000    999.000    999.000
REACT_2            0.000      0.000    999.000    999.000
REACT_3            0.000      0.000    999.000    999.000
REACT_4            0.000      0.000    999.000    999.000
REACT_5            0.000      0.000    999.000    999.000
REACT_6            0.000      0.000    999.000    999.000
REACT_7            0.000      0.000    999.000    999.000
REACT_8            0.000      0.000    999.000    999.000
REACT_9            0.000      0.000    999.000    999.000
Variances
FI               565.516    265.266      2.132      0.033
FS                32.682     13.573      2.408      0.016
Residual Variances
REACT_0          654.941     77.186      8.485      0.000
REACT_1          654.941     77.186      8.485      0.000
REACT_2          654.941     77.186      8.485      0.000
REACT_3          654.941     77.186      8.485      0.000
REACT_4          654.941     77.186      8.485      0.000
REACT_5          654.941     77.186      8.485      0.000
REACT_6          654.941     77.186      8.485      0.000
REACT_7          654.941     77.186      8.485      0.000
REACT_8          654.941     77.186      8.485      0.000
REACT_9          654.941     77.186      8.485      0.000
QUALITY OF NUMERICAL RESULTS
Condition Number for the Information Matrix              0.453E-04
(ratio of smallest to largest eigenvalue)

I came across the following plot while reading “Ensuring Fair and Reliable Measures of Effective Teaching“.

Maybe I’ve spent too much time reading Andrew Gelman’s blog because I noticed two issues with the plot that could be improved. The most serious issue is that the first two sets of bars display correlation coefficients, but the last set of bars displays squared correlation coefficients. It doesn’t matter that the first two sets represent a type of validity and the last represents reliability. What matters is that the bars incorrectly imply a common scale for the entire figure and could mislead readers. Perhaps the authors made the somewhat common mistake of assuming that a reliability coefficient is a correlation coefficient because both are denoted r? It’s probably just a graphic design mistake that the authors didn’t catch. The second issue I noticed is far less serious (and is debatable). Since the bars do not represent counts, the coefficients should be displayed as dots.

I used to create an improved (but less colorful) version of their plot that displays only correlation coefficients with dots instead of bars. The new version shows more clearly that each variety of teacher effectiveness composite score “predicts” itself (from a test-retest perspective) better than any of them predict the criteria. In other words, the improved plot illustrates how score reliability is a necessary but insufficient condition that can act as a ceiling on validity.

I have contributed my generalizability functions to the QME: Quantitative Methods in Education package. I described my efforts in an earlier post. This project continues to be a great learning experience for me, especially in terms of seeing how classical test theory, generalizability theory, and multilevel modeling overlap. I will probably submit a proposal to demonstrate the package at either NCME or AERA in 2014. You can use the following commands to install the package:

library(devtools)
install_github("QME", "zief0002")

Minnesota has large and persistent achievement gaps. It’s an empirically supported proposition no matter how you look at it. But if we are going to tackle the problem, it does matter how we look at it! Well-intentioned advocates and researchers sometimes talk about “the achievement gap” as if there is one definition. Are they referring to test score differences? Or graduation rates? Or gaps in educational opportunities? Are they referring to math? Or reading? Or science? Are they referring to students of color? Or students in poverty? Or English learners? Or special education students? Or all four groups? They don’t often say. Referring to “the achievement gap” is good rhetoric, but I contend that a lack of specificity is impeding progress. I also contend that we can learn from schools that are closing achievement gaps.

Advocates and researchers would be wise to use the Minnesota Department of Education’s (MDE) achievement gap reduction measure when possible. MDE quantifies gaps as the difference between the growth of historically under-achieving groups relative to more advantaged groups. The calculation yields a negative number if a school helped reduce achievement gaps in Minnesota and a positive number if a school added to existing gaps. (See page 78 of Minnesota’s approved No Child Left Behind (NCLB) waiver for the methodology.) Here’s why I think advocates and researchers should consistently publicize and track the achievement gap reduction measure:

• It’s official. Minnesota’s accountability system holds schools accountable for achievement gap reduction (i.e., it’s one of the Multiple Measures). It’s recognized by the U.S. Department of Education, it’s reported annually by MDE, and it’s increasingly familiar to teachers, administrators, and parents.
• It’s fair. Student growth is a better measure of the contribution of a school or intervention than proficiency rates, which reflect the economic affluence and selectivity. When we over-emphasize proficiency it prevents us from finding schools in lower-income areas that are closing gaps. Growth measures only compare students and schools to those with similar achievement at the beginning of the school year.
• It’s embodies a theory of action. As long as some groups are starting the school year behind, then the only way to close gaps is to accelerate their learning relative to their more advantaged peers. It’s a race, and traditional achievement gap measures paint a picture of the starting line. The achievement gap reduction measure tells the pace of learning over a school year.

I used to create an interactive map to help advocates and researchers become familiar with the achievement gap reduction measure and encourage recognition of schools that are helping close achievement gaps in Minnesota. The data underlying the map are publicly available from MDE’s Data Center. MDE does not publish achievement gap reduction for each student group–it only publishes overall achievement gap reduction measures (i.e., student-weighted averages across all historically lower-achieving student groups). I averaged the annual values over three years (2010-2012) to improve precision. You can zoom in and click on a school to learn more about it. Now that we know which schools are helping close achievement gaps in Minnesota, we should recognize their successes and learn from them. What are the successful schools doing? How can we encourage other schools to adopt their strategies?

View Larger Map

Remembering Hap Casmey inspired me to write this entry. I enjoyed getting to know Hap later in his life and hearing about his influence on education policy in Minnesota.

The annual conference of the American Evaluation Association came to Minneapolis last week. I enjoyed getting re-acquainted with many of my evaluation colleagues and learning about evaluation studies from around the world without leaving the city I call home. I presented a poster titled “Teacher evaluation and the Standards of Effective Instruction rating instrument: Psychometric considerations“.