I was recently tasked with estimating the reliability of scores composed of items on different scales. The lack of a common scale could be seen in widely varying item score means and variances. As discussed by Graham, it’s not widely known in applied educational research that Cronbach’s alpha assumes essential tau-equivalence and underestimates reliability when the assumption isn’t met. A friend made a good suggestion to consider stratified alpha, which is commonly used to estimate internal consistency when scores are composed of multiple choice items (scored 0-1) and constructed response items (e.g., scored 0-4). However, the assessment with which I was concerned does not have clear item-type strata. I decided to estimate congeneric reliability because it makes few assumptions (e.g., unidimensionality) and doesn’t require grouping items into essentially tau-equivalent strata.
With the help of Graham’s paper and a LISREL tutorial by Raykov I wrote an 
program that estimates congeneric reliability. The program uses Fox’s sem() library to conduct the confirmatory factor analysis with minimal constraints (variance of common true score fixed at 1 for identifiability). The estimated loadings and error variances are then summed to calculate reliability (i.e., the ratio of true score variance to observed score variance) as:
.
I obtained a congeneric reliability estimate of 0.80 and an internal consistency estimate of 0.58 for the scores I was analyzing. If the items had been essentially tau-equivalent, then the reliability estimates would have been the same. If I had assumed tau-equivalence, than I would have underestimated the reliability of the total scores (and overestimated standard errors of measurement). The example below replicates Graham’s heuristic example.
> ############################################
> #Replicate results from Graham (2006) to check congeneric reliability code/calculations.
>
> library(sem)
> library(psych)
> library(stringr)
>
> #Variance/covariance matrix
> S.graham <- readMoments(diag = T, names = paste("x", 1:7, sep = ""))
1: 4.98
2: 4.60 5.59
4: 4.45 4.42 6.30
7: 3.84 3.81 3.66 6.44
11: 5.71 5.67 5.52 4.91 11.86
16: 23.85 23.68 22.92 19.87 34.28 127.65
22: 46.53 46.20 44.67 38.57 62.30 244.36 471.95
29:
Read 28 items
>
> ############################################
> #A function to estimate and compare congeneric reliability and internal consistency
> funk.congeneric <- function(cfa.out) {
+ names.loadings <- str_detect(names(cfa.out$coeff), "loading")
+ names.errors <- str_detect(names(cfa.out$coeff), "error")
+ r.congeneric <- sum(cfa.out$coeff[names.loadings]) ^ 2 /
+ (sum(cfa.out$coeff[names.loadings]) ^ 2 + sum(cfa.out$coeff[names.errors]))
+ round(c("Congeneric" = r.congeneric, "Alpha" = alpha(cfa.out$S)$total$raw_alpha), 2)
+ }
>
> ############################################
> #Congeneric model; tau-equivalent items
> model.graham <- specifyModel()
1: T -> x1, loading1
2: T -> x2, loading2
3: T -> x3, loading3
4: T -> x4, loading4
5: T -> x5, loading5
6: x1 <-> x1, error1
7: x2 <-> x2, error2
8: x3 <-> x3, error3
9: x4 <-> x4, error4
10: x5 <-> x5, error5
11: T <-> T, NA, 1
12:
Read 11 records
> cfa.out <- sem(model = model.graham, S = S.graham, N = 60)
> summary(cfa.out)
Model Chisquare = 0.11781 Df = 5 Pr(>Chisq) = 0.99976
Chisquare (null model) = 232.13 Df = 10
Goodness-of-fit index = 0.9992
Adjusted goodness-of-fit index = 0.99761
RMSEA index = 0 90% CI: (NA, NA)
Bentler-Bonnett NFI = 0.99949
Tucker-Lewis NNFI = 1.044
Bentler CFI = 1
SRMR = 0.0049092
AIC = 20.118
AICc = 4.6076
BIC = 41.061
CAIC = -25.354
Normalized Residuals
Min. 1st Qu. Median Mean 3rd Qu. Max.
-0.038700 -0.008860 -0.000002 0.004700 0.002450 0.119000
R-square for Endogenous Variables
x1 x2 x3 x4 x5
0.9286 0.8175 0.6790 0.4962 0.5968
Parameter Estimates
Estimate Std Error z value Pr(>|z|)
loading1 2.15045 0.21595 9.9580 2.3263e-23 x1 <--- T
loading2 2.13776 0.24000 8.9073 5.2277e-19 x2 <--- T
loading3 2.06828 0.26941 7.6770 1.6281e-14 x3 <--- T
loading4 1.78754 0.29136 6.1352 8.5040e-10 x4 <--- T
loading5 2.66040 0.38141 6.9752 3.0535e-12 x5 <--- T
error1 0.35559 0.17469 2.0356 4.1793e-02 x1 <--> x1
error2 1.02000 0.25339 4.0255 5.6861e-05 x2 <--> x2
error3 2.02222 0.41688 4.8509 1.2293e-06 x3 <--> x3
error4 3.24471 0.62679 5.1767 2.2583e-07 x4 <--> x4
error5 4.78227 0.94911 5.0387 4.6871e-07 x5 <--> x5
Iterations = 21
> pathDiagram(cfa.out, edge.labels = "values", ignore.double = F, rank.direction = "TB")
> funk.congeneric(cfa.out) Congeneric Alpha 0.91 0.91 > > ############################################ > #Congeneric model; tau-inequivalent items > model.graham <- specifyModel() 1: T -> x1, loading1 2: T -> x2, loading2 3: T -> x3, loading3 4: T -> x4, loading4 5: T -> x7, loading7 6: x1 <-> x1, error1 7: x2 <-> x2, error2 8: x3 <-> x3, error3 9: x4 <-> x4, error4 10: x7 <-> x7, error7 11: T <-> T, NA, 1 12: Read 11 records > cfa.out <- sem(model = model.graham, S = S.graham, N = 60) > summary(cfa.out) Model Chisquare = 0.0072298 Df = 5 Pr(>Chisq) = 1 Chisquare (null model) = 353.42 Df = 10 Goodness-of-fit index = 0.99995 Adjusted goodness-of-fit index = 0.99985 RMSEA index = 0 90% CI: (NA, NA) Bentler-Bonnett NFI = 0.99998 Tucker-Lewis NNFI = 1.0291 Bentler CFI = 1 SRMR = 0.0010915 AIC = 20.007 AICc = 4.497 BIC = 40.951 CAIC = -25.464 Normalized Residuals Min. 1st Qu. Median Mean 3rd Qu. Max. -2.76e-02 -1.27e-04 -1.00e-07 -1.92e-03 1.96e-04 3.70e-03 R-square for Endogenous Variables x1 x2 x3 x4 x7 0.9303 0.8171 0.6778 0.4942 0.9902 Parameter Estimates Estimate Std Error z value Pr(>|z|) loading1 2.15247 0.212984 10.10624 5.1835e-24 x1 <--- T loading2 2.13719 0.237279 9.00711 2.1156e-19 x2 <--- T loading3 2.06646 0.266419 7.75646 8.7336e-15 x3 <--- T loading4 1.78392 0.287599 6.20279 5.5470e-10 x4 <--- T loading7 21.61720 2.015041 10.72792 7.5272e-27 x7 <--- T error1 0.34688 0.090227 3.84451 1.2080e-04 x1 <--> x1 error2 1.02240 0.200635 5.09584 3.4720e-07 x2 <--> x2 error3 2.02973 0.382466 5.30694 1.1148e-07 x3 <--> x3 error4 3.25764 0.605376 5.38118 7.3998e-08 x4 <--> x4 error7 4.64661 6.387765 0.72742 4.6697e-01 x7 <--> x7 Iterations = 38 > pathDiagram(cfa.out, edge.labels = "values", ignore.double = F, rank.direction = "TB")
> funk.congeneric(cfa.out) Congeneric Alpha 0.99 0.56