One of the things I like about my current job is getting to apply generalizability theory. I have to confess that I didn’t appreciate G theory at first, but now that I’m analyzing data from performance assessments (i.e., ratings of instructional effectiveness) I think it’s great for a number of reasons. I like the concept of universe scores, treating items as a random facet, estimating multiple sources of error, the ability to focus on different objects of measurement (e.g., ratees and raters), and G theory’s strong emphasis on finding ways to increase generalizability (i.e., decision studies). And as someone who has far more experience with item response theory (IRT) where conditional standard errors of measurement (CSEMs) are the norm, I’m especially intrigued by the prospect of using G theory to estimate CSEMs.
It’s pretty obvious from this blog that I like statistical programming with 
because it helps me learn methods far better than simply reading about them or using a pre-existing software package. In that vein, I decided to write some functions to conduct G and D studies with . I’ve been using synthetic data from Brennan’s book and Shavelson and Webb’s book to ensure that my results match theirs. The nice thing about using for G theory is that you can choose functions for many different packages to fully apply G theory. For example, by using lme4 to estimate variance components, arm to extract standard errors of random intercepts, and ggplot2 to create faceted plots, I am able to create “vertical ruler” plots, which are prevalent among practitioners of many-facets Rasch. Such plots make it easy to identify the (statistically significantly) highest and lowest performers as well as the most lenient and stringent raters.
The plots below illustrate how I’ve been implementing G theory with . The plots were produced from Brennan’s synthetic data set number 4, and the dependability coefficients match those from GENOVA. I’ve never developed an package, but given that no one else has so far, I’d be interested in doing so with the code I’m developing.
