Quasi-experiments at Google: Evaluation insights

Google’s statisticians routinely use randomized experiments to improve their products (and profit), but did you know they also conduct quasi-experiments when random assignment is not feasible? I receive the American Statistical Association’s (ASA) membership magazine called Amstat News. Daryl Pregibon, a Google statistician (or “engineer” as they are called internally), was invited to write about the company’s statistical practices in the May issue. He writes that Google users can be randomly assigned to treatment conditions, but

“it is usually not possible to randomly assign advertisers to treatment groups due to contractual obligations and/or their willingness to be ‘experimental units’ for a service for which they are paying. In such cases, we … use statistical methods that try to tease out causal inferences. Propensity score matching, inverse propensity weighting, and double robust estimates are some of the methods established in social and biological sciences currently in use at Google when randomization is not possible.”

That approach mirrors best practices in quantitative evaluation. Randomized field trials are considered the gold standard for judging the degree to which a program or its components cause a desired outcome; when random assignment is not feasible, quasi-experiments provide a valuable alternative. Evaluation researchers rarely have as much control over conditions as Google’s “engineers.” Consequently, evaluators must rely more on quasi-experiments to “tease out causal inferences.” Another key difference is that no matter how enormous a program data set may seem and no matter how many parameters a client might want an evaluator to estimate, those amounts will never reach the terabytes of data or the millions of parameter estimates that Pregibon describes as commonplace in life of a Google statistician.

By the way, my master’s paper involved applying inverse propensity weighting to account for self selection into a local public school district. Does that mean a career as a Google statistician is in my future?

This entry was posted in Praxes. Bookmark the permalink.