There is limited evidence on the degree to which differences in implementation among institutions matter for program effectiveness. To examine this question, we conducted an experiment in rural China in which public schools were randomly assigned to one of three treatments: a computer-assisted learning program (CAL) implemented by a government agency, the same program implemented by an NGO, and a pure control. Results show that compared to the pure control condition and unlike the NGO program, the government program did not improve student achievement. Analyzing impacts along the causal chain, we find that government officials were more likely to substitute CAL for regular instruction (contrary to protocol) and less likely to directly monitor program progress. Correlational analyses suggest that these differences in program implementation were responsible for the lack of impacts.
Keywords: Computer-assisted learning; education; external validity of program; implementation by government; randomized controlled trial