Each time I log into Facebook it asks me “what’s on your mind?”. My mind can be a muddled place where there are rather too many musings about the meaning of Leonard Cohen lyrics (is there really a crack in everything? What about time? In Crazy to Love You does he make reference to the Wire?) and attempts to make myself giggle by replaying Phil Dunphy’s most brilliant lines (“believe me, you don’t want to make that call to a bunch of former male cheerleaders. They will mock you with a hurtful rhythmic chant”). But this week Leonard and Phil have been pushed aside and my mind has been full of ideas about impact. And how we can get better at evidencing it.
A company would not take a new product to market without evidence to confirm its consumer desirability and viability. Likewise, a company would not roll out or “scale up” a new product without first generating evidence of impact. In that case “impact” is profit. In our case “impact” is social. Yet there is little consensus about what we mean by social impact. At our most timid we claim that nearly any philanthropic investment in the developing world will create impact. At the other end of the spectrum we may pay great attention to impact at design stage, but then have limited capacity post-investment to really understand the drivers of that impact. This is a serious situation to be in. It means that knowledge about what is actually working is limited. And therefore how do we make evidence-based decisions to scale?
This blog, and the next one, mulls over two impact dilemmas that are top of this developmentista’s mind:
- Within a proven model, how do we know what are the actual drivers of impact?
- How can we compare impact across our investment portfolio? Do we need to?
The drivers of impact
I spent a week in India last month strategizing with colleagues about the future of an education investment. The “product” is a phonics-based English language package for primary school children that has achieved great results. In two years children receiving the intervention have progressed at a rate up to five times faster than their peers in the control group. But there is a big caveat: the business model is high-touch, being reliant on a significant degree of face-to-face interaction, and is therefore expensive. Perhaps prohibitively expensive if the ultimate aspiration is to achieve real scale. So the purpose of the strategy session was to analyze whether the cost can be compressed without compromising on quality.
The product has three components: teacher training, a phonics curriculum and ongoing mentoring to teachers delivering the lessons. The third component – mentoring – requires intensive human interaction and therefore constitutes over fifty percent of the cost. Instinctively my colleagues believe that this component is the weightiest impact driver and have some great ideas around replacing (expensive) human interaction with (cheaper) innovative technology (tech-utopian alert*). Instinct is important and certainly provides useful consumer insight, but needs to be examined with a skeptical eye and assumptions validated (as far as possible) by theory and evidence from elsewhere.
I loved A-Level chemistry because I love experiments. And, as I have blogged before, I firmly believe that the development sector must get better at experimentation: quick and cheap experiments with preparedness to fail, to iterate and then (hopefully) succeed; prototyping with rapid market-testing and consumer feedback. When applied to our phonics product, this kind of tinkering will allow rapid testing of innovative technologies to compress cost (and may also reveal new opportunities to increase its potential in ways that we have not yet even conceptualized). Controlling variables and reducing, adapting and compressing each component in turn will generate insight about the actual role and contribution to impact of the three components. And will allow us to better understand where the sweet spot is between cost and quality.
But are experiments enough? Can funders and governments justify scaling a new product without more robust evidence behind it?
A randomized control trial would enable us to scientifically measure the impact of controlling each variable in turn. This rigorous methodology allows us to move from a broad conclusion of correlation (student achievement seems to increase when teachers receive more mentoring) to a definitive statement (more teacher mentoring increases educational achievement).
Randomized control trials are the gold standard. The best way to deliver the evidence that policy-makers and investors are hungry for. I spoke with J-PAL at length this week about the optimal way to generate the right kind of evidence in a context like this one. J-PAL is the awesome group of randomistas based at MIT who use randomised evaluations to answer questions critical to poverty alleviation. If anyone knows about evidence in this business, they do. And they gave me great advice. Multiple variables within an RCT, with incremental differences between versions, are expensive and ultimately not viable or necessary. Probably not the right thing for us to do immediately. J-PAL agreed that there is a strong case for more “quick and dirty” interventions: rapid testing and experimenting to generate insight, iron out “process” kinks and back up those gut instincts. With this experimental approach we will generate better evidence of the real drivers of impact. Then a subsequent RCT can rigorously scrutinize that evidence when it is ready to scale up.
An experimental approach will provide an ongoing stream of insight and good-quality data. It does not need to cost a fortune but it will require a curious and a skeptical mind.
Let’s get experimenting.
* Note: this is an area where I would self-identify as a tech-skeptic. Teachers possess a lot of unleveraged knowledge. Is giving them access to more knowledge really the answer?