Example of a statement of long-term impact of a program: “Most notable is the increase in the enrollment rates of girls in school in areas where Org X was focused on girls education. In the District of X, enrollment of girls went from 43% to 46% of total enrollment from 2002-3 to 2008-9. The overall graduation rate from primary schools in District X grew from 45 percent in 2002-3 to 79 percent in 2008-9.
Anyone reading this can ask the obvious questions related to the comparison and attribution. Of course there may be many factors in these increases, but this report was on a grant that was less than US$20,000, a relatively small amount when you consider the scale of most development projects. Let’s always consider what is the appropriate cost and complexity needed for measurement, especially given the size and scope of the program.
Proportional expectations for the applicability of RCTs, as well as the potential consequences of poorly-done RCTs for those who are being studied are also important, especially when people are in the process of organizing at the local level. Rather than an afterthought, let’s talk simultaneously about how local partner organizations become drivers of the use of RCTs, rather than just being consulted or included in them. As a commenter on Owen Barder’s blog shared, “Great tools, we economists undoubtedly do have. In studying development issues, they are often used unhelpfully due to hubris and a shocking level of comfort with ignorance about the phenomenon being studied.”
Despite behavioral economists’ so-called acceptance of the rationality of the poor’s decision-making, I find phrases like “the bizarre thing was that Oti didn’t seem to mind wasting his own time,” “he might have spent his last twenty rupees on [flower garlands],” “people showed they had both the will and desire to save,” “people were learning” (as if these were a surprise) and the most striking, “were [the research subjects] just thickheaded?” contained in the book’s anecdotes to reveal subtle, underlying, and perhaps unexamined judgment, if not contempt, to which I am admittedly very sensitive.
If an assumption is operating that poor people don’t know what’s good for them, then the flip side of this assumption is that someone else must. As Tom Murphy comments on Bottom Up Thinking, “In development, can’t we say that the batch of behavioral economists are exercising some amount of paternalism? They are using ‘nudges’ to encourage behaviors but that inherently comes from a place of knowing.” In my career in the aid sector, I’ve learned the hard way to become comfortable with stepping away from the role of “expert.” In fact, much of my work is focused on encouraging aid workers, donors, and international do-gooders to do the same.
Sixty years of development aid hasn’t reduced poverty using existing methods, yes. But sixty years of development aid that has squashed local initiatives by not giving the due attention to how that aid (and the accompanying monitoring, surveys, etc.) makes people feel, is, I believe, perhaps one of our biggest challenges in making aid more effective. The prevalent, yet not often exposed negative attitudes, behaviors and perceptions towards local people and organizations in the aid world is something that has been under-reported, insufficiently documented, and poorly-studied. Consider—How would programs change if an equal amount of curiosity and energy that we spend in conducting RCTs were invested in well-facilitated listening exercises in which we had to learn about people’s experiences of being on the receiving side of aid?
Let’s not forget our Sen. Freedom, power and poverty are inexorably intertwined and the mechanics of economic transformation just a part of development. In all of the discussions of RCTs and their usefulness, the reality of power relationships within the aid system and the lack of humility that continues to plague us is not something we can escape.
Undoubtedly, soundly-interpreted data provides important new perspectives for us all. There remains, however, quite a lot we cannot know.
And I, for one, am okay with that.
This post originally appeared at: http://www.how-matters.org/2011/05/24/rcts-band-aid-on-deeper-issue/
See also related how-matters.org posts, RCTs: Much to be said and RCTs: "how matters" advice for donors.
Got ‘Em: An Evaluation Story
Trying to Quit
Did I fund Organization X?
More on Why ‘How Matters’
How to build strong relationships with grassroots organizations (Pa...