I finished the book weeks ago. When I closed Dean Karlan and Jacob Appel’s “More Than Good Intentions: How a New Economics Is Helping to Solve ...,” I had many notes and reactions jotted down but they hadn’t yet gelled. Something just didn’t sit quite right with me and I was having trouble articulating it.

Of course that’s related to intuition and inspiration, which can’t be measured…but I digress.

After my Friday “tweet debate” with @poverty_action and after reading many other bloggers’ reactions to the book (see related post), I realized that Karlan and Appel have heard and refuted it all by now. As seasoned and skilled randomistas, they have ready responses in support of incorporating randomized control trials (RCTs) into more aid projects.

And this is not something I’m necessarily against. In a development discourse that is still ruled by economic academics, I respect Innovations for Poverty Action’s work, which attempts to bridge this knowledge base with aid practitioners’ experience. In fact, Karlan, Appel and I agree on many aspects of their main arguments. We all agree that aid can be more effective and that well-formed questions and well-executed, applied research can offer many relevant clues about this. We all want to see deeper thinking behind the doing.

Where I think we differ is on some fundamental beliefs about what prevents this and what ails the aid industry overall. Is it a lack of information about “what works”? Or is it a lack of respect for local initiatives and understanding about complex power dynamics that impede authentic relationships among development partners? And if it’s the latter, are RCTs just a band-aid on a deeper issue?

I am, like many others, worried about the implications of donors moving towards making RCTs yet another conditionality of aid, more food to satisfy their seemingly insatiable appetite for evidence. (See my related post providing "how matters" advice for donors on RCTs.) As someone working within the extensive web of local organizations and grassroots movements in the developing world, this is especially troubling for nascent, non-“formal” and under-resourced organizations that are already marginalized from the aid system.

Whether RCTs gain momentum as just the latest fad in aid, or whether they become a part of accepted practice, I am also afraid that now “unsubstantiated claims” such as the example below, which I recently read in a report, will now be considered invalid (and un-fundable), rather than be probed for clearer understanding and viewed as a opportunity for learning.

Example of a statement of long-term impact of a program: “Most notable is the increase in the enrollment rates of girls in school in areas where Org X was focused on girls education. In the District of X, enrollment of girls went from 43% to 46% of total enrollment from 2002-3 to 2008-9. The overall graduation rate from primary schools in District X grew from 45 percent in 2002-3 to 79 percent in 2008-9.

Anyone reading this can ask the obvious questions related to the comparison and attribution. Of course there may be many factors in these increases, but this report was on a grant that was less than US$20,000, a relatively small amount when you consider the scale of most development projects. Let’s always consider what is the appropriate cost and complexity needed for measurement, especially given the size and scope of the program.

Proportional expectations for the applicability of RCTs, as well as the potential consequences of poorly-done RCTs for those who are being studied are also important, especially when people are in the process of organizing at the local level. Rather than an afterthought, let’s talk simultaneously about how local partner organizations become drivers of the use of RCTs, rather than just being consulted or included in them. As a commenter on Owen Barder’s blog shared, “Great tools, we economists undoubtedly do have. In studying development issues, they are often used unhelpfully due to hubris and a shocking level of comfort with ignorance about the phenomenon being studied.”

Despite behavioral economists’ so-called acceptance of the rationality of the poor’s decision-making, I find phrases like “the bizarre thing was that Oti didn’t seem to mind wasting his own time,” “he might have spent his last twenty rupees on [flower garlands],” “people showed they had both the will and desire to save,” “people were learning” (as if these were a surprise) and the most striking, “were [the research subjects] just thickheaded?” contained in the book’s anecdotes to reveal subtle, underlying, and perhaps unexamined judgment, if not contempt, to which I am admittedly very sensitive.

If an assumption is operating that poor people don’t know what’s good for them, then the flip side of this assumption is that someone else must. As Tom Murphy comments on Bottom Up Thinking, “In development, can’t we say that the batch of behavioral economists are exercising some amount of paternalism? They are using ‘nudges’ to encourage behaviors but that inherently comes from a place of knowing.” In my career in the aid sector, I’ve learned the hard way to become comfortable with stepping away from the role of “expert.” In fact, much of my work is focused on encouraging aid workers, donors, and international do-gooders to do the same.

Sixty years of development aid hasn’t reduced poverty using existing methods, yes. But sixty years of development aid that has squashed local initiatives by not giving the due attention to how that aid (and the accompanying monitoring, surveys, etc.) makes people feel, is, I believe, perhaps one of our biggest challenges in making aid more effective. The prevalent, yet not often exposed negative attitudes, behaviors and perceptions towards local people and organizations in the aid world is something that has been under-reported, insufficiently documented, and poorly-studied. Consider—How would programs change if an equal amount of curiosity and energy that we spend in conducting RCTs were invested in well-facilitated listening exercises in which we had to learn about people’s experiences of being on the receiving side of aid?

Let’s not forget our Sen. Freedom, power and poverty are inexorably intertwined and the mechanics of economic transformation just a part of development. In all of the discussions of RCTs and their usefulness, the reality of power relationships within the aid system and the lack of humility that continues to plague us is not something we can escape.

Undoubtedly, soundly-interpreted data provides important new perspectives for us all. There remains, however, quite a lot we cannot know.

And I, for one, am okay with that.

***

This post originally appeared at: http://www.how-matters.org/2011/05/24/rcts-band-aid-on-deeper-issue/

See also related how-matters.org posts, RCTs: Much to be said and RCTs: "how matters" advice for donors.

***

Related Posts

Got ‘Em: An Evaluation Story

Trying to Quit

Did I fund Organization X?

More on Why ‘How Matters’

How to build strong relationships with grassroots organizations (Pa...