One of the big challenges in evaluating development projects that involve social change is the question of proving that what you did led to specific results. There is a fascinating discussion about this question on Alex Jacob’s NGO Performance blog.

Coincidentally, this morning I also read about a new study that compared the costs of providing HIV/AIDS drugs to 3.5 million people vs. the benefits in terms of increased productivity, fewer orphaned children, and savings on care otherwise needed for TB and other opportunistic infections. Spending $14.2 billion between 2011 and 2020 would save 18.5 million life-years and $12 to $34 billion on those costs, the study found.

This reminded me of something I learned a number of years ago when I was the executive director of a quasi-governmental women’s organization in the Northwest Territories. The Council, which directed our work, had asked me to find ways to make progress on making family violence a higher priority for government.

Up to then, the usual approach had been to talk to government about the needs of women who were experiencing violence. Government usually saw dollar signs when the Council or other groups lobbied them – this was going to cost them money. And when it came to spending money, there were always a lot of competing priorities.

Investment vs costs
So I went to the internet and spent some time doing research. That was how I discovered some fascinating work being done by a women’s group in British Columbia. Instead of focusing on how much government needed to spend on shelters and support programs for family violence, they had decided to look at how much family violence cost government. And they cast a broad net in doing so.

They realized that many agencies and departments were dealing with the consequences of family violence. The police; the courts; the prisons; the social welfare agency; the hospitals; the probation service; the shelters; women’s advocates; and so on. That added up to a great deal of spending on the results of a behavior, with much less being spent on addressing the causes. And it didn’t include the costs experienced by employers when women had to take time off work as a result of family violence.

They set out to see if they could estimate how much all of that was costing, and the figures were completely astounding. They had to do a lot of estimating, of course, because (at that time, at any rate) no one was collecting those kinds of statistics in a related way. When the costs of running a shelter and family violence education programs were compared with those other costs, it represented a miniscule investment. Government was spending millions of dollars on dealing with the results of the behavior, compared with only a few thousand dollars on preventing the behavior. This wasn’t a sensible investment in cost-benefit terms.

A new perspective for government
So I decided to begin explaining spending on family violence as being an investment in reducing substantial costs that appeared elsewhere in the government’s budget. This was an argument that was easily understood by government. Most government officials had never thought of the issue this way. They were used to thinking about categorizing spending by department, rather than by activity or by addressing a multi-faceted problem. Suddenly, they saw the money they were spending on shelters and supporting educational programs about family violence in a completely different way. It represented a way of saving money, not a way of spending money. (And this new view brings a lot more allies into programs to end family violence – agencies and people who otherwise would not be involved with ‘women’s’ issues.)

Many social agencies, and many local peacebuilders, don’t think of this way of approaching the evaluation of their work. It is a sea change in thinking. Suddenly, we are not asking them to think about what their program costs, in isolation. We are asking them to think about what that investment in their program represents in terms of savings.

For example, in one community in South Kordofan that had experienced years of conflict (before the current problems arose), peace meant that last year, the community residents were able to plant their fields again for the first time in many years. When there was peace, they could cultivate without fear and they were able to harvest their crops without having them stolen by armed groups. The result was that the price of the two staple crops, sorghum and maize, began to drop dramatically – and that had a marked impact on the cost of living. Instead of having to buy these staples, people had a stock of them. And even those people who couldn’t cultivate fields benefitted, because it cost them less to buy these staples at the market.

People in the village had begun thinking about how they could now attract regional traders to their local market to buy the surplus crops, as they would get better prices from those traders than if they took the goods to more remote markets. And attracting more traders would benefit the rest of the community, too, as those traders would likely bring other goods with them that people otherwise could not obtain easily.

A cost benefit analysis of peacebuilding
Last year, I began asking local peacebuilders to begin thinking about this kind of cost-benefit analysis of their work. Don’t just look at what your program costs, I urged them. Think about what your program saves. It is not a usual way people think in social development circles and so it takes a bit of stretching of the mind. In peacebuilding terms, it means estimating some costs because data is not easy to obtain.

Think, for example, about running a community-based activity to provide jobs for young men by starting a trucking company (a real-life example). As their business prospers, they start to turn in their guns to the local chief, because they don’t need their guns to make a living. Effectively, you are running a DDR program while also building the local economy, creating jobs, and providing cheaper and better local transportation. How do you measure those benefits, vs. the investment in buying two trucks and providing some basic training in management and book-keeping? One way is to compare it with other local disarmament activities.

One of those, in southern Sudan, had involved having 1,500 soldiers living near a village for six months, before eventually having a battle with the young white army herder/soldiers scattered around the bush. During those six months, the soldiers used local community food to support themselves – the community felt the impact for years afterwards. Many people were killed in the fighting, and the cattle that the young men were caring for scattered all over the region.

The costs were clear to other communities, which used this as a salutary example of what happened if the community could not find a way to disarm these young men peacefully. But no one actually totted them up in columns, and the NGO that inspired the trucking company had never thought of using those costs as an example of what its program had saved.

The cost-benefit analysis of locally-led peacebuilding is just in its infancy. But it has huge possibilities for changing how this work is currently funded and monitored. In doing this, local knowledge and capacity represents a huge plus on the ‘credit’ side of the comparison. For international organizations, developing the capacity that local peacebuilders bring to the program would represent a huge investment of both money and time before they could start. But in evaluating programs, the value of this local knowledge is never computed.

More reading:
Resch S, Korenromp E, Stover J, Blakley M, Krubiner C, et al. (2011) Economic Returns to Investment in AIDS Treatment in Low and Middle Income Countries. PLoS ONE 6(10): e25310. doi:10.1371/journal.pone.0025310

Aids treatment is good value for money, says new study. Sarah Bosely’s Global Health Blog, The Guardian

Throwing off our results chains, by Alex Jacobs. NGO Performance blog. October 5, 2011.