One of the challenges of ‘evaluation’ is that it requires judgment of someone else’s work. Especially if the evaluation takes place at the end of a project, the evaluator must assess work done by others over a long period of time. As someone whose work has been evaluated, as well as someone who evaluates others’ work, I find this need to make a ‘judgment’ one of the most challenging aspects of my work.

I am always aware that no matter how many questions I ask and how many reports I read, I will never understand all that has been accomplished in the same way as those who have worked on a project day in and day out for one or two years. I remember what it felt like to have two years of intensive work ‘evaluated’ by two people who came with extensive CVs, assumptions picked up in head office, and a distinct reluctance to work as hard as we did. Instead of giving us a helpful and longer-term perspective on our work, they cut down the terms of reference in the field, asked few questions and made few notes, and produced a report that worsened the existing politics within the project.

So I read as much as I can. I ask questions – many questions – of everyone I meet. I do my best to test my conclusions while in the field, to see if I have understood well or not. I work as hard as I can, for as many hours a day as I can, while in the field. I do my best to let people know that I am there to tap into their expertise and knowledge.

A learner with power

I know that a few weeks in the field allows one to see only a small part of an NGO’s work. I know that an NGO may not know how to package and present their work to an evaluator. Even if they do, it is entirely likely that both NGO and evaluator may have made assumptions about the other that are never surfaced – and thus never confirmed or dismissed. In short, I know that I am in essence evaluating the map rather than the territory.

Yet while I try to be as much as possible a learner, I also know that I am a learner with power. I am acutely aware that in many ways, my evaluation will affect their activities, their view of their achievements, and possibly even the future of their organization. I also know that I have a responsibility to the donor, to produce a report that is as clear and coherent as possible and that gives the donor a good idea of how their money has been spent. Did the funding have the results they expected? Was the funding effective? Were the project goals achieved? If not, why not?

This is where the judgment comes in. And in some projects, like a project to test whether a medicine is effective against a particular disease or not, it is relatively easy. One can see from the numbers whether the medicine is effective or not, although the data may not necessarily show you the behavior changes that may have contributed to those results.

But when the results are focused on peacebuilding or community development, in a situation where there has been conflict or civil war, finding numbers becomes considerably more problematic. The challenge, for the evaluator, is much more akin to how Meas Nee describes the process of recovering from conflict in Cambodia. The village, he says, is like a basket that has been torn apart.  The people in the village still remember what the basket looked like, but they are living in its shreds, physically and metaphorically. So what does one report – the destroyed basket, the strands that may have been rewoven, or the progress towards the rebuilding of the basket? And in doing so, whose perspective or perspectives takes precedence?

Rebuilding seems overwhelming

The idea of rebuilding after conflict is at first overwhelming.  Think about the challenge of rebuilding from devastating floods, even in a city with a modern infrastructure and services, like Calgary, Alberta. Within that infrastructure, individuals who have lost their homes and possessions may be in a state of shock, unable to figure out even where to begin. For the city, the infrastructure rebuilding needs may seem enormous. Planning can be extremely challenging, but still, there is a reasonable certainty that with time, things can be back to normal.

Then visualize a situation where a village may have been burned to the ground.  Where any infrastructure (stores, offices, government services) that existed has been destroyed. Where many people have been killed, wounded, mutilated, or raped. Where there are no lights, no one to protect you, and the possibility that militias may come at night in the dark. Where there is no form of public transportation. Where there may be no clean water anywhere, and little if any wood to light a fire to boil water. Where there is no food, because gardens have been destroyed or looted, and everyone is afraid to go into the forest to find wild foods. And where, because security is seen as precarious, there is little or no help on the ground from the outside world – or even from your own government, which is located far away.

Even when it may look to an outsider as if things are returning to ‘normal’, you can still see the terrible impacts of the conflict. You can see the young people who have been orphaned and have no one to care for them. You can see the people who have no way to feed their family because there is no work and their fields have been devastated.  You might see hundreds or thousands of people who have been displaced and have no homes to return to. You might see how relatives are becoming overburdened by the challenge of caring for those who are living with them, and how that is causing conflicts among people.

And yet, in this setting, you can still see the basket that once was, and you have a vision – to rebuild that basket. But you know you can’t do it alone. You cannot do it for people; you must do it with them. And you know that this is a long term proposition. And so you begin by finding the leaders, the people who are known in their communities as the ones who solve problems and resolve disputes – because you know that people will listen to them. You invite those people to come together and talk about what you can do together. And even as you do that, you work with people, one on one, to help them begin to recover their capacity to work together. You weep with people, and you laugh with people.  You listen to them, and you encourage them when they have ideas for taking some action for themselves. You do this even as you struggle to feed and clothe your own family.

In every situation where there has been conflict, there are some people with this vision of the village as it used to be. And as they gradually find the people who are able to work with them to rebuild the strands, and then to weave them together, the village gradually becomes to come back into itself. At first it doesn’t look much like the village that used to be, says Meas Nee, but in time, it will be once again.

The outside perspective

Often, it is at this point of recovery that outside organizations and donors arrive. And they are focused on what seems most urgent, and visible, to them – the needs. And being good hearted, they want to do their best to meet as many needs as they can. What is much harder for them to see is the capacity that lies within the village and its people. Without fully appreciating that capacity, and without intending to do so, they may well disrupt the process of rebuilding peoples’ spirits and lives that is already underway and that will lead to – over time – rebuilding the physical infrastructure.

As outsiders, they need to be wise; to sit with people, to listen to them, to let them recover their spirits and their capacity to begin working together to rebuild; to identify the support they need, and to provide that support at the time and in the way it is needed. Such patience, however, is hard to fit into a business plan or a project schedule.

For example, outsiders might see that the roads are rutted and badly damaged, and they might think that in order to best help, they should bring in a contractor to repair and pave the roads using heavy equipment.  That is very efficient, and it would result in a good road that would make it easier to repair peoples’ houses, get food to outlying areas, and get economic development underway.  They might even get some local support for this idea because, even if it is not really what is needed, people think it might bring in more resources into their community, and some might see a personal opportunity in it.

What outsiders probably wouldn’t see is that in such a case, efficiency would mean that the opportunity for many men to earn some money repairing the road with hand tools would be lost. That the opportunity for former combatants to show the villagers that they are human, by working together to repair the roads by hand, would be lost. That the message they inadvertently sent to the village was that you are not capable of rebuilding by yourself; that maybe you have gotten this far in rebuilding by yourselves, but you can’t do the rest without us.

The evaluation challenge

An evaluator, coming in from outside, might look at this situation and see a newly paved road. They might talk to the mayor, who is hoping for even more outside aid to arrive, and get a very positive response to the donor-funded initiative in hopes that it will bring even more donor aid. They might hear from some newly-arrived international aid agencies that the new road is going to make it much easier to reach the people who need food.  If they are from a country that has good roads, they are less likely to even think about the possibility that rebuilding the road by hand might have created jobs for local people, that in turn might have made it possible for them to feed their families and thus reduce the need for outside food aid.

The evaluator might have only a few days in the field to talk to people. Probably was briefed about the project, with reams of paper and nicely-organized log frames, before their arrival. Knows how much money was spent on the project. Has probably have spoken to the contractor, who may be full of the stories about how difficult it was to get the project completed, or conversely, how efficiently the work was done. Probably a few local people got jobs, and they will say positive things. The evaluator is focused on what is in front of his or her eyes – the paved road – and not what might have been, had things been done differently. And in that situation, of course, it is difficult for him or her to do otherwise.

We have no way (at least none I am aware of) to assess the lost ‘opportunity’ costs of rebuilding in a way that is more labour intensive and less equipment intensive. To assess how this intervention might have affected the progress people had been making for themselves, in rebuilding, before the outside world arrived. And most likely, no way to assess what impact the newly paved road might have on travel by combatants if the conflict resumes because the community peacebuilding process has been interrupted. We are evaluating the visible, and not the less visible.

NEXT – Evaluating the ‘invisible’ parts of the basket