Accreditation – the word alone strikes either hope or panic into the hearts and minds of many evaluators. As a result, it is much talked about, but rarely acted upon. Let me make the case why I believe it’s time for accreditation in Development Evaluation. I have deliberately left out other areas of evaluation for reasons that will hopefully become evident below (apart from the fact that I know much less about the other areas).
Let me start with a true event that some of you may remember from the 2002 Winter Olympics. During the pairs figure skating event, Russian and France were found to have conspired leaving the Canadian pair in second place. This was a great surprise to experienced skaters (the practitioners) and coaches (the managers) and former judges (the evaluators) who had immediately upon completion of the Canadians’ performance agreed that they clearly had won gold. This scandal triggered a complete revamping of the judging system (or evaluation framework).
In Development evaluation, there is a very large body of work from international conferences and UN Resolution that provides the basic framework of what any development program or project ought to contribute to – poverty reduction, environmental sustainability, gender equality, etc. There are also a vast variety or projects and programs, ranging from the relatively simple to the nearly impossibly complex. The question is: would anyone find it acceptable for highly complex development projects involving perhaps the livelihoods of hundreds of thousands of people to be supervised and implemented by poorly skilled administrators who show little appreciation for the high stakes involved? Similarly, would anyone find it acceptable for the same project to have been supervised and implemented by highly proficient and experienced individuals and institutions, yet evaluated by someone (or a team led by someone) with little experience in the areas that are pertinent to the success of the project? Sadly, the answer is all too often ‘yes’ and while the former may lead to the symbolic dismissal of high-level officials, the latter rarely leads to significant consequences, other than perhaps a lost opportunity to excerpt influence on the future design, or the gradual erosion of the reputation of the commissioners of the evaluation (independent or not).
Back to skating for a moment. Based on years of experience and the significant risk of the International Skating Union (the commissioners of evaluations) losing its credibility, Figure skating’s international body developed very precise evaluation criteria, using quantitative and qualitative methods (yes, they have specialists who review the quantitative aspects if necessary in super slow motion to make sure they get it right). They also built in a principle that, on average, the ‘how’ (is he or she displaying good skating skills?) and the ‘what’ (was it a difficult jump or spin?), get valued equally.
To rise up through the ranks, both as a skater and as a judge, both have to pass ‘tests’ and achieve sufficient scores before they are allowed to compete or judge at the next higher level. For the judges, maintaining proficiency at all levels is a requirement, and statistics are being kept on how one judge's scores compare with those of the other judges. Most importantly, how precisely each element is to be performed and judged is determined in a collaborative fashion by a committee including representatives of the skaters and coaches, as well as the judges. Technical changes to the detailed ‘evaluation handbook’ are communicated on a weekly basis and are effective as early as the following Monday – ‘I didn’t check the website’ is no excuse.
What would this look like for development evaluation? The starting point would have to be already existing internationally agreed goals and purposes of development programs, or their national equivalents (depending who the funder is). Accreditation has to be a three-way collaboration. The staff carrying out or supervising development projects need to be accredited to understand the linkage between their work and the higher-level objectives in development. Higher levels of accreditation are required to manage more complex projects. The commissioners of evaluation (e.g. mid-level or senior officials) need to be accredited in understanding the process of commissioning evaluations and understanding their findings. Higher-level accreditation is needed to commission evaluations for complex programs and projects. Finally, who can evaluate a given program or project depends on the level of complexity and the corresponding requirements on the skills of the evaluator.
It took figure skating 10 years of experience with the new system, and still changes occur on a weekly basis. And it certainly doesn’t ensure that everyone agrees with the final ‘evaluation’. But it is a giant step forward, though it took a major scandal to get out of the comfortable system that had been working ‘just fine’ since the early years of the 20th century.
Considering the stakes involved in international development, it is surprising that accreditation is even an issue. There is no lack of international cooperation, competencies, methods, and national and international evaluation associations. Let’s hope that it doesn’t take a scandal to get out of the comfort zone as many previous enormous development policy failures – which are often failures to learn from previous evaluations – have changed little. The hypothetical scandal necessary to force a change in the evaluation system would have to be truly devastating.
Hans-Martin Boehmer, former Senior Manager in the World Bank Group's Independent Evaluation Group and US Figure Skating accredited Test Judge for Singles and Pairs.