In early 2015 MDF Asia launched the Impact Evaluation course to the public for the first time. The course started with 7 participants in Hanoi, Vietnam who were very satisfied after the training and spread the word. Later courses in Cambodia and Vietnam were filled with participants who came from various international organisations (eg. GIZ, UNDP, Habitat, Lux Development, LIFT, ILO, Plan etc.) in Laos, Cambodia, Germany, Vietnam, Myanmar, Indonesia, Uganda etc. “It’s an excellent training in terms of both organisation and facilitation” – one of the participants commented. Another said: ”You should be proud of this training!”
How was the training designed? This is an applied course about Impact Evaluation. This means it is relevant for those “in the field” – project and M&E officers working, for example, in NGOs or bilateral and multilateral donor agencies, rather than for university degree students or academics.
The training is designed such that attention is paid to both the quantitative and qualitative details of impact evaluation designs, as well as planning and management issues. This includes:
- developing an overall mixed-methods impact evaluation design
- the process to choose appropriate evaluation tools
- planning and specifying tasks
- working within a given budget; and
- managing consultants.
Along the way, participants are introduced to the key concepts behind quality impact evaluations, notably the counterfactual question, and control groups.
Our course follows three basic steps:
- Getting the project design right;
- Specifying a value-for-money impact evaluation design; and
- Managing the evaluation well.
We argue that a clear Theory of Change (ToC) is the core of a good project design. The ToC can be represented as a simple flow diagram, but that needs interrogation to draw out all causal assumptions and alternative hypotheses.
The ToC will lead to specifying the questions that need answering in the impact evaluation design (which can change if outcome mapping is used). That design will be influenced by many considerations, including available budgets, existing secondary data, similar donor projects, the complexity of the intervention, the quantifiability of intervention outcomes, the ability to identify meaningful control groups, and the importance of doing the impact evaluation. Against these many considerations we consider an even longer list of evaluation method options (“tools”) to determine what is the most effective mix. This is more art than science, and is best learnt by working through examples.
When understanding theory and practice as described above, the development practitioner is ready to specify the details of tasks to be done, who will do them, by when, and what consulting inputs are needed.
The management of impact evaluations is an important skill, yet most project (or M&E) officers do not engage in formal training to understand the issues involved. We rectify that with particular attention to drafting terms of reference, managing procurement and contracting, and then managing consultants undertaking the tasks. For staff working in developing countries this is a course not to be missed.
“After decades of seeing so many poorly designed and poorly managed impact evaluations, I am really excited to lead the design and teaching of this course. Academics talk about evaluation “gold standards”, but in the field we are closer to the stone age – this course will raise the quality of the “typical” budget-constrained evaluation design and hence outcomes.” – Adam McCarty, the MDF course design team leader/senior economist said.