1. Recognise the early signs of success
A director of an NGO in the Balkans sighed; after many years of advocating participatory democracy, still government officials only try to convince me why they cannot do it. What do we need to do to get them to move? Interestingly, her colleague then mentioned ‘Well, at least they now engage in a serious discussion’ indicating that until recently the concept of ‘participatory democracy’ had been unknown and not worthy of discussion to them.
Lobby & Advocacy is pushing people to change, and (almost) every change meets misunderstanding, suspicion and denial first. This is then followed by resistance, before understanding, adaptation and actual change can be expected.
Most Theories of Change and results frameworks do not reflect this common knowledge about change but reflect outcomes that may take years to be accomplished. These outcomes reflect the end-result of a longer term and gradual changes process consisting of many small yet meaningful steps. Not recognising and reflecting these initial steps in result frameworks often means that they are not captured in regular monitoring. Making sure that your evaluation is alert and able to capture these early and less predictable signs of progress, helps in reassuring what courses of action to be proud of and which ones need adjustment.
2. Conscientiously consider contribution
The core questions of most L&A evaluations revolve around measuring meaningful effects on targeted audiences and understanding the significance of the programme’s contribution to those effects. There is now an increasing consensus that we need to look for ‘contribution’ rather than ‘attribution'. Putting a successful contribution analysis in practice is not easy. It requires a careful selection of cases, access to reliable sources outside the programme that can inform you about the (significance of) alternative factors, and investing additional time for data collection and (joint) analysis.
In cases, where these conditions were in place, systematic contribution analysis has the potential of providing remarkable insights. It can illustrate the significance of an intervention vis-à-vis external factors, but also which strategies within a particular intervention were the most effective. For instance, contribution analysis in a programme to combat human trafficking in Ethiopia showed that creating space for families affected by trafficking to share their stories was much more impactful - and efficient - than improving facilities and training programmes. A well-planned contribution analysis can truly increase the benefits of an evaluation, provided that it is well understood, prepared and resourced.
3. Deep-dive into dynamics
Partnerships and alliances are not organisations. Management control and reporting systems that would be normal in an organisation, do not work in partnerships that lack conventional hierarchy. Instead, partnerships are driven by the commitment and energy of their members. This notion is widely accepted, but scarcely used in partnership evaluations when researching the ‘internal’ factors that explain partnership performance.
Good models and tools for network and partnership analysis are available, ranging from GiZ’s Capacity Works model, to the Alliance Thermometer, and the Free Network Approach. Unfortunately, these are scarcely or superficially used, leading to recommendations about systems and structure, while the crux lies in getting a grip on the inter-organisational dynamics, coloured by individual behaviour. Investigating how interactions actually take place and affect the motivation of members to become and stay active in the partnership is a key element in every ‘virtuous evaluation’. Such an exercise can create inspiring new ideas about how to work differently together. Making partnerships more successful but also more fun to take part in, are the key benefits of a truly virtuous evaluation.