Designing an effective outcome monitoring framework is tough enough for programmes aiming to deliver concrete changes, such as the expansion of sanitation services and infrastructure. It’s even more difficult for programmes seeking to contribute to changes in policies and institutions. But the demand to measure these types of outcomes is growing fast. Initiatives working to effect change in complex institutional environments face the challenge of rapidly designing and operationalising programme-level frameworks for outcome monitoring that both (a) contribute to organisational development and (b) build evidence to test the validity of the programme strategy.
Through a Collaborative Learning Seed fund from the Open Gov Hub, Akvo had the opportunity to design such a framework with a new partner, Collaborating for Resilience (CoRe). CoRe is an international non-profit change initiative working to foster innovations that sustain critical ecosystem services, build resilient local economies, and increase social and economic wellbeing. Their approach includes working with civil society actors and building multi-stakeholder platforms to promote shifts in the governance of land, water, forests, and fisheries. Contributing to complex change like this requires robust and comprehensive programme design, and this is where Akvo could help.
The goal of the partnership was initially twofold: to develop a framework that CoRe could pilot and to codify learning about considerations needed when designing such a framework. In practice, however, the collaboration took a detour from its original plan, which has led to valuable insights for working on similar partnerships.
Making a ToC operational for monitoring
Programmes based on a Theory of Change (ToC) offer a systems approach to understanding complex environments, and therefore provide a backbone to the design of monitoring and evaluation (M&E) frameworks. Because of this, an early step in the collaboration was to review the planning documents that CoRe had developed for the programme. We soon realised that the Theory of Change, which would be the basis for the monitoring system, required additional outcome specification for a monitoring process to be effectively set up.
CoRe’s prior Theory of Change offered a high-level description of the programme approach, but lacked specificity when it came to articulating how particular types of actors contribute to the changes sought, the sequence of outcomes envisioned, and a more fine-grained picture of the end results. It worked as a summary to quickly communicate the overall approach, but to translate that into a basis for monitoring, we needed to dig deeper.
Above: CoRe’s prior, high-level Theory of Change diagram.
This meant that we needed to take a step back, to restate the Theory of Change so that the expected outcomes reflected a change in behaviours, relationships, actions, activities, policies or practices of stakeholders within the sphere of influence of the programme. Outcomes were refined to respond to the question “who does what differently?”. These outcomes were then connected according to the sequence in which CoRe believes they occur and assumptions were added to explain why that outcome is expected to occur. The form used was as follows: if x then y because of z - where z is the causal assumption between outcomes x and y.
For example, CoRe’s initial Theory of Change framed one of the outcomes as “civil society collective action.” The statement in itself did not provide enough of a description to be measurable. It also didn’t specify which civil society groups would be expected to do things differently because of the intervention. In restating the ToC, that outcome was broken down with more specificity and became a sequence of outcomes about the different ways the programme expected civil society groups to behave as a result of CoRe’s actions.
The image below shows a portion of the revised ToC. The pink box shows CoRe’s strategy to strengthen the awareness and capacity of civil society actors; the yellow squares are the changes that are expected upon civil society groups due to the intervention (outcomes); and the blue circles are the causal assumptions, or the reasoning that explains why CoRe believes one change or outcome will lead to the next one. These assumptions reveal a great deal about the distinctive character of the approach.
Above: A portion of CoRe’s revised draft Theory of Change diagram (in progress).
The new iteration of the Theory of Change resulted in a more detailed and comprehensive image of how CoRe believes change occurs, with various pathways of change stemming from the three main thrusts of CoRe’s strategy.
Key Lessons
This revised ToC has now set the ground for designing a programme-level monitoring, evaluation and learning (MEL) system. Akvo and CoRe will continue to collaborate to build upon the new ToC to design the MEL system, which will include a series of consultations with CoRe’s partners to ensure that the framework is designed in a way that complements, integrates and builds upon their existing monitoring an evaluation (M&E) efforts.
The experience so far, however, has already underscored important lessons:
1. Design with the end in mind
CoRe doesn’t implement its own field programmes. Instead, it works to strengthen and support the capabilities of diverse changemakers in civil society, in government, and in foundations and development agencies working across Asia, Africa, and Latin America. That means the outcome monitoring framework it’s seeking to implement needs to reflect both a shared strategic approach and the diversity of particular contexts in which its partners work. And the aim is to support both organisational learning and generate robust evidence that can independently be judged and validated. Our challenge is to help design a framework that responds to these imperatives.2. Assess, reflect and adapt
This partnership is a good example of how pursuing a clear goal often requires deviation from initial plans. In particular, we found that the rich thinking behind CoRe’s programming approach needed to be articulated in a new way to provide a basis for effective monitoring. This necessity to remain nimble underpins our best collaborative work.
3. Probe underlying assumptions
An MEL framework is only as good as the programme design, in this case the ToC. Investing time and effort into crafting a thorough ToC is essential. As partners in the process, one of the most important ways we can provide support is repeatedly asking questions like “why do you think this will happen?” and “what about your approach will enable this?” then listening carefully to identify the assumptions that animate the strategy. Clarifying the causal connections between outcomes and the corresponding assumptions provides a foundation for prioritising outcomes to measure and identify critical assumptions where evidence is most needed, which may eventually lead to formative evaluation questions.
4. Validate the approach from multiple perspectives
With a newly revised draft Theory of Change in hand, our next steps include seeking feedback from CoRe’s international advisory committee, and its diverse partners. In particular, we’re looking to ensure that the outcome pathways reflect the approaches as understood by different partners, to identify which outcomes are being measured already and how, and to understand the different ways outcome and impact measures will be communicated and used. These are all essential to ensure the resulting MEL system can be implemented in a way that is both low cost and high value.
Stay tuned for part two of this blog series, in which we’ll share how partner consultations have helped shape the MEL system design.
