This expert post comes from Gary Woller, President of Woller & Associates and a M&E consultant to the STRIVE Program.
This is the question I was asked about a year ago when I began an assignment as a monitoring and evaluation consultant to the STRIVE Program. Since then, I have worked closely with this multi-organizational initiative to develop results indicators for economic strengthening projects and to develop M&E ‘systems’ to accurately track, capture, analyze and use the results indicators to improve interventions. Our goal has been to design M&E systems that are as rigorous as possible, while remaining practical for individual projects to implement. Achieving this has proven to be no small feat and, frankly, we still have a ways to go.
STRIVE is currently working in poor, remote areas of the Philippines, Liberia, and Mozambique and in urban areas of Kabul, Afghanistan. Economic interventions include, respectively: traditional value chain activities, value chain activities linked to public schools, village savings and loans, and youth apprenticeship programs. Coming up with enterprise and households indicators that can be compared across these interventions has been a challenge, but developing practical indicators to monitor the impact of the interventions on children and/or youth has been, if possible, an even bigger challenge. I suspect this is not news to many of you who have faced this challenge in your own work.
I often find that economic development projects know (or can plausibly estimate) the outcome of their activity on the people they work with directly (for example, adoption of new production methods/technologies, forging of new commercial relationships, increased sales prices and sales, new financial products developed, producer associates established or strengthened, etc). Where there is much less clarity is on what impact, if any, these improvements have had on children or youth. Are children healthier, safer, or better educated? Are youth less likely to drop out of school, resort to violence to settle disputes, take up arms, or enter into exploitative relationships? By contrast, projects focused on non-economic interventions (e.g., nutrition, vaccinations, other health provision, vocational education) to improve the lives of children and youth can point to improved health and education but struggle to understand the implications of those improvements on the overall security of the child or the household within which they live. Ultimately, that is what CYES is about, but if we cannot track how we are doing, how can we know whether we are successful? And if we cannot know whether we are successful, how can we improve?
As one step toward addressing this, I have outlined the STRIVE M&E process and lessons we have learned to date below. I hope to hear readers’ reactions to our process and lessons as well as their own experience in this area.
STRIVE M&E Development Process (Note: The process does not necessarily proceed sequentially in the order presented below.):
- Develop an overall Learning Agenda including guiding research questions and child and youth level ‘impact domains’ to be monitored.
- Develop country-level intervention strategies.
- Establish detailed causal model(s) and Performance Monitoring Plans (PMPs) for each country-level intervention strategy. Clarify anticipated benefit for children and/or youth and assumptions being made by the project.
- Identify a range of ‘common indicators’ falling within each impact domain, to the extent possible, across each project.
- Identify areas of overlap with program Learning Agenda (including indicators and information gathering methodologies) and incorporate those into the project M&E design.
- Where overlap does not exist, negotiate with the project to incorporate the program Learning Agenda into project M&E design.
- Select M&E methods appropriate to answer the primary research questions and the common indicators within each child and youth impact domain and consistent with the level of validity sought and resources available.
- Develop a tailored M&E system in collaboration with each country project team that includes indicators belonging to common indicators within child and youth impact domains, along with indicators of interest to the individual projects. The M&E system includes both routine data collection and reporting and complementary information gathering using a tool box of methodologies.
That is about as far as we have gotten to this point.
Lessons Learned to Date:
- Measuring the child/youth impacts of economic strengthening projects is particularly challenging because the impacts tend to be indirect rather than direct. It is necessary to trace through the cause and effect logic that leads to child/youth impacts identifying each important causal link. Causal model development takes time but can pay substantial dividends down the line.
- Researchers in child and youth areas have identified seven common impact domains: (1) family economic well-being, (2) health, (3) education, (4) safety/behavioral concerns, (5) community connectedness, (6) social relationships (family & peers), and (7) emotional/spiritual well-being. We have recommended the first three as common impact domains for STRIVE given the contexts and nature of its projects. The fourth impact domain also appears to be relevant for the STRIVE Afghanistan project. We are further considering child use of time (e.g., children working) as an additional cross-cutting indicator for the STRIVE projects based on feedback in the Philippines that stressed child use of time as a critical impact domain.
- Economic strengthening projects also affect children and youth in a variety of other ways (many of them possibly surprising) outside of the three aforementioned impact domains. Again, these are best discovered through a careful, complete construction of the project’s casual model.
- One of the most substantial challenges to improved integrated programming is the language and assumption barriers that exist between different areas of specialization. The causal model development process forces implementers to identify, question and test assumptions about how a particular intervention is going to achieve the project’s objectives. These are likely to include project sustainability, clear exit strategies, and the improved wellbeing of children and youth (whether health, education, social, physical, financial, etc.).
- Work from the bottom up. In implementing a large, complex program like STRIVE, the best approach is to begin with the individual projects and work up from there to a more generalized M&E framework. Beginning with each project’s causal model, we have been able to identify themes and related indicators that make sense to track across the diverse initiatives.
- Be flexible about monitoring and evaluation methodologies. Sometimes getting accurate information is more important (such as with a formal evaluation), while many other times ‘reasonable accuracy’ (or approximate accuracy) is sufficient, particularly when the purpose is to generate ongoing information to guide project management. The best M&E systems will use a ‘toolbox’ of information gathering activities.
- Projects can rarely afford or manage control groups of non-beneficiaries, in which case formally attributing observed results to project operations is not possible. In such cases (which are the majority), focus monitoring activities on outcomes over which the project has more or less control and for which it can, thus, claim ‘plausible attribution.’ This includes, particularly, outcomes measuring behavioral changes adopted by beneficiaries and other sector actors and that are targeted by project activities.
- Adjust M&E activities to project design attributes where possible. STRIVE Mozambique, for example, planned to implement three variations of its ‘service delivery package’--villages savings and loans (VSLs), rotating labor groups, and VSLs with rotating savings groups. By randomizing the placement of service delivery packages in selected areas, STRIVE Mozambique has created a natural experiment that uses its normal M&E activities to test the comparative effectiveness of the three service delivery packages.
Moving forward the STRIVE program will continue to develop and refine its M&E strategy, document its experience, and synthesize the lessons learned.
We hope that our experience will benefit other child and youth development practitioners, and particularly those that are using economic strengthening interventions as a means to benefit children and youth. Meanwhile, I encourage you visit the microLINKS page on the Monitoring and Evaluation of Value Chain Projects for a good overview of M&E issues related to economic strengthening projects.
I look forward to your comments and feedback.
- Gary.
