This expert post comes from Gary Woller, President of Woller & Associates and a M&E consultant to the STRIVE Program.
This is the question I was asked about a year ago when I began an assignment as a monitoring and evaluation consultant to the STRIVE Program. Since then, I have worked closely with this multi-organizational initiative to develop results indicators for economic strengthening projects and to develop M&E ‘systems’ to accurately track, capture, analyze and use the results indicators to improve interventions. Our goal has been to design M&E systems that are as rigorous as possible, while remaining practical for individual projects to implement. Achieving this has proven to be no small feat and, frankly, we still have a ways to go.
STRIVE is currently working in poor, remote areas of the Philippines, Liberia, and Mozambique and in urban areas of Kabul, Afghanistan. Economic interventions include, respectively: traditional value chain activities, value chain activities linked to public schools, village savings and loans, and youth apprenticeship programs. Coming up with enterprise and households indicators that can be compared across these interventions has been a challenge, but developing practical indicators to monitor the impact of the interventions on children and/or youth has been, if possible, an even bigger challenge. I suspect this is not news to many of you who have faced this challenge in your own work.
I often find that economic development projects know (or can plausibly estimate) the outcome of their activity on the people they work with directly (for example, adoption of new production methods/technologies, forging of new commercial relationships, increased sales prices and sales, new financial products developed, producer associates established or strengthened, etc). Where there is much less clarity is on what impact, if any, these improvements have had on children or youth. Are children healthier, safer, or better educated? Are youth less likely to drop out of school, resort to violence to settle disputes, take up arms, or enter into exploitative relationships? By contrast, projects focused on non-economic interventions (e.g., nutrition, vaccinations, other health provision, vocational education) to improve the lives of children and youth can point to improved health and education but struggle to understand the implications of those improvements on the overall security of the child or the household within which they live. Ultimately, that is what CYES is about, but if we cannot track how we are doing, how can we know whether we are successful? And if we cannot know whether we are successful, how can we improve?
As one step toward addressing this, I have outlined the STRIVE M&E process and lessons we have learned to date below. I hope to hear readers’ reactions to our process and lessons as well as their own experience in this area.
STRIVE M&E Development Process (Note: The process does not necessarily proceed sequentially in the order presented below.):
- Develop an overall Learning Agenda including guiding research questions and child and youth level ‘impact domains’ to be monitored.
- Develop country-level intervention strategies.
- Establish detailed causal model(s) and Performance Monitoring Plans (PMPs) for each country-level intervention strategy. Clarify anticipated benefit for children and/or youth and assumptions being made by the project.
- Identify a range of ‘common indicators’ falling within each impact domain, to the extent possible, across each project.
- Identify areas of overlap with program Learning Agenda (including indicators and information gathering methodologies) and incorporate those into the project M&E design.
- Where overlap does not exist, negotiate with the project to incorporate the program Learning Agenda into project M&E design.
- Select M&E methods appropriate to answer the primary research questions and the common indicators within each child and youth impact domain and consistent with the level of validity sought and resources available.
- Develop a tailored M&E system in collaboration with each country project team that includes indicators belonging to common indicators within child and youth impact domains, along with indicators of interest to the individual projects. The M&E system includes both routine data collection and reporting and complementary information gathering using a tool box of methodologies.
That is about as far as we have gotten to this point.
Lessons Learned to Date:
- Measuring the child/youth impacts of economic strengthening projects is particularly challenging because the impacts tend to be indirect rather than direct. It is necessary to trace through the cause and effect logic that leads to child/youth impacts identifying each important causal link. Causal model development takes time but can pay substantial dividends down the line.
- Researchers in child and youth areas have identified seven common impact domains: (1) family economic well-being, (2) health, (3) education, (4) safety/behavioral concerns, (5) community connectedness, (6) social relationships (family & peers), and (7) emotional/spiritual well-being. We have recommended the first three as common impact domains for STRIVE given the contexts and nature of its projects. The fourth impact domain also appears to be relevant for the STRIVE Afghanistan project. We are further considering child use of time (e.g., children working) as an additional cross-cutting indicator for the STRIVE projects based on feedback in the Philippines that stressed child use of time as a critical impact domain.
- Economic strengthening projects also affect children and youth in a variety of other ways (many of them possibly surprising) outside of the three aforementioned impact domains. Again, these are best discovered through a careful, complete construction of the project’s casual model.
- One of the most substantial challenges to improved integrated programming is the language and assumption barriers that exist between different areas of specialization. The causal model development process forces implementers to identify, question and test assumptions about how a particular intervention is going to achieve the project’s objectives. These are likely to include project sustainability, clear exit strategies, and the improved wellbeing of children and youth (whether health, education, social, physical, financial, etc.).
- Work from the bottom up. In implementing a large, complex program like STRIVE, the best approach is to begin with the individual projects and work up from there to a more generalized M&E framework. Beginning with each project’s causal model, we have been able to identify themes and related indicators that make sense to track across the diverse initiatives.
- Be flexible about monitoring and evaluation methodologies. Sometimes getting accurate information is more important (such as with a formal evaluation), while many other times ‘reasonable accuracy’ (or approximate accuracy) is sufficient, particularly when the purpose is to generate ongoing information to guide project management. The best M&E systems will use a ‘toolbox’ of information gathering activities.
- Projects can rarely afford or manage control groups of non-beneficiaries, in which case formally attributing observed results to project operations is not possible. In such cases (which are the majority), focus monitoring activities on outcomes over which the project has more or less control and for which it can, thus, claim ‘plausible attribution.’ This includes, particularly, outcomes measuring behavioral changes adopted by beneficiaries and other sector actors and that are targeted by project activities.
- Adjust M&E activities to project design attributes where possible. STRIVE Mozambique, for example, planned to implement three variations of its ‘service delivery package’--villages savings and loans (VSLs), rotating labor groups, and VSLs with rotating savings groups. By randomizing the placement of service delivery packages in selected areas, STRIVE Mozambique has created a natural experiment that uses its normal M&E activities to test the comparative effectiveness of the three service delivery packages.
Moving forward the STRIVE program will continue to develop and refine its M&E strategy, document its experience, and synthesize the lessons learned.
We hope that our experience will benefit other child and youth development practitioners, and particularly those that are using economic strengthening interventions as a means to benefit children and youth. Meanwhile, I encourage you visit the microLINKS page on the Monitoring and Evaluation of Value Chain Projects for a good overview of M&E issues related to economic strengthening projects.
I look forward to your comments and feedback.
- Gary.
I recently had the opportunity to attend the Orphans and Vulnerable Children Forum hosted on 25-26 June 2008 by Catholic Relief Services. The event showcased CRS’s work in OVC programming and, of particular interest to me, the tools the agency has used in monitoring and evaluating program effects on children. The following overview of key sessions on this topic is meant to provide some background for others who are interested. I would also like very much to hear from practitioners and researchers what tools you find effective for monitoring and evaluation and impact assessment with children.
Child Protection Needs
In the “Promising Practices for Child Involvement” session, Shepherd Mupfumi and Daphyne Williams addressed the need to include children in OVC research and project design, while at the same time protecting them from harm. Children’s lack of power in research settings due to their youth and limited capacity to provide informed consent is compounded when working with OVC by factors such as the loss of one or both parents, poverty, displacement, and stigma associated with HIV/AIDS. (An overview of risk and ethics in human research is available here, and background on the ethics of research involving children can be found here). In large part due to the HIV/AIDS crisis, development practitioners have grown increasingly sensitive to the importance of child protection within ethical research design.
Child Protection Mechanisms
CRS employs a variety of mechanisms to protect children in project research and implementation. For example, in rural Zimbabwe, CRS partners work with Child Protection Committees composed of child representatives, caregivers and community representatives throughout the program cycle, seeking their input on decision making, project design, implementation and monitoring (see Mr. Mupfumi’s presentation, “Involving Children” for further detail). This is in keeping with the agency’s “Do No Harm” principles, which mandate that risk to OVC must be assessed and addressed throughout the project cycle. CRS has also instituted guidelines for HIV-related research in field projects. These include recommendations about when it is necessary to seek external review of proposed research activities, as the agency, like most non-profits, does not have its own internal review board (IRB) process. (Refer to Ms. Williams presentation on the “Do No Harm Implementation Strategy” for more information.)
In a subsequent session, “Making a Difference: Documenting Results” the focus was on specific tools used to gather information from children themselves. Featured tools included:
- The Child Status Index (CSI) Tool
- CRS’s OVC Wellbeing Tool (OWT)
- The “Station Days” data-collection methodology used by CRS in Zimbabwe.
The Child Status Index (CSI)
Karen O’Donnell of Duke University Medical Center presented the CSI. The index was developed in response to OGAC’s need for a simple, child-centered tool that would show change in child well being over time, identify current and future child needs, and guide longer-term program interventions. The CSI uses a community-based, participatory methodology to gather data on children’s food and nutritional status, shelter and care, protection, health care, psychosocial well being, and education and vocational skills. See Dr O’Donnell’s presentation on the OVC Forum resource sitefor further detail on the CSI methodology and indicators.
OVC Wellbeing Tool (OWT)
Shannon Senefeld presented the results obtained from the CRS OWT during a pilot which incorporated the tool into 5 PEPFAR country evaluations. In conjunction with other measurement tools, the OWT, a self-reporting tool for the 13 – 18 year-old cohort, helped identify issues important to children. For example, through the OWT, OVC in institutions in Haiti reported stronger and more consistent feelings of self-worth and self-confidence than children in communities because they felt connected to their peer group, whereas many children living in communities were more likely to feel isolated from their peers. The OWT also helped identify gender disparities in OVC well being – while orphans overall suffered more negative nutritional impacts, maternal orphans fared even worse, and boys overall had better nutritional results than girls. For more information on the OWT, download CRS’s presentation on the tool at the International AIDS Conference in Mexico City in August, or download the OWT at CRS’s website.
"Station Days" Methodology
Finally, Joyce Tamuka Chitemere of CRS Zimbabwe presented the “Station Days” methodology, which provides OVC aged 5 to 18 with information and material assistance while also collecting quantitative and qualitative data about their well being. On a quarterly basis, “Station Days” brings children to a central location (such as local school) where they pass through a series of stations, providing information and receiving care. For instance, at the station where children’s weight and height is measured, staff perform small personal care tasks for the children. School performance is assessed by reviewing children’s exercise books. Interactions with counselors and elders both provide and check the effectiveness of psychosocial support. On a quarterly basis, the CRS Zimbabwe program uses this approach to sample approximately 180 OVC per day, involve their technical staff more directly in monitoring and evaluation, and provide the children with an opportunity for self-expression and direct input into program goals. The “Station Days” methodology has been effective at raising awareness of issues affecting children but is not without risks, including encountering emotional triggers (particularly in the counseling stations) and raising children’s expectations (staff have found that children anticipate that their feedback will be quickly incorporated, which is not always feasible).
All in all, the day offered insight into a number of tools and current initiatives to improve M&E for child-focused programming by using methods that directly involve children. To see what else was covered and/or learn more about the sessions I described above, visit the OVC Forum resource page, where both presentation slides and video of the presentations are archived. Have you used any of these tools or techniques? What was your experience with them? What other tools for involving children in M&E and research have you used?
