The SIM TIG sponsored the following sessions at Eval 23 in Indianapolis in October 2023.
Qualitative Data Tools for Impact Measurement and Management
Slavica Stevanovic, Qatalyst Research Group
Wednesday, October 11
4:15-5:15pm
Room 309/310
“The truth about stories is that that’s all we are,” Thomas Kind tells us in his book ‘The Truth About Stories – A Native Narrative.’ Stories shape who we are, where we came from, and how we understand and interact with others. Qualitative data or storytelling has become an accepted methodology in decolonized and Indigenous-based approaches for evaluating the effectiveness of programs. However, there continues to be some resistance to incorporating stories in the emerging field of impact measurement and measurement. Investors, particularly private sector investors, tend to prioritize the use of quantitative data and the efficiency of data collection. Using personal stories to demonstrate or illustrate impacts is often considered time-consuming, expensive, and not representative. It is also messy. How would we choose whose story to tell? Do we only tell positive stories? What happens to other stories? It is important to have strong quantitative data. We need to collect extensive, representative and defensible data on impacts (e.g., data on financial returns, scalability, employment, gender balance, health outcomes, emissions reductions, technology development, etc.). However, we must not forget the stories that numbers cannot tell. This presentation will demonstrate how stories draw us in, bring impacts to life, engage our emotions, improve understanding, create empathy, and even raise funding. The stories can also deceive us and create illusions of success. We will use examples from multiple projects to illustrate how storytelling identified challenges, gaps, and unintended outcomes, painted a more complete picture of the impacts, and contributed to significant improvements in future investments.
Download the slides here: Eval 23 Qualitative IMM.pdf
Social Return on Investment (SROI) – An Evaluation Framework for Localization? (Poster)
Shubha Kumar, USC
Sara Olsen, SVT Group
Aaron Mallett, SVT Group
Saji Prelis, Search for Common Ground
Wednesday, October 11
5:30-7:00pm
Poster #170
The youth, peace and security (YPS) agenda is gaining momentum across policy, funding and practice communities. To match this growing interest, the YPS agenda requires a change in the mindset of how we think of impact and how we quantify that impact as it relates to youth-led and adult-supported interventions that impact the lives of people in youthful majority countries. Measuring the right impacts changes the narrative and puts power where it matters – in the hands of young people locally, something critical in understanding the role of young people in preventing violence and sustaining peace. The international community critically needs a way to assess the impacts of youth-led and youth-supporting peacebuilding interventions to better support and sustain these efforts.
While evaluations of peacebuilding programming are conducted regularly, little is known about the overall impact and return on investment of such programs. Traditional economic evaluations often shed light on cost savings to the State but are hard to come by in the peacebuilding field. Furthermore, traditional approaches do not capture the value of interventions to all key stakeholders, including youth and communities, and what matters most to them. To fill this gap, a research team led by the Institute on Inequalities in Global Health at the University of Southern California in collaboration with Search for Common Ground and multiple partners, conducted a proof-of-concept study of the impact of peacebuilding interventions using social return on investment (SROI) methodology, a stakeholder-centered participatory approach to impact measurement and management, to evaluate youth-led and youth-supporting peacebuilding interventions in Kenya.
This poster will describe the approach used, results that were found, and implications for practice, including for localization in international aid and development.
Download the poster here: Eval23 SROI Localization.pdf
What is outcome valuation and how it can help tell a story: lessons from the field (Poster)
Seth Tucker, TCC Group
Wednesday, October 11
5:30-7:00pm
Poster #171
Social and environmental outcome valuation is the process of converting project outcomes to monetary values. Through this conversion, project inputs and outcomes can be measured in the same monetary unit which can demonstrate the amount of value a project creates and whether it creates more value than it costs to implement. By presenting, or supplementing, a project with this information, potentially abstract concepts like environmental and social impact can be presented in a tangible and understandable unit of measurement to a variety of stakeholders. This can help projects raise funds, build coalitions, and tell their story efficiently and transparently. Using lessons of what went right and what went 'less right' from their experiences with outcome valuation in the field, the presenter will explain what social and environmental outcomes valuation is, methods used to value outcomes, when these methods are appropriate, how these methods are used in designs such as Social Return on Investment and Social Impact Bonds, and how it can be used to tell or enhance a story of whether, and for who, a project is creating value.
Download the poster here: Eval23 Outcome Valuation.pdf
Using Most Significant Change Stories in Social Finance System Change Initiatives that Advance Racial Equity
Jasmine Johnson, Jasmine Johnson Consulting
John Sherman, Sherman Impact Consulting
Thursday, October 12
10:15-11:15am
Room 201
Most Significant Change stories (MSC) have been widely practiced in the development context but have been less visible in the U.S. context. First developed in the 1990s by Rick Davies, it wasn't until 2005 that Davies and his colleague Jess Dart published a comprehensive guide for the technique of using Most Significant Change stories. While developing the evaluation plan for assessing a complex systems change initiative which is based upon using innovative finance, our evaluation team viewed MSCs as an ideal method to meet our challenge. This session will demonstrate our use case in which this storytelling method benefits an evaluation of an impact investing initiative aiming to disrupt patterns of racial inequity in lending decisions. We will show how we applied it to elevate and better understand the experiences of beneficiaries who are removed from the direct intervention. We will also speak to the power of story as data for audiences who do not typically engage with (and may even be resistant to) evaluation.
Download slides here: Eval23 Most Significant Change.pdf
Story's Power for Measuring Impact in a Decolonizing World
Andrea Nelson Trice, National Institute for Social Impact
Thursday, October 12
3:45-4:45pm
Room 104
As we continue to pursue decolonization, more Majority World leaders are now sitting at the table, making decisions with Western leaders about how development programs will proceed and who will lead the work. But often decisions about what outcomes will matter still reside with those who control the funds. This presentation explores the stories of 90 leaders who are involved in international development work, 45 from Majority World countries and 45 from the West. Differences in Majority World and Western perceptions of the outcomes that truly matter, and the cultural values that underly these beliefs form the core of this presentation.
Two key themes emerged from the stories related to impact measurement. First, according to Majority World leaders, empowerment is crucial to measure. They operationalized this to include proximity of leaders to the people of focus, use of local resources, regular training on topics of value to the people of focus, and regular feedback from them. Paths to empowerment that require significant time – building trust with community leaders, using word-of-mouth to attract people to the work, tapping the social capital of local organizations – were also described as key elements to measure according to enterprise leaders.
This session will explore how both Majority World and Western stories brought to light important, culturally-bound assumptions that influence how impact is conceptualized and measured. The session will also include time to grapple together with how we are measuring impact today, even as we continue to decolonize.
Download slides here: Eval23 Power of Story Decolonizing.pdf
The stories we tell ourselves about the big, bad private sector – repositioning the value proposition of evaluation in corporate contexts
Jared Raynor, TCC Group
Morgan Buras-Finlay, Raya Cooper Impact Consulting
Veronica Olazabal, BHP Foundation
John Sherman, Sherman Impact Consulting
Friday, October 13
3:45-4:45
Room 204
Evaluation and evaluative thinking (also known as impact measurement and management across the private sector) is gaining traction in market-based contexts (i.e. impact investing, corporate settings) especially among entities that invest in their foundation, corporate responsibility, and in their ESG reporting. But unlike nonprofit and traditional foundations where evaluation is often “in the drinking water”, evaluation and market-based contexts still have a lot to learn about one another.
When the value of evaluation is not implicitly understood, how do evaluators talk about what they do? How do they build buy-in for measuring impact in market-based contexts? Does the story of the “big, bad private sector” need to change? What stories do they need to tell about the value propositions? And conversely, what stories do businesses and private-sector collaborators tell about evaluation? How does evaluation battle “green washing”, and become a substantive player in corporate social impact efforts?
In this interactive session, we will share stories of how we have communicated the value of evaluation, and facilitate a conversation with the audience about methods, strategies and obstacles.
Download slides here: Eval23 Corporate Contexts.pptx