Theme Thursdays: Is an impact evaluation right for you?


(Stacey McDonald) #1

We use the word impact a lot in the nonprofit sector, and often when we talk about telling our story or demonstrating the difference we make. However, when we’re talking about evaluation, this word has a very specific meaning. Better Evaluation defines impact evaluation as going beyond describing or measuring impacts that have occurred to seeking to understand the role of the intervention in producing these (understanding if the program caused the change/impact or something else).

Today I read an interesting article in the Stanford Social Innovation Review: Ten Reasons Not to Measure Impact—and What to Do Instead. While there are definite benefits to engaging in an impact evaluation, it’s not always the best type of evaluation to meet your needs. The authors break down their top 10 reasons NOT to measure impact into 4 buckets:

  • Not the right tool (it can’t answer the questions you have)
  • Not now (design or implementation isn’t ready yet, or it’s too late)
  • Not feasible (don’t have the resources, or large enough sample, can’t measure well other factors that might influence outcomes, not possible to stick to research protocols)
  • Not worth it (not worth the cost, since it’s already been studied, or you won’t learn enough to make it worth the cost)

If you think it might be right for you, and you want to learn more, check out this comprehensive guide from the World Bank. For more resources, check out this list.

Evaluating Your Pilot: Setting yourself up for success
(Stacey McDonald) #2

If it’s clear that an impact evaluation isn’t right for you, either because it’s too costly, it’s not possible to do it well, it’s not the right time, etc, then what should you do instead if you want to understand if you’re work is making a difference, and you want to document (create evidence) of that difference?

Nesta has developed standards for evidence, which lays out a continuum that can show what you can work towards.

For Level 1, organizations can do this through developing a thorough theory of change that is supported by existing research and data.

For Level 2, you can engage in an outcome evaluation using multiple methods, including pre- and post-surveys, or other methods that can track changes that are occurring.

For more, check out this article: Creating evidence of your impact by Design for Europe