How to collect stories for your program evaluation?


(Stacey McDonald) #1

There are many ways that you can systematically collect stories for your program evaluation. Below are a few popular options and some links to find out more.

Here are few tips that can apply to many of the methods below:

  • Don’t leave anything out! In person, don’t just collect what’s said, but also how it’s said. Body language is also a type of data that can tell us a lot. Capture context, too.
  • Start with a list of key questions, but the real gold often comes from digging deeper and being open to going where the participant wants to go. Don’t ignore that. They want to talk about something for a reason.
  • Always let people know when you are collecting information for an evaluation. You don’t want to write down stories people are telling you as they are casually interacting with you, without any knowledge of your role in collecting data. You don’t want people wondering if they have to watch what they are saying in front of you, so be clear about your role as evaluator, and when you’re wearing that hat (if it’s not your full time job).

Use another method that isn’t included? Share it! Want to know more about one of these? Let me know! We can explore it together, and if there’s interest, I could offer a webinar and/or workshop on the topic (or find others that do this).

  • Interviews: a conversation in which an interviewer asks a set of questions to the interviewee to get information from them on a topic. They are useful when you want to collect in-depth information on a topic, and want to understand the context and motivations of participants. Learn more here.

  • Focus Groups: a small group discussion (6-10 people) that typically follows a structured plan to investigate a set of questions. It’s led by a facilitator, and sometimes a co-facilitator or note taker. Great when the interaction between participants will be helpful to analysis, and you want to generate in-depth understanding about an issue, including people’s opinions and motivations. Check out this resource to learn more.

  • Most Significant Change: is a way to generate and collect personal stories of change, and then a process to deciding which of these changes is the most significant and why. This method can be helpful when trying to understand how a change happens, and when (in what situations and contexts). Want to learn more? Check out this resource and this guide from Rick Davies and Jess Dart.

  • Journals, Diaries or Logs: journals or diaries require that participants provide information over a long period of time, and filled out on an ongoing basis (write immediately, or close to, the timing of the subject). Staff members or volunteers could also be asked to fill out a log, documenting conversations or observations. Check out this article about using diaries.

  • Observation: evaluator observes and records behaviour either as a participant or a non-participant to understand the goals, cultures, challenges, motivations, and themes that emerge (a lot of observing). Alternatively, you could analyze photography/video/recordings to discern changes that might be taking place over time.

  • Surveys: You can ask open-ended questions as part of a survey, but this is a limited way of collecting stories, as you won’t be able to follow up, or dig deeper. Open-ended questions are often included in surveys to help explain answers to quantitative questions. Be wary of including too-many open-ended survey questions - they can lower your survey response rate if people feel they take too long, or don’t want to answer the questions.

I’ll likely update this resource as time goes on, so please let me know if there’s a great resource out there I should add.