Many great questions were asked during the webinar. I did my best to answer them at the time, but I didn’t get to answer to all of them. I’m going to start responding to them here, as well as expand on questions already answered.
Q1. Can someone look over your evaluation plan with you on the Knowledge Centre?
A1. Yes! If you’d like to share your draft plan here, I would be happy to look it over, and others would also be able to provide feedback.
Q2. Would partnership development with groups who can help evaluate be a legitimate expense?
A2. I’m not entirely sure what “partnership development” was meant to refer to in this question. However, all funds needed to carry out an evaluation are legitimate expenses. All funds requested for the evaluation need to be explained clearly in the budget, and the corresponding activities should also be listed in the project plan. I feel that partnership development here could be one of three things: 1) funds needed to hire, or compensate a partner that might help develop an evaluation plan, tools or measures; 2) funds needed to bring together a group of organizations to develop a shared evaluation plan, tools and measures; or 3) funds needed to engage stakeholders (beneficiaries, community members, etc.) as part of the evaluation. All of these would be legitimate expenses. If the person that posted the question meant something else, please follow up!
Q3. Say your pilot is just one of a few programs that your organization provides that all desire a similar outcome for your population. How would you best determine which program is causing the desired outcome?
A3. To be able to understand if a pilot caused a change or outcome, you would need to do an evaluation using an experimental (randomly assigned control and test groups) or quasi-experimental design (comparison group design). For more on these research designs, check out Sage Publications’ chapter on Reseach Designs. Also see the discussion above and this post acknowledging that these designs aren’t always possible, or even desirable. It’s much easier (but still not always easy) to understand if a change occurred, and then try to understand a specific program’s contribution to that change. However, if you are running multiple programs that all have a common desired outcome, then you could develop some common measures and evaluation tools (such as a common survey, or common interview questions), and then analyze and compare the results. Do people have better outcomes with some programs than others, with a combination of programs, with a certain level of participation? Also explore if there are any other plausible explanations for the results you are seeing (other services, changes in the community, etc.).
Q4. Can you give any advice for getting a diverse team to work together to create a strong evaluation process? Also, who should ‘own’ the responsibility for overseeing the evaluation in an ideal world - recognizing that everyone has a stake, but someone needs to keep the ball moving and help keep a team accountable.
A4. While many evaluations are carried out by a team, there usually does need to be someone assigned with keeping the ball moving, as you say. Who on your team has the interest, and ability (time & skills) to play this role? It needn’t be the person with the most experience with evaluation, but rather the ability to coordinate the work, bring team members together, and keep things on track.
I’m also going to make a suggestion here to consider engaging team members outside of staff. It can be very beneficial to include a variety of stakeholders.
Q5. Is there help to evaluate the data you collect? Sort it and decipher your results?
A5. Generally, there are three options: 1) a staff member or volunteer has an interest in evaluation and learns these skills (either through a course, series or workshops or free online materials); 2) you hire an evaluator or researcher to help you do this; or 3) you see if there is a community partner that can assist you (academic institution, social planning council, health unit, etc.). Ideally, you would have funds earmarked for your evaluation within your project budget (in the future, request funds for this work as part of your grant request), but if that’s currently not possible, there are options.
Have more questions? Post them here!