Does it work?
It’s an easy question to answer when talking about the blender you picked up at a neighbor’s garage sale.
The question’s just a bit tougher to answer when asked by a funding agency trying to save the world through a comprehensive intervention.
“Does it Work?” is the kind of question that fuels the field of evaluation. Everyone involved in an intervention wants to know that they accomplished something and who wants to fund something that doesn’t work?
So how do you approach the discussion?
Below you will find 5 cartoons I’ve designed that speak to the issue of attribution. These are discussion tools, not answers.
About the illustrations
This post was inspired by Kylie Hutchinson, who gave me the idea and provided me with resources I could use to dive into the issue. The primary resource I used for this post was [PDF] Addressing the Question of Attribution in Evaluation by the Evaluation Unit of Canada’s The International Development Research Centre.
A few notes:
- If you like the post, write a comment and let me know.
- Share it with colleagues. Seeing people sharing my cartoons inspires me to create more cartoons.
- If you think I’m missing critical pieces to the overall discussion, let me know in the comments.
- Please feel free to use my cartoons in presentations, training materials, etc.
Separating from the crowd
One reason attribution is so hard is that there are often so many additional factors at play.
Impact at the micro level
Did my contribution accomplish anything?
Modeling a comprehensive intervention
The more comprehensive the intervention, the harder it is to gauge its contribution. It’s also much harder to explain and model.
Simple and complex
I tried to get at the difference between simple systems and complex systems. I’m ok with this cartoon but it still doesn’t feel quite right. What analogies do you use? Let me know in the comments.
Doing the best with uncertainty
Additional illustrations
What’s missing from this post? Let me know in the comments and I might just add more cartoons.
Additions
Susan Kistler was kind enough to offer some feedback on the simple/complex cartoon. Here is a new version based on her comments.
According to Matt Keene two trees and three whales will be saved at the 2013 EEN Pacific Forum. Since Matt didn’t specify in his post which whales, I thought I would help out. Print it out, circle your choice, then deliver to Matt.
@evalcentral: 5 illustrations on attribution bit.ly/10dPrsV @clysy we’ll save 2 trees & 3 whales at #EEN13 ow.ly/jKduV…
— Matt Keene (@mwkeene) April 4, 2013
clysy EvaluationMaven I’m also a big fan of John Mayne on contribution vs attribution. Thanks for the cartoons. #outcomemapping #eval…
leslieaj thanks for the compliment 🙂
Thanks @clysy ! @EvaluationMaven regularly inspires me as well. Love the cartoons. I tried to think a bit more about the simple/complex metaphor. How about watering with a hose versus watering with rain? The latter is relatively uncontrolled, less predictable, waters everywhere (not just where you want it), takes an entire system to replenish, etc. Not sure this does it. Am still thinking.
@susanjkistler @clysy @EvaluationMaven Thanks Susan, I like the whole rain thing better than the river I came up with. A river is more complex than a hose but it’s not as complex as rain.
Once again, spot on! Great work Chris, and thanks to Kylie for providing inspiration behind this post.
@AnnKEmery thanks Ann 🙂
AnnKEmery omg I love these! http://t.co/ijGxJgZ1RV I’m totally using these to explain attribution issues to clients #eval
david_henderson clysy Welcome to my office. http://t.co/ECpA8oY8fC
Great post Chris! This is a topic I spend a great deal of time on when I teach my graduate Program Evaluation course. I love the blender analogy and the cartoons and will be sharing these with my next batch of students in May! Most of my work (and most of my students’ work) is in PK-12 education and the question “does it work?” (and its 1st cousin “is it effective?”) is easier asked than answered. My struggle often lies in explaining the challenges in assigning attribution to program effects to non-evaluators. My students get it, but only after they’ve been with me awhile! 🙂
@SheilaBRobinson thanks Sheila 🙂
@SheilaBRobinson thanks Sheila 🙂
Now with all of the whale printing, we have to save another tree. And A-Hab-n’t you heard that its so difficult to save big and scary whales! What about a minke as big as your pinky?…As if I dont have enough to do. We’ll just aim for 2 whales next time so we are sure to succeed. Thanks chris.
@mattkeene that’s a good point Matt. Should I ask that, instead of printing out the sheet, people just call you and vote using the appropriate whale call?
Here’s a great example. I work with an organization attempting to reduce infant mortality. This graph http://theincidentaleconomist.com/wordpress/wp-content/uploads/2013/04/KindigFigure1.jpg shows that female mortality in the county they serve has improved dramatically in the past 20 years. How much of that improvement is due to this organization? I have no idea and no way to answer that question with any degree of reliability. Mortality is such a complex issue that it would take a chaos theorist to tease out all the confounding factors.
I want to save the narwhal.because they are so awesome.
http://www.youtube.com/watch?v=ykwqXuMPsoc (guilty pleasure – the narwhal song)