Describing how a process works is valuable for two reasons. It forces you to make sure you know how it works. Then it forces you to take the reader through the same sequence of ideas and deductions that made the process clear to you.
William Zinsser
The whole point of having an evaluation mindset is that by knowing how something works (or does not work) you can try to make it work better. Or it can inform you as to when you should abandon your plans and take a different path. Or it can give you insight on other pathways to reaching your target consequences.
But mindset doesn’t mean just keeping the thoughts in your head. There is incredible value in putting ideas down on paper.
Modeling your evaluation thought process creates something that can be built upon. It makes your thoughts and evidence tangible, so they can be measured and tweaked.
And it gives you something you can share with others. A representation of your thought process.
All Models are Wrong
But some are useful.
I have always liked that quote by the late British statistician George Box.
In the field of evaluation, models are important. We use them to ground our work and communicate our assumptions.
They carry different names depending on your context, sometimes called logic models or log frames and other times theories of change. Over time these concepts have naturally evolved, increasing complexity and decreasing accessibility.
But for your evaluation mindset, I encourage you to think of them as something else.
Representations.
They can be simple depictions of the actions and consequences discussed in my last post.
And like all models, they will be wrong.
But our goal isn’t to make them right.
Our goal is to make them useful.
Doodle Modeling
So let’s draw some useful models.
We’ll start with a basic action – consequence pair. Then we’ll expand up on it, adding additional details and space for measurements/context.
We’ll doodle it out on a piece of paper using arrows and boxes to connect different pieces.
Action: We Teach ______
Consequence: They Learn _____
Now let’s expand upon it a bit.
Action: We Teach _____
Measure: How much did we actually teach? How many people did we teach? How long did we spend teaching them?
Consequence: They Learn _____
Measure: Did they actually learn _____? How much of what was taught did they learn _____?
Now let’s go a little bit further with more detail.
Goal: We want to teach all high schoolers in our county personal finance.
Context: We are given an hour once a week to speak to a single class of high school seniors. This class is mandatory and we will see the same students over and over again. We were awarded a local government grant to put on a single class and cover a little bit of prep time.
Action: We teach practical personal finance lessons for students who will soon be off to higher education or fully into the working world.
Measure: How many lessons were taught? Were all of the students present for all lessons?
Consequence: They learned how to create a budget and tips for staying on that budget. They also learned how to use tools to monitor their progress.
Measure: Did they actually learn how to create a budget? Will they start creating/following budgets on their own after the program has ended?
Doodle Modeling an Indie Business
Alright, so let’s do another example.
Action: Create Websites
Consequence: Get Paid
It doesn’t take long before it dawns on you that just creating websites doesn’t get you paid.
You need to find people who will hire you to create websites. Then you can create the websites and get paid. So the tricky part of the business, most of the time, is finding clients.
It’s not that getting paid to create websites isn’t the work. It’s just not particularly useful to focus a lot on that aspect. So instead of building out that model, let’s build out another one.
Action: Attending Networking Events
Consequence: Get Clients
Okay, so this pair is a bit more useful in my opinion, at least for most of the indie businesses I know who are constantly struggling to find new clients.
Let’s expand it out a little.
Action: Attending Networking Events
Measure: How many networking events did you go to? How many prospective customers did you meet?
Consequence: Get Clients
Measure: How many follow-up coffee chats were setup? How many new clients were found that can be directly connected to that event? How many indirectly connected new clients?
Models make actions and consequences tangible.
The goal of most models isn’t academic.
It’s about translating a bunch of ideas, evidence, goals, and strategies into something tangible. Something that you can wrap your mind around. Something that you can use to discuss your plans with other people.
You are creating a useful representation.
You don’t need a special degree or certification to create a model. It doesn’t have to be super fancy or made with super fancy software.
A pen and paper is a great place to start.
Where Doodle Models meet Logic Models
So for all of you professional evaluators and nonprofit program managers who are trying to tie this to the work you do right now. This is how I connect the two.
Outputs are measurable actions.
Outcomes are measurable consequences.
Tip: Start Simple then Expand
The action-consequence pair is the heart and soul of the model.
If you begin with that, you can then start to expand and add more detail. You can add more measurements, more context, more detail.
And while in our complex world, the idea of boiling our projects down to action-consequence pairs might seem wrong, it can certainly be useful.
Practice Time
Time to doodle some of your own models. To get you started, here are some prompts.
Let’s say you want to have a fantastic weekend (consequence) what actions will help make that consequence a reality? How would you measure both your actions and consequences?
Maybe you would like to run, not walk, an entire 5K (consequence). What actions would help make that consequence a reality? How would you measure both your actions and consequences?
Let’s say you want to help the people who are important in your life feel more appreciation (consequence). What actions would help make that consequence a reality? How would you measure both your actions and consequences?
Thank You!
As our collective attention gets pulled in every which direction, it becomes increasingly meaningful to have your work read, discussed, and shared.
So thank you for reading, commenting, and sharing, it really means a lot.
Good stuff. There is too much jargon in Results-Based Management. We shouldn’t expect doctors, farmers, engineers, teachers, judges etc to learn the RBM jargon, just to explain in simple terms what they want to change, and why.
Thanks Greg, I think it’s particularly hard for any field to see their own jargon. When talking among peers precision can be useful, but when you have to take that same talk to wider audiences…
Yes. This approach to action-consequence pairs (concepts and causal connections) is the same as how we make a practical map (also called knowledge map). This lets us create models (maps) that fit how we see things, rather than forcing things into categories to fit a model. http://meaningfulevidence.com/news-resources/collaborative-mapping
Nice Resource Bernadette, thanks for sharing.
I like your focus on collaborative mapping. The process is often far more powerful than the product!
I like that this helps get people on the same page – both internal and external stakeholders. It breaks it down into opportunities for people to share their expertise and ideas as well as list out assumptions. I would argue, this reflection of actions and consequences should be part of any culturally competent evaluation practice!
That’s a good point Catherine, and an argument I can certainly get behind.
I think there is a tendency to use specific terms to avoid some of the baggage associated with common everyday words. But different terms are almost guaranteed to limit opportunities for people to participate.
Thanks for this post, Chris — LOVE the logic model repair shop!! I still struggle with when I should use graphics and pictures versus lots of words versus linear arrows despite nonlinear work and contexts. I have come around to “it depends” on the audience and use and purpose. Sometimes the logic model is just for me to keep my evaluation efforts focused and it needs validation from the program folks but they don’t “groove” on it like I do. Other times it is a helpful communications tool to increase stakeholder understanding of complex pathways. And stories and metaphors help to communicate in addition to what is on the page — I have heard program teams talk about planets and gravitational pull, friction, flowers trying to grow in the shade, oil wells and the building up of (community) pressure, etc. And if the metaphor works then I try and use it, shape it. https://www.inciter.io/thinking-like-aesop-to-visualize-theory-of-change/ And they CAN CHANGE OVER TIME as we do more, learn more, and contexts change. Logic models and theories should be more about the process of continuous evaluative thinking and reflection than a final pretty picture.
I am also reminded that as evaluators we need to be extra careful that what we prescribe as a theory or logic model better have the basics of good theory and logic. Both Thomas Schwandt and Nick Smith have been telling the field that we need better theories and better research on our evaluation theories.
Not that I disagree Tom.
But I feel like the story “Evaluation Theorists and Researchers Advocate for Better Evaluation Theory and Research” is quite similar to other stories I hear on daily basis.
(Marketers often say we need better marketing practice. Data scientists say we need better data science practice. Designers say we need better design practice.)
Personally I think we need more evaluators, or people who feel somewhat qualified to take on the role of an evaluator, more than I think we need more evaluation research. And I worry about adding academic disqualifiers, which I think are persistent in so much of what we see around modeling.
I love all of these and especially the logic model repair shop. I find your simple way of conveying stuff so helpful and effective. The idea of logical thinking has really landed in my organisation at the moment and we have huge demand to input into designing projects and programmes. Thanks Chris – all of this is much appreciated!