There are all sorts of purposeful goal-driven people and organizations. Intent on following a path towards a destination that is clear in their minds. They might have even put the destination down on paper, charting the course and knowing the way forward.
But that is not all organizations, and certainly not all people.
We don’t need the expensive tools to navigate our way through the modern world. We just need the discipline to see the evidence, understand our origin, imagine our destination, and plot our course.
The Evaluation Mindset: The Role of the Evaluator
So what happens when the destination is not so clear?
When you only have a problem in need of a solution? Or when the problem is far too complex for any single intervention or initiative to resolve? Or when changing circumstances change the problem, and the tools you used in the past are clearly not the tools you will use in the future?
How do you even start?
After falling down the rabbit hole.
“Would you tell me, please, which way I ought to go from here?”
Alice’s Adventures in Wonderland
“That depends a good deal on where you want to get to,” said the Cat.
“I don’t much care where —–” said Alice.
“Then it doesn’t matter which way you go,” said the Cat.
“—— so long as I get somewhere,” Alice added as an explanation.
“Oh, you’re sure enough to do that,” said the Cat, “if you only walk long enough.”
I love that quote.
At that moment, the Cheshire Cat is simultaneously 100% correct and totally unhelpful.
Alice, for her part, isn’t really one to be told where to go. She followed her curiosity down the rabbit hole and, lost as she was, didn’t have a destination. She wasn’t longing for home, just kind of wandering forward.
Most evaluators are at least part Cheshire Cat. But pointing out that a project has no clear direction is not always helpful, even if it’s true.
All sorts of projects, initiatives, programs, and non-profits don’t have a clear direction. That’s not a flaw, it’s a feature. It also doesn’t mean that they don’t need help from evaluators, or that we don’t have a role until they find their direction.
Having direction is not the starting point.
Problems/opportunities, not actions, are the prerequisites for evaluation.
Most projects are reactions to societal problems.
Poverty, homelessness, hunger, domestic violence, illiteracy, joblessness, drug abuse, climate change, inequity, conflict, and countless other societal problems trigger the need for solutions.
Some other projects leverage opportunities. Such as a pot of money left behind by a wealthy industrialist or soon-to-be college seniors on summer break in desperate need of quality work experience.
Just like how questions precede answers, problems and opportunities precede solutions.
In an earlier post I talked about how our actions have consequences. And in a traditional summative or formative evaluation, how our actions lead to desired consequences is most certainly the focus.
But until we can imagine our destination, taking action is just wandering forward like Alice, lost and directionless.
Brainstorming your desired consequences.
Every project is an attempt to change the world, from a tiny localized change to huge sweeping global change. Being honest about the level of change you hope to bring at the front of the project can reduce a lot of headaches later.
What we want to get away from at the start is the “just do some stuff and hope for the best” mentality. Playing it safe while hoping for a huge win. It makes failure impossible, but it puts limits on success.
You start with just a simple brainstorming question.
What would the world look like if you were successful?
- Write it down on in the middle of a piece a paper and start jotting down ideas.
- Send it in an email to all of your colleagues.
- Put it up on a whiteboard during your next staff meeting.
- Create a slide with that question and screen share it during your next Zoom call.
Keep it simple at first, then start to add some “what if” qualifiers.
- What if we had to do our project on a shoestring budget?
- What if we were given a 1 million dollar grant for this project alone?
- What if a big company decided to partner with us?
- What if we had 3 new hires that could be dedicated to this project?
- What if we put a central focus on increasing equity?
- What if we increased our scope?
- What if we reduced our scope?
- What if we only had 3 months to see results?
- What if we had 5 years to see results?
- What if the entire world was told to stay and work from home for months?
- What if we could only meet digitally?
Painting the picture of your imagined destination using rubrics.
As you start processing the brainstorm, it’s helpful to develop a tangible product to help organize your thoughts. Rubrics are a really nice way to paint the picture of your imagined destination and make it measurable.
At this point you’ll be tempted to dive into the actions and consequences that will be necessary to help you change the world. But I would encourage creating a rubric of your destination first. You can always change it, but having a sense of where you are going can give you the guidance necessary to set your course.
Let’s take our brainstorming question and split it into a sequence of statements.
- This is how the world will look if we are successful.
- This is how the world will look if we are somewhat successful.
- This is how the world will look if we are not at all successful.
You’ll need a timeframe, base this is on your project. How long until you expect to see some results? How long until you think your project can be judged as a success or failure?
Now write a short paragraph under each statement using information gathered during the brainstorming activities. Try to find ways to describe your new world that would be measurable.
The evidence you use should ideally use a mix of methods, both qualitative and quantitative. It should also be realistic that you can obtain the evidence. Rubrics are practical tools.
Want to dive deeper?
- Create similar kinds of rubrics for waypoints along your journey, what types of evidence should you start to see to know you are heading in the right direction?
- Subdivide your rubric into mini rubrics, taking on different aspects of your project. This can make charting a course easier helping you connect your destination with your actions.
- Create short term or super long term rubrics. Be honest about the level of change you believe you can bring.
- Finally, use these rubrics, and allow them to evolve over time as your thinking evolves. Rubrics are really great formative evaluation tools as well as summative tools for measuring impact.
Upcoming UnWebinar with Carolyn Camman.
Register now for Eval Central’s event on Crowdcast, scheduled to go live on Wednesday, April 08, 2020 at 3:00 pm EDT.
Click here to register for the series> crowdcast.io/e/evalcentral
This coming week we have a special guest: Carolyn Camman
This week’s seed topic: What does it mean to put Developmental Evaluation into Practice?
Topic inspiration: Carolyn’s lastest blog post: Complexity and Equity
Upcoming Doing Nothing Webinar
On Friday, April 10 I’ll be spending an hour doing nothing starting at 11 AM Eastern time.
Want to join me? You can register to do so here: https://www.crowdcast.io/e/doingnothing
Leave a Reply