Why simply defining evaluation is not enough.

So I’ve only been independent for half a year but I’ve certainly learned one thing.  If you have to spend a lot of time defining or explaining something, it’s going to be a hard sell.
Which brings me to evaluation…
Some of my most popular cartoons fit under the what is evaluation theme.  The fact that many evaluators have to spend a lot of their time explaining evaluation is also a common gripe you find in the field.
But lately, I’ve found it much easier to explain.  And I think I know why.  So I’ll share.

This is what I thought I did.

So over the last few years, I would often define myself as someone who specializes in the overlap between design and evaluation.
But here’s the problem, the more time I spent designing, the more I realized the Venn doesn’t really work here.

Design and Evaluation are Interdependent.

The reason why the Venn doesn’t work, is that what we call design and what we call evaluation are almost completely interdependent.
As they go about their day to day work, many designers end up evaluating and many evaluators end up designing.  They do not call the things they do by the same name but there is so much overlap between the fields that it is really hard to disentangle the two.
So I looked for a definition…just not for evaluation.

Defining Design

The following definition comes from graphic design icon Milton Glaser.

“Design is simply moving from an existing condition to a preferred one,” he said. “That’s it. It covers all cases. Nothing else is involved. It has nothing to do with beauty or effectiveness.”

And that was it, my mind blown.
Okay, so I’m a nerd and you’re probably thinking, “I don’t get it, why is his mind blown?”
Evaluation has everything to do with beauty, effectiveness, and preference.  It is what design is not.  And design is what evaluation is not.
But let me elaborate with some actual evaluation explanations, using the design definition as context.

First let’s visualize the design definition.

Pretty simple, right?
You’re just changing something into something else.  But let’s use it to talk evaluation.

1. Understanding the current state, that’s your needs assessment, asset identification, or your baseline studies.
2. That whole arrow thing, the change that happens.  This is why evaluators get so jazzed up about logic models, logframes, and theories of change.  You’re modeling that prospective change.
3. The preferred state thing.  That’s your outcome, the thing you are trying to make happen.  This is why we do pre-post kinds of things to refer back to the original current state.
4. For you experimental/quasi-experimental people, comparing the preferred state with the current state is sort of like comparing a treatment group with a control or counterfactual.
5. Understanding that both of these points (the current state and the preferred state) are constantly in motion and impacted by all sorts of other things leads to systems theory.  It’s also the motivation behind experimental design versus a simple pre-post.
6. The fact that the preferred state is entirely based on who defines that preferred state is why evaluators talk so much about culture.
7. Since the people doing the design often don’t know what the preferred state is or how to get there, means we have to do a lot more than summative and formative evaluation.
8. If we are evaluating for the sake of all stakeholders, not just those who pay us, then we need other methods to account for the fact that what is preferred and worthwhile is always dependent on point of view.

Evaluation serves design, good design relies on evaluation

If I have a point with this whole post, it’s this.
Evaluation is so much easier to explain when you start by defining design.  Scriven’s evaluation definition is awesome and accurate but it will always be inadequate for the purposes of explaining what it is that we do.
I look forward to hearing what you think.