The Evaluation Mindset: Evaluation in a Crisis

This post is a day late as recent events hit me with blogger’s block.

I didn’t want to just write another post, seemingly oblivious to the very real global pandemic that is on everyone’s mind. But I didn’t just want to add to the COVID-19 noise.

But the more I thought about it, the more I started to consider all of the super useful ways evaluators can contribute in times of crisis.

This post is a reflection of that brainstorm. It is what I see as evaluator roles during a crisis. How we can use our evaluation expertise and skills to support our society in an unprecedented time.

It is not comprehensive, after you read it, I would love to hear your thoughts and additions.

Remembering to Breathe

Breathe in, breathe out.

When you breathe in, you bring oxygen into your body.

When you breathe out, you expel carbon dioxide.

Hold your breath and your body doesn’t get the amount of oxygen it needs to work. This is particularly challenging when physically active and your body needs more oxygen.

On the other end, breathing in too much oxygen can also be bad. Hyperventilation causes our oxygen levels to go up and carbon dioxide to go down. But if the carbon dioxide goes down too far we suffer, as we have nothing to breath out.

Healthy breathing is about balance.

Evaluation is the oxygen that powers decision making. Too little and we are likely to make poor decisions. And when faced with big challenges, we need more than usual.

Too much evaluation without action leads to hyperventilation. Analysis paralysis.

As an evaluator, it is your responsibility to keep the breathing steady.

Informing decision makers

Dr. Bicknell believed “public health is much more dangerous than medicine. We deal with populations, doctors deal mainly with individuals. So bad public health professionals are far more dangerous than bad doctors. We can kill more people with less accountability.”

From Rationing, Data, and the Ethics of our Decisions by Amanda Makulec

During times of crisis decisions must be made, with or without data.

Acting quickly, with the right decisions, can save lives.

Acting quickly, with the wrong decisions, can cost lives.

Most of the time decision making isn’t about yes or no, black or white, and wrong or right. It’s about making a choice at the time, with the information you have at hand.

There are decision makers who stay uninformed by choice. Deciding to follow their gut or only listen to others who are similarly uninformed.

But there are others who prefer to be informed. To show up with at least a sense of the decision that seems most supported by the information we have at hand. It isn’t always going to be the right choice, but it is less likely to be the wrong choice.

Informing those who wish to be informed, is also with the domain of evaluation.

Causality without control groups

Contrary to popular opinion, there are actually a lot of options [to measure attribution]. They range from the high-powered ones to some fairly low-tech common-sense options you can use even in small-scale community projects, and even if ALL of your evidence is qualitative. Yes, really!

From Understanding Causes by Jane Davidson

There is not always a control group.

When the situation at hand is complex and emerging, there might not be a proper comparison group. But that doesn’t mean we can’t determine how certain actions lead to desired consequences.

The often touted “gold standard” RCT methodology is not the only evaluation method capable of determining if something works. Especially for emerging social responses to crises.

Creating feedback loops, opportunities to understand and learn from our successes and mistakes, are critical to fueling rapid improvement.

As an evaluator, you have a role in the iterative process of rapid response.

Speaking for the vulnerable

Social vulnerability refers to the resilience of communities when confronted by external stresses on human health, stresses such as natural or human-caused disasters, or disease outbreaks. 

From the CDC’s Social Vulnerability Index

Vulnerable populations need advocates even at the best of times.

But when the worst happens, the most vulnerable populations are likely to be most at risk. Vulnerability amplified.

When all the restaurants close their doors, a tip-reliant single-mother already living paycheck to paycheck, can get left without any options. Her child who might have been getting most of her meals at school, is now sheltering at home without that benefit.

When decisions are made to support the greater good, it is right to ask, “just what is that greater good?”

“Who doesn’t have a seat at the table?”

“Who is likely to see the most negative consequences from our actions?”

Out of mind, out of sight. As an evaluator, you can use your talents to help keep those with the highest needs stay in mind and in sight.

Not just an afterthought

Developmental Evaluation (DE) is an evaluation approach that can assist social innovators develop social change initiatives in complex or uncertain environments.

Better Evaluation’s Overview of Developmental Evaluation

Even among the most data savvy of organizations, I think there is still a prevailing thought on the timing of evaluation. That it’s something you plan at the beginning, and do at the end.

But the role an evaluator, as a guide, is one that can be useful during every stage of a program or activity. It’s not just about lessons learned, but informing change.

Our methods are not simply about saying whether or not a program worked. They are not simply about improving designs. Our methods are also useful in discovering the right way to move forward.

In challenging times, we need step up.

Evaluation is an anytime activity.

Staying true to your values

Everyday narratives that continue to marginalize, minimize, and disrespect people of color and those with less privilege could be replaced with ones that do not demonize and place blame on the individual. They could instead lift up the historical, contextual, and powerful dynamics that create and sustain oppression and shed light on the strategies and solutions which can shift the “rules of the game” so that equity is achievable.

From the Equitable Evaluation Initiative

It is easy to live our values when everything is going along swimmingly. When the economy is solid and the uncertainty level is low.

As evaluators we have the responsibility to hold our organizations accountable for core values, especially when times are tough.

***

Facilitating Online Meetings

Let’s be honest. A poorly designed run meeting is a waste of time, energy and resources. This is true face to face and online.

From So You Want to Host a Web Meeting? by Nancy White, with Pete Cranston, Susan Stewart and Bonnie Koenig

The guide above is from a few years ago. But it’s well written and full of useful insight on hosting effective web meetings.

Wherever you are, stay safe. Flatten the curve. Lookout for your neighbors.

We’ll get through this.