Evaluation and COVID-19

Social media is not a monologue. It’s a collection of voices seen (or unseen) based almost entirely on who you do or do not follow.

But the world we see is based, as it always has been, on the people that surround us. Historically this has been primarily determined by where we live, and the communities where we interact. But increasingly, what we see it is based at least somewhat on who we choose to follow in the digital realm. And who we choose to invite into our own conversations.

I am an on and off lurker* of the American Evaluation Association’s Evaltalk listserv and recently stumbled into this comment by Doug Fraser.

It raises the same question Bill Fear has tried to raise a couple of times, namely: why, at this of all times, when the world faces the biggest and most vital evaluation challenge in its recent history, is the evaluation profession missing in action?

I don’t see it that way.

I see evaluators across the globe stepping up. They are using their approaches, methods, and other tools in their toolkits to make contributions. And they are using their voices to share their perspectives and guidance across the globe.

In this post I wanted to create a compilation of some of those viewpoints. And as I collected, I found that I could have kept going and going.

If you have read (or written) something recently that inspired you, please share in the comments.

*EvalTalk has a long history, but listserv technology has a way of elevating the perspectives of those members who prefer shouting to discourse. Subsequently, I find many of the conversations lacking sufficient diversity in both participation and perspective. But I digress.

When Was There Certainty?

The pandemic has laid bare that which many of us (meaning people of the global majority that live in the United States) have always known: this country is designed to maintain a power dynamic that privileges white male power and wealth at the sacrifice of most everything else.

Jara Dean-Coffey in A note from EEI’s Director: “I have feelings.”

On May 13, at 12PM Eastern/3PM Pacific, Jara will be joining me for an Eval Central UnWebinar to talk about being of service when you are not essential. Join us!

The Lines Between Work and Life and Life and Work

The resources designed to help us adapt to Covid-19 don’t match up with our lives right now. Every day, I sit down at my kitchen table — the same table where I (used to) host dinners and put together puzzles — in front of a make-shift workstation where I do my job. My living space is also my workspace. We’re managing a whole new definition of work-life balance right now and it turns out work is part of our lives.

Alissa Marchant in 18 Resources helping me in work and life with Covid-19

Don’t Overthink, Just Do

@timbidey/@traversepeople Tim is an experienced qualitative researcher with a passion for helping charities explore what works (and what doesn’t) and why to inform their project design and practice.

It’s clear that many of the voluntary and community sector organisations that Traverse works with are struggling at the moment in the UK. Charities have had to adapt to continue delivering frontline services or develop new ones to meet emerging needs in a world of new public health restrictions – all amid a catastrophic loss of funding and, in some cases, lack of staff where people have been furloughed.

In some cases, evaluation has fallen fast down the list of priorities – but it’s important to remember that ‘evaluation’ in itself is not a homogenous practice. Sure, now is not the right moment for continued impact evaluation of multi-year programmes, but evaluation has many faces and need not be so traditional or comprehensive.

My advice to organisations has been to keep it simple. Evaluation is so often seen as a ‘mystifying practice’, but now, more than ever, it’s better to collect something than not collect anything through fear of being seen as unsystematic.

After all, evaluation’s value lies in its utility – it needs to serve the information needs of its users. Right now, the needs of voluntary and community sector organisations demand real-time data to inform weekly decisions about delivery, rather than demonstrating the differences that they’ve made for funders or members of the public. 

Evaluation for this purpose does not need to be theory-based with perfectly rounded edges, it just needs to capture ‘good enough’ data about the essentials on a regular basis. For new or developed services these might include: what is the problem? Who is affected, how? Who are we reaching or not reaching – why? What do people think? What, if any differences are we making? What do we need to do to improve?

A sense of intended outcomes, basic monitoring data, simple service user feedback or even reflective, anecdotal data from staff can all provide ‘good enough’ insights into these questions for the situation at hand – so long as people remain honest with themselves (and others) about how insights were generated and what limitations sit behind them. 

So don’t overthink, just do.

Tim Bidey of Traverse is part of the FreshSpectrum Panel of Experts. These were his words.

Becoming Developmental Evaluators

All evaluators must now become developmental evaluators, capable of adapting to complex dynamics systems, preparing for the unknown, for uncertainties, turbulence, lack of control, nonlinearities, and for emergence of the unexpected. This is the current context around the world in general and this is the world in which evaluation will exist for the foreseeable future.

Michael Quinn Patton in the Evaluation Implications of the Coronavirus Global Health Pandemic Emergency

Because Our Decisions Have Consequences

 Some of us want to do everything we can to stop the spread of the pandemic and minimize the overall harm it will cause. Others of us are more concerned with managing the indirect effects of the crisis on communities or causes we care about. Still others of us are just trying to figure out the role we can and ought to play. All of us, though, can benefit from approaching these challenges with thoughtfulness and rigor.

Ian David Moss in Deciding Well in Tumultuous Times

Changing Definitions

But now, with the uncertainty of what visitation will look like over the coming months and potentially years as museums phase into reopening with limitations on visitor capacity and new social distancing measures, I wonder what does a “representative sample” mean now?  

Katie Chandler in Sampling: What does “representative” mean during and after coronavirus?

Who is Afraid of Rigor?

Bio: @b3consults/www.b3consults.com  Betsy brings all the tools of data & program evaluation in harmony with the heart and intuition-led world of coaching, to increase the impact of results-driven organizations.

My gut reaction, though I’m scared to put it out there, is two fold.  One, what about the organizations that were ready for the pandemic? Two, why shouldn’t we see a call for increased rigor?

To the first: I believe some organizations planned for something along the lines of a global pandemic.  As early as 2012, some organizations were making the case that a COVID-19 like event was coming.  The crisis was foreseen, and some organizations were ready for it. 

To the second: I’ve seen implied that we should be more adaptive and iterative, in a sense softening the rigor as organizations lean into change and adapt.  
But what if COVID-19 isn’t a one-time event (and most experts will tell you it isn’t)? And what if our ability to measure, with a high degree of rigor, how organizations served critical populations during this time will be crucial to inform what we need to do next time? 

I’m an unlikely architect of highly rigorous evaluation design, I tend to focus in the PrEvaluation space. However, when I think of my clients, the ones that know that their ability to rise up and serve critical populations well NOW will prevent further exacerbating equity issues in the future, I want to hear the kind of evaluation for which they would strive.  And I would hesitate to assume they want less rigor, or that they are afraid of rigor in a crisis. 

Ask yourself: what if they crave rigor right now?

Betsy Block of B3 Consults is part of the FreshSpectrum Panel of Experts. These were her words.

Responsible COVID-19 Data Visualization

Amanda said, “There are different points in which we make decisions about how and what we visualize, and then how we publish and share. What are we creating and doing more for our own exploration and understanding? And what are doing so that we can share it with the public to help others make sense of information?”

From Ann K Emery’s interview with Amanda Makulec in Visualizing COVID-19 Data Responsibly: An Interview with Amanda Makulec

Evaluation Contingency Plan

We’re often providing funders an evaluation plan that includes best case scenario and prioritize in-person interactions. Rarely do plans require the evaluation team to offer contingency options. As many of us are now tailoring methods to respond to social distancing and travel recommendations, we’re switching to virtual interviews and other methods of web-based data collection. Building contingencies into future evaluation plans will leave us better prepared to pivot and could save time and resources on creating post-hoc plans.

Martena Reed in Reflex or Reflection: Three Lessons for Evaluators Amid COVID-19

Changing Your Data Strategy

Consider what sort of data collection activities are going to help you use your data and collect data that is useful.

Michelle Molina in her Video on Nonprofit Data Adjustments to COVID-19

Forget Returning to Normal

So no, I don’t want us to return to normal. I want us to use this an opportunity to change, to create systems and social structures that create deep and lasting equity and a world where we work together for the common good. One can dream, right? If anything, this crisis should teach us that we are all connected. 

Ann Price in There are words I really hate right now.

The Importance of Our Work

We work to support better evaluation globally. Good evaluation helps people identify the information they need and make sense of it.  It helps inform decisions about what to do and how to improve results. Good evaluation is essential to guide the best use of resources and to ensure accountability and learning. During this pandemic and in the post-pandemic world our work is more important than ever.

Patricia Rogers in BetterEvaluation COVID-19 Statement

Compensating for a Lack of Monitoring

Una evaluación en tiempo real (RTE) está diseñada para proporcionar retroalimentación inmediata (en tiempo real) a aquellos que planifican o ejecutan un proyecto o programa, para que puedan realizar mejoras. Esta retroalimentación generalmente se proporciona durante el trabajo de campo de la evaluación, en lugar de después.

Carlos Rodriguez-Ariza in La evaluación en tiempo real en emergencias

[English Translation] A real-time evaluation (RTE) is designed to provide immediate (real-time) feedback to those who plan or execute a project or program, so they can make improvements. This feedback is generally provided during the assessment fieldwork, rather than afterwards.

Access for On-Site Data Collection

On 1 April 2020, USAID and IDEAL hosted a webinar on ‘Challenges and Strategies for Monitoring and Evaluation (M&E) in the Time of COVID-19’. The virtual meeting was attended by 500 M&E professionals.

The participants of webinar completed a poll that yielded the following results.

Ann-Murray Brown in A New Dawn: Monitoring and Evaluation during COVID-19

Before Expanding Boundaries

This is the smallest visible system (SVS) in which you can make a difference. Once you can act wisely on this system, you can expand the boundaries and scope to work larger.

Cameron D. Norman in Acting in Complex Times

Looking to the Desired Present

I’m looking to the desired present instead of a desired future. Not because I have no hopes or aspirations for the future, but because I don’t find it helpful right now to aim for something I can’t see. I don’t know what the future will hold. I don’t know where this moment goes. I’m hoping there’s a future out there so different from this one that I can’t even imagine it fully much less trace a path to it by design. All I want to do is find the best part of whatever moment I am in, and work with that.

Carolyn Camman in Entering the Clearing