Like many evaluators I just kind of fell into the field from a career in research (my MA is in sociology). One day I found myself in a non-profit data position doing evaluation work. Then over the years I’ve learned more and more about the field through a lot of self-initiated learning, experience and participation in the American Evaluation Association.
I believe strongly that evaluative thinking should be embedded into the core of organizations. I also believe that there are many people out there doing evaluation work that don’t currently know they are evaluators. So what about credentials?
The whole doctor and lawyer analogies get used a lot. I don’t want more people without the training thinking of themselves as doctors. Now I do want more people embedded within organizations thinking of themselves as evaluators. But if they are going to be doing high stakes work (which includes a lot of evaluation work), or selling themselves as professional evaluators, that seems different.
To put it another way. If my sink gets clogged, I can probably fix it. It’s good to know how to do these things. If I can’t fix it, maybe a handyman can or maybe I’ll ask a friend who knows a little more than I. But if I have a major plumbing issue, I’m going to call a plumber, and they are certified.
Inspired by Caroline Heider, Director General of the World Bank’s Independent Evaluation Group, in Professionalizing Evaluation: What More Is There to Say?
From my perspective, there is a range of pressures and factors at play that are nudging us, as a global evaluation community, ever closer towards professionalization.
These include pressures on improving the quality of evaluations.
- Effectiveness of Evaluation. Users of evaluation are increasingly inclined to ask questions about what difference evaluation makes, about value-for-money, and about evaluation quality. These, legitimate questions will intensify, and push us to better demonstrate the effectiveness of our work and how we influence change – the competence of evaluators has a big role to play in this.
- Delivering timely, high quality evaluations. For some time, commissioners of evaluation have struggled to get the right people for the right jobs. At least part of the drive towards professionalization is motivated by frustrations with the costs – not just in money, but also in time, reputation, and effort – associated with poor quality evaluations.
The risks to the users of evaluation findings – making ill-informed decisions and suffering the consequences – are real, and so are the risks to the evaluation profession. If evaluation cannot positively influence change – and quality of evaluation is an important determinant in that – its own relevance might be questioned.
Although I had graduate courses in research design, program evaluation, and I evaluated a statewide energy conservation program for my Master’s Paper, it was not until I had “real-life” experiences of working with consultants in designing research evaluations and critiquing them that I really felt I was qualified as a competent evaluator or programs.
I know that feeling. I’ve been a supporting player in evaluations for years now, but I still think of myself mostly in that support role.
I appreciate your thoughts here. While I believe that almost anyone can do some evaluative components and can think and act with evaluation in mind, I wholeheartedly think that training in evaluation is crucial for more intensive and sustaining evaluation work that can lead to program improvements.
Maybe we have branding issue. Instead of credentialing evaluators, we should create a “Super Awesome Evaluator” credential.
I could hear it now, “well so and so is an evaluator but Wendy Tackett is a credentialed Super Awesome Evaluator.” Of course they might already be saying that.
I think you’ve already hit it in your original post–a lot of us are relatively handy and can, say, make some repairs, renovations, improvements, etc. around our homes, especially if it isn’t high stakes. I am, for example, restoring a master bathroom in my home–a fairly major project–but, we have an existing, functional bathroom, and it isn’t urgent that I get the master bath functional right away, so I can take my time, research what I don’t know, ask for help, etc. Because it’s a major project, I have to get permits for some of the work–for that, and for the plumbing where mistakes could have major consequences, I call a *professional*. I happen to have a friend who is a plumber, who is willing to give me pointers, inspect my work, etc., but I insist on actually “hiring him” on these occasions.
Another role for professional evaluators that overlaps these two areas is *meta*evaluation. Like my plumber friend, professional evaluators can “inspect” or evaluate the evaluation work of otherwise qualified amateurs, offering tips and making sure they’ll meet requirement on those occasions when more official requirements or higher stakes are involved.
Thanks James for adding your perspective!
Thanks for referencing and quoting my blog! Wow, nice surprise! I agree that there are many people out there who exercise evaluative thinking. The more the better, as critical thinking and immediate corrective actions when needed are essential for success. But, I would argue that’s a bit different to commissioning an evaluation, where whoever commissions it has specific and normally high expectations in the reliability of the results. So even if the professionalism in evaluators wouldn’t drive them to do a great job, it’s the people who ask and pay for an evaluation and want to use it, who will push for getting high quality. The question though remains whether formal professionalization can actually guarantee such high performance. As Gene Shackman writes: we don’t have evidence to prove that.
Thanks for the comment and the additional thoughts Caroline!
An evaluation’s high quality and high performance may also look quite different in the eyes of the organizations commissioning the evaluation and in the eyes of experienced evaluators. Questions like why are they funding the evaluation, what are they looking for, and are they pushing for a specific method even if it is inappropriate for that specific context can also have an impact on the evaluations quality.
In addition to better evaluations, credentials should provide of a certain level of social proof for the evaluator.
Of course no matter the goal, supporting evidence would be a good thing 🙂
I think there may be some merit in “professionalization”/certification. But I think that is outweighed by the danger of defining what is “evaluation” too narrowly. Once you do that, then the certification criteria and all of the supporting stuff that grows up around it, including certification training offered by those who somehow are given the job of defining “evaluation”, becomes narrow. This works against the real-world need for people who can think and work flexibly in measuring results and in working with and communicating with policy makers and managers. It also creates a group of powerful gate-keepers whose incentives are squarely aligned with the status quo that they have set up.
Thanks John, nice addition to the overall conversation. I tend to worry about that stuff too.
I am like you, Chris in that I, well I won’t say fell into, but was sort of tossed into evaluation. My background is in research and although they use similar methodologies, the transition was slow and at times painful. To return to the main premise of your blog, the professionalisation of evaluation is a desirable objective but who will define it, and how will it be defined and measured. Given that evaluation has evolved overtime and continues to do so, any definition or measures would need to reflect this. And we do need to be careful as John said about being too restrictive.
Regarding the issue of quality, this is extremely important to maintaining the credibility of the discipline as a whole and building an M&E culture. One thing I have found in my experience, persons commissioning evaluations will either state or imply they want a high quality evaluation but this desire can and will change if the evaluation results are not what they wanted. For me, professionalizing evaluation is not just about the experts or practitioners but it should include an appreciation by those who commission such activities, understanding and appreciating that evaluation is a value-added exercise which, irrespective of the results provided, can only benefit entities if the information is taken into account and used for improvement and/or to build on efficiency gains.
Thanks Nicole!