Why Evaluation Research is Pretty Cool...

Despite this current climate of economic belt-tightening - and maybe even because of it - there is growing pressure on organizations to evaluate their programs. More and more, funders are asking for a comprehensive evaluation plan that produces concrete data on the process of implementation and the outcomes of the program. While federal and state grants have historically required evaluation as a part of their budgets, the emphasis has  increased in this tough economy and private foundations are not far behind. Funders want to know that their money is being used to do what was promised, that the outcomes promised were achieved, or if not, that there were legitimate reasons.

Many programs approach the notion of evaluation with the fear of being judged ("What if they find that we screwed up?"), annoyance ("This is a waste of time."), or resentment. ("Why do they have to spend so much money on evaluation instead of where it belongs, in the program?").

These reactions may be influenced by people's previous experience with evaluation or a lack of it. My own first experience with an evaluator - nearly 30 years ago - was downright disappointing. I was directing a federally-funded educational program based in state government. And our evaluator "came with" the project; that is, we had no input on his selection. He "interviewed" me a couple of times and seemed friendly enough, but I never really felt he cared about the project or really understood what we were doing, nor did he make much effort to do so. He didn't seem to have an evaluation research design, didn't ask me for input on the interview protocols he used, and I honestly have no idea who he even talked to in the process of evaluating the program. 

At the end, he generated a report that was totally useless (to us at least) and didn't really resonate with our experience of the program. Nonetheless, he concluded that we had done a good job, so we were not about to argue! We could say an independent evaluator liked the program... Sad to say, my colleagues and I just thought it was a big joke. But in reality, what a waste of public dollars, and what a missed opportunity for us to actually reflect on our program experience!

A sound evaluation is focused on identifying and supporting the strength of a program, identifying the challenges, and targeting ways to overcome the obstacles. In my view, the goal of evaluation must be to help organizational decision makers make good choices about their programs and/or policies.

All evaluation is aimed at assessing some form of transformation, whether at the micro or macro level, including understanding if all the elements are in place to achieve the desired goals. In the vernacular of sociologists, I bring a "third eye" to help solve problems in diverse organizational settings, such as foundations, nonprofits and universities.

To these "clients", I am an evaluation researcher, organizational analyst, educator, strategic planner, and increasingly a management coach. Some of the questions my colleagues and I have grappled with this year include:

* To what extent do early childhood teachers benefit from state policies aimed at improving quality in child care programs?

* How can a university-community partnership overcome differences to work together more effectively?

* What role can the arts play in improving student outcomes and school culture?

* How can a foundation-initiated leadership program strengthen the nonprofit sector?

When I do evaluation planning with clients, I ask them:

Who wants to know what for what purpose?

* In other words, who is the audience for the evaluation?

* What does the audience want to know?

* And why do they want to know it?

Usually the audience for an evaluation is both internal and external. An internal audience is the program leadership and staff who need robust information about how a program is being implemented and/or developing, so it can stay on course or improve if there are difficulties. An external audience generally refers to funders or policymakers, those who require an evaluation or will use evaluation data for their own decision-making purposes.

That said, I (and most of my evaluator colleagues) often choose to work with organizations that are trying to make a positive change in society with the clients they serve, the policies they promote, the campaigns they run and/or the advocacy efforts they promote. As an evaluator, I present myself to organizations as a "critical friend" who can think big-picture about what they are trying to accomplish and who can work closely with them to understand what is working and what needs some help. 

Evaluation can play a key role in supporting program leaders and staff to make good decisions to maximize the possibility of achieving goals, or dare I say, "success", however that is defined.... And it is also key to highlighting the real world challenges programs face that may impede program success. 

Increasingly, I am finding that funders want evaluation research to document the process of program implementation, warts and all. Those of us who run or evaluate programs have known for a long time that programs will run into obstacles when they are being implemented. Evaluation can bring a wide lens to understanding and documenting these obstacles, what it takes to overcome them, or if they simply are insurmountable.

In my next posting, I will write about how to select an independent evaluator. It is inspired by my experience and that of my evaluator colleagues who have some very wise things to say. Stay tuned!  Meanwhile, here are a few quotes that relate to evaluation research because, as I said, evaluation research is pretty cool...

Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted."  Albert Einstein

"Failure is the condiment that gives success its flavor."  Truman Capote

"My mother drew a distinction between achievement and success. She said that 'achievement is the knowledge that you have studied and worked hard and done the best that is the new. Success is being praised by others, and that's nice too, but not as important or satisfying. Always aim for achievement and forget about success..'" Helen Hayes

"First get your facts; then you can distort them at your leisure."  Mark Twain

"Great things are not done by impulse, but by a series of small things brought together." Vincent van Gogh

 "True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information." Winston Churchill

 "Each time he sees me, while all the rest go on with their old measurements and expect me to fit them." George Bernard Shaw

Friend to Groucho Marx:  "Life is difficult!"

Marx to Friend:  "Compared to what?"