Evaluation

a few weeks ago, I received a note from one of our funders asking if PTP used any pre and post program evaluation tools in our training program.

I freaked out.

I think I freaked out because the culture of non-profits trains us to be afraid of funders asking about evaluation. After freaking out, I wrote up what we did, and am posting that (with some small edits for clarity) here.

I don't really know why we haven't written about this before, since evaluation has been woven into our program design and thinking from the very beginning. I write this up not because I think we have the answers or are “doing it right” in every way possible. Instead, I write this up because I think that in many ways we are doing a good job, and because I want to engage more people in the question of how we measure and evaluate the impact of capacity building programs in general.

---

From: Arif Mamdani
To: one of our funders
Subject: PTP COaTI Evaluation Tools

We use a series of evaluation tools as part of COaTI and are currently in the middle of a set of interviews with previous COaTI participants to try to assess the longer term impact of the program.

Through doing the COaTI program, we’ve learned that in general, when we first talk with groups about building their ability to use technology effectively, they tend to focus on lack of concrete skills. While this is definitely a challenge, our experience also tells us that groups often have weaknesses in other areas that hinder their ability to use technology to its fullest extent. These areas include factors like understanding of what is possible with particular technologies, capacity for training, access to quality technical assistance, and overall vision for how technology can complement and enhance their on-the-ground organizing work.

COaTI is largely designed to address the latter factors. Measuring progress with respect to these factors is in many cases much “fuzzier” than assessing whether someone has become more proficient with something like Microsoft Word, so we’ve developed a set of tools and in-session processes to help us gain a fuller picture of our impact. Finally, because many of these factors take time to change, we have tried to approach components of our evaluation process as an ongoing collaboration with participants, hence the interviews we’re currently engaged in.

So, here’s a summary of how we evaluate COaTI referencing the attached documents where applicable:

1. We try to have a conversation with every participating organization to get a sense of their “most pressing” technology capacity needs, and to encourage them to start thinking about their personal and organizational goals for COaTI
2. At the first COaTI session, we place participants into predetermined pairs and ask them to interview each other using a set of (suggested) questions that probe the individual’s and organization’s goals for participating in COaTI. In the setup for the interviews, we strongly encourage participants to use their buddies much like the “swim buddy” concept from summer camp – as a person to help them stick to their goals, a resource for ideas on how to maximize what they’re getting out of COaTI, and as a source of support in the frustration that can arise when tackling the issues that COaTI raises. We invite participants to report back to us on the results of the interviews. The set of questions we provide participants is here.
3. At the end of every day, we conduct an oral and written evaluation of the day. I’ve attached the written evaluation form we use.
4. Between COaTI sessions, we try to contact each group at least once to check-in with them on the progress of their COaTI tech projects, and see where and how they are making use of the material we’ve covered up to that point.
5. At the conclusion of the entire COaTI, we ask participants to fill out a longer evaluation designed to look at the impact of the entire program. I’ve attached the full program evaluation form (part one, part two).
6. Finally, this year, with three COaTIs completed, we are conducting a series of interviews aimed at understanding what aspects of COaTI had lasting impact on participants. Attached is the set of questions we’re using for these interviews.

So, evaluation is definitely part of what we do. Having said that, we always think we can do a better job evaluating our work. I’m curious about any questions, comments, or feedback you have about the process I’ve described here. Are there other organizations doing work similar to PTP who have evaluation models that you find particularly interesting? What types of tools have you seen used that have produced useful evaluation reports?

That covers the “standard” evaluation work that we do. We’re always refining and improving it, and I certainly see ways that we could develop a pre-COaTI evaluation that would provide us some comparison information for the final evaluation. Beyond evaluating the factors I’ve already listed, we’ve been doing a lot of thinking and exploring lately around a factor that we’re loosely calling an organization’s culture of learning. What we’re starting to see is that often, the biggest indicator of COaTI’s impact on an organization is the degree to which participants have real, concrete, internalized goals for their participation in the program, and their degree of commitment in reaching those goals. To some degree, it is a question of how actively each participant is engaged in their own learning process within COaTI. Additionally, we’re starting to see that COaTI has a greater impact in organizations that have a deep organizational commitment to learning and are open and interested in new ideas and ways of working. Of course, we have no idea how we’d go about assessing that - pre or post COaTI. As I said, this is an area of evaluation that we are trying to get our arms around – we’re not yet at a point where we even try to measure or assess this, and I’m very interested in thoughts you have in this area.

---

So, that's pretty much what I sent out, and I'd love to know what you think/do/wonder about when you evaluate capacity building work. Also, now that you know what we do, I'm curious about other things that you think we should consider adding to our evaluation process?