PTP Blog

I picked up this article by Eugene Eric Kim via Jon Stahl’s blog.

In it, Eugene writes about a training experience in which he clearly sees the impact of differences in technology understanding - what he calls the technology understanding gap:

The following day, I co-led a session on this topic with AngusParker. Two of the participants were dealing with the specific challenge of connecting members of a national network of leaders in reproductive health, so we used that as a case study. We decided to use Clay’s contention to frame the problem, resulting in this whiteboard:

What do you notice about this picture? Obviously, the Tools column is completely empty. That’s a dead giveaway that I’m facilitating this discussion. (That and the horrific handwriting.) Figure out the basics first. Don’t let the question about technology drive the discussion. During the discussion, one of the participants asked, “What tools can we use?” I responded, “Let’s not worry about that now.” So we kept talking and talking, and I noticed the two non-technical participants in the group squirming like crazy. So I stopped, noticed how gaping the Tools column looked, and said, “You’re uncomfortable about not having discussed the tools, aren’t you.” She nodded. “Don’t worry about it,” I responded. “The tools part will be easy, once we figure everything else out.” “Easy for you, maybe,” she said. “You already know what goes there.” That was not quite true, but I got her point, and the force of it struck me so hard, I had to stop for a moment. I looked at the gap, and I saw possibilities. She looked at the gap, and she saw a void. That was upsetting for her. It made it hard for her to think about the other aspects of the problem. It made me realize how much I take my technology literacy for granted. But it also created an opportunity to discuss how easily we are sidetracked by technology. “Tool” does not have to mean software, and making that assumption prevents us from exploring other viable, possibly better solutions.

Jon follows up with this observation:

often, people who are less technically literate think that if they only fill in the right answer in that middle “Tools” column, that their problems will all be solved. When, really, it is more important to get the Promise and the Bargain right. I like to call this pattern “magical tool thinking.” It results in a lot of wasted time and effort trying to identify that magical, right tool — effort which should go into thinking about process, objectives and how to sustain the non-technological parts of the organizing effort.

I couldn’t agree more - with both of them. In fact, it is this “Technology Understanding Gap” and the its impacts that Jon notes that we address with our Community Organizing and Technology Institute.

Our goal with COaTI - and really all of our programs - is to help organizers understand what’s possible with technology so that organizers can put the “tools” in their proper context and focus on the goals and strategy that should drive any technology’s use.

a few weeks ago, I received a note from one of our funders asking if PTP used any pre and post program evaluation tools in our training program.

I freaked out.

I think I freaked out because the culture of non-profits trains us to be afraid of funders asking about evaluation. After freaking out, I wrote up what we did, and am posting that (with some small edits for clarity) here.

I don’t really know why we haven’t written about this before, since evaluation has been woven into our program design and thinking from the very beginning. I write this up not because I think we have the answers or are “doing it right” in every way possible. Instead, I write this up because I think that in many ways we are doing a good job, and because I want to engage more people in the question of how we measure and evaluate the impact of capacity building programs in general.

---

From: Arif Mamdani
To: one of our funders
Subject: PTP COaTI Evaluation Tools

We use a series of evaluation tools as part of COaTI and are currently in the middle of a set of interviews with previous COaTI participants to try to assess the longer term impact of the program.

Through doing the COaTI program, we’ve learned that in general, when we first talk with groups about building their ability to use technology effectively, they tend to focus on lack of concrete skills. While this is definitely a challenge, our experience also tells us that groups often have weaknesses in other areas that hinder their ability to use technology to its fullest extent. These areas include factors like understanding of what is possible with particular technologies, capacity for training, access to quality technical assistance, and overall vision for how technology can complement and enhance their on-the-ground organizing work.

COaTI is largely designed to address the latter factors. Measuring progress with respect to these factors is in many cases much “fuzzier” than assessing whether someone has become more proficient with something like Microsoft Word, so we’ve developed a set of tools and in-session processes to help us gain a fuller picture of our impact. Finally, because many of these factors take time to change, we have tried to approach components of our evaluation process as an ongoing collaboration with participants, hence the interviews we’re currently engaged in.

So, here’s a summary of how we evaluate COaTI referencing the attached documents where applicable:

1. We try to have a conversation with every participating organization to get a sense of their “most pressing” technology capacity needs, and to encourage them to start thinking about their personal and organizational goals for COaTI
2. At the first COaTI session, we place participants into predetermined pairs and ask them to interview each other using a set of (suggested) questions that probe the individual’s and organization’s goals for participating in COaTI. In the setup for the interviews, we strongly encourage participants to use their buddies much like the “swim buddy” concept from summer camp – as a person to help them stick to their goals, a resource for ideas on how to maximize what they’re getting out of COaTI, and as a source of support in the frustration that can arise when tackling the issues that COaTI raises. We invite participants to report back to us on the results of the interviews. The set of questions we provide participants is here.
3. At the end of every day, we conduct an oral and written evaluation of the day. I’ve attached the written evaluation form we use.
4. Between COaTI sessions, we try to contact each group at least once to check-in with them on the progress of their COaTI tech projects, and see where and how they are making use of the material we’ve covered up to that point.
5. At the conclusion of the entire COaTI, we ask participants to fill out a longer evaluation designed to look at the impact of the entire program. I’ve attached the full program evaluation form (part one, part two).
6. Finally, this year, with three COaTIs completed, we are conducting a series of interviews aimed at understanding what aspects of COaTI had lasting impact on participants. Attached is the set of questions we’re using for these interviews.

So, evaluation is definitely part of what we do. Having said that, we always think we can do a better job evaluating our work. I’m curious about any questions, comments, or feedback you have about the process I’ve described here. Are there other organizations doing work similar to PTP who have evaluation models that you find particularly interesting? What types of tools have you seen used that have produced useful evaluation reports?

That covers the “standard” evaluation work that we do. We’re always refining and improving it, and I certainly see ways that we could develop a pre-COaTI evaluation that would provide us some comparison information for the final evaluation. Beyond evaluating the factors I’ve already listed, we’ve been doing a lot of thinking and exploring lately around a factor that we’re loosely calling an organization’s culture of learning. What we’re starting to see is that often, the biggest indicator of COaTI’s impact on an organization is the degree to which participants have real, concrete, internalized goals for their participation in the program, and their degree of commitment in reaching those goals. To some degree, it is a question of how actively each participant is engaged in their own learning process within COaTI. Additionally, we’re starting to see that COaTI has a greater impact in organizations that have a deep organizational commitment to learning and are open and interested in new ideas and ways of working. Of course, we have no idea how we’d go about assessing that - pre or post COaTI. As I said, this is an area of evaluation that we are trying to get our arms around – we’re not yet at a point where we even try to measure or assess this, and I’m very interested in thoughts you have in this area.

---

So, that’s pretty much what I sent out, and I’d love to know what you think/do/wonder about when you evaluate capacity building work. Also, now that you know what we do, I’m curious about other things that you think we should consider adding to our evaluation process?

well, not so much about blogs themselves, but more about how to talk to people about blogs.

Specifically, this musing is a request for your thoughts about how you’d go about designing a session for a training program for community organizers about the blogosphere, how it works, its potential value for community organizing groups, and how to engage bloggers around the issues community organizers are addressing.

I’ve been working on this idea for the past few weeks, and will start to get some ideas up here in the coming days, but I wanted to get this idea out in the world before the weekend in hopes that some of you are thinking about this very thing right now and want to share your thoughts.

Weekends tend to be good times for me to think about things like this, so if you’re inclined, please share your thoughts and help start this conversation.

Thanks to Jan Adams for the picture.

We wrapped up the first session of our in-progress COaTI on Saturday, and though the curriculum still isn’t where I want it to be, we’re steadily getting better.

One challenge that’s been on my mind a lot over the last few days is that we try to create a certain culture at COaTI - largely through a series of warm-up sessions that are designed to help foster a sense of safety, a sense of not being alone with technology challenges, and an understanding that peer knowledge is at least as valuable, if not more valuable than what the techies know. This presents a challenge because it takes time, and I think gets frustrating for some folks participating.

Right now, I’m not sure if and how to change that - shorten the “warm-up” maybe, or be more clear up front about the flow of all the sessions so that we’re clear about what we’re doing and why we’re doing.

We do try to do that at the beginning of every session, but I’m thinking that if we can really tighten up how to introduce the whole program at the beginning, things will make more sense to folks who are participating.

this will make sense after you read the first post about COaTI recipes.

It just occurred to me that it would be really cool if I could define “nutritional requirements” for a COaTI.  Think the food pyramid, but more like this:  1 serving of data merge skills, 2 servings of peer to peer story sharing, 2 servings of hands-on practice with queries, etc.  Then, I’d be able to indicate on each recipe what “nutritional requirements” it fulfils, and the menu function of the software would help manage and keep track of those requirements ensuring that each COaTI met its nutritional requirements.

This probably all sounds a bit crazy, but if you’ve ever managed a large curriculum with multiple goals, seriously give this idea some thought.  I really don’t think its that far off what I do when I design and manage the COaTI curriculum… at least not far off what I’d like to be able to do.

this morning I was thinking about the upcoming COaTI and the work that I’m doing now to try to weave together the different curriculum strands into clearer themes running throughout the three sessions.  All the elements are there, though as always some need more work to get them to a place where I’m happy.

I’m a very visual person, and I love notecards, so I was thinking about writing out a shot description of each session on a notecard and arranging them all on a big table so I could see the whole program and have the ability to move things around to see how the flow worked.  As I was imagining this, my brain thought “hey, this is kind of like having a box of recipes on notecards.”  Followed shortly by “hey, I wonder if I could use recipe software to manage COaTI curriculum.” 

It would work something like this: basically, each session would be a recipe.  Every recipe has a title (session title), description, ingredients (materials needed), and instructions for cooking (methodology).  When its time to assemble the facilitation plan, I’d just select the recipes I wanted in the order I wanted them, and the software would produce:

1. the ”menu” - a list of the sessions and a description of each which would be agenda we send to participants.
2. the shopping list - the materials we need to assemble for the COaTI. 
3. The list of recipes - in this case the facilitation plan for that COaTI.

I think it just might work.  What do you think?

requesting help via the lazyweb again:

We’re gearing up for another COaTI and are looking for a retreat-ish facility in either California or New Mexico.

What we need:

1. easy access to airports
2. highspeed internet access, wifi a plus but we can bring our access point
3. ability to deal with vegetarians
4. one large meeting room plus two breakout rooms
5. progressive politics friendly

any tips would be greatly appreciated

so, we’re spinning up for another COaTI program launching in a few months and concluding sometime next year from the looks of it.

COaTI is one of PTP’s core training programs, and its a multi-session, multi-day extravaganza.  When we run COaTI, we bring together 15 organizations, as you can imagine, scheduling is. . . tricky.

Last year we started a decidedly low-tech system for scheduling that A) seems to be working, and B) is actually somewhat less work than calling back and forth between 15 organizations.

Here’s what we do: 

1. I print out calendars for a 6 to 9 month span of time during which we’d like to schedule the COaTI sessions.
2. I PDF that calendar and send it out to the participants, with instructions to cross off days on which they can’t attend a training, and a request to send it back to me.
3. Once I’ve got all the calendars, I merge the details back into a single calendar (using Outlook/iCal, recently Google Calendars because that makes it easy to share), and use the results of that merged calendar to pick dates. 

And as I said, the system is VERY low-tech, but so far, its worked better than anything else we’ve tried.