PTP Blog

a few weeks ago, I received a note from one of our funders asking if PTP used any pre and post program evaluation tools in our training program.

I freaked out.

I think I freaked out because the culture of non-profits trains us to be afraid of funders asking about evaluation. After freaking out, I wrote up what we did, and am posting that (with some small edits for clarity) here.

I don’t really know why we haven’t written about this before, since evaluation has been woven into our program design and thinking from the very beginning. I write this up not because I think we have the answers or are “doing it right” in every way possible. Instead, I write this up because I think that in many ways we are doing a good job, and because I want to engage more people in the question of how we measure and evaluate the impact of capacity building programs in general.

---

From: Arif Mamdani
To: one of our funders
Subject: PTP COaTI Evaluation Tools

We use a series of evaluation tools as part of COaTI and are currently in the middle of a set of interviews with previous COaTI participants to try to assess the longer term impact of the program.

Through doing the COaTI program, we’ve learned that in general, when we first talk with groups about building their ability to use technology effectively, they tend to focus on lack of concrete skills. While this is definitely a challenge, our experience also tells us that groups often have weaknesses in other areas that hinder their ability to use technology to its fullest extent. These areas include factors like understanding of what is possible with particular technologies, capacity for training, access to quality technical assistance, and overall vision for how technology can complement and enhance their on-the-ground organizing work.

COaTI is largely designed to address the latter factors. Measuring progress with respect to these factors is in many cases much “fuzzier” than assessing whether someone has become more proficient with something like Microsoft Word, so we’ve developed a set of tools and in-session processes to help us gain a fuller picture of our impact. Finally, because many of these factors take time to change, we have tried to approach components of our evaluation process as an ongoing collaboration with participants, hence the interviews we’re currently engaged in.

So, here’s a summary of how we evaluate COaTI referencing the attached documents where applicable:

1. We try to have a conversation with every participating organization to get a sense of their “most pressing” technology capacity needs, and to encourage them to start thinking about their personal and organizational goals for COaTI
2. At the first COaTI session, we place participants into predetermined pairs and ask them to interview each other using a set of (suggested) questions that probe the individual’s and organization’s goals for participating in COaTI. In the setup for the interviews, we strongly encourage participants to use their buddies much like the “swim buddy” concept from summer camp – as a person to help them stick to their goals, a resource for ideas on how to maximize what they’re getting out of COaTI, and as a source of support in the frustration that can arise when tackling the issues that COaTI raises. We invite participants to report back to us on the results of the interviews. The set of questions we provide participants is here.
3. At the end of every day, we conduct an oral and written evaluation of the day. I’ve attached the written evaluation form we use.
4. Between COaTI sessions, we try to contact each group at least once to check-in with them on the progress of their COaTI tech projects, and see where and how they are making use of the material we’ve covered up to that point.
5. At the conclusion of the entire COaTI, we ask participants to fill out a longer evaluation designed to look at the impact of the entire program. I’ve attached the full program evaluation form (part one, part two).
6. Finally, this year, with three COaTIs completed, we are conducting a series of interviews aimed at understanding what aspects of COaTI had lasting impact on participants. Attached is the set of questions we’re using for these interviews.

So, evaluation is definitely part of what we do. Having said that, we always think we can do a better job evaluating our work. I’m curious about any questions, comments, or feedback you have about the process I’ve described here. Are there other organizations doing work similar to PTP who have evaluation models that you find particularly interesting? What types of tools have you seen used that have produced useful evaluation reports?

That covers the “standard” evaluation work that we do. We’re always refining and improving it, and I certainly see ways that we could develop a pre-COaTI evaluation that would provide us some comparison information for the final evaluation. Beyond evaluating the factors I’ve already listed, we’ve been doing a lot of thinking and exploring lately around a factor that we’re loosely calling an organization’s culture of learning. What we’re starting to see is that often, the biggest indicator of COaTI’s impact on an organization is the degree to which participants have real, concrete, internalized goals for their participation in the program, and their degree of commitment in reaching those goals. To some degree, it is a question of how actively each participant is engaged in their own learning process within COaTI. Additionally, we’re starting to see that COaTI has a greater impact in organizations that have a deep organizational commitment to learning and are open and interested in new ideas and ways of working. Of course, we have no idea how we’d go about assessing that - pre or post COaTI. As I said, this is an area of evaluation that we are trying to get our arms around – we’re not yet at a point where we even try to measure or assess this, and I’m very interested in thoughts you have in this area.

---

So, that’s pretty much what I sent out, and I’d love to know what you think/do/wonder about when you evaluate capacity building work. Also, now that you know what we do, I’m curious about other things that you think we should consider adding to our evaluation process?

Its a new day in philanthropy! Here’s a great example of the online social networking phenomena having a relationship to philanthropy - especially the younger generation of philanthopists. It also takes an interesting look at bringing visibility to social causes using these online tools that are used by many of us young folk. Peace

A New Generation
Reinvents Philanthropy
Blogs, Social-Networking Sites
Give 20-Somethings a Means
To Push, Fund Favorite Causes
By RACHEL EMMA SILVERMAN

August 21, 2007; Page D1
Joe Alamo didn’t set out to become a do-gooder. But late last year, when the Geneva, N.Y., Web designer was surfing on MySpace, he chanced onto the profile of Kiva.org1, a nonprofit that allows people to make zero-interest “microfinance” loans over the Internet to needy entrepreneurs in developing countries.
Soon after, Mr. Alamo not only became a lender through Kiva, but he also started a new Web site, Kivafriends.org2, devoted to Kiva enthusiasts. He also now volunteers to run Kiva’s MySpace page. “This is the first time I’ve ever gotten so involved with a charity,” says Mr. Alamo, now 30 years old.
YOUNG MONEY

• Plus: The 27-year-old founder of Change.org6 explains how he got started and how he earns a living. Read the interview7.
Young donors and volunteers, snubbing traditional appeals such as direct mail and phone calls, are satisfying their philanthropic urges on the Internet. They’re increasingly turning to blogs and social-networking Web sites, such as MySpace and Facebook, to spread the word about — and raise funds for — their favorite nonprofits and causes. They’re sending Web-based fund-raising pitches to their friends and families, encouraging them, in turn, to forward the appeals to their own contacts.
At the same time, a growing number of charities — ranging from start-ups to established names such as the Salvation Army — are launching profiles on popular social-networking sites, hoping that young people will link up to the pages. Some are also encouraging bloggers to mention the causes on their sites, raising thousands of dollars in small donations from readers.
Many of the nonprofits that have embraced social networking are themselves run by people in their 20s and 30s, who already spend a good portion of their lives online. Some of them also appeal to donors by offering them tangible results of their gifts by directly linking contributors with recipients.
Social-networking sites, for their part, are offering new tools to help attract nonprofits and contributors. In May, a social-action start-up called Project Agape launched a new program on Facebook called “Causes,” in which users can create online communities to advocate for various issues, charities and political candidates. Since then, the program has attracted more than 2.5 million Facebook users, raising some $300,000 for nonprofits and politicians, says Joe Green, 24, the project’s co-founder.
That move comes after MySpace — which already hosts thousands of nonprofits’ profiles — launched its “Impact” awards late last year, honoring individuals and nonprofit groups that have successfully used the site to make a difference. MySpace members vote on the winners, who get $10,000.
Visitors to another social-networking site, Change.org8, which also launched in May, can join “virtual foundations” of peers dedicated to specific causes, such as fighting AIDS in Africa, and raise money for the charities or political candidates that support those issues. So far, its 30,000 members have raised nearly $50,000, says founder Ben Rattray, 27. Before launching the site, Mr. Rattray had never made a charitable donation, finding charities’ traditional pitches to be “unengaging.”
Some young philanthropists are turning to “viral fund raising” — sending appeals to their network of contacts that are forwarded on to others — to maximize the power of small donations. Users of SixDegrees.org9, a program launched by nonprofit site Network for Good, can create a “charity badge” listing their favorite cause and send it out to their acquaintances. The badge keeps a running tally of how much has been raised and how many donors have contributed. Since the program was launched in January, users have created some 6,000 charity badges, raising some $740,000.
“When you’re young and starting out, it’s very difficult to take this meager paycheck you have and donate part of it to something else,” says Samantha Millman, 26, of Los Angeles, who works for a real-estate investment firm.
Several months ago, Ms. Millman created a badge on behalf of Bet Tzedek — the House of Justice, a legal-aid program in Los Angeles. “I basically blasted everyone I knew,” says Ms. Millman, raising more than $15,000 from 406 donors. “I was not only surprised by the dollar amount, but to have 400 people somehow hearing about this through word of mouth was phenomenal,” she says, adding that many of the donations were for just $10.
Bloggers are also pounding the drums for their favorite causes. Sarah D. Bunting, 34, who writes Tomato Nation, a culture and humor blog, offered to shave her head if her readers donated $30,000 to DonorsChoose.org10, a charity that allows contributors to directly purchase school supplies for needy classrooms. The funds were raised within days. (For a video of Ms. Bunting’s head being shorn last year, go to www.tinyurl.com/32svqs11.) DonorsChoose.org now features a “Blogger Challenge” on its site where bloggers compete to raise money among their readers.
Established charities, to be sure, have long had programs targeted to young donors, and many of them, especially health-related causes, have used peer-to-peer fund raising to help raise money, often in connection with races and other sporting events. But many of these charities either tend to target wealthier donors or are focused on occasional events, rather than on ongoing operations.
Some of the newer Web-based nonprofits, such as DonorsChoose and Kiva, are attractive because contributors say they allow them to connect directly with their recipients. Donors or lenders can hand over money directly to, respectively, teachers and students in urban public schools or individual entrepreneurs in developing countries, rather than sending a check that ends up with an abstract recipient.
“You can donate money to a charity, but it seems like it just goes into a pile and you never know what really goes on there,” says Mr. Alamo, the Kiva lender. “With Kiva, you just pick someone out and lend to them directly and watch what they do and how they succeed. That was the main appeal.”
Kiva, which started in the fall of 2005, has already drawn more than 89,600 lenders who have lent $10 million. Mr. Alamo’s Kivafriends.org Web site has attracted about 600 members since it was launched in March.
Some older charities are grappling with how to best take advantage of social-networking sites. The Salvation Army, for instance, has had a MySpace profile for “Red Kettle,” its online persona, since last year. But the site has only roughly 80 online “friends,” or people who have linked to it. (By contrast, Kiva has some 7,000 online friends on MySpace.)
Melissa Temme, 28 years old, a Salvation Army spokeswoman who came up with the MySpace page, says that in order to be successful on social-networking sites, charities need to spend a lot of time updating content and communicating with members, which can be difficult for a stretched nonprofit staff. “There has to be a certain level of interaction with other people and their pages,” she says.
In order to keep its online postings up-to-date, DoSomething.org12, which seeks to get young people involved in social action, enlisted an Ohio University college student. “We’re trying to use social networking as much as possible,” says Aria Finger, 24, DoSomething’s business development director. “We want to reach young people where they already are.” The charity boasts roughly 5,500 MySpace friends and more than 2,200 on Facebook.