Which Charity Will Do the Most Good with Your Donation? This Simple Tool Can Tell You.

It’s a thornier question than you might think, explains Dean Karlan, a professor of economics and finance at the Kellogg School.

Karlan says that many organizations simply report the outcomes of their programs. “It’s a job-training program and they say, ‘We’ve trained a thousand people and 950 of them got jobs,’” he says.

But such metrics have a fatal flaw: they fail to consider what might have happened without that program. “Would these people have gotten the jobs anyhow?” Karlan asks. “Did you actually change the world in any way? It just doesn’t tell you that. It’s a deeply flawed method and could be leading us really far from the right answer.”

Researchers who study public or nonprofit social programs can avoid this problem by running randomized controlled trials of the programs or using sophisticated statistical techniques. But such evaluations often require expertise, resources, and scale that few nonprofits can muster. Furthermore, some organizations (scholarship charities, for example) may not see the full impact of their work for decades yet need to estimate their effectiveness—and raise money—right away.

So what if there was a way to evaluate the impact of an organization not by conducting an entirely new study, but by drawing on rigorous research that was already done?

This idea is what motivated Karlan and colleagues to develop ImpactMatters, a new interactive platform that allows prospective donors to assess the effectiveness of more than 1,000 nonprofits across the United States.

Cofounded by Karlan and Elijah Goldberg, now its executive director, ImpactMatters draws upon existing research to estimate an organization’s impact using certain key data. By plugging data from the organization (most of which is publicly available) into mathematical models based on previous peer-reviewed research, the platform spits out a simple one- to five-star rating of that organization’s net impact per dollar.

Karlan believes that the new platform has the capacity to improve how donors make decisions about their contributions—and to help organizations learn more about their peers.

“Not every organization can run a randomized controlled trial,” he says. “But that doesn’t mean we should throw up our hands and say, ‘We have nothing to say about it.’ So, let’s take our best shot.”

Letting Donors Decide What Matters Most to Them

The idea for ImpactMatters originated in 2014, when Goldberg was a student in Karlan’s course on effective philanthropy at Yale, where Karlan used to teach. At the time there were two major platforms that let prospective donors evaluate organizations: Charity Navigator and GiveWell.

Charity Navigator rates thousands of organizations based on metrics of financial health and accountability, including the portion of donated dollars that go towards overhead expenses. However, while this may be useful for weeding out the tiny portion of organizations engaging in fraud or egregiously wasting money, Karlan says, “there’s no reason to assert that overhead is positively correlated with impact, and in fact it may even be an indicator of being a more thoughtful, rigorous organization.”

The other ratings platform, GiveWell, draws upon careful evaluations to pinpoint the organizations across the world that are doing the most good, measured primarily in terms of human lives saved per dollar spent. As a result, GiveWell only rates organizations operating in the developing world, where they can fight malaria or engage in deworming at miniscule costs.

Karlan says that this utilitarian approach makes sense if you simply want to do the most possible good in the abstract, without regard for where, or to which causes, your money is going.

“The reality, though, is that for a lot of donors, that’s just not how they think about things,” he says. “They’re not sitting there saying, ‘I have no preferences. I don’t care whether it’s education or health. I don’t care if it’s Arkansas or Iowa or Niger, I just want to get the biggest bang for the buck.’ A lot of people say, ‘No, I grew up in Cleveland and I care about the issues affecting my local community.’”

Discussing these problems in the course, Goldberg and Karlan bemoaned that there was no tool that could apply the rigor of GiveWell at the scale of Charity Navigator.

So they decided to create one.

Rigorous Nonprofit Evaluations at Scale

When ImpactMatters launched in 2015, it focused on conducting what it called impact audits. Karlan’s team would spend three months intensively examining one organization at a time—meeting with its staff and diving deeply into the organization’s data—and would then produce detailed reports summarizing their conclusions.

However, it soon became clear that this approach was not scalable—nor was it delivering results that prospective donors could easily utilize. “There was no critical mass of people coming to our website just to download these reports,” Karlan recalls.

So over the next several years, Karlan and Goldberg evolved the platform into something that could rigorously evaluate additional organizations more efficiently and provide a simple takeaway donors could use. Eventually, ImpactMatters landed on the approach it uses today.

For a given nonprofit, ImpactMatters first tries to understand the long-term social impact that the program is trying to have. Tree-planting organizations seek to offset carbon emissions, for example, while college-scholarship foundations seek to improve recipients’ lifetime earnings.

The platform also identifies the intermediate steps that are required to achieve that final impact—what’s known as the organization’s “theory of change.” Scholarships, for example, are intended to increase one’s odds of attending and graduating from college, which can lead to a higher-paying job, which in turns leads to greater lifetime earnings.

ImpactMatters estimates the effect of each step in an organization’s theory of change by drawing on research and reliable data. It then builds a unique impact model to estimate how much each organization improves outcomes over the counterfactual (i.e., if the organization did not exist), based on the specific programs undertaken by the nonprofit.

For instance, to evaluate a scholarship organization that provides funding to undergraduates on the basis of both financial need and academic merit, Karlan and colleagues might draw upon a previous study of a similar scholarship that was geared towards similar students.

Suppose that the earlier study had found that the scholarship increased graduation rates by eight percent compared to the counterfactual, and that that scholarship covered 100 percent of tuition. If the scholarship organization now being analyzed covers 50 percent of total tuition, then ImpactMatters will scale the eight percent figure downward (according to a mathematical formula) to estimate the organization’s impact on graduation rates.

“And then the next step is we look at, well, what does the social-science literature tell us about the increase in expected earnings as a result of going to college?” Karlan says.

Combining those outcomes and scaling for the number of scholarships the organization provides, ImpactMatters produces an estimate of the organization’s average total impact in dollars.

It then calculates the total costs undertaken by the organization, using IRS filings as well as financial reports from the organizations. From the impact estimate and the total expenses, it can deduce the organization’s impact per dollar.

Finally, ImpactMatters uses this cost-effectiveness measurement to give each nonprofit a rating of one to five stars.

“We live in a world where everyone’s accustomed to ratings, and it does actually change consumer behavior,” Karlan says. “We recognized that, to get to scale, we need to reduce things down to the most usable form possible.”

A Resource for Donors—and Nonprofits, Too

Some have criticized the ImpactMatters methodology for focusing only on directly measurable outcomes and ignoring organizations that work towards structural change by engaging in advocacy or lobbying rather than providing direct services.

While Karlan points out that the vast majority of charitable organizations are not engaged in lobbying, ImpactMatters hopes to someday address this gap by evaluating advocacy organizations as well. “We’re at the early stages of exploring how to expand the platform to those organizations,” he says. But for now, the platform is focused on direct-service-delivery programs and nonprofits with more standardized intervention models.

Karlan hopes that ImpactMatters will empower donors with a specific goal in mind to direct their gifts to organizations that show real progress on that goal. He also suspects that it could spur new donations from individuals who were previously skeptical that nonprofits have much capacity for real impact. “They’ll give more because they’re now more confident that it’s actually doing good,” says Karlan.

But beyond helping donors make better decisions, Karlan is optimistic that the platform could nudge the nonprofit world towards providing more information—to researchers and to one another.

For instance, he says, if one homeless shelter realizes that another shelter nearby is delivering the same services at half the cost, that might motivate the less efficient shelter to try and learn what the other is doing right.

“The hope is that it can help push organizations to share more information, learn more from each other, and be better stewards of the resources they’re given,” says Karlan.