The Thorny Challenge of Measuring Success in “Systems-of-Care” Networks

While AmericaServes is geared toward assisting veterans and their families, other similar service networks have popped up in the U.S. with different focuses: mental health, family and children’s services, or homelessness, for example. The general goal of service networks is to address the usually siloed nature of service delivery by integrating medical, social, and other services to make it easier for people to access—and benefit from—them. After all, multifaceted issues require multifaceted interventions.

“These new interorganizational arrangements try to get people to the right benefits programs and services they’re eligible to receive,” says Michelle Shumate, Northwestern professor of communication studies and associate faculty member of the school’s Institute for Policy Research.

“I grew up without a lot of money,” she continues, “and the people around me often didn’t know where to go for help. These systems-of-care networks are about making the process better, to provide guidance people need.”

But how well do they deliver on their mission?

That’s the broad question tackled in new research from Shumate and Karen Smilowitz, Kellogg professor of operations, along with Northwestern collaborators Zachary Gibson, Mariana Escallon-Barrios, and Joshua-Paul Miles—and four coauthors from Syracuse University. The study was funded by a grant from the Army Research Office.

They worked closely with AmericaServes to better understand the nature of service networks and how network success should be evaluated.

The researchers found that different metrics for measuring performance were correlated: directing users to the correct service provider was associated with users actually receiving services, for example. But they also found that measuring performance in these networks is highly context-dependent. High-complexity services will inevitably have lower performance across some metrics, so it’s not always helpful to focus only on which networks are top performers on these dimensions. In fact, if funders make decisions based only on, say, how quickly users are served, they may be incentivizing networks to prioritize easy-to-solve issues.

This is a new area of study, Smilowitz says. “Our study is among the first to capitalize on real-time, grounded data to understand how the work of systems of care affects their overall performance.”

Partnering with AmericaServes

The team partnered with AmericaServes for their study because it is a broad network that tackles a range of service requests, including more complicated ones.

“My husband is a veteran, and both of my grandfathers were veterans. I know the frustration of trying to get benefits and figuring out eligibility in the VA system,” Shumate says. “Even people who work in these networks have a hard time keeping track of an evolving set of programs and eligibility requirements.”

Both she and Smilowitz were impressed by AmericaServes’s advanced technology systems, which gave the researchers access to every referral and service episode in recent years. They also engaged in informal discussions with AmericaServes employees as part of the study. “By interviewing the staff at AmericaServes we could get the context behind the data and interpret it, which led to fantastic insight,” Smilowitz says.

They studied 30 days of service-episode data across all 11 AmericaServes networks—over 1,500 service episodes total—from early 2020. The researchers focused on three key measures: effectiveness, or whether a given individual seeking services actually receives some form of service; efficiency, measured as the number of days before an individual begins receiving care; and accuracy, or whether a service provider to whom an individual is referred accepts the referral, meaning they are the right provider for that person’s needs. While past service-network studies have examined effectiveness, efficiency and accuracy are newer variables of focus in this research arena.

“These metrics really help us to understand how things in this network are currently working, like understanding trade-offs in achieving efficiency, effectiveness, and accuracy,” Smilowitz says.

Complexity Matters

Overall, the results suggested the service network performed well, with 72 percent of referrals resulting in resolution of the expressed need and 88 percent of referrals accepted by the first provider in an average of just under three days.

The researchers also examined correlations among effectiveness, efficiency, and accuracy. They found that accuracy and effectiveness were positively correlated, suggesting that referrals that are accepted for care on the first attempt generally resolve the person’s request. Efficiency and effectiveness were not correlated, however, meaning faster entry into a system doesn’t necessarily result in resolution of the request.

At the same time, the researchers saw that context matters deeply—that is, expectations for effectiveness, efficiency, and accuracy must take into account the type of service in question.

For example, many of the services needed are low in complexity—food and clothing, for instance. Based on study data, such low-complexity services took an average of only 1.3 days to be addressed, with 93 percent accuracy and 78 percent efficiency. But more complicated services, like housing and legal services, saw efficiency numbers balloon to nearly 6 days and accuracy and effectiveness drop to 71 percent and 57 percent, respectively.

“You might be quick to label one particular network as being very efficient or very accurate,” Smilowitz says. “But then if you actually looked at the composition of their service load, that factors in significantly. So you can’t just say, ‘This is a good score for accuracy’ or ‘This is a good score for efficiency’ without knowing the context. That was a big ‘aha’ moment for us.”

Better Policy and Coordination

The findings here have significant policy-related and operational implications.

“There is now an increasing number of state and federal efforts to create and sustain these kinds of networks,” Shumate says. “They want to attach metrics to them, and if they rely mostly on efficiency and effectiveness, they will incentivize low-complexity services. That will incentivize ‘cream-skimming,’ where networks focus on the easiest options, like working on food instead of housing, which is a complex but critical service that a lot of the other things rely on.”

Another potential application of the findings, Smilowitz explains, is that “the work can help anticipate what kinds of needs might emerge, and when, among veterans and their families. So we can think about that kind of sequencing of services to provide, based on that.”