The Thorny Challenge of Measuring Success in “Systems-of-Care” Networks
Skip to content
Policy Operations Sep 6, 2022

The Thorny Challenge of Measuring Success in “Systems-of-Care” Networks

These networks, which help direct people to the medical and social services they need, must tread carefully in choosing which metrics to use in decision-making.

person being raised up by large hands

Riley Mann

Based on the research of

Karen Smilowitz

Michelle Shumate

and coauthors

When veterans and military families are seeking services, they often reach out to AmericaServes. The organization aims to provide one-stop shopping for military families, directing them to a comprehensive set of services, including healthcare, financial support, and employment assistance. The organization does this by partnering with public, private, and nonprofit groups to operate 11 networks nationwide that help people find and use 20 different types of services.

Add Insight
to your inbox.

While AmericaServes is geared toward assisting veterans and their families, other similar service networks have popped up in the U.S. with different focuses: mental health, family and children’s services, or homelessness, for example. The general goal of service networks is to address the usually siloed nature of service delivery by integrating medical, social, and other services to make it easier for people to access—and benefit from—them. After all, multifaceted issues require multifaceted interventions.

“These new interorganizational arrangements try to get people to the right benefits programs and services they’re eligible to receive,” says Michelle Shumate, Northwestern professor of communication studies and associate faculty member of the school’s Institute for Policy Research.

“I grew up without a lot of money,” she continues, “and the people around me often didn’t know where to go for help. These systems-of-care networks are about making the process better, to provide guidance people need.”

But how well do they deliver on their mission?

That’s the broad question tackled in new research from Shumate and Karen Smilowitz, Kellogg professor of operations, along with Northwestern collaborators Zachary Gibson, Mariana Escallon-Barrios, and Joshua-Paul Miles—and four coauthors from Syracuse University. The study was funded by a grant from the Army Research Office.

They worked closely with AmericaServes to better understand the nature of service networks and how network success should be evaluated.

The researchers found that different metrics for measuring performance were correlated: directing users to the correct service provider was associated with users actually receiving services, for example. But they also found that measuring performance in these networks is highly context-dependent. High-complexity services will inevitably have lower performance across some metrics, so it’s not always helpful to focus only on which networks are top performers on these dimensions. In fact, if funders make decisions based only on, say, how quickly users are served, they may be incentivizing networks to prioritize easy-to-solve issues.

“Even people who work in these networks have a hard time keeping track of an evolving set of programs and eligibility requirements.”

— Michelle Shumate

This is a new area of study, Smilowitz says. “Our study is among the first to capitalize on real-time, grounded data to understand how the work of systems of care affects their overall performance.”

Partnering with AmericaServes

The team partnered with AmericaServes for their study because it is a broad network that tackles a range of service requests, including more complicated ones.

“My husband is a veteran, and both of my grandfathers were veterans. I know the frustration of trying to get benefits and figuring out eligibility in the VA system,” Shumate says. “Even people who work in these networks have a hard time keeping track of an evolving set of programs and eligibility requirements.”

Both she and Smilowitz were impressed by AmericaServes’s advanced technology systems, which gave the researchers access to every referral and service episode in recent years. They also engaged in informal discussions with AmericaServes employees as part of the study. “By interviewing the staff at AmericaServes we could get the context behind the data and interpret it, which led to fantastic insight,” Smilowitz says.

They studied 30 days of service-episode data across all 11 AmericaServes networks—over 1,500 service episodes total—from early 2020. The researchers focused on three key measures: effectiveness, or whether a given individual seeking services actually receives some form of service; efficiency, measured as the number of days before an individual begins receiving care; and accuracy, or whether a service provider to whom an individual is referred accepts the referral, meaning they are the right provider for that person’s needs. While past service-network studies have examined effectiveness, efficiency and accuracy are newer variables of focus in this research arena.

“These metrics really help us to understand how things in this network are currently working, like understanding trade-offs in achieving efficiency, effectiveness, and accuracy,” Smilowitz says.

Complexity Matters

Overall, the results suggested the service network performed well, with 72 percent of referrals resulting in resolution of the expressed need and 88 percent of referrals accepted by the first provider in an average of just under three days.

The researchers also examined correlations among effectiveness, efficiency, and accuracy. They found that accuracy and effectiveness were positively correlated, suggesting that referrals that are accepted for care on the first attempt generally resolve the person’s request. Efficiency and effectiveness were not correlated, however, meaning faster entry into a system doesn’t necessarily result in resolution of the request.

At the same time, the researchers saw that context matters deeply—that is, expectations for effectiveness, efficiency, and accuracy must take into account the type of service in question.

For example, many of the services needed are low in complexity—food and clothing, for instance. Based on study data, such low-complexity services took an average of only 1.3 days to be addressed, with 93 percent accuracy and 78 percent efficiency. But more complicated services, like housing and legal services, saw efficiency numbers balloon to nearly 6 days and accuracy and effectiveness drop to 71 percent and 57 percent, respectively.

“You might be quick to label one particular network as being very efficient or very accurate,” Smilowitz says. “But then if you actually looked at the composition of their service load, that factors in significantly. So you can’t just say, ‘This is a good score for accuracy’ or ‘This is a good score for efficiency’ without knowing the context. That was a big ‘aha’ moment for us.”

Better Policy and Coordination

The findings here have significant policy-related and operational implications.

“There is now an increasing number of state and federal efforts to create and sustain these kinds of networks,” Shumate says. “They want to attach metrics to them, and if they rely mostly on efficiency and effectiveness, they will incentivize low-complexity services. That will incentivize ‘cream-skimming,’ where networks focus on the easiest options, like working on food instead of housing, which is a complex but critical service that a lot of the other things rely on.”

Another potential application of the findings, Smilowitz explains, is that “the work can help anticipate what kinds of needs might emerge, and when, among veterans and their families. So we can think about that kind of sequencing of services to provide, based on that.”

About the Writer

Sachin Waikar is a freelance writer based in Evanston, Illinois.

About the Research

Gibson, Zachary, Mariana Escallon-Barrios, Joshua-Paul Miles, Catherine Annis, Julia Carboni, Karen Smilowitz, Gilly Cantor, Nicholas Armstrong, and Michelle Shumate. 2022. “Beyond Network Effectiveness: The Case for Network Efficiency and Accuracy.” Working paper.

Read the original

Most Popular This Week
  1. Will AI Eventually Replace Doctors?
    Maybe not entirely. But the doctor–patient relationship is likely to change dramatically.
    doctors offices in small nodules
  2. 3 Tips for Reinventing Your Career After a Layoff
    It’s crucial to reassess what you want to be doing instead of jumping at the first opportunity.
    woman standing confidently
  3. What Happens to Worker Productivity after a Minimum Wage Increase?
    A pay raise boosts productivity for some—but the impact on the bottom line is more complicated.
    employees unload pallets from a truck using hand carts
  4. 6 Takeaways on Inflation and the Economy Right Now
    Are we headed into a recession? Kellogg’s Sergio Rebelo breaks down the latest trends.
    inflatable dollar sign tied down with mountains in background
  5. What Is the Purpose of a Corporation Today?
    Has anything changed in the three years since the Business Roundtable declared firms should prioritize more than shareholders?
    A city's skyscrapers interspersed with trees and rooftop gardens
  6. How to Get the Ear of Your CEO—And What to Say When You Have It
    Every interaction with the top boss is an audition for senior leadership.
    employee presents to CEO in elevator
  7. Why We Can’t All Get Away with Wearing Designer Clothes
    In certain professions, luxury goods can send the wrong signal.​
    Man wearing luxury-brand clothes walks with a cold wind behind him, chilling three people he passes.
  8. Why You Should Skip the Easy Wins and Tackle the Hard Task First
    New research shows that you and your organization lose out when you procrastinate on the difficult stuff.
    A to-do list with easy and hard tasks
  9. How Are Black–White Biracial People Perceived in Terms of Race?
    Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
    How are biracial people perceived in terms of race
  10. Which Form of Government Is Best?
    Democracies may not outlast dictatorships, but they adapt better.
    Is democracy the best form of government?
  11. When Do Open Borders Make Economic Sense?
    A new study provides a window into the logic behind various immigration policies.
    How immigration affects the economy depends on taxation and worker skills.
  12. Why Do Some People Succeed after Failing, While Others Continue to Flounder?
    A new study dispels some of the mystery behind success after failure.
    Scientists build a staircase from paper
  13. How Has Marketing Changed over the Past Half-Century?
    Phil Kotler’s groundbreaking textbook came out 55 years ago. Sixteen editions later, he and coauthor Alexander Chernev discuss how big data, social media, and purpose-driven branding are moving the field forward.
    people in 1967 and 2022 react to advertising
  14. How Old Are Successful Tech Entrepreneurs?
    A definitive new study dispels the myth of the Silicon Valley wunderkind.
    successful entrepreneurs are most often middle aged
  15. How Offering a Product for Free Can Backfire
    It seems counterintuitive, but there are times customers would rather pay a small amount than get something for free.
    people in grocery store aisle choosing cheap over free option of same product.
  16. Immigrants to the U.S. Create More Jobs than They Take
    A new study finds that immigrants are far more likely to found companies—both large and small—than native-born Americans.
    Immigrant CEO welcomes new hires
  17. College Campuses Are Becoming More Diverse. But How Much Do Students from Different Backgrounds Actually Interact?
    Increasing diversity has been a key goal, “but far less attention is paid to what happens after we get people in the door.”
    College quad with students walking away from the center
  18. How Peer Pressure Can Lead Teens to Underachieve—Even in Schools Where It’s “Cool to Be Smart”
    New research offers lessons for administrators hoping to improve student performance.
    Eager student raises hand while other student hesitates.
More in Policy