The Thorny Challenge of Measuring Success in “Systems-of-Care” Networks
Skip to content
Policy Operations Sep 6, 2022

The Thorny Challenge of Measuring Success in “Systems-of-Care” Networks

These networks, which help direct people to the medical and social services they need, must tread carefully in choosing which metrics to use in decision-making.

person being raised up by large hands

Riley Mann

Based on the research of

Karen Smilowitz

Michelle Shumate

and coauthors

When veterans and military families are seeking services, they often reach out to AmericaServes. The organization aims to provide one-stop shopping for military families, directing them to a comprehensive set of services, including healthcare, financial support, and employment assistance. The organization does this by partnering with public, private, and nonprofit groups to operate 11 networks nationwide that help people find and use 20 different types of services.

While AmericaServes is geared toward assisting veterans and their families, other similar service networks have popped up in the U.S. with different focuses: mental health, family and children’s services, or homelessness, for example. The general goal of service networks is to address the usually siloed nature of service delivery by integrating medical, social, and other services to make it easier for people to access—and benefit from—them. After all, multifaceted issues require multifaceted interventions.

“These new interorganizational arrangements try to get people to the right benefits programs and services they’re eligible to receive,” says Michelle Shumate, Northwestern professor of communication studies and associate faculty member of the school’s Institute for Policy Research.

“I grew up without a lot of money,” she continues, “and the people around me often didn’t know where to go for help. These systems-of-care networks are about making the process better, to provide guidance people need.”

But how well do they deliver on their mission?

That’s the broad question tackled in new research from Shumate and Karen Smilowitz, Kellogg professor of operations, along with Northwestern collaborators Zachary Gibson, Mariana Escallon-Barrios, and Joshua-Paul Miles—and four coauthors from Syracuse University. The study was funded by a grant from the Army Research Office.

They worked closely with AmericaServes to better understand the nature of service networks and how network success should be evaluated.

The researchers found that different metrics for measuring performance were correlated: directing users to the correct service provider was associated with users actually receiving services, for example. But they also found that measuring performance in these networks is highly context-dependent. High-complexity services will inevitably have lower performance across some metrics, so it’s not always helpful to focus only on which networks are top performers on these dimensions. In fact, if funders make decisions based only on, say, how quickly users are served, they may be incentivizing networks to prioritize easy-to-solve issues.

“Even people who work in these networks have a hard time keeping track of an evolving set of programs and eligibility requirements.”

— Michelle Shumate

This is a new area of study, Smilowitz says. “Our study is among the first to capitalize on real-time, grounded data to understand how the work of systems of care affects their overall performance.”

Partnering with AmericaServes

The team partnered with AmericaServes for their study because it is a broad network that tackles a range of service requests, including more complicated ones.

“My husband is a veteran, and both of my grandfathers were veterans. I know the frustration of trying to get benefits and figuring out eligibility in the VA system,” Shumate says. “Even people who work in these networks have a hard time keeping track of an evolving set of programs and eligibility requirements.”

Both she and Smilowitz were impressed by AmericaServes’s advanced technology systems, which gave the researchers access to every referral and service episode in recent years. They also engaged in informal discussions with AmericaServes employees as part of the study. “By interviewing the staff at AmericaServes we could get the context behind the data and interpret it, which led to fantastic insight,” Smilowitz says.

They studied 30 days of service-episode data across all 11 AmericaServes networks—over 1,500 service episodes total—from early 2020. The researchers focused on three key measures: effectiveness, or whether a given individual seeking services actually receives some form of service; efficiency, measured as the number of days before an individual begins receiving care; and accuracy, or whether a service provider to whom an individual is referred accepts the referral, meaning they are the right provider for that person’s needs. While past service-network studies have examined effectiveness, efficiency and accuracy are newer variables of focus in this research arena.

“These metrics really help us to understand how things in this network are currently working, like understanding trade-offs in achieving efficiency, effectiveness, and accuracy,” Smilowitz says.

Complexity Matters

Overall, the results suggested the service network performed well, with 72 percent of referrals resulting in resolution of the expressed need and 88 percent of referrals accepted by the first provider in an average of just under three days.

The researchers also examined correlations among effectiveness, efficiency, and accuracy. They found that accuracy and effectiveness were positively correlated, suggesting that referrals that are accepted for care on the first attempt generally resolve the person’s request. Efficiency and effectiveness were not correlated, however, meaning faster entry into a system doesn’t necessarily result in resolution of the request.

At the same time, the researchers saw that context matters deeply—that is, expectations for effectiveness, efficiency, and accuracy must take into account the type of service in question.

For example, many of the services needed are low in complexity—food and clothing, for instance. Based on study data, such low-complexity services took an average of only 1.3 days to be addressed, with 93 percent accuracy and 78 percent efficiency. But more complicated services, like housing and legal services, saw efficiency numbers balloon to nearly 6 days and accuracy and effectiveness drop to 71 percent and 57 percent, respectively.

“You might be quick to label one particular network as being very efficient or very accurate,” Smilowitz says. “But then if you actually looked at the composition of their service load, that factors in significantly. So you can’t just say, ‘This is a good score for accuracy’ or ‘This is a good score for efficiency’ without knowing the context. That was a big ‘aha’ moment for us.”

Better Policy and Coordination

The findings here have significant policy-related and operational implications.

“There is now an increasing number of state and federal efforts to create and sustain these kinds of networks,” Shumate says. “They want to attach metrics to them, and if they rely mostly on efficiency and effectiveness, they will incentivize low-complexity services. That will incentivize ‘cream-skimming,’ where networks focus on the easiest options, like working on food instead of housing, which is a complex but critical service that a lot of the other things rely on.”

Another potential application of the findings, Smilowitz explains, is that “the work can help anticipate what kinds of needs might emerge, and when, among veterans and their families. So we can think about that kind of sequencing of services to provide, based on that.”

Featured Faculty

Professor of Operations; Professor of Industrial Engineering and Management Sciences

About the Writer

Sachin Waikar is a freelance writer based in Evanston, Illinois.

About the Research

Gibson, Zachary, Mariana Escallon-Barrios, Joshua-Paul Miles, Catherine Annis, Julia Carboni, Karen Smilowitz, Gilly Cantor, Nicholas Armstrong, and Michelle Shumate. 2022. “Beyond Network Effectiveness: The Case for Network Efficiency and Accuracy.” Working paper.

Read the original

Most Popular This Week
  1. Sitting Near a High-Performer Can Make You Better at Your Job
    “Spillover” from certain coworkers can boost our productivity—or jeopardize our employment.
    The spillover effect in offices impacts workers in close physical proximity.
  2. Podcast: How to Discuss Poor Performance with Your Employee
    Giving negative feedback is not easy, but such critiques can be meaningful for both parties if you use the right roadmap. Get advice on this episode of The Insightful Leader.
  3. 2 Factors Will Determine How Much AI Transforms Our Economy
    They’ll also dictate how workers stand to fare.
    robot waiter serves couple in restaurant
  4. Will AI Kill Human Creativity?
    What Fake Drake tells us about what’s ahead.
    Rockstars await a job interview.
  5. How Are Black–White Biracial People Perceived in Terms of Race?
    Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
    How are biracial people perceived in terms of race
  6. 5 Tips for Growing as a Leader without Burning Yourself Out
    A leadership coach and former CEO on how to take a holistic approach to your career.
    father picking up kids from school
  7. Will AI Eventually Replace Doctors?
    Maybe not entirely. But the doctor–patient relationship is likely to change dramatically.
    doctors offices in small nodules
  8. What Should Leaders Make of the Latest AI?
    As ChatGPT flaunts its creative capabilities, two experts discuss the promise and pitfalls of our coexistence with machines.
    person working on computer next to computer working at a computer
  9. Today’s Gig Workers Are Subject to Endless Experimentation
    “It raises the question, do we want to be a society where experimentation is just the norm?”
    gig worker at computer with three scientists studying them through a window
  10. How to Make Inclusivity More Than Just an Office Buzzword
    Tips for turning good intentions into actions.
    A group of coworkers sit in various chairs.
  11. China’s Youth Unemployment Problem
    If the record-breaking joblessness persists, as seems likely, China will have an even harder time supporting its rapidly aging population.
    college graduate standing before Chinese flag
  12. The Psychological Factor That Helps Shape Our Moral Decision-Making
    We all have a preferred motivation style. When that aligns with how we’re approaching a specific goal, it can impact how ethical we are in sticky situations.
    a person puts donuts into a bag next to a sign that reads "limit one"
  13. How to Manage a Disengaged Employee—and Get Them Excited about Work Again
    Don’t give up on checked-out team members. Try these strategies instead.
    CEO cheering on team with pom-poms
  14. Why Do Some People Succeed after Failing, While Others Continue to Flounder?
    A new study dispels some of the mystery behind success after failure.
    Scientists build a staircase from paper
  15. Why Are We So Quick to Borrow When the Value of Our Home Rises?
    The reason isn’t as simple as just feeling wealthier.
    A homeowner uses the value of their home to buy things.
  16. One Key to a Happy Marriage? A Joint Bank Account.
    Merging finances helps newlyweds align their financial goals and avoid scorekeeping.
    married couple standing at bank teller's window
  17. What’s at Stake in the Debt-Ceiling Standoff?
    Defaulting would be an unmitigated disaster, quickly felt by ordinary Americans.
    two groups of politicians negotiate while dangling upside down from the ceiling of a room
  18. Take 5: Research-Backed Tips for Scheduling Your Day
    Kellogg faculty offer ideas for working smarter and not harder.
    A to-do list with easy and hard tasks
  19. The Second-Mover Advantage
    A primer on how late-entering companies can compete with pioneers.
More in Policy