How a Good Analytics Strategy Can Become the Victim of Its Own Success
Skip to content
Data Analytics Marketing Jan 5, 2018

How a Good Analytics Strategy Can Become the Victim of Its Own Success

The best firms “purposely mess stuff up” to get the data they need to grow.

Two data analysts sort marbles.

Michael Meier

There’s a parable that Eric Anderson, a professor of marketing at the Kellogg School, likes to tell, one he’s deemed the “Analytics Paradox.”

Add Insight
to your inbox.

“A young firm starts out making many mistakes. Eager to improve, they collect lots of data and build cool new models,” he says. “Over time, these models allow the young firm to find the best answers and implement these with great precision. The young firm becomes a mature firm that is great at analytics. Then one day the models stop working. Mistakes that fueled the models are now gone and the analytic models are starved.”

The paradox is that the better the firm gets at gleaning insights from analytics—and acting on those insights—the more streamlined their operations become. This in turn makes the data resulting from those operations more homogeneous. But over time, homogeneity becomes a problem: variable data—and, yes, mistakes—allow algorithms to continue to learn and optimize. As the variability in the new data shrinks, the algorithms don’t have much to work with anymore.

The paradox leads to a rather startling recommendation: “occasionally you need to purposely mess stuff up,” Florian Zettelmeyer says.

“You design variation into your data in order to be able to derive long-run insight,” explains Zettelmeyer, also a professor of marketing at Kellogg.

Zettelmeyer and Anderson are academic directors of Kellogg’s Executive Education program on Leading with Big Data and Analytics; they are also writing a book about data science for leaders.

Here, they offer a look at how the best firms have found a way to sidestep the Analytics Paradox.

From Optimization to Stagnation

In some sense, the value in big data lies in its messiness—in the often unexpected variation in how events play out and the myriad ways these events help establish connections between variables that can help people make better decisions.

“In theory, the best manager for analytics is the one who walks into the office every morning and flips a coin to make all decisions,” Anderson says. “Because if you make all your decisions by flipping a coin, you will generate the best possible data for your analytics engine.”

“The problem,” he adds, “is that at every company, the manager flipping coins gets fired very quickly. The managers who survive are the ones who are really good at implementing decisions with great precision.”

To understand how the best teams can find their operations too optimized for their own good, Anderson offers this hypothetical example.

Florian Zettelmeyer and Eric Anderson are the academic directors for Kellogg Executive Education’s Leading With Big Data and Analytics program. Learn more about Kellogg Exec Ed’s Analytics for Better Marketing Decisions for marketing executives here.

“Right now your company offers two-day delivery, and someone says to you, ‘I would like you to go back and analyze the historical data. Tell me whether we should have two-day delivery or move to one-day delivery.’ Could you answer that question with your data?”

If your delivery process is being overseen by a high-performing team focused squarely on efficiency, then you likely cannot answer this question with data.

“If you are really good at delivery—if you’ve been running operations efficiently—how many days does it take? Two days,” says Anderson. “The guy who was messing up and taking four days to deliver a package was fired. The one who was delivering in three days sometimes and one day other times got fired. You’re left with all of the managers who deliver in two days—you’ve built an organization that is so good at delivering things that it almost always happens in two days.”

“If I don’t occasionally do the wrong thing, I will never know whether what I think is the best actually still is the best.” — Florian Zettelmeyer.  

Hamstrung by your own success, you do not have the data to know whether a better possible delivery strategy exists, or how you might successfully move to a new model.

“If I don’t occasionally do the wrong thing, I will never know whether what I think is the best actually still is the best,” says Zettelmeyer.

What the Best Firms Are Doing

Of course, firms have plenty of good reasons for not wanting to richly reward incompetence, or promote a manager whose decision-making seems limited to coin flips.

Instead, top firms have adopted a fundamentally different strategy for thinking about big data.

“The best firms are heavily investing now in creating data, designing data,” says Anderson. “They’re purposefully injecting variability in the data.”

Whether they are experimenting with how many days it takes to deliver a package, how to set prices, or how to best maintain an aging fleet of vehicles, these elite firms understand that experimentation and variability need to be built into the organization’s DNA.

“It’s just a miniscule fraction of firms” doing this, maybe five percent, says Anderson.

So what do most managers need to do differently?

“When you take a business action, you need to keep in mind what the effect is on the usefulness of the data that are going to emerge from it,” says Zettelmeyer.

That requires the foresight to understand the questions you may wish to answer in the future, as well as the discipline to work backward from those questions to ensure that you set yourself up to get data that are rich and helpful.

A company rolling out a national advertising campaign, for instance, might decide to tweak the campaign in important ways only in select markets, or to stagger the rollout by region. While there may be short-term costs in terms of efficiency and optimization, the resulting data have the potential to teach the company going forward.

Don’t Relegate Data Science to the Data Scientists

Such foresight cannot be the purview of a single employee or team at an organization, the pair stress. That’s because decisions about how to experiment should be made with specific problems in mind.

“It cuts across the whole organization, so it has to be a cultural change in how we think about our day-to-day operations,” says Anderson.

The key, Zettelmeyer says, is “to transport yourself into the situation you’re going to find yourself in in the future.” What data would be helpful to have in order to make the next decision, and the next one? What relationship between variables do you want to demonstrate? And how could you design an experiment to demonstrate that link, given your existing capabilities and constraints?

And keep in mind that the infrastructure this requires may be quite different from what is necessary for managing much of the big data that flows through an organization. For instance, the high-level dashboards that senior leaders are used to may not be capable of distinguishing among many subtle but important differences in when a campaign was rolled out, for instance, or how a delivery route was established.

“It’s a very different thought process in terms of how you would actually build an IT system to support experimentation,” says Anderson.

Thus, rather than try to outsource this work to a dedicated data-science team—or worse, a single piece of software—Anderson and Zettelmeyer recommend that firms train managers on how to think and ask questions about data.

“It requires a working knowledge of data science,” says Zettelmeyer. “This is a skill set that managers need in order to even be conscious that this is something they need to take charge of.”

Featured Faculty

Polk Bros. Chair in Retailing; Professor of Marketing; Director Kellogg-McCormick MBAi

Nancy L. Ertle Professor of Marketing; Faculty Director, Program on Data Analytics at Kellogg; Chair of Marketing Department

About the Writer
Jessica Love is editor-in-chief of Kellogg Insight.
Most Popular This Week
  1. Will AI Eventually Replace Doctors?
    Maybe not entirely. But the doctor–patient relationship is likely to change dramatically.
    doctors offices in small nodules
  2. 3 Tips for Reinventing Your Career After a Layoff
    It’s crucial to reassess what you want to be doing instead of jumping at the first opportunity.
    woman standing confidently
  3. What Happens to Worker Productivity after a Minimum Wage Increase?
    A pay raise boosts productivity for some—but the impact on the bottom line is more complicated.
    employees unload pallets from a truck using hand carts
  4. 6 Takeaways on Inflation and the Economy Right Now
    Are we headed into a recession? Kellogg’s Sergio Rebelo breaks down the latest trends.
    inflatable dollar sign tied down with mountains in background
  5. What Is the Purpose of a Corporation Today?
    Has anything changed in the three years since the Business Roundtable declared firms should prioritize more than shareholders?
    A city's skyscrapers interspersed with trees and rooftop gardens
  6. How to Get the Ear of Your CEO—And What to Say When You Have It
    Every interaction with the top boss is an audition for senior leadership.
    employee presents to CEO in elevator
  7. Why We Can’t All Get Away with Wearing Designer Clothes
    In certain professions, luxury goods can send the wrong signal.​
    Man wearing luxury-brand clothes walks with a cold wind behind him, chilling three people he passes.
  8. Why You Should Skip the Easy Wins and Tackle the Hard Task First
    New research shows that you and your organization lose out when you procrastinate on the difficult stuff.
    A to-do list with easy and hard tasks
  9. How Are Black–White Biracial People Perceived in Terms of Race?
    Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
    How are biracial people perceived in terms of race
  10. Which Form of Government Is Best?
    Democracies may not outlast dictatorships, but they adapt better.
    Is democracy the best form of government?
  11. When Do Open Borders Make Economic Sense?
    A new study provides a window into the logic behind various immigration policies.
    How immigration affects the economy depends on taxation and worker skills.
  12. Why Do Some People Succeed after Failing, While Others Continue to Flounder?
    A new study dispels some of the mystery behind success after failure.
    Scientists build a staircase from paper
  13. How Has Marketing Changed over the Past Half-Century?
    Phil Kotler’s groundbreaking textbook came out 55 years ago. Sixteen editions later, he and coauthor Alexander Chernev discuss how big data, social media, and purpose-driven branding are moving the field forward.
    people in 1967 and 2022 react to advertising
  14. How Old Are Successful Tech Entrepreneurs?
    A definitive new study dispels the myth of the Silicon Valley wunderkind.
    successful entrepreneurs are most often middle aged
  15. How Offering a Product for Free Can Backfire
    It seems counterintuitive, but there are times customers would rather pay a small amount than get something for free.
    people in grocery store aisle choosing cheap over free option of same product.
  16. Immigrants to the U.S. Create More Jobs than They Take
    A new study finds that immigrants are far more likely to found companies—both large and small—than native-born Americans.
    Immigrant CEO welcomes new hires
  17. College Campuses Are Becoming More Diverse. But How Much Do Students from Different Backgrounds Actually Interact?
    Increasing diversity has been a key goal, “but far less attention is paid to what happens after we get people in the door.”
    College quad with students walking away from the center
  18. How Peer Pressure Can Lead Teens to Underachieve—Even in Schools Where It’s “Cool to Be Smart”
    New research offers lessons for administrators hoping to improve student performance.
    Eager student raises hand while other student hesitates.
More in Data Analytics