How a Good Analytics Strategy Can Become the Victim of Its Own Success
Skip to content
Data Analytics Marketing Jan 5, 2018

How a Good Analytics Strategy Can Become the Victim of Its Own Success

The best firms “purposely mess stuff up” to get the data they need to grow.

Two data analysts sort marbles.

Michael Meier

There’s a parable that Eric Anderson, a professor of marketing at the Kellogg School, likes to tell, one he’s deemed the “Analytics Paradox.”

“A young firm starts out making many mistakes. Eager to improve, they collect lots of data and build cool new models,” he says. “Over time, these models allow the young firm to find the best answers and implement these with great precision. The young firm becomes a mature firm that is great at analytics. Then one day the models stop working. Mistakes that fueled the models are now gone and the analytic models are starved.”

The paradox is that the better the firm gets at gleaning insights from analytics—and acting on those insights—the more streamlined their operations become. This in turn makes the data resulting from those operations more homogeneous. But over time, homogeneity becomes a problem: variable data—and, yes, mistakes—allow algorithms to continue to learn and optimize. As the variability in the new data shrinks, the algorithms don’t have much to work with anymore.

The paradox leads to a rather startling recommendation: “occasionally you need to purposely mess stuff up,” Florian Zettelmeyer says.

“You design variation into your data in order to be able to derive long-run insight,” explains Zettelmeyer, also a professor of marketing at Kellogg.

Zettelmeyer and Anderson are academic directors of Kellogg’s Executive Education program on Leading with Big Data and Analytics; they are also writing a book about data science for leaders.

Here, they offer a look at how the best firms have found a way to sidestep the Analytics Paradox.

From Optimization to Stagnation

In some sense, the value in big data lies in its messiness—in the often unexpected variation in how events play out and the myriad ways these events help establish connections between variables that can help people make better decisions.

“In theory, the best manager for analytics is the one who walks into the office every morning and flips a coin to make all decisions,” Anderson says. “Because if you make all your decisions by flipping a coin, you will generate the best possible data for your analytics engine.”

“The problem,” he adds, “is that at every company, the manager flipping coins gets fired very quickly. The managers who survive are the ones who are really good at implementing decisions with great precision.”

To understand how the best teams can find their operations too optimized for their own good, Anderson offers this hypothetical example.

Florian Zettelmeyer and Eric Anderson are the academic directors for Kellogg Executive Education’s Leading With Big Data and Analytics program. Learn more about Kellogg Exec Ed’s Analytics for Better Marketing Decisions for marketing executives here.

“Right now your company offers two-day delivery, and someone says to you, ‘I would like you to go back and analyze the historical data. Tell me whether we should have two-day delivery or move to one-day delivery.’ Could you answer that question with your data?”

If your delivery process is being overseen by a high-performing team focused squarely on efficiency, then you likely cannot answer this question with data.

“If you are really good at delivery—if you’ve been running operations efficiently—how many days does it take? Two days,” says Anderson. “The guy who was messing up and taking four days to deliver a package was fired. The one who was delivering in three days sometimes and one day other times got fired. You’re left with all of the managers who deliver in two days—you’ve built an organization that is so good at delivering things that it almost always happens in two days.”

“If I don’t occasionally do the wrong thing, I will never know whether what I think is the best actually still is the best.” — Florian Zettelmeyer.  

Hamstrung by your own success, you do not have the data to know whether a better possible delivery strategy exists, or how you might successfully move to a new model.

“If I don’t occasionally do the wrong thing, I will never know whether what I think is the best actually still is the best,” says Zettelmeyer.

What the Best Firms Are Doing

Of course, firms have plenty of good reasons for not wanting to richly reward incompetence, or promote a manager whose decision-making seems limited to coin flips.

Instead, top firms have adopted a fundamentally different strategy for thinking about big data.

“The best firms are heavily investing now in creating data, designing data,” says Anderson. “They’re purposefully injecting variability in the data.”

Whether they are experimenting with how many days it takes to deliver a package, how to set prices, or how to best maintain an aging fleet of vehicles, these elite firms understand that experimentation and variability need to be built into the organization’s DNA.

“It’s just a miniscule fraction of firms” doing this, maybe five percent, says Anderson.

So what do most managers need to do differently?

“When you take a business action, you need to keep in mind what the effect is on the usefulness of the data that are going to emerge from it,” says Zettelmeyer.

That requires the foresight to understand the questions you may wish to answer in the future, as well as the discipline to work backward from those questions to ensure that you set yourself up to get data that are rich and helpful.

A company rolling out a national advertising campaign, for instance, might decide to tweak the campaign in important ways only in select markets, or to stagger the rollout by region. While there may be short-term costs in terms of efficiency and optimization, the resulting data have the potential to teach the company going forward.

Don’t Relegate Data Science to the Data Scientists

Such foresight cannot be the purview of a single employee or team at an organization, the pair stress. That’s because decisions about how to experiment should be made with specific problems in mind.

“It cuts across the whole organization, so it has to be a cultural change in how we think about our day-to-day operations,” says Anderson.

The key, Zettelmeyer says, is “to transport yourself into the situation you’re going to find yourself in in the future.” What data would be helpful to have in order to make the next decision, and the next one? What relationship between variables do you want to demonstrate? And how could you design an experiment to demonstrate that link, given your existing capabilities and constraints?

And keep in mind that the infrastructure this requires may be quite different from what is necessary for managing much of the big data that flows through an organization. For instance, the high-level dashboards that senior leaders are used to may not be capable of distinguishing among many subtle but important differences in when a campaign was rolled out, for instance, or how a delivery route was established.

“It’s a very different thought process in terms of how you would actually build an IT system to support experimentation,” says Anderson.

Thus, rather than try to outsource this work to a dedicated data-science team—or worse, a single piece of software—Anderson and Zettelmeyer recommend that firms train managers on how to think and ask questions about data.

“It requires a working knowledge of data science,” says Zettelmeyer. “This is a skill set that managers need in order to even be conscious that this is something they need to take charge of.”

Featured Faculty

Polk Bros. Chair in Retailing; Professor of Marketing; Director Kellogg-McCormick MBAi

Nancy L. Ertle Professor of Marketing; Faculty Director, Program on Data Analytics at Kellogg

About the Writer
Jessica Love is editor-in-chief of Kellogg Insight.
Most Popular This Week
  1. Sitting Near a High-Performer Can Make You Better at Your Job
    “Spillover” from certain coworkers can boost our productivity—or jeopardize our employment.
    The spillover effect in offices impacts workers in close physical proximity.
  2. 5 Tips for Growing as a Leader without Burning Yourself Out
    A leadership coach and former CEO on how to take a holistic approach to your career.
    father picking up kids from school
  3. How Are Black–White Biracial People Perceived in Terms of Race?
    Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
    How are biracial people perceived in terms of race
  4. 2 Factors Will Determine How Much AI Transforms Our Economy
    They’ll also dictate how workers stand to fare.
    robot waiter serves couple in restaurant
  5. Podcast: How to Discuss Poor Performance with Your Employee
    Giving negative feedback is not easy, but such critiques can be meaningful for both parties if you use the right roadmap. Get advice on this episode of The Insightful Leader.
  6. What Should Leaders Make of the Latest AI?
    As ChatGPT flaunts its creative capabilities, two experts discuss the promise and pitfalls of our coexistence with machines.
    person working on computer next to computer working at a computer
  7. Today’s Gig Workers Are Subject to Endless Experimentation
    “It raises the question, do we want to be a society where experimentation is just the norm?”
    gig worker at computer with three scientists studying them through a window
  8. Will AI Eventually Replace Doctors?
    Maybe not entirely. But the doctor–patient relationship is likely to change dramatically.
    doctors offices in small nodules
  9. How to Make Inclusivity More Than Just an Office Buzzword
    Tips for turning good intentions into actions.
    A group of coworkers sit in various chairs.
  10. China’s Youth Unemployment Problem
    If the record-breaking joblessness persists, as seems likely, China will have an even harder time supporting its rapidly aging population.
    college graduate standing before Chinese flag
  11. Will AI Kill Human Creativity?
    What Fake Drake tells us about what’s ahead.
    Rockstars await a job interview.
  12. Why Are We So Quick to Borrow When the Value of Our Home Rises?
    The reason isn’t as simple as just feeling wealthier.
    A homeowner uses the value of their home to buy things.
  13. Take 5: Research-Backed Tips for Scheduling Your Day
    Kellogg faculty offer ideas for working smarter and not harder.
    A to-do list with easy and hard tasks
  14. Why Do Some People Succeed after Failing, While Others Continue to Flounder?
    A new study dispels some of the mystery behind success after failure.
    Scientists build a staircase from paper
  15. How to Manage a Disengaged Employee—and Get Them Excited about Work Again
    Don’t give up on checked-out team members. Try these strategies instead.
    CEO cheering on team with pom-poms
  16. Which Form of Government Is Best?
    Democracies may not outlast dictatorships, but they adapt better.
    Is democracy the best form of government?
  17. The Second-Mover Advantage
    A primer on how late-entering companies can compete with pioneers.
  18. What Happens to Worker Productivity after a Minimum Wage Increase?
    A pay raise boosts productivity for some—but the impact on the bottom line is more complicated.
    employees unload pallets from a truck using hand carts
More in Data Analytics