How a Good Analytics Strategy Can Become the Victim of Its Own Success
Skip to content
Data Analytics Marketing Jan 5, 2018

How a Good Ana­lyt­ics Strat­e­gy Can Become the Vic­tim of Its Own Success

The best firms pur­pose­ly mess stuff up” to get the data they need to grow.

Two data analysts sort marbles.

Michael Meier

There’s a para­ble that Eric Ander­son, a pro­fes­sor of mar­ket­ing at the Kel­logg School, likes to tell, one he’s deemed the Ana­lyt­ics Paradox.”

Add Insight
to your inbox.

We’ll send you one email a week with content you actually want to read, curated by the Insight team.

A young firm starts out mak­ing many mis­takes. Eager to improve, they col­lect lots of data and build cool new mod­els,” he says. Over time, these mod­els allow the young firm to find the best answers and imple­ment these with great pre­ci­sion. The young firm becomes a mature firm that is great at ana­lyt­ics. Then one day the mod­els stop work­ing. Mis­takes that fueled the mod­els are now gone and the ana­lyt­ic mod­els are starved.” 

The para­dox is that the bet­ter the firm gets at glean­ing insights from ana­lyt­ics — and act­ing on those insights — the more stream­lined their oper­a­tions become. This in turn makes the data result­ing from those oper­a­tions more homo­ge­neous. But over time, homo­gene­ity becomes a prob­lem: vari­able data — and, yes, mis­takes — allow algo­rithms to con­tin­ue to learn and opti­mize. As the vari­abil­i­ty in the new data shrinks, the algo­rithms don’t have much to work with anymore. 

The para­dox leads to a rather star­tling rec­om­men­da­tion: occa­sion­al­ly you need to pur­pose­ly mess stuff up,” Flo­ri­an Zettelmey­er says. 

You design vari­a­tion into your data in order to be able to derive long-run insight,” explains Zettelmey­er, also a pro­fes­sor of mar­ket­ing at Kellogg. 

Zettelmey­er and Ander­son are aca­d­e­m­ic direc­tors of Kellogg’s Exec­u­tive Edu­ca­tion pro­gram on Lead­ing with Big Data and Ana­lyt­ics; they are also writ­ing a book about data sci­ence for leaders. 

Here, they offer a look at how the best firms have found a way to side­step the Ana­lyt­ics Paradox. 

From Opti­miza­tion to Stagnation 

In some sense, the val­ue in big data lies in its messi­ness — in the often unex­pect­ed vari­a­tion in how events play out and the myr­i­ad ways these events help estab­lish con­nec­tions between vari­ables that can help peo­ple make bet­ter decisions. 

In the­o­ry, the best man­ag­er for ana­lyt­ics is the one who walks into the office every morn­ing and flips a coin to make all deci­sions,” Ander­son says. Because if you make all your deci­sions by flip­ping a coin, you will gen­er­ate the best pos­si­ble data for your ana­lyt­ics engine.” 

The prob­lem,” he adds, is that at every com­pa­ny, the man­ag­er flip­ping coins gets fired very quick­ly. The man­agers who sur­vive are the ones who are real­ly good at imple­ment­ing deci­sions with great precision.” 

To under­stand how the best teams can find their oper­a­tions too opti­mized for their own good, Ander­son offers this hypo­thet­i­cal example. 

Flo­ri­an Zettelmey­er and Eric Ander­son are the aca­d­e­m­ic direc­tors for Kel­logg Exec­u­tive Education’s Lead­ing With Big Data and Ana­lyt­ics pro­gram. Learn more about Kel­logg Exec Ed’s Ana­lyt­ics for Bet­ter Mar­ket­ing Deci­sions for mar­ket­ing exec­u­tives here.

Right now your com­pa­ny offers two-day deliv­ery, and some­one says to you, I would like you to go back and ana­lyze the his­tor­i­cal data. Tell me whether we should have two-day deliv­ery or move to one-day deliv­ery.’ Could you answer that ques­tion with your data?” 

If your deliv­ery process is being over­seen by a high-per­form­ing team focused square­ly on effi­cien­cy, then you like­ly can­not answer this ques­tion with data. 

If you are real­ly good at deliv­ery — if you’ve been run­ning oper­a­tions effi­cient­ly — how many days does it take? Two days,” says Ander­son. The guy who was mess­ing up and tak­ing four days to deliv­er a pack­age was fired. The one who was deliv­er­ing in three days some­times and one day oth­er times got fired. You’re left with all of the man­agers who deliv­er in two days — you’ve built an orga­ni­za­tion that is so good at deliv­er­ing things that it almost always hap­pens in two days.” 

If I don’t occa­sion­al­ly do the wrong thing, I will nev­er know whether what I think is the best actu­al­ly still is the best.” — Flo­ri­an Zettelmeyer. 

Ham­strung by your own suc­cess, you do not have the data to know whether a bet­ter pos­si­ble deliv­ery strat­e­gy exists, or how you might suc­cess­ful­ly move to a new model. 

If I don’t occa­sion­al­ly do the wrong thing, I will nev­er know whether what I think is the best actu­al­ly still is the best,” says Zettelmeyer. 

What the Best Firms Are Doing 

Of course, firms have plen­ty of good rea­sons for not want­i­ng to rich­ly reward incom­pe­tence, or pro­mote a man­ag­er whose deci­sion-mak­ing seems lim­it­ed to coin flips. 

Instead, top firms have adopt­ed a fun­da­men­tal­ly dif­fer­ent strat­e­gy for think­ing about big data. 

The best firms are heav­i­ly invest­ing now in cre­at­ing data, design­ing data,” says Ander­son. They’re pur­pose­ful­ly inject­ing vari­abil­i­ty in the data.” 

Whether they are exper­i­ment­ing with how many days it takes to deliv­er a pack­age, how to set prices, or how to best main­tain an aging fleet of vehi­cles, these elite firms under­stand that exper­i­men­ta­tion and vari­abil­i­ty need to be built into the organization’s DNA

It’s just a minis­cule frac­tion of firms” doing this, maybe five per­cent, says Anderson. 

So what do most man­agers need to do differently? 

When you take a busi­ness action, you need to keep in mind what the effect is on the use­ful­ness of the data that are going to emerge from it,” says Zettelmeyer. 

That requires the fore­sight to under­stand the ques­tions you may wish to answer in the future, as well as the dis­ci­pline to work back­ward from those ques­tions to ensure that you set your­self up to get data that are rich and help­ful.

A com­pa­ny rolling out a nation­al adver­tis­ing cam­paign, for instance, might decide to tweak the cam­paign in impor­tant ways only in select mar­kets, or to stag­ger the roll­out by region. While there may be short-term costs in terms of effi­cien­cy and opti­miza­tion, the result­ing data have the poten­tial to teach the com­pa­ny going forward. 

Don’t Rel­e­gate Data Sci­ence to the Data Scientists 

Such fore­sight can­not be the purview of a sin­gle employ­ee or team at an orga­ni­za­tion, the pair stress. That’s because deci­sions about how to exper­i­ment should be made with spe­cif­ic prob­lems in mind. 

It cuts across the whole orga­ni­za­tion, so it has to be a cul­tur­al change in how we think about our day-to-day oper­a­tions,” says Anderson. 

The key, Zettelmey­er says, is to trans­port your­self into the sit­u­a­tion you’re going to find your­self in in the future.” What data would be help­ful to have in order to make the next deci­sion, and the next one? What rela­tion­ship between vari­ables do you want to demon­strate? And how could you design an exper­i­ment to demon­strate that link, giv­en your exist­ing capa­bil­i­ties and constraints? 

And keep in mind that the infra­struc­ture this requires may be quite dif­fer­ent from what is nec­es­sary for man­ag­ing much of the big data that flows through an orga­ni­za­tion. For instance, the high-lev­el dash­boards that senior lead­ers are used to may not be capa­ble of dis­tin­guish­ing among many sub­tle but impor­tant dif­fer­ences in when a cam­paign was rolled out, for instance, or how a deliv­ery route was established. 

It’s a very dif­fer­ent thought process in terms of how you would actu­al­ly build an IT sys­tem to sup­port exper­i­men­ta­tion,” says Anderson. 

Thus, rather than try to out­source this work to a ded­i­cat­ed data-sci­ence team — or worse, a sin­gle piece of soft­ware — Ander­son and Zettelmey­er rec­om­mend that firms train man­agers on how to think and ask ques­tions about data. 

It requires a work­ing knowl­edge of data sci­ence,” says Zettelmey­er. This is a skill set that man­agers need in order to even be con­scious that this is some­thing they need to take charge of.” 

Featured Faculty

Eric T. Anderson

Hartmarx Professor of Marketing, Professor of Marketing, Director of the Center for Global Marketing Practice

Florian Zettelmeyer

Nancy L. Ertle Professor of Marketing

About the Writer

Jessica Love is editor-in-chief of Kellogg Insight.

Suggested For You

Most Popular


How Are Black – White Bira­cial Peo­ple Per­ceived in Terms of Race?

Under­stand­ing the answer — and why black and white Amer­i­cans’ respons­es may dif­fer — is increas­ing­ly impor­tant in a mul­tira­cial society.


Why Warmth Is the Under­ap­pre­ci­at­ed Skill Lead­ers Need

The case for demon­strat­ing more than just competence.

Most Popular Podcasts


Pod­cast: Our Most Pop­u­lar Advice on Improv­ing Rela­tion­ships with Colleagues

Cowork­ers can make us crazy. Here’s how to han­dle tough situations.

Social Impact

Pod­cast: How You and Your Com­pa­ny Can Lend Exper­tise to a Non­prof­it in Need

Plus: Four ques­tions to con­sid­er before becom­ing a social-impact entrepreneur.


Pod­cast: Attract Rock­star Employ­ees — or Devel­op Your Own

Find­ing and nur­tur­ing high per­form­ers isn’t easy, but it pays off.


Pod­cast: How Music Can Change Our Mood

A Broad­way song­writer and a mar­ket­ing pro­fes­sor dis­cuss the con­nec­tion between our favorite tunes and how they make us feel.