Data Analytics Marketing Jan 5, 2018

How a Good Ana­lyt­ics Strat­e­gy Can Become the Vic­tim of Its Own Success

The best firms pur­pose­ly mess stuff up” to get the data they need to grow.

Michael Meier

Based on insights from

Eric T. Anderson

Florian Zettelmeyer

There’s a para­ble that Eric Ander­son, a pro­fes­sor of mar­ket­ing at the Kel­logg School, likes to tell, one he’s deemed the Ana­lyt­ics Paradox.”

Add Insight
to your inbox.

We’ll send you one email a week with content you actually want to read, curated by the Insight team.

Sign up here

A young firm starts out mak­ing many mis­takes. Eager to improve, they col­lect lots of data and build cool new mod­els,” he says. Over time, these mod­els allow the young firm to find the best answers and imple­ment these with great pre­ci­sion. The young firm becomes a mature firm that is great at ana­lyt­ics. Then one day the mod­els stop work­ing. Mis­takes that fueled the mod­els are now gone and the ana­lyt­ic mod­els are starved.” 

The para­dox is that the bet­ter the firm gets at glean­ing insights from ana­lyt­ics — and act­ing on those insights — the more stream­lined their oper­a­tions become. This in turn makes the data result­ing from those oper­a­tions more homo­ge­neous. But over time, homo­gene­ity becomes a prob­lem: vari­able data — and, yes, mis­takes — allow algo­rithms to con­tin­ue to learn and opti­mize. As the vari­abil­i­ty in the new data shrinks, the algo­rithms don’t have much to work with anymore. 

The para­dox leads to a rather star­tling rec­om­men­da­tion: occa­sion­al­ly you need to pur­pose­ly mess stuff up,” Flo­ri­an Zettelmey­er says. 

You design vari­a­tion into your data in order to be able to derive long-run insight,” explains Zettelmey­er, also a pro­fes­sor of mar­ket­ing at Kellogg. 

Zettelmey­er and Ander­son are aca­d­e­m­ic direc­tors of Kellogg’s Exec­u­tive Edu­ca­tion pro­gram on Lead­ing with Big Data and Ana­lyt­ics; they are also writ­ing a book about data sci­ence for leaders. 

Here, they offer a look at how the best firms have found a way to side­step the Ana­lyt­ics Paradox. 

From Opti­miza­tion to Stagnation 

In some sense, the val­ue in big data lies in its messi­ness — in the often unex­pect­ed vari­a­tion in how events play out and the myr­i­ad ways these events help estab­lish con­nec­tions between vari­ables that can help peo­ple make bet­ter decisions. 

In the­o­ry, the best man­ag­er for ana­lyt­ics is the one who walks into the office every morn­ing and flips a coin to make all deci­sions,” Ander­son says. Because if you make all your deci­sions by flip­ping a coin, you will gen­er­ate the best pos­si­ble data for your ana­lyt­ics engine.” 

The prob­lem,” he adds, is that at every com­pa­ny, the man­ag­er flip­ping coins gets fired very quick­ly. The man­agers who sur­vive are the ones who are real­ly good at imple­ment­ing deci­sions with great precision.” 

To under­stand how the best teams can find their oper­a­tions too opti­mized for their own good, Ander­son offers this hypo­thet­i­cal example. 

Flo­ri­an Zettelmey­er and Eric Ander­son are the aca­d­e­m­ic direc­tors for Kel­logg Exec­u­tive Education’s Lead­ing With Big Data and Ana­lyt­ics pro­gram. Learn more about Kel­logg Exec Ed’s Ana­lyt­ics for Bet­ter Mar­ket­ing Deci­sions for mar­ket­ing exec­u­tives here.

Right now your com­pa­ny offers two-day deliv­ery, and some­one says to you, I would like you to go back and ana­lyze the his­tor­i­cal data. Tell me whether we should have two-day deliv­ery or move to one-day deliv­ery.’ Could you answer that ques­tion with your data?” 

If your deliv­ery process is being over­seen by a high-per­form­ing team focused square­ly on effi­cien­cy, then you like­ly can­not answer this ques­tion with data. 

If you are real­ly good at deliv­ery — if you’ve been run­ning oper­a­tions effi­cient­ly — how many days does it take? Two days,” says Ander­son. The guy who was mess­ing up and tak­ing four days to deliv­er a pack­age was fired. The one who was deliv­er­ing in three days some­times and one day oth­er times got fired. You’re left with all of the man­agers who deliv­er in two days — you’ve built an orga­ni­za­tion that is so good at deliv­er­ing things that it almost always hap­pens in two days.” 

If I don’t occa­sion­al­ly do the wrong thing, I will nev­er know whether what I think is the best actu­al­ly still is the best.” — Flo­ri­an Zettelmeyer. 

Ham­strung by your own suc­cess, you do not have the data to know whether a bet­ter pos­si­ble deliv­ery strat­e­gy exists, or how you might suc­cess­ful­ly move to a new model. 

If I don’t occa­sion­al­ly do the wrong thing, I will nev­er know whether what I think is the best actu­al­ly still is the best,” says Zettelmeyer. 

What the Best Firms Are Doing 

Of course, firms have plen­ty of good rea­sons for not want­i­ng to rich­ly reward incom­pe­tence, or pro­mote a man­ag­er whose deci­sion-mak­ing seems lim­it­ed to coin flips. 

Instead, top firms have adopt­ed a fun­da­men­tal­ly dif­fer­ent strat­e­gy for think­ing about big data. 

The best firms are heav­i­ly invest­ing now in cre­at­ing data, design­ing data,” says Ander­son. They’re pur­pose­ful­ly inject­ing vari­abil­i­ty in the data.” 

Whether they are exper­i­ment­ing with how many days it takes to deliv­er a pack­age, how to set prices, or how to best main­tain an aging fleet of vehi­cles, these elite firms under­stand that exper­i­men­ta­tion and vari­abil­i­ty need to be built into the organization’s DNA

It’s just a minis­cule frac­tion of firms” doing this, maybe five per­cent, says Anderson. 

So what do most man­agers need to do differently? 

When you take a busi­ness action, you need to keep in mind what the effect is on the use­ful­ness of the data that are going to emerge from it,” says Zettelmeyer. 

That requires the fore­sight to under­stand the ques­tions you may wish to answer in the future, as well as the dis­ci­pline to work back­ward from those ques­tions to ensure that you set your­self up to get data that are rich and help­ful.

A com­pa­ny rolling out a nation­al adver­tis­ing cam­paign, for instance, might decide to tweak the cam­paign in impor­tant ways only in select mar­kets, or to stag­ger the roll­out by region. While there may be short-term costs in terms of effi­cien­cy and opti­miza­tion, the result­ing data have the poten­tial to teach the com­pa­ny going forward. 

Don’t Rel­e­gate Data Sci­ence to the Data Scientists 

Such fore­sight can­not be the purview of a sin­gle employ­ee or team at an orga­ni­za­tion, the pair stress. That’s because deci­sions about how to exper­i­ment should be made with spe­cif­ic prob­lems in mind. 

It cuts across the whole orga­ni­za­tion, so it has to be a cul­tur­al change in how we think about our day-to-day oper­a­tions,” says Anderson. 

The key, Zettelmey­er says, is to trans­port your­self into the sit­u­a­tion you’re going to find your­self in in the future.” What data would be help­ful to have in order to make the next deci­sion, and the next one? What rela­tion­ship between vari­ables do you want to demon­strate? And how could you design an exper­i­ment to demon­strate that link, giv­en your exist­ing capa­bil­i­ties and constraints? 

And keep in mind that the infra­struc­ture this requires may be quite dif­fer­ent from what is nec­es­sary for man­ag­ing much of the big data that flows through an orga­ni­za­tion. For instance, the high-lev­el dash­boards that senior lead­ers are used to may not be capa­ble of dis­tin­guish­ing among many sub­tle but impor­tant dif­fer­ences in when a cam­paign was rolled out, for instance, or how a deliv­ery route was established. 

It’s a very dif­fer­ent thought process in terms of how you would actu­al­ly build an IT sys­tem to sup­port exper­i­men­ta­tion,” says Anderson. 

Thus, rather than try to out­source this work to a ded­i­cat­ed data-sci­ence team — or worse, a sin­gle piece of soft­ware — Ander­son and Zettelmey­er rec­om­mend that firms train man­agers on how to think and ask ques­tions about data. 

It requires a work­ing knowl­edge of data sci­ence,” says Zettelmey­er. This is a skill set that man­agers need in order to even be con­scious that this is some­thing they need to take charge of.” 

Featured Faculty

Eric T. Anderson

Hartmarx Professor of Marketing, Professor of Marketing, Director of the Center for Global Marketing Practice

Florian Zettelmeyer

Nancy L. Ertle Professor of Marketing

About the Writer

Jessica Love is editor-in-chief of Kellogg Insight.

Editor’s Picks

Careers

Pod­cast: How to Be a Great Mentor

Plus, some valu­able career advice that applies to just about everyone.

Marketing

A New Way to Per­suade Kids to Drink More Water and Less Soda

Get­ting chil­dren to make healthy choic­es is tricky — and the wrong mes­sage can backfire.

Innovation

How Can Social Sci­ence Become More Solutions-Oriented?

A con­ver­sa­tion between researchers at Kel­logg and Microsoft explores how behav­ioral sci­ence can best be applied.

Innovation

Buy­ing a Com­pa­ny for Its Tal­ent? Beware of Hid­den Legal Risks.

Acquir­ing anoth­er firm’s trade secrets — even unin­ten­tion­al­ly — could prove costly.

Careers

Take 5: Tips for Widen­ing — and Improv­ing — Your Can­di­date Pool

Com­mon bias­es can cause com­pa­nies to over­look a wealth of top talent.

Innovation

Every­one Wants Phar­ma­ceu­ti­cal Break­throughs. What Dri­ves Drug Com­pa­nies to Pur­sue Them?

A new study sug­gests that firms are at their most inno­v­a­tive after a finan­cial windfall.

Careers

4 Key Steps to Prepar­ing for a Busi­ness Presentation

Don’t let a lack of prep work sab­o­tage your great ideas.

Healthcare

Video: How Open Lines of Com­mu­ni­ca­tion Can Improve Health­care Outcomes

Train­ing physi­cians to be bet­ter com­mu­ni­ca­tors builds trust with patients and their loved ones.

Operations

Here’s a Bet­ter Way to Sched­ule Surgeries

A new tool could dri­ve sav­ings of 20 per­cent while still keep­ing sur­geons happy.

Politics & Elections

Why Eco­nom­ic Crises Trig­ger Polit­i­cal Turnover in Some Coun­tries but Not Others

The fall­out can hinge on how much a country’s peo­ple trust each other.

Marketing

Build­ing Strong Brands: The Inside Scoop on Brand­ing in the Real World

Tim Calkins’s blog draws lessons from brand mis­steps and triumphs.

Economics

How the Cof­fee Indus­try Is Build­ing a Sus­tain­able Sup­ply Chain in an Unsta­ble Region

Three experts dis­cuss the chal­lenges and rewards of sourc­ing cof­fee from the Demo­c­ra­t­ic Repub­lic of Congo.