Podcast: Think You Understand Why Ideas Go Viral? Big Data May Change Your Mind
Skip to content
Data Analytics Marketing Oct 3, 2016

Pod­cast: Think You Under­stand Why Ideas Go Viral? Big Data May Change Your Mind

From tweets to sci­en­tif­ic dis­cov­er­ies, human behav­ior is sur­pris­ing­ly predictable.

A researcher uses big data to understand patterns

Yevgenia Nayberg

Based on the research of

Dashun Wang

Duncan Watts

Listening: Big Data and Ideas Going Viral

download
0:00 Skip back button Play Skip forward button 16:47

Why do some ideas go viral while oth­ers go nowhere? Is it all about reach­ing that myth­i­cal tip­ping point, or is some­thing else at work?

Kel­logg Insight talked with two researchers who are start­ing to find answers by ana­lyz­ing huge amounts of data. Microsoft’s Dun­can Watts explains why we should stop wor­ry­ing about a tip­ping point, and Kel­logg Pro­fes­sor Dashun Wang dis­cuss­es how human behav­ior is more pre­dictable than you might think.

Add Insight
to your inbox.

We’ll send you one email a week with content you actually want to read, curated by the Insight team.

Pod­cast transcript

[music pre­lude]

Emi­ly STONE: Remem­ber the Ice Buck­et Chal­lenge? Or that striped dress that some peo­ple swore was blue and black and oth­ers were sure was white and gold? Or the lat­est hilar­i­ous cat video? You like­ly do, because they went viral. They were shared by mil­lions on social media and picked up as sto­ries in the news.

But for every Ice Buck­et Chal­lenge, there’s also a fundrais­ing gim­mick that falls flat. Is it pos­si­ble to know what’s going to be a run­away suc­cess ver­sus a dress that’s … just a dress? To put it anoth­er way: Is human behav­ior predictable?

[music inter­lude]

STONE: Hel­lo, and wel­come to Kel­logg Insight’s month­ly pod­cast. I’m your host, Emi­ly Stone. In this episode, we talk with two researchers who use huge amounts of data to try to pre­dict how humans will col­lec­tive­ly act, despite all our diver­si­ty and com­plex­i­ty. So stay with us.

[music inter­lude]

Dun­can WATTS: We are inter­est­ed in a wide range of ques­tions about col­lec­tive human behav­ior: So what is the struc­ture of orga­ni­za­tion­al net­works and how can we under­stand that and map those out using e-mail logs? How does infor­ma­tion spread over large social net­works and media net­works like Twit­ter? How do peo­ple coop­er­ate or solve prob­lems in groups? So we’re inter­est­ed, gen­er­al­ly, in how peo­ple behave col­lec­tive­ly and how we can shed new light on these old ques­tions using mod­ern dig­i­tal technology.

STONE: That’s Dun­can Watts, a researcher in the com­pu­ta­tion­al and social sci­ence group at Microsoft Research.

This line of ques­tion­ing is not new. Sci­en­tists have long been inter­est­ed in how peo­ple work togeth­er and how ideas, to use the lan­guage of social sci­en­tists, diffuse.

WATTS: For most of that time, it’s been a very the­o­ret­i­cal exer­cise, where you sort of sit and think deeply about how you think things are and maybe you have some sort of anec­do­tal obser­va­tions from your own expe­ri­ence, or you go and sit and watch a small group of peo­ple in a nat­ur­al set­ting, or you admin­is­ter some survey.

STONE: Even the ide­al sce­nario, where you would run a for­mal exper­i­ment, has its down­sides. You end up with a lim­it­ed amount of data about a lim­it­ed num­ber of peo­ple to demon­strate real­ly broad the­o­ries about what makes humans tick.

But this type of research is chang­ing on a fun­da­men­tal lev­el. Today sci­en­tists have vast amounts of data avail­able to use in their stud­ies. Think of all those bits and bytes of infor­ma­tion that pour out of your com­put­er, your phone — per­haps even your ther­mo­stat or refrig­er­a­tor — every sin­gle sec­ond. All of that can be col­lect­ed and crunched by researchers who want to bet­ter under­stand our basic behav­ioral patterns.

And yet with that data comes the real­iza­tion that we may have pre­vi­ous­ly got­ten things com­plete­ly wrong.

WATTS: For exam­ple, in the dif­fu­sion lit­er­a­ture, there’s many, many the­o­ret­i­cal mod­els about how things spread on net­works and a lot of the focus is on what’s called the epi­dem­ic thresh­old or the tip­ping point.

It’s sort of this tran­si­tion from when things don’t spread to when things start spread­ing. If you’re a mar­ket­ing per­son, you want to get things above the thresh­old and if you’re an epi­demi­ol­o­gist, you want to get them below the thresh­old. But all the focus is on this threshold.

STONE: Right.

We’ve all heard of the tip­ping point. And it’s such an appeal­ing con­cept that once ideas reach that tip­ping point, they spread viral­ly just like a dis­ease. Think for a minute about how often you say some­thing went viral” when it real­ly has noth­ing what­so­ev­er to do with an actu­al virus.

Don’t feel bad. Watts found this to be a com­pelling par­al­lel, too.

WATTS: I’ve writ­ten sev­er­al papers about social con­ta­gion using dif­fer­ent kinds of math­e­mat­i­cal mod­els that have exact­ly the sort of bio­log­i­cal metaphor that I just described. So that’s how I was think­ing things worked as well. But when you look, when you real­ly look, you don’t see that. That’s just not how the world works.

STONE: His team dis­cov­ered this by study­ing the dif­fu­sion of near­ly a bil­lion sto­ries, videos, pic­tures, and peti­tions on Twit­ter. They learned that there real­ly is no patient zero” for social dis­sem­i­na­tion. Instead, they found anoth­er key player.

WATTS: The media is, first of all, an indis­pens­able ele­ment in any kind of social dif­fu­sion. You always have these enti­ties that are either for­mal media orga­ni­za­tions or increas­ing­ly online celebri­ties who have, at this point, many tens of mil­lions of fol­low­ers and effec­tive­ly act like media orga­ni­za­tions. And any­thing that becomes pop­u­lar invari­ably goes through these chan­nels, right? There’s no real equiv­a­lent in epidemiology.

STONE: So how did so many social sci­en­tists get this wrong for so long? The prob­lem, in large part, boils down to not hav­ing the right data to examine.

WATTS: Even though our men­tal mod­el of how things spread is: a per­son gets infect­ed with an idea or a new behav­ior or some­thing and then pass­es that along to the peo­ple that he or she inter­acts with, and they pass it along to peo­ple they inter­act with, and you can imag­ine this net­work in the back­ground and it’s kind of light­ing up as this enti­ty, this con­ta­gion spreads through it.

The kind of data that was avail­able was not that kind of data.

You want indi­vid­ual, per­son-to-per­son lev­el trans­mis­sion data, and what you actu­al­ly had his­tor­i­cal­ly was aggre­gate counts over time.

STONE: There was anoth­er big prob­lem with the data: the sin of omission.

WATTS: You’re try­ing to under­stand what makes some­thing suc­cess­ful, but you’re only study­ing suc­cess­ful things.

But of course, most of the things that suc­cess­ful peo­ple do are also done by unsuc­cess­ful peo­ple. So all suc­cess­ful peo­ple have break­fast, right? Maybe hav­ing break­fast is the key to being suc­cess­ful. Well, it turns out that’s not a very pre­dic­tive fea­ture. But you already know that if you study suc­cess­ful peo­ple and unsuc­cess­ful peo­ple. You have to have a total sam­ple of the pop­u­la­tion, and the same is true for diffusion.

You have to look at not only the things that do spread but also the things that don’t spread, which are mas­sive­ly more numerous.

STONE: It’s easy to see why this research is of great inter­est to mar­keters, who are eager to dif­fuse their ideas and prod­ucts to the masses.

But under­stand­ing net­works and how peo­ple inter­act with­in them is cru­cial for busi­ness lead­ers, as well. Take, for exam­ple, the goal of improv­ing how peo­ple com­mu­ni­cate with­in a com­pa­ny or get­ting teams to inter­act more effi­cient­ly when they tack­le a cre­ative challenge.

WATTS: If you think about the size of the econ­o­my and how much of the econ­o­my depends on firms and on teams with­in firms, it’s sort of a mul­ti-tril­lion-dol­lar ques­tion. If you could even improve effi­cien­cy by a small frac­tion, it would have a huge impact.

STONE: And, again, the advent of big data is key. Because just like the faulty notion of an epi­dem­ic tip­ping point for Tweets, lead­ers are mak­ing assump­tions about improv­ing their orga­ni­za­tions that the data may com­plete­ly invalidate.

If a team or firm is floun­der­ing, a CEO may be con­vinced of the need for lay­offs or a reor­ga­ni­za­tion. But are those tru­ly the best options?

WATTS: It’s real­ly sort of stun­ning how lit­tle we real­ly know about any of these things.

STONE: Instead of bas­ing these enor­mous­ly con­se­quen­tial deci­sions on intu­ition, lead­ers could use big data, Watts says. For exam­ple, he’s start­ing to ana­lyze email com­mu­ni­ca­tions. Even com­plete­ly anonymized data about how Per­son A com­mu­ni­cates fre­quent­ly with Per­son B but nev­er with Per­son Z could shed light on net­works of collaboration.

This type of data could be used to address a vari­ety of ques­tions about how we nur­ture suc­cess at work.

WATTS: These are going to be tricky ques­tions, it’s hard to define per­for­mance, it’s hard to mea­sure it, it’s real­ly hard to pre­dict it, but we might be able to build some­thing like a com­pre­hen­sive the­o­ry of per­for­mance that will allow man­agers to make more data-dri­ven deci­sions about, not just who to pro­mote, but how to reduce attri­tion, or how to com­pose teams, or when to move a team into an open office floor plan ver­sus keep­ing them in some oth­er kind of floor plan. I mean, these are all deci­sions that could be addressed with data, both obser­va­tion­al and exper­i­men­tal, and this is sort of a big project that we’re real­ly just get­ting start­ed on.

[music inter­lude]

STONE: We’ve been talk­ing about how data can help pre­dict suc­cess in a spe­cif­ic con­text: whether that Tweet will go viral, whether re-orga­niz­ing a team will improve communication.

But what can big data teach us about suc­cess more broadly?

Dashun WANG: Suc­cess by nature is a col­lec­tive phe­nom­e­non. What this means is that you can only be suc­cess­ful if every­body else thinks you’re successful.

STONE: That’s Dashun Wang, an asso­ciate pro­fes­sor of man­age­ment and orga­ni­za­tions at Kellogg.

WANG: If we start to accept that suc­cess or influ­ence fun­da­men­tal­ly is a col­lec­tive phe­nom­e­non, then what this means is that there must be fin­ger­prints in the data sur­round­ing how peo­ple react to that arti­fact, that inno­va­tion, that individual.

STONE: Wang’s research looks for those fin­ger­prints inside giant data sets. In one study, he and his col­leagues looked at suc­cess with­in the world of aca­d­e­m­ic publishing.

In the same way that the suc­cess of a Tweet depends in large part on how often it’s retweet­ed, the suc­cess of a sci­en­tif­ic paper depends in part on how often it’s cit­ed by oth­er academics.

But think about all the fac­tors that could go into this. Is a paper cit­ed more when there are more col­lab­o­ra­tors, or when the authors are from a more pres­ti­gious institution?

WANG: We can now pre­cise­ly mea­sure and mod­el this phe­nom­e­non. We’re able to show that in a sys­tem that we thought was very, very noisy and unpre­dictable, there’s deep reg­u­lar­i­ties under­ly­ing this system.

STONE: In oth­er words, they devel­oped a way to pre­dict how much of a hit” a giv­en paper would be among fel­low scientists.

They found three main fac­tors that dri­ve cita­tion suc­cess. The first is what Wang refers to as the rich get rich­er” phe­nom­e­non. When a paper gets cit­ed, it is con­sid­ered a well-cit­ed paper and thus gar­ners more cita­tions. The sec­ond is what he calls the aging effect.” New papers are fresh and excit­ing and get cit­ed more often than they do once they start to age. The third has to do with the actu­al ideas in the paper: Are they high-qual­i­ty? And how much of the sci­en­tif­ic com­mu­ni­ty could find them relevant?

WANG: What we find in our stud­ies is by com­bin­ing these fac­tors we’re able to build pre­cise math­e­mat­i­cal for­mu­las that are ana­lyt­i­cal­ly solved that will be able to help us pre­dict, under­stand what is the under­ly­ing for­mu­la that gov­erns how cita­tion is being cited.

STONE: Wang points out that once you know that these are the three crit­i­cal fac­tors, it makes sense. But if he had told you that cita­tion suc­cess depends on the num­ber of col­lab­o­ra­tors and how pres­ti­gious their insti­tu­tions are, you prob­a­bly would also say, sure, that makes sense. It’s only through ana­lyz­ing huge reams of data that we know which intu­itive mod­el is correct.

Fur­ther­more, Wang’s more recent research shows that these same fac­tors often dic­tate suc­cess in oth­er areas, too.

WANG: What becomes inter­est­ing is this idea of influ­ence or suc­cess, if you will, that’s real­ly gen­er­al­iz­able across many, many dif­fer­ent domains. They share a set of com­mon under­ly­ing fin­ger­prints and prin­ci­ples that they follow.

Think about tweets in the social media space. How things get viral? Think about how tech­nol­o­gy pen­e­trates a pop­u­la­tion. Or think about an indi­vid­ual, how some­one starts to pro­duce a lot of work and that by itself cre­ates a rich-get-rich­er effect.

STONE: Of course, con­duct­ing sci­ence is very dif­fer­ent from get­ting your ad to go viral. And, indeed, Wang stress­es that in some con­texts, oth­er fac­tors are also impor­tant. But these are gen­er­al­ly in addi­tion to, rather than in place of, the under­ly­ing prin­ci­ples that Wang uncovered.

Wang is quick to point out that while giant data sets are rich with poten­tial insights into human behav­ior, he would have no idea how to gain those insights with­out the com­pu­ta­tion­al tools that have been devel­oped over the past 15 or so years.

For exam­ple, con­sid­er a study that Wang recent­ly con­duct­ed that ana­lyzed data on mil­lions of mobile-phone users in three coun­tries: Who called whom, how often, and from where?

STONE: He used sophis­ti­cat­ed tools to ana­lyze this mas­sive dataset and saw a rela­tion­ship that nobody had found before: a link between the pat­terns in where we trav­el and who we call.

[music inter­lude]

STONE: Most of us tend to make lots of short trips from home, with the occa­sion­al long-dis­tance for­ay. The same pat­tern plays out in our com­mu­ni­ca­tions. We make lots of phone calls to peo­ple who live near­by — col­leagues, local busi­ness­es — and just a few to peo­ple who live fur­ther away — a fam­i­ly mem­ber once a week, or an old friend once a month.

In pop­u­la­tions where long trips are more com­mon, you can pre­dict that long-dis­tance phone calls will also be more com­mon. And vice versa.

WANG: What we real­ized is that these two aspects, pre­vi­ous­ly pur­sued as dif­fer­ent lines of inquiry, are actu­al­ly in fact con­nect­ed through pre­cise math­e­mat­i­cal for­mu­las, because they actu­al­ly rep­re­sent two facets of the same phe­nom­e­na. Then once we have one set of phe­nom­e­na, we’ll be able to derive the infor­ma­tion for the oth­er side.

It’s just fas­ci­nat­ing to see this kind of deep math­e­mat­i­cal rela­tion­ship in the human behavior.

STONE: OK, so sci­en­tists can pre­dict how far we’re like­ly to trav­el based on who we call; they can pre­dict whether Tweet A or aca­d­e­m­ic paper B will be a huge suc­cess while sim­i­lar ones go nowhere; they may soon be able to pre­dict whether our office will be more pro­duc­tive with an open floor plan. Is human nature real­ly so predictable?

Here’s Watts again.

WATTS: One reac­tion that peo­ple have when they hear about advances in com­pu­ta­tion­al social sci­ence is you know that it’s going to kill all the mys­tery in the world, right? That every­thing will be pre­dict­ed, and free will will dis­ap­pear, and human expe­ri­ence will sort of be reduced to algo­rithms and num­bers, and that just sounds sort of dis­mal, and there’ll be blanked-face social sci­en­tists kind of pulling the levers of soci­ety behind the scenes.

I don’t think that’s a plau­si­ble out­come. I’m more con­cerned that we won’t be able to fig­ure any­thing out, right? That every­thing will be so com­pli­cat­ed and so con­tin­gent and so depen­dent on ran­dom­ness and con­text that noth­ing will gen­er­al­ize at all. What I would like to see is that there are some things that we can fig­ure out well enough, that we can do bet­ter than just going with our guts, which is sort of how we’ve been doing it for­ev­er basically.

[music inter­lude]

STONE: This pro­gram was pro­duced by Jes­si­ca Love, Fred Schmalz, Emi­ly Stone, and Michael Spikes.

Spe­cial thanks to Microsoft’s Dun­can Watts and Kel­logg School pro­fes­sor Dashun Wang.

You can stream or down­load our month­ly pod­cast from iTunes, Google Play, or from our web­site, where you can read more on data ana­lyt­ics, col­lab­o­ra­tion, and lead­er­ship. Vis­it us at insight​.kel​logg​.north​west​ern​.edu. We’ll be back next month with anoth­er Kel­logg Insight podcast.

Featured Faculty

Dashun Wang

Associate Professor of Management & Organizations

Suggested For You

Most Popular

Organizations

How Are Black – White Bira­cial Peo­ple Per­ceived in Terms of Race?

Under­stand­ing the answer — and why black and white Amer­i­cans’ respons­es may dif­fer — is increas­ing­ly impor­tant in a mul­tira­cial society.

Careers

Pod­cast: Our Most Pop­u­lar Advice on Advanc­ing Your Career

Here’s how to con­nect with head­hunters, deliv­er with data, and ensure you don’t plateau professionally.

Most Popular Podcasts

Careers

Pod­cast: Our Most Pop­u­lar Advice on Improv­ing Rela­tion­ships with Colleagues

Cowork­ers can make us crazy. Here’s how to han­dle tough situations.

Social Impact

Pod­cast: How You and Your Com­pa­ny Can Lend Exper­tise to a Non­prof­it in Need

Plus: Four ques­tions to con­sid­er before becom­ing a social-impact entrepreneur.

Careers

Pod­cast: Attract Rock­star Employ­ees — or Devel­op Your Own

Find­ing and nur­tur­ing high per­form­ers isn’t easy, but it pays off.

Marketing

Pod­cast: How Music Can Change Our Mood

A Broad­way song­writer and a mar­ket­ing pro­fes­sor dis­cuss the con­nec­tion between our favorite tunes and how they make us feel.