Organizations May 1, 2023
Today’s Gig Workers Are Subject to Endless Experimentation
“It raises the question, do we want to be a society where experimentation is just the norm?”
Experimentation is thriving in the digital age. Now that algorithms can be marshalled in service of conducting large-scale, low-cost experiments, the scope and pervasiveness of workplace experimentation has reached a level that, not so long ago, would’ve been unfathomable.
Uber, for example, has announced that it has over one thousand experiments running on its platform at any given time. LinkedIn recently subjected 20 million users to thousands of experiments (without their knowledge) over a five-year period.
It’s not just the app’s customers who are affected; it’s also the people who rely on these apps for gig work. Some research estimates that nearly one-fifth of people in the United States used a digital platform to find work in 2021.
How might all this experimentation be affecting workers?
A new paper from Hatim Rahman of Kellogg, Tim Weiss of Imperial College London, and Arvind Karunakaran of Stanford takes up this question.
The research team analyzed nearly two decades of qualitative data from QuickHire, a pseudonym for one of the world’s largest digital labor platforms. They discovered that over their study period, QuickHire had moved through three “experimentation regimes,” which were characterized by distinctly different approaches to experimentation on workers—and very different responses from workers themselves. At first, experimentation was explicit and conducted only on those who had voluntarily opted into it; in the second stage, experimentation became concealed, with workers participating in studies without their notification or consent; and in the final stage, experimentation was unbounded, with many experiments running simultaneously and continuously.
The researchers came to think of these successive shifts in approach as an “experimental hand” that, they write, reshapes workers’ sense of autonomy “by initially increasing it, then diminishing it, and finally by normalizing its diminishment.”
Rahman notes that the prevalence of pervasive, quiet experimentation is increasing outside of digital-platform settings, too; even public-policy ideas can be tested on unwitting subjects in this way. And while Rahman understands the benefits of rigorous experimentation, he fears that if left unchecked, its overuse could start to weaken the social fabric by eroding trust.
“It raises the question, do we want to be a society where experimentation is just the norm,” he says, “and people don’t necessarily have any say in, or awareness about, what’s being experimented on?”
Taking a long view
The paper builds on Rahman’s previous research, which has also examined the experiences of gig workers on digital platforms in an effort to reveal how opaque rating algorithms affect workers and their behavior on the job.
Rahman says the newest project grew from a desire shared among the study’s coauthors: to look further back in time to better understand the present.
“We are generally curious about how we arrived where we are now, where we have this unbounded experimentation,” Rahman says. “It’s unlikely that it just happened all of a sudden.”
They decided to take a more longitudinal approach in their study of QuickHire, a digital platform on which gig workers around the globe can register to provide services like software development, marketing, and graphic design to clients. The researchers collected extensive data from the platform between 2004 (when QuickHire launched its discussion board) and 2020. These data included discussion-board conversations among QuickHire workers and moderators, and archival records of QuickHire’s terms of service, company announcements, and blog posts. For the period between 2014 and 2020, the data also included Rahman’s own observations as a registered client and worker.
“As platforms or organizations gain market advantage and power, they’re no longer incentivized to make [experimentation] mutually beneficial.”
Tracing QuickHire’s history in this way revealed to the researchers just how significant a role experimentation seemed to play in workers’ experience on the platform—and how dramatically that role shifted with time.
Culling from company announcements and discussion-board posts, the researchers determined that QuickHire had launched its first worker experiments in about 2007, thus beginning the era the researchers termed the “explicit experimentation regime.” In this phase, QuickHire offered details about the features they were testing—for example, a change to the platform’s “job search,” which would update how workers and clients found one another. In its public discussion-board post about this experiment, QuickHire invited workers to “beta test” the new feature and provide feedback. During this period, QuickHire even solicited worker input on what parts of the platform to subject to experimentation. Workers could either consent or opt out of experiments.
Not only did this experimental phase increase worker autonomy—since workers had control over their participation in experiments and the ability to voice their opinions about how those experiments should work—but it also seemingly established a precedent for what workers could expect going forward.
“This regime established workers’ baseline expectations about how platform-based experimentation worked,” the authors wrote.
Starting in about 2014, the researchers noticed a shift. Around this time, workers had coincidentally discovered that QuickHire was experimenting with the design and display of workers’ performance metrics on their profiles—seemingly benign changes that, as one worker put it, were nonetheless “going to be the very first thing that clients notice on my profile.” Later, the company acknowledged that it had indeed been testing the design for a group of workers who hadn’t been notified. QuickHire followed a similar order of operations when it launched another concealed experiment: the platform began sending some users automated messages threatening to suspend their accounts when they communicated with clients or workers outside of the site.
These events ushered in the “concealed experimentation regime,” in which QuickHire ran experiments without workers’ consent. During this phase, the researchers saw workers’ autonomy decrease, as their ability to control their participation in experiments (and related work features) diminished.
In 2017, something changed again: experiments on the platform were no longer as episodic or issue-focused; they lacked a clear end point and encompassed a range of platform features. The researchers’ data showed that QuickHire subjected workers to concurrent tests of search filters, pricing suggestions, communication tools, and the information displayed in worker profiles, among others. Workers voiced that “all that new stuff” was changing “all the time” and affected their ability to do their jobs.
Unlike in past experimentation regimes, the platform offered no acknowledgement—belated or otherwise—of its experimentation on workers. Rahman and his colleagues were surprised by how workers met this “unbounded experimentation regime”: quietly, without pushing back or leaving the platform en masse.
“Rather than observing collective worker exit, we found workers responded with resigned acceptance,” the researchers wrote, “as they had come to expect that they will be subjected to experimentation every time they used the platform.”
Rahman says the order of the phases in this experimental regime change is important to note—even though it likely wasn’t intentional on the part of QuickHire. After initially securing buy-in for worker experimentation, the platform proceeded into the concealed and unbounded experiments simply because it could.
“I do think there’s this initial notion of wanting to begin by getting buy-in and scaling [experimentation] in ways that are mutually beneficial,” Rahman says. “But as platforms or organizations gain market advantage and power, they’re no longer incentivized to make it mutually beneficial.”
The Results of Past Experiments
Still, organizations do themselves a disservice by failing to account for the long-term impacts of concealed and unbounded experimentation, Rahman argues. For one thing, the integrity of experiments themselves can be undermined by jaded participants who expect any small change they see to be part of some larger test. They can also pose a challenge to morale; the researchers found “indications of worker anxiety, frustration, and stress” among the gig workers.
In their paper, the research team recommends that organizations form internal boards—or external oversight units—to oversee experimentation and ensure helpful, insightful outcomes for both leaders and workers.
The movement for fairness for experimental subjects isn’t a new one, Rahman points out, and digital platforms like QuickHire would do well to learn from similar movements in medicine and academia.
“This isn’t something we have to imagine,” Rahman says. “Look at medicine: you can still do randomized control trials, but you have to have very explicit informed consent from subjects. It’s the same with academia.”
Eventually, it’s possible that regulators will force the point. States are already beginning to pass data privacy laws giving users more control over the information that online platforms can collect about them. Rahman can imagine something similar happening around experimentation and informed consent.
At a minimum, he argues, we need more transparency about the nature of these experiments. A natural starting point could be requiring websites that ask users to accept cookies to be explicit about the consequences of opting in or not.
But in the meantime, he says, organizations have a decision to make: “Do you want to wait and be reactive about this? Or do you want to be at the forefront of trying to think of more mutually beneficial ways of implementing experimentation?”
Katie Gilbert is a freelance writer in Philadelphia.
Rahman, Hatim, Tim Weiss, and Arvind Karunakaran. 2023. "The Experimental Hand: How Platform-Based Experimentation Reconfigures Worker Autonomy." Academy of Management Journal.
One Key to a Happy Marriage? A Joint Bank Account.Merging finances helps newlyweds align their financial goals and avoid scorekeeping.
Take 5: Yikes! When Unintended Consequences StrikeGood intentions don’t always mean good results. Here’s why humility, and a lot of monitoring, are so important when making big changes.
How Are Black–White Biracial People Perceived in Terms of Race?Understanding the answer—and why black and white Americans may percieve biracial people differently—is increasingly important in a multiracial society.
Will AI Eventually Replace Doctors?Maybe not entirely. But the doctor–patient relationship is likely to change dramatically.
Entrepreneurship Through Acquisition Is Still EntrepreneurshipETA is one of the fastest-growing paths to entrepreneurship. Here's how to think about it.
Take 5: Research-Backed Tips for Scheduling Your DayKellogg faculty offer ideas for working smarter and not harder.
How to Manage a Disengaged Employee—and Get Them Excited about Work AgainDon’t give up on checked-out team members. Try these strategies instead.
Which Form of Government Is Best?Democracies may not outlast dictatorships, but they adapt better.
What Went Wrong at AIG?Unpacking the insurance giant's collapse during the 2008 financial crisis.
The Appeal of Handmade in an Era of AutomationThis excerpt from the book “The Power of Human" explains why we continue to equate human effort with value.
2 Factors Will Determine How Much AI Transforms Our EconomyThey’ll also dictate how workers stand to fare.
When Do Open Borders Make Economic Sense?A new study provides a window into the logic behind various immigration policies.
Why Do Some People Succeed after Failing, While Others Continue to Flounder?A new study dispels some of the mystery behind success after failure.
Sitting Near a High-Performer Can Make You Better at Your Job“Spillover” from certain coworkers can boost our productivity—or jeopardize our employment.
How the Wormhole Decade (2000–2010) Changed the WorldFive implications no one can afford to ignore.
What’s at Stake in the Debt-Ceiling Standoff?Defaulting would be an unmitigated disaster, quickly felt by ordinary Americans.
What Happens to Worker Productivity after a Minimum Wage Increase?A pay raise boosts productivity for some—but the impact on the bottom line is more complicated.
Immigrants to the U.S. Create More Jobs than They TakeA new study finds that immigrants are far more likely to found companies—both large and small—than native-born Americans.
How Has Marketing Changed over the Past Half-Century?Phil Kotler’s groundbreaking textbook came out 55 years ago. Sixteen editions later, he and coauthor Alexander Chernev discuss how big data, social media, and purpose-driven branding are moving the field forward.
3 Traits of Successful Market-Creating EntrepreneursCreating a market isn’t for the faint of heart. But a dose of humility can go a long way.