Today’s Gig Workers Are Subject to Endless Experimentation

Uber, for example, has announced that it has over one thousand experiments running on its platform at any given time. LinkedIn recently subjected 20 million users to thousands of experiments (without their knowledge) over a five-year period.

It’s not just the app’s customers who are affected; it’s also the people who rely on these apps for gig work. Some research estimates that nearly one-fifth of people in the United States used a digital platform to find work in 2021.

How might all this experimentation be affecting workers?

A new paper from Hatim Rahman of Kellogg, Tim Weiss of Imperial College London, and Arvind Karunakaran of Stanford takes up this question.

The research team analyzed nearly two decades of qualitative data from QuickHire, a pseudonym for one of the world’s largest digital labor platforms. They discovered that over their study period, QuickHire had moved through three “experimentation regimes,” which were characterized by distinctly different approaches to experimentation on workers—and very different responses from workers themselves. At first, experimentation was explicit and conducted only on those who had voluntarily opted into it; in the second stage, experimentation became concealed, with workers participating in studies without their notification or consent; and in the final stage, experimentation was unbounded, with many experiments running simultaneously and continuously.

The researchers came to think of these successive shifts in approach as an “experimental hand” that, they write, reshapes workers’ sense of autonomy “by initially increasing it, then diminishing it, and finally by normalizing its diminishment.”

Rahman notes that the prevalence of pervasive, quiet experimentation is increasing outside of digital-platform settings, too; even public-policy ideas can be tested on unwitting subjects in this way. And while Rahman understands the benefits of rigorous experimentation, he fears that if left unchecked, its overuse could start to weaken the social fabric by eroding trust.

“It raises the question, do we want to be a society where experimentation is just the norm,” he says, “and people don’t necessarily have any say in, or awareness about, what’s being experimented on?”

Taking a long view

The paper builds on Rahman’s previous research, which has also examined the experiences of gig workers on digital platforms in an effort to reveal how opaque rating algorithms affect workers and their behavior on the job.

Rahman says the newest project grew from a desire shared among the study’s coauthors: to look further back in time to better understand the present.

“We are generally curious about how we arrived where we are now, where we have this unbounded experimentation,” Rahman says. “It’s unlikely that it just happened all of a sudden.”

They decided to take a more longitudinal approach in their study of QuickHire, a digital platform on which gig workers around the globe can register to provide services like software development, marketing, and graphic design to clients. The researchers collected extensive data from the platform between 2004 (when QuickHire launched its discussion board) and 2020. These data included discussion-board conversations among QuickHire workers and moderators, and archival records of QuickHire’s terms of service, company announcements, and blog posts. For the period between 2014 and 2020, the data also included Rahman’s own observations as a registered client and worker.

Tracing QuickHire’s history in this way revealed to the researchers just how significant a role experimentation seemed to play in workers’ experience on the platform—and how dramatically that role shifted with time.

Culling from company announcements and discussion-board posts, the researchers determined that QuickHire had launched its first worker experiments in about 2007, thus beginning the era the researchers termed the “explicit experimentation regime.” In this phase, QuickHire offered details about the features they were testing—for example, a change to the platform’s “job search,” which would update how workers and clients found one another. In its public discussion-board post about this experiment, QuickHire invited workers to “beta test” the new feature and provide feedback. During this period, QuickHire even solicited worker input on what parts of the platform to subject to experimentation. Workers could either consent or opt out of experiments.

Not only did this experimental phase increase worker autonomy—since workers had control over their participation in experiments and the ability to voice their opinions about how those experiments should work—but it also seemingly established a precedent for what workers could expect going forward.

“This regime established workers’ baseline expectations about how platform-based experimentation worked,” the authors wrote.

Starting in about 2014, the researchers noticed a shift. Around this time, workers had coincidentally discovered that QuickHire was experimenting with the design and display of workers’ performance metrics on their profiles—seemingly benign changes that, as one worker put it, were nonetheless “going to be the very first thing that clients notice on my profile.” Later, the company acknowledged that it had indeed been testing the design for a group of workers who hadn’t been notified. QuickHire followed a similar order of operations when it launched another concealed experiment: the platform began sending some users automated messages threatening to suspend their accounts when they communicated with clients or workers outside of the site.

These events ushered in the “concealed experimentation regime,” in which QuickHire ran experiments without workers’ consent. During this phase, the researchers saw workers’ autonomy decrease, as their ability to control their participation in experiments (and related work features) diminished.

In 2017, something changed again: experiments on the platform were no longer as episodic or issue-focused; they lacked a clear end point and encompassed a range of platform features. The researchers’ data showed that QuickHire subjected workers to concurrent tests of search filters, pricing suggestions, communication tools, and the information displayed in worker profiles, among others. Workers voiced that “all that new stuff” was changing “all the time” and affected their ability to do their jobs.

Unlike in past experimentation regimes, the platform offered no acknowledgement—belated or otherwise—of its experimentation on workers. Rahman and his colleagues were surprised by how workers met this “unbounded experimentation regime”: quietly, without pushing back or leaving the platform en masse.

“Rather than observing collective worker exit, we found workers responded with resigned acceptance,” the researchers wrote, “as they had come to expect that they will be subjected to experimentation every time they used the platform.”

Rahman says the order of the phases in this experimental regime change is important to note—even though it likely wasn’t intentional on the part of QuickHire. After initially securing buy-in for worker experimentation, the platform proceeded into the concealed and unbounded experiments simply because it could.

“I do think there’s this initial notion of wanting to begin by getting buy-in and scaling [experimentation] in ways that are mutually beneficial,” Rahman says. “But as platforms or organizations gain market advantage and power, they’re no longer incentivized to make it mutually beneficial.”

The Results of Past Experiments

Still, organizations do themselves a disservice by failing to account for the long-term impacts of concealed and unbounded experimentation, Rahman argues. For one thing, the integrity of experiments themselves can be undermined by jaded participants who expect any small change they see to be part of some larger test. They can also pose a challenge to morale; the researchers found “indications of worker anxiety, frustration, and stress” among the gig workers.

In their paper, the research team recommends that organizations form internal boards—or external oversight units—to oversee experimentation and ensure helpful, insightful outcomes for both leaders and workers.

The movement for fairness for experimental subjects isn’t a new one, Rahman points out, and digital platforms like QuickHire would do well to learn from similar movements in medicine and academia.

“This isn’t something we have to imagine,” Rahman says. “Look at medicine: you can still do randomized control trials, but you have to have very explicit informed consent from subjects. It’s the same with academia.”

Eventually, it’s possible that regulators will force the point. States are already beginning to pass data privacy laws giving users more control over the information that online platforms can collect about them. Rahman can imagine something similar happening around experimentation and informed consent.

At a minimum, he argues, we need more transparency about the nature of these experiments. A natural starting point could be requiring websites that ask users to accept cookies to be explicit about the consequences of opting in or not.

But in the meantime, he says, organizations have a decision to make: “Do you want to wait and be reactive about this? Or do you want to be at the forefront of trying to think of more mutually beneficial ways of implementing experimentation?”