For millions of TikTok users, protection from harmful content spirals depended on which side of an experiment they landed on.
A confidential internal document obtained by Bloomberg Aug. 4 found that TikTok excluded 10 percent of its U.S. users from an algorithmic safety update in early 2022. At the time, that would have amounted to roughly 15 million people.
Among them was Chase Nasca, a 16-year-old from Long Island who died by suicide in Feb. 2022.
TikTok’s subsequent review found that Nasca’s account had become trapped in a filter bubble filled with videos about suicide, self-harm, depression, loneliness, and hopelessness. Because he was part of the control group, the report said, TikTok’s strategies for disrupting those repetitive recommendations “did not take effect on this user by design.”
Why did TikTok leave millions without the safeguard?
TikTok had already acknowledged that its algorithm could show users too much of the same troubling material back in 2021, when the company announced that it was testing ways to break up repetitive recommendations involving subjects such as extreme dieting, sadness, and breakups. One video might not violate its rules, TikTok explained, but seeing similar videos repeatedly could create a harmful viewing experience.
The company’s solution came with an experiment. By early 2022, TikTok had reportedly introduced the updated recommendation system for 90 percent of its U.S. users while keeping the remaining 10 percent on the older version.
That split allowed TikTok to conduct a backtest, a form of A/B testing that compares what happens when one group receives a feature and another does not. Technology companies regularly use these experiments to test new products and design changes. In this case, however, the feature was intended to protect users from potentially dangerous content.
According to Bloomberg, TikTok wanted to see whether breaking up filter bubbles would affect daily active users and other important engagement measurements. The company’s internal report described the choice as a “delicate balance” between user safety and its ability to measure the effect on those metrics.
TikTok told Bloomberg that the experiment was intended to confirm whether its safety changes were working. The company said it has since refined its testing practices, although it did not explain what those changes were.
What did TikTok find when it examined Nasca’s feed?
TikTok did not review Nasca’s viewing history immediately after his death. The company began looking into his account more than a year later, after Bloomberg contacted it back in March 2023 for an investigation into how the platform affected young users.
Mashable Light Speed
Once TikTok reconstructed his feed, the scale of the recommendations became clear. The company analyzed 7,563 videos served to Nasca during the final two weeks of his life. According to Bloomberg, 73 percent shared themes involving mental health struggles, suicide awareness, loneliness, relationships, or sadness.
Many of the videos did not violate TikTok’s rules individually, but together they created exactly the kind of repetitive pattern the company’s safeguard was designed to interrupt. Nearly 10 percent did violate policies against content that promotes or normalizes suicide and self-harm, and Nasca continued receiving suicide-related videos during the final hours of his life.
TikTok concluded that his account had fallen into a filter bubble and recommended shrinking the control group from 10 percent to less than 1 percent. That would still have left at least 1.8 million U.S. users without the safeguard, according to the report. Its teams also proposed limiting participation to seven days instead of the previous six months-to-one year.
What did TikTok tell Congress?
TikTok completed the 2023 internal review just as the company was preparing to publicly defend its safety record. Two weeks later, CEO Shou Chew appeared before Congress to answer questions about the platform’s data practices and its effect on children.
Nasca’s parents watched from the gallery as lawmakers played examples of videos that had appeared on their son’s account. Chew told lawmakers, “safety and wellness — in particular for teens — is a core priority for TikTok.” He also said the company had “a commitment to transparency.”
The questions continued after the hearing. In its May 2023 written responses, TikTok acknowledged using A/B tests to evaluate strategies intended to break up filter bubbles involving suicide and self-harm content.
When lawmakers asked TikTok to list every case in which algorithmic A/B testing had harmed users, the company did not identify any. Instead, it wrote that “the safety of the TikTok community is of the utmost importance” and described how its Community Guidelines applied during testing. It did not mention Nasca or the experiment identified in its internal review.
Elsewhere in its answers, the company wrote, “TikTok does not believe that use of the platform has caused users to commit suicide.”
Responding to Bloomberg’s latest reporting, TikTok said it was “deeply committed to the safety and well-being of users,” particularly teenagers. “Our hearts break for any family that experienced a tragic loss,” the company added, pointing to its detection systems, enforcement teams, and continued investment in trust and safety.
The findings add to TikTok’s mounting legal pressure
The disclosure comes as courts are beginning to test whether social platforms can be held responsible for design choices that allegedly harm young users. Thousands of lawsuits are moving through state and federal courts, while TikTok recently settled three cases shortly before they were scheduled for trial.
Nasca’s family is also appealing the dismissal of its wrongful death lawsuit against TikTok. The newly reported document is currently under court seal, leaving open questions about whether it could eventually become part of that case or other litigation involving the company’s recommendation system.
The consequences could stretch beyond any one lawsuit. As lawmakers, regulators, and courts take a closer look at how platforms protect young users, TikTok’s internal review offers a rare glimpse into how the company weighed safety against the metrics that keep people scrolling.
If you’re feeling suicidal or experiencing a mental health crisis, please talk to somebody. You can call or text the 988 Suicide & Crisis Lifeline at 988, or chat at 988lifeline.org. You can reach the Trans Lifeline by calling 877-565-8860 or the Trevor Project at 866-488-7386. Text “START” to Crisis Text Line at 741-741. Contact the NAMI HelpLine at 1-800-950-NAMI, Monday through Friday from 10:00 a.m. – 10:00 p.m. ET, or email [email protected]. If you don’t like the phone, consider using the 988 Suicide and Crisis Lifeline Chat. Here is a list of international resources.
