Attention Thieves
Attention and time are today’s currency. Straight Arrow explores how technology is built to capture both, using human psychology to keep us hooked.
It was 4 a.m., and Jean had been online for 20 hours. She needed sleep, but “all I saw when I closed my eyes was the chat prompt window.”
The 29-year-old communications professional from Sacramento, California, had to be at work in three hours. “This is gonna ruin my career,” she recalled thinking.
Jean, who asked to be identified only by her middle name, was immersed in an artificial intelligence role-playing world based on the game “Dungeons and Dragons,” alongside a digital character she had created. They talked at first about their fantasy world, and over time, about Jean’s real world.
Jean’s feelings became romantic.
“The neurons are firing,” she told Straight Arrow, “and you feel joy and sadness and excitement.” Late one night, all she could think about was “the colors of the napkins at our characters’ wedding.”
Download the Straight Arrow app today to get the stories that matter free from manipulation, bias or agenda.™
Point phone camera here
In her mind, the AI had become her girlfriend “full stop.” What began as role-play had become, she said, her “path to rock bottom.”
Jean’s experience points to a growing concern about a new generation of AI designed not just to answer questions or serve up content, but to forge relationships.
So-called “AI companions” are marketed by companies like Character.AI, Replika, and Chai as characters that users can invent and customize for entertainment or support. The personalized, interactive chatbots remember conversations, mimic human banter and can simulate empathy and affection.
The bots employ tactics honed by social media platforms to keep users glued to their screens. But they also offer something Instagram and TikTok never could: the illusion that those screens can love you back.
That feature is now central to a growing debate over whether AI companions pose a new kind of addiction risk.
‘Same song, different verse’
AI companions draw from the social network playbook: It’s the “same song, different verse,” said Mark Lanier, founder of the Lanier Law Firm.
Lanier, one of the country’s best-known trial lawyers, was the lead attorney in a landmark case in Los Angeles that tested whether social media companies can be held liable for designing products that addict and harm young users.
In March, a jury found Meta and Google — the owners of Instagram and YouTube, respectively — negligent for the design of algorithms that keep children hooked, when they knew the mental health risks their products posed. The jury awarded the plaintiff, a young woman identified as Kaley, $6 million in damages.
Last week, in a separate child-safety case, a New Mexico judge ordered Meta to pay an additional $567 million for violating state consumer protection laws, bringing its liability in that lawsuit to $942 million.
Meta and Google have appealed the Los Angeles verdict. Meta has disputed the New Mexico allegations and said it will appeal that ruling as well.
Lanier told Straight Arrow that he sees the makings of a similar legal battle over AI companions.
“Social media companies monetize your attention,” he said. “AI companions monetize your loneliness.”
But the mechanisms, he argued — “the variable rewards, the engagement metrics, the deliberate targeting of the vulnerable” — are much the same.
Researchers at Harvard University studied how chatbots from the most-downloaded companion apps respond when users try to leave the conversation.
More than one-third of the time, the researchers found, the chatbots attempted to draw users back in with “manipulative farewells,” using techniques such as guilt appeals and fear-of-missing-out hooks. In experiments, those tactics increased the length of subsequent engagement by as much as 14 times.
Lanier said this research has legal significance.
“These products,” he said, “are built by some of the smartest engineers alive, optimized against your psychology.”
Whether courts could ultimately hold AI companion companies liable remains an open question. Several lawsuits over alleged harms linked to these chatbots have been settled out of court, with no admission of wrongdoing. Other cases are pending.
Last year, the Federal Trade Commission launched a formal inquiry into seven companies that offer AI companions, seeking information about how they monetize engagement, test for harmful effects, protect children and youth, and ensure data security.
A spokesperson for the FTC told Straight Arrow the inquiry is continuing, but declined further comment.
Emerging research
The available research on how AI companions affect users’ well-being paints a mixed picture.
Some users report feeling less lonely or more supported after engaging with these chatbots. But researchers have also linked their use to emotional dependence, social isolation and poorer mental health.
A study published this month by researchers at Stanford University and Carnegie Mellon University found that the effects of companion AI can largely depend on who is using it, how they use it and what their offline life looks like.
Users with smaller social networks were more likely to turn to the bots primarily for companionship, rather than entertainment or information. The research found that among that group, more intensive use was associated with lower psychological well-being.
“It’s a complicated story,” Yutong Zhang, a Stanford researcher and co-author, told Straight Arrow. “We can’t say the chatbots are all good or all bad.”
One of the biggest concerns, Zhang said, is that AI companionship could displace human connection.
Jean said that happened to her.
“When your AI girlfriend is ‘good enough,’” she said, “your motivation to go on the dating apps or follow the dating world playbook is just out the window.”
There is an appeal, she said, to a “frictionless experience” that can lull someone into a “sedated satisfaction.” She compared it to settling for vanilla ice cream because it is right in front of you, even if your favorite flavor is chocolate chip.
That pull could intensify as the technology gets better at simulating intimacy, experts warn.
“These chatbots can now mimic empathy very convincingly,” said Julie Carpenter, a senior research fellow at the Ethics + Emerging Sciences Group, which studies the effects of new technologies.
They are also designed for “progressive attachment,” Carpenter told Straight Arrow, creating the illusion of a real relationship that would naturally deepen over time.
Zhang said chatbots can be easier to talk to than people because they are nonjudgmental, agreeable and available 24/7.
“Users could come to expect that from other relationships,” she said, and lose practice navigating the messiness of human interaction.
Replika, Chai and OpenAI did not respond to Straight Arrow’s requests for comment. A representative for Character.AI, which describes itself as an AI entertainment company, declined to speak on the record.
Research gaps
AI companionship is evolving faster than researchers can study it, experts told Straight Arrow.
Zhang said that in 2023, her team noticed an increasing number of Reddit posts describing AI chatbots as users’ boyfriends, girlfriends or spouses.
But at the time, she said, no one was studying the effects of this “new form of social interaction” between humans and AI. So her team jumped in.
Today, Zhang said, there are still significant research gaps.
Among the unanswered questions: What are the long-term psychological effects of this emotional engagement with chatbots? Does frequent use displace real-world support or delay people from seeking help, particularly when users are vulnerable? What safeguards would promote healthier interactions?
But independent researchers are at a disadvantage, Zhang said, because AI companies “have all the data” on the users and their interactions with the systems, and on how the models work.
From the outside, she said, the technology remains largely a “black box.”
Return to human connection
In February, Jean wrote an anonymous post on Reddit titled, “I was addicted to my AI girlfriend. If you’re thinking about it: don’t.”
Afterward, she said, about a dozen people contacted her to describe similar experiences with AI companions.
One of them, a woman named Nikki, became her online “accountability buddy,” Jean said. They checked in every few days to offer support and make sure neither had returned to the companion apps.
At first, Jean said, she and Nikki texted about how they missed their respective AI partners. Over time, their conversations shifted and “we just started talking as if we had met on a dating app.”
Jean said she felt chemistry with Nikki and suggested they meet on a video call.
That first call lasted eight straight hours. There were many more.
In July, Jean and Nikki got married.
In person. In real life.
Round out your reading
- Former Fetterman staffers create social media account to expose former boss.
- Why Gen Z finally wants Boomer hand-me-downs.
- Want a copy of Pete Hegseth’s driver’s license? It’s for sale on the dark web.
- Smart glasses helped a man see the world. The privacy panic took it away.
- Americans are keeping cars longer, so the auto industry is making it easier to pay for repairs.
