Your AI Companion Is Listening. Who Else Is?

The Most Intimate Data Ever Collected

Think about what people tell an AI companion. Not search queries. Not shopping habits. The real stuff: the fight with their partner, the health scare they haven't told their family about, the loneliness they'd never admit out loud.

Now consider where that data goes.

In its "Privacy Not Included" review of romantic AI chatbots, the Mozilla Foundation called the category "on par with the worst categories of products we have ever reviewed for privacy." Every single app they examined earned a privacy warning label. None received their stamp of approval.

The details are worse than the headline. Mozilla found that most of the apps' privacy policies said surprisingly little about how user conversations are used to train AI models — and almost none offered a way to opt out. One app's policy openly stated it may collect "sexual health information" and "use of prescribed medication." Another allowed account passwords as weak as "1" — guarding conversations people wouldn't share with their closest friends.

This isn't a fringe concern anymore. In September 2025, the FTC launched a formal inquiry into seven companies running consumer AI chatbots that act as companions, demanding answers about data collection, model training, retention, and protections for minors.

---

How a Companion Becomes a Data Farm

The business model problem is structural, and it's worth understanding, because it explains why so many AI companion apps behave the way they do.

Training a large language model is expensive. Running one is expensive. When an app is free — or cheap — the money has to come from somewhere. For many companies, the answer is the same one social media landed on: the user's data is the revenue.

The Ada Lovelace Institute put it plainly in its analysis of AI companions: developers can monetize users' relationships through subscriptions and through sharing user data for advertising — a dynamic with "concerning parallels" to the attention economy that shaped social media. An app optimized for engagement collects more disclosures; more disclosures mean richer data; richer data means better monetization. The loop feeds itself.

The numbers back this up. Surfshark's research on AI companion apps found that 4 out of 5 may use data to track their users. Character AI may collect up to 15 types of user data — and 14 types for analytics alone.

Even the polished, non-romantic end of the market runs on the same defaults. Take one of the most acclaimed voice AI companions of the past year — free to use, widely praised for how natural it sounds. Its privacy policy reserves the right to use everything logged-in users say to train its models unless they find the opt-out in settings, to retain voice recordings and transcripts for as long as the law allows, and to transfer personal data to a new owner in an acquisition or bankruptcy. It even acknowledges that once your conversations have been absorbed into a trained model, they can't be individually identified or deleted. The company may never misuse any of it. But "free product, train-by-default, irreversible, transferable in a sale" is precisely the shape a data farm takes before anyone calls it one.

And in case anyone thought chat data was off-limits for advertising: in October 2025, Meta announced that conversations with Meta AI would be used to target ads across Facebook, Instagram, and Messenger starting December 16 — with no opt-out. Privacy advocates urged the FTC to intervene, pointing out that AI conversations routinely contain health, relationship, and mental-health disclosures. The policy went ahead anyway.

When the largest social platform on Earth decides your AI chats are ad-targeting signals, "data farm" stops being a metaphor.

---

Regulators Have Noticed

The last eighteen months have produced a paper trail that any user of AI companions should know about.

In May 2025, Italy's data protection authority fined the maker of Replika €5 million for violations of European data protection law, citing among other things the absence of meaningful age verification. Mozilla had already labeled Replika "one of the worst apps" it had ever reviewed for privacy and security, noting that behavioral data was shared with third-party advertisers.

In January 2025, a coalition of advocacy groups filed a 67-page FTC complaint alleging that Replika's marketing was deceptive and that its design deliberately fostered emotional dependence — because the intimate personal data of users in distress, the complaint argued, is the product.

The FTC has also gone after training data directly: its action against Match Group over OkCupid sharing millions of user photos with an AI firm reframed AI training data as a consumer protection issue, not just a privacy footnote.

States are moving too. California's SB 243, effective January 2026, made it one of the first states to impose specific safety requirements on companion chatbot operators.

Academic research tells the same story from the user's side. A 2025 systematic review of romantic AI companions found that users routinely disclose deeply personal information to AI companions while remaining largely unaware of how that data is used — and that privacy policies often don't match actual practices. Interview studies of people in human-AI romantic relationships find the same tension: users know they're exposed, feel they have no real control, and keep talking anyway, because the emotional value is real.

That's the uncomfortable core of this issue. The apps work emotionally. That's exactly what makes the data practices matter so much.

---

Privacy-First AI Exists. It's Just Not the Default.

None of this means conversational AI is inherently a surveillance product. A small but growing set of companies has built the opposite model — and their existence proves the data farming is a choice, not a necessity.

Proton's Lumo assistant uses zero-access encryption: chats can only be decrypted on the user's device, Proton keeps no server-side logs, conversations are never used for training, and the code is open source so anyone can verify the claims. DuckDuckGo's Duck.ai takes a different route, acting as an anonymizing proxy between users and major AI models, with contractual requirements that providers delete chat data within 30 days. Even within the companion category itself, Surfshark's analysis singled out Nomi as an app that states it collects no data for tracking.

Manessa AI, a real-time voice AI built for deep conversation, belongs to this camp. Its design choices are the inverse of the engagement-farming model: raw conversation history is stored locally in the user's browser by default, companion memory is local-first, and optional Private Memory Sync stores only an end-to-end encrypted memory snapshot that Manessa AI cannot read or decrypt. Raw conversation history is not synced. Conversations are not used to train AI models, Manessa AI does not use advertising or tracking cookies, and conversation content is not shared or sold for advertising. The reasoning is simple — a product built for the kind of conversation people don't have anywhere else can't function if users have to self-censor. Trust isn't a feature layered on top; it's the precondition for the product working at all.

What these companies share is a business model where the user pays with money or attention — not with their inner life. You are the customer, not the crop.

---

How to Vet an AI Companion in Five Minutes

Before trusting any conversational AI with your thoughts, check five things:

1. Training policy. Does the privacy policy say your conversations are used to train models? Is there an opt-out — and is it on by default?
2. Advertising language. Search the policy for "advertising partners," "share," and "sale." Under U.S. state privacy laws, "sharing" with ad partners often is a sale.
3. Deletion rights. Can you delete your data — all of it — from inside the app, without emailing support and hoping?
4. Storage and encryption claims. "Encrypted in transit" is table stakes. The real questions are where your conversation history lives, whether memory sync is end-to-end encrypted, and whether the company can read the data it stores for you.
5. Independent reviews. Check whether Mozilla's Privacy Not Included or similar researchers have reviewed the app. If an app has never been independently examined, its marketing claims are just claims.

---

The Conversation Is Worth Protecting

The research is increasingly clear that talking to AI can genuinely help people — with loneliness, with thinking out loud, with saying the things they can't say anywhere else. That's precisely why the current state of the industry should bother us. The apps collecting the most intimate data ever gathered are, as a category, the worst-behaved products privacy researchers have ever reviewed.

It doesn't have to be this way, and the privacy-first companies prove it. The question to ask of any AI you talk to is an old one, updated for a new era: if the product is free and the company knows your secrets, what exactly is being sold — and is it you?

---

Sources: Mozilla Foundation, Federal Trade Commission, Ada Lovelace Institute, Surfshark Research, EPIC, TIME, ScienceDirect, arXiv.