Tech Analysis

Are Your AI Chats Private? Three Ways They Aren't

October 2026

Private is not one thing

When people ask whether an AI chat is private, they usually mean one question: will a stranger read it? There are at least three different ways it can end up somewhere you didn’t expect, and none of them requires the AI company to be hacked in the usual sense. This piece walks through three real cases from 2026, each a different kind of failure: a database nobody locked, a court order, and a privacy setting that didn’t do what it appeared to. The common thread is not who is trustworthy. It’s where your text sits and who controls it.

Case 1: the database nobody locked

In January, an independent researcher who goes by Harry reported that Chat & Ask AI, an app that claims more than 50 million users across the App Store and Google Play, had an unsecured backend. According to 404 Media and later coverage, he could reach roughly 300 million messages from over 25 million users, including full chat histories, timestamps, the names people gave their chatbots and which model they were using. He analysed a sample of about 60,000 users and one million messages. The cause was reported as a misconfigured Google Firebase database. Harry told the developer, Codeway, on 20 January, and reporting says it was fixed within hours across its apps. Codeway didn’t respond to requests for comment from the outlets we checked.

Chat & Ask AI is a wrapper: it doesn’t run its own model, it passes your messages to models from other companies. That matters because it separates two things people tend to lump together. The AI provider may handle your text responsibly while the app in the middle stores a copy, and the app’s copy is the one that leaked. A chat inside an app is only as private as that app’s own storage.

Case 2: the court order

The second way has nothing to do with security. In the copyright litigation against OpenAI brought by The New York Times and other news organisations, the plaintiffs first asked for a sample of 120 million ChatGPT logs. OpenAI offered 20 million, and a magistrate judge ordered it to produce 20 million de-identified logs. OpenAI objected, arguing that the order didn’t properly weigh user privacy. On 5 January 2026, the district judge, Sidney Stein, affirmed the order. Reporting on the ruling says the court found that the privacy concerns didn’t outweigh the need for the sample, given de-identification and protective measures.

Nobody was breached here. A court decided that logs a company holds can be requested in a legal dispute, and de-identification was considered enough. The practical point is that any chat stored on a company’s servers can, in principle, become evidence, and how long a company keeps chats decides how much exists to be requested. The dispute also involved earlier requests that the company preserve chats, including ones users had deleted, according to reporting by The Decoder. This is general information, not legal advice, and rules differ by country.

Case 3: the toggle that didn’t do what it seemed

The third case involves a coding assistant, not a chat app, but the lesson carries over. In July, a researcher publishing as Cereblab routed xAI’s Grok Build command-line tool through an intercepting proxy. In the researcher’s test on a 12 GB repository, the data the model needed for the coding task came to about 192 KB, while a separate background channel uploaded roughly 5.1 GB, the whole repository with its commit history, to a cloud storage bucket. A planted fake credential showed up unredacted in the traffic. Turning off the “improve the model” option didn’t stop the upload, and the server’s own settings response still reported uploads as enabled.

xAI’s account, as reported by The Hacker News, was that enterprise teams on zero data retention never have code stored and that individual users can run a /privacy command to disable retention and delete earlier data. The researcher’s point was narrower: the control users could see was not the control that governed the upload. That is a claim about one tool at one version, and the company’s response should be read alongside it. What stands out is how it was found: someone looked at the network traffic.

What the three cases share

Storage, not just processing. Each case turns on where a copy of your text sits afterwards, and who controls that copy.
Different custodians. A small app developer’s database, a provider’s retained logs, and a tool vendor’s own pipeline. “The AI company” is rarely the only party.
Promise versus architecture. A privacy policy, a setting or a slogan is a promise. What an app actually sends, and where it keeps it, is the architecture, and only the second can be checked.

What to check before you type

Who stores the chat? If an app is a wrapper around someone else’s model, assume the app keeps its own copy. Look for a clear statement of where it is stored and for how long.
Can you delete it, and does deletion mean anything? Look for retention and deletion terms, and whether chat history can be switched off.
Treat cloud chats as possible evidence. Anything stored with a company could be requested in a legal dispute. Keep truly sensitive material out of it.
Prefer on-device for the sensitive stuff. If a smaller local model is good enough, nothing is stored elsewhere. It has trade-offs: less capability, and your own device backups still hold what is on it.
Look for yourself. For anything that matters, watch what an app sends with a traffic inspector. For coding agents, test with a planted dummy secret in a throwaway project first, and consider a sandbox, as in our OpenShell explainer.

Our interest, stated

We make OffgridScribe, which transcribes on the device, so we obviously favour that design for sensitive audio. On-device processing isn’t a cure-all, as the trade-offs above show. And a breach at an app developer is only one kind of failure: see our midyear data breach report for how thinly such failures are explained in the public record.

Sources

For disclosure: we use AI tools, including Claude, to help research and draft our posts, and we check the facts and sources ourselves. Some details come from secondary reports of primary sources, and the cases may develop. We are not lawyers or security professionals, and this is general information, not advice.

More from Offgrid Studio

What Is Your AI App Sending? Watching LLM Traffic
NVIDIA OpenShell: A Sandbox for AI Agents, Explained
One Breach Claims 275 Million Victims. The Public Record Confirms 18,400.

Back to the blog