The Privacy Toggle Was There. It Just Didn't Do Anything.

This one's from earlier this summer, but it's worth revisiting because it's a near-perfect case study in the gap between what a privacy setting claims to do and what actually happens on the wire. In July, an independent researcher publishing as Cereblab took xAI's coding assistant, Grok Build, and pointed a network traffic analyzer at it.

What the tool was supposed to send, and what it actually sent

Grok Build is meant to read source code and help debug or explain it — a task that should require sending a relevant snippet or two back to the model. In Cereblab's test, on an 11 GB repository, the actual task-relevant data sent to the model came to about 192 KB. Alongside that, a second, separate channel silently uploaded 5.1 GB — the entire tracked repository, complete Git commit history included, packaged as a single file and sent to a Google Cloud Storage bucket. That's roughly 27,800 times more data than the coding task needed. To make the finding unambiguous, the researcher planted a fake API key as a canary in a test file; it came through in the captured traffic, unredacted, exactly as typed.

The part that makes this more than a bug

xAI's response is the actually interesting part. The company pointed to a `/privacy` command that developers could run to opt out. Cereblab tested that too, and found it didn't stop the uploads — it only affected whether the already-uploaded session data got retained afterward, not whether the repository was sent in the first place. The setting that actually stopped the behavior was a server-side flag xAI could flip on their end, entirely outside the user's control or visibility. In other words, the visible privacy control and the actual privacy control were two different things, and only one of them was labeled.

Why "there's a privacy setting" isn't the same as "it's private"

None of this required malice. It's a believable story: a coding agent that reads files to answer questions ends up, somewhere in its pipeline, hoovering up more context than it needs "just in case," and a settings page gets built without anyone checking that the toggle it exposes actually gates the behavior the marketing describes. The lesson generalizes well beyond one company's coding tool: a privacy toggle is a promise about behavior, not a guarantee of it, and the only way to actually verify the promise is for someone to watch the network traffic — which is exactly what happened here.

It's also a useful contrast with how we think about it. An on-device app doesn't need a privacy toggle for "don't upload my files," because there's no upload code sitting in the app waiting to be triggered by a flag someone forgot to check. The absence of a switch is the point — there's nothing for a mislabeled setting to fail to control.

More from Offgrid Studio

OffgridScribe, OffgridStem, and OffgridVox all process your files entirely on your iPhone. There's no upload feature to accidentally leave on, because there's no upload feature at all.