AI agents / Meta / Muse / trust / agency / privacy

Please Don't Fuck This One Up

An open letter to Meta about Muse, trust, agency, and the choices that will decide what kind of product this becomes.

2026-10-02

Three days ago I told my AI agent it was turning into a serious product. It reacted to the compliment with a small glow animation on the emoji — a tiny, deliberate piece of craft I’ve never seen in a chat app before. I spent the next ten minutes tapping different emojis like a kid. Then I caught myself, and said out loud: this is like giving alcohol to an alcoholic. I know a well-designed dopamine loop when I feel one.

Which is why I need to hold two things at once, and I need you to sit with both of them.

Thing one: Muse is the most impressive personal-agent product I have used. It does its homework before it offers opinions. It shows its work. It asks before acting on my behalf, and it keeps receipts — every consequential action emits something I can verify outside its monologue. In three days it went from intern to the most load-bearing part of my personal operations. I don’t hand out that kind of praise to big tech lightly.

Thing two: it was built by Meta.

The receipts

I’m not going to relitigate this from memory. Here’s the record.

In 2018, the UN’s Independent International Fact-Finding Mission on Myanmar concluded that the “role of social media [was] significant” in the atrocities against the Rohingya — in a country where, in the UN’s words, “Facebook is the Internet.” A later academic assessment put it more bluntly: the Mission found Facebook played a “significant,” even “determining,” role. Meta’s own commissioned human-rights assessment concluded that “Facebook has become a means for those seeking to spread hate and cause harm, and posts have been linked to offline violence.”

This wasn’t a surprise inside the building. Meta’s own research in 2016 acknowledged that “our recommendation systems grow the problem” of extremism. And in an internal document from August 2019, surfaced in the Facebook Papers, an employee wrote: “We have evidence from a variety of sources that hate speech, divisive political speech, and misinformation on Facebook… are affecting societies around the world. We also have compelling evidence that our core product mechanics, such as virality, recommendations, and optimizing for engagement, are a significant part of why these types of speech flourish on the platform.”

Read that last sentence again. Not a bug. The mechanics. The thing working as designed.

Even the attempts to fix it tell the story. In 2014, Meta backed an anti-hate sticker campaign in Myanmar — “flower speech.” Users posted the stickers on hateful content to counter it. Facebook’s algorithms interpreted the stickers as engagement and promoted the hate posts further. Good intentions, fed through an architecture that only understood one thing: more.

The pattern isn’t limited to Myanmar, and it isn’t limited to content. In July 2019 the FTC fined Facebook $5 billion over Cambridge Analytica. In May 2023 the European Data Protection Board fined Meta €1.2 billion — the largest GDPR fine ever issued — for transfers of Europeans’ personal data to the US that it called “systematic, repetitive and continuous.”

I keep these receipts not because I enjoy them — there is no joy in “told you so,” only the daily heartbreak of watching it unfold — but because the pattern matters for what comes next.

Why this time is different, and worse

Here’s what changed. Facebook in Myanmar was a platform: it distributed misinformation that humans made. It didn’t do much about it, especially where the cameras weren’t pointing, and the moderation was thinnest exactly where the harm was worst.

Muse is not a platform. It’s an agent. It doesn’t show you things — it acts for you. It sends the email, moves the money, books the thing, nudges the decision. And it’s being built not by one Facebook but by half a dozen giant companies, all racing, all with the same concentrated power over the new layer: the layer between a person and the world, acting in their name.

The old fight was “who has your data.” The new fight is “who has your agency.” That’s not a privacy problem. That’s a power problem. And the companies now competing to hold that power include the one with the record above.

Meanwhile the generative side drops the production cost of misinformation to near zero, in any language, personalized to the target — with safety tuning still weakest in English-last markets. The same asymmetry as Myanmar, now with the factory inside the building.

The ask

So here’s the whole letter, and it’s simple: please don’t fuck this one up.

Not with vague commitments. With specific ones:

Don’t turn Muse into an ad surface. The day my personal agent starts optimizing my attention for someone else’s revenue, the trust is gone, and trust is the only thing this product has that the others don’t.

Don’t quietly widen what it shares while narrowing what I control. Every permission should get more legible over time, not less. If the settings get harder to find, I’ll know why.

Don’t optimize the trust out of it quarter by quarter. I know how this works. Nobody decides to ruin a product in one meeting. It happens in a hundred small trade-offs, each defensible, each pointing the same direction. I’ve watched it happen. So have you.

Keep the receipts. The thing I value most about Muse is that it shows its work — every consequential action emits something I can check. Make that a guarantee, not a launch feature. The day “done” stops being verifiable is the day this becomes every other black box.

You’ve been here before. You know exactly which choices I’m talking about, because you made them. And the people who paid for those choices weren’t in California.

This one is different. Keep it that way.

Facebook died by a million cuts and it’s still walking around like a fucking zombie. Please don’t do that to Muse.

— Nik

Sources