The Git History You Can't Actually Delete

Part 1 ended with a docs folder that had to move. An internal architecture-review write-up quoted real hostnames, a secret-block name, internal work-pool names, an NFS mount path, and the names of other private repos, all as illustrative detail from the original investigation. Deleting it from the current tree was the easy part. The tracking issue for what came next filed itself as an obvious mechanical task: rewrite git history, done.

It was not done. It took most of a week, involved three completely different kinds of “you can’t actually remove that,” and ended with GitHub Support telling me the blast radius was five times bigger than what I’d scoped. This is the story of that week — so you don’t have to live it yourself.

The tracking issue was itself the leak

First pass: sweep media-api for anywhere else the same content might be sitting. It turned out not to be that isolated. The PR that had originally removed the secrets quoted the leak as a “before” example in its own description (thanks, AI!). A stale issue, auto-filed months earlier by an old Logfire-to-GitHub-issues integration, had a real hostname sitting in a stack trace (that integration is gone now, removed entirely rather than just disconnected). And the tracking issue for this very cleanup effort had, in the course of describing what needed scrubbing, quoted several of the exact strings it was supposed to get rid of. These were unforced errors on my part.

Getting AI to help with this kind of redaction pass comes with its own version of the same trap. More than once, a commit message or a branch name written to describe what was being fixed came close to repeating the thing being fixed, because describing a leak precisely enough to explain the diff means naming it, and naming it is the one thing I was trying to stop doing. Worth watching for, not assuming a redaction pass is safe just because it’s automated. AI is enormously helpful…occasionally too helpful.

A sweep across roughly 74 issues and PRs found and fixed six more places carrying the same content, plus the stale issue that was better off being deleted outright than edited. GitHub draws a sharp line here that isn’t obvious until you hit it: an issue can be edited or deleted. A pull request can be edited, but never deleted, only closed. That asymmetry is the root cause of that week’s effort.

What a force-push doesn’t reach

Rewriting main’s history and force-pushing feels like it should be the end of it. It isn’t, and the reason is specific to how GitHub hosts a repo, not how git itself works. Every pull request gets its own server-managed refs, refs/pull/<n>/head and refs/pull/<n>/merge, that GitHub keeps regardless of what happens to the branches that fed them. A handful of merged PRs that had touched the leaked content anchored full, permanent, unrewritten snapshots of it, completely untouched by rewriting main.

This “wrong mental model” theme underpins the whole week: I’d been treating “rewrite the branch” and “the repo no longer has this content” as the same statement. They aren’t. A GitHub-hosted repo has more surface area than its branches alone.

A third kind of stuck

While mapping out what was actually reachable, by whatever means, one more category turned up, stricter than either of the first two.

A Copilot-authored PR review comment can’t be edited through the API ("body is not editable") and can’t be deleted either ("cannot be deleted"). This is a harder restriction than the ordinary “PRs can’t be deleted, only closed” rule that applies to everything else. Whatever mechanism protects Copilot’s own comments from tampering doesn’t have an exception for “the author wants to redact a leaked hostname.”

The rationale behind this limitation is easy to understand: you can’t put words in the AI’s mouth (by omission, addition or editing).

And so, we land at three different flavours of immutable, each for a different reason, discovered in the space of about a day:

The false alarm

Before rewriting anything, I wanted to know exactly how far back the exposure went, so I ran a full-history credential scan rather than trusting my memory of what could have leaked and when. This turned up two Postgres credentials that had never been documented, on a host that wasn’t part of the original rotation work, reachable only through roughly 32 old pre-squash PR refs.

Adrenaline spike, then a quick check: both credentials belonged to infrastructure that had already been decommissioned. Not live, not exploitable, a false alarm. But the method that found them, scanning every PR ref rather than just the current branch, is the value-add lesson here.

main’s history is not the whole of what a scan needs to cover; it just happens to be the part that’s easy to remember to check.

Counting the damage wrong

Next question: how many PRs actually needed attention? A naive approach: grep the full tree of every PR’s head snapshot for the strings in question. That came back saying roughly 60 of 61 PRs were affected, a number that didn’t pass the smell test for a leak that had actually shipped in a handful of specific commits.

The bug was in the method, not the data. A full-tree grep answers “does this snapshot contain the string,” which is a different question from “did this PR actually introduce or remove it.” Almost the entire inflation traced back to one unfixed line in .gitignore’s own comment text, present in nearly every snapshot simply because it had never been touched, not because every PR had anything to do with the leak. Once that line was fixed, the real count dropped to five — the actual number of PRs whose own diffs had introduced or removed the leaked content.

Getting the sequence right

The plan going in was to request GitHub Support’s help before doing the git-history rewrite, on the theory that the two fixes should land together. However, GitHub’s documented process is rewrite and force-push first, then file a support ticket referencing the rewrite’s “First Changed Commit” so Support can find and purge whatever the rewrite couldn’t reach on its own.

Support needs something to compare against; asking them to purge PR content before the rewrite exists gives them nothing to anchor the request to. So — clean up what can be cleaned first, then pull GitHub Support in to assist.

The actual rewrite

git filter-repo against a fresh mirror clone, two kinds of change. Path-based removal for anything no longer needed in the tree at all: the architecture-review docs, a couple of workflow files, a script tied to the retired Logfire integration. Text replacement for everything still living in current files, using --replace-text for blob content and --replace-message separately for commit messages, since the first doesn’t touch the second and it’s an easy detail to miss.

Two replacement patterns needed real care. A short string tied to one leaked secret name was close enough to a currently-legitimate environment variable name that a careless match would have quietly corrupted a real, still-in-use config value. Another pattern, meant to catch a specific internal reference, was close enough to legitimate “Generated with Claude Code” attribution lines that a blunt match would have mangled commit messages that had nothing to do with the leak. Both needed scoping tighter than the obvious first pass.

One operational note carried forward from an earlier incident: the org’s branch-protection ruleset applies to every repo, not just this one, and a previous rewrite (a different repo, a different day) had disabled it organization-wide for the duration of the push, which was more blast radius than the job needed. This time the exclusion was scoped to just the one repo.

A clean landing

git push --force --mirror: two branches, twenty-nine tags, zero rejections. A fresh clone confirmed zero hits for any of the target content across the entire history. The rewrite even caught one more live leftover along the way, a leaked reference sitting in a .gitignore comment that earlier scrub passes had missed because nobody thinks to grep the comments in a file whose whole job is being invisible.

Production was untouched throughout: the current tree came out byte-identical to before the rewrite except for that one fix, and the single PR that was open at the time stayed mergeable the whole way through. The support ticket, referencing the rewrite’s First Changed Commit, went out the same day.

The bill came back five times bigger

Two days later, GitHub Support replied. Instead of the five PRs I’d flagged as containing the leaked content, they’d found 28. The reason was simple in hindsight and not something I’d have guessed going in: the “First Changed Commit” I’d handed them was the repo’s own squashed root commit, and every PR merged since that squash shares it as an ancestor, whether or not that specific PR’s own diff ever showed the leaked content. I’d asked a precise question and gotten a coarse, ancestry-shaped answer back, because ancestry was the only thing “First Changed Commit” was precise enough to express.

Support offered a choice per PR: delete it outright, or dereference it, which makes the diff permanently inaccessible while leaving the title and comments intact. Full deletion would have taken real project history down with it. Several of the 28 documented things worth keeping on the record: a security-policy addition, a significant internal rename, some documentation work. Dereferencing all 28 kept the paper trail and lost only the diffs, which was the only part that actually mattered.

Confirmation landed a day after that. Spot-checking four of the 28: fetching each one’s diff returned an explicit “not available” response where it used to render in full, and the commit list for the same PRs came back empty where it hadn’t before. Titles and comments were untouched, exactly as requested.

One thing never got fixed, and I’d rather make it plain than handwave it away: the “immortal” Copilot-authored comment I mentioned earlier still to-this-day names a private sibling repo. Neither the git rewrite nor GitHub Support’s dereferencing work reached it, because it isn’t governed by either mechanism, it’s a separate immutability rule of its own. It wasn’t worth a dedicated support ticket for one low-severity mention, so it’s still sitting there. That’s the honest ending, not a tidy one, but a deliberate trade-off.

Every layer of this looked solved right up until it wasn’t. HEAD was clean, but history wasn’t. History was clean, but PR refs weren’t. Issue text was clean, but a Copilot comment couldn’t be touched by anything. A plan that felt uncontroversial and correctly sequenced did not match GitHub’s well-reasoned documented order of operations. A purge request that felt properly scoped came back five times bigger, for a reason that made total sense the moment it was explained and hadn’t occurred to me before that. Same fix, every single time: stop reasoning from what seems like it should be true, and go check what the actual system does.

Part 3 is the quieter piece: what it took to keep a private deploy pipeline running against a codebase that no longer knows anything about the infrastructure it runs on.

The repo: github.com/Glitchedpixel-io/media-api