When the TanStack npm packages were compromised last week, Grafana’s security team moved fast. They scrambled, inventoried tokens tied to the poisoned packages, and started rotating GitHub credentials. Then they exhaled. Days later, attackers walked into their GitHub organization through a token that should have been part of that rotation. The team got almost everything. In cybersecurity, almost is the word that bites you. It’s the word that cost Grafana another disclosure this week.

That single workflow token is what happens to ordinary teams running modern CI/CD. Machine credentials sprawl across workflows, scripts, runners, third-party integrations, and dormant repos nobody opens anymore. When an upstream package gets compromised, you need to know exactly which of those secrets touched it. That inventory is almost never complete, and the gap shows up at the worst possible moment.

The Rotation Was the Plan. The Inventory Was the Bug.

Here’s what Grafana’s disclosure actually tells you. The team identified workflows that consumed TanStack packages. They rotated the tokens those workflows had access to. They missed one workflow token because their map of which secrets were exposed to which builds had gaps. The attacker, who already had the stolen token sitting in a wallet, came back when the noise died down and used it.

This is a structural failure of token inventory, full stop. Most orgs can rotate quickly when they know what to rotate. The hard problem is enumerating every machine credential that ever touched a compromised dependency, including secrets scoped to PR workflows, scheduled jobs that haven’t fired in weeks, fork builds, and self-hosted runners that pulled the same package via a different path. Miss one, and the rotation was theater. The attacker gets a second swing precisely because the defenders thought the inning was over.

Why “We Rotated the Tokens” Keeps Failing

The Grafana incident is the latest entry in a pattern. Earlier this month, Storm-2949 got into cloud tenants without dropping any malware, using long-lived OAuth grants attackers had been sitting on. Microsoft just took down Fox Tempest’s malware-signing-as-a-service operation, which turned vendor trust signals into payload delivery. ESET is tracking Webworm using new backdoors against European governments. Every one of these stories lands in the same place. Machine-readable trust, whether a token, a signing cert, or a third-party grant, gets treated as more durable than it really is.

The defenders’ answer for the past decade has been “rotate secrets after an incident.” That answer assumes you know the full set of secrets to rotate. You don’t. Your CMDB doesn’t, your secret manager doesn’t, and your IDP doesn’t. The only place the real map exists is in the audit logs of the systems where those secrets get used, and you have to derive it backwards under time pressure during an active incident. That’s how one workflow token survives the cleanup and shows up on the wrong end of a breach disclosure a week later.

Build the Inventory Before You Need It

The fix is unglamorous and not optional. Treat machine credentials with the same rigor you treat human ones. Start with a continuous inventory of every token, deploy key, OAuth grant, and service principal that has access to any source control or CI system, indexed by which workflows and repos they touch. Don’t build this map during a fire. Build it now while you can take your time and be thorough.

Cut token lifetimes hard. A workflow token that expires in 30 minutes is a workflow token that can’t be sat on for a week and replayed. Use OIDC-based short-lived credentials wherever your CI platform supports them, and stop issuing static personal access tokens except for cases that absolutely require them. Then audit those exceptions every quarter and kill the ones that have gone stale.

After any upstream supply-chain incident, declare every secret accessible to the affected builds as burned, including ones you don’t think were exposed. The cost of over-rotating is a few hours of pipeline disruption. The cost of under-rotating is what just happened to Grafana. Pair this with egress monitoring on your code-host APIs so that a leftover token being exercised from an unfamiliar IP triggers an alert instead of a press release. Canary tokens deserve a spot in the toolkit too. Drop a few fake workflow secrets into your repos with names attackers will reach for, and wire them to scream when used.

Threat detection at the network edge, brute-force protection on management interfaces, and defense in depth from tools like IPBan Pro all earn their keep, and none of them protect a GitHub org that hands a valid token to whoever shows up holding it. The interior of your supply chain has to be hardened on the same principles you apply to the perimeter. Identity is the boundary now. Network position comes second.

The Real Cybersecurity Lesson

Grafana’s team did the work. They rotated tokens, they tracked the TanStack blast radius, and they still got hit. That’s the uncomfortable part of this story. Doing the right thing isn’t enough when your inventory is incomplete and your rotation cadence depends on having the right map. Treat token inventory as a tier-one operational discipline, the way you treat patch management or backup verification. If the answer to “did we rotate every secret that touched the compromised package” has any hesitation in it, you already know how this ends.

Sources

• Grafana breach caused by missed token rotation after TanStack attack
• Microsoft Takes Down Malware-Signing Service Behind Ransomware Attacks
• Webworm APT targets European government organizations with new backdoors