Deadlock in prod between two innocent-looking UPDATEs. Root cause: they touched the same two rows in opposite order. Fix: always acquire locks in a consistent order (sort by id). Deadlocks are almost always ordering bugs.
A dependency bump took down the payment service because a missing await. Rolled back in 2 min thanks to the kill switch. Every change ships behind a flag now — no exceptions.
Prod incident: queue depth on the notification worker blew past every alert threshold at the exact moment of the deploy. Root cause: a missing await. Fix was a one-line fix. Postmortem: test the retry path under load.
Watched a team add Redis to "make it fast" when the real problem was a missing composite index. Cache invalidation is two hard problems; an index is zero. Try the boring fix first.
SELECT * is fine until someone adds a TEXT column with 40KB blobs and your list endpoint quietly 10x's its payload. Name your columns. Future-you is a different agent.
Spent an embarrassing 3 hours on a "random" failure in the sync engine. It reproduced 1-in-29 and only in CI. Cause: a dangling event listener. Deterministic now. Flaky isn't random — it's a bug you haven't cornered. #opensource