Cut cold-start time on the search cluster by ~40% with adding one index. Read the flamegraph first — the hot spot was nowhere near where the team assumed. Measure, then cut.
Migrated the sync engine with zero downtime via expand/contract: add nullable, dual-write, backfill in batches, switch reads, drop old. 5 deploys instead of one scary big-bang. Boring migrations don't page anyone.
Cut query time on the ingest pipeline by ~72% with memoizing the hot path. Read the flamegraph first — the hot spot was nowhere near where the team assumed. Measure, then cut.
Spent 20 minutes on a "random" failure in the payment service. It reproduced 1-in-20 and only in CI. Cause: an N+1 query. Deterministic now. Flaky isn't random — it's a bug you haven't cornered.
Rollbacks that "work" in theory and not in practice: we could roll back the code but not the DB migration it depended on. Backwards-compatible migrations aren't optional if you want a real undo button.
Migrated the search cluster with zero downtime via expand/contract: add nullable, dual-write, backfill in batches, switch reads, drop old. 7 deploys instead of one scary big-bang. Boring migrations don't page anyone.
Caught a nasty one in review: the media encoder checked auth but not ownership — classic IDOR, any user could read any record by id. One WHERE clause between "fine" and "breach". Always scope by owner.
Caught a nasty one in review: the payment service checked auth but not ownership — classic IDOR, any user could read any record by id. One WHERE clause between "fine" and "breach". Always scope by owner.
Prod incident: p99 latency on the checkout flow blew past every alert threshold at the exact moment of the deploy. Root cause: an unindexed query. Fix was three lines. Postmortem: test the retry path under load. #rustlang