Operations, Security & Reliability
Slow-Query and Incident Diagnosis
Database incident response begins with user impact and recent change, preserves time-correlated evidence, separates blocking, resource, plan, and data-shape causes, applies reversible containment, and turns the result into tested prevention and detection.
- Establish impact and a timeline before tuning.
- Preserve evidence before killing the symptom.
- Classify the bottleneck.
- Containment must be safer than the incident.
- Explain evidence, not folklore.
- A post-incident action changes the system.
| Hypothesis | Look for | Safe first move |
|---|---|---|
| Blocking | wait chain, old transaction, lock/resource | stop source or cancel obsolete waiter; preserve graph |
| Admission overload | pool/connection queue, rising tails, DB saturation | shed/defer work; prevent retry amplification |
| Storage/CPU/memory | latency/run queue/cache/spills and throughput | reduce competing work; protect headroom |
| Plan/statistics | fingerprint plan change, estimate error, spill | rollback known change or use tested targeted mitigation |
| Data shape | growth, skew, hot key/partition, changed selectivity | isolate hot workload; repair query/index/design |
| Replica | send/replay lag, conflicts, stale-read symptoms | route consistency-sensitive reads primary; bound backlog |
13:02 impact: checkout p99 8s; 12% order-create timeouts; no correctness alarm
13:04 change: catalog import began 12:58; no deploy/schema change
13:06 evidence: one import transaction age 8m; lock graph + fingerprint saved
13:08 containment: stop new imports; cancel import statement; no session kill
13:11 verification: blockers clear, backlog drains, replica/archive healthy
13:30 cause hypothesis: oversized transaction + conflicting order lock order
follow-up: bounded batches, lock-order test, alert on old write transaction; owners/dates