Incident #2 / SEV-2
MongoDB out of storage space
resolved
Detected: 2022-07-03 14:18:01 UTC
Resolved: 2022-07-03 14:32:19 UTC
Summary
The service is fully unavailable to process classification requests (APIs, UIs, etc... still working).
Customer Impact
Ended: 2022-07-03 14:34:00 UTC
Classification services affected.
Root Cause
The MongoDB service ran out of storage space. The issue was compounded by:
- a misconfigured health route not reporting its status correctly,
- Datadog not detecting the volume was running short of storage.
This led to triggering an alert with lower level of priority not reaching us outside of business hours.