checkout-service is returning HTTP 5xx errors

criticalresolved

Source: alertmanager
Description: checkout-service has served HTTP 5xx responses at a nonzero average rate over the last 5 minutes, sustained for at least 2 minutes. A genuine client-side 4xx (e.g. a failed request validation) never triggers this rule — only http_response_status_code=~"5.." is counted.
First seen: 2026-11-01 17:46:10 UTC
Last seen: 2026-11-01 17:46:20 UTC
Resolved at: 2026-11-01 17:51:10 UTC
Incident ID: e05f2bc4-f3e2-446c-a4e6-56bb0f0903f3

This incident was investigated once, as part of the recorded run below. View the recorded investigation

Audit timeline

created by alertmanager
(none) → open
2026-11-01 17:46:20 UTC
status_transition by operator
open → investigating
2026-11-01 17:46:25 UTC
status_transition by operator
investigating → remediating
2026-11-01 17:46:27 UTC
status_transition by alertmanager
remediating → resolved
2026-11-01 17:51:20 UTC

Remediation history

clear_payment_authorization_fault on payment-service

succeeded

deterministic policy match: alertmanager-sourced critical incident matching the CheckoutServerErrors condition, with an existing investigation and tool-gateway independently reporting the controlled payment authorization fault as active

Proposed at: 2026-11-01 17:46:26 UTC
Decided at: 2026-11-01 17:46:27 UTC
Execution started: 2026-11-01 17:46:27 UTC
Execution completed: 2026-11-01 17:46:27 UTC

View recorded investigation

2026-11-01 17:46:26 UTC -- proposed (agent)
2026-11-01 17:46:27 UTC -- approved (operator)
2026-11-01 17:46:27 UTC -- execution_started (executor)
2026-11-01 17:46:27 UTC -- execution_succeeded (executor)