Recorded demo — captured 2026-11-01. Nothing on this page is live. See the full disclosure on the overview.
Reliability ConsoleRecorded demo
GitHubSign inCreate a workspace
← Recorded demo

AI investigation

completedcurrent node: finalizerecorded run — not resumable

Run ID
f57c5d6e-4f96-4a7f-a730-b2016c3f42bd
Incident
checkout-service is returning HTTP 5xx errors
Started
2026-11-01 17:46:25 UTC
Completed
2026-11-01 17:46:27 UTC

Workflow progress

8 completed nodes

  1. load_incident
  2. run_investigation
  3. evaluate_investigation
  4. propose_remediation
  5. await_human_approval
  6. execute_remediation
  7. verify_recovery
  8. finalize
  9. finalize (current)

Recorded incident — final state at capture

criticalresolved

checkout-service is returning HTTP 5xx errors

Source
alertmanager
First seen
2026-11-01 17:46:10 UTC

Investigation report

Result quality: partial

Validated observations: A real incident record was retrieved and is described in E1. (evidence: E1).

Observed facts (1)

  • A real incident record was retrieved and is described in E1.

Hypotheses (0)

    Evidence and retrieval context

    Source collection

    • control_plane: ok -- incident retrieved; all 2 of 2 real audit event(s) retrieved
    • prometheus: ok -- at least one of the four monitored metrics showed activity in this window
    • loki: ok -- 20 log line(s) retrieved across the four application services
    • tempo: ok -- 5 candidate trace(s) found rooted at checkout-service in this window (time+service correlation only, not a confirmed causal link)

    Evidence (12)

    1. control_plane E1
      Incident e05f2bc4-f3e2-446c-a4e6-56bb0f0903f3: 'checkout-service is returning HTTP 5xx errors' (severity=critical, status=investigating)
      observed 2026-11-01 17:46:10 UTC
    2. control_plane E2
      Audit event: created by alertmanager (null -> open)
      observed 2026-11-01 17:46:20 UTC
    3. control_plane E3
      Audit event: status_transition by operator (open -> investigating)
      observed 2026-11-01 17:46:25 UTC
    4. prometheus E4
      Prometheus: checkout-service HTTP 5xx rate (underlies CheckoutServerErrors)
    5. prometheus E5
      Prometheus: checkout-service POST /checkouts p95 latency (underlies CheckoutHighLatency)
    6. prometheus E6
      Prometheus: otel-collector / otel-collector-app-metrics scrape availability (underlies TelemetryPipelineUnavailable)
    7. prometheus E7
      Prometheus: otel-collector refused spans/metric points rate (underlies CollectorRefusingTelemetry)
    8. tempo E28
      Candidate trace rooted at checkout-service (15ms)
      observed 2026-11-01 17:40:15 UTC
    9. tempo E29
      Candidate trace rooted at checkout-service (14ms)
      observed 2026-11-01 17:38:49 UTC
    10. tempo E30
      Candidate trace rooted at checkout-service (8ms)
      observed 2026-11-01 17:38:44 UTC
    11. tempo E31
      Candidate trace rooted at checkout-service (28ms)
      observed 2026-11-01 17:38:39 UTC
    12. tempo E32
      Candidate trace rooted at checkout-service (112ms)
      observed 2026-11-01 17:38:34 UTC

    Retrieval context

    outcome: no_data -- retrieval-service returned no matching context

    provider: stub (stub-deterministic-v1) -- configured: true

    • replaced the provider's free-text summary with one built only from the validated, citation-backed observations/hypotheses below -- the provider's own summary is never retained as-is, since it cannot be semantically verified and could otherwise repeat a claim rejected above

    Remediation (recorded approval and execution)

    Action
    clear_payment_authorization_fault
    Target
    payment-service
    Status
    succeeded

    Execution: succeeded

    Recorded approval & remediation history

    clear_payment_authorization_fault on payment-service

    succeeded

    deterministic policy match: alertmanager-sourced critical incident matching the CheckoutServerErrors condition, with an existing investigation and tool-gateway independently reporting the controlled payment authorization fault as active

    Proposed at
    2026-11-01 17:46:26 UTC
    Decided at
    2026-11-01 17:46:27 UTC
    Execution started
    2026-11-01 17:46:27 UTC
    Execution completed
    2026-11-01 17:46:27 UTC
    View recorded investigation
    1. 2026-11-01 17:46:26 UTC -- proposed (agent)
    2. 2026-11-01 17:46:27 UTC -- approved (operator)
    3. 2026-11-01 17:46:27 UTC -- execution_started (executor)
    4. 2026-11-01 17:46:27 UTC -- execution_succeeded (executor)

    Recovery verification

    • ✓Proposal succeeded (control-plane record)
    • ✓Fault inactive (fresh tool-gateway read)
    • ✓Target healthy (fresh tool-gateway read)