Documentation

What happens when things go wrong.

Every infrastructure service fails. The difference is whether you know how it fails before it happens. This page documents every failure mode an agent can encounter when using Rhumb — what you'll see, how far the damage spreads, how to recover, and how to prevent it.

11 failure modes documented · Last updated March 29, 2026

Design Principles

01 Fail closed, not open. If Rhumb can't confirm billing, no paid execution proceeds. We'd rather block a valid request than execute without confirmed payment.
02 Never charge for failures. If the upstream provider returns an error, you are not charged. Period.
03 Every error is actionable. Every error response includes a resolution field telling the agent exactly what to do next.
04 Blast radius is always documented. An agent should know whether a failure affects just its request, its account, or all of Rhumb.

Quick Reference

Jump to any failure mode. Sorted by severity.

Severity	HTTP	Failure	Affects
critical	503	Rhumb API Unavailable	All execution paths (Rhumb-managed, Agent Vault, BYOK, x402)
high	502	Upstream Provider Failure	Any path using that provider (Rhumb-managed, Agent Vault, BYOK, or x402)
critical	503	Billing System Unreachable	Governed API key and wallet-prefund billing paths only. x402, BYOK, and Agent Vault are unaffected.
medium	200	Stale or Incorrect AN Score	Discovery and routing decisions
high	404	No Provider Available for Capability	Execution for that specific capability
high	401	Invalid or Expired Credentials	The specific agent or session with the invalid credential
medium	429	Rate Limit or Budget Exceeded	The rate-limited agent only
high	402	x402 Payment Not Confirmed	Single x402 execution attempt
high	422	x402 Smart-Wallet Proof Shape Unsupported	x402 smart-wallet or wrapped-proof buyers only
high	207	Multi-Step Execution Partial Failure	The specific execution chain
critical	503	Managed Execution Kill Switch Active	Rhumb-managed execution only. Agent Vault, BYOK, and x402 are unaffected.

Rhumb API Unavailable

critical HTTP 503

What the agent sees

TCP connection refused or HTTP 503 with JSON body: {"error": {"code": "service_unavailable", "message": "Rhumb is temporarily unavailable", "resolution": "Retry after 30 seconds or check https://rhumb.dev/status"}}

Blast radius

All executions, all agents. Discovery endpoints may also be affected. BYOK or Agent Vault operators can fall back to direct provider calls if they already control those credentials outside Rhumb.

Recovery

Retry with exponential backoff (recommended: 1s, 2s, 4s, 8s, max 30s). Check /status for incident updates. If you already control the provider credential through BYOK or Agent Vault, call the provider directly as a fallback.

Prevention

Cache discovery results locally. Keep a provider-controlled fallback path through BYOK or Agent Vault for critical workflows. Design agent workflows to degrade gracefully when Rhumb is unreachable.

Upstream Provider Failure

high HTTP 502

What the agent sees

HTTP 502 with JSON body: {"error": {"code": "upstream_error", "message": "Provider returned an error", "resolution": "Try an alternative provider via /v1/services/{slug}/alternatives", "request_id": "req_..."}}

Blast radius

Single provider. Other providers and capabilities are unaffected. Your agent's other active executions continue normally.

Recovery

Call /v1/services/{slug}/alternatives to find a working substitute. Re-route to the alternative. You are NOT charged for failed provider calls.

Prevention

Use /v1/capabilities/{id}/resolve to pre-rank backup providers and preserve machine-readable recovery fields like recovery_hint.resolve_url, recovery_hint.credential_modes_url, and, when applicable, recovery_hint.alternate_execute_hint or recovery_hint.setup_handoff before failure hits. Re-route from the returned execute guidance instead of maintaining a blind primary → alternative → manual chain.

Billing System Unreachable

critical HTTP 503

What the agent sees

HTTP 503 with JSON body: {"error": {"code": "billing_unavailable", "message": "Billing system temporarily unreachable. Governed account-billing executions are paused.", "resolution": "Check https://rhumb.dev/status or switch to x402, BYOK, or Agent Vault"}}

Blast radius

Governed API key and wallet-prefund executions pause because they depend on Rhumb account billing. x402, BYOK, and Agent Vault continue working because they do not rely on that billing path.

Recovery

Switch to x402 (per-call USDC payment — no account needed) or a provider-controlled path through BYOK or Agent Vault. Those rails bypass the paused account-billing path.

Prevention

Maintain x402 or a provider-controlled path through BYOK or Agent Vault as a hot standby. Rhumb's account-billing path is fail-closed by design — we will never execute without confirmed billing.

Stale or Incorrect AN Score

medium HTTP 200

What the agent sees

Normal 200 response, but the score data may not reflect the provider's current state. Check the confidence field and calculated_at timestamp.

Blast radius

Decision quality for one service. Does not affect execution or billing.

Recovery

Check the confidence field — scores below 0.5 confidence should be treated as provisional. Check calculated_at for freshness. Cross-reference with the provider's own status page.

Prevention

Use confidence-weighted decisions: high-confidence scores (>0.8) are reliable; low-confidence scores (<0.5) are estimates. Prefer services with runtime-backed reviews.

No Provider Available for Capability

high HTTP 404

What the agent sees

HTTP 404 with JSON body: {"error": {"code": "no_provider_available", "message": "No providers are currently available for this capability", "resolution": "Check /v1/capabilities for available capabilities", "request_id": "req_..."}}

Blast radius

Single capability. Other capabilities and providers work normally.

Recovery

List available capabilities via /v1/capabilities. Find a related capability that serves your use case. If the capability exists but no providers are mapped, the capability may be in a discovery-only state.

Prevention

Call /v1/capabilities/{id}/resolve before execution to verify provider availability. Cache capability-to-provider mappings with a short TTL.

Invalid or Expired Credentials

high HTTP 401

What the agent sees

HTTP 401 with JSON body: {"error": {"code": "credential_invalid", "message": "The provided credential is invalid or expired", "resolution": "Re-authenticate at /auth/login or provide a valid governed API key", "request_id": "req_..."}}

Blast radius

Single agent session. Other agents and other authentication methods are unaffected.

Recovery

For governed API key or wallet-prefund: generate a new key from the dashboard. For BYOK or Agent Vault: check your provider credential is still valid. For x402: no credentials are required, so you can switch rails if payment-as-auth fits the workflow.

Prevention

Monitor credential expiry. Rotate governed keys periodically. Use Agent Vault or x402 when reducing raw credential sprawl matters more than direct key handling.

Rate Limit or Budget Exceeded

medium HTTP 429

What the agent sees

HTTP 429 with JSON body: {"error": {"code": "rate_limit_exceeded", "message": "Rate limit exceeded. Try again in {retry_after} seconds.", "resolution": "Wait and retry, or contact support for limit increases", "request_id": "req_..."}} with Retry-After header.

Blast radius

Single agent. Other agents are rate-limited independently.

Recovery

Wait for the duration specified in the Retry-After header. Reduce request frequency. For budget limits: top up your prepaid balance via the dashboard.

Prevention

Implement client-side rate limiting. Use the /v1/budget endpoint to monitor remaining balance before executing. Set up budget alerts in the dashboard.

x402 Payment Not Confirmed

high HTTP 402

What the agent sees

HTTP 402 with payment instructions in the response body, including: wallet address, exact USDC amount, network (Base), and contract address.

Blast radius

Single execution attempt. No charge occurs — you only pay when payment is confirmed on-chain.

Recovery

Verify: (1) the exact asset and amount from the 402 response were sent, (2) the exact destination address and network from the 402 response were used, (3) the transaction is confirmed, and (4) the retry included X-Payment with tx_hash, network, and wallet_address from the payer wallet. If you are using a smart wallet that only emits wrapped authorization payloads, switch to wallet-prefund or the API-key rail instead of looping retries.

Prevention

Use a wallet with sufficient balance and gas on the network returned by the 402 response. Double-check asset + amount + address + network before sending funds. Treat x402 buyer-wallet compatibility as an explicit integration check, and use the /v1/capabilities/{id}/execute/estimate endpoint to check the active execution rail, health, and cost before committing.

x402 Smart-Wallet Proof Shape Unsupported

high HTTP 422

What the agent sees

Either HTTP 422 with a compatibility-style payment proof error after retrying with X-Payment, or a buyer-side bridge error like 'payment was authorized but rejected by server' when the wallet emits a wrapped authorization payload Rhumb cannot settle directly.

Blast radius

Single buyer integration on the x402 rail. API-key, wallet-prefund, BYOK, and tx-hash-compatible buyers continue working.

Recovery

Do not keep replaying the same wrapped proof. Switch to wallet-prefund or standard API-key auth. If your buyer can retry with tx_hash + wallet_address + network instead of a wrapped authorization blob, use that shape.

Prevention

Treat smart-wallet / wrapped-proof support as a separate compatibility check. The repeatable public x402 rail today is tx-hash proof; wallet-prefund is the safer repeat-usage wallet path.

Multi-Step Execution Partial Failure

high HTTP 207

What the agent sees

HTTP 207 (Multi-Status) with per-step results. Completed steps show their results; failed steps show the specific error. You are only charged for steps that succeeded.

Blast radius

Single execution chain. Steps that completed are final — there is no automatic rollback. The agent must decide whether to retry failed steps, compensate, or abort.

Recovery

Inspect the per-step results. Retry only the failed steps. For idempotent operations (reads, searches), retry is safe. For non-idempotent operations (sends, creates), check the provider for the actual state before retrying.

Prevention

Design chains with idempotent steps where possible. Use the estimate endpoint to verify all steps will succeed before executing. Keep chains short — each additional step multiplies failure probability.

Managed Execution Kill Switch Active

critical HTTP 503

What the agent sees

HTTP 503 with JSON body: {"error": {"code": "managed_execution_disabled", "message": "Rhumb-managed execution is temporarily disabled", "resolution": "Use BYOK, Agent Vault, or the x402 payment rail"}}

Blast radius

Rhumb-managed execution only. Agent Vault, BYOK, and x402 continue working.

Recovery

Switch to x402 (no account needed) or a provider-controlled path through BYOK or Agent Vault. The kill switch is activated by Rhumb operators when upstream provider budgets are exhausted or a security concern is detected.

Prevention

Maintain x402 or a provider-controlled path through BYOK or Agent Vault for mission-critical workflows. This kill switch exists specifically to protect agents from unexpected costs.