Honeycomb

Honeycomb takes a fundamentally different approach to observability: every request generates a wide event (potentially hundreds of fields), stored without pre-aggregation, and queryable at high cardinality. For agents analyzing production behavior, this means any dimension can be sliced and diced without predefined dashboards — BubbleUp automatically surfaces anomalous field values. OpenTelemetry is the recommended ingestion path, making Honeycomb interoperable with the OTel ecosystem. The Query API enables agents to run GROUP BY, HEATMAP, and percentile calculations over event data. The data model favors exploration over predefined metrics — strong for debugging agents, less conventional for agents expecting Prometheus-style metric queries. Free tier includes 20M events/month.

Honeycomb's core API operations are event ingestion and querying. Events are sent via POST to /1/events/{dataset} or via OpenTelemetry's OTLP endpoint. Each event is a flat JSON object with arbitrary fields. The Query API lets agents construct queries with breakdowns, filters, calculations, and time ranges — POST to /1/queries/{dataset}. Results include series data suitable for visualization. Derived columns allow computed fields. SLOs and Burn Alerts are manageable via API. Triggers provide threshold-based alerting. The Markers API records deployment events. The API design is clean and event-centric, but agents accustomed to Prometheus-style instant/range queries need to adapt to Honeycomb's calculate-then-fetch pattern (create query, get results, poll for completion on large datasets).

Event ingestion is forgiving: Honeycomb accepts events with arbitrary schemas and handles schema evolution automatically — new fields appear immediately as queryable dimensions. Invalid JSON returns 400. Rate limits are tied to your plan's events-per-second allocation; exceeding it returns 429 with Retry-After. The Query API returns results synchronously for small datasets and asynchronously for large ones — agents must handle both patterns. Queries that scan too much data return timeout errors with suggestions to narrow the time range. Column data type conflicts (string vs. number for the same field name) are handled by creating type-suffixed columns, which can confuse agents expecting a consistent schema. Event ingestion latency is typically under 10 seconds to queryability.

API keys are the primary credential, with two types: Ingest keys (write-only, for sending events) and Configuration keys (read/write for queries, triggers, SLOs, markers). This clean separation means agents can have write-only credentials for telemetry emission without query access. Keys are scoped to an environment (production, staging). No fine-grained per-dataset permissions on API keys — a Configuration key can access all datasets in the environment. Team management uses roles (member, admin) for console access. No OAuth. OpenTelemetry ingestion uses the Ingest key as the x-honeycomb-team header or OTLP Authorization header. Key rotation is manual. No temporary credential mechanism.

Documentation at docs.honeycomb.io is well-written and conceptually oriented — Honeycomb invests in explaining their observability philosophy, not just API mechanics. Getting started guides cover OpenTelemetry integration for major languages. The Query API reference includes working examples. Honeycomb's blog and Pollinators community (Slack) provide strong secondary resources. Beeline SDKs (Honeycomb's proprietary auto-instrumentation) are being deprecated in favor of OpenTelemetry, which is the right long-term direction but means some older tutorial content references deprecated libraries. The sandbox environment allows experimentation without production data. Ecosystem is smaller than Datadog or New Relic — fewer third-party integrations and community-maintained tools.