Honeyhive

HoneyHive provides the evaluation layer for iterating on LLM pipelines — trace production LLM calls, run automated evaluations (using LLM-as-judge, custom functions, or human review), manage prompt versions, and catch regressions before shipping. For agents: instrument with honeyhive.init() + @honeyhive.trace decorator for automatic tracing, trigger evaluation runs via REST API, compare prompt version performance via A/B test dashboards, and manage evaluation datasets. Integrates with LangChain, LlamaIndex, and OpenAI SDK. Confidence is docs-derived.

Python SDK: pip install honeyhive. Init: import honeyhive; honeyhive.init(api_key="hh_...", project="my-project"). Trace decorator: @honeyhive.trace() def my_llm_call(prompt): return openai_client.chat.completions.create(...). Session tracking: honeyhive.trace(session_id="...", inputs={"prompt": "..."}, outputs={"response": "..."}, metrics={"latency_ms": 250}). REST API (v1): POST /session/start starts a session. POST /session/{session_id}/end ends with outputs + metrics. POST /runs creates an evaluation run. GET /runs/{run_id} retrieves run status. POST /datapoints adds examples to a dataset. GET /prompts retrieves prompt versions. Feedback: POST /feedback/{session_id} adds human feedback to a session.

API key auth: Authorization: Bearer {api_key}. Keys from app.honeyhive.ai → Settings → API Keys. HTTPS enforced. Project isolation: data scoped to project. No OAuth2 for API. Team roles in HoneyHive dashboard. Rate limits: apply per API key tier. Data: LLM prompts/responses stored in HoneyHive — review data retention and privacy policy for sensitive applications.

Tracing: async background upload — minimal latency impact on production LLM calls. Failed traces: SDK retries on network errors; does not block on trace failure. Eval runs: async — triggered via API, progress via GET /runs/{id}. Evaluation functions: Python callables (custom evals) or built-in LLM judge metrics (relevance, faithfulness, toxicity). Dataset management: upload CSV/JSON or programmatically add datapoints. Rate limits: 429 with Retry-After. Large eval runs: batch processing — status polling required.

docs.honeyhive.ai covers Python SDK, REST API reference, evaluation guide, prompt versioning, and dataset management. Getting started: pip install honeyhive, honeyhive.init() — first trace in 3 minutes. Free tier available (limited sessions/month). GitHub: honeyhive-ai (private). Community via Discord. Documentation is solid for an early-stage platform; evaluation configuration is well-documented.