Opik

Opik is Comet ML's open-source LLM observability stack: instrument LLM calls with the Python SDK for automatic tracing, then evaluate traces using built-in metrics (hallucination detection, factuality, toxicity, answer relevance) or custom scoring functions. Opik Cloud provides hosted storage and dashboards; self-hosted via Docker for air-gapped environments. For agents: integrate Opik decorators into LLM pipeline code, run automated eval suites in CI/CD, and track quality metrics across prompt/model changes over time. Integrations with LangChain, LlamaIndex, OpenAI, Anthropic for auto-tracing with minimal code changes. Confidence is docs-derived.

Python SDK (opik): decorate functions with @opik.track for automatic trace capture. opik.evaluate() runs eval dataset through task function + scoring functions and logs results. REST API: POST /api/v1/traces ingests trace data. POST /api/v1/datasets/{id}/items adds items to an eval dataset. GET /api/v1/experiments/{id}/items retrieves experiment results. LangChain integration: OpikTracer callback auto-traces all LangChain calls. OpenAI integration: opik.configure() + opik_context.track_openai(client) auto-logs all OpenAI completions. LlamaIndex: OpikCallbackHandler for automatic trace capture.

API key auth: OPIK_API_KEY environment variable (SDK) or Authorization: Bearer {api_key} (REST). Keys from Opik Cloud Settings → API Keys. Self-hosted: no auth by default; configure auth layer for exposed instances. HTTPS enforced on Opik Cloud. Self-hosted: operator-managed TLS. No OAuth2. Project/workspace scoping for team isolation.

SDK: async trace ingestion with local queue; opik.flush() ensures all traces sent before exit. REST API: standard HTTP status codes with JSON error bodies. 401 for invalid API key, 400 for malformed trace payload. Trace data eventually consistent in the Opik dashboard (seconds latency). Eval runs: synchronous within SDK — blocking until all eval items processed. Self-hosted reliability: operator-managed. Opik Cloud uptime at status.comet.com. Open-source means community support for self-hosted issues.

comet.com/docs/opik provides SDK reference, quickstarts for common LLM frameworks, REST API reference, eval metrics documentation, and self-hosting guide (Docker Compose). Getting started: pip install opik, configure API key, first trace captured in 2 lines of code. Self-hosted: docker compose up from GitHub repo. Open-source MIT license (GitHub: comet-ml/opik, 2k+ stars). Community via GitHub Issues and Comet Discord. Actively maintained; Comet ML backing provides engineering resources.