Splunk V2

Splunk is the dominant enterprise log analysis and SIEM platform — ingests massive volumes of machine data and provides SPL (Search Processing Language) for powerful log analytics, anomaly detection, and security investigations. For agents: HTTP Event Collector (HEC) is the high-throughput ingestion path (POST events with a token — no auth per-event overhead); the REST API runs SPL searches, retrieves search results, manages saved searches, and triggers alert actions. On-premises or Splunk Cloud. Very widely deployed in financial services, healthcare, and government. Confidence is docs-derived.

REST API at {splunk_host}:8089/services. HEC: POST {splunk_host}:8088/services/collector/event (Authorization: Splunk {hec_token}, body: {event: {...}, sourcetype, index}). Search: POST /services/search/jobs creates a search job ({search: "search index=main sourcetype=access_combined | stats count by status"}). GET /services/search/jobs/{sid} checks job status. GET /services/search/jobs/{sid}/results retrieves results (output_mode=json). Saved searches: POST /services/saved/searches creates a saved search + alert. GET /services/saved/searches/{name}/history retrieves job history. One-shot search: POST /services/search/jobs (exec_mode=oneshot) for small, fast queries. KV Store: REST API for key-value lookup tables.

Token-based auth: Authorization: Bearer {token} for modern Splunk instances. Or basic auth: Authorization: Basic base64(username:password). Token management: Settings → Tokens in Splunk Web. HEC tokens: separate from user tokens — HEC-specific tokens with index/sourcetype defaults. HTTPS enforced (self-signed cert common for on-premises — trust or add CA). Splunk Cloud: Splunk-managed TLS. Role-based access: Splunk user roles (admin, power, user) control search and data access. Service accounts: create dedicated Splunk user for agent automation.

Standard HTTP status codes with XML or JSON error bodies (configurable via output_mode). Search jobs: async — create job → poll status (RUNNING/DONE) → retrieve results. TTL: search jobs expire (default 600s after completion — retrieve results promptly). HEC: 200 for successful event acceptance; 400 for malformed JSON; 403 for invalid HEC token; 503 if Splunk is busy. Splunk Cloud uptime at splunkstatus.com. On-premises: operator-managed reliability. SPL: search performance depends on data volume and index configuration. Results pagination: count/offset for large result sets.

docs.splunk.com covers REST API reference, HEC documentation, SPL reference, and Splunk Cloud vs. on-premises differences. Getting started: Splunk Free (500MB/day, 60-day full trial), install, configure HEC, first event in under 15 minutes. Splunk Python SDK: pip install splunk-sdk. SPL quickstart: simple queries like "search index=_internal | stats count by sourcetype". Community via Splunk Community (community.splunk.com) — very active. Extensive documentation for a mature enterprise platform; SPL is powerful but has a learning curve.