Apache Kafka

Apache Kafka is the de facto standard for high-throughput event streaming — used by virtually every large-scale data engineering team for event sourcing, ETL pipelines, activity tracking, and microservice communication. For agents: produce messages to topics (async, high-throughput), consume from topics (consumer groups for load-distributed processing), manage topics via Admin API (create, configure retention/compaction), and monitor consumer lag. Self-hostable for full control; managed options (Confluent Cloud, MSK, Aiven) reduce operational overhead. Confidence is docs-derived.

Kafka Producer/Consumer (confluent-kafka-python). Producer: p = Producer({bootstrap.servers: "localhost:9092"}); p.produce("topic", key, value, callback=delivery_report); p.flush(). Consumer: c = Consumer({bootstrap.servers, group.id, auto.offset.reset: "earliest"}); c.subscribe(["topic"]); msg = c.poll(timeout=1.0); c.commit(). Admin API: a = AdminClient({bootstrap.servers}); a.create_topics([NewTopic("my-topic", num_partitions=3, replication_factor=1)]); a.list_topics() lists topics; a.delete_topics(["old-topic"]). Kafka Streams: Java library for stateful stream processing (joins, aggregations, windowing). REST Proxy (Confluent): HTTP-based produce/consume without native clients. Schema Registry: Avro/Protobuf schema management for strong typing.

Production auth: SASL/PLAIN or SASL/SCRAM-SHA-512 with TLS. Config: security.protocol=SASL_SSL, sasl.mechanisms=PLAIN, sasl.username={user}, sasl.password={password}. Self-hosted development: no auth by default. ACLs: topic-level read/write permissions per user (bin/kafka-acls.sh or AdminClient). HTTPS: configure listener TLS certificates. Managed services (Confluent, MSK): use service-specific credentials (API key/secret or IAM).

Delivery guarantee: at-least-once by default (enable idempotent producer for exactly-once within session). Consumer group rebalancing: brief pause during partition reassignment — handle gracefully. Lag monitoring: measure consumer lag (consumer_position - latest_offset) for pipeline health. Self-hosted reliability: operator-managed (ZooKeeper or KRaft for cluster coordination). Confluent/MSK uptime at their respective status pages. Partitioning: key-based routing to partitions — choose keys carefully for even distribution. Retention: configure retention.ms and retention.bytes per topic.

kafka.apache.org/documentation covers producer/consumer API, Admin API, configuration reference, Kafka Streams guide, and security documentation. Getting started: docker run confluentinc/cp-kafka — running in minutes; produce first message in under 5 minutes. confluent-kafka-python: pip install confluent-kafka. Community via Kafka Mailing Lists, Confluent Community Forum, and Stack Overflow (apache-kafka tag). Mature, extensive documentation with decades of community resources.