Assemblyai

AssemblyAI provides speech-to-text transcription as its core, with AI-powered features layered on top: summarization, sentiment analysis, topic detection, content moderation, entity detection, and speaker diarization. LeMUR (Language Model for Understanding Recorded Audio) enables LLM-powered Q&A and analysis over transcribed content. For agents processing audio — podcast analysis, meeting transcription, customer call review, content moderation — AssemblyAI's API covers the full pipeline from upload to structured output. The async processing model means agents submit audio and poll or receive webhook notification when transcription completes. Real-time streaming transcription via WebSocket is available for live audio. Accuracy is competitive with major cloud providers. The API design is clean and focused on its domain.

Single API key passed via authorization header. Keys are generated in the AssemblyAI dashboard. The key grants full API access — no permission scoping. No OAuth or delegated access for multi-tenant applications. Uploaded audio files are accessible only via the API with the same key — no public URLs for uploaded content. Audio data is deleted after processing by default (configurable retention). For agents processing sensitive audio (customer calls, legal recordings), the data handling policy is relevant: AssemblyAI does not use customer data for model training. No IP restriction on API keys. No temporary credential mechanism. The auth model is simple — appropriate for the single-purpose nature of the API.

Documentation at assemblyai.com/docs is well-structured, with quickstart guides, API reference, and feature-specific guides. Code examples are provided in Python, JavaScript/TypeScript, Go, Java, and Ruby. The official Python SDK (assemblyai) and JavaScript SDK provide high-level abstractions that handle the upload-submit-poll workflow automatically. LeMUR documentation includes prompt engineering guidance specific to audio analysis use cases. The blog covers practical applications (meeting summarization, podcast analysis, content moderation). API reference includes all parameters with descriptions and example responses. Playground in the dashboard allows testing transcription without code. Documentation quality is consistently strong across the API surface — one of the better-documented AI API platforms.

The transcription workflow is straightforward: upload audio (POST to /v2/upload or provide a URL), submit for transcription (POST to /v2/transcript with configuration), and retrieve results (GET /v2/transcript/{id}). Audio intelligence features are enabled via boolean flags on the transcription request: speaker_labels, auto_chapters, sentiment_analysis, entity_detection, content_safety, etc. This single-request configuration model is agent-friendly — no separate API calls for each feature. LeMUR requests reference completed transcripts by ID and accept natural language prompts. Real-time streaming uses WebSocket with JSON message frames. Paragraphs and sentences are returned as structured arrays, not just raw text. Word-level timestamps enable precise audio-text alignment. Response payloads are well-structured JSON throughout.

API errors return JSON with error field. Transcription processing status transitions through queued → processing → completed or error. Failed transcriptions include an error message describing the failure (unsupported format, audio too short, etc.). Webhook delivery for completion events retries on failure. The async model means agents must handle polling or webhook-based completion tracking. Processing time varies: 15-30% of audio duration for standard transcription, longer with audio intelligence features enabled. Rate limits are generous — documented at 100 concurrent transcriptions and 5 requests per second for management endpoints. Real-time streaming requires proper WebSocket connection management including reconnection logic. Audio format support is broad (mp3, wav, m4a, flac, ogg, webm, and more).