Databricks

Databricks unifies data engineering (Spark-based ETL), data warehousing (Databricks SQL on Delta Lake), and machine learning (MLflow) in one platform. For agents: the Jobs API enables programmatic job submission and monitoring; the SQL Statement Execution API runs SQL queries against Delta tables; the MLflow API tracks experiments and models. Unity Catalog provides fine-grained data governance with column-level security and lineage. Delta Live Tables automate streaming and batch pipeline orchestration. Databricks SQL competes with Snowflake on query performance for analytical workloads. Serverless compute eliminates cluster management overhead. Confidence is docs-derived.

REST API at {workspace_url}/api. Resources: jobs, clusters, sql/statements, notebooks, experiments (MLflow), models. POST /api/2.1/jobs/create defines a job ({tasks: [{notebook_task, spark_python_task, sql_task}], clusters}). POST /api/2.1/jobs/run-now triggers a job run. GET /api/2.1/jobs/runs/get?run_id={id} polls run status. POST /api/2.0/sql/statements executes SQL ({statement: "SELECT ...", warehouse_id}). GET /api/2.0/sql/statements/{id} polls SQL result (sync or async mode). POST /api/2.0/mlflow/experiments/create creates ML experiment. Databricks Python SDK: from databricks.sdk import WorkspaceClient; w = WorkspaceClient(); w.jobs.run_now(job_id=...). SQL connector: databricks-sql-connector for pandas/SQLAlchemy workflows.

PAT (Personal Access Token): Authorization: Bearer {token}. Tokens from Databricks workspace → User Settings → Developer → Access Tokens. Workspace-scoped. HTTPS enforced. No OAuth2 for API. Service principals: create for CI/CD automation (recommended over personal tokens for production). OAuth M2M: service principal + client secret for advanced scenarios. Unity Catalog: GRANT SELECT ON TABLE to PRINCIPAL — column-level security, row filters, dynamic data masking.

Standard HTTP status codes with JSON error bodies. Job runs: PENDING → RUNNING → SUCCEEDED/FAILED/CANCELLED. SQL statements: PENDING → RUNNING → SUCCEEDED/FAILED. SQL errors include error_code, message, and SQL position. Cluster auto-termination: idle clusters terminate after configurable time (cost management). Serverless compute: no cold start concern. Job failure notifications: configure webhook or email alerts. Databricks uptime at status.databricks.com. Long-running Spark jobs: monitor via run_id + cluster event log. Unity Catalog access errors: PERMISSION_DENIED with detailed resource and permission info.

docs.databricks.com is comprehensive — REST API reference, Python SDK guide, Databricks SQL docs, Delta Lake reference, MLflow integration, Unity Catalog governance guide, and Terraform provider docs. Getting started: Databricks Community Edition (free), or trial on AWS/Azure/GCP. Python SDK: pip install databricks-sdk. SQL connector: pip install databricks-sql-connector. Databricks Academy for structured learning. Community via Databricks Community Forum and GitHub. Extensive enterprise documentation reflects large customer base.