Bigquery V2

BigQuery is Google's serverless data warehouse — no cluster provisioning, automatic scaling to petabytes, and pay-per-query pricing (or flat-rate slots for predictable cost). SQL-standard with powerful extensions: ARRAY, STRUCT, GEOGRAPHY, and BigQuery ML for running ML models directly on warehouse data. For agents: the client libraries (Python, Go, Java, Node.js) handle query job submission, result streaming, and dataset/table management. REST API for management operations and custom integrations. Streaming inserts for real-time data ingestion. Free tier (1TB queries/month, 10GB storage) covers meaningful development workloads. Confidence is docs-derived.

REST API at bigquery.googleapis.com/bigquery/v2. Resources: jobs, tables, datasets, tabledata. POST /projects/{project}/jobs submits a query job ({configuration: {query: {query: "SELECT ...", useLegacySql: false}}}). GET /projects/{project}/jobs/{jobId} polls job status and retrieves results. POST /projects/{project}/datasets/{dataset}/tables/{table}/insertAll streams rows to a table. Python client (google-cloud-bigquery): client = bigquery.Client(); query_job = client.query("SELECT ..."); results = query_job.result(). Storage Read API: BigQuery Storage API for high-throughput columnar reads (Arrow/Avro format) — much faster than REST for large result sets. BigQuery ML: CREATE MODEL / ML.PREDICT directly in SQL.

Service Account JSON key: set GOOGLE_APPLICATION_CREDENTIALS env var or use google.oauth2.service_account.Credentials. Application Default Credentials (ADC): gcloud auth application-default login for local dev. OAuth2 for user-level access. IAM roles: roles/bigquery.dataViewer (read), roles/bigquery.dataEditor (write), roles/bigquery.jobUser (run queries). Column-level security via policy tags. Row-level security via row access policies. HTTPS enforced. Service account keys should be managed via Secret Manager, not hardcoded.

Async job model: POST /jobs → get jobId → poll status → retrieve results (or use client library result() which handles polling). Query timeout configurable (default 6 hours). Cost control: set maximumBytesBilled to abort expensive queries before running. Job failure: detailed error message in job status response. Streaming inserts: individual row errors reported in insertErrors array (partial success possible). BigQuery uptime at status.cloud.google.com. SLA: 99.99% for data access. Slot contention: on-demand queries may be slower during peak — use flat-rate slots for consistent latency.

cloud.google.com/bigquery/docs is comprehensive — SQL reference, client library guides, pricing calculator, BigQuery ML docs, Storage API reference, and data loading guides (batch, streaming, transfer service). Getting started: GCP free tier, BigQuery sandbox (no credit card, 60-day trial with free tier limits), first query in under 5 minutes. Python client: pip install google-cloud-bigquery. bq CLI for command-line queries. Community via Google Cloud Community, Stack Overflow (google-bigquery tag), and Google Cloud Discord. Excellent official cookbook and best practices documentation.