Aws Glue

AWS Glue is the standard serverless ETL service in the AWS data ecosystem — deeply integrated with S3, Redshift, Athena, and Lake Formation. For agents: start Glue jobs (triggers PySpark or Python shell scripts), poll job run status for completion detection, manage the Data Catalog (crawlers that auto-discover schema from S3/databases), and handle table partitions. Key use cases: S3-to-Redshift ETL, incremental data lake ingestion, schema discovery and cataloging. Confidence is docs-derived.

Glue API via boto3 (Python). glue.start_job_run(JobName="my-etl-job", Arguments={"--source": "s3://bucket/path", "--output": "s3://output/path"}) starts job. glue.get_job_run(JobName, RunId) retrieves run status. glue.get_job_runs(JobName, MaxResults=10) lists recent runs. glue.start_crawler(Name="my-crawler") triggers crawler. glue.get_crawler(Name) retrieves crawler status. glue.get_tables(DatabaseName="my-db") lists catalog tables. glue.get_table(DatabaseName, Name) retrieves table schema. glue.create_table(DatabaseName, TableInput={Name, StorageDescriptor}) creates catalog table. glue.start_trigger(Name) triggers a workflow trigger. glue.get_workflow_run(Name, RunId) retrieves workflow status.

IAM-based auth: SigV4 via boto3. IAM roles: glue:StartJobRun, glue:GetJobRun, glue:GetJobRuns, glue:StartCrawler, glue:GetCrawler, glue:GetTables. Glue Job service role: separate IAM role assumed by Glue job execution — must have S3 access, Redshift access, etc. HTTPS enforced. No OAuth2. VPC connectivity: Glue can connect to VPC resources (RDS, Redshift in private subnets) via Glue connection with VPC config.

Job run: async — start_job_run returns RunId; poll get_job_run until SUCCEEDED/FAILED/STOPPED. Crawler: async similarly. Job failure: check ErrorMessage in job run details. DPU (Data Processing Units): Glue bills per DPU-hour — set MaxCapacity carefully. Job timeout: default 2880 minutes — set explicit timeout for runaway jobs. Glue uptime at status.aws.amazon.com. Spark context: Glue PySpark jobs get GlueContext (extension of SparkContext) — use Glue DynamicFrames or standard Spark DataFrames. Bookmarks: Glue job bookmarks track processed data for incremental loads.

docs.aws.amazon.com/glue covers job authoring (PySpark, Python shell), crawler configuration, Data Catalog management, connection types, and Glue Studio (visual ETL). Getting started: AWS free tier, create Glue IAM role, create job — first job run in under 15 minutes. boto3 glue client: pip install boto3. AWS Glue Studio for no-code ETL design. Community via AWS re:Post and Stack Overflow (amazon-web-services + glue tags). Comprehensive documentation for a complex ETL platform.