Aws S3

S3 is the gravitational center of cloud storage — agents that need to stage files, persist artifacts, or read from data lakes will encounter it almost inevitably. The REST API is mature and well-understood, with SDKs in every major language (Boto3 being the gold standard). Pre-signed URL generation lets agents hand off secure temporary access without exposing credentials. Multi-part upload handles large objects cleanly. The main friction for agents is IAM: configuring the right policies requires understanding of resource ARNs, condition keys, and the subtle difference between bucket policies and IAM policies. Region-specific endpoints add another layer of configuration. Despite this complexity, the sheer ecosystem depth makes S3 the most capable object store available.

S3's REST API established the conventions that R2, B2, and Wasabi all emulate. ListObjectsV2 with ContinuationToken provides reliable pagination over arbitrarily large buckets. GET/PUT/DELETE on individual keys is straightforward. Response formats are XML by default on raw REST calls — most agents will want SDK wrappers that normalize to JSON. Content-addressed operations via ETag enable efficient change detection. HeadObject is useful for metadata checks without downloading content. The API surface is large (bucket lifecycle, versioning, replication, inventory) but agents typically need only the core CRUD plus pre-signed URL generation. Batch operations via S3 Batch Operations exist but are overkill for most agent workloads.

IAM access keys (access key ID + secret) are the standard credential pattern. For agents, the recommended approach is STS AssumeRole with temporary credentials — avoids long-lived secrets and supports fine-grained scoping. Bucket policies provide resource-level control but interact with IAM policies in non-obvious ways (explicit deny wins). CORS configuration is required for browser-side access — irrelevant for server-side agents but a common misconfiguration source. Service-linked roles can simplify cross-service access patterns. The main agent pitfall: a single misconfigured bucket policy can silently deny access even when IAM allows it, and 403 responses don't indicate which policy layer rejected the request.

Documentation is exhaustive — thousands of pages spanning API reference, SDK guides, best practices, and solution architectures. This depth is both S3's strength and weakness: finding the specific guidance for an agent use case requires navigating a sprawling doc tree. Boto3 documentation for Python is the most agent-relevant and includes clear examples. The AWS CLI (aws s3 and aws s3api) is excellent for testing and debugging. CloudTrail provides audit trails of all S3 operations, useful for verifying agent behavior. The console UI is functional but not API-oriented. Getting-started guides assume human operators, not agent callers — agents need the SDK reference more than the tutorials.

Error responses include a RequestId that maps to CloudTrail entries — essential for debugging production agent issues. SlowDown (503) errors occur under sustained high-throughput to a single partition and require exponential backoff with jitter. AccessDenied (403) messages are famously unhelpful — they don't indicate whether the bucket policy, IAM policy, or SCP blocked the request. NoSuchKey (404) is clean and reliable. S3 achieved strong read-after-write consistency in 2020, eliminating a long-standing eventual consistency gotcha on overwrite PUTs. Rate limits are per-prefix (3,500 PUT/POST/DELETE and 5,500 GET per second per prefix), which agents can work around with key prefix distribution.