Behavior & Limits

Alien's Storage binding is built on the Apache Arrow object_store crate — the same library used by DataFusion, Delta Lake, and the wider Arrow ecosystem.

Guarantees

On cloud platforms (AWS, GCP, Azure), Alien provisions and manages the storage backing service. These guarantees apply:

Atomic Writes. A put() call either succeeds completely or fails completely. No partial writes are visible to other readers.

Strong Read-After-Write. After a successful put(), any subsequent get() returns the new data immediately. All three cloud providers guarantee this.

Strong List Consistency. After a put() or delete(), list() reflects the change immediately.

Durability. AWS S3 and GCP Cloud Storage: 99.999999999% (11 nines). Azure Blob Storage: 12+ nines with zone-redundant storage.

Conditional Operations. Conditional creates are atomic — if two concurrent callers create the same key, exactly one succeeds. Backed by object_store's PutMode::Create (Rust put_opts).

Conditional Updates (ETag). Compare-and-swap via ETags — update an object only if it hasn't changed since you last read it.

Limits

Alien does not impose rate limits — backend rate limits apply (see platform details).

Operation Semantics

These come from the object_store crate and hold across all backends:

List ordering. list() does not guarantee ordering. Sort on the client if needed.

List prefix matching. Segment-based: prefix "foo/bar" matches "foo/bar/x.json" but not "foo/bar_baz/x.json".

Copy and rename. copy() is atomic when the backend supports it. The Rust-only rename() is copy + delete — not atomic.

Multipart uploads (Rust only). Parts uploaded concurrently in any order. Finalized atomically on complete(). Abandoned uploads cleaned up by the backend.

Object attributes. put() can store provider-neutral attributes (content type, content disposition, content encoding, content language, cache control, custom metadata); get() and head() return them, plus the provider-reported storage class. Writes with unsupported attributes are rejected before any payload is written — the operation fails cleanly rather than storing an object with attributes silently dropped. list() is metadata-only: cloud list APIs don't return attributes.

ETags and versions. ObjectMeta carries the provider eTag and version when available, and put() returns them for the written object — the basis for compare-and-swap flows.

Platform Details

AWS (S3)

• Request rates auto-scale per prefix. No ramp-up required.
• Baseline: 5,500 GET/HEAD + 3,500 PUT/DELETE per second per prefix.
• Versioning and lifecycle rules fully supported via stack configuration.

GCP (Cloud Storage)

• Baseline: ~5,000 reads/sec + ~1,000 writes/sec per bucket.
• Must ramp up gradually (double every 20 minutes) for sustained high load.
• IAM/ACL changes are eventually consistent (~1 minute).
• contentEncoding: "gzip" is rejected before upload: GCS decompressive transcoding omits the response length required for byte-exact reads. Other encodings (e.g. br) are preserved.

Azure (Blob Storage)

• Per-blob limit: 500 requests/sec, 60 MB/s throughput.
• Versioning not supported per container. The versioning stack option is ignored with a warning.
• Lifecycle rules not supported per container. The lifecycle stack option is ignored.
• Requires a Storage Account (provisioned automatically by Alien).

Kubernetes / On-Prem

Storage is not provisioned by Alien on Kubernetes. The cluster operator provides the backing service (S3, MinIO, GCS, etc.) via Helm values. Guarantees depend entirely on the backing service.

Local

• Filesystem-backed directory. Durability depends on OS fsync behavior.
• Deleting a non-existent key returns a not-found error (unlike cloud platforms where it's a no-op).
• Object attributes are not supported — the filesystem cannot persist them, so attribute-bearing put() calls are rejected before any payload is written.

Triggers

Design Decisions

Built on object_store. Battle-tested foundation rather than a custom abstraction. Same library used by DataFusion and Delta Lake.

No Alien-level rate limiting. Cloud provider native rate limits apply. AWS auto-scales, GCP requires ramp-up — we document the difference rather than hiding it.

Permissive key charset. Unlike KV, Storage allows any valid UTF-8 key. Matches developer expectations for file paths.