Secure Ingest Pipeline

Every object written to IronShard passes through four stages: compress, encrypt, encode, distribute. The order is deliberate. Compression precedes encryption because ciphertext does not compress, and encryption precedes encoding so that every fragment is derived from ciphertext.

1. Compression

Data is compressed with adaptive algorithms that balance throughput against size reduction. The gain depends on the input: text, logs, and columnar data shrink substantially, while encrypted and already-compressed formats gain little. Compressible formats commonly shrink enough to cover the redundancy we add for extended locality and ultra-high availability. For details, see Erasure Coding and Distribution.

2. Encryption

The payload is divided into slices, and each slice is encrypted with its own cryptographically secure random key. Data encryption keys are KMS-encrypted with and held in IronShard's metadata plane, never stored alongside the fragments, and never seen by fragment storage providers. A key is only released after the read request passed the access policy check. Data is also encrypted with industry-standard TLS in transit between the client, the gateway, and storage locations. No IronShard component accepts plain http requests or deprecated ciphers.

3. Erasure Coding

The encrypted slices are encoded into k + m fragments. Any k reconstruct the object, and up to m can be lost with no data loss. The encoding and the storage footprint against replication are covered in Erasure Coding and Distribution; how k and m adapt to demand is covered in Adaptive Storage Representation.

4. Distribution

Fragments are streamed to distinct locations across independent providers and regions. Placement draws on the access history of similar objects, the geolocation of similar datasets, and the predicted access routes of agents and compute, and it stays within a geo-fenced region.

Guarantees

A storage provider only holds fragments of encrypted slices. It cannot reconstruct any plaintext byte of the user object. Collecting a sufficient number of fragments allows reconstructing the ciphertext, but the secrecy of encryption still protects the object. The threshold adds containment on top: no single provider ever holds even that, because obtaining a full ciphertext means gathering k fragments from independent locations. Losing a provider costs at most the fragments it held, which the code tolerates up to m. Data leaving the gateway is therefore smaller than it arrived, unreadable to anyone without the key, and recoverable from a subset of its fragments.