Erasure Coding and Distribution

IronShard stores every object as a set of erasure-coded fragments spread across providers and regions, not as full replicas. An object is split into k slices and encoded into k + m fragments. Any k of them reconstruct the object, so up to m fragments can be lost at once with no data loss and no degraded reads. The storage cost is (k + m) / k of the object's size, against n times the size for n-way replication.

This page covers the encoding, the storage footprint compared to replication, how fragments are placed across failure domains, and geo-fencing. How the fragment count and gateway caches adapt to demand is covered in Adaptive Storage Representation.

How an Object Is Encoded

Every write passes through the ingest pipeline: the payload is compressed, divided into slices that are each encrypted with their own key, then encoded and distributed. Encoding works on the ciphertext:

1. The compressed payload is divided into k equal slices, and each slice is encrypted with its own key.
2. An encoder produces k + m fragments, each a linear combination of the k slices with random coefficients over a finite field. Every fragment is the size of one slice.
3. The fragments are written to distinct storage locations across providers and regions, with fewer than k at any one provider and all of them inside the geo-fence set on the bucket or object.

Reconstruction reads any k fragments and solves for the k slices. There are no "data" and "parity" fragments, so the gateway reads the k nearest or fastest and ignores the rest.

Erasure coding provides durability, not secrecy. Fewer than k fragments cannot reconstruct the payload but still carry constraints on it. Confidentiality comes from the per-slice encryption before encoding: every fragment is ciphertext-derived and unreadable without the keys.

Three numbers describe any scheme:

Storage Footprint Compared to Replication

Replication buys fault tolerance one full copy at a time. Surviving m losses costs m + 1 copies. Erasure coding buys the same tolerance for m extra slices, each 1 / k of the object. The wider the code, the smaller the premium per unit of tolerance.

The table applies the formula to a 1 PB dataset. The coding schemes are illustrative points on the curve, not a list of fixed product tiers; IronShard chooses k and m per object and adjusts them as described in Adaptive Storage Representation.

Two things follow from the table:

• The same tolerance at less than half the cost. An 8+2 code survives the same two simultaneous losses as three-way replication and stores 1.25 PB instead of 3 PB. The 1.75 PB difference is storage that is never bought, never transferred to a second and third provider, and never repaired.
• Tolerance scales without linear cost. Going from tolerating 2 losses to 4 costs 200 percentage points under replication and 15 under a 10+4 code. Wider codes spread each object over more locations, so a full read touches more places; IronShard offsets this with parallel reads and gateway caching.

Because compression precedes encoding, the bytes on disk for compressible data fall below the footprint figure. The footprint is a ratio against the encrypted payload, not against the logical object size.

Failure Domains Across Providers and Regions

Tolerance of m fragments only protects against correlated failure if the fragments sit in independent failure domains. IronShard places the k + m fragments of an object at distinct locations across multiple providers and regions, so losing a location, a region, or an entire provider removes only the fragments held there. Reads continue as long as k remain reachable, and the missing fragments are rebuilt in the background. Because no provider holds k or more fragments, no provider ever holds a complete ciphertext, let alone a complete object.

Geo-Fencing

A geo-fence defines the approved regions for a bucket or an individual object. IronShard blocks storage, caching, and reconstruction outside the permitted zones, so every fragment, every cached slice, and every reassembly of the object stays inside them. A fence narrows which locations are eligible for the k + m fragments; the tolerance and footprint arithmetic is unchanged. Fragment writes, placement decisions, reconstructions, and fence enforcement outcomes are all recorded in the audit log.

What This Means for AI Workloads

• Training corpora and feature stores are stored once at a 125% to 150% footprint rather than three times over, which is where most of the cost difference against replicated multi-cloud storage comes from.
• Retrieval and inference pipelines read from the nearest k fragments, so a provider or regional outage changes which locations are read, not whether the read succeeds.

Summary