S3 Quickstart
Connect any S3 client to IronShard in a few minutes.
Getting Started
Overview
IronShard speaks the Amazon S3 API. Any tool that talks to S3, including the AWS CLI, Boto3, the JavaScript, Go, Java, .NET, and PHP SDKs, rclone, and s3cmd, works against it with a changed endpoint and no client library of ours. What differs is underneath: every object is encrypted, erasure-coded into fragments, and distributed across the providers and locations you allow, so no single provider holds a complete object.
Every client needs the same four settings:
| Setting | Value |
|---|---|
| Endpoint | https://s3.ironshard.ai |
| Signature | AWS Signature Version 4 (s3v4) |
| Addressing | Path-style bucket addressing (required) |
| Multipart size | 32 MB threshold and chunk size. Matches IronShard's internal chunk size and gives the fastest transfers |
1. Create an Access Key
In the console, open S3 Access Keys → New Key. When asked where the key can write, choose A new bucket if it will create buckets; the examples below do.
The secret access key is shown once, when the key is created. Copy it into your secret manager before closing the dialog. A lost secret cannot be recovered; you create a replacement key and delete the old one.
In the description, name the service or job that will use the key, not the person creating it. One key per consumer keeps request logs legible and lets you retire a single workload without rotating credentials everywhere else.
The examples on this page use the placeholders IRONSHARD_ACCESS_KEY and IRONSHARD_SECRET_KEY. Prefer environment variables or a secrets store over hard-coded keys.
2. Pick a Storage Policy
Every bucket is created under a Storage Policy, the rule set that decides which providers and locations hold its fragments. S3 clients have no storage-policy field, so IronShard reads the policy from the one field every S3 client already sends at bucket creation: the region (--region in the CLI, LocationConstraint in the SDKs, bucket_location in s3cmd, location_constraint in rclone).
Region = Storage Policy. Wherever the examples below say STORAGE_POLICY_NAME, substitute the name of a Storage Policy from your console. An empty region (--region "") selects your account's default Storage Policy. The region configured on the client itself (for example us-east-1 in the SDK constructors) is different: the SDK requires it for request signing, any value works, and IronShard ignores it. Only the value sent at bucket creation selects the policy.
Command Line
AWS CLI
Install AWS CLI v2. Version 1 does not support the endpoint_url profile setting used below. See also the configuration files reference.
Config
~/.aws/config:
[profile ironshard]
endpoint_url = https://s3.ironshard.ai
signature_version = s3v4
s3 =
addressing_style = path
multipart_threshold = 32MB
multipart_chunksize = 32MB
s3api =
addressing_style = path
Credentials
~/.aws/credentials:
[ironshard]
aws_access_key_id = IRONSHARD_ACCESS_KEY
aws_secret_access_key = IRONSHARD_SECRET_KEY
Try It
# Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
aws s3 mb s3://my-new-bucket --profile ironshard --region "STORAGE_POLICY_NAME"
# Upload
aws s3 cp ./report.parquet s3://my-new-bucket/ --profile ironshard
# List
aws s3 ls s3://my-new-bucket --profile ironshard
# Download (byte-range reads are supported)
aws s3 cp s3://my-new-bucket/report.parquet ./restored.parquet --profile ironshard
# Delete (writes a delete marker, see Versioning below)
aws s3 rm s3://my-new-bucket/report.parquet --profile ironshard
SDKs
Python (Boto3)
Official page · Configuration guide
pip install boto3
example.py:
import boto3
from boto3.s3.transfer import TransferConfig
from botocore.config import Config
# Prefer environment variables or a secrets store over hard-coded keys
s3 = boto3.client(
"s3",
endpoint_url="https://s3.ironshard.ai",
aws_access_key_id="IRONSHARD_ACCESS_KEY",
aws_secret_access_key="IRONSHARD_SECRET_KEY",
config=Config(
signature_version="s3v4",
s3={"addressing_style": "path"}, # required: path-style bucket addressing
),
)
# Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
s3.create_bucket(
Bucket="my-new-bucket",
CreateBucketConfiguration={"LocationConstraint": "STORAGE_POLICY_NAME"},
)
# List the objects in a bucket
for obj in s3.list_objects_v2(Bucket="my-new-bucket").get("Contents", []):
print(obj["Key"])
# 32 MB multipart parts give the fastest transfers
MB = 1024 * 1024
transfer = TransferConfig(multipart_threshold=32 * MB, multipart_chunksize=32 * MB)
s3.upload_file("local-file.bin", "my-new-bucket", "remote-file.bin", Config=transfer)
JavaScript
AWS SDK for JavaScript v3: official page · S3 client reference
npm install @aws-sdk/client-s3
example.mjs:
import { S3Client, CreateBucketCommand, ListObjectsV2Command } from "@aws-sdk/client-s3";
// Prefer environment variables or a secrets store over hard-coded keys
const s3 = new S3Client({
endpoint: "https://s3.ironshard.ai",
region: "us-east-1", // required by the SDK for request signing, any value works
forcePathStyle: true, // required: path-style bucket addressing
credentials: {
accessKeyId: "IRONSHARD_ACCESS_KEY",
secretAccessKey: "IRONSHARD_SECRET_KEY",
},
});
// Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
await s3.send(new CreateBucketCommand({
Bucket: "my-new-bucket",
CreateBucketConfiguration: { LocationConstraint: "STORAGE_POLICY_NAME" },
}));
// List the objects in a bucket
const { Contents = [] } = await s3.send(new ListObjectsV2Command({ Bucket: "my-new-bucket" }));
console.log(Contents.map((o) => o.Key));
For multipart uploads, use @aws-sdk/lib-storage with a 32 MB partSize.
Go
AWS SDK for Go v2: official page · configuring the SDK
go get github.com/aws/aws-sdk-go-v2/aws \
github.com/aws/aws-sdk-go-v2/config \
github.com/aws/aws-sdk-go-v2/credentials \
github.com/aws/aws-sdk-go-v2/service/s3
main.go:
package main
import (
"context"
"fmt"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types"
)
func main() {
ctx := context.TODO()
// Prefer environment variables or a secrets store over hard-coded keys
cfg, err := config.LoadDefaultConfig(ctx,
config.WithRegion("us-east-1"), // required by the SDK for request signing, any value works
config.WithCredentialsProvider(
credentials.NewStaticCredentialsProvider("IRONSHARD_ACCESS_KEY", "IRONSHARD_SECRET_KEY", ""),
),
)
if err != nil {
panic(err)
}
client := s3.NewFromConfig(cfg, func(o *s3.Options) {
o.BaseEndpoint = aws.String("https://s3.ironshard.ai")
o.UsePathStyle = true // required: path-style bucket addressing
})
// Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
_, err = client.CreateBucket(ctx, &s3.CreateBucketInput{
Bucket: aws.String("my-new-bucket"),
CreateBucketConfiguration: &types.CreateBucketConfiguration{
LocationConstraint: types.BucketLocationConstraint("STORAGE_POLICY_NAME"),
},
})
if err != nil {
panic(err)
}
// List the objects in a bucket
out, err := client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{Bucket: aws.String("my-new-bucket")})
if err != nil {
panic(err)
}
for _, o := range out.Contents {
fmt.Println(*o.Key)
}
}
Java
AWS SDK for Java v2: official page · working with S3
pom.xml:
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
<version>[2.0,3.0)</version>
</dependency>
Example.java:
import java.net.URI;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;
public class Example {
public static void main(String[] args) {
// Prefer environment variables or a secrets store over hard-coded keys
S3Client s3 = S3Client.builder()
.endpointOverride(URI.create("https://s3.ironshard.ai"))
.region(Region.of("us-east-1")) // required by the SDK for request signing, any value works
.forcePathStyle(true) // required: path-style bucket addressing
.credentialsProvider(StaticCredentialsProvider.create(
AwsBasicCredentials.create("IRONSHARD_ACCESS_KEY", "IRONSHARD_SECRET_KEY")))
.build();
// Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
s3.createBucket(b -> b
.bucket("my-new-bucket")
.createBucketConfiguration(c -> c.locationConstraint("STORAGE_POLICY_NAME")));
// List the objects in a bucket
s3.listObjectsV2(b -> b.bucket("my-new-bucket")).contents()
.forEach(o -> System.out.println(o.key()));
}
}
.NET
AWS SDK for .NET v4: official page · configuring the SDK
dotnet add package AWSSDK.S3
Program.cs:
using Amazon.S3;
using Amazon.S3.Model;
// Prefer environment variables or a secrets store over hard-coded keys
var config = new AmazonS3Config
{
ServiceURL = "https://s3.ironshard.ai",
ForcePathStyle = true, // required: path-style bucket addressing
};
var s3 = new AmazonS3Client("IRONSHARD_ACCESS_KEY", "IRONSHARD_SECRET_KEY", config);
// Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
await s3.PutBucketAsync(new PutBucketRequest
{
BucketName = "my-new-bucket",
BucketRegionName = "STORAGE_POLICY_NAME",
});
// List the objects in a bucket
var response = await s3.ListObjectsV2Async(new ListObjectsV2Request { BucketName = "my-new-bucket" });
foreach (var obj in response.S3Objects ?? []) // null when the bucket is empty
{
Console.WriteLine(obj.Key);
}
PHP
AWS SDK for PHP v3: official page · configuration guide
composer require aws/aws-sdk-php
example.php:
<?php
require 'vendor/autoload.php';
use Aws\S3\S3Client;
// Prefer environment variables or a secrets store over hard-coded keys
$s3 = new S3Client([
'version' => 'latest',
'region' => 'us-east-1', // required by the SDK for request signing, any value works
'endpoint' => 'https://s3.ironshard.ai',
'use_path_style_endpoint' => true, // required: path-style bucket addressing
'credentials' => [
'key' => 'IRONSHARD_ACCESS_KEY',
'secret' => 'IRONSHARD_SECRET_KEY',
],
]);
// Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
$s3->createBucket([
'Bucket' => 'my-new-bucket',
'CreateBucketConfiguration' => ['LocationConstraint' => 'STORAGE_POLICY_NAME'],
]);
// List the objects in a bucket
foreach ($s3->listObjectsV2(['Bucket' => 'my-new-bucket'])['Contents'] ?? [] as $object) {
echo $object['Key'] . PHP_EOL;
}
Tools
rclone
Official page · S3 backend reference
~/.config/rclone/rclone.conf:
[ironshard]
type = s3
provider = Other
access_key_id = IRONSHARD_ACCESS_KEY
secret_access_key = IRONSHARD_SECRET_KEY
endpoint = https://s3.ironshard.ai
force_path_style = true
location_constraint = STORAGE_POLICY_NAME
chunk_size = 32M
Try It
# Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
rclone mkdir ironshard:my-new-bucket
rclone ls ironshard:my-new-bucket
rclone copy ./local-folder ironshard:my-new-bucket
s3cmd
Official page · usage reference
Setting host_bucket to the plain host, without %(bucket)s, selects path-style bucket addressing.
~/.s3cfg:
[default]
access_key = IRONSHARD_ACCESS_KEY
secret_key = IRONSHARD_SECRET_KEY
host_base = s3.ironshard.ai
host_bucket = s3.ironshard.ai
use_https = True
signature_v2 = False
bucket_location = STORAGE_POLICY_NAME
multipart_chunk_size_mb = 32
Try It
# Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
s3cmd mb s3://my-new-bucket
s3cmd ls s3://my-new-bucket
Reference
Versioning and Limits
Versioning Is Always On
Every bucket is versioned and versioning cannot be switched off. A delete writes a delete marker rather than removing data, and earlier versions stay until you delete them explicitly by version ID. Plan for this when you estimate stored volume, and when an agent's cleanup step "frees" space.
What Is and Isn't Supported Today
- Supported: byte-range reads on
GetObject, user-defined object metadata, multipart uploads. - Overriding response headers on
GetObject(for exampleresponse-cache-control) is not supported. - Object tagging is not yet available; user-defined object metadata is.
- There is no CORS configuration, so browser-based clients cannot talk to the endpoint directly. Use a server, an SDK, or the console.
Next Steps
- Agent Buckets: let an autonomous agent create and operate a bucket over MCP, with no account.
- Erasure Coding and Distribution: how objects are erasure-coded and distributed across providers, plus geo-fencing.
