S3 Quickstart

Connect any S3 client to IronShard in a few minutes.

Getting Started

Overview

IronShard speaks the Amazon S3 API. Any tool that talks to S3, including the AWS CLI, Boto3, the JavaScript, Go, Java, .NET, and PHP SDKs, rclone, and s3cmd, works against it with a changed endpoint and no client library of ours. What differs is underneath: every object is encrypted, erasure-coded into fragments, and distributed across the providers and locations you allow, so no single provider holds a complete object.

Every client needs the same four settings:

SettingValue
Endpointhttps://s3.ironshard.ai
SignatureAWS Signature Version 4 (s3v4)
AddressingPath-style bucket addressing (required)
Multipart size32 MB threshold and chunk size. Matches IronShard's internal chunk size and gives the fastest transfers

1. Create an Access Key

In the console, open S3 Access Keys → New Key. When asked where the key can write, choose A new bucket if it will create buckets; the examples below do.

The secret access key is shown once, when the key is created. Copy it into your secret manager before closing the dialog. A lost secret cannot be recovered; you create a replacement key and delete the old one.

In the description, name the service or job that will use the key, not the person creating it. One key per consumer keeps request logs legible and lets you retire a single workload without rotating credentials everywhere else.

The examples on this page use the placeholders IRONSHARD_ACCESS_KEY and IRONSHARD_SECRET_KEY. Prefer environment variables or a secrets store over hard-coded keys.

2. Pick a Storage Policy

Every bucket is created under a Storage Policy, the rule set that decides which providers and locations hold its fragments. S3 clients have no storage-policy field, so IronShard reads the policy from the one field every S3 client already sends at bucket creation: the region (--region in the CLI, LocationConstraint in the SDKs, bucket_location in s3cmd, location_constraint in rclone).

Region = Storage Policy. Wherever the examples below say STORAGE_POLICY_NAME, substitute the name of a Storage Policy from your console. An empty region (--region "") selects your account's default Storage Policy. The region configured on the client itself (for example us-east-1 in the SDK constructors) is different: the SDK requires it for request signing, any value works, and IronShard ignores it. Only the value sent at bucket creation selects the policy.

Command Line

AWS CLI

Install AWS CLI v2. Version 1 does not support the endpoint_url profile setting used below. See also the configuration files reference.

Config

~/.aws/config:

[profile ironshard]
endpoint_url = https://s3.ironshard.ai
signature_version = s3v4
s3 =
  addressing_style = path
  multipart_threshold = 32MB
  multipart_chunksize = 32MB
s3api =
  addressing_style = path

Credentials

~/.aws/credentials:

[ironshard]
aws_access_key_id = IRONSHARD_ACCESS_KEY
aws_secret_access_key = IRONSHARD_SECRET_KEY
export AWS_PROFILE=ironshard
export AWS_ACCESS_KEY_ID="IRONSHARD_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="IRONSHARD_SECRET_KEY"

With the profile exported, drop --profile ironshard from the commands below.

Try It

# Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
aws s3 mb s3://my-new-bucket --profile ironshard --region "STORAGE_POLICY_NAME"

# Upload
aws s3 cp ./report.parquet s3://my-new-bucket/ --profile ironshard

# List
aws s3 ls s3://my-new-bucket --profile ironshard

# Download (byte-range reads are supported)
aws s3 cp s3://my-new-bucket/report.parquet ./restored.parquet --profile ironshard

# Delete (writes a delete marker, see Versioning below)
aws s3 rm s3://my-new-bucket/report.parquet --profile ironshard

SDKs

Python (Boto3)

Official page · Configuration guide

pip install boto3

example.py:

import boto3
from boto3.s3.transfer import TransferConfig
from botocore.config import Config

# Prefer environment variables or a secrets store over hard-coded keys
s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.ironshard.ai",
    aws_access_key_id="IRONSHARD_ACCESS_KEY",
    aws_secret_access_key="IRONSHARD_SECRET_KEY",
    config=Config(
        signature_version="s3v4",
        s3={"addressing_style": "path"},  # required: path-style bucket addressing
    ),
)

# Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
s3.create_bucket(
    Bucket="my-new-bucket",
    CreateBucketConfiguration={"LocationConstraint": "STORAGE_POLICY_NAME"},
)

# List the objects in a bucket
for obj in s3.list_objects_v2(Bucket="my-new-bucket").get("Contents", []):
    print(obj["Key"])

# 32 MB multipart parts give the fastest transfers
MB = 1024 * 1024
transfer = TransferConfig(multipart_threshold=32 * MB, multipart_chunksize=32 * MB)
s3.upload_file("local-file.bin", "my-new-bucket", "remote-file.bin", Config=transfer)

JavaScript

AWS SDK for JavaScript v3: official page · S3 client reference

npm install @aws-sdk/client-s3

example.mjs:

import { S3Client, CreateBucketCommand, ListObjectsV2Command } from "@aws-sdk/client-s3";

// Prefer environment variables or a secrets store over hard-coded keys
const s3 = new S3Client({
  endpoint: "https://s3.ironshard.ai",
  region: "us-east-1", // required by the SDK for request signing, any value works
  forcePathStyle: true, // required: path-style bucket addressing
  credentials: {
    accessKeyId: "IRONSHARD_ACCESS_KEY",
    secretAccessKey: "IRONSHARD_SECRET_KEY",
  },
});

// Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
await s3.send(new CreateBucketCommand({
  Bucket: "my-new-bucket",
  CreateBucketConfiguration: { LocationConstraint: "STORAGE_POLICY_NAME" },
}));

// List the objects in a bucket
const { Contents = [] } = await s3.send(new ListObjectsV2Command({ Bucket: "my-new-bucket" }));
console.log(Contents.map((o) => o.Key));

For multipart uploads, use @aws-sdk/lib-storage with a 32 MB partSize.

Go

AWS SDK for Go v2: official page · configuring the SDK

go get github.com/aws/aws-sdk-go-v2/aws \
  github.com/aws/aws-sdk-go-v2/config \
  github.com/aws/aws-sdk-go-v2/credentials \
  github.com/aws/aws-sdk-go-v2/service/s3

main.go:

package main

import (
    "context"
    "fmt"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/credentials"
    "github.com/aws/aws-sdk-go-v2/service/s3"
    "github.com/aws/aws-sdk-go-v2/service/s3/types"
)

func main() {
    ctx := context.TODO()

    // Prefer environment variables or a secrets store over hard-coded keys
    cfg, err := config.LoadDefaultConfig(ctx,
        config.WithRegion("us-east-1"), // required by the SDK for request signing, any value works
        config.WithCredentialsProvider(
            credentials.NewStaticCredentialsProvider("IRONSHARD_ACCESS_KEY", "IRONSHARD_SECRET_KEY", ""),
        ),
    )
    if err != nil {
        panic(err)
    }

    client := s3.NewFromConfig(cfg, func(o *s3.Options) {
        o.BaseEndpoint = aws.String("https://s3.ironshard.ai")
        o.UsePathStyle = true // required: path-style bucket addressing
    })

    // Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
    _, err = client.CreateBucket(ctx, &s3.CreateBucketInput{
        Bucket: aws.String("my-new-bucket"),
        CreateBucketConfiguration: &types.CreateBucketConfiguration{
            LocationConstraint: types.BucketLocationConstraint("STORAGE_POLICY_NAME"),
        },
    })
    if err != nil {
        panic(err)
    }

    // List the objects in a bucket
    out, err := client.ListObjectsV2(ctx, &s3.ListObjectsV2Input{Bucket: aws.String("my-new-bucket")})
    if err != nil {
        panic(err)
    }
    for _, o := range out.Contents {
        fmt.Println(*o.Key)
    }
}

Java

AWS SDK for Java v2: official page · working with S3

pom.xml:

<dependency>
  <groupId>software.amazon.awssdk</groupId>
  <artifactId>s3</artifactId>
  <version>[2.0,3.0)</version>
</dependency>

build.gradle.kts:

implementation("software.amazon.awssdk:s3:2.+")

Example.java:

import java.net.URI;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3Client;

public class Example {
    public static void main(String[] args) {
        // Prefer environment variables or a secrets store over hard-coded keys
        S3Client s3 = S3Client.builder()
            .endpointOverride(URI.create("https://s3.ironshard.ai"))
            .region(Region.of("us-east-1")) // required by the SDK for request signing, any value works
            .forcePathStyle(true) // required: path-style bucket addressing
            .credentialsProvider(StaticCredentialsProvider.create(
                AwsBasicCredentials.create("IRONSHARD_ACCESS_KEY", "IRONSHARD_SECRET_KEY")))
            .build();

        // Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
        s3.createBucket(b -> b
            .bucket("my-new-bucket")
            .createBucketConfiguration(c -> c.locationConstraint("STORAGE_POLICY_NAME")));

        // List the objects in a bucket
        s3.listObjectsV2(b -> b.bucket("my-new-bucket")).contents()
            .forEach(o -> System.out.println(o.key()));
    }
}

.NET

AWS SDK for .NET v4: official page · configuring the SDK

dotnet add package AWSSDK.S3

Program.cs:

using Amazon.S3;
using Amazon.S3.Model;

// Prefer environment variables or a secrets store over hard-coded keys
var config = new AmazonS3Config
{
    ServiceURL = "https://s3.ironshard.ai",
    ForcePathStyle = true, // required: path-style bucket addressing
};
var s3 = new AmazonS3Client("IRONSHARD_ACCESS_KEY", "IRONSHARD_SECRET_KEY", config);

// Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
await s3.PutBucketAsync(new PutBucketRequest
{
    BucketName = "my-new-bucket",
    BucketRegionName = "STORAGE_POLICY_NAME",
});

// List the objects in a bucket
var response = await s3.ListObjectsV2Async(new ListObjectsV2Request { BucketName = "my-new-bucket" });
foreach (var obj in response.S3Objects ?? []) // null when the bucket is empty
{
    Console.WriteLine(obj.Key);
}

PHP

AWS SDK for PHP v3: official page · configuration guide

composer require aws/aws-sdk-php

example.php:

<?php
require 'vendor/autoload.php';

use Aws\S3\S3Client;

// Prefer environment variables or a secrets store over hard-coded keys
$s3 = new S3Client([
    'version' => 'latest',
    'region' => 'us-east-1', // required by the SDK for request signing, any value works
    'endpoint' => 'https://s3.ironshard.ai',
    'use_path_style_endpoint' => true, // required: path-style bucket addressing
    'credentials' => [
        'key' => 'IRONSHARD_ACCESS_KEY',
        'secret' => 'IRONSHARD_SECRET_KEY',
    ],
]);

// Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
$s3->createBucket([
    'Bucket' => 'my-new-bucket',
    'CreateBucketConfiguration' => ['LocationConstraint' => 'STORAGE_POLICY_NAME'],
]);

// List the objects in a bucket
foreach ($s3->listObjectsV2(['Bucket' => 'my-new-bucket'])['Contents'] ?? [] as $object) {
    echo $object['Key'] . PHP_EOL;
}

Tools

rclone

Official page · S3 backend reference

~/.config/rclone/rclone.conf:

[ironshard]
type = s3
provider = Other
access_key_id = IRONSHARD_ACCESS_KEY
secret_access_key = IRONSHARD_SECRET_KEY
endpoint = https://s3.ironshard.ai
force_path_style = true
location_constraint = STORAGE_POLICY_NAME
chunk_size = 32M

Try It

# Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
rclone mkdir ironshard:my-new-bucket
rclone ls ironshard:my-new-bucket
rclone copy ./local-folder ironshard:my-new-bucket

s3cmd

Official page · usage reference

Setting host_bucket to the plain host, without %(bucket)s, selects path-style bucket addressing.

~/.s3cfg:

[default]
access_key = IRONSHARD_ACCESS_KEY
secret_key = IRONSHARD_SECRET_KEY
host_base = s3.ironshard.ai
host_bucket = s3.ironshard.ai
use_https = True
signature_v2 = False
bucket_location = STORAGE_POLICY_NAME
multipart_chunk_size_mb = 32

Try It

# Create a bucket with the selected Storage Policy (the access key must be allowed to create buckets)
s3cmd mb s3://my-new-bucket
s3cmd ls s3://my-new-bucket

Reference

Versioning and Limits

Versioning Is Always On

Every bucket is versioned and versioning cannot be switched off. A delete writes a delete marker rather than removing data, and earlier versions stay until you delete them explicitly by version ID. Plan for this when you estimate stored volume, and when an agent's cleanup step "frees" space.

What Is and Isn't Supported Today

  • Supported: byte-range reads on GetObject, user-defined object metadata, multipart uploads.
  • Overriding response headers on GetObject (for example response-cache-control) is not supported.
  • Object tagging is not yet available; user-defined object metadata is.
  • There is no CORS configuration, so browser-based clients cannot talk to the endpoint directly. Use a server, an SDK, or the console.

Next Steps