Performance Tuning Guide¶

Performance optimization strategies for S3 infrastructure provisioned by the S3 Provisioner.

Table of Contents¶

S3 Request Rate Optimization
Transfer Performance
ML Workflow Optimization
Provisioner Performance
Monitoring Performance

S3 Request Rate Optimization¶

S3 automatically scales to high request rates per prefix:

3,500 PUT/COPY/POST/DELETE requests per second per prefix
5,500 GET/HEAD requests per second per prefix

Optimize Folder Structure¶

The provisioned folder structure already distributes data across multiple prefixes:

solutions/customer-churn/
  data/
    raw/              # 5,500 GET/s for raw data reads
    curated/          # 5,500 GET/s for curated data reads
    processed/
      train/          # 5,500 GET/s for training reads
      validation/     # 5,500 GET/s for validation reads
      test/           # 5,500 GET/s for test reads
    inference/        # 5,500 GET/s for inference reads

Each folder is a separate prefix — the provisioned structure gives you parallel throughput by default.

Date Partitioning for High-Volume Data¶

For high-volume ingestion, partition within the provisioned folders:

# Within the provisioned raw/ folder, add date partitions
solutions/customer-churn/data/raw/
  2026-04-01/         # 5,500 GET/s
  2026-04-02/         # 5,500 GET/s
  2026-04-03/         # 5,500 GET/s

Hash Partitioning for Maximum Throughput¶

For extreme throughput requirements (>50,000 requests/second):

solutions/customer-churn/data/processed/train/
  a1/file1.parquet    # 5,500 GET/s
  b2/file2.parquet    # 5,500 GET/s
  c3/file3.parquet    # 5,500 GET/s

Transfer Performance¶

Multipart Upload (Files > 100 MB)¶

import boto3
from boto3.s3.transfer import TransferConfig

config = TransferConfig(
    multipart_threshold=100 * 1024 * 1024,  # 100 MB
    max_concurrency=10,
    multipart_chunksize=10 * 1024 * 1024,   # 10 MB chunks
)

s3 = boto3.client('s3')
s3.upload_file(
    'large-dataset.parquet',
    'edge-prod-b001-us-west-1-s3',
    'solutions/customer-churn/data/raw/large-dataset.parquet',
    Config=config
)

Transfer Acceleration¶

Enable for uploads from distant regions (adds ~$0.04/GB):

aws s3api put-bucket-accelerate-configuration \
  --bucket edge-prod-b001-us-west-1-s3 \
  --accelerate-configuration Status=Enabled

# Use accelerated endpoint
aws s3 cp large-file.csv \
  s3://edge-prod-b001-us-west-1-s3/solutions/customer-churn/data/raw/ \
  --endpoint-url https://edge-prod-b001-us-west-1-s3.s3-accelerate.amazonaws.com

Parallel Downloads¶

import boto3
from boto3.s3.transfer import TransferConfig

config = TransferConfig(
    max_concurrency=20,          # Parallel threads
    multipart_threshold=8 * 1024 * 1024,
)

s3 = boto3.client('s3')
s3.download_file(
    'edge-prod-b001-us-west-1-s3',
    'solutions/customer-churn/models/training/model.tar.gz',
    'model.tar.gz',
    Config=config
)

AWS CLI Tuning¶

# Increase concurrent requests for bulk operations
aws configure set default.s3.max_concurrent_requests 50
aws configure set default.s3.multipart_chunksize 64MB

# Sync with high concurrency
aws s3 sync s3://edge-prod-b001-us-west-1-s3/solutions/customer-churn/data/ ./local-data/

ML Workflow Optimization¶

SageMaker Data Channels¶

Use S3 data channels with ShardedByS3Key for distributed training:

from sagemaker.inputs import TrainingInput

train_input = TrainingInput(
    s3_data=f"s3://edge-prod-b001-us-west-1-s3/solutions/customer-churn/data/processed/train/",
    distribution="ShardedByS3Key",  # Each instance gets a subset
    s3_data_type="S3Prefix"
)

SageMaker Pipe Mode¶

For large datasets, use Pipe mode to stream data directly from S3:

estimator = Estimator(
    input_mode="Pipe",  # Stream from S3 instead of downloading
    # ... other parameters
)

Feature Store Integration¶

Store processed features in the provisioned feature engineering folder:

solutions/customer-churn/data/processed/feature_engineering/
  features_v1.parquet
  features_v2.parquet

Provisioner Performance¶

Typical Operation Times¶

Action	Typical Duration	Notes
validate-config	< 1 second	Local only
create-policy	< 1 second	Local only
create-prov-template	< 1 second	Local only
validate-prov-template	< 1 second	Local only
prep-master	60-120 seconds	Creates bucket + CloudFormation stack
deploy-solution	5-15 seconds	Creates folder structure via S3 API
show-changes	10-30 seconds	Creates and deletes ChangeSet
check-drift	15-45 seconds	Drift detection API
test-deploy	60-120 seconds	Full stack with test suffix
tear-down	30-90 seconds	Stack deletion + bucket cleanup

Optimizing Deployment Speed¶

Run validate-config and validate-prov-template before deploying to catch errors early
Use test-deploy to verify before production — avoids failed production deployments
Deploy multiple solutions in sequence (each deploy-solution takes only 5-15 seconds)

Monitoring Performance¶

S3 Request Metrics¶

Enable S3 request metrics for your bucket:

aws s3api put-bucket-metrics-configuration \
  --bucket edge-prod-b001-us-west-1-s3 \
  --id EntireBucket \
  --metrics-configuration '{"Id":"EntireBucket"}'

CloudWatch Metrics¶

Monitor these S3 CloudWatch metrics:

AllRequests — total request count
GetRequests / PutRequests — read/write breakdown
4xxErrors — client errors (access denied, not found)
5xxErrors — server errors (throttling)
FirstByteLatency — time to first byte
TotalRequestLatency — total request time

S3 Access Logs¶

Enable access logging for detailed request analysis:

aws s3api put-bucket-logging \
  --bucket edge-prod-b001-us-west-1-s3 \
  --bucket-logging-status '{
    "LoggingEnabled": {
      "TargetBucket": "edge-prod-b001-us-west-1-s3",
      "TargetPrefix": "solutions/master-solution/artifacts/logs/s3-access/"
    }
  }'

For detailed S3 pricing, see COST_OPTIMIZATION.md. For folder structure details, see S3_FOLDERS.md.