User Guide

Complete reference for all commands and features.

Table of Contents

Prerequisites

Planning Your Deployment

Before installing tools or configuring credentials, understand the key decisions you’ll need to make:

Bucket Naming: You can use auto-generated bucket names based on your organization’s naming convention, or provide custom names. This affects how you identify and organize your S3 resources.

Deployment Pattern: You can deploy multiple ML solutions to a single shared bucket, or create separate buckets for each solution. This affects cost, management complexity, and access control.

Detailed guidance on these decisions is provided in Bucket Naming and Deployment Strategy sections.

Docker Installation

This tool requires Docker to run. Verify Docker is installed:

docker --version

If Docker is installed you should get the line similar to this: Docker version 29.2.1, build a5c7197

If you get a “command not found” error, Docker is likely not installed or not in your PATH.

If Docker is not installed follow the steps:

  • Linux (Ubuntu/Debian):

If Docker is not installed, follow these steps:

**Linux (NOT WSL2):**
```bash
# Update package index
sudo apt update

# Install prerequisites
sudo apt install -y apt-transport-https ca-certificates curl software-properties-common

# Add Docker's official GPG key
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update package index again
sudo apt update

# Install Docker
sudo apt install -y docker-ce docker-ce-cli containerd.io

# Start and enable Docker
sudo systemctl start docker
sudo systemctl enable docker

# Add user to docker group (optional, to run without sudo)
sudo usermod -aG docker $USER
newgrp docker

macOS: Download and install Docker Desktop for Mac

Windows: Download and install Docker Desktop for Windows

Verify Docker installation:

docker run hello-world

Python Installation (Optional - for AWS CLI)

Python 3.8+ is required if you plan to use AWS CLI for credential configuration.

Verify Python is installed:

python3 --version

Expected output:

Python 3.8.10 (or higher)

If Python is not installed:

Linux (Ubuntu/Debian):

sudo apt update
sudo apt install -y python3 python3-pip

macOS:

# Python 3 comes pre-installed on macOS 10.15+
# Or install via Homebrew:
brew install python3

Windows:

  • Download and install from python.org

  • During installation, check “Add Python to PATH”

Note: The S3 Provisioner tool itself runs in Docker and does not require Python on your host system. Python is only needed if you want to use AWS CLI for credential management (Method 1 in the next section).

AWS Credentials Configuration

This tool requires AWS credentials to provision S3 buckets and CloudFormation stacks.

Verify AWS credentials are configured:

aws sts get-caller-identity

Expected output:

{
    "UserId": "AIDAXXXXXXXXXXXXXXXXX",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/your-username"
}

If you get an error like “Unable to locate credentials”, configure AWS using one of the following methods:

Method 1: AWS CLI (Recommended)

# Install AWS CLI if not already installed
# Linux/macOS:
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

# Configure credentials
aws configure

# You'll be prompted for:
# AWS Access Key ID: [Your access key]
# AWS Secret Access Key: [Your secret key]
# Default region name: [e.g., us-west-1]
# Default output format: [json]

Method 2: Environment Variables

export AWS_ACCESS_KEY_ID="your-access-key"
export AWS_SECRET_ACCESS_KEY="your-secret-key"
export AWS_DEFAULT_REGION="us-west-1"

Method 3: IAM Role (For EC2/ECS)

  • If running on AWS EC2 or ECS, attach an IAM role with appropriate permissions. For detailed IAM permissions required, see IAM Permissions section.

  • No manual configuration needed

System Requirements

Disk Space:

  • Docker image: ~500 MB

  • Working directory: ~50 MB (for configs, templates, policies, reports)

  • Minimum recommended: 1 GB free space

Network Access:

  • Internet connection to pull Docker image from AWS Marketplace

  • AWS API endpoints accessibility

  • If behind corporate firewall/proxy, ensure Docker and AWS CLI can reach external endpoints

Operating System:

  • Linux: Ubuntu 20.04+, Debian 10+, RHEL 8+, or equivalent

  • macOS: 10.15+ (Catalina or later)

  • Windows: Windows 10/11 with WSL2 (for Docker Desktop integration)

Docker Version:

  • Docker Engine 20.10+ or Docker Desktop 4.0+

AWS CLI Version:

  • AWS CLI v2 recommended (v1 also supported)

IAM Permissions Check

Verify your AWS user/role has necessary permissions:

# Check current identity
aws sts get-caller-identity

# Test S3 permissions
aws s3 ls

# Test CloudFormation permissions
aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE

For detailed IAM permissions required, see IAM Permissions section.

Minimum required permissions:

  • S3: CreateBucket, PutBucketPolicy, PutBucketVersioning, PutLifecycleConfiguration

  • CloudFormation: CreateStack, UpdateStack, DeleteStack, DescribeStacks

  • IAM: PassRole (if using VPC endpoints)

AWS Account Limits

S3 Bucket Limit:

  • Default: 100 buckets per account

  • View current buckets: aws s3 ls

  • Request increase via AWS Support if needed

CloudFormation Stack Limit:

  • Default: 200 stacks per region

  • View current stacks: aws cloudformation list-stacks --stack-status-filter CREATE_COMPLETE UPDATE_COMPLETE

  • Request increase via AWS Support if needed

Bucket Name Availability:

  • S3 bucket names are globally unique across all AWS accounts

  • Tool will fail if bucket name already exists

  • Choose unique names or use auto-generated naming

Docker Image Availability

The S3 Provisioner tool is distributed as a Docker container image through AWS Marketplace.

Pull the image from AWS Marketplace:

# Pull the latest version from AWS Marketplace container registry
docker pull public.ecr.aws/<marketplace-id>/s3-provisioner:latest

# Tag it locally for convenience
docker tag public.ecr.aws/<marketplace-id>/s3-provisioner:latest s3-provisioner:latest

Note: Replace <marketplace-id> with the actual registry path provided in your AWS Marketplace subscription details.

Verify the image is available:

docker images s3-provisioner

Expected output:

REPOSITORY        TAG       IMAGE ID       CREATED        SIZE
s3-provisioner    latest    abc123def456   2 days ago     150MB

Security Scanning (Optional)

You can scan the S3 Provisioner Docker image for security vulnerabilities using Trivy or any container scanning tool your organization uses.

Install Trivy:

# macOS
brew install trivy

# Linux
sudo apt-get install trivy

Run scan:

trivy image s3-provisioner:latest

Filter by severity:

trivy image --severity HIGH,CRITICAL s3-provisioner:latest

Note: OS-level vulnerabilities in the base image (Debian) are common and typically have no fixed version available yet. Python package vulnerabilities are more actionable — report any findings to support.

Important: The Docker image is a pre-built, tested product. Do not attempt to rebuild or modify the image as this will void support and may introduce security vulnerabilities.

Step 1: Understand Bucket Naming

S3 bucket names must be globally unique across all AWS accounts and meet AWS naming requirements: 3–63 characters, lowercase letters, numbers, hyphens (-), or periods (.) only. Must start and end with a letter or number, cannot contain consecutive periods, and cannot be formatted as an IP address.

You have two options for naming your buckets:

Option A: Auto-Generated Names

The tool constructs bucket names from your configuration:

{company_prefix}-{env}-{tenant_id}-{region}

Example: edge-prod-b001-us-west-1-s3

Field

Description

Example

Constraints

company_prefix

Company identifier

“edge”

Max 10 characters

env

Environment

“prod”

dev/staging/prod/test

tenant_id

AWS account identifier

“a001”

1-4 alphanumeric characters

region

AWS region

“us-west-1”

Valid AWS region

Option B: Custom Names

Provide your own bucket name using bucket_name_override in the configuration:

s3:
  bucket_name_override: "my-company-ml-prod-bucket"

When to use each option:

  • Auto-generated: Consistent naming across multiple buckets, easier to manage at scale

  • Custom names: Specific naming requirements, existing naming conventions, single bucket deployments

Decision Point: Will you use auto-generated names or provide custom names?

Choose Deployment Strategy

You need to decide how to organize your ML solutions in S3 buckets. There are two main patterns:

Pattern A: Shared Bucket (Multiple Solutions)

Deploy multiple ML solutions within a single S3 bucket, each in its own folder structure.

edge-prod-b001-us-west-1-s3/
├── customer-churn/
│   ├── data/
│   ├── models/
│   ├── notebooks/
│   └── ...
├── fraud-detection/
│   ├── data/
│   ├── models/
│   ├── notebooks/
│   └── ...
└── recommendation-engine/
    ├── data/
    ├── models/
    ├── notebooks/
    └── ...

Advantages:

  • Lower cost (single bucket, consolidated storage)

  • Simpler management (one bucket to monitor)

  • Easier to share resources across solutions

  • Fewer buckets to track (stays under AWS limits)

Disadvantages:

  • All solutions share the same bucket policies

  • Less isolation between solutions

  • Harder to apply solution-specific access controls

  • Single point of failure affects all solutions

Best for: Organizations with many ML solutions, centralized ML teams, cost-sensitive deployments

Pattern B: Dedicated Buckets (One Solution Per Bucket)

Create a separate S3 bucket for each ML solution.

edge-prod-b001-us-west-1-s3-customer-churn/
├── data/
├── models/
├── notebooks/
└── ...

edge-prod-b001-us-west-1-s3-fraud-detection/
├── data/
├── models/
├── notebooks/
└── ...

edge-prod-b001-us-west-1-s3-recommendation/
├── data/
├── models/
├── notebooks/
└── ...

Advantages:

  • Strong isolation between solutions

  • Independent access controls per solution

  • Easier to manage solution-specific policies

  • Can delete entire solution cleanly

  • Better for compliance and security requirements

Disadvantages:

  • Higher cost (multiple buckets)

  • More buckets to manage and monitor

  • Can hit AWS bucket limits (default 100 per account)

  • More complex cross-solution resource sharing

Best for: Regulated industries, multi-tenant environments, solutions with different security requirements, independent solution lifecycles

Comparison Summary

Factor

Pattern A (Shared)

Pattern B (Dedicated)

Cost

Lower

Higher

Management

Simpler

More complex

Isolation

Low

High

Access Control

Bucket-level

Solution-level

Scalability

High (many solutions)

Limited (bucket quota)

Security

Shared policies

Independent policies

Decision Point: Will you use a shared bucket for multiple solutions (Pattern A) or dedicated buckets per solution (Pattern B)?

Quick Start

Pre-Deployment Checklist

Before deploying, ensure you have:

  • Docker 20.10+ installed and running (docker --version)

  • AWS credentials configured (aws sts get-caller-identity works)

  • AWS Marketplace subscription active for S3 Provisioner

  • IAM permissions verified (see IAM_PERMISSIONS.md)

  • Configuration file created and validated

  • Bucket names verified as globally unique (S3 bucket names are unique across all AWS accounts)

  • Reviewed generated CloudFormation template

  • Tested in dev/staging environment first

Setup

This section provides two complete workflows based on your deployment strategy choice from Step 2. Both patterns can coexist for the same environment, tenant, and AWS region. Choose the workflow that matches your needs.

Note: This tool provisions S3 folder structures for your ML solutions. You will develop and deploy your actual ML solutions separately.

Setup (Required for Both Patterns)

Create the working directory structure and copy documentation from the Docker image.

cd ~
mkdir -p mlops-infra-suite/s3/{configs,policies,reports,templates}
cd mlops-infra-suite
mkdir -p s3/docs

# Create a temporary container
container_id=$(docker create s3-provisioner:latest)

# Copy docs from container to host
docker cp $container_id:/app/docs/. s3/docs/

# Remove the temporary container
docker rm $container_id

ls s3/docs/

The following directory tree is created and the documentation files are copied.

└── mlops-infra-suite
    └── s3
        ├── configs
        ├── docs
        │   ├── _sources/
        │   ├── _static/
        │   ├── APPLICATION_ARCHITECTURE.html
        │   ├── BACKUP_RECOVERY.html
        │   ├── CONFIGURATION.html
        │   ├── GOVERNANCE_COMPLIANCE.html
        │   ├── IAM_PERMISSIONS.html
        │   ├── ML_LIFECYCLE_POLICIES.html
        │   ├── MONITORING_HEALTH_CHECKS.html
        │   ├── README.html
        │   ├── RELEASE_NOTES.html
        │   ├── ROADMAP.html
        │   ├── S3_FOLDERS.html
        │   ├── SECURITY.html
        │   ├── SECURITY_GUIDELINES.html
        │   ├── SUPPORT.html
        │   ├── TROUBLESHOOTING.html
        │   ├── UPDATE_PROCEDURES.html
        │   ├── USER_GUIDE.html
        │   ├── _sources
        │   ├── _static
        │   ├── api_reference.html
        │   ├── genindex.html
        │   ├── index.html
        │   ├── objects.inv
        │   ├── py-modindex.html
        │   ├── search.html
        │   └── searchindex.js
        ├── policies
        ├── reports
        └── templates

Open s3/docs/index.html in your browser to view the documentation.”

Pattern A: Shared Bucket Workflow

Provision a single S3 bucket with folder structures for multiple ML solutions.

1. Create Configuration File

Create s3/configs/edge-prod-b001-us-west-1-s3.yaml with auto-generated bucket name (leave bucket_name_override empty).

client:
  company_name: Edge Corp
  company_prefix: edge
  account_id: "123456789012"
  tenant_id: "a001"

environment:
  env: prod
  region: us-west-1

s3:
  bucket_name_override: ""
  versioning: true
  lifecycle_policy: ml-optimized
  vpc_id: ""
  route_table_ids: ""
  tags:
    Purpose: ML Solutions Portfolio
    ManagedBy: CloudFormation

2. Validate Configuration

Verify the configuration file is valid.

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action validate-config

3. Create IAM Policy

Generate the IAM policy JSON file for bucket permissions.

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-policy

4. Generate CloudFormation Template

Create the CloudFormation template for the master bucket structure.

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-prov-template \
  --solution master-solution

5. Provision Bucket

Deploy the S3 bucket and master folder structure using CloudFormation.

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  -v $(pwd)/s3/templates:/app/templates \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action prep-master \
  --solution master-solution \
  --force

6. Deploy First ML Solution Folder Structure

Create the folder structure for your first ML solution (e.g., customer-churn).

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action deploy-solution \
  --solution customer-churn

7. Deploy Additional Solution Folder Structures

Add folder structures for more ML solutions to the same bucket by repeating step 6 with different solution names.

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action deploy-solution \
  --solution fraud-detection

Pattern B: Dedicated Bucket Workflow

Provision a separate S3 bucket with folder structure for each ML solution.

1. Create Configuration for First Solution

Create s3/configs/edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml with custom bucket name including solution name.

client:
  company_name: Edge Corp
  company_prefix: edge
  account_id: "123456789012"
  tenant_id: "a001"

environment:
  env: prod
  region: us-west-1

s3:
  bucket_name_override: "edge-prod-b001-us-west-1-s3-customer-churn"
  versioning: true
  lifecycle_policy: ml-optimized
  vpc_id: ""
  route_table_ids: ""
  tags:
    Purpose: Customer Churn ML Solution
    ManagedBy: CloudFormation

2. Validate Configuration

Verify the configuration file is valid.

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action validate-config

3. Create IAM Policy

Generate the IAM policy JSON file for this bucket.

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action create-policy

4. Generate CloudFormation Template

Create the CloudFormation template for this solution’s bucket.

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action create-prov-template \
  --solution master-solution

5. Provision Bucket and Solution Folder Structure

Deploy the dedicated bucket with the ML solution folder structure.

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action create-bucket \
  --solution customer-churn \
  --force

6. Repeat for Additional Solutions

For each additional ML solution, create a new configuration file with a different bucket name and repeat steps 1-5.

Example for fraud-detection solution:

# s3/configs/edge-prod-b001-us-west-1-s3-fraud-detection-s3.yaml
s3:
  bucket_name_override: "edge-prod-b001-us-west-1-s3-fraud-detection"
  # ... rest of config

Then run steps 2-5 with --config edge-prod-b001-us-west-1-s3-fraud-detection-s3.yaml.

Cleanup

To remove a deployment, use the tear-down action.

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config <your-config-file>.yaml \
  --action tear-down \
  --force

Deployment Patterns

The S3 Provisioner supports two deployment patterns:

Pattern A: Shared Bucket (Multiple Solutions)

Use Case: Cost-effective deployment where multiple ML solutions share a single S3 bucket.

Configuration: Single config file without bucket_name_override

Example: edge-prod-b001-us-west-1-s3.yaml

client:
  company_name: Edge Corp
  company_prefix: edge
  account_id: "123456789012"
  tenant_id: "a001"

environment:
  env: prod
  region: us-west-1

s3:
  bucket_name_override: ""  # Empty = auto-generated name
  versioning: true
  lifecycle_policy: ml-optimized

Bucket Name: edge-prod-b001-us-west-1-s3

Workflow:

# 1. Validate, create policy, generate template
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action validate-config

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-policy

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-prov-template \
  --solution master-solution

# 2. Create bucket and prepare master structure
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action prep-master \
  --solution master-solution \
  --force

# 3. Deploy multiple solutions to same bucket
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action deploy-solution \
  --solution customer-churn

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action deploy-solution \
  --solution fraud-detection

Result: One bucket with multiple solution folders:

edge-prod-b001-us-west-1-s3/
  solutions/
    master-solution/
    customer-churn/
    fraud-detection/

Verify Deployment:

# Check bucket exists
aws s3 ls s3://edge-prod-b001-us-west-1-s3/

# List solution folders
aws s3 ls s3://edge-prod-b001-us-west-1-s3/solutions/

# View master-solution structure
aws s3 ls s3://edge-prod-b001-us-west-1-s3/solutions/master-solution/ --recursive | head -20

# Check CloudFormation stack (if deployed via CloudFormation)
aws cloudformation describe-stacks --stack-name edge-prod-b001-us-west-1-s3-stack --region us-west-1 --query 'Stacks[0].StackStatus'

# Count total folders created
aws s3 ls s3://edge-prod-b001-us-west-1-s3/solutions/master-solution/ --recursive | grep '.gitkeep' | wc -l

Artifacts created:

/mlops-infra-suite/s3
├── configs
│   └── edge-prod-b001-us-west-1-s3.yaml
├── policies
│   └── edge-prod-b001-us-west-1-s3-iam-policy.json
├── reports
│   ├── edge-prod-b001-us-west-1-s3-create-bucket-master-solution-20260228-020806-166.html
│   ├── edge-prod-b001-us-west-1-s3-create-policy-None-20260228-020700-792.log
│   ├── edge-prod-b001-us-west-1-s3-create-prov-template-master-solution-20260228-020702-061.log
│   ├── edge-prod-b001-us-west-1-s3-create-prov-template-master-solution-20260228-020702-154.html
│   ├── edge-prod-b001-us-west-1-s3-deploy-solution-customer-churn-20260228-020813-436.log
│   ├── edge-prod-b001-us-west-1-s3-deploy-solution-fraud-detection-20260228-020824-170.log
│   ├── edge-prod-b001-us-west-1-s3-prep-master-master-solution-20260228-020703-485.log
│   ├── edge-prod-b001-us-west-1-s3-prep-master-master-solution-20260228-020704-179.html
│   └── edge-prod-b001-us-west-1-s3-validate-config-None-20260228-020659-555.log
└── templates
    └── edge-prod-b001-us-west-1-s3_master-solution_s3_template.yaml

Pattern B: Dedicated Buckets (One Solution Per Bucket)

Use Case: Isolated deployment where each ML solution has its own dedicated S3 bucket.

Configuration: Separate config files with bucket_name_override for each solution

Example 1: edge-prod-b001-us-west-1-s3-customer-churn.yaml

client:
  company_name: Edge Corp
  company_prefix: edge
  account_id: "123456789012"
  tenant_id: "a001"

environment:
  env: prod
  region: us-west-1

s3:
  bucket_name_override: "edge-prod-b001-us-west-1-s3-customer-churn"
  versioning: true
  lifecycle_policy: ml-optimized

Example 2: edge-prod-b001-us-west-1-s3-fraud-detection.yaml

client:
  company_name: Edge Corp
  company_prefix: edge
  account_id: "123456789012"
  tenant_id: "a001"

environment:
  env: prod
  region: us-west-1

s3:
  bucket_name_override: "edge-prod-b001-us-west-1-s3-fraud-detection"
  versioning: false
  lifecycle_policy: development

Workflow for Customer Churn:

# 1. Validate and prepare
  -v ~/.aws:/home/s3user/.aws:ro \
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action validate-config
  -v ~/.aws:/home/s3user/.aws:ro \

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  -v ~/.aws:/home/s3user/.aws:ro \
  --action create-policy

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action create-prov-template \
  --solution customer-churn

# 2. Create bucket with solution structure
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action create-bucket \
  --solution customer-churn \
  --force

-v ~/.aws:/home/s3user/.aws:ro
Workflow for Fraud Detection (repeat with fraud-detection config):

# 1. Validate and prepare
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  -v ~/.aws:/home/s3user/.aws:ro \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-fraud-detection-s3.yaml \
  --action validate-config

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-fraud-detection-s3.yaml \
  --action create-policy

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-fraud-detection-s3.yaml \
  --action create-prov-template \
  --solution fraud-detection

# 2. Create bucket with solution structure
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-fraud-detection-s3.yaml \
  --action create-bucket \
  --solution fraud-detection \
  --force

Result: Two separate buckets, each with one solution:

edge-prod-b001-us-west-1-s3-customer-churn/
  solutions/
    customer-churn/

edge-prod-b001-us-west-1-s3-fraud-detection/
  solutions/
    fraud-detection/

Verify Deployment:

# Check customer-churn bucket
aws s3 ls s3://edge-prod-b001-us-west-1-s3-customer-churn/solutions/
aws s3 ls s3://edge-prod-b001-us-west-1-s3-customer-churn/solutions/customer-churn/ --recursive | head -20

# Check fraud-detection bucket
aws s3 ls s3://edge-prod-b001-us-west-1-s3-fraud-detection/solutions/
aws s3 ls s3://edge-prod-b001-us-west-1-s3-fraud-detection/solutions/fraud-detection/ --recursive | head -20

# Check CloudFormation stacks
aws cloudformation describe-stacks --stack-name edge-prod-b001-us-west-1-s3-customer-churn-s3-stack --region us-west-1 --query 'Stacks[0].StackStatus' 2>/dev/null || echo "Customer-churn stack not found"
aws cloudformation describe-stacks --stack-name edge-prod-b001-us-west-1-s3-fraud-detection-s3-stack --region us-west-1 --query 'Stacks[0].StackStatus' 2>/dev/null || echo "Fraud-detection stack not found"

# Count folders in each bucket
aws s3 ls s3://edge-prod-b001-us-west-1-s3-customer-churn/ --recursive | grep '.gitkeep' | wc -l
aws s3 ls s3://edge-prod-b001-us-west-1-s3-fraud-detection/ --recursive | grep '.gitkeep' | wc -l

Artifacts created:

/mlops-infra-suite/s3
├── configs
│   ├── edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml
│   └── edge-prod-b001-us-west-1-s3-fraud-detection-s3.yaml
├── policies
│   ├── edge-prod-b001-us-west-1-s3-customer-churn-iam-policy.json
│   └── edge-prod-b001-us-west-1-s3-fraud-detection-iam-policy.json
├── reports
│   ├── edge-prod-b001-us-west-1-s3-customer-churn-create-bucket-customer-churn-20260228-023020-467.log
│   ├── edge-prod-b001-us-west-1-s3-customer-churn-create-bucket-customer-churn-20260228-023021-194.html
│   ├── edge-prod-b001-us-west-1-s3-customer-churn-create-bucket-customer-churn-20260228-023123-188.html
│   ├── edge-prod-b001-us-west-1-s3-customer-churn-create-policy-None-20260228-022949-415.log
│   ├── edge-prod-b001-us-west-1-s3-customer-churn-create-prov-template-customer-churn-20260228-022958-989.log
│   ├── edge-prod-b001-us-west-1-s3-customer-churn-create-prov-template-customer-churn-20260228-022959-083.html
│   ├── edge-prod-b001-us-west-1-s3-customer-churn-validate-config-None-20260228-022941-327.log
│   ├── edge-prod-b001-us-west-1-s3-fraud-detection-create-bucket-fraud-detection-20260228-024112-852.log
│   ├── edge-prod-b001-us-west-1-s3-fraud-detection-create-bucket-fraud-detection-20260228-024113-544.html
│   ├── edge-prod-b001-us-west-1-s3-fraud-detection-create-bucket-fraud-detection-20260228-024215-593.html
│   ├── edge-prod-b001-us-west-1-s3-fraud-detection-create-policy-None-20260228-024042-486.log
│   ├── edge-prod-b001-us-west-1-s3-fraud-detection-create-prov-template-fraud-detection-20260228-024102-231.log
│   ├── edge-prod-b001-us-west-1-s3-fraud-detection-create-prov-template-fraud-detection-20260228-024102-324.html
│   └── edge-prod-b001-us-west-1-s3-fraud-detection-validate-config-None-20260228-024035-244.log
└── templates
    ├── edge-prod-b001-us-west-1-s3-customer-churn_customer-churn_s3_template.yaml
    └── edge-prod-b001-us-west-1-s3-fraud-detection_fraud-detection_s3_template.yaml

Pattern Comparison:

Aspect

Pattern A (Shared)

Pattern B (Dedicated)

Cost

Lower (one bucket)

Higher (multiple buckets)

Isolation

Shared resources

Complete isolation

Management

Simpler (one stack)

More complex (multiple stacks)

Permissions

Shared IAM policies

Granular per-bucket policies

Use Case

Dev/test, cost-sensitive

Production, compliance, multi-tenant

Configuration

Configuration File Structure

The S3 Provisioner uses a three-section YAML configuration:

client:
  company_name: string          # Full company name
  company_prefix: string        # Short identifier (lowercase)
  account_id: "string"          # 12-digit AWS account ID (quoted)
  tenant_id: "string"       # 4-character alphanumeric alias (e.g., "a001", "b002")

environment:
  env: string                   # Environment: prod, dev, test, staging
  region: string                # AWS region

s3:
  bucket_name_override: string  # Custom bucket name (optional)
  versioning: boolean           # Enable versioning (true/false)
  lifecycle_policy: string      # Lifecycle profile: ml-optimized, compliance, development, none
  vpc_id: string                # VPC ID for endpoint (optional)
  route_table_ids: string       # Comma-separated route table IDs (optional)
  tags:                         # Custom tags (optional)
    key: value

For detailed configuration reference, see CONFIGURATION.md.

Tenant ID Format

The tenant_id field uses a 4-character alphanumeric format to reduce resource name length:

Format: ^[a-z0-9]{4}$

Examples:

  • a001 - First account

  • b002 - Second account

  • c003 - Third account

  • x999 - Custom identifier

Benefits:

  • Saves 8 characters compared to 12-digit account IDs

  • Prevents Lambda function name length issues (64-char AWS limit)

  • Maintains uniqueness within organization

  • Human-readable and memorable

Bucket Name Override

The auto-generated bucket name format is: <company-prefix>-<env>-<account-alias>-<region>

Example: edge-prod-b001-us-west-1-s3

Use bucket_name_override when you need:

  • Multiple dedicated buckets per solution (Pattern B)

  • Custom naming for compliance requirements

  • Legacy naming compatibility

Example with Override:

s3:
  bucket_name_override: "edge-prod-b001-us-west-1-s3-customer-churn"

Artifact Naming Reference

All artifacts created by the S3 Provisioner follow consistent naming conventions. The bucket name serves as the universal stem for all generated artifacts, ensuring global uniqueness and consistency.

Artifact Type

Naming Pattern

Derived From

Example

AWS Resources

S3 Bucket

{company_prefix}-{env}-{tenant_id}-{region}

Config: Auto-generated

edge-prod-b001-us-west-1-s3

S3 Bucket (Override)

{bucket_name_override}

Config: bucket_name_override

edge-prod-b001-us-west-1-s3-customer-churn

CloudFormation Stack

{bucket_name}-s3-stack

Bucket name + -s3-stack

edge-prod-b001-us-west-1-s3-stack

Lambda Function

{stack_name}-creator

Stack name + -creator

edge-prod-b001-us-west-1-s3-stack-creator

Local Files

Config File (Pattern A)

{company_prefix}-{env}-{tenant_id}-{region}-s3.yaml

User-defined (note: -s3 suffix in filename only)

edge-prod-b001-us-west-1-s3.yaml

Config File (Pattern B)

{company_prefix}-{env}-{tenant_id}-{region}-{solution}-s3.yaml

User-defined (note: -s3 suffix in filename only)

edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml

CloudFormation Template

{bucket_name}_{solution}_s3_template.yaml

Bucket name

edge-prod-b001-us-west-1-s3_master-solution_s3_template.yaml

IAM Policy

{bucket_name}-iam-policy.json

Bucket name

edge-prod-b001-us-west-1-s3-iam-policy.json

Log File

{bucket_name}_{action}_{solution}_{timestamp}.log

Bucket name

edge-prod-b001-us-west-1-s3-create-bucket-master-solution-20260226-211116-934.log

HTML Report (Template)

{bucket_name}-create-prov-template-{solution}_{timestamp}.html

Bucket name

edge-prod-b001-us-west-1-s3-create-prov-template-master-solution-20260228-020702-154.html

HTML Report (Deployment)

{bucket_name}_{action}_{solution}_{timestamp}.html

Bucket name

edge-prod-b001-us-west-1-s3-create-bucket-master-solution-20260228-020806-166.html

S3 Objects

Solution Folder

solutions/{solution}/

Solution parameter

solutions/customer-churn/

Folder Marker

solutions/{solution}/{path}/.gitkeep

Solution + folder path

solutions/customer-churn/data/raw/.gitkeep

Uploaded Template

solutions/{solution}/templates/{template_filename}

Solution + template name

solutions/master-solution/templates/edge-prod-b001-us-west-1-s3_master-solution_s3_template.yaml

Key Naming Components:

  • {bucket_name}: Either auto-generated from config values or from bucket_name_override. This is the universal stem for all local artifacts, ensuring global uniqueness (guaranteed by AWS S3 bucket naming constraints)

  • {stack_name}: Always {bucket_name}-s3-stack (adds -s3-stack suffix to bucket name)

  • {solution}: Solution name from --solution parameter (e.g., master-solution, customer-churn)

  • {timestamp}: UTC timestamp in format YYYYMMDD_HHMMSS_mmm (e.g., 20260226_211116_934)

  • {action}: Command action (e.g., create-bucket, deploy-solution, validate-config)

Naming Best Practices:

  1. Config Files: Use descriptive names that include environment and purpose

    • Pattern A (shared): {company_prefix}-{env}-{tenant_id}-{region}-s3.yaml

    • Pattern B (dedicated): {company_prefix}-{env}-{tenant_id}-{region}-{solution}-s3.yaml

  2. Bucket Names: Keep company_prefix short (max 10 chars) to avoid AWS 63-character bucket name limit

  3. Solution Names: Use lowercase with hyphens (e.g., customer-churn, not CustomerChurn)

  4. Consistency: All artifacts for a deployment share the same {bucket_name} prefix, making them easy to identify and manage

Note on Early Initialization Errors: If the configuration file cannot be found, contains YAML syntax errors, or fails validation, these errors will only appear in console/stderr output (not in log files) because the bucket name cannot be determined until the configuration is successfully loaded and validated.

Commands Reference

Validation Commands

validate-config

Validates configuration file against schema.

Purpose: Verify YAML syntax and schema compliance before deployment

Requirements: No AWS credentials needed

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action validate-config

Output:

  • Logs: reports/{bucket_name}-validate-config-None_{timestamp}.log

Policy Commands

create-policy

Generates IAM policy JSON file for S3 bucket and CloudFormation operations.

Purpose: Create IAM policy document with required permissions

Requirements: No AWS credentials needed (generates file only)

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-policy

Use Case: Attach generated policy to IAM user/role before running AWS operations

CloudFormation Template Commands

create-prov-template

Generates CloudFormation template for S3 bucket provisioning.

Purpose: Create infrastructure-as-code template for bucket deployment

Requirements: No AWS credentials needed (generates file only)

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-prov-template \
  --solution master-solution

Output: templates/<config-name>_<solution>_s3_template.yaml

Template Includes:

  • S3 bucket with configured properties

  • Versioning configuration

  • Lifecycle rules (if lifecycle_policy specified)

  • Tags (system + custom)

  • VPC endpoint (if vpc_id specified)

  • Lambda function for folder creation

validate-prov-template

Validates the generated CloudFormation template locally without making any AWS calls.

Purpose: Catch template errors early — YAML syntax, required keys, and internal reference integrity

Requirements: No AWS credentials needed

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action validate-prov-template \
  --solution master-solution

Output: Report in console (template size, resource count, output count)

What It Checks:

  1. YAML syntax is valid

  2. Required top-level keys present (AWSTemplateFormatVersion, Resources)

  3. All !Ref and !GetAtt targets resolve to parameters or resources within the template

Note: If the template file does not exist, it will be generated automatically.

Change Preview and Drift Detection Commands

show-changes

Previews what would change in the deployed stack without applying any modifications.

Purpose: Review pending changes before updating infrastructure

Requirements: AWS credentials, deployed stack

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action show-changes \
  --solution master-solution

Output: Report in console (list of pending resource changes)

How It Works:

  1. Creates a temporary CloudFormation ChangeSet against the deployed stack

  2. Displays pending resource changes (Add, Modify, Remove)

  3. Deletes the ChangeSet after display (no changes applied)

Dry Run: Use --dry-run to simulate without creating a ChangeSet:

s3-prov -con edge-prod-b001-us-west-1-s3.yaml -act show-changes --dry-run

check-drift

Detects infrastructure drift between the deployed stack and its template.

Purpose: Identify resources that have been modified outside of CloudFormation

Requirements: AWS credentials, deployed stack

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action check-drift

Output: Report in console (per-resource drift status with property-level diffs)

What It Detects:

  • Bucket policies modified manually

  • Lifecycle rules changed outside CloudFormation

  • Tags modified or removed

  • VPC endpoint configurations altered

Dry Run: Use --dry-run to simulate without initiating drift detection:

s3-prov -con edge-prod-b001-us-west-1-s3.yaml -act check-drift --dry-run

test-deploy

Deploys S3 infrastructure with a random test suffix for safe testing.

Purpose: Validate that the template deploys successfully without affecting production resources

Requirements: AWS credentials, --solution

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action test-deploy \
  --solution master-solution

How It Works:

  1. Appends a random 6-character suffix to the bucket name (e.g., edge-prod-b001-us-west-1-s3-test-a1b2c3)

  2. All resource names inherit the test suffix — no collision with production

  3. Generates template in-memory (not persisted to disk)

  4. Deploys via CloudFormation with the test-suffixed stack name

  5. Restores original bucket name after deployment

Cleanup: Delete the test stack when done:

aws cloudformation delete-stack --stack-name edge-prod-b001-us-west-1-s3-test-a1b2c3-stack --region us-west-1

Note: This action does not require --force. Test deployments are designed to be disposable. Custom bucket_name_override values must not exceed 51 characters to stay within the S3 63-character limit after the test suffix is appended.

Bucket Provisioning Commands

create-bucket

Output:

  • Files: templates/{bucket_name}_{solution}_s3_template.yaml

  • Reports: reports/{bucket_name}-create-bucket-{solution}_{timestamp}.html

  • Logs: reports/{bucket_name}-create-bucket-{solution}_{timestamp}.log

  • AWS Resources: S3 bucket {bucket_name}, CloudFormation stack {bucket_name}-s3-stack

  • SSM Parameters: /s3/<bucket-name>/<OutputKey>

Creates S3 bucket using CloudFormation stack.

Purpose: Deploy S3 bucket infrastructure via CloudFormation

Requirements: AWS credentials with S3 and CloudFormation permissions

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-bucket \
  --solution master-solution \
  --force

What It Does:

  1. Generates and saves CloudFormation template to templates directory

  2. Creates CloudFormation stack

  3. Provisions S3 bucket with all configurations

  4. Applies tags, versioning, lifecycle rules

  5. Creates VPC endpoint (if configured)

  6. Stores stack outputs in SSM Parameter Store (/s3/<bucket-name>/<OutputKey>)

    • e.g., /s3/globalbank-prod-c001-us-west-2-s3/BucketName

    • Available keys: BucketName, BucketArn, VPCEndpointId, TotalFolders, CompanyPrefix, Region

Stack Name: {bucket-name}-stack

Example: edge-prod-b001-us-west-1-s3-stack

prep-master

Prepares S3 bucket with master-solution folder structure.

Purpose: Create master folder template for ML solutions

Requirements: AWS credentials, bucket must exist

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  -v $(pwd)/s3/templates:/app/templates \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action prep-master \
  --solution master-solution \
  --force

What It Does:

  1. Generates CloudFormation template

  2. Creates S3 bucket from the template

  3. Stores stack outputs in SSM Parameter Store (/s3/<bucket-name>/<OutputKey>)

    • e.g., /s3/globalbank-prod-c001-us-west-2-s3/BucketName

    • Available keys: BucketName, BucketArn, VPCEndpointId, TotalFolders, CompanyPrefix, Region

  4. Adds .gitkeep files to all folders in the master-solution structure

What It Creates:

solutions/master-solution/
  data/
    raw/
    curated/
    processed/
    inference/
  models/
  notebooks/
  artifacts/
  code/
  config/

Output:

  • reports/{bucket_name}-prep-master-{solution}_{timestamp}.html

  • reports/{bucket_name}-prep-master-{solution}_{timestamp}.log

  • S3 folder structure under s3://{bucket_name}/solutions/{solution}/

  • SSM Parameters: /s3/<bucket-name>/<OutputKey>

delete-bucket

Output:

  • Logs: reports/{bucket_name}_delete-bucket_None_{timestamp}.log

Deletes S3 bucket directly (bypasses CloudFormation).

Purpose: Remove bucket and all objects (emergency cleanup)

Requirements: AWS credentials, --force flag

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action delete-bucket \
  --force

Warning: This empties and deletes the bucket directly. Use tear-down for proper CloudFormation cleanup.

delete-cfn-stack

Output:

  • Logs: reports/{bucket_name}_delete-cfn-stack_None_{timestamp}.log

Deletes CloudFormation stack (keeps bucket if deletion protection enabled).

Purpose: Remove CloudFormation stack

Requirements: AWS credentials, --force flag

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action delete-cfn-stack \
  --force

tear-down

Output:

  • Logs: reports/{bucket_name}_tear-down_None_{timestamp}.log

Deletes CloudFormation stack with all underlying S3 resources.

Purpose: Complete infrastructure cleanup (stack + bucket + objects)

Requirements: AWS credentials, --force flag

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action tear-down \
  --force

Recommended: Use this for clean infrastructure removal instead of delete-bucket.

Cost Estimation Commands

cost-traffic

Generate a usage assumptions file for cost estimation.

Purpose: Create an editable YAML file with default monthly usage values for storage, requests, data transfer, and VPC Endpoint traffic

Requirements: AWS credentials, provisioning template must exist

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action cost-traffic \
  --solution master-solution

Output:

  • File configs/<bucket-name>-usage.yaml

  • File reports/<bucket-name>-cost-traffic-{solution}_{timestamp}.log

Generated Usage File:

# Auto-generated S3 usage assumptions for cost estimation
# Edit values to match your expected monthly usage

usage:
  storage:
    storage_class: Standard
    data_gb: 100
  requests:
    put_requests_per_month: 10000
    get_requests_per_month: 50000
  transfer:
    data_out_gb_per_month: 10
  vpc_endpoint:
    S3VPCEndpoint:
      type: AWS::EC2::VPCEndpoint
      data_gb_per_month: 50

Note: Edit the values to match your expected usage before running cost-estimate. Requires the provisioning template — run create-prov-template first.

cost-estimate

Calculate estimated monthly infrastructure costs with a detailed breakdown.

Purpose: Produce a cost breakdown showing storage, request, data transfer, and VPC Endpoint costs with monthly and annual totals

Requirements: AWS credentials, provisioning template and usage file must exist

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action cost-estimate \
  --solution master-solution

Output:

  • Cost breakdown in console

  • File reports/<bucket-name>-cost-estimate-{timestamp}.html

  • File reports/<bucket-name>-cost-estimate-{solution}_{timestamp}.log

HTML Report Includes:

  • Cost summary with monthly and annual totals

  • Cost breakdown table (storage, requests, transfer, VPC Endpoint)

  • Usage assumptions and per-unit rates

  • Pricing source and region information

Note: Requires both the provisioning template and usage assumptions file. Run create-prov-template and cost-traffic first. Edit the usage file and re-run to model different scenarios.

cost-refresh-prices

Refresh S3 resource pricing from the AWS Pricing API.

Purpose: Update the built-in pricing data with the latest on-demand rates from AWS for all regions

Requirements: AWS credentials with Pricing API access

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action cost-refresh-prices \
  --solution master-solution

Output:

  • Report in console (region count, pricing for current region)

  • File reports/<bucket-name>-cost-refresh-prices-{solution}_{timestamp}.log

What It Fetches:

  • S3 Standard storage per-GB rates

  • PUT/COPY/POST/LIST request rates

  • GET and other request rates

  • Data transfer out per-GB rates

  • VPC Endpoint hourly and per-GB rates

Note: This action is optional. The tool ships with pre-loaded pricing data. Run this action periodically to ensure pricing accuracy.

Solution Deployment Commands

deploy-solution

Output:

  • Logs: reports/{bucket_name}-deploy-solution-{solution}_{timestamp}.log

  • S3 Objects: Folder structure under s3://{bucket_name}/solutions/{solution}/

Deploys ML solution folder structure to bucket.

Purpose: Create solution-specific folder structure

Requirements: AWS credentials, bucket must exist

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action deploy-solution \
  --solution customer-churn

Available Solutions:

  • customer-churn

  • demand-forecasting

  • fraud-detection

  • master-solution

deploy-folders

Output:

  • Logs: reports/{bucket_name}_deploy-folders_{solution}_{timestamp}.log

  • S3 Objects: Cloned folder structure under s3://{bucket_name}/solutions/{solution}/

Clones S3 folder structures from master-solution to specific solution.

Purpose: Copy folder structure from master template

Requirements: AWS credentials, master-solution must exist

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action deploy-folders \
  --solution customer-churn

upload-template

Output:

  • Logs: reports/{bucket_name}_upload-template_{solution}_{timestamp}.log

  • S3 Objects: s3://{bucket_name}/solutions/{solution}/templates/{bucket_name}_{solution}_s3_template.yaml

Uploads YAML template to S3 location.

Purpose: Store CloudFormation template in S3 for reference

Requirements: AWS credentials, bucket must exist

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action upload-template \
  --solution master-solution

Upload Location: s3://<bucket>/solutions/<solution>/templates/

GitKeep Management Commands

gitkeep-full

Output:

  • Logs: reports/{bucket_name}_gitkeep-full_{solution}_{timestamp}.log

  • S3 Objects: .gitkeep files in all folders under s3://{bucket_name}/solutions/{solution}/

Creates .gitkeep files for all folders under solutions/.

Purpose: Preserve empty folder structure in Git

Requirements: AWS credentials, solution must exist

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action gitkeep-full \
  --solution customer-churn

gitkeep-none

Output:

  • Logs: reports/{bucket_name}_gitkeep-none_{solution}_{timestamp}.log

Removes .gitkeep files from all folders in solutions/.

Purpose: Clean up all .gitkeep files

Requirements: AWS credentials, solution must exist

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action gitkeep-none \
  --solution customer-churn

gitkeep-partial

Output:

  • Logs: reports/{bucket_name}_gitkeep-partial_{solution}_{timestamp}.log

Removes .gitkeep files from folders two levels below root.

Purpose: Selective .gitkeep cleanup

Requirements: AWS credentials, solution must exist

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action gitkeep-partial \
  --solution customer-churn

purge-bucket

Output:

  • Logs: reports/{bucket_name}_purge-bucket_None_{timestamp}.log

Removes all .gitkeep files from the entire bucket.

Purpose: Complete .gitkeep cleanup across all solutions

Requirements: AWS credentials, bucket must exist

Command:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action purge-bucket \
  --force

Common Workflows

Pattern A: Shared Bucket Deployment

Deploy a single bucket with multiple ML solutions.

# Step 1: Validate configuration
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action validate-config

# Step 2: Generate IAM policy
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-policy

# Step 3: Generate CloudFormation template
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-prov-template \
  --solution master-solution

# Step 4: Create bucket and master solution structure
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action prep-master \
  --solution master-solution \
  --force

# Step 5: Deploy first ML solution
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action deploy-solution \
  --solution customer-churn

# Step 6: Deploy additional ML solutions (repeat as needed)
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action deploy-solution \
  --solution fraud-detection

Pattern B: Dedicated Bucket Deployment

Deploy a separate bucket for each ML solution.

# Step 1: Validate configuration
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action validate-config

# Step 2: Generate IAM policy
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action create-policy

# Step 3: Generate CloudFormation template
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action create-prov-template \
  --solution customer-churn

# Step 4: Create dedicated bucket with solution structure
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action create-bucket \
  --solution customer-churn \
  --force

# Step 5: Create .gitkeep files for all nodes
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3-customer-churn-s3.yaml \
  --action gitkeep-full \
  --solution customer-churn

# Repeat steps 1-4 for each additional solution with its own config file

Quick Test Workflow

# Validate only
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action validate-config

# Generate template only (no AWS deployment)
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-prov-template \
  --solution master-solution

# Review generated template
cat templates/edge-prod-b001-us-west-1-s3_master-solution_s3_template.yaml

Cleanup Workflow

# Complete teardown (recommended)
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action tear-down \
  --force

Cost Estimation Workflow

# Step 1: Generate CloudFormation template (if not already done)
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-prov-template \
  --solution master-solution

# Step 2: Generate usage assumptions with defaults
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action cost-traffic \
  --solution master-solution

# Step 3: Edit usage assumptions to match expected usage
# Edit configs/edge-prod-b001-us-west-1-s3-usage.yaml

# Step 4: Calculate cost estimate
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action cost-estimate \
  --solution master-solution

# Step 5: View HTML report
# Open reports/edge-prod-b001-us-west-1-s3-cost-estimate-*.html

# Optional: Refresh pricing data from AWS Pricing API
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action cost-refresh-prices \
  --solution master-solution

Volume Mounts

Mount

Purpose

Required For

configs/

Input configuration files and usage assumptions

All actions

policies/

Generated IAM policies

create-policy

reports/

Execution logs and HTML reports

All actions (recommended)

templates/

CloudFormation templates

create-prov-template, prep-master, upload-template, cost-traffic, cost-estimate

~/.aws/

AWS credentials

All actions (required for license validation)

Example with all mounts:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v $(pwd)/s3/reports:/app/reports \
  -v $(pwd)/s3/templates:/app/templates \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action prep-master \
  --solution master-solution \
  --force

AWS Credentials

Option 2: Environment Variables

docker run --rm \
  -e AWS_ACCESS_KEY_ID=AKIA... \
  -e AWS_SECRET_ACCESS_KEY=wJal... \
  -e AWS_DEFAULT_REGION=us-west-1 \
  -v $(pwd)/s3/configs:/app/configs \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-bucket \
  --solution master-solution \
  --force

Option 3: IAM Role (EC2/ECS)

When running on EC2 or ECS with IAM role attached, no credentials needed:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-bucket \
  --solution master-solution \
  --force

Best Practices

Always Validate First

Run validate-config before any AWS operations:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action validate-config

Review Generated Templates

Always review CloudFormation templates before deployment:

# Generate template
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action create-prov-template \
  --solution master-solution

# Review template
cat templates/your-config_master-solution_s3_template.yaml

Use IAM Roles in Production

Prefer IAM roles over access keys when running on EC2/ECS.

Enable Versioning in Production

s3:
  versioning: true  # Always enable for production

Use Lifecycle Policies

s3:
  lifecycle_policy: ml-optimized  # or compliance, development

Test in Dev First

Test configurations in dev environment before production:

environment:
  env: dev  # Test here first
  region: us-west-2

Use CloudFormation for Infrastructure

Always use create-bucket (CloudFormation) instead of manual bucket creation.

Proper Cleanup

Use tear-down instead of delete-bucket for proper infrastructure cleanup:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action tear-down \
  --force

Monitor Logs

Always mount reports/ directory to capture execution logs:

-v $(pwd)/s3/reports:/app/reports

Version Control Configurations

Store configuration files in Git for change tracking and rollback capability.

Command Summary

Action

AWS Creds

–solution

–force

Purpose

validate-config

❌

❌

❌

Validate YAML

create-policy

❌

❌

❌

Generate IAM policy

create-prov-template

❌

✅

❌

Generate CFN template

validate-prov-template

❌

❌

❌

Validate CFN template locally

show-changes

✅

❌

❌

Preview pending changes

check-drift

✅

❌

❌

Detect infrastructure drift

test-deploy

✅

✅

❌

Safe test deployment

create-bucket

✅

✅

✅

Create S3 bucket

prep-master

✅

✅

✅

Create master folders

deploy-solution

✅

✅

❌

Deploy solution folders

deploy-folders

✅

✅

❌

Clone folder structure

upload-template

✅

✅

❌

Upload template to S3

gitkeep-full

✅

✅

❌

Add .gitkeep files

gitkeep-none

✅

✅

❌

Remove .gitkeep files

gitkeep-partial

✅

✅

❌

Partial .gitkeep removal

purge-bucket

✅

❌

✅

Remove all .gitkeep

delete-bucket

✅

❌

✅

Delete bucket directly

delete-cfn-stack

✅

❌

✅

Delete CFN stack

tear-down

✅

❌

✅

Complete cleanup

cost-traffic

✅

✅

❌

Generate usage assumptions

cost-estimate

✅

✅

❌

Calculate cost estimate

cost-refresh-prices

✅

✅

❌

Refresh resource pricing

Troubleshooting

See TROUBLESHOOTING.md for common issues and solutions.

What’s Next

After provisioning your S3 infrastructure, explore these resources:

Enterprise Implementation:

  • GOVERNANCE_COMPLIANCE.md - Reference architecture for implementing governance, compliance, and audit capabilities with ready-to-use JSON schemas and AWS service recommendations

Structure Reference:

  • S3_FOLDERS.md - Complete technical reference for the provisioned folder structure with detailed hierarchy at all levels

Additional Documentation:

  • ML_LIFECYCLE_POLICIES.md - Detailed lifecycle policy configurations for cost optimization

  • CONFIGURATION.md - Complete YAML configuration reference

  • IAM_PERMISSIONS.md - Required AWS permissions and security best practices

Support

See SUPPORT.md for help and contact information.

Configuration Reference

See CONFIGURATION.md for complete configuration documentation.

IAM Permissions

See IAM_PERMISSIONS.md for required AWS permissions.

Copyright © 2025 Axon Tech Labs All rights reserved.

See LICENSE.txt for terms and conditions.

Frequently Asked Questions

Q: Can I modify the generated CloudFormation template? A: Yes, but changes will be overwritten on next create-prov-template. Use configuration parameters (bucket_name_override, lifecycle_policy, tags, etc.) instead.

Q: How do I upgrade to a new version? A: Pull the latest Docker image. Existing buckets and data are not affected unless you redeploy.

Q: Can I use this with existing S3 buckets? A: No, this tool creates new buckets via CloudFormation. For existing buckets, continue managing them with AWS Console or CLI.

Q: What happens if deployment fails? A: CloudFormation automatically rolls back all resources. Check stack events for details. See TROUBLESHOOTING.md.

Q: Can I deploy to multiple regions? A: Yes, create separate configuration files for each region and run the tool for each config.

Q: How much does the infrastructure cost? A: Use the built-in cost estimation feature to calculate costs for your specific configuration. Run cost-traffic to generate usage assumptions, edit the values to match your expected storage, requests, and data transfer, then run cost-estimate for a detailed breakdown with monthly and annual totals. S3 buckets are free to create — costs come from storage, requests, and data transfer. VPC Endpoints incur an hourly charge. See COST_OPTIMIZATION.md for optimization strategies.

Q: Can I have multiple solutions in one bucket? A: Yes — that’s Pattern A (Shared Bucket). Use deploy-solution to add solution folder structures to an existing bucket.

Q: What’s the difference between prep-master and create-bucket? A: prep-master creates the bucket via CloudFormation and deploys the master folder structure. create-bucket creates a dedicated bucket for a single solution (Pattern B).

Q: How do I delete everything? A: Use tear-down --force to delete the CloudFormation stack, bucket, and all objects.