Application Architecture¶
Table of Contents¶
Executive Summary¶
S3 Provisioner is an enterprise-grade AWS infrastructure automation tool that provisions secure, production-ready S3 bucket environments for machine learning workloads through a configuration-driven approach. Built for cloud infrastructure and MLOps teams, the tool deploys a complete ML folder hierarchy, lifecycle policies, VPC endpoint integration, and SSM parameter outputs in minutes — enforcing AWS Well-Architected Framework best practices for storage security, data governance, and compliance.
Key Capabilities:
Automated S3 bucket provisioning with 130+ ML-optimized folder structure via CloudFormation
Multi-solution support — shared bucket (multiple solutions) or dedicated buckets (one per solution)
VPC endpoint integration for private S3 access with no internet exposure
SSM Parameter Store outputs managed as CloudFormation resources (auto-cleaned on stack deletion)
Multi-region and multi-account support with standardized naming conventions
AWS Marketplace integration with license validation
Comprehensive audit trails and HTML reporting
Built-in cost estimation with region-specific AWS Pricing API integration
Docker containerization for consistent deployment
Cython-compiled core modules for code protection
1. Introduction & Context¶
1.1 Purpose & Scope¶
Purpose: Automate the provisioning of AWS S3 bucket infrastructure for machine learning workloads, providing production-ready storage environments with standardized folder hierarchies, lifecycle policies, and governance controls across multiple AWS accounts, regions, and environments.
Scope:
S3 bucket creation with configurable versioning, lifecycle policies, and tagging
130+ folder ML folder hierarchy (data, models, notebooks, artifacts, code, config)
CloudFormation-based infrastructure as code
Lambda-based folder creation for leaf nodes
VPC endpoint provisioning for private S3 access (optional)
SSM Parameter Store outputs (
/s3/{bucket_name}/BucketName,BucketArn,VPCEndpointId)IAM policy generation for least-privilege access
HTML report generation for pre-deployment and post-deployment documentation
Built-in cost estimation with AWS Pricing API integration
AWS Marketplace license validation
Out of Scope:
VPC provisioning (handled by VPC Provisioner)
Security group management (handled by SG Provisioner)
ML pipeline provisioning (handled by ML Provisioner)
Application deployment within S3 buckets
1.2 Business Goals¶
Operational Efficiency:
Reduce S3 ML infrastructure setup from days (manual) to minutes (automated)
Eliminate human error in folder hierarchy and lifecycle policy configuration
Enable self-service infrastructure provisioning for data science and MLOps teams
Security & Compliance:
Enforce S3 security best practices by design (public access blocking, versioning, encryption)
Ensure consistent governance posture across all environments and tenants
Provide audit trails for compliance requirements (SOC 2, ISO 27001, HIPAA, GDPR)
Implement least-privilege IAM policies automatically
Private S3 access via VPC endpoints — no data traverses the public internet
Cost Optimization:
4 pre-configured lifecycle policy profiles (ml-optimized, compliance, development, none)
Built-in cost estimation before deployment — no surprises on the bill
Resource tagging for accurate cost allocation and chargeback
Standardization:
Consistent ML folder hierarchy across all clients, environments, and regions
Reusable configuration templates for common deployment patterns
Version-controlled infrastructure definitions
Multi-Region Strategy:
Rapid deployment of consistent storage infrastructure across AWS regions
Geographic distribution for data locality and latency optimization
Compliance with data residency requirements
1.3 Stakeholders¶
Primary Users:
MLOps Engineers: Deploy and manage S3 ML infrastructure
Data Engineers: Provision storage for ML data pipelines
DevOps Engineers: Integrate S3 provisioning into CI/CD pipelines
Platform Engineers: Maintain standardized ML storage blueprints
Secondary Users:
Security Teams: Audit bucket configurations and enforce policies
Compliance Officers: Verify adherence to regulatory requirements
Data Scientists: Consume S3 bucket infrastructure for ML workloads
Finance Teams: Track infrastructure costs via resource tagging and cost reports
Executive Stakeholders:
CTO/VP Engineering: Oversee infrastructure automation strategy
CISO: Ensure security and compliance posture
CFO: Monitor infrastructure costs and ROI
1.4 Success Criteria¶
Functional:
✅ Provision S3 bucket with complete ML folder structure in < 5 minutes
✅ Support multiple AWS regions and accounts
✅ Generate valid CloudFormation templates (74 resources, 6 outputs)
✅ Validate configurations before deployment
✅ Provide comprehensive pre-deployment and post-deployment HTML reports
Non-Functional:
✅ 99.9% success rate for S3 provisioning operations
✅ Zero security vulnerabilities in application code
✅ Complete audit trail for all operations
✅ Support for 100+ concurrent bucket deployments
✅ Documentation coverage > 90%
2. Architectural Representation¶
2.1 System Context Diagram¶
┌───────────────────────────────────────────────────────────────────┐
│ External Context │
│ │
│ ┌──────────────┐ │
│ │ MLOps / │ │
│ │ DevOps │ │
│ │ Engineer │ │
│ └──────┬───────┘ │
│ │ YAML Config │
│ │ CLI Commands │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ S3 Provisioner Application │ │
│ │ ┌────────────────────────────────────────────────────┐ │ │
│ │ │ • Configuration Validation │ │ │
│ │ │ • CloudFormation Template Generation │ │ │
│ │ │ • AWS API Integration (boto3) │ │ │
│ │ │ • License Validation (AWS Marketplace) │ │ │
│ │ │ • HTML Report Generation │ │ │
│ │ │ • Cost Estimation (AWS Pricing API) │ │ │
│ │ │ • Audit Logging │ │ │
│ │ └────────────────────────────────────────────────────┘ │ │
│ └─────────────────┬───────────────────────────────────────┘ │
│ │ AWS API Calls (HTTPS/TLS) │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ Amazon Web Services (AWS) │ │
│ │ ┌────────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │
│ │ │ CloudFormation │ │ Amazon S3 │ │ AWS Marketplace │ │ │
│ │ │ Service │ │ Service │ │ License Manager │ │ │
│ │ └────────────────┘ └──────────────┘ └──────────────────┘ │ │
│ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────────┐ │ │
│ │ │ IAM │ │ AWS Lambda │ │ SSM Parameter │ │ │
│ │ │ Service │ │ Service │ │ Store │ │ │
│ │ └──────────────┘ └──────────────┘ └──────────────────┘ │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ Amazon EC2 │ │ AWS Pricing │ │ │
│ │ │ (VPC Endpt) │ │ API │ │ │
│ │ └──────────────┘ └──────────────┘ │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │
└───────────────────────────────────────────────────────────────────┘
Key Relationships:
User provides YAML configuration defining bucket requirements
Application validates configuration against JSON schema
Application generates CloudFormation template dynamically
Application calls AWS APIs to create/manage S3 resources
AWS CloudFormation orchestrates resource provisioning
Lambda function creates S3 folder structure at deployment time
SSM Parameter Store receives bucket outputs as CloudFormation-managed resources
Application validates AWS Marketplace license (if applicable)
Application generates HTML reports and audit logs
2.2 Container View (Deployment Units)¶
┌────────────────────────────────────────────────────────────────┐
│ Docker Container │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Python 3.13 Runtime Environment │ │
│ │ ┌────────────────────────────────────────────────────┐ │ │
│ │ │ S3 Provisioner Application │ │ │
│ │ │ • CLI Interface (cli.py) │ │ │
│ │ │ • Core Logic (.so compiled modules) │ │ │
│ │ │ • Configuration Loader │ │ │
│ │ │ • License Validator │ │ │
│ │ │ • HTML Generator │ │ │
│ │ │ • Cost Adapter │ │ │
│ │ └────────────────────────────────────────────────────┘ │ │
│ │ ┌────────────────────────────────────────────────────┐ │ │
│ │ │ Dependencies (installed via uv) │ │ │
│ │ │ • boto3 (AWS SDK) │ │ │
│ │ │ • PyYAML (config parsing) │ │ │
│ │ │ • jsonschema (validation) │ │ │
│ │ │ • urllib3 >=2.7.0 (CVE fix) │ │ │
│ │ └────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Mounted Volumes │ │
│ │ • /app/configs (read-only) - YAML configurations │ │
│ │ • /app/policies (read-write) - Generated IAM policies │ │
│ │ • /app/templates (read-write) - CFN templates │ │
│ │ • /app/reports (read-write) - Logs and HTML reports │ │
│ │ • ~/.aws (read-only) - AWS credentials │ │
│ └──────────────────────────────────────────────────────────┘ │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ Baked-in Assets (read-only) │ │
│ │ • /app/examples/configs - 6 sample configuration files │ │
│ │ • /app/docs - Sphinx-generated HTML documentation │ │
│ │ • /app/schemas - JSON validation schemas │ │
│ │ • /app/templates - CloudFormation template fragments │ │
│ └──────────────────────────────────────────────────────────┘ │
│ │
│ Non-root user: s3user (UID 1000) │
│ Base image: python:3.13-slim (Debian) │
│ Health check: Python import validation │
└────────────────────────────────────────────────────────────────┘
Container Characteristics:
Isolation: Runs as non-root user (s3user) with restricted permissions
Portability: Consistent execution environment across development, staging, production
Security: No hardcoded credentials, read-only credential mounts
Observability: Health checks for container monitoring
Immutability: Application code compiled to .so files, preventing tampering
2.3 Component View (Internal Modules)¶
s3_provisioner/
├── cli.py # Command-line interface entry point
├── __main__.py # Python module execution entry
│
├── config/
│ ├── loader.py (.so) # Configuration loading and validation
│ └── app_config.yaml # Application-level configuration
│
├── core/
│ └── s3_manager.py (.so) # Core S3 provisioning logic
│
├── license/
│ └── validator.py (.so) # AWS Marketplace license validation
│
├── utils/
│ └── html_generator.py (.so) # HTML report generation
│
└── cost/
└── s3_cost_adapter.py (.so) # Cost estimation with AWS Pricing API
common/ (shared library)
├── utils/
│ ├── config_loader.py # YAML configuration parsing
│ ├── aws_access.py # AWS credential verification
│ ├── aws_helpers.py # AWS name length limits and validation
│ └── timing.py # Performance timing utilities
│
└── validators/
└── schema_validator.py # JSON schema validation
Module Responsibilities:
cli.py (Entry Point):
Parse command-line arguments (–config, –action, –solution, –force, –dry-run)
Initialize logging configuration
Dispatch actions to S3 Manager
Handle top-level exception handling
Display user-friendly error messages
config/loader.py (Configuration Management):
Load YAML configuration files
Validate against JSON schema
Flatten nested configuration into attributes
Compute derived values (bucket name, stack name, artifact paths)
Handle configuration defaults
core/s3_manager.py (Core Business Logic):
Orchestrate S3 provisioning workflow
Generate CloudFormation templates (74 resources, 6 outputs)
Manage AWS CloudFormation stacks (create, update, delete, drift detection)
Generate Lambda function code for folder creation
Manage SSM Parameter Store outputs as CloudFormation resources
Handle waiter configuration for async operations
Generate IAM policies
Create HTML pre-deployment and post-deployment reports
license/validator.py (License Management):
Validate AWS Marketplace product code
Check license entitlements via License Manager
Handle license expiration
Report license status
utils/html_generator.py (Reporting):
Generate HTML pre-deployment reports (template structure, resource inventory)
Generate HTML post-deployment reports (stack outputs, physical resource IDs)
Generate HTML cost estimation reports
Include metadata (timestamps, user, region, bucket details)
cost/s3_cost_adapter.py (Cost Estimation):
Fetch region-specific S3 pricing from AWS Pricing API
Calculate storage, request, data transfer, and VPC endpoint costs
Generate usage assumptions files for scenario modeling
Produce HTML cost reports with monthly and annual totals
Fall back to bundled pricing data when offline
common/utils/ (Shared Utilities):
config_loader: Reusable YAML parsing logic
aws_access: AWS credential verification and region validation
aws_helpers: AWS resource name length limits and validation
timing: Performance measurement decorators
3. Technical Strategy & Decisions¶
3.1 Technology Stack¶
Core Technologies:
Python 3.13: Primary programming language
Modern type hints for code clarity
Comprehensive standard library
Cython: Compilation of core modules to .so files
Code protection (obfuscation)
Performance optimization
Intellectual property protection
boto3 1.42+: AWS SDK for Python
S3 API operations
CloudFormation stack management
Lambda function management
SSM Parameter Store management
EC2 VPC endpoint management
IAM policy operations
AWS Marketplace License Manager
uv: Modern Python package manager
Fast dependency resolution
Reproducible builds via uv.lock
Virtual environment management
Infrastructure Technologies:
AWS CloudFormation: Infrastructure as Code engine
Declarative resource definitions
Atomic operations with rollback
Dependency management
Change sets for preview
Drift detection
AWS Lambda: Folder creation execution
Triggered at CloudFormation stack creation
Creates 67 leaf S3 folders programmatically
Automatic cleanup on stack deletion
AWS SSM Parameter Store: Stack output registry
Managed as CloudFormation resources (auto-deleted on stack teardown)
Enables downstream integration with ML Provisioner and other tools
Parameters under
/s3/{bucket_name}/
Docker: Containerization platform
Consistent runtime environment
Multi-stage builds for optimization
Non-root user execution
Health check integration
Configuration & Validation:
YAML: Human-readable configuration format
JSON Schema: Configuration validation
Development Tools:
Git: Version control
Make: Build automation
Sphinx: Documentation generation
Trivy: Container security scanning
3.2 Architecture Decision Records (ADRs)¶
ADR-001: CloudFormation vs. Direct Boto3 Resource Creation¶
Decision: Use AWS CloudFormation for S3 resource provisioning
Context: Need to provision multiple interdependent AWS resources (S3 bucket, Lambda function, IAM roles, VPC endpoint, SSM parameters) with proper dependency management and error handling.
Alternatives Considered:
Direct boto3 API calls for each resource
Terraform
AWS CDK
CloudFormation (chosen)
Rationale:
Atomic Operations: CloudFormation treats all resources as a single unit, ensuring all-or-nothing deployment
Automatic Rollback: Failed deployments automatically roll back to previous state
Dependency Management: CloudFormation handles resource dependencies automatically
State Management: CloudFormation maintains resource state, enabling updates and drift detection
Native AWS Integration: No third-party tools required
Change Sets: Preview changes before applying them
Consequences:
✅ Simplified error handling and recovery
✅ Consistent resource state management
✅ Built-in drift detection
⚠️ CloudFormation-specific limitations (e.g., resource limits per stack)
⚠️ Async operations require waiter pattern implementation
ADR-002: Configuration-Driven Architecture¶
Decision: All S3 topology defined in YAML configuration files
Context: Need to support multiple clients, environments, regions, and bucket configurations without code changes.
Rationale:
Separation of Concerns: Bucket topology (data) separated from provisioning logic (code)
Version Control: Configurations can be versioned, reviewed, and audited in Git
Reusability: Common patterns can be templated and reused
Self-Service: Non-developers can modify bucket configurations
Validation: JSON schema ensures configuration correctness before deployment
Consequences:
✅ Flexible and extensible without code changes
✅ Configuration can be reviewed and approved separately
✅ Easy to create environment-specific variations
⚠️ Requires robust validation to prevent misconfigurations
ADR-003: Lambda for S3 Folder Creation¶
Decision: Use AWS Lambda within the CloudFormation stack to create S3 folders
Context: CloudFormation does not natively support creating S3 objects. The ML folder hierarchy requires 130 folders, of which 67 are leaf nodes.
Rationale:
CloudFormation Integration: Lambda as a custom resource executes during stack creation and deletion
Atomicity: Lambda failure causes CloudFormation rollback — no partial deployments
Cleanup: Lambda automatically removes folders on stack deletion
Scalability: Lambda handles any number of folders without timeout concerns
Consequences:
✅ Fully integrated with CloudFormation lifecycle
✅ Automatic cleanup on stack deletion
✅ No external tooling required
⚠️ Lambda function name must stay within 64-character AWS limit
ADR-004: SSM Parameters as CloudFormation Resources¶
Decision: Manage SSM Parameter Store outputs as AWS::SSM::Parameter CloudFormation resources
Context: Stack outputs need to be accessible to downstream tools (ML Provisioner, custom pipelines). Previous approach used programmatic ssm:PutParameter calls outside CloudFormation.
Rationale:
Lifecycle Management: SSM params are automatically deleted when the stack is torn down — no orphaned parameters
Consistency: All resources tracked in a single CloudFormation stack
Tagging: SSM params receive the same tags as other stack resources
Auditability: Parameter creation/deletion captured in CloudFormation events
Parameters Published:
/s3/{bucket_name}/BucketName/s3/{bucket_name}/BucketArn/s3/{bucket_name}/VPCEndpointId(when VPC integration enabled)
Consequences:
✅ No orphaned SSM parameters after stack deletion
✅ Single source of truth for stack lifecycle
✅ Consistent tagging and auditing
⚠️ Shared tag dictionaries must use per-resource copies to avoid YAML alias issues
ADR-005: Cython Compilation for Code Protection¶
Decision: Compile core Python modules to .so files using Cython
Context: Need to protect intellectual property and prevent code tampering in commercial product.
Modules Compiled:
config/loader.py→loader.cpython-313-x86_64-linux-gnu.socore/s3_manager.py→s3_manager.cpython-313-x86_64-linux-gnu.solicense/validator.py→validator.cpython-313-x86_64-linux-gnu.soutils/html_generator.py→html_generator.cpython-313-x86_64-linux-gnu.socost/s3_cost_adapter.py→s3_cost_adapter.cpython-313-x86_64-linux-gnu.so
Consequences:
✅ Protected intellectual property
✅ Harder to reverse engineer or tamper with
✅ Professional commercial product appearance
⚠️ Platform-specific binaries (Linux x86_64)
⚠️ Debugging more difficult (no source line numbers)
ADR-006: Docker Containerization¶
Decision: Distribute application as Docker container
Context: Need consistent execution environment across development, staging, production, and customer environments.
Container Design:
Multi-stage build (builder + runtime)
Minimal base image (python:3.13-slim)
Non-root user (s3user, UID 1000)
Health checks for monitoring
OCI metadata labels
Example configs baked in at
/app/examples/configs/
Consequences:
✅ Consistent execution environment
✅ Simplified dependency management
✅ Enhanced security through isolation
✅ Easy to deploy and scale
⚠️ Requires Docker runtime on host
ADR-007: Waiter Pattern for Async Operations¶
Decision: Implement explicit waiter configuration for CloudFormation operations
Configuration:
waiter_config = {
'Delay': 10, # Poll every 10 seconds
'MaxAttempts': 60 # Maximum 10 minutes (60 * 10s)
}
Consequences:
✅ Predictable operation timeouts
✅ Better error messages for timeout scenarios
⚠️ Requires tuning for large stack deployments
ADR-008: Naming Convention with Tenant ID¶
Decision: Use tenant ID instead of account ID in resource names
Naming Pattern:
{company_prefix}-{environment}-{tenant_id}-{region}-s3
Example: edge-prod-a001-us-west-1-s3
Rationale:
Readability:
a001more readable than123456789012Brevity: Shorter names in AWS console and CLI output
Consistency: Standardized naming across all resources
Lambda limit compliance: Shorter names stay within Lambda’s 64-character function name limit
Consequences:
✅ Human-readable resource names
✅ Easier to identify resources in AWS console
✅ Shorter CloudFormation stack names
⚠️
bucket_name_overrideneeded for names that still exceed limits
3.3 Design Patterns¶
Command Pattern:
22 actions mapped to methods in S3 Manager
Centralized action dispatch
Consistent error handling across all actions
Factory Pattern:
Dynamic CloudFormation template generation based on configuration
Customized bucket configurations for different environments
Reusable template components (lifecycle policies, SSM params, VPC endpoints)
Strategy Pattern:
Different lifecycle policy strategies (ml-optimized, compliance, development, none)
Pluggable cost estimation per resource type
Flexible bucket naming (auto-generated vs. override)
Template Method Pattern:
Common provisioning workflow with customizable steps
Consistent validation → generation → deployment flow
4. Component Architecture¶
4.1 CLI Interface (cli.py)¶
Responsibilities:
Parse command-line arguments
Initialize logging configuration
Load application configuration
Dispatch actions to S3 Manager
Handle exceptions and display user-friendly errors
Supported Actions (22 total):
validate-config— Validate configuration YAML against schemacreate-policy— Generate IAM policy JSONcreate-prov-template— Generate CloudFormation templatevalidate-prov-template— Validate generated template locallyshow-changes— Preview pending changes via ChangeSetcheck-drift— Detect infrastructure drifttest-deploy— Deploy with isolated test suffixcreate-bucket— Create S3 bucket via CloudFormationprep-master— Prepare master solution structuredeploy-solution— Deploy ML solution foldersdeploy-folders— Clone folder structure from masterupload-template— Upload template to S3gitkeep-full/none/partial— GitKeep file managementpurge-bucket— Remove all .gitkeep from bucketdelete-bucket— Delete bucket directlydelete-cfn-stack— Delete CloudFormation stacktear-down— Complete infrastructure cleanupcost-traffic— Generate usage assumptions filecost-estimate— Calculate estimated monthly costscost-refresh-prices— Refresh pricing from AWS Pricing APIprep-master— Prepare master solution folder structure
4.2 Configuration Loader (config/loader.py)¶
Responsibilities:
Load YAML configuration files
Validate against JSON schema
Flatten nested configuration into attributes
Compute derived values (bucket name, stack name, artifact paths)
Handle configuration defaults
Configuration Schema Validation:
Client information (company_name, company_prefix, account_id, tenant_id)
Environment (env, region)
S3 settings (bucket_name_override, versioning, lifecycle_policy, vpc_id, route_table_ids, tags)
Derived Values:
Bucket Name:
{company_prefix}-{env}-{tenant_id}-{region}-s3(or override)Stack Name:
{bucket_name}-stackLog File:
{bucket_name}-{action}-{timestamp}.logTemplate File:
{bucket_name}-{solution}-template.yamlPolicy File:
{bucket_name}-iam-policy.jsonReport File:
{bucket_name}-{action}-{timestamp}.html
4.3 S3 Manager (core/s3_manager.py)¶
Responsibilities:
Orchestrate S3 provisioning workflow
Generate CloudFormation templates
Manage CloudFormation stacks (create, delete, drift, change sets)
Generate Lambda function code for folder creation
Manage SSM Parameter Store outputs
Generate IAM policies
Create HTML reports
CloudFormation Template Structure (74 resources, 6 outputs):
AWSTemplateFormatVersion: '2010-09-09'
Resources:
MLSolutionsBucket: # AWS::S3::Bucket
BucketPolicy: # AWS::S3::BucketPolicy
FolderCreatorRole: # AWS::IAM::Role
FolderCreatorFunction: # AWS::Lambda::Function
FolderCreatorInvocation: # AWS::CloudFormation::CustomResource
SsmParamBucketName: # AWS::SSM::Parameter
SsmParamBucketArn: # AWS::SSM::Parameter
SsmParamVPCEndpointId: # AWS::SSM::Parameter (conditional)
S3VPCEndpoint: # AWS::EC2::VPCEndpoint (conditional)
# + 65 additional leaf folder resources
Outputs:
BucketName, BucketArn, TotalFolders, CompanyPrefix, Region, VPCEndpointId
4.4 License Validator (license/validator.py)¶
License Validation Flow:
Read product SKU from environment variable
Call AWS Marketplace License Manager API
Verify entitlement status
Return validation result
4.5 HTML Generator (utils/html_generator.py)¶
Report Types:
Pre-Deployment Report (create-prov-template):
Configuration summary (bucket name, region, lifecycle policy, VPC integration)
CloudFormation resource inventory (74 resources, 6 outputs)
Template structure preview
Metadata (timestamp, user, action)
Post-Deployment Report (create-bucket, prep-master):
All pre-deployment content
CloudFormation stack details (stack ID, status)
Physical resource IDs from
describe_stack_resourcesDeployment timeline
SSM parameter paths published
4.6 Cost Adapter (cost/s3_cost_adapter.py)¶
Cost Categories:
S3 storage (Standard, IA, Glacier tiers)
S3 requests (PUT, GET, LIST)
Data transfer (inbound free, outbound charged)
VPC Endpoint (hourly + data processing)
Pricing Data:
Fetched from AWS Pricing API per region
Bundled offline fallback pricing included in image
Refreshable via
cost-refresh-pricesaction
5. Data Architecture¶
5.1 Data Model¶
Storage Strategy: Configuration-driven, no persistent database required
Data Sources:
YAML Configuration Files: Client-provided bucket definitions
JSON Schema Files: Configuration validation rules
Application Config: Tool-level settings (logging, timeouts, defaults)
AWS State: CloudFormation stack state (managed by AWS)
Pricing Data: AWS Pricing API responses (bundled + refreshable)
Configuration Structure:
client:
company_name: "GlobalBank Financial"
company_prefix: "globalbank"
account_id: "123456789012"
tenant_id: "c001"
environment:
env: "prod"
region: "us-west-1"
s3:
bucket_name_override: ""
versioning: true
lifecycle_policy: "ml-optimized"
vpc_id: ""
route_table_ids: ""
tags:
Project: "Customer Churn ML"
Owner: "data-science-team"
CostCenter: "ML-Engineering"
5.2 Data Flow¶
Provisioning Flow:
1. User provides YAML config
↓
2. ConfigLoader reads and parses YAML
↓
3. Schema validation against JSON schema
↓
4. Configuration flattened to attributes
↓
5. S3Manager generates CloudFormation template (74 resources)
↓
6. Template saved to file system (optional)
↓
7. boto3 creates CloudFormation stack
↓
8. CloudFormation provisions S3 bucket, Lambda, IAM, SSM params
↓
9. Lambda creates 130-folder ML hierarchy
↓
10. S3Manager waits for stack completion
↓
11. HTML post-deployment report generated with physical resource IDs
↓
12. Artifacts saved (logs, templates, policies, reports)
5.3 Artifact Management¶
Artifact Types:
Logs: Timestamped operation logs
Templates: CloudFormation YAML templates
Policies: IAM policy JSON documents
Reports: HTML pre-deployment and post-deployment reports
Usage Files: Cost estimation assumptions YAML
Naming Conventions:
Logs:
{bucket_name}-{action}-{solution}-{timestamp}.logTemplates:
{bucket_name}-{solution}-template.yamlPolicies:
{bucket_name}-iam-policy.jsonReports:
{bucket_name}-{action}-{solution}-{timestamp}.html
6. Security Architecture¶
6.1 Security Principles¶
Least Privilege:
Minimal IAM permissions for S3 operations
Non-root container execution
Read-only credential mounts
Scoped AWS API access
Security by Design:
S3 Block Public Access enforced by default
Versioning enabled by default
VPC endpoint support for private access with no internet exposure
Mandatory resource tagging for governance
6.2 Application Security¶
Code Protection:
Core modules compiled to .so files (Cython)
Binary obfuscation prevents reverse engineering
Intellectual property protection
Credential Management:
No hardcoded credentials in code or containers
AWS credentials mounted read-only at runtime
Support for IAM roles (EC2, ECS, Lambda)
Support for AWS SSO and temporary credentials
Input Validation:
JSON schema validation for all configurations
Bucket name validation (length, character set, AWS constraints)
Lambda function name length pre-flight validation (64-char limit)
Path traversal prevention
Dependency Security:
Pinned dependency versions (uv.lock)
urllib3 pinned to
>=2.7.0(CVE fix)Regular security scanning (Trivy / Amazon Inspector)
Minimal base image (python:3.13-slim)
6.3 Container Security¶
Non-Root Execution:
RUN useradd -m -u 1000 s3user
USER s3user
Read-Only Mounts:
-v ~/.aws:/home/s3user/.aws:ro # Read-only credentials
-v ./configs:/app/configs:ro # Read-only configurations
Health Checks:
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
CMD python -c "import sys; sys.exit(0)" || exit 1
6.4 S3 Security Controls¶
All provisioned S3 buckets enforce:
Block Public Access: All four block public access settings enabled
Versioning: Configurable (enabled by default)
Lifecycle Policies: 4 pre-configured profiles for cost and compliance
VPC Endpoints: Optional private access — all traffic stays within AWS private network
Resource Tagging: 7 system tags + custom tags for governance
6.5 Known Vulnerabilities¶
See Security for the full list of known base image vulnerabilities and mitigations.
v1.2.0 Security Fixes:
urllib3 upgraded to
>=2.7.0— resolves CVE findings in Python dependency chainperl-base upgraded to latest Debian patch in base image
7. Deployment Architecture¶
7.1 Deployment Models¶
Local Execution (Development):
python -m s3_provisioner.cli \
--config configs/my-config.yaml \
--action validate-config
Docker Execution (Production):
docker run --rm \
-v ~/.aws:/home/s3user/.aws:ro \
-v $(pwd)/s3/configs:/app/configs:ro \
-v $(pwd)/s3/reports:/app/reports \
s3-provisioner:latest \
--config my-config.yaml \
--action prep-master \
--solution master-solution \
--force
CI/CD Pipeline (Automation):
# GitLab CI example
deploy-s3:
stage: deploy
image: s3-provisioner:latest
script:
- s3-provisioner --config $CONFIG_FILE --action prep-master --solution master-solution --force
only:
- main
7.2 Docker Architecture¶
Multi-Stage Build:
# Stage 1: Builder (dependency installation via uv)
FROM python:3.13-slim AS builder
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
WORKDIR /build
COPY packages/s3-provisioner-tool/pyproject.toml ...
RUN uv sync --no-install-project
# Stage 2: Runtime (minimal image)
FROM python:3.13-slim
RUN useradd -m -u 1000 s3user
WORKDIR /app
COPY --from=builder .venv/lib/python3.13/site-packages ...
COPY src/s3_provisioner ./s3_provisioner
COPY configs/examples/ ./examples/configs/
COPY docs/sphinx/build/html ./docs
USER s3user
ENTRYPOINT ["/app/entrypoint.sh"]
7.3 Infrastructure Requirements¶
Host Requirements:
Docker 20.10+ or compatible runtime
AWS credentials configured (IAM role, access keys, or SSO)
Network access to AWS API endpoints (HTTPS/443)
Sufficient disk space for logs, templates, and reports
AWS Requirements:
AWS account with S3 and CloudFormation permissions
IAM permissions per IAM Permissions
Service quotas: S3 buckets (default 100 per account, can be increased)
7.4 Scalability¶
Horizontal Scaling:
Multiple buckets across regions (parallel execution)
Multiple environments per client (dev, staging, prod)
Multiple clients (configuration-driven)
Multiple solutions per bucket (master + customer-churn + fraud-detection + …)
Limits:
AWS S3 bucket limits (100 per account default, can be increased)
CloudFormation stack limits (500 resources per stack — current templates use 74)
Lambda function name limit (64 characters — validated pre-deployment)
7.5 High Availability¶
Application HA:
Stateless application (no local state)
Idempotent operations (safe to retry)
CloudFormation handles resource state
Failure Scenarios:
CloudFormation failure: Automatic rollback to previous state
Lambda failure: CloudFormation rollback removes partial S3 folders
Application failure: Retry operation (idempotent)
8. Quality Attributes¶
8.1 Performance¶
Provisioning Time:
Bucket creation: 2-5 minutes (typical, includes Lambda folder creation)
Template generation: < 1 second
Configuration validation: < 1 second
IAM policy generation: < 1 second
Cost estimation: 2-5 seconds (AWS Pricing API)
8.2 Reliability¶
Success Rate: 99.9% (target)
Error Handling:
Configuration validation before deployment
CloudFormation automatic rollback on failure
Retry logic for transient AWS API errors
Comprehensive error messages
Idempotency:
CloudFormation ensures idempotent operations
Re-running with same configuration converges to desired state
No duplicate resource creation
8.3 Maintainability¶
Code Organization:
Modular design (config, core, license, utils, cost)
Shared utilities in common library
Clear separation of concerns
Consistent naming conventions
8.4 Usability¶
User Experience:
# Simple command
s3-prov --config my-config.yaml --action prep-master --solution master-solution --force
# Clear output
⚠️ License validation skipped (development mode)
✅ Configuration validated
✅ CloudFormation template generated (74 resources)
📦 Creating CloudFormation stack...
⏳ Waiting for stack creation (2-5 minutes)
✅ Bucket created successfully
✅ Post-deployment report saved: reports/globalbank-prod-c001-us-west-1-s3-create-bucket-...html
9. Integration Architecture¶
9.1 AWS Service Integration¶
AWS CloudFormation:
Purpose: Infrastructure as Code engine
Operations: CreateStack, DeleteStack, DescribeStacks, DescribeStackEvents, DescribeStackResources, CreateChangeSet, DetectStackDrift
Amazon S3:
Purpose: Core provisioned resource
Resources: Bucket, BucketPolicy, lifecycle configuration, versioning, tags, public access block
AWS Lambda:
Purpose: S3 folder creation at deployment time
Integration: Custom resource within CloudFormation stack
Cleanup: Automatically invoked on stack deletion
AWS SSM Parameter Store:
Purpose: Publish bucket outputs for downstream consumption
Integration:
AWS::SSM::ParameterCloudFormation resourcesParameters:
/s3/{bucket_name}/BucketName,BucketArn,VPCEndpointId
Amazon EC2 (VPC Endpoints):
Purpose: Private S3 access within customer VPC
Integration:
AWS::EC2::VPCEndpointCloudFormation resource (conditional)Dependency: Requires existing VPC and route tables (from VPC Provisioner)
AWS IAM:
Purpose: Lambda execution role
Integration:
AWS::IAM::RoleCloudFormation resource
AWS Marketplace License Manager:
Purpose: License validation
Operations: CheckoutLicense, CheckInLicense
AWS Pricing API:
Purpose: Region-specific S3 cost data
Integration: boto3 pricing client (us-east-1 endpoint)
9.2 Integration with Other Provisioners¶
VPC Provisioner (upstream dependency for VPC endpoint):
S3 Provisioner reads
vpc_idandroute_table_idsfrom configurationThese values are published by VPC Provisioner to SSM Parameter Store under
/vpc/{vpc_name}/When VPC integration is enabled, S3 Provisioner creates a VPC endpoint within the customer’s VPC
ML Provisioner (downstream consumer):
ML Provisioner reads S3 bucket outputs from SSM Parameter Store
Parameters consumed:
/s3/{bucket_name}/BucketName,/s3/{bucket_name}/BucketArnML Provisioner deploys ML pipeline infrastructure that uses the provisioned S3 bucket
SG Provisioner (optional upstream):
Not a direct dependency for S3 provisioning
Security groups may be used alongside VPC endpoints for additional network controls
9.3 External Tool Integration¶
CI/CD Pipelines:
GitLab CI, GitHub Actions, Jenkins, AWS CodePipeline
Integration via Docker image in pipeline
Use case: Automated S3 provisioning on infrastructure changes
10. Operational Architecture¶
10.1 Monitoring & Observability¶
Application Logging:
All operations logged with timestamps and log levels
User identity captured (AWS IAM principal via
sts:GetCallerIdentity)Configuration changes tracked
CloudFormation events logged
Log Destinations:
File system:
reports/{bucket_name}-{action}-{timestamp}.logConsole: Real-time feedback to user
HTML Reports:
Pre-deployment: Template structure and resource inventory
Post-deployment: Physical resource IDs and stack outputs
Cost estimation: Monthly/annual cost breakdown
10.2 Backup & Recovery¶
Recovery Time Objective (RTO): < 10 minutes
Redeploy bucket from configuration file
CloudFormation recreates all resources
Recovery Point Objective (RPO): 0 (infrastructure is stateless)
Configuration files version-controlled in Git
CloudFormation templates stored in templates directory
Recovery Procedure:
Retrieve configuration from Git
Execute
prep-masterorcreate-bucketactionCloudFormation recreates all resources
Verify bucket name and SSM parameter outputs
10.3 Maintenance¶
Updates:
Application updates: Docker image tags
Dependency updates: uv lock file regeneration
Base image updates: Rebuild Docker image
Patching:
Security patches: Immediate rebuild and deployment
Bug fixes: Regular release cycle
11. Future Roadmap¶
See Roadmap for planned features including Terraform export, S3 Object Lock, multi-region deployment, S3 Access Points, and REST API/SDK support.
12. Appendices¶
12.1 Glossary¶
S3: Amazon Simple Storage Service — object storage service
CloudFormation: AWS Infrastructure as Code service
Lambda: AWS serverless compute — used for folder creation
SSM Parameter Store: AWS Systems Manager Parameter Store — key-value store for configuration and outputs
VPC Endpoint: Enables private connectivity between VPC and S3 without internet exposure
IAM: Identity and Access Management — AWS authentication and authorization
Waiter: Polling mechanism for asynchronous AWS operations
Lifecycle Policy: S3 rules for transitioning objects between storage classes or expiring them
Cython: Tool to compile Python to C extensions (.so files)
uv: Fast Python package manager used for dependency management
12.2 References¶
AWS Documentation:
Internal Documentation:
User Guide - Complete command reference
Security - Security policy and known vulnerabilities
Troubleshooting - Troubleshooting guide
IAM Permissions - IAM permission requirements
Configuration Reference - Configuration file reference
12.3 Version History¶
Version |
Date |
Changes |
|---|---|---|
1.0.0 |
2026-04-01 |
Initial architecture documentation |
1.2.0 |
2026-06-13 |
Full rewrite — corrected VPC-specific content, added SSM CFN resources, Lambda folder creation, cost adapter, example configs, 1.2.0 security fixes |
Document Metadata:
Author: S3 Provisioner Team
Last Updated: 2026-06-13
Next Review: 2026-09-30
Status: Production Ready
Classification: Internal Use