API Reference

This section contains the complete API documentation auto-generated from source code docstrings.

Core Modules

S3 Manager

class s3_provisioner.core.s3_manager.S3Manager(config_file_name, action, solution=None, dry_run=False, log_level='INFO', force=False, app_config=None)

Bases: object

Manages S3 bucket provisioning and ML solution folder structures for AWS.

This class provides comprehensive S3 infrastructure management for ML workloads, including CloudFormation-based bucket creation, folder structure generation, lifecycle policies, VPC endpoints, and Parameter Store integration.

The S3Manager supports multiple ML solutions with standardized folder hierarchies for data, models, notebooks, artifacts, code, and configuration files. It uses .gitkeep files to maintain folder visibility in S3 and provides flexible visibility control through depth-based filtering.

Key Features:

  • CloudFormation template generation for S3 infrastructure

  • Automated ML folder structure creation (70+ folders per solution)

  • VPC endpoint configuration for private S3 access

  • Lifecycle policy management (ml-optimized, compliance, development)

  • IAM policy generation for least-privilege access

  • Parameter Store integration for infrastructure outputs

  • Dry-run mode for safe testing

  • Force deletion with automatic cleanup

bucket_name

Name of the S3 bucket being managed

Type:

str

region

AWS region for bucket deployment

Type:

str

action

Current action being performed

Type:

str

solution

ML solution name (e.g., ‘master-solution’, ‘customer-churn’)

Type:

str

dry_run

If True, simulates operations without making changes

Type:

bool

force

If True, bypasses confirmation prompts for destructive operations

Type:

bool

stack_name

CloudFormation stack name for the bucket

Type:

str

s3_client

Boto3 S3 client

cf_client

Boto3 CloudFormation client

ssm_client

Boto3 Systems Manager client

logger

Configured logger instance

Class Constants:

DEFAULT_LAMBDA_RUNTIME (str): Lambda runtime version for custom resources DEFAULT_LAMBDA_TIMEOUT (int): Lambda timeout in seconds STACK_CREATE_WAITER_DELAY (int): Delay between stack creation checks (seconds) STACK_CREATE_WAITER_MAX_ATTEMPTS (int): Maximum stack creation wait attempts STACK_DELETE_WAITER_DELAY (int): Delay between stack deletion checks (seconds) STACK_DELETE_WAITER_MAX_ATTEMPTS (int): Maximum stack deletion wait attempts S3_DELETE_BATCH_SIZE (int): Maximum objects to delete in single batch GITKEEP_PARTIAL_DEPTH (int): Folder depth for partial visibility control DEFAULT_REGION (str): Default AWS region if not specified

Example

>>> from s3_provisioner.core import S3Manager
>>> manager = S3Manager(
...     config_file_name='config.yaml',
...     action='create-bucket',
...     solution='customer-churn',
...     dry_run=False
... )
>>> exit_code = manager.execute()

Note

  • Requires valid AWS credentials with appropriate IAM permissions

  • Configuration file must pass JSON schema validation

  • Some actions require –solution parameter

  • Use dry_run=True to preview changes before execution

DEFAULT_LAMBDA_RUNTIME = 'python3.12'
DEFAULT_LAMBDA_TIMEOUT = 120
STACK_CREATE_WAITER_DELAY = 10
STACK_CREATE_WAITER_MAX_ATTEMPTS = 60
STACK_DELETE_WAITER_DELAY = 10
STACK_DELETE_WAITER_MAX_ATTEMPTS = 60
S3_DELETE_BATCH_SIZE = 1000
GITKEEP_PARTIAL_DEPTH = 5
DEFAULT_REGION = 'us-east-1'
DRIFT_POLL_INTERVAL = 10
DRIFT_POLL_TIMEOUT = 600
CHANGESET_POLL_INTERVAL = 5
CHANGESET_MAX_ATTEMPTS = 60
__init__(config_file_name, action, solution=None, dry_run=False, log_level='INFO', force=False, app_config=None)

Inintialize class members from config file and parameters.

Parameters:

  • config_file_name (str) – Name of the configuration file.

  • action (str) – Action to be performed.

  • solution (Optional[str]) – Name of the business solution (optional for some actions).

  • dry_run (bool) – Dry run flag, if True no changes are made.

  • log_level (str) – Logging level for the operation.

  • force (bool) – Force flag, if True certain validations are skipped.

  • app_config (AppConfig) – Application configuration object (AppConfig instance).

execute()

Executes the specified action on the S3 folder. This method serves as the main entry point for executing the desired action based on the command provided. It first displays the class members for debugging purposes, then validates any requirements for the specified action (e.g., checking if the –solution parameter is provided for actions that require it). Finally, it calls the _execute_action method to perform the actual execution of the action and returns 0 for success or 1 for failure based on the result.

Returns:

0 for success, 1 for failure.

Return type:

int

Configuration

Configuration Loader

class s3_provisioner.config.loader.AppConfig(config_dict)

Bases: object

Application configuration with embedded defaults.

Loads configuration from YAML files and provides convenient access to configuration values with automatic path resolution.

Features:

  • Loads config from provided dictionary (from YAML)

  • Resolves directory paths to absolute paths

  • Provides access to config values via get() method with dot notation

  • Handles directory path resolution and validation

__init__(config_dict)

Initializes the config object with config dictionary.

Parameters:

config_dict (Dict[str, Any]) – Configuration dictionary loaded from YAML.

get(key_path, default=None)

Gets a config value using dot notation.

Parameters:

  • key_path (str) – Dot-separated path (e.g., ‘directories.configs_dir’)

  • default (Any) – Default value if key not found

Returns:

Config value or default

Return type:

Any

get_directory(dir_name)

Gets a directory path as a Path object.

Parameters:

dir_name (str) – Directory name (configs_dir, reports_dir, etc.)

Returns:

Path object

Return type:

Path

Raises:

ValueError – If directory not configured

s3_provisioner.config.loader.load_app_config(custom_config_path=None)

Loads and returns application configuration.

Resolution order: 1. Custom config file (if provided via –app-config) 2. Embedded default config (always available)

Parameters:

custom_config_path (Optional[str]) – Optional path to custom config file

Returns:

AppConfig instance

Return type:

AppConfig

Raises:

Utilities

HTML Generator

HTML documentation generator for S3 provisioning.

Provides two report types:

s3_provisioner.utils.html_generator.generate_template_documentation(config_file, template_file, cfn_template, solution, output_path)

Generate HTML documentation for CloudFormation template.

Parameters:

  • config_file (str) – Path to configuration YAML file.

  • template_file (str) – Path to generated CloudFormation template.

  • cfn_template (Dict[str, Any]) – Parsed CloudFormation template dictionary.

  • solution (str) – Solution name.

  • output_path (str) – Path where HTML file should be saved.

Returns:

Path to generated HTML file.

Return type:

str

s3_provisioner.utils.html_generator.generate_deployment_documentation(config_file, stack_name, region, stack_info, solution, output_path)

Generate HTML documentation for S3 deployment.

Parameters:

  • config_file (str) – Path to configuration YAML file.

  • stack_name (str) – CloudFormation stack name.

  • region (str) – AWS region.

  • stack_info (Dict[str, Any]) – Stack information from describe_stacks.

  • solution (str) – Solution name.

  • output_path (str) – Path where HTML file should be saved.

Returns:

Path to generated HTML file.

Return type:

str

s3_provisioner.utils.html_generator.generate_cost_report(bucket_name, region, config_file, cost_result, pricing_source, output_path)

Generate HTML cost estimation report for S3 infrastructure.

Produces a professional HTML document with a cost breakdown table showing storage, request, data transfer, and VPC Endpoint costs with monthly and annual totals.

Parameters:

  • bucket_name (str) – Name of the S3 bucket.

  • region (str) – AWS region code.

  • config_file (str) – Path to the S3 configuration YAML file.

  • cost_result (Dict[str, Any]) – Cost breakdown dictionary from calculate().

  • pricing_source (str) – Description of the pricing source.

  • output_path (str) – Filesystem path where the HTML file should be saved.

Returns:

Path to the generated HTML file.

Return type:

str

Raises:

License

License Validator

class s3_provisioner.license.validator.LicenseValidator

Bases: object

Validates product licenses against the AWS Marketplace License Manager.

This class handles the end-to-end license verification workflow, including secure communication with AWS APIs to confirm customer entitlements. To optimize performance and minimize latency, it employs an in-memory caching mechanism, reducing API calls for frequently validated tokens or customers.

product_sku

AWS Marketplace product sku to validate against.

Type:

str

region

AWS region for the Marketplace API.

Type:

str

license_client

Boto3 client for AWS Marketplace License Manager.

key_fingerprint

The fingerprint of the public key used for license validation.

entitlement_name

The name of the entitlement to check for in the AWS License Manager.

license_cache_file

Path to the local cache file.

Type:

str

cache_duration

Duration for which the cache is considered valid (hours).

Type:

timedelta

Integration Workflow:

  1. Checks cache for active, unexpired validation result.

  2. If missing/expired, calls CheckoutLicense API.

  3. Validates product SKU and entitlement status.

  4. Updates cache and returns validation status.

__init__()

Initialize the LicenseValidator with AWS Marketplace configuration.

Reads configuration from environment variables and sets up the AWS License Manager client for license validation.

Environment Variables:

MARKETPLACE_PRODUCT_SKU: AWS Marketplace product SKU (required for validation) AWS_REGION: AWS region for Marketplace API (default: ‘us-east-1’)

verify_aws_marketplace_subscription()

Validate AWS Marketplace subscription via License Manager CheckoutLicense API.

Returns:

True if valid license, False otherwise

Return type:

bool

CLI Interface

class s3_provisioner.cli.S3ProvisionerCLI

Bases: BaseCLI

CLI for S3 Provisioner Tool.

Extends BaseCLI to implement S3-specific actions and arguments. Supports 16 actions grouped by safety level:

  • Local: validate-config, create-prov-template, validate-prov-template, create-policy

  • Bucket operations: create-bucket, deploy-solution, deploy-folders, upload-template, prep-master

  • Gitkeep management: gitkeep-full, gitkeep-none, gitkeep-partial

  • Read-only: show-changes, check-drift

  • Testing: test-deploy

  • Destructive: purge-bucket, delete-bucket, delete-cfn-stack, tear-down

Includes AWS Marketplace license validation before executing any actions.

__init__()

Initialize the S3 Provisioner CLI.

is_action_required()

Determine if an action argument is required for this CLI.

Return type:

bool

Returns:

Always True for this CLI

get_actions()

Return the list of valid actions for the S3 Provisioner CLI.

Return type:

List[str]

Returns:

List of action strings

get_actions_help()

Return help text describing all available actions.

Return type:

str

Returns:

Formatted help text for all S3 actions

get_manager_class()

Return the manager class for this CLI.

Return type:

Type[Any]

Returns:

S3Manager class reference

get_example_usage()

Return example usage instructions.

Detects Docker environment via /.dockerenv or /proc/1/cgroup and adjusts command examples accordingly.

Returns:

Example usage string with Docker or native CLI commands.

Return type:

str

add_custom_arguments(parser)

Add S3 Provisioner-specific CLI arguments.

Parameters:

parser (Any) – The argparse parser to add arguments to.

Return type:

None

create_manager_instance(args)

Create a new S3Manager instance.

Parameters:

args (argparse.Namespace) – Parsed command-line arguments.

Returns:

Instance initialized with CLI arguments and app config.

Return type:

S3Manager

requires_force(action)

Determine if an action requires the --force flag.

Parameters:

action (str) – The action being performed.

Returns:

True for destructive operations (delete-bucket, delete-cfn-stack, tear-down).

Return type:

bool

s3_provisioner.cli.main()

Entry point for S3 Provisioner CLI.

Validates AWS Marketplace license before running the CLI. Exits with code 1 if license validation fails.