Troubleshooting¶

Table of Contents¶

Quick Diagnostics
Common Errors
Performance Issues
VPC Endpoint Issues
Advanced Troubleshooting
Getting Help
Additional Resources

Quick Diagnostics¶

# Check AWS credentials
aws sts get-caller-identity

# Check Docker version
docker --version

# Test AWS S3 access
aws s3 ls

# Validate configuration
docker run --rm \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action validate-config

Common Pitfalls¶

1. Not quoting account_id in YAML

❌ account_id: 123456789012 (loses leading zeros)
✅ account_id: "123456789012" (preserves format)

2. Skipping configuration validation

Always run validate-config before AWS operations
Catches most errors before deployment — saves time and avoids failed stacks

3. Not reviewing generated templates

Review CloudFormation templates before prep-master or create-bucket
Use validate-prov-template to catch reference errors locally
Prevents unexpected resource creation

4. Using access keys in production

❌ Environment variables with long-lived access keys
✅ IAM roles (EC2/ECS) or AWS profiles with MFA
Access keys are visible in process lists and Docker history

Common Errors¶

AWS Credentials¶

Error: Unable to locate credentials¶

Error: Unable to locate credentials. You can configure credentials by running "aws configure".

Solution:

# Option 1: Environment variables
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
export AWS_DEFAULT_REGION=us-west-1

# Option 2: AWS profile
export AWS_PROFILE=default

# Verify
aws sts get-caller-identity

Error: The security token included in the request is invalid¶

Error: The security token included in the request is invalid

Causes:

Expired temporary credentials
Invalid access key
Credentials from different account

Solution:

# Refresh credentials
aws sts get-session-token

# Or use long-term credentials
cat ~/.aws/credentials

# Verify current identity
aws sts get-caller-identity

Error: Access Denied¶

Error: An error occurred (AccessDenied) when calling the CreateBucket operation

Solution:

# Check current user
aws sts get-caller-identity

# Generate IAM policy
docker run --rm \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action create-policy

# Review and attach generated policy
cat policies/{bucket_name}-iam-policy.json

# See IAM_PERMISSIONS.md for required permissions

Configuration Errors¶

Error: Bucket name already exists¶

Error: Bucket name already exists and is owned by another account

Solution:

# Use bucket_name_override with unique name
s3:
  bucket_name_override: "edge-ml-data-prod-2024"

Error: Invalid bucket name¶

Error: Invalid bucket name: must be 3-63 characters, lowercase only

Solution:

# Valid names
s3:
  bucket_name_override: "my-data-bucket"
  bucket_name_override: "company-prod-2024"

# Invalid names (will fail)
s3:
  bucket_name_override: "My_Bucket"      # uppercase and underscore
  bucket_name_override: "ab"             # too short
  bucket_name_override: "my..bucket"     # consecutive dots

Error: Configuration validation failed¶

Error: Configuration validation failed: 'client' is a required property

Solution: Ensure configuration has all three required sections:

client:
  company_name: Edge Corp
  company_prefix: edge
  account_id: "123456789012"
  tenant_id: "a001"

environment:
  env: prod
  region: us-west-1

s3:
  bucket_name_override: ""
  versioning: true
  lifecycle_policy: ml-optimized
  vpc_id: ""
  route_table_ids: ""

Error: Invalid lifecycle_policy value¶

Error: 'custom' is not one of ['ml-optimized', 'compliance', 'development', 'none']

Solution:

# Valid values only
s3:
  lifecycle_policy: ml-optimized  # or compliance, development, none

Error: Account ID must be quoted¶

Error: Account ID lost leading zeros

Solution:

# Wrong (loses leading zeros)
client:
  account_id: 123456789012

# Correct (preserves leading zeros)
client:
  account_id: "123456789012"

Lifecycle Policy Errors¶

Error: Lifecycle rules not applied¶

Symptoms: CloudFormation template generated but no LifecycleConfiguration section

Causes:

lifecycle_policy set to ‘none’
lifecycle_policy field missing (defaults to ‘none’)

Solution:

s3:
  lifecycle_policy: ml-optimized  # Explicitly set profile

Verify:

# Generate template
docker run --rm \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action create-prov-template \
  --solution master-solution

# Check for LifecycleConfiguration
grep -A 10 "LifecycleConfiguration" templates/your-config_master-solution_s3_template.yaml

CloudFormation Errors¶

Error: Stack already exists¶

Error: Stack [edge-prod-a001-us-west-1-s3-stack] already exists

Solution:

# Option 1: Delete existing stack first
docker run --rm \
  -e AWS_PROFILE=default \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action tear-down \
  --force

# Option 2: Use different solution name
docker run --rm \
  -e AWS_PROFILE=default \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action create-bucket \
  --solution master-solution-v2 \
  --force

Error: CloudFormation rollback¶

Error: Stack creation failed and rolled back

Solution:

# Check CloudFormation events
aws cloudformation describe-stack-events \
  --stack-name edge-prod-a001-us-west-1-s3-stack \
  --max-items 20

# Common causes:
# 1. Insufficient IAM permissions
# 2. Invalid VPC ID
# 3. Invalid route table IDs
# 4. Bucket name conflict

# Review logs
cat reports/*.log

Template Validation Errors¶

Error: Template file not found¶

Error: Template file not found: templates/your-config_master-solution_s3_template.yaml

Solution: Generate the template first, or let validate-prov-template auto-generate it:

docker run --rm \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action validate-prov-template \
  --solution master-solution

Note: If the template does not exist, it will be generated automatically before validation.

Error: Invalid !Ref target¶

Error: !Ref target 'InvalidResource' not found in Resources or Parameters

Causes:

Template references a resource that doesn’t exist
Typo in resource name
Template was manually edited

Solution: Regenerate the template from configuration:

docker run --rm \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action create-prov-template \
  --solution master-solution

Change Preview Errors¶

Error: Stack does not exist for show-changes¶

Error: Stack [edge-prod-a001-us-west-1-s3-stack] does not exist

Cause: show-changes requires a deployed stack to compare against.

Solution: Deploy the stack first, then preview changes:

# Deploy first
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action prep-master \
  --solution master-solution \
  --force

# Then preview changes
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action show-changes \
  --solution master-solution

Error: No changes detected¶

Symptoms: show-changes reports no pending changes.

Cause: The deployed stack matches the current template.

Solution: This is expected when no configuration changes have been made. Modify your configuration or template, then re-run show-changes.

Drift Detection Errors¶

Error: Stack does not exist for check-drift¶

Error: Stack [edge-prod-a001-us-west-1-s3-stack] does not exist

Cause: check-drift requires a deployed stack to detect drift against.

Solution: Deploy the stack first:

docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action prep-master \
  --solution master-solution \
  --force

Error: Drift detection timeout¶

Symptoms: Drift detection takes too long or times out.

Causes:

Large number of resources in the stack
AWS API throttling

Solution:

# Check drift detection status manually
aws cloudformation describe-stack-drift-detection-status \
  --stack-drift-detection-id <detection-id>

# Retry
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action check-drift

Docker Errors¶

Error: Cannot connect to Docker daemon¶

Error: Cannot connect to the Docker daemon at unix:///var/run/docker.sock

Solution:

# Start Docker
sudo systemctl start docker

# Or Docker Desktop
# Start Docker Desktop application

# Verify
docker ps

Error: Permission denied accessing volume¶

Error: Permission denied: '/app/configs/your-config.yaml'

Solution:

# Check file permissions
ls -la configs/

# Fix permissions
chmod 644 configs/your-config.yaml

# Or run with user mapping
docker run --rm --user $(id -u):$(id -g) \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action validate-config

Error: Volume mount not found¶

Error: No such file or directory: '/app/configs/your-config.yaml'

Solution:

# Verify file exists on host
ls -la configs/your-config.yaml

# Use absolute path
docker run --rm \
  -v /full/path/to/configs:/app/configs \
  -v /full/path/to/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action validate-config

# Or use $(pwd)
docker run --rm \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action validate-config

Deletion Errors¶

Error: Bucket not empty¶

Error: The bucket you tried to delete is not empty

Solution:

# Use tear-down action (recommended)
docker run --rm \
  -e AWS_PROFILE=default \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action tear-down \
  --force

# Or manually empty bucket first
aws s3 rm s3://bucket-name --recursive

# Delete all versions (if versioning enabled)
aws s3api delete-objects \
  --bucket bucket-name \
  --delete "$(aws s3api list-object-versions \
    --bucket bucket-name \
    --output json \
    --query '{Objects: Versions[].{Key:Key,VersionId:VersionId}}')"

Error: Stack deletion failed¶

Error: CloudFormation stack deletion failed

Solution:

# Check stack events
aws cloudformation describe-stack-events \
  --stack-name edge-prod-a001-us-west-1-s3-stack

# Common causes:
# 1. Bucket not empty
# 2. VPC endpoint in use
# 3. Lambda function still exists

# Force delete stack (use with caution)
aws cloudformation delete-stack \
  --stack-name edge-prod-a001-us-west-1-s3-stack

Solution Deployment Errors¶

Error: Solution not found¶

Error: Unknown solution: my-custom-solution

Example Solutions (for reference only):

master-solution
customer-churn
demand-forecasting
fraud-detection

Note: You can specify any solution name that follows the naming convention (lowercase alphanumeric with hyphens, starting and ending with alphanumeric character, max 63 characters). The examples above are for reference only.

Solution: Deploy master-solution first, then deploy your custom solution with any valid name.

Error: Bucket does not exist¶

Error: Bucket does not exist for solution deployment

Solution:

# Create bucket first
docker run --rm \
  -e AWS_PROFILE=default \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action create-bucket \
  --solution master-solution \
  --force

# Then deploy solution
docker run --rm \
  -e AWS_PROFILE=default \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action deploy-solution \
  --solution customer-churn

Cost Estimation Errors¶

Error: Template not found¶

Message: Template not found. Run 'create-prov-template' first.

Cause: The cost-traffic or cost-estimate action requires a generated CloudFormation template.

Solution:

# Generate the template first
--action create-prov-template --solution master-solution

# Then run cost estimation
--action cost-traffic --solution master-solution
--action cost-estimate --solution master-solution

Error: Usage file not found¶

Message: Usage file not found. Run 'cost-traffic' first.

Cause: The cost-estimate action requires a usage assumptions file.

Solution:

# Generate usage assumptions
--action cost-traffic --solution master-solution

# Edit the file to match your expected usage
# Then run cost estimate
--action cost-estimate --solution master-solution

Error: Failed to refresh pricing¶

Message: Failed to refresh pricing: ...

Cause: The cost-refresh-prices action could not reach the AWS Pricing API. Common reasons: no AWS credentials, no internet access, or insufficient permissions.

Solution:

Verify AWS credentials are configured
Ensure network connectivity to AWS APIs
The tool will use built-in pricing data as fallback — cost-estimate still works without refreshing

License Validation Errors¶

Error: License validation failed¶

Error: AWS Marketplace subscription not found

Solution:

Verify AWS Marketplace subscription is active
Check IAM permissions for AWS Marketplace
Contact AWS Marketplace support

Performance Issues¶

Slow CloudFormation Stack Creation¶

Symptoms: Stack creation takes longer than expected (>5 minutes)

Causes:

Lambda function creation
VPC endpoint creation
AWS API throttling

Solution:

# Monitor stack events
aws cloudformation describe-stack-events \
  --stack-name edge-prod-a001-us-west-1-s3-stack

# Check AWS service health
curl https://status.aws.amazon.com/

# Wait for completion (can take 2-5 minutes)

Timeout Errors¶

Symptoms: Operation times out before completion

Solution:

# Check CloudFormation stack status
aws cloudformation describe-stacks \
  --stack-name edge-prod-a001-us-west-1-s3-stack

# If stack is still creating, wait for completion
aws cloudformation wait stack-create-complete \
  --stack-name edge-prod-a001-us-west-1-s3-stack

VPC Endpoint Issues¶

Error: Invalid VPC ID¶

Error: VPC vpc-12345 does not exist

Solution:

# List VPCs
aws ec2 describe-vpcs

# Use correct VPC ID or leave empty
s3:
  vpc_id: "vpc-0a1b2c3d4e5f6g7h8"
  # OR
  vpc_id: ""

Error: Invalid route table IDs¶

Error: Route table rtb-12345 does not exist

Solution:

# List route tables for VPC
aws ec2 describe-route-tables \
  --filters "Name=vpc-id,Values=vpc-0a1b2c3d4e5f6g7h8"

# Use correct route table IDs (comma-separated)
s3:
  vpc_id: "vpc-0a1b2c3d4e5f6g7h8"
  route_table_ids: "rtb-0a1b2c3d,rtb-4e5f6g7h"

Advanced Troubleshooting¶

Enable Debug Logging¶

# Set verbose logging
docker run --rm \
  -e AWS_PROFILE=default \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action create-bucket \
  --solution master-solution \
  --verbose debug \
  --force

Inspect Container¶

# Run interactive shell
docker run --rm -it \
  -v $(pwd)/s3/configs:/app/configs \
  --entrypoint /bin/bash \
  s3-provisioner:latest

# Inside container
ls -la /app/
cat /app/configs/your-config.yaml
python -m s3_provisioner.cli --help

Check AWS API Calls¶

# Enable CloudTrail logging
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateBucket \
  --max-results 10

# Check for errors
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateBucket \
  --query 'Events[?ErrorCode!=`null`]'

Network Connectivity¶

# Test AWS endpoint connectivity
curl -I https://s3.us-west-1.amazonaws.com

# Test from container
docker run --rm \
  --entrypoint /bin/bash \
  s3-provisioner:latest \
  -c "curl -I https://s3.us-west-1.amazonaws.com"

Review Generated Files¶

# Check generated IAM policy
cat policies/your-config_iam_policy.json

# Check generated CloudFormation template
cat templates/your-config_master-solution_s3_template.yaml

# Check execution logs
cat reports/*.log

Getting Help¶

Collect Diagnostic Information¶

# System info
docker --version
aws --version
uname -a

# AWS identity
aws sts get-caller-identity

# Configuration (sanitized)
cat configs/your-config.yaml

# Error output
docker run --rm \
  -v $(pwd)/s3/configs:/app/configs \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config your-config.yaml \
  --action validate-config 2>&1 | tee error.log

Contact Support¶

Include in support request:

Docker version
AWS region
Sanitized configuration file
Complete error message
Steps to reproduce
Expected vs actual behavior
Log files from reports/ directory

See SUPPORT.md for contact information.

Additional Resources¶

USER_GUIDE.md - Complete command reference
CONFIGURATION.md - Configuration reference
S3_FOLDERS.md - Complete folder hierarchy reference
GOVERNANCE_COMPLIANCE.md - Enterprise governance implementation guide
IAM_PERMISSIONS.md - Required permissions
SUPPORT.md - Support information
AWS S3 Troubleshooting: https://docs.aws.amazon.com/AmazonS3/latest/userguide/troubleshooting.html
AWS CloudFormation Troubleshooting: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/troubleshooting.html

See LICENSE.txt for terms and conditions.