Backup and Recovery Procedures

Procedures for backing up and recovering S3 Provisioner configurations and deployed infrastructure.

Table of Contents

• What to Backup

• Backup Strategy

• Recovery Procedures

• S3 Infrastructure Recovery

• Testing Recovery

• Retention Policy

---

What to Backup

Critical Data

1. Configuration Files (s3/configs/)

   • Client configuration YAML files

   • These define your entire S3 infrastructure — losing them means recreating from scratch

2. CloudFormation Templates (s3/templates/)

   • Generated provisioning templates

   • Can be regenerated from configs, but useful to have as backup

3. IAM Policies (s3/policies/)

   • Generated IAM policy files

   • Can be regenerated from configs

4. Reports and Logs (s3/reports/)

   • Deployment logs and HTML reports

   • Audit trail for compliance

Priority Order

Priority	Data	Why
1 (Critical)	configs/	Cannot be regenerated — source of truth
2 (Important)	reports/	Audit trail, deployment history
3 (Regenerable)	templates/	Can regenerate from configs
4 (Regenerable)	policies/	Can regenerate from configs

Backup Strategy

Manual Backup

# Create dated backup of all S3 provisioner artifacts
DATE=$(date +%Y%m%d)
tar -czf s3-provisioner-backup-$DATE.tar.gz \
  s3/configs/ \
  s3/templates/ \
  s3/policies/ \
  s3/reports/

Upload Backup to S3

# Upload to a separate backup bucket
aws s3 cp s3-provisioner-backup-$DATE.tar.gz \
  s3://your-backup-bucket/s3-provisioner/$DATE/ \
  --storage-class STANDARD_IA \
  --sse AES256

Automated Daily Backup

#!/bin/bash
# backup-s3-provisioner.sh

set -e

DATE=$(date +%Y%m%d)
BACKUP_DIR="backups/s3-provisioner"
BACKUP_BUCKET="your-backup-bucket"

mkdir -p "$BACKUP_DIR"

# Backup configs (critical)
tar -czf "$BACKUP_DIR/configs-$DATE.tar.gz" s3/configs/

# Backup templates and policies (regenerable but convenient)
tar -czf "$BACKUP_DIR/templates-$DATE.tar.gz" s3/templates/
tar -czf "$BACKUP_DIR/policies-$DATE.tar.gz" s3/policies/

# Backup recent reports (last 7 days)
find s3/reports/ -mtime -7 -name "*.log" -o -name "*.html" | \
  tar -czf "$BACKUP_DIR/reports-$DATE.tar.gz" -T -

# Upload to S3
aws s3 sync "$BACKUP_DIR/" "s3://$BACKUP_BUCKET/s3-provisioner/$DATE/" \
  --storage-class STANDARD_IA --sse AES256

echo "Backup completed: $DATE"

Recovery Procedures

Restore Configuration Files

# Download backup
aws s3 cp s3://your-backup-bucket/s3-provisioner/20260401/configs-20260401.tar.gz .

# Restore
tar -xzf configs-20260401.tar.gz

# Validate restored configs
docker run --rm \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action validate-config

Regenerate Templates and Policies

If templates or policies are lost but configs are intact:

# Regenerate IAM policy
docker run --rm \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/policies:/app/policies \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-policy

# Regenerate CloudFormation template
docker run --rm \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-prov-template \
  --solution master-solution

S3 Infrastructure Recovery

Document Existing Infrastructure

Before any destructive operations, capture the current state:

# Export bucket configuration
aws s3api get-bucket-versioning --bucket edge-prod-b001-us-west-1-s3
aws s3api get-bucket-lifecycle-configuration --bucket edge-prod-b001-us-west-1-s3
aws s3api get-bucket-tagging --bucket edge-prod-b001-us-west-1-s3

# Export CloudFormation stack
aws cloudformation get-template \
  --stack-name edge-prod-b001-us-west-1-s3-stack \
  --region us-west-1 > stack-template-backup.json

# Check stack status
aws cloudformation describe-stacks \
  --stack-name edge-prod-b001-us-west-1-s3-stack \
  --region us-west-1

Recreate S3 Infrastructure from Config

If the bucket and stack were deleted but you have the config file:

# Validate config
docker run --rm \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action validate-config

# Regenerate template
docker run --rm \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action create-prov-template \
  --solution master-solution

# Redeploy
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/templates:/app/templates \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action prep-master \
  --solution master-solution \
  --force

# Redeploy solution folders
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action deploy-solution \
  --solution customer-churn

Note: This recreates the bucket structure but does not restore data that was in the bucket. S3 data recovery depends on versioning and replication settings.

Verify Recovery

# Check bucket exists
aws s3 ls s3://edge-prod-b001-us-west-1-s3/

# Check solution folders
aws s3 ls s3://edge-prod-b001-us-west-1-s3/solutions/

# Check CloudFormation stack
aws cloudformation describe-stacks \
  --stack-name edge-prod-b001-us-west-1-s3-stack \
  --region us-west-1 \
  --query 'Stacks[0].StackStatus'

# Run drift detection
docker run --rm \
  -v ~/.aws:/home/s3user/.aws:ro \
  -v $(pwd)/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action check-drift

Testing Recovery

Run a recovery test periodically to verify your backups are usable:

# 1. Create backup
tar -czf /tmp/s3-recovery-test.tar.gz s3/configs/

# 2. Restore to temp location
mkdir -p /tmp/s3-recovery-test
tar -xzf /tmp/s3-recovery-test.tar.gz -C /tmp/s3-recovery-test

# 3. Validate restored configs
docker run --rm \
  -v /tmp/s3-recovery-test/s3/configs:/app/configs:ro \
  -v $(pwd)/s3/reports:/app/reports \
  s3-provisioner:latest \
  --config edge-prod-b001-us-west-1-s3.yaml \
  --action validate-config

# 4. Cleanup
rm -rf /tmp/s3-recovery-test /tmp/s3-recovery-test.tar.gz

Retention Policy

Backup Type	Retention	Storage Class
Daily	7 days	S3 Standard-IA
Weekly	4 weeks	S3 Standard-IA
Monthly	12 months	S3 Glacier
Yearly	Indefinite	S3 Glacier Deep Archive

---

Copyright © 2025 Axon Tech Labs All rights reserved.

See LICENSE.txt for terms and conditions.