Update Procedures

Procedures for safely updating the ML Provisioner tool and maintaining deployed infrastructure.

Table of Contents

Version Strategy

Semantic Versioning

We follow Semantic Versioning 2.0.0:

  • MAJOR (1.x.x): Breaking changes, incompatible configuration format changes

  • MINOR (x.1.x): New features, new actions, backward compatible

  • PATCH (x.x.1): Bug fixes, backward compatible

Image Tags

ML Provisioner uses explicit version tags per tier. Only version-pinned tags are published to AWS Marketplace ECR:

Image

Description

ml-provisioner:starter-1.0.1

Starter tier — version 1.0.1

ml-provisioner:professional-1.0.1

Professional tier — version 1.0.1

ml-provisioner:enterprise-1.0.1

Enterprise tier — version 1.0.1

Tier-only tags (e.g. ml-provisioner:enterprise) are used locally during development and testing but are not published to Marketplace ECR.

Pre-Update Checklist

Before updating:

  • Review Release Notes for breaking changes

  • Backup current configuration files

  • Note down current image tags before pulling the new version (for rollback reference): docker images ml-provisioner

  • Test new version against a non-production config first

  • Schedule maintenance window for production updates

  • Notify stakeholders

  • Prepare rollback plan (keep previous image tagged locally)

Updating the Docker Image

Note on tag resolution: Only explicit version tags are published to AWS Marketplace ECR (e.g. ml-provisioner:enterprise-1.0.1). Tier-only tags (e.g. ml-provisioner:enterprise) are not published — you must always specify the version when pulling. This ensures every deployment is traceable and prevents accidental upgrades.

Get New Version from AWS Marketplace

When a new version is available, you will be notified via AWS Marketplace. To update:

  1. Log into AWS Console

  2. Navigate to AWS Marketplace → Manage Subscriptions

  3. Select ML Provisioner for your tier

  4. Follow the instructions to pull the latest image

Verify New Version

# Check which versions you have locally
docker images ml-provisioner

The TAG column shows all available version tags (e.g. enterprise-1.0.0, enterprise-1.0.1). Both old and new versions coexist in your local Docker system until you explicitly remove the old one. Set IMAGE to the version you want to run:

# Validate your existing config file against the new version
# to confirm it is still compatible before using it in production
CONFIG=globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml
IMAGE=enterprise-1.0.1  # set to the new version you just pulled

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} \
  -con ${CONFIG} -act validate-config

Test Before Production

Run a test deployment with the new image to verify compatibility:

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} \
  -con ${CONFIG} -act test-deploy

This creates an isolated test stack with a random suffix — no impact on production resources. Delete the test stack when done:

# Stack name is printed by test-deploy — replace <test-stack-name> below
aws cloudformation delete-stack --stack-name <test-stack-name> --region us-west-2
aws cloudformation wait stack-delete-complete --stack-name <test-stack-name> --region us-west-2

Rollback Procedures

Quick Rollback

If the new version (enterprise-1.0.1) causes issues, simply set IMAGE back to the previous version (enterprise-1.0.0) — it is still available locally since both versions coexist:

# Roll back by setting IMAGE to the previous version
CONFIG=globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml
IMAGE=enterprise-1.0.0

# Verify the previous version works with your config
docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} \
  -con ${CONFIG} -act validate-config

No re-tagging or AWS Marketplace contact needed — the previous version is already on your machine.

Configuration Updates

Design scope: The ML Provisioner is an initial provisioning tool. It sets up the ML infrastructure stack and hands it over. Ongoing lifecycle management of live resources — including modifications to stacks that are actively consumed by downstream services — is outside the scope of this tool. See ROADMAP.

Modifying an Existing Configuration

⚠️ Important: The ML Provisioner is an initial provisioning tool — modifying live stacks that are actively consumed is outside its design scope. show-changes can be used to assess the impact of a config change, but no changes are applied.

Key constraints

  • Config filename must not change. The stack name is derived from the config filename. Renaming the file would cause show-changes to look for a different stack.

  • create-prov-template overwrites the existing YAML template silently. Back it up first.

  • Editing the config file overwrites the original. Back it up first.

  • Same Docker image is used throughout — this is not an image update procedure.

Procedure

CONFIG=globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml
IMAGE=enterprise-1.0.0

# Derive the template filename — same ml_name prefix, ends with -template.yaml
# Example: globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-sgprov-template.yaml
# Check ml/templates/ if unsure of the exact name:
#   ls ml/templates/ | grep globalbank-prod-c001-us-west-2-demand-forecasting-ml
TEMPLATE=globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-sgprov-template.yaml

# 1. Back up config and provisioning template before making any changes
cp ml/configs/${CONFIG} ml/configs/${CONFIG}.bak
cp ml/templates/${TEMPLATE} ml/templates/${TEMPLATE}.bak

# 2. Edit the config file — keep the filename unchanged
vi ml/configs/${CONFIG}

# 3. Validate the edited config
docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act validate-config

# 4. Regenerate the provisioning template (overwrites the existing one)
docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/templates:/app/templates \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act create-prov-template

# 5. Preview what would change against the deployed stack
docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/templates:/app/templates \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act show-changes

show-changes displays the ChangeSet — additions, modifications, and replacements — then deletes it. No changes are applied to the stack.

If you want to revert to the original state:

cp ml/configs/${CONFIG}.bak ml/configs/${CONFIG}
cp ml/templates/${TEMPLATE}.bak ml/templates/${TEMPLATE}

Applying changes

Stack updates are not currently supported. For any change, the stack must be deleted and reprovisioned from scratch — see Replacing a Deployment.

Config fields and their impact

Before deciding to delete-and-redeploy, understand what each config change means for your deployed resources. Use show-changes to confirm the impact before proceeding.

Change

Impact

Consequence

Changing ml_name (any field that drives it)

🔴 Critical

All SSM paths change — consuming services break. Old SSM params orphaned. Never do this on an existing stack.

Changing KMS key settings (enterprise)

🔴 Critical

Key replacement leaves existing encrypted data inaccessible

Changing S3 bucket name

🔴 Critical

Bucket replaced — all pipeline artifacts lost

Changing source_control (codecommit ↔ s3)

🟠 High

CodeCommit repos deleted or created — pipeline source changes

Changing vpc_integration.mode

🟠 High

Security group created or deleted — VPC endpoint traffic disrupted

Changing tags

🟢 Safe

In-place update — no resource replacement

Changing alerts_email (enterprise)

🟢 Safe

SNS subscription updated in place

Changing log_retention_days (enterprise)

🟢 Safe

LogGroup retention updated in place

Infrastructure Maintenance

Drift Detection

Check if deployed resources have been modified outside of CloudFormation:

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act check-drift

If drift is detected, check-drift is informational only — the provisioner cannot remediate drift automatically. Your options are:

  • Minor drift (e.g. a tag was changed manually): fix the drifted resource directly via AWS Console or CLI to bring it back in line with the template

  • Significant drift: delete-and-redeploy using the Replacing a Deployment procedure below

Replacing a Deployment

Danger

Delete-and-redeploy is a destructive, irreversible operation. Once delete-product runs, all associated resources are permanently removed — CodeCommit repositories, CodePipeline pipelines, S3 artifacts bucket, pipeline artifacts, and SSM parameters. There is no undo.

This operation is only appropriate for stacks with no live data or active downstream dependencies. Modifying live stacks is outside the design scope of the ML Provisioner.

To delete and reprovision a stack with no active dependencies:

CONFIG=globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml
IMAGE=enterprise-1.0.0

# 1. Delete existing stack
docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act delete-product --force

# 2. Redeploy — follow the full provisioning sequence in USER_GUIDE

Monitoring After Update

Verify Deployed Infrastructure

CONFIG=globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml
IMAGE=enterprise-1.0.0
ML_NAME=globalbank-prod-c001-us-west-2-demand-forecasting-ml

# Check CloudFormation stack status
aws cloudformation describe-stacks \
  --stack-name ${ML_NAME}-stack \
  --region us-west-2 \
  --query 'Stacks[0].StackStatus'

# Check SSM parameters
aws ssm get-parameters-by-path \
  --path /ml/${ML_NAME}/ \
  --recursive \
  --region us-west-2 \
  --query 'Parameters[*].Name' \
  --output table

# Run drift detection
docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act check-drift

Review Logs

# Check recent execution logs
ls -lt ml/reports/*.log | head -10

# Check for errors in logs
grep -i error ml/reports/*.log

Update Schedule

New versions of the ML Provisioner are published to AWS Marketplace. You will be notified via AWS Marketplace when a new version is available. When to pull and deploy a new version is at your discretion based on your organization’s change management process.

See Updating the Docker Image for the update procedure.