Integration Test — globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm¶

Table of Contents¶

Integration Test — globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm

Scenario Summary¶

Field	Value
Config file	globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml
ml_name	globalbank-prod-c001-us-west-2-demand-forecasting-ml
Stack name	globalbank-prod-c001-us-west-2-demand-forecasting-ml-stack
Tier	enterprise
Source control	codecommit
VPC mode	sg-provisioner
VPC source	parameter-store
Workload	(none)
Region	us-west-2
Account	123456789012
CFN resources	39 (no endpoint SecurityGroup — managed by SG Provisioner)

1. Prerequisites¶

This scenario uses vpc_integration.mode: sg-provisioner and vpc_source: parameter-store. Before running the ML Provisioner test cycle, the following upstream provisioners must be deployed and their outputs must be present in SSM Parameter Store.

1.1 VPC Provisioner¶

The VPC Provisioner must be deployed for globalbank-prod-c001-us-west-2. It publishes the VPC ID and private subnet IDs to SSM Parameter Store, which the ML Provisioner reads at deploy time via the CloudFormation parameter resolution mechanism.

Deploy command:

docker run --rm \
  -v ~/.aws:/home/vpcuser/.aws:ro \
  -v $(pwd)/vpc/configs:/app/configs:ro \
  -v $(pwd)/vpc/reports:/app/reports \
  -v $(pwd)/vpc/templates:/app/templates \
  vpc-provisioner:latest \
  --config globalbank-prod-c001-us-west-2-vpc.yaml \
  --action create-vpc \
  --force

SSM paths required by this scenario:

/vpc/globalbank-prod-c001-us-west-2-vpc/VPCId
/vpc/globalbank-prod-c001-us-west-2-vpc/PrivateSubnetIds

Verify VPC SSM outputs are present:

aws ssm get-parameters-by-path \
  --path /vpc/globalbank-prod-c001-us-west-2-vpc \
  --recursive \
  --region us-west-2 \
  --query 'Parameters[*].[Name,Value]' \
  --output table

Results:

--------------------------------------------------------------------------------------------
|                                                            GetParametersByPath           |
+------------------------------------------------------------+-----------------------------+
|  /vpc/globalbank-prod-c001-us-west-2-vpc/DatabaseSubnet1Id |  subnet-09320fac506c4d48b   |
|  /vpc/globalbank-prod-c001-us-west-2-vpc/DatabaseSubnet2Id |  subnet-03697eea9405a7fb8   |
|  /vpc/globalbank-prod-c001-us-west-2-vpc/PrivateSubnet1Id  |  subnet-03ef04e345a08b008   |
|  /vpc/globalbank-prod-c001-us-west-2-vpc/PrivateSubnet2Id  |  subnet-059d4326e512a5a33   |
|  /vpc/globalbank-prod-c001-us-west-2-vpc/PublicSubnet1Id   |  subnet-096d98235628484b5   |
|  /vpc/globalbank-prod-c001-us-west-2-vpc/PublicSubnet2Id   |  subnet-052a773b05ee88b95   |
|  /vpc/globalbank-prod-c001-us-west-2-vpc/PublicSubnet3Id   |  subnet-01e2faec0be5b56cd   |
|  /vpc/globalbank-prod-c001-us-west-2-vpc/VPCCidr           |  10.3.0.0/16                |
|  /vpc/globalbank-prod-c001-us-west-2-vpc/VPCId             |  vpc-01d82c12b5b84da89      |
|  /vpc/globalbank-prod-c001-us-west-2-vpc/PrivateSubnet3Id  |  subnet-038b75b317519b7d4   |
+------------------------------------------------------------+-----------------------------+

ML Provisioner config expects combined PrivateSubnetIds parameter.

Let’s build this parameter:

aws ssm put-parameter \
  --name /vpc/globalbank-prod-c001-us-west-2-vpc/PrivateSubnetIds \
  --value "subnet-03ef04e345a08b008,subnet-059d4326e512a5a33,subnet-038b75b317519b7d4" \
  --type StringList \
  --region us-west-2

Let’s check that the parameter exists:

aws ssm get-parameter \
  --name /vpc/globalbank-prod-c001-us-west-2-vpc/PrivateSubnetIds \
  --region us-west-2 \
  --query Parameter.Value \
  --output text

Output:

subnet-03ef04e345a08b008,subnet-059d4326e512a5a33,subnet-038b75b317519b7d4

1.2 SG Provisioner¶

The SG Provisioner must be deployed for globalbank-prod-c001-us-west-2. It creates and manages the endpoint Security Group and publishes its ID to SSM Parameter Store. The ML Provisioner reads this ID and attaches it to the VPC endpoints — it does not create its own Security Group in this mode.

Rebuild common and sg-provisioner:

make common
make sg

Regenerate the template from the 3-tier config:

docker run --rm \
  -v ~/.aws:/home/sguser/.aws:ro \
  -v $(pwd)/sg/configs:/app/configs:ro \
  -v $(pwd)/sg/templates:/app/templates \
  -v $(pwd)/sg/reports:/app/reports \
  sg-provisioner:latest \
  -con globalbank-prod-c001-us-west-2-sg-ssm-3tier.yaml \
  -act create-prov-template

Deploy SG:

docker run --rm \
  -v ~/.aws:/home/sguser/.aws:ro \
  -v $(pwd)/sg/configs:/app/configs:ro \
  -v $(pwd)/sg/templates:/app/templates \
  -v $(pwd)/sg/reports:/app/reports \
  sg-provisioner:latest \
  -con globalbank-prod-c001-us-west-2-sg-ssm-3tier.yaml \
  -act create-security-groups \
  --force

Verify that security groups were created

aws ssm get-parameters-by-path \
  --path /sg/globalbank-prod-c001-us-west-2-sg \
  --recursive \
  --region us-west-2 \
  --query 'Parameters[*].[Name,Value]' \
  --output table

Output:

------------------------------------------------------------------------------------------
|                                   GetParametersByPath                                  |
+---------------------------------------------------------------+------------------------+
|  /sg/globalbank-prod-c001-us-west-2-sg/AppSecurityGroupOutput |  sg-09468cab28bcf1020  |
|  /sg/globalbank-prod-c001-us-west-2-sg/DbSecurityGroupOutput  |  sg-0ad5e7a9bcb202a2c  |
|  /sg/globalbank-prod-c001-us-west-2-sg/WebSecurityGroupOutput |  sg-0e340fa9323e1c080  |
|  /sg/globalbank-prod-c001-us-west-2-sg/app/SecurityGroupId    |  sg-09468cab28bcf1020  |
|  /sg/globalbank-prod-c001-us-west-2-sg/db/SecurityGroupId     |  sg-0ad5e7a9bcb202a2c  |
|  /sg/globalbank-prod-c001-us-west-2-sg/web/SecurityGroupId    |  sg-0e340fa9323e1c080  |
+---------------------------------------------------------------+------------------------+

NOTE:

There are two separate mechanisms writing SSM parameters, each serving a different consumer:

/sg/.../web/SecurityGroupId — written by CloudFormation directly (via AWS::SSM::Parameter resource in the generated template). This is the path consumed by the ML Provisioner (sg_parameter_store_path in the enterprise config points here) and other downstream provisioners that need to reference the SG by tier name.

/sg/.../WebSecurityGroupOutput — written by _store_sg_parameters in sg_manager.py after stack creation, from the CloudFormation stack outputs. This was likely added for operational visibility — a flat list of all SG IDs under the stack’s SSM namespace, mirroring the CloudFormation outputs pattern used consistently across all provisioners in the suite.

In short — the /tier/SecurityGroupId paths are for machine consumption by downstream provisioners, and the *Output paths are for operational reference and consistency with the suite’s SSM output pattern. Whether both are strictly necessary is worth revisiting — the *Output paths are redundant if the /tier/SecurityGroupId paths already serve all consumers.

SSM path required by this scenario:

/sg/globalbank-prod-c001-us-west-2-sg/app/SecurityGroupId

Verify SG SSM output is present:

aws ssm get-parameter \
  --name /sg/globalbank-prod-c001-us-west-2-sg/app/SecurityGroupId  \
  --region us-west-2 \
  --query Parameter.Value \
  --output text

Output:

sg-09468cab28bcf1020

2. Artifacts¶

The following local artifacts are created during the test cycle. All paths are relative to the ml-provisioner-tool package root. Timestamps follow the pattern YYYYMMDD_HHMMSS_mmm.

2.1 Log Files — reports/¶

One log file is created per action run.

Action	File pattern
`validate-config`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-validate-config-{timestamp}.log`
`list-products`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-list-products-{timestamp}.log`
`show-product`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-show-product-{timestamp}.log`
`create-policy`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-create-policy-{timestamp}.log`
`create-prov-template`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-create-prov-template-{timestamp}.log`
`validate-prov-template`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-validate-prov-template-{timestamp}.log`
`create-review-report`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-create-review-report-{timestamp}.log`
`deploy-product`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-deploy-product-{timestamp}.log`
`show-changes`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-show-changes-{timestamp}.log`
`check-drift`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-check-drift-{timestamp}.log`
`delete-product`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-delete-product-{timestamp}.log`

2.2 HTML Files — reports/¶

Action	File pattern
`create-review-report`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-create-review-report-{timestamp}.html`
`deploy-product`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-deployment-{timestamp}.html`

2.3 JSON Files — policies/¶

Action	File
`create-policy`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-iam-policy.json`

2.4 YAML Files — templates/¶

Action	File
`create-prov-template`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-sgprov-template.yaml`

3. Resources¶

All 39 AWS resources provisioned by this scenario. No endpoint SecurityGroup is created because vpc_integration.mode: sg-provisioner — the SG Provisioner manages it.

3.1 SageMaker (x1)¶

Resource type	Physical name
`AWS::SageMaker::ModelPackageGroup`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-models`

3.2 CodeCommit (x2)¶

Resource type	Physical name
`AWS::CodeCommit::Repository`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-model-build`
`AWS::CodeCommit::Repository`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-model-deploy`

3.3 CodeBuild (x2)¶

Resource type	Physical name
`AWS::CodeBuild::Project`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-build`
`AWS::CodeBuild::Project`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-deploy`

3.4 CodePipeline (x2)¶

Resource type	Physical name
`AWS::CodePipeline::Pipeline`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-build-pipeline`
`AWS::CodePipeline::Pipeline`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-deploy-pipeline`

3.5 S3 (x1)¶

Resource type	Physical name
`AWS::S3::Bucket`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-artifacts`

3.6 IAM Roles (x3)¶

Resource type	Physical name
`AWS::IAM::Role`	`globalbank-prod-c001-demand-forecasting-ml-codebuild-role`
`AWS::IAM::Role`	`globalbank-prod-c001-demand-forecasting-ml-pipeline-role`
`AWS::IAM::Role`	`globalbank-prod-c001-demand-forecasting-ml-sm-exec-role`

3.7 IAM Managed Policies (x3)¶

Resource type	Physical name
`AWS::IAM::ManagedPolicy`	`globalbank-prod-c001-demand-forecasting-ml-build-policy`
`AWS::IAM::ManagedPolicy`	`globalbank-prod-c001-demand-forecasting-ml-deploy-policy`
`AWS::IAM::ManagedPolicy`	`globalbank-prod-c001-demand-forecasting-ml-perm-boundary`

3.8 EventBridge (x1)¶

Resource type	Physical name
`AWS::Events::Rule`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-event-rule`

3.9 CloudWatch (x3)¶

Resource type	Physical name
`AWS::CloudWatch::Dashboard`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-dashboard`
`AWS::CloudWatch::Alarm`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-unauthorized-api-calls`
`AWS::CloudWatch::Alarm`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-root-account-usage`

3.10 KMS (x2)¶

Resource type	Physical name / Alias
`AWS::KMS::Key`	(no physical name — referenced via alias)
`AWS::KMS::Alias`	`alias/globalbank-prod-c001-us-west-2-demand-forecasting-ml-key`

3.11 CloudWatch Logs (x1) + Metric Filters (x2)¶

Resource type	Physical name
`AWS::Logs::LogGroup`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-compliance-logs`
`AWS::Logs::MetricFilter`	(filter for unauthorized-api-calls alarm — no physical name)
`AWS::Logs::MetricFilter`	(filter for root-account-usage alarm — no physical name)

3.12 SNS (x2)¶

Resource type	Physical name
`AWS::SNS::Topic`	`globalbank-prod-c001-us-west-2-demand-forecasting-ml-security-alerts`
`AWS::SNS::Subscription`	(no physical name — linked to topic above, endpoint: ml-alerts@globalbank.com)

3.13 VPC Endpoints (x4)¶

Resource type	Service	Type
`AWS::EC2::VPCEndpoint`	`sagemaker.api`	Interface
`AWS::EC2::VPCEndpoint`	`sagemaker.runtime`	Interface
`AWS::EC2::VPCEndpoint`	`s3`	Gateway
`AWS::EC2::VPCEndpoint`	`sts`	Interface

3.14 SSM Parameters (x10 — all CFN-managed)¶

Parameter path
`/ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/ModelPackageGroupArn`
`/ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/RepositoryUrl`
`/ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/BucketName`
`/ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/DashboardName`
`/ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/KmsKeyArn`
`/ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/LogGroupName`
`/ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdSagemakerApi`
`/ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdSagemakerRuntime`
`/ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdS3`
`/ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdSts`

Note: SecurityGroupId is NOT written to SSM in sg-provisioner mode.

4. Test Steps¶

All commands run from the monorepo root (~/mlops-infra-suite). The config file is globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml. The image tag is ml-provisioner:enterprise.

4.1 validate-config¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act validate-config

4.2 list-products¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act list-products

4.3 show-product¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act show-product

4.4 create-policy¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/policies:/app/policies \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act create-policy

4.5 create-prov-template¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/templates:/app/templates \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act create-prov-template

4.6 validate-prov-template¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/templates:/app/templates \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act validate-prov-template

4.7 create-review-report¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/templates:/app/templates \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act create-review-report

4.8 deploy-product¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/templates:/app/templates \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act deploy-product --force

4.9 show-changes¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/templates:/app/templates \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act show-changes

4.10 check-drift¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act check-drift

4.11 test-deploy¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act test-deploy

4.12 delete-product¶

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:enterprise \
  -con globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml \
  -act delete-product --force

5. Verification¶

AWS CLI commands to verify all resources were provisioned. All commands target us-west-2.

5.1 List all provisioned resources¶

aws cloudformation describe-stack-resources \
  --stack-name globalbank-prod-c001-us-west-2-demand-forecasting-ml-stack \
  --region us-west-2 \
  --query "StackResources[*].[ResourceType,PhysicalResourceId]" \
  --output table

5.2 CloudFormation Stack¶

aws cloudformation describe-stacks \
  --stack-name globalbank-prod-c001-us-west-2-demand-forecasting-ml-stack \
  --region us-west-2 \
  --query "Stacks[0].StackStatus" --output text
  
# Expected: CREATE_COMPLETE

5.3 SageMaker Model Package Group¶

aws sagemaker describe-model-package-group \
  --model-package-group-name globalbank-prod-c001-us-west-2-demand-forecasting-ml-models \
  --region us-west-2 \
  --query "ModelPackageGroupStatus" \
  --output text

# Expected: Completed

5.4 CodeCommit Repositories¶

aws codecommit list-repositories \
  --region us-west-2 \
  --query "repositories[?contains(repositoryName,'demand-forecasting')].repositoryName" \
  --output table

output:

-----------------------------------------------------------------------
|                          ListRepositories                           |
+---------------------------------------------------------------------+
|  globalbank-prod-c001-us-west-2-demand-forecasting-ml-model-build   |
|  globalbank-prod-c001-us-west-2-demand-forecasting-ml-model-deploy  |
+---------------------------------------------------------------------+

aws codecommit get-repository \
  --repository-name globalbank-prod-c001-us-west-2-demand-forecasting-ml-model-build \
  --region us-west-2 \
  --query "repositoryMetadata.repositoryName" \
  --output text

aws codecommit get-repository \
  --repository-name globalbank-prod-c001-us-west-2-demand-forecasting-ml-model-deploy \
  --region us-west-2 \
  --query "repositoryMetadata.repositoryName" \
  --output text
  
# Expected: repository name returned for each

5.5 CodeBuild Projects¶

aws codebuild batch-get-projects \
  --names \
    globalbank-prod-c001-us-west-2-demand-forecasting-ml-build \
    globalbank-prod-c001-us-west-2-demand-forecasting-ml-deploy \
  --region us-west-2 \
  --query "projects[*].name" \
  --output table

# Expected: both project names returned

output:

-----------------------------------------------------------------
|                       BatchGetProjects                        |
+---------------------------------------------------------------+
|  globalbank-prod-c001-us-west-2-demand-forecasting-ml-build   |
|  globalbank-prod-c001-us-west-2-demand-forecasting-ml-deploy  |
+---------------------------------------------------------------+

5.6 CodePipeline Pipelines¶

aws codepipeline list-pipelines \
  --region us-west-2 \
  --query "pipelines[?contains(name,'demand-forecasting')].name" \
  --output table

output:

--------------------------------------------------------------------------
|                              ListPipelines                             |
+------------------------------------------------------------------------+
|  globalbank-prod-c001-us-west-2-demand-forecasting-ml-build-pipeline   |
|  globalbank-prod-c001-us-west-2-demand-forecasting-ml-deploy-pipeline  |
+------------------------------------------------------------------------+

aws codepipeline get-pipeline \
  --name globalbank-prod-c001-us-west-2-demand-forecasting-ml-build-pipeline \
  --region us-west-2 \
  --query "pipeline.name" \
  --output text

aws codepipeline get-pipeline \
  --name globalbank-prod-c001-us-west-2-demand-forecasting-ml-deploy-pipeline \
  --region us-west-2 \
  --query "pipeline.name" \
  --output text

# Expected: pipeline name returned for each

5.7 S3 Artifacts Bucket¶

aws s3api head-bucket \
  --bucket globalbank-prod-c001-us-west-2-demand-forecasting-ml-artifacts \
  --region us-west-2
  
# Expected: exit code 0 (no error)

aws s3api get-bucket-versioning \
  --bucket globalbank-prod-c001-us-west-2-demand-forecasting-ml-artifacts \
  --region us-west-2 \
  --query "Status" \
  --output text

# Expected: Enabled

5.8 IAM Roles¶

aws iam get-role \
  --role-name globalbank-prod-c001-demand-forecasting-ml-codebuild-role \
  --query "Role.RoleName" \
  --output text

aws iam get-role \
  --role-name globalbank-prod-c001-demand-forecasting-ml-pipeline-role \
  --query "Role.RoleName" \
  --output text

aws iam get-role \
  --role-name globalbank-prod-c001-demand-forecasting-ml-sm-exec-role \
  --query "Role.RoleName" \
  --output text

# Expected: role name returned for each

5.9 IAM Managed Policies¶

aws iam get-policy \
  --policy-arn arn:aws:iam::123456789012:policy/globalbank-prod-c001-demand-forecasting-ml-build-policy \
  --query "Policy.PolicyName" --output text

aws iam get-policy \
  --policy-arn arn:aws:iam::123456789012:policy/globalbank-prod-c001-demand-forecasting-ml-deploy-policy \
  --query "Policy.PolicyName" --output text

aws iam get-policy \
  --policy-arn arn:aws:iam::123456789012:policy/globalbank-prod-c001-demand-forecasting-ml-perm-boundary \
  --query "Policy.PolicyName" --output text

# Expected: policy name returned for each

5.10 EventBridge Rule¶

aws events describe-rule \
  --name globalbank-prod-c001-us-west-2-demand-forecasting-ml-event-rule \
  --region us-west-2 \
  --query "State" \
  --output text

# Expected: ENABLED

5.11 CloudWatch Dashboard¶

aws cloudwatch get-dashboard \
  --dashboard-name globalbank-prod-c001-us-west-2-demand-forecasting-ml-dashboard \
  --region us-west-2 \
  --query "DashboardName" \
  --output text

# Expected: dashboard name returned

5.12 CloudWatch Alarms¶

aws cloudwatch describe-alarms \
  --alarm-names \
    globalbank-prod-c001-us-west-2-demand-forecasting-ml-unauthorized-api-calls \
    globalbank-prod-c001-us-west-2-demand-forecasting-ml-root-account-usage \
  --region us-west-2 \
  --query "MetricAlarms[*].AlarmName" \
  --output text

# Expected: both alarm names returned

5.13 KMS Key and Alias¶

aws kms describe-key \
  --key-id alias/globalbank-prod-c001-us-west-2-demand-forecasting-ml-key \
  --region us-west-2 \
  --query "KeyMetadata.KeyState" \
  --output text

# Expected: Enabled

aws kms list-aliases \
  --region us-west-2 \
  --query "Aliases[?AliasName=='alias/globalbank-prod-c001-us-west-2-demand-forecasting-ml-key'].AliasName" \
  --output text

# Expected: alias name returned

5.14 CloudWatch Log Group¶

aws logs describe-log-groups \
  --log-group-name-prefix globalbank-prod-c001-us-west-2-demand-forecasting-ml-compliance-logs \
  --region us-west-2 \
  --query "logGroups[0].logGroupName" \
  --output text

# Expected: log group name returned

aws logs describe-log-groups \
  --log-group-name-prefix globalbank-prod-c001-us-west-2-demand-forecasting-ml-compliance-logs \
  --region us-west-2 \
  --query "logGroups[0].retentionInDays" \
  --output text

# Expected: 365

5.16 VPC Endpoints¶

aws ec2 describe-vpc-endpoints \
  --region us-west-2 \
  --filters "Name=tag:Name,Values=*globalbank-prod-c001-us-west-2-demand-forecasting-ml*" \
  --query "VpcEndpoints[*].{Service:ServiceName,State:State,Type:VpcEndpointType}" \
  --output table

# Expected: 4 endpoints — sagemaker.api (Interface), sagemaker.runtime (Interface),
#           s3 (Gateway), sts (Interface) — all in state 'available'

output:

-------------------------------------------------------------------------
|                         DescribeVpcEndpoints                          |
+--------------------------------------------+------------+-------------+
|                   Service                  |   State    |    Type     |
+--------------------------------------------+------------+-------------+
|  com.amazonaws.us-west-2.s3                |  available |  Gateway    |
|  com.amazonaws.us-west-2.sagemaker.api     |  available |  Interface  |
|  com.amazonaws.us-west-2.sts               |  available |  Interface  |
|  com.amazonaws.us-west-2.sagemaker.runtime |  available |  Interface  |
+--------------------------------------------+------------+-------------+

5.17 SSM Parameters¶

aws ssm get-parameters-by-path \
  --path /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/ \
  --recursive \
  --region us-west-2 \
  --query "Parameters[*].Name" \
  --output table

# Expected: 10 parameters returned — all managed as AWS::SSM::Parameter CFN resources

For example:

--------------------------------------------------------------------------------------------
|                                    GetParametersByPath                                   |
+------------------------------------------------------------------------------------------+
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/BucketName                     |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/DashboardName                  |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/KmsKeyArn                      |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/LogGroupName                   |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/ModelPackageGroupArn           |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/RepositoryUrl                  |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdS3                |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdSagemakerApi      |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdSagemakerRuntime  |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdSts               |
+------------------------------------------------------------------------------------------+

5.18 Verify SecurityGroupId is NOT in SSM (sg-provisioner mode)¶

aws ssm get-parameter \
  --name /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/SecurityGroupId \
  --region us-west-2 2>&1 | grep -c "ParameterNotFound"
  
# Expected: 1 (parameter must not exist in sg-provisioner mode)

Integration Test — globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm¶

Table of Contents¶

Scenario Summary¶

1. Prerequisites¶

1.1 VPC Provisioner¶

1.2 SG Provisioner¶

2. Artifacts¶

2.1 Log Files — reports/¶

2.2 HTML Files — reports/¶

2.3 JSON Files — policies/¶

2.4 YAML Files — templates/¶

3. Resources¶

3.1 SageMaker (x1)¶

3.2 CodeCommit (x2)¶

3.3 CodeBuild (x2)¶

3.4 CodePipeline (x2)¶

3.5 S3 (x1)¶

3.6 IAM Roles (x3)¶

3.7 IAM Managed Policies (x3)¶

3.8 EventBridge (x1)¶

3.9 CloudWatch (x3)¶

3.10 KMS (x2)¶

3.11 CloudWatch Logs (x1) + Metric Filters (x2)¶

3.12 SNS (x2)¶

3.13 VPC Endpoints (x4)¶

3.14 SSM Parameters (x10 — all CFN-managed)¶

4. Test Steps¶

4.1 validate-config¶

4.2 list-products¶

4.3 show-product¶

4.4 create-policy¶

4.5 create-prov-template¶

4.6 validate-prov-template¶

4.7 create-review-report¶

4.8 deploy-product¶

4.9 show-changes¶

4.10 check-drift¶

4.11 test-deploy¶

4.12 delete-product¶

5. Verification¶

5.1 List all provisioned resources¶

5.2 CloudFormation Stack¶

5.3 SageMaker Model Package Group¶

5.4 CodeCommit Repositories¶

5.5 CodeBuild Projects¶

5.6 CodePipeline Pipelines¶

5.7 S3 Artifacts Bucket¶

5.8 IAM Roles¶

5.9 IAM Managed Policies¶

5.10 EventBridge Rule¶

5.11 CloudWatch Dashboard¶

5.12 CloudWatch Alarms¶

5.13 KMS Key and Alias¶

5.14 CloudWatch Log Group¶

5.15 SNS Topic¶

5.16 VPC Endpoints¶

5.17 SSM Parameters¶

5.18 Verify SecurityGroupId is NOT in SSM (sg-provisioner mode)¶