Security Guidelines¶

Security best practices for deploying and operating the ML Provisioner.

Table of Contents¶

Credential Management
IAM Least Privilege
KMS Encryption (Enterprise Tier)
VPC Endpoint Security (Enterprise Tier)
Permission Boundaries (Enterprise Tier)
Compliance Monitoring (Enterprise Tier)
CloudFormation Security
Docker Security
Logging and Monitoring
Security Checklist
References

Credential Management¶

AWS Credentials¶

Never hardcode credentials. Use one of these methods:

1. AWS IAM Roles (Recommended for EC2/ECS)¶

If you are running your Docker containers on AWS services like EC2, ECS, or EKS:

Assign an IAM Instance Profile or ECS Task Role directly to the infrastructure.
The AWS SDK inside your container will automatically detect and fetch temporary credentials securely.

# No credentials needed — automatic from instance metadata
docker run --rm \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:starter \
  -con my-config.yaml \
  -act validate-config

This method leverages the host machine’s IAM instance metadata instead of passing secret keys.

It mounts your local configuration folder as read-only (:ro) to protect your source files.
It mounts a local reports folder with read-write access to save the output.
The --rm flag ensures the container is deleted immediately after execution, keeping your host system clean.
Using $(pwd) requires you to run this exact command from the root directory of your project, or it will fail to find the files.
The :starter tag is marked with the specific version tag (e.g., :1.2.3) to ensure consistent behavior.

2. AWS Profiles (Recommended for Developers)¶

Instead of passing the strings, mount your local, pre-configured AWS directory into the container as a read-only volume.

# Mount credentials read-only
docker run -v ~/.aws:/home/mluser/.aws:ro my-image

# Set profile
export AWS_PROFILE=production

Why This Configuration Is Excellent

Prevents Credential Theft: The :ro flag ensures the container cannot overwrite, delete, or inject malicious data into your local .aws directory.
Mitigates Container Compromise: If an attacker compromises your application code, they still cannot modify your host credentials.
Simplifies Local Development: It gives your container immediate access to AWS services without hardcoding sensitive access keys inside the Docker image.
Enforces Scope Isolation: Explicitly setting AWS_PROFILE=production ensures the container targets the correct environment instead of defaulting to your primary local profile.

Important Technical Considerations

Path Matching: Ensure the container user mluser actually has a home directory at /home/mluser. If the path is wrong, the AWS SDK inside the container will look in the wrong place and fail.
File Permissions: The files inside your host ~/.aws directory must have read permissions allowed for the specific User ID (UID) running inside the container.
Host Overreliance: While great for local testing, do not use this volume-mounting approach in actual cloud production. Use IAM Roles (like ECS Task Roles or EKS Service Accounts) instead.

3. Environment Variables¶

-e AWS_ACCESS_KEY_ID=<access_key> \
-e AWS_SECRET_ACCESS_KEY=<secret_key> \
-e AWS_DEFAULT_REGION=us-west-2

Passing AWS credentials directly into a Docker container using environment flags (-e) poses a severe security risk because these credentials will expose your cloud infrastructure to anyone with access to the host machine.

Why This Is Dangerous

Process Visibility: Anyone running ps aux on the host system can view your plaintext secrets.
Docker Inspection: The credentials remain permanently baked into the container configuration and can be read by anyone running docker inspect <container_id>.
Command History: Your secret keys will be saved in plaintext inside your shell’s history file (e.g., ~/.bash_history).

4. Use an Environment File¶

If you must use environment variables locally, save them to a file excluded from version control (add to .gitignore) and reference it securely:

docker run --env-file .env ml-provisioner:${IMAGE}  # e.g. IMAGE=starter

Security Practices¶

Use temporary credentials (STS AssumeRole) when possible — temporary credentials automatically expire and cannot be reused if leaked
Prefer IAM roles with short-lived session tokens over long-lived access keys — use IAM Identity Center (formerly AWS SSO) for CLI access instead of permanent .aws/credentials keys
Rotate access keys every 90 days — automate using AWS Secrets Manager or CI/CD pipeline scripts
Monitor credential age with IAM Credential Report — use AWS Config rules (access-keys-rotated) combined with EventBridge to alert or disable keys that exceed 90 days

IAM Least Privilege¶

Use the Generated Policy¶

Always use create-policy to generate a scoped policy rather than granting broad permissions:

CONFIG=globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml
IMAGE=enterprise

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/policies:/app/policies \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} \
  -con ${CONFIG} \
  -act create-policy

The generated policy is scoped to your specific ml_name — it cannot create, modify, or delete resources outside your naming scope.

What is ml_name? It is the unique identifier for your ML product, auto-generated from your config file values. Examples:

techcorp-prod-a001-us-west-2-customer-churn-ml (starter, no workload)

edge-prod-b001-us-west-2-fraud-detection-realtime-ml (professional, with workload)

globalbank-prod-c001-us-west-2-demand-forecasting-ml (enterprise)

See NAMING_CONVENTIONS.md for the full construction rules.

Separate Roles by Environment¶

Use distinct IAM roles per environment:

Environment	Role	Permissions
dev	`ml-provisioner-dev-role`	Full deploy/delete
staging	`ml-provisioner-staging-role`	Deploy only (no delete)
prod	`ml-provisioner-prod-role`	Deploy with MFA requirement

MFA for Production¶

Add MFA condition for production deployment actions:

{
  "Condition": {
    "Bool": {
      "aws:MultiFactorAuthPresent": "true"
    }
  }
}

KMS Encryption (Enterprise Tier)¶

Enterprise tier creates a Customer Managed Key (CMK) for encrypting:

ML artifacts S3 bucket (via BucketEncryption)
CodePipeline artifact store (via artifactStore.encryptionKey)

Key Rotation¶

Enable automatic key rotation (enabled by default in the generated template):

EnableKeyRotation: true

Key Access Control¶

The KMS key policy is scoped to:

The CodeBuild role — for artifact encryption/decryption
The CodePipeline role — for pipeline artifact store
The SageMaker execution role — for model artifact access

Never grant kms:* to broad principals. Review key policy after deployment.

Key ARN in SSM¶

The KMS key ARN is published to SSM at:

/ml/{ml_name}/KmsKeyArn

This path is consumed by the SageMaker Provisioner to encrypt Studio EBS volumes.

Example:

aws ssm get-parameters-by-path \
  --path /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/ \
  --region us-west-2 \
  --query 'Parameters[?contains(Name, `Kms`)].{Name:Name,Value:Value}' \
  --output table

Result:

------------------------------------------------------------------------------------------------------------------------------------------------------
|                                                                 GetParametersByPath                                                                 |
+---------------------------------------------------------------------+-------------------------------------------------------------------------------+
|                                Name                                 |                                     Value                                     |
+---------------------------------------------------------------------+-------------------------------------------------------------------------------+
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/KmsKeyArn |  arn:aws:kms:us-west-2:123456789012:key/95997aeb-aad2-466a-adec-852e51609002  |
+---------------------------------------------------------------------+-------------------------------------------------------------------------------+

VPC Endpoint Security (Enterprise Tier)¶

Enterprise tier creates four VPC endpoints to keep ML traffic off the public internet:

com.amazonaws.{region}.sagemaker.api — Interface type
com.amazonaws.{region}.sagemaker.runtime — Interface type
com.amazonaws.{region}.s3 — Gateway type
com.amazonaws.{region}.sts — Interface type

Interface vs Gateway endpoints:

	Interface	Gateway
How it works	Creates an ENI with a private IP in your subnet	Injects a route entry into your route table
Cost	~$0.01/hour per AZ	Free
Requires security group	Yes	No
Requires route table association	No	Yes

This is why the S3 endpoint behaves differently from the other three — it requires route table associations to function, which is the purpose of the route_table_ids config field. See Prerequisites for setup details.

Security Group for Endpoints¶

In standalone mode, the ML Provisioner creates a dedicated AWS::EC2::SecurityGroup for the VPC endpoints. The security group controls inbound traffic to the endpoints.

In sg-provisioner mode, the existing SG Provisioner security group is reused — no new security group is created.

Endpoint Policy¶

VPC endpoints accept all traffic by default. For stricter control, add endpoint policies to restrict which principals and actions can use each endpoint. This is not currently done by the ML Provisioner — it is the responsibility of the security team.

Verify Endpoints are Active¶

After deployment, verify all 4 endpoints are in available state:

aws ssm get-parameters-by-path \
  --path /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/ \
  --region us-west-2 \
  --query 'Parameters[?contains(Name, `VpcEndpoint`)].{Name:Name,Value:Value}' \
  --output table

Example result:

---------------------------------------------------------------------------------------------------------------------
|                                                 GetParametersByPath                                                |
+------------------------------------------------------------------------------------------+-------------------------+
|                                           Name                                           |          Value          |
+------------------------------------------------------------------------------------------+-------------------------+
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdS3                |  vpce-0ba9dc4a4d68f4059 |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdSagemakerApi      |  vpce-04550e929a6afeb4d |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdSagemakerRuntime  |  vpce-00875dbd429b19c5f |
|  /ml/globalbank-prod-c001-us-west-2-demand-forecasting-ml/VpcEndpointIdSts               |  vpce-01ebcd15b74afab98 |
+------------------------------------------------------------------------------------------+-------------------------+

Permission Boundaries (Enterprise Tier)¶

Enterprise tier creates an AWS::IAM::ManagedPolicy as a permission boundary applied to all IAM roles provisioned in the stack.

Purpose¶

Permission boundaries limit the maximum permissions a role can have, even if its role policy grants broader access. This prevents privilege escalation — a compromised role cannot grant itself permissions beyond what the boundary allows.

Boundary Scope¶

The permission boundary is scoped to the ML product’s own resources — the provisioned roles cannot access resources outside the ml_name naming scope.

Review Before Production¶

Review the generated permission boundary policy before deploying to production:

For example:

cat ml/policies/globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-ssm-iam-policy.json

Compliance Monitoring (Enterprise Tier)¶

Enterprise tier provisions a compliance monitoring stack:

CloudWatch Log Group — captures CloudTrail events for the ML stack
Metric Filters — two filters detecting security violations:
- Unauthorized API calls
- Root account usage
CloudWatch Alarms — triggered when violations are detected
SNS Topic + Subscription — delivers alerts to alerts_email

Log Retention¶

Set log_retention_days in your config according to your compliance requirements:

Framework	Minimum Retention
General	90 days (minimum enforced by schema)
PCI-DSS / SOC2	365 days
HIPAA	2190 days (6 years)

Alert Email¶

Confirm the SNS subscription after first deployment — AWS sends a confirmation email to alerts_email. Alerts will not be delivered until the subscription is confirmed.

Verify Compliance Infrastructure¶

# Scenario — globalbank codecommit sgprov ssm
ML_NAME=globalbank-prod-c001-us-west-2-demand-forecasting-ml

echo "=== 1. CloudWatch Log Group ==="
aws logs describe-log-groups \
  --log-group-name-prefix "${ML_NAME}-compliance-logs" \
  --query "logGroups[*].logGroupName" \
  --output text

echo "=== 2. Metric Filters ==="
aws logs describe-metric-filters \
  --log-group-name "${ML_NAME}-compliance-logs" \
  --query "metricFilters[*].[filterName, metricTransformations[0].metricName]" \
  --output table

echo "=== 3. CloudWatch Alarms ==="
aws cloudwatch describe-alarms \
  --alarm-name-prefix "${ML_NAME}" \
  --query "MetricAlarms[*].[AlarmName, StateValue]" \
  --output table

echo "=== 4. SNS Topic and Subscription ==="
TOPIC_ARN=$(aws sns list-topics \
  --query "Topics[?ends_with(TopicArn, '${ML_NAME}-security-alerts')].TopicArn" \
  --output text)
echo "Topic ARN: $TOPIC_ARN"
aws sns list-subscriptions-by-topic \
  --topic-arn "$TOPIC_ARN" \
  --query "Subscriptions[*].[Protocol, Endpoint, SubscriptionArn]" \
  --output table

Example result:

=== 1. CloudWatch Log Group ===
globalbank-prod-c001-us-west-2-demand-forecasting-ml-compliance-logs
=== 2. Metric Filters ===
--------------------------------------------------------------------------------------------------------------
|                                            DescribeMetricFilters                                           |
+------------------------------------------------------------------------------------+-----------------------+
|  GlobalbankDemandForecastingSecurityAlarmsRootAccountUsageFilter-Xphu7jGUwkAt      |  RootAccountUsage     |
|  GlobalbankDemandForecastingSecurityAlarmsUnauthorizedApiCallsFilter-WXEkgJBKHTKn  |  UnauthorizedAPICalls |
+------------------------------------------------------------------------------------+-----------------------+
=== 3. CloudWatch Alarms ===
---------------------------------------------------------------------------------------
|                                   DescribeAlarms                                    |
+-------------------------------------------------------------------------------+-----+
|  globalbank-prod-c001-us-west-2-demand-forecasting-ml-root-account-usage      |  OK |
|  globalbank-prod-c001-us-west-2-demand-forecasting-ml-unauthorized-api-calls  |  OK |
+-------------------------------------------------------------------------------+-----+
=== 4. SNS Topic and Subscription ===
Topic ARN: arn:aws:sns:us-west-2:123456789012:globalbank-prod-c001-us-west-2-demand-forecasting-ml-security-alerts
--------------------------------------------------------------
|                  ListSubscriptionsByTopic                  |
+-------+----------------------------+-----------------------+
|  email|  ml-alerts@globalbank.com  |  PendingConfirmation  |
+-------+----------------------------+-----------------------+

CloudFormation Security¶

Review Before Deployment¶

Always validate and review before deploying:

CONFIG=globalbank-prod-c001-us-west-2-demand-forecasting-ml-codecommit-sgprov-ssm.yaml
IMAGE=enterprise

# Validate template locally
docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/templates:/app/templates \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act validate-prov-template

# Generate review report
docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/templates:/app/templates \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act create-review-report

# Preview changes against deployed stack
docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/templates:/app/templates \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act show-changes

Drift Detection¶

Run drift detection regularly to catch manual changes:

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act check-drift

If drift is detected:

Identify who made the change (CloudTrail)
Determine if the change is desired
Either update your config to match, or redeploy to revert

Test Deployments¶

Always test in isolation before production:

docker run --rm \
  -v ~/.aws:/home/mluser/.aws:ro \
  -v $(pwd)/ml/configs:/app/configs:ro \
  -v $(pwd)/ml/reports:/app/reports \
  ml-provisioner:${IMAGE} -con ${CONFIG} -act test-deploy

This creates a separate stack with a random suffix — no impact on the production stack.

Stack Protection¶

Use --force flag only when intentional
Keep generated templates in version control
Do not modify generated templates manually — regenerate via create-prov-template

Docker Security¶

Runtime Security¶

Read-only credentials (:ro)¶

-v ~/.aws:/home/mluser/.aws:ro

What it does: Shares your host’s AWS credentials with the container but prevents the container from modifying or deleting them.
Why it matters: If malicious code runs inside the container, it cannot overwrite your AWS config, append unauthorized credentials, or delete your access keys.
Limitation: The container can still read and leak these keys if compromised. For production, use temporary IAM roles (like AWS IAM Roles for Tasks) instead of mounting static root keys

Read-only config¶

-v $(pwd)/ml/configs:/app/configs:ro

What it does: Permits the container to read application configurations while blocking any write permissions.
Why it matters: Attackers often try to alter configuration files to redirect traffic, change API endpoints, or disable security flags. This mount guarantees your application configuration remains untampered

No privileged mode¶

What it does: Keeps the container isolated without the –privileged flag.
Why it matters: Running a container with –privileged gives it capabilities nearly equal to the host root user. It breaks container isolation, allowing the container to access all host devices and easily break out into the host system.

No host networking¶

What it does: Avoids using --network=host, keeping the container inside its isolated virtual bridge network.
Why it matters: Host networking allows the container to see and interact with all network traffic on the host. It could manipulate local network services, bypass host firewalls, or intercept traffic from other containers.

No extra capabilities¶

What it does: Refrains from using --cap-add to grant Linux kernel capabilities.
Why it matters: Linux breaks down root privileges into distinct capabilities (like CAP_SYS_ADMIN or CAP_NET_ADMIN). Restricting these ensures the container cannot perform administrative tasks like modifying system clocks, altering network interfaces, or mounting external filesystems.

Image Verification¶

# Scan image before use
trivy image ml-provisioner:enterprise

Note: OS-level vulnerabilities in the Debian base image are common and typically have no fixed version available yet — the Fixed Version column will be empty for these. Focus on Python package vulnerabilities which are more actionable. Report any HIGH or CRITICAL Python package findings to support@axontechlabs.com.

Logging and Monitoring¶

Provisioner Audit Logs¶

Every action generates a log file in ml/reports/:

{ml_name}-{scenario}-{action}-{timestamp}.log

Retain these logs alongside your deployment artifacts for audit purposes.