Integration Examples¶
Examples showing how to use VPC infrastructure provisioned by the VPC Provisioner with common AWS services and workflows.
Table of Contents¶
EC2 Integration¶
Launch Instance in Private Subnet¶
# Get private subnet ID from provisioned VPC
SUBNET_ID=$(aws ec2 describe-subnets \
--filters "Name=tag:Name,Values=edge-prod-b001-us-west-2-private-app-subnet-1" \
--query 'Subnets[0].SubnetId' --output text)
# Launch instance
aws ec2 run-instances \
--image-id ami-0abcdef1234567890 \
--instance-type ml.m5.xlarge \
--subnet-id $SUBNET_ID \
--security-group-ids sg-0a1b2c3d4e5f6g7h8 \
--iam-instance-profile Name=SageMakerExecutionRole \
--tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=ml-training-node}]'
Security Group for ML Workloads¶
# Get VPC ID
VPC_ID=$(aws ec2 describe-vpcs \
--filters "Name=tag:Name,Values=edge-prod-b001-us-west-2-vpc" \
--query 'Vpcs[0].VpcId' --output text)
# Create security group for ML training
aws ec2 create-security-group \
--group-name ml-training-sg \
--description "Security group for ML training instances" \
--vpc-id $VPC_ID
# Allow internal communication within VPC
aws ec2 authorize-security-group-ingress \
--group-name ml-training-sg \
--protocol -1 \
--cidr 10.1.0.0/16
SageMaker Integration¶
SageMaker in VPC Mode¶
import sagemaker
from sagemaker.estimator import Estimator
# Use provisioned VPC subnets and security groups
estimator = Estimator(
image_uri="123456789012.dkr.ecr.us-west-2.amazonaws.com/ml-training:latest",
role="arn:aws:iam::123456789012:role/edge-prod-b001-role-sagemaker-execution",
instance_count=2,
instance_type="ml.m5.xlarge",
subnets=[
"subnet-0a1b2c3d", # private-app-subnet-1
"subnet-4e5f6g7h", # private-app-subnet-2
],
security_group_ids=["sg-0a1b2c3d4e5f6g7h8"],
output_path="s3://edge-prod-b001-us-west-2-s3/solutions/customer-churn/models/",
)
estimator.fit({
"train": "s3://edge-prod-b001-us-west-2-s3/solutions/customer-churn/data/processed/train/"
})
SageMaker Notebook in VPC¶
import boto3
sagemaker_client = boto3.client('sagemaker')
sagemaker_client.create_notebook_instance(
NotebookInstanceName='ml-notebook',
InstanceType='ml.t3.medium',
RoleArn='arn:aws:iam::123456789012:role/edge-prod-b001-role-sagemaker-execution',
SubnetId='subnet-0a1b2c3d', # private-app-subnet-1
SecurityGroupIds=['sg-0a1b2c3d4e5f6g7h8'],
DirectInternetAccess='Disabled', # Access via NAT Gateway only
)
RDS Integration¶
Database in Database Subnet¶
# Get database subnet group
DB_SUBNET_1=$(aws ec2 describe-subnets \
--filters "Name=tag:Name,Values=edge-prod-b001-us-west-2-database-subnet-1" \
--query 'Subnets[0].SubnetId' --output text)
DB_SUBNET_2=$(aws ec2 describe-subnets \
--filters "Name=tag:Name,Values=edge-prod-b001-us-west-2-database-subnet-2" \
--query 'Subnets[0].SubnetId' --output text)
# Create DB subnet group
aws rds create-db-subnet-group \
--db-subnet-group-name ml-database-subnets \
--db-subnet-group-description "Database subnets for ML platform" \
--subnet-ids $DB_SUBNET_1 $DB_SUBNET_2
# Create RDS instance
aws rds create-db-instance \
--db-instance-identifier ml-metadata-db \
--db-instance-class db.t3.medium \
--engine postgres \
--master-username admin \
--master-user-password <password> \
--db-subnet-group-name ml-database-subnets \
--vpc-security-group-ids sg-database-sg \
--no-publicly-accessible
ECS/EKS Integration¶
ECS Service in Private Subnet¶
# Create ECS service using provisioned subnets
aws ecs create-service \
--cluster ml-cluster \
--service-name inference-service \
--task-definition ml-inference:1 \
--desired-count 2 \
--launch-type FARGATE \
--network-configuration "awsvpcConfiguration={
subnets=[subnet-0a1b2c3d,subnet-4e5f6g7h],
securityGroups=[sg-0a1b2c3d4e5f6g7h8],
assignPublicIp=DISABLED
}"
EKS Cluster in VPC¶
# Create EKS cluster using provisioned VPC
aws eks create-cluster \
--name ml-platform \
--role-arn arn:aws:iam::123456789012:role/eks-cluster-role \
--resources-vpc-config \
subnetIds=subnet-0a1b2c3d,subnet-4e5f6g7h,subnet-9i0j1k2l,\
securityGroupIds=sg-0a1b2c3d4e5f6g7h8
CI/CD Pipeline Integration¶
GitHub Actions¶
name: Deploy VPC
on:
push:
branches: [main]
paths: ['vpc/configs/**']
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/GitHubActions
aws-region: us-west-2
- name: Validate configuration
run: |
docker run --rm \
-v ~/.aws:/home/vpcuser/.aws:ro \
-v $(pwd)/vpc/configs:/app/configs:ro \
-v $(pwd)/vpc/reports:/app/reports \
vpc-provisioner:latest \
--config edge-prod-b001-us-west-2-vpc.yaml \
--action validate-config
- name: Preview changes
run: |
docker run --rm \
-v ~/.aws:/home/vpcuser/.aws:ro \
-v $(pwd)/vpc/configs:/app/configs:ro \
-v $(pwd)/vpc/templates:/app/templates \
-v $(pwd)/vpc/reports:/app/reports \
vpc-provisioner:latest \
--config edge-prod-b001-us-west-2-vpc.yaml \
--action show-changes
- name: Deploy VPC
run: |
docker run --rm \
-v ~/.aws:/home/vpcuser/.aws:ro \
-v $(pwd)/vpc/configs:/app/configs:ro \
-v $(pwd)/vpc/templates:/app/templates \
-v $(pwd)/vpc/reports:/app/reports \
vpc-provisioner:latest \
--config edge-prod-b001-us-west-2-vpc.yaml \
--action create-vpc \
--force
Terraform Integration¶
Reference Provisioned VPC¶
# Look up VPC created by VPC Provisioner
data "aws_vpc" "provisioned" {
filter {
name = "tag:Name"
values = ["edge-prod-b001-us-west-2-vpc"]
}
}
# Look up private subnets
data "aws_subnets" "private" {
filter {
name = "vpc-id"
values = [data.aws_vpc.provisioned.id]
}
filter {
name = "tag:Name"
values = ["*private*"]
}
}
# Use in other resources
resource "aws_security_group" "app" {
vpc_id = data.aws_vpc.provisioned.id
name = "app-security-group"
ingress {
from_port = 443
to_port = 443
protocol = "tcp"
cidr_blocks = [data.aws_vpc.provisioned.cidr_block]
}
}
resource "aws_lb" "app" {
name = "app-load-balancer"
internal = true
load_balancer_type = "application"
subnets = data.aws_subnets.private.ids
security_groups = [aws_security_group.app.id]
}
Cross-Provisioner Integration¶
S3 VPC Endpoint¶
The S3 Provisioner can create a VPC endpoint for private S3 access using the VPC created by the VPC Provisioner:
# S3 config references VPC
s3:
vpc_id: "vpc-0a1b2c3d4e5f6g7h8"
route_table_ids: "rtb-0a1b2c3d,rtb-4e5f6g7h"
SEC Provisioner Roles in VPC¶
IAM roles created by the SEC Provisioner are used by services running in the VPC:
# SageMaker execution role (from SEC) used in VPC subnets
estimator = Estimator(
role="arn:aws:iam::123456789012:role/edge-prod-b001-role-sagemaker-execution",
subnets=["subnet-0a1b2c3d", "subnet-4e5f6g7h"], # From VPC Provisioner
output_path="s3://edge-prod-b001-us-west-1-s3/...", # From S3 Provisioner
)
Discover Resources via SSM¶
# Get VPC stack outputs
aws ssm get-parameters-by-path \
--path /vpc/edge-prod-b001-us-west-2-vpc/ \
--query 'Parameters[].{Name:Name,Value:Value}'
# Get SEC role ARNs
aws ssm get-parameters-by-path \
--path /security/edge-prod-b001-us-west-1-medium-sec/ \
--query 'Parameters[].{Name:Name,Value:Value}'
SDK Examples¶
Boto3 — Discover VPC Resources¶
import boto3
ec2 = boto3.client('ec2', region_name='us-west-2')
# Find provisioned VPC
vpcs = ec2.describe_vpcs(
Filters=[{'Name': 'tag:Name', 'Values': ['edge-prod-b001-us-west-2-vpc']}]
)
vpc_id = vpcs['Vpcs'][0]['VpcId']
# List subnets
subnets = ec2.describe_subnets(
Filters=[{'Name': 'vpc-id', 'Values': [vpc_id]}]
)
for subnet in subnets['Subnets']:
name = next((t['Value'] for t in subnet['Tags'] if t['Key'] == 'Name'), 'unnamed')
print(f"{name}: {subnet['SubnetId']} ({subnet['CidrBlock']}) - {subnet['AvailabilityZone']}")
AWS CLI — Verify Deployment¶
# Check VPC
aws ec2 describe-vpcs \
--filters "Name=tag:Name,Values=edge-prod-b001-us-west-2-vpc" \
--query 'Vpcs[0].{VpcId:VpcId,CIDR:CidrBlock,State:State}'
# Check subnets
aws ec2 describe-subnets \
--filters "Name=tag:Name,Values=edge-prod-b001-us-west-2-*" \
--query 'Subnets[].{Name:Tags[?Key==`Name`].Value|[0],CIDR:CidrBlock,AZ:AvailabilityZone}'
# Check NAT Gateways
aws ec2 describe-nat-gateways \
--filter "Name=vpc-id,Values=vpc-0a1b2c3d4e5f6g7h8" \
--query 'NatGateways[].{Id:NatGatewayId,State:State,SubnetId:SubnetId}'