CDK Infrastructure Pattern

Purpose

Define a standard two-stack CDK architecture for service modules: a shared CommonResourcesStack for networking and database resources, and a per-module stack for Lambda functions, queues, and event wiring.

Stack Composition

┌─────────────────────────────────────────────┐
│  CommonResourcesStack                        │
│                                              │
│  ├── VPC (imported from existing)            │
│  ├── Lambda Security Group                   │
│  ├── RDS Writer Proxy                        │
│  ├── RDS Reader Proxy (read-only endpoint)   │
│  ├── SNS Topic (imported or created)         │
│  └── Exports: proxy endpoints, ARNs          │
└──────────────────┬──────────────────────────┘
                   │ addDependency()
                   │ Fn.importValue()
┌──────────────────▼──────────────────────────┐
│  DocLoad Stack (per-module)                  │
│                                              │
│  ├── Python Package Layer                    │
│  ├── Document Loader Lambda                  │
│  ├── Job Processor Lambda + SQS Queue + DLQ  │
│  ├── Batch Processor Lambda + SQS Queue + DLQ│
│  ├── SNS Subscriptions with Filter Policies  │
│  └── IAM Roles and Policies                  │
└──────────────────────────────────────────────┘

CommonResourcesStack

Shared resources created once per environment/region, consumed by all modules.

Exported Resources

// Cross-stack exports using CDK Fn.importValue()
new cdk.CfnOutput(this, 'WriterProxyEndpoint', {
  value: writerProxy.endpoint,
  exportName: `${stackNamePrefix}-WriterProxyEndpoint`,
});
new cdk.CfnOutput(this, 'ReaderProxyEndpoint', {
  value: readerEndpoint.attrEndpoint,
  exportName: `${stackNamePrefix}-ReaderProxyEndpoint`,
});
new cdk.CfnOutput(this, 'WriterProxyArn', {
  value: writerProxy.dbProxyArn,
  exportName: `${stackNamePrefix}-WriterProxyArn`,
});
new cdk.CfnOutput(this, 'ReaderProxyArn', {
  value: readerProxy.dbProxyArn,
  exportName: `${stackNamePrefix}-ReaderProxyArn`,
});

RDS Proxy Configuration

Separate writer and reader proxies with environment-specific connection limits:

// Writer Proxy — for mutations
const writerProxy = new rds.DatabaseProxy(this, 'WriterProxy', {
  proxyTarget: rds.ProxyTarget.fromCluster(dbCluster),
  secrets: [dbSecret],
  vpc,
  dbProxyName: `${stackNamePrefix}-database-writer-proxy`,
  maxConnectionsPercent: config.rdsProxyConfig.writerMaxConnections,
  maxIdleConnectionsPercent: config.rdsProxyConfig.writerMaxIdleConnections,
  requireTLS: false,
  iamAuth: false,
});

// Reader Proxy — for read-only queries
const readerProxy = new rds.DatabaseProxy(this, 'ReaderProxy', {
  // Same config but with reader connection limits
  maxConnectionsPercent: config.rdsProxyConfig.readerMaxConnections,
  maxIdleConnectionsPercent: config.rdsProxyConfig.readerMaxIdleConnections,
});

// Dedicated read-only endpoint on reader proxy
const readerEndpoint = new rds.CfnDBProxyEndpoint(this, 'ReaderEndpoint', {
  dbProxyEndpointName: `${stackNamePrefix}-reader-endpoint`,
  dbProxyName: readerProxy.dbProxyName,
  vpcSubnetIds: privateSubnetIds,
  targetRole: 'READ_ONLY',
});

Connection Limits by Environment

Environment	Writer Max	Writer Idle	Reader Max	Reader Idle
Development	60	50	60	50
QA	20	3	20	3
Staging	60	varies	60	varies
Production	20	varies	20	varies

SNS Topic Strategy

// Production/Staging/QA: reuse topic from CommonResourcesStack
if (envName !== 'development') {
  snsStatusMonitorTopic = commonResources.snsStatusMonitorTopic;
} else {
  // Development: create a new local topic per developer
  snsStatusMonitorTopic = new sns.Topic(this, 'StatusMonitorTopic', {
    topicName: snsTopicName,
  });
}

Global Tags

Applied to all stack resources:

function applyGlobalTags(stack, env) {
  cdk.Tags.of(stack).add('map-migrated', 'mig6UGCR2HJGG');
  cdk.Tags.of(stack).add('Source', 'CDK');
  cdk.Tags.of(stack).add('APPLICATION', 'next-gen');
  cdk.Tags.of(stack).add('MODULE', 'document-loader');
  cdk.Tags.of(stack).add('ENVIRONMENT', env);
}

Module Stack (DocLoad)

Lambda Function Pattern

const documentLoader = new lambda.Function(this, 'DocumentLoader', {
  functionName: `${stackNamePrefix}-DocumentLoader`,
  runtime: lambda.Runtime.PYTHON_3_13,
  handler: 'index.lambda_handler',
  code: lambda.Code.fromAsset('lambda/src/loader'),
  memorySize: 256,
  timeout: cdk.Duration.minutes(15),
  layers: [pythonPackageLayer],
  vpc: commonResources.vpc,
  vpcSubnets: { subnets: privateSubnets },
  securityGroups: [lambdaSecurityGroup],
  environment: {
    ENV: envName,
    STACK_NAME: stackName,
    RDS_DBNAME: dbClusterDbName,
    RDS_HOST: writerProxyEndpoint,
    WRITER_PROXY_ENDPOINT: writerProxyEndpoint,
    READER_PROXY_ENDPOINT: readerProxyEndpoint,
    RDS_SECRET_ARN: dbSecret.secretArn,
    RDS_SECRET_NAME: dbSecret.secretName,
    SNS_STATUS_MONITOR_ARN: snsStatusMonitorTopic.topicArn,
    DEPLOY_REGION: awsRegion,
  },
});

SQS Queue + DLQ Pattern

// Dead Letter Queue
const dlq = new sqs.Queue(this, 'JobProcessingDLQ', {
  queueName: `${queueName}-dead-letter`,
  retentionPeriod: cdk.Duration.days(14),
});

// Main Queue
const jobQueue = new sqs.Queue(this, 'JobProcessingQueue', {
  queueName: `${stackNamePrefix}-DocLoaderJobsSQSQueue`,
  visibilityTimeout: cdk.Duration.minutes(15),  // Match Lambda timeout
  deadLetterQueue: {
    queue: dlq,
    maxReceiveCount: 3,
  },
});

SNS → SQS Subscription with Filter

snsStatusMonitorTopic.addSubscription(
  new subs.SqsSubscription(jobQueue, {
    rawMessageDelivery: false,
    filterPolicyWithMessageBody: {
      eventType: sns.FilterOrPolicy.filter(
        sns.SubscriptionFilter.stringFilter({
          allowlist: ['JOB_STARTED', 'JOB_FINISHED', 'IMPORT_CANCELLED', 'BATCH_END_FINISHED'],
        })
      ),
    },
  })
);

SQS → Lambda Event Source

jobProcessor.addEventSource(new SqsEventSource(jobQueue, {
  batchSize: 1,
  maxConcurrency: 40,
}));

Lambda Layer

const pythonPackageLayer = new lambda.LayerVersion(this, 'PythonPackageLayer', {
  code: lambda.Code.fromAsset('assets/layer'),
  compatibleRuntimes: [lambda.Runtime.PYTHON_3_13],
  description: 'Python packages: sqlalchemy, pymysql, pyyaml, etc.',
});

Layer Contents

The layer contains production dependencies only. Test dependencies are installed locally via pip install -r requirements-dev.txt.

Package	Version	Purpose
sqlalchemy	2.x	ORM — database models, sessions, queries
pymysql	1.x	MySQL driver for SQLAlchemy
pyyaml	6.x	YAML parsing (ES mappings, config files)
pydantic	2.x	Data validation schemas (if used by module)
elasticsearch	7.x	ES client (if module interacts with ES)
requests	2.x	HTTP client (if module calls external APIs)

NOT included in the layer (available in Lambda runtime or dev-only): - boto3 / botocore — pre-installed in Lambda runtime - pytest, moto, pytest-mock, pytest-cov — test dependencies only - black, isort — formatting tools, not runtime dependencies

Layer is built from requirements.txt in the module's assets/layer/ directory. Pin exact versions to prevent cross-environment drift.

IAM Least-Privilege Pattern

const lambdaRole = new iam.Role(this, 'LambdaRole', {
  assumedBy: new iam.ServicePrincipal('lambda.amazonaws.com'),
  managedPolicies: [
    iam.ManagedPolicy.fromAwsManagedPolicyName('service-role/AWSLambdaBasicExecutionRole'),
  ],
});

// Scoped permissions — specific resource ARNs
lambdaRole.addToPolicy(new iam.PolicyStatement({
  actions: ['s3:GetObject', 's3:ListBucket'],
  resources: bucketArns,
}));

lambdaRole.addToPolicy(new iam.PolicyStatement({
  actions: ['rds-db:connect'],
  resources: [writerProxyArn, readerProxyArn],
}));

// Secret access
dbSecret.grantRead(lambdaRole);

Environment Configuration

Infrastructure config is a TypeScript file mapping environment/region to resource IDs:

interface EnvironmentConfig {
  account?: string;
  region: string;
  vpcId: string;
  dbClusterEndpoint: string;
  dbClusterIdentifier: string;
  dbClusterDbName: string;
  snsStatusMonitorTopicArn: string;
  rdsProxyConfig: {
    writerMaxConnections: number;
    readerMaxConnections: number;
    writerMaxIdleConnections: number;
    readerMaxIdleConnections: number;
  };
  engine: string;
  engineVersion: string;
  privateSubnetIds: string[];
  lambdaSG: string;
  rdsSG: string;
  rdsSecretArn: string;
  bucketAccess: string[];
}

Stack Naming

{regionCode}-{envName}              # e.g., "use1-staging"
{regionCode}-{envName}-{alias}      # Dev includes developer alias

Lambda Configurations Summary

Lambda	Memory	Timeout	Reserved Concurrency	Batch Size	Handler
DocumentLoader	256 MB	15 min	None (dynamic)	Per-batch	index.lambda_handler
JobProcessor	128 MB	15 min	40	1	job_processor_index.lambda_handler
BatchProcessor	256 MB	15 min	40	1	batch_index.lambda_handler

Key Rules

1. Visibility timeout = Lambda timeout — prevents messages reappearing while Lambda runs
2. DLQ retention = 14 days — enough time to investigate and redrive
3. maxReceiveCount = 3 — 3 failures before DLQ (separate from application-level retry)
4. Batch size 1 for orchestrators — JobProcessor and BatchProcessor need single-record semantics
5. Reserved concurrency on orchestrators — prevents runaway scaling
6. RecursiveLoop = ALLOW — job processor re-enqueues to its own queue by design
7. SNS topic shared in prod, separate in dev — dev isolation without infrastructure cost
8. All env vars in CDK — Lambda code reads os.environ, CDK sets the values
9. Layer for Python deps — single source of truth for package versions
10. Private subnets only — Lambda in VPC with no public internet access

Sign In

Sign In

Please set a new password.

Set Password

Cancel

?

Ask the Architecture ×

Ask questions about Nextpoint architecture, patterns, rules, or any module. Powered by Claude Opus 4.6.

Ask