Cross-Module Communication Pattern¶

Purpose¶

Define how Nextpoint service modules communicate with each other. The primary mechanism is asynchronous SNS events. Synchronous communication (HTTP APIs, Lambda invocation) exists for specific use cases but is the exception, not the rule.

Communication Hierarchy¶

Prefer asynchronous over synchronous. Use the simplest mechanism that meets the need.

Priority	Mechanism	Use When
1 (default)	SNS events	Module publishes a fact; zero or more consumers react
2	S3 data staging	Payload exceeds SQS 256 KB limit
3	HTTP API (API Gateway)	Caller needs a synchronous response
4	Lambda invocation	Legacy integration from Rails (no SNS path)

The standard inter-module communication pattern. A module publishes an event; any interested module subscribes via filter policy.

DocumentExtractor                       DocumentLoader
┌──────────┐    SNS Topic    ┌───────────────┐
│ Extracts  │───▶ DOCUMENT_ ──▶│ SQS Queue     │
│ document  │    PROCESSED    │ (filter policy)│
└──────────┘                  │    ▼           │
                              │ Lambda loads   │
                              │ into database  │
                              └───────────────┘

Key properties: - Publisher doesn't know about subscribers — loose coupling - Fan-out: one event can trigger multiple consumers - Filter policies route events — subscribers only get what they need - See patterns/sns-event-publishing.md for implementation

Pattern 2: S3 Data Staging¶

When the data payload exceeds SQS's 256 KB message limit, stage it in S3 and pass the S3 reference in the event.

# Publisher: stage large payload in S3
def publish_large_payload(case_id: int, batch_id: int, data: dict) -> None:
    key = f"{Config.ENV}/staging/extractor/loader/{batch_id}/payload.json"
    s3_client.put_object(
        Bucket=Config.S3_BUCKET,
        Key=key,
        Body=json.dumps(data),
    )

    SNS.publish_sns_message({
        "eventType": EventType.EXTRACTION_COMPLETE.value,
        "eventDetail": json.dumps({
            "s3Bucket": Config.S3_BUCKET,
            "s3Key": key,
        }),
    })

# Consumer: retrieve from S3
def handle_extraction_complete(message: dict) -> None:
    detail = parse_event_detail(message.get("eventDetail"))
    response = s3_client.get_object(
        Bucket=detail["s3Bucket"],
        Key=detail["s3Key"],
    )
    payload = json.loads(response["Body"].read())
    process_extracted_data(session, payload)

Key properties: - Event is still the trigger — S3 is just the payload transport - Consumer deletes the staged file after processing - Use staging/ prefix with lifecycle rules for auto-cleanup

Pattern 3: HTTP API (API Gateway)¶

For cases where the caller needs a synchronous response. Used by documentexchanger (dual entry point) and Rails → NGE integration.

Rails App                        NGE Module
┌──────────┐    API Gateway    ┌───────────┐
│ POST      │───▶ /exchange ──▶│ Lambda    │
│ /exchange │                  │ (sync)    │
│           │◀── 200 OK ◀─────│ Response  │
└──────────┘                   └───────────┘

API Gateway Lambda Handler¶

# handlers/api_handler.py

def api_handler(event: dict, context) -> dict:
    """API Gateway Lambda handler — synchronous request/response."""
    try:
        body = json.loads(event.get("body", "{}"))
        validate_request(body)

        result = process_request(session, body)

        return {
            "statusCode": 200,
            "headers": {"Content-Type": "application/json"},
            "body": json.dumps(result),
        }

    except ValidationError as e:
        return {"statusCode": 400, "body": json.dumps({"error": str(e)})}
    except Exception as e:
        log_message("error", f"API error: {e}", exc_info=True)
        return {"statusCode": 500, "body": json.dumps({"error": "Internal error"})}

When to Use HTTP APIs¶

Rails needs a synchronous response (e.g., start exchange, get status)
Client needs confirmation before proceeding (e.g., validate before import)
Health checks or status polling

When NOT to Use HTTP APIs¶

Fire-and-forget operations — use SNS instead
Long-running operations — return 202 Accepted, then publish completion event
Module-to-module communication — use SNS events

Pattern 4: Lambda Invocation (Legacy)¶

Rails invokes NGE Lambdas directly for operations that don't have an SNS path. This is a Legacy integration pattern — prefer SNS for new module-to-module communication.

# Rails — invokes Lambda directly
lambda_client = Aws::Lambda::Client.new
response = lambda_client.invoke(
  function_name: "use1-production-DocumentLoader",
  invocation_type: "Event",  # Async — fire and forget
  payload: { caseId: case_id, batchId: batch_id }.to_json,
)

Invocation types: - Event (async) — fire and forget, Lambda runs in background - RequestResponse (sync) — waits for Lambda to complete, gets return value

When to Use Lambda Invocation¶

Rails → NGE integration where SNS subscription doesn't exist
One-off operations triggered by admin actions
Migration path: start with Lambda invocation, migrate to SNS events

When NOT to Use Lambda Invocation¶

Module-to-module communication — use SNS events
High-throughput operations — Lambda invocation has lower concurrency limits
When the caller doesn't need to know which Lambda handles it

Anti-Patterns¶

Direct Database Access (NEVER)¶

# WRONG: Module A reads Module B's database
with writer_session(npcase_id=case_id) as session:
    result = session.execute(text("SELECT * FROM module_b_table"))

Modules MUST NOT read or write another module's database tables directly. Share data via events.

Direct Module Import (NEVER)¶

# WRONG: Import another module's code
from documentloader.core.process import create_exhibit

Modules are independently deployable. Direct imports create coupling that breaks independent deployment.

Synchronous Chain (AVOID)¶

Module A → HTTP → Module B → HTTP → Module C → HTTP → Module A

Synchronous chains create cascading failures. If Module C is slow, Module A's request times out. Use events to decouple.

Key Rules¶

SNS events are the default — use synchronous communication only when async won't work
Events are facts, not commands — publish what happened, not what should happen
S3 for large payloads — SQS 256 KB limit means large data goes through S3
HTTP APIs return fast — long operations return 202, then publish completion event
Never read another module's database — share data through events
Never import another module's code — modules are independently deployable
Lambda invocation is Legacy glue — new inter-module communication uses SNS
Event schemas are the public API — changes are breaking changes

Ask the Architecture ×

Ask questions about Nextpoint architecture, patterns, rules, or any module. Powered by Claude Opus 4.6.