Reconciliation Workflow Agent

A sample agentic reconciliation platform. Work arrives either from an IDP (Intelligent Document Processing) pipeline or as a structured dataset posted to an API, and both become reconciliation items. Whatever matches deterministically clears without a model touching it. The rest goes to an LLM agent, which classifies the break, runs whichever investigation skills apply, and then either resolves the item itself or writes up a proposal for an analyst.

It resolves on its own only when the computed confidence clears an admin threshold and there is a clean, provable action available. That gate does not live in the prompt: an AgentCore Policy (Cedar) on the tools gateway checks the confidence server-side, so a below-threshold model cannot write even if it talks itself into trying. Proposals that do reach a human get approved or corrected, and the corrections come back to the agent as lessons.

The Tier-2 agent has two interchangeable backends, and a single SSM parameter (agent_backend) picks between them, which makes A/B and rollback instant. One is a container AgentCore Runtime running a hand-rolled Strands/Bedrock loop. The other is the managed AgentCore Harness, declared in config with no orchestration code of ours. Alongside both, an evaluation pipeline scores sessions against analyst decisions as ground truth and surfaces prompt and tool recommendations in the Evals tab.

Solution Architecture

Full interactive version: assets/Solution Architecture.html

The architecture has three planes:

Plane	Purpose	Key services
Ingestion	Email/document intake → unique event id → raw storage → classification → field extraction (Bedrock LLM) → schema validation → per-field confidence scoring	IDP pipeline (Lambda, S3, DynamoDB)
Application	Human-in-the-loop review frontend + backend API; agent runtime for the Reconciliation Agent	Frontend (React/Next.js), ECS/ALB, Backend API, AgentCore Runtime/Harness
Shared services	Tool access, memory, identity, policy, observability, evaluation for all agents; LLM access	AgentCore Gateway, Memory, Identity, Policy, Evaluation; Bedrock Knowledge Base; Bedrock foundation models

High-level architecture components

Concern	Implementation
Entry points	The IDP post-processing hook Lambda (`recon-dev-idp-hook`, invoked by the IDP stack on document completion), and an intake HTTP API (API Gateway + Cognito JWT) for structured datasets
Item / case stores	Four DynamoDB tables: `recon-dev-items` (canonical `ReconItem` inputs, stream-enabled), `recon-dev-cases` (case lifecycle, status GSI), `recon-dev-audit` (append-only status-transition log), `recon-dev-lessons` (analyst decisions: approval, correction, auto-resolution, one row per item+trigger)
Deterministic tier	A Tier-1 Lambda consuming the items stream. Sided items match within tolerance; sides-less (IDP) items are looked up in a mocked general ledger (Athena over S3) and auto-clear only on an unambiguous attribute match: account name, entry-type direction, and amount within tolerance. Toggleable via SSM or the Config tab
Agent	Two interchangeable backends selected by the `agent_backend` SSM parameter: an AgentCore Runtime container (Strands `Agent` agentic loop), or the managed AgentCore Harness (config-declared). Skills and the system prompt are live from S3, with a ~60 s cache on the runtime and per-session on the harness. Two AgentCore gateways (AWS_IAM/SigV4): the egress tools gateway (6 targets, 2 of them conditional) with the Cedar Policy confidence gate, and an ingress agent gateway fronting the runtime. AgentCore Memory holds the `lessons_learned` semantic strategy, and a Bedrock Knowledge Base holds guidance
Evaluation	AgentCore Online Evaluation (a custom analyst-agreement evaluator plus 3 builtins) over harness OTel traces, on-demand batch re-scores, managed recommendations, and a versioned harness-config store (immutable S3 docs + SSM pointer). All of it surfaces in the Evals tab
Frontend	Next.js on ECS Fargate behind an ALB and CloudFront, with a WAFv2 web ACL (`AWSManagedRulesCommonRuleSet`) on the distribution, which is the single internet entry point. Okta OIDC login (`auth_provider`, swappable to Entra) and same-origin BFF routes (`/api/recon/*`) running under the task role
Notifications	Microsoft Graph is the only channel (app-only, from the shared mailbox), reached through the egress gateway's OpenAPI target. It carries resolution emails on approve/auto-resolve (`cases/notify.py` plus the frontend BFF calling `sendSharedMailboxMail` through the gateway with SigV4), counterparty email sent by the BFF from an analyst-approved draft, and mailbox reads (`listSharedMailboxMessages`, reached only through the `search_correspondence` wrapper). No agent holds a send tool on either backend: the model writes the counterparty message into its proposal and a human approves a specific revision of it. Sends are gated at the gateway REQUEST interceptor, because Cedar cannot gate the OpenAPI op — a `notification` send may only address `RECON_NOTIFY_EMAIL`, and a `counterparty` send must match, byte for byte, the draft approved on that case at that revision
IaC	Terraform (`infra/`) with S3-backed state. The AgentCore Harness lifecycle is driven by boto3 (`manage_harness.py`) through a `terraform_data` provisioner

IDP decoupling. Two channels reach the independently-deployed IDP solution and no others: the inbound hook invocation and the IDP MCP tool. One storage read is sanctioned, and it happens at ingest, when the hook copies extracted field values and page images into the recon item. After that the runtime never touches IDP storage.

Repository layout

backend/                Python 3.12 Lambda handlers
  recon_core/           Shared domain: schema, cases, status, confidence, auto_resolve,
                        lessons_recall, errors (ToolDenied), skills_s3, prompt_source
                        (shared-core + harness-contract composition), otel_client
                        (client-side spans, baggage, trace propagation)
  tier1/                DynamoDB stream consumer + agent-worker (ingress-gateway / direct / harness selector)
  harness_agent/        Managed-Harness backend: worker, stream, intake (proposal validation +
                        reference derivation), prompting, session, config_store
  gl_tool/              General-ledger read + set_draw_status write (status allowlist only;
                        threshold via Cedar Policy, provenance via the gateway interceptor)
  status_tool/          recon-status gateway target: recon_update_status (platform-only,
                        state-machine-guarded + audited case-status writes)
  gateway_interceptor/  Gateway REQUEST interceptor: set_draw_status provenance +
                        recon_update_status transition re-check + the sendSharedMailboxMail
                        email gate — token + sendPurpose, where `notification` may only
                        address RECON_NOTIFY_EMAIL and `counterparty` must match the draft
                        approved on the case at that revision (log/enforce modes), plus
                        OData argument normalization on the listSharedMailboxMessages read
  notify_tool/          Microsoft Graph email: graph.py (app-only client), send + search handlers
  correspondence_tool/  correspondence-search gateway target: search_correspondence(query, top)
                        — sanitizes the model's arguments into Graph's OData form ($search
                        double-quoted, $top an integer) and re-enters this gateway to call
                        listSharedMailboxMessages, so the Graph credential stays in the vault
  eval_agreement/       Analyst-agreement custom evaluator Lambda
  kb_tool/              Knowledge-base search handler
  intake/               Intake API handler
  idp_hook/             IDP post-processing hook Lambda + mapper + explainability (aggregates
                        IDP's per-field extraction confidences into the harness composite's
                        classification signal — IDP emits no document_class.confidence)
  cases/                Resolution-email helper (notify.py), shared by the proposal service +
                        approve path — the JWT cases BFF was removed (status writes now go
                        through the recon-status gateway tool)
  skills_api/           Skills BFF (CRUD)
  lessons_api/          Lessons BFF

agent-blueprint/
  recon-agent/          AgentCore Runtime container: agent.py, strands_investigator.py, llm.py,
                        classifier.py, proposal.py, gateway_mcp.py, skills_loader.py,
                        skills/*.md, system-prompt.md, Dockerfile
  recon-agent-harness/  Harness blueprint: harness_config.py (tools/schema), system-prompt.md

chatbot-app/
  frontend/             Next.js app: /recon/* pages + /api/recon/* BFF routes. Other api/ route
                        groups are scaffolding inherited with the fork; 10 of them are
                        non-functional in this deployment and now fail loudly naming the missing
                        env var (src/lib/deployment-env.ts)

infra/
  modules/              Terraform modules: foundation, intake, tier1, idp-hook, recon-agent,
                        recon-agent-harness, agent-evals, gl-mock, api, frontend-ecs,
                        lambda-package, lambda-logs, network, observability, microsoft-graph-obo
  environments/recon/   Dev environment root (S3-backed state via a partial backend config)
  bootstrap/            Terraform-state bucket bootstrap (local state; import-first — see
                        "Getting Started" step 1)
  scripts/              Utility scripts (deploy-recon.sh, spike_harness.py, spike_evals.md)

data/                   Synthetic sample documents + mocked general-ledger CSV
tests/                  60 pytest test files (moto-mocked AWS); frontend: chatbot-app/frontend/__tests__
assets/                 Architecture diagrams (SVG/HTML), screenshots, CUJ walkthrough + template

Design records, implementation plans and security-audit reports are kept outside this repository.

Tech stack

Layer	Stack
Backend	Python 3.12, `strands-agents==1.50.2`, `bedrock-agentcore==1.18.1`, `boto3==1.43.57`, `pydantic==2.13.4`, `aws-opentelemetry-distro==0.19.0` (runtime container)
Frontend	Next.js 16, React 18, Tailwind CSS, Radix UI, `@aws-sdk/client-bedrock-agentcore`, MSAL / `@okta/okta-auth-js`
IaC	Terraform (AWS provider `~> 6.55, != 6.57.0` — 6.57.0 corrupts request bodies under parallel refresh), S3 backend
Agent	Amazon Bedrock AgentCore (Runtime, Harness, Gateway, Memory, Policy, Evaluation, Identity)
LLMs	Claude Sonnet 5 (default for both the runtime and harness backends; selectable per backend)
Testing	pytest + moto (backend), vitest + testing-library (frontend)

Getting Started

You need a clone of this repo, Terraform >= 1.11, and AWS credentials for the target account. Steps 1 to 3 are one-time setup for a fresh account or a fresh checkout; from then on step 4 is the whole deployment.

1. Bootstrap the Terraform state bucket (once per account)

infra/bootstrap creates the S3 bucket every environment stores its state in. It keeps local state, because it cannot use the bucket it is about to create as its own backend.

In a fresh account, terraform init && terraform apply there is all it takes. If the bucket already exists — it does in the dev account, where recon-dev-tfstate-<account_id> was created out-of-band before it was expressed as code — a plain apply fails with BucketAlreadyOwnedByYou instead of adopting it. Import the four resources first:

cd infra/bootstrap && terraform init
BUCKET="recon-dev-tfstate-$(aws sts get-caller-identity --profile huthmac --query Account --output text)"
terraform import aws_s3_bucket.tfstate                                      "$BUCKET"
terraform import aws_s3_bucket_versioning.tfstate                           "$BUCKET"
terraform import aws_s3_bucket_server_side_encryption_configuration.tfstate "$BUCKET"
terraform import aws_s3_bucket_public_access_block.tfstate                  "$BUCKET"
terraform plan   # expect "No changes" — the live bucket already has all four settings

Skipping the import is not free: an unimported bucket has no Terraform source, so drift in its versioning, encryption or public-access settings appears in no plan.

2. Create the two local config files

Neither file is committed, because both carry the AWS account ID — the state bucket name embeds it, and a backend block cannot read variables, which is why the bucket name arrives through a separate -backend-config file.

cd infra/environments/recon
cp backend.hcl.example backend.hcl            # then set the state bucket name from step 1
cp terraform.tfvars.example terraform.tfvars  # then fill in the required values

otel_layer_account declares no default, so both plan and apply stop until it is set. It is AWS's own public layer-publisher account rather than a secret. See Prerequisites & configuration for every variable.

3. Deploy

./infra/scripts/deploy-recon.sh plan
./infra/scripts/deploy-recon.sh apply

The script changes into infra/environments/recon, refuses to run at all without backend.hcl, and wraps terraform init -backend-config=backend.hcl plus plan, apply or destroy with -input=false. By hand it is terraform init -backend-config=backend.hcl && terraform apply.

One apply deploys everything. It packages the Lambdas, has CodeBuild build and push the agent container and the frontend image (the build driver blocks until the push succeeds, before the AgentCore Runtime is created), provisions the managed Harness (manage_harness.py via terraform_data), attaches the Cedar Policy, seeds the skills, system prompts and KB corpus to S3, wires the CloudFront domain and agent runtime ARN through Terraform's dependency graph, and rolls the ECS service. No second apply, no manual build step. Most of the wall-clock time is CodeBuild.

4. Do the two things Terraform cannot

Both live in systems Terraform does not own, so an apply can succeed and the platform still not work. The post_deploy_checklist output names them after every apply.

Register the okta_redirect_uri_to_register output as a sign-in redirect URI on the Okta OIDC app, and frontend_url as a sign-out redirect URI. This needs an Okta org admin. Until the callback URI is registered, login cannot complete and every route stops at 400 invalid_request.

Set the IDP solution's PostProcessingLambdaHookFunctionArn to the idp_hook_function_arn output, so extracted documents reach the recon pipeline.

5. Optional flips

# Harness instead of the runtime backend (instant A/B; flip back with agent_backend=runtime)
terraform apply -var="agent_backend=harness"

# Policy is ENFORCE by default; LOG_ONLY observes decisions without blocking
terraform apply -var="policy_enforcement_mode=LOG_ONLY"

6. Run the tests

# Backend (repo root)
python -m pytest -q            # 312 passed, 11 skipped
#                              # the 11 skips are all in tests/integration/ — 10 need
#                              # RECON_GATEWAY_URL (+ dev-account creds), 1 also needs
#                              # EMAIL_CONFIRMATION_TOKEN

# Frontend tests + build — include __tests__/api/ (the BFF route tests: harness configs,
# config deploy, evals recommendations) alongside the lib tests
cd chatbot-app/frontend && npx vitest run __tests__/lib/ __tests__/api/ && npm run build
#                          # 17 files, 222 passed

Prerequisites & configuration

Variable	Required	Purpose
`region`	yes	AWS region (default `us-east-1`)
`name_prefix`	yes	Resource name prefix (e.g. `recon-dev`)
`otel_layer_account`	yes	AWS's own public publisher account for the `AWSOpenTelemetryDistroPython` layer. It declares no default on purpose: a wrong or absent value composes a valid-looking layer ARN that fails at apply with an opaque Lambda error, so Terraform stops and names the variable instead. Not a secret. It lives in tfvars only because the repo's pre-push guard rejects any 12-digit run in a committed file. Only read when `enable_worker_tracing = true`, though `terraform plan` requires it either way
`hosted_ui_prefix`	yes	Cognito Hosted UI domain prefix. It must be globally unique, so the `recon-dev-login` default will collide. Required only because `aws_cognito_user_pool_domain` is unconditional in `modules/foundation`; the Hosted UI login path itself has been orphaned since the Okta switch (`hosted_ui_domain` → `NEXT_PUBLIC_COGNITO_HOSTED_UI` → `src/lib/auth.ts` `buildLoginUrl`/`exchangeCode`, which nothing calls). The user pool is still live, as the issuer for the intake API's JWT authorizer, but that authorizer uses the pool's `cognito-idp` endpoint rather than this domain, so the domain can be dropped on its own
`idp_gateway_target_url`	no	IDP MCP endpoint (enables `document-extraction` target)
`idp_mcp_secret_json`	no	IDP OAuth2 client credentials JSON
`recon_domain`	no	Recon domain the IDP hook stamps on ingested items (default `cash`)
`graph_enabled`	no	Enable the `microsoft-graph` OpenAPI target (the platform's single email interface)
`graph_mailbox`	no	Shared mailbox SMTP address all Graph email is sent from / read (must be a real mailbox in the Entra tenant)
`notify_email`	no	Resolution-notification recipient (human approve + auto-resolve), sent from `graph_mailbox` via the gateway's `sendSharedMailboxMail` tool. Empty disables the email step. The dev environment points it at the shared mailbox itself, so notifications land in the same inbox the agent reads
`entra_tenant_id/client_id/client_secret`	no	Entra app-only credentials for Graph email
`auth_provider`	no	Frontend IdP: `okta` (deployed) or `entra` (var default)
`okta_issuer` / `okta_client_id`	no	Required when `auth_provider=okta`
`agent_backend`	no	`runtime` (default) or `harness`
`harness_model_id`	no	Override harness LLM (default `us.anthropic.claude-sonnet-5`)
`policy_enforcement_mode`	no	`ENFORCE` (default) or `LOG_ONLY` (observe only)
`interceptor_mode`	no	Gateway REQUEST interceptor: `log` (default) or `enforce`
`enable_worker_tracing`	no	`true` (default) attaches the ADOT layer and OTel env to the agent-worker Lambda so its invocations share one trace with the agent's own spans. `false` means no layer, no OTel env, PassThrough X-Ray
`otel_layer_version`	no	Version of AWS's public `AWSOpenTelemetryDistroPython` Lambda layer (default `30`; pinned rather than `latest`, which AWS does not publish)
`reprocess_cap`	no	Max re-process attempts before a case ages out (default `3`)
`private_vpc`	no	`false` (default) = public CloudFront + internet-facing ALB. `true` = the whole private topology in one flag: internal ALB on private subnets, Fargate with no public IP, no CloudFront, plus the interface endpoints. Does not remove the NAT — see Private VPC deployment
`private_ingress_cidrs`	no	CIDRs allowed to reach the internal ALB when `private_vpc=true` (VPN/corporate ranges). Empty ⇒ the VPC CIDR only. Ignored when `private_vpc=false`

Copy infra/environments/recon/terraform.tfvars.example → terraform.tfvars and fill values. terraform.tfvars.example is the only committed record of which variables an environment is expected to set — add a placeholder entry there (never a real credential) in the same change that adds a variable.

Status & Process Flow

stateDiagram-v2
    [*] --> PENDING : item ingested (IDP hook / intake API)
    PENDING --> AUTO_CLEARED : Tier-1 deterministic match
    PENDING --> IN_PROGRESS : Tier-1 miss → escalate to agent
    IN_PROGRESS --> PROPOSED : agent proposes resolution
    IN_PROGRESS --> APPROVED : autonomous execution (confidence ≥ threshold + clean action, Policy-permitted)
    PROPOSED --> APPROVED : analyst approves (+ optional comment)
    APPROVED --> RESOLVED : notification email sent
    PROPOSED --> REJECTED : analyst disapproves (comment required)
    REJECTED --> CLOSED_NO_ACTION : outcome "no further action"
    REJECTED --> IN_PROGRESS : outcome "re-process" (correction fed to agent)
    IN_PROGRESS --> AGED : re-process cap reached (default 3)
    AUTO_CLEARED --> [*]
    RESOLVED --> [*]
    CLOSED_NO_ACTION --> [*]
    AGED --> [*]

Step-by-step

#	Step	Status	What happens
1	Ingest	→ `PENDING`	A completed IDP document invokes the hook Lambda, or a structured dataset hits the intake API. The hook maps the event to a canonical `ReconItem` (`item_id = idp-<documentId>`, idempotent) and embeds the IDP results at ingest: per-section classification, extracted field values, and page-preview images copied into recon's own assets bucket. Writing the item opens a `PENDING` case.
2	Tier-1 deterministic	→ `AUTO_CLEARED` or → `IN_PROGRESS`	A DynamoDB-stream Lambda runs rule-based matching. Sided items match within tolerance. Sides-less (IDP) items are matched against the mocked general ledger on the cash item's economic identity (`backend/tier1/gl_match.py`): account name (IDP `BorrowerName` → GL `borrower`), the entry-type direction (CREDIT/DEBIT, derived by keyword from the opaque IDP document class), and an amount within ±0.05 of an IDP-extracted amount. It never keys on the document filename or reference. Auto-clear requires exactly one surviving GL row; zero means no match, more than one means ambiguous, and both escalate to `IN_PROGRESS` with the candidate rows attached as `gl_candidates` context before invoking the agent worker. Tier-1 can be disabled at runtime from the Config tab (SSM-backed).
3	Tier-2 agent	`IN_PROGRESS` → `PROPOSED`	The agent characterizes the break (a calibration signal, not a skill selector), investigates by invoking one or more relevant skills from the live SKILL.md library (gateway tools, with reasoning, confidence and cited evidence per step), then proposes a resolution. The composite confidence comes from classification self-consistency (0.45), evidence grounding (0.35) and model self-report (0.20), with a −10% IDP low-confidence penalty where applicable.
3b	Autonomous execution + auto-resolve	`IN_PROGRESS` → `APPROVED` → `RESOLVED`	When confidence clears the admin threshold and a single matched ledger reference exists, the agent executes `set_draw_status` through the Policy-gated egress gateway. The AgentCore Policy (Cedar, ENFORCE) gates it at the gateway, and the write Lambda additionally verifies provenance by checking the reference against the persisted proposal. On success the case auto-resolves: notification, `AUTO_RESOLVED` lesson, `RESOLVED`. Below threshold, or with no clean action, it halts at `PROPOSED` for human review.
4	Human review	`PROPOSED`	The analyst reviews the case: IDP document panel (section tabs ⇄ page images + extracted fields), classification and reasoning, the proposed resolution, and the step-by-step agent trace. Bulk status updates are supported, with comments.
5a	Approve	→ `APPROVED` → `RESOLVED`	Optional comment. A notification email goes out and the case closes as `RESOLVED`. The decision is captured as a `USER_APPROVED` lesson.
5b	Disapprove	→ `REJECTED` → …	A correction comment is required, plus an outcome: No further action → `CLOSED_NO_ACTION`, or Re-process → stores the correction and re-invokes the agent (`IN_PROGRESS`). Re-processing is capped (default 3) and ages out at the cap. Captured as a `USER_CORRECTION` lesson.
6	Lessons learned	(parallel)	Every analyst decision is captured twice: in the `recon-lessons` DynamoDB ledger, and as an AgentCore Memory event (`lessons_learned` SEMANTIC strategy). Before classifying, the agent retrieves consolidated lessons and weights them in both classification and investigation.

Terminal states

AUTO_CLEARED · RESOLVED · CLOSED_NO_ACTION · AGED

The Reconciliation Agent & pre-created skills

The Tier-2 agent runs on one of two interchangeable backends (below). Skills are a composable library of investigation and resolution procedures, not classification categories: the agent gets handed the whole library and invokes as many of them as the item warrants. Each escalated item goes through one loop:

Recall lessons. Retrieve the most relevant consolidated analyst lessons for the item's domain from AgentCore Memory (lessons_learned strategy). Advisory, fail-soft.
Characterize the break. The model assesses what kind of exception the item is (name, confidence, reasoning). This is a calibration signal that seeds the case class and the confidence composite; it does not restrict which skills may run. Below the global DEFAULT_CLASS_THRESHOLD (0.6) it is recorded as unknown.
Investigate. The agent receives the full skill library, where each skill's markdown body is its procedure, and runs whichever ones apply, composing several when the evidence warrants. Their gateway tools (search_ledger, search_guidance, get_results, search_correspondence — all reads) each land in the trace as a typed ReasoningStep carrying reasoning, cited evidence, and tool I/O.
Propose. A final pass produces the resolution, the composite confidence, and a structured proposed_action. The ledger reference in it is derived by the worker from the search_ledger results rather than supplied by the model; zero matches or more than one distinct match means no action, which forces an escalation.
Execute or escalate. If the composite clears the threshold and a clean action exists, the platform (the worker or runtime process, never the model) performs the Policy-gated set_draw_status write on the agent's behalf and the case auto-resolves. Otherwise it halts at PROPOSED for human review.

Two backends (`agent_backend` — instant A/B + rollback)

Backend	Where	How it runs
`runtime` (default)	`agent-blueprint/recon-agent/`	An arm64 AgentCore Runtime container running a Strands `Agent` agentic loop (`strands_investigator.py`) with k-sample self-consistency classification. That classification is also a Strands call: every Bedrock request this container makes goes through the SDK with `streaming=False`. It makes autonomous gateway tool calls over MCP (SigV4) and returns a JSON proposal. Tools: `search_ledger`, `search_guidance`, `get_results`, `search_correspondence` — reads only, no send.
`harness`	`agent-blueprint/recon-agent-harness/` + `backend/harness_agent/`	The managed AgentCore Harness, declared in config with no orchestration container of ours. The harness calls the egress gateway (the `agentCoreGateway` tool) plus an `inline_function submit_proposal`, and a thin worker drives the round-trip, assembles the trace from the event stream, derives the reference, computes the composite and persists.

Both backends share the egress gateway, Memory, KB, DynamoDB, and the same SKILL.md skill set, and both reach the same Microsoft Graph surface — for reading only. Mailbox reads go through the sanitized search_correspondence(query, top) wrapper, because the raw Graph op's $-prefixed OData arguments are not legal tool-schema property names. Neither backend holds a send tool: a counterparty email is data the model writes into its proposal, and the BFF sends the revision an analyst approved. Both models are propose-only. Neither is given set_draw_status, and the confidence-gated write happens after the proposal in auto_resolve.autonomous_execute, where the harness worker and the runtime container run the same code. What differs between the two is the calling mechanics, meaning who drives the loop and where the trace comes from, not the tool surface.

The runtime is reached through the ingress agent gateway (SigV4, with a direct InvokeAgentRuntime fallback).

Skills live as SKILL.md files in S3, one directory per skill (skills/<name>/SKILL.md), with frontmatter (name, description, tools: [<gateway tools>], optional model) and a free-text procedure body. The catalog is a composable library rather than a one-of-N classification registry: the agent is handed every skill and invokes the relevant ones, and unknown is the escalate-with-context fallback for when nothing conclusive applies. Create or edit a skill in the Skills tab and the agent picks it up within about 60 s, no redeploy. The system prompt is just as live-editable. One global DEFAULT_CLASS_THRESHOLD (0.6) gates the break-characterization step.

Pre-created skills

These ship as a starting library. They are composable rather than mutually exclusive, and one reconciliation often uses several: document-cross-reference plus record-match-review plus correspondence-search, then ledger-status-resolution to act on what they turned up.

Skill	Tools (`tools:` frontmatter)	Purpose
`record-match-review`	`search_ledger`	Compare the two sides' key attributes (amount, date, identifier) with tolerance and aggregation to confirm or refute a match
`document-cross-reference`	`get_results`, `search_ledger`	Retrieve and compare fields from the source document (via the IDP `document-extraction` MCP tool) against ledger records to confirm or refute a candidate match
`consult-guidance`	`search_guidance`	Retrieve reconciliation guidance/playbooks from the recon Knowledge Base
`correspondence-search`	`search_correspondence`	Search the shared mailbox for messages that clarify the item — via the sanitized `correspondence-search` target, which builds the Graph OData arguments for the model
`counterparty-contact-draft`	— (`tools: []`)	Draft a counterparty email into `submit_proposal`'s `email_draft` and stop. No send tool is offered on either backend; the analyst approves a revision on the case and the BFF sends that exact text
`ledger-status-resolution`	`set_draw_status`	Resolve a confirmed break via a ledger status update (`{Confirmed, Cancelled, OnHold, Amended}`); executed by the worker/human-approve path as the Policy-gated write — reference derived from `search_ledger`, never model-supplied
`unknown`	—	Escalate-with-context fallback when no skill conclusively applies — gather context and escalate (not deletable)

The model is configurable per backend: runtime MODEL_ID (default us.anthropic.claude-sonnet-5), harness harness_model_id (default us.anthropic.claude-sonnet-5).

Confidence & auto-resolution (straight-through processing)

The Overall Confidence shown on the case, which is also what gets compared against the admin threshold, is a computed composite (confidence.py). Both backends run the identical formula and weights out of a single shared _composite() core, so the same document scores the same whichever backend reconciled it:

Signal	Weight	What it measures
Classification confidence	0.45	How sure the pipeline is of the break's class
Evidence grounding	0.35	Fraction of cited evidence literally present in item data
Model self-report	0.20	Verbalized confidence (weak signal)
IDP low-confidence penalty	×0.9	Applied when IDP flagged any extracted field below its own threshold

The one backend-specific detail is the source of the 0.45 classification-confidence signal, and it differs because only the runtime owns its own inference loop. The runtime runs a dedicated classification step before the agentic loop, so it can draw k independent votes. On the harness the loop belongs to the managed AgentCore service and the class label arrives as a single field (submit_proposal.class_name) from one session, so there is no agreement fraction to compute. There is only one vote.

runtime: k-sample self-consistency, the agreement across 3 independent classification samples (majority vote, temp 0.7). Each sample is a fresh single-turn call rather than another turn on one agent, so sample n cannot see sample n−1's answer and collapse the very independence the signal is supposed to measure (agent-blueprint/recon-agent/llm.py).
harness: IDP's per-field extraction confidence, averaged over the fields IDP actually populated (backend/idp_hook/explainability.py). The harness has no self-consistency loop.

When the classification signal is unavailable, both paths renormalize the remaining two weights identically (grounding/verbalized → 0.35/0.55 and 0.20/0.55 of the composite, i.e. 0.636 × grounding + 0.364 × verbalized). Given the same classification / grounding / verbalized inputs, runtime and harness therefore return the exact same score (unit-tested in tests/recon_core/test_confidence_idp.py::test_both_backends_agree_for_same_inputs).

The default threshold is set to 0.85, which still demands a strong result on all three signals at once (roughly classification ≥0.93, grounding ≥0.9 and verbalized ≥0.7 together) and sits deliberately at the conservative end of the reachable band.

Treat 0.85 as a stated assumption rather than a derived answer. No target auto-execute rate was ever specified, and n=10 is far too small to fit a calibration curve to. Operators should re-tune it against their own observed distribution; it is a live knob in the Config tab, needs no redeploy, and rewrites the Cedar gate immediately.

The auto-resolve threshold (Config tab, default 0.85, disableable) is enforced by the AgentCore Policy Cedar gate on the gateway in ENFORCE mode, and that gate is the single source of truth for whether an autonomous write is permitted. Editing the threshold in the Config tab rewrites the Cedar policy at runtime (reconPolicy.ts calling the Policy UpdatePolicy API), so the gate tracks the admin value without a redeploy. The SSM value the worker reads is only an app-level hint, used to decide execute-vs-escalate before it attempts the write at all.

AgentCore Gateways, Policy & preconfigured tools

Two gateways, both AWS_IAM inbound (SigV4):

Egress tools gateway (recon-dev-gateway) fronts all tool traffic: the agent's, the worker's, and the frontend BFF's platform calls. Two gateway-native enforcement layers sit on it.
- AgentCore Policy (Cedar, ENFORCE by default). recon_write_gate permits set_draw_status when context.input.confidence ≥ threshold, which covers the agents and the worker. recon_write_human permits it for the BFF principal, since human approval carries no confidence value to test. recon_status_platform and recon_status_forbid_agents together make recon_update_status platform-only, so the model can never move its own case. Reads are unconditional. The Config-tab threshold edit rewrites recon_write_gate through the Policy API.
- REQUEST interceptor (Lambda, backend/gateway_interceptor/) holds the three guards Cedar cannot express: provenance for the ledger write, where the reference must equal the persisted proposed_action.reference; a case state-machine re-check for recon_update_status; and the email gate on sendSharedMailboxMail. Every send needs a confirmationToken matching EMAIL_CONFIRMATION_TOKEN (stripped before the request reaches Graph) plus a sendPurpose saying which kind of send it is. A notification send may only address RECON_NOTIFY_EMAIL. A counterparty send must name a reconItemId whose persisted draft is approved, at the revision that was approved, and must match that draft's recipient, subject and body byte for byte — so the guard is provenance ("is this the text a human approved?"), not just capability ("does the caller hold the token?"). A missing or unrecognized sendPurpose is denied. Rolled out as interceptor_mode = "log" first, then "enforce".
  
  It also carries one behaviour that enforces nothing: OData argument normalization on the listSharedMailboxMessages read ($top numeric-string to int, bare $search to double-quoted). That runs regardless of interceptor_mode, fails open, and logs every coercion at WARNING. It is a backstop now rather than the primary fix, since the correspondence-search target and the runtime's wrapper both emit correct OData and the coercions are idempotent, but it stays because those wrappers are not the only callers the raw op could ever have.
Ingress agent gateway (recon-dev-ingress-gateway) is an http/agentcoreRuntime target fronting the Runtime: one controlled SigV4 entry point for the Tier-1 worker and the BFF.

Egress tools

Target	Type	Enabled by	Tools
`general-ledger`	Lambda (Athena over S3)	always	`search_ledger(reference, borrower, facility, amount, date)`
`set-draw-status`	Lambda (DynamoDB GL status overlay)	always	`set_draw_status(reference, status, reason, item_id, confidence)` — Policy-gated + interceptor-provenance-checked write; executed by the WORKER (autonomous) or the BFF (human approve) — never by the model directly on the harness backend
`recon-status`	Lambda (cases + audit tables)	always	`recon_update_status(item_id, new_status, comment, actor)` — platform-only workflow-status tool (Cedar forbids agent principals); guarded by the case state machine + audited
`knowledge-base`	Lambda (Bedrock KB Retrieve)	always	`search_guidance(query, top_k)`
`microsoft-graph`	OpenAPI target (app-only, client_credentials)	`graph_enabled` + Entra app credentials	The one Graph interface for email. `sendSharedMailboxMail` sends from the shared mailbox, called only by the platform — the approve/auto-resolve notification and the BFF's counterparty send of an analyst-approved draft, never by a model; `listSharedMailboxMessages` reads and searches it, used by `correspondence-search`. The schema also declares `getUserProfile` and `searchSharePointSites`, neither of which is reachable. See The Graph target in detail.
`correspondence-search`	Lambda (re-enters this gateway)	`graph_enabled` (shares `graph_mailbox`)	`search_correspondence(query, top)` — the model-safe mailbox read. Declares only pattern-legal property names, then assembles the OData form (`$search` double-quoted, `$top` an integer, `mailboxAddress` from `GRAPH_MAILBOX`) and calls `microsoft-graph___listSharedMailboxMessages` back through this gateway with SigV4. It re-enters rather than calling Graph directly because the Graph credential lives in the AgentCore OAuth2 provider and there is no Lambda-readable copy — so the read still passes Cedar and the interceptor. Cedar permits both the wrapper action and the inner Graph action (the inner call arrives as the wrapper's own role).
`document-extraction`	MCP server (IDP endpoint, client_credentials)	`idp_gateway_target_url` + `idp_mcp_secret_json`	`get_results(document_id)` — full IDP extraction results. The parameter is snake_case (`document_id`); `documentId` and `batch_id` both fail for a single document.

The Graph target in detail

Auth is the configured Entra app (client-credentials, auth_mode pinned in environments/recon/main.tf). The app holds admin-consented application permissions Mail.Read and Mail.Send, so both send and read are live, and a read through the gateway comes back with a normal Graph payload. The target mailbox comes from graph_mailbox / GRAPH_MAILBOX and has to be a real mailbox in the tenant.

None of these ops are confidence-gated, because OpenAPI ops carry no confidence argument for Cedar to compare against. sendSharedMailboxMail is gated at the REQUEST interceptor instead, which rejects any send arriving without a valid confirmationToken and a sendPurpose whose conditions hold (see the interceptor bullet above).

getUserProfile and searchSharePointSites are denied by Cedar, not by Graph. cedar_reads permits exactly six actions and omits both, and the policy engine denies by default, so a call returns No policy applies to the request (denied by default) and never reaches Graph. Whatever the Entra app is consented for is beside the point for these two. Don't read that denial as a missing permission grant.

sendSharedMailboxMail reaches the model on neither backend, and this one is a design choice rather than a technical limit — its argument names are all pattern-legal and it would work if offered. An earlier release did offer it (allowlisted on the harness, wrapped as send_mail on the runtime) and leaned on the interceptor to deny every model-originated send, which it reliably did: the model cannot obtain EMAIL_CONFIRMATION_TOKEN. Safe, but the wrong shape. It put a send affordance in front of a model whose every send was destined to be refused, and each refusal landed on the case trace looking like an attempted outbound email. The tool is gone from both backends now: the model writes the message into submit_proposal's email_draft, an analyst approves a specific revision, and the BFF sends that text.

listSharedMailboxMessages, as the gateway advertises it, reaches the model on neither backend. Its $-prefixed OData arguments surface as tool-schema property names and violate Bedrock's ^[a-zA-Z0-9_.-]{1,64}$ pattern, so the model always goes through a search_correspondence wrapper (the runtime's in-process one, or the correspondence-search target) and only those wrappers call the op. On the harness the raw op sits in the decorative GATEWAY_TOOLS list but is deliberately absent from the enforced ALLOWED_TOOLS. The op wants two strict argument forms, $top an integer and $search a double-quoted string, which the interceptor normalizes; verified live, a $top of "3" is coerced and the read succeeds.

One naming trap worth knowing: the runtime registers its in-process wrapper under the raw op name as a tolerant-matching alias (strands_investigator.py). So microsoft-graph___listSharedMailboxMessages in a runtime model's tool list means the clean query/top wrapper, not the raw Graph schema. There is no such alias for the send op any more — gateway_mcp.py maps no short name to it, so no agent tool can reach it even by accident.

Agent evaluation & optimization

The harness emits OTel traces, and a continuous evaluation pipeline scores them.

Online evaluation (100% sampling, 4 evaluators): the GoalSuccessRate, Helpfulness and Correctness builtins, plus a custom analyst-agreement evaluator (a code-based Lambda scoring against the lessons ledger). Online eval scores each session about 5 min after it closes, which is before any analyst decision exists, so the agreement evaluator abstains at that point.

The authoritative agreement pass is a batch re-score. Nothing schedules it weekly; it fires two ways. Every analyst decision triggers one automatically, so approving or correcting a case starts a targeted StartBatchEvaluation for that case's latest session with the agreement evaluator alone, and the metric updates within a few minutes with no manual step. The other way is the Evals-tab "Re-run evaluation" button, which re-scores the active backend's recent sessions against all evaluators.
Recommendations: managed SYSTEM_PROMPT_RECOMMENDATION and TOOL_DESCRIPTION_RECOMMENDATION over a trace window. It is backend-agnostic, since the API distinguishes only the trace source, and the prompt sent for optimization is the shared policy core s3://<assets>/system-prompt.md that both backends run (see "One prompt, two backends" below). The harness's calling contract is deliberately kept out of what the optimizer sees, so an applied recommendation can never paraphrase the submit_proposal field list into the shared core.

Two service behaviours are worth knowing about. The submitted prompt is screened by prompt-attack protection, so text reading as an injected role delimiter fails in about 2 s with ValidationException: The provided content was detected as unsafe…, and the Evals error names the prompt artifact to edit. And 5 concurrent recommendations is the account limit; an IN_PROGRESS job cannot be deleted, so a burst locks the account out for the duration.
Versioned config store: harness-configs/v<NNNN>.json in S3 plus an SSM active-pointer. A deploy writes the version's system_prompt into the shared core object (system-prompt.md) first, then moves the pointer, and rollback is just deploying the older version. That order is what makes a deploy reach both backends: the runtime container reads the prompt object and never the pointer, so the prompt write is the deploy and the pointer is the harness worker's view of it. A version with a blank system_prompt returns 409 instead of blanking the live prompt.

A version's system_prompt is a snapshot at save time and is never rewritten, so it stays an honest record of what ran. The live prompt object, though, is also writable from the Skills tab (Skills → System Prompt), which does not move the pointer, so the deployed version can stop being the live text. The list endpoint compares the two and returns liveMatchesDeployed; when that is false the row reads LIVE · edited since and the panel explains that the edited text is what both backends run. null, meaning the prompt object could not be read, shows as LIVE ? and never as agreement.

Archive, not delete. A bad version can be soft-archived out of the list (PATCH /api/recon/harness/configs {version, archived}), and "Show archived (n)" brings them back. There is no delete at all: the deployed version is both the rollback target and the drift baseline, and the documents are the record of every prompt that ever ran. Archiving the deployed version returns 409. Version numbering counts archived documents too, so a number is never reused.

Prerequisites: the harness backend active, and account-level CloudWatch Transaction Search enabled.

Client → agent trace continuity

AgentCore traces the agent side for you, but the caller is a separate trace unless the client propagates context. The SDK does not forward traceparent or baggage, so a worker invocation and the agent's own spans land as two unrelated traces. The agent-worker Lambda closes that gap with five pieces:

Piece	Where	What it does
ADOT layer `AWSOpenTelemetryDistroPython`	`enable_worker_tracing` / `otel_layer_version` (`infra/.../recon`)	supplies the `opentelemetry` packages + `/opt/otel-instrument` (`AWS_LAMBDA_EXEC_WRAPPER`). Deliberately not vendored into the shared Lambda zip, which every other Lambda uses.
`backend/recon_core/otel_client.py`	worker + harness worker	`traced(...)` custom spans around each invoke, `set_recon_baggage(...)` for item/domain/backend/session, `register_trace_propagation(client)`.
boto3 `before-send` hook	`register_trace_propagation`	injects `traceparent` + `baggage` (and forces `X-Amzn-Trace-Id` to `Sampled=1`) after SigV4, so the headers ride along unsigned and cannot invalidate the signature. The ingress path is hand-signed urllib with no botocore event system, so it takes the same headers through `invoke_via_ingress(extra_headers=…)`, merged after `signed_headers` — `ingress_invoke.py` itself stays OTel-free.
`OTEL_BAGGAGE_SPAN_ATTRIBUTE_KEYS`	set identically on all three participants: the worker Lambda, the harness definition, and the Runtime container	promotes the allow-listed baggage keys onto the agent-side spans — this is what makes `recon.item_id` / `session.id` searchable in Transaction Search. The allow-list is per-participant: the header propagates either way, but a participant without its own copy records nothing (the container runtime's spans showed only `session.id` until it got one).
Harness `environmentVariables`	`infra/modules/recon-agent-harness` (`HARNESS_ENV_JSON`)	span-noise reduction (`OTEL_PYTHON_EXCLUDED_URLS`, `OTEL_PYTHON_DISABLED_INSTRUMENTATIONS`) plus the baggage allow-list; hashed into `config_hash` so an edit is never a no-op.

Two settings are asymmetric on purpose. AWS_GENAI_CONTENT_EXTRACTION_OPT_OUT and OTEL_SEMCONV_STABILITY_OPT_IN are on for the Lambda, which emits no gen-ai content, and off for the harness, because the live online evaluators score the gen-ai content records the harness emits; opting out there would silently starve them.

Set enable_worker_tracing = false to detach the layer and make the whole client-side path inert. In a private VPC the xray interface endpoint is required, and without it spans are dropped silently.

One prompt, two backends

The agent's instructions live in one editable artifact, and both Tier-2 backends read it:

Artifact	Holds	Read by	Written by
`s3://<assets>/system-prompt.md`	the shared policy core — role, skills-as-procedures, workflow, autonomy, principles	runtime container and harness worker	Skills-tab prompt editor; config-version deploy
`s3://<assets>/system-prompt-harness.md`	the harness's calling contract only — `submit_proposal` fields, prefixed tool names	harness worker (appended after the core)	repo seed (create-only; `aws s3 cp` to update)

backend/recon_core/prompt_source.py owns the composition and fails loudly on an empty core. Switching agent_backend therefore cannot change the agent's policy, only its calling mechanics.

The split exists because two full copies of a prompt drift, and an optimizer fed a drifted copy reproduces its errors with more confidence. One core object keeps the policy single-sourced, and the harness file holds only calling mechanics, which cannot drift into policy. Both S3 seeds are ignore_changes create-only, so a repo-side edit reaches an existing environment only via aws s3 cp; editing the live text is the prompt editor's job, not Terraform's.

Frontend tabs

Tab	Purpose
Dashboard	Lifecycle status counts across all cases; click-through to filtered history
Queue	Open exceptions (PENDING / IN_PROGRESS / PROPOSED); class, confidence meter; multi-select bulk actions; click-through to case detail
Case detail	IDP document split view (section ⇄ page images + extracted fields), classification + reasoning, proposed resolution, agent trace (tool calls + evidence), approve/disapprove with comments
Skills	Browse/create/edit/delete SKILL.md files (live, ~60 s). Each tile shows tools; click opens read-only (Edit is explicit). System prompt also editable here
Lessons	Captured analyst decisions/corrections fed back to the agent
Evals	Last-7-days evaluation metrics; on-demand batch; managed recommendations; versioned harness-config (save/deploy/rollback)
Config	Toggle Tier-1 (with inline read-only source); set/disable auto-resolve threshold (rewrites Cedar); switch agent backend runtime↔harness (with inline code viewer / harness skill list); model selection

Authentication

NEXT_PUBLIC_AUTH_PROVIDER (build-time, from the auth_provider Terraform var) selects one of two providers:

okta is the Okta OIDC redirect flow (@okta/okta-auth-js), and needs okta_issuer plus okta_client_id. This is what the dev environment is deployed with.
entra is Microsoft Entra ID via MSAL. It is the Terraform variable's default, so it applies when auth_provider is unset.

UserMenu shows the signed-in user's name and a Logout button. Cognito survives only as the intake HTTP API's JWT authorizer; it is not the frontend login.

Networking

All recon backend compute runs inside the VPC on private subnets (infra/modules/network). The Lambdas are VPC-attached with a shared egress-only security group, and the AgentCore Runtime uses network_mode = VPC on the same subnets and SG. Gateway VPC endpoints cover S3 and DynamoDB; interface endpoints for ecr.api, ecr.dkr and logs handle container image refresh and logging without leaving the VPC. A single NAT gateway carries the remaining AWS API egress: Bedrock, SSM, the AgentCore control plane.

Two components sit outside that shape:

AgentCore Gateway. No gateway-level VPC attribute exists (confirmed July 2026), so both gateways are managed public endpoints behind AWS_IAM, and targets are invoked via the gateway's own IAM role.
Frontend. In the deployed dev environment the ECS/ALB tier is cost-optimized: the Fargate task runs in dedicated public subnets with assign_public_ip = true, pulling its image and reaching CloudFront and AWS APIs directly with no NAT, locked down by security groups and fronted by the ALB. The next section describes the hardened private-VPC alternative.

CloudFront is the only internet entry point, since the ALB's sole port-80 ingress is the CloudFront managed prefix list, so it carries a WAFv2 web ACL (aws_wafv2_web_acl.frontend in infra/modules/frontend-ecs) with AWSManagedRulesCommonRuleSet and a default action of allow. That default is deliberate: this is a filter in front of an already-authenticated app, not an allowlist perimeter. SizeRestrictions_BODY is overridden to count, because real config-save and proposal bodies exceed its 8 KB limit.

A CLOUDFRONT-scoped ACL has to be created in us-east-1, and the module inherits the root provider, so a lifecycle.precondition asserts var.region == "us-east-1" rather than failing later with an opaque WAF error. In private-VPC mode the ACL is count = 0, since there is no distribution to attach it to. No rate-based rule is configured on the ACL, so request-rate abuse is unmitigated at the edge.

Private VPC deployment

The dev default trades isolation for cost: public Fargate subnets and a single NAT. For a regulated or internet-restricted deployment, the frontend and backend both run entirely on private subnets with no route to an Internet Gateway, and every AWS dependency is reached over PrivateLink interface endpoints instead of the public internet. CloudFront is an optional edge layer here rather than part of the isolation: the workload isolation is identical without it, and only the ingress hop differs.

Ingress options (CloudFront optional)

Option	Ingress path	When to use
A — CloudFront (prod)	CloudFront → PrivateLink VPC origin → internal ALB	Production: global edge, WAF attachment point, managed TLS, no internet-facing ALB
B — Direct ALB (testing)	Internet-facing ALB in the two public subnets, SG locked to tester CIDRs → private Fargate task	Easier testing without CloudFront: the workloads stay exactly as private; only the ALB is reachable, and only from allowlisted IPs. Needs an ACM cert on the ALB (or HTTP for quick tests) and the ALB DNS name added to the Okta/Entra redirect URIs
C — Fully private	Internal ALB, reached via Client VPN / Direct Connect, or an SSM port-forward for ad-hoc tests	Internet-restricted environments; nothing is reachable from the internet at all

What changes vs. the dev default

Concern	Dev default (cost-optimized)	Private VPC mode
Fargate placement	Public subnets, `assign_public_ip = true`, no NAT	Private subnets, `assign_public_ip = false`, no public IP
ALB	Internet-facing, open to the CloudFront prefix list	Option A/C: internal ALB · Option B: internet-facing but CIDR-allowlisted (testing)
Egress to AWS APIs	Direct (Fargate) + single NAT (Lambdas/Runtime)	All AWS access via VPC interface endpoints (PrivateLink) + S3/DynamoDB gateway endpoints. No IGW route for the frontend; the NAT is what the endpoints make removable — the flag does not delete it (see "Enabling it")
Bedrock / AgentCore	Over NAT to public endpoints	`bedrock-runtime`, `bedrock-agentcore`, `bedrock-agentcore.gateway` (Gateway has its own PrivateLink service) and `bedrock-agent-runtime` (KB `Retrieve`) interface endpoints
Blast radius	Task can reach the internet	Task can reach only the enumerated endpoint services

Interface (PrivateLink) endpoints required

Beyond the S3 + DynamoDB gateway endpoints and the ecr.api / ecr.dkr / logs interface endpoints already provisioned, the root private_vpc = true flag (which sets the network module's enable_private_endpoints) adds the following (infra/modules/network/main.tf, _private_interface_endpoints) so nothing needs the NAT: bedrock-runtime, bedrock-agentcore, bedrock-agentcore.gateway, bedrock-agent-runtime, ssm, secretsmanager, sts, elasticloadbalancing, ecs / ecs-agent / ecs-telemetry, and xray (OTel span export — without it the VPC-attached worker drops every span while otherwise working normally). Each carries a security group allowing 443 from the workload SGs.

bedrock-agent-runtime is in the list for the KB tool. backend/kb_tool/handler.py calls Bedrock KB Retrieve through boto3.client("bedrock-agent-runtime"), a different service from bedrock-runtime (model inference), so without the endpoint search_guidance and the consult-guidance skill hang in a no-NAT deployment. states is deliberately absent.

AgentCore Gateway PrivateLink Support: AgentCore publishes three PrivateLink services, and Gateway is supported on both data and control plane:

Service name Private DNS Purpose

com.amazonaws.<region>.bedrock-agentcore bedrock-agentcore.<region>.amazonaws.com data plane (Runtime, Memory, …)

com.amazonaws.<region>.bedrock-agentcore.gateway *.gateway.bedrock-agentcore.<region>.amazonaws.com Gateway invocation

com.amazonaws.<region>.bedrock-agentcore-control bedrock-agentcore-control.<region>.amazonaws.com control plane (Runtime/Memory)

The gateway endpoint is not redundant with the data-plane one. A gateway URL is <gateway-id>.gateway.bedrock-agentcore.<region>.amazonaws.com, a subdomain the data-plane endpoint's exact-name private zone does not resolve. An earlier version of this README claimed egress-tool traffic "still leaves via the bedrock-agentcore interface endpoint's service"; that was wrong, and in a no-NAT deployment those calls had no private path at all. bedrock-agentcore.gateway is in _private_interface_endpoints for exactly this reason.

Both recon gateways use AWS_IAM/SigV4 inbound auth, so the default full-access endpoint policy works and a custom policy can scope Principal to specific IAM identities. There is an asymmetry to watch if the ingress is ever switched to OAuth/JWT: endpoint policies can only match IAM principals, so an OAuth-ingress gateway requires Principal: "*" or every call over the endpoint is denied.

Separately, gateway targets can now reach private VPC resources through a privateEndpoint / managedVpcResource block (VPC Lattice) on MCP and OpenAPI targets, so a target no longer has to be publicly reachable. Nothing here uses it: Lambda targets need no configuration and already run in the VPC, and the Graph OpenAPI target is a public third-party API. It is the supported route if the IDP MCP endpoint is ever moved inside a VPC. It does not apply to Smithy targets, and API Gateway targets need the export-as-OpenAPI workaround with routingDomain set to the API's VPCE DNS name.

Architecture (private-VPC mode)

flowchart TB
    user([Analyst / Browser])
    user -.->|"A (prod, optional): HTTPS"| cf["CloudFront (OPTIONAL)<br/>WAF / edge TLS"]
    cf -.->|PrivateLink VPC origin| alb
    user -->|"B (testing): HTTPS direct,<br/>SG allowlisted CIDRs · ACM cert"| alb
    user -.->|"C: VPN / SSM port-forward"| alb

    subgraph aws["AWS account / Region"]
      subgraph vpc["VPC (no IGW / no NAT on workload subnets)"]
        subgraph ingress["Ingress (2 AZs)"]
          alb["ALB<br/>internal (A/C) · internet-facing + CIDR allowlist (B)"]
        end

        subgraph app["Private subnets — Application"]
          fe["ECS Fargate<br/>Next.js + BFF /api/recon/*<br/>assign_public_ip = false"]
          rt["AgentCore Runtime<br/>container (network_mode = VPC)"]
          lam["Lambdas<br/>idp-hook · intake · tier1 · agent-worker ·<br/>gl · recon-status · kb · interceptor · eval-agreement"]
        end

        subgraph pl["VPC Endpoints (PrivateLink)"]
          gw["Gateway endpoints:<br/>S3 · DynamoDB"]
          ife["Interface endpoints:<br/>bedrock-runtime · bedrock-agentcore ·<br/>bedrock-agentcore.gateway · bedrock-agent-runtime ·<br/>ecr.api · ecr.dkr · logs · ssm ·<br/>secretsmanager · sts · xray ·<br/>elasticloadbalancing · ecs"]
        end
      end

      subgraph data["Regional AWS services (via PrivateLink)"]
        ddb[(DynamoDB<br/>items · cases · audit · lessons)]
        s3[(S3<br/>assets · skills · configs)]
        bedrock["Bedrock<br/>Foundation models · Knowledge Base"]
        acore["AgentCore<br/>Gateway · Memory · Policy · Identity · Evaluation"]
        cw["CloudWatch Logs / X-Ray<br/>(aws/spans, eval results)"]
      end
    end

    alb --> fe
    fe -->|SigV4| rt
    fe --> lam
    lam --> rt

    fe --> ife
    rt --> ife
    lam --> ife
    fe --> gw
    rt --> gw
    lam --> gw

    gw --> ddb
    gw --> s3
    ife --> bedrock
    ife --> acore
    ife --> cw

    classDef vpcbox fill:#eef6ff,stroke:#4a90d9;
    classDef svc fill:#f5f5f5,stroke:#999;
    classDef optional stroke-dasharray:5 5,fill:#fffbe6,stroke:#b8962e;
    class vpc,app,pl,ingress vpcbox;
    class data,ddb,s3,bedrock,acore,cw svc;
    class cf optional;

Request path. The user arrives by exactly one of three routes: (A) CloudFront → PrivateLink VPC origin → internal ALB, (B) allowlisted HTTPS straight to the ALB for testing without CloudFront, or (C) a VPN/SSM tunnel to the internal ALB. All three land on the same ECS Fargate task (Next.js plus the same-origin /api/recon/* BFF, running under the task role). The OIDC login runs in the SPA and behaves identically on every option; each origin URL, whether the CloudFront domain or the ALB DNS name, just has to be registered as a redirect URI in the IdP app.

From the task inward nothing differs between the options. BFF and agent calls reach the AgentCore Runtime over SigV4, through the ingress gateway or a direct InvokeAgentRuntime, and every AWS API call (Bedrock models, the AgentCore control and data planes, DynamoDB, S3, SSM, Secrets Manager, CloudWatch) flows through the S3/DynamoDB gateway endpoints and the interface endpoints. No workload holds a public IP, and in option B the only internet-reachable component is the CIDR-allowlisted ALB.

The diagram shows the end-state topology. private_vpc gets you the endpoints and the private frontend; removing the NAT gateway is the last manual step, described under "Enabling it".

Enabling it

One flag does it. terraform apply -var="private_vpc=true" switches the whole topology to option C in a single change, with no per-module wiring to do:

cd infra/environments/recon
terraform apply -var="private_vpc=true" -var='private_ingress_cidrs=["10.0.0.0/8"]'

What the flag does	Where
Adds the twelve interface endpoints listed above (`enable_private_endpoints`)	`infra/modules/network`
ALB becomes internal; ingress from `private_ingress_cidrs` (empty ⇒ the VPC CIDR only)	`infra/modules/frontend-ecs`
Fargate moves to the private subnets, `assign_public_ip = false`, private-endpoint SG	`infra/modules/frontend-ecs`
CloudFront is not created at all — distribution, WAF web ACL, log bucket, and origin-request policy all `count = 0`, and no public subnets / IGW route for the frontend	`infra/modules/frontend-ecs`

private_ingress_cidrs is your VPN or corporate range; leave it empty to allow the VPC CIDR only. Register the internal ALB's DNS name as an OIDC redirect URI in Okta/Entra, then reach the UI over Client VPN / Direct Connect or aws ssm start-session --document-name AWS-StartPortForwardingSessionToRemoteHost.

Three things stay manual, and the flag does none of them:

The NAT gateway survives. aws_nat_gateway.this, its EIP and public subnet, and the private route table's 0.0.0.0/0 → NAT route are unconditional in infra/modules/network/main.tf. A private_vpc = true apply therefore gives you the interface endpoints and a fully private frontend while the backend subnets still hold a default route out. Removing the NAT, and the ~$35/mo it costs, is a follow-up edit to that module. Do it only after confirming the endpoint set covers every dependency. The two easiest to miss are bedrock-agent-runtime (KB Retrieve, or search_guidance hangs) and bedrock-agentcore.gateway (every egress tool call), both now in the list.
Option A is out of reach of the flag, which removes CloudFront outright. A CloudFront → PrivateLink-VPC-origin front end has to be added back on top of the internal ALB.
Option B is a hand edit too. Keep private_vpc = false, replace the CloudFront prefix-list ingress rule on the ALB SG with your tester CIDRs, and attach an ACM certificate. Workloads stay as private as the module makes them, but no variable expresses this.

The dev environment root defaults to private_vpc = false, the cost-optimized public-subnet variant. Private mode is the hardened profile of the same modules, not a different deployment.

Cost Estimation (caution: AI generated & unverified)

Sample monthly cost for the deployed dev/demo configuration in US East (N. Virginia), on-demand pricing, no savings plans/free-tier. This is a low-volume demo profile — the always-on infrastructure (Fargate, NAT, ALB) sets the floor, and Bedrock is the largest single line even at this volume. There is no provisioned vector-store cost: the Knowledge Base uses S3 Vectors, which bills per stored vector and per query.

Assumptions

Volume is low: ~1,000 reconciliation items a month, of which ~300 escalate to the Tier-2 agent while the other ~700 auto-clear in Tier-1, at roughly 2 investigation and analyst-decision cycles per escalated item.

The frontend is one always-on ECS Fargate task (0.5 vCPU / 1 GB) behind an ALB, fronted by CloudFront, with a single NAT gateway, running 24×7.

Bedrock runs Claude Sonnet as both the agent and the LLM-judge model: ~300 investigations at ~50K input and ~3K output tokens each, plus the online-eval judges (~4 evaluators over sampled sessions).

The Knowledge Base is Bedrock-managed and backed by S3 Vectors (storage_configuration { type = "S3_VECTORS" } in infra/modules/recon-agent/main.tf), a 1024-dimension float32/cosine index over a small seed corpus embedded with amazon.titan-embed-text-v2:0. There is no OpenSearch Serverless collection and no OCU floor.

DynamoDB, Lambda, S3 and Athena are all on-demand at demo volume.

Estimated monthly cost (demo profile, us-east-1)

Service	Driver	Est. $/mo
Amazon Bedrock — Claude Sonnet	~300 investigations + eval judges (~20M in / ~1M out tokens)	~$70
NAT Gateway	1 gateway (~$0.045/hr) + data processing	~$35
CloudWatch (logs, metrics, Transaction Search spans, Logs Insights)	OTel spans + eval queries	~$25
AgentCore (Runtime/Harness, Gateway, Memory, Evaluations)	low invocation volume; consumption-priced	~$20
ECS Fargate (frontend)	1 task, 0.5 vCPU + 1 GB, 24×7	~$18
Application Load Balancer	1 ALB, low LCU	~$18
WAF (CloudFront web ACL)	1 web ACL + 1 managed rule group + low request volume	~$6
Secrets Manager / SSM / ECR / CodeBuild	few secrets, params, image builds	~$5
Bedrock — Titan embeddings + KB queries	seed corpus + retrievals	~$3
Lambda (idp-hook, tier1, worker, gl, kb, interceptor, evaluator, etc.)	demo invocations, mostly free-tier-adjacent	~$3
DynamoDB (items, cases, audit, lessons — on-demand)	low RCU/WCU	~$3
S3 (assets, skills, configs, GL, IDP page copies)	few GB + requests	~$2
CloudFront	low egress	~$2
Athena (GL queries via `search_ledger`)	small scans, $5/TB	~$1
S3 Vectors (Bedrock KB vector store)	small seed corpus; per-vector storage + query requests	~$1
Total (demo profile)		≈ $212/mo

What moves the number:

The vector store is not worth optimising. S3 Vectors has no provisioned capacity, so the KB costs roughly nothing while idle. An OpenSearch Serverless collection would have added a ~$350/mo floor (2 OCU minimum) and more than doubled this bill, which makes the storage type the one KB decision with real money attached. The ~$1 in the table is a guess at demo scale rather than a checked figure; confirm S3 Vectors rates before quoting it.
Bedrock is already the top line at ~$70, a third of the bill, and it scales with volume while the ~$71 of always-on NAT, ALB and Fargate does not. No large fixed cost is left to cut here. The one remaining lever on the floor is the NAT gateway.
private_vpc = true is the most expensive flag in the repo (see Private VPC deployment). It adds the 12 endpoints in _private_interface_endpoints, and each one gets an ENI in both private subnets (subnet_ids = aws_subnet.private[*].id, count = 2). At ~$0.01/ENI-hour that is ~$15/mo per endpoint, so ≈ +$175/mo, not the ~$75 an earlier version of this table claimed by counting one AZ. Removing CloudFront and its WAF web ACL gives back ≈ $8, and dropping the NAT gateway another $35, but the NAT removal is a manual follow-up edit and the flag alone leaves it running. Net: roughly +$167/mo, or +$132 once the NAT goes.
At production volume the shape inverts. Ten to a hundred times the traffic leaves the ~$71 infrastructure floor untouched and takes Bedrock past everything else combined.

These are rough list-price estimates for planning only. Validate them against the AWS Pricing Calculator and your actual traffic before relying on them.

License

This library is licensed under the MIT-0 License. See the LICENSE file.

Security

See CONTRIBUTING for more information.

Authors

Felix Huthmacher, Senior Applied AI Architect github - fhuthmacher

Name		Name	Last commit message	Last commit date
agent-blueprint		agent-blueprint
assets		assets
backend		backend
chatbot-app/frontend		chatbot-app/frontend
data		data
infra		infra
tests		tests
.gitignore		.gitignore
.gitleaks.toml		.gitleaks.toml
.semgrepignore		.semgrepignore
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
conftest.py		conftest.py
pyproject.toml		pyproject.toml

Service name	Private DNS	Purpose
`com.amazonaws.<region>.bedrock-agentcore`	`bedrock-agentcore.<region>.amazonaws.com`	data plane (Runtime, Memory, …)
`com.amazonaws.<region>.bedrock-agentcore.gateway`	`*.gateway.bedrock-agentcore.<region>.amazonaws.com`	Gateway invocation
`com.amazonaws.<region>.bedrock-agentcore-control`	`bedrock-agentcore-control.<region>.amazonaws.com`	control plane (Runtime/Memory)

Uh oh!

Latest commit

History

Folders and files

Repository files navigation