Warning
π§ This repository is under active development. Watch the repo, monitor branches and issues, and check the Changelog for the latest updates.
π§ Navigation:
π΅ Home | Vision LLM Theory | UI | Deployment | CDK Stacks | Runtime | S3 Files | Lambda Specialists | Prompting System
Broad Agentic Document Generative Extraction & Recognition System
BADGERS transforms document processing through vision-enabled AI and deep layout analysis. Unlike traditional text extraction tools, BADGERS understands document structure and meaning by recognizing visual hierarchies, reading patterns, and contextual relationships between elements.
Traditional document processing tools extract text but lose context. They can't distinguish a header from body text, understand table relationships, or recognize that a diagram explains the adjacent paragraph. BADGERS solves this by:
- ποΈ Preserving semantic structure - Maintains document hierarchy and element relationships
- ποΈ Understanding visual context - Recognizes how layout conveys meaning
- π Processing diverse content - Handles 21+ element types from handwriting to equations
- π€ Automating complex workflows - Orchestrates multiple specialized specialists via an AI agent
Use cases: research acceleration, compliance automation, content management, accessibility remediation.
A single React + Express app is both the testing workbench and the deployment/ops console β the same code runs locally via npm run dev or on ECS behind Cognito OIDC. Tabs are role-gated: the Testing row below is visible to all users, while an admin-only Deploy row (Stacks, Specialists, S3 Configs, Deploy Tags) is not pictured here. See the UI README for the full tab and role breakdown.
The deployment menu tracks the eight ordered steps β Lambda layers, foundational infrastructure, prompts/manifests/schemas, specialist Lambdas, gateway, runtime, UI image, UI ECS service β per deployment ID and stack suffix. Steps can be run individually, all at once, or resumed from wherever the last run stopped.
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β AgentCore Runtime β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
β β PDF Analysis Agent (Strands) β β
β β - Claude Sonnet 4.5 with Extended Thinking β β
β β - Session state management β β
β β - MCP tool orchestration β β
β βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
βΌ
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β AgentCore Gateway β
β - MCP Protocol (2025-03-26) β
β - Cognito JWT Authentication β
β - Semantic tool search β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β
ββββββββββββββββββββΌβββββββββββββββββββ
β β β
βΌ βΌ βΌ
βββββββββββββββ βββββββββββββββ βββββββββββββββ
β Lambda β β Lambda β β Lambda β
β Specialist β β Specialist β β Specialist β
β (26 tools) β β β β β
βββββββββββββββ βββββββββββββββ βββββββββββββββ
β β β
ββββββββββββββββββββΌβββββββββββββββββββ
βΌ
βββββββββββββββ
β Bedrock β
β Claude β
βββββββββββββββ
- π User submits a document with analysis instructions
- π§ Strands Agent (running in AgentCore Runtime) interprets the request
- π§ Agent selects tools from a library of specialists via MCP Gateway
- π Agent opens a job on the first specialist call, tagging every invocation in the run with the job and document it belongs to
- β‘ Lambda specialists (standardized and domain-specific functions, including container-based) process document elements using Claude vision models, each recording its own outcome
- π Results aggregate with preserved structure and semantic relationships
| Component | Technology |
|---|---|
| π€ Agent Framework | Strands Agents |
| π Agent Hosting | Amazon Bedrock AgentCore Runtime |
| πͺ Tool Gateway | Amazon Bedrock AgentCore Gateway (MCP Protocol) |
| π§ Foundation Model | Claude Sonnet 4.5 (via Amazon Bedrock) |
| β‘ Compute | AWS Lambda (modular specialist functions, including container-based) |
| π¦ Storage | Amazon S3 (configs, prompts, outputs) |
| π Job Tracking | Amazon DynamoDB (document β job β subtask state) |
| π₯οΈ UI Hosting | Amazon ECS Express Gateway service (in a VPC) |
| π Auth | Amazon Cognito (OIDC + PKCE for the UI, OAuth 2.0 M2M for the Gateway) |
| ποΈ IaC | AWS CDK (Python) |
| π Observability | CloudWatch Logs, X-Ray Transaction Search |
| π Cost Tracking | Bedrock Application Inference Profiles |
| Specialist | Purpose |
|---|---|
πΈ pdf_to_images_converter |
Convert PDF pages to images |
π·οΈ classify_pdf_content |
Classify document content type |
π full_text_specialist |
Extract all text content |
π table_specialist |
Extract and structure tables |
π charts_specialist |
Analyze charts and graphs |
π diagram_specialist |
Process diagrams and flowcharts |
π layout_specialist |
Document structure analysis |
π₯ decision_tree_specialist |
Medical/clinical document analysis |
π¬ scientific_specialist |
Scientific paper analysis |
βοΈ handwriting_specialist |
Handwritten text recognition |
π’ handwriting_math_specialist |
Handwritten mathematical notation recognition |
π» code_block_specialist |
Extract code snippets |
ποΈ metadata_generic_specialist |
Generic metadata extraction |
ποΈ metadata_mads_specialist |
MADS metadata format extraction |
ποΈ metadata_mods_specialist |
MODS metadata format extraction |
π keyword_topic_specialist |
Extract keywords and topics |
π§ remediation_specialist |
PDF accessibility remediation (container, content stream tagging + structure tree builder) |
π page_specialist |
Single page content analysis |
π§± elements_specialist |
Document element detection |
π§± robust_elements_specialist |
Enhanced element detection with fallbacks |
ποΈ general_visual_analysis_specialist |
General-purpose visual content analysis |
βοΈ editorial_specialist |
Editorial content and markup analysis |
πΊοΈ war_map_specialist |
Historical war map analysis |
π edu_transcript_specialist |
Educational transcript analysis |
π correlation_specialist |
Correlate multi-specialist results per page |
πΌοΈ image_enhancer |
Image enhancement and preprocessing |
- βοΈ AWS CLI configured with credentials
- π¦ AWS CDK v2 (
npm install -g aws-cdk) - π³ Docker (running)
- π Python 3.12+
- β‘ uv
./deploy.shThat is the whole command. deploy.sh asks which deployment to work on β listing anything
it finds in .deploy-state/, or offering to start a new one β and then presents a menu.
It is resumable and every step is idempotent, so re-run it after a failure and it picks up
where it stopped.
DEPLOYMENT_IDis not read from the environment. It is always chosen interactively, because a value left exported in your shell silently targets another deployment's stacks.
Pick option 9 for a full deployment, or 12 to run only what is still outstanding.
You can jump straight to one option β ./deploy.sh 6 β and the deployment is still chosen
interactively first.
The eight steps:
| # | Step | What it does |
|---|---|---|
| 1 | Lambda Layers | foundation, PDF processing, Poppler/qpdf |
| 2 | Foundational Infra | S3, Cognito, DynamoDB, IAM, ECR, Inference Profiles, X-Ray, Memory, VPC |
| 3 | Upload Config | prompts, manifests and schemas to the config bucket |
| 4 | Specialist Lambdas | container images, then the Lambda stack (26 specialists) |
| 5 | Gateway | AgentCore MCP Gateway, records the Gateway URL |
| 6 | Runtime | builds and pushes the agent image, then deploys the Runtime |
| 7 | UI β Build | generates ui/.env from Cognito, builds the bundle and image |
| 8 | UI β Deploy | ECS Express Gateway service, forces the rollout, waits for it |
Plus 9 full deployment, 12 resume, 10 status, 11 reset state (deletes nothing in AWS), 0 exit.
Step 8 asks once whether the UI should be publicly reachable. That answer is fixed for the life of the VPC β see Network exposure.
For the full procedure, prerequisites in depth, and every environment variable, see the Deployment Guide.
DEPLOYMENT_ID is a short label you choose β lowercase, starting with a letter, 16
characters or fewer. deploy.sh generates a three-character random STACK_SUFFIX once and
persists both in .deploy-state/{DEPLOYMENT_ID}.json:
- Stack names are
BADGERS-{Name}-{DEPLOYMENT_ID}-{suffix}β for exampleBADGERS-S3-dev-a1b - Resource names carry both parts β for example
badgers-config-dev-a1b, and SSM parameters under/badgers-dev-a1b/
Because both are unique per deployment, several deployments can coexist in one account and region. Stack names include the deployment id as well as the suffix so each stack is self-describing β tooling reads a deployment's identity off the stack name, and a mistyped id matches no stacks instead of resolving someone else's. The state file also tracks which steps completed, which is what makes the script resumable.
./destroy.shLike deploy.sh it asks what to tear down, but it discovers deployments from
CloudFormation rather than from .deploy-state/ β a state file can be deleted while the
stacks are still live. You are then required to type the DEPLOYMENT_ID to confirm.
It empties the S3 buckets, deletes the ECS Express service and the AgentCore runtime before the VPC (CloudFormation cannot delete a VPC while any ENI is still attached), sweeps leftover ENIs, destroys every stack in reverse dependency order, verifies they are gone, and only then schedules the KMS key for deletion so its alias is freed for redeployment. A teardown that leaves stacks standing exits non-zero and says so rather than reporting success.
If a VPC stack still gets stuck on a lingering ENI:
DEPLOYMENT_ID=dev STACK_SUFFIX=a1b ./destroy.sh --vpc-cleanup-onlyTo tear down by hand when the script cannot run, follow Manual Teardown in the Console β the stack deletion order matters, and two resources have to be removed before any stack.
βββ deployment/
β βββ app.py # CDK app entry point
β βββ stacks/ # CDK stack definitions
β βββ lambdas/code/ # Specialist Lambda functions
β βββ runtime/ # AgentCore Runtime container
β βββ s3_files/ # Prompts, schemas, manifests
β βββ badgers-foundation/ # Shared specialist framework
βββ ui/ # BADGERS UI (React + Express, runs locally or deployed via Docker)
β βββ src/ # React components (testing + admin tabs, role-gated)
β βββ server/ # Express API server (testing + admin routes, OIDC auth)
β βββ Dockerfile # Container image for AWS deployment
βββ pyproject.toml
BADGERS uses Lambda layers shared across specialist functions:
ποΈ Foundation Layer (layer.zip)
- Built via
deployment/lambdas/build_foundation_layer.sh - Contains the specialist framework (7 Python modules)
- Includes dependencies: boto3, botocore
- Includes core system prompts used by all specialists
layer/python/
βββ foundation/
β βββ specialist_foundation.py # π― Main orchestration class
β βββ bedrock_client.py # π Bedrock API with retry/fallback
β βββ configuration_manager.py # βοΈ Config loading/validation
β βββ image_processor.py # πΌοΈ Image optimization
β βββ message_chain_builder.py # π¬ Claude message formatting
β βββ prompt_loader.py # π Prompt file loading (local/S3)
β βββ response_processor.py # π€ Response extraction
βββ config/
β βββ config.py
βββ prompts/core_system_prompts/
βββ *.xml
π Poppler Layer (poppler-qpdf-layer.zip)
- PDF rendering library for
pdf_to_images_converter - Built via
deployment/lambdas/build_poppler_qdf_layer.sh
Each specialist follows the same pattern using SpecialistFoundation:
# Lambda handler (simplified)
def lambda_handler(event, context):
# 1οΈβ£ Load config from S3 manifest
config = load_manifest_from_s3(bucket, "full_text_specialist")
# 2οΈβ£ Initialize foundation with S3-aware prompt loader
specialist = SpecialistFoundation(...)
# 3οΈβ£ Run analysis pipeline
result = specialist.analyze(image_data)
# 4οΈβ£ Save result to S3 and return
save_result_to_s3(result, session_id)
return {"result": result}The analyze() method orchestrates:
- πΌοΈ Image processing - Resize/optimize for Claude's vision API
- π Prompt loading - Combine wrapper + specialist prompts from S3
- π¬ Message building - Format for Bedrock Converse API
- β‘ Dynamic token estimation - Score image complexity and set token budget (when enabled)
- π€ Model invocation - Call Claude with retry/fallback logic
- β Response processing - Extract and validate result
Prompts are modular XML files composed at runtime:
s3://config-bucket/
βββ core_system_prompts/
β βββ prompt_system_wrapper.xml # π Main template with placeholders
β βββ core_rules/rules.xml # π Shared rules for all specialists
β βββ error_handling/*.xml # β οΈ Error response templates
βββ prompts/{specialist_name}/
β βββ {specialist}_job_role.xml # π€ Role definition
β βββ {specialist}_context.xml # π Domain context
β βββ {specialist}_rules.xml # π Specialist-specific rules
β βββ {specialist}_tasks.xml # β
Task instructions
β βββ {specialist}_format.xml # π Output format spec
βββ wrappers/
βββ prompt_system_wrapper.xml
The PromptLoader composes the final system prompt:
<!-- prompt_system_wrapper.xml -->
<system_prompt>
{core_rules} <!-- π Injected from core_rules/rules.xml -->
{composed_prompt} <!-- π§© Injected from specialist prompt files -->
{error_handler_general}
{error_handler_not_found}
</system_prompt>Placeholders like [[PIXEL_WIDTH]] and [[PIXEL_HEIGHT]] are replaced with actual image dimensions at runtime.
Each specialist has a manifest file in S3:
// s3://config-bucket/manifests/full_text_specialist.json
{
"tool": {
"name": "analyze_full_text_tool",
"description": "Extracts text content maintaining reading order...",
"inputSchema": {
"type": "object",
"properties": {
"image_path": { "type": "string" },
"session_id": { "type": "string" },
"audit_mode": { "type": "boolean" }
},
"required": ["image_path", "session_id"]
}
},
"specialist": {
"name": "full_text_specialist",
"enhancement_eligible": true,
"model_selections": {
"primary": "us.anthropic.claude-sonnet-4-5-20250929-v1:0",
"fallback_list": [
"us.anthropic.claude-haiku-4-5-20251001-v1:0",
"us.amazon.nova-premier-v1:0"
]
},
"max_retries": 3,
"prompt_files": [
"full_text_job_role.xml",
"full_text_context.xml",
"full_text_rules.xml",
"full_text_tasks_extraction.xml",
"full_text_format.xml"
],
"max_examples": 0,
"analysis_text": "full text content",
"expected_output_tokens": 6000,
"output_extension": "xml"
}
}Key configuration features:
- π Model fallback chain - Primary model with ordered fallbacks
- π Retry logic - Configurable retry count per specialist
- π§© Prompt composition - List of XML files to combine
- π Tool schema - MCP-compatible input schema for Gateway
- πΌοΈ Enhancement eligible - Flag indicating specialist benefits from image preprocessing (used by
image_enhancertool)
Global settings (from environment or defaults):
{
"max_tokens": 8000,
"temperature": 0.1,
"max_image_size": 20971520, # 20MB
"max_dimension": 2048,
"jpeg_quality": 85,
"throttle_delay": 1.0,
"aws_region": "us-west-2"
}When enabled, BADGERS estimates the optimal max_tokens per image based on visual complexity, reducing cost on simple documents and avoiding truncation on dense ones. The scorer runs on the already-processed image bytes β no extra I/O.
Four metrics are combined into a complexity score: text pixel ratio, grayscale entropy, edge density, and color standard deviation. The score maps to a token budget (8K / 12K / 16K / 24K).
Enabling: Toggle "Dynamic Token Estimation" in the chat UI, or set the Lambda environment variable DYNAMIC_TOKENS_ENABLED=true.
Tuning: Add a dynamic_tokens block to an specialist manifest to customize weights and thresholds:
"dynamic_tokens": {
"weights": {
"text_ratio": 0.2,
"entropy": 0.3,
"edge_density": 0.3,
"color_std": 0.2
},
"thresholds": [
{"max_score": 0.20, "max_tokens": 8000},
{"max_score": 0.30, "max_tokens": 12000},
{"max_score": 0.45, "max_tokens": 16000},
{"max_score": 1.00, "max_tokens": 24000}
]
}Observability: When active, logs report the estimated budget, actual token usage, and utilization percentage for calibration.
BADGERS uses Application Inference Profiles to enable cost allocation and usage monitoring. The system maps model IDs to profile ARNs at runtime:
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β Inference Profile Flow β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β 1. CDK deploys InferenceProfilesStack β
β ββ> Creates ApplicationInferenceProfile for each model β
β β’ badgers-claude-sonnet-{id} (US) β
β β’ badgers-claude-haiku-{id} (US) β
β β’ badgers-claude-opus-{id} (US) β
β β’ badgers-nova-premier-{id} (US) β
β β
β 2. Runtime receives profile ARNs as environment variables β
β ββ> CLAUDE_SONNET_PROFILE_ARN, CLAUDE_HAIKU_PROFILE_ARN, etc. β
β β
β 3. At invocation, bedrock_client.py maps model_id β profile ARN β
β ββ> "us.anthropic.claude-sonnet-4-5-*" β $CLAUDE_SONNET_PROFILE_ARN β
β β
β 4. Bedrock invoked with profile ARN (enables cost tracking) β
β ββ> Falls back to model ID if no profile configured β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
Model ID to environment variable mapping:
| Model Pattern | Environment Variable |
|---|---|
*claude-sonnet-4-5* |
CLAUDE_SONNET_PROFILE_ARN |
*claude-haiku-4-5* |
CLAUDE_HAIKU_PROFILE_ARN |
*claude-opus-4-6* |
CLAUDE_OPUS_PROFILE_ARN |
*nova-premier* |
NOVA_PREMIER_PROFILE_ARN |
Option 1: Use the Wizard (Recommended)
cd local_testing
npm run devThe Specialist Creation Wizard is available as the π§ Create Specialist tab in the UI.
Option 2: Manual Creation
- π Create prompt files in
deployment/s3_files/prompts/{specialist_name}/ - π Create manifest in
deployment/s3_files/manifests/{specialist_name}.json - π Create schema in
deployment/s3_files/schemas/{specialist_name}.json - β‘ Create Lambda code in
deployment/lambdas/code/{specialist_name}/lambda_handler.py - π Register in
deployment/stacks/lambda_stack.py - π Redeploy:
cdk deploy BADGERS-Lambda-{id}-{suffix} BADGERS-Gateway-{id}-{suffix}
If your AWS organization uses strict SCPs that deny cross-region Bedrock operations, you may see:
AccessDeniedException: ... is not authorized to perform: bedrock:InvokeModelWithResponseStream
on resource: arn:aws:bedrock:::foundation-model/anthropic.claude-* with an explicit deny
in a service control policy
BADGERS defaults to regional (us.anthropic.*) inference profiles which avoid cross-region routing. If you previously deployed with global.anthropic.* profiles, redeploy after pulling the latest code.
After a fresh deployment, the first model invocation may fail with:
AccessDeniedException: Model access is denied due to IAM user or service role is not authorized
to perform the required AWS Marketplace actions (aws-marketplace:ViewSubscriptions,
aws-marketplace:Subscribe)
The IAM stack now includes aws-marketplace:ViewSubscriptions and aws-marketplace:Subscribe permissions. If you see this error on an older deployment, redeploy the IAM stack. As a workaround, manually invoke the model once in the Bedrock console playground to trigger the Marketplace subscription.
Customers are responsible for making their own independent assessment of the information in this Guidance. This Guidance: (a) is for informational purposes only, (b) represents AWS current product offerings and practices, which are subject to change without notice, and (c) does not create any commitments or assurances from AWS and its affiliates, suppliers or licensors. AWS products or services are provided "as is" without warranties, representations, or conditions of any kind, whether express or implied. AWS responsibilities and liabilities to its customers are controlled by AWS agreements, and this Guidance is not part of, nor does it modify, any agreement between AWS and its customers.
- Randall Potter
- Amazon Bedrock Developer Experience - Foundation model choice and customization
- Anthropic's Claude in Amazon Bedrock - Claude Opus 4.6, Sonnet 4.5, Haiku 4.5 hybrid reasoning models
- Claude Sonnet 4.5 in Amazon Bedrock - Most intelligent model for coding and complex agents
- Claude Opus 4.6 in Amazon Bedrock - Tool search, extended thinking, and agent capabilities
- Amazon Nova Foundation Models - Nova Micro, Lite, Pro, Premier - frontier intelligence
- Using Amazon Nova in AI Agents - Nova as foundation model for agents
- Amazon Bedrock AgentCore Overview - Build, deploy, and operate agents at scale
- AgentCore Gateway Guide - Set up unified tool connectivity
- AgentCore Gateway Blog - Transforming enterprise AI agent tool development
- AgentCore Runtime - Secure serverless hosting for AI agents
- Lambda Layers Overview - Managing dependencies with layers
- Python Lambda Layers - Working with layers for Python functions
- Adding Layers to Functions - Layer configuration and management
- OAuth 2.0 Grants - Authorization code, implicit, and client credentials
- M2M Authorization - Scopes, resource servers, and machine-to-machine auth
- M2M Security Best Practices - Monitor, optimize, and secure M2M authorization
- CloudWatch + X-Ray Integration - Enhanced application monitoring
- Cross-Account Tracing - Distributed tracing across accounts
- AWS Observability Best Practices - Logs, metrics, and traces
- S3 as Data Lake Storage - Central storage platform best practices
- S3 Performance Optimization - Design patterns for optimal performance
- Amazon Kiro Overview - Agentic IDE for spec-driven development
- Kiro with AWS Builder ID - Sign in and get started with Kiro
- Nova Act IDE Extension - Accelerate AI agent development in Kiro
- Production-Ready AI Agents at Scale - Kiro as part of the agent development ecosystem






