AWS Serverless Metadata Workflow · Reliable metadata processing without servers
An event-driven AWS pipeline that turns S3 uploads into normalized DynamoDB records through a Python Lambda, with deterministic identity, duplicate-safe writes, failure routing, monitoring, infrastructure as code, and automated tests.File uploads need a reliable metadata trail
An object landing in storage is only the start of a workflow. Downstream systems need a consistent record of where the object lives, its type, size, checksum, and upload time. The processor also has to handle URL-encoded object keys, repeated event delivery, retries, and failed invocations without silently producing duplicate or missing records.
The goal was to build that processing path with managed AWS services, keep the runtime small, and make the operational behavior inspectable through tests, alarms, and documented recovery procedures.
S3 events become normalized DynamoDB records
An Amazon S3 object-created event invokes a Python 3.12 Lambda function. The handler validates the event, URL-decodes the object key, and calls HeadObject to retrieve metadata without downloading or parsing the file body. It then normalizes that information into a DynamoDB record containing the bucket, key, size, content type, ETag, timestamps, and source details.
Event and data path
- A file is uploaded to the configured S3 bucket.
- S3 publishes an object-created event to Lambda.
- Lambda validates the event and retrieves object headers.
- A deterministic SHA-256 record ID is generated from stable object identity fields.
- DynamoDB stores the normalized record using a conditional write.
Why this design
- No always-on application server to provision or maintain
- No file-body download when metadata alone is required
- Deterministic identity makes repeated event delivery predictable
- AWS SAM keeps infrastructure reviewable and reproducible
- Least-privilege IAM limits the function to the resources it needs
Duplicate delivery and failures are designed into the workflow
AWS event delivery is asynchronous, so retries and duplicates are normal operating conditions—not edge cases to ignore. The workflow creates a deterministic RecordId and uses DynamoDB's attribute_not_exists(RecordId) condition so a repeated event does not create a second metadata row.
Failure handling
- Two asynchronous retry attempts
- One-hour maximum event age
- Encrypted SQS destination for exhausted failures
- Documented queue inspection and redrive procedure
Operational visibility
- CloudWatch alarms for Lambda errors and throttles
- An alarm for messages visible in the failure queue
- Optional SNS notifications for operators
- Structured logs with request and object identifiers
From internship architecture to a public, independently verifiable implementation
I originally built the workflow architecture during an AWS Support Engineering internship in isolated training and project environments. I later reconstructed and expanded it as this public repository so the design, code quality, tests, infrastructure, and operational decisions could be reviewed directly.
- Designed the event-driven S3, Lambda, and DynamoDB data path
- Implemented the Python handler and deterministic idempotency strategy
- Defined the AWS resources, permissions, encryption, and retention behavior in SAM
- Added retry handling, the SQS failure destination, alarms, and optional notifications
- Built unit tests for success, malformed events, duplicate delivery, and AWS failures
- Created CI checks, architecture documentation, deployment guidance, and an operator runbook
The public repository contains no confidential Amazon source code, customer data, production account information, or support-access material. Its GitHub Pages presentation is a separate documentation site; it does not claim that a live AWS stack is currently attached to the page.
Code, tests, infrastructure, and operations are open for review
The repository had 114 commits when this write-up was prepared. Its August 3, 2026 verification report records 11 passing tests with 100% statement and branch coverage. The continuous-integration workflow runs Python compilation, Ruff linting, branch-coverage tests, static-site validation, SAM validation, and a SAM build.
- Live project: Explore the architecture, implementation, and verification evidence
- GitHub repository: Review the source, SAM template, tests, workflows, diagrams, and documentation
- Runtime: Python 3.12 on ARM64 with 256 MB memory and a 30-second timeout
- Data protection: S3 and DynamoDB encryption, S3 versioning and public-access blocking, plus DynamoDB point-in-time recovery
- Deployment model: AWS SAM and CloudFormation with retained stateful resources
- Honest limitation: The public implementation reconstructs and expands the original architecture; it is not a release of internal Amazon code or a currently connected production AWS account.