A production-grade, event-driven cloud platform designed to automate and streamline automotive repair shop operations. This system integrates with third-party shop management software, processes multimedia content, and delivers real-time insights to technicians and service advisors.
This platform demonstrates my approach to building scalable, resilient cloud architectures that handle complex business workflows across distributed systems. The architecture processes thousands of daily transactions while maintaining data integrity with external APIs that have strict rate limits.
- Multi-Stage Data Processing Pipeline - Orchestrated workflow that transforms raw input data through validation, enrichment, and synchronization stages
- Real-Time Media Processing - Video transcription, compression, and intelligent content pairing
- Third-Party API Integration - Resilient integration layer with OAuth token management and adaptive rate limiting
- WebSocket-Based Live Updates - Push notifications to mobile and web clients for status changes
- AI-Powered Content Generation - LLM integration for expanding technical descriptions and generating estimates
I chose an event-driven architecture over a monolithic approach for several reasons:
- Decoupled Evolution - Each processing stage can be modified, scaled, or replaced independently without affecting others
- Natural Retry Boundaries - Failed operations are isolated to specific stages, making partial retries possible without reprocessing entire workflows
- Visibility - EventBridge events create an auditable trail of system state transitions
- Cost Efficiency - Pay only for actual compute time rather than idle server capacity
The workload is inherently bursty - shops are busiest during business hours with minimal overnight activity. Serverless computing aligns costs with actual usage patterns, while eliminating operational overhead of managing server fleets.
┌─────────────────────────────────────────────────────────────────────────────┐
│ CLIENT LAYER │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Mobile App │ │ Web App │ │Chrome Extend.│ │ Edge Device │ │
│ │ (iOS/And.) │ │ (React) │ │ │ │ (Raspberry) │ │
│ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ │
└─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────┘
│ │ │ │
▼ ▼ ▼ ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ API LAYER │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ API Gateway (REST + WebSocket) │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Upload API │ │ Vehicle API │ │ Estimate API│ │ Auth/Token │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
│ ▲ │
│ │ JWT/Cognito Validation │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ Custom Lambda Authorizer (Multi-Provider) │ │
│ │ Supports: Cognito + Auth0 + Custom JWT │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ PROCESSING LAYER │
│ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ EventBridge (Event Router) │ │
│ │ │ │
│ │ Stage Transitions Failure Alerts S3 Triggers Schedules │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │
│ │ Stage 1 │────▶│ Stage 2 │───▶│ Stage 3 │ │
│ │ (Ingest) │ │ (Process) │ │ (Finalize) │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ Step Functions Orchestrator │ │
│ │ │ │
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │
│ │ │ Assign │──▶│ Verify │──▶│ Upload │──▶│ Confirm │──▶│Complete │ │ │
│ │ │ Slots │ │Pre-Send │ │ w/Rate │ │Post-Send│ │ │ │ │
│ │ └─────────┘ └─────────┘ │ Limit │ └─────────┘ └─────────┘ │ │
│ │ └─────────┘ │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌──────────────────────────────────────────────────────────────────────┐ │
│ │ SQS Queues (Work Distribution) │ │
│ │ │ │
│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ │
│ │ │ Upload Queue │ │ Estimate Queue│ │ Media Queue │ │ │
│ │ │ (FIFO) │ │ (FIFO) │ │ (Standard) │ │ │
│ │ └───────┬───────┘ └───────┬───────┘ └───────┬───────┘ │ │
│ │ │ │ │ │ │
│ │ ▼ ▼ ▼ │ │
│ │ ┌───────────────┐ ┌───────────────┐ ┌───────────────┐ │ │
│ │ │ Dead Letter │ │ Dead Letter │ │ Dead Letter │ │ │
│ │ │ Queue │ │ Queue │ │ Queue │ │ │
│ │ └───────────────┘ └───────────────┘ └───────────────┘ │ │
│ └──────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ MEDIA PROCESSING LAYER │
│ │
│ ┌────────────────────────────────────────────────────────────────────┐ │
│ │ Multi-Stage Media Pipeline │ │
│ │ │ │
│ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌───────┐ │ │
│ │ │ Ingest │──▶│Transcode│──▶│Transcribe──▶│ Match │──▶│ Concat│ │ │
│ │ │ │ │Compress │ │ (AI) │ │ Visuals │ │ │ │ │
│ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ └───────┘ │ │
│ └────────────────────────────────────────────────────────────────────┘ │
│ │ │
│ ▼ │
│ ┌──────────────────┐ │
│ │ MediaConvert │ │
│ │ (AWS Service) │ │
│ └──────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ DATA LAYER │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ DynamoDB Tables │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Queue │ │ Processing │ │ Entity │ │ Config │ │ │
│ │ │ Tables │ │ Status │ │ Tables │ │ Tables │ │ │
│ │ │ │ │ │ │ │ │ │ │ │
│ │ │ - Findings │ │ - RO Status │ │ - Vehicles │ │ - Auth Tkns │ │ │
│ │ │ - Estimates │ │ - Uploads │ │ - Customers │ │ - Rate Lmts │ │ │
│ │ │ - Media │ │ - Locks │ │ - Orders │ │ - Employees │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ S3 │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Media │ │ Static │ │ Processed │ │ Lambda │ │ │
│ │ │ Uploads │ │ Assets │ │ Outputs │ │ Deploys │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ CONTAINERIZED SERVICES (EC2) │
│ │
│ ┌─────────────────────────────────────────────────────────────────────┐ │
│ │ Docker Compose Stack │ │
│ │ │ │
│ │ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │ │
│ │ │ Long-Running│ │ Scheduled │ │ Stateful │ │ │
│ │ │ Workers │ │ Jobs │ │ Services │ │ │
│ │ └─────────────┘ └─────────────┘ └─────────────┘ │ │
│ └─────────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
The Problem: Processing a single unit of work requires multiple sequential operations (fetch, enrich, validate, sync), each with different failure modes and retry characteristics.
My Approach: I decomposed the workflow into discrete stages, using EventBridge as the choreographer. Each stage:
- Emits a "Stage N Complete" event upon success
- Emits a "Stage N Failed" event with error details on failure
- Is triggered by the previous stage's completion event
Why This Works:
- Failures are isolated - a transcription failure doesn't block data sync
- Each stage can have custom retry policies and timeouts
- Operations can be replayed from any stage without starting over
- New stages can be inserted without modifying existing code
The Problem: When new work items arrive, processing must begin immediately without polling.
My Approach: DynamoDB Streams trigger Lambda functions whenever items are inserted into queue tables. The stream captures:
- The full item content (NewImage)
- Whether it's an insert, modify, or delete
- The sequence of changes for ordering guarantees
Why This Works:
- Zero-latency trigger from data write to processing start
- Built-in ordering within partition keys
- Automatic batching for efficiency
- No separate polling infrastructure to manage
The Problem: Certain operations (like uploading to third-party APIs) must happen in order and exactly once to maintain data integrity.
My Approach: FIFO SQS queues with message group IDs based on the entity being processed:
- Content-based deduplication prevents duplicate processing
- Message groups ensure per-entity ordering
- Dead letter queues capture persistent failures for manual review
Why This Works:
- Guarantees each item is processed exactly once
- Maintains ordering when it matters (same entity)
- Parallelizes when safe (different entities)
- Failed messages don't block the entire queue
The Problem: Uploading to the third-party API requires multiple coordinated steps with verification at each stage.
My Approach: A state machine that orchestrates:
- Slot Assignment - Match local data to available slots in the external system
- Pre-Upload Verification - Validate the mapping before any API calls
- Rate-Limited Upload - Sequential API calls with enforced delays (500ms between each)
- Post-Upload Verification - Fetch back from API to confirm success
Why This Works:
- Visual debugging through the Step Functions console
- Built-in retry with exponential backoff
- Timeouts at each stage prevent runaway executions
- Graceful handling of partial success ("CompletedWithErrors" state)
The Problem: Third-party APIs have strict rate limits. Exceeding them results in temporary bans.
My Approach: A multi-layer rate limiting strategy:
- Token Bucket in DynamoDB - Track API calls per time window
- Sequential Processing - Step Functions enforce 500ms delays between calls
- Circuit Breaker - Exponential backoff when rate limit errors occur
- Per-Shop Counters - Distribute load fairly across tenants
Why This Works:
- Respects API limits while maximizing throughput
- Automatically adapts when limits change
- Prevents one tenant from consuming all capacity
- Graceful degradation under load
The Problem: Different clients use different identity providers - web app uses Cognito, mobile uses Auth0, and internal tools use custom tokens.
My Approach: A unified Lambda authorizer that:
- Inspects the token format to determine provider
- Validates against the appropriate provider
- Returns a normalized principal for downstream use
- Caches validation results to reduce latency
Why This Works:
- Single authorization logic for all APIs
- Clients don't need to change - only the auth layer
- Easy to add new providers
- Token caching reduces auth latency
The Problem: Clients need real-time updates without polling.
My Approach: API Gateway WebSocket endpoints with:
- Connection IDs stored in DynamoDB
- Topic-based subscriptions (e.g., "subscribe to order 123")
- Fan-out on state changes to relevant connections
- Connection cleanup on disconnect
Why This Works:
- True push notifications without client polling
- Scales to thousands of concurrent connections
- Cost-effective (pay per message, not per connection)
- Works through firewalls and proxies
The Problem: Video content needs transcription, compression, matching with visual templates, and concatenation.
My Approach: A staged media pipeline:
- Ingestion - S3 event triggers processing
- Transcription - Audio extraction and speech-to-text
- Compression - MediaConvert for format normalization
- Visual Matching - AI-powered matching with template library
- Concatenation - Final assembly of clips
Why This Works:
- Each stage can scale independently
- Expensive operations (transcription, compression) are isolated
- Failures in one stage don't require restarting the pipeline
- Can prioritize stages differently (e.g., rush transcription, defer compression)
Every SQS queue has a paired DLQ. After a configured number of failures, messages move to the DLQ for:
- Manual inspection
- Automated alerting
- Replay after fixes
Critical operations (especially API calls) use idempotency keys stored in DynamoDB:
- Prevents duplicate processing on retry
- Keys expire after processing window
- Enables safe retry without side effects
External API calls use a circuit breaker:
- Closed (normal) - Requests flow through
- Open (tripped) - Requests fail fast, no API calls
- Half-Open (testing) - Limited requests to test recovery
For operations that must be singleton (one at a time per entity):
- DynamoDB conditional writes for lock acquisition
- TTL-based lock expiration to prevent deadlocks
- Lock holder recorded for debugging
EventBridge rules capture failure events and route to:
- SNS topics for email/SMS alerts
- Slack webhooks for team notifications
- CloudWatch for metrics and dashboards
All Lambda functions emit structured JSON logs with:
- Correlation IDs for request tracing
- Stage identifiers for pipeline position
- Entity IDs for debugging specific items
For this workload profile (business hours heavy, overnight light):
- Lambda costs ~40% less than equivalent always-on EC2
- No capacity planning required
- Automatic scaling to zero overnight
For this access pattern (key-value lookups, no complex joins):
- On-demand pricing aligns with bursty traffic
- No connection pool management
- Global tables for future multi-region if needed
Some workloads run on EC2 with Docker:
- Long-running processes (>15 min Lambda limit)
- Stateful services requiring local storage
- Third-party tools not designed for Lambda
-
Start with events, add orchestration - Not everything needs Step Functions. EventBridge handles most workflows; reserve Step Functions for complex branching logic.
-
Design for partial failure - Third-party APIs fail randomly. Every integration point needs retry logic, timeout handling, and graceful degradation.
-
Rate limits are features, not bugs - Designing around rate limits forced better queue management and fairer resource distribution.
-
DynamoDB Streams are powerful - They're not just for replication. Using them as triggers eliminates entire categories of polling infrastructure.
-
FIFO isn't always necessary - Standard queues with idempotency keys often suffice. Reserve FIFO for true ordering requirements.
| Layer | Technologies |
|---|---|
| Compute | AWS Lambda (Python, Node.js), EC2 with Docker Compose |
| API | API Gateway (REST, WebSocket), Lambda Authorizers |
| Events | EventBridge, SQS (Standard + FIFO), SNS, DynamoDB Streams |
| Orchestration | Step Functions |
| Data | DynamoDB, S3 |
| Media | AWS MediaConvert, AI Transcription Services |
| Auth | Cognito, Auth0, Custom JWT |
| Monitoring | CloudWatch, Custom EventBridge Alerting |
| Attribute | Implementation |
|---|---|
| Scalability | Auto-scaling Lambda, on-demand DynamoDB |
| Resilience | Multi-stage retry, DLQs, circuit breakers |
| Observability | Structured logging, event-based alerting |
| Cost Efficiency | Pay-per-use compute, right-sized resources |
| Maintainability | Decoupled services, event-driven integration |
This architecture documentation describes the system design patterns and decisions. It does not include implementation details, credentials, or proprietary business logic.