Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ podwhisperer/
│ ├── index.ts # Lambda handler (durable workflow)
│ ├── types.ts # Shared types
│ ├── steps/ # Pipeline step implementations
│ │ ├── fill-timing-gaps.ts
│ │ ├── llm-refinement.ts
│ │ ├── replacement.ts
│ │ └── segments-normalization.ts
Expand Down Expand Up @@ -100,7 +101,7 @@ podwhisperer/
2. **Pipeline Lambda** (durable execution) orchestrates the workflow:
- Sends job to SQS queue
- Waits for callback from GPU worker
- Applies post-processing steps (replacement, LLM, normalization)
- Applies post-processing steps (fill timing gaps, replacement, LLM, normalization)
- Generates captions (VTT, SRT, JSON)
- Sends EventBridge notification
3. **GPU Worker** (ECS container on Managed Instances):
Expand All @@ -124,7 +125,7 @@ podwhisperer/

**Configuration** (`packages/config/index.ts`):
- Zod schemas for all pipeline configuration
- Exported types: `PipelineConfig`, `TranscriptionConfig`, etc.
- Exported types: `PipelineConfig`, `TranscriptionConfig`, `FillTimingGapsConfig`, etc.
- Used by both CDK (at synth time) and Lambda (at runtime)

## Important Implementation Notes
Expand Down
29 changes: 22 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ flowchart TD
CheckExists{Output Exists?}
SQS[Send to SQS]
WaitCallback[Wait for Callback]
FillGaps[Fill Timing Gaps]
Replacement[Replacement Rules]
LLM[LLM Refinement]
Normalize[Segments Normalization]
Expand Down Expand Up @@ -67,10 +68,10 @@ flowchart TD

S3Upload --> EB --> CheckExists
CheckExists -->|No| SQS --> Poll
CheckExists -->|Yes| Replacement
CheckExists -->|Yes| FillGaps
Poll --> Download --> Transcribe --> Align --> Diarize --> Upload --> Callback
Upload --> Raw
Callback --> WaitCallback --> Replacement --> LLM --> Normalize --> Finalize --> Captions --> Notify
Callback --> WaitCallback --> FillGaps --> Replacement --> LLM --> Normalize --> Finalize --> Captions --> Notify
Finalize --> Refined
Captions --> VTT & SRT & JSON
```
Expand All @@ -88,7 +89,15 @@ The WhisperX container runs on GPU-enabled ECS Managed Instances:
5. **Diarization** - Identifies and labels different speakers
6. **Upload** - Saves raw transcript to S3 `output/` prefix

### Step 1: Replacement Rules (Optional)
### Step 1: Fill Timing Gaps

WhisperX sometimes produces words with missing `start`/`end` timestamps, particularly for words containing numbers or currency symbols (e.g., `"15"`, `"$8"`, `"$0.25"`). This step fills those gaps using character-proportional interpolation between neighboring anchor words, giving all downstream steps clean timing data.

- Gaps are distributed proportionally based on word character length
- Padding is applied dynamically based on the segment's speech rate
- Filled words are marked with `score: 0` to distinguish them from original timestamps

### Step 2: Replacement Rules (Optional)

Apply regex or literal string replacements to fix common transcription errors:

Expand All @@ -101,23 +110,23 @@ Apply regex or literal string replacements to fix common transcription errors:
}
```

### Step 2: LLM Refinement (Optional)
### Step 3: LLM Refinement (Optional)

Uses Amazon Bedrock to improve transcript quality:

- Fixes spelling and grammar errors while preserving meaning
- Identifies speakers by name when possible (e.g., "SPEAKER_00" becomes "Luciano")
- Validates suggestions to prevent aggressive rewrites

### Step 3: Segments Normalization
### Step 4: Segments Normalization

Splits long transcript segments into caption-friendly chunks:

- Respects character and word limits per segment
- Splits at natural punctuation boundaries
- Handles speaker changes within segments

### Step 4: Caption Generation
### Step 5: Caption Generation

Generates multiple caption formats from the refined transcript:

Expand All @@ -129,7 +138,7 @@ Optional features:
- Word-by-word highlighting (underline, bold, or italic)
- Speaker name prefixes (always, when speaker changes, or never)

### Step 5: Notification
### Step 6: Notification

Sends an EventBridge event when the pipeline completes, including:

Expand Down Expand Up @@ -270,6 +279,12 @@ The pipeline will automatically trigger and generate outputs in `s3://$BUCKET/ou
| `skipIfOutputExists` | boolean | `false` | Skip transcription if output already exists |
| `hfTokenSsmPath` | string | `"/podwhisperer/hf_token"` | SSM parameter path for HuggingFace token |

### Fill Timing Gaps

| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `enabled` | boolean | `true` | Enable fill-timing-gaps step |

### Replacement Rules

Array of replacement rules, each with:
Expand Down
37 changes: 37 additions & 0 deletions lambdas/pipeline/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ import {
import { SendMessageCommand, SQSClient } from '@aws-sdk/client-sqs'
import { PipelineConfigSchema } from '@podwhisperer/config'
import type { EventBridgeEvent } from 'aws-lambda'
import { fillTimingGaps } from './steps/fill-timing-gaps'
import { llmRefinement } from './steps/llm-refinement'
import { applyReplacements } from './steps/replacement'
import { normalizeSegments } from './steps/segments-normalization'
Expand Down Expand Up @@ -167,6 +168,42 @@ const handler = async (event: S3EventBridgeEvent, context: DurableContext) => {

// refining steps

// Fill timing gaps — fills missing word timestamps before any other processing
await context.step('fill-timing-gaps', async () => {
if (!pipelineConfig.fillTimingGaps.enabled) {
context.logger.info('Fill timing gaps disabled, skipping step')
return
}

// Load the transcript from S3
const response = await s3.send(
new GetObjectCommand({
Bucket: BUCKET_NAME,
Key: transcriptProcessingKey,
}),
)
const bodyStr = await response.Body?.transformToString()
if (!bodyStr) {
throw new Error('Empty response body from S3')
}
const transcript = JSON.parse(bodyStr) as WhisperxResult

// Fill timing gaps
const stats = fillTimingGaps(transcript, pipelineConfig.fillTimingGaps)

// Save the updated transcript back to S3
await s3.send(
new PutObjectCommand({
Bucket: BUCKET_NAME,
Key: transcriptProcessingKey,
Body: JSON.stringify(transcript),
ContentType: 'application/json',
}),
)

context.logger.info('Filled word timing gaps', { stats })
})

// Replacement rules
await context.step('replacement-rules', async () => {
if (typeof pipelineConfig.replacementRules === 'undefined') {
Expand Down
Loading