Repository navigation
fix(asr): seam-gap repair probes a short window around every inter-word gap - #1061
Conversation
…rd gap The v3 CoreML export drops short phrases mid-window as well as at seams, concentrated in non-English speech: on ten-minute Italian and German oral-history excerpts 16 and 5–14 phrases per file, English near zero. They sit in 0.5–1.4 s gaps between words, below the pass's 1.5 s threshold, and a full-length probe window re-drops them exactly as the original chunk did; a window of the gap plus 0.8 s either side recovers every phrase tested. - Probe window is gap ± 0.8 s first; the full-length placements stay as fallbacks for gaps of 1.5 s and more, the seam-drop class the pass was built for. - Default minimum gap 0.5 s (the clamp floor); CLI --seam-gap-min. - Probe budget scales with the audio: 12 per minute, at least 32. - Gaps inside a word (between pieces, or before a lone apostrophe) are not probed. - A splice is accepted on median token confidence ≥ 0.7, not a per-token floor, which threw away "Are you ready? Just a little bit." for one "a" at 0.38. - spliceCandidate completes a word the gap end cut through, drops a completion that runs past the gap end (the probe disagreeing with the tail word), treats an in-word apostrophe as part of the word, and dedupes whole words punctuation-, case- and diacritic-insensitively, or by position alone within two frames. Cost (v3, M-series, CLI best of three against main): Italian ten minutes 2.47 s → 4.12 s, German 2.72 → 3.29, English debate 1.56 → 2.15, 20-minute English conversation unchanged.
Alex-Wengg
left a comment
Alex-Wengg
left a comment
There was a problem hiding this comment.
Thanks, the short ±0.8 s probe window is well motivated, and every splice rule has a test that reproduces its case (the 35 tests in SeamGapRepairTests and ChunkProcessorUnspacedScriptTests pass locally, and the branch merges cleanly into main).
My main concern is that two of the new rules also change behaviour for the ≥ 1.5 s seam-gap class that #758 was built and validated on (inline comments below). Please run the Earnings-22 benchmark you offered on this branch, since the default minimum gap also drops from 1.5 s to 0.5 s.
| // A gap inside a word (slow "mille … novecento" between the | ||
| // pieces of "1929", or "E" … "'" written for "È") has nothing | ||
| // to splice: only gaps before a word start are probed. | ||
| if let safeIds = spliceSafeTokenIds, |
There was a problem hiding this comment.
This skip applies to every gap, not just the short ones. spliceSafeTokenIds contains punctuation-only pieces, and they are excluded here, so a gap followed by "." is never probed. For a long seam drop like "…to the [store and bought milk].", the period lands right after the gap, and on main that gap is probed.
Could this be limited to gaps shorter than fullWindowMinGapFrames, so the ≥ 1.5 s class keeps its existing behaviour?
| if !candidate.isEmpty { return candidate } | ||
| // A low-confidence candidate is a hallucination over noise, not | ||
| // a recovery; try the next placement. | ||
| if !candidate.isEmpty, isConfidentSplice(candidate) { return candidate } |
There was a problem hiding this comment.
The median-confidence gate now also applies to the full-length cold-start and gap-centred placements, which previously accepted any non-empty candidate. Those placements target the low-SNR seam drops, where confidence is likely to run lower, and that class was not re-measured with a 0.7 floor.
Could the gate apply only to the short window? If not, an Earnings-22 or other English long-form WER comparison against main would show whether it costs any recoveries there.
| // stutters ("I I") sit beyond the tolerance. | ||
| func echoesNeighbor(_ range: Range<Int>, _ neighbor: (text: String, timestamp: Int)) -> Bool { | ||
| let distance = abs(candidate[range.lowerBound].timestamp - neighbor.timestamp) | ||
| if distance <= samePositionFrames { return true } |
There was a problem hiding this comment.
On the tail side, tailWord.timestamp is gapEndFrame (the gap skip guarantees next is a word start), and in-gap tokens start at gapEndFrame - 2 or earlier. So a genuine word starting exactly 2 frames (160 ms) before the tail word is always dropped, whatever its text. Short function words ("e", "a", "di", "und") can land right there in the languages this PR targets.
Would < samePositionFrames, or requiring the texts to differ only a little for the position-only match, be enough for the "Meu"/"Mio" case?
| ) -> (text: String, timestamp: Int) { | ||
| let touching = wordNeighbor(in: stream, from: index, step: step, vocabulary: vocabulary) | ||
| guard let safeIds = spliceSafeTokenIds, | ||
| let touchingIndex = stream.firstIndex(where: { $0.timestamp == touching.timestamp && $0.token == touching.token }) |
There was a problem hiding this comment.
Nit: this scans the whole stream again to find the token wordNeighbor already found, once per probe, and matches it by timestamp and token. Having wordNeighbor return its index would make this O(1) and avoid the matching.
| let windowEnd = min(windowStart + length, totalSamples) | ||
| guard windowEnd > windowStart else { continue } | ||
| let windowAudio = try readSamples(offset: windowStart, count: windowEnd - windowStart) | ||
| placements.append((windowStart, windowAudio, windowEnd >= totalSamples)) |
There was a problem hiding this comment.
Nit: the short window is decoded with isLastChunk: false unless it reaches the end of the file, so the decoder skips its final flush. That seems harmless because the padded tail is handled by the dedupe, but a short comment saying so would help the next reader.
| parsed.dualDecodeArbitration = true | ||
| case "--no-seam-gap-repair": | ||
| parsed.seamGapRepair = false | ||
| case "--seam-gap-min": |
There was a problem hiding this comment.
A few small things with this flag:
- It is missing from the help text (
--no-seam-gap-repairis listed around line 1195). - A value that does not parse falls back to the default without any message.
- Values below 0.5 are clamped to 0.5 by
ASRConfigwithout telling the user.
A help line that mentions the 0.5 s floor would cover most of this.
… gaps On a 13 s Earnings-22 seam gap (4474506, 50.9-64.1 s) the short window, tried first, recovered only the gap's back half; the residual front kept the probed start, so no later pass re-probed it and 16 words that main recovers were lost. Multi-second gaps now try main's placements first (cold start at the gap, then centred) and fall back to the short window; shorter gaps use the short window only. Also applies the swift-format 6.1 line breaks CI asks for.
…bes only long seam gaps (#1098) Closes #1031. ### Why is this change needed? #1032 fixed the issue's first finding, the splice guard. The second finding was a question: should tdtJa resolve `melChunkContext` to off? It should. Today's short-gap seam repair (#1061) also adds a tdtJa-specific problem. I measured it on real read speech: 500 JSUT basic5000 clips (BASIC5000_4501–5000, 16 kHz), joined into 31 min with their transcripts as the reference. A second copy trims each clip's edge silence so more seams land in speech. CER is NFKC, without punctuation or spaces; "drops" are runs of 5 or more deleted reference characters. | tdtJa on main b20d209 | joined: CER / drops | trimmed: CER / drops | |---|---|---| | default (mel context on), seam repair off | 23.48% / 65 | 22.29% / 67 | | default (mel context on) | 10.37% / 7 | 9.56% / 3 | | `--no-mel-context` | 8.90% / 2 | 8.52% / 2 | | `--no-mel-context --seam-gap-min 1.5` | 8.49% / 3 | 8.19% / 2 | | **this PR, defaults** | **8.49% / 3** | **8.19% / 2** | Before #1061, the no-mel path already scored 8.49% / 8.17%. #1061's 0.5 s gap floor added 19 insertions on the joined file, and every one is re-heard neighbour text ("モデルモデル", "ころはころは", "ノーベルノーベル"). Its short-gap rules (skip gaps inside a word, complete a cut word, drop re-heard bordering words) need word boundaries, and tdtJa's unspaced vocabulary has none: with the splice guard off, every piece counts as a word. ### What changes? - `resolvedMelChunkContext(for:)` resolves to `false` for `.tdtJa`. An explicit `melChunkContext:` still wins. - `ChunkProcessor.seamGapMinGapFrames`: when there's no word guard (`spliceSafeTokenIds == nil`), the repair pass probes only gaps of 1.5 s or more. Vocabularies with a guard keep the configured floor. - Docs and CLI help updated. v3 output is byte-identical on three Earnings-22 calls (4469088, 4474506, 4483937): text and word timings. The tdtJa default is also faster here: 3.6 s against 6.9 s for the 31 min file. 190 chunk, seam and config tests pass, including the new ones. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
Closes #1059.
Problem
Parakeet v3 (CoreML, 15 s windows) drops short phrases mid-window, not only at seams, and the drops concentrate in non-English speech. Measured on the first ten minutes of six recordings plus 20 minutes of English conversation (gap scan → re-decode of every speech-bearing inter-word gap ≥ 0.5 s):
Examples: "Ora si parla un po' in inglese, are you ready?" and "E ora si cambia, se parla un po'" (the second straddles a chunk end). The seam-gap repair pass (#758/#761) never saw them: its minimum gap was 1.5 s and the drops sit in 0.5–1.4 s gaps, and when it did probe, a 15 s window cold-started at the gap or centred on it re-dropped the same phrase. The palettized int8 encoder (#760) is one of the two mechanisms;
int8-v2moves the errors around more than it removes them (Italian +34/−18 words, German +133/−88).Change
seamGapRepairMinGapSeconds0.5 s; new CLI flag--seam-gap-min.spliceCandidate: completes a word the gap end cuts through ("Dicev"|"ano"), drops a completion that runs past the gap end (the probe hearing the tail word differently: "nasceamo" vs "parliamo"), treats an in-word apostrophe as part of the word ("l'infanzia"), dedupes whole words (neighborWord) punctuation/case/diacritic-insensitively ("è" vs "E'") and by position alone within 2 frames ("Meu" for "Mio").ChunkProcessorUnspacedScriptTestsupdated for the new signature; docs updated.Results (v3,
--mel-context)Both example recordings recover their phrases. Italian ten-minute excerpt: 21 phrases inserted, all plausible Italian. English: two files unchanged, 20-minute conversation two "um"s, a debate five short crosstalk recoveries and one fragment. Output of this branch matches the same change on 0.17.4 within a handful of words on every file.
Cost, CLI best of three vs
main:A probe costs a full model window (~45 ms) however short its audio, so the cost is proportional to the gap count, which is why English pays almost nothing.
Note on the ladder note in LongTranscription.md ("a window sized to each gap plus 2 s of context was tried and lost"): that measurement was WER on English long-form with Phonon-2's 5 s ladder function, where mid-window drops are rare; this change targets the non-English drop class with a tighter window and a confidence gate, and keeps the previous placements for the long gaps that measurement covered. Happy to run the Earnings-22 benchmark if you want the WER number on this branch.
Maintainer changes
mainrecovers ("identified from time to time in the company's public statements…") were lost. Gaps ≥ 1.5 s now trymain's full-length placements first (cold start at the gap, then centred), with the short window as the fallback. Shorter gaps use the short window only. The order now lives inseamGapProbePlacements, with two tests.main, including fix(asr): recover dropped opening speech in long-form v3 decoding #1056. The conflict inrepairSeamGapskeeps this PR's word-level edge dedupe with an optional lead word for fix(asr): recover dropped opening speech in long-form v3 decoding #1056's leading gap, and fix(asr): recover dropped opening speech in long-form v3 decoding #1056's leading-window placement.Measured with
mainmerged (macOS 27, M5 Pro, v3 defaults):The 24 h file reproduces #1059:
main's 32-probe budget runs out at 16 min 45 s, leaving 621 speech-bearing gaps unprobed; the scaled budget probes them. The FLEURS read speech shows no mid-window drops, so the non-English gain isn't reproduced on public audio here. 170 seam/chunk tests pass.