Skip to content

Pull requests: huggingface/tokenizers

New pull request New
128 Open 1,005 Closed
128 Open 1,005 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

perf: simd literal matching run-pipeline-bench
#2312 opened Aug 6, 2026 by SBrandeis Contributor • Draft
[BUG] | The docs listing page for tokenizers is throwing 404
#2311 opened Aug 6, 2026 by xiaoyaoqilan Loading…
1
perf(bpe): decide the fold at load time, drop fold_by_flag
#2310 opened Aug 6, 2026 by ArthurZucker Collaborator Loading…
1
TARGET POC: bitsplit target
#2306 opened Aug 6, 2026 by ArthurZucker Collaborator Loading…
1
perf(pipeline): give the model a whole chunk of pre-tokens at a time
#2304 opened Aug 6, 2026 by ArthurZucker Collaborator • Draft
TARGET MINIMAL RUST CRATE + .tok FORMAT target
#2293 opened Aug 5, 2026 by ArthurZucker Collaborator Loading…
1
Make the encode pool opt-out behind a parallel feature
#2289 opened Aug 5, 2026 by ArthurZucker Collaborator Loading…
1
feat(node): expose the pipeline encode path over napi
#2281 opened Aug 4, 2026 by ArthurZucker Collaborator Loading…
1
feat(atomsplit): run deepseek's pre-tokenizer with no regex backend
#2265 opened Jul 29, 2026 by SBrandeis Contributor • Draft
1
Fix usize underflow in BPE / Strip decoders on empty / short input
#2245 opened Jul 27, 2026 by weixlu Loading…
3
Update fancy-regex dependency to 0.18
#2243 opened Jul 25, 2026 by musicinmybrain Contributor Loading…
1
Update monostate dependency to v1
#2242 opened Jul 25, 2026 by musicinmybrain Contributor Loading…
1
Fix panic in Strip decoder on short all-content tokens
#2239 opened Jul 24, 2026 by v-code01 Loading…
feat(pipeline+bindings): Support encoding pairs of sequences
#2237 opened Jul 23, 2026 by SBrandeis Contributor • Draft
2
implement ultra-basic, erroneous decode
#2232 opened Jul 22, 2026 by SBrandeis Contributor • Draft
1
feat(bindings): Python bindings rework, plugging PipelineTokenizer
#2229 opened Jul 22, 2026 by SBrandeis Contributor • Draft
1
bench(pipeline): add sebpop/upstream as a second reference series
#2221 opened Jul 20, 2026 by SBrandeis Contributor • Draft
2
Fix malformed BPE merges panicking during build
#2219 opened Jul 19, 2026 by 0xzr Loading…
1
Preserve backslashes in generated Python stub docstrings
#2218 opened Jul 18, 2026 by cupkk Loading…
1
5
Add a ParityBpeTrainer example and list it in the trainers API docs
#2217 opened Jul 18, 2026 by cimeister Contributor Loading…
Fix XSS in EncodingVisualizer HTML attributes
#2216 opened Jul 18, 2026 by varneeth01 Loading…
1
1
Fix dead wikitext-103 S3 link in quicktour.mdx
#2215 opened Jul 18, 2026 by RudrenduPaul Loading…
1
Fix token_to_sequence returning Some(0) for out-of-bounds index
#2214 opened Jul 17, 2026 by Osamaali313 Loading…
1
feat: multi threaded pipeline target
#2213 opened Jul 17, 2026 by McPatate Member • Draft
57
Fix Metaspace decoder dropping internal spaces in the first token
#2212 opened Jul 17, 2026 by sohumt123 Loading…
ProTip! Mix and match filters to narrow down what you’re looking for.