Optimize statistical unigram tokenizer `decode_forward` - #63

Open

aria42 wants to merge 1 commit into

JuliaText:masterfrom

aria42:faster-decode-forward

Commits on Dec 30, 2021

truncate decode_forward pass to skip spans longer than any word in SentencePieceModel vocab
aria42
committed
7931208