Implement stateful tokenization - #59

Open

Leon Puchinger (LeonPuchinger) wants to merge 13 commits into

microsoft:mainfrom

LeonPuchinger:stateful-lexer

Commits on Mar 6, 2025

define types to allow building a stateful lexer
Leon Puchinger (LeonPuchinger)
committed
713ea05
analyze the rules in nested lexer states
Leon Puchinger (LeonPuchinger)
committed
218a1f7
Implement the stateful lexer
Leon Puchinger (LeonPuchinger)
committed
ccba6f6
Resolve naming conflict
Leon Puchinger (LeonPuchinger)
committed
5294107
apply tslint suggestions to follow project rules
Leon Puchinger (LeonPuchinger)
committed
c44c98e
Test stateful tokenization on c-style block comments
Leon Puchinger (LeonPuchinger)
committed
4c174ac

Commits on Apr 1, 2025

Commits on Apr 2, 2025

update documentation
Leon Puchinger (LeonPuchinger)
committed
6a4c084