Skip to content

Allow non-ASCII characters in identifiers#3028

Draft
ksss wants to merge 1 commit into
ruby:masterfrom
ksss:ksss/allow-non-ascii-identifiers
Draft

Allow non-ASCII characters in identifiers#3028
ksss wants to merge 1 commit into
ruby:masterfrom
ksss:ksss/allow-non-ascii-identifiers

Conversation

@ksss

@ksss ksss commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

Ruby accepts identifiers with non-ASCII code points — a class name like Ú (U+00DA) or Ω (U+03A9) is a valid constant, and a method name like únicos or 日本語 is a valid local identifier. The RBS lexer, however, only recognised identifiers when the surrounding characters were ASCII ([a-zA-Z0-9_]), so writing the same names in a signature file failed to parse. See rubocop-on-rbs#151 for the linter-side motivation.

This PR:

  1. Extends the lexer's identifier grammar so any valid multibyte character can appear as either the leading or a continuation code point.
  2. Aligns TypeName#kind on the Ruby and Rust sides with the same Unicode Uppercase semantics the lexer now uses.

Lexer

rbs_next_char already maps every non-ASCII code point to a fixed sentinel U+30EB, so a single mb_char = [ル] class covers all multibyte characters. word and the identifier alias are widened to include mb_char, so existing rules like [a-z] word* and "@" [a-zA-Z_] word* automatically accept trailing multibyte code points without changing their token kind.

Two new rules handle a multibyte leading code point:

mb_char word*       { return rbs_next_mb_ident_token(lexer, 0, false); }
"_" mb_char word*   { return rbs_next_mb_ident_token(lexer, 1, true); }

rbs_next_mb_ident_token reads the real bytes of the identifier's first code point and asks the encoding's isupper_char whether it belongs to Unicode's Uppercase property — the same test Ruby's rb_sym_constant_char_p uses to decide constant vs. local. That picks tUIDENT vs. tLIDENT (or tULIDENT vs. tULLIDENT for the _-prefixed variants).

Depends on #2983

The double-sentinel design introduced in #2983 (valid multibyte → U+30EB, invalid byte → U+FFFD) is what makes this change safe: mb_char matches only U+30EB, so an invalid UTF-8 byte inside an identifier still surfaces as an ErrorToken rather than being silently absorbed into the token.

Behaviour changes

Given the semantics above, the following now parse:

  • class ServicioÚltimaVez, class Ωmega (Unicode uppercase → tUIDENT)
  • def enviar_últimas_interacciones , def 日本語 (any letter → tLIDENT)
  • @日本語 , @@クラス変数 , $グローバル

The following are still rejected, matching Ruby:

  • class αlpha — Greek α is Unicode lowercase, so the token is tLIDENT and the parser rejects it as a class name.
  • class 日本語 — kanji is Other_Letter (a local identifier in Ruby), same story.
  • Identifiers containing an invalid UTF-8 byte still surface as ErrorToken.

TypeName#kind

TypeName#kind previously classified Ú and 日本語 as :class only because it fell through the default branch. αlpha also landed on :class for the same accidental reason, which was wrong.

Both sides now use the same Unicode Uppercase property the lexer uses, keeping ASCII on a fast path:

  • Ruby (lib/rbs/type_name.rb) checks [A-Z] / [a-z] first, then \p{Uppercase} for non-ASCII code points.
  • Rust (rust/ruby-rbs/src/type_name.rs) checks the leading byte with is_ascii_uppercase when it is ASCII, else decodes one code point with chars().next() and calls char::is_uppercase.

ASCII names — the overwhelming majority — pay the same single comparison as before. Non-ASCII names pay one code point decode plus a Unicode table lookup.

src/lexer.c is regenerated with re2c 4.3.

Ruby accepts identifiers with non-ASCII code points — a class name like
`Ú` (U+00DA) or `Ω` (U+03A9) is a valid constant, and a method name like
`únicos` or `日本語` is a valid local identifier. The RBS lexer, however,
recognised identifiers only when the surrounding characters were ASCII
(`[a-zA-Z0-9_]`), so writing the same names in a signature file failed
to parse. rubocop-on-rbs issue ruby#151 tracks this on the linter side.

Extend the lexer's identifier grammar to accept any valid multibyte
character as either the leading or a continuation code point. The lexer
already maps every non-ASCII code point to a fixed sentinel (U+30EB) via
`rbs_next_char`, so a single `mb_char = [ル]` class covers all
multibyte characters. When the leading character is multibyte, the token
is dispatched to `rbs_next_mb_ident_token`, which reads the real bytes
of the identifier's first code point and asks the encoding's
`isupper_char` whether it belongs to Unicode's `Uppercase` property —
the same test Ruby's `rb_sym_constant_char_p` uses to classify a
constant. That decides `tUIDENT` vs. `tLIDENT` (or `tULIDENT` vs.
`tULLIDENT` for the `_`-prefixed variants).

Depends on the invalid-byte-to-U+FFFD sentinel introduced in ruby#2983:
the `mb_char` class matches U+30EB (valid multibyte) but not U+FFFD
(invalid byte), so an invalid UTF-8 byte inside an identifier still
surfaces as an ErrorToken rather than being silently swallowed.

Align `TypeName#kind` on both the Ruby and Rust sides with the same
Unicode semantics. Ruby now falls through to `\p{Uppercase}` for
non-ASCII code points after the ASCII fast paths, and Rust decodes the
first code point with `char::is_uppercase` only when the leading byte
is non-ASCII. ASCII names — the overwhelming majority — keep their
current single-comparison cost on both sides.

test/rbs/type_parsing_test.rb's `test_parse__byte_range_incorrect`
documented an incidental side effect where starting `byte_range` in
the middle of a multibyte character produced an ErrorToken. That was
only true because non-ASCII was rejected wholesale; the lexer already
rounds `start_pos` up to the next character boundary, so the parse now
succeeds on the next valid identifier. Rewrite the test to describe
the actual behaviour and to demonstrate how `require_eof: true` still
catches the leftover trailing token.

src/lexer.c is regenerated with re2c 4.3.
@ksss ksss added this to the RBS 4.1 milestone Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant