Skip to content

feat(lexer): record regexp encoding flags n/u/e/s - #23

Merged
tannevaled merged 1 commit into
mainfrom
wave18/usen
Sep 6, 2026
Merged

tannevaled merged 1 commit into
mainfrom
wave18/usen

Conversation

@tannevaled

Copy link
Copy Markdown
Contributor

What

The regexp trailing-flag loop recorded only i/m/x/o and silently discarded the encoding modifiers n/u/e/s. That dropped information is exactly what rbgo needs to build a Regexp with the right encoding, and it has blocked regexp encoding conformance across two waves.

This records n/u/e/s into Flags in source order, for both regexp forms:

  • /…/flags — lexRegexp
  • %r{…}flags — lexPercentRXS

Any other trailing lowercase letter is still consumed harmlessly (unchanged behavior).

MRI reference

MRI records an ARG_ENCODING for each encoding letter (rb_char_to_option_kcode in re.c, driven from regx_options in parse.y):

letter encoding
n ASCII-8BIT (BINARY), ARG_ENCODING_NONE
u UTF-8, ARG_ENCODING_FIXED
e EUC-JP, ARG_ENCODING_FIXED
s Windows-31J (SJIS), ARG_ENCODING_FIXED

i/m/x remain the onig option bits and o remains "interpolate once"; these are unchanged.

Changes

  • lexer/lexer.go: append n/u/e/s in both trailing-flag loops (lexRegexp and lexPercentRXS); doc comment updated.
  • token/token.go, ast/ast.go: Flags doc comments now note the encoding flags and source-order semantics.
  • percent_rxs_test.go: TestRegexpEncodingFlagsStillDropped → TestRegexpEncodingFlagsRecorded (asserts /x/n→n, /x/u→u, /x/e→e, /x/s→s, and order for /a/mn, /a/un); plus TestPercentRegexpEncodingFlagsRecorded locking the %r{…} path.

Verification

GOWORK=off go build ./..., go vet ./..., and go test ./... all green; the four touched files are gofmt-clean.

🤖 Generated with Claude Code

The regexp trailing-flag loop previously collected only i/m/x/o and
silently discarded the encoding modifiers n/u/e/s, which blocked rbgo
regexp encoding conformance. MRI records an ARG_ENCODING for each of
these (rb_char_to_option_kcode in re.c): n=ASCII-8BIT, u=UTF-8,
e=EUC-JP, s=Windows-31J. Append them to Flags in source order for both
the /…/ (lexRegexp) and %r{…} (lexPercentRXS) forms; any other trailing
lowercase letter is still consumed harmlessly.

Update the Flags doc comments on token.Token and ast.RegexpLit, and
replace the old TestRegexpEncodingFlagsStillDropped with
TestRegexpEncodingFlagsRecorded (plus a %r variant).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@tannevaled
tannevaled merged commit 3139fc6 into main Sep 6, 2026
11 checks passed
@tannevaled
tannevaled deleted the wave18/usen branch September 6, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant