Repository navigation
parser: keyword parameters in a paren-less lambda list, the r/i-suffix panic, and seven more one-liner gaps - #39
Merged
Merged
Conversation
…one-liner gaps
Seven refusals, each traced to one production in MRI's grammar.
1. `-> x: 1 { x }` — a paren-less lambda parameter list beginning with a keyword
parameter. `f_larglist: f_args` (parse.y v3_4_0 5216) is the SAME `f_args` the
parenthesised form takes, and `f_args: args_tail` (6473) makes a bare
`args_tail` a whole list; `args_tail` starts with `f_kwarg(f_kw)` (6371),
`f_kw: f_label arg_value | f_label` (6611), `f_label: tLABEL` (6591). The
guard that decided whether a paren-less list was present named IDENT, `*`,
`**` and `&` but not tLABEL, so the list was invisible and the `{` that
followed was read as the body opener.
2. `-3r` / `-3i` / `-3ri` — negateLiteral ended in an unchecked
`n.(*ast.FloatLit)`, and applyNumSuffix wraps the literal in RationalLit /
ImaginaryLit. Now a full switch with a failing default. Negating the innermost
numeric matches MRI's `negate_lit`, which flags the outermost node.
3. `0x10r`, `0b10i`, `0o7r`, `0d9i` were unlexed: lexRadixInt read no suffix.
The suffix scan is now one helper shared by the decimal and radix paths,
implementing MRI's number_literal_suffix (9044) — including the two rules that
were missing entirely: an `r` after an `i` is masked off (`2ri` yes, `2ir` no),
and a letter or `_` ending the scan rejects the WHOLE suffix, which is why
`2rescue nil` is `2` with a modifier rescue.
4. `def M::name(v)`, `def self::m` — `dot_or_colon: '.' | tCOLON2` (6839) in
`defs_head` (3405), so `::` is the same separator as `.`. Also `def nil.m` /
`def true.m` / `def false.m` (`var_ref: keyword_variable`, 6711). Reaches
rbconfig.rb and mkmf.rb. `def M::Bar::baz` correctly becomes a REFUSAL: the
receiver is a single `var_ref`, never a constant path, and we used to accept it
by mis-reading it as `def M` with a `::Bar::baz` body.
5. `when`, `in` and `case` now continue across a newline, as `if`/`while`/`and`
already did. MRI has no rule for it: each sets a value-expecting lexer state
and a newline there is discarded, so `case_body: k_when case_args then` (6191)
never sees a terminator it does not allow. Reaches rubygems/request.rb:125.
`k_case terms?` (6136) also makes `case when x then ... end` legal with no
terminator at all.
6. `-> (**nil) { }` / `{ |**nil| }` — `block_args_tail: f_any_kwrest ...` (5006),
`f_any_kwrest: f_kwrest | f_no_kwarg`, `f_no_kwarg: kwrest_mark keyword_nil`.
7. `BEGIN { }` / `END { }` (3020/3029 and 3193), with `BEGIN` refused anywhere but
a `top_stmts` list — the whole purpose of MRI's `stmt_or_begin` (3093) — and
still allowed inside another BEGIN, because `begin_block` reads a
`top_compstmt`. BEGIN hoists to the front of the program as MRI's
`eval_tree_begin` does; END desugars to `at_exit`, the replacement MRI's own
warning names, measured equivalent for local visibility on ruby 4.0.5.
8. `x.%(1)` and `6.0./(2)` — the four literal openers were each guarded by
`prevType != token.DEF`, which named `def` but not a dot, so `%` after `.`
opened a %Q string and `/` opened a regexp. One `atMethodName()` predicate now
carries the rule (MRI's EXPR_FNAME|EXPR_DOT, neither in EXPR_BEG_ANY).
Two premises this measurement contradicted: the leading `then` on its own line
and `-> () { ... rescue ... }` are ALREADY parsed on v0.7.0, and the "DIED" on
core/proc/parameters_spec.rb is a plain one-line parse error with exit 1 — no
panic, no hang, and no rational literal anywhere in that file.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…rom continuing its line, and add the tests
Two corrections and the test tables for the batch.
The regression the sweep caught: making `in` continue across a newline turned
`def in` + a body on the next line into `def in in_loc.slice`, and
prism/node.rb — one of the 728 stdlib files — stopped parsing. A reserved word in
a method-NAME position is a name, not the operator it usually spells: MRI's states
for that are EXPR_FNAME (after `def`) and EXPR_DOT (after a dot), where a keyword
sets EXPR_ENDFN and nothing value-expecting. The lexer now records whether each
token stood in such a position (prevFname, from the same atMethodName predicate)
and refuses to continue the line from one, which also fixes the pre-existing
`def if` / `def while` / `def and` cases of the same shape.
The over-acceptance the shape table caught: `def (1).m`, `def ("s").m`,
`def (:sym).m`, `def ([]).m`, `def (/re/).m`, `def (2r).m`, `def (`x`).m` and
`def __FILE__.m` are all "can't define singleton method for literals" in MRI,
over a fixed node list in the `singleton` production (parse.y v3_4_0 6724-6744).
14 shapes; one helper, called from both receiver paths so they cannot drift.
Ruby 4.0 MOVED that check, which is worth recording: v3_4_0 applies it only to
the `'(' expr rparen` alternative, while 4.0.5 restructured `singleton` into
`value_expr(singleton_expr)` and applies it to the bare `var_ref` spelling too.
`def __FILE__.m` is the shape that distinguishes the two grammars — legal under
v3_4_0, refused by ruby 4.0.5 under both its front ends. The oracle is 4.0.5.
One shape is knowingly left accepted: `def (:"a#{b}").m`. This parser desugars an
interpolated symbol to `"a#{b}".to_sym`, an *ast.Call indistinguishable from a
real `.to_sym` call, which MRI accepts as a receiver — refusing on that shape
would refuse a legal program.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Coverage, measured with Go 1.27.1 (1.26 disagrees about basic-block counts, so a
1.26 figure cannot witness this gate).
The parser package went 100.0% -> 99.9% with two uncovered blocks, both new:
- `def (foo)m` — the refusal when something other than `.` or `::` follows a
parenthesised receiver. Reachable, and now in the refuse table (MRI refuses it
too, under both parsers).
- negateLiteral's default arm. NOT reachable from any Ruby source: parseUnary
calls the helper only after parsePrimary consumed an INT or FLOAT token. It is
kept rather than deleted, because deleting it means restoring the trailing
`n.(*ast.FloatLit)` assertion that panicked on every `-3r`; its job is to keep
the never-panics contract for a node kind that does not belong there. A
white-box internal test witnesses that job, the same seam parseHook uses in
robustness_internal_test.go.
The lexer package's own tests never reached lexRadixInt (0.0% on main), so the two
new predicates there — atMethodName and numberLiteralSuffix — landed below 100 as
well. lexer/method_name_and_num_suffix_test.go covers both, including the two
rules a "consume r then consume i" reading gets wrong, and as a side effect
radixDigit LEAVES the below-100 list.
Result against origin/main, Go 1.27.1: parser 100.0% -> 100.0%, lexer 44.1% ->
49.9%; the below-100 LIST gains nothing and loses radixDigit; the FULL function
set gains five functions and loses none, so nothing was orphaned.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Eight refusals and one over-acceptance family, each traced to a named production in MRI's grammar, measured against
origin/main(v0.7.0,8afa48e).What the "DIED" actually was
core/proc/parameters_spec.rbscored 0 of 27 in the census and the run was reported as having died. It had not. Onorigin/main,rbgo core/proc/parameters_spec.rbprints exactly one line and exits 1:No panic, no hang — the parser refusing input, which is the least alarming of the three possibilities the brief named.
parser.Parserecovers every panic by contract, so even the-3rdefect below surfaced asinternal parser error: …, never as a crash. "DIED" is the sweep harness classifying a file that emits noRBGO_RESULTline, and a file that cannot be loaded emits none.It is also not the same defect as
-3r:core/proc/parameters_spec.rbcontains no rational or imaginary literal at all (grep '[0-9][ri]\b'→ nothing). The two were independent.The productions
-> x: 1 { x }f_larglist: f_args(5216) — the SAMEf_argsthe parenthesised form takes;f_args: args_tail(6473);args_tail: f_kwarg(f_kw) …(6371);f_kw: f_label arg_value | f_label(6611);f_label: tLABEL(6591). Also checked against 4.0'sf_largs: f_args-list(largs_tail, none)with itstail-only-args(tail)alternative, which says the same.-3r,-3i,-3rinegate_litsetsminuson the OUTERMOST node (NODE_RATIONAL and NODE_IMAGINARY each carry their own);arg: tUMINUS_NUM simple_numeric(6256).0x10r,0b10i,0o7r,0d9inumber_literal_suffix(9044), called withNUM_SUFFIX_ALLon the hex (9839), binary (9863), decimal (9892) and octal (9920) paths alike.def M::name(v),def self::mdefs_head: k_def singleton dot_or_colon def_name(3405);dot_or_colon: '.' | tCOLON2(6839);singleton: var_ref | '(' expr rparen(6711);var_ref: user_variable | keyword_variable;fname: ident_or_const | tFID | op | reswords(3755).when⏎ value;in⏎;case⏎case_body: k_when case_args then(6191) has notermsbetweenk_whenandcase_args;then: term | keyword_then | term keyword_then(4890);k_case terms?(6136) makes the terminator after a subject-lesscaseoptional. MRI has no grammar rule for the newline at all — each keyword sets a value-expecting lexer state andparser_yylex's'\n'case never emits a terminator in one.-> (**nil),{ |**nil| }block_args_tail: f_any_kwrest opt_f_block_arg(5006);f_any_kwrest: f_kwrest | f_no_kwarg;f_no_kwarg: p_kwnorest: kwrest_mark keyword_nil.BEGIN { },END { }top_stmt: keyword_BEGIN begin_block(3020);begin_block: '{' top_compstmt '}'(3029);stmt_or_begin(3093) is where "BEGIN is permitted only at toplevel" lives;stmt: k_END allow_exits '{' compstmt '}'(3193).x.%(1),6.0./(2)parser_yylextests before callingparse_percentor opening a regexp.def (1).m,def __FILE__.msingletonitself (6724-6744), over NODE_STR/DSTR/XSTR/DXSTR/REGX/DREGX/SYM/DSYM/LINE/FILE/ENCODING/INTEGER/FLOAT/RATIONAL/IMAGINARY/LIST/ZLIST.Two mechanisms were extended rather than duplicated, per the standing rule that a rule restated at several call sites drifts:
atMethodName()replaces theprevType != token.DEFtest that each of the four %-literal openers and the regexp opener carried separately. That form nameddefbut not a dot, which is defect 8 — one predicate now carries EXPR_FNAME and EXPR_DOT for all five.numberLiteralSuffix()is one helper for the decimal and every radix path, so the two rules in MRI's function that a naive reading misses are stated once: anrafter aniis masked off (2riyes,2irno), and a letter /_/ non-ASCII byte ending the scan rejects the whole suffix and rewinds. That second rule is whyp 2rescue nilis(p 2) rescue niland prints 2.checkSingletonRecv()is called from bothdefreceiver paths.The gate: the enumerated table
135 shapes, each judged by both ruby 4.0.5 front ends (
ruby -candruby --parser=parse.y -c), with every refusal's message kept and classified rather than counted by exit status.All 16 Y are over-acceptances MRI also refuses:
def M::Bar::baz(the receiver is a singlevar_ref, never a constant path — we used to read it asdef Mwith a::Bar::bazbody, defining the wrong method silently),def obj&.m(dot_or_colonhas notANDDOTalternative), and the 14 literal receivers.The 10 disagreements asserted neither:
a[foo bar do end],f(foo bar do end), and eightnot-in-exprshapes (a = not x,f(not x),p not x,!not x,def f; return not x; end,x = not not y,a[not x],x = not y and z) — prism accepts, parse.y refuses, and this parser sides with parse.y.The mandated controls all hold, base and branch, under both MRI parsers: refused —
while begin; y do end; end; end,while y do end; end,[foo bar],if x end,while x end,case x; when 1 end,p(x rescue y); accepted —a[foo bar],f(foo bar, 1),x = (y do end),[defined? foo],{k: defined? foo},p((x rescue y)).The floor: two sweeps
8afa48e)rubylibdir, 728.rblanguage/+core/— 2206*_spec.rb.rbSPEC_SHA
87b1631; the tree also holds 3815*_spec.rbin total, of which 2206 sit underlanguage/+core/.Gained:
rbconfig.rb,mkmf.rb(bothdef M::name),rubygems/request.rb(when⏎ value),core/proc/parameters_spec.rb(the 27-example file),language/fixtures/begin_file.rb(BEGIN),shared/kernel/fixtures/END.rb(END),library/bigdecimal/shared/modulo.rb(6543.21.%(137)).The 2 files still refused in the whole tree are
command_line/fixtures/bad_syntax.rbandcore/exception/fixtures/syntax_error.rb— fixtures that exist to be invalid;ruby -cexits 1 on both under both parsers, so refusing them is correct and they must stay refused.Sweep integrity. Output is indexed by input index from a file, so a dropped placeholder cannot silently produce an empty key column. Asserted per sweep: driver exit 0; rows == inputs; distinct paths == inputs; distinct indices == inputs; zero empty key columns; more than one verdict value; and a floor on the count itself.
Two of the three populations now parse completely, which makes "more than one verdict value" unsatisfiable from the real inputs alone — a saturated population and a broken probe look identical. Each sweep therefore appends three control files: two that must come back REFUSED (
def f(,x = = 1) and one that must come back OK (p 1). Every run reportscontrols=REFUSED/REFUSED/OK, so the assertion has something to bite on at 100%.Regression caught by the floor, and fixed
Making
incontinue across a newline turnedinto
def in in_loc.slice, andprism/node.rb— one of the 728 stdlib files — stopped parsing. A reserved word in a method-name position is a name, not the operator it usually spells: MRI's states for that are EXPR_FNAME and EXPR_DOT, where a keyword yields EXPR_ENDFN and nothing value-expecting. The lexer now records whether each token stood in such a position (prevFname, from the sameatMethodNamepredicate) and will not continue a line from one. That also fixes the pre-existingdef if/def while/def andcases of the same shape.Coverage, Go 1.27.1
Measured with Go 1.27.1, not the host 1.26.4, and each
cover -funcresolved in the tree that produced its profile.parser.go.radixDigit(the new lexer test is the first to reachlexRadixInt, 0.0% on main).Two new below-100 entries appeared on the first pass and were resolved by asking whether each was reachable first:
def (foo)m— the refusal when something other than.or::follows a parenthesised receiver. Reachable; it is now in the refuse table (MRI refuses it too).negateLiteral's default arm — not reachable from any Ruby source, sinceparseUnarycalls the helper only afterparsePrimaryconsumed an INT or FLOAT token. It is kept rather than deleted, because deleting it means restoring the trailingn.(*ast.FloatLit)assertion that panicked on every-3r; its job is to keep the never-panics contract for a node kind that should not arrive. A white-box internal test witnesses that job, using the same seamparseHookuses inrobustness_internal_test.go.gofmt -l .under Go 1.27.1 reports onlyparen_singleton_def_test.go, which was already non-empty onorigin/mainand is untouched here.Every new test was A/B-ed with a mutation that still compiles — 12 of them, one per fix — and each made its own test fail and pass again on revert. The first attempt at one of those mutations silently hit the wrong call site (
s///without/gpatched thedefparameter guard instead of the lambda one) and the test kept passing; that is what the A/B is for.Premises this measurement contradicted
r/iliteral.Parserecovers by contract and returnsinternal parser error: …; the crash risk was real for the AST but never for the caller.thenon its own line was already fixed on v0.7.0.if x⏎then⏎1⏎endandcase x⏎when 1⏎then 2⏎endboth parse on8afa48e.ripper/filter.rbandripper/lexer.rbare not among main's three stdlib failures.-> () { … rescue … }was already fixed too, in all three spellings (-> () {},-> {},-> (a) {}).-> x: 1 { x }is[[:key, :x]], not[[:keyreq, :x]]. Thekeyreqin the brief belongs to-> x: { x }, the spelling with no default — two different rows ofcore/proc/parameters_spec.rb.{k: not x}is not one of thenot-in-exprdisagreements. Both 4.0.5 parsers refuse it; they agree. Eight disagreeingnotshapes were found, not nine.[yield 1]is not a semantic refusal. Written inside the method that makesyieldlegal —def f; [yield 1]; end— both MRI parsers still refuse it: it is the[foo bar]rule, a command call in anargposition. The other five shapes on that list (yield,break,next, top-levelreturn,super) do become legal inside their contexts, and all five parse here.parse.yv3_4_0 is what this repo cites;ruby 4.0.5is what judges it.singletonis one production where they differ: v3_4_0 applies its literal check only to the'(' expr rparenalternative, while 4.0 restructured the rule intovalue_expr(singleton_expr)and applies it to the barevar_refspelling too.def __FILE__.mparses under v3_4_0 and is refused by 4.0.5 under both front ends. This branch follows 4.0.5 and says so in the code. 4.0 also renamed the lambda'sf_argstof_largs, which is why both grammars were checked for gap 1.Not fixed
def (:"a#{b}").mstays accepted where MRI refuses it. This parser desugars an interpolated symbol to"a#{b}".to_sym, an*ast.Callindistinguishable from a real.to_symcall — which MRI does accept as a receiver. Refusing on that shape would refuse a legal program; it needs a distinct AST node for a dsym.core/proc/parameters_spec.rbare now reachable but the pass count is unmeasured: that needs anrbgobuild,go-embedded-ruby/rubyis a different repository, and its local checkout is on another session's feature branch.🤖 Generated with Claude Code