Skip to content

JS_08_Iterators_Generators.md

Latest commit

 

History

History
137 lines (79 loc) · 21.1 KB

JS_08_Iterators_Generators.md

File metadata and controls

137 lines (79 loc) · 21.1 KB
Raw

LambdaJS — Iterators, Generators & Destructuring

Last verified against tree: 2026-08-13 (initial stamp from git history)

Part of the LambdaJS detailed-design set. This document covers the iteration protocol (GetIterator/IteratorStep/IteratorClose and the done sentinel), the lightweight fast-path iterators for arrays/strings/typed-arrays, how for-of/for-in compile (with per-iteration bindings and IteratorClose), the two-function generator state-machine transform (yield, yield*), and array/object destructuring with spread/rest.

Primary sources: lambda/js/js_runtime.cpp (iterator + generator runtime), lambda/js/js_mir_iterator.cpp (MIR iterator helpers), lambda/js/js_mir_function_class_lowering.cpp (generator state-machine emission), lambda/js/js_mir_statement_lowering.cpp (for-of/for-in), lambda/js/js_mir_expression_lowering.cpp (yield, destructuring, spread), lambda/js/js_mir_completion.cpp (abrupt completions), lambda/js/js_mir_analysis.cpp (yield counting, spill slots). Audience: engine developers. Convention: file:line references drift; confirm against symbol names.

1. Purpose & scope

Iteration in LambdaJS is built on a runtime IteratorRecord-style protocol and a compile-time generator transform that turns a generator body into a resumable state machine sharing Lambda's Item/GC runtime. Three runtime entry points — js_get_iterator, js_iterator_step, js_iterator_close — are the single ABI through which the compiler drives every iterable, and the MIR side emits calls to them via thin wrappers in js_mir_iterator.cpp. Generators reuse this protocol (a generator is its own iterable) but layer a separate JsGenerator state machine on top.

This document owns the iterator protocol, the generator state-machine transform, for-of compilation, and destructuring/spread. Promises, the microtask/event loop, modules, and await-suspension are owned by JS_09 — Async, Promises & Modules; async generators are described here as a generator mechanism, but their await-points route through the same suspension machinery JS_09 documents. The underlying value/shape representation used by iterator objects (the Map + map_kind discriminator) is in JS_03 — Value Model and JS_06 — Objects, Properties & Prototypes.

2. Iterator protocol & the done sentinel

The protocol avoids allocating a {value, done} object on every step. Instead js_iterator_step returns the value directly, and signals completion with a unique sentinel: JS_ITER_DONE_SENTINEL == 0x7F00DEAD00000000ULL (js_runtime.h:31). The high tag byte 0x7F cannot collide with any real Item — null, undefined, false, 0, or the empty string all have distinct encodings — so callers test step_result.item == JS_ITER_DONE_SENTINEL rather than reading a done property (js_runtime.cpp:27726+). The MIR side encodes this test as a single MIR_EQ against the constant in jm_emit_iterator_done_test (js_mir_iterator.cpp:23).

js_iterator_step (js_runtime.cpp:27726) dispatches fixed-layout Map/Set/string/typed-array iterators through js_is_fixed_layout_iterator (§3), generators through their typed trailing carrier, and all other iterators through the ordinary next property protocol. On any thrown exception it returns the done sentinel so the caller's exception check fires.

The for-await variant uses a parallel path: js_async_iterator_step_result (js_runtime.cpp:26839) always returns a real {value, done} object (wrapping fast-path iterators and generators) so the loop can await it before reading done via js_iterator_result_done/js_iterator_result_value (:26831).

3. Fast-path iterators

Arrays, strings, and typed arrays get a lightweight iterator with no public next property: a valid metadata-qualified Map with physical MAP_KIND_ITERATOR storage and a typed trailing JsIteratorMapCarrier { Map base; JsIterData payload; } (js_runtime.cpp:30549). JsIterData { Item source; int64_t index; int64_t length; } is precisely traced by the iterator carrier hook; it is not stored in Map.data and no fake TypeMap marker is used. Creation is js_create_array_iterator / js_create_string_iterator / js_create_typed_array_iterator (:30565+).

The fixed-layout helper is used wherever the payload layout matters; generic prototype/property APIs see a valid TypeMap and therefore do not dereference Map.data as iterator state. Stepping reads the trailing JsIterData directly: the array iterator re-reads the live observable length each step via js_array_iterator_source_length, the string iterator advances by a full UTF-8 code point with WTF-8/CESU-8 surrogate-pair combining, and the typed-array iterator checks out-of-bounds detach before each read.

js_get_iterator_impl (js_runtime.cpp:27565) chooses the fast path only when it is safe: for arrays it first consults js_check_array_sym_iterator (:27520, gated by g_array_sym_iter_ever_set) and js_array_iterator_next_is_default (:27411) — if user code overrode Array.prototype[Symbol.iterator] or %ArrayIteratorPrototype%.next, it falls back to a property-based array iterator object or calls the user @@iterator. Maps/Sets dispatch through JsCollectionData to the proper collection-iterator builtin; plain objects and elements look up __sym_1 (the interned key for Symbol.iterator, see JS_06 §9) or a bare next method. js_get_iterator caches the resolved next method as __iter_next__; js_get_iterator_lazy (:27718) skips that caching and is used by destructuring.

4. for-of / for-in compilation

jm_transpile_for_of (js_mir_statement_lowering.cpp:3557) handles both for-of and for-in (discriminated by node_type), plus for await. The for-in branch collects keys eagerly via js_for_in_keys and walks them with a liveness re-check (js_for_in_key_is_live, :3895); the rest of this section covers for-of.

for-of compilation with IteratorClose

The loop obtains its iterator with jm_emit_get_iterator (:3989) and pushes it onto the mt->for_of_iterators[32] stack (js_mir_context.hpp:375) so nested loops can be closed by break/continue to an outer label. It then opens a synthetic try context (mt->try_ctx_stack, depth-bounded at 16) whose catch target is l_iter_error and whose end target is l_forit_ret (:4087). This try context is what makes IteratorClose happen on every abrupt exit, implementing ES §13.7.5.13:

  • normal done → branch to l_end (no close needed; the iterator reported done);
  • break → l_break calls jm_emit_iterator_close then falls through (:4269);
  • throw from the body → l_iter_error saves the routed ERROR Item, closes the iterator, preserves the original payload across any close-time failure, and re-throws it (:4281);
  • return from the body → jm_transpile_return stores into forit_return_val/forit_has_return and jumps to l_forit_ret, which closes the iterator before propagating the return to an outer try or emitting the real ret (:4296).

Abrupt jumps that skip over a for-of (a labeled break/continue to an enclosing loop) are handled by jm_emit_close_intervening_iterators (js_mir_completion.cpp:30), which walks mt->loop_stack closing each entry's iterator_to_close (js_mir_context.hpp:187, set at js_mir_statement_lowering.cpp:4038).

Each iteration installs a fresh binding for a let/const loop variable. The loop variable register is created anew (not reused) for block-scoped declarations (:3659), the per-iteration value is initialized to the ITEM_JS_TDZ sentinel until the step value is assigned (:3680), and const loop variables are tagged so body writes throw. The RHS-during-TDZ rule and per-iteration closure semantics are enforced by the "P19" save/restore of mt->last_closure_* around the whole loop (:3564, :4329) — a closure created in one iteration must not write back into a sibling iteration's env. Inside generators, the iterator and loop variable are additionally registered as env-backed slots (:4001) so they survive yield suspension, as are forit_return_val/forit_has_return (:4060). If the loop variable is a pattern, the step value is fed to jm_emit_array_destructure/jm_emit_object_destructure (§7) with an exception check that routes failures to l_iter_error for IteratorClose.

5. Generator state machines

A generator function compiles to two functions: the ordinary callable (which, when invoked, calls js_generator_create to allocate a generator object) and a separate state-machine function named gen_sm_<name>_<n> with signature (Item* env, Item input, int64_t state) -> Item (js_mir_function_class_lowering.cpp:677). The state machine returns a 2-element array [value, next_state] (built by js_gen_yield_result, js_runtime.cpp:26591); next_state == -1 means done.

generator state machine

Env layout & state dispatch. All locals, params, captures, this, and arguments are collected up front and assigned env slots (jm_collect_body_locals, :627), because a register cannot survive a return-and-resume. The env reserves padding for dynamically allocated loop vars and a 128-slot spill region for temporaries that straddle a yield (:670). At entry the state machine emits a switch over state to the matching gen_state_labels[k] (:752); the label array is fixed at 64 entries (js_mir_context.hpp:460), so the yield count is capped at 63 (:621).

Yield emission. jm_transpile_yield (the JS_AST_NODE_YIELD_EXPRESSION case, js_mir_expression_lowering.cpp:12560) implements the save/return/resume/reload cycle: it allocates the next state index, saves every env-backed local into the env, returns [value, next_state], then emits the resume label gen_state_labels[next_state] and reloads every local from the env. After reload it re-initializes try-context registers (which are plain registers and do not survive the return, :12643) and checks js_gen_is_return_signal on the resumed input so a Generator.prototype.return() arriving at a suspended yield routes through enclosing finally blocks and closes any active for-of iterator (:12671). The yield expression's own value is the input parameter. Sub-expression temporaries that contain a nested yield are preserved across it with jm_gen_spill_save/jm_gen_spill_load (js_mir_analysis.cpp:293), used pervasively in destructuring and member-access lowering.

Eager param binding. ES requires parameter destructuring to run synchronously at call time, not on first .next(). The transform emits an implicit param-binding yield after binding params (:954): it saves env-backed params and returns [undefined, 1], so the body proper begins at state 1. js_generator_create (js_runtime.cpp:26684) then eagerly runs state 0 at creation time so a destructuring error throws synchronously; if it throws, the generator is marked done immediately.

Runtime driving. js_generator_next (js_runtime.cpp:26912) guards re-entrancy (executing), routes an active yield* delegate first, then calls the state-machine function and interprets its [value, next_state] result, marking done on next_state < 0. js_generator_return (:27053) and js_generator_throw inject a return/throw at the suspension point.

yield* delegation. A yield* returns a 3-element marker array [iterable, resume_state, 1] from js_gen_yield_delegate_result (:26599). js_generator_next detects the flag, calls js_get_iterator on the iterable, stores it as gen->delegate with delegate_resume, and on each subsequent .next() advances the delegate via js_yield_delegate_next_result (:26867); when the delegate reports done its return value becomes the yield* expression's value and the machine resumes at delegate_resume.

Generator pool. JsGenerator records are stored in a fixed array js_generators[JS_MAX_GENERATORS] with JS_MAX_GENERATORS == 4096; the generator object is a metadata-qualified Map-prefix carrier whose trailing generator_index selects the record. Slots are not recycled until the array is full, at which point the oldest done slot is reused.

6. Async generators

Async generators reuse the entire §5 mechanism: the same two-function transform, the same JsGenerator record (with is_async = true), and the same gen_state_labels dispatch. The difference is purely at the result-wrapping and suspension boundary. When is_async, js_generator_next/return/throw wrap their results in Promises: a yielded value goes through js_async_generator_yield_result (js_runtime.cpp:26580), which resolves the value and .then()-maps it into a {value, done} result (js_async_generator_wrap_yield_value, :26576); a done result is wrapped with js_promise_resolve. The async-generator prototype installs the async next/return/throw builtins and Symbol.asyncIterator (js_get_generator_shared_proto, :26655; js_get_async_iterator_proto, :26632).

Inside an async-generator body, await is lowered as a second kind of yield point in the same state machine — the JS_AST_NODE_AWAIT_EXPRESSION case (js_mir_expression_lowering.cpp:12743) allocates a state from the same gen_yield_index counter, with the await-count folded into the state budget (:621, js_mir_function_class_lowering.cpp:1261). The actual promise-pending check (js_async_must_suspend), the resolved-value fast path (js_async_get_resolved), and the microtask resumption are owned by JS_09 — Async, Promises & Modules; this document only notes that an async generator interleaves yield-states and await-states in one machine.

7. Destructuring & spread

array/object destructuring

Array destructuring — jm_emit_array_destructure (js_mir_expression_lowering.cpp:3539) follows ES iterator-destructuring exactly. It validates that a rest element is last and has no default (:3542), obtains a lazy iterator (js_get_iterator_lazy, no next caching), tracks an iter_done flag, and opens a synthetic try context so a thrown target binding triggers IteratorClose. Each element calls js_iterator_step; a defaulted target applies its initializer when the step value is undefined; a rest element drains the remainder with js_iterator_collect_rest (js_runtime.cpp:27994). Crucially, IteratorClose only runs when the iterator is not yet exhausted — jm_emit_iterator_close_on_exception_if_open (js_mir_iterator.cpp:60) checks the iter_done flag before calling close, per spec. In a generator, a yield inside any element spills iterator and iter_done across the suspension and parks the iterator in gen_active_iterator_slot so Generator.prototype.return() can close it.

Object destructuring — jm_emit_object_destructure (:3874) pre-initializes all target variables to undefined (assignment mode), calls js_require_object_coercible on the source, then for each property computes its key (js_to_property_key for computed/non-identifier keys), reads it with js_property_get, and binds the target. There is no iterator and no IteratorClose — object patterns are plain property reads.

Object rest — { ...rest } collects the exclude-key list at compile time into a small js_alloc_env buffer and calls js_object_rest(src, exclude_keys, exclude_count) (js_runtime.cpp:26499). The runtime enumerates keys via js_reflect_own_keys (preserving the spec own-property key order — integer indices ascending, then string keys in insertion order, then symbols), skips excluded keys, copies only own enumerable properties (checking each descriptor's enumerable flag), and skips deleted-sentinel values. Strings are handled by index.

Spread — in call arguments and array/object literals, spread is materialized by jm_build_spread_args_array (js_mir_function_collection_class_inference.cpp:2744): it allocates a fresh array and, for each JS_AST_NODE_SPREAD_ELEMENT, converts the operand to an array with js_iterable_to_array (js_runtime.cpp:28007 — fast paths for typed arrays, generators, and Map/Set; falls back to the iterator protocol otherwise) and appends each element. Generator spill keeps the accumulating array alive across a yield inside a spread operand (:2752).

Known Issues & Future Improvements

  1. Max 64 yield/await states per generator. gen_state_labels is a fixed [64] array (js_mir_context.hpp:460) and the yield count is clamped to 63 (js_mir_function_class_lowering.cpp:621); a generator with more than 63 syntactic yield/await points silently loses resume labels. jm_transpile_yield guards this with "returning undefined instead of crashing" when next_state >= 64 (js_mir_expression_lowering.cpp:12582). Improvement: size the label array from the pre-scanned count.

  2. Yield-in-destructuring relies on conservative spill heuristics. Destructuring lowering spills iterator/iter_done around yields based on jm_has_yield subtree scans (js_mir_expression_lowering.cpp:3634+), and jm_count_yields is acknowledged to over-count resume points (js_mir_function_class_lowering.cpp:1208), so unused state labels are emitted defensively at the end of the body to avoid dangling-label link crashes. Yields hidden in patterns may also be under-counted relative to jm_count_yields, which is exactly why the next_state >= 64 fallback exists.

  3. for-of early-exit .return() gaps for fixed-layout iterators. Built-in fixed-layout iterators have no return method, so js_iterator_close is a no-op for them — correct for built-ins, but a user iterator is honored through the generic property path. IteratorClose is also bounded by the depth-16 synthetic try-context stack (js_mir_statement_lowering.cpp:4049); deeper nesting silently skips the close.

  4. Async-generator await modeled as yield. await and yield share one state counter and one state-machine function (§6), with the await-suspension semantics living in JS_09. This conflation keeps the transform simple but means the await/yield state budget is shared against the same 63-state cap.

  5. Fixed generator pool, no GC reclamation. js_generators[4096] (js_runtime.cpp:26562) never frees a slot; exhaustion only recycles the oldest done slot. Long-running programs that create more than 4096 live generators, or churn through them, can collide indices or fail allocation (js_generator_create logs "exceeded max generators"). Improvement: tie generator records to GC lifetime.

  6. Prototype override fallback. If a built-in iterator prototype is overridden, construction falls back to a shape-backed iterator object so the user-visible next property and custom prototype are honored; the fixed-layout carrier remains the default path.

Appendix A — Source map

File Responsibility (this doc)
lambda/js/js_runtime.cpp js_get_iterator/js_iterator_step/js_iterator_close, JsIterData + fast-path iterators, JsGenerator + js_generator_create/next/return/throw, yield* delegation, js_object_rest, js_iterable_to_array, done sentinel.
lambda/js/js_mir_iterator.cpp MIR wrappers: jm_emit_get_iterator(_lazy), jm_emit_iterator_step, jm_emit_iterator_done_test, jm_emit_iterator_close[_on_exception[_if_open]].
lambda/js/js_mir_function_class_lowering.cpp Generator/async-generator two-function transform, env layout, state dispatch, eager param-binding yield.
lambda/js/js_mir_statement_lowering.cpp for-of/for-in/for await lowering, synthetic try context, per-iteration binding + TDZ, IteratorClose targets.
lambda/js/js_mir_expression_lowering.cpp yield/yield* emission, await state point, jm_emit_array_destructure/jm_emit_object_destructure, object-rest call.
lambda/js/js_mir_completion.cpp Abrupt-completion cleanup, jm_emit_close_intervening_iterators.
lambda/js/js_mir_analysis.cpp jm_count_yields, jm_gen_spill_save/load, jm_has_yield.

Appendix B — Related documents