Skip to content

Introduce Menudet 1.0 draft for portable array and table computation - #741

Open
FrancescAlted wants to merge 196 commits into
mainfrom
improved-miniexpr
Open

FrancescAlted wants to merge 196 commits into
mainfrom
improved-miniexpr

Conversation

@FrancescAlted

@FrancescAlted FrancescAlted commented Oct 23, 2026 •

Copy link
Copy Markdown
Member

Supersedes #735.

Menudet is a little language for portable computation on arrays and tables, with Python-like syntax and native execution through miniexpr. Validated kernels can be saved and loaded without reconstructing Python functions, while preserving defined numeric semantics and logical computation groups. Menudet 1.0 remains a draft specification.

Main changes

  • Add standalone, validated kernel artifacts, @blosc2.menudet authoring with annotated or inferred dtypes, and native execution without Python reconstruction or execution fallback.
  • Define NumPy-style numerical semantics, including wrapping array integers, checked weak-scalar conversions, operand-driven floating-point types, and interpreted complex arithmetic. Support control flow, fixed-width strings, logical coordinates and ordered block reductions.
  • Add native expression and staged graph execution, parallel resident maps/reductions, and bounded readers for direct computation over compressed operands.
  • Remove NumExpr as a runtime dependency. Automatically interpret single-return expressions and prefer JIT for multi-statement programs, while honoring explicit execution settings. Unsupported optional JIT falls back before execution; execution failures propagate without replaying numerical work.
  • Improve NumPy-output evaluation with explicit output buffers, layout/casting handling, reusable evaluators and operand-free preparation caches. Reduce cold overhead through shared syntax analysis, prepared-plan reuse, fewer Menudet artifact imports and native uniform-cast reuse.
  • Execute supported DSL-backed LazyUDFs and array constructors through Menudet. Preserve logical reduction groups through persistence, partial reads and output rechunking.
  • Integrate portable persistence across LazyUDFs, frames, MessagePack and object stores. Restore validated numerical LazyExpr recipes under safe loading; retain explicit deserialize="full" for legacy Python-specific recipes and reject unsupported saves before destination writes.
  • Add CTable computed/generated transformers with explicit independent-row semantics, and regression coverage for pandas-free Arrow dictionary imports.
  • Expose explicit math-library and floating-point policies, update the pinned native dependency, and expand native/WASM qualification, documentation, examples and benchmarks.
  • Retain independent math certification with 284 samples across 40 functions, float32 and float64; these are finite-corpus checks, not universal accuracy bounds.

Performance and validation

For x + (y - 1) over 100K float64 NumPy operands on macOS arm64, with 12 configured threads and fresh NumPy outputs, preparation-cold medians from 151 samples were:

Engine Median
numexpr (comparison only) 210 µs
LazyExpr-NumPy 168 µs
evaluate 186 µs
Menudet 177 µs

Preparation/compiler caches were cleared before each cold sample; inputs were prebuilt and CPU/OS caches were not flushed. Measurements used a rebuilt native extension containing the pinned uniform-cast optimization. Compressed-output LazyExpr is excluded from this comparison.

  • Full local Python suite after merging current main: 11,796 passed, 87 skipped, including a subprocess that blocks pandas imports and the unsigned dictionary overflow regression.
  • Native uniform-cast parity tests and 15 additional native graph/portable checks passed.
  • Ruff, formatting and applicable commit hooks passed. Heavy/network tests and release-wheel qualification were not rerun for these final changes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant