Skip to content

Infer exact rows for #table expressions - #286

Merged
Alex Vallone (amv146) merged 25 commits into
masterfrom
alexvallone/table-row-inference
Aug 27, 2026
Merged

Infer exact rows for #table expressions#286
Alex Vallone (amv146) merged 25 commits into
masterfrom
alexvallone/table-row-inference

Conversation

@amv146

@amv146 Alex Vallone (amv146) commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Why

PQLS already inferred a schema for static #table calls, but it discarded literal row values. As a result, later operations such as numeric item access, field projection, and table concatenation could not use row data that was known at inspection time.

This PR retains exact rows when it is safe to do so and keeps a schema-only table when the schema is known but the rows are not.

Behavior

Input Inferred result
#table({"Name"}, {{"Betty"}}) Closed DefinedTable with Name: Any and the exact row
#table(type table [Name = text], rows) with dynamic rows Schema-only DefinedTable with Name: text
Known columns with malformed, incompatible, or more than 100 rows Schema-only DefinedTable
Dynamic or open columns Primitive table

List-form columns continue to use Any; row contents do not infer or widen column types. Explicit type table [...] schemas retain their declared field types, and exact rows are kept only when their values are compatible with those declarations.

Downstream operations

  • Numeric item access returns the exact retained row, reports an out-of-range exact index as None, or returns a record derived from the known schema when rows are unavailable.
  • Field projection projects both the schema and every retained row. If an open table projects an undeclared field, the field is Any and exact rows are dropped.
  • Table concatenation combines closed schemas, unions shared field types, makes one-sided fields nullable, and fills missing row values with null. Exact rows are retained only when both operands have them and the combined total is at most 100 rows; otherwise the combined schema is preserved without rows.
  • Operations involving an unknown/open schema or a primitive table operand continue to return primitive table.

For example:

#table(type table [A = number], {{1}})
    & #table(type table [B = text], {{"two"}})

produces a schema equivalent to [A = nullable number, B = nullable text] and exact rows equivalent to {[A = 1, B = null], [A = null, B = "two"]}.

Non-goals

Inferring column types from list-form row contents is intentionally deferred to a follow-up change.

Dependency

This uses @microsoft/powerquery-parser@2.0.0, where DefinedTable.rows is optional: omitted rows mean "schema known, rows unknown," while [] means an exact empty table.

Validation

  • npm ci --registry https://registry.npmjs.org
  • npm run build
  • npm run lint
  • npm test (776 passing)

alexvallone added 3 commits August 18, 2026 14:21
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 9839e176-efce-4a70-811b-7e2668cdb7f3
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 9839e176-efce-4a70-811b-7e2668cdb7f3
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 9839e176-efce-4a70-811b-7e2668cdb7f3
alexvallone added 2 commits August 18, 2026 14:36
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 9839e176-efce-4a70-811b-7e2668cdb7f3
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 9839e176-efce-4a70-811b-7e2668cdb7f3
Base automatically changed from alexvallone/table-column-inference to master August 24, 2026 18:24
alexvallone and others added 13 commits August 24, 2026 16:28
Migrate #table row inference to the intrinsic dispatcher and the released parser 1.0.0 defined-table contract.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Limit this PR to retaining exact row values without inferring or widening list-form column schemas from their contents.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Use one schema, row extraction, validation, and factory path for both explicit and list-form #table constructors.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Cover nullable schemas, generalized and heterogeneous values, zero-column rows, out-of-range access, multi-row projection, and additional malformed row shapes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Union closed table schemas, normalize missing cells to null, concatenate exact rows within the shared retention limit, and preserve conservative fallbacks.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Use existing type guards for dispatch and combine ordered table fields in one pass while preserving exact concatenation behavior.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Preserve the existing examples that explain primitive, defined, and mixed record/table concatenation branches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Separate record opening and field merging from exact table concatenation while preserving the existing branch comments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
@amv146
Alex Vallone (amv146) marked this pull request as ready for review August 27, 2026 17:50
alexvallone added 6 commits August 27, 2026 10:53
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 946bd35e-0c84-49f0-90ff-a15d522d34f8
@amv146
Alex Vallone (amv146) merged commit dfa1878 into master Aug 27, 2026
5 checks passed
@amv146
Alex Vallone (amv146) deleted the alexvallone/table-row-inference branch August 27, 2026 21:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants