Repository navigation
JVM columnar readers return the values the row readers do - #180
Merged
Merged
Conversation
to_dataframe keeps microseconds and dates (C2): ColumnBatcher takes a micros flag and writes datetime columns as epoch microseconds, DATE and tz-aware columns as tagged JSON; to_columns and to_arrow keep their documented milliseconds. A json column is written with TypedRows' tagged encoding plus a side array of engine objects, so a date, datetime, Decimal or int map key inside a list or map reads as to_list() reads it (B3). An empty float vector has no dim in the header and reads as empty (B2). insert_many refuses a subnormal double for the bulk JSON path, whose parser read 5e-324 as a BigDecimal (A4). Java is not compiled here; CI builds it. Against the main 26.11.1.dev0 wheels the JVM wheel fails the new tests (33 failed) and the native wheel passes the whole file (67 passed). The Java change is compiled and tested by CI. Closes #178 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
tae898
added a commit
that referenced
this pull request
Nov 1, 2026
#180 writes nested list/map values of the columnar readers with the TypedRows tagged encoding and decodes them with _typed_hook, the path to_list() uses, which keeps NaN and the infinities; its version of the json column (ColumnBatcher and results.py) is kept, and this branch's keepNonFinite/_loads_nonfinite now cover to_json_list and iter_json_batches (RowBatcher). tests/test_nonfinite_floats.py still checks every reader. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 1, 2026
Merged
tae898
added a commit
that referenced
this pull request
Nov 1, 2026
…#183) #180 sends a bulk JSON chunk through a per-row Python walk when its text contains "e-3", to catch a subnormal double the engine's parser would read as a BigDecimal. "e-3" is common in ordinary text (hex ids, UUIDs, "name-3"), so such loads walked every chunk. The text is now searched for the exponent json.dumps writes for a subnormal (e-308 to e-329 after a digit) with a compiled pattern: the same C-speed scan, no false hits on ordinary text. Test: UUID-like and "name-3" text never reaches the walk; 5e-324 and the largest subnormal still raise ValueError (per-row path). Closes #182 Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #178
JVM build only (the native readers already return these values).
src/java/com/arcadedb/python/ColumnBatcher.java:nextColumnBatchreturns{buffer, side}; a json column (nested list or map values) is written withTypedRows' tagged encoding, so dates, datetimes, Decimals and int map keys keep their type; values JSON cannot carry go to the side array. A newmicrosform writes datetime columns as epoch microseconds forto_dataframe.TypedRows.appendis package-private for this.src/arcadedb/results.py:to_dataframereads through_read_columns(..., micros=True)(datetime64[us], dates asdatetime.date);to_columnsandto_arrowkeep their documented milliseconds; json columns decode with the same_typed_hookasto_list; an empty float32 vector reads as empty instead ofKeyError: 'dim'.src/arcadedb/type_conversion.py: a subnormal double in a bulk row sends that row down the per-row path, so5e-324stays a float.tests/test_columnar_readers.py(C2, B3 throughto_columns/to_dataframeandto_arrow, B2, A4, plus a guard thatto_columns/to_arrowkeep milliseconds).Verification: against the 26.11.1.dev0 wheels of main the JVM wheel fails the new tests (33) and the native wheel passes the whole file (67). Java is compiled and the fixed JVM wheel is tested by this PR's CI. #179 (NaN and inf on the JVM readers) touches the same json-column path and is rebased on this one after it merges.
🤖 Generated with Claude Code