[python] Add efficient VARIANT path get and replace by XiaoHongbo-Hope · Pull Request #9147 · apache/paimon

XiaoHongbo-Hope · 2026-08-10T10:03:01Z

Purpose

Add Arrow APIs for reading and replacing existing VARIANT paths without decoding unrelated fields.

current = variant_get(payload, "$.velocity.y", pa.float64())
updated = pc.negate(current)
payload = variant_replace(payload, "$.velocity.y", updated)

Semantics

The Arrow type must match the stored VARIANT type; these APIs do not cast values.
Exact reads support scalar types and nested struct, list, and string-keyed map types.
Replacements support scalar types.
Missing paths read as NULL and remain unchanged by default; strict=True rejects them.
Path mappings read or replace multiple fields in one pass.

Implementation

Validate metadata, row bounds, DECIMAL values, and container child offsets before access.
Vectorize FLOAT/DOUBLE lookup and replacement per Arrow chunk.
Keep sparse layout exceptions on a bounded row-wise fallback.
Patch fixed-width values in one copied value buffer and rebuild only affected rows when serialized sizes change.
Reuse compatible metadata and Arrow buffers.

Performance

A local 20K-row benchmark updating four DOUBLE paths completed get + replace at about 720K rows/s.

Tests

VARIANT regression suite: 133 passed, 15 subtests passed.

discivigour · 2026-08-10T12:23:59Z

Just DOUBLE?

XiaoHongbo-Hope · 2026-08-10T12:40:46Z

Just DOUBLE?

Add float support too.

JingsongLi · 2026-08-10T12:42:12Z

Could we clarify whether this is intended to be a general public VARIANT API or an internal fast path?

I could not find an equivalent VARIANT mutation function in Databricks; its documented VARIANT functions are primarily for extraction, inspection, and explosion. JSON APIs, however, have fairly established mutation semantics:

MySQL JSON_REPLACE updates existing paths and treats a missing path as a no-op, while JSON_SET also creates missing paths.
BigQuery JSON_SET(..., create_if_missing => false) provides similar existing-path-only behavior.
Oracle JSON_TRANSFORM is an operation DSL (SET, REPLACE, REMOVE, etc.), rather than a callback-based transform.

MySQL can optimize JSON_SET and JSON_REPLACE into a partial in-place binary JSON update under suitable conditions, which is conceptually close to this implementation. Importantly, that is an implementation optimization behind stable set/replace semantics.

Would an API such as variant_replace, or variant_set(..., create_if_missing=False), be clearer than variant_transform? The current binary patch can remain the fast path for existing fixed-width values. If a path is missing, replace semantics could leave the value unchanged (optionally with a strict mode), while set semantics could fall back to reconstruction because adding a field may require changing metadata and offset tables. This would keep the public contract independent of the current optimization and leave room for supporting more types later.

References: Databricks VARIANT functions, MySQL JSON modification functions, MySQL partial JSON updates, BigQuery JSON_SET, Oracle JSON_TRANSFORM.

JingsongLi · 2026-08-10T12:42:25Z

One additional implementation idea: the fast path could be based on encoded-length invariance rather than being hard-coded to FLOAT and DOUBLE.

For an existing path, after computing and encoding the replacement, if the encoded value has the same length as the old encoded value, the implementation can patch the complete encoded slice directly. The metadata dictionary and all ancestor offset tables remain valid because neither the path nor any byte positions change. This would naturally cover all fixed-width values and potentially variable-width values whose encoded length is unchanged. If the encoded length differs, or the path is missing and must be created, the implementation can fall back to rebuilding the affected value.

There is also an Arrow-level optimization opportunity. The current to_pylist() -> bytearray(value) -> pa.array(values) path materializes Python objects, copies every row separately, and rebuilds the binary offsets. Since the supported fast path preserves each row length, we could reuse the validity bitmap, offsets, and metadata buffers, allocate/copy the value data buffer once per chunk, and patch the located bytes at their absolute offsets. This should be copy-on-write rather than mutating the input buffer, because Arrow arrays can share buffers and are logically immutable.

XiaoHongbo-Hope · 2026-08-10T12:54:27Z

Could we clarify whether this is intended to be a general public VARIANT API or an internal fast path?

I could not find an equivalent VARIANT mutation function in Databricks; its documented VARIANT functions are primarily for extraction, inspection, and explosion. JSON APIs, however, have fairly established mutation semantics:

MySQL JSON_REPLACE updates existing paths and treats a missing path as a no-op, while JSON_SET also creates missing paths.

BigQuery JSON_SET(..., create_if_missing => false) provides similar existing-path-only behavior.

Oracle JSON_TRANSFORM is an operation DSL (SET, REPLACE, REMOVE, etc.), rather than a callback-based transform.

MySQL can optimize JSON_SET and JSON_REPLACE into a partial in-place binary JSON update under suitable conditions, which is conceptually close to this implementation. Importantly, that is an implementation optimization behind stable set/replace semantics.

Would an API such as variant_replace, or variant_set(..., create_if_missing=False), be clearer than variant_transform? The current binary patch can remain the fast path for existing fixed-width values. If a path is missing, replace semantics could leave the value unchanged (optionally with a strict mode), while set semantics could fall back to reconstruction because adding a field may require changing metadata and offset tables. This would keep the public contract independent of the current optimization and leave room for supporting more types later.

References: Databricks VARIANT functions, MySQL JSON modification functions, MySQL partial JSON updates, BigQuery JSON_SET, Oracle JSON_TRANSFORM.

Maybe variant_set? Would we also need variant_get so users can compute replacement values from existing paths?

JingsongLi · 2026-08-10T12:58:35Z

Yes, I think variant_set would be a good direction, with normal set/upsert semantics:

if the path exists, replace its value;
if the final path component is missing, create it;
an existing-path-only operation should be named variant_replace.

With that contract, I would not add create_if_missing to variant_set: creation is part of the meaning of set. For an initial implementation, it would be reasonable to support creation only when the parent container already exists, and explicitly reject missing intermediate containers, incompatible parent types, and undefined array-extension cases until their semantics are specified.

I also agree that variant_get would make the API more composable and closer to Databricks. For example:

current = variant_get(column, "$.velocity.y", pa.float64())
updated = pc.negate(current)
result = variant_set(column, "$.velocity.y", updated)

Here variant_set could accept either an Arrow scalar or an Arrow array. This avoids placing arbitrary Python callbacks in the per-row hot path and lets users build updates from Arrow compute expressions. The existing position-planning code could also be shared by variant_get and variant_set.

The implementation can then choose its strategy independently of the public semantics:

existing path plus equal encoded length: copy-on-write patch, reusing metadata and offsets;
existing path plus different encoded length: rebuild the value and affected offsets;
missing final field: rebuild the value; reuse metadata only if the key is already present in the metadata dictionary, otherwise rebuild metadata as well.

If missing-path creation is out of scope for this PR, I would expose the current operation as variant_replace first and add the broader variant_set once the reconstruction fallback is implemented.

JingsongLi

Re-reviewed the latest head. The sliced-buffer copy issue is fixed; the remaining reproducible findings are inline below.

JingsongLi · 2026-08-10T14:07:56Z

+                )
+                if new_size is None:
+                    new_size = len(encoded)
+                if new_size != _value_size(value, pos):


[P1] Please reject a truncated target before entering the fast patch path. _value_size returns the fixed logical size for a primitive, but it does not prove that those bytes remain inside the current BinaryArray row. I reproduced this with an object whose child contains only a DOUBLE header followed by a valid row: new_size == _value_size(...), so the 9-byte patch crosses the first row boundary and overwrites the next row, which then decodes as NULL. Please validate the resolved child and require pos + checked_size <= len(value) (including container offset/sentinel invariants), and explicitly bound every absolute patch range to the current row. A malformed value should raise instead of corrupting an unrelated row in the result.

JingsongLi · 2026-08-10T14:07:56Z

+                )
+        for path, _, target_type in parsed:
+            result_chunks[path].append(
+                pa.array(results[path], type=target_type))


[P1] Constructing an Arrow array with target_type does not implement Paimon's established variant_get cast contract. For example, LONG -> STRING and OBJECT/ARRAY -> STRING currently raise ArrowTypeError; the Java VariantGet.cast implementation supports primitive casts, JSON serialization for OBJECT/ARRAY -> STRING, and recursive complex-type casts. This makes the same named public API behave differently across Paimon clients. Please implement explicit cast semantics matching Java (including invalid-cast behavior) and add parity tests, or expose this as exact-type extraction rather than accepting a cast-like target_type.

JingsongLi · 2026-08-10T14:07:56Z

+        return struct.unpack_from('<d', value, pos + 1)[0]
+    if value_type == _Type.FLOAT:
+        return struct.unpack_from('<f', value, pos + 1)[0]
+    return GenericVariant(bytes(value), metadata, pos).to_python()


[P1] DECIMAL values decoded through this fallback can silently lose precision. GenericVariant.to_python() currently computes the decimal using the default Python Decimal context (precision 28), so a valid DECIMAL16 value such as 12345678901234567890123456789012345678 is returned as 12345678901234567890123456790000000000 by variant_get(..., pa.decimal128(38, 0)). Please decode the unscaled integer and scale exactly (for example with a Decimal tuple), ideally fixing the shared decoder, and add an exact 38-digit regression test.

JingsongLi · 2026-08-10T14:07:56Z

+        or pa.types.is_large_binary(data_type)
+        or pa.types.is_date32(data_type)
+        or pa.types.is_timestamp(data_type)
+        or pa.types.is_decimal128(data_type)


[P2] decimal128 is advertised as a supported replacement type here, but ordinary valid Arrow decimals can still fail in the encoder. Replacing with pa.array([Decimal('100.00')], type=pa.decimal128(10, 2)) normalizes the value to 1E+2 and raises append_decimal requires a non-positive exponent. Please encode from the Arrow precision/scale and exact unscaled integer instead of normalizing the Python Decimal, and cover trailing-zero and negative-scale values.

JingsongLi · 2026-08-10T14:07:56Z

+        )
+
+
+def _decode_scalar(value, metadata: bytes, pos: int):


[P2] The non-FLOAT/DOUBLE fallback below still copies the entire serialized VARIANT row via bytes(value) before decoding one selected subtree. Extracting a one-byte string from a row with an unrelated 2 MiB binary field allocates roughly 2 MiB, and this repeats for every requested non-float path. That undermines the path API's bounded-work goal. Please decode primitives directly from the memoryview, or copy only value[pos:pos + checked_value_size] and decode it at offset zero.

JingsongLi

Three follow-up findings on the latest head.

JingsongLi · 2026-08-10T15:10:42Z

+        if pa.types.is_boolean(target_type):
+            if isinstance(value, str):
+                lowered = value.strip().lower()
+                if lowered not in ('true', 'false'):
+                    raise ValueError
+                return lowered == 'true'
+            if isinstance(value, (bool, int, decimal.Decimal)):
+                return value != 0
+            raise TypeError
+        if pa.types.is_signed_integer(target_type):
+            if not isinstance(value, (bool, int, float, decimal.Decimal, str)):
+                raise TypeError
+            return _cast_integer(value, target_type)


[P2] Preserve Java's source-specific string cast semantics

This routes strings through _cast_integer, which first calls int() and then wraps modulo the target width. As a result, "2147483648" becomes -2147483648 for int32, whereas Java VariantGet rejects the overflow; conversely, "1.9" is rejected here even though BinaryStringUtils truncates it to 1. The boolean branch also rejects Java-supported tokens such as "1" and "yes". Since this API is intended to match Java casts, please dispatch by both source and target type: use the same bounds-checked string parsers and boolean token set for string inputs, and retain narrowing wraparound only for numeric-to-numeric casts.

JingsongLi · 2026-08-10T15:10:42Z

+        fixed_size = _PRIMITIVE_FIXED_SIZES.get(type_info)
+        if fixed_size is not None:
+            end = pos + fixed_size


[P2] Validate DECIMAL precision and scale before accepting its fixed size

Treating every decimal header as size-only allows malformed values such as DECIMAL4 with scale 10, or unscaled 2147483647 at scale 0, to be returned by Python even though Java GenericVariantUtil.getDecimalWithOriginalScale rejects them. Please decode the scale and signed unscaled integer during validation and enforce the DECIMAL4/8/16 precision and scale limits of 9/18/38 before returning the size.

JingsongLi · 2026-08-10T15:10:42Z

+    if (valid_count != len(values.array)
+            or values.array.null_count
+            or metadata.null_count
+            or not len(values.array)):
+        return None


[P2] Avoid de-vectorizing the entire chunk because of one NULL

This guard sends every row through the Python slow path when a chunk contains even one parent NULL. On 50,000 otherwise identical rows, one NULL changed variant_get from 0.0019 s to 0.319 s and variant_replace from 0.0022 s to 0.257 s (roughly 118–165× slower). Nullable VARIANT columns are common, so could we use the struct validity bitmap as a row mask, vectorize the valid rows, then scatter NULLs back for get and skip them for replace?

JingsongLi · 2026-08-11T08:34:45Z

+1

XiaoHongbo-Hope added 4 commits August 10, 2026 03:02

[python] Avoid full decoding for VARIANT path updates

77ccfd2

[python] Limit VARIANT path transforms to DOUBLE

7095ff1

[python] Tighten DOUBLE path transformations

80de6ef

[python] Reject duplicate VARIANT transform paths

be3df2d

XiaoHongbo-Hope marked this pull request as ready for review August 10, 2026 11:33

[python] Support FLOAT VARIANT path transforms

c232052

[python] Add composable VARIANT get and replace APIs

7f09def

XiaoHongbo-Hope changed the title ~~[python] Avoid full VARIANT decoding for path updates~~ [python] Add efficient VARIANT get and replace APIs Aug 10, 2026

[python] Bound memory for VARIANT replacements

742a465

JingsongLi reviewed Aug 10, 2026

View reviewed changes

XiaoHongbo-Hope added 2 commits August 10, 2026 07:09

[python] Vectorize numeric VARIANT path updates

0bfc056

[python] Harden VARIANT path access and casts

5a5f8ac

JingsongLi reviewed Aug 10, 2026

View reviewed changes

XiaoHongbo-Hope added 5 commits August 10, 2026 08:45

[python] Harden VARIANT path semantics and sparse layouts

bd14294

[python] Keep sparse VARIANT values on the fast path

94e277f

[python] Bound VARIANT object children by offsets

c9c5041

[python] Preserve VARIANT timestamp and cast semantics

57e1a46

[python] Preserve VARIANT float string precision

6372245

XiaoHongbo-Hope marked this pull request as draft August 11, 2026 01:24

XiaoHongbo-Hope added 2 commits August 10, 2026 18:25

[python] Preserve nested VARIANT numeric semantics

b66d7ed

[python] Match legacy Java floating strings

7e5af8e

XiaoHongbo-Hope marked this pull request as ready for review August 11, 2026 02:45

XiaoHongbo-Hope added 2 commits August 10, 2026 19:58

[python] Validate VARIANT timestamp nanosecond range

fe2929e

[python] Clarify JDK 8 floating-point formatting

d47c1bb

XiaoHongbo-Hope added 4 commits August 10, 2026 21:21

[python] Scope floating formatting to VARIANT

b02316d

[python] Harden VARIANT path edge cases

a5ddb1e

[python] Align VARIANT casts and missing paths

a90eb1d

[python] Scope VARIANT path updates to floating types

0915832

XiaoHongbo-Hope changed the title ~~[python] Add efficient VARIANT get and replace APIs~~ [python] Add efficient VARIANT FLOAT/DOUBLE path updates Aug 11, 2026

[python] Support exact VARIANT path types

9a2d916

XiaoHongbo-Hope changed the title ~~[python] Add efficient VARIANT FLOAT/DOUBLE path updates~~ [python] Add efficient VARIANT path get and replace Aug 11, 2026

JingsongLi merged commit 4b0d658 into apache:master Aug 11, 2026
10 checks passed

XiaoHongbo-Hope mentioned this pull request Aug 11, 2026

[python] Align exact VARIANT integer and decimal types #9168

Merged

Uh oh!

Conversation

XiaoHongbo-Hope commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Semantics

Implementation

Performance

Tests

Uh oh!

discivigour commented Aug 10, 2026

Uh oh!

XiaoHongbo-Hope commented Aug 10, 2026

Uh oh!

JingsongLi commented Aug 10, 2026

Uh oh!

JingsongLi commented Aug 10, 2026

Uh oh!

XiaoHongbo-Hope commented Aug 10, 2026

Uh oh!

JingsongLi commented Aug 10, 2026

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingsongLi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JingsongLi commented Aug 11, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

XiaoHongbo-Hope commented Aug 10, 2026 •

edited

Loading