Query: Optimize schema traversal for non-leading tag IN predicates

[Caideyipi]

Description

Problem

Table-model tag predicates are converted to OR-concatenated precise-filter branches. Both an IN predicate and its equivalent OR expression therefore produce multiple device patterns.

When the predicate is on a non-leading tag and an earlier tag level is not fixed, every expanded pattern traverses the same wildcard prefix independently. With N values, the schema tree prefix is visited N times. Rewriting IN as OR does not avoid this behavior.

Solution

• Keep predicate expansion in the relational planner so complete device IDs can still benefit from the existing device cache and partition lookup paths.
• Compact branches in DeviceFilterUtil only when they differ by a precise value at a non-leading tag behind an unfixed earlier tag level.
• Represent the merged values as an InFilter attached to ExtendedPartialPath.
• Add multi-exact transitions to SimpleNFA so the shared wildcard prefix is traversed only once.
• At each in-memory MTree parent, compare the effective multi-exact value count with the current child key-set size. Iterate child keys when their count is less than or equal to the IN count; otherwise perform precise child lookups. When the complete child count is unavailable, retain precise child lookup.

The comparison intentionally uses a stable 1:1 candidate-count heuristic rather than a wall-clock threshold. An opt-in performance UT confirmed that direct memory-tree child iteration wins the equal-count tie because it avoids one direct child lookup per transition.

Leading-tag predicates and fully fixed prefixes remain as separate precise paths. Branches with null precise values, additional filters on the same tag, or correlated conditions such as (card = a AND device = x) OR (card = b AND device = y) are not merged.

For the motivating case, six values on the second tag are reduced from six schema readers over the same first-tag wildcard subtree to one adaptive reader.

Manual performance UT

SchemaRegionTableDevicePerformanceTest is disabled by default with a JUnit assumption. It compares the legacy N-reader expansion with the merged adaptive reader, verifies identical results, and reports median CPU time, thread allocation, peak-heap delta, and speedup. There are no wall-clock performance assertions.

The values property accepts a comma-separated sweep, and hit.values=0 models the important empty-result case:

mvn -pl iotdb-core/datanode \
  -Dtest=SchemaRegionTableDevicePerformanceTest \
  -Diotdb.schema.non.leading.in.perf.enabled=true \
  -Diotdb.schema.non.leading.in.perf.meters=50000 \
  -Diotdb.schema.non.leading.in.perf.children.per.meter=8 \
  -Diotdb.schema.non.leading.in.perf.values=7,8,9 \
  -Diotdb.schema.non.leading.in.perf.hit.values=0 \
  -Diotdb.schema.non.leading.in.perf.warmups=2 \
  -Diotdb.schema.non.leading.in.perf.iterations=3 \
  -Diotdb.schema.non.leading.in.perf.rounds=5 \
  test

Representative local Windows results with 400,000 devices and zero matches:

IN values	Selected side	Legacy CPU/op	Optimized CPU/op	Speedup
7	precise IN lookups	98.958 ms	41.667 ms	2.38x
8	child-key iteration	98.958 ms	31.250 ms	3.17x
9	child-key iteration	125.000 ms	31.250 ms	4.00x

For the equal-count case, switching from precise lookup to child-key iteration reduced the optimized path from 46.875 ms/op to 31.250 ms/op in the same 50,000-by-8 empty-result workload (about 1.5x faster).

Real-dataset validation

At PR head a50d9ca, I loaded the two TAG levels relevant to this predicate from a real-world dataset into an in-memory SchemaRegion: 405,421 input rows, 404,060 distinct devices, and 402,032 distinct second-TAG values. The relevant topology had one second-TAG child per first-TAG node.

The same schema tree was scanned with the legacy six-reader expansion and the optimized single adaptive reader. Both result-equivalence checks passed. Measurements used 3 warmups, 5 iterations per round, and the median of 7 rounds on Windows 11 with JDK 17.

Case	Legacy matches	Optimized matches	Legacy CPU/op	Optimized CPU/op	Speedup	Allocation reduction
6-value IN, zero hits	0	0	687.500 ms	187.500 ms	3.67x	35.7%
6-value IN, six hits	6	6	618.750 ms	171.875 ms	3.60x	35.7%

Peak-heap delta was effectively unchanged at about 331-332 MiB, so no peak-heap improvement is claimed.

Tests

• DeviceFilterUtilTest: 10 tests passed, including value-smaller, child-smaller, and equal-count adaptive directions
• SchemaPredicateUtilTest: 2 tests passed, covering both expanded IN and equivalent OR
• SchemaRegionTableDeviceTest: 20 tests passed
• SchemaRegionTableDevicePerformanceTest: enabled benchmark passed; disabled mode skips all 4 parameterized instances
• Spotless, Checkstyle, RAT, and staged diff checks passed

---

This PR has:

• been self-reviewed.
• added comments explaining the intent of the optimization and heuristic.
• added unit tests covering the new code paths and safety guards.

---

Key changed/added classes

• DeviceFilterUtil
• ExtendedPartialPath
• SimpleNFA
• AbstractTreeVisitor
• Traverser
• DeviceFilterUtilTest
• SchemaPredicateUtilTest
• SchemaRegionTableDevicePerformanceTest

[alpass163gmail]

its child class do not override the method hasMultiExactMatchTransitions(IFAState state)

[Caideyipi]

SimpleNFA already overrides this method. PatternDFA intentionally inherits the default false because multi-exact transitions are only constructed by SimpleNFA. The false fallback keeps the existing precise-lookup behavior for FA implementations that do not support multi-exact transitions.