Skip to content

Fix flaky response merging after array size changes - #1866

Open
hoco-scy wants to merge 1 commit into
masterfrom
fix/flaky-body-array-merge
Open

hoco-scy wants to merge 1 commit into
masterfrom
fix/flaky-body-array-merge

Conversation

@hoco-scy

@hoco-scy hoco-scy commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Problem

While running OCVN with handleFlakiness=true, static inference enabled and execNumForDetectFlakiness=2 (a one-hour search budget), we encountered 27 test-writing failures out of 1,439 tests selected for output. The stack traces pointed to mergeJsonDiffFromOriginal: NullPointerException: get(...) must not be null. The writer caught these exceptions and omitted the affected tests, leaving 1,412 generated test methods even though the overall process reported success.

The array merge checks the original and current observation lengths, but not the accumulated merged array length. An earlier observation can replace that array with a shorter one. A later observation matching the original length then accesses a missing merged[i].

Minimal reproduction:

original:      {"items":[1,2]}
observation 1: {"items":[]}
observation 2: {"items":[1,3]}

Fix

Before merging array elements by index, retain the accumulated array if its length already differs from the original. This avoids indexing missing elements and preserves the previously observed size difference for flaky assertion handling. Simply replacing it with the latest original-length observation would discard that evidence.

This is a small, independent fix to response-body merging. It does not change phase budgets, re-execution scheduling, or exception handling in the test writer.

Validation

  • Added regression coverage for an empty, shorter, or longer array followed by an original-length observation with changed values.
  • Added a nested-array case that also checks that independent field differences continue to be merged.
  • Both new regression test methods reproduce the NPE before the fix.
  • After the fix, all 24 tests pass: 17 in RestCallResultTest and 7 in FlakinessInferenceUtilTest, with no failures, errors, or skips.

The full OCVN experiment has not been rerun with this patch; validation here is focused regression testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants