Skip to content

local-sort.max-num-file-handles below 2 causes a data-volume-dependent crash on spill #10292

Description

@LuciferYang

Search before asking

  • I searched in the issues and found no similar issues.

Paimon version

master (1.5-SNAPSHOT)

Compute Engine

Any engine doing an external sort that spills (large sort / merge).

Minimal reproduce step

  1. Set local-sort.max-num-file-handles = 1.
  2. Run a sort large enough to produce more than one spill file.

What doesn't meet your expectations?

The job crashes with an IllegalArgumentException (a negative subList argument), but only after the second spill file appears. local-sort.max-num-file-handles (the external-merge fan-in, default 128) has no lower-bound validation, so it can be set to 1. With fan-in = 1, AbstractBinaryExternalMerger.mergeChannelList divides by maxFanIn - 1 = 0, yielding Integer.MAX_VALUE merges and a negative subList bound. Because it only triggers on the second spill, it passes small-data testing and then crash-loops in production as data grows (the job restarts on the same config and fails again). It should fail fast at construction with a clear message.

Anything else?

A fan-in of 1 is degenerate for a merge sort (it can never converge N runs), so rejecting it up front is correct.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions