fix milvus forceMerge support targetSize config

[frankleaf]

I noticed that Milvus currently defaults to forcing merges into segments with a fixed size of int64 max value. In practice, this introduces uncertainty in segment sizing, particularly affected by DataNode memory usage.

MILVUS_FORCE_MERGE_TARGET_SIZE_MB = ((1 << 63) - 1) // (1024**2)
compaction_id = self.client.compact(self.collection_name, target_size=MILVUS_FORCE_MERGE_TARGET_SIZE_MB)

I'd like to make this fixed value configurable via a parameter. The default behavior should remain unchanged (using the current maximum value), but when explicitly specified through configuration, the fixed segment size should become adjustable.

milvushnsw:
  case_type: Performance1536D50K
  uri: http://localhost:19530
  m: 16
  ef_construction: 128
  ef_search: 128
  force_merge_target_size_mb: 4096   

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: frankleaf
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[XuanYang-cn]

@frankleaf Could you clarify the benchmark use case for this option?
target_size is an upper bound, not a fixed output segment size. Milvus may choose a smaller value based on memory, topology, shards, and replicas.
For example, if Milvus calculates a safe limit of 2 GB:

• MaxInt64 → up to 2 GB
• 4 GB → still up to 2 GB
• 1 GB → up to 1 GB, usually with more segments
  This calculation is deterministic for the same deployment, so the current behavior does not introduce uncertainty.
  The PR describes this as a “fixed segment size” and says MaxInt64 merges a collection into one segment, but neither is guaranteed. This also shows how easily the parameter can be misunderstood.
  Exposing it has a high usage and explanation cost: users must understand Milvus compaction, memory constraints, and topology, or the benchmark results may become misleading and less comparable.
  It may make sense for a specialized segment-size experiment, but I do not think it should be a general VectorDBBench option without a clear use case and documentation.

[frankleaf]

@XuanYang-cn thank you, but I understand that this safe limit is constrained by cluster resources(dn/qn) and derived through a system-level calculation. The intention behind exposing this parameter is to make it more practically useful and easier to understand, as well as to ensure fairness in benchmarking where applicable. because I've noticed that when people submit benchmark results only indicate computing resources. does the benchmark standard imply that no additional constraints other than computing resources should be considered, focusing solely on system-level results? For example, for 16c64g-sq8-force_merge or ElasticCloud-8c60g-force_merge, I only need to consider the computing resources, while other system components are free to be optimized as needed.

[XuanYang-cn]

@frankleaf Thanks for the clarification. I still do not think we should merge this PR, for three reasons:

1. There is no fairness issue in our standard results.
   All standard results we publish use the standard end-to-end workflow, default benchmark parameters, and no product-specific tuning. Resource labels only describe the hardware environment. Users may tune parameters for private experiments, but those are not standard benchmark results.
2. target_size is easy to misunderstand.
   It is a Milvus-specific upper bound, not a fixed output segment size. For example, if Milvus calculates a safe limit of 2 GB, both MaxInt64 and 4 GB result in segments up to 2 GB, while 1 GB only lowers the limit. The “fixed segment size” and “single segment” descriptions in this PR are therefore inaccurate. Using this option correctly requires significant knowledge of Milvus topology, memory constraints, shards, replicas, and compaction behavior.
3. The PR changes the compaction flow without sufficient validation.
   For a custom target size, it skips the existing compaction-plan coverage check and retry logic. This is based on an assumption about Milvus behavior, but the PR only verifies it with mocks—not with an end-to-end test against a real Milvus deployment. It could allow the benchmark to continue after an incomplete compaction.

Flexibility for private testing does not mean every database-specific tuning parameter should become part of the public benchmark configuration. Given the unclear standard use case, high explanation cost, and unverified compaction-flow change, I do not think we should accept this PR.

[XuanYang-cn]

One additional concern: the PR's custom flow appears to be based on a misunderstanding of Milvus compaction semantics. This also shows why exposing this parameter can be misleading—without understanding the underlying compaction behavior, users are unlikely to get the segment layout or benchmark result they expect.

[frankleaf]

@XuanYang-cn thank you, What I actually meant is that the MILVUS_FORCE_MERGE_TARGET_SIZE_MB parameter exhibits different behavior under varying resource conditions, making it an "X-factor." Moreover, this size can be practically derived based on available resources. If the resource capacity is fixed and the target size is configurable, it would enable more flexible testing strategies. That said, I agree with you that what you mentioned earlier does not fall under any publicly documented standard.Based on your answer, I now understand that for benchmarking purposes, segment size and quantity are not dimensions that should be compared — everyone can adjust them freely according to their actual situation. Therefore, I will cancel this PR. Thank you for your response.