[Bug] 顺序消费丢失消息一条（生产环境）

[DeerplayHub]

Before Creating the Bug Report

• I found a bug, not just asking a question, which should be created in GitHub Discussions.

• I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.

• I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

OpenSUSE
Push模式+顺序消费+CONSUME_FROM_LAST_OFFSET
同步刷盘

RocketMQ version

Server：RocketMQ 4.9.7
Client：RocketMQ 4.7.1

JDK Version

java8

Describe the Bug

消费配置

• 消费模式：Push 模式 + 顺序消费
• 消费位点策略：CONSUME_FROM_LAST_OFFSET
• 启用TRACE，但无任何消费TRACE信息

业务消费方式

每个整点启动消费者进行消费，消费结束后立即 shutdown，下一个整点再重新启动。

问题概述

• 丢失消息：broker-b 的 5 号队列，第 9 号位置消息
• 消息发送时间（bornTime）：2026-07-31 15:00:00.398
• 消费者启动时间：2026-07-31 15:00:00.230
• 内部首次计算消费位置时间：2026-07-31 15:00:00.230
• 消费者首次拉取消息时间：2026-07-31 15:00:01.348

注：消费者与消息发送方非同机，两端时间可能存在时差。

消费者日志分析

阶段结论： 消息从未进入 RocketMQ 客户端，在拉取（pull）过程中被直接跳过，导致丢失。

Broker-B 5 号队列索引分析

阶段结论： 该队列历史上断续地持续有消息写入，并非首次消费。commitlog 中曾有对应消息，只是后续被清理；但 consumeQueue 对应的消费 offset 记录理论上应当一直存在。由于是事后调查，consumerOffset 文件中实际是否存在该 offset 已无法考证。

关键时间线（生产、消费可能存在时差）

时间	事件
15:00:00.230	消费者启动，内部首次计算出消费位置 offset = 9
15:00:00.398	消息实际写入（bornTime）
15:00:01.348	消费者首次真正发起 pull，此时计算出的 offset = 10

首次计算 offset（9）与首次真正 pull（对应 offset 10）之间相隔约 1.1 秒，且消息的 bornTime 正好落在这个窗口中间。

结论

1. 消息从未被拉取到消费者端——即消费者的 pull 请求返回结果中不包含该消息。

2. 消费者启动时首次计算出的 offset = 9，但首次真正发起 pull 时使用的 offset 已经变成 10，中间 9 号消息被跳过。对 offset=9 的来源存在两种假设，单独看均有矛盾之处：

   情形 A：offset=9 为上次消费位置（记录在 consumerOffset 文件中）
   若如此，则无法解释为何 1.1 秒后首次 pull 时 offset 又变成了 10——如果只发生过一次 computePullFromWhere，pull 应该直接用 9 去拉取，而不会跳过 9 号消息。

   情形 B：offset=9 为 maxOffset（队列末尾位置，因未查到已持久化的 consumerOffset 而走的兜底逻辑）
   若如此，说明系统出现了类似 apache/rocketmq#2708 描述的问题。但排查各方日志均未发现相关异常记录，该假设同样缺乏直接证据支撑。

下一步

• 目前基于现有证据暂时无法进一步闭环，暂无其他明确排查方向，烦请各位老师帮忙提供一些思路，看看是否还有遗漏的排查点。

Steps to Reproduce

无法复现

What Did You Expect to See?

目前还未彻底分析到消息丢失的底层原因，还无修复方案

What Did You See Instead?

预期：offset=9 的消息应被恰好一次地正常消费（无论是被交接前的旧实例还是新实例）。
实际：所有已排查的消费者实例日志中均未发现该消息被拉取或处理的记录，业务侧也未感知到该消息被处理过。

Additional Context

No response

[RockteMQ-AI]

Issue Evaluation

Category: type/bug | Status: Confirmed

Thank you for the detailed analysis. The issue describes a plausible race condition in ordered consumption with CONSUME_FROM_LAST_OFFSET.

Root Cause Analysis:

The timeline you provided points to a race between offset computation and the first pull request:

1. T=0.230s — Consumer starts, computePullFromWhere() resolves CONSUME_FROM_LAST_OFFSET → maxOffset = 9
2. T=0.398s — A new message is written at offset 9 on queue 5 (bornTime)
3. T=1.348s — First actual PullMessageRequest is sent, but by this time maxOffset has advanced to 10, so the pull starts from offset 10

The message at offset 9 falls in the ~1.1s gap and is never fetched. This is consistent with the behavior where computePullFromWhere() is called during rebalance, but the actual pull request uses a freshly resolved maxOffset at pull time rather than the cached offset from rebalance.

Key Code Path:

In ordered consumption, ConsumeMessageOrderlyService relies on ProcessQueue locking and sequential pull. The issue likely stems from the interaction between:

• RebalanceService computing the initial offset via computePullFromWhere() → consumeFromWhere.computePullOffset()
• PullAPIWrapper or PullMessageService resolving the offset again at pull time

If the offset is re-resolved against the broker's current maxOffset instead of using the cached value from rebalance, a message arriving in the gap will be skipped.

Related: This pattern is similar to #2708 where offset resolution timing causes message skip in specific consumption modes.

Version Note: You are running Server 4.9.7 + Client 4.7.1. The version mismatch between client and server may contribute to edge cases in offset negotiation. Consider testing with matched versions if possible.

Severity: Medium — data loss in ordered consumption, but requires a specific timing window (message arriving between rebalance offset computation and first pull).

Suggested Investigation:

1. Check if computePullFromWhere() result is cached and reused for the first pull, or if the offset is re-resolved
2. Verify whether PullMessageService uses the ProcessQueue's cached nextOffset or calls computePullFromWhere() again
3. Check if this has been addressed in newer releases (5.x branch)

---

Automated evaluation by github-manager

[RockteMQ-AI]

Hi @DeerplayHub, thanks for reporting this!

I've analyzed this issue against the codebase and can confirm this appears to be a real bug.

Root Cause: In DefaultMQPushConsumerImpl.pullMessage() (line 308), for an orderly consumer's first pull, computePullFromWhereWithException is called a second time to "fix" the offset. When the consumer has no committed offset (first start, CONSUME_FROM_LAST_OFFSET), this re-queries maxOffset from the broker. If any new message arrived between the rebalance call and this second call (~1 second later), maxOffset has advanced, and pullRequest.setNextOffset(offset) overwrites the originally-computed offset with the new higher value, silently skipping the messages that arrived in between.
Affected Files: client/src/main/java/org/apache/rocketmq/client/impl/consumer/DefaultMQPushConsumerImpl.java, client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalancePushImpl.java, client/src/main/java/org/apache/rocketmq/client/impl/consumer/RebalanceImpl.java

Analysis: The double-call to computePullFromWhere — once at rebalance (RebalanceImpl.java:477) and again at first pull time (DefaultMQPushConsumerImpl.java:308) — is intentional for correcting stale offsets, but it creates a race window: for a fresh consumer with no committed offset, both calls invoke maxOffset(), and if the queue advances between them, the initial offset is discarded. The messages produced in that gap are lost.

I'll prepare a fix spec and work on a PR. The community is welcome to provide feedback on the approach before implementation.

---

🤖 Automated issue analysis by github-manager

[RockteMQ-AI]

Issue Evaluation

Category: bug | Status: Needs Info

Thank you for the detailed report with timestamps and queue-level analysis. A few clarifications are needed before verification can proceed:

1. Version mismatch: Server is 4.9.7, client is 4.7.1 — could you confirm if this issue is reproducible with matched versions (e.g., both 4.9.7)?
2. Consumer startup pattern: You mention starting the consumer every hour and shutting down after consumption completes. Could you clarify whether the CONSUME_FROM_LAST_OFFSET strategy is applied on each fresh start, and whether the offset is persisted to the broker or only in memory?
3. Reproducibility: Has this been observed more than once? Is it specific to broker-b queue 5, or has it occurred on other queues/brokers?
4. TRACE data: You mention TRACE is enabled but no consumption TRACE exists for the lost message. Does the TRACE data show the message was successfully stored (bornTimestamp matches)?

The scenario you describe — where a message at offset 9 is skipped while offsets 0-8 and 10-14 are consumed — suggests a potential offset calculation issue during consumer startup with CONSUME_FROM_LAST_OFFSET in ordered consumption mode. This warrants investigation.

---

Automated evaluation by github-manager

[yx9o]

This behavior matches the orderly-consumption locking issue fixed by #5466. The fix was included starting from RocketMQ 4.9.5, while the affected client is still using 4.7.1. Could you upgrade the client to 4.9.5 or later, preferably 4.9.7 to match the server, and verify whether the issue still occurs?