Repository navigation
Supercache - #9
Merged
Merged
Conversation
- 新增基于 trace_id 的确定性高频日志采样，支持跨进程和线程的请求级日志关联 - 在存储后端添加细粒度读性能统计（计划、文件打开、磁盘读取时间等） - 为传输层添加队列深度和传输诊断日志，包含提交和等待时间细分 - 扩展批量操作日志，包含每个键的详细信息、副本类型和端点地址 - 在 SSD 卸载 RPC 中传播 trace_id 以实现端到端的诊断关联
引入 mooncake_logging.h 头文件，为 file_storage.cpp 提供日志记录功能。
添加 `local_endpoints_us` 和 `select_replica_us` 两个新的诊断字段，用于更精细地测量获取缓冲区操作中锁定/复制本地挂载端点与扫描选择最优副本两个子阶段的耗时。同时更新了相关日志输出和文档说明。
- 在 offload RPC 客户端和服务器端添加细粒度性能计时点 - 新增 offload_rpc_client_breakdown 和 offload_rpc_server_breakdown 日志 - 在 get_into_breakdown 日志中增加 offload 子阶段耗时 - 为 release_offload_buffer 添加 trace_id 参数和详细日志 - 新增多个性能监控点以跟踪 RPC 各阶段耗时
- 使用 std::call_once 确保日志初始化只执行一次 - 在 offload 读取操作中添加细粒度耗时统计和结构化日志 - 重构 RPC 调用以分离提交和等待阶段的耗时统计 - 添加缺失的头文件引用
将 RpcTiming 结构体中的 `request_submit_us` 和 `response_wait_us` 分别重命名为 `rpc_call_us` 和 `result_get_us`，使其更准确地反映 RPC 调用和结果获取两个阶段的耗时。同步更新了对应的性能监控点名称和日志输出字段，以保持一致性。
添加对RPC调用各阶段耗时的细粒度监控，包括连接池查找、RPC调用、结果获取和结果解析时间。通过trace_id进行采样控制，仅在需要时记录高性能日志，避免对正常性能产生影响。这有助于诊断RPC调用的性能瓶颈。
避免日志文件过早达到大小限制而被轮转，减少日志轮转频率，提高日志记录的连续性。
修复RPC服务器线程数不足导致的性能瓶颈，提升批处理操作的并发处理能力。
新增 `--shuffle_seed` 标志，允许使用指定种子对分段读取的键进行确定性洗牌。当种子不为零时，使用 `std::mt19937_64` 随机数生成器对 `all_keys` 进行洗牌，确保每次运行具有可重复的键访问顺序，便于性能对比和故障复现。
- 新增 GetOffloadEndpoints RPC 接口，供客户端获取主节点已知的卸载端点 - 为 RealClient 构造函数添加 offload_rpc_thread_num 参数 - 在 Environ 类中添加相关环境变量读取方法，用于配置RPC连接池 - 扩展 MasterClient 配置，支持通过环境变量设置最大连接数 - 在 ClientRequester 中实现 RPC 连接池预热逻辑，避免流量到来时延迟建立连接
添加GetOffloadEndpoints方法用于发现所有可用的offload端点，并在客户端初始化时预热RPC连接池。新增WarmupRpcPool方法通过并发RPC调用预先建立连接，减少首次请求延迟。支持通过环境变量配置连接池大小和预热连接数，优化offload RPC服务器的线程数配置。
在压力集群基准测试中新增 `client_init_wait_seconds` 标志，允许在 RealClient 设置成功后、基准测试运行前等待指定秒数。这为异步 RPC/传输预热和后台初始化提供了稳定时间，确保基准测试开始时系统已就绪。
避免日志文件过快轮转，减少因频繁创建新日志文件导致的性能开销，提升系统稳定性。
添加 `MC_YLT_RPC_POOL_IDLE_TIMEOUT_MS` 和 `MC_YLT_RPC_POOL_SHORT_IDLE_TIMEOUT_MS` 环境变量支持，用于分别配置长连接和短连接的空闲超时时间。这允许运维根据实际负载动态调整连接池行为，优化资源利用。
github-actions
Bot
added
documentation
| mooncake::logging::ShouldSampleHiFreqLog(trace_id); | ||
| int jetty_index = SimpleRandom::Get().next(jetty_list_.size()); | ||
| const int jetty_before = wr_depth_list_[jetty_index]; | ||
| const int jfc_before = *jfc_outstanding_; |
Author
There was a problem hiding this comment.
1093-1095,1099,1100放到queue_log判断里，减小开销。
通过环境变量 MC_YLT_LOG_PATH、MC_YLT_LOG_MAX_FILE_SIZE 和 MC_YLT_LOG_MAX_FILES 灵活配置日志输出路径与文件轮转策略，替代原有硬编码默认值。更新相关文档说明。 同时，优化 URMA 端点 submitPostSend 中的日志开销：仅在需要采样输出时计算队列深度、字节数等统计信息，避免非必要计算。 更新 yalantinglibs 子模块以支持新功能。
Connor-Matthew
pushed a commit
that referenced
this pull request
Nov 3, 2026
…B contexts (kvcache-ai#5104) * [TransferEngine] Fix URMA module lifecycle for multi-device UB contexts urma_init()/urma_uninit() are process-global, but the UB transport paired them wrongly: UrmaContext::deconstruct() called urma_uninit() once per context, while UbTransport::init() called urma_init() once per transport and ~UbTransport() never released it. With a single device the extra uninit() was harmless. With two or more devices the first context destructor released the whole URMA module while sibling contexts still held jetties, and the next teardown faulted: #0 urma_unbind_jetty () #1 UrmaEndpoint::deconstruct() urma_endpoint.cpp:680 #2 UbSIEVEEndpointStore::destroy() ub_context.cpp:145 #3 UrmaContext::~UrmaContext() #16 UbTransport::~UbTransport() Reproduced on a Kunpeng 950 host with 10 URMA devices: any target or initiator configured with two devices crashed on exit while the same single-device configuration ran fine. Let the transport own the module lifetime: drop the per-context uninit(), record whether init() ran, make uninit() idempotent, and pair init() in the destructor after the contexts are gone. install() failure paths already call uninit(), so the guard keeps those from releasing the module twice. * [TransferEngine] Do not dereference a null URMA context while logging UbContext::doConstruct() reports a construction failure with LOG(INFO) << "failed construct context " << toString(); and UrmaContext::toString() calls getAsyncFd(), which dereferenced urma_context_ unconditionally. Opening a device can fail - for example when no configured device name matches, or the mock build is used on a host with real URMA devices - so the failure was reported as a SIGSEGV instead of an error. Return -1 when the context has not been created so the existing error path survives long enough to log and return ERR_DEVICE_NOT_FOUND. * [TransferEngine] Fix a data race in UrmaContext::retrieveRemoteSeg retrieveRemoteSeg() is reached from every submitting thread through UbWorkerPool::submitPostSend(), but it did its cache lookup, malloc and push_back into remote_seg_list_ with no synchronisation. Two threads that both miss the cache therefore push concurrently, and when the vector reallocates both free the old buffer: AddressSanitizer reports ERROR: AddressSanitizer: attempting double-free #7 UrmaContext::retrieveRemoteSeg() urma_endpoint.cpp:364 #9 UbWorkerPool::submitPostSend() ub_context.cpp:271 After the corruption a later malloc()/free() segfaults somewhere unrelated, which is what a multi-initiator benchmark sees as a random SIGSEGV. The same function also published the handle in remote_seg_list_ before the import succeeded, so a failed urma_import_seg() left it for deconstruct() to free a second time. Guard the map and the lists with a shared_mutex: shared lock for the hot lookup path, unique lock for publishing, with a second lookup so a thread that lost the race drops its own import. Publish only after a successful import. Verified with four concurrent initiators (2 read + 2 write, 2 devices, 64 KiB blocks) under ASan: before the fix 3 of 8 processes reported a double-free, after it 8 of 8 are clean across repeated runs, and the non-instrumented build finishes 5 mixed scenarios with no crash. * ci: retrigger build (previous run was cancelled by a runner shutdown) * [TransferEngine] Read the UB device NUMA node from ubcore sysfs UbContext::socketId() built its path from /sys/class/infiniband/<dev>/device/numa_node. URMA devices have no infiniband entry, so the lookup always failed and every UbWorkerPool thread was pinned to node 0 via bindToSocket() regardless of where the device actually sits. The node is published two directories above the /sys/class/ubcore/<dev> link, next to the ub_bus_controller directory, and is formatted as hex (0x01), so a plain '>> int' would have truncated it to 0 even with the right path. Resolve the link with realpath(), walk two levels up, and parse the value with base 0 so the hex form is handled. On a 4-node Kunpeng 950 (node 0: CPU 0-95, node 1: 96-191, node 2: 192-287, node 3: 288-383) with udmac1d1e2, which reports numa=0x03: before: 3 worker threads Cpus_allowed_list=0-95 after: 3 worker threads Cpus_allowed_list=288-383 Devices udmac0* report 0x01 and udmac1* report 0x03, so the two device groups are now pinned to their own nodes instead of both landing on node 0. * docs: correct the device-name comments in UbContext The deviceName()/nicPath() comments were copied from the RDMA transport and still said 'mlx5_3'. URMA devices are named after their ubcore entry (udmac0d1e2, udmac1d1e6, ...) and the NIC path is '${server}:${port}@${device}', for example 127.0.0.1:15479@udmac0d1e3. * docs: point the mock URMA device path at ubcore sysfs mock_urma.cpp advertised /sys/class/infiniband/mock_device, which does not exist for any UB device. Nothing reads urma_device_t::path (listUBDevices() builds /sys/class/ubcore/<name> from the device name), so this was dead data, but it kept the RDMA spelling that misleads readers of the UB transport. Verified the mock build still compiles and passes: cmake .. -DUSE_UB=ON -DURMA_LIBRARY= ... # 'building ub_transport with Mock URMA library' ./ub_transport_test --metadata_server=P2PHANDSHAKE ... [ PASSED ] 2 tests. * docs: call the UB endpoint a device, not an RNIC Two diagnostics in the UB worker pool reported problems on a 'local RNIC'. UB has no NIC; the value logged next to the message is nicPath(), i.e. <host>:<port>@<ubep device>, so 'local UB device' describes it accurately. The '<Transport>: No available RNIC' messages in rdma/barex/ub are left untouched: that sentence is deliberately shared across transports. Both builds still compile and the mock UB test passes: make -j64 transfer_engine_bench -> 0 errors make -j64 ub_transport_test (mock) -> 0 errors ./ub_transport_test -> [ PASSED ] 2 tests
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Module
mooncake-transfer-engine)mooncake-store)mooncake-ep)mooncake-pg)mooncake-integration)mooncake-p2p-store)mooncake-wheel)mooncake-common)mooncake-rl)Type of Change
How Has This Been Tested?
Test commands:
# Example: bash scripts/run_ci_test.shTest results:
Checklist
./scripts/code_format.shpre-commit run --all-filesand all hooks passAI Assistance Disclosure