Repository navigation
Fix #2103: fix: Search_memories在默认dedup=mmr下调试用 info 日志打印完整 embedding 向量 - #2104
Memtensor-AI wants to merge 3 commits into
Conversation
…ing vectors
Under the default `APISearchRequest` (`mode=fast`, `dedup=mmr`),
`_fast_search` calls `format_memory_item(..., include_embedding=True)`, so
every memory keeps its full `metadata.embedding` list. The debug line
`self.logger.info(f"Search memories result: {memories_result}")` in
`SingleCubeView.search_memories` therefore serialized MB of float vectors
into logs on every default /product/search — drowning useful signal and
leaking vector content that can be used to approximate semantic contents.
Fix (logging-only, no API contract change):
* Add `_summarize_search_result_for_log` helper producing a compact,
embedding-free summary: per-bucket counts across text_mem / pref_mem /
tool_mem / skill_mem, a bounded (max 5) sample of memory ids /
ref_ids / types / relativity scores, and a boolean `has_embedding`
plus `embedding_len` — never the raw vector.
* Replace the two offending log lines with a single INFO log line
emitting that summary. The old `len(memories_result)` also reported
the number of top-level dict keys (~7 constants) instead of the
number of matched memories; the new summary carries an accurate
`total_memories` field instead.
* Regression suite `tests/multi_mem_cube/test_search_log_sanitization.py`
covers the helper directly (embedding leak / bucket counts / bounded
sampling / malformed input) and drives `search_memories` end-to-end
under both `dedup=mmr` (embeddings populated) and `dedup=no`
branches, asserting the log line never contains embedding floats.
Fixes MemTensor#2103
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🤖 Open Code ReviewTarget: PR #2104 🔍 OpenCodeReview found 2 issue(s) in this PR. 1.
|
…ict test assertion Round 1/2 of automated review fixes for PR MemTensor#2104 (issue MemTensor#2103). Two findings from the review: 1. src/memos/multi_mem_cube/single_cube.py L94-97 — bucket_count undercount. The outer `break` on the `for group in groups` loop exited the moment `_LOG_SAMPLE_PER_BUCKET` samples were collected, so any remaining groups were silently skipped by `bucket_count += len(mems)`. In a multi-cube deployment where a single bucket (e.g. text_mem) can have one group per cube, `total_memories` in the log undercounted the real number of memories. Fix: always accumulate `bucket_count += len(mems)` for every group, guard only the sampling loop with `len(samples) < _LOG_SAMPLE_PER_BUCKET`. Sampling stays bounded, counting is now complete. 2. tests/multi_mem_cube/test_search_log_sanitization.py L113 — tautological assertion. `"embedding" not in sample or sample["embedding_len"] is not None` was vacuously true because the sample dict never has an "embedding" key (only "has_embedding" and "embedding_len"). If a future change accidentally added a raw "embedding" key with a real value, the check would still pass. Replaced with a direct structural check that asserts the raw "embedding" key is absent, with a descriptive message listing sample keys. Also added regression test `test_count_is_accurate_across_groups_after_sample_cap` that pins the multi-group counting behaviour (two cubes x 10 memories per group; sampling caps at 5 in the first group; count must still be 20). Verification (all 12 tests in tests/multi_mem_cube/test_search_log_sanitization.py pass): $ python3 -m pytest tests/multi_mem_cube/test_search_log_sanitization.py -v ... ======================== 12 passed, 2 warnings in 4.17s ======================== Ruff: All checks passed! Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🤖 Open Code ReviewTarget: PR #2104 🔍 OpenCodeReview found 2 issue(s) in this PR. 1.
|
…f_note + drop duplicate test condition Round 2/2 of automated review fixes for PR MemTensor#2104 (issue MemTensor#2103). Two findings from the review: 1. src/memos/multi_mem_cube/single_cube.py L101-L103 — pref_note silent type coercion. When `pref_note` was a truthy non-string value (list, dict, etc.), `isinstance(pref_note, str)` returned False and `pref_note_len` was silently logged as 0. Operators saw `pref_note_len: 0` and could not distinguish "empty pref note" from "wrong type from upstream". Replaced the `else 0` fallback with a type-marker string `f"non-str({type(pref_note).__name__})"` so the anomaly is visible in logs. Behaviour when `pref_note` is a str (length) or falsy (key omitted) is unchanged. 2. tests/multi_mem_cube/test_search_log_sanitization.py L338 — duplicate condition in `or` chain. The third condition `"'count': 2" in joined` was an exact copy-paste of the first, dead code. Replaced with the intended JSON-key variant `'"count": 2' in joined` so all three branches now cover distinct rendering formats (Python repr / kwargs / JSON). Also added regression test `test_pref_note_non_string_surfaces_type` pinning the four cases: str (numeric length), list / dict (type marker), falsy (key omitted). Verification (13 tests, all pass): $ python3 -m pytest tests/multi_mem_cube/test_search_log_sanitization.py -v ... ======================== 13 passed, 2 warnings in 4.24s ======================== Ruff: All checks passed! Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
🤖 Open Code ReviewTarget: PR #2104 🔍 OpenCodeReview found 4 issue(s) in this PR. 1.
|
|
Description
Fixed issue #2103:
SingleCubeView.search_memorieswas writing the full assembledmemories_result— including high-dimensional per-memorymetadata.embeddingvectors — to the INFO log on every default/product/searchcall (becauseAPISearchRequestdefaultsmode=fast,dedup=mmr, which causesformat_memory_item(..., include_embedding=True)in_fast_search). Each request therefore serialized MB of float data to logs, drowned useful signal, and leaked vector content — a privacy / compliance risk.The fix is logging-only, with no change to
search_memories' return payload or any API contract. A new module-level helper_summarize_search_result_for_loginsrc/memos/multi_mem_cube/single_cube.pyreturns a compact, embedding-free summary: per-bucket counts acrosstext_mem/pref_mem/tool_mem/skill_mem, a bounded (max 5 per bucket) sample of memory ids / ref_ids / memory_types / relativity scores, and — for each sample — a booleanhas_embeddingplus an integerembedding_len. The two offending log statements atsingle_cube.py:132-133are collapsed into a single sanitized INFO log line; the oldlen(memories_result)(which returned ~7, the number of top-level dict keys, not the number of matched memories) is replaced by the summary'stotal_memoriesfield.Regression coverage is added at
tests/multi_mem_cube/test_search_log_sanitization.py(11 tests, all passing): it exercises the helper directly (embedding leak, bucket counts, bounded sampling, malformed input) and drivesSingleCubeView.search_memoriesend-to-end for bothdedup="mmr"(embeddings populated) anddedup="no"branches, asserting that distinctive embedding-vector floats never appear in any captured log record while the summary line still emits the useful debug signals.ruff checkandruff format --checkon the changed files are clean; pre-existing failures intests/test_add_stage_logging.pyandtests/api/test_mcp_serve.pywere confirmed unrelated by re-running against a stash of the branch tip.Related Issue (Required): Fixes #2103
Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Not run; documentation-only change.
Checklist
@MatthewZhuang, @CarltonXiang, @syzsunshine219, @World-controller please review this PR.
Reviewer Checklist