Skip to content

fix(reorganizer): optimize all graph tenants, run scheduled jobs, and survive non-JSON LLM output - #2481

Merged
endxxxx merged 2 commits into
MemTensor:dev-v2.0.37from
akushonkamen:fix/2480-reorganizer-multitenant
Nov 19, 2026
Merged

endxxxx merged 2 commits into
MemTensor:dev-v2.0.37from
akushonkamen:fix/2480-reorganizer-multitenant

Conversation

@akushonkamen

@akushonkamen akushonkamen commented Oct 11, 2026 •

Copy link
Copy Markdown

Description

Graph reorganization previously optimized only the default tenant, while its registered periodic jobs were never executed. Malformed model output could interrupt a cluster or produce an empty parent.

The final implementation covers three areas.

  • Enumerate named tenants in shared Neo4j and Community stores and pass each tenant through candidate reads, parent writes and edge writes. A tenant failure is isolated from the remaining tenants. Backends without enumeration use their configured tenant, while physically isolated database mode processes its configured database once. Shared stores require an explicit valid scope for scheduled work.
  • Give each reorganizer its own schedule.Scheduler. Both 100-second jobs execute through that scheduler, immediate ingestion triggers reset the jobs, and triggers arriving during optimization remain pending. Shutdown interrupts the wait and clears the instance's jobs.
  • Parse plain/fenced JSON without removing json from memory text. Reject non-object and malformed output, validate nonempty summary key/value and field types before embedding, and tolerate malformed subcluster entries while filtering unknown/duplicate IDs. Invalid clusters are skipped so valid work can continue.

The original contribution history by akushonkamen is retained, with dev-v2.0.37 merged into the PR branch and follow-up regression fixes applied.

Fixes #2480.

Type of change

  • Bug fix

How Has This Been Tested?

  • Unit tests cover tenant enumeration, explicit fallback, per-tenant failure isolation, private schedulers, real due-job execution, trigger preservation and shutdown
  • Regression cases cover plain text, malformed JSON/arrays, empty objects, invalid summary fields and valid strings containing json
  • Native Neo4j integration tests cover Neo4j and Community backends with valid and malformed tenant responses

Actual verification output

make format
All checks passed!
675 files left unchanged

poetry run pytest tests/graph_dbs/ tests/memories/ tests/api/ -q -ra --tb=short
504 passed, 9 skipped, 36 warnings in 13.90s

MEMOS_TEST_REORGANIZE_NEO4J_URI pointed to a disposable Neo4j 5.26.6 instance. All four new native graph cases passed, including parent and PARENT-edge isolation and one tenant receiving non-JSON output. Model/embedding calls and the Community vector client are mocked with synthetic data.

The nine skips are pre-existing opt-in database integrations whose separate service variables were unconfigured. Existing warnings cover deprecated UTC calls, Pydantic serialization and the intentional thread-exception fixture.

Repository pre-commit hooks passed.

Checklist

  • Self-review completed
  • Comments explain scheduler ownership, scoped fallback and trigger handling
  • Regression and native integration tests added
  • Documentation applicability reviewed for these internal fixes
  • Related issue linked
  • Review context and original contributor akushonkamen identified

Reviewer Checklist

… survive non-JSON LLM output

- enumerate tenants via new get_all_user_names() (Neo4jGraphDB impl +
  BaseGraphDB fallback) and route both the _reorganize_needed trigger and
  the schedule jobs through _optimize_all_users(scope), isolating
  per-tenant failures; backends without the capability keep the previous
  single-tenant behavior
- call schedule.run_pending() in the structure optimizer loop so the
  registered 100s jobs actually run instead of being dead code
- make _parse_json_result return None for anything that is not a valid
  JSON object and short-circuit _summarize_cluster/_local_subcluster, so
  a malformed LLM reply skips the cluster instead of crashing the whole
  optimization round or silently writing empty parent nodes

Fixes MemTensor#2480
@Memtensor-AI Memtensor-AI added area:database graph_db + vector_db | 图数据库与向量数据库 area:memory 记忆存储、检索、更新、召回逻辑 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理 labels Oct 11, 2026
@Memtensor-AI

Copy link
Copy Markdown
Collaborator

🤖 Open Code Review

Target: PR #2481
Task: 591c0052b6b55317
Base: main
Head: fix/2480-reorganizer-multitenant

✅ OpenCodeReview: Review complete: 0 finding(s) across 5 selected item(s).

Generated by cloud-assistant via Open Code Review.

@Memtensor-AI

Copy link
Copy Markdown
Collaborator

⚠️ Automated Test Results: ENV ISSUE

The test environment encountered an issue that requires manual attention.

Details: Both failures are ModuleNotFoundError for the third-party 'schedule' package, raised when the new tests trigger the reorganizer's background loop. The production code under test already imports 'schedule', so the dependency is simply not installed in the test environment.
Branch: fix/2480-reorganizer-multitenant

@github-actions

Copy link
Copy Markdown
Contributor

This PR has been automatically marked as stale due to inactivity.

@github-actions github-actions Bot added the status:stale Auto-marked stale after 30d inactivity; closes 7d later unless exempt. label Nov 12, 2026
@endxxxx
endxxxx changed the base branch from main to dev-v2.0.37 November 19, 2026 12:46
@Memtensor-AI Memtensor-AI removed the status:stale Auto-marked stale after 30d inactivity; closes 7d later unless exempt. label Nov 19, 2026
@endxxxx

endxxxx commented Nov 19, 2026

Copy link
Copy Markdown
Collaborator

Thanks for the original fix. I pushed follow-up changes in 475be4f and retargeted the PR to the development branch. The update adds explicit tenant scoping and per-tenant failure isolation, gives each reorganizer its own scheduler, preserves triggers received during optimization, and makes shutdown interrupt the scheduler wait. It also preserves literal json text and skips malformed or empty summaries before embedding or writing parent nodes. Regression tests finished with 504 passed and 9 optional integration tests skipped, including four passing native Neo4j cases covering both backends and tenant isolation.

@endxxxx
endxxxx merged commit 4e290c4 into MemTensor:dev-v2.0.37 Nov 19, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:database graph_db + vector_db | 图数据库与向量数据库 area:memory 记忆存储、检索、更新、召回逻辑 status:in-progress Someone or AI is working on it | 人工或 AI 正在处理

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: GraphStructureReorganizer only optimizes the default cube tenant; periodic schedule never runs; cluster summary crashes on non-JSON LLM output

5 participants