Repository navigation
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved scope-validation and vector-lifecycle defects can expose out-of-scope content, lose vectors, or break retrieval.
6 open issues
What changed in this PR
Adds Milvus as an optional vector backend for MaxKB knowledge retrieval, while retaining PostgreSQL metadata, full-text search, and legacy vector fallback. It also consolidates database migrations for v2-to-v3 upgrades.
Changes:
- Adds Milvus REST operations, backend selection, and backend-specific index management.
- Adds hybrid retrieval helpers, regression tests, and a local HTTP simulation.
- Consolidates upgrade migrations and tests preservation of existing data.
| File | Description |
|---|---|
apps/users/migrations/0004_alter_systemusergrouprelation_group.py |
Removes migration folded into the upgrade entry. |
apps/users/migrations/0003_systemusergroup_systemusergrouprelation.py |
Removes consolidated user-group migration. |
apps/users/migrations/0002_v3_upgrade.py |
Consolidates user schema upgrades. |
apps/users/migrations/0002_alter_user_nick_name.py |
Removes consolidated nickname migration. |
apps/system_manage/migrations/0008_add_chat_user_token_quota.py |
Removes consolidated quota migration. |
apps/system_manage/migrations/0007_workspaceusergroupresourcepermission.py |
Removes consolidated group-permission migration. |
apps/system_manage/migrations/0006_v3_upgrade.py |
Consolidates schema upgrades and token backfill. |
apps/system_manage/migrations/0006_alter_chatuser_nick_name_chatuserapikey.py |
Removes consolidated chat-user migration. |
apps/maxkb/conf.py |
Adds vector backend and Milvus connection settings. |
apps/knowledge/vector/pg_vector.py |
Implements PostgreSQL index lifecycle hooks. |
apps/knowledge/vector/pg_fulltext.py |
Adds text-ranking and legacy-vector fallback helpers. |
apps/knowledge/vector/milvus_vector.py |
Implements Milvus storage, synchronization, and retrieval. |
apps/knowledge/vector/base_vector.py |
Defines backend index lifecycle hooks. |
apps/knowledge/tests/sync/test_workflow_embedding.py |
Updates index-delegation regression coverage. |
apps/knowledge/tests/retrieval/test_milvus_write.py |
Tests Milvus writes and failure propagation. |
apps/knowledge/tests/retrieval/test_milvus_scope.py |
Tests retrieval filters and multimodal scope. |
apps/knowledge/tests/retrieval/README.md |
Documents retrieval and HTTP regression tests. |
apps/knowledge/tests/fixtures/milvus/default.js |
Simulates Milvus operations and failures. |
apps/knowledge/tests/fixtures/milvus/config.yaml |
Configures the HTTP simulation scenario. |
apps/knowledge/tests/fixtures/milvus.sim.yaml |
Defines simulation routing. |
apps/knowledge/sql/embedding_recall.sql |
Adds PostgreSQL fallback vector recall. |
apps/knowledge/sql/blend_text_rank.sql |
Adds text ranking for recalled candidates. |
apps/knowledge/serializers/common.py |
Delegates index management to the selected backend. |
apps/knowledge/models/knowledge.py |
Allows NULL PostgreSQL vectors for Milvus-backed rows. |
apps/knowledge/migrations/0015_knowledge_external_service.py |
Removes consolidated external-service migration. |
apps/knowledge/migrations/0014_knowledgeworkflow_default_model_setting_and_more.py |
Removes consolidated workflow-settings migration. |
apps/knowledge/migrations/0013_paragraph_asset_recall.py |
Removes consolidated asset-recall migration. |
apps/knowledge/migrations/0012_v3_upgrade.py |
Consolidates knowledge upgrades and nullable vectors. |
apps/knowledge/migrations/0011_publicfileaccess.py |
Removes the former public-access migration path. |
apps/knowledge/migrations/0011_knowledgeworkflow_default_model_setting_and_more.py |
Restores the v2 workflow-settings migration. |
apps/knowledge/migrations/0010_publicfileaccess.py |
Restores the v2 public-access migration path. |
apps/knowledge/migrations/0010_file_storage_type.py |
Removes consolidated file-storage migration. |
apps/common/tests/test_v2_migrations.py |
Tests v2 upgrades and historical data preservation. |
apps/common/db/README.md |
Documents upgrade scope and verification precautions. |
apps/common/config/embedding_config.py |
Registers and selects the Milvus backend. |
apps/application/migrations/0019_applicationversion_publish_desc.py |
Removes consolidated publish-description migration. |
apps/application/migrations/0017_application_is_portal_and_more.py |
Removes consolidated portal migration. |
apps/application/migrations/0016_v3_upgrade.py |
Consolidates application and chat upgrades. |
apps/application/migrations/0016_chatrecord_messages_chatrecord_question_and_more.py |
Removes consolidated chat-record migration. |
apps/application/migrations/0015_chat_execute_type.py |
Removes consolidated chat execution migration. |
apps/application/migrations/0015_application_default_model_setting_and_more.py |
Restores the v2 application-settings migration. |
Review effort: Balanced
Comment on lines
+346
to
+347
| if not ok: | ||
| return [] |
Comment on lines
+471
to
+472
| .filter(id__in=[str(hit.get("id") or "") for hit in hits]) | ||
| .values("id", "paragraph_id", "source_id", "source_type", "meta") |
| meta, | ||
| (1 - (embedding::vector(%s) <=> %s)) AS similarity | ||
| FROM embedding ${embedding_query} | ||
| AND embedding IS NOT NULL |
Comment on lines
+130
to
+131
| if knowledge_id in MilvusVector._loaded: | ||
| return |
| pg_instance = {k: v for k, v in instance.items() if k != "embedding"} | ||
| if pg_instance: | ||
| QuerySet(Embedding).filter(**filter_kwargs).update(**pg_instance) | ||
| self._sync_update(matched, instance) |
| top_n, | ||
| ], | ||
| ) | ||
| return [ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it?
Summary of your change
Please indicate you've done the following: