diff --git a/.agents b/.agents new file mode 120000 index 0000000000..c8161850a4 --- /dev/null +++ b/.agents @@ -0,0 +1 @@ +.claude \ No newline at end of file diff --git a/.cursor/skills/spec-coding/SKILL.md b/.cursor/skills/spec-coding/SKILL.md new file mode 100644 index 0000000000..45647e8677 --- /dev/null +++ b/.cursor/skills/spec-coding/SKILL.md @@ -0,0 +1,448 @@ +--- +name: spec-coding +description: Use for Nexent feature work, architecture changes, database/API changes, multi-file refactors, or any implementation that should be driven by SPEC documentation. Enforces documentation-first development through the Nexent Development SPECs Wiki: organize by implementation status, then feature scope, then lifecycle documents; update requirements, functional design, technical design, and development plan before coding. +--- + +# Spec Coding（规格编码） + +本技能适用于改变产品行为、架构、数据模型、API、持久化、运行时流程或多个模块的 Nexent 编码工作。目标是受控实现：先文档，后开发，并保持 Wiki 作为真相来源。 + +## 事实来源 + +使用飞书 Wiki，名为 `Nexent Development SPECs`。URL: https://dcnvjn24oieg.feishu.cn/wiki/KyU6wFj3siGJ1WkWlu8cTHgwnYb + +**顶级组织按实现状态分类：** + +```text +00 - Wiki Governance and Reading Guide（Wiki治理和阅读指南） +10 - Proposed Specs（提案中的规格） +20 - In Development Specs（开发中的规格） +30 - Implemented Specs（已实现的规格） +40 - Paused or Superseded Specs（暂停或已替代的规格） +90 - Templates and Standards（模板和标准） +``` + +在每个状态分类内，按功能范围组织。在每个功能范围内，使用生命周期文档： + +```text +<功能范围> +├── 00 - Requirement Analysis（需求分析） +├── 01 - Functional Design（功能设计） +├── 02 - Technical Design（技术设计） +└── 03 - Development Plan（开发计划） + └── + └── + └── ... +``` + +如果父页面有子页面，其正文可以包含 `Quick Access`（快速访问）表格，但仅限于直接子页面（深度1）。`Quick Access` 中的每个条目必须是子页面的可点击链接。不要在页面正文中维护全局目录；Wiki UI 已经提供了这个功能。 + +## Mandatory Workflow（强制工作流） + +**所有文档编写必须使用中文。** + +### 阶段一：需求澄清（需求不明确时执行） + +在开始编写任何 SPEC 文档之前，必须先确认需求是否足够清晰以指导代码实现。 + +**触发条件**：当用户需求存在以下任一情况时： +- 功能边界不明确 +- 输入/输出/异常处理未定义 +- 关键设计决策未确定 +- 存在多种实现路径未选择 + +**执行方式**： + +加载并使用 `reference/grilling.md`，逐条向用户澄清需求。 + +示例澄清问题： +- "这个功能的核心输入是什么？数据类型和格式是什么？" +- "输出结果的格式要求是什么？" +- "边界条件和异常场景有哪些？" +- "这个功能和现有模块的集成点在哪里？" +- "性能要求是什么？延迟/吞吐量/TPS 目标？" + +### 阶段二：识别与定位 + +1. **识别功能范围和当前实现状态** + - 确定功能属于哪个状态分类（Proposal/In Development/Implemented/Paused） + - 确定功能范围（Feature Scope） + +2. **定位或创建功能范围** + - 在正确状态分类下找到或创建功能范围节点 + +3. **确保生命周期文档完整** + - `00 - Requirement Analysis`（需求分析） + - `01 - Functional Design`（功能设计） + - `02 - Technical Design`（技术设计） + - `03 - Development Plan`（开发计划） + - 包含多个 Phase 子页面 + +4. **阅读相关生命周期文档后再编辑代码** + +### 阶段三：编码与同步 + +编码过程中： + +- 保持实现与 `03 - Development Plan` 中的 Phase/PR 拆分对齐 +- 如果代码发现使文档失效，先停止广泛实现，先更新相关生命周期文档 +- 保持验收标准、测试、迁移和兼容性要求与文档同步 + +编码完成后： + +- 仅当变更计划、设计或验收标准时，才更新相关生命周期文档的**实现笔记** +- 当生命周期状态变化时，在状态分类之间移动功能范围： + - `10 - Proposed Specs` → `20 - In Development Specs`：实现开始时 + - `20 - In Development Specs` → `30 - Implemented Specs`：实现和验收后 + - 任何活跃状态 → `40 - Paused or Superseded Specs`：暂停、放弃或替换时 + +## Lifecycle Page Responsibilities + +`00 - Requirement Analysis` (需求分析): + +- 问题陈述 +- 目标和非目标 +- 用户或系统影响 +- 约束条件 +- 风险评估 + +`01 - Functional Design` (功能设计): + +- 用户可见或系统可见的行为 +- 能力边界 +- 功能分解 +- 错误处理、空状态、兼容性和迁移行为（如适用） + +`02 - Technical Design` (技术设计): + +- 架构设计 +- 接口和契约 +- 数据模型和schema变更 +- 运行时集成点 +- 向后兼容性策略 + +`03 - Development Plan` (开发计划): + +**核心原则：所有文档必须使用中文编写。** + +开发计划由以下部分组成： +- **主页面**：概述所有 Phase，定义 Phase 之间的依赖关系 +- **Phase 子页面**：每个 Phase 拆分为独立的子文档 + +### 03 - Development Plan 主页面结构 + +```markdown +# <功能范围> - 开发计划 + +## Phase 概览 + +| Phase | 名称 | 状态 | 依赖 | +|-------|------|------|------| +| Phase 1 | 实现配置加载模块 | [ ] | - | +| Phase 2 | 集成内存服务 | [ ] | Phase 1 | +| Phase 3 | 添加单元测试 | [ ] | Phase 2 | + +## Phase 1: 实现配置加载模块 + +- **子页面**：[03.1 - 实现配置加载模块](./03.1%20-%20实现配置加载模块.md) +- **PR**: [待创建] + +## Phase 2: 集成内存服务 + +- **子页面**：[03.2 - 集成内存服务](./03.2%20-%20集成内存服务.md) +- **PR**: [待创建] + +... +``` + +### Phase 子页面结构 + +**每个 Phase 必须是一个独立的子文档**，命名为 `03.N - .md`。 + +--- + +## Phase 子页面模板 + +````markdown +# Phase N: <阶段名称> + +## 基本信息 + +| 属性 | 值 | +|------|-----| +| 所属功能 | <功能范围> | +| 预计工时 | | +| 依赖 Phase | | +| 状态 | [ ] 未开始 / [ ] 进行中 / [ ] 已完成 | + +## 代码设计 + +| 文件 | 类/函数 | 职责 | 伪代码/逻辑说明 | +|------|---------|------|-----------------| +| `src/module_a.py` | `class AgentProcessor` | 处理代理核心逻辑 | 参见 `references/pseudocode-patterns.md` | +| `src/module_a.py` | `AgentProcessor.process()` | 主处理方法 | 完整类伪代码 | +| `src/module_b.py` | `validate_config()` | 配置校验 | 简单的参数校验逻辑 | + +## 伪代码 + +**完整模板与示例见**：`references/pseudocode-patterns.md` + +按需引用以下模板之一： +- 完整类伪代码（适用于主协调器类） +- 数据流伪代码（适用于向量检索、数据变换） +- 状态机伪代码（适用于任务生命周期） + +## 关键设计决策 + +- 决策点 A：说明为什么选择这种实现方式 +- 决策点 B：替代方案及未采用原因 + +## 任务清单 + +### 基础设施与准备 +[ ] N.1 初始化任务描述 +[ ] N.2 依赖安装或配置 + +### 核心实现 +[ ] N.3 核心功能实现 +[ ] N.4 辅助方法实现 + +### 测试与验证 +[ ] N.5 单元测试编写 +[ ] N.6 集成测试验证 + +### 文档与收尾 +[ ] N.7 更新相关文档 +[ ] N.8 代码审查准备 + +## 验收标准 + +[ ] 配置加载模块可正确读取 YAML 配置 +[ ] 配置校验在参数缺失时抛出 ValidationError +[ ] 默认超时时间为 30 秒（可通过配置覆盖） +[ ] 单元测试覆盖率达到 90% 以上 + +## 实现笔记 + +（编码完成后填写，记录实际实现与设计的差异） +```` + +--- + +## Checkbox 格式说明 + +**⚠️ 重要：必须使用 `[ ]` 格式作为 Checkbox，不带前导的 `-` 或数字** + +| 格式 | 含义 | +|------|------| +| `[ ] 任务描述` | 未完成的任务 | +| `[x] 任务描述` | 已完成的任务 | + +**禁止使用以下格式：** +- `- [ ] 任务描述`（错误：多了前导 `-`） +- `1. [ ] 任务描述`（错误：多了数字前缀） + +## 任务清单编写原则 + +- 任务必须小到可以在单次开发会话（1-2小时）内完成 +- 按依赖顺序排列，确保前置任务在前 +- 每个任务可独立验证完成 +- 引用 `01 - Functional Design` 说明要构建什么 +- 引用 `02 - Technical Design` 说明如何构建 + +## Phase/PR 拆分指南 + +- 每个 Phase 对应一个 PR，便于独立审查和回滚 +- Phase 粒度：包含完整功能闭环，可测试、可演示 +- 建议 Phase 数量：每个功能 2-5 个 Phase +- Phase 命名：使用动词短语，如"实现配置加载模块"、"集成内存服务" +- Phase 命名格式：`03.N - <阶段名称>`，例如 `03.1 - 实现配置加载模块` + +## 参考资料（按需加载） + +| 文档 | 何时读取 | +|------|----------| +| `references/pseudocode-patterns.md` | 编写 Phase 子页面时查阅伪代码模板 | +| `references/grilling.md` | 需求不明确、需要澄清时使用 | +| `references/lark-wiki-push-python.md` | 将 Markdown 拆分为 XML 时参考 Python 脚本模式（由 `lark-wiki-spec-push` skill 提供） | + +## Nexent 特定检查 + +**所有文档编写必须使用中文。** + +对于后端工作，保持 `AGENTS.md` 中描述的 app/service/const 层边界。 + +对于环境变量，保持 `backend/consts/const.py` 作为唯一真相来源。SDK 代码不得直接读取环境变量。 + +对于数据库 schema 变更，更新所有必需的位置： + +- `docker/sql/*.sql` 下的版本化迁移脚本 +- Docker Compose 全新部署的 init SQL +- K8s 全新部署的 init SQL +- 如果项目版本控制规则要求，更新 `APP_VERSION` + +对于测试，遵循项目的 pytest 约定，并添加与文档化验收标准匹配的针对性覆盖率。 + +## 何时可以轻量化文档 + +当以下条件全部满足时，小型机械修复可以使用简短的现有范围说明代替完整的生命周期文档集： + +- 变更单一目的且低风险 +- 无 API、数据库、运行时契约或用户可见行为变更 +- 无需跨模块协调 +- 用户明确要求小型修复 + +即使如此，也要提及相关的现有 SPEC 或解释为何不需要更新 SPEC。 + + + +## Feishu Wiki Push 实践指南 + +本文档记录将 Markdown 设计文档推送至飞书 Wiki 的完整工作流，适用于 `Nexent Development SPECs` 空间（space_id = `7660349659210091744`）。 + +### Wiki Token 速查表 + +飞书 Wiki 和 Docx 操作涉及三类 token，含义不同： + +| Token 类型 | 用途 | 获取方式 | +|---|---|---| +| `node_token` | Wiki 层级导航（`wiki +node-*` 系列命令） | `wiki +node-create` 返回 `node_token` | +| `obj_token`（即 `doc_token`）| 文档内容读写（`docs +fetch` / `docs +update`） | `wiki +node-create` 同时返回 `obj_token`，也是文档 URL 中 `/docx/` 后的字段 | +| `space_id` | Wiki 空间标识 | `wiki +space-list` 返回 | + +**常见错误**：将 `node_token` 用于 `docs +update` 的 `--doc` 参数。应始终使用 `obj_token`（即 `doc_token`）。 + +### 认证状态处理 + +```bash +lark-cli auth status --json --verify +``` + +| `user.status` | 含义 | 是否需要干预 | +|---|---|---| +| `ready` | 用户身份可用 | 不需要干预 | +| `needs_refresh` | token 即将过期但仍可写 | 不需要干预，所有写操作仍成功；lark-cli 会在下次 API 调用时自动刷新 | + +### Lifecycle Page 推送：Initial vs Republish + +首次推送一个新功能范围时，使用 `append`（安全，因为失败后可恢复）。 + +**Republish（重新推送已存在的 scope）时的策略**： + +| 页面状态 | 推荐命令 | 原因 | +|---|---|---| +| 页面为空（新创建） | `append` | 安全 | +| 页面有旧内容，需要完整替换 | `overwrite` | 避免重复内容 | +| 页面有旧内容，只需修一个单元格 | `str_replace` | 精确修补已有块 | + +**常见场景**：同一 scope 的 lifecycle 页面 repush 时，00/01/02 通常已存在旧内容，用 `overwrite` 替换；03 主页面用 `overwrite` 或 `append`（取决于是否要保留旧 Phase 概览表）；新增的 Phase 子页面用 `append`。 + +### `--content @file` 的路径基准 + +`--content @filename` 的文件路径**必须相对于当前工作目录（cwd）**，不是脚本所在目录，也不是绝对路径。 + +```bash +# ✅ 正确：cd 到文件所在目录后使用相对路径 +cd .spec_tmp/memory && lark-cli docs +update --doc "$DOC" --command append --content @"./phase_1.xml" + +# ❌ 错误：绝对路径会被 lark-cli 拒绝 +lark-cli docs +update --doc "$DOC" --content @"/mnt/c/Project/nexent/.spec_tmp/memory/phase_1.xml" +``` + +### Phase 子页面推送完整流程 + +创建 Phase 子页面需要三步：**创建 Wiki 节点 → 生成内容 XML → 上传内容 → 更新父页面 sub-page-list**。 + +**步骤 1：创建 Wiki 子节点** + +```bash +# 03 页面的 node_token = OToHwx7p0iQhAmkFQTtcrIs7nmh（已知） +lark-cli wiki +node-create \ + --space-id 7660349659210091744 \ + --parent-node-token OToHwx7p0iQhAmkFQTtcrIs7nmh \ + --title "03.1 - 协议与抽象" \ + --as user --format json +``` + +返回中的 `obj_token`（即 `doc_token`）用于后续 `docs +update`。 + +**步骤 2：用 Python 生成 XML 内容** + +参见 `references/lark-wiki-push-python.md`（由 `lark-wiki-spec-push` skill 提供），或直接复用该 skill 的 `examples.md` 中的 `build_spec_xml.py` 脚本来构建 Phase 内容。 + +**步骤 3：上传内容** + +```bash +cd /path/to/.spec_tmp// && lark-cli docs +update \ + --doc "$OBJ_TOKEN" --command append \ + --as user --format json \ + --content @"./phase_N.xml" +``` + +**步骤 4：在父页面插入 sub-page-list** + +在父 03 页面追加 `` 块（Wiki 特殊块），lark-cli 会自动将所有子节点的 doc_token 填充进去： + +```xml +补充子页面导航 + +``` + +```bash +lark-cli docs +update \ + --doc "$PARENT_OBJ_TOKEN" \ + --command append \ + --as user --format json \ + --content @"sub_page_list.xml" +``` + +验证方式：`wiki +node-list --parent-node-token "$PARENT_NODE_TOKEN"` 确认子节点数量正确。 + +### str_replace 精确修补典型场景 + +当需要修改已有页面中的某个单元格或链接时，用 `str_replace`： + +```bash +# 场景：Quick Access 表格中 03 Development Plan 的链接是占位符，需要替换为真实 URL +lark-cli docs +update \ + --doc "$SCOPE_OBJ_TOKEN" \ + --command str_replace \ + --as user --format json \ + --pattern 'Memory Architecture — 03 Development Plan' \ + --content 'Memory Architecture — 03 Development Plan' +``` + +`str_replace` 在 XML 模式下是行内匹配，`--pattern` 必须完整匹配目标字符串。 + +### 常见陷阱与处理 + +| 陷阱 | 症状 | 处理方式 | +|---|---|---| +| 用 `node_token` 而不是 `doc_token` 调用 `docs +update` | `"ok": false, "error": "invalid doc"` | 确认使用 `wiki +node-create` 返回的 `obj_token` | +| `--content @` 使用绝对路径 | `--file must be a relative path within the current directory` | 先 `cd` 到文件所在目录，再使用相对路径 | +| 对已存在页面用 `append` | 页面出现重复内容 | 用 `overwrite` 替换（republish 场景） | +| 对空页面（新 node-create 无内容）用 `append` | API 返回 badluck 或内容消失 | 改用 `block_insert_after`，在目标 block_id 前插入（block_id 可以是任意合法 id，即使块为空） | +| `needs_refresh` auth 状态 | 担心写操作失败 | 不需要干预，lark-cli 会自动刷新，所有写操作实际成功 | +| for 循环中变量传入 python heredoc | 变量为空 | 放弃 bash 循环，每个 doc 单独写一个命令 | +| 生成的 XML 超过 lark-cli 单次容量 | 上传失败 | 拆分为 ≤ 15 KB 的小块后多次 append | + +### Status Section Token（Nexent SPECs 固定值） + +| Status | node_token | 用途 | +|---|---|---| +| `10 - Proposed Specs` | `KGAEwAceZizF7AkMjCfcVhH6n9e` | 尚未开始实现 | +| `20 - In Development Specs` | `JhSIwbBd2i0e5DkmoqicqC3Dn5e` | 设计 + 开发中（git 有相关代码） | +| `30 - Implemented Specs` | `Kls1wtADQiI3sYk97ilc81v0nYc` | 已合并验收 | +| `40 - Paused or Superseded Specs` | `BJeHwwiiYiEXT4krOhIc2N87nuc` | 暂停或被替代 | + +**判断规则**：git status 有相关代码变更时，推送至 `20 - In Development Specs`；完全无代码时推至 `10 - Proposed Specs`。 + +### Cleanup 强制规则 + +每次推送完成后**必须**立即清理临时文件： + +```bash +rm -rf .spec_tmp// +``` + +`.spec_tmp/` 如果不在 `.gitignore` 中，残留文件会污染 `git status`。本次推送产生的所有 `.spec_tmp/memory/` 文件必须在任务结束前删除。 \ No newline at end of file diff --git a/.cursor/skills/spec-coding/references/grilling.md b/.cursor/skills/spec-coding/references/grilling.md new file mode 100644 index 0000000000..219930f78b --- /dev/null +++ b/.cursor/skills/spec-coding/references/grilling.md @@ -0,0 +1,12 @@ +--- +name: grilling +description: Grill the user relentlessly about a plan or design. Use when the user wants to stress-test a plan before building, or uses any 'grill' trigger phrases. +--- + +Interview me relentlessly about every aspect of this plan until we reach a shared understanding. Walk down each branch of the design tree, resolving dependencies between decisions one-by-one. For each question, provide your recommended answer. + +Ask the questions one at a time, waiting for feedback on each question before continuing. Asking multiple questions at once is bewildering. + +If a *fact* can be found by exploring the codebase, look it up rather than asking me. The *decisions*, though, are mine — put each one to me and wait for my answer. + +Do not enact the plan until I confirm we have reached a shared understanding. diff --git a/.cursor/skills/spec-coding/references/pseudocode-patterns.md b/.cursor/skills/spec-coding/references/pseudocode-patterns.md new file mode 100644 index 0000000000..fa23a2b0d0 --- /dev/null +++ b/.cursor/skills/spec-coding/references/pseudocode-patterns.md @@ -0,0 +1,262 @@ +# 伪代码规范与示例参考 + +本文档详细说明 SPEC-Coding 中伪代码的编写规范和示例。如需查阅具体模板，请阅读对应的小节。 + +## 目录 + +- [1. 伪代码必须包含的要素](#1-伪代码必须包含的要素) +- [2. 完整类伪代码示例](#2-完整类伪代码示例) +- [3. 数据流伪代码示例](#3-数据流伪代码示例) +- [4. 状态机伪代码示例](#4-状态机伪代码示例) +- [5. 选用指南](#5-选用指南) + +--- + +## 1. 伪代码必须包含的要素 + +```pseudocode +# 必须包含： +# 1. 输入/输出：明确参数类型和返回值类型 +# 2. 步骤编号：使用序号标注执行顺序 +# 3. 条件分支：if/else/elif 必须完整写出 +# 4. 循环逻辑：for/while 必须标注边界条件 +# 5. 异常处理：try/except 必须标注可能的异常类型 +# 6. 关键数据流：标注数据从输入到输出的变换过程 +``` + +| 要素 | 作用 | 示例 | +|------|------|------| +| 输入/输出 | 明确函数契约 | `输入: task - Task 对象`
`输出: Result - 包含 status, data, error 属性` | +| 步骤编号 | 标注执行顺序 | `# Step 1: 输入校验` + `1.1. if ...` | +| 条件分支 | 覆盖所有逻辑路径 | `if/else/elif + endif` 完整闭合 | +| 循环边界 | 避免无限循环 | `for each X in items` + `endfor` | +| 异常处理 | 标注异常类型 | `raise InvalidTaskError(...)` | +| 关键数据流 | 数据变换过程 | `query_vec = normalize(query_embedding)` | + +--- + +## 2. 完整类伪代码示例 + +适用于：主协调器类、复杂服务类、含多个方法协作的模块。 + +```pseudocode +class AgentProcessor: + """代理处理器 - 负责执行代理任务的主协调器""" + + # 属性定义 + config: AgentConfig # 配置对象，包含 agent_id, timeout, retry_policy 属性 + cache: Dict[str, Result] # 内存缓存，key 为 memory_id，value 为执行结果 + memory_service: MemoryService # 内存服务依赖，用于获取上下文 + + def __init__(self, config: AgentConfig, memory_service: MemoryService): + """ + 初始化处理器 + 输入: config - AgentConfig 配置对象 + 输入: memory_service - MemoryService 内存服务实例 + """ + self.config = config + self.memory_service = memory_service + self.cache = {} + + async def process(self, task: Task) -> Result: + """ + 主处理方法 - 协调任务执行流程 + 输入: task - Task 对象，包含 task_id, memory_id, payload 属性 + 输出: Result - 包含 status, data, error 属性 + 异常: InvalidTaskError - 任务校验失败时抛出 + """ + # Step 1: 输入校验 + 1.1. if not self._validate_task(task): + raise InvalidTaskError(f"Task {task.task_id} validation failed") + endif + + # Step 2: 检查缓存（避免重复执行） + 2.1. if task.memory_id in self.cache: + return self.cache[task.memory_id] # 命中缓存，直接返回 + endif + + # Step 3: 准备上下文（从 memory_service 获取） + 3.1. context = await self.memory_service.get_context(task.memory_id) + 3.2. if context is None: + context = self._create_empty_context() + endif + + # Step 4: 执行核心逻辑 + 4.1. result = await self._execute_core(task, context) + 4.2. if result.status == "error": + result = await self._handle_error(result.error) + endif + + # Step 5: 更新缓存 + 5.1. self.cache[task.memory_id] = result + + # Step 6: 返回结果 + return result + + def _validate_task(self, task: Task) -> bool: + """ + 校验任务有效性 + 输入: task - Task 对象 + 输出: bool - 校验是否通过 + """ + 1. if task is None: return False + 2. if not hasattr(task, 'task_id'): return False + 3. if not hasattr(task, 'memory_id'): return False + 4. if task.task_id == "": return False + 5. return True + + async def _execute_core(self, task: Task, context: Context) -> Result: + """ + 执行核心业务逻辑 + 输入: task - Task 对象 + 输入: context - Context 对象，包含 memories, metadata 属性 + 输出: Result - 业务执行结果 + """ + 1. agent = self._load_agent(task.agent_id) + 2. prompt = self._build_prompt(task.payload, context) + 3. response = await agent.run(prompt) + 4. return Result(status="success", data=response) + + def _handle_error(self, error: Error) -> Result: + """ + 错误处理逻辑 + 输入: error - Error 对象 + 输出: Result - 错误处理结果 + """ + 1. if error.type == "timeout": + return Result(status="timeout", data=None, error=str(error)) + 2. elif error.type == "rate_limit": + return Result(status="retry_later", data=None, error=str(error)) + 3. else: + return Result(status="error", data=None, error=str(error)) +``` + +**使用场景**： +- 类需要多个方法协作 +- 包含状态管理（缓存、依赖注入） +- 业务流程较长（≥4 个步骤） + +--- + +## 3. 数据流伪代码示例 + +适用于：向量检索、数据变换、ETL 流程、管道处理。 + +```pseudocode +# 数据流伪代码示例：内存向量检索流程 + +INPUT: + - query_embedding: List[float] # 查询向量，维度 1536 + - top_k: int = 5 # 返回前 k 个结果 + - filters: Dict[str, Any] # 元数据过滤条件 + +OUTPUT: + - results: List[MemoryItem] # 检索结果列表 + +PROCESS: + 1. 构建查询向量 + 1.1. query_vec = normalize(query_embedding) # L2 归一化 + 1.2. assert len(query_vec) == 1536, "向量维度必须为 1536" + + 2. 构建过滤条件 + 2.1. if filters is not None: + filter_expr = build_filter_expression(filters) + else: + filter_expr = None + endif + + 3. 执行向量检索 + 3.1. candidates = vector_db.search( + vector=query_vec, + top_k=top_k * 2, # 多取一些，用于后续过滤 + filter=filter_expr + ) + + 4. 后处理与排序 + 4.1. for each candidate in candidates: + 4.1.1. score = cosine_similarity(query_vec, candidate.vector) + 4.1.2. if score >= THRESHOLD: + results.append(candidate) + endif + endfor + + 5. 返回最终结果 + 5.1. return results[:top_k] +``` + +**使用场景**： +- 数据变换流程清晰，输入输出明确 +- 无需维护内部状态 +- 处理步骤是无状态的转换 + +--- + +## 4. 状态机伪代码示例 + +适用于：任务生命周期、订单状态、审批流、有明确状态转换的流程。 + +```pseudocode +# 状态机伪代码示例：任务生命周期管理 + +STATES: + - PENDING: 待处理 + - RUNNING: 执行中 + - COMPLETED: 已完成 + - FAILED: 失败 + - CANCELLED: 已取消 + +INITIAL_STATE: PENDING + +TRANSITIONS: + PENDING -> RUNNING: + 触发条件: worker 接收到任务 + 动作: + 1. update_status(RUNNING) + 2. record_start_time() + 3. acquire_resource() + + RUNNING -> COMPLETED: + 触发条件: 任务正常执行完成 + 动作: + 1. update_status(COMPLETED) + 2. record_end_time() + 3. release_resource() + 4. notify_callback() + + RUNNING -> FAILED: + 触发条件: 执行过程中发生异常 + 动作: + 1. update_status(FAILED) + 2. record_error(error_info) + 3. release_resource() + 4. schedule_retry() if retry_count < MAX_RETRIES + + PENDING/RUNNING -> CANCELLED: + 触发条件: 用户主动取消 + 动作: + 1. update_status(CANCELLED) + 2. release_resource() + 3. cleanup_partial_results() +``` + +**使用场景**： +- 实体具有有限且明确的状态集合 +- 状态之间的转换有明确的触发条件 +- 需要描述转换时的副作用 + +--- + +## 5. 选用指南 + +| 场景特征 | 推荐模板 | +|----------|----------| +| 含类的多个方法、需要状态管理 | 完整类伪代码 | +| 数据变换流程、输入输出明确 | 数据流伪代码 | +| 有有限状态集合和明确转换 | 状态机伪代码 | +| 简单工具函数 | 单函数伪代码（无需模板） | +| API 调用编排 | 序列图（可选） | + +**混用提示**： +- 同一 Phase 中可以混用多种模板 +- 每个文件/类对应一种模板，保持单一职责 +- 状态机适合顶层流程，类伪代码适合具体实现 \ No newline at end of file diff --git a/.github/workflows/auto-unit-test.yml b/.github/workflows/auto-unit-test.yml index 5ade1bad7e..b844a529e6 100644 --- a/.github/workflows/auto-unit-test.yml +++ b/.github/workflows/auto-unit-test.yml @@ -59,6 +59,7 @@ jobs: run: | bash deploy/tests/test_common.sh bash deploy/tests/test_sql_migrations.sh + bash deploy/tests/test_super_admin_init.sh bash deploy/tests/test_build_offline_package.sh - name: Set up Python diff --git a/.github/workflows/auto-web-check-dev.yml b/.github/workflows/auto-web-check-dev.yml index ae831a3fb1..bcaaa8bfdc 100644 --- a/.github/workflows/auto-web-check-dev.yml +++ b/.github/workflows/auto-web-check-dev.yml @@ -36,7 +36,7 @@ jobs: - name: Install dependencies run: | cd frontend - npm install + npm install --legacy-peer-deps - name: Run TypeScript type check run: | diff --git a/.github/workflows/docker-build-push-mainland.yml b/.github/workflows/docker-build-push-mainland.yml index b2ce9453ed..b63d2deb33 100644 --- a/.github/workflows/docker-build-push-mainland.yml +++ b/.github/workflows/docker-build-push-mainland.yml @@ -23,7 +23,7 @@ jobs: strategy: fail-fast: false matrix: - image: [main, web, data-process, mcp, terminal] + image: [main, web, data-process, mcp, terminal, sandbox] steps: - name: Free disk space for data-process if: matrix.image == 'data-process' diff --git a/.github/workflows/docker-build-push-overseas.yml b/.github/workflows/docker-build-push-overseas.yml index ea02dd4100..4757bc5db7 100644 --- a/.github/workflows/docker-build-push-overseas.yml +++ b/.github/workflows/docker-build-push-overseas.yml @@ -23,7 +23,7 @@ jobs: strategy: fail-fast: false matrix: - image: [main, web, data-process, mcp, terminal] + image: [main, web, data-process, mcp, terminal, sandbox] steps: - name: Free disk space for data-process if: matrix.image == 'data-process' diff --git a/.gitignore b/.gitignore index ebce75ae74..8353c14a0f 100644 --- a/.gitignore +++ b/.gitignore @@ -63,7 +63,6 @@ model-assets/ openspec/ logs/ -.agents/ .devspace/ devspace.yaml deploy/k8s/helm/**/*.tgz @@ -77,6 +76,7 @@ sdk/benchmark/.env .venv .pytest-tmp +test/ext_components/aidp/mock_servers/_state/ doc/mermaid _doc/ @@ -85,3 +85,8 @@ _doc/ /deploy/env/.env.bak agent_repository_frontend +.omo/ +.tokensave +.playwright-mcp/ +# Added by code-review-graph +.code-review-graph/ diff --git a/AGENTS.md b/AGENTS.md index a09a544736..f33d7ee0fa 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -24,6 +24,12 @@ Usage notes: + +spec-coding +Use for Nexent feature work, architecture changes, database/API changes, multi-file refactors, or any implementation that should be driven by SPEC documentation. Enforces documentation-first development through the Nexent Development SPECs Wiki: organize by implementation status, then feature scope, then lifecycle documents; update requirements, functional design, technical design, and development plan before coding. +project + + prompts-writing Create, refine, and optimize high-quality YAML prompts for AI assistants. Use when working with prompt templates, system prompts, agent prompts, or any prompt engineering tasks. Provides structure guidelines, template patterns, and quality standards for YAML-based prompts. @@ -53,6 +59,31 @@ Nexent is a zero-code platform for auto-generating AI agents. Monorepo with: --- +## SPEC Coding Workflow (Mandatory) + +For any Nexent feature work, architecture change, database/API change, multi-file refactor, runtime behavior change, or other implementation that can affect product behavior, **invoke and follow the `spec-coding` skill before coding**. + +Development must be documentation-first: +- Use the Feishu Wiki `Nexent Development SPECs` as the source of truth. +- Organize SPEC documents by implementation status first, then feature Scope, then lifecycle document type. +- The expected lifecycle pages are `00 - Requirement Analysis`, `01 - Functional Design`, `02 - Technical Design`, and `03 - Development Plan`. +- Read the relevant lifecycle pages before editing code. +- If required SPEC pages are missing or stale, update the Wiki first, then implement. +- Code changes must trace back to the documented requirements, design, development plan, and acceptance criteria. +- If implementation discoveries invalidate the SPEC, update the relevant lifecycle page before continuing broad code changes. + +Only tiny mechanical fixes may skip a full SPEC update, and only when they do not change API, DB schema, runtime contracts, cross-module behavior, or user-visible behavior. In that case, state why no SPEC update was needed. + +Development must be test-verified against the documented acceptance criteria: +- Unit tests should relact the acceptance criteria and edge cases from the SPEC. +- Unit tests must reach 90% coverage for any new or modified module. +- Integration tests must verify cross-module behavior and runtime flows. +- For all frontend-affected changes, `playwright` must be used to verify user-visible behavior and acceptance criteria. +- For all backend-affected changes, `curl` or `wget` must be used to verify API behavior and acceptance criteria. +- For all SDK-affacted changes, when actual model calls are required to perform functional test, ask the user to provide one, and test with `LangFuse` to trace every step's input and output. + +--- + ## Developer Commands ### Backend (Python 3.11) @@ -165,4 +196,4 @@ Existing instruction files with detailed rules: - `CLAUDE.md` - Backend architecture, env var management, app/service layer rules - `.cursor/rules/environment_variable.mdc` - Env var centralization - `.cursor/rules/pytest_unit_test_rules.mdc` - Testing patterns -- `.cursor/rules/english_comments.mdc` - Comment language enforcement \ No newline at end of file +- `.cursor/rules/english_comments.mdc` - Comment language enforcement diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index 91c809459e..0000000000 --- a/CLAUDE.md +++ /dev/null @@ -1,185 +0,0 @@ -# Claude Code Rules - -## Code Quality Standards - -### English-Only Comments and Documentation -- All comments and docstrings must be written in clear, concise English -- Do not use non-English characters in comments (string literals may contain any language) -- Use proper grammar and spelling; avoid ambiguous abbreviations -- Apply to: docstrings, inline comments, TODO/FIXME/NOTE, header comments, configuration comments - -**Good:** -```python -# Initialize cache for 60 seconds -self.cache_ttl = 60 -``` - -**Bad:** -```python -# 初始化缓存 60 秒 - FORBIDDEN -# データキャッシュ60秒 - FORBIDDEN -``` - -### Environment Variable Management -- All environment variable access must go through `backend/consts/const.py` -- No direct `os.getenv()` or `os.environ.get()` calls outside of `const.py` -- SDK modules (`sdk/`) should never read environment variables directly - accept configuration via parameters -- Services (`backend/services/`) read from `consts.const` and pass config to SDK -- Apps (`backend/apps/`) read from `consts.const` and pass through to services/SDK - -**Good:** -```python -# backend/consts/const.py -APPID = os.getenv("APPID", "") -TOKEN = os.getenv("TOKEN", "") - -# other modules -from consts.const import APPID, TOKEN -``` - -**Bad:** -```python -# direct calls in other modules -import os -appid = os.getenv("APPID") -token = os.environ.get("TOKEN") -``` - -## Backend Architecture Rules - -### App Layer Rules (`backend/apps/**/*.py`) -**Purpose:** HTTP boundary for the backend - parse/validate input, call services, map domain errors to HTTP - -**Responsibilities:** -- Parse and validate HTTP inputs using Pydantic models -- Call underlying services; do not implement core business logic -- Translate domain/service exceptions into `HTTPException` with proper status codes -- Return `JSONResponse(status_code=HTTPStatus.OK, content=payload)` on success - -**Routing and URL Design:** -- Keep existing top-level prefixes for compatibility (e.g., `"/agent"`, `"/memory"`) -- Use plural nouns for collection-style resources (e.g., `"/agents"`, `"/memories"`) -- Use snake_case for all path segments -- Path parameters must be singular, semantic nouns: `"/agents/{agent_id}"` - -**HTTP Methods:** -- GET: Read and list operations only -- POST: Create resources, perform searches, or trigger actions with side effects -- DELETE: Delete resources or clear collections (ensure idempotency) -- PUT/PATCH: Update resources - -**Authorization:** -- Retrieve bearer token via header injection: `authorization: Optional[str] = Header(None)` -- Use utility helpers from `utils.auth_utils` (e.g., `get_current_user_id`) - -**Exception Mapping:** -- `UnauthorizedError` → 401 UNAUTHORIZED -- `LimitExceededError` → 429 TOO_MANY_REQUESTS -- Parameter/validation errors → 400 BAD_REQUEST or 406 NOT_ACCEPTABLE -- Unexpected errors → 500 INTERNAL_SERVER_ERROR - -**Correct Example:** -```python -from http import HTTPStatus -import logging -from fastapi import APIRouter, HTTPException -from starlette.responses import JSONResponse - -from consts.exceptions import LimitExceededError, AgentRunException -from services.agent_service import run_agent - -logger = logging.getLogger(__name__) -router = APIRouter() - -@router.post("/agent/run") -def run_agent_endpoint(payload: dict): - try: - result = run_agent(payload) - return JSONResponse(status_code=HTTPStatus.OK, content=result) - except LimitExceededError as exc: - raise HTTPException(status_code=HTTPStatus.TOO_MANY_REQUESTS, detail=str(exc)) - except AgentRunException as exc: - raise HTTPException(status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail=str(exc)) -``` - -### Service Layer Rules (`backend/services/**/*.py`) -**Purpose:** Implement core business logic orchestration; coordinate repositories/SDKs - -**Requirements:** -- Implement core business logic and orchestrate complex workflows -- Raise domain/service exceptions from `backend/consts/exceptions.py` -- No HTTP concerns (no HTTPException, JSONResponse, etc.) -- No direct environment variable access (use `consts.const`) -- Return plain Python objects, not HTTP responses - -**Available Exceptions:** -- `AgentRunException`: When agent run fails -- `LimitExceededError`: When outer platform calls too frequently -- `UnauthorizedError`: When user from outer platform is unauthorized -- `SignatureValidationError`: When X-Signature header validation fails -- `MemoryPreparationException`: When memory preprocessing/retrieval fails - -**Correct Example:** -```python -from typing import Any, Dict -from consts.exceptions import LimitExceededError, AgentRunException, MemoryPreparationException - -def run_agent(task_payload: Dict[str, Any]) -> Dict[str, Any]: - """Run agent core workflow and return domain result dict.""" - if _is_rate_limited(task_payload): - raise LimitExceededError("Too many requests for this tenant.") - - try: - memory = _prepare_memory(task_payload) - except Exception as exc: - raise MemoryPreparationException("Failed to prepare memory.") from exc - - try: - result = _execute_core_logic(task_payload, memory) - except Exception as exc: - raise AgentRunException("Agent execution failed.") from exc - - return {"status": "ok", "data": result} -``` - -## Migration Checklist - -### Environment Variables -1. Add new vars to `backend/consts/const.py` -2. Update `deploy/env/.env.example` -3. Remove all direct `os.getenv()`/`os.environ.get()` outside `const.py` -4. Import from `consts.const` in backend modules -5. Pass configuration as parameters to SDK -6. Remove `from_env()` methods from config classes - -### Code Quality -1. Convert all non-English comments to English -2. Ensure docstrings use proper English grammar -3. Add module-level loggers: `logger = logging.getLogger(__name__)` -4. Follow existing async/sync conventions in each module - -## File Structure - -``` -backend/ -├── apps/ # HTTP API layer (FastAPI endpoints) -├── services/ # Business logic orchestration -├── consts/ -│ ├── const.py # Single source of truth for env vars -│ └── exceptions.py # Domain exceptions -└── utils/ - └── auth_utils.py # Authentication utilities - -sdk/ # Pure configuration-based, no env vars -``` - -## Validation Rules - -- No direct env access outside `const.py` -- No `from_env()` in config classes -- All env vars defined in `const.py` -- SDK modules accept configuration via parameters -- All comments in English -- Service layer raises domain exceptions only -- App layer maps domain exceptions to HTTP status codes -- Use structured logging with module-level loggers \ No newline at end of file diff --git a/SECURITY.md b/SECURITY.md index d5a81a2f30..9288510449 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -3,7 +3,7 @@ **Please do not report security vulnerabilities through public GitHub issues, discussions, or other public channels.** Instead, please disclose them responsibly by contacting our security team at: -📧 [chenshuangrui@gmail.com](mailto:chenshuangrui@gmail.com) +📧 [zhenggaoqi@huawei.com](mailto:zhenggaoqi@huawei.com) ## What to Include: - Detailed description of the vulnerability diff --git a/VERSION b/VERSION index b1d18bc43f..8721bbc46a 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -v2.3.0 +v2.4.0 diff --git a/backend/agents/agent_run_manager.py b/backend/agents/agent_run_manager.py index 33703fafaa..843660aa71 100644 --- a/backend/agents/agent_run_manager.py +++ b/backend/agents/agent_run_manager.py @@ -1,14 +1,11 @@ -import logging -import threading -from typing import TYPE_CHECKING, Any, Dict, Union - -from nexent.core.agents.agent_model import AgentRunInfo -from services.runtime_state_service import runtime_state_service - -if TYPE_CHECKING: - from nexent.core.agents.agent_context import ContextManager, ContextManagerConfig - -logger = logging.getLogger("agent_run_manager") +import logging +import threading +from typing import Dict, Union + +from nexent.core.agents.agent_model import AgentRunInfo +from services.runtime_state_service import runtime_state_service + +logger = logging.getLogger("agent_run_manager") class AgentRunManager: @@ -25,12 +22,8 @@ def __new__(cls): def __init__(self): if not self._initialized: - # user_id:conversation_id -> agent_run_info - self.agent_runs: Dict[str, AgentRunInfo] = {} - # conversation_id -> ContextManager (conversation-level lifetime) - self._conversation_context_managers: Dict[str, Any] = {} - # conversation_id -> active run count for safe cleanup - self._conversation_run_counts: Dict[str, int] = {} + # user_id:conversation_id -> agent_run_info + self.agent_runs: Dict[str, AgentRunInfo] = {} self._initialized = True def _get_run_key(self, conversation_id: Union[int, str], user_id: str) -> str: @@ -42,28 +35,22 @@ def register_agent_run(self, conversation_id: Union[int, str], agent_run_info, u with self._lock: run_key = self._get_run_key(conversation_id, user_id) self.agent_runs[run_key] = agent_run_info - conv_key = str(conversation_id) - self._conversation_run_counts[conv_key] = self._conversation_run_counts.get(conv_key, 0) + 1 logger.info( f"register agent run instance, user_id: {user_id}, conversation_id: {conversation_id}") - runtime_state_service.register_run(user_id=user_id, conversation_id=conversation_id) + runtime_state_service.register_run(user_id=user_id, conversation_id=conversation_id) - def unregister_agent_run(self, conversation_id: Union[int, str], user_id: str, status: str = "completed"): + def unregister_agent_run(self, conversation_id: Union[int, str], user_id: str, status: str = "completed"): """unregister agent run instance""" with self._lock: run_key = self._get_run_key(conversation_id, user_id) if run_key in self.agent_runs: del self.agent_runs[run_key] - conv_key = str(conversation_id) - self._conversation_run_counts[conv_key] = max( - 0, self._conversation_run_counts.get(conv_key, 0) - 1 - ) logger.info( f"unregister agent run instance, user_id: {user_id}, conversation_id: {conversation_id}") else: logger.info( f"no agent run instance found for user_id: {user_id}, conversation_id: {conversation_id}") - runtime_state_service.mark_run_finished(user_id=user_id, conversation_id=conversation_id, status=status) + runtime_state_service.mark_run_finished(user_id=user_id, conversation_id=conversation_id, status=status) def get_agent_run_info(self, conversation_id: Union[int, str], user_id: str): """get agent run instance""" @@ -72,47 +59,17 @@ def get_agent_run_info(self, conversation_id: Union[int, str], user_id: str): def stop_agent_run(self, conversation_id: Union[int, str], user_id: str) -> bool: """stop agent run for specified conversation_id and user_id""" - remote_signal_set = runtime_state_service.set_cancel_signal( - user_id=user_id, - conversation_id=conversation_id, - ) + remote_signal_set = runtime_state_service.set_cancel_signal( + user_id=user_id, + conversation_id=conversation_id, + ) agent_run_info = self.get_agent_run_info(conversation_id, user_id) if agent_run_info is not None: agent_run_info.stop_event.set() logger.info( f"agent run stopped, user_id: {user_id}, conversation_id: {conversation_id}") return True - return remote_signal_set - - def get_or_create_context_manager( - self, - conversation_id: Union[int, str], - config: "ContextManagerConfig", - max_steps: int - ) -> "ContextManager": - """Get or create a conversation-level ContextManager instance.""" - from nexent.core.agents.agent_context import ContextManager - - conv_key = str(conversation_id) - with self._lock: - cm = self._conversation_context_managers.get(conv_key) - if cm is None: - cm = ContextManager(config=config, max_steps=max_steps) - self._conversation_context_managers[conv_key] = cm - logger.info( - f"Created new ContextManager for conversation_id: {conv_key}") - return cm - - def clear_conversation_context_manager(self, conversation_id: Union[int, str]): - """Explicitly clear the ContextManager for a conversation.""" - conv_key = str(conversation_id) - with self._lock: - cm = self._conversation_context_managers.pop(conv_key, None) - self._conversation_run_counts.pop(conv_key, None) - if cm: - logger.info( - f"Cleared ContextManager for conversation_id: {conv_key}") - + return remote_signal_set # create singleton instance agent_run_manager = AgentRunManager() diff --git a/backend/agents/create_agent_info.py b/backend/agents/create_agent_info.py index e5708904da..474044f891 100644 --- a/backend/agents/create_agent_info.py +++ b/backend/agents/create_agent_info.py @@ -1,13 +1,18 @@ -import json -import threading +import asyncio +import copy +import json import logging +import threading from typing import Any, Dict, List, Optional from urllib.parse import urljoin -from jinja2 import Template, StrictUndefined from nexent.core.utils.observer import MessageObserver from nexent.core.agents.agent_model import AgentRunInfo, ModelConfig, AgentConfig, ToolConfig, ExternalA2AAgentConfig, AgentHistory, AgentVerificationConfig -from nexent.core.agents.summary_config import ContextManagerConfig +from nexent.core.agents.context import ( + ContextManagerConfig, + PolicyLayers, + resolve_policy, +) from nexent.core.models.prompt_cache import resolve_prompt_cache_profile from nexent.core.models.capacity_resolver import ( ModelCapacitySnapshot, @@ -20,7 +25,8 @@ SafeInputBudgetCalculator, UncertaintyReserveBasisUnknown, ) -from nexent.memory.memory_service import search_memory_in_levels +from nexent.core.tools.parallel_executor import ParallelExecutorTool +from nexent.core.agents.sandbox import SandboxConfig from consts.capability_profiles import CATALOG as CAPABILITY_CATALOG @@ -42,20 +48,69 @@ resolve_sub_agent_version_no, ) from database.agent_version_db import query_current_version_no -from database.tool_db import search_tools_for_sub_agent +from database import skill_db +from database.tool_db import query_tools_by_ids, search_tools_for_sub_agent from database.model_management_db import get_model_records, get_model_by_model_id from database.knowledge_db import get_knowledge_name_map_by_index_names from database.client import minio_client from utils.model_name_utils import add_repo_to_name from utils.prompt_template_utils import get_agent_prompt_template from utils.config_utils import tenant_config_manager, get_model_name_from_config -from utils.context_utils import build_context_components -from consts.const import LOCAL_MCP_SERVER, MODEL_CONFIG_MAPPING, LANGUAGE, DATA_PROCESS_SERVICE, MINIO_DEFAULT_BUCKET -from consts.model import AgentToolParamsRequest, ToolParamsRequest +from utils.memory_tool_prompt import build_memory_tool_policy +from utils.automation_tool_prompt import build_automation_tool_policy +from utils.context_utils import build_context_inputs +from utils.redis_utils import get_redis_client +from consts.const import ( + AIDP_API_KEY, + AIDP_SERVER_URL, + AIDP_TENANT_ID, + DATA_PROCESS_SERVICE, + LANGUAGE, + LOCAL_MCP_SERVER, + MINIO_DEFAULT_BUCKET, + MODEL_CONFIG_MAPPING, +) +from consts.model import ToolParamsRequest from consts.exceptions import ValidationError logger = logging.getLogger("create_agent_info") -logger.setLevel(logging.DEBUG) +logger.setLevel(logging.INFO) + + +def _create_fixed_search_memory_tool(): + """Create the internal search tool lazily to keep import boundaries stable.""" + from nexent.core.tools.search_memory_tool import SearchMemoryTool + + return SearchMemoryTool() + + +def _format_long_term_memory_prompt(search_context: Any, language: str) -> str: + """Render tenant and user long-term memories as a system prompt block.""" + sections = [] + section_specs = ( + ( + "tenant_long_term", + "### 租户长期记忆" if language == "zh" else "### Tenant Long-term Memory", + ), + ( + "user_long_term", + "### 用户长期记忆" if language == "zh" else "### User Long-term Memory", + ), + ) + for attribute, heading in section_specs: + entries = [] + for item in getattr(search_context, attribute, ()) or (): + content = ( + item.get("content", "") + if isinstance(item, dict) + else getattr(item, "content", "") + ) + normalized = str(content or "").strip() + if normalized: + entries.append(f"- {normalized}") + if entries: + sections.append("\n".join((heading, *entries))) + return "\n\n".join(sections) # Safe fallback for context-manager token_threshold when no capacity is known. @@ -211,7 +266,7 @@ def _resolve_safe_input_budget( exc, ) return None - logger.info( + logger.debug( "W2 safe input budget resolved: tenant_id=%s model=%s requested_output_tokens=%s " "soft_input_budget_tokens=%s hard_input_budget_tokens=%s fingerprint=%s warnings=%s", tenant_id, @@ -490,7 +545,6 @@ def _get_external_a2a_agents( Returns: List of ExternalA2AAgentConfig for external A2A sub-agents """ - logger.info(f"[_get_external_a2a_agents] START - agent_id={agent_id}, tenant_id={tenant_id}") try: from database import a2a_agent_db @@ -499,26 +553,21 @@ def _get_external_a2a_agents( tenant_id=tenant_id, version_no=version_no, ) - logger.info(f"[_get_external_a2a_agents] DB query returned {len(external_agents)} agents") - logger.debug(f"[_get_external_a2a_agents] agent details: {external_agents}") result = [] for agent in external_agents: agent_url = agent.get("agent_url", "") or _extract_url_from_card(agent.get("raw_card")) if not agent_url: logger.warning( - f"[_get_external_a2a_agents] Skipping agent '{agent.get('name')}' - no URL available" + f"Skipping agent '{agent.get('name')}' - no URL available" ) continue result.append(_build_external_agent_config(agent, agent_url)) - logger.info(f"[_get_external_a2a_agents] returning {len(result)} ExternalA2AAgentConfig") - for i, config in enumerate(result): - logger.info(f" [{i}] name={config.name}, description={config.description}") return result except Exception as e: - logger.error(f"[_get_external_a2a_agents] FAILED: {e}", exc_info=True) + logger.error(f"Get external A2A agents failed: {e}", exc_info=True) return [] @@ -545,6 +594,40 @@ def _get_skill_script_tools( "version_no": version_no, } + skill_config_values: Dict[str, Dict[str, Any]] = {} + try: + from services.skill_service import SkillService + + enabled_skills = SkillService(tenant_id=tenant_id).get_enabled_skills_for_agent( + agent_id=agent_id, + tenant_id=tenant_id, + version_no=version_no, + ) + skill_config_values = { + skill.get("name", ""): dict(skill.get("config_values") or {}) + for skill in enabled_skills + if skill.get("name") + } + except Exception as exc: + logger.debug("Failed to resolve effective skill configuration: %s", exc) + + skill_config_values: Dict[str, Dict[str, Any]] = {} + try: + from services.skill_service import SkillService + + enabled_skills = SkillService(tenant_id=tenant_id).get_enabled_skills_for_agent( + agent_id=agent_id, + tenant_id=tenant_id, + version_no=version_no, + ) + skill_config_values = { + skill.get("name", ""): dict(skill.get("config_values") or {}) + for skill in enabled_skills + if skill.get("name") + } + except Exception as exc: + logger.warning(f"Failed to resolve effective skill configuration: {exc}", exc_info=True) + try: return [ ToolConfig( @@ -575,7 +658,10 @@ def _get_skill_script_tools( description="Read the config.yaml file from a skill directory. Returns JSON containing configuration variables needed for skill workflows.", inputs='{"skill_name": "str"}', output_type="string", - params={"local_skills_dir": CONTAINER_SKILLS_PATH}, + params={ + "local_skills_dir": CONTAINER_SKILLS_PATH, + "config_overrides": skill_config_values, + }, source="builtin", usage="builtin", metadata=skill_context, @@ -656,6 +742,42 @@ async def create_model_config_list(tenant_id): return model_list +def _inject_plan_tools(tools: List[ToolConfig], enable_planning: bool) -> None: + """Inject plan tool configs into the given tools list if enable_planning is True.""" + if not enable_planning: + return + + plan_names = {"create_plan", "update_plan_step"} + if any(t.name in plan_names for t in tools): + return + + # description_zh/zh pairs match the bilingual descriptions in plan_tools.py + tools.extend([ + ToolConfig( + class_name="CreatePlanTool", + name="create_plan", + description="为当前任务创建执行计划。开始执行前调用一次，传入 3-8 个功能块步骤。" + "每个步骤必须有稳定的 id（step-1、step-2、...）、简短标题和详细描述。" + "返回创建的计划 id 和步骤数量。", + inputs='{"plan_id": "string", "title": "string", "steps": "array"}', + output_type="object", + params={}, + source="builtin", + ), + ToolConfig( + class_name="UpdatePlanStepTool", + name="update_plan_step", + description="更新单个计划步骤的状态。完成后调用 status='completed'，不再需要时调用" + " status='skipped'，开始执行时调用 status='in_progress'。" + "返回被更新的步骤 id 和状态。", + inputs='{"step_id": "string", "status": "string"}', + output_type="object", + params={}, + source="builtin", + ), + ]) + + async def create_agent_config( agent_id, tenant_id, @@ -667,6 +789,13 @@ async def create_agent_config( override_model_id: int | None = None, request_requested_output_tokens: int | None = None, tool_params: Optional[ToolParamsRequest | Dict[str, Any]] = None, + conversation_id: Optional[int] = None, + request_context_policy: Optional[Dict[str, Any]] = None, + enable_planning: bool = False, + include_automation_tool: bool = False, + automation_user_message: Optional[str] = None, + automation_model_id: Optional[int] = None, + automation_has_attachments: bool = False, ): normalized_tool_params = _normalize_tool_params_request(tool_params) agent_info = search_agent_info_by_agent_id( @@ -693,6 +822,8 @@ async def create_agent_config( version_no=sub_agent_version_no, override_model_id=None, tool_params=normalized_tool_params, + conversation_id=conversation_id, + include_automation_tool=False, ) managed_agents.append(sub_agent_config) @@ -706,15 +837,51 @@ async def create_agent_config( version_no=version_no, tool_params=normalized_tool_params, ) + memory_tool_names = {"store_memory", "search_memory"} + tool_list = [tool for tool in tool_list if tool.name not in memory_tool_names] + + # Append parallel_executor as an always-available system-managed tool. + # Memory handling is wired separately below: only store_memory is exposed + # to the model, while search_memory runs once during preparation. + tool_list.append(ToolConfig( + class_name=ParallelExecutorTool.__name__, + name=ParallelExecutorTool.name, + description=ParallelExecutorTool.description, + inputs=json.dumps(ParallelExecutorTool.inputs, ensure_ascii=False), + output_type=ParallelExecutorTool.output_type, + params={}, + source="local", + )) + + if ( + include_automation_tool + and conversation_id is not None + and automation_user_message + ): + from services.agent_automation.tool_adapter import ( + agent_loop_automation_tool_adapter, + ) + tool_list.append( + agent_loop_automation_tool_adapter.build_tool_config( + tenant_id=tenant_id, + user_id=user_id, + conversation_id=int(conversation_id), + agent_id=int(agent_id), + user_message=automation_user_message, + agent_version_no=version_no, + model_id=automation_model_id, + tool_params=normalized_tool_params.model_dump(mode="json"), + has_attachments=automation_has_attachments, + language=language, + ) + ) # Build system prompt: prioritize segmented fields, fallback to original prompt field if not available duty_prompt = agent_info.get("duty_prompt", "") constraint_prompt = agent_info.get("constraint_prompt", "") few_shots_prompt = agent_info.get("few_shots_prompt", "") - # Get template content (use manager template if has any sub-agents) is_manager = len(managed_agents) > 0 or len(external_a2a_agents) > 0 - prompt_template = get_agent_prompt_template(is_manager=is_manager, language=language) # Get app information default_app_description = 'Nexent 是一个开源智能体SDK和平台' if language == 'zh' else 'Nexent is an open-source agent SDK and platform' @@ -723,107 +890,210 @@ async def create_agent_config( app_description = tenant_config_manager.get_app_config( 'APP_DESCRIPTION', tenant_id=tenant_id) or default_app_description - # Get memory list - memory_context = build_memory_context(user_id, tenant_id, agent_id, skip_query=not allow_memory_search) - memory_list = [] - if allow_memory_search and memory_context.user_config.memory_switch: - logger.debug("Retrieving memory list...") - memory_levels = ["tenant", "agent", "user", "user_agent"] - if memory_context.user_config.agent_share_option == "never": - memory_levels.remove("agent") - if memory_context.agent_id in memory_context.user_config.disable_agent_ids: - memory_levels.remove("agent") - if memory_context.agent_id in memory_context.user_config.disable_user_agent_ids: - memory_levels.remove("user_agent") + # Memory list population: in the new Memory system this is performed by + # the backend's ``memory_context_service`` via the + # ``MemoryService.search_memory`` facade. The legacy + # ``search_memory_in_levels`` multi-level fan-out has been removed; the + # streaming layer and tool wiring below remain in place. + memory_list: list = [] + long_term_memory_prompt = "" + pre_run_tool_events: list[dict[str, Any]] = [] + memory_context = build_memory_context( + user_id, tenant_id, agent_id, skip_query=not allow_memory_search + ) + # Append active memory tools if memory is enabled + if memory_context.user_config.memory_switch: try: - search_res = await search_memory_in_levels( - query_text=last_user_query, - memory_config=memory_context.memory_config, - tenant_id=memory_context.tenant_id, - user_id=memory_context.user_id, - agent_id=memory_context.agent_id, - memory_levels=memory_levels, + from services.memory_record_service import ( + _resolve_tenant_embedding_model_info, ) - memory_list = search_res.get("results", []) - logger.debug(f"Retrieved memory list: {memory_list}") - except Exception as e: - # Bubble up to streaming layer so it can emit and fall back - raise Exception(f"Failed to retrieve memory list: {e}") - # Append active memory tools if memory is enabled - if memory_context.user_config.memory_switch and memory_context.memory_config: - try: + embedding_configured = ( + _resolve_tenant_embedding_model_info( + str(memory_context.tenant_id or "") + ) + is not None + ) memory_metadata = { - "memory_config": memory_context.memory_config, "memory_user_config": memory_context.user_config, "tenant_id": memory_context.tenant_id, "user_id": memory_context.user_id, "agent_id": memory_context.agent_id, + "conversation_id": ( + str(conversation_id) if conversation_id is not None else "" + ), + "embedding_configured": embedding_configured, } - memory_tool_names = {"store_memory", "search_memory"} - tool_list = [t for t in tool_list if t.name not in memory_tool_names] - - store_tool_config = ToolConfig( - class_name="StoreMemoryTool", - name="store_memory", - description=( - "Save important information to long-term memory for future recall. " - "Use this when the user shares personal preferences, facts about themselves, " - "project context, or instructions that should persist across conversations. " - "Do NOT store transient information like temporary calculations, information " - "already in the knowledge base, or data the user explicitly says to forget." - ), - inputs=json.dumps({ - "content": { - "type": "string", - "description": "The information to remember", - "description_zh": "需要记住的信息" - } - }, ensure_ascii=False), - output_type="string", - params={}, - source="local", - usage=None, - metadata=memory_metadata, - ) - tool_list.append(store_tool_config) - - search_tool_config = ToolConfig( - class_name="SearchMemoryTool", - name="search_memory", - description=( - "Search long-term memory for relevant information from previous interactions. " - "Use this when you need context about the user's preferences, past decisions, " - "or previously discussed topics that aren't in the current conversation. " - "The system already provides some memory context automatically -- use this tool " - "when you need to search for specific information not already available." - ), - inputs=json.dumps({ - "query": { - "type": "string", - "description": "Natural language query describing what to search for", - "description_zh": "描述要搜索内容的自然语言查询" + # Wire the SDK ``MemoryService`` facade to the + # backend services via the adapter. The facade handles policy + # enforcement, embedding lookup, and idempotency on its own + # and dispatches persistence/retrieval to + # ``services.memory_record_service`` / + # ``services.memory_retrieval_service``. + memory_service = None + try: + from services.memory_backend_adapter import build_memory_service_for_agent + + memory_service = build_memory_service_for_agent( + tenant_id=memory_context.tenant_id, + user_id=memory_context.user_id, + agent_id=str(memory_context.agent_id or ""), + ) + if memory_service is None: + raise RuntimeError("MemoryService builder returned no service") + memory_metadata["memory_service"] = memory_service + except Exception as exc: + logger.warning( + "event=memory_service_init_failed tenant_id=%s user_id=%s " + "agent_id=%s error_type=%s", + tenant_id, + user_id, + agent_id, + type(exc).__name__, + ) + + # Hand the internal fixed SearchMemoryTool a backend + # ``MemoryContextService`` so the pre-run search executes + # through the retrieval pipeline (normalize / fusion / + # decay / MMR / token-budget selection) instead of + # bypassing it. The service is reused for prompt injection, + # so a single instance per agent is sufficient. + memory_context_service = None + try: + from services.memory_context_service import get_memory_context_service + + memory_context_service = get_memory_context_service() + if memory_context_service is None: + raise RuntimeError( + "MemoryContextService provider returned no service" + ) + memory_metadata["memory_context_service"] = memory_context_service + except Exception as exc: + logger.warning( + "event=memory_context_service_init_failed tenant_id=%s " + "user_id=%s agent_id=%s error_type=%s", + tenant_id, + user_id, + agent_id, + type(exc).__name__, + ) + + if memory_context_service is not None: + try: + long_term_search_context = await memory_context_service.build_context( + tenant_id=str(memory_context.tenant_id or ""), + user_id=str(memory_context.user_id or ""), + agent_id=str(memory_context.agent_id or "") or None, + conversation_id=( + str(conversation_id) + if conversation_id is not None + else None + ), + query=None, + layers=["tenant", "user"], + ) + long_term_memory_prompt = _format_long_term_memory_prompt( + long_term_search_context, + language, + ) + except Exception as exc: + logger.warning( + "event=long_term_memory_load_failed tenant_id=%s " + "user_id=%s agent_id=%s error_type=%s", + tenant_id, + user_id, + agent_id, + type(exc).__name__, + ) + + if memory_service is not None: + store_tool_config = ToolConfig( + class_name="StoreMemoryTool", + name="store_memory", + description=( + "Store one model-selected and summarized short-term memory extracted only " + "from the conversation between the user and the current agent. Eligible " + "information is limited to user preferences, task goals, action plans and " + "latest progress, or reflections on user feedback and errors. Consider the " + "user question, tool or code execution results, and the final answer. Do not " + "store whole conversations, transient calculations, unverified guesses, " + "duplicates, secrets, or information the user asks to forget. Before every " + "final answer, assess whether an eligible memory was added or updated; if so, " + "calling this tool is mandatory." + ), + inputs=json.dumps({ + "content": { + "type": "string", + "description": ( + "One concise, reusable short-term memory entry already judged, " + "summarized, and deduplicated by the model" + ), + "description_zh": ( + "由模型判断、总结并去重后的一条简洁、可复用的短期记忆" + ) + } + }, ensure_ascii=False), + output_type="string", + params={}, + source="local", + usage=None, + metadata=memory_metadata, + ) + tool_list.append(store_tool_config) + + if memory_context_service is not None: + fixed_search_tool = _create_fixed_search_memory_tool() + fixed_search_tool.memory_context_service = memory_context_service + fixed_search_tool.tenant_id = str(memory_context.tenant_id or "") + fixed_search_tool.user_id = str(memory_context.user_id or "") + fixed_search_tool.agent_id = str(memory_context.agent_id or "") + fixed_search_tool.conversation_id = ( + str(conversation_id) if conversation_id is not None else "" + ) + fixed_search_tool.embedding_configured = embedding_configured + fixed_search_result = await asyncio.to_thread( + fixed_search_tool.forward, + last_user_query or "", + 5, + ) + else: + fixed_search_result = ( + "Memory search unavailable: retrieval pipeline is not configured. " + "Continuing without memory results." + ) + logger.warning( + "event=memory_presearch_skipped tenant_id=%s user_id=%s " + "agent_id=%s conversation_id=%s " + "reason=context_service_unavailable", + tenant_id, + user_id, + agent_id, + conversation_id, + ) + pre_run_tool_events.extend([ + { + "type": "tool", + "content": "", + "tool_name": "search_memory", + "tool_arguments": { + "query": last_user_query or "", + "top_k": 5, }, - "top_k": { - "type": "integer", - "description": "Maximum number of results to return", - "description_zh": "返回结果的最大数量", - "default": 5, - "nullable": True - } - }, ensure_ascii=False), - output_type="string", - params={}, - source="local", - usage=None, - metadata=memory_metadata, - ) - tool_list.append(search_tool_config) - logger.debug("Active memory tools appended to agent tool list") + }, + { + "type": "execution_logs", + "content": fixed_search_result, + }, + ]) + if fixed_search_result.startswith("Found "): + memory_list.append({ + "memory": fixed_search_result, + "memory_level": "agent", + }) except Exception as e: - logger.warning(f"Failed to append active memory tools: {e}") + logger.error(f"Failed to load memory tools: {e}", exc_info=True) # Build knowledge base summary knowledge_base_summary = "" @@ -853,18 +1123,27 @@ async def create_agent_config( except Exception as e: logger.error(f"Failed to build knowledge base summary: {e}") - # Select the context path once. Managed assembly receives raw components - # and must never consume a Jinja-rendered legacy prompt. + # This compatibility flag controls compression only. ContextManager remains + # the single context assembly path when compression is disabled. enable_context_manager = agent_info.get("enable_context_manager", False) - # Assemble legacy system_prompt only for the isolated fallback path. - # Get skills list for prompt template + # Get the skills included in ContextManager items. skills = _get_skills_for_template(agent_id, tenant_id, version_no) is_manager = len(managed_agents) > 0 or len(external_a2a_agents) > 0 builtin_tools = _get_skill_script_tools(agent_id, tenant_id, version_no) available_tools = tool_list + builtin_tools + _inject_plan_tools(available_tools, enable_planning) + memory_tool_policy = build_memory_tool_policy( + language, + (tool.name for tool in available_tools), + ) + automation_tool_policy = build_automation_tool_policy( + language, + (tool.name for tool in available_tools), + ) + render_kwargs = { "duty": duty_prompt, "constraint": constraint_prompt, @@ -879,13 +1158,9 @@ async def create_agent_config( "knowledge_base_summary": knowledge_base_summary, "user_id": user_id, } - system_prompt = "" - if not enable_context_manager: - system_prompt = Template( - prompt_template["system_prompt"], undefined=StrictUndefined - ).render(render_kwargs) - - model_id_to_use = override_model_id if override_model_id else agent_info.get("model_id") + # AgentInfo stores model_ids (a list); pick the first for the primary model lookup + agent_model_ids = agent_info.get("model_ids") + model_id_to_use = override_model_id if override_model_id else (agent_model_ids[0] if agent_model_ids else None) model_info = None if model_id_to_use is not None: model_info = get_model_by_model_id(model_id_to_use, tenant_id=tenant_id) @@ -919,55 +1194,80 @@ async def create_agent_config( hard_input_budget_tokens = 0 context_token_threshold = input_budget - logger.info( - "Agent main LLM: agent_id=%s, model_id=%s, display_name=%s, model_name=%s", - agent_id, - model_id_to_use, - model_info.get("display_name") if model_info else model_name, - model_info.get("model_name") if model_info else model_name, + context_window_tokens = ( + resolved_capacity_snapshot.context_window_tokens + if resolved_capacity_snapshot is not None + and resolved_capacity_snapshot.context_window_tokens is not None + else input_budget ) - # Managed context assembly starts from raw sources. No legacy rendered - # prompt is supplied on this path. - context_components = [] - if enable_context_manager: - context_components = build_context_components( - duty=duty_prompt, - constraint=constraint_prompt, - few_shots=few_shots_prompt, - app_name=app_name, - app_description=app_description, - user_id=user_id, - language=language, - is_manager=is_manager, - tools=render_kwargs["tools"], - skills=skills, - managed_agents=render_kwargs["managed_agents"], - external_a2a_agents=render_kwargs["external_a2a_agents"], - memory_list=memory_list, - memory_search_query=last_user_query, - knowledge_base_summary=knowledge_base_summary, - kb_ids=kb_ids, - ) + context_items = build_context_inputs( + duty=duty_prompt, + constraint=constraint_prompt, + few_shots=few_shots_prompt, + app_name=app_name, + app_description=app_description, + user_id=user_id, + language=language, + is_manager=is_manager, + enable_planning=enable_planning, + tools=render_kwargs["tools"], + skills=skills, + managed_agents=render_kwargs["managed_agents"], + external_a2a_agents=render_kwargs["external_a2a_agents"], + memory_list=memory_list, + memory_search_query=last_user_query, + memory_tool_policy=memory_tool_policy, + automation_tool_policy=automation_tool_policy, + long_term_memory_prompt=long_term_memory_prompt, + knowledge_base_summary=knowledge_base_summary, + kb_ids=kb_ids, + ) - logger.info( - f"Agent {agent_id} context assembly: " - f"skills_count={len(skills)}, " - f"components={[f'{type(c).__name__}(type={c.component_type},priority={c.priority})' for c in context_components]}" - ) + logger.debug( + f"Agent {agent_id} context assembly: " + f"skills_count={len(skills)}, " + f"items={[f'{item.id}(type={item.type.value},priority={item.priority})' for item in context_items]}" + ) + policy_layers = PolicyLayers.model_validate({ + "platform": { + "processing_mode": "adaptive_compact" if enable_context_manager else "passthrough" + }, + "tenant": tenant_config_manager.get_context_policy(tenant_id), + "agent": agent_info.get("context_policy"), + "request": request_context_policy, + }) + effective_context_policy = resolve_policy(policy_layers) + effective_processing_mode = getattr( + effective_context_policy.processing_mode, + "value", + effective_context_policy.processing_mode, + ) + policy_layers_payload = ( + policy_layers.model_dump(mode="json") + if hasattr(policy_layers, "model_dump") + else policy_layers + ) + logger.info( + "Agent %s effective context policy: processing_mode=%s layers=%s", + agent_id, + effective_processing_mode, + policy_layers_payload, + ) cm_config = ContextManagerConfig( - enabled=enable_context_manager, token_threshold=context_token_threshold, + context_window_tokens=context_window_tokens, soft_input_budget_tokens=soft_input_budget_tokens, hard_input_budget_tokens=hard_input_budget_tokens, - strategy="full", + policy_layers=policy_layers, ) + + agent_config = AgentConfig( name="undefined" if agent_info["name"] is None else agent_info["name"], description="undefined" if agent_info["description"] is None else agent_info["description"], prompt_templates=await prepare_prompt_templates( is_manager=len(managed_agents) > 0 or len(external_a2a_agents) > 0, - system_prompt=system_prompt, language=language, agent_id=agent_id ), @@ -979,14 +1279,88 @@ async def create_agent_config( managed_agents=managed_agents, external_a2a_agents=external_a2a_agents, context_manager_config=cm_config, - context_components=context_components, + context_items=context_items, + pre_run_tool_events=pre_run_tool_events, capacity_snapshot=capacity_snapshot, safe_input_budget_snapshot=safe_input_budget_snapshot, verification_config=AgentVerificationConfig.model_validate(agent_info.get("verification_config") or {}), + enable_planning=enable_planning, ) return agent_config +def _resolve_runtime_tool_records( + agent_id: int, + tenant_id: str, + version_no: int = 0, +) -> List[Dict[str, Any]]: + """Merge explicitly enabled tools with tools required by enabled skills.""" + explicit_tools = search_tools_for_sub_agent( + agent_id, + tenant_id, + version_no=version_no, + ) + explicit_tool_ids = { + tool.get("tool_id") for tool in explicit_tools if tool.get("tool_id") is not None + } + + dependency_values: Dict[int, Dict[str, Any]] = {} + dependency_sources: Dict[int, Dict[str, str]] = {} + enabled_skill_instances = skill_db.search_skills_for_agent( + agent_id=agent_id, + tenant_id=tenant_id, + version_no=version_no, + ) + for skill_instance in enabled_skill_instances: + skill = skill_db.get_skill_by_id(skill_instance.get("skill_id"), tenant_id) + if not skill: + continue + effective_config = dict(skill.get("config_values") or {}) + effective_config.update(skill_instance.get("config_values") or {}) + skill_name = skill.get("name") or str(skill.get("skill_id")) + for tool_id in skill.get("tool_ids") or []: + if tool_id in explicit_tool_ids: + continue + values = dependency_values.setdefault(tool_id, {}) + sources = dependency_sources.setdefault(tool_id, {}) + for name, value in effective_config.items(): + if name in values and values[name] != value: + raise ValidationError( + f"Skills '{sources[name]}' and '{skill_name}' configure " + f"tool ID {tool_id} parameter '{name}' with different values." + ) + values[name] = value + sources[name] = skill_name + + implicit_tool_ids = set(dependency_values) - explicit_tool_ids + if not implicit_tool_ids: + return explicit_tools + + implicit_definitions = query_tools_by_ids(list(implicit_tool_ids)) + definitions_by_id = {tool.get("tool_id"): tool for tool in implicit_definitions} + missing_tool_ids = implicit_tool_ids - set(definitions_by_id) + if missing_tool_ids: + raise ValidationError( + f"Enabled skills require missing tools: {sorted(missing_tool_ids)}" + ) + + implicit_tools = [] + for tool_id in sorted(implicit_tool_ids): + tool = copy.deepcopy(definitions_by_id[tool_id]) + if tool.get("is_available") is False: + raise ValidationError( + f"Enabled skills require unavailable tool '{tool.get('name') or tool_id}'." + ) + configured_values = dependency_values[tool_id] + for param in tool.get("params") or []: + param_name = param.get("name") + if param_name in configured_values: + param["default"] = configured_values[param_name] + implicit_tools.append(tool) + + return explicit_tools + implicit_tools + + async def create_tool_config_list( agent_id, tenant_id, @@ -998,8 +1372,11 @@ async def create_tool_config_list( langchain_tools = await discover_langchain_tools() normalized_tool_params = _normalize_tool_params_request(tool_params) - # now only admin can modify the agent, user_id is not used - tools_list = search_tools_for_sub_agent(agent_id, tenant_id, version_no=version_no) + tools_list = _resolve_runtime_tool_records( + agent_id=agent_id, + tenant_id=tenant_id, + version_no=version_no, + ) # Look up agent name for use in error messages. # Agent name is optional for tool_params matching (matching uses tool identifiers only), @@ -1024,6 +1401,44 @@ async def create_tool_config_list( override_params = agent_tool_overrides[tool.get("class_name")] param_dict = _merge_tool_params(tool, override_params) + if tool.get("class_name") == "AidpSearchTool": + # Credentials are backend-owned since the v7.1 permission + # redesign; populate them from the central constants (the + # database row may carry a stale value). + param_dict.pop("server_url", None) + param_dict.pop("api_key", None) + param_dict.pop("tenant_id", None) + param_dict.update({ + "server_url": AIDP_SERVER_URL, + "api_key": AIDP_API_KEY, + "tenant_id": AIDP_TENANT_ID, + }) + + # v7.1: inject the runtime whitelist for AidpSearchTool. The + # permission service recomputes it on every agent call so per-KB + # permission changes take effect immediately without re-publishing + # the agent. Falls back to the configured ``kds_list`` when the + # whitelist lookup fails (defensive path). + _allowed_kds_set: set[str] = set() + _kds_name_to_id_map: dict[str, str] = {} + if tool.get("class_name") == "AidpSearchTool": + try: + from ext_components.aidp.services import ( + aidp_permission_service as _aidp_perms, + ) + _allowed_kds_set = set( + _aidp_perms.get_allowed_kds_list( + user_id=user_id, tenant_id=tenant_id, + ) + ) + _kds_name_to_id_map = _aidp_perms.get_kds_name_to_id_map( + user_id=user_id, tenant_id=tenant_id, + ) + except Exception as exc: # pragma: no cover - defensive + logger.warning( + "Aidp permission lookup failed: %s", exc, + ) + tool_config = ToolConfig( class_name=tool.get("class_name"), name=tool.get("name"), @@ -1035,7 +1450,28 @@ async def create_tool_config_list( usage=tool.get("usage") ) - if tool.get("source") == "langchain": + if tool.get("class_name") == "AidpSearchTool": + # Carry over the runtime whitelist; merge into any existing + # metadata so langchain_tool references that may already be + # attached are preserved. ``tool_config.metadata`` defaults to + # None on ToolConfig, so guard the spread accordingly. + existing = tool_config.metadata if isinstance(tool_config.metadata, dict) else {} + tool_config.metadata = { + **existing, + "allowed_kds_set": _allowed_kds_set, + "kds_name_to_id_map": _kds_name_to_id_map, + } + tool_class_name = tool.get("class_name") + for langchain_tool in langchain_tools: + if langchain_tool.name == tool_class_name: + existing2 = tool_config.metadata if isinstance(tool_config.metadata, dict) else {} + tool_config.metadata = { + **existing2, + "langchain_tool": langchain_tool, + } + break + + if tool.get("source") == "langchain" and tool.get("class_name") != "AidpSearchTool": tool_class_name = tool.get("class_name") for langchain_tool in langchain_tools: if langchain_tool.name == tool_class_name: @@ -1065,10 +1501,28 @@ async def create_tool_config_list( # Build display_name to index_name mapping for LLM parameter conversion # Also build reverse mapping (index_name -> display_name) for knowledge_base_summary index_names = tool_config.params.get("index_names", []) + + # Enforce knowledge-base-level read permission for the chatting user. + # Agent-level permission controls "who can use this agent", but each knowledge + # base has its own "who can read" permission (group_ids + ingroup_permission). + # Filter out any index the current user does NOT have at least read access to, + # so the tool, its display-name mapping, and the injected KB summary all honour + # the per-KB ACL. + if index_names: + index_names = ElasticSearchService.filter_accessible_indices( + index_names, user_id=user_id, tenant_id=tenant_id, + ) + # Persist the filtered list back into params so downstream consumers + # (knowledge_base_summary builder, metadata) see only accessible indices. + tool_config.params["index_names"] = index_names + display_name_to_index_map = {} index_name_to_display_map = {} if index_names: - knowledge_name_map = get_knowledge_name_map_by_index_names(index_names) + knowledge_name_map = get_knowledge_name_map_by_index_names( + index_names, + tenant_id=tenant_id, + ) # Reverse the mapping: display_name (knowledge_name) -> index_name for idx_name, kb_name in knowledge_name_map.items(): display_name_to_index_map[kb_name] = idx_name @@ -1082,12 +1536,27 @@ async def create_tool_config_list( "index_name_to_display_map": index_name_to_display_map, # Internal access control: restrict results to specific document paths (path_or_urls) "document_paths": document_paths, + # Defense-in-depth whitelist: forward() will reject any index not in this list, + # even if the LLM fabricates an unauthorized index name. + "allowed_index_names": list(index_names), } if not index_names: - raise ValidationError( - f"[{agent_name or agent_id}] knowledge_base_search tool requires index_names, " - f"but it is not configured in the agent and not provided via tool_params.") + # Empty after permission filtering means the current user has no read access + # to any of the agent's configured knowledge bases. Instead of skipping the tool + # (which would cause the LLM to hallucinate tool calls against a non-existent tool), + # we keep the tool in the list with empty index_names. The SDK forward() will return + # a clear "no accessible knowledge base" message, allowing the LLM to explain + # the situation to the user instead of entering a retry loop. + logger.warning( + "Keeping knowledge_base_search tool for agent '%s' with no accessible " + "knowledge bases for user '%s' after permission filtering. " + "Tool will return a permission-denial message at search time.", + agent_name or agent_id, user_id, + ) + # Append the tool and skip embedding model lookup (no index to lookup from) + tool_config_list.append(tool_config) + continue embedding_model, _, _ = get_embedding_model_by_index_name(tenant_id, index_names[0]) if not embedding_model: @@ -1174,7 +1643,6 @@ async def discover_langchain_tools(): async def prepare_prompt_templates( is_manager: bool, - system_prompt: str, language: str = 'zh', agent_id: int = None, ): @@ -1183,7 +1651,6 @@ async def prepare_prompt_templates( Args: is_manager: Whether it is a manager mode - system_prompt: System prompt content language: Language code ('zh' or 'en') agent_id: Agent ID for fetching skill instances @@ -1191,7 +1658,10 @@ async def prepare_prompt_templates( dict: Prompt template configuration """ prompt_templates = get_agent_prompt_template(is_manager, language) - prompt_templates["system_prompt"] = system_prompt + # Stable context is assembled exclusively by ContextManager. Keep the key + # for smolagents prompt-template compatibility, but never source it from a + # second rendering path. + prompt_templates["system_prompt"] = "" return prompt_templates @@ -1254,7 +1724,7 @@ async def join_minio_file_description_to_query( # Enforce file count limit (keep most recent files by truncating from the end) if len(all_files) > max_files: all_files = all_files[:max_files] - logger.debug(f"File list truncated from {len(all_files)} to {max_files} files") + logger.info(f"File list truncated from {len(all_files)} to {max_files} files") if all_files: file_descriptions: list[str] = [] @@ -1284,7 +1754,7 @@ async def join_minio_file_description_to_query( # Check if adding this description would exceed the character limit if total_len > max_chars: - logger.debug( + logger.info( f"File descriptions truncated at {len(file_descriptions)} files " f"to stay within {max_chars} character limit" ) @@ -1396,6 +1866,10 @@ async def create_agent_run_info( override_model_id: int | None = None, requested_output_tokens: int | None = None, tool_params: Optional[ToolParamsRequest | Dict[str, Any]] = None, + conversation_id: Optional[int] = None, + context_policy: Optional[Dict[str, Any]] = None, + enable_planning: bool = False, + enable_automation_tool: bool = True, ): # Determine which version_no to use based on is_debug flag # If is_debug=false, use the current published version (current_version_no) @@ -1424,11 +1898,22 @@ async def create_agent_run_info( "last_user_query": final_query, "allow_memory_search": allow_memory_search, "version_no": version_no, + "conversation_id": conversation_id, + "enable_planning": enable_planning, } + if enable_automation_tool and not is_debug and conversation_id is not None: + create_config_kwargs.update({ + "include_automation_tool": True, + "automation_user_message": query, + "automation_model_id": override_model_id, + "automation_has_attachments": bool(minio_files), + }) if override_model_id is not None: create_config_kwargs["override_model_id"] = override_model_id if requested_output_tokens is not None: create_config_kwargs["request_requested_output_tokens"] = requested_output_tokens + if context_policy is not None: + create_config_kwargs["request_context_policy"] = context_policy agent_config = await create_agent_config(**create_config_kwargs, tool_params=tool_params) @@ -1477,6 +1962,16 @@ async def create_agent_run_info( # Convert HistoryItem (from API) to AgentHistory (expected by SDK) converted_history = _convert_history_with_minio_files(history) + # Resolve sandbox config: DB policy overrides env-var defaults. + # build_sandbox_policy returns None when level=local (backward-compatible). + # Import inside function body to avoid circular dependency. + from services.agent_service import build_sandbox_policy, get_sandbox_minio_client + sandbox_policy = build_sandbox_policy(tenant_id=tenant_id, agent_type="") + agent_db_policy = getattr(agent_config, "sandbox_policy", None) + merged_policy = sandbox_policy if sandbox_policy else agent_db_policy + sandbox_config = SandboxConfig.from_dict(merged_policy) if merged_policy else None + minio_client = get_sandbox_minio_client() if sandbox_config and sandbox_config.auto_sync_outputs else None + agent_run_info = AgentRunInfo( query=final_query, model_config_list=model_list, @@ -1491,5 +1986,8 @@ async def create_agent_run_info( "safe_input_budget_snapshot", None, ), + sandbox_config=sandbox_config, + minio_client=minio_client, + redis_client=get_redis_client(), ) return agent_run_info diff --git a/backend/agents/nl2agent_agent.py b/backend/agents/nl2agent_agent.py new file mode 100644 index 0000000000..2a635a300d --- /dev/null +++ b/backend/agents/nl2agent_agent.py @@ -0,0 +1,50 @@ +"""Build the ephemeral NL2Agent configuration.""" + +from jinja2 import StrictUndefined, Template +from nexent.core.agents.agent_model import AgentConfig + +from consts.const import LANGUAGE +from tool_collection.mcp.nl2agent_mcp_tools import ( + MAX_TOOL_RECOMMENDATIONS, + NL2A_WRAPPER_NAME, + SEARCH_INSTALLED_MCP_TOOLS_NAME, + create_nl2agent_mcp_tool_configs, +) +from utils.prompt_template_utils import get_prompt_template + +NL2AGENT_NAME = "__nl2agent_runtime__" + + +def build_nl2agent_system_prompt( + language: str, + tool_name: str = SEARCH_INSTALLED_MCP_TOOLS_NAME, + wrapper_name: str = NL2A_WRAPPER_NAME, + max_results: int = MAX_TOOL_RECOMMENDATIONS, +) -> str: + """Load and render the localized NL2Agent system prompt.""" + + template_language = ( + LANGUAGE["EN"] if language == LANGUAGE["EN"] else LANGUAGE["ZH"] + ) + template = get_prompt_template("nl2agent", template_language)["system_prompt"] + return Template(template, undefined=StrictUndefined).render( + tool_name=tool_name, + wrapper_name=wrapper_name, + max_results=max_results, + ) + + +def create_nl2agent_agent_config(language: str) -> AgentConfig: + """Create the in-memory AgentConfig for one NL2Agent request.""" + + return AgentConfig( + name=NL2AGENT_NAME, + description="Ephemeral natural-language agent builder", + prompt_templates=None, + tools=create_nl2agent_mcp_tool_configs(), + max_steps=5, + model_name="main_model", + provide_run_summary=False, + instructions=build_nl2agent_system_prompt(language), + enable_planning=False, + ) diff --git a/backend/apps/a2a_client_app.py b/backend/apps/a2a_client_app.py index ea149ac319..894da5cac7 100644 --- a/backend/apps/a2a_client_app.py +++ b/backend/apps/a2a_client_app.py @@ -6,7 +6,7 @@ """ import logging import uuid -from typing import Annotated, List, Optional +from typing import Annotated, Dict, List, Optional from http import HTTPStatus from fastapi import APIRouter, Depends, Header, HTTPException, Query, Request @@ -27,9 +27,10 @@ class DiscoverFromUrlRequest(BaseModel): - """Request to discover external A2A agent from URL.""" + """Request to discover an external A2A agent from an Agent Card URL.""" url: str name: Optional[str] = None + custom_headers: Optional[Dict[str, str]] = None class DiscoverFromNacosRequest(BaseModel): @@ -46,6 +47,12 @@ class UpdateAgentProtocolRequest(BaseModel): ) +class UpdateExternalAgentSecurityCredentialsRequest(BaseModel): + """Request to configure credentials required by an Agent Card.""" + security_credentials: Dict[str, str] = Field(default_factory=dict) + selected_security_requirement_index: Optional[int] = Field(default=None, ge=0) + + class TestNacosConnectionRequest(BaseModel): """Request to test Nacos connectivity without saving the config.""" nacos_addr: str = Field(description="Nacos server address (e.g., http://nacos-server:8848)") @@ -74,7 +81,8 @@ async def discover_from_url( result = await a2a_client_service.discover_from_url( url=request.url, tenant_id=tenant_id, - user_id=user_id + user_id=user_id, + custom_headers=request.custom_headers ) return JSONResponse( @@ -203,6 +211,91 @@ async def get_external_agent( ) +@router.put("/agents/{external_agent_id}/security-credentials") +async def update_external_agent_security_credentials( + external_agent_id: int, + request: UpdateExternalAgentSecurityCredentialsRequest, + authorization: Annotated[Optional[str], Header()] = None, + http_request: Request = None, +): + """Save the credential values required to call an external A2A agent.""" + try: + user_id, tenant_id, _ = get_current_user_info(authorization, http_request) + if not request.security_credentials and request.selected_security_requirement_index is None: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="At least one security credential or a security requirement selection is required" + ) + if any(not scheme_id or not value for scheme_id, value in request.security_credentials.items()): + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="Security credential names and values must be non-empty" + ) + agent = a2a_agent_db.get_external_agent_by_id( + external_agent_id, + tenant_id, + include_security_credentials=True, + ) + if not agent: + raise HTTPException( + status_code=HTTPStatus.NOT_FOUND, + detail=f"Agent {external_agent_id} not found" + ) + supported_scheme_ids = set((agent.get("security_schemes") or {}).keys()) + if not set(request.security_credentials).issubset(supported_scheme_ids): + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="Security credentials must use scheme names declared by the Agent Card" + ) + requirements = agent.get("security_requirements") or [] + if request.selected_security_requirement_index is not None: + selected_index = request.selected_security_requirement_index + if selected_index >= len(requirements): + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="Selected security requirement does not exist" + ) + selected_schemes = (requirements[selected_index] or {}).get("schemes", {}) + configured_credentials = dict(agent.get("security_credentials") or {}) + configured_credentials.update(request.security_credentials) + if not selected_schemes or not set(selected_schemes).issubset(set(configured_credentials)): + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="Credentials must satisfy the selected security requirement" + ) + result = a2a_agent_db.update_external_agent_security_credentials( + external_agent_id=external_agent_id, + tenant_id=tenant_id, + user_id=user_id, + security_credentials=request.security_credentials, + selected_security_requirement_index=request.selected_security_requirement_index, + ) + if not result: + raise HTTPException( + status_code=HTTPStatus.NOT_FOUND, + detail=f"Agent {external_agent_id} not found" + ) + agent = a2a_agent_db.get_external_agent_by_id(external_agent_id, tenant_id) + return JSONResponse( + status_code=HTTPStatus.OK, + content={ + "status": "success", + "data": { + "configured_security_scheme_ids": agent["configured_security_scheme_ids"], + "selected_security_requirement_index": agent.get("selected_security_requirement_index"), + }, + }, + ) + except HTTPException: + raise + except Exception as e: + logger.error(f"Update agent security credentials failed: {e}", exc_info=True) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail="Failed to update agent security credentials" + ) + + @router.post("/agents/{external_agent_id}/refresh") async def refresh_agent_card( external_agent_id: int, diff --git a/backend/apps/agent_app.py b/backend/apps/agent_app.py index a0aa9d3838..5d47f9f823 100644 --- a/backend/apps/agent_app.py +++ b/backend/apps/agent_app.py @@ -5,11 +5,33 @@ from fastapi import APIRouter, Body, Header, HTTPException, Request, Query from fastapi.encoders import jsonable_encoder -from starlette.responses import JSONResponse, Response +from starlette.responses import JSONResponse, Response, StreamingResponse from consts.const import ASSET_OWNER_TENANT_ID -from consts.model import AgentRequest, AgentInfoRequest, AgentIDRequest, ConversationResponse, AgentImportRequest, AgentNameBatchCheckRequest, AgentNameBatchRegenerateRequest, VersionPublishRequest, VersionListResponse, VersionDetailResponse, VersionRollbackRequest, VersionStatusRequest, CurrentVersionResponse, VersionCompareRequest, VersionUpdateRequest -from consts.exceptions import SkillDuplicateError +from consts.model import ( + AgentRequest, + AgentInfoRequest, + AgentIDRequest, + ConversationResponse, + AgentImportRequest, + AgentNameBatchCheckRequest, + AgentNameBatchRegenerateRequest, + VersionPublishRequest, + VersionListResponse, + VersionDetailResponse, + VersionRollbackRequest, + VersionStatusRequest, + CurrentVersionResponse, + VersionCompareRequest, + VersionUpdateRequest, + NL2AgentRunRequest, +) +from consts.exceptions import ( + ForbiddenError, + SkillDuplicateError, + AppException, + UnauthorizedError, +) from services.asset_owner_visibility import apply_agent_detail_prompt_visibility from services.agent_service import ( @@ -30,6 +52,8 @@ export_agent_with_skills_impl, import_agent_with_skills_impl, ) +from services.prompt_service import generate_guardrail_rules_impl +from services.nl2agent_service import create_nl2agent_stream from services.agent_version_service import ( publish_version_impl, get_version_list_impl, @@ -70,6 +94,8 @@ async def agent_run_api( authorization=authorization, resume=resume, ) + except ForbiddenError as e: + raise HTTPException(status_code=HTTPStatus.FORBIDDEN, detail=str(e)) from e except Exception as e: logger.error(f"Agent run error: {str(e)}") # Only expose actual error in debug mode for better diagnosis @@ -79,6 +105,38 @@ async def agent_run_api( status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail=error_detail) +@agent_runtime_router.post("/nl2agent/run") +async def nl2agent_run_api( + nl2agent_request: NL2AgentRunRequest, + http_request: Request, + authorization: Optional[str] = Header(None), +): + """Run one non-persistent NL2Agent turn.""" + + try: + _, tenant_id, language = get_current_user_info( + authorization, http_request + ) + stream = await create_nl2agent_stream( + request=nl2agent_request, + tenant_id=tenant_id, + language=language, + authorization=authorization, + ) + return StreamingResponse(stream, media_type="text/event-stream") + except UnauthorizedError as exc: + raise HTTPException( + status_code=HTTPStatus.UNAUTHORIZED, + detail=str(exc), + ) from exc + except Exception as exc: + logger.exception("NL2Agent run error") + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail="NL2Agent run error.", + ) from exc + + @agent_runtime_router.get("/stop/{conversation_id}") async def agent_stop_api(conversation_id: int, authorization: Optional[str] = Header(None)): """ @@ -161,6 +219,57 @@ async def update_agent_info_api(request: AgentInfoRequest, authorization: Option status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail="Agent update error.") +@agent_config_router.post("/generate_guardrail_rules") +async def generate_guardrail_rules_api( + http_request: Request, + description: str = Body(..., embed=True), + model_id: int = Body(..., embed=True), + language: str = Body("zh", embed=True), + authorization: Optional[str] = Header(None), +): + """Generate guardrail regex rules from a natural-language description. + + Derives tenant_id and language from the authenticated caller, delegates to + :func:`generate_guardrail_rules_impl`, and wraps the result as JSON. + + Args: + http_request: Incoming HTTP request, used for auth-context resolution. + description: Natural-language description of what to match or block. + model_id: ID of the LLM model to use for generation. + language: Language override ('zh' or 'en'); falls back to the auth + context language when not provided or empty. + authorization: Bearer token header used to derive the tenant_id. + + Returns: + JSONResponse with ``{"message": "Success", "data": }`` on + success, or ``{"message": , "data": null}`` on AppException. + """ + _, tenant_id, auth_language = get_current_user_info(authorization, http_request) + try: + result = generate_guardrail_rules_impl( + description=description, + model_id=model_id, + tenant_id=tenant_id, + language=auth_language or language, + ) + return JSONResponse( + status_code=HTTPStatus.OK, + content={"message": "Success", "data": result}, + ) + except AppException as e: + logger.exception(f"Generate guardrail rules error: {e}") + return JSONResponse( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + content={"message": str(e), "data": None}, + ) + except Exception as e: + logger.exception(f"Generate guardrail rules error: {e}") + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail="Generate guardrail rules error.", + ) + + @agent_config_router.delete("") async def delete_agent_api( request: AgentIDRequest, @@ -301,18 +410,22 @@ async def list_all_agent_info_api( list all agent info """ try: - user_id, tenant_id, _ = get_current_user_info( + user_id, auth_tenant_id, _ = get_current_user_info( authorization, request) - agent_list = await list_all_agent_info_impl( + if tenant_id is None: + agent_list = await list_all_agent_info_impl( + tenant_id=auth_tenant_id, user_id=user_id + ) + if auth_tenant_id != ASSET_OWNER_TENANT_ID: + asset_agent_list = await list_all_agent_info_impl( + tenant_id=ASSET_OWNER_TENANT_ID, user_id=user_id + ) + return agent_list + asset_agent_list + return agent_list + return await list_all_agent_info_impl( tenant_id=tenant_id, user_id=user_id ) - if tenant_id != ASSET_OWNER_TENANT_ID: - asset_agent_list = await list_all_agent_info_impl( - tenant_id=ASSET_OWNER_TENANT_ID, user_id=user_id - ) - return agent_list + asset_agent_list - return agent_list except Exception as e: logger.error(f"Agent list error: {str(e)}") raise HTTPException( @@ -627,5 +740,3 @@ async def list_published_agents_api( raise HTTPException( status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail="Published agents list error." ) - - diff --git a/backend/apps/agent_automation_app.py b/backend/apps/agent_automation_app.py new file mode 100644 index 0000000000..75137b0f1f --- /dev/null +++ b/backend/apps/agent_automation_app.py @@ -0,0 +1,243 @@ +import logging +from http import HTTPStatus +from typing import Annotated, Optional + +from fastapi import APIRouter, Header, HTTPException, Query + +from consts.exceptions import UnauthorizedError +from services.agent_automation.errors import ( + AgentAutomationError, + AutomationConversationAlreadyBoundError, + AutomationNotFoundError, +) +from services.agent_automation.facade import agent_automation_facade +from services.agent_automation.models import ( + AutomationProposalConfirmRequest, + AutomationProposalCreateRequest, + AutomationProposalPatchRequest, + AutomationResponse, + AutomationTaskPatchRequest, +) +from utils.auth_utils import get_current_user_id + +logger = logging.getLogger("agent_automation_app") + +router = APIRouter(prefix="/agent/automations") +conversation_automation_router = APIRouter(prefix="/conversation") + + +def _get_current_user(authorization: Optional[str]) -> tuple[str, str]: + try: + return get_current_user_id(authorization) + except UnauthorizedError as exc: + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(exc)) from exc + + +def _map_error(exc: AgentAutomationError) -> HTTPException: + if isinstance(exc, AutomationNotFoundError): + status = HTTPStatus.NOT_FOUND + elif isinstance(exc, AutomationConversationAlreadyBoundError): + status = HTTPStatus.CONFLICT + else: + status = HTTPStatus.BAD_REQUEST + return HTTPException( + status_code=status, + detail={ + "code": exc.error_code, + "message": exc.message, + "details": exc.details, + }, + ) + + +@router.post("/proposals", response_model=AutomationResponse) +async def create_proposal(request: AutomationProposalCreateRequest, authorization: Optional[str] = Header(None)): + try: + user_id, tenant_id = _get_current_user(authorization) + data = await agent_automation_facade.create_proposal(request, tenant_id, user_id) + return AutomationResponse(data=data) + except AgentAutomationError as exc: + raise _map_error(exc) + except HTTPException: + raise + except Exception as exc: + logger.error("Failed to create automation proposal: %s", exc, exc_info=True) + raise HTTPException(status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail=str(exc)) + + +@router.post("/proposals/{proposal_id}/confirm", response_model=AutomationResponse) +async def confirm_proposal( + proposal_id: int, + request: AutomationProposalConfirmRequest | None = None, + authorization: Optional[str] = Header(None), +): + try: + user_id, tenant_id = _get_current_user(authorization) + data = await agent_automation_facade.confirm_proposal( + proposal_id, + request or AutomationProposalConfirmRequest(), + tenant_id, + user_id, + ) + return AutomationResponse(data=data) + except AgentAutomationError as exc: + raise _map_error(exc) + except HTTPException: + raise + except Exception as exc: + logger.error("Failed to confirm automation proposal: %s", exc, exc_info=True) + raise HTTPException(status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail=str(exc)) + + +@router.patch("/proposals/{proposal_id}", response_model=AutomationResponse) +async def update_proposal( + proposal_id: int, + request: AutomationProposalPatchRequest, + authorization: Optional[str] = Header(None), +): + try: + user_id, tenant_id = _get_current_user(authorization) + data = await agent_automation_facade.update_proposal( + proposal_id, + request, + tenant_id, + user_id, + ) + return AutomationResponse(data=data) + except AgentAutomationError as exc: + raise _map_error(exc) + except HTTPException: + raise + except Exception as exc: + logger.error("Failed to update automation proposal: %s", exc, exc_info=True) + raise HTTPException(status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail=str(exc)) + + +@router.get("", response_model=AutomationResponse) +async def list_tasks( + status: Optional[str] = Query(default=None), + search: Optional[str] = Query(default=None, max_length=200), + agent_name: Optional[str] = Query(default=None, max_length=200), + page: Annotated[int, Query(ge=1)] = 1, + page_size: Annotated[int, Query(ge=1, le=100)] = 20, + authorization: Optional[str] = Header(None), +): + try: + user_id, tenant_id = _get_current_user(authorization) + return AutomationResponse( + data=agent_automation_facade.list_tasks( + tenant_id, + user_id, + status, + search, + agent_name, + page, + page_size, + ) + ) + except HTTPException: + raise + except Exception as exc: + logger.error("Failed to list automation tasks: %s", exc, exc_info=True) + raise HTTPException(status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail=str(exc)) + + +@router.get("/{task_id}", response_model=AutomationResponse) +async def get_task(task_id: int, authorization: Optional[str] = Header(None)): + try: + user_id, tenant_id = _get_current_user(authorization) + return AutomationResponse(data=agent_automation_facade.get_task(task_id, tenant_id, user_id)) + except AgentAutomationError as exc: + raise _map_error(exc) + + +@router.patch("/{task_id}", response_model=AutomationResponse) +async def patch_task(task_id: int, request: AutomationTaskPatchRequest, authorization: Optional[str] = Header(None)): + try: + user_id, tenant_id = _get_current_user(authorization) + data = await agent_automation_facade.patch_task(task_id, request, tenant_id, user_id) + return AutomationResponse(data=data) + except AgentAutomationError as exc: + raise _map_error(exc) + + +@router.post("/{task_id}/pause", response_model=AutomationResponse) +async def pause_task(task_id: int, authorization: Optional[str] = Header(None)): + try: + user_id, tenant_id = _get_current_user(authorization) + return AutomationResponse(data=agent_automation_facade.pause_task(task_id, tenant_id, user_id)) + except AgentAutomationError as exc: + raise _map_error(exc) + + +@router.post("/{task_id}/resume", response_model=AutomationResponse) +async def resume_task(task_id: int, authorization: Optional[str] = Header(None)): + try: + user_id, tenant_id = _get_current_user(authorization) + return AutomationResponse(data=agent_automation_facade.resume_task(task_id, tenant_id, user_id)) + except AgentAutomationError as exc: + raise _map_error(exc) + + +@router.post("/{task_id}/run", response_model=AutomationResponse) +async def run_task_now(task_id: int, authorization: Optional[str] = Header(None)): + try: + user_id, tenant_id = _get_current_user(authorization) + return AutomationResponse(data=await agent_automation_facade.run_task_now(task_id, tenant_id, user_id)) + except AgentAutomationError as exc: + raise _map_error(exc) + + +@router.delete("/{task_id}", response_model=AutomationResponse) +async def delete_task(task_id: int, authorization: Optional[str] = Header(None)): + try: + user_id, tenant_id = _get_current_user(authorization) + return AutomationResponse(data=agent_automation_facade.delete_task(task_id, tenant_id, user_id)) + except AgentAutomationError as exc: + raise _map_error(exc) + + +@router.get("/{task_id}/runs", response_model=AutomationResponse) +async def list_runs( + task_id: int, + authorization: Optional[str] = Header(None), + page: Annotated[int, Query(ge=1)] = 1, + page_size: Annotated[int, Query(ge=1, le=100)] = 20, +): + try: + user_id, tenant_id = _get_current_user(authorization) + return AutomationResponse(data=agent_automation_facade.list_runs(task_id, tenant_id, user_id, page, page_size)) + except AgentAutomationError as exc: + raise _map_error(exc) + + +@router.post("/runs/{run_id}/cancel", response_model=AutomationResponse) +async def cancel_run(run_id: int, authorization: Optional[str] = Header(None)): + try: + user_id, tenant_id = _get_current_user(authorization) + return AutomationResponse(data=agent_automation_facade.cancel_run(run_id, tenant_id, user_id)) + except AgentAutomationError as exc: + raise _map_error(exc) + + +@router.delete("/runs/{run_id}", response_model=AutomationResponse) +async def delete_run(run_id: int, authorization: Optional[str] = Header(None)): + try: + user_id, tenant_id = _get_current_user(authorization) + return AutomationResponse(data=agent_automation_facade.delete_run(run_id, tenant_id, user_id)) + except AgentAutomationError as exc: + raise _map_error(exc) + + +@conversation_automation_router.get("/{conversation_id}/automation", response_model=AutomationResponse) +async def get_conversation_automation(conversation_id: int, authorization: Optional[str] = Header(None)): + try: + user_id, tenant_id = _get_current_user(authorization) + return AutomationResponse( + data=agent_automation_facade.get_task_for_conversation(conversation_id, tenant_id, user_id) + ) + except HTTPException: + raise + except Exception as exc: + logger.error("Failed to get conversation automation: %s", exc, exc_info=True) + raise HTTPException(status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail=str(exc)) diff --git a/backend/apps/agent_repository_app.py b/backend/apps/agent_repository_app.py index f7f0632826..37a023b8d1 100644 --- a/backend/apps/agent_repository_app.py +++ b/backend/apps/agent_repository_app.py @@ -76,6 +76,10 @@ async def list_my_editable_agents_api( False, description="Reserve first slot on page 1 for create-agent placeholder", ), + agent_id: Optional[int] = Query( + None, + description="Filter to a single agent by agent_id", + ), authorization: str = Header(None), ): """List editable draft agents for the current user with repository listing info.""" @@ -89,6 +93,7 @@ async def list_my_editable_agents_api( page_size=page_size, search=search, new_agent_padding=new_agent_padding, + agent_id=agent_id, ) return JSONResponse(status_code=HTTPStatus.OK, content=result) except UnauthorizedError as e: @@ -140,6 +145,16 @@ async def update_agent_repository_status_api( "rejected (审核驳回) / shared (已共享)" ), ), + notify_content: Optional[str] = Body( + None, + embed=True, + description="Optional reviewer message for approve/reject notification", + ), + content: Optional[str] = Body( + None, + embed=True, + description="Review opinion or resubmit note", + ), authorization: str = Header(None), ): """Update marketplace repository listing status (share, unshare, approve, reject).""" @@ -150,6 +165,8 @@ async def update_agent_repository_status_api( status=status, user_id=user_id, tenant_id=tenant_id, + notify_content=notify_content, + content=content, ) return JSONResponse(status_code=HTTPStatus.OK, content=result) except UnauthorizedError as e: diff --git a/backend/apps/aidp_app.py b/backend/apps/aidp_app.py deleted file mode 100644 index 49f7006f9a..0000000000 --- a/backend/apps/aidp_app.py +++ /dev/null @@ -1,72 +0,0 @@ -""" -AIDP App Layer -FastAPI endpoints for AIDP knowledge base list proxy. -""" -import logging -from http import HTTPStatus -from typing import Annotated - -from fastapi import APIRouter, Query -from fastapi.responses import JSONResponse - -from consts.error_code import ErrorCode -from consts.exceptions import AppException -from services.aidp_service import ( - fetch_aidp_knowledge_bases_impl, - fetch_all_aidp_knowledge_bases_impl, -) - -router = APIRouter(prefix="/aidp") -logger = logging.getLogger("aidp_app") - - -@router.get("/knowledge-bases") -async def fetch_aidp_knowledge_bases_api( - server_url: Annotated[str, Query(description="AIDP API server URL")], - api_key: Annotated[str, Query(description="AIDP API key")], - page: Annotated[int, Query(ge=1, description="Page number starting from 1")] = 1, - page_size: Annotated[int, Query(ge=1, le=100, description="Page size from 1 to 100")] = 10, -) -> JSONResponse: - """Fetch a single page of knowledge bases from the external AIDP API.""" - try: - result = fetch_aidp_knowledge_bases_impl( - server_url=server_url, - api_key=api_key, - page=page, - page_size=page_size, - ) - return JSONResponse(status_code=HTTPStatus.OK, content=result) - except AppException: - raise - except Exception as e: - logger.exception("Failed to fetch AIDP knowledge bases: %s", e) - raise AppException( - ErrorCode.AIDP_SERVICE_ERROR, - f"Failed to fetch AIDP knowledge bases: {str(e)}", - ) - - -@router.get("/knowledge-bases-all") -async def fetch_all_aidp_knowledge_bases_api( - server_url: Annotated[str, Query(description="AIDP API server URL")], - api_key: Annotated[str, Query(description="AIDP API key")], -) -> JSONResponse: - """Fetch ALL knowledge bases from AIDP (accumulates every page internally). - - Use this when you need the total count and want to handle pagination - entirely on the client side. - """ - try: - result = fetch_all_aidp_knowledge_bases_impl( - server_url=server_url, - api_key=api_key, - ) - return JSONResponse(status_code=HTTPStatus.OK, content=result) - except AppException: - raise - except Exception as e: - logger.exception("Failed to fetch all AIDP knowledge bases: %s", e) - raise AppException( - ErrorCode.AIDP_SERVICE_ERROR, - f"Failed to fetch all AIDP knowledge bases: {str(e)}", - ) diff --git a/backend/apps/app_factory.py b/backend/apps/app_factory.py index 02816cec1d..a6ffc3cb70 100644 --- a/backend/apps/app_factory.py +++ b/backend/apps/app_factory.py @@ -7,7 +7,7 @@ from fastapi.middleware.cors import CORSMiddleware from fastapi.responses import JSONResponse -from consts.exceptions import AppException +from consts.exceptions import AppException, QuotaExceededError logger = logging.getLogger(__name__) @@ -95,6 +95,86 @@ async def app_exception_handler(request, exc): }, ) + @app.exception_handler(QuotaExceededError) + async def quota_exceeded_exception_handler(request, exc): + logger.warning("QuotaExceededError: %s", exc) + return JSONResponse( + status_code=413, + content={ + "error": "TenantStorageFull", + "message": str(exc), + "usage_bytes": exc.usage_bytes, + "hard_limit_bytes": exc.hard_limit_bytes, + "exceeded_by_bytes": exc.exceeded_by_bytes, + }, + ) + + # ---- AIDP permission subsystem exceptions (v7.1) ---- + # These are domain exceptions (inherit from plain Exception) that map to + # HTTP status codes per v7.1 design. Without explicit handlers they would + # fall through to the generic handler below and surface as 500, hiding + # the real error from the client. + try: + from ext_components.aidp.consts.aidp_exceptions import ( + AidpKbNotFoundError, + AidpKbPermissionDeniedError, + AidpKbConflictError, + AidpKbSyncError, + AidpGroupValidationError, + ) + + @app.exception_handler(AidpKbNotFoundError) + async def aidp_kb_not_found_handler(request, exc): + logger.warning("AidpKbNotFoundError: %s", exc) + return JSONResponse( + status_code=404, + content={"message": str(exc), "code": "AIDP_KB_NOT_FOUND", "kb_id": exc.kb_id}, + ) + + @app.exception_handler(AidpKbPermissionDeniedError) + async def aidp_permission_denied_handler(request, exc): + logger.warning("AidpKbPermissionDeniedError: user=%s kb=%s required=%s", exc.user_id, exc.kb_id, exc.required) + return JSONResponse( + status_code=403, + content={ + "message": str(exc), + "code": "AIDP_PERMISSION_DENIED", + "kb_id": exc.kb_id, + "required": exc.required, + }, + ) + + @app.exception_handler(AidpKbConflictError) + async def aidp_conflict_handler(request, exc): + logger.warning("AidpKbConflictError: %s", exc) + return JSONResponse( + status_code=409, + content={"message": str(exc), "code": "AIDP_KB_CONFLICT", "kb_id": exc.kb_id}, + ) + + @app.exception_handler(AidpKbSyncError) + async def aidp_sync_error_handler(request, exc): + logger.error("AidpKbSyncError: %s", exc) + return JSONResponse( + status_code=502, + content={"message": str(exc), "code": "AIDP_SYNC_ERROR", "operation": exc.operation}, + ) + + @app.exception_handler(AidpGroupValidationError) + async def aidp_group_validation_handler(request, exc): + logger.warning("AidpGroupValidationError: %s", exc) + return JSONResponse( + status_code=400, + content={ + "message": str(exc), + "code": "AIDP_GROUP_VALIDATION", + "invalid_ids": exc.invalid_ids, + }, + ) + except ImportError: + # AIDP subsystem not installed in this deployment; safe to skip + logger.debug("AIDP exception classes not available, skipping AIDP exception handlers") + @app.exception_handler(Exception) async def generic_exception_handler(request, exc): # Don't catch AppException - it has its own handler diff --git a/backend/apps/config_app.py b/backend/apps/config_app.py index b8998f19fa..9db130e175 100644 --- a/backend/apps/config_app.py +++ b/backend/apps/config_app.py @@ -31,15 +31,23 @@ from apps.group_app import router as group_router from apps.user_app import router as user_router from apps.invitation_app import router as invitation_router +from apps.notification_app import router as notification_router from apps.a2a_client_app import router as a2a_client_router from apps.monitoring_app import router as monitoring_router from apps.a2a_server_app import router as a2a_server_router from apps.haotian_app import router as haotian_router from apps.evaluation_set_app import router as evaluation_set_router from apps.agent_evaluation_app import router as agent_evaluation_router -from apps.aidp_app import router as aidp_router from apps.cas_app import router as cas_router -from consts.const import IS_SPEED_MODE +from apps.memory_config_app import router as memory_config_router +from apps.memory_record_app import router as memory_record_router +from apps.quota_app import tenant_quota_router, platform_quota_router +from consts.const import ( + AIDP_API_KEY, + AIDP_SERVER_URL, + ENABLE_AIDP_KNOWLEDGE, + IS_SPEED_MODE, +) from services.prompt_template_service import sync_system_default_prompt_template # Create logger instance @@ -51,7 +59,12 @@ @app.on_event("startup") async def sync_default_prompt_template_on_startup(): - """Sync the YAML-backed system default prompt template into the database on startup.""" + """Sync defaults and validate enabled external service configuration.""" + if ENABLE_AIDP_KNOWLEDGE and (not AIDP_SERVER_URL or not AIDP_API_KEY): + raise RuntimeError( + "AIDP_SERVER_URL and AIDP_API_KEY are required when ENABLE_AIDP_KNOWLEDGE=true" + ) + try: sync_system_default_prompt_template() logger.info("System default prompt template synced successfully.") @@ -96,9 +109,17 @@ async def sync_default_prompt_template_on_startup(): app.include_router(group_router) app.include_router(user_router) app.include_router(invitation_router) +app.include_router(notification_router) app.include_router(a2a_client_router) app.include_router(a2a_server_router) app.include_router(haotian_router) app.include_router(evaluation_set_router) app.include_router(agent_evaluation_router) -app.include_router(aidp_router) +if ENABLE_AIDP_KNOWLEDGE: + from ext_components.aidp.apps.aidp_mgmt_app import aidp_mgmt_router + app.include_router(aidp_mgmt_router) +# New memory architecture routers (upstream #3497) +app.include_router(memory_config_router) +app.include_router(memory_record_router) +app.include_router(tenant_quota_router) +app.include_router(platform_quota_router) diff --git a/backend/apps/conversation_share_app.py b/backend/apps/conversation_share_app.py index 64806c8e99..451eb5f80b 100644 --- a/backend/apps/conversation_share_app.py +++ b/backend/apps/conversation_share_app.py @@ -28,6 +28,7 @@ class CreateConversationShareRequest(BaseModel): mode: str = "selected" selected_user_message_ids: Optional[List[int]] = None expire_time: Optional[datetime] = None + render_version: str = "legacy" def _parse_range_header(range_header: Optional[str], total_size: int) -> Optional[Tuple[int, int]]: @@ -74,6 +75,7 @@ async def create_conversation_share_endpoint( mode=request.mode, selected_user_message_ids=request.selected_user_message_ids, expire_time=request.expire_time, + render_version=request.render_version, ) result["url"] = f"/share/{result['share_id']}" return {"code": 0, "message": "success", "data": result} diff --git a/backend/apps/file_management_app.py b/backend/apps/file_management_app.py index 427bde6f3e..45711252bb 100644 --- a/backend/apps/file_management_app.py +++ b/backend/apps/file_management_app.py @@ -10,8 +10,14 @@ from fastapi.responses import JSONResponse, RedirectResponse, StreamingResponse from starlette.background import BackgroundTask -from consts.exceptions import FileTooLargeException, NotFoundException, UnsupportedFileTypeException +from consts.exceptions import ( + FileTooLargeException, + NotFoundException, + QuotaExceededError, + UnsupportedFileTypeException, +) from consts.model import ProcessParams +from apps.permission_utils import require_knowledge_base_edit_permission from services.file_management_service import upload_to_minio, upload_files_impl, \ get_file_url_impl, get_file_stream_impl, delete_file_impl, list_files_impl, \ resolve_preview_file, get_preview_stream, check_file_access, check_file_access_batch, \ @@ -102,25 +108,39 @@ async def upload_files( detail="No files in the request") user_id, tenant_id = get_current_user_id(authorization) - errors, uploaded_file_paths, uploaded_filenames = await upload_files_impl( - destination, file, folder, index_name, user_id, uploader_tenant_id=tenant_id + if index_name: + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) + upload_result = await upload_files_impl( + destination, + file, + folder, + index_name, + user_id, + uploader_tenant_id=tenant_id, ) + errors, uploaded_file_paths, uploaded_filenames = upload_result + quota_status = getattr(upload_result, "quota_status", None) if uploaded_file_paths: + response_content = { + "message": f"Files uploaded successfully to {destination}, ready for processing.", + "uploaded_filenames": uploaded_filenames, + "uploaded_file_paths": uploaded_file_paths, + "errors": errors, + } + if quota_status: + response_content["quota_status"] = quota_status.get("quota_status") return JSONResponse( status_code=HTTPStatus.OK, - content={ - "message": f"Files uploaded successfully to {destination}, ready for processing.", - "uploaded_filenames": uploaded_filenames, - "uploaded_file_paths": uploaded_file_paths, - "errors": errors - } + content=response_content, ) else: raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail="No valid files uploaded") except HTTPException: raise + except QuotaExceededError: + raise except Exception as e: logger.error(f"File upload error: {str(e)}") raise HTTPException( @@ -144,6 +164,9 @@ async def process_files( index_name: index name in elasticsearch destination: 'local' or 'minio' """ + user_id, tenant_id = get_current_user_id(authorization) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) + process_params = ProcessParams( chunking_strategy=chunking_strategy, source_type=destination, diff --git a/backend/apps/knowledge_summary_app.py b/backend/apps/knowledge_summary_app.py index ab45170fbb..6f664f5ea0 100644 --- a/backend/apps/knowledge_summary_app.py +++ b/backend/apps/knowledge_summary_app.py @@ -6,6 +6,7 @@ from nexent.vector_database.base import VectorDatabaseCore from consts.model import ChangeSummaryRequest +from apps.permission_utils import require_knowledge_base_edit_permission from services.vectordatabase_service import ElasticSearchService, get_vector_db_core from utils.auth_utils import get_current_user_id, get_current_user_info from utils.config_utils import tenant_config_manager @@ -28,8 +29,9 @@ async def auto_summary( ): """Summary Elasticsearch index_name by model""" try: - _, tenant_id, language = get_current_user_info( + user_id, tenant_id, language = get_current_user_info( authorization, http_request) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) service = ElasticSearchService() # Get model_id from tenant config if not provided @@ -53,6 +55,8 @@ async def auto_summary( language=language, model_id=model_id ) + except HTTPException: + raise except Exception as e: logger.error( f"Knowledge base summary generation failed: {e}", exc_info=True) @@ -73,9 +77,12 @@ def change_summary( ): """Summary Elasticsearch index_name by user""" try: - user_id = get_current_user_id(authorization)[0] + user_id, tenant_id = get_current_user_id(authorization) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) summary_result = change_summary_request.summary_result return ElasticSearchService().change_summary(index_name=index_name, summary_result=summary_result, user_id=user_id) + except HTTPException: + raise except Exception as e: raise HTTPException( status_code=500, detail=f"Knowledge base summary update failed: {str(e)}") diff --git a/backend/apps/mcp_management_app.py b/backend/apps/mcp_management_app.py index f94e03e1f9..5bf9ce894e 100644 --- a/backend/apps/mcp_management_app.py +++ b/backend/apps/mcp_management_app.py @@ -90,6 +90,7 @@ async def list_community_mcp_services_api( user_id, tenant_id, _ = get_current_user_info(authorization, http_request) data = await list_community_mcp_services( tenant_id=tenant_id, + user_id=user_id, search=query.search, tag=query.tag, transport_type=query.transport_type, @@ -214,6 +215,10 @@ async def publish_community_mcp_service_api( tags=payload.tags, mcp_server=payload.mcp_server, config_json=payload.config_json, + group_ids=payload.group_ids, + ingroup_permission=payload.ingroup_permission, + shared_fields=payload.shared_fields, + content=payload.content, ) return JSONResponse( status_code=HTTPStatus.OK, @@ -257,6 +262,10 @@ async def update_community_mcp_service_legacy_api( mcp_server=payload.mcp_server, config_json=payload.config_json, transport_type=payload.transport_type, + group_ids=payload.group_ids, + ingroup_permission=payload.ingroup_permission, + shared_fields=payload.shared_fields, + content=payload.content, ) return JSONResponse(status_code=HTTPStatus.OK, content={"status": "success"}) except McpNotFoundError as exc: @@ -319,6 +328,7 @@ async def approve_community_mcp_service_api( tenant_id=tenant_id, user_id=user_id, market_id=payload.review_id, + content=payload.content, ) return JSONResponse(status_code=HTTPStatus.OK, content={"status": "success"}) except McpNotFoundError as exc: @@ -348,6 +358,7 @@ async def reject_community_mcp_service_api( tenant_id=tenant_id, user_id=user_id, market_id=payload.review_id, + content=payload.content, ) return JSONResponse(status_code=HTTPStatus.OK, content={"status": "success"}) except McpNotFoundError as exc: @@ -386,6 +397,10 @@ async def create_community_mcp_service_api( tags=payload.tags, mcp_server=payload.mcp_server, config_json=payload.config_json, + group_ids=payload.group_ids, + ingroup_permission=payload.ingroup_permission, + shared_fields=payload.shared_fields, + content=payload.content, ) return JSONResponse( status_code=HTTPStatus.OK, @@ -430,6 +445,10 @@ async def update_community_mcp_service_api( mcp_server=payload.mcp_server, config_json=payload.config_json, transport_type=payload.transport_type, + group_ids=payload.group_ids, + ingroup_permission=payload.ingroup_permission, + shared_fields=payload.shared_fields, + content=payload.content, ) return JSONResponse(status_code=HTTPStatus.OK, content={"status": "success"}) except McpNotFoundError as exc: @@ -494,6 +513,7 @@ async def change_community_mcp_status_api( user_id=user_id, market_id=market_id, new_status=payload.status, + content=payload.content, ) return JSONResponse(status_code=HTTPStatus.OK, content={"status": "success"}) except McpNotFoundError as exc: diff --git a/backend/apps/memory_config_app.py b/backend/apps/memory_config_app.py index c1f56dd70e..6247a1769d 100644 --- a/backend/apps/memory_config_app.py +++ b/backend/apps/memory_config_app.py @@ -1,39 +1,28 @@ -"""Memory configuration and CRUD API endpoints for the app layer. - -This module exposes HTTP endpoints under the `/memory` prefix. It follows the -app-layer responsibilities: -- Parse and validate HTTP inputs -- Delegate business logic to the service layer -- Convert unexpected exceptions to error JSON responses - -Routes: -- GET `/memory/config/load`: Load memory-related configuration for current user -- POST `/memory/config/set`: Set a single configuration entry -- POST `/memory/config/disable_agent`: Add a disabled agent id -- DELETE `/memory/config/disable_agent/{agent_id}`: Remove a disabled agent id -- POST `/memory/config/disable_useragent`: Add a disabled user-agent id -- DELETE `/memory/config/disable_useragent/{agent_id}`: Remove a disabled user-agent id -- POST `/memory/add`: Add memory items (optionally with LLM inference) -- POST `/memory/search`: Semantic search memory items -- GET `/memory/list`: List memory items -- DELETE `/memory/delete/{memory_id}`: Delete a single memory item -- DELETE `/memory/clear`: Clear memory items by scope +"""Memory configuration API endpoints for the app layer. + +This module exposes HTTP endpoints under the `/memory` prefix for managing +user-level memory preferences that the new Memory system reads at agent +build time. CRUD endpoints that delegated to the legacy mem0-based +``nexent.memory.memory_service`` (``/memory/add``, ``/memory/search``, +``/memory/list``, ``/memory/delete/{memory_id}``, ``/memory/clear``) have +been removed; their callers now use the in-process ``MemoryService`` +directly (or the agent-side ``StoreMemoryTool`` / ``SearchMemoryTool``). + +Routes retained: +- GET `/memory/config/load`: Load memory-related configuration for current user. +- POST `/memory/config/set`: Set a single configuration entry. +- POST `/memory/config/disable_agent`: Add a disabled agent id. +- DELETE `/memory/config/disable_agent/{agent_id}`: Remove a disabled agent id. +- POST `/memory/config/disable_useragent`: Add a disabled user-agent id. +- DELETE `/memory/config/disable_useragent/{agent_id}`: Remove a disabled user-agent id. """ -import asyncio import logging -from typing import Any, Dict, List, Optional +from typing import Any, Optional from http import HTTPStatus -from fastapi import APIRouter, Body, Header, Path, Query, HTTPException +from fastapi import APIRouter, Body, Header, Path, HTTPException from fastapi.responses import JSONResponse -from nexent.memory.memory_service import ( - add_memory as svc_add_memory, - clear_memory as svc_clear_memory, - delete_memory as svc_delete_memory, - list_memory as svc_list_memory, - search_memory as svc_search_memory, -) from consts.const import ( MEMORY_AGENT_SHARE_KEY, MEMORY_SWITCH_KEY, @@ -50,24 +39,20 @@ set_agent_share, set_memory_switch, ) +from services.memory_record_service import ( + get_tenant_memory_index_name, + is_tenant_embedding_configured, +) from utils.auth_utils import get_current_user_id -from utils.memory_utils import build_memory_config logger = logging.getLogger("memory_config_app") -logger.setLevel(logging.DEBUG) +logger.setLevel(logging.INFO) router = APIRouter(prefix="/memory") -# --------------------------------------------------------------------------- -# Configuration Endpoints -# --------------------------------------------------------------------------- @router.get("/config/load") def load_configs(authorization: Optional[str] = Header(None)): - """Load all memory-related configuration for the current user. - - Args: - authorization: Optional authorization header used to identify the user. - """ + """Load all memory-related configuration for the current user.""" try: user_id, _ = get_current_user_id(authorization) configs = get_user_configs(user_id) @@ -80,6 +65,28 @@ def load_configs(authorization: Optional[str] = Header(None)): detail="Failed to load configuration") +@router.get("/config/embedding-status") +def get_embedding_status(authorization: Optional[str] = Header(None)): + """Return tenant embedding availability and the active memory index.""" + try: + _, tenant_id = get_current_user_id(authorization) + return JSONResponse( + status_code=HTTPStatus.OK, + content={ + "configured": is_tenant_embedding_configured(tenant_id), + "current_es_index_name": get_tenant_memory_index_name(tenant_id), + }, + ) + except UnauthorizedError as e: + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(e)) + except Exception as e: + logger.error("get_embedding_status failed: %s", e) + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="Failed to load embedding configuration status", + ) + + @router.post("/config/set") def set_single_config( key: str = Body(..., embed=True, description="Configuration key"), @@ -89,13 +96,8 @@ def set_single_config( """Set a single-value configuration item for the current user. Supported keys: - - `MEMORY_SWITCH_KEY`: Toggle memory system on/off (boolean-like values accepted) - - `MEMORY_AGENT_SHARE_KEY`: Set agent share mode (`always`/`ask`/`never`) - - Args: - key: Configuration key to update. - value: New value for the configuration key. - authorization: Optional authorization header used to identify the user. + - `MEMORY_SWITCH_KEY`: Toggle memory system on/off (boolean-like values accepted). + - `MEMORY_AGENT_SHARE_KEY`: Set agent share mode (`always`/`ask`/`never`). """ user_id, _ = get_current_user_id(authorization) @@ -125,12 +127,7 @@ def add_disable_agent( agent_id: str = Body(..., embed=True), authorization: Optional[str] = Header(None), ): - """Add an agent id to the user's disabled agent list. - - Args: - agent_id: Identifier of the agent to disable. - authorization: Optional authorization header used to identify the user. - """ + """Add an agent id to the user's disabled agent list.""" user_id, _ = get_current_user_id(authorization) ok = add_disabled_agent_id(user_id, agent_id) if ok: @@ -144,12 +141,7 @@ def remove_disable_agent( agent_id: str = Path(...), authorization: Optional[str] = Header(None), ): - """Remove an agent id from the user's disabled agent list. - - Args: - agent_id: Identifier of the agent to remove from the disabled list. - authorization: Optional authorization header used to identify the user. - """ + """Remove an agent id from the user's disabled agent list.""" user_id, _ = get_current_user_id(authorization) ok = remove_disabled_agent_id(user_id, agent_id) if ok: @@ -163,12 +155,7 @@ def add_disable_useragent( agent_id: str = Body(..., embed=True), authorization: Optional[str] = Header(None), ): - """Add a user-agent id to the user's disabled user-agent list. - - Args: - agent_id: Identifier of the user-agent to disable. - authorization: Optional authorization header used to identify the user. - """ + """Add a user-agent id to the user's disabled user-agent list.""" user_id, _ = get_current_user_id(authorization) ok = add_disabled_useragent_id(user_id, agent_id) if ok: @@ -182,170 +169,10 @@ def remove_disable_useragent( agent_id: str = Path(...), authorization: Optional[str] = Header(None), ): - """Remove a user-agent id from the user's disabled user-agent list. - - Args: - agent_id: Identifier of the user-agent to remove from the disabled list. - authorization: Optional authorization header used to identify the user. - """ + """Remove a user-agent id from the user's disabled user-agent list.""" user_id, _ = get_current_user_id(authorization) ok = remove_disabled_useragent_id(user_id, agent_id) if ok: return JSONResponse(status_code=HTTPStatus.OK, content={"success": True}) raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail="Failed to remove disable user-agent id") - - -# --------------------------------------------------------------------------- -# Memory CRUD Endpoints -# --------------------------------------------------------------------------- -@router.post("/add") -def add_memory( - messages: List[Dict[str, Any] - ] = Body(..., description="Chat messages list"), - memory_level: str = Body(..., embed=True, - description="Memory level: tenant/agent/user/user_agent"), - agent_id: Optional[str] = Body(None, embed=True), - infer: bool = Body( - True, embed=True, description="Whether to run LLM inference during add"), - authorization: Optional[str] = Header(None), -): - """Add memory records for the given scope. - - Args: - messages: List of chat messages as dictionaries. - memory_level: Scope for the memory record (tenant/agent/user/user_agent). - agent_id: Optional agent identifier when scope is agent-related. - infer: Whether to run LLM inference during add. - authorization: Optional authorization header used to identify the user. - """ - user_id, tenant_id = get_current_user_id(authorization) - try: - result = asyncio.run(svc_add_memory( - messages=messages, - memory_level=memory_level, - memory_config=build_memory_config(tenant_id), - tenant_id=tenant_id, - user_id=user_id, - agent_id=agent_id, - infer=infer, - )) - return JSONResponse(status_code=HTTPStatus.OK, content=result) - except Exception as e: - logger.error("add_memory error: %s", e, exc_info=True) - raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) - - -@router.post("/search") -def search_memory( - query_text: str = Body(..., embed=True, description="Query text"), - memory_level: str = Body(..., embed=True), - top_k: int = Body(5, embed=True), - agent_id: Optional[str] = Body(None, embed=True), - authorization: Optional[str] = Header(None), -): - """Search memory semantically for the given scope. - - Args: - query_text: Natural language query to search memory. - memory_level: Scope for search (tenant/agent/user/user_agent). - top_k: Maximum number of results to return. - agent_id: Optional agent identifier when scope is agent-related. - authorization: Optional authorization header used to identify the user. - """ - user_id, tenant_id = get_current_user_id(authorization) - try: - results = asyncio.run(svc_search_memory( - query_text=query_text, - memory_level=memory_level, - memory_config=build_memory_config(tenant_id), - tenant_id=tenant_id, - user_id=user_id, - top_k=top_k, - agent_id=agent_id, - )) - return JSONResponse(status_code=HTTPStatus.OK, content=results) - except Exception as e: - logger.error("search_memory error: %s", e, exc_info=True) - raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) - - -@router.get("/list") -def list_memory( - memory_level: str = Query(..., - description="Memory level: tenant/agent/user/user_agent"), - agent_id: Optional[str] = Query( - None, description="Filter by agent id if applicable"), - authorization: Optional[str] = Header(None), -): - """List memory for the given scope. - - Args: - memory_level: Scope for listing (tenant/agent/user/user_agent). - agent_id: Optional agent filter when scope is agent-related. - authorization: Optional authorization header used to identify the user. - """ - user_id, tenant_id = get_current_user_id(authorization) - try: - payload = asyncio.run(svc_list_memory( - memory_level=memory_level, - memory_config=build_memory_config(tenant_id), - tenant_id=tenant_id, - user_id=user_id, - agent_id=agent_id, - )) - return JSONResponse(status_code=HTTPStatus.OK, content=payload) - except Exception as e: - logger.error("list_memory error: %s", e, exc_info=True) - raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) - - -@router.delete("/delete/{memory_id}") -def delete_memory( - memory_id: str = Path(..., description="ID of memory to delete"), - authorization: Optional[str] = Header(None), -): - """Delete a specific memory record by id. - - Args: - memory_id: Identifier of the memory record to delete. - authorization: Optional authorization header used to identify the user. - """ - _user_id, tenant_id = get_current_user_id(authorization) - try: - result = asyncio.run(svc_delete_memory( - memory_id=memory_id, memory_config=build_memory_config(tenant_id))) - return JSONResponse(status_code=HTTPStatus.OK, content=result) - except Exception as e: - logger.error("delete_memory error: %s", e, exc_info=True) - raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) - - -@router.delete("/clear") -def clear_memory( - memory_level: str = Query(..., - description="Memory level: tenant/agent/user/user_agent"), - agent_id: Optional[str] = Query( - None, description="Filter by agent id if applicable"), - authorization: Optional[str] = Header(None), -): - """Clear memory records for the given scope. - - Args: - memory_level: Scope for clearing (tenant/agent/user/user_agent). - agent_id: Optional agent filter when scope is agent-related. - authorization: Optional authorization header used to identify the user. - """ - user_id, tenant_id = get_current_user_id(authorization) - try: - result = asyncio.run(svc_clear_memory( - memory_level=memory_level, - memory_config=build_memory_config(tenant_id), - tenant_id=tenant_id, - user_id=user_id, - agent_id=agent_id, - )) - return JSONResponse(status_code=HTTPStatus.OK, content=result) - except Exception as e: - logger.error("clear_memory error: %s", e, exc_info=True) - raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) diff --git a/backend/apps/memory_record_app.py b/backend/apps/memory_record_app.py new file mode 100644 index 0000000000..8c1e50c1b4 --- /dev/null +++ b/backend/apps/memory_record_app.py @@ -0,0 +1,319 @@ +"""HTTP endpoints for managing internal memory records (Phase 2). + +These endpoints are intentionally restricted to manual management of +tenant/user long-term memory. Agent short-term memory writes are routed +through the in-process ``StoreMemoryTool`` -> ``MemoryService`` pipeline, +not through HTTP, to keep the agent side lightweight. + +Routes: + +- POST ``/memory/records`` Create a memory record +- GET ``/memory/records/{memory_id}`` Read a record +- GET ``/memory/records`` List records (with filters) +- PATCH ``/memory/records/{memory_id}`` Update a record +- DELETE ``/memory/records/{memory_id}`` Soft-delete a record +- POST ``/memory/records/search`` Run a retrieval +- GET ``/memory/context`` Build an agent prompt context block + +All endpoints scope results by ``(tenant_id, user_id, ...)`` derived from +the auth token. Tenant isolation keys are required and cannot be supplied +by the client. +""" + +from __future__ import annotations + +import logging +from http import HTTPStatus +from typing import Any, Dict, List, Optional + +from fastapi import APIRouter, Body, Header, HTTPException, Path, Query +from fastapi.responses import JSONResponse +from pydantic import BaseModel, Field + +from database.user_tenant_db import get_user_tenant_by_user_id +from services.memory_context_service import get_memory_context_service +from services.memory_record_service import ( + MemoryRecordError, + get_memory_record_service, +) +from services.memory_retrieval_service import get_memory_retrieval_service +from utils.auth_utils import get_current_user_id + + +logger = logging.getLogger("memory_record_app") +logger.setLevel(logging.INFO) +router = APIRouter(prefix="/memory") + + +def _require_tenant_admin(user_id: str) -> None: + """Reject tenant-memory writes from non-ADMIN users.""" + user_tenant = get_user_tenant_by_user_id(user_id) or {} + if str(user_tenant.get("user_role") or "").upper() != "ADMIN": + raise HTTPException( + status_code=HTTPStatus.FORBIDDEN, + detail="Tenant memory creation requires the ADMIN role", + ) + + +# --------------------------------------------------------------------------- +# Request / response models +# --------------------------------------------------------------------------- + + +class CreateMemoryRequest(BaseModel): + layer: str = Field(..., description="tenant | user | agent") + content: str = Field(..., min_length=1) + memory_type: Optional[str] = Field( + default=None, + description="long_term | short_term; defaults by layer", + ) + agent_id: Optional[str] = None + conversation_id: Optional[str] = None + concept_tags: List[str] = Field(default_factory=list) + idempotency_key: Optional[str] = None + + +class UpdateMemoryRequest(BaseModel): + content: Optional[str] = None + status: Optional[str] = None + concept_tags: Optional[List[str]] = None + + +class SearchMemoryRequest(BaseModel): + query: str + agent_id: Optional[str] = None + conversation_id: Optional[str] = None + layers: Optional[List[str]] = None + top_k: int = 5 + threshold: float = 0.65 + hybrid: bool = Field( + default=False, + description=( + "When true, agent short-term memory is retrieved via a hybrid " + "(BM25 + kNN) query against Elasticsearch instead of pure " + "kNN. Defaults to false to preserve prior behavior." + ), + ) + weight_accurate: float = Field( + default=0.3, + ge=0.0, + le=1.0, + description=( + "Weight of the fuzzy (BM25) branch when hybrid=true. The " + "complementary 1 - weight is given to the semantic kNN branch." + ), + ) + + +# --------------------------------------------------------------------------- +# Routes +# --------------------------------------------------------------------------- + + +@router.post("/records") +def create_record( + payload: CreateMemoryRequest, + authorization: Optional[str] = Header(None), +): + """Create a memory record. + + Agents should not call this endpoint for short-term memory; they go + through ``StoreMemoryTool``. The endpoint exists for tenant/user + manual management and for Dreaming promotion. + """ + user_id, tenant_id = get_current_user_id(authorization) + if payload.layer.strip().lower() == "tenant": + _require_tenant_admin(user_id) + service = get_memory_record_service() + try: + result = service.create_memory( + tenant_id=tenant_id, + user_id=user_id, + content=payload.content, + layer=payload.layer, + memory_type=payload.memory_type, + agent_id=payload.agent_id, + conversation_id=payload.conversation_id, + concept_tags=payload.concept_tags, + idempotency_key=payload.idempotency_key, + created_by=user_id, + actor="system", + ) + except MemoryRecordError as exc: + raise HTTPException( + status_code=HTTPStatus.NOT_ACCEPTABLE, detail=str(exc) + ) + + return JSONResponse(status_code=HTTPStatus.OK, content=result) + + +@router.get("/records/{memory_id}") +def read_record( + memory_id: int = Path(..., description="Auto-incremented memory primary key."), + authorization: Optional[str] = Header(None), +): + user_id, tenant_id = get_current_user_id(authorization) + service = get_memory_record_service() + record = service.get_memory_for_user( + memory_id=memory_id, tenant_id=tenant_id, user_id=user_id + ) + if record is None: + raise HTTPException( + status_code=HTTPStatus.NOT_FOUND, detail="Memory record not found" + ) + return JSONResponse(status_code=HTTPStatus.OK, content=record) + + +@router.get("/records") +def list_records( + authorization: Optional[str] = Header(None), + layer: Optional[str] = Query(default=None), + memory_type: Optional[str] = Query(default=None), + status: Optional[str] = Query(default="active"), + agent_id: Optional[str] = Query(default=None), + conversation_id: Optional[str] = Query(default=None), + limit: int = Query(default=100, ge=1, le=1000), + offset: int = Query(default=0, ge=0), +): + user_id, tenant_id = get_current_user_id(authorization) + normalized_layer = layer.strip().lower() if layer else None + service = get_memory_record_service() + rows = service.list_memories( + tenant_id, + user_id=None if normalized_layer == "tenant" else user_id, + agent_id=agent_id, + conversation_id=conversation_id, + layer=normalized_layer, + memory_type=memory_type, + status=status, + limit=limit, + offset=offset, + ) + return JSONResponse( + status_code=HTTPStatus.OK, + content={"items": rows, "count": len(rows)}, + ) + + +@router.patch("/records/{memory_id}") +def update_record( + memory_id: int = Path(..., description="Auto-incremented memory primary key."), + payload: UpdateMemoryRequest = Body(...), + authorization: Optional[str] = Header(None), +): + user_id, tenant_id = get_current_user_id(authorization) + update_data: Dict[str, Any] = {"updated_by": user_id} + if payload.content is not None: + update_data["content"] = payload.content + if payload.status is not None: + update_data["status"] = payload.status + if payload.concept_tags is not None: + update_data["concept_tags"] = payload.concept_tags + + service = get_memory_record_service() + ok = service.update_memory(memory_id, tenant_id, update_data) + if not ok: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="Failed to update memory record", + ) + return JSONResponse( + status_code=HTTPStatus.OK, + content={"success": True, "memory_id": memory_id}, + ) + + +@router.delete("/records/{memory_id}") +def delete_record( + memory_id: int = Path(..., description="Auto-incremented memory primary key."), + authorization: Optional[str] = Header(None), +): + user_id, tenant_id = get_current_user_id(authorization) + service = get_memory_record_service() + ok = service.soft_delete_memory(memory_id, tenant_id, updated_by=user_id) + if not ok: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="Failed to delete memory record", + ) + return JSONResponse( + status_code=HTTPStatus.OK, content={"success": True} + ) + + +@router.post("/records/search") +async def search_records( + payload: SearchMemoryRequest, + authorization: Optional[str] = Header(None), +): + """Run a memory retrieval and return ranked hits. + + Layer resolution, embedding model lookup, and query vector computation are + all handled inside the service layer. + """ + user_id, tenant_id = get_current_user_id(authorization) + retrieval = get_memory_retrieval_service() + results = await retrieval.search_memories( + tenant_id=tenant_id, + user_id=user_id, + query=payload.query, + agent_id=payload.agent_id, + conversation_id=payload.conversation_id, + layers=payload.layers, + top_k=payload.top_k, + threshold=payload.threshold, + hybrid=payload.hybrid, + weight_accurate=payload.weight_accurate, + ) + return JSONResponse( + status_code=HTTPStatus.OK, + content={ + "items": [result.model_dump() for result in results], + "count": len(results), + }, + ) + + +@router.get("/context") +async def build_context( + authorization: Optional[str] = Header(None), + query: Optional[str] = Query(default=None), + agent_id: Optional[str] = Query(default=None), + conversation_id: Optional[str] = Query(default=None), + layers: Optional[str] = Query( + default=None, description="Comma-separated layer names" + ), + top_k: int = Query(default=5, ge=1, le=100), + threshold: float = Query(default=0.65, ge=0.0, le=1.0), +): + """Return a memory context block ready for prompt injection.""" + user_id, tenant_id = get_current_user_id(authorization) + + parsed_layers: Optional[List[str]] = None + if layers: + parsed_layers = [v.strip().lower() for v in layers.split(",")] + + service = get_memory_context_service() + context = await service.build_context( + tenant_id=tenant_id, + user_id=user_id, + agent_id=agent_id, + conversation_id=conversation_id, + query=query, + top_k=top_k, + threshold=threshold, + layers=parsed_layers, + ) + + return JSONResponse( + status_code=HTTPStatus.OK, + content={ + "tenant_long_term": [r.model_dump() for r in context.tenant_long_term], + "user_long_term": [r.model_dump() for r in context.user_long_term], + "agent_short_term": [ + r.model_dump() for r in context.agent_short_term + ], + "external": [r.model_dump() for r in context.external], + "prompt_text": context.to_prompt_text(), + }, + ) diff --git a/backend/apps/model_managment_app.py b/backend/apps/model_managment_app.py index edc64ef148..ce448af6dd 100644 --- a/backend/apps/model_managment_app.py +++ b/backend/apps/model_managment_app.py @@ -517,7 +517,8 @@ async def manage_check_model_health( result = await check_model_connectivity( request.display_name, - request.tenant_id + request.tenant_id, + request.model_type ) return JSONResponse(status_code=HTTPStatus.OK, content={ "message": "Successfully checked model connectivity", diff --git a/backend/apps/northbound_app.py b/backend/apps/northbound_app.py index 997909c94a..f048392be7 100644 --- a/backend/apps/northbound_app.py +++ b/backend/apps/northbound_app.py @@ -18,6 +18,7 @@ start_streaming_chat, stop_chat, get_agent_info_list, + get_agent_info_by_name_for_northbound, update_conversation_title, upload_files_for_northbound, ) @@ -348,6 +349,32 @@ async def list_agents(request: Request): status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail="Internal Server Error") +@router.get("/agents/{agent_name}") +async def get_agent_by_name( + request: Request, + agent_name: str, +): + try: + ctx: NorthboundContext = await _get_northbound_context(request) + return await get_agent_info_by_name_for_northbound(ctx=ctx, agent_name=agent_name) + except ValueError as e: + logging.error(f"Invalid agent detail request: {str(e)}", exc_info=e) + raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) + except LookupError as e: + logging.info(f"Published agent not found: {agent_name}") + raise HTTPException(status_code=HTTPStatus.NOT_FOUND, detail=str(e)) + except LimitExceededError as e: + logging.error(f"Too Many Requests: rate limit exceeded: {str(e)}", exc_info=e) + raise HTTPException(status_code=HTTPStatus.TOO_MANY_REQUESTS, + detail="Too Many Requests: rate limit exceeded") + except HTTPException as e: + raise e + except Exception as e: + logging.error(f"Failed to get agent by name: {str(e)}", exc_info=e) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail="Internal Server Error") + + @router.get("/conversations") async def list_convs(request: Request): try: diff --git a/backend/apps/northbound_knowledge_app.py b/backend/apps/northbound_knowledge_app.py index a5b190ee89..a8265d2fdf 100644 --- a/backend/apps/northbound_knowledge_app.py +++ b/backend/apps/northbound_knowledge_app.py @@ -98,7 +98,7 @@ async def create_new_index( Optional[Dict[str, Any]], Body( description=( - "Request body with optional fields (ingroup_permission, group_ids, embedding_model_name, preserve_source_file)" + "Request body containing embedding_model_id and optional knowledge-base settings" ), ), ] = None, @@ -115,14 +115,17 @@ async def create_new_index( ingroup_permission = None group_ids = None - embedding_model_name = None + embedding_model_id = None preserve_source_file = None if body: ingroup_permission = body.get("ingroup_permission") group_ids = body.get("group_ids") - embedding_model_name = body.get("embedding_model_name") + embedding_model_id = body.get("embedding_model_id") preserve_source_file = body.get("preserve_source_file") + if isinstance(embedding_model_id, bool) or not isinstance(embedding_model_id, int): + raise ValueError("embedding_model_id must be an integer") + return ElasticSearchService.create_knowledge_base( knowledge_name=index_name, embedding_dim=embedding_dim, @@ -131,7 +134,7 @@ async def create_new_index( tenant_id=ctx.tenant_id, ingroup_permission=ingroup_permission, group_ids=group_ids, - embedding_model_name=embedding_model_name, + embedding_model_id=embedding_model_id, preserve_source_file=preserve_source_file, ) except LimitExceededError as e: @@ -144,6 +147,9 @@ async def create_new_index( status_code=HTTPStatus.UNAUTHORIZED, detail=str(e)) except HTTPException: raise + except (TypeError, ValueError) as e: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) except Exception: logger.exception("Error creating index") raise HTTPException( diff --git a/backend/apps/notification_app.py b/backend/apps/notification_app.py new file mode 100644 index 0000000000..7ed75c9085 --- /dev/null +++ b/backend/apps/notification_app.py @@ -0,0 +1,82 @@ +"""Notification API endpoints.""" +import logging +from http import HTTPStatus +from typing import Annotated, Optional + +from fastapi import APIRouter, Body, Header, HTTPException, Query +from starlette.responses import JSONResponse + +from consts.exceptions import NotFoundException, UnauthorizedError +from services.notification_service import list_notifications, mark_notifications_read +from utils.auth_utils import get_current_user_id + +logger = logging.getLogger(__name__) +router = APIRouter(prefix="/notifications", tags=["notifications"]) + + +@router.get("") +async def list_notifications_endpoint( + only_unread: bool = Query(False, description="Return only unread notifications"), + page: Annotated[int, Query(ge=1, description="Page number starting from 1")] = 1, + page_size: Annotated[ + int, Query(ge=1, le=100, description="Page size from 1 to 100") + ] = 10, + authorization: Optional[str] = Header(None), +) -> JSONResponse: + """List the current user's notifications.""" + try: + user_id, _ = get_current_user_id(authorization) + result = list_notifications( + user_id, + only_unread=only_unread, + page=page, + page_size=page_size, + ) + return JSONResponse( + status_code=HTTPStatus.OK, + content={"message": "OK", "data": result}, + ) + except UnauthorizedError as exc: + logger.warning("Unauthorized notification list access: %s", exc) + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(exc)) from exc + except Exception as exc: + logger.error("Unexpected error listing notifications: %s", exc) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail="Failed to list notifications", + ) from exc + + +@router.post("/read") +async def mark_notifications_read_endpoint( + mark_all: bool = Body(False, embed=True, description="Mark all unread as read"), + receiver_id: Optional[int] = Body( + None, embed=True, description="Receiver row ID when mark_all is false" + ), + authorization: Optional[str] = Header(None), +) -> JSONResponse: + """Mark one or all of the current user's notifications as read.""" + try: + user_id, _ = get_current_user_id(authorization) + result = mark_notifications_read( + user_id, + mark_all=mark_all, + receiver_id=receiver_id, + ) + return JSONResponse( + status_code=HTTPStatus.OK, + content={"message": "OK", "data": result}, + ) + except UnauthorizedError as exc: + logger.warning("Unauthorized mark-read access: %s", exc) + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(exc)) from exc + except NotFoundException as exc: + raise HTTPException(status_code=HTTPStatus.NOT_FOUND, detail=str(exc)) from exc + except ValueError as exc: + raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail=str(exc)) from exc + except Exception as exc: + logger.error("Unexpected error marking notifications read: %s", exc) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail="Failed to mark notifications as read", + ) from exc diff --git a/backend/apps/oauth_app.py b/backend/apps/oauth_app.py index b36e9b3dbb..6712e097d1 100644 --- a/backend/apps/oauth_app.py +++ b/backend/apps/oauth_app.py @@ -7,6 +7,7 @@ from pydantic import ValidationError as PydanticValidationError +from consts.const import JWT_EXPIRY_SECONDS from consts.model import OAuthCompleteRequest from consts.exceptions import OAuthLinkError, OAuthProviderError, UnauthorizedError from consts.oauth_providers import get_all_provider_definitions @@ -202,8 +203,10 @@ async def callback( username=username, ) - expiry_seconds = 3600 - jwt_token = generate_session_jwt(supabase_user_id, expires_in=expiry_seconds) + jwt_token = generate_session_jwt( + supabase_user_id, expires_in=JWT_EXPIRY_SECONDS + ) + expiry_seconds = JWT_EXPIRY_SECONDS expires_at = calculate_expires_at(jwt_token) return JSONResponse( diff --git a/backend/apps/permission_utils.py b/backend/apps/permission_utils.py new file mode 100644 index 0000000000..2bd770eecc --- /dev/null +++ b/backend/apps/permission_utils.py @@ -0,0 +1,32 @@ +from http import HTTPStatus + +from fastapi import HTTPException + +from services.vectordatabase_service import ElasticSearchService + + +def require_knowledge_base_edit_permission(index_name: str, user_id: str, tenant_id: str) -> None: + try: + ElasticSearchService.require_knowledge_base_edit_permission( + index_name=index_name, + user_id=user_id, + tenant_id=tenant_id, + ) + except ValueError as exc: + raise HTTPException(status_code=HTTPStatus.NOT_FOUND, detail=str(exc)) + except PermissionError as exc: + raise HTTPException(status_code=HTTPStatus.FORBIDDEN, detail=str(exc)) + + +def require_knowledge_base_read_permission(index_name: str, user_id: str, tenant_id: str) -> None: + """FastAPI adapter: raise 404 if KB is missing, 403 if the user cannot read it.""" + try: + ElasticSearchService.require_knowledge_base_read_permission( + index_name=index_name, + user_id=user_id, + tenant_id=tenant_id, + ) + except ValueError as exc: + raise HTTPException(status_code=HTTPStatus.NOT_FOUND, detail=str(exc)) + except PermissionError as exc: + raise HTTPException(status_code=HTTPStatus.FORBIDDEN, detail=str(exc)) diff --git a/backend/apps/quota_app.py b/backend/apps/quota_app.py new file mode 100644 index 0000000000..86969e7da7 --- /dev/null +++ b/backend/apps/quota_app.py @@ -0,0 +1,478 @@ +""" +Quota management API endpoints. + +Provides tenant-level and platform-level quota configuration and usage tracking. +""" + +import logging +from http import HTTPStatus +from typing import Any, Dict, Optional + +from fastapi import APIRouter, Body, Header, HTTPException, Path, Query +from fastapi.responses import JSONResponse + +from consts.const import ASSET_OWNER_TENANT_ID +from consts.exceptions import PlatformQuotaConflictError +from database.user_tenant_db import get_user_tenant_by_user_id +from services.quota_service import QuotaService +from utils.auth_utils import get_current_user_id + +logger = logging.getLogger(__name__) + +# Tenant-level quota router +tenant_quota_router = APIRouter(prefix="/tenants") + +# Platform-level quota router +platform_quota_router = APIRouter(prefix="/platform/quota") + + +def _platform_quota_conflict_response(exc: PlatformQuotaConflictError) -> JSONResponse: + """Serialize allocation conflicts consistently for quota clients.""" + return JSONResponse( + status_code=HTTPStatus.CONFLICT, + content={"error": exc.error, "message": str(exc), **exc.details}, + ) + + +# ── Role Helpers ──────────────────────────────────────────────────────── + +def _get_user_role(authorization: Optional[str]) -> str: + """Extract user role from the authorization token.""" + if not authorization: + return "USER" + try: + user_id, _ = get_current_user_id(authorization) + user_info = get_user_tenant_by_user_id(user_id) + return (user_info.get("user_role") or "USER").upper() if user_info else "USER" + except Exception: + return "USER" + + +def _require_admin_or_su(authorization: Optional[str]) -> str: + """Require ADMIN, SU, or ASSET_OWNER role. Returns the role string.""" + role = _get_user_role(authorization) + if role not in ("SU", "ADMIN", "ASSET_OWNER"): + raise HTTPException( + status_code=HTTPStatus.FORBIDDEN, + detail="This operation requires ADMIN, SU, or ASSET_OWNER role", + ) + return role + + +def _require_platform_quota_manager(authorization: Optional[str]) -> str: + """Require a role that can manage platform-level quotas.""" + role = _get_user_role(authorization) + if role not in ("SU", "ASSET_OWNER", "SPEED"): + raise HTTPException( + status_code=HTTPStatus.FORBIDDEN, + detail="This operation requires SU, ASSET_OWNER, or SPEED role", + ) + return role + + +def _get_manageable_index_names(tenant_id: str, user_id: str) -> set[str]: + """Return KB index names the current user can manage.""" + from services.vectordatabase_service import ( + ElasticSearchService, + get_vector_db_core, + ) + + result = ElasticSearchService.list_indices( + pattern="*", + include_stats=False, + target_tenant_id=tenant_id, + user_id=user_id, + vdb_core=get_vector_db_core(), + ) + if not isinstance(result, dict): + return set() + permissions = result.get("index_permissions", {}) + return { + index_name + for index_name, permission in permissions.items() + if permission in ("EDIT", "CREATOR") + } + + +# ═══════════════════════════════════════════════════════════════════════════ +# Tenant-Level Quota Endpoints +# ═══════════════════════════════════════════════════════════════════════════ + + +@tenant_quota_router.get("/{tenant_id}/quota") +def get_tenant_quota( + tenant_id: str = Path(..., description="Tenant ID"), + authorization: Optional[str] = Header(None), +): + """ + Get tenant quota configuration and summary. + Accessible to any authenticated tenant user. + """ + try: + user_id, auth_tenant_id = get_current_user_id(authorization) + # Use auth tenant_id if accessing own tenant; SU can see any + role = _get_user_role(authorization) + if role not in ("SU", "ASSET_OWNER") and tenant_id != auth_tenant_id: + raise HTTPException( + status_code=HTTPStatus.FORBIDDEN, + detail="Cannot access quota config for another tenant", + ) + + service = QuotaService(tenant_id, user_id) + hard_limit = service.get_hard_limit() + warning_config = service.get_warning_config() + summary = service.get_quota_summary() + + return JSONResponse( + status_code=HTTPStatus.OK, + content={ + **hard_limit, + **warning_config, + "summary": summary, + }, + ) + except HTTPException: + raise + except Exception as exc: + logger.exception("Error getting tenant quota for %s", tenant_id) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail=f"Error getting quota config: {str(exc)}", + ) + + +@tenant_quota_router.put("/{tenant_id}/quota") +def update_tenant_quota( + tenant_id: str = Path(..., description="Tenant ID"), + payload: Dict[str, Any] = Body(..., description="Quota config payload"), + authorization: Optional[str] = Header(None), +): + """ + Update tenant quota configuration (hard limit + warning config). + Restricted to ADMIN, SU, and ASSET_OWNER roles. + Rejects if hard limit was set by SU (hard_limit_editable = false). + """ + try: + user_id, auth_tenant_id = get_current_user_id(authorization) + role = _require_admin_or_su(authorization) + + # Non-SU users can only modify their own tenant + if role not in ("SU", "ASSET_OWNER") and tenant_id != auth_tenant_id: + raise HTTPException( + status_code=HTTPStatus.FORBIDDEN, + detail="Cannot modify quota config for another tenant", + ) + + service = QuotaService(tenant_id, user_id) + + # Apply hard limit + hard_limit_gb = payload.get("hard_limit_gb") + hard_limit_mb = payload.get("hard_limit_mb") + + if hard_limit_gb is not None or hard_limit_mb is not None: + # Check if hard limit is editable (task 9.6) + hard_limit_info = service.get_hard_limit() + if not hard_limit_info.get("hard_limit_editable", True): + # Only allow SU to modify SU-set limits + if role not in ("SU", "ASSET_OWNER"): + raise HTTPException( + status_code=HTTPStatus.FORBIDDEN, + detail="Tenant hard quota is managed by the platform administrator", + ) + + # SU sets tenant hard limit → mark as non-editable for tenant admin + if role in ("SU", "ASSET_OWNER"): + QuotaService.set_tenant_hard_limit( + tenant_id, + limit_gb=hard_limit_gb, + limit_mb=hard_limit_mb, + su_user_id=user_id, + ) + else: + service.set_hard_limit( + limit_gb=hard_limit_gb, + limit_mb=hard_limit_mb, + ) + + # Apply warning config + warning_enabled = payload.get("warning_enabled") + warning_pct = payload.get("warning_threshold_pct") + critical_pct = payload.get("critical_threshold_pct") + + if any(v is not None for v in [warning_enabled, warning_pct, critical_pct]): + try: + service.set_warning_config( + enabled=warning_enabled, + warning_pct=warning_pct, + critical_pct=critical_pct, + ) + except ValueError as exc: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, detail=str(exc) + ) + + # Return updated config + updated_hard_limit = service.get_hard_limit() + updated_warning = service.get_warning_config() + + return JSONResponse( + status_code=HTTPStatus.OK, + content={ + **updated_hard_limit, + **updated_warning, + "message": "Quota configuration updated successfully", + }, + ) + except PlatformQuotaConflictError as exc: + return _platform_quota_conflict_response(exc) + except HTTPException: + raise + except Exception as exc: + logger.exception("Error updating tenant quota for %s", tenant_id) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail=f"Error updating quota config: {str(exc)}", + ) + + +@tenant_quota_router.delete("/{tenant_id}/quota") +def delete_tenant_quota( + tenant_id: str = Path(..., description="Tenant ID"), + authorization: Optional[str] = Header(None), +): + """ + Remove all tenant quota configuration. + Restricted to ADMIN, SU, and ASSET_OWNER roles. + """ + try: + user_id, auth_tenant_id = get_current_user_id(authorization) + role = _require_admin_or_su(authorization) + + if role not in ("SU", "ASSET_OWNER") and tenant_id != auth_tenant_id: + raise HTTPException( + status_code=HTTPStatus.FORBIDDEN, + detail="Cannot modify quota config for another tenant", + ) + + service = QuotaService(tenant_id, user_id) + service.delete_hard_limit() + + return JSONResponse( + status_code=HTTPStatus.OK, + content={"message": "Quota configuration removed successfully"}, + ) + except PlatformQuotaConflictError as exc: + return _platform_quota_conflict_response(exc) + except HTTPException: + raise + except Exception as exc: + logger.exception("Error deleting tenant quota for %s", tenant_id) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail=f"Error deleting quota config: {str(exc)}", + ) + + +@tenant_quota_router.get("/{tenant_id}/quota/usage") +def get_tenant_quota_usage( + tenant_id: str = Path(..., description="Tenant ID"), + force_refresh: bool = Query(False, description="Bypass cache and recompute usage"), + detail: bool = Query(False, description="Include per-KB breakdown"), + authorization: Optional[str] = Header(None), +): + """ + Get tenant storage usage with optional per-KB breakdown. + Accessible to any authenticated tenant user. + """ + try: + user_id, auth_tenant_id = get_current_user_id(authorization) + role = _get_user_role(authorization) + if role not in ("SU", "ASSET_OWNER") and tenant_id != auth_tenant_id: + raise HTTPException( + status_code=HTTPStatus.FORBIDDEN, + detail="Cannot access usage data for another tenant", + ) + + service = QuotaService(tenant_id, user_id) + usage = service.get_usage(force_refresh=force_refresh, detail=detail) + if detail and role in ("USER", "DEV"): + manageable_index_names = _get_manageable_index_names( + tenant_id, user_id + ) + usage["breakdown"] = [ + item + for item in usage.get("breakdown", []) + if item.get("index_name") in manageable_index_names + ] + + return JSONResponse(status_code=HTTPStatus.OK, content=usage) + except HTTPException: + raise + except Exception as exc: + logger.exception("Error getting usage for tenant %s", tenant_id) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail=f"Error getting usage data: {str(exc)}", + ) + + +# ═══════════════════════════════════════════════════════════════════════════ +# Platform-Level Quota Endpoints (SU/ASSET_OWNER/SPEED only) +# ═══════════════════════════════════════════════════════════════════════════ + + +@platform_quota_router.get("/overview") +def get_platform_overview( + authorization: Optional[str] = Header(None), +): + """ + Get platform-level storage overview: all tenants' quotas and usage. + Restricted to SU, ASSET_OWNER, and SPEED roles. + """ + try: + _require_platform_quota_manager(authorization) + user_id, _ = get_current_user_id(authorization) + + overview = QuotaService.get_platform_overview(ASSET_OWNER_TENANT_ID) + return JSONResponse(status_code=HTTPStatus.OK, content=overview) + except HTTPException: + raise + except Exception as exc: + logger.exception("Error getting platform overview") + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail=f"Error getting platform overview: {str(exc)}", + ) + + +@platform_quota_router.put("/capacity") +def set_platform_capacity( + payload: Dict[str, Any] = Body(..., description="Capacity payload"), + authorization: Optional[str] = Header(None), +): + """ + Set platform-wide declared storage capacity. + Restricted to SU, ASSET_OWNER, and SPEED roles. + """ + try: + _require_platform_quota_manager(authorization) + user_id, _ = get_current_user_id(authorization) + + capacity_gb = payload.get("capacity_gb") + result = QuotaService.set_platform_capacity( + capacity_gb, ASSET_OWNER_TENANT_ID, user_id + ) + return JSONResponse( + status_code=HTTPStatus.OK, + content={**result, "message": "Platform capacity updated successfully"}, + ) + except PlatformQuotaConflictError as exc: + return _platform_quota_conflict_response(exc) + except HTTPException: + raise + except Exception as exc: + logger.exception("Error setting platform capacity") + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail=f"Error setting platform capacity: {str(exc)}", + ) + + +@platform_quota_router.delete("/capacity") +def delete_platform_capacity( + authorization: Optional[str] = Header(None), +): + """ + Remove platform capacity declaration. + Restricted to SU, ASSET_OWNER, and SPEED roles. + """ + try: + _require_platform_quota_manager(authorization) + user_id, _ = get_current_user_id(authorization) + + QuotaService.set_platform_capacity(None, ASSET_OWNER_TENANT_ID, user_id) + return JSONResponse( + status_code=HTTPStatus.OK, + content={"message": "Platform capacity removed successfully"}, + ) + except HTTPException: + raise + except Exception as exc: + logger.exception("Error deleting platform capacity") + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail=f"Error deleting platform capacity: {str(exc)}", + ) + + +@platform_quota_router.put("/tenants/{tenant_id}") +def set_tenant_hard_quota( + tenant_id: str = Path(..., description="Target tenant ID"), + payload: Dict[str, Any] = Body(..., description="Quota payload"), + authorization: Optional[str] = Header(None), +): + """ + SU sets a hard quota on a target tenant. + Restricted to SU, ASSET_OWNER, and SPEED roles. + """ + try: + _require_platform_quota_manager(authorization) + user_id, _ = get_current_user_id(authorization) + + hard_limit_gb = payload.get("hard_limit_gb") + hard_limit_mb = payload.get("hard_limit_mb") + result = QuotaService.set_tenant_hard_limit( + tenant_id, + limit_gb=hard_limit_gb, + limit_mb=hard_limit_mb, + su_user_id=user_id, + ) + return JSONResponse( + status_code=HTTPStatus.OK, + content={ + **result, + "tenant_id": tenant_id, + "message": "Tenant hard quota updated successfully", + }, + ) + except PlatformQuotaConflictError as exc: + return _platform_quota_conflict_response(exc) + except HTTPException: + raise + except Exception as exc: + logger.exception("Error setting tenant hard quota for %s", tenant_id) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail=f"Error setting tenant hard quota: {str(exc)}", + ) + + +@platform_quota_router.delete("/tenants/{tenant_id}") +def delete_tenant_hard_quota( + tenant_id: str = Path(..., description="Target tenant ID"), + authorization: Optional[str] = Header(None), +): + """ + SU removes a tenant's hard quota. + Restricted to SU, ASSET_OWNER, and SPEED roles. + """ + try: + _require_platform_quota_manager(authorization) + user_id, _ = get_current_user_id(authorization) + + QuotaService.delete_tenant_hard_limit(tenant_id, user_id) + return JSONResponse( + status_code=HTTPStatus.OK, + content={ + "tenant_id": tenant_id, + "message": "Tenant hard quota removed successfully", + }, + ) + except HTTPException: + raise + except Exception as exc: + logger.exception("Error deleting tenant hard quota for %s", tenant_id) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail=f"Error deleting tenant hard quota: {str(exc)}", + ) diff --git a/backend/apps/remote_mcp_app.py b/backend/apps/remote_mcp_app.py index aa362fcbda..b547a67dc4 100644 --- a/backend/apps/remote_mcp_app.py +++ b/backend/apps/remote_mcp_app.py @@ -170,10 +170,15 @@ async def add_mcp_service_endpoint( authorization_token=payload.authorization_token, custom_headers=payload.custom_headers, container_config=payload.container_config, + container_port=payload.container_port, registry_json=payload.registry_json, config_json=payload.config_json, market_id=payload.market_id, enabled=payload.enabled if payload.enabled is not None else False, + group_ids=payload.group_ids, + ingroup_permission=payload.ingroup_permission, + shared_fields=payload.shared_fields, + skip_health_check=payload.skip_health_check if payload.skip_health_check is not None else False, ) return JSONResponse( @@ -186,7 +191,10 @@ async def add_mcp_service_endpoint( raise HTTPException(status_code=HTTPStatus.CONFLICT, detail="MCP name already exists") except MCPConnectionError as e: logger.error(f"Failed to add MCP service: {e}") - raise HTTPException(status_code=HTTPStatus.SERVICE_UNAVAILABLE, detail="MCP connection failed") + raise HTTPException( + status_code=HTTPStatus.SERVICE_UNAVAILABLE, + detail=str(e) or "MCP connection failed" + ) except McpValidationError as e: raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) except Exception as e: @@ -222,6 +230,9 @@ async def add_container_mcp_service_endpoint( market_id=payload.market_id, port=payload.port, mcp_config=payload.mcp_config, + group_ids=payload.group_ids, + ingroup_permission=payload.ingroup_permission, + shared_fields=payload.shared_fields, ) return JSONResponse( @@ -254,7 +265,7 @@ async def add_container_mcp_service_endpoint( logger.error(f"MCP connection failed when adding container service: {e}") raise HTTPException( status_code=HTTPStatus.SERVICE_UNAVAILABLE, - detail="MCP connection failed" + detail=str(e) or "MCP connection failed" ) except Exception as e: logger.error(f"Failed to add container MCP service: {e}") @@ -293,6 +304,9 @@ async def update_mcp_service_endpoint( config_json=payload.config_json, tags=payload.tags, market_id=payload.market_id, + group_ids=payload.group_ids, + ingroup_permission=payload.ingroup_permission, + shared_fields=payload.shared_fields, ) return JSONResponse( @@ -302,6 +316,8 @@ async def update_mcp_service_endpoint( except McpNotFoundError as e: raise HTTPException(status_code=HTTPStatus.NOT_FOUND, detail=str(e)) + except McpNameConflictError as e: + raise HTTPException(status_code=HTTPStatus.CONFLICT, detail=str(e)) except McpValidationError as e: raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) except Exception as e: @@ -833,6 +849,12 @@ async def upload_mcp_image( None, description="Name for the MCP service (auto-generated if not provided)"), env_vars: Optional[str] = Form( None, description="Environment variables as JSON string"), + group_ids: Optional[str] = Form( + None, description="Comma-separated group IDs that can access this MCP"), + ingroup_permission: Optional[str] = Form( + None, description="Permission level: EDIT, READ_ONLY, PRIVATE"), + shared_fields: Optional[str] = Form( + None, description="JSON string of field-level sharing flags"), tenant_id: Optional[str] = Form( None, description="Tenant ID for filtering (uses auth if not provided)"), authorization: Optional[str] = Header(None), @@ -858,6 +880,9 @@ async def upload_mcp_image( port=port, service_name=service_name, env_vars=env_vars, + group_ids=group_ids, + ingroup_permission=ingroup_permission, + shared_fields=json.loads(shared_fields) if shared_fields else None, ) return JSONResponse(status_code=HTTPStatus.OK, content=result) diff --git a/backend/apps/runtime_app.py b/backend/apps/runtime_app.py index 49fe1286b8..5e6859036c 100644 --- a/backend/apps/runtime_app.py +++ b/backend/apps/runtime_app.py @@ -2,10 +2,10 @@ from apps.app_factory import create_app from apps.agent_app import agent_runtime_router as agent_router +from apps.agent_automation_app import conversation_automation_router, router as agent_automation_router from apps.voice_app import voice_runtime_router as voice_router from apps.conversation_management_app import router as conversation_management_router from apps.conversation_share_app import router as conversation_share_router -from apps.memory_config_app import router as memory_config_router from apps.file_management_app import file_management_runtime_router as file_management_router from apps.skill_app import skill_creator_router from middleware.exception_handler import ExceptionHandlerMiddleware @@ -20,9 +20,24 @@ app.add_middleware(ExceptionHandlerMiddleware) app.include_router(agent_router) +app.include_router(agent_automation_router) +app.include_router(conversation_automation_router) app.include_router(conversation_management_router) app.include_router(conversation_share_router) -app.include_router(memory_config_router) app.include_router(file_management_router) app.include_router(voice_router) app.include_router(skill_creator_router) + + +@app.on_event("startup") +async def start_agent_automation_scheduler(): + from services.agent_automation.scheduler import agent_automation_scheduler + + await agent_automation_scheduler.start() + + +@app.on_event("shutdown") +async def stop_agent_automation_scheduler(): + from services.agent_automation.scheduler import agent_automation_scheduler + + await agent_automation_scheduler.stop() diff --git a/backend/apps/skill_app.py b/backend/apps/skill_app.py index 7cc5b9f829..928357fb6a 100644 --- a/backend/apps/skill_app.py +++ b/backend/apps/skill_app.py @@ -17,6 +17,7 @@ stream_skill_creation, update_skill_list, get_official_skills_with_status, + install_skills_from_zip_for_tenant, ) from consts.model import SkillInstanceInfoRequest, SkillCreateRequest, SkillCreateInteractiveRequest, SkillUpdateRequest, SkillResponse from utils.auth_utils import get_current_user_id, get_current_user_info @@ -46,6 +47,8 @@ def _build_skill_update_data(request: SkillUpdateRequest) -> Dict[str, Any]: "content", "tags", "source", + "group_ids", + "ingroup_permission", "config_schemas", "config_values", ): @@ -66,11 +69,14 @@ async def list_skills( ) -> JSONResponse: """List all available skills for the current tenant (or a specific tenant for super admin).""" try: - _, current_tenant_id = get_current_user_id(authorization) + user_id, current_tenant_id = get_current_user_id(authorization) # Super admin can query a specific tenant's skills; otherwise use current user's tenant effective_tenant_id = tenant_id if tenant_id else current_tenant_id service = SkillService(tenant_id=effective_tenant_id) - skills = service.list_skills(tenant_id=effective_tenant_id) + skills = service.list_visible_skills( + tenant_id=effective_tenant_id, + user_id=user_id, + ) return JSONResponse(content={"skills": skills}) except SkillException as e: raise HTTPException(status_code=500, detail=str(e)) @@ -122,8 +128,6 @@ async def install_skills( """ try: user_id, current_tenant_id = get_current_user_id(authorization) - from services.skill_service import install_skills_from_zip_for_tenant - effective_tenant_id = tenant_id if tenant_id else current_tenant_id installed_names = install_skills_from_zip_for_tenant( skill_names=request.skill_names, @@ -165,6 +169,8 @@ async def create_skill( "tool_ids": tool_ids, "tags": request.tags, "source": request.source, + "group_ids": request.group_ids, + "ingroup_permission": request.ingroup_permission, "config_schemas": request.config_schemas, "config_values": request.config_values, "files": request.files if request.files else [], @@ -200,6 +206,7 @@ async def create_skill_from_file( """ try: user_id, tenant_id = get_current_user_id(authorization) + service = SkillService(tenant_id=tenant_id) content = await file.read() @@ -222,6 +229,8 @@ async def create_skill_from_file( except UnauthorizedError as e: logger.warning(f"Unauthorized: {e}") raise HTTPException(status_code=401, detail=str(e)) + except ForbiddenError as e: + raise HTTPException(status_code=403, detail=str(e)) except SkillException as e: error_msg = str(e).lower() logger.warning(f"SkillException: {e}") @@ -262,6 +271,8 @@ async def get_skill_file_tree( raise except UnauthorizedError as e: raise HTTPException(status_code=401, detail=str(e)) + except ForbiddenError as e: + raise HTTPException(status_code=403, detail=str(e)) except SkillException as e: raise HTTPException(status_code=500, detail=str(e)) except Exception as e: @@ -302,6 +313,8 @@ async def get_skill_file_content( raise except UnauthorizedError as e: raise HTTPException(status_code=401, detail=str(e)) + except ForbiddenError as e: + raise HTTPException(status_code=403, detail=str(e)) except SkillException as e: raise HTTPException(status_code=500, detail=str(e)) except Exception as e: @@ -309,7 +322,10 @@ async def get_skill_file_content( raise HTTPException(status_code=500, detail="Internal server error") -@router.put("/{skill_name}/upload") +@router.put( + "/{skill_name}/upload", + responses={403: {"description": "Not authorized to update this skill"}}, +) async def update_skill_from_file( skill_name: str, file: UploadFile = File(..., description="SKILL.md file or ZIP archive"), @@ -342,6 +358,8 @@ async def update_skill_from_file( return JSONResponse(content=skill) except UnauthorizedError as e: raise HTTPException(status_code=401, detail=str(e)) + except ForbiddenError as e: + raise HTTPException(status_code=403, detail=str(e)) except SkillException as e: if _NOT_FOUND_TEXT in str(e).lower(): raise HTTPException(status_code=404, detail=str(e)) @@ -382,18 +400,16 @@ async def get_skill_instance( # The instance's per-agent overrides are mapped to config_values for the frontend. skill = service.get_skill_by_id(skill_id, tenant_id) if skill: + instance_config_values = instance.get("config_values") or {} instance["skill_name"] = skill.get("name") instance["skill_description"] = skill.get("description", "") instance["skill_content"] = skill.get("content", "") # Template defaults from YAML-enriched skill instance["config_schemas"] = skill.get("config_schemas") or [] - instance["config_values"] = skill.get("config_values") or {} # Per-agent overrides from SkillInstance.config_values override the template defaults - instance_params = instance.get("config_values") or {} - if instance_params: - merged = dict(instance.get("config_values") or {}) - merged.update(instance_params) - instance["config_values"] = merged + merged = dict(skill.get("config_values") or {}) + merged.update(instance_config_values) + instance["config_values"] = merged return JSONResponse(content=instance) except UnauthorizedError as e: @@ -434,16 +450,14 @@ async def update_skill_instance( ) # Enrich with template info so the frontend gets config_schemas and config_values + instance_config_values = instance.get("config_values") or {} instance["skill_name"] = skill.get("name") instance["skill_description"] = skill.get("description", "") instance["skill_content"] = skill.get("content", "") instance["config_schemas"] = skill.get("config_schemas") or [] - instance["config_values"] = skill.get("config_values") or {} - instance_params = instance.get("config_values") or {} - if instance_params: - merged = dict(instance.get("config_values") or {}) - merged.update(instance_params) - instance["config_values"] = merged + merged = dict(skill.get("config_values") or {}) + merged.update(instance_config_values) + instance["config_values"] = merged return JSONResponse(content={"message": "Skill instance updated", "instance": instance}) except UnauthorizedError as e: @@ -602,7 +616,10 @@ async def get_skill(skill_name: str, authorization: Optional[str] = Header(None) raise HTTPException(status_code=500, detail="Internal server error") -@router.put("/{skill_name}") +@router.put( + "/{skill_name}", + responses={403: {"description": "Not authorized to update this skill"}}, +) async def update_skill( skill_name: str, request: SkillUpdateRequest, @@ -643,6 +660,8 @@ async def update_skill( return JSONResponse(content=skill) except UnauthorizedError as e: raise HTTPException(status_code=401, detail=str(e)) + except ForbiddenError as e: + raise HTTPException(status_code=403, detail=str(e)) except SkillException as e: if _NOT_FOUND_TEXT in str(e).lower(): raise HTTPException(status_code=404, detail=str(e)) @@ -695,7 +714,7 @@ def _build_model_config_from_tenant(tenant_id: str) -> ModelConfig: url=quick_config.get("base_url", ""), temperature=0.1, top_p=0.95, - ssl_verify=True, + ssl_verify=quick_config.get("ssl_verify", False), model_factory=model_factory, prompt_cache=resolve_prompt_cache_profile(model_factory), ) diff --git a/backend/apps/skill_repository_app.py b/backend/apps/skill_repository_app.py index 108642f6ad..0dbe4d6b74 100644 --- a/backend/apps/skill_repository_app.py +++ b/backend/apps/skill_repository_app.py @@ -8,7 +8,9 @@ from consts.exceptions import ForbiddenError, SkillDuplicateError, UnauthorizedError from consts.model import SkillRepositoryInstallRequest, SkillRepositoryListingCreateRequest from services.skill_repository_service import ( + count_my_editable_skills_impl, create_skill_repository_listing_impl, + ensure_skill_repository_access, get_skill_repository_listing_detail_impl, install_skill_from_repository_impl, list_my_editable_skills_impl, @@ -46,9 +48,11 @@ async def list_skill_repository_listings_api( ): """List all skill marketplace repository listings with optional filters.""" try: - _, tenant_id = get_current_user_id(authorization) + user_id, tenant_id = get_current_user_id(authorization) + ensure_skill_repository_access(user_id) result = list_skill_repository_listings_impl( tenant_id, + user_id=user_id, status=status, skill_id=skill_id, category_id=category_id, @@ -63,6 +67,11 @@ async def list_skill_repository_listings_api( f"Unauthorized skill repository listings access attempt: {str(e)}" ) raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(e)) + except ForbiddenError as e: + logger.warning( + f"Forbidden skill repository listings access attempt: {str(e)}" + ) + raise HTTPException(status_code=HTTPStatus.FORBIDDEN, detail=str(e)) except ValueError as e: logger.warning( f"Invalid skill repository listings request parameters: {str(e)}" @@ -115,6 +124,25 @@ async def list_my_editable_skills_api( raise HTTPException(status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) +@skill_repository_router.get("/mine/counts") +async def count_my_editable_skills_api( + authorization: str = Header(None), +): + """Count visible skills by ownership without loading full skill records.""" + try: + user_id, tenant_id = get_current_user_id(authorization) + result = count_my_editable_skills_impl( + tenant_id=tenant_id, + user_id=user_id, + ) + return JSONResponse(status_code=HTTPStatus.OK, content=result) + except UnauthorizedError as e: + logger.warning( + f"Unauthorized my editable skill counts access attempt: {str(e)}" + ) + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(e)) + + @skill_repository_router.get("/{skill_repository_id}") async def get_skill_repository_listing_detail_api( skill_repository_id: int, @@ -122,7 +150,8 @@ async def get_skill_repository_listing_detail_api( ): """Get detailed skill marketplace repository listing by primary key.""" try: - _, tenant_id = get_current_user_id(authorization) + user_id, tenant_id = get_current_user_id(authorization) + ensure_skill_repository_access(user_id) result = get_skill_repository_listing_detail_impl( skill_repository_id, tenant_id, @@ -134,6 +163,12 @@ async def get_skill_repository_listing_detail_api( f"(id={skill_repository_id}): {str(e)}" ) raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(e)) + except ForbiddenError as e: + logger.warning( + f"Forbidden skill repository listing detail access attempt " + f"(id={skill_repository_id}): {str(e)}" + ) + raise HTTPException(status_code=HTTPStatus.FORBIDDEN, detail=str(e)) except ValueError as e: logger.warning( f"Skill repository listing not found (id={skill_repository_id}): {str(e)}" @@ -151,16 +186,23 @@ async def update_skill_repository_status_api( "New status: not_shared / pending_review / rejected / shared" ), ), + content: Optional[str] = Body( + None, + embed=True, + description="Review opinion or resubmit note", + ), authorization: str = Header(None), ): """Update skill marketplace repository listing status.""" try: user_id, tenant_id = get_current_user_id(authorization) + ensure_skill_repository_access(user_id) result = update_skill_repository_status_impl( skill_repository_id=skill_repository_id, status=status, user_id=user_id, tenant_id=tenant_id, + content=content, ) return JSONResponse(status_code=HTTPStatus.OK, content=result) except UnauthorizedError as e: @@ -244,11 +286,17 @@ async def install_skill_from_repository_api( ) raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(e)) except ValueError as e: + message = str(e) + status_code = ( + HTTPStatus.NOT_FOUND + if "not found" in message.lower() + else HTTPStatus.BAD_REQUEST + ) logger.warning( - f"Skill repository listing not found for install " - f"(id={skill_repository_id}): {str(e)}" + f"Invalid skill repository install request " + f"(id={skill_repository_id}): {message}" ) - raise HTTPException(status_code=HTTPStatus.NOT_FOUND, detail=str(e)) + raise HTTPException(status_code=status_code, detail=message) except SkillDuplicateError as e: logger.warning( f"Duplicate skill repository install attempt " diff --git a/backend/apps/tenant_app.py b/backend/apps/tenant_app.py index 291cd22fa9..9ce1e6aa50 100644 --- a/backend/apps/tenant_app.py +++ b/backend/apps/tenant_app.py @@ -13,15 +13,15 @@ TenantCreateRequest, TenantUpdateRequest, ) -from consts.exceptions import NotFoundException, ValidationError, UnauthorizedError +from consts.exceptions import ForbiddenError, NotFoundException, ValidationError, UnauthorizedError from services.tenant_service import ( create_tenant, - get_tenant_info, - get_tenants_paginated, + get_tenant_info_for_user, + get_tenants_paginated_for_user, update_tenant_info, delete_tenant, ) -from utils.auth_utils import get_current_user_id +from utils.auth_utils import get_current_user_context, get_current_user_id logger = logging.getLogger(__name__) router = APIRouter(prefix="/tenants", tags=["tenants"]) @@ -86,7 +86,10 @@ async def create_tenant_endpoint( @router.get("/{tenant_id}") -async def get_tenant_endpoint(tenant_id: str) -> JSONResponse: +async def get_tenant_endpoint( + tenant_id: str, + authorization: Optional[str] = Header(None), +) -> JSONResponse: """ Get tenant information by tenant ID @@ -97,8 +100,12 @@ async def get_tenant_endpoint(tenant_id: str) -> JSONResponse: JSONResponse: Tenant information """ try: - # Get tenant info - tenant_info = get_tenant_info(tenant_id) + _, requester_tenant_id, requester_role = get_current_user_context(authorization) + tenant_info = get_tenant_info_for_user( + tenant_id, + requester_tenant_id=requester_tenant_id, + requester_role=requester_role, + ) return JSONResponse( status_code=HTTPStatus.OK, @@ -108,6 +115,10 @@ async def get_tenant_endpoint(tenant_id: str) -> JSONResponse: } ) + except UnauthorizedError as exc: + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(exc)) + except ForbiddenError as exc: + raise HTTPException(status_code=HTTPStatus.FORBIDDEN, detail=str(exc)) except NotFoundException as exc: logger.warning(f"Tenant not found: {tenant_id}") raise HTTPException( @@ -124,7 +135,8 @@ async def get_tenant_endpoint(tenant_id: str) -> JSONResponse: @router.post("/tenant-list") async def get_all_tenants_endpoint( - pagination: PaginationRequest = Body(...) + pagination: PaginationRequest = Body(...), + authorization: Optional[str] = Header(None), ) -> JSONResponse: """ Get all tenants with pagination support @@ -136,8 +148,12 @@ async def get_all_tenants_endpoint( JSONResponse: Paginated list of tenants with total count """ try: - # Get paginated tenants - result = get_tenants_paginated(page=pagination.page, page_size=pagination.page_size) + _, _, requester_role = get_current_user_context(authorization) + result = get_tenants_paginated_for_user( + page=pagination.page, + page_size=pagination.page_size, + requester_role=requester_role, + ) return JSONResponse( status_code=HTTPStatus.OK, @@ -151,6 +167,10 @@ async def get_all_tenants_endpoint( } ) + except UnauthorizedError as exc: + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(exc)) + except ForbiddenError as exc: + raise HTTPException(status_code=HTTPStatus.FORBIDDEN, detail=str(exc)) except Exception as exc: logger.error(f"Unexpected error retrieving tenants: {str(exc)}") raise HTTPException( diff --git a/backend/apps/tenant_config_app.py b/backend/apps/tenant_config_app.py index cd67f0c8fe..dfddd97f4d 100644 --- a/backend/apps/tenant_config_app.py +++ b/backend/apps/tenant_config_app.py @@ -4,7 +4,7 @@ from fastapi import APIRouter, HTTPException from fastapi.responses import JSONResponse -from consts.const import DEPLOYMENT_VERSION, APP_VERSION +from consts.const import DEPLOYMENT_VERSION, APP_VERSION, ENABLE_AIDP_KNOWLEDGE logger = logging.getLogger("tenant_config_app") router = APIRouter(prefix="/tenant_config") @@ -20,6 +20,7 @@ def get_deployment_version(): status_code=HTTPStatus.OK, content={"deployment_version": DEPLOYMENT_VERSION, "app_version": APP_VERSION, + "enable_aidp_knowledge": ENABLE_AIDP_KNOWLEDGE, "status": "success"} ) except Exception as e: diff --git a/backend/apps/tool_config_app.py b/backend/apps/tool_config_app.py index e0fcda6522..188c4820c6 100644 --- a/backend/apps/tool_config_app.py +++ b/backend/apps/tool_config_app.py @@ -19,6 +19,7 @@ delete_openapi_service, _refresh_openapi_services_in_mcp, ) +from database.user_tenant_db import get_user_email_map from utils.auth_utils import get_current_user_id router = APIRouter(prefix="/tool") @@ -301,9 +302,15 @@ async def update_tool_labels_api( status_code=HTTPStatus.NOT_FOUND, detail="Tool not found or access denied" ) + updated_by_name = get_user_email_map([user_id]).get(user_id, "") return JSONResponse( status_code=HTTPStatus.OK, - content={"message": "Labels updated successfully", "status": "success", "labels": labels} + content={ + "message": "Labels updated successfully", + "status": "success", + "labels": labels, + "updated_by_name": updated_by_name, + }, ) except HTTPException: raise diff --git a/backend/apps/user_app.py b/backend/apps/user_app.py index a311695a6e..a2e2a8925e 100644 --- a/backend/apps/user_app.py +++ b/backend/apps/user_app.py @@ -11,11 +11,12 @@ from consts.model import ( UserListRequest, UserUpdateRequest ) +from consts.exceptions import ForbiddenError, NotFoundException, UnauthorizedError from services.user_service import ( - get_users, update_user, delete_user_and_cleanup + delete_user_and_cleanup, get_users_for_requester, update_user_for_requester ) from database.user_tenant_db import get_user_tenant_by_user_id -from utils.auth_utils import get_current_user_id +from utils.auth_utils import get_current_user_context, get_current_user_id logger = logging.getLogger("user_app") router = APIRouter(prefix="/users", tags=["users"]) @@ -24,6 +25,7 @@ @router.post("/list") async def get_users_endpoint( request: UserListRequest, + authorization: Optional[str] = Header(None), ) -> JSONResponse: """ Get users belonging to a specific tenant with pagination @@ -36,9 +38,16 @@ async def get_users_endpoint( JSONResponse: List of users in the tenant (paginated or all) """ try: - # Get tenant users with pagination and sorting - result = get_users(request.tenant_id, request.page, request.page_size, - request.sort_by, request.sort_order) + _, requester_tenant_id, requester_role = get_current_user_context(authorization) + result = get_users_for_requester( + request.tenant_id, + request.page, + request.page_size, + request.sort_by, + request.sort_order, + requester_tenant_id=requester_tenant_id, + requester_role=requester_role, + ) # Build response content content = { @@ -60,6 +69,10 @@ async def get_users_endpoint( status_code=HTTPStatus.OK, content=content ) + except UnauthorizedError as exc: + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(exc)) + except ForbiddenError as exc: + raise HTTPException(status_code=HTTPStatus.FORBIDDEN, detail=str(exc)) except Exception as exc: logger.error(f"Unexpected error retrieving users for tenant {request.tenant_id}: {str(exc)}") # Include the actual error message for debugging @@ -88,10 +101,16 @@ async def update_user_endpoint( """ try: # Get current user ID from token for access control - current_user_id, _ = get_current_user_id(authorization) + current_user_id, requester_tenant_id, requester_role = get_current_user_context(authorization) # Update user - updated_user = await update_user(user_id, request.model_dump(), current_user_id) + updated_user = await update_user_for_requester( + user_id, + request.model_dump(), + updated_by=current_user_id, + requester_tenant_id=requester_tenant_id, + requester_role=requester_role, + ) logger.info(f"Updated user {user_id} by user {current_user_id}") @@ -103,6 +122,12 @@ async def update_user_endpoint( } ) + except UnauthorizedError as exc: + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(exc)) + except ForbiddenError as exc: + raise HTTPException(status_code=HTTPStatus.FORBIDDEN, detail=str(exc)) + except NotFoundException as exc: + raise HTTPException(status_code=HTTPStatus.NOT_FOUND, detail=str(exc)) except ValueError as exc: logger.warning(f"User update validation error for user {user_id}: {str(exc)}") raise HTTPException( @@ -175,4 +200,3 @@ async def delete_user_endpoint( status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail=f"Failed to delete user: {str(exc)}" ) - diff --git a/backend/apps/vectordatabase_app.py b/backend/apps/vectordatabase_app.py index 505c395590..657e9a3945 100644 --- a/backend/apps/vectordatabase_app.py +++ b/backend/apps/vectordatabase_app.py @@ -24,6 +24,10 @@ from utils.file_management_utils import get_all_files_status from database.knowledge_db import get_index_name_by_knowledge_name, get_knowledge_record from database.model_management_db import get_model_by_model_id +from apps.permission_utils import ( + require_knowledge_base_edit_permission, + require_knowledge_base_read_permission, +) router = APIRouter(prefix="/indices") service = ElasticSearchService() @@ -76,7 +80,7 @@ def create_new_index( embedding_dim: Optional[int] = Query( None, description="Dimension of the embedding vectors"), request: Dict[str, Any] = Body( - None, description="Request body with optional fields (ingroup_permission, group_ids, embedding_model_name, preserve_source_file)"), + None, description="Request body containing embedding_model_id and optional knowledge-base settings"), vdb_core: VectorDatabaseCore = Depends(get_vector_db_core), authorization: Optional[str] = Header(None) ): @@ -87,15 +91,18 @@ def create_new_index( # Extract optional fields from request body ingroup_permission = None group_ids = None - embedding_model_name: Optional[str] = None - is_multimodal: Optional[bool] = None + embedding_model_id: Optional[int] = None preserve_source_file: Optional[bool] = None + quota_limit_bytes: Optional[int] = None if request: ingroup_permission = request.get("ingroup_permission") group_ids = request.get("group_ids") - embedding_model_name = request.get("embeddingModel") - is_multimodal = request.get("is_multimodal") + embedding_model_id = request.get("embedding_model_id") preserve_source_file = request.get("preserve_source_file") + quota_limit_bytes = request.get("quota_limit_bytes") + + if isinstance(embedding_model_id, bool) or not isinstance(embedding_model_id, int): + raise ValueError("embedding_model_id must be an integer") # Treat path parameter as user-facing knowledge base name for new creations return ElasticSearchService.create_knowledge_base( @@ -106,10 +113,15 @@ def create_new_index( tenant_id=tenant_id, ingroup_permission=ingroup_permission, group_ids=group_ids, - embedding_model_name=embedding_model_name, - is_multimodal=is_multimodal, + embedding_model_id=embedding_model_id, preserve_source_file=preserve_source_file, + quota_limit_bytes=quota_limit_bytes, ) + except HTTPException: + raise + except (TypeError, ValueError) as e: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, detail=str(e)) except Exception as e: raise HTTPException( status_code=HTTPStatus.INTERNAL_SERVER_ERROR, detail=f"Error creating index: {str(e)}") @@ -125,9 +137,12 @@ async def delete_index( logger.debug(f"Received request to delete knowledge base: {index_name}") try: user_id, tenant_id = get_current_user_id(authorization) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) # Call the centralized full deletion service result = await ElasticSearchService.full_delete_knowledge_base(index_name, vdb_core, user_id) return result + except HTTPException: + raise except Exception as e: logger.error( f"Error during API call to delete index '{index_name}': {str(e)}", exc_info=True) @@ -147,21 +162,25 @@ async def update_index( user_id, auth_tenant_id = get_current_user_id(authorization) # Use explicit tenant_id if provided, otherwise fall back to auth tenant_id tenant_id = request.get("tenant_id") or auth_tenant_id + require_knowledge_base_edit_permission(index_name, user_id, auth_tenant_id) # Extract update fields knowledge_name = request.get("knowledge_name") ingroup_permission = request.get("ingroup_permission") group_ids = request.get("group_ids") - # Call service layer to update knowledge base - result = ElasticSearchService.update_knowledge_base( - index_name=index_name, - knowledge_name=knowledge_name, - ingroup_permission=ingroup_permission, - group_ids=group_ids, - tenant_id=tenant_id, - user_id=user_id, - ) + update_kwargs = { + "index_name": index_name, + "knowledge_name": knowledge_name, + "ingroup_permission": ingroup_permission, + "group_ids": group_ids, + "tenant_id": tenant_id, + "user_id": user_id, + } + if "quota_limit_bytes" in request: + update_kwargs["quota_limit_bytes"] = request["quota_limit_bytes"] + + result = ElasticSearchService.update_knowledge_base(**update_kwargs) if result: return JSONResponse( @@ -197,6 +216,7 @@ async def update_summary_frequency_endpoint( """Update the auto-summary frequency for a knowledge base.""" try: user_id, tenant_id = get_current_user_id(authorization) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) summary_frequency = request.get("summary_frequency") valid_frequencies = VALID_SUMMARY_FREQUENCIES @@ -337,6 +357,7 @@ def update_embedding_model( """ try: user_id, tenant_id = get_current_user_id(authorization) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) model_id = request.get("model_id") if not model_id: @@ -456,6 +477,7 @@ def create_index_documents( """ try: user_id, tenant_id = get_current_user_id(authorization) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) # Get the knowledge base record to retrieve the saved embedding model knowledge_record = get_knowledge_record({'index_name': index_name}) @@ -477,6 +499,8 @@ def create_index_documents( large_mode=large_mode, model_id=saved_embedding_model_id, ) + except HTTPException: + raise except Exception as e: error_msg = str(e) logger.error(f"Error indexing documents: {error_msg}") @@ -519,10 +543,13 @@ async def delete_documents( "full: delete ES documents, MinIO source, and Redis task records" ), ), - vdb_core: VectorDatabaseCore = Depends(get_vector_db_core) + vdb_core: VectorDatabaseCore = Depends(get_vector_db_core), + authorization: Optional[str] = Header(None), ): """Delete a document by scope: source file only or full removal from the index.""" try: + user_id, tenant_id = get_current_user_id(authorization) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) result = await ElasticSearchService.delete_document_by_scope( index_name, path_or_url, scope, vdb_core ) @@ -567,6 +594,8 @@ async def delete_documents( raise HTTPException( status_code=HTTPStatus.BAD_REQUEST, detail=str(exc) ) + except HTTPException: + raise except Exception as e: raise HTTPException( status_code=HTTPStatus.INTERNAL_SERVER_ERROR, @@ -701,6 +730,7 @@ def create_chunk( """Create a manual chunk.""" try: user_id, tenant_id = get_current_user_id(authorization) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) result = ElasticSearchService.create_chunk( index_name=index_name, chunk_request=payload, @@ -714,6 +744,8 @@ def create_chunk( status_code=HTTPStatus.NOT_FOUND, detail=str(e) ) + except HTTPException: + raise except Exception as exc: logger.error( "Error creating chunk for index %s: %s", index_name, exc, exc_info=True @@ -736,6 +768,7 @@ def update_chunk( """Update an existing chunk.""" try: user_id, tenant_id = get_current_user_id(authorization) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) result = ElasticSearchService.update_chunk( index_name=index_name, chunk_id=chunk_id, @@ -750,6 +783,8 @@ def update_chunk( status_code=HTTPStatus.NOT_FOUND, detail=str(e) ) + except HTTPException: + raise except Exception as exc: logger.error( "Error updating chunk %s for index %s: %s", @@ -773,7 +808,8 @@ def delete_chunk( ): """Delete a chunk.""" try: - get_current_user_id(authorization) + user_id, tenant_id = get_current_user_id(authorization) + require_knowledge_base_edit_permission(index_name, user_id, tenant_id) result = ElasticSearchService.delete_chunk( index_name=index_name, chunk_id=chunk_id, @@ -785,6 +821,8 @@ def delete_chunk( status_code=HTTPStatus.NOT_FOUND, detail=str(e) ) + except HTTPException: + raise except Exception as exc: logger.error( "Error deleting chunk %s for index %s: %s", @@ -806,7 +844,7 @@ async def hybrid_search( ): """Run a hybrid (accurate + semantic) search across indices.""" try: - _, tenant_id = get_current_user_id(authorization) + user_id, tenant_id = get_current_user_id(authorization) resolved_index_names: List[str] = [] for requested_name in payload.index_names: try: @@ -815,6 +853,11 @@ async def hybrid_search( ) except Exception: resolved_name = requested_name + # Enforce per-KB read permission before searching. The permission layer + # maps ValueError (KB not found) -> 404 and PermissionError (no access) -> 403. + require_knowledge_base_read_permission( + index_name=resolved_name, user_id=user_id, tenant_id=tenant_id, + ) resolved_index_names.append(resolved_name) result = ElasticSearchService.search_hybrid( index_names=resolved_index_names, @@ -839,6 +882,9 @@ async def hybrid_search( except ValueError as exc: raise HTTPException( status_code=HTTPStatus.BAD_REQUEST, detail=str(exc)) + except HTTPException: + # Re-raise HTTP exceptions (e.g. 403 from permission check) as-is + raise except Exception as exc: logger.error(f"Hybrid search failed: {exc}", exc_info=True) raise HTTPException( diff --git a/backend/consts/a2a_models.py b/backend/consts/a2a_models.py index 29eb104149..2ec309006b 100644 --- a/backend/consts/a2a_models.py +++ b/backend/consts/a2a_models.py @@ -133,9 +133,13 @@ class A2AAgentCard(BaseModel): # ============================================================================= class DiscoverFromUrlRequest(BaseModel): - """Request to discover an external A2A agent from URL.""" + """Request to discover an external A2A agent from an Agent Card URL.""" url: str = Field(description="Direct URL to the Agent Card") name: Optional[str] = Field(default=None, description="Optional display name override") + custom_headers: Optional[Dict[str, str]] = Field( + default=None, + description="Headers saved for Agent Card discovery and refresh only" + ) class DiscoverFromNacosRequest(BaseModel): diff --git a/backend/consts/const.py b/backend/consts/const.py index eabf6a8b37..fffda132d0 100644 --- a/backend/consts/const.py +++ b/backend/consts/const.py @@ -46,6 +46,29 @@ class VectorDatabaseType(str, Enum): PER_WAVE_TIMEOUT = int(os.getenv("DP_SPLIT_WAIT_TIMEOUT_PER_WAVE_S", "30")) MAX_TIMEOUT = int(os.getenv("DP_SPLIT_WAIT_TIMEOUT_MAX_S", "1800")) +# Agent automation runtime configuration +AGENT_AUTOMATION_ENABLED = os.getenv( + "AGENT_AUTOMATION_ENABLED", "true" +).lower() in ("true", "1", "yes", "on") +AGENT_AUTOMATION_POLL_INTERVAL_SECONDS = int( + os.getenv("AGENT_AUTOMATION_POLL_INTERVAL_SECONDS", "5") +) +AGENT_AUTOMATION_MAX_CONCURRENT_RUNS = int( + os.getenv("AGENT_AUTOMATION_MAX_CONCURRENT_RUNS", "2") +) +AGENT_AUTOMATION_LEASE_SECONDS = int( + os.getenv("AGENT_AUTOMATION_LEASE_SECONDS", "120") +) +AGENT_AUTOMATION_DEFAULT_TIMEOUT_SECONDS = int( + os.getenv("AGENT_AUTOMATION_DEFAULT_TIMEOUT_SECONDS", "1800") +) +AGENT_AUTOMATION_SHUTDOWN_GRACE_SECONDS = int( + os.getenv("AGENT_AUTOMATION_SHUTDOWN_GRACE_SECONDS", "30") +) +AGENT_AUTOMATION_MIN_INTERVAL_SECONDS = int( + os.getenv("AGENT_AUTOMATION_MIN_INTERVAL_SECONDS", "5") +) + # Container-internal skills storage path CONTAINER_SKILLS_PATH = os.getenv("SKILLS_PATH") @@ -85,7 +108,7 @@ class VectorDatabaseType(str, Enum): # OAuth Configuration -OAUTH_CALLBACK_BASE_URL = os.getenv("OAUTH_CALLBACK_BASE_URL", "") +OAUTH_CALLBACK_BASE_URL = os.getenv("OAUTH_CALLBACK_BASE_URL", "").rstrip("/") OAUTH_SSL_VERIFY = os.getenv("OAUTH_SSL_VERIFY", "true").lower() == "true" OAUTH_CA_BUNDLE = os.getenv("OAUTH_CA_BUNDLE", "") # OAuth login mode: @@ -183,6 +206,12 @@ class VectorDatabaseType(str, Enum): # Deployment Version Configuration DEPLOYMENT_VERSION = os.getenv("DEPLOYMENT_VERSION", "speed") IS_SPEED_MODE = DEPLOYMENT_VERSION == "speed" + +# AIDP Knowledge Base configuration +ENABLE_AIDP_KNOWLEDGE = os.getenv("ENABLE_AIDP_KNOWLEDGE", "false").lower() in ("true", "1", "yes", "on") +AIDP_SERVER_URL = os.getenv("AIDP_SERVER_URL", "") +AIDP_API_KEY = os.getenv("AIDP_API_KEY", "") +AIDP_TENANT_ID = os.getenv("AIDP_TENANT_ID", "aidp") DEFAULT_APP_DESCRIPTION_ZH = "Nexent 是一个开源智能体平台，基于 MCP 工具生态系统，提供灵活的多模态问答、检索、数据分析、处理等能力。" DEFAULT_APP_DESCRIPTION_EN = "Nexent is an open-source agent platform built on the MCP tool ecosystem, providing flexible multi-modal Q&A, retrieval, data analysis, and processing capabilities." DEFAULT_APP_NAME_ZH = "Nexent 智能体" @@ -194,6 +223,7 @@ class VectorDatabaseType(str, Enum): MINIO_SECRET_KEY = os.getenv("MINIO_SECRET_KEY") MINIO_REGION = os.getenv("MINIO_REGION") MINIO_DEFAULT_BUCKET = os.getenv("MINIO_DEFAULT_BUCKET") +MINIO_SECURE = os.getenv("MINIO_SECURE", "true").lower() == "true" S3_URL_PREFIX = "s3://" @@ -220,12 +250,11 @@ class VectorDatabaseType(str, Enum): NORTHBOUND_RATE_LIMIT_ENABLED = os.getenv("NORTHBOUND_RATE_LIMIT_ENABLED", "true").lower() == "true" NORTHBOUND_RATE_LIMIT_PER_MINUTE = int(os.getenv("NORTHBOUND_RATE_LIMIT_PER_MINUTE", "120")) FLOWER_PORT = int(os.getenv("FLOWER_PORT", "5555")) -DP_REDIS_CHUNKS_WAIT_TIMEOUT_S = int( - os.getenv("DP_REDIS_CHUNKS_WAIT_TIMEOUT_S", "30")) -DP_REDIS_CHUNKS_POLL_INTERVAL_MS = int( - os.getenv("DP_REDIS_CHUNKS_POLL_INTERVAL_MS", "200")) -FORWARD_REDIS_RETRY_DELAY_S = int( - os.getenv("FORWARD_REDIS_RETRY_DELAY_S", "5")) +DP_REDIS_CHUNKS_WAIT_TIMEOUT_S = int(os.getenv("DP_REDIS_CHUNKS_WAIT_TIMEOUT_S", "300")) +DP_REDIS_CHUNKS_POLL_INTERVAL_MS = int(os.getenv("DP_REDIS_CHUNKS_POLL_INTERVAL_MS", "200")) +REDIS_ERROR_INFO_TTL_SECONDS = int(os.getenv("REDIS_ERROR_INFO_TTL_SECONDS", str(1 * 24 * 60 * 60))) +REDIS_ERROR_INFO_SCAN_COUNT = int(os.getenv("REDIS_ERROR_INFO_SCAN_COUNT", "500")) +FORWARD_REDIS_RETRY_DELAY_S = int(os.getenv("FORWARD_REDIS_RETRY_DELAY_S", "5")) FORWARD_REDIS_RETRY_MAX = int(os.getenv("FORWARD_REDIS_RETRY_MAX", "12")) @@ -302,6 +331,65 @@ class VectorDatabaseType(str, Enum): # Boolean value representations for configuration parsing BOOLEAN_TRUE_VALUES = {"true", "1", "y", "yes", "on"} +# ===== Memory System ===== + +# MMR (Maximal Marginal Relevance) configuration +MMR_LAMBDA = float(os.getenv("MMR_LAMBDA", "0.7")) +MMR_CANDIDATE_TOP_K = int(os.getenv("MMR_CANDIDATE_TOP_K", "10")) +MMR_FINAL_TOP_K = int(os.getenv("MMR_FINAL_TOP_K", "5")) +MMR_DUPLICATE_THRESHOLD = float(os.getenv("MMR_DUPLICATE_THRESHOLD", "0.92")) + +# Temporal decay (only applied to internal agent short-term memory) +AGENT_SHORT_TERM_HALF_LIFE_DAYS = int( + os.getenv("AGENT_SHORT_TERM_HALF_LIFE_DAYS", "14") +) + +# Score fusion source weights +W_AGENT_SHORT_TERM = float(os.getenv("W_AGENT_SHORT_TERM", "1.0")) +W_EXTERNAL = float(os.getenv("W_EXTERNAL", "0.8")) + +# Token budget selection +MEMORY_TOKEN_BUDGET = int(os.getenv("MEMORY_TOKEN_BUDGET", "2000")) + +# Dreaming promotion thresholds +LIGHT_SLEEP_WINDOW_DAYS = int(os.getenv("LIGHT_SLEEP_WINDOW_DAYS", "7")) +RECENCY_HALF_LIFE_DAYS = int(os.getenv("RECENCY_HALF_LIFE_DAYS", "14")) +MIN_PROMOTION_SCORE = float(os.getenv("MIN_PROMOTION_SCORE", "0.72")) +MIN_RECALL_COUNT = int(os.getenv("MIN_RECALL_COUNT", "3")) +MIN_UNIQUE_QUERIES = int(os.getenv("MIN_UNIQUE_QUERIES", "2")) +# Scheduling/cron constants are intentionally not defined here: the +# background Dreaming scheduler is not part of Phase 2 (an agent-driven +# timer will be added in a later phase, at which point the cron expression +# and heartbeat can be reintroduced). + +# External provider retry / timeout +PROVIDER_RETRY_MAX_ATTEMPTS = int(os.getenv("PROVIDER_RETRY_MAX_ATTEMPTS", "3")) +PROVIDER_RETRY_BACKOFF_BASE_SECONDS = int( + os.getenv("PROVIDER_RETRY_BACKOFF_BASE_SECONDS", "1") +) +PROVIDER_REQUEST_TIMEOUT_SECONDS = int( + os.getenv("PROVIDER_REQUEST_TIMEOUT_SECONDS", "30") +) + +# External provider toggles (configured per provider elsewhere; these constants +# describe protocol-level defaults) +EXTERNAL_MEMORY_DEFAULT_ALLOWED_UNIT_TYPES = ( + "model_output", + "model_output_thinking", + "model_output_deep_thinking", + "model_output_code", + "final_answer", + "error", + "search_content", + "tool", + "parse", + "execution_logs", + "picture_web", + "memory_search", + "verification", + "max_steps_reached", +) + DEFAULT_LLM_MAX_TOKENS = 4096 @@ -319,6 +407,11 @@ class VectorDatabaseType(str, Enum): # Invite code INVITE_CODE = os.getenv("INVITE_CODE") +# Access-token lifetime in seconds. This must match GoTrue's GOTRUE_JWT_EXP. +JWT_EXPIRY_SECONDS = int(os.getenv("JWT_EXPIRY", "7200") or 7200) +if JWT_EXPIRY_SECONDS <= 0: + raise ValueError("JWT_EXPIRY must be a positive number of seconds") + # Debug JWT expiration time (seconds), not set or 0 means not effective DEBUG_JWT_EXPIRE_SECONDS = int(os.getenv('DEBUG_JWT_EXPIRE_SECONDS', '0') or 0) @@ -563,6 +656,50 @@ def _resolve_app_version(default: str = "v2.2.1") -> str: APP_VERSION = _resolve_app_version() +# ============================================================================= +# Agent Sandbox Configuration +# ============================================================================= + +NEXENT_SANDBOX_DEFAULT_LEVEL = os.getenv("NEXENT_SANDBOX_DEFAULT_LEVEL", "local").lower() +"""Default sandbox isolation level: local / docker / wasm. + Default 'local' preserves backward-compatibility for existing deployments.""" + +NEXENT_SANDBOX_DEFAULT_SCOPE = os.getenv("NEXENT_SANDBOX_DEFAULT_SCOPE", "system").lower() +"""Default sandbox container lifecycle scope: session / system. + session = one container per agent_run, destroyed on run end (strictest isolation). + system = persistent warm pool shared by all runs (lowest cold-start latency).""" + +NEXENT_SANDBOX_DOCKER_IMAGE = os.getenv( + "NEXENT_SANDBOX_DOCKER_IMAGE", "nexent/nexent-sandbox:latest" +) +"""Docker image used when level is 'docker'.""" + +NEXENT_SANDBOX_MEMORY_LIMIT_MB = int(os.getenv("NEXENT_SANDBOX_MEMORY_LIMIT_MB", "512")) + +NEXENT_SANDBOX_CPU_QUOTA = float(os.getenv("NEXENT_SANDBOX_CPU_QUOTA", "1.0")) + +NEXENT_SANDBOX_TIMEOUT_S = int(os.getenv("NEXENT_SANDBOX_TIMEOUT_S", "30")) + +NEXENT_SANDBOX_NETWORK_DISABLED = ( + os.getenv("NEXENT_SANDBOX_NETWORK", "disabled").lower() == "disabled" +) + +NEXENT_SANDBOX_SHELL_POLICY = os.getenv( + "NEXENT_SANDBOX_SHELL_POLICY", "disabled" +).lower() +"""Shell execution policy: disabled / restricted / boxed. + 'disabled' is recommended — blocks subprocess/os shell calls at AST-parse time.""" + +NEXENT_SANDBOX_OUTPUT_BUCKET = os.getenv( + "NEXENT_SANDBOX_OUTPUT_BUCKET", "nexent-artifacts" +) +"""MinIO bucket for sandbox output file sync.""" + +NEXENT_SANDBOX_AUTO_SYNC_OUTPUTS = ( + os.getenv("NEXENT_SANDBOX_AUTO_SYNC_OUTPUTS", "true").lower() == "true" +) + + # Skill Creation Streaming Configuration STREAMABLE_CONTENT_TYPES = frozenset([ "model_output_thinking", diff --git a/backend/consts/error_code.py b/backend/consts/error_code.py index a6326668fd..c31873f732 100644 --- a/backend/consts/error_code.py +++ b/backend/consts/error_code.py @@ -194,6 +194,8 @@ class ErrorCode(Enum): AIDP_CONFIG_INVALID = "130502" # Invalid AIDP configuration AIDP_CONNECTION_ERROR = "130503" # AIDP connection error AIDP_AUTH_ERROR = "130504" # AIDP auth error + AIDP_RATE_LIMIT = "130505" # AIDP rate limit + AIDP_RESPONSE_ERROR = "130506" # AIDP response error # 06 - RAGFlow Service RAGFLOW_SERVICE_ERROR = "130507" # RAGFlow service error @@ -281,6 +283,9 @@ class ErrorCode(Enum): ErrorCode.AIDP_CONFIG_INVALID: 400, ErrorCode.AIDP_AUTH_ERROR: 502, ErrorCode.AIDP_CONNECTION_ERROR: 502, + ErrorCode.AIDP_RATE_LIMIT: 429, + ErrorCode.AIDP_RESPONSE_ERROR: 502, + ErrorCode.AIDP_SERVICE_ERROR: 502, # OAuth (module 16) ErrorCode.OAUTH_PROVIDER_NOT_CONFIGURED: 400, ErrorCode.OAUTH_PROVIDER_DISABLED: 400, diff --git a/backend/consts/error_message.py b/backend/consts/error_message.py index bb36416041..d8885a901b 100644 --- a/backend/consts/error_message.py +++ b/backend/consts/error_message.py @@ -133,6 +133,8 @@ class ErrorMessage: ErrorCode.AIDP_CONFIG_INVALID: "AIDP configuration invalid. Please check URL and API key format.", ErrorCode.AIDP_CONNECTION_ERROR: "Failed to connect to AIDP. Please check network connection and URL.", ErrorCode.AIDP_AUTH_ERROR: "AIDP authentication failed. Please check your API key.", + ErrorCode.AIDP_RATE_LIMIT: "AIDP API rate limit exceeded. Please try again later.", + ErrorCode.AIDP_RESPONSE_ERROR: "Failed to parse AIDP response. Please check API URL.", # ==================== 14 Northbound / 北向接口 ==================== ErrorCode.NORTHBOUND_REQUEST_FAILED: "Northbound request failed.", diff --git a/backend/consts/exceptions.py b/backend/consts/exceptions.py index 54fce6d82f..79215ee4a0 100644 --- a/backend/consts/exceptions.py +++ b/backend/consts/exceptions.py @@ -264,6 +264,25 @@ class SkillException(Exception): pass +class QuotaExceededError(Exception): + """Raised when tenant storage hard limit is exceeded during file upload.""" + + def __init__(self, message: str, usage_bytes: int = 0, hard_limit_bytes: int = 0, exceeded_by_bytes: int = 0): + super().__init__(message) + self.usage_bytes = usage_bytes + self.hard_limit_bytes = hard_limit_bytes + self.exceeded_by_bytes = exceeded_by_bytes + + +class PlatformQuotaConflictError(Exception): + """Raised when a platform or tenant quota update violates allocation rules.""" + + def __init__(self, message: str, error: str, details: dict): + super().__init__(message) + self.error = error + self.details = details + + class OAuthProviderError(Exception): """Raised when OAuth provider configuration is invalid or provider returns an error.""" diff --git a/backend/consts/model.py b/backend/consts/model.py index 0ec8ee4079..650537bd4d 100644 --- a/backend/consts/model.py +++ b/backend/consts/model.py @@ -7,6 +7,16 @@ from consts.prompt_template import PROMPT_GENERATE_TEMPLATE_FIELD_ALIAS_MAP +def _validated_context_policy(value: Optional[Dict[str, Any]]) -> Optional[Dict[str, Any]]: + """Validate a partial request/agent policy while preserving its layer shape.""" + if value is None: + return None + from nexent.core.agents.context import PolicyLayers, resolve_policy + + resolve_policy(PolicyLayers(request=value)) + return value + + class ModelConnectStatusEnum(Enum): """Enum class for model connection status""" NOT_DETECTED = "not_detected" @@ -32,9 +42,18 @@ class UserSignUpRequest(BaseModel): """User registration request model""" email: EmailStr password: str = Field(..., min_length=8) - invite_code: Optional[str] = None + invite_code: str = Field(..., min_length=1) auto_login: Optional[bool] = True # Whether to return session after signup + @field_validator("invite_code") + @classmethod + def validate_invite_code(cls, value: str) -> str: + """Reject empty or whitespace-only invitation codes.""" + normalized = value.strip() + if not normalized: + raise ValueError("Invitation code is required") + return normalized + class UserSignInRequest(BaseModel): """User login request model""" @@ -59,7 +78,7 @@ class UserUpdateRequest(BaseModel): """User update request model""" username: Optional[str] = Field(None, min_length=1, max_length=50) email: Optional[EmailStr] = None - role: Optional[str] = Field(None, pattern="^(SUPER_ADMIN|ADMIN|DEV|USER)$") + role: Optional[str] = Field(None, pattern="^(ADMIN|DEV|USER)$") class UserDeleteRequest(BaseModel): @@ -316,11 +335,37 @@ class AgentRequest(BaseModel): version_no: Optional[int] = None is_debug: Optional[bool] = False tool_params: Optional[ToolParamsRequest] = None + context_policy: Optional[Dict[str, Any]] = Field( + default=None, + description="Optional request-scoped context policy override", + ) + + @field_validator("context_policy") + @classmethod + def validate_context_policy(cls, value): + return _validated_context_policy(value) + enable_plan: Optional[bool] = Field( + default=False, + description="Whether to enable the planning phase before execution" + ) + enable_automation_tool: bool = Field( + default=True, + description="Whether the root interactive Agent may create scheduled-task proposals", + ) + + +class NL2AgentRunRequest(BaseModel): + """Request payload for one ephemeral NL2Agent turn.""" + + query: str = Field(min_length=1) + history: Optional[List[HistoryItem]] = None + minio_files: Optional[List[Dict[str, Any]]] = None class MessageUnit(BaseModel): type: str content: str + tool_call_id: Optional[str] = None class MessageRequest(BaseModel): @@ -346,7 +391,6 @@ class RenameRequest(BaseModel): conversation_id: int name: str - # Pydantic models for API class TaskRequest(BaseModel): source: str @@ -538,7 +582,21 @@ class GenerateTitleRequest(BaseModel): question: str +class AgentSkillInstanceRequest(BaseModel): + """Skill selection and per-agent configuration saved with an agent.""" + + skill_id: int + enabled: bool = True + config_values: Dict[str, Any] = Field(default_factory=dict) + + # used in agent/search agent/update for save agent info +class RelatedAgentInfo(BaseModel): + """Related agent info with pinned version.""" + agent_id: int + version_no: Optional[int] = None + + class AgentInfoRequest(BaseModel): agent_id: Optional[int] = None name: Optional[str] = None @@ -549,6 +607,7 @@ class AgentInfoRequest(BaseModel): model_ids: Optional[List[int]] = None max_steps: Optional[int] = Field(default=None, ge=1) requested_output_tokens: Optional[int] = Field(default=None, gt=0) + is_main_agent: Optional[bool] = None provide_run_summary: Optional[bool] = None duty_prompt: Optional[str] = None constraint_prompt: Optional[str] = None @@ -560,12 +619,16 @@ class AgentInfoRequest(BaseModel): prompt_template_name: Optional[str] = None enabled_tool_ids: Optional[List[int]] = None enabled_skill_ids: Optional[List[int]] = None + skill_instances: Optional[List[AgentSkillInstanceRequest]] = None related_agent_ids: Optional[List[int]] = None + related_agents: Optional[List[RelatedAgentInfo]] = None # Related agents with pinned versions related_external_agent_ids: Optional[List[int]] = None group_ids: Optional[List[int]] = None ingroup_permission: Optional[str] = None enable_context_manager: Optional[bool] = None verification_config: Optional[Dict[str, Any]] = None + context_policy: Optional[Dict[str, Any]] = None + greeting_message: Optional[str] = None example_questions: Optional[List[str]] = None version_no: int = 0 @@ -577,6 +640,11 @@ def normalize_verification_config(cls, value): return None return AgentVerificationConfig.model_validate(value).model_dump() + @field_validator("context_policy") + @classmethod + def validate_context_policy(cls, value): + return _validated_context_policy(value) + class AgentIDRequest(BaseModel): agent_id: int @@ -650,8 +718,10 @@ class ExportAndImportAgentInfo(BaseModel): author: Optional[str] = None max_steps: int requested_output_tokens: Optional[int] = Field(default=None, gt=0) + is_main_agent: bool = True provide_run_summary: bool verification_config: Optional[Dict[str, Any]] = None + context_policy: Optional[Dict[str, Any]] = None duty_prompt: Optional[str] = None constraint_prompt: Optional[str] = None few_shots_prompt: Optional[str] = None @@ -666,6 +736,11 @@ class ExportAndImportAgentInfo(BaseModel): prompt_template_id: Optional[int] = None prompt_template_name: Optional[str] = None + @field_validator("context_policy") + @classmethod + def validate_context_policy(cls, value): + return _validated_context_policy(value) + class Config: arbitrary_types_allowed = True @@ -720,6 +795,9 @@ class AgentRepositoryListingCreateRequest(BaseModel): tool_count: Optional[int] = Field( None, ge=0, description="Total tool count across all agents in the bundle" ) + content: Optional[str] = Field( + None, description="Listing note when submitting for review" + ) class AgentRepositoryListingDetailResponse(BaseModel): @@ -746,11 +824,18 @@ class SkillRepositoryListingCreateRequest(BaseModel): downloads: int = Field(0, ge=0, description="Initial download count for card display") tags: Optional[List[str]] = Field(None, description="Marketplace tags") category_id: Optional[int] = Field(0, description="Optional marketplace category ID") + content: Optional[str] = Field( + None, description="Listing note when submitting for review" + ) class SkillRepositoryInstallRequest(BaseModel): """Request body for installing a repository skill into current tenant.""" - target_name: Optional[str] = Field(None, description="Target skill name in current tenant") + target_name: Optional[str] = Field( + None, + max_length=100, + description="Target skill name in current tenant", + ) class SkillRepositoryListingDetailResponse(BaseModel): @@ -1108,6 +1193,14 @@ class ManageTenantModelCreateRequest(BaseModel): access_token: Optional[str] = Field(None, description="Access token for STT models (e.g., Volcano Engine)") timeout_seconds: Optional[int] = Field(None, description="Request timeout in seconds") concurrency_limit: Optional[int] = Field(None, description="Maximum concurrent requests for this model") + # W1 capacity fields (see W1 ADR). All nullable; resolver applies precedence. + context_window_tokens: Optional[int] = Field(None, description="Total combined input/output context window in tokens") + max_input_tokens: Optional[int] = Field(None, description="Provider hard input-token limit") + max_output_tokens: Optional[int] = Field(None, description="Provider-supported completion output cap") + default_output_reserve_tokens: Optional[int] = Field(None, description="Default output allowance reserved per request") + tokenizer_family: Optional[str] = Field(None, description="Token-counting strategy or tokenizer identifier") + capacity_source: Optional[str] = Field(None, description="Source of the persisted capacity value") + capability_profile_version: Optional[str] = Field(None, description="Version of the approved capability profile") # W11 accept-signal fields. Same audit-only contract as ModelRequest: # the app layer pops them off model_data before the dict reaches the # service/DB layer and forwards them to @@ -1139,6 +1232,14 @@ class ManageTenantModelUpdateRequest(BaseModel): access_token: Optional[str] = Field(None, description="Access token for STT models") timeout_seconds: Optional[int] = Field(None, description="Request timeout in seconds") concurrency_limit: Optional[int] = Field(None, description="Maximum concurrent requests for this model") + # W1 capacity fields (see W1 ADR). All nullable; resolver applies precedence. + context_window_tokens: Optional[int] = Field(None, description="Total combined input/output context window in tokens") + max_input_tokens: Optional[int] = Field(None, description="Provider hard input-token limit") + max_output_tokens: Optional[int] = Field(None, description="Provider-supported completion output cap") + default_output_reserve_tokens: Optional[int] = Field(None, description="Default output allowance reserved per request") + tokenizer_family: Optional[str] = Field(None, description="Token-counting strategy or tokenizer identifier") + capacity_source: Optional[str] = Field(None, description="Source of the persisted capacity value") + capability_profile_version: Optional[str] = Field(None, description="Version of the approved capability profile") # W11 accept-signal fields. See ManageTenantModelCreateRequest for the # contract. The app layer pops them before calling the service so # update_model_record never sees them. @@ -1156,6 +1257,10 @@ class ManageTenantModelHealthcheckRequest(BaseModel): """Request model for checking model connectivity in a specific tenant (admin/manage operation)""" tenant_id: str = Field(..., min_length=1, description="Target tenant ID to check model connectivity") display_name: str = Field(..., description="Display name of the model to check") + model_type: Optional[str] = Field( + None, + description="Model type to disambiguate models with the same display name", + ) class ManageBatchCreateModelsRequest(BaseModel): @@ -1275,6 +1380,8 @@ class SkillCreateRequest(BaseModel): tool_names: Optional[List[str]] = [] tags: Optional[List[str]] = [] source: Optional[str] = "custom" + group_ids: Optional[List[int]] = None + ingroup_permission: Optional[str] = None config_schemas: Optional[Dict[str, Any]] = None config_values: Optional[Dict[str, Any]] = None files: Optional[List[Dict[str, str]]] = Field( @@ -1300,6 +1407,8 @@ class SkillUpdateRequest(BaseModel): tool_names: Optional[List[str]] = None tags: Optional[List[str]] = None source: Optional[str] = None + group_ids: Optional[List[int]] = None + ingroup_permission: Optional[str] = None config_schemas: Optional[Dict[str, Any]] = None config_values: Optional[Dict[str, Any]] = None files: Optional[List[SkillFileData]] = Field( @@ -1318,6 +1427,8 @@ class SkillResponse(BaseModel): tool_ids: List[int] tags: List[str] source: str + group_ids: Optional[List[int]] = None + ingroup_permission: Optional[str] = None config_schemas: Optional[Dict[str, Any]] = None config_values: Optional[Dict[str, Any]] = None created_by: Optional[str] = None @@ -1355,10 +1466,15 @@ class AddMcpServiceRequest(BaseModel): authorization_token: Optional[str] = Field(None, description="Authorization token for MCP server") custom_headers: Optional[Dict[str, Any]] = Field(None, description="Custom HTTP headers as JSON object") container_config: Optional[Dict[str, Any]] = Field(None, description="Container configuration") + container_port: Optional[int] = Field(None, ge=1, le=65535, description="Container host port") registry_json: Optional[Dict[str, Any]] = Field(None, description="Registry metadata JSON") config_json: Optional[Dict[str, Any]] = Field(None, description="MCP configuration JSON (e.g. OpenAPI spec for API-type MCP)") market_id: Optional[int] = Field(None, gt=0, description="Linked market record ID") enabled: Optional[bool] = Field(default=False, description="Whether the MCP is enabled after creation") + group_ids: Optional[str] = Field(None, description="Comma-separated group IDs that can access this MCP") + ingroup_permission: Optional[str] = Field(None, description="Permission level: EDIT, READ_ONLY, PRIVATE") + shared_fields: Optional[Dict[str, Any]] = Field(None, description="JSON object of field-level sharing flags") + skip_health_check: Optional[bool] = Field(None, description="Skip MCP protocol health check (for community URL fallback)") @field_validator("name", "server_url", "description", "authorization_token", mode="before") @classmethod @@ -1379,6 +1495,9 @@ class AddContainerMcpServiceRequest(BaseModel): market_id: Optional[int] = Field(None, gt=0, description="Linked market record ID") port: int = Field(..., ge=1, le=65535, description="Host port for the container") mcp_config: MCPConfigRequest = Field(..., description="MCP server configuration") + group_ids: Optional[str] = Field(None, description="Comma-separated group IDs that can access this MCP") + ingroup_permission: Optional[str] = Field(None, description="Permission level: EDIT, READ_ONLY, PRIVATE") + shared_fields: Optional[Dict[str, Any]] = Field(None, description="JSON object of field-level sharing flags") @field_validator("name", "description", "authorization_token", mode="before") @classmethod @@ -1393,15 +1512,18 @@ class UpdateMcpServiceRequest(BaseModel): mcp_id: int = Field(..., gt=0, description="MCP record ID") name: str = Field(..., min_length=1, description="New MCP service name") description: Optional[str] = Field(None, description="MCP service description") - server_url: str = Field(..., min_length=1, description="New MCP server URL") + server_url: Optional[str] = Field(None, description="New MCP server URL") tags: List[str] = Field(default_factory=list, description="MCP tags") authorization_token: Optional[str] = Field(None, description="Authorization token for MCP server") custom_headers: Optional[Dict[str, Any]] = Field(None, description="Custom HTTP headers as JSON object") config_json: Optional[Dict[str, Any]] = Field(None, description="MCP configuration JSON") version: Optional[str] = Field(None, description="MCP version") market_id: Optional[int] = Field(None, gt=0, description="Linked market record ID") + group_ids: Optional[str] = Field(None, description="Comma-separated group IDs that can access this MCP") + ingroup_permission: Optional[str] = Field(None, description="Permission level: EDIT, READ_ONLY, PRIVATE") + shared_fields: Optional[Dict[str, Any]] = Field(None, description="JSON object of field-level sharing flags") - @field_validator("name", "server_url", "description", "authorization_token", "version", mode="before") + @field_validator("name", "description", "authorization_token", "version", mode="before") @classmethod def _strip_text(cls, value: Any): if isinstance(value, str): @@ -1505,6 +1627,15 @@ def _strip_status(cls, value: Any): class CommunityReviewActionRequest(BaseModel): """Request model for approving or rejecting an MCP community submission""" review_id: int = Field(..., gt=0, description="Review record ID") + content: Optional[str] = Field(None, description="Review opinion on approve/reject") + + @field_validator("content", mode="before") + @classmethod + def _strip_review_content(cls, value: Any): + if isinstance(value, str): + stripped = value.strip() + return stripped or None + return value class CommunityPublishRequest(BaseModel): @@ -1516,8 +1647,12 @@ class CommunityPublishRequest(BaseModel): tags: Optional[List[str]] = Field(None, description="Tags override") mcp_server: Optional[str] = Field(None, max_length=500, description="Remote MCP server URL override (URL / HTTP / SSE transports)") config_json: Optional[Dict[str, Any]] = Field(None, description="Container MCP configuration JSON override") + group_ids: Optional[List[int]] = Field(None, description="Group IDs that can access this MCP") + ingroup_permission: Optional[str] = Field(None, description="Permission level: EDIT, READ_ONLY, PRIVATE") + shared_fields: Optional[Dict[str, Any]] = Field(None, description="JSON object of field-level sharing flags") + content: Optional[str] = Field(None, description="Listing note on submit") - @field_validator("name", "description", "mcp_server", mode="before") + @field_validator("name", "description", "mcp_server", "content", mode="before") @classmethod def _strip_publish_optional_text(cls, value: Any): if isinstance(value, str): @@ -1539,8 +1674,12 @@ class CommunityUpdateRequest(BaseModel): None, description="Container MCP configuration JSON (omit to leave unchanged)", ) + group_ids: Optional[List[int]] = Field(None, description="Group IDs that can access this MCP") + ingroup_permission: Optional[str] = Field(None, description="Permission level: EDIT, READ_ONLY, PRIVATE") + shared_fields: Optional[Dict[str, Any]] = Field(None, description="JSON object of field-level sharing flags") + content: Optional[str] = Field(None, description="Listing note on resubmit") - @field_validator("name", "description", "mcp_server", "transport_type", mode="before") + @field_validator("name", "description", "mcp_server", "transport_type", "content", mode="before") @classmethod def _strip_text(cls, value: Any): if isinstance(value, str): @@ -1552,6 +1691,15 @@ def _strip_text(cls, value: Any): class CommunityStatusUpdateRequest(BaseModel): """Request model for changing MCP market listing status (PATCH).""" status: str = Field(..., description="New status: shared / rejected / not_shared / pending_review") + content: Optional[str] = Field(None, description="Review opinion or resubmit listing note") + + @field_validator("content", mode="before") + @classmethod + def _strip_status_content(cls, value: Any): + if isinstance(value, str): + stripped = value.strip() + return stripped or None + return value class DeleteMcpServiceRequest(BaseModel): diff --git a/backend/consts/notification.py b/backend/consts/notification.py new file mode 100644 index 0000000000..066e5974f7 --- /dev/null +++ b/backend/consts/notification.py @@ -0,0 +1,44 @@ +"""Domain constants for in-app notifications.""" + +# Resource types (drive frontend deep-link target) +RESOURCE_TYPE_AGENT_REPOSITORY = "agent_repository" +RESOURCE_TYPE_SKILL_REPOSITORY = "skill_repository" +RESOURCE_TYPE_MCP_REPOSITORY = "mcp_repository" + +VALID_RESOURCE_TYPES = frozenset({ + RESOURCE_TYPE_AGENT_REPOSITORY, + RESOURCE_TYPE_SKILL_REPOSITORY, + RESOURCE_TYPE_MCP_REPOSITORY, +}) + +# Event types (drive title copy / icon) +EVENT_TYPE_REPOSITORY_REVIEW_APPROVED = "repository_review_approved" +EVENT_TYPE_REPOSITORY_REVIEW_REJECTED = "repository_review_rejected" +EVENT_TYPE_REPOSITORY_REVIEW_PENDING = "repository_review_pending" + +VALID_EVENT_TYPES = frozenset({ + EVENT_TYPE_REPOSITORY_REVIEW_APPROVED, + EVENT_TYPE_REPOSITORY_REVIEW_REJECTED, + EVENT_TYPE_REPOSITORY_REVIEW_PENDING, +}) + +# Audience scope for a notification +SCOPE_SU = "SU" # all super admins (global, tenant-agnostic) +SCOPE_TENANT = "TENANT" # all users in a given tenant +SCOPE_TENANT_ADMIN = "TENANT_ADMIN" # admins of a given tenant +SCOPE_TENANT_USER = "TENANT_USER" # regular users of a given tenant +SCOPE_USER = "USER" # a specific single user + +VALID_NOTIFICATION_SCOPES = frozenset({ + SCOPE_SU, SCOPE_TENANT, SCOPE_TENANT_ADMIN, SCOPE_TENANT_USER, SCOPE_USER, +}) + +# Scopes that require a target tenant_id +TENANT_REQUIRED_SCOPES = frozenset({ + SCOPE_TENANT, SCOPE_TENANT_ADMIN, SCOPE_TENANT_USER, +}) + +# Role sets used to resolve receivers (user_tenant_t.user_role values) +SU_ROLES = frozenset({"SU", "SUPER_ADMIN"}) +TENANT_ADMIN_ROLES = frozenset({"ADMIN"}) +TENANT_USER_ROLES = frozenset({"USER"}) diff --git a/backend/consts/oauth_providers.py b/backend/consts/oauth_providers.py index 7429855b69..9fb2d65b3a 100644 --- a/backend/consts/oauth_providers.py +++ b/backend/consts/oauth_providers.py @@ -111,11 +111,36 @@ enabled_check="ENABLE_WECHAT_OAUTH", ) +HUAWEI_PROVIDER = OAuthProviderDefinition( + name="huawei", + display_name="Huawei", + icon="huawei", + authorize_url=f"{os.getenv('UNIPORTAL_URL')}/saaslogin1/oauth2/authorize", + authorize_params={"scope": "base.profile", "response_type": "code"}, + token_url=f"{os.getenv('UNIPORTAL_URL')}/saaslogin1/oauth2/accesstoken", + token_error_key="error", + token_error_message_key="error_description", + userinfo_url=f"{os.getenv('UNIPORTAL_URL')}/saaslogin1/oauth2/userinfo", + userinfo_params={ + "access_token": "{access_token}", + "client_id": f"{os.getenv('HUAWEI_OAUTH_CLIENT_ID')}", + "scope": "base.profile" + }, + userinfo_field_map={ + "id": "globalUserID", + "email": "email", + "username": "givenName", + }, + client_id_env="HUAWEI_OAUTH_CLIENT_ID", + client_secret_env="HUAWEI_OAUTH_CLIENT_SECRET", +) + OAUTH_PROVIDER_REGISTRY: Dict[str, OAuthProviderDefinition] = { "github": GITHUB_PROVIDER, "wechat": WECHAT_PROVIDER, "gde": GDE_PROVIDER, "link_app": LINK_APP_PROVIDER, + "huawei": HUAWEI_PROVIDER } diff --git a/backend/consts/tool_labels.py b/backend/consts/tool_labels.py index 649e635747..f8f2b6f4fc 100644 --- a/backend/consts/tool_labels.py +++ b/backend/consts/tool_labels.py @@ -13,7 +13,7 @@ - This module is the only hook that fires at the exact moment tools are inserted — the earliest lifecycle point where the data exists. -Keep in sync with: deploy/sql/migrations/v2.3.0_0624_add_labels_to_ag_tool_info.sql +Keep in sync with: deploy/sql/migrations/v2.3_merged_migrations.sql """ # tool_name → [label, ...] @@ -54,4 +54,5 @@ BUILTIN_LABEL_MAP.update(_category_memory) BUILTIN_LABEL_MAP.update(_category_terminal) -SYSTEM_MANAGED_TOOL_NAMES = frozenset({"store_memory", "search_memory"}) +PARALLEL_EXECUTOR_TOOL_NAME = "parallel_executor" +SYSTEM_MANAGED_TOOL_NAMES = frozenset({"store_memory", "search_memory", PARALLEL_EXECUTOR_TOOL_NAME}) diff --git a/backend/data_process/app.py b/backend/data_process/app.py index e70403c36d..aaccfd95ea 100644 --- a/backend/data_process/app.py +++ b/backend/data_process/app.py @@ -64,8 +64,8 @@ worker_prefetch_multiplier=1, # Fair scheduling; avoid batchy prefetch worker_max_tasks_per_child=1000, # Reduce restart frequency # Important for task chains - task_acks_late=True, # Tasks are acknowledged after completion - task_reject_on_worker_lost=True, # Tasks are rejected if worker is lost + task_acks_late=False, + task_reject_on_worker_lost=False, # Result storage settings result_expires=None, # Results never expire result_persistent=True, # Persist results to backend @@ -84,7 +84,7 @@ broker_connection_retry=True, broker_connection_retry_on_startup=True, broker_connection_max_retries=10, - broker_heartbeat=30, # Heartbeat check + broker_heartbeat=300, # Heartbeat check broker_pool_limit=10, # Connection pool size # Add transport options diff --git a/backend/data_process/tasks.py b/backend/data_process/tasks.py index 4dd6edd695..82d0ed7072 100644 --- a/backend/data_process/tasks.py +++ b/backend/data_process/tasks.py @@ -98,7 +98,7 @@ def _compute_split_wait_timeout(parts_count: int) -> int: waves = math.ceil(max(1, parts_count) / _estimate_parallel_parts()) dynamic_timeout = base_timeout + \ max(0, waves - 1) * max(1, PER_WAVE_TIMEOUT) - return min(MAX_TIMEOUT, max(base_timeout, dynamic_timeout)) + return min(MAX_TIMEOUT, dynamic_timeout) def _count_image_metadata_chunks(chunks: Optional[List[Dict[str, Any]]]) -> int: @@ -107,7 +107,9 @@ def _count_image_metadata_chunks(chunks: Optional[List[Dict[str, Any]]]) -> int: return sum( 1 for chunk in chunks - if isinstance(chunk, dict) and chunk.get("process_source") == IMAGE_METADATA_PROCESS_SOURCE + if isinstance(chunk, dict) + and (chunk.get("process_source") or chunk.get("metadata", {}).get("process_source")) + == IMAGE_METADATA_PROCESS_SOURCE ) @@ -338,6 +340,7 @@ def _delete_source_file_via_http_sync( index_name: str, path_or_url: str, scope: str, + authorization: Optional[str] = None, timeout_s: float = 30.0, ) -> Dict[str, Any]: base = (base_url or "").rstrip("/") @@ -345,8 +348,13 @@ def _delete_source_file_via_http_sync( raise RuntimeError("ELASTICSEARCH_SERVICE is not configured") url = f"{base}/indices/{index_name}/documents" params = {"path_or_url": path_or_url, "scope": scope} + headers: Dict[str, str] = {} + if authorization: + headers["Authorization"] = authorization - resp = requests.delete(url, params=params, timeout=timeout_s) + resp = requests.delete( + url, params=params, headers=headers, timeout=timeout_s + ) body_text = getattr(resp, "text", "") parsed = None try: @@ -1957,12 +1965,19 @@ def forward( name="data_process.tasks.cleanup_source", queue="forward_q", ) -def cleanup_source(self, forward_result: Dict[str, Any]) -> Dict[str, Any]: +def cleanup_source( + self, + forward_result: Dict[str, Any], + authorization: Optional[str] = None, +) -> Dict[str, Any]: """ Conditionally delete the MinIO source file after successful indexing. If the knowledge base is configured with preserve_source_file=false, call: DELETE /indices/{index_name}/documents?path_or_url=...&scope=source_only + + authorization is passed as a Celery signature kwarg (not via forward_result) + so tokens are not persisted in task result payloads. """ index_name = (forward_result or {}).get("index_name") source = (forward_result or {}).get("source") @@ -2010,6 +2025,7 @@ def cleanup_source(self, forward_result: Dict[str, Any]) -> Dict[str, Any]: index_name=index_name, path_or_url=source, scope="source_only", + authorization=authorization, ) cleanup_info["http_status"] = resp.get("http_status") cleanup_info["response"] = ( @@ -2083,7 +2099,7 @@ def submit_process_forward_chain( original_filename=original_filename, authorization=authorization ).set(queue='forward_q'), - cleanup_source.s().set(queue='forward_q'), + cleanup_source.s(authorization=authorization).set(queue='forward_q'), ) result = task_chain.apply_async() diff --git a/backend/data_process/utils.py b/backend/data_process/utils.py index f4ed5631c6..13dea244ca 100644 --- a/backend/data_process/utils.py +++ b/backend/data_process/utils.py @@ -15,6 +15,27 @@ logger = logging.getLogger("data_process.utils") +def _parse_failure_info(info: Any) -> tuple[Optional[Dict[str, Any]], Optional[str]]: + """Parse Celery failure metadata as structured JSON or plain error text.""" + if isinstance(info, dict): + return info, None + if info is None: + return None, None + + info_text = str(info).strip() + if not info_text: + return None, None + + try: + parsed_info = json.loads(info_text) + except (json.JSONDecodeError, TypeError): + return None, info_text + + if isinstance(parsed_info, dict): + return parsed_info, None + return None, info_text + + def get_all_task_ids_from_redis(redis_client: redis.Redis) -> List[str]: """ Get all task IDs from Redis backend @@ -25,10 +46,8 @@ def get_all_task_ids_from_redis(redis_client: redis.Redis) -> List[str]: task_ids = [] try: # Get all keys matching Celery result pattern - result_keys = redis_client.keys('celery-task-meta-*') - - # Extract task IDs from keys - for key in result_keys: + for key in redis_client.scan_iter( + match='celery-task-meta-*', count=500): if isinstance(key, bytes): key = key.decode('utf-8') @@ -145,18 +164,10 @@ def sync_get(): # Add error information for failed tasks if result.failed(): try: - info = str(result.info) - error_json = None - if isinstance(info, str): - try: - error_json = json.loads(info) - except Exception as e: - logger.error( - f"Failed to load result.info as a json: {str(e)}") - error_json = None - else: - logger.warning( - f"Cannot parse result.info into a string: {type(result.info)}") + error_json, plain_error = _parse_failure_info( + result.info) + if plain_error: + status_info['error'] = plain_error if error_json: if error_json.get('message') is not None: @@ -174,7 +185,7 @@ def sync_get(): if error_json.get('original_filename') is not None: status_info['original_filename'] = error_json.get( 'original_filename') - else: + elif not status_info['error']: # fallback: compatible with previous format status_info['error'] = str( result.result) if result.result else "Unknown error" diff --git a/backend/data_process_service.py b/backend/data_process_service.py index 23d3497d98..c162cfaaa8 100644 --- a/backend/data_process_service.py +++ b/backend/data_process_service.py @@ -72,6 +72,9 @@ def _check_redis_connection(self, redis_url: str) -> bool: import redis redis_client = redis.from_url(redis_url, socket_timeout=5, socket_connect_timeout=5) redis_client.ping() + from services.redis_service import get_redis_service + cleanup_stats = get_redis_service().cleanup_error_info_keys() + logger.info(f"Redis error info cleanup stats: {cleanup_stats}") logger.info(f"✅ Redis connection successful: {redis_url}") return True except ImportError: diff --git a/backend/database/a2a_agent_db.py b/backend/database/a2a_agent_db.py index c1d9982721..982be32cd2 100644 --- a/backend/database/a2a_agent_db.py +++ b/backend/database/a2a_agent_db.py @@ -145,6 +145,11 @@ def _extract_protocol_type(supported_interfaces: Optional[List[Dict[str, Any]]]) return PROTOCOL_JSONRPC +def _configured_security_scheme_ids(security_credentials: Optional[Dict[str, str]]) -> List[str]: + """Return credential scheme IDs without exposing their secret values.""" + return sorted(scheme_id for scheme_id, value in (security_credentials or {}).items() if value) + + def create_external_agent_from_url( source_url: str, name: str, @@ -157,6 +162,9 @@ def create_external_agent_from_url( streaming: bool = False, supported_interfaces: Optional[List[Dict[str, Any]]] = None, base_url: Optional[str] = None, + agent_card_headers: Optional[Dict[str, str]] = None, + security_schemes: Optional[Dict[str, Any]] = None, + security_requirements: Optional[List[Dict[str, Any]]] = None, ) -> Dict[str, Any]: """Create or update an external A2A agent discovered from URL. @@ -172,6 +180,9 @@ def create_external_agent_from_url( streaming: Whether this agent supports SSE streaming. supported_interfaces: All supported protocol interfaces. base_url: Base URL for health checks (service root address). + agent_card_headers: Headers saved only for Agent Card discovery and refresh. + security_schemes: Security schemes declared by the Agent Card. + security_requirements: Security requirements declared by the Agent Card. Returns: Created agent information dict. @@ -202,6 +213,10 @@ def create_external_agent_from_url( existing.streaming = streaming existing.supported_interfaces = supported_interfaces existing.raw_card = raw_card + existing.security_schemes = security_schemes + existing.security_requirements = security_requirements + if agent_card_headers is not None: + existing.agent_card_headers = agent_card_headers existing.cached_at = now existing.cache_expires_at = expires_at existing.updated_by = user_id @@ -220,6 +235,11 @@ def create_external_agent_from_url( supported_interfaces=supported_interfaces, source_type="url", source_url=source_url, + agent_card_headers=agent_card_headers, + security_schemes=security_schemes, + security_requirements=security_requirements, + security_credentials=None, + selected_security_requirement_index=None, tenant_id=tenant_id, created_by=user_id, updated_by=user_id, @@ -242,6 +262,10 @@ def create_external_agent_from_url( "protocol_type": agent.protocol_type, "streaming": agent.streaming, "supported_interfaces": agent.supported_interfaces, + "security_schemes": agent.security_schemes, + "security_requirements": agent.security_requirements, + "configured_security_scheme_ids": _configured_security_scheme_ids(agent.security_credentials), + "selected_security_requirement_index": agent.selected_security_requirement_index, "source_type": agent.source_type, "base_url": agent.base_url, "is_available": agent.is_available, @@ -263,6 +287,8 @@ def create_external_agent_from_nacos( streaming: bool = False, supported_interfaces: Optional[List[Dict[str, Any]]] = None, base_url: Optional[str] = None, + security_schemes: Optional[Dict[str, Any]] = None, + security_requirements: Optional[List[Dict[str, Any]]] = None, ) -> Dict[str, Any]: """Create or update an external A2A agent discovered from Nacos. @@ -279,6 +305,8 @@ def create_external_agent_from_nacos( streaming: Whether this agent supports SSE streaming. supported_interfaces: All supported protocol interfaces. base_url: Base URL for health checks (service root address). + security_schemes: Security schemes declared by the Agent Card. + security_requirements: Security requirements declared by the Agent Card. Returns: Created agent information dict. @@ -309,6 +337,8 @@ def create_external_agent_from_nacos( existing.streaming = streaming existing.supported_interfaces = supported_interfaces existing.raw_card = raw_card + existing.security_schemes = security_schemes + existing.security_requirements = security_requirements existing.cached_at = now existing.cache_expires_at = expires_at existing.updated_by = user_id @@ -327,6 +357,10 @@ def create_external_agent_from_nacos( source_type="nacos", nacos_config_id=nacos_config_id, nacos_agent_name=nacos_agent_name, + security_schemes=security_schemes, + security_requirements=security_requirements, + security_credentials=None, + selected_security_requirement_index=None, tenant_id=tenant_id, created_by=user_id, updated_by=user_id, @@ -349,6 +383,10 @@ def create_external_agent_from_nacos( "protocol_type": agent.protocol_type, "streaming": agent.streaming, "supported_interfaces": agent.supported_interfaces, + "security_schemes": agent.security_schemes, + "security_requirements": agent.security_requirements, + "configured_security_scheme_ids": _configured_security_scheme_ids(agent.security_credentials), + "selected_security_requirement_index": agent.selected_security_requirement_index, "source_type": agent.source_type, "base_url": agent.base_url, "is_available": agent.is_available, @@ -357,12 +395,19 @@ def create_external_agent_from_nacos( } -def get_external_agent_by_id(external_agent_id: int, tenant_id: str) -> Optional[Dict[str, Any]]: +def get_external_agent_by_id( + external_agent_id: int, + tenant_id: str, + include_agent_card_headers: bool = False, + include_security_credentials: bool = False, +) -> Optional[Dict[str, Any]]: """Get an external agent by its id. Args: external_agent_id: The external agent database ID. tenant_id: Tenant ID for isolation. + include_agent_card_headers: Include headers for internal Agent Card refresh only. + include_security_credentials: Include credentials for internal agent calls only. Returns: Agent information dict or None if not found. @@ -377,7 +422,7 @@ def get_external_agent_by_id(external_agent_id: int, tenant_id: str) -> Optional if not agent: return None - return { + result = { "id": agent.id, "name": agent.name, "description": agent.description, @@ -386,6 +431,10 @@ def get_external_agent_by_id(external_agent_id: int, tenant_id: str) -> Optional "streaming": agent.streaming, "protocol_type": agent.protocol_type, "supported_interfaces": agent.supported_interfaces, + "security_schemes": agent.security_schemes, + "security_requirements": agent.security_requirements, + "configured_security_scheme_ids": _configured_security_scheme_ids(agent.security_credentials), + "selected_security_requirement_index": agent.selected_security_requirement_index, "source_type": agent.source_type, "source_url": agent.source_url, "base_url": agent.base_url, @@ -399,6 +448,11 @@ def get_external_agent_by_id(external_agent_id: int, tenant_id: str) -> Optional "cache_expires_at": agent.cache_expires_at.isoformat() if agent.cache_expires_at else None, "create_time": agent.create_time.isoformat() if agent.create_time else None, } + if include_agent_card_headers: + result["agent_card_headers"] = agent.agent_card_headers + if include_security_credentials: + result["security_credentials"] = agent.security_credentials + return result def list_external_agents( @@ -442,9 +496,13 @@ def list_external_agents( "version": agent.version, "agent_url": agent.agent_url, "streaming": agent.streaming, - "protocol_type": agent.protocol_type, - "supported_interfaces": agent.supported_interfaces, - "source_type": agent.source_type, + "protocol_type": agent.protocol_type, + "supported_interfaces": agent.supported_interfaces, + "security_schemes": agent.security_schemes, + "security_requirements": agent.security_requirements, + "configured_security_scheme_ids": _configured_security_scheme_ids(agent.security_credentials), + "selected_security_requirement_index": agent.selected_security_requirement_index, + "source_type": agent.source_type, "source_url": agent.source_url, "base_url": agent.base_url, "is_available": agent.is_available, @@ -455,6 +513,33 @@ def list_external_agents( ] +def update_external_agent_security_credentials( + external_agent_id: int, + tenant_id: str, + user_id: str, + security_credentials: Dict[str, str], + selected_security_requirement_index: Optional[int] = None, +) -> bool: + """Save configured security credential values for an external agent.""" + with _get_db_session() as session: + agent = session.query(A2AExternalAgent).filter( + A2AExternalAgent.id == external_agent_id, + A2AExternalAgent.tenant_id == tenant_id, + A2AExternalAgent.delete_flag != 'Y' + ).first() + if not agent: + return False + + existing_credentials = dict(agent.security_credentials or {}) + existing_credentials.update(security_credentials) + agent.security_credentials = existing_credentials + if selected_security_requirement_index is not None: + agent.selected_security_requirement_index = selected_security_requirement_index + agent.updated_by = user_id + session.flush() + return True + + def delete_external_agent(external_agent_id: int, tenant_id: str) -> bool: """Soft delete an external agent. @@ -569,6 +654,8 @@ def update_external_agent_protocol( "protocol_type": agent.protocol_type, "streaming": agent.streaming, "supported_interfaces": agent.supported_interfaces, + "security_schemes": agent.security_schemes, + "security_requirements": agent.security_requirements, "source_type": agent.source_type, "source_url": agent.source_url, "nacos_config_id": agent.nacos_config_id, @@ -596,6 +683,8 @@ def refresh_external_agent_cache( new_streaming: Optional[bool] = None, new_supported_interfaces: Optional[List[Dict[str, Any]]] = None, new_protocol_type: Optional[str] = None, + new_security_schemes: Optional[Dict[str, Any]] = None, + new_security_requirements: Optional[List[Dict[str, Any]]] = None, ) -> Optional[Dict[str, Any]]: """Refresh the cache for an external agent. @@ -611,6 +700,8 @@ def refresh_external_agent_cache( new_streaming: Updated streaming capability. new_supported_interfaces: Updated supported interfaces. new_protocol_type: Updated protocol type (JSONRPC, HTTP+JSON, or GRPC). + new_security_schemes: Updated Agent Card security schemes. + new_security_requirements: Updated Agent Card security requirements. Returns: Updated agent information dict or None if not found. @@ -630,8 +721,6 @@ def refresh_external_agent_cache( if new_raw_card is not None: agent.raw_card = new_raw_card - if new_agent_url is not None: - agent.agent_url = new_agent_url if new_name is not None: agent.name = new_name if new_description is not None: @@ -640,17 +729,30 @@ def refresh_external_agent_cache( agent.version = new_version if new_streaming is not None: agent.streaming = new_streaming + selected_protocol_type = new_protocol_type or agent.protocol_type if new_supported_interfaces is not None: agent.supported_interfaces = new_supported_interfaces - if new_protocol_type is not None: - agent.protocol_type = new_protocol_type - # Update agent_url based on the selected protocol type interface = _find_interface_by_protocol_type( - agent.supported_interfaces, - new_protocol_type + new_supported_interfaces, + selected_protocol_type, ) if interface: agent.agent_url = interface.get("url", agent.agent_url) + elif new_agent_url is not None: + agent.agent_url = new_agent_url + if new_security_schemes is not None: + agent.security_schemes = new_security_schemes + if new_security_requirements is not None: + agent.security_requirements = new_security_requirements + if new_protocol_type is not None: + agent.protocol_type = new_protocol_type + if new_supported_interfaces is None: + interface = _find_interface_by_protocol_type( + agent.supported_interfaces, + new_protocol_type, + ) + if interface: + agent.agent_url = interface.get("url", agent.agent_url) agent.cached_at = now agent.cache_expires_at = expires_at diff --git a/backend/database/agent_automation_db.py b/backend/database/agent_automation_db.py new file mode 100644 index 0000000000..9ba300055a --- /dev/null +++ b/backend/database/agent_automation_db.py @@ -0,0 +1,658 @@ +from datetime import datetime, timezone +from typing import Any, Dict, List, Optional + +from sqlalchemy import desc, func, insert, select, text, update + +from .client import as_dict, get_db_session +from .db_models import AgentAutomationProposal, AgentAutomationRun, AgentAutomationTask +from .utils import add_creation_tracking, add_update_tracking + + +def _utcnow() -> datetime: + return datetime.now(timezone.utc) + + +def create_task(task_data: Dict[str, Any], user_id: str) -> Dict[str, Any]: + data = { + **task_data, + "delete_flag": "N", + } + data = add_creation_tracking(data, user_id) + with get_db_session() as session: + stmt = insert(AgentAutomationTask).values(**data).returning(AgentAutomationTask) + task = session.execute(stmt).scalar_one() + return as_dict(task) + + +def get_task(task_id: int, tenant_id: str, user_id: str) -> Optional[Dict[str, Any]]: + with get_db_session() as session: + task = session.execute( + select(AgentAutomationTask).where( + AgentAutomationTask.task_id == task_id, + AgentAutomationTask.tenant_id == tenant_id, + AgentAutomationTask.user_id == user_id, + AgentAutomationTask.delete_flag == "N", + ) + ).scalar_one_or_none() + return as_dict(task) if task else None + + +def get_task_by_conversation( + conversation_id: int, + user_id: str, + include_deleted: bool = False, +) -> Optional[Dict[str, Any]]: + with get_db_session() as session: + conditions = [ + AgentAutomationTask.conversation_id == conversation_id, + AgentAutomationTask.user_id == user_id, + ] + if not include_deleted: + conditions.extend([ + AgentAutomationTask.delete_flag == "N", + AgentAutomationTask.status != "DELETED", + ]) + task = session.execute(select(AgentAutomationTask).where(*conditions)).scalar_one_or_none() + return as_dict(task) if task else None + + +def _escape_like_pattern(value: str) -> str: + return value.replace("\\", "\\\\").replace("%", "\\%").replace("_", "\\_") + + +def _task_list_conditions( + tenant_id: str, + user_id: str, + status: Optional[str] = None, + search: Optional[str] = None, +): + conditions = [ + AgentAutomationTask.tenant_id == tenant_id, + AgentAutomationTask.user_id == user_id, + AgentAutomationTask.delete_flag == "N", + AgentAutomationTask.status != "DELETED", + ] + if status: + conditions.append(AgentAutomationTask.status == status) + normalized_search = search.strip() if search else "" + if normalized_search: + pattern = f"%{_escape_like_pattern(normalized_search)}%" + conditions.append(AgentAutomationTask.title.ilike(pattern, escape="\\")) + return conditions + + +def list_tasks( + tenant_id: str, + user_id: str, + status: Optional[str] = None, + search: Optional[str] = None, +) -> List[Dict[str, Any]]: + with get_db_session() as session: + conditions = _task_list_conditions(tenant_id, user_id, status, search) + rows = session.execute( + select(AgentAutomationTask) + .where(*conditions) + .order_by(desc(AgentAutomationTask.update_time)) + ).scalars().all() + return [as_dict(row) for row in rows] + + +def list_tasks_paginated( + tenant_id: str, + user_id: str, + status: Optional[str], + search: Optional[str], + page: int, + page_size: int, +) -> Dict[str, Any]: + with get_db_session() as session: + conditions = _task_list_conditions(tenant_id, user_id, status, search) + total = session.execute( + select(func.count()).select_from(AgentAutomationTask).where(*conditions) + ).scalar_one() + rows = session.execute( + select(AgentAutomationTask) + .where(*conditions) + .order_by(desc(AgentAutomationTask.update_time)) + .offset((page - 1) * page_size) + .limit(page_size) + ).scalars().all() + return { + "items": [as_dict(row) for row in rows], + "total": int(total or 0), + "page": page, + "page_size": page_size, + } + + +def update_task(task_id: int, tenant_id: str, user_id: str, values: Dict[str, Any]) -> Optional[Dict[str, Any]]: + data = add_update_tracking({ + **values, + "update_time": _utcnow(), + }, user_id) + with get_db_session() as session: + task = session.execute( + update(AgentAutomationTask) + .where( + AgentAutomationTask.task_id == task_id, + AgentAutomationTask.tenant_id == tenant_id, + AgentAutomationTask.user_id == user_id, + AgentAutomationTask.delete_flag == "N", + ) + .values(**data) + .returning(AgentAutomationTask) + ).scalar_one_or_none() + return as_dict(task) if task else None + + +def update_task_if_lock_owner( + task_id: int, + tenant_id: str, + user_id: str, + lock_owner: str, + values: Dict[str, Any], +) -> Optional[Dict[str, Any]]: + """Apply a scheduled-run result only while the caller owns the task lease.""" + data = add_update_tracking({ + **values, + "update_time": _utcnow(), + }, user_id) + with get_db_session() as session: + task = session.execute( + update(AgentAutomationTask) + .where( + AgentAutomationTask.task_id == task_id, + AgentAutomationTask.tenant_id == tenant_id, + AgentAutomationTask.user_id == user_id, + AgentAutomationTask.lock_owner == lock_owner, + AgentAutomationTask.lock_until > func.now(), + AgentAutomationTask.delete_flag == "N", + ) + .values(**data) + .returning(AgentAutomationTask) + ).scalar_one_or_none() + return as_dict(task) if task else None + + +def soft_delete_task(task_id: int, tenant_id: str, user_id: str) -> bool: + result = update_task(task_id, tenant_id, user_id, { + "status": "DELETED", + "delete_flag": "Y", + "lock_owner": None, + "lock_until": None, + }) + return result is not None + + +def soft_delete_task_by_conversation(conversation_id: int, user_id: str) -> int: + with get_db_session() as session: + result = session.execute( + update(AgentAutomationTask) + .where( + AgentAutomationTask.conversation_id == conversation_id, + AgentAutomationTask.user_id == user_id, + AgentAutomationTask.delete_flag == "N", + ) + .values( + status="DELETED", + delete_flag="Y", + lock_owner=None, + lock_until=None, + update_time=_utcnow(), + updated_by=user_id, + ) + ) + return result.rowcount or 0 + + +def create_proposal(proposal_data: Dict[str, Any], user_id: str) -> Dict[str, Any]: + data = add_creation_tracking({**proposal_data, "delete_flag": "N"}, user_id) + with get_db_session() as session: + stmt = insert(AgentAutomationProposal).values(**data).returning(AgentAutomationProposal) + proposal = session.execute(stmt).scalar_one() + return as_dict(proposal) + + +def get_proposal(proposal_id: int, tenant_id: str, user_id: str) -> Optional[Dict[str, Any]]: + with get_db_session() as session: + proposal = session.execute( + select(AgentAutomationProposal).where( + AgentAutomationProposal.proposal_id == proposal_id, + AgentAutomationProposal.tenant_id == tenant_id, + AgentAutomationProposal.user_id == user_id, + AgentAutomationProposal.delete_flag == "N", + ) + ).scalar_one_or_none() + return as_dict(proposal) if proposal else None + + +def get_proposal_by_source_message( + source_message_id: int, + tenant_id: str, + user_id: str, +) -> Optional[Dict[str, Any]]: + with get_db_session() as session: + proposal = session.execute( + select(AgentAutomationProposal).where( + AgentAutomationProposal.source_message_id == source_message_id, + AgentAutomationProposal.tenant_id == tenant_id, + AgentAutomationProposal.user_id == user_id, + AgentAutomationProposal.delete_flag == "N", + ) + ).scalar_one_or_none() + return as_dict(proposal) if proposal else None + + +def update_proposal_status(proposal_id: int, tenant_id: str, user_id: str, status: str) -> bool: + with get_db_session() as session: + result = session.execute( + update(AgentAutomationProposal) + .where( + AgentAutomationProposal.proposal_id == proposal_id, + AgentAutomationProposal.tenant_id == tenant_id, + AgentAutomationProposal.user_id == user_id, + AgentAutomationProposal.delete_flag == "N", + ) + .values(status=status, update_time=_utcnow(), updated_by=user_id) + ) + return bool(result.rowcount) + + +def update_proposal_task( + proposal_id: int, + tenant_id: str, + user_id: str, + proposed_task: Dict[str, Any], +) -> bool: + with get_db_session() as session: + result = session.execute( + update(AgentAutomationProposal) + .where( + AgentAutomationProposal.proposal_id == proposal_id, + AgentAutomationProposal.tenant_id == tenant_id, + AgentAutomationProposal.user_id == user_id, + AgentAutomationProposal.delete_flag == "N", + ) + .values( + proposed_task=proposed_task, + update_time=_utcnow(), + updated_by=user_id, + ) + ) + return bool(result.rowcount) + + +def link_proposal_message_unit( + proposal_id: int, + tenant_id: str, + user_id: str, + message_id: int, + unit_id: int, +) -> bool: + """Link a proposal to the assistant unit that rendered its confirmation card.""" + with get_db_session() as session: + proposal = session.execute( + select(AgentAutomationProposal).where( + AgentAutomationProposal.proposal_id == proposal_id, + AgentAutomationProposal.tenant_id == tenant_id, + AgentAutomationProposal.user_id == user_id, + AgentAutomationProposal.delete_flag == "N", + ) + ).scalar_one_or_none() + if proposal is None: + return False + proposed_task = dict(proposal.proposed_task or {}) + proposed_task["_conversation_message_id"] = message_id + proposed_task["_conversation_unit_id"] = unit_id + proposal.proposed_task = proposed_task + proposal.update_time = _utcnow() + proposal.updated_by = user_id + return True + + +def update_proposal( + proposal_id: int, + tenant_id: str, + user_id: str, + proposed_task: Dict[str, Any], + capability_resolution: Dict[str, Any], +) -> bool: + with get_db_session() as session: + result = session.execute( + update(AgentAutomationProposal) + .where( + AgentAutomationProposal.proposal_id == proposal_id, + AgentAutomationProposal.tenant_id == tenant_id, + AgentAutomationProposal.user_id == user_id, + AgentAutomationProposal.status.in_(["PENDING", "ACCEPTED"]), + AgentAutomationProposal.delete_flag == "N", + ) + .values( + proposed_task=proposed_task, + capability_resolution=capability_resolution, + update_time=_utcnow(), + updated_by=user_id, + ) + ) + return bool(result.rowcount) + + +def create_run(run_data: Dict[str, Any], user_id: str) -> Dict[str, Any]: + data = add_creation_tracking({**run_data, "delete_flag": "N"}, user_id) + with get_db_session() as session: + stmt = insert(AgentAutomationRun).values(**data).returning(AgentAutomationRun) + run = session.execute(stmt).scalar_one() + return as_dict(run) + + +def update_run( + run_id: int, + values: Dict[str, Any], + user_id: Optional[str] = None, + expected_statuses: Optional[List[str]] = None, +) -> Optional[Dict[str, Any]]: + data = { + **values, + "update_time": _utcnow(), + } + if user_id: + data = add_update_tracking(data, user_id) + with get_db_session() as session: + conditions = [ + AgentAutomationRun.run_id == run_id, + AgentAutomationRun.delete_flag == "N", + ] + if expected_statuses: + conditions.append(AgentAutomationRun.status.in_(expected_statuses)) + run = session.execute( + update(AgentAutomationRun) + .where(*conditions) + .values(**data) + .returning(AgentAutomationRun) + ).scalar_one_or_none() + return as_dict(run) if run else None + + +def get_run(run_id: int, tenant_id: str, user_id: str) -> Optional[Dict[str, Any]]: + with get_db_session() as session: + run = session.execute( + select(AgentAutomationRun).where( + AgentAutomationRun.run_id == run_id, + AgentAutomationRun.tenant_id == tenant_id, + AgentAutomationRun.user_id == user_id, + AgentAutomationRun.delete_flag == "N", + ) + ).scalar_one_or_none() + return as_dict(run) if run else None + + +def cancel_run(run_id: int, tenant_id: str, user_id: str, reason: str) -> Optional[Dict[str, Any]]: + with get_db_session() as session: + run = session.execute( + update(AgentAutomationRun) + .where( + AgentAutomationRun.run_id == run_id, + AgentAutomationRun.tenant_id == tenant_id, + AgentAutomationRun.user_id == user_id, + AgentAutomationRun.status.in_(["QUEUED", "RUNNING"]), + AgentAutomationRun.delete_flag == "N", + ) + .values( + status="CANCELED", + error_code="AUTOMATION_RUN_CANCELED", + error_message=reason, + finished_at=_utcnow(), + update_time=_utcnow(), + updated_by=user_id, + ) + .returning(AgentAutomationRun) + ).scalar_one_or_none() + return as_dict(run) if run else None + + +def soft_delete_run( + run_id: int, + tenant_id: str, + user_id: str, + expected_statuses: List[str], +) -> Optional[Dict[str, Any]]: + with get_db_session() as session: + run = session.execute( + update(AgentAutomationRun) + .where( + AgentAutomationRun.run_id == run_id, + AgentAutomationRun.tenant_id == tenant_id, + AgentAutomationRun.user_id == user_id, + AgentAutomationRun.status.in_(expected_statuses), + AgentAutomationRun.delete_flag == "N", + ) + .values( + delete_flag="Y", + update_time=_utcnow(), + updated_by=user_id, + ) + .returning(AgentAutomationRun) + ).scalar_one_or_none() + return as_dict(run) if run else None + + +def cancel_runs_by_conversation(conversation_id: int, user_id: str, reason: str) -> int: + with get_db_session() as session: + result = session.execute( + update(AgentAutomationRun) + .where( + AgentAutomationRun.conversation_id == conversation_id, + AgentAutomationRun.user_id == user_id, + AgentAutomationRun.status.in_(["QUEUED", "RUNNING"]), + AgentAutomationRun.delete_flag == "N", + ) + .values( + status="CANCELED", + error_code="AUTOMATION_RUN_CANCELED", + error_message=reason, + finished_at=_utcnow(), + update_time=_utcnow(), + updated_by=user_id, + ) + ) + return result.rowcount or 0 + + +def list_runs(task_id: int, tenant_id: str, user_id: str, limit: int = 50) -> List[Dict[str, Any]]: + with get_db_session() as session: + rows = session.execute( + select(AgentAutomationRun) + .where( + AgentAutomationRun.task_id == task_id, + AgentAutomationRun.tenant_id == tenant_id, + AgentAutomationRun.user_id == user_id, + AgentAutomationRun.delete_flag == "N", + ) + .order_by(desc(AgentAutomationRun.scheduled_fire_at)) + .limit(limit) + ).scalars().all() + return [as_dict(row) for row in rows] + + +def list_runs_paginated( + task_id: int, + tenant_id: str, + user_id: str, + page: int, + page_size: int, +) -> Dict[str, Any]: + conditions = [ + AgentAutomationRun.task_id == task_id, + AgentAutomationRun.tenant_id == tenant_id, + AgentAutomationRun.user_id == user_id, + AgentAutomationRun.delete_flag == "N", + ] + with get_db_session() as session: + total = session.execute( + select(func.count()).select_from(AgentAutomationRun).where(*conditions) + ).scalar_one() + rows = session.execute( + select(AgentAutomationRun) + .where(*conditions) + .order_by(desc(AgentAutomationRun.scheduled_fire_at)) + .offset((page - 1) * page_size) + .limit(page_size) + ).scalars().all() + return { + "items": [as_dict(row) for row in rows], + "total": int(total or 0), + "page": page, + "page_size": page_size, + } + + +def get_active_run_task_ids(task_ids: List[int], tenant_id: str, user_id: str) -> set[int]: + """Return task IDs that currently have a queued or running execution.""" + if not task_ids: + return set() + with get_db_session() as session: + rows = session.execute( + select(AgentAutomationRun.task_id) + .where( + AgentAutomationRun.task_id.in_(task_ids), + AgentAutomationRun.tenant_id == tenant_id, + AgentAutomationRun.user_id == user_id, + AgentAutomationRun.status.in_(["QUEUED", "RUNNING"]), + AgentAutomationRun.delete_flag == "N", + ) + .distinct() + ).scalars().all() + return {int(task_id) for task_id in rows} + + +def has_active_run_for_conversation(conversation_id: int) -> bool: + with get_db_session() as session: + run = session.execute( + select(AgentAutomationRun.run_id) + .where( + AgentAutomationRun.conversation_id == conversation_id, + AgentAutomationRun.status.in_(["QUEUED", "RUNNING"]), + AgentAutomationRun.delete_flag == "N", + ) + .limit(1) + ).scalar_one_or_none() + return run is not None + + +def claim_due_tasks(instance_id: str, batch_size: int, lease_seconds: float) -> List[Dict[str, Any]]: + sql = text(""" + WITH due AS ( + SELECT task_id + FROM nexent.agent_automation_task_t + WHERE delete_flag = 'N' + AND status = 'ACTIVE' + AND next_fire_at <= now() + AND (lock_until IS NULL OR lock_until < now()) + ORDER BY next_fire_at ASC + LIMIT :batch_size + FOR UPDATE SKIP LOCKED + ), claimed AS ( + UPDATE nexent.agent_automation_task_t AS task + SET lock_owner = :instance_id, + lock_until = now() + (:lease_seconds * interval '1 second'), + update_time = now() + FROM due + WHERE task.task_id = due.task_id + RETURNING task.* + ), orphaned_runs AS ( + UPDATE nexent.agent_automation_run_t AS run + SET status = 'TIMEOUT', + error_code = 'AUTOMATION_LEASE_EXPIRED', + error_message = 'The previous scheduler lease expired before the run completed.', + finished_at = now(), + update_time = now() + WHERE run.task_id IN (SELECT task_id FROM claimed) + AND run.trigger_type = 'SCHEDULED' + AND run.status IN ('QUEUED', 'RUNNING') + AND run.delete_flag = 'N' + RETURNING run.run_id + ) + SELECT claimed.* + FROM claimed + WHERE (SELECT count(*) FROM orphaned_runs) >= 0 + """) + with get_db_session() as session: + rows = session.execute(sql, { + "instance_id": instance_id, + "batch_size": batch_size, + "lease_seconds": lease_seconds, + }).fetchall() + return [dict(row._mapping) for row in rows] + + +def release_task_lock(task_id: int, lock_owner: Optional[str] = None) -> bool: + conditions = [AgentAutomationTask.task_id == task_id] + if lock_owner: + conditions.append(AgentAutomationTask.lock_owner == lock_owner) + with get_db_session() as session: + result = session.execute( + update(AgentAutomationTask) + .where(*conditions) + .values(lock_owner=None, lock_until=None, update_time=_utcnow()) + ) + return bool(result.rowcount) + + +def renew_task_lock(task_id: int, lock_owner: str, lease_seconds: float) -> bool: + sql = text(""" + UPDATE nexent.agent_automation_task_t + SET lock_until = now() + (:lease_seconds * interval '1 second'), + update_time = now() + WHERE task_id = :task_id + AND lock_owner = :lock_owner + AND lock_until > now() + AND delete_flag = 'N' + AND status = 'ACTIVE' + RETURNING task_id + """) + with get_db_session() as session: + renewed_task_id = session.execute(sql, { + "task_id": task_id, + "lock_owner": lock_owner, + "lease_seconds": lease_seconds, + }).scalar_one_or_none() + return renewed_task_id is not None + + +def recover_orphaned_runs() -> int: + """Finish runs whose task no longer has a live scheduler lease. + + This is safe to run on every replica startup because a live lease is never + modified. Due tasks are retried after their expired lease is reclaimed. + """ + sql = text(""" + UPDATE nexent.agent_automation_run_t AS run + SET status = 'TIMEOUT', + error_code = 'AUTOMATION_LEASE_EXPIRED', + error_message = 'The scheduler stopped before the run completed.', + finished_at = now(), + update_time = now() + FROM nexent.agent_automation_task_t AS task + WHERE run.task_id = task.task_id + AND run.delete_flag = 'N' + AND run.trigger_type = 'SCHEDULED' + AND run.status IN ('QUEUED', 'RUNNING') + AND (task.lock_until IS NULL OR task.lock_until < now()) + """) + with get_db_session() as session: + result = session.execute(sql) + return result.rowcount or 0 + + +def release_expired_locks() -> int: + with get_db_session() as session: + result = session.execute( + update(AgentAutomationTask) + .where( + AgentAutomationTask.delete_flag == "N", + AgentAutomationTask.lock_until.is_not(None), + AgentAutomationTask.lock_until < func.now(), + ) + .values(lock_owner=None, lock_until=None, update_time=_utcnow()) + ) + return result.rowcount or 0 diff --git a/backend/database/agent_db.py b/backend/database/agent_db.py index 5a4ac7baa1..ecdcb63369 100644 --- a/backend/database/agent_db.py +++ b/backend/database/agent_db.py @@ -198,7 +198,9 @@ def create_agent(agent_info, tenant_id: str, user_id: str): """ info_with_metadata = dict(agent_info) info_with_metadata.setdefault("max_steps", 15) + info_with_metadata.setdefault("is_main_agent", True) info_with_metadata.setdefault("verification_config", None) + info_with_metadata.setdefault("context_policy", None) info_with_metadata.update({ "tenant_id": tenant_id, "version_no": 0, # Default to draft version @@ -226,6 +228,7 @@ def create_agent(agent_info, tenant_id: str, user_id: str): "few_shots_prompt": new_agent.few_shots_prompt, "parent_agent_id": new_agent.parent_agent_id, "enabled": new_agent.enabled, + "is_main_agent": new_agent.is_main_agent, "provide_run_summary": new_agent.provide_run_summary, "business_description": new_agent.business_description, "business_logic_model_id": new_agent.business_logic_model_id, @@ -237,6 +240,7 @@ def create_agent(agent_info, tenant_id: str, user_id: str): "enable_context_manager": new_agent.enable_context_manager, "requested_output_tokens": new_agent.requested_output_tokens, "verification_config": new_agent.verification_config, + "context_policy": getattr(new_agent, "context_policy", None), "greeting_message": new_agent.greeting_message, "example_questions": new_agent.example_questions, "current_version_no": new_agent.current_version_no, @@ -335,7 +339,32 @@ def query_all_agent_info_by_tenant_id(tenant_id: str, version_no: int = 0): return [as_dict(agent) for agent in agents] -def insert_related_agent(parent_agent_id: int, child_agent_id: int, tenant_id: str, user_id: str, version_no: int = 0) -> bool: +def batch_search_agent_display_names(agent_ids: List[int], tenant_id: str) -> dict: + """ + Batch query agent display names by agent IDs. + Returns a dict mapping agent_id -> display_name (falls back to name). + + Args: + agent_ids: List of agent IDs to query + tenant_id: Tenant ID + """ + if not agent_ids: + return {} + with get_db_session() as session: + agents = session.query( + AgentInfo.agent_id, + AgentInfo.display_name, + AgentInfo.name + ).filter( + AgentInfo.agent_id.in_(agent_ids), + AgentInfo.tenant_id == tenant_id, + AgentInfo.version_no == 0, + AgentInfo.delete_flag != 'Y' + ).all() + return {a.agent_id: (a.display_name or a.name) for a in agents} + + +def insert_related_agent(parent_agent_id: int, child_agent_id: int, tenant_id: str, user_id: str, version_no: int = 0, selected_agent_version_no: Optional[int] = None) -> bool: """ Insert a related agent. Default version_no=0 creates the draft version. @@ -345,7 +374,8 @@ def insert_related_agent(parent_agent_id: int, child_agent_id: int, tenant_id: s child_agent_id: Child agent ID tenant_id: Tenant ID user_id: User ID - version_no: Version number. Default 0 = draft/editing state + version_no: Parent agent version number. Default 0 = draft/editing state + selected_agent_version_no: Pinned version of child agent. None = runtime fallback to child current_version_no """ try: relation_info = { @@ -353,6 +383,7 @@ def insert_related_agent(parent_agent_id: int, child_agent_id: int, tenant_id: s "selected_agent_id": child_agent_id, "tenant_id": tenant_id, "version_no": version_no, + "selected_agent_version_no": selected_agent_version_no, "created_by": user_id, "updated_by": user_id } @@ -394,22 +425,77 @@ def delete_related_agent(parent_agent_id: int, child_agent_id: int, tenant_id: s return False -def update_related_agents(parent_agent_id: int, related_agent_ids: List[int], tenant_id: str, user_id: str, version_no: int = 0): +def _parse_related_agents(related_agents: Optional[List[dict]]) -> tuple: + """Extract agent_id set and version_map from related_agents list.""" + new_related_ids: set = set() + version_map: dict = {} + if not related_agents: + return new_related_ids, version_map + for rel in related_agents: + agent_id = rel.get("agent_id") + if agent_id is None: + continue + new_related_ids.add(agent_id) + version_no_val = rel.get("version_no") + if version_no_val is not None: + version_map[agent_id] = version_no_val + return new_related_ids, version_map + + +def _add_new_relations(session, parent_agent_id, tenant_id, user_id, version_no, ids_to_add, version_map): + """Insert new agent relations into the database.""" + for child_agent_id in ids_to_add: + relation_info = { + "parent_agent_id": parent_agent_id, + "selected_agent_id": child_agent_id, + "tenant_id": tenant_id, + "version_no": version_no, + "created_by": user_id, + "updated_by": user_id, + } + if child_agent_id in version_map: + relation_info["selected_agent_version_no"] = version_map[child_agent_id] + new_relation = AgentRelation(**filter_property(relation_info, AgentRelation)) + session.add(new_relation) + + +def _update_existing_relations(current_relations, ids_to_update, version_map, user_id): + """Update version_no for existing relations.""" + if not ids_to_update or not version_map: + return + for rel in current_relations: + if rel.selected_agent_id not in ids_to_update: + continue + new_version_no = version_map.get(rel.selected_agent_id) + if new_version_no is not None: + rel.selected_agent_version_no = new_version_no + rel.updated_by = user_id + + +def update_related_agents( + parent_agent_id: int, + tenant_id: str, + user_id: str, + related_agents: Optional[List[dict]] = None, + version_no: int = 0, +): """ Update related agents for a parent agent by replacing all existing relations. Default version_no=0 updates the draft version. This function handles both creation and deletion of relations in a single transaction. + related_agents is the single source of truth: each item has 'agent_id' and optional 'version_no'. Args: parent_agent_id: ID of the parent agent - related_agent_ids: List of child agent IDs to be related tenant_id: Tenant ID user_id: User ID for audit trail + related_agents: List of dicts with 'agent_id' and optional 'version_no' keys version_no: Version number to filter. Default 0 = draft/editing state """ + new_related_ids, version_map = _parse_related_agents(related_agents) + with get_db_session() as session: - # Get current relations current_relations = session.query(AgentRelation).filter( AgentRelation.parent_agent_id == parent_agent_id, AgentRelation.tenant_id == tenant_id, @@ -417,17 +503,12 @@ def update_related_agents(parent_agent_id: int, related_agent_ids: List[int], te AgentRelation.delete_flag != 'Y' ).all() - current_related_ids = { - rel.selected_agent_id for rel in current_relations} - new_related_ids = set( - related_agent_ids) if related_agent_ids else set() + current_related_ids = {rel.selected_agent_id for rel in current_relations} - # Find IDs to delete (in current but not in new) ids_to_delete = current_related_ids - new_related_ids - # Find IDs to add (in new but not in current) ids_to_add = new_related_ids - current_related_ids + ids_to_update = current_related_ids & new_related_ids - # Soft delete removed relations if ids_to_delete: session.query(AgentRelation).filter( AgentRelation.parent_agent_id == parent_agent_id, @@ -439,19 +520,11 @@ def update_related_agents(parent_agent_id: int, related_agent_ids: List[int], te synchronize_session=False ) - # Add new relations - for child_agent_id in ids_to_add: - relation_info = { - "parent_agent_id": parent_agent_id, - "selected_agent_id": child_agent_id, - "tenant_id": tenant_id, - "version_no": version_no, - "created_by": user_id, - "updated_by": user_id - } - new_relation = AgentRelation( - **filter_property(relation_info, AgentRelation)) - session.add(new_relation) + _add_new_relations( + session, parent_agent_id, tenant_id, user_id, version_no, ids_to_add, version_map + ) + + _update_existing_relations(current_relations, ids_to_update, version_map, user_id) def delete_agent_relationship(agent_id: int, tenant_id: str, user_id: str, version_no: int = 0): diff --git a/backend/database/agent_repository_db.py b/backend/database/agent_repository_db.py index 0937d69df3..f5fb71b072 100644 --- a/backend/database/agent_repository_db.py +++ b/backend/database/agent_repository_db.py @@ -63,17 +63,7 @@ def insert_agent_repository_record( return int(new_record.agent_repository_id) -def get_agent_repository_by_id(repository_id: int) -> Optional[dict]: - """Fetch a repository listing by primary key.""" - with get_db_session() as session: - record = session.query(AgentRepository).filter( - AgentRepository.agent_repository_id == repository_id, - AgentRepository.delete_flag != "Y", - ).first() - return as_dict(record) if record else None - - -def get_agent_repository_by_id_and_publisher( +def get_agent_repository_by_id( repository_id: int, publisher_tenant_id: str, ) -> Optional[dict]: @@ -196,6 +186,7 @@ def list_agent_repository_summaries( AgentRepository.version_name, AgentRepository.icon, AgentRepository.downloads, + AgentRepository.content, ).filter( AgentRepository.delete_flag != "Y", AgentRepository.publisher_tenant_id == publisher_tenant_id, @@ -220,6 +211,7 @@ def list_agent_repository_summaries( "version_name": row.version_name, "icon": row.icon, "downloads": row.downloads, + "content": row.content, } for row in rows ] @@ -246,6 +238,7 @@ def update_agent_repository_by_id( "version_no", "agent_info_json", "status", + "content", } update_fields = { key: value @@ -279,6 +272,7 @@ def update_agent_repository_status_by_id( publisher_tenant_id: Optional[str] = None, publisher_user_id: Optional[str] = None, submitted_by: Optional[str] = None, + content: Optional[str] = None, ) -> int: """Update repository listing status by primary key. Returns affected row count.""" update_values: Dict[str, Any] = { @@ -291,6 +285,8 @@ def update_agent_repository_status_by_id( update_values["publisher_user_id"] = publisher_user_id if submitted_by is not None: update_values["submitted_by"] = submitted_by + if content is not None: + update_values["content"] = content with get_db_session() as session: where_clauses = [ @@ -332,26 +328,6 @@ def reset_agent_repository_status( return int(result.rowcount or 0) -def soft_delete_agent_repository_by_id( - *, - repository_id: int, - publisher_tenant_id: str, - user_id: str, -) -> int: - """Soft-delete a repository listing owned by the publisher tenant.""" - with get_db_session() as session: - result = session.execute( - update(AgentRepository) - .where( - AgentRepository.agent_repository_id == repository_id, - AgentRepository.publisher_tenant_id == publisher_tenant_id, - AgentRepository.delete_flag != "Y", - ) - .values(delete_flag="Y", updated_by=user_id) - ) - return int(result.rowcount or 0) - - def list_agent_repository_by_publisher( publisher_tenant_id: str, *, @@ -392,6 +368,7 @@ def list_agent_repository_by_agent_ids( AgentRepository.version_no, AgentRepository.version_name, AgentRepository.create_time, + AgentRepository.content, ) .filter( AgentRepository.delete_flag != "Y", @@ -414,6 +391,7 @@ def list_agent_repository_by_agent_ids( "version_no": row.version_no, "version_name": row.version_name, "create_time": row.create_time, + "content": row.content, } for row in rows ] diff --git a/backend/database/agent_version_db.py b/backend/database/agent_version_db.py index c895cb249d..391e5d5e8a 100644 --- a/backend/database/agent_version_db.py +++ b/backend/database/agent_version_db.py @@ -35,6 +35,36 @@ def search_version_by_version_no( return as_dict(version) if version else None +def batch_search_version_names( + agent_ids: List[int], + tenant_id: str, + version_nos: List[int], +) -> List[dict]: + """ + Batch query version names for multiple (agent_id, version_no) pairs. + + Returns list of dicts: [{"agent_id": int, "version_no": int, "version_name": Optional[str]}] + """ + if not agent_ids or not version_nos: + return [] + + with get_db_session() as session: + versions = session.query(AgentVersion).filter( + AgentVersion.agent_id.in_(agent_ids), + AgentVersion.version_no.in_(version_nos), + AgentVersion.tenant_id == tenant_id, + ).all() + + result = [] + for v in versions: + result.append({ + "agent_id": v.agent_id, + "version_no": v.version_no, + "version_name": v.version_name, + }) + return result + + def search_version_by_id( version_id: int, tenant_id: str, @@ -87,6 +117,42 @@ def query_current_version_no( return agent.current_version_no if agent else None +def batch_query_current_version_nos( + agent_ids: List[int], + tenant_id: str, +) -> dict: + """ + Batch query current published version_no for multiple agents. + + Returns a dict mapping agent_id -> current_version_no (only includes agents + that have a non-null current_version_no). + + Args: + agent_ids: List of agent IDs to query + tenant_id: Tenant ID + """ + if not agent_ids: + return {} + with get_db_session() as session: + agents = session.query( + AgentInfo.agent_id, + AgentInfo.current_version_no, + ).filter( + AgentInfo.agent_id.in_(agent_ids), + or_( + AgentInfo.tenant_id == tenant_id, + AgentInfo.tenant_id == ASSET_OWNER_TENANT_ID, + ), + AgentInfo.version_no == 0, + AgentInfo.delete_flag == 'N', + ).all() + return { + a.agent_id: a.current_version_no + for a in agents + if a.current_version_no is not None + } + + def query_agent_snapshot( agent_id: int, tenant_id: str, @@ -502,13 +568,15 @@ def get_next_version_no( tenant_id: str, ) -> int: """ - Calculate the next version number for an agent + Calculate the next version number from all historical published snapshots. + + Soft-deleted snapshots remain in the table and retain the composite primary + key, so their version numbers must not be reused. """ with get_db_session() as session: max_version = session.query(func.max(AgentInfo.version_no)).filter( AgentInfo.agent_id == agent_id, AgentInfo.tenant_id == tenant_id, - AgentInfo.delete_flag == 'N', ).scalar() return (max_version or 0) + 1 diff --git a/backend/database/conversation_db.py b/backend/database/conversation_db.py index 9efd49881b..176d3f26bf 100644 --- a/backend/database/conversation_db.py +++ b/backend/database/conversation_db.py @@ -24,6 +24,13 @@ class MessageRecord(TypedDict): opinion_flag: Optional[str] +def _serialize_unit_content(content: Any) -> str: + """Serialize structured unit content for the text database column.""" + if isinstance(content, str): + return content + return json.dumps(content, ensure_ascii=False) + + class SearchRecord(TypedDict): message_id: int source_type: str @@ -53,8 +60,50 @@ class ConversationHistory(TypedDict): image_records: List[ImageRecord] +HISTORY_SUMMARY_UNIT_TYPE = "history_summary" + + +class HistorySummaryPersistenceError(ValueError): + """Raised when a history-summary candidate violates persistence rules.""" + + +def _parse_history_summary_content(content: Any) -> Optional[Dict[str, Any]]: + """Return a valid summary payload, or ``None`` for malformed/stale units.""" + try: + payload = json.loads(content) if isinstance(content, str) else content + if not isinstance(payload, dict) or not isinstance(payload.get("summary"), dict): + return None + boundary = payload.get("covered_through_message_id") + if isinstance(boundary, bool) or int(boundary) <= 0: + return None + payload["covered_through_message_id"] = int(boundary) + return payload + except (TypeError, ValueError, json.JSONDecodeError): + return None + + +def _get_user_tenant(user_id: str) -> Optional[Dict[str, Any]]: + """Resolve tenant ownership lazily to keep database modules decoupled.""" + from .user_tenant_db import get_user_tenant_by_user_id + + return get_user_tenant_by_user_id(user_id) + + +def _get_effective_tenant_id(user_tenant: Dict[str, Any]) -> str: + """Resolve legacy empty tenant fields consistently with authentication.""" + from consts.const import ASSET_OWNER_ROLE, ASSET_OWNER_TENANT_ID, DEFAULT_TENANT_ID + + tenant_id = user_tenant.get("tenant_id") + if tenant_id: + return tenant_id + if (user_tenant.get("user_role") or "").upper() == ASSET_OWNER_ROLE: + return ASSET_OWNER_TENANT_ID + return DEFAULT_TENANT_ID + + def create_conversation(conversation_title: str, user_id: Optional[str] = None, - agent_id: Optional[int] = None) -> Dict[str, Any]: + agent_id: Optional[int] = None, + chat_mode: Optional[str] = None) -> Dict[str, Any]: """ Create a new conversation record @@ -62,6 +111,8 @@ def create_conversation(conversation_title: str, user_id: Optional[str] = None, conversation_title: Conversation title user_id: Reserved parameter for created_by and updated_by fields agent_id: Agent used by the latest run in this conversation + chat_mode: Initial UI chat mode ('planning' or 'execution'). Defaults + to the column default ('execution') when omitted. Returns: Dict[str, Any]: Dictionary containing complete information of the newly created conversation @@ -71,6 +122,8 @@ def create_conversation(conversation_title: str, user_id: Optional[str] = None, data = {"conversation_title": conversation_title, "delete_flag": 'N'} if agent_id is not None: data["agent_id"] = agent_id + if chat_mode is not None: + data["chat_mode"] = chat_mode if user_id: data = add_creation_tracking(data, user_id) @@ -78,6 +131,7 @@ def create_conversation(conversation_title: str, user_id: Optional[str] = None, ConversationRecord.conversation_id, ConversationRecord.conversation_title, ConversationRecord.agent_id, + ConversationRecord.chat_mode, (func.extract('epoch', ConversationRecord.create_time) * 1000).label('create_time'), (func.extract('epoch', ConversationRecord.update_time) @@ -91,6 +145,7 @@ def create_conversation(conversation_title: str, user_id: Optional[str] = None, "conversation_id": record.conversation_id, "conversation_title": record.conversation_title, "agent_id": record.agent_id, + "chat_mode": record.chat_mode or "execution", "create_time": int(record.create_time), "update_time": int(record.update_time) } @@ -151,6 +206,7 @@ def create_message_units(message_units: List[Dict[str, Any]], message_id: int, c message_units: List of message units, each containing: - type: Unit type - content: Unit content + - tool_call_id (optional): ID of the originating tool invocation message_id: Message ID (integer) conversation_id: Conversation ID (integer) user_id: Reserved parameter for created_by and updated_by fields @@ -175,7 +231,9 @@ def create_message_units(message_units: List[Dict[str, Any]], message_id: int, c "conversation_id": conversation_id, "unit_index": idx, "unit_type": unit['type'], - "unit_content": unit['content'], + "unit_content": _serialize_unit_content(unit['content']), + "tool_call_id": unit.get("tool_call_id"), + "invocation_id": unit.get("invocation_id"), "delete_flag": 'N' } @@ -194,9 +252,11 @@ def create_message_units(message_units: List[Dict[str, Any]], message_id: int, c def create_message_unit(message_id: int, conversation_id: int, unit_index: int, - unit_type: str, unit_content: str, + unit_type: str, unit_content: Any, user_id: Optional[str] = None, - unit_status: str = 'completed') -> int: + unit_status: str = 'completed', + tool_call_id: Optional[str] = None, + invocation_id: Optional[str] = None) -> int: """ Insert a single ConversationMessageUnit row. @@ -208,6 +268,12 @@ def create_message_unit(message_id: int, conversation_id: int, unit_index: int, unit_content: Complete content of the unit user_id: Reserved parameter for created_by and updated_by fields unit_status: Lifecycle status (streaming / completed) + tool_call_id: Unique ID of the originating tool invocation. None for + units that are not tied to a specific tool call. The frontend uses + this field to attribute side-channel output in parallel execution. + invocation_id: Identifies which sub-agent invocation produced this unit. + Used by the frontend history adapter to route deep-thinking / + reasoning chunks into the correct nested sub-agent card. Returns: int: Newly created unit ID (auto-increment ID) @@ -216,14 +282,15 @@ def create_message_unit(message_id: int, conversation_id: int, unit_index: int, message_id = int(message_id) conversation_id = int(conversation_id) unit_index = int(unit_index) - row_data = { "message_id": message_id, "conversation_id": conversation_id, "unit_index": unit_index, "unit_type": unit_type, - "unit_content": unit_content, + "unit_content": _serialize_unit_content(unit_content), "unit_status": unit_status, + "tool_call_id": tool_call_id, + "invocation_id": invocation_id, "delete_flag": 'N', } if user_id: @@ -314,7 +381,7 @@ def update_message_unit_status(unit_id: int, status: str, ) -def update_message_unit_content(unit_id: int, content: str, +def update_message_unit_content(unit_id: int, content: Any, user_id: Optional[str] = None) -> None: """ Update the unit_content field of a message unit. @@ -327,7 +394,7 @@ def update_message_unit_content(unit_id: int, content: str, with get_db_session() as session: unit_id = int(unit_id) update_data = { - "unit_content": content, + "unit_content": _serialize_unit_content(content), "update_time": func.current_timestamp(), } if user_id: @@ -340,17 +407,29 @@ def update_message_unit_content(unit_id: int, content: str, ) -def get_conversation(conversation_id: int, user_id: Optional[str] = None) -> Optional[Dict[str, Any]]: +def get_conversation( + conversation_id: int, + user_id: Optional[str] = None, + tenant_id: Optional[str] = None, +) -> Optional[Dict[str, Any]]: """ Get conversation details Args: conversation_id: Conversation ID (integer) - user_id: Reserved parameter for created_by and updated_by fields + user_id: User that must own the conversation + tenant_id: Tenant that the owning user must currently belong to Returns: Optional[Dict[str, Any]]: Conversation details, or None if it doesn't exist """ + if tenant_id and not user_id: + raise ValueError("user_id is required when tenant_id is provided") + if tenant_id: + user_tenant = _get_user_tenant(user_id) + if not user_tenant or _get_effective_tenant_id(user_tenant) != tenant_id: + return None + with get_db_session() as session: # Ensure conversation_id is integer type conversation_id = int(conversation_id) @@ -365,7 +444,6 @@ def get_conversation(conversation_id: int, user_id: Optional[str] = None) -> Opt stmt = stmt.where( ConversationRecord.created_by == user_id ) - # Execute the query record = session.scalars(stmt).first() return None if record is None else as_dict(record) @@ -441,6 +519,7 @@ def get_conversation_list(user_id: Optional[str] = None) -> List[Dict[str, Any]] ConversationRecord.conversation_id, ConversationRecord.conversation_title, ConversationRecord.agent_id, + ConversationRecord.chat_mode, (func.extract('epoch', ConversationRecord.create_time) * 1000).label('create_time'), (func.extract('epoch', ConversationRecord.update_time) @@ -464,6 +543,7 @@ def get_conversation_list(user_id: Optional[str] = None) -> List[Dict[str, Any]] conversation = as_dict(record) conversation['create_time'] = int(conversation['create_time']) conversation['update_time'] = int(conversation['update_time']) + conversation['chat_mode'] = conversation.get('chat_mode') or 'execution' result.append(conversation) return result @@ -499,6 +579,50 @@ def update_conversation_agent_id(conversation_id: int, agent_id: int, user_id: O return result.rowcount > 0 +# Allowed values for conversation_record_t.chat_mode. Anything outside this set +# is rejected at the service boundary so the column never stores free-form text. +CHAT_MODE_VALUES = {"planning", "execution"} + + +def update_conversation_chat_mode( + conversation_id: int, + chat_mode: str, + user_id: Optional[str] = None, +) -> bool: + """ + Update the persisted UI chat mode of a conversation. + + Args: + conversation_id: Conversation ID (integer) + chat_mode: New mode. Must be one of 'planning' or 'execution'. + user_id: Reserved parameter for updated_by field + + Returns: + bool: Whether the operation was successful + """ + if chat_mode not in CHAT_MODE_VALUES: + raise ValueError( + f"Invalid chat_mode '{chat_mode}'. Allowed values: {sorted(CHAT_MODE_VALUES)}" + ) + + with get_db_session() as session: + conversation_id = int(conversation_id) + update_data = { + "chat_mode": chat_mode, + "update_time": func.current_timestamp(), + } + if user_id: + update_data = add_update_tracking(update_data, user_id) + + stmt = update(ConversationRecord).where( + ConversationRecord.conversation_id == conversation_id, + ConversationRecord.delete_flag == 'N', + ).values(update_data) + + result = session.execute(stmt) + return result.rowcount > 0 + + def rename_conversation(conversation_id: int, new_title: str, user_id: Optional[str] = None) -> bool: """ Rename a conversation @@ -717,6 +841,7 @@ def get_conversation_history(conversation_id: int, user_id: Optional[str] = None check_stmt = select( ConversationRecord.conversation_id, ConversationRecord.agent_id, + ConversationRecord.chat_mode, (func.extract('epoch', ConversationRecord.create_time) * 1000).label('create_time') ).where( @@ -741,7 +866,9 @@ def get_conversation_history(conversation_id: int, user_id: Optional[str] = None 'unit_type', ConversationMessageUnit.unit_type, 'unit_content', ConversationMessageUnit.unit_content, 'unit_status', ConversationMessageUnit.unit_status, - 'unit_index', ConversationMessageUnit.unit_index + 'unit_index', ConversationMessageUnit.unit_index, + 'tool_call_id', ConversationMessageUnit.tool_call_id, + 'invocation_id', ConversationMessageUnit.invocation_id ) ) ).select_from( @@ -749,7 +876,7 @@ def get_conversation_history(conversation_id: int, user_id: Optional[str] = None ).where( ConversationMessageUnit.message_id == ConversationMessage.message_id, ConversationMessageUnit.delete_flag == 'N', - ConversationMessageUnit.unit_type is not None + ConversationMessageUnit.unit_type.is_not(None) ).scalar_subquery() query = select( @@ -765,7 +892,10 @@ def get_conversation_history(conversation_id: int, user_id: Optional[str] = None ConversationMessage.conversation_id == conversation_id, ConversationMessage.delete_flag == 'N' - ).order_by(ConversationMessage.message_index) + ).order_by( + asc(ConversationMessage.message_index), + asc(ConversationMessage.message_id), + ) message_records = session.execute(query).all() @@ -809,6 +939,7 @@ def get_conversation_history(conversation_id: int, user_id: Optional[str] = None return { 'conversation_id': conversation['conversation_id'], 'agent_id': conversation.get('agent_id'), + 'chat_mode': conversation.get('chat_mode') or 'execution', 'create_time': int(conversation['create_time']), 'message_records': message_list, 'search_records': [as_dict(record) for record in search_records], @@ -1255,6 +1386,22 @@ def get_latest_assistant_message_id(conversation_id: int, user_id: Optional[str] return result +def get_latest_user_message_id(conversation_id: int, user_id: str) -> Optional[int]: + """Return the latest user message in a non-deleted conversation owned by user.""" + with get_db_session() as session: + stmt = select(ConversationMessage.message_id).join( + ConversationRecord, + ConversationMessage.conversation_id == ConversationRecord.conversation_id, + ).where( + ConversationMessage.conversation_id == int(conversation_id), + ConversationMessage.message_role == 'user', + ConversationMessage.delete_flag == 'N', + ConversationRecord.created_by == user_id, + ConversationRecord.delete_flag == 'N', + ).order_by(desc(ConversationMessage.message_index)).limit(1) + return session.execute(stmt).scalar() + + def get_latest_assistant_message(conversation_id: int, user_id: Optional[str] = None) -> Optional[Dict[str, Any]]: """ Get the latest assistant message for a conversation, including its status field. @@ -1370,3 +1517,198 @@ def update_message_minio_files(message_id: int, skill_file_uploads: List[Dict[st record.minio_files = json.dumps(existing, ensure_ascii=False) return True + + +def save_history_summary( + conversation_id: int, user_id: str, tenant_id: str, + summary: Dict[str, Any], covered_through_message_id: int, + previous_summary_unit_id: Optional[int] = None, + trigger: Optional[str] = None, +) -> int: + """Persist a validated checkpoint on its last covered assistant message.""" + if not user_id or not tenant_id or not isinstance(summary, dict): + raise HistorySummaryPersistenceError( + "user_id, tenant_id and an object summary are required") + conversation_id = int(conversation_id) + covered_through_message_id = int(covered_through_message_id) + user_tenant = _get_user_tenant(user_id) + if not user_tenant or _get_effective_tenant_id(user_tenant) != tenant_id: + raise HistorySummaryPersistenceError("conversation is not accessible") + + with get_db_session() as session: + owner = session.execute(select(ConversationRecord.conversation_id).where( + ConversationRecord.conversation_id == conversation_id, + ConversationRecord.created_by == user_id, + ConversationRecord.delete_flag == 'N')).first() + if not owner: + raise HistorySummaryPersistenceError("conversation is not accessible") + + covered = session.execute(select( + ConversationMessage.message_id, ConversationMessage.message_index, + ConversationMessage.message_role, ConversationMessage.status, + ).where( + ConversationMessage.message_id == covered_through_message_id, + ConversationMessage.conversation_id == conversation_id, + ConversationMessage.delete_flag == 'N')).first() + if (not covered or covered.message_role != 'assistant' + or covered.status != 'completed'): + raise HistorySummaryPersistenceError( + "coverage must end at a completed assistant message") + + previous_boundary_index = -1 + if previous_summary_unit_id is not None: + previous = session.execute(select( + ConversationMessageUnit.unit_content, + ConversationMessage.message_index, + ).join(ConversationMessage, + ConversationMessage.message_id == ConversationMessageUnit.message_id).where( + ConversationMessageUnit.unit_id == int(previous_summary_unit_id), + ConversationMessageUnit.conversation_id == conversation_id, + ConversationMessageUnit.unit_type == HISTORY_SUMMARY_UNIT_TYPE, + ConversationMessageUnit.unit_status == 'completed', + ConversationMessageUnit.delete_flag == 'N', + ConversationMessage.delete_flag == 'N')).first() + if not previous or not _parse_history_summary_content(previous.unit_content): + raise HistorySummaryPersistenceError("previous summary is invalid") + previous_boundary_index = previous.message_index + if previous_boundary_index >= covered.message_index: + raise HistorySummaryPersistenceError("summary coverage must advance") + + incomplete_count = session.scalar(select(func.count()).select_from( + ConversationMessage).where( + ConversationMessage.conversation_id == conversation_id, + ConversationMessage.message_index > previous_boundary_index, + ConversationMessage.message_index <= covered.message_index, + ConversationMessage.status != 'completed', + ConversationMessage.delete_flag == 'N')) + if incomplete_count: + raise HistorySummaryPersistenceError( + "history summaries may cover completed messages only") + + max_index = session.scalar(select(func.max( + ConversationMessageUnit.unit_index)).where( + ConversationMessageUnit.message_id == covered_through_message_id, + ConversationMessageUnit.delete_flag == 'N')) + payload: Dict[str, Any] = { + "summary": summary, + "covered_through_message_id": covered_through_message_id, + } + if previous_summary_unit_id is not None: + payload["previous_summary_unit_id"] = int(previous_summary_unit_id) + if trigger: + payload["trigger"] = trigger + row = add_creation_tracking({ + "message_id": covered_through_message_id, + "conversation_id": conversation_id, + "unit_index": (max_index if max_index is not None else -1) + 1, + "unit_type": HISTORY_SUMMARY_UNIT_TYPE, + "unit_content": json.dumps(payload, ensure_ascii=False), + "unit_status": 'completed', "delete_flag": 'N', + }, user_id) + return session.execute(insert(ConversationMessageUnit).values(**row).returning( + ConversationMessageUnit.unit_id)).scalar_one() + + +def get_historical_context( + conversation_id: int, current_user_message_id: int, + user_id: str, tenant_id: str, +) -> Optional[Dict[str, Any]]: + """Load the authorized latest checkpoint and completed turns before a run.""" + if not user_id or not tenant_id: + return None + user_tenant = _get_user_tenant(user_id) + if not user_tenant or _get_effective_tenant_id(user_tenant) != tenant_id: + return None + conversation_id = int(conversation_id) + current_user_message_id = int(current_user_message_id) + with get_db_session() as session: + current = session.execute(select( + ConversationMessage.message_id, ConversationMessage.message_index, + ).join(ConversationRecord, + ConversationRecord.conversation_id == ConversationMessage.conversation_id).where( + ConversationMessage.message_id == current_user_message_id, + ConversationMessage.conversation_id == conversation_id, + ConversationMessage.message_role == 'user', + ConversationMessage.delete_flag == 'N', + ConversationRecord.created_by == user_id, + ConversationRecord.delete_flag == 'N')).first() + if not current: + return None + + candidates = session.execute(select( + ConversationMessageUnit.unit_id, + ConversationMessageUnit.unit_content, + ConversationMessageUnit.unit_index, + ConversationMessage.message_index, + ).join(ConversationMessage, + ConversationMessage.message_id == ConversationMessageUnit.message_id).where( + ConversationMessageUnit.conversation_id == conversation_id, + ConversationMessageUnit.unit_type == HISTORY_SUMMARY_UNIT_TYPE, + ConversationMessageUnit.unit_status == 'completed', + ConversationMessageUnit.delete_flag == 'N', + ConversationMessage.status == 'completed', + ConversationMessage.delete_flag == 'N', + ConversationMessage.message_index < current.message_index, + ).order_by(desc(ConversationMessage.message_index), + desc(ConversationMessageUnit.unit_index))).all() + + summary_record = None + summary_payload = None + boundary_index = -1 + for candidate in candidates: + payload = _parse_history_summary_content(candidate.unit_content) + if not payload: + continue + boundary = session.execute(select( + ConversationMessage.message_index, + ConversationMessage.message_role, + ConversationMessage.status, + ).where( + ConversationMessage.message_id == payload["covered_through_message_id"], + ConversationMessage.conversation_id == conversation_id, + ConversationMessage.delete_flag == 'N')).first() + if (boundary and boundary.message_role == 'assistant' + and boundary.status == 'completed' + and boundary.message_index == candidate.message_index + and boundary.message_index < current.message_index): + summary_record, summary_payload = candidate, payload + boundary_index = boundary.message_index + break + + messages = session.execute(select( + ConversationMessage.message_id, + ConversationMessage.message_index, + ConversationMessage.message_role, + ConversationMessage.message_content, + ConversationMessage.minio_files, + ).where( + ConversationMessage.conversation_id == conversation_id, + ConversationMessage.message_index > boundary_index, + ConversationMessage.message_index < current.message_index, + ConversationMessage.status == 'completed', + ConversationMessage.delete_flag == 'N', + ConversationMessage.message_role.in_(['user', 'assistant']), + ).order_by(asc(ConversationMessage.message_index))).all() + + turns: List[Dict[str, Any]] = [] + pending_user = None + for message in messages: + if message.message_role == 'user': + pending_user = message + elif pending_user is not None: + turns.append({ + "user_message": pending_user.message_content or "", + "assistant_final_answer": message.message_content or "", + "attachments": pending_user.minio_files, + "user_message_id": pending_user.message_id, + "assistant_message_id": message.message_id, + }) + pending_user = None + + summary_result = None + if summary_record and summary_payload: + summary_result = { + "unit_id": summary_record.unit_id, + **summary_payload, + } + return {"history_summary": summary_result, "conversation_turns": turns} diff --git a/backend/database/db_models.py b/backend/database/db_models.py index b50e801fb6..370da0d2ec 100644 --- a/backend/database/db_models.py +++ b/backend/database/db_models.py @@ -16,6 +16,7 @@ _PUBLISHER_TENANT_ID_DOC = "Publisher tenant ID" _PUBLISHER_USER_ID_DOC = "Publisher user ID" _MCP_NAME_DOC = "MCP name" +_INGROUP_PERMISSION_DOC = "In-group permission: EDIT, READ_ONLY, PRIVATE" # Base class for tables without audit fields @@ -47,6 +48,12 @@ class ConversationRecord(TableBase): "conversation_record_t_conversation_id_seq", schema=SCHEMA), primary_key=True, nullable=False) conversation_title = Column(String(100), doc="Conversation title") agent_id = Column(Integer, doc="Agent ID used by the latest run in this conversation") + chat_mode = Column( + String(16), + nullable=False, + server_default=text("'execution'"), + doc="UI chat mode for the conversation: 'planning' or 'execution'", + ) class ConversationMessage(TableBase): @@ -95,6 +102,155 @@ class ConversationMessageUnit(TableBase): unit_status = Column( String(30), default='completed', doc="Lifecycle status: streaming (still aggregating) or completed (fully persisted)") + tool_call_id = Column( + String(36), doc="Unique ID of the originating tool invocation. Used to attribute side-channel units to the correct tool call when multiple calls run in parallel.") + invocation_id = Column( + String(36), doc="Identifies which sub-agent invocation produced this unit. Used by the frontend history adapter to route deep-thinking / reasoning chunks into the correct nested sub-agent card.") + + +class AgentAutomationTask(TableBase): + """User-managed scheduled automation task bound to one conversation.""" + + __tablename__ = "agent_automation_task_t" + __table_args__ = ( + Index( + "idx_agent_automation_due", + "status", + "next_fire_at", + postgresql_where=text("delete_flag = 'N'"), + ), + Index( + "idx_agent_automation_owner", + "tenant_id", + "user_id", + "status", + postgresql_where=text("delete_flag = 'N'"), + ), + Index( + "uq_agent_automation_conversation_active", + "conversation_id", + unique=True, + postgresql_where=text("delete_flag = 'N' AND status <> 'DELETED'"), + ), + {"schema": SCHEMA}, + ) + + task_id = Column(BigInteger, Sequence( + "agent_automation_task_t_task_id_seq", schema=SCHEMA), primary_key=True, nullable=False) + tenant_id = Column(String(100), nullable=False, doc="Tenant ID") + user_id = Column(String(100), nullable=False, doc="Owner user ID") + conversation_id = Column(BigInteger, nullable=False, doc="Bound conversation ID") + agent_id = Column(BigInteger, nullable=False, doc="Bound agent ID") + agent_version_no = Column(Integer, nullable=True, doc="Pinned agent version") + title = Column(String(255), nullable=False, doc="Task title") + instruction = Column(Text, nullable=False, doc="Base instruction for every automation run") + status = Column(String(32), nullable=False, doc="Task lifecycle status") + source = Column(String(32), nullable=False, doc="Creation source") + schedule_mode = Column(String(16), nullable=False, doc="ONCE or RECURRING") + schedule_rule_type = Column(String(16), nullable=False, doc="AT, INTERVAL, or CRON") + schedule_expr = Column(Text, nullable=True, doc="Display schedule expression") + schedule_config = Column(JSONB, nullable=False, doc="Normalized ScheduleTrigger payload") + capability_requirements = Column(JSONB, doc="Capability requirements parsed from user intent") + capability_bindings = Column(JSONB, doc="Confirmed matched capabilities") + runtime_snapshot = Column(JSONB, doc="Agent/runtime capability snapshot at creation time") + timezone = Column(String(64), nullable=False, doc="IANA timezone") + next_fire_at = Column(TIMESTAMP(timezone=True), nullable=True, doc="Next scheduled fire time") + last_fire_at = Column(TIMESTAMP(timezone=True), nullable=True, doc="Last scheduled fire time") + fire_count = Column(Integer, default=0, nullable=False, doc="Number of scheduled fires") + last_run_status = Column(String(32), nullable=True, doc="Latest run status") + last_error = Column(Text, nullable=True, doc="Latest run error") + consecutive_failures = Column(Integer, default=0, nullable=False, doc="Consecutive failure count") + timeout_seconds = Column(Integer, nullable=False, doc="Single-run timeout") + overlap_policy = Column(String(16), nullable=False, doc="Overlap policy") + misfire_policy = Column(String(16), nullable=False, doc="Misfire policy") + lock_owner = Column(String(128), nullable=True, doc="Scheduler lease owner") + lock_until = Column(TIMESTAMP(timezone=True), nullable=True, doc="Scheduler lease expiry") + + +class AgentAutomationRun(TableBase): + """Execution history for an automation task fire.""" + + __tablename__ = "agent_automation_run_t" + __table_args__ = ( + Index( + "idx_agent_automation_run_task", + "task_id", + "scheduled_fire_at", + postgresql_where=text("delete_flag = 'N'"), + ), + Index( + "idx_agent_automation_run_conversation", + "conversation_id", + "status", + postgresql_where=text("delete_flag = 'N'"), + ), + Index( + "uq_agent_automation_active_occurrence", + "task_id", + "scheduled_fire_at", + unique=True, + postgresql_where=text( + "delete_flag = 'N' AND trigger_type = 'SCHEDULED' " + "AND status IN ('QUEUED', 'RUNNING')" + ), + ), + {"schema": SCHEMA}, + ) + + run_id = Column(BigInteger, Sequence( + "agent_automation_run_t_run_id_seq", schema=SCHEMA), primary_key=True, nullable=False) + task_id = Column(BigInteger, nullable=False, doc="Automation task ID") + tenant_id = Column(String(100), nullable=False, doc="Tenant ID") + user_id = Column(String(100), nullable=False, doc="Owner user ID") + conversation_id = Column(BigInteger, nullable=False, doc="Bound conversation ID") + scheduled_fire_at = Column(TIMESTAMP(timezone=True), nullable=False, doc="Scheduled fire time") + actual_fire_at = Column(TIMESTAMP(timezone=True), nullable=True, doc="Actual fire time") + trigger_type = Column(String(32), nullable=False, doc="SCHEDULED or MANUAL") + status = Column(String(32), nullable=False, doc="Run lifecycle status") + generated_prompt = Column(Text, nullable=True, doc="Prompt appended to the conversation") + user_message_id = Column(BigInteger, nullable=True, doc="Automation user message ID") + assistant_message_id = Column(BigInteger, nullable=True, doc="Assistant message ID") + started_at = Column(TIMESTAMP(timezone=True), nullable=True, doc="Run start time") + finished_at = Column(TIMESTAMP(timezone=True), nullable=True, doc="Run finish time") + duration_ms = Column(BigInteger, nullable=True, doc="Run duration in milliseconds") + error_code = Column(String(64), nullable=True, doc="Automation error code") + error_message = Column(Text, nullable=True, doc="Automation error message") + + +class AgentAutomationProposal(TableBase): + """Pending automation task proposal created from chat intent.""" + + __tablename__ = "agent_automation_proposal_t" + __table_args__ = ( + Index( + "idx_agent_automation_proposal_owner", + "tenant_id", + "user_id", + "status", + postgresql_where=text("delete_flag = 'N'"), + ), + Index( + "uq_agent_automation_proposal_source_message", + "tenant_id", + "user_id", + "source_message_id", + unique=True, + postgresql_where=text("delete_flag = 'N' AND source_message_id IS NOT NULL"), + ), + {"schema": SCHEMA}, + ) + + proposal_id = Column(BigInteger, Sequence( + "agent_automation_proposal_t_proposal_id_seq", schema=SCHEMA), primary_key=True, nullable=False) + tenant_id = Column(String(100), nullable=False, doc="Tenant ID") + user_id = Column(String(100), nullable=False, doc="Owner user ID") + conversation_id = Column(BigInteger, nullable=False, doc="Source conversation ID") + agent_id = Column(BigInteger, nullable=False, doc="Bound agent ID") + source_message_id = Column(BigInteger, nullable=True, doc="User message that requested the proposal") + proposed_task = Column(JSONB, nullable=False, doc="Proposed automation task payload") + capability_resolution = Column(JSONB, nullable=False, doc="Capability matching result") + status = Column(String(32), nullable=False, doc="PENDING, ACCEPTED, REJECTED, or EXPIRED") + expires_at = Column(TIMESTAMP(timezone=True), nullable=False, doc="Proposal expiry time") class ConversationSourceImage(TableBase): @@ -458,6 +614,7 @@ class AgentInfo(TableBase): parent_agent_id = Column(Integer, doc="Parent Agent ID") tenant_id = Column(String(100), doc="Belonging tenant") enabled = Column(Boolean, doc="Enabled") + is_main_agent = Column(Boolean, default=True, nullable=False, doc="Whether this agent is a main agent") provide_run_summary = Column( Boolean, doc="Whether to provide the running summary to the manager agent") business_description = Column( @@ -473,7 +630,7 @@ class AgentInfo(TableBase): group_ids = Column(String, doc="Agent group IDs list") is_new = Column(Boolean, default=False, doc="Whether this agent is marked as new for the user") current_version_no = Column(Integer, nullable=True, doc="Current published version number. NULL means no version published yet") - ingroup_permission = Column(String(30), doc="In-group permission: EDIT, READ_ONLY, PRIVATE") + ingroup_permission = Column(String(30), doc=_INGROUP_PERMISSION_DOC) requested_output_tokens = Column( Integer, doc=( @@ -483,6 +640,7 @@ class AgentInfo(TableBase): ) enable_context_manager = Column(Boolean, default=True, doc="Whether to enable context management (compression) for this agent") verification_config = Column(JSONB, doc="Layered ReAct self-verification configuration") + context_policy = Column(JSONB, doc="Agent-level context processing policy override") greeting_message = Column(Text, doc="Agent greeting message displayed on chat initial screen") example_questions = Column(JSONB, doc="List of example questions for starting a conversation with this agent") @@ -569,7 +727,7 @@ class KnowledgeRecord(TableBase): tenant_id = Column(String(100), doc="Tenant ID") group_ids = Column(String, doc="Knowledge base group IDs list") ingroup_permission = Column( - String(30), doc="In-group permission: EDIT, READ_ONLY, PRIVATE") + String(30), doc=_INGROUP_PERMISSION_DOC) summary_frequency = Column(String(10), nullable=True, doc="Auto-summary frequency: '3h', '5h', '1d', '1w', or NULL (disabled)") last_summary_time = Column(TIMESTAMP(timezone=False), nullable=True, @@ -581,6 +739,10 @@ class KnowledgeRecord(TableBase): default=True, doc="Whether to preserve uploaded source documents after vectorization", ) + quota_limit_bytes = Column( + BigInteger, nullable=True, + doc="Per-KB soft storage quota in bytes. NULL means no per-KB limit (shares tenant pool freely)." + ) class TenantConfig(TableBase): @@ -617,6 +779,174 @@ class MemoryUserConfig(TableBase): config_value = Column(String(10000), doc="the value of the config") +# --------------------------------------------------------------------------- +# Phase 2 Memory System tables +# --------------------------------------------------------------------------- + +# Identifier lengths used by memory_records_t and memory_retrieval_hits_t. +# memory_id is auto-incremented by PostgreSQL (serial4) on insert; callers do +# not supply a value. ES mirrors it as `str(memory_id)` in the document `_id`. +class MemoryRecord(TableBase): + """Internal memory records persisted in PostgreSQL. + + This is the authoritative store for tenant/user/agent memory. Tenant and + user long-term memories live here exclusively; agent short-term memory + additionally mirrors the content into Elasticsearch (managed by + ``services.memory_index_service``). + + The isolation contract from ``memory_design.md`` is enforced by the + database access layer, not by the schema: + - tenant layer: tenant_id + - user layer: tenant_id + user_id + - agent layer: tenant_id + user_id + agent_id (+ conversation_id) + """ + + __tablename__ = "memory_records_t" + __table_args__ = ( + Index("idx_memory_records_tenant", "tenant_id"), + Index("idx_memory_records_user", "tenant_id", "user_id"), + Index( + "idx_memory_records_agent", + "tenant_id", + "user_id", + "agent_id", + "conversation_id", + ), + Index( + "idx_memory_records_idempotency", + "tenant_id", + "idempotency_key", + ), + Index( + "idx_memory_records_status", + "tenant_id", + "user_id", + "layer", + "status", + ), + {"schema": SCHEMA}, + ) + + memory_id = Column(Integer, primary_key=True, nullable=False, autoincrement=True, + doc="Auto-incremented memory primary key (serial4).") + tenant_id = Column(String(100), nullable=False, + doc="Tenant ID (isolation key).") + user_id = Column(String(100), nullable=False, + doc="User ID (isolation key for user/agent layers).") + agent_id = Column(String(100), nullable=True, + doc="Agent ID (isolation key for agent short-term layer).") + conversation_id = Column(String(100), nullable=True, + doc="Conversation ID (further isolation key for agent).") + + layer = Column(String(30), nullable=False, + doc="Memory layer: tenant | user | agent.") + memory_type = Column(String(30), nullable=True, + doc="Memory type: long_term | short_term.") + status = Column(String(30), nullable=False, default="active", + doc="Status: active | archived | disabled.") + + content = Column(Text, nullable=False, doc="Memory content.") + concept_tags = Column(ARRAY(Text), nullable=True, + doc="Optional concept tags from Dreaming REM phase.") + + es_index_name = Column(String(255), nullable=True, + doc="Elasticsearch index for agent short-term memory " + "(mem_{model_name}_{dimension}); null for PG-only layers.") + + create_time = Column(TIMESTAMP(timezone=False), server_default=func.now(), + doc="Creation timestamp.") + update_time = Column(TIMESTAMP(timezone=False), server_default=func.now(), + onupdate=func.now(), + doc="Last update timestamp.") + created_by = Column(String(100), nullable=True, doc="Creator user id.") + updated_by = Column(String(100), nullable=True, doc="Last updater user id.") + delete_flag = Column(String(1), nullable=False, default="N", + doc="Soft-delete flag (Y/N).") + + idempotency_key = Column(String(128), nullable=False, + doc="Idempotency key for write deduplication.") + + recall_count = Column(Integer, nullable=False, default=0, + doc="Total recall hit count.") + daily_count = Column(Integer, nullable=False, default=0, + doc="Recall hit count for the most recent active day.") + grounded_count = Column(Integer, nullable=False, default=0, + doc="Count of grounded (verified) recalls.") + last_recalled_at = Column(TIMESTAMP(timezone=False), nullable=True, + doc="Most recent recall timestamp.") + query_hashes = Column(ARRAY(Text), nullable=True, + doc="Hashes of queries that recalled this memory.") + recall_days = Column(ARRAY(Text), nullable=True, + doc="ISO date strings of recall days.") + + light_hits = Column(Integer, nullable=False, default=0, + doc="Light Sleep phase hit count.") + rem_hits = Column(Integer, nullable=False, default=0, + doc="REM Sleep phase hit count.") + last_light_at = Column(TIMESTAMP(timezone=False), nullable=True, + doc="Last Light Sleep timestamp.") + last_rem_at = Column(TIMESTAMP(timezone=False), nullable=True, + doc="Last REM Sleep timestamp.") + + +class MemoryRetrievalHit(TableBase): + """Per-hit memory retrieval log row, sourced by ``search_memory`` tools. + + Phase 2 only writes rows from internal PG-backed recalls. Dreaming + aggregates these rows in batch to update ``memory_records_t`` statistics. + """ + + __tablename__ = "memory_retrieval_hits_t" + __table_args__ = ( + Index("idx_memory_retrieval_hits_memory", "memory_id", "occurred_at"), + Index( + "idx_memory_retrieval_hits_tenant_user_agent", + "tenant_id", + "user_id", + "agent_id", + "day", + ), + {"schema": SCHEMA}, + ) + + hit_id = Column(Integer, primary_key=True, nullable=False, autoincrement=True, + doc="Hit primary key (serial4).") + tenant_id = Column(String(100), nullable=True, doc="Tenant ID.") + user_id = Column(String(100), nullable=True, doc="User ID.") + agent_id = Column(String(100), nullable=True, doc="Agent ID.") + conversation_id = Column(String(100), nullable=True, + doc="Conversation ID.") + memory_id = Column(Integer, nullable=True, + doc="Recalled memory id (null on miss rows).") + query_text = Column(Text, nullable=True, + doc="Original search query text.") + query_hash = Column(String(128), nullable=True, + doc="Stable hash of the query text.") + retrieval_score = Column(Numeric(38, 18), nullable=True, + doc="Similarity score reported by the backend.") + source = Column(String(100), nullable=False, default="nexent", + doc="Hit origin: nexent | external_provider.") + occurred_at = Column(TIMESTAMP(timezone=False), nullable=False, + server_default=func.now(), + doc="Time the hit was recorded.") + day = Column(String(100), nullable=True, + doc="ISO date string (occurred_at::date).") + grounded = Column(Boolean, nullable=False, default=False, + doc="Whether the hit was verified/grounded.") + create_time = Column(TIMESTAMP(timezone=False), nullable=True, + server_default=func.now(), + doc="Row creation time.") + update_time = Column(TIMESTAMP(timezone=False), nullable=True, + server_default=func.now(), + doc="Row last update time.") + created_by = Column(String(100), nullable=True, + doc="User that created the row.") + updated_by = Column(String(100), nullable=True, + doc="User that last updated the row.") + delete_flag = Column(String(1), nullable=False, default="N", + doc="Soft delete flag (N = active, Y = deleted).") + + class McpRecord(TableBase): """ MCP (Model Context Protocol) records table @@ -661,6 +991,11 @@ class McpRecord(TableBase): enabled = Column(Boolean, default=True, doc="Enabled") tags = Column(ARRAY(Text), doc="Tags") description = Column(Text, doc="Description") + group_ids = Column(String, doc="Comma-separated group IDs that can access this MCP") + ingroup_permission = Column(String(30), default="READ_ONLY", + doc="In-group permission: EDIT, READ_ONLY, PRIVATE") + shared_fields = Column(JSON, default=None, + doc="JSON object of field-level sharing flags (e.g. {\"serverUrl\": true, \"authorizationToken\": false})") class McpCommunityRecord(TableBase): @@ -716,11 +1051,17 @@ class McpMarketRecord(TableBase): config_json = Column(JSON, doc="Public-shareable MCP configuration JSON") tags = Column(ARRAY(Text), doc="Tags") description = Column(Text, doc="Description") + content = Column(Text, doc="Listing note on submit or review opinion on approve/reject") download_count = Column(Integer, default=0, doc="Cumulative download/install count") review_status = Column(String(30), default="not_shared", doc="Listing status: not_shared / pending_review / rejected / shared") submitted_by = Column(String(100), doc="Submitter email when listing enters pending_review") source_mcp_id = Column(Integer, doc="Local MCP record ID that created this market record") + group_ids = Column(String, doc="Comma-separated group IDs that can access this MCP") + ingroup_permission = Column(String(30), default="READ_ONLY", + doc="In-group permission: EDIT, READ_ONLY, PRIVATE") + shared_fields = Column(JSON, default=None, + doc="Snapshot of shared_fields at submission time") class UserTenant(TableBase): @@ -742,18 +1083,19 @@ class UserTenant(TableBase): class AgentRelation(TableBase): """ Agent parent-child relationship table + Primary key: (relation_id, version_no) """ __tablename__ = "ag_agent_relation_t" __table_args__ = {"schema": SCHEMA} relation_id = Column(Integer, Sequence("ag_agent_relation_t_relation_id_seq", schema=SCHEMA), primary_key=True, nullable=False, doc="Relationship ID, primary key") + version_no = Column(Integer, primary_key=True, default=0, nullable=False, + doc="Version number. 0 = draft/editing state, >=1 = published snapshot") selected_agent_id = Column( - Integer, primary_key=True, doc="Selected agent ID") + Integer, doc="Selected agent ID") parent_agent_id = Column(Integer, doc="Parent agent ID") tenant_id = Column(String(100), doc="Tenant ID") - version_no = Column(Integer, default=0, nullable=False, - doc="Version number. 0 = draft/editing state, >=1 = published snapshot") selected_agent_version_no = Column( Integer, nullable=True, doc="Pinned version of selected_agent_id. NULL = runtime fallback to child current_version_no", @@ -921,6 +1263,7 @@ class AgentRepository(TableBase): doc="Frozen ExportAndImportDataFormat snapshot with optional skills") status = Column(String(30), default="not_shared", doc="Listing status: not_shared (未共享) / pending_review (待审核) / rejected (审核驳回) / shared (已共享)") + content = Column(Text, doc="Listing note on submit or review opinion on approve/reject") class SkillRepository(TableBase): @@ -947,6 +1290,7 @@ class SkillRepository(TableBase): skill_zip_base64 = Column(Text, nullable=False, doc="Frozen skill ZIP payload encoded as base64") status = Column(String(30), default="not_shared", doc="Listing status: not_shared / pending_review / rejected / shared") + content = Column(Text, doc="Listing note on submit or review opinion on approve/reject") class UserTokenInfo(TableBase): @@ -1041,12 +1385,31 @@ class SkillInfo(TableBase): Skill information table - stores skill metadata and content. """ __tablename__ = "ag_skill_info_t" - __table_args__ = {"schema": SCHEMA} + __table_args__ = ( + Index( + "uq_skill_info_tenant_name_active", + "tenant_id", + "skill_name", + unique=True, + postgresql_where=text( + "tenant_id IS NOT NULL AND delete_flag = 'N'" + ), + ), + Index( + "uq_skill_info_global_name_active", + "skill_name", + unique=True, + postgresql_where=text( + "tenant_id IS NULL AND delete_flag = 'N'" + ), + ), + {"schema": SCHEMA}, + ) skill_id = Column(Integer, Sequence("ag_skill_info_t_skill_id_seq", schema=SCHEMA), primary_key=True, nullable=False, autoincrement=True, doc="Skill ID") skill_name = Column(String(100), nullable=False, - unique=True, doc="Unique skill name") + doc="Skill name, unique among active skills within its tenant scope") tenant_id = Column(String(100), nullable=True, doc="Tenant ID for multi-tenancy. NULL for pre-existing skills.") skill_description = Column(String(1000), doc="Skill description") @@ -1058,6 +1421,8 @@ class SkillInfo(TableBase): JSON, doc="Runtime parameter values from config/config.yaml") source = Column(String(30), nullable=False, default="official", doc="Skill source: official, custom, etc.") + group_ids = Column(String, doc="Skill group IDs list") + ingroup_permission = Column(String(30), doc=_INGROUP_PERMISSION_DOC) class SkillToolRelation(TableBase): @@ -1214,6 +1579,22 @@ class A2AExternalAgent(TableBase): # For URL mode source_url = Column(String(512), doc="Direct URL to agent card") + agent_card_headers = Column( + JSON, + doc="Headers used only to retrieve and refresh the Agent Card" + ) + + # Security declared by the Agent Card and credentials configured by the user + security_schemes = Column(JSON, doc="Security schemes declared by the Agent Card") + security_requirements = Column(JSON, doc="Security requirements declared by the Agent Card") + security_credentials = Column( + JSON, + doc="Credential values for Agent Card security schemes, never exposed by APIs" + ) + selected_security_requirement_index = Column( + Integer, + doc="Selected Agent Card security requirement index used for external agent calls" + ) # For Nacos mode nacos_config_id = Column( @@ -1597,3 +1978,64 @@ class AgentEvaluationCase(TableBase): Index("ix_agent_eval_case_pass_status", "tenant_id", "agent_evaluation_id", "pass_status"), {"schema": SCHEMA}, ) + + +class Notification(TableBase): + """ + In-app notification message table. One row per message; actual per-user + delivery and read state live in notification_receiver_t (fan-out). + """ + __tablename__ = "notification_t" + + notification_id = Column( + BigInteger, + Sequence("notification_t_notification_id_seq", schema=SCHEMA), + primary_key=True, nullable=False, + doc="Notification ID, unique primary key") + event_type = Column(String(50), nullable=False, + doc="Event type, e.g. repository_review_approved / repository_review_rejected") + resource_type = Column(String(50), nullable=False, + doc="Resource type, e.g. agent_repository / skill_repository / mcp_repository") + unique_id = Column(BigInteger, + doc="Related resource primary key (e.g. agent_repository_id)") + details = Column(JSONB, doc="i18n interpolation details for the event template") + scope = Column(String(20), nullable=False, + doc="Audience scope: SU / TENANT / TENANT_ADMIN / USER") + tenant_id = Column(String(100), + doc="tenant for TENANT / TENANT_ADMIN scope; NULL for SU") + is_active = Column(Boolean, nullable=False, default=True, + doc="Whether this notification is still active/valid") + + __table_args__ = ( + Index( + "ix_notification_event_resource_unique_active", + "event_type", "resource_type", "unique_id", "is_active", + ), + {"schema": SCHEMA}, + ) + + +class NotificationReceiver(TableBase): + """ + Per-user notification delivery and read status (fan-out from notification_t). + """ + __tablename__ = "notification_receiver_t" + + receiver_id = Column( + BigInteger, + Sequence("notification_receiver_t_receiver_id_seq", schema=SCHEMA), + primary_key=True, nullable=False, + doc="Receiver row ID, unique primary key") + notification_id = Column(BigInteger, nullable=False, + doc="FK to notification_t.notification_id") + receiver_user_id = Column(String(100), nullable=False, + doc="Receiver user ID") + tenant_id = Column(String(100), doc=_TENANT_ID_DOC) + is_read = Column(Boolean, default=False, + doc="Whether this receiver has read the notification") + + __table_args__ = ( + Index("ix_notification_receiver_user_read", "receiver_user_id", "is_read"), + Index("ix_notification_receiver_notification_id", "notification_id"), + {"schema": SCHEMA}, + ) diff --git a/backend/database/group_db.py b/backend/database/group_db.py index 6066e85fcc..5b0b8c2385 100644 --- a/backend/database/group_db.py +++ b/backend/database/group_db.py @@ -250,6 +250,54 @@ def query_group_users(group_id: int) -> List[Dict[str, Any]]: return [as_dict(record) for record in result] +def query_group_ids_by_user_in_tenant(user_id: str, tenant_id: str) -> List[int]: + """Return ``user_id``'s group IDs restricted to ``tenant_id``. + + Joins ``tenant_group_user_t`` with ``tenant_group_info_t`` so a user that + happens to share a group ID across tenants cannot leak access. Filters on + ``delete_flag='N'`` for both tables. + """ + if not user_id or not tenant_id: + return [] + with get_db_session() as session: + result = ( + session.query(TenantGroupUser.group_id) + .join( + TenantGroupInfo, + TenantGroupInfo.group_id == TenantGroupUser.group_id, + ) + .filter( + TenantGroupUser.user_id == user_id, + TenantGroupUser.delete_flag == "N", + TenantGroupInfo.tenant_id == tenant_id, + TenantGroupInfo.delete_flag == "N", + ) + .all() + ) + return [row[0] for row in result] + + +def filter_tenant_group_ids(group_ids: List[int], tenant_id: str) -> List[int]: + """Return the subset of ``group_ids`` that exist in ``tenant_id``. + + Used by the AIDP permission service to validate submitted ``group_ids`` + against the current tenant. Returns an empty list when no IDs match. + """ + if not group_ids or not tenant_id: + return [] + with get_db_session() as session: + rows = ( + session.query(TenantGroupInfo.group_id) + .filter( + TenantGroupInfo.group_id.in_(group_ids), + TenantGroupInfo.tenant_id == tenant_id, + TenantGroupInfo.delete_flag == "N", + ) + .all() + ) + return [row[0] for row in rows] + + def query_group_ids_by_user(user_id: str) -> List[int]: """ Query all group IDs for a user @@ -292,6 +340,41 @@ def query_groups_by_user(user_id: str) -> List[Dict[str, Any]]: return [as_dict(record) for record in result] +def query_groups_by_users(user_ids: List[str]) -> Dict[str, List[str]]: + """ + Batch query group names for multiple users in a single query. + + Args: + user_ids: List of user IDs + + Returns: + Dict mapping user_id -> list of group names + """ + if not user_ids: + return {} + + result_map: Dict[str, List[str]] = {uid: [] for uid in user_ids} + + with get_db_session() as session: + rows = ( + session.query(TenantGroupUser.user_id, TenantGroupInfo.group_name) + .join( + TenantGroupInfo, + TenantGroupInfo.group_id == TenantGroupUser.group_id, + ) + .filter( + TenantGroupUser.user_id.in_(user_ids), + TenantGroupUser.delete_flag == "N", + TenantGroupInfo.delete_flag == "N", + ) + .all() + ) + for uid, group_name in rows: + result_map[uid].append(group_name) + + return result_map + + def check_user_in_group(user_id: str, group_id: int) -> bool: """ Check if user is in a specific group diff --git a/backend/database/knowledge_db.py b/backend/database/knowledge_db.py index 8fc60d6bd4..de519d63bf 100644 --- a/backend/database/knowledge_db.py +++ b/backend/database/knowledge_db.py @@ -61,6 +61,10 @@ def create_knowledge_record(query: Dict[str, Any]) -> Dict[str, Any]: "preserve_source_file": query.get("preserve_source_file", True), } + # Per-KB soft quota (optional, null = unlimited) + if "quota_limit_bytes" in query: + data["quota_limit_bytes"] = query["quota_limit_bytes"] + # For backward compatibility: if caller explicitly provides index_name, # respect it and do not regenerate; otherwise generate after flush. explicit_index_name = query.get("index_name") @@ -132,6 +136,10 @@ def upsert_knowledge_record(query: Dict[str, Any]) -> Dict[str, Any]: existing_record.updated_by = query.get('user_id') existing_record.update_time = func.current_timestamp() + # Update per-KB soft quota if provided + if "quota_limit_bytes" in query: + existing_record.quota_limit_bytes = query["quota_limit_bytes"] + session.flush() session.commit() return { @@ -191,6 +199,10 @@ def update_knowledge_record(query: Dict[str, Any]) -> bool: if query.get("group_ids") is not None: record.group_ids = query["group_ids"] + # Update per-KB soft quota + if "quota_limit_bytes" in query: + record.quota_limit_bytes = query["quota_limit_bytes"] + # Update timestamp and user if query.get("user_id"): record.updated_by = query["user_id"] @@ -439,13 +451,14 @@ def get_index_name_by_knowledge_name(knowledge_name: str, tenant_id: str) -> str raise e -def get_knowledge_name_map_by_index_names(index_names: List[str]) -> Dict[str, str]: +def get_knowledge_name_map_by_index_names(index_names: List[str], tenant_id: str) -> Dict[str, str]: """ Get a mapping from index_name to knowledge_name (display name) for the given index_names. Used to build user-friendly knowledge base summaries in prompts. Args: index_names: List of internal index names + tenant_id: Tenant that owns the knowledge bases Returns: Dict[str, str]: Mapping of index_name -> knowledge_name. @@ -462,6 +475,7 @@ def get_knowledge_name_map_by_index_names(index_names: List[str]) -> Dict[str, s KnowledgeRecord.knowledge_name ).filter( KnowledgeRecord.index_name.in_(index_names), + KnowledgeRecord.tenant_id == tenant_id, KnowledgeRecord.delete_flag != 'Y' ).all() diff --git a/backend/database/market_mcp_db.py b/backend/database/market_mcp_db.py index a27b139b13..7a26ce795b 100644 --- a/backend/database/market_mcp_db.py +++ b/backend/database/market_mcp_db.py @@ -1,7 +1,7 @@ import logging -from typing import Any, Dict, List +from typing import Any, Dict, List, Optional -from sqlalchemy import func, or_ +from sqlalchemy import func, or_, text as sa_text from database.client import as_dict, filter_property, get_db_session from database.db_models import McpMarketRecord @@ -9,6 +9,29 @@ logger = logging.getLogger("market_mcp_db") +def _apply_group_permission_filter(query, user_id: str, user_group_ids: List[int]): + """Apply group-based visibility filter to a market record query. + + Users see MCPs where: + - They are the creator, OR + - The MCP has no group restriction (group_ids IS NULL/empty), OR + - They belong to at least one of the MCP's allowed groups + """ + conditions = [ + McpMarketRecord.user_id == user_id, + McpMarketRecord.group_ids.is_(None), + McpMarketRecord.group_ids == "", + ] + if user_group_ids: + group_ids_str = ",".join(str(g) for g in user_group_ids) + conditions.append( + sa_text( + f"string_to_array(group_ids, ',') && ARRAY[{group_ids_str}]::text[]" + ) + ) + return query.filter(or_(*conditions)) + + def get_mcp_market_records( *, tenant_id: str | None = None, @@ -17,6 +40,8 @@ def get_mcp_market_records( transport_type: str | None = None, cursor: str | None = None, limit: int = 30, + user_id: str | None = None, + user_group_ids: Optional[List[int]] = None, ) -> Dict[str, Any]: """Cursor-paginated listing of shared (approved) market records scoped to a tenant.""" with get_db_session() as session: @@ -54,6 +79,9 @@ def get_mcp_market_records( if cursor_id is not None: query = query.filter(McpMarketRecord.market_id < cursor_id) + if user_id is not None and user_group_ids is not None: + query = _apply_group_permission_filter(query, user_id, user_group_ids) + rows: List[McpMarketRecord] = ( query.order_by(McpMarketRecord.market_id.desc()) .limit(limit + 1) @@ -122,8 +150,8 @@ def get_mcp_market_record_by_id(market_id: int) -> Dict[str, Any] | None: return as_dict(record) if record else None -def check_mcp_market_name_exists(mcp_name: str) -> bool: - """Check if a shared market record with the given name already exists. +def check_mcp_market_name_exists(mcp_name: str, tenant_id: str) -> bool: + """Check if a shared market record with the given name already exists in this tenant. Matches the partial unique index uq_mcp_market_name_active: WHERE delete_flag = 'N' AND review_status = 'shared' @@ -131,6 +159,7 @@ def check_mcp_market_name_exists(mcp_name: str) -> bool: with get_db_session() as session: record = session.query(McpMarketRecord).filter( McpMarketRecord.mcp_name == mcp_name, + McpMarketRecord.tenant_id == tenant_id, McpMarketRecord.delete_flag != "Y", McpMarketRecord.review_status == "shared", ).first() @@ -148,6 +177,10 @@ def update_mcp_market_record( mcp_server: str | None = None, config_json: Dict[str, Any] | None = None, transport_type: str | None = None, + group_ids: str | None = None, + ingroup_permission: str | None = None, + shared_fields: dict | None = None, + content: str | None = None, ) -> None: """Update editable fields on a market record (does not change status).""" update_fields: Dict[str, Any] = {"updated_by": user_id} @@ -165,6 +198,14 @@ def update_mcp_market_record( update_fields["config_json"] = config_json if transport_type is not None: update_fields["transport_type"] = transport_type + if group_ids is not None: + update_fields["group_ids"] = group_ids + if ingroup_permission is not None: + update_fields["ingroup_permission"] = ingroup_permission + if shared_fields is not None: + update_fields["shared_fields"] = shared_fields + if content is not None: + update_fields["content"] = content with get_db_session() as session: session.query(McpMarketRecord).filter( @@ -179,11 +220,14 @@ def update_mcp_market_status( user_id: str, review_status: str, submitted_by: str | None = None, + content: str | None = None, ) -> None: """Atomically update the review_status, optionally recording the submitter.""" update_fields: Dict[str, Any] = {"updated_by": user_id, "review_status": review_status} if submitted_by is not None: update_fields["submitted_by"] = submitted_by + if content is not None: + update_fields["content"] = content with get_db_session() as session: session.query(McpMarketRecord).filter( diff --git a/backend/database/memory_record_db.py b/backend/database/memory_record_db.py new file mode 100644 index 0000000000..d81f684b24 --- /dev/null +++ b/backend/database/memory_record_db.py @@ -0,0 +1,611 @@ +"""Database access helpers for the authoritative ``memory_records_t`` table. + +The functions in this module translate the SDK-level +``MemoryRecord`` payload into SQLAlchemy inserts/updates against +``memory_records_t``. They are intentionally thin: policy enforcement and +index/embedding orchestration live in ``services/memory_record_service``. + +Layer rules: +- ``tenant`` and ``user`` long-term memory are stored exclusively in PG. +- ``agent`` short-term memory is stored in PG and mirrored into Elasticsearch + by ``services/memory_index_service``. + +All write operations are soft-delete aware (``delete_flag='N'``). +""" + +from __future__ import annotations + +import logging +from typing import Any, Dict, Iterable, List, Optional, Sequence + +from sqlalchemy import String, and_, cast + +from .client import filter_property, get_db_session +from .db_models import AgentInfo, ConversationRecord, MemoryRecord + + +logger = logging.getLogger("memory_record_db") + + +# --------------------------------------------------------------------------- +# Inserts / updates +# --------------------------------------------------------------------------- + + +def generate_memory_id() -> None: + """No-op placeholder kept for API compatibility. + + ``memory_id`` is allocated by the PostgreSQL ``serial4`` column on insert. + Callers that previously passed a generated uuid must omit ``memory_id`` so + the database can assign the primary key. The function is preserved so that + upstream services / tests continue to compile, but it always returns + ``None`` and never produces a value to seed into the payload. + """ + return None + + +def insert_memory_record(record: Dict[str, Any]) -> Optional[int]: + """Insert a new memory record. + + Args: + record: Payload describing the memory. Required keys: ``tenant_id``, + ``user_id``, ``layer``, ``content``, ``idempotency_key``. + ``memory_id`` must be omitted - the database assigns the serial + primary key on insert. + + Returns: + The persisted memory id (int) assigned by the database, or ``None`` + on failure. + """ + payload = dict(record) + payload.pop("memory_id", None) + payload.setdefault("status", "active") + payload.setdefault("delete_flag", "N") + + with get_db_session() as session: + try: + payload = filter_property(payload, MemoryRecord) + row = MemoryRecord(**payload) + session.add(row) + session.commit() + return row.memory_id + except Exception: + session.rollback() + logger.exception("insert_memory_record failed") + return None + + +def upsert_memory_record_by_idempotency(record: Dict[str, Any]) -> Optional[int]: + """Insert a memory record, or update it when the idempotency key exists. + + Idempotency is scoped by ``(tenant_id, idempotency_key)`` - the same key + from a different tenant is treated as a distinct memory. If a matching + row exists, ``content`` and ``concept_tags`` are refreshed and + ``update_time`` is bumped; ``memory_id`` is preserved. + + Args: + record: Same shape as ``insert_memory_record``. ``memory_id`` is + ignored on insert - the database allocates the serial id. + + Returns: + The persisted memory id (int, existing or new), or ``None`` on failure. + """ + tenant_id = record.get("tenant_id") + idempotency_key = record.get("idempotency_key") + if not tenant_id or not idempotency_key: + raise ValueError( + "upsert_memory_record_by_idempotency requires tenant_id and idempotency_key" + ) + + with get_db_session() as session: + try: + existing = ( + session.query(MemoryRecord) + .filter( + MemoryRecord.tenant_id == tenant_id, + MemoryRecord.idempotency_key == idempotency_key, + MemoryRecord.delete_flag == "N", + ) + .first() + ) + if existing is not None: + update_payload = filter_property( + { + "content": record.get("content", existing.content), + "concept_tags": record.get("concept_tags"), + "status": record.get("status", existing.status), + "updated_by": record.get("updated_by"), + "memory_type": record.get("memory_type", existing.memory_type), + "es_index_name": record.get("es_index_name", existing.es_index_name), + }, + MemoryRecord, + ) + for key, value in update_payload.items(): + setattr(existing, key, value) + session.commit() + return existing.memory_id + + payload = filter_property(record, MemoryRecord) + payload.pop("memory_id", None) + payload.setdefault("status", "active") + payload.setdefault("delete_flag", "N") + row = MemoryRecord(**payload) + session.add(row) + session.commit() + return row.memory_id + except Exception: + session.rollback() + logger.exception("upsert_memory_record_by_idempotency failed") + return None + + +def update_memory_record( + memory_id: int, + tenant_id: str, + update_data: Dict[str, Any], +) -> bool: + """Update fields of an active memory record. + + Args: + memory_id: Primary key of the record (integer). + tenant_id: Tenant isolation key (defence-in-depth on top of PK). + update_data: Columns to update. + + Returns: + True on success, False on failure. + """ + with get_db_session() as session: + try: + payload = filter_property(update_data, MemoryRecord) + rows = ( + session.query(MemoryRecord) + .filter( + MemoryRecord.memory_id == memory_id, + MemoryRecord.tenant_id == tenant_id, + MemoryRecord.delete_flag == "N", + ) + .update(payload) + ) + session.commit() + return bool(rows) + except Exception: + session.rollback() + logger.exception("update_memory_record failed") + return False + + +def soft_delete_memory_record( + memory_id: int, + tenant_id: str, + updated_by: Optional[str] = None, +) -> bool: + """Soft-delete a single record by id.""" + with get_db_session() as session: + try: + rows = ( + session.query(MemoryRecord) + .filter( + MemoryRecord.memory_id == memory_id, + MemoryRecord.tenant_id == tenant_id, + MemoryRecord.delete_flag == "N", + ) + .update( + { + "delete_flag": "Y", + "status": "archived", + "updated_by": updated_by, + } + ) + ) + session.commit() + return bool(rows) + except Exception: + session.rollback() + logger.exception("soft_delete_memory_record failed") + return False + + +# --------------------------------------------------------------------------- +# Reads +# --------------------------------------------------------------------------- + + +def get_memory_record( + memory_id: int, + tenant_id: str, + *, + include_deleted: bool = False, +) -> Optional[Dict[str, Any]]: + """Fetch a single memory record by id.""" + with get_db_session() as session: + try: + query = session.query(MemoryRecord).filter( + MemoryRecord.memory_id == memory_id, + MemoryRecord.tenant_id == tenant_id, + ) + if not include_deleted: + query = query.filter(MemoryRecord.delete_flag == "N") + record = query.first() + if record is None: + return None + return _record_to_dict(record) + except Exception: + session.rollback() + logger.exception("get_memory_record failed") + return None + + +def list_memory_records( + tenant_id: str, + *, + user_id: Optional[str] = None, + agent_id: Optional[str] = None, + conversation_id: Optional[str] = None, + layer: Optional[str] = None, + memory_type: Optional[str] = None, + status: Optional[str] = None, + limit: int = 100, + offset: int = 0, + include_deleted: bool = False, +) -> List[Dict[str, Any]]: + """List memory records with the standard isolation filters. + + Args: + tenant_id: Tenant isolation key (required). + user_id: Optional user filter. + agent_id: Optional agent filter (implies agent layer). + conversation_id: Optional conversation filter. + layer: Optional layer filter (``tenant``/``user``/``agent``). + memory_type: Optional memory type filter. + status: Optional status filter; defaults to ``active``. + limit: Maximum number of rows to return. + offset: Pagination offset. + include_deleted: When ``True``, soft-deleted rows are included. + + Returns: + A list of records serialized as plain dicts. + """ + # Normalize empty-string filters to ``None`` so that empty query params + # (e.g. ``?status=``) do not translate into ``WHERE status = ''`` filters + # that match no rows. + if user_id == "": + user_id = None + if agent_id == "": + agent_id = None + if conversation_id == "": + conversation_id = None + if layer == "": + layer = None + if memory_type == "": + memory_type = None + all_statuses = status == "" + if all_statuses: + status = None + + with get_db_session() as session: + try: + query = ( + session.query( + MemoryRecord, + AgentInfo.display_name.label("agent_display_name"), + AgentInfo.name.label("agent_name"), + ConversationRecord.conversation_title, + ) + .outerjoin( + AgentInfo, + and_( + cast(AgentInfo.agent_id, String) == MemoryRecord.agent_id, + AgentInfo.version_no == 0, + AgentInfo.delete_flag != "Y", + ), + ) + .outerjoin( + ConversationRecord, + and_( + cast(ConversationRecord.conversation_id, String) + == MemoryRecord.conversation_id, + ConversationRecord.delete_flag != "Y", + ), + ) + .filter(MemoryRecord.tenant_id == tenant_id) + ) + if user_id is not None: + query = query.filter(MemoryRecord.user_id == user_id) + if agent_id is not None: + query = query.filter(MemoryRecord.agent_id == agent_id) + if conversation_id is not None: + query = query.filter( + MemoryRecord.conversation_id == conversation_id + ) + if layer is not None: + query = query.filter(MemoryRecord.layer == layer) + if memory_type is not None: + query = query.filter(MemoryRecord.memory_type == memory_type) + if status is not None: + query = query.filter(MemoryRecord.status == status) + elif not include_deleted and not all_statuses: + query = query.filter(MemoryRecord.status == "active") + if not include_deleted: + query = query.filter(MemoryRecord.delete_flag == "N") + + query = query.order_by(MemoryRecord.update_time.desc()) + query = query.limit(limit).offset(offset) + result = [] + for ( + record, + agent_display_name, + agent_name, + conversation_title, + ) in query.all(): + item = _record_to_dict(record) + item["agent_name"] = agent_display_name or agent_name + item["conversation_title"] = conversation_title + result.append(item) + return result + except Exception: + session.rollback() + logger.exception("list_memory_records failed") + return [] + + +def list_active_memory_ids_by_layer( + tenant_id: str, + layer: str, + *, + user_id: Optional[str] = None, + agent_id: Optional[str] = None, +) -> List[int]: + """Return memory ids for the given layer (used by Dreaming pre-load).""" + with get_db_session() as session: + try: + query = session.query(MemoryRecord.memory_id).filter( + MemoryRecord.tenant_id == tenant_id, + MemoryRecord.layer == layer, + MemoryRecord.delete_flag == "N", + MemoryRecord.status == "active", + ) + if user_id is not None: + query = query.filter(MemoryRecord.user_id == user_id) + if agent_id is not None: + query = query.filter(MemoryRecord.agent_id == agent_id) + return [row[0] for row in query.all()] + except Exception: + session.rollback() + logger.exception("list_active_memory_ids_by_layer failed") + return [] + + +def get_memory_records_by_ids( + memory_ids: Sequence[int], + tenant_id: str, +) -> List[Dict[str, Any]]: + """Bulk-fetch memory records by id (used by Dreaming aggregation).""" + if not memory_ids: + return [] + with get_db_session() as session: + try: + rows = ( + session.query(MemoryRecord) + .filter( + MemoryRecord.tenant_id == tenant_id, + MemoryRecord.memory_id.in_(list(memory_ids)), + ) + .all() + ) + return [_record_to_dict(row) for row in rows] + except Exception: + session.rollback() + logger.exception("get_memory_records_by_ids failed") + return [] + + +def find_by_idempotency( + tenant_id: str, + idempotency_key: str, +) -> Optional[Dict[str, Any]]: + """Return the existing record for a (tenant, idempotency_key) pair, if any.""" + with get_db_session() as session: + try: + row = ( + session.query(MemoryRecord) + .filter( + MemoryRecord.tenant_id == tenant_id, + MemoryRecord.idempotency_key == idempotency_key, + MemoryRecord.delete_flag == "N", + ) + .first() + ) + return _record_to_dict(row) if row is not None else None + except Exception: + session.rollback() + logger.exception("find_by_idempotency failed") + return None + + +# --------------------------------------------------------------------------- +# Dreaming aggregation helpers +# --------------------------------------------------------------------------- + + +def increment_recall_stats( + memory_id: int, + tenant_id: str, + *, + query_hash: Optional[str] = None, + day: Optional[str] = None, + grounded: bool = False, +) -> bool: + """Bump recall counters on a memory row in a single transaction. + + Used by Dreaming aggregation after it batches hit rows. The function + keeps ``query_hashes`` and ``recall_days`` deduplicated. + """ + with get_db_session() as session: + try: + row = ( + session.query(MemoryRecord) + .filter( + MemoryRecord.memory_id == memory_id, + MemoryRecord.tenant_id == tenant_id, + MemoryRecord.delete_flag == "N", + ) + .first() + ) + if row is None: + return False + + row.recall_count = (row.recall_count or 0) + 1 + row.daily_count = (row.daily_count or 0) + 1 + if grounded: + row.grounded_count = (row.grounded_count or 0) + 1 + row.last_recalled_at = _utcnow() + + existing_hashes: List[str] = list(row.query_hashes or []) + if query_hash and query_hash not in existing_hashes: + existing_hashes.append(query_hash) + row.query_hashes = existing_hashes + + existing_days: List[str] = list(row.recall_days or []) + if day and day not in existing_days: + existing_days.append(day) + row.recall_days = existing_days + + session.commit() + return True + except Exception: + session.rollback() + logger.exception("increment_recall_stats failed") + return False + + +def apply_dreaming_phase( + memory_id: int, + tenant_id: str, + *, + phase: str, +) -> bool: + """Apply Light Sleep or REM Sleep phase counters.""" + with get_db_session() as session: + try: + row = ( + session.query(MemoryRecord) + .filter( + MemoryRecord.memory_id == memory_id, + MemoryRecord.tenant_id == tenant_id, + MemoryRecord.delete_flag == "N", + ) + .first() + ) + if row is None: + return False + now = _utcnow() + if phase == "light": + row.light_hits = (row.light_hits or 0) + 1 + row.last_light_at = now + elif phase == "rem": + row.rem_hits = (row.rem_hits or 0) + 1 + row.last_rem_at = now + else: + raise ValueError(f"Unknown dreaming phase: {phase}") + session.commit() + return True + except Exception: + session.rollback() + logger.exception("apply_dreaming_phase failed") + return False + + +def list_memories_for_dreaming( + tenant_id: str, + *, + user_id: str, + layer: str = "agent", + min_recall_count: int = 0, + window_days: int = 7, +) -> List[Dict[str, Any]]: + """Return agent memories eligible for Dreaming promotion. + + Filters: same tenant/user, target layer (default agent), status active, + recall_count >= min_recall_count, and the row has been recalled within + the last ``window_days`` days. + """ + with get_db_session() as session: + try: + query = session.query(MemoryRecord).filter( + and_( + MemoryRecord.tenant_id == tenant_id, + MemoryRecord.user_id == user_id, + MemoryRecord.layer == layer, + MemoryRecord.status == "active", + MemoryRecord.delete_flag == "N", + MemoryRecord.recall_count >= min_recall_count, + ) + ) + rows = query.all() + cutoff = _utcnow().timestamp() - window_days * 86400 + eligible: List[Dict[str, Any]] = [] + for row in rows: + last = row.last_recalled_at + if last is None: + continue + if last.timestamp() < cutoff: + continue + eligible.append(_record_to_dict(row)) + return eligible + except Exception: + session.rollback() + logger.exception("list_memories_for_dreaming failed") + return [] + + +# --------------------------------------------------------------------------- +# Internal helpers +# --------------------------------------------------------------------------- + + +def _record_to_dict(record: MemoryRecord) -> Dict[str, Any]: + """Serialize a ``MemoryRecord`` ORM row into a JSON-friendly dict.""" + return { + "memory_id": record.memory_id, + "tenant_id": record.tenant_id, + "user_id": record.user_id, + "agent_id": record.agent_id, + "conversation_id": record.conversation_id, + "layer": record.layer, + "memory_type": record.memory_type, + "status": record.status, + "content": record.content, + "concept_tags": list(record.concept_tags or []), + "es_index_name": record.es_index_name, + "create_time": _isoformat_or_none(record.create_time), + "update_time": _isoformat_or_none(record.update_time), + "created_by": record.created_by, + "updated_by": record.updated_by, + "delete_flag": record.delete_flag, + "idempotency_key": record.idempotency_key, + "recall_count": record.recall_count or 0, + "daily_count": record.daily_count or 0, + "grounded_count": record.grounded_count or 0, + "last_recalled_at": _isoformat_or_none(record.last_recalled_at), + "query_hashes": list(record.query_hashes or []), + "recall_days": list(record.recall_days or []), + "light_hits": record.light_hits or 0, + "rem_hits": record.rem_hits or 0, + "last_light_at": _isoformat_or_none(record.last_light_at), + "last_rem_at": _isoformat_or_none(record.last_rem_at), + } + + +def _isoformat_or_none(value): + """Return ``value.isoformat()`` for datetimes, otherwise ``None``/passthrough.""" + if value is None: + return None + if hasattr(value, "isoformat"): + return value.isoformat() + return value + + +def _utcnow(): + from datetime import datetime + + return datetime.utcnow() diff --git a/backend/database/memory_retrieval_hit_db.py b/backend/database/memory_retrieval_hit_db.py new file mode 100644 index 0000000000..9028ad41bf --- /dev/null +++ b/backend/database/memory_retrieval_hit_db.py @@ -0,0 +1,226 @@ +"""Database access helpers for ``memory_retrieval_hits_t``. + +These helpers are intentionally append-only: every row represents a single +recall observed by the ``search_memory`` flow. Dreaming aggregates them +into ``memory_records_t`` statistics in batch. +""" + +from __future__ import annotations + +import logging +from datetime import datetime +from typing import Any, Dict, Iterable, List, Optional, Sequence + +from sqlalchemy import Integer, func + +from .client import filter_property, get_db_session +from .db_models import MemoryRetrievalHit + + +logger = logging.getLogger("memory_retrieval_hit_db") + + +def insert_retrieval_hits(hits: Iterable[Dict[str, Any]]) -> int: + """Append a batch of retrieval hit rows. + + Each hit dict must include at least ``memory_id`` (nullable on miss + rows), ``tenant_id``, ``user_id``, ``agent_id`` (all nullable for miss + rows), and ``occurred_at`` (defaults to current timestamp when omitted). + Unknown columns are dropped via ``filter_property``. + + Returns: + Number of rows inserted. + """ + rows: List[MemoryRetrievalHit] = [] + for hit in hits: + payload = dict(hit) + payload.setdefault("source", "nexent") + payload.setdefault("grounded", False) + if "occurred_at" not in payload or payload["occurred_at"] is None: + payload["occurred_at"] = datetime.utcnow() + payload.setdefault("day", payload["occurred_at"].date().isoformat()) + payload = filter_property(payload, MemoryRetrievalHit) + rows.append(MemoryRetrievalHit(**payload)) + + if not rows: + return 0 + + with get_db_session() as session: + try: + session.add_all(rows) + session.commit() + return len(rows) + except Exception: + session.rollback() + logger.exception("insert_retrieval_hits failed") + return 0 + + +def count_hits_since( + tenant_id: str, + *, + user_id: Optional[str] = None, + agent_id: Optional[str] = None, + since: Optional[datetime] = None, +) -> int: + """Count hits matching the given isolation scope.""" + with get_db_session() as session: + try: + query = session.query(func.count(MemoryRetrievalHit.hit_id)).filter( + MemoryRetrievalHit.tenant_id == tenant_id, + ) + if user_id is not None: + query = query.filter(MemoryRetrievalHit.user_id == user_id) + if agent_id is not None: + query = query.filter(MemoryRetrievalHit.agent_id == agent_id) + if since is not None: + query = query.filter( + MemoryRetrievalHit.occurred_at >= since + ) + return int(query.scalar() or 0) + except Exception: + session.rollback() + logger.exception("count_hits_since failed") + return 0 + + +def list_hits_for_memory( + memory_id: int, + *, + since: Optional[datetime] = None, + limit: int = 1000, +) -> List[Dict[str, Any]]: + """Return hit rows for a single memory, ordered chronologically.""" + with get_db_session() as session: + try: + query = session.query(MemoryRetrievalHit).filter( + MemoryRetrievalHit.memory_id == memory_id, + ) + if since is not None: + query = query.filter(MemoryRetrievalHit.occurred_at >= since) + query = query.order_by(MemoryRetrievalHit.occurred_at.asc()).limit(limit) + return [_hit_to_dict(row) for row in query.all()] + except Exception: + session.rollback() + logger.exception("list_hits_for_memory failed") + return [] + + +def list_hits_for_user( + tenant_id: str, + user_id: str, + *, + since: Optional[datetime] = None, + limit: int = 5000, +) -> List[Dict[str, Any]]: + """Return hit rows for a user (used by Dreaming aggregation).""" + with get_db_session() as session: + try: + query = session.query(MemoryRetrievalHit).filter( + MemoryRetrievalHit.tenant_id == tenant_id, + MemoryRetrievalHit.user_id == user_id, + ) + if since is not None: + query = query.filter(MemoryRetrievalHit.occurred_at >= since) + query = query.order_by(MemoryRetrievalHit.occurred_at.asc()).limit(limit) + return [_hit_to_dict(row) for row in query.all()] + except Exception: + session.rollback() + logger.exception("list_hits_for_user failed") + return [] + + +def aggregate_memory_stats( + tenant_id: str, + *, + user_id: Optional[str] = None, + agent_id: Optional[str] = None, + since: Optional[datetime] = None, +) -> List[Dict[str, Any]]: + """Aggregate recall statistics grouped by memory_id. + + Returns rows shaped like: + ``{"memory_id": int, "hit_count": int, "grounded_count": int, + "days": set[str], "query_hashes": set[str]}`` + """ + with get_db_session() as session: + try: + query = session.query( + MemoryRetrievalHit.memory_id, + func.count(MemoryRetrievalHit.hit_id).label("hit_count"), + func.sum( + func.cast(MemoryRetrievalHit.grounded, Integer) + ).label("grounded_count"), + ).filter(MemoryRetrievalHit.tenant_id == tenant_id) + if user_id is not None: + query = query.filter(MemoryRetrievalHit.user_id == user_id) + if agent_id is not None: + query = query.filter(MemoryRetrievalHit.agent_id == agent_id) + if since is not None: + query = query.filter(MemoryRetrievalHit.occurred_at >= since) + query = query.group_by(MemoryRetrievalHit.memory_id) + raw_rows = query.all() + except Exception: + session.rollback() + logger.exception("aggregate_memory_stats failed") + return [] + + out: List[Dict[str, Any]] = [] + for memory_id, hit_count, grounded_count in raw_rows: + if memory_id is None: + continue + hits = list_hits_for_memory(memory_id, since=since, limit=10000) + out.append( + { + "memory_id": memory_id, + "hit_count": int(hit_count or 0), + "grounded_count": int(grounded_count or 0), + "days": {hit["day"] for hit in hits if hit.get("day")}, + "query_hashes": { + hit["query_hash"] + for hit in hits + if hit.get("query_hash") + }, + } + ) + return out + + +def delete_hits_before(cutoff: datetime) -> int: + """Delete hit rows older than ``cutoff`` (housekeeping).""" + with get_db_session() as session: + try: + rows = ( + session.query(MemoryRetrievalHit) + .filter(MemoryRetrievalHit.occurred_at < cutoff) + .delete(synchronize_session=False) + ) + session.commit() + return int(rows or 0) + except Exception: + session.rollback() + logger.exception("delete_hits_before failed") + return 0 + + +# --------------------------------------------------------------------------- +# Internal helpers +# --------------------------------------------------------------------------- + + +def _hit_to_dict(row: MemoryRetrievalHit) -> Dict[str, Any]: + return { + "hit_id": row.hit_id, + "tenant_id": row.tenant_id, + "user_id": row.user_id, + "agent_id": row.agent_id, + "conversation_id": row.conversation_id, + "memory_id": row.memory_id, + "query_text": row.query_text, + "query_hash": row.query_hash, + "retrieval_score": float(row.retrieval_score) if row.retrieval_score is not None else None, + "source": row.source, + "occurred_at": row.occurred_at, + "day": row.day, + "grounded": bool(row.grounded), + } \ No newline at end of file diff --git a/backend/database/model_management_db.py b/backend/database/model_management_db.py index 64b00b9f37..16c76bd2d9 100644 --- a/backend/database/model_management_db.py +++ b/backend/database/model_management_db.py @@ -183,10 +183,12 @@ def get_model_by_display_name(display_name: str, tenant_id: str, model_type: str """ filters = {'display_name': display_name} - if model_type in ["multiEmbedding", "multi_embedding"]: - filters['model_type'] = "multi_embedding" - elif model_type == "embedding": - filters['model_type'] = "embedding" + if model_type: + filters['model_type'] = ( + "multi_embedding" + if model_type in ["multiEmbedding", "multi_embedding"] + else model_type + ) records = get_model_records(filters, tenant_id) if not records: @@ -358,5 +360,3 @@ def get_model_by_name_factory(model_name: str, model_factory: str, tenant_id: st } records = get_model_records(filters, tenant_id) return records[0] if records else None - - diff --git a/backend/database/notification_db.py b/backend/database/notification_db.py new file mode 100644 index 0000000000..3eeff3bc11 --- /dev/null +++ b/backend/database/notification_db.py @@ -0,0 +1,261 @@ +""" +Database operations for in-app notifications. + +Notifications use a fan-out model: one row in notification_t holds the message, +and one row per resolved receiver is written to notification_receiver_t. +""" +import logging +from typing import Any, Dict, List, Optional + +from consts.notification import ( + VALID_EVENT_TYPES, + VALID_RESOURCE_TYPES, + VALID_NOTIFICATION_SCOPES, + TENANT_REQUIRED_SCOPES, + SCOPE_SU, + SCOPE_TENANT, + SCOPE_TENANT_ADMIN, + SCOPE_TENANT_USER, + SCOPE_USER, + SU_ROLES, + TENANT_ADMIN_ROLES, + TENANT_USER_ROLES, +) +from database.client import get_db_session +from database.db_models import Notification, NotificationReceiver, UserTenant + +logger = logging.getLogger(__name__) + + +def _resolve_receivers( + session, + scope: str, + tenant_id: Optional[str], + receiver_user_id: Optional[str] = None, +) -> List[Dict[str, str]]: + """Resolve (user_id, tenant_id) receivers for a scope. Deduplicated by user_id.""" + if scope == SCOPE_USER: + return [{"user_id": receiver_user_id, "tenant_id": tenant_id}] + + query = session.query(UserTenant.user_id, UserTenant.tenant_id).filter( + UserTenant.delete_flag == "N" + ) + if scope == SCOPE_SU: + query = query.filter(UserTenant.user_role.in_(SU_ROLES)) + elif scope == SCOPE_TENANT: + query = query.filter(UserTenant.tenant_id == tenant_id) + elif scope == SCOPE_TENANT_ADMIN: + query = query.filter( + UserTenant.tenant_id == tenant_id, + UserTenant.user_role.in_(TENANT_ADMIN_ROLES), + ) + elif scope == SCOPE_TENANT_USER: + query = query.filter( + UserTenant.tenant_id == tenant_id, + UserTenant.user_role.in_(TENANT_USER_ROLES), + ) + + seen: set = set() + receivers: List[Dict[str, str]] = [] + for user_id, uid_tenant in query.all(): + if user_id in seen: + continue + seen.add(user_id) + receivers.append({"user_id": user_id, "tenant_id": uid_tenant}) + return receivers + + +def create_notification(*, event_type: str, resource_type: str, scope: str, + details: Optional[dict] = None, + tenant_id: Optional[str] = None, + receiver_user_id: Optional[str] = None, + unique_id: Optional[int] = None, + created_by: Optional[str] = None) -> Dict[str, Any]: + """Create a notification and fan-out receiver rows based on scope. + + Args: + event_type: Event type identifier (e.g. repository_review_approved). + resource_type: Resource type identifier (e.g. agent_repository). + scope: Audience scope (SU / TENANT / TENANT_ADMIN / TENANT_USER / USER). + details: i18n interpolation details for the event template. + tenant_id: Target tenant; required for tenant-scoped notifications. + receiver_user_id: Target user; required for the USER scope. + unique_id: Related resource primary key (e.g. agent_repository_id). + created_by: Actor who created the notification. + + Returns: + Dict with the new notification_id and the number of receiver rows. + """ + if event_type not in VALID_EVENT_TYPES: + raise ValueError(f"Invalid event_type: {event_type}") + if resource_type not in VALID_RESOURCE_TYPES: + raise ValueError(f"Invalid resource_type: {resource_type}") + if scope not in VALID_NOTIFICATION_SCOPES: + raise ValueError(f"Invalid notification scope: {scope}") + if scope in TENANT_REQUIRED_SCOPES and not tenant_id: + raise ValueError(f"tenant_id is required for scope {scope}") + if scope == SCOPE_USER and not receiver_user_id: + raise ValueError(f"receiver_user_id is required for scope {scope}") + + with get_db_session() as session: + notification = Notification( + event_type=event_type, + resource_type=resource_type, + unique_id=unique_id, + details=details, + scope=scope, + tenant_id=tenant_id, + is_active=True, + created_by=created_by, + updated_by=created_by, + delete_flag="N", + ) + session.add(notification) + session.flush() # obtain notification_id + notification_id = int(notification.notification_id) + + receivers = _resolve_receivers(session, scope, tenant_id, receiver_user_id) + session.add_all([ + NotificationReceiver( + notification_id=notification_id, + receiver_user_id=receiver["user_id"], + tenant_id=receiver["tenant_id"], + is_read=False, + created_by=created_by, + updated_by=created_by, + delete_flag="N", + ) + for receiver in receivers + ]) + return {"notification_id": notification_id, "receiver_count": len(receivers)} + + +def deactivate_notifications( + *, + event_type: str, + resource_type: str, + unique_id: int, + updated_by: Optional[str] = None, +) -> int: + """Deactivate active notifications matching event_type + resource_type + unique_id. + + Args: + event_type: Event type identifier. + resource_type: Resource type identifier. + unique_id: Related resource primary key. + updated_by: Actor who deactivated the notifications. + + Returns: + Number of notification rows updated. + """ + with get_db_session() as session: + return session.query(Notification).filter( + Notification.event_type == event_type, + Notification.resource_type == resource_type, + Notification.unique_id == unique_id, + Notification.is_active.is_(True), + Notification.delete_flag != "Y", + ).update( + {"is_active": False, "updated_by": updated_by}, + synchronize_session=False, + ) + + +def list_notifications_by_user( + user_id: str, + *, + only_unread: bool = False, + page: int = 1, + page_size: int = 10, +) -> Dict[str, Any]: + """List a user's notifications (joined with message body), newest first.""" + with get_db_session() as session: + base = session.query(NotificationReceiver, Notification).join( + Notification, + Notification.notification_id == NotificationReceiver.notification_id, + ).filter( + NotificationReceiver.receiver_user_id == user_id, + NotificationReceiver.delete_flag != "Y", + Notification.delete_flag != "Y", + Notification.is_active.is_(True), + ) + if only_unread: + base = base.filter(NotificationReceiver.is_read.is_(False)) + + total = base.count() + rows = ( + base.order_by(Notification.create_time.desc()) + .offset((page - 1) * page_size) + .limit(page_size) + .all() + ) + + return { + "items": [ + { + "receiver_id": recv.receiver_id, + "notification_id": recv.notification_id, + "event_type": notif.event_type, + "resource_type": notif.resource_type, + "details": notif.details, + "scope": notif.scope, + "is_read": recv.is_read, + "create_time": recv.create_time, + } + for recv, notif in rows + ], + "pagination": { + "page": page, + "page_size": page_size, + "total": total, + "total_pages": (total + page_size - 1) // page_size if total else 0, + }, + } + + +def count_unread_by_user(user_id: str) -> int: + """Count unread active notifications for a user.""" + with get_db_session() as session: + return session.query(NotificationReceiver).join( + Notification, + Notification.notification_id == NotificationReceiver.notification_id, + ).filter( + NotificationReceiver.receiver_user_id == user_id, + NotificationReceiver.is_read.is_(False), + NotificationReceiver.delete_flag != "Y", + Notification.is_active.is_(True), + Notification.delete_flag != "Y", + ).count() + + +def mark_notifications_read( + user_id: str, + *, + mark_all: bool = False, + receiver_id: Optional[int] = None, +) -> int: + """Mark notification receiver rows as read for the given user. + + Always scopes updates to receiver_user_id == user_id so callers cannot + mark another user's notifications. + + Args: + user_id: Authenticated receiver user ID. + mark_all: If True, mark all unread rows for the user. + receiver_id: Specific receiver row to mark when mark_all is False. + + Returns: + Number of rows updated. + """ + with get_db_session() as session: + query = session.query(NotificationReceiver).filter( + NotificationReceiver.receiver_user_id == user_id, + NotificationReceiver.is_read.is_(False), + NotificationReceiver.delete_flag != "Y", + ) + if not mark_all: + query = query.filter(NotificationReceiver.receiver_id == receiver_id) + return query.update( + {"is_read": True, "updated_by": user_id}, + synchronize_session=False, + ) diff --git a/backend/database/remote_mcp_db.py b/backend/database/remote_mcp_db.py index d6223628f6..4359f465d9 100644 --- a/backend/database/remote_mcp_db.py +++ b/backend/database/remote_mcp_db.py @@ -20,7 +20,8 @@ def create_mcp_record(mcp_data: Dict[str, Any], tenant_id: str, user_id: str): allowed_fields = { 'mcp_name', 'mcp_server', 'status', 'container_id', 'container_port', 'authorization_token', 'custom_headers', 'source', 'market_id', - 'registry_json', 'config_json', 'enabled', 'tags', 'description' + 'registry_json', 'config_json', 'enabled', 'tags', 'description', + 'group_ids', 'ingroup_permission', 'shared_fields' } filtered_data = {k: v for k, v in mcp_data.items() if k in allowed_fields and v is not None} @@ -139,26 +140,34 @@ def update_mcp_record_manage_fields_by_id( custom_headers: Dict[str, Any] | None, config_json: Dict[str, Any] | None, market_id: int | None, + group_ids: str | None = None, + ingroup_permission: str | None = None, + shared_fields: dict | None = None, ) -> None: with get_db_session() as session: + update_data = { + "mcp_name": name, + "mcp_server": server_url, + "description": description, + "tags": tags or [], + "source": source, + "authorization_token": authorization_token, + "custom_headers": custom_headers, + "config_json": config_json, + "market_id": market_id, + "updated_by": user_id, + } + if group_ids is not None: + update_data["group_ids"] = group_ids + if ingroup_permission is not None: + update_data["ingroup_permission"] = ingroup_permission + if shared_fields is not None: + update_data["shared_fields"] = shared_fields session.query(McpRecord).filter( McpRecord.mcp_id == mcp_id, McpRecord.tenant_id == tenant_id, McpRecord.delete_flag != 'Y' - ).update( - { - "mcp_name": name, - "mcp_server": server_url, - "description": description, - "tags": tags or [], - "source": source, - "authorization_token": authorization_token, - "custom_headers": custom_headers, - "config_json": config_json, - "market_id": market_id, - "updated_by": user_id, - } - ) + ).update(update_data) def update_mcp_record_market_id_by_id( diff --git a/backend/database/skill_db.py b/backend/database/skill_db.py index 8f51849bbe..57849e11f7 100644 --- a/backend/database/skill_db.py +++ b/backend/database/skill_db.py @@ -5,11 +5,12 @@ from datetime import datetime from typing import Any, Dict, List, Optional -from sqlalchemy import update as sa_update +from sqlalchemy import or_, update as sa_update from database.client import get_db_session, filter_property, as_dict from database.db_models import SkillInfo, SkillToolRelation, SkillInstance, ToolInfo from utils.skill_params_utils import strip_params_comments_for_db +from utils.str_utils import convert_list_to_string, convert_string_to_list logger = logging.getLogger(__name__) @@ -80,6 +81,34 @@ def query_skill_instances_by_agent_id(agent_id: int, tenant_id: str, version_no: return [as_dict(skill_instance) for skill_instance in skill_instances] +def get_valid_skill_ids(tenant_id: str, skill_ids: List[int]) -> set: + """Return skill IDs that still exist in ag_skill_info_t and are not soft-deleted. + + Checks both tenant-scoped skills and global (tenant_id IS NULL) skills. + + Used as a fallback check when skill instances may reference deleted skills. + + Args: + tenant_id: Tenant ID for filtering tenant-scoped skills. + skill_ids: Candidate skill IDs to validate. + + Returns: + Set of valid (non-deleted) skill IDs. + """ + if not skill_ids: + return set() + with get_db_session() as session: + rows = session.query(SkillInfo.skill_id).filter( + SkillInfo.skill_id.in_(skill_ids), + SkillInfo.delete_flag != 'Y', + or_( + SkillInfo.tenant_id == tenant_id, + SkillInfo.tenant_id.is_(None), + ), + ).all() + return {row[0] for row in rows} + + def query_enabled_skill_instances(agent_id: int, tenant_id: str, version_no: int = 0): """Query enabled SkillInstance in the database.""" with get_db_session() as session: @@ -203,10 +232,16 @@ def _build_skill_update_values( "content": "skill_content", "tags": "skill_tags", "source": "source", + "ingroup_permission": "ingroup_permission", } for input_field, model_field in field_mapping.items(): if input_field in skill_data: row_values[model_field] = skill_data[input_field] + if "group_ids" in skill_data: + group_ids = skill_data["group_ids"] + row_values["group_ids"] = ( + convert_list_to_string(group_ids) if isinstance(group_ids, list) else group_ids + ) for field in ("config_schemas", "config_values"): if field in skill_data: @@ -218,6 +253,7 @@ def _replace_skill_tool_relations( session, skill_id: int, tool_ids: List[int], + updated_by: Optional[str] = None, ) -> None: session.query(SkillToolRelation).filter( SkillToolRelation.skill_id == skill_id @@ -226,7 +262,10 @@ def _replace_skill_tool_relations( session.add(SkillToolRelation( skill_id=skill_id, tool_id=tool_id, + created_by=updated_by, create_time=datetime.now(), + updated_by=updated_by, + update_time=datetime.now(), )) @@ -242,6 +281,8 @@ def _to_dict(skill: SkillInfo) -> Dict[str, Any]: "config_schemas": skill.config_schemas, "config_values": skill.config_values, "source": skill.source, + "group_ids": convert_string_to_list(skill.group_ids), + "ingroup_permission": skill.ingroup_permission, "created_by": skill.created_by, "create_time": skill.create_time.isoformat() if skill.create_time else None, "updated_by": skill.updated_by, @@ -259,6 +300,9 @@ def list_skills(tenant_id: str) -> List[Dict[str, Any]]: skills = session.query(SkillInfo).filter( SkillInfo.tenant_id == tenant_id, SkillInfo.delete_flag != 'Y' + ).order_by( + SkillInfo.create_time.desc(), + SkillInfo.skill_id.desc(), ).all() results = [] for s in skills: @@ -268,6 +312,29 @@ def list_skills(tenant_id: str) -> List[Dict[str, Any]]: return results +def list_skill_permission_summaries(tenant_id: str) -> List[Dict[str, Any]]: + """List only the fields required to resolve skill visibility and ownership.""" + with get_db_session() as session: + rows = session.query( + SkillInfo.skill_id, + SkillInfo.created_by, + SkillInfo.group_ids, + SkillInfo.ingroup_permission, + ).filter( + SkillInfo.tenant_id == tenant_id, + SkillInfo.delete_flag != 'Y', + ).all() + return [ + { + "skill_id": row.skill_id, + "created_by": row.created_by, + "group_ids": convert_string_to_list(row.group_ids), + "ingroup_permission": row.ingroup_permission, + } + for row in rows + ] + + def get_skill_by_name(skill_name: str, tenant_id: str) -> Optional[Dict[str, Any]]: """Get skill by name within a tenant. @@ -368,6 +435,12 @@ def create_skill(skill_data: Dict[str, Any], tenant_id: str) -> Dict[str, Any]: config_values=_params_value_for_db( skill_data.get("config_values")), source=skill_data.get("source", "custom"), + group_ids=( + convert_list_to_string(skill_data.get("group_ids")) + if isinstance(skill_data.get("group_ids"), list) + else skill_data.get("group_ids") + ), + ingroup_permission=skill_data.get("ingroup_permission"), created_by=skill_data.get("created_by"), create_time=datetime.now(), updated_by=skill_data.get("updated_by"), @@ -380,11 +453,16 @@ def create_skill(skill_data: Dict[str, Any], tenant_id: str) -> Dict[str, Any]: tool_ids = skill_data.get("tool_ids", []) if tool_ids: + relation_created_by = skill_data.get("created_by") or skill_data.get("updated_by") + relation_updated_by = skill_data.get("updated_by") or relation_created_by for tool_id in tool_ids: rel = SkillToolRelation( skill_id=skill_id, tool_id=tool_id, - create_time=datetime.now() + created_by=relation_created_by, + create_time=datetime.now(), + updated_by=relation_updated_by, + update_time=datetime.now(), ) session.add(rel) @@ -441,6 +519,7 @@ def update_skill( session, skill_id, skill_data["tool_ids"], + updated_by=updated_by, ) session.commit() @@ -509,6 +588,7 @@ def update_skill_by_id( session, skill_id, skill_data["tool_ids"], + updated_by=updated_by, ) session.commit() diff --git a/backend/database/skill_repository_db.py b/backend/database/skill_repository_db.py index e999974762..7647cd22a8 100644 --- a/backend/database/skill_repository_db.py +++ b/backend/database/skill_repository_db.py @@ -19,6 +19,7 @@ "skill_info_json", "skill_zip_base64", "status", + "content", }) @@ -66,6 +67,7 @@ def get_skill_repository_by_skill_id( skill_id: int, *, publisher_tenant_id: Optional[str] = None, + statuses: Optional[Collection[str]] = None, ) -> Optional[dict]: """Fetch an active repository listing by source skill_id.""" with get_db_session() as session: @@ -77,7 +79,9 @@ def get_skill_repository_by_skill_id( query = query.filter( SkillRepository.publisher_tenant_id == publisher_tenant_id, ) - record = query.first() + if statuses is not None: + query = query.filter(SkillRepository.status.in_(list(statuses))) + record = query.order_by(SkillRepository.update_time.desc()).first() return as_dict(record) if record else None @@ -133,6 +137,7 @@ def list_skill_repository_summaries( query = session.query( SkillRepository.skill_repository_id, SkillRepository.skill_id, + SkillRepository.publisher_user_id, SkillRepository.submitted_by, SkillRepository.name, SkillRepository.description, @@ -142,6 +147,7 @@ def list_skill_repository_summaries( SkillRepository.tags, SkillRepository.icon, SkillRepository.downloads, + SkillRepository.content, SkillRepository.create_time, ) query = _apply_skill_repository_filters( @@ -171,6 +177,7 @@ def list_skill_repository_summaries( { "skill_repository_id": row.skill_repository_id, "skill_id": row.skill_id, + "publisher_user_id": row.publisher_user_id, "submitted_by": row.submitted_by, "name": row.name, "description": row.description, @@ -180,6 +187,7 @@ def list_skill_repository_summaries( "tags": row.tags or [], "icon": row.icon, "downloads": row.downloads or 0, + "content": row.content, "created_at": row.create_time.isoformat() if row.create_time else None, } for row in rows @@ -235,6 +243,7 @@ def update_skill_repository_status_by_id( publisher_tenant_id: Optional[str] = None, publisher_user_id: Optional[str] = None, submitted_by: Optional[str] = None, + content: Optional[str] = None, ) -> int: """Update repository listing status by primary key. Returns affected row count.""" update_values: Dict[str, Any] = { @@ -247,6 +256,8 @@ def update_skill_repository_status_by_id( update_values["publisher_user_id"] = publisher_user_id if submitted_by is not None: update_values["submitted_by"] = submitted_by + if content is not None: + update_values["content"] = content with get_db_session() as session: where_clauses = [ @@ -265,6 +276,29 @@ def update_skill_repository_status_by_id( return int(result.rowcount or 0) +def reset_skill_repository_status( + *, + repository_id: int, + skill_id: int, + status: str, + publisher_tenant_id: str, +) -> int: + """Set other active listings with the same skill and status to not_shared.""" + with get_db_session() as session: + result = session.execute( + update(SkillRepository) + .where( + SkillRepository.skill_id == skill_id, + SkillRepository.status == status, + SkillRepository.skill_repository_id != repository_id, + SkillRepository.publisher_tenant_id == publisher_tenant_id, + SkillRepository.delete_flag != "Y", + ) + .values(status=STATUS_NOT_SHARED) + ) + return int(result.rowcount or 0) + + def increment_skill_repository_downloads( *, repository_id: int, @@ -307,6 +341,7 @@ def list_skill_repository_by_skill_ids( SkillRepository.skill_repository_id, SkillRepository.skill_id, SkillRepository.status, + SkillRepository.content, SkillRepository.create_time, ) .filter( @@ -327,6 +362,7 @@ def list_skill_repository_by_skill_ids( "skill_repository_id": row.skill_repository_id, "skill_id": row.skill_id, "status": row.status, + "content": row.content, "create_time": row.create_time, } for row in rows diff --git a/backend/database/tool_db.py b/backend/database/tool_db.py index 81ef89eefb..f7d1d0aefc 100644 --- a/backend/database/tool_db.py +++ b/backend/database/tool_db.py @@ -1,6 +1,6 @@ import re import json -from typing import List +from typing import List, Optional from database.agent_db import logger from database.client import get_db_session, filter_property, as_dict from database.db_models import ToolInstance, ToolInfo @@ -251,11 +251,27 @@ def check_tool_list_initialized(tenant_id: str) -> bool: return count > 0 -def update_tool_table_from_scan_tool_list(tenant_id: str, user_id: str, tool_list: List[ToolInfo]): +def update_tool_table_from_scan_tool_list( + tenant_id: str, + user_id: str, + tool_list: List[ToolInfo], + enabled_mcp_names: Optional[set] = None, +): """ scan all tools and update the tool table in PG database, remove the duplicate tools For MCP tools, use name&source&usage as unique key to allow same tool name from different MCP servers + + Args: + tenant_id: Tenant ID + user_id: User ID who triggered the scan + tool_list: Tools successfully gathered during this scan + enabled_mcp_names: Names of MCP services that are enabled. Tools belonging + to these MCPs keep their previous availability when the MCP was + unreachable during this scan — a single transient connection failure + (e.g. a container still warming up) should not hide a healthy MCP's + tools from the tool list. Only disabled/deleted MCPs get downgraded. """ + enabled_mcp_names = enabled_mcp_names or set() with get_db_session() as session: # get all existing tools (including complete information) existing_tools = session.query(ToolInfo).filter(ToolInfo.delete_flag != 'Y', @@ -271,8 +287,15 @@ def update_tool_table_from_scan_tool_list(tenant_id: str, user_id: str, tool_lis key = f"{tool.name}&{tool.source}" existing_tool_dict[key] = tool - # set all tools to unavailable + # Set tools to unavailable. Tools belonging to enabled MCPs keep their + # previous availability so a transient fetch failure this scan does not + # remove a healthy MCP's tools from the tool list. for tool in existing_tools: + if ( + tool.source == ToolSourceEnum.MCP.value + and (tool.usage or "") in enabled_mcp_names + ): + continue tool.is_available = False for tool in tool_list: @@ -310,6 +333,32 @@ def update_tool_table_from_scan_tool_list(tenant_id: str, user_id: str, tool_lis logger.info("Updated tool table in PG database") +def set_mcp_tools_unavailable(*, tenant_id: str, mcp_server_name: str, user_id: str) -> int: + """ + Mark all tool records belonging to a deleted MCP server as unavailable. + + Keeps the tool rows in place (agents may still reference them via tool + instances) but hides them from the agent tool selection list, which only + shows tools where is_available != False. + + Args: + tenant_id: Tenant ID (stored as ToolInfo.author) + mcp_server_name: MCP server name (stored as ToolInfo.usage) + user_id: User ID performing the deletion + + Returns: + Number of tool records updated + """ + with get_db_session() as session: + updated = session.query(ToolInfo).filter( + ToolInfo.delete_flag != 'Y', + ToolInfo.author == tenant_id, + ToolInfo.source == ToolSourceEnum.MCP.value, + ToolInfo.usage == mcp_server_name, + ).update({"is_available": False, "updated_by": user_id}) + return updated or 0 + + def add_tool_field(tool_info): with get_db_session() as session: # Query if there is an existing ToolInstance diff --git a/backend/database/user_tenant_db.py b/backend/database/user_tenant_db.py index b147eac491..9578f44737 100644 --- a/backend/database/user_tenant_db.py +++ b/backend/database/user_tenant_db.py @@ -11,6 +11,27 @@ logger = logging.getLogger(__name__) +def get_user_role_by_tenant(user_id: str, tenant_id: str) -> str: + """Return the user's role within the given tenant. + + Joins ``user_tenant_t`` by ``(user_id, tenant_id)`` so the result is + strictly tenant-scoped. Returns the empty string when no active row + exists; callers should treat that as "no role". + """ + if not user_id or not tenant_id: + return "" + with get_db_session() as session: + result = session.query(UserTenant).filter( + UserTenant.user_id == user_id, + UserTenant.tenant_id == tenant_id, + UserTenant.delete_flag == "N", + ).first() + # Access ORM attributes INSIDE the session context — the session + # closes on context-manager exit and lazy-loaded attributes are not + # reachable after that (would raise DetachedInstanceError). + return (result.user_role or "") if result is not None else "" + + def get_user_tenant_by_user_id(user_id: str) -> Optional[Dict[str, Any]]: """ Get user tenant relationship by user ID @@ -32,6 +53,25 @@ def get_user_tenant_by_user_id(user_id: str) -> Optional[Dict[str, Any]]: return None +def get_user_email_map(user_ids: List[str]) -> Dict[str, str]: + """Return active user email addresses keyed by user ID.""" + unique_user_ids = list({user_id for user_id in user_ids if user_id}) + if not unique_user_ids: + return {} + + with get_db_session() as session: + rows = session.query(UserTenant.user_id, UserTenant.user_email).filter( + UserTenant.user_id.in_(unique_user_ids), + UserTenant.delete_flag == "N", + ).all() + + return { + user_id: user_email + for user_id, user_email in rows + if user_email + } + + def get_all_tenant_ids() -> list[str]: """ Get all unique tenant IDs from the database diff --git a/docker/init.sql b/backend/ext_components/__init__.py similarity index 100% rename from docker/init.sql rename to backend/ext_components/__init__.py diff --git a/backend/ext_components/aidp/__init__.py b/backend/ext_components/aidp/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/backend/ext_components/aidp/apps/__init__.py b/backend/ext_components/aidp/apps/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/backend/ext_components/aidp/apps/aidp_mgmt_app.py b/backend/ext_components/aidp/apps/aidp_mgmt_app.py new file mode 100644 index 0000000000..d233e8bd20 --- /dev/null +++ b/backend/ext_components/aidp/apps/aidp_mgmt_app.py @@ -0,0 +1,575 @@ +"""AIDP Management App Layer (v7.1). + +FastAPI endpoints for AIDP knowledge base CRUD with permission enforcement. + +* Every handler calls :func:`_auth` to resolve ``(user_id, tenant_id)`` from + the ``Authorization`` header. Missing or invalid auth raises 401. +* Resource-level operations call :func:`require_permission` to enforce + the v7.1 permission matrix and raise 403/404 when violated. +* Creation is idempotent: the AIDP call uses ``kds_id`` returned from AIDP + as the dedup key; collisions surface as 409 without compensating deletes. +* KB metadata is fetched lazily for the visible page only; failures mark + ``resource_status = UNAVAILABLE`` so the frontend can render the row + gracefully instead of treating it as a hard error. +""" +from __future__ import annotations + +import logging +from http import HTTPStatus +from typing import Annotated, List, Optional + +from fastapi import APIRouter, File, Path, Query, Request, UploadFile +from fastapi.responses import JSONResponse +from pydantic import BaseModel, Field +from sqlalchemy.exc import IntegrityError + +from consts.const import AIDP_API_KEY, AIDP_SERVER_URL +from consts.error_code import ErrorCode +from consts.exceptions import AppException, UnauthorizedError +from ext_components.aidp.consts.aidp_exceptions import ( + AidpKbConflictError, + AidpKbNotFoundError, + AidpKbPermissionDeniedError, + AidpKbSyncError, + AidpGroupValidationError, +) +from ext_components.aidp.database import aidp_permission_db +from ext_components.aidp.services import aidp_permission_service as perms +from ext_components.aidp.services.aidp_service import ( + count_aidp_docs_impl, + create_aidp_kb_impl, + delete_aidp_kb_impl, + get_aidp_kb_impl, + list_aidp_docs_impl, + list_aidp_models_impl, + update_aidp_kb_impl, + upload_aidp_docs_impl, +) +from ext_components.aidp.services.aidp_permission_service import ( + EDIT, + READ_ONLY, + _validate_group_ids_strict, +) +from utils import auth_utils as auth_utils_module + +aidp_mgmt_router = APIRouter(prefix="/aidp-mgmt") +logger = logging.getLogger("aidp_mgmt_app") + + +# --------------------------------------------------------------------------- +# Request Models +# --------------------------------------------------------------------------- + + +class CreateKbRequest(BaseModel): + """Request body for creating a knowledge base.""" + + name: str = Field(..., description="Knowledge base name (required)") + description: Optional[str] = Field(None, description="Knowledge base description") + embedding_model: Optional[str] = Field(None, description="Embedding model identifier") + is_multimodal: Optional[bool] = Field(None, description="Whether KB supports multimodal content") + vision_model: Optional[str] = Field(None, description="Vision model identifier for multimodal KBs") + chunk_token_num: Optional[int] = Field(None, description="Chunk size in tokens (> 0)") + chunk_overlap_num: Optional[int] = Field(None, description="Chunk overlap in tokens (>= 0)") + vlm_model: Optional[str] = Field(None, description="VLM model identifier for caption generation") + is_personal: Optional[int] = Field(None, ge=0, le=1, description="Personal KB flag, int 0 or 1") + topk: Optional[int] = Field(None, description="Top-K retrieval count") + similarity: Optional[float] = Field(None, description="Similarity score threshold") + smartsplit: Optional[int] = Field(None, ge=0, le=1, description="Smart chunking mode, int 0 or 1") + caption_enable: Optional[int] = Field(None, ge=0, le=1, description="Caption generation toggle, int 0 or 1") + # Nexent-side permission payload. Never forwarded to AIDP. + ingroup_permission: Optional[str] = Field( + "READ_ONLY", + description="Permission level for authorised groups: EDIT / READ_ONLY / PRIVATE", + ) + group_ids: Optional[List[int]] = Field( + None, + description="Group IDs granted the in-group permission. Empty/ignored when PRIVATE.", + ) + + +class UpdateKbRequest(BaseModel): + """Request body for updating a knowledge base.""" + + name: Optional[str] = Field(None, description="Knowledge base name") + description: Optional[str] = Field(None, description="Knowledge base description") + + +class SetPermissionRequest(BaseModel): + """Request body for setting a KB's group-level permission. + + The AIDP platform is not invoked; the change is purely a local table + write that controls who can see the KB in subsequent list/search calls. + """ + + ingroup_permission: str = Field(..., description="EDIT / READ_ONLY / PRIVATE") + group_ids: Optional[List[int]] = Field( + None, + description="Group IDs granted the in-group permission. Ignored when PRIVATE.", + ) + + +# --------------------------------------------------------------------------- +# Auth helpers +# --------------------------------------------------------------------------- + + +async def _auth(request: Request) -> tuple[str, str]: + """Resolve ``(user_id, tenant_id)`` from the Authorization header. + + Raises 401 for missing/invalid tokens or empty tenant contexts so the + caller never has to defend against partially-authenticated state. + """ + auth = request.headers.get("Authorization") + if not auth: + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail="Missing Authorization header") + try: + user_id, tenant_id = auth_utils_module.get_current_user_id(auth) + except UnauthorizedError as exc: + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail=str(exc)) + if not tenant_id: + raise HTTPException(status_code=HTTPStatus.UNAUTHORIZED, detail="No tenant context") + return user_id, tenant_id + + +def _infer_is_multimodal(detail: dict) -> bool: + """Reverse-derive ``is_multimodal`` from AIDP detail response. + + AIDP does not return an ``is_multimodal`` field — it is a Nexent-side + concept. On create the SDK mapper translates it one-to-one into + ``caption_enable`` (``sdk/nexent/core/knowledge_base/mapper.py``): + + caption_enable = 1 if is_multimodal else DEFAULT_CAPTION_ENABLE + + So the reverse mapping only needs to inspect ``caption_enable``. The + ``vlm_model`` field is a separate, optional identifier that the user + may or may not supply — we deliberately do NOT gate on it being + non-empty, because (a) the user can choose any VLM model from the + AIDP catalog (not a fixed name) and (b) AIDP may not even return + the field for a given KB. + + Returns ``True`` iff ``caption_enable ∈ {1, "1", True}``. + """ + if not isinstance(detail, dict): + return False + caption = detail.get("caption_enable") + return caption in (1, "1", True) + + +def _raise_aidp_conflict(exc: IntegrityError) -> None: + """Translate a unique-index violation into an HTTP 409 conflict.""" + logger.warning("AIDP permission unique constraint violated: %s", exc) + raise HTTPException( + status_code=HTTPStatus.CONFLICT, + detail="Knowledge base already exists for this tenant", + ) + + +# HTTPException is imported lazily to keep FastAPI's exception handler in +# control of the response body. +from fastapi import HTTPException # noqa: E402 (placed here to avoid editing mid-file) + + +def _credentials() -> tuple[str, str]: + return AIDP_SERVER_URL, AIDP_API_KEY + + +# --------------------------------------------------------------------------- +# Permission-aware helpers +# --------------------------------------------------------------------------- + + +def _serialize_permission(decision) -> dict: + return { + "permission": decision.permission, + "matched_group_ids": list(decision.matched_group_ids), + "is_management_role": decision.is_management_role, + } + + +# --------------------------------------------------------------------------- +# Handlers +# --------------------------------------------------------------------------- + + +@aidp_mgmt_router.get("/knowledge-bases") +async def list_knowledge_bases( + request: Request, + page: Annotated[int, Query(ge=1, description="Page number starting from 1")] = 1, + page_size: Annotated[int, Query(ge=1, le=100, description="Page size from 1 to 100")] = 10, +) -> JSONResponse: + """List KBs the caller can access. + + Resolution order: + 1. Read active rows from the local DB for the tenant (tenant + active + filter ensures we never leak across tenants). + 2. For each row, compute the effective permission using the role + + ownership + group intersection matrix. + 3. Fetch AIDP-side metadata lazily for the visible page only; failures + mark ``resource_status = UNAVAILABLE`` rather than failing the list. + """ + user_id, tenant_id = await _auth(request) + + total_count = perms.count_accessible_kbs(user_id=user_id, tenant_id=tenant_id) + if total_count == 0: + return JSONResponse( + status_code=HTTPStatus.OK, + content={"value": [], "total_count": 0, "has_more": False, "total_reliable": True}, + ) + + rows = perms.get_accessible_kbs( + user_id=user_id, tenant_id=tenant_id, page=page, page_size=page_size + ) + + server_url, api_key = _credentials() + items: list[dict] = [] + for row in rows: + kb_id = row["kb_id"] + try: + detail = get_aidp_kb_impl(server_url, api_key, kb_id) or {} + resource_status = "ACTIVE" + except AppException as exc: + logger.warning("AIDP detail fetch failed for %s: %s", kb_id, exc) + perms.update_resource_status( + kb_id=kb_id, tenant_id=tenant_id, status="UNAVAILABLE", + updated_by=user_id, + ) + detail = {} + resource_status = "UNAVAILABLE" + + items.append({ + "kds_id": kb_id, + "kds_name": detail.get("kds_name") or detail.get("name") or "", + "description": detail.get("description", ""), + "document_count": detail.get("document_count", 0), + "chunk_count": detail.get("chunk_count", 0), + "embedding_model": detail.get("embedding_model", ""), + # ``is_multimodal`` is a Nexent-side concept (frontend sends it + # when creating a KB; the SDK mapper converts it to + # ``caption_enable`` + ``vlm_model``). AIDP does NOT return this + # field, so we reverse-derive it from ``caption_enable == 1`` + # and a non-empty ``vlm_model``. Matches the forward mapping + # in ``sdk/nexent/core/knowledge_base/mapper.py``. + "is_multimodal": _infer_is_multimodal(detail), + "vlm_model": detail.get("vlm_model") or "", + "caption_enable": detail.get("caption_enable", 0), + "created_at": detail.get("created_at"), + "permission": row.get("permission"), + "ingroup_permission": row.get("ingroup_permission"), + "group_ids": row.get("group_ids"), + "created_by": row.get("owner_user_id"), + "resource_status": resource_status, + }) + + has_more = page * page_size < total_count + return JSONResponse( + status_code=HTTPStatus.OK, + content={ + "value": items, + "total_count": total_count, + "has_more": has_more, + "total_reliable": True, + }, + ) + + +@aidp_mgmt_router.get("/knowledge-bases/count") +async def count_knowledge_bases(request: Request) -> JSONResponse: + """Return the accessible KB count for the calling user/tenant.""" + user_id, tenant_id = await _auth(request) + total = perms.count_accessible_kbs(user_id=user_id, tenant_id=tenant_id) + return JSONResponse(status_code=HTTPStatus.OK, content={"total_count": total}) + + +@aidp_mgmt_router.post("/knowledge-bases") +async def create_knowledge_base( + request: Request, + body: CreateKbRequest, +) -> JSONResponse: + """Create a KB. Idempotent via ``kds_id`` unique-index backstop.""" + user_id, tenant_id = await _auth(request) + + ingroup = body.ingroup_permission or READ_ONLY + if ingroup not in {EDIT, READ_ONLY, "PRIVATE"}: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail=f"Unsupported ingroup_permission: {ingroup!r}", + ) + + if ingroup != "PRIVATE": + if not body.group_ids: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="group_ids is required when ingroup_permission is READ_ONLY or EDIT", + ) + try: + valid_group_ids = perms._validate_group_ids_strict(body.group_ids, tenant_id) + except AidpGroupValidationError as exc: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail=str(exc), + ) + else: + valid_group_ids = [] + + server_url, api_key = _credentials() + aidp_payload = body.model_dump( + exclude={"ingroup_permission", "group_ids"}, + exclude_none=True, + ) + try: + aidp_result = create_aidp_kb_impl(server_url, api_key, aidp_payload) + except AppException: + raise + except Exception as exc: + logger.exception("AIDP create failed: %s", exc) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"Failed to create AIDP knowledge base: {exc}", + ) + + kds_id = aidp_result.get("kds_id") or aidp_result.get("id") + if not kds_id: + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + "AIDP did not return a kds_id for the created knowledge base", + ) + # Normalize to string. AIDP may return kds_id as int or str; the DB + # schema declares ``kb_id VARCHAR(64)``, so PostgreSQL rejects a + # mixed-type comparison (``varchar = integer``) with an + # ``UndefinedFunction`` error. Cast once here so every downstream + # use (DB lookup, permission record insert, log messages) is a str. + kds_id = str(kds_id) + + if aidp_permission_db.get_permission_by_kb_id(kds_id, tenant_id): + raise AidpKbConflictError(kds_id, tenant_id).__class__( + kds_id=kds_id, tenant_id=tenant_id + ) if False else HTTPException( # construct HTTPException directly to keep mapping simple + status_code=HTTPStatus.CONFLICT, + detail=f"Knowledge base {kds_id} already exists in this tenant", + ) + + try: + perms.create_permission( + kb_id=kds_id, + kds_name=body.name or aidp_result.get("kds_name") or aidp_result.get("name") or "", + owner_user_id=user_id, + tenant_id=tenant_id, + ingroup_permission=ingroup, + group_ids=valid_group_ids, + resource_status="CREATING", + created_by=user_id, + ) + except IntegrityError as exc: + _raise_aidp_conflict(exc) + except Exception as db_err: + logger.error("Failed to save KB permission, rolling back AIDP: %s", db_err) + try: + delete_aidp_kb_impl(server_url, api_key, kds_id) + except Exception as rollback_err: + logger.critical( + "AIDP rollback failed for kds_id=%s (orphan remains): %s", + kds_id, rollback_err, + ) + perms.update_resource_status( + kb_id=kds_id, tenant_id=tenant_id, status="ORPHANED", + updated_by=user_id, + ) + raise HTTPException( + status_code=HTTPStatus.INTERNAL_SERVER_ERROR, + detail="Failed to save knowledge base permission record", + ) + + perms.update_resource_status( + kb_id=kds_id, tenant_id=tenant_id, status="ACTIVE", updated_by=user_id, + ) + + aidp_result = dict(aidp_result or {}) + aidp_result["permission"] = EDIT + return JSONResponse(status_code=HTTPStatus.OK, content=aidp_result) + + +@aidp_mgmt_router.get("/knowledge-bases/{kds_id}") +async def get_knowledge_base( + request: Request, + kds_id: Annotated[str, Path(description="Knowledge base ID")], +) -> JSONResponse: + user_id, tenant_id = await _auth(request) + decision = perms.require_permission(kds_id, user_id, tenant_id, required="READ") + + server_url, api_key = _credentials() + try: + detail = get_aidp_kb_impl(server_url, api_key, kds_id) or {} + resource_status = "ACTIVE" + except AppException as exc: + logger.warning("AIDP detail fetch failed for %s: %s", kds_id, exc) + perms.update_resource_status( + kb_id=kds_id, tenant_id=tenant_id, status="UNAVAILABLE", + updated_by=user_id, + ) + detail = {} + resource_status = "UNAVAILABLE" + + detail = dict(detail) + detail["kds_id"] = kds_id + detail["permission"] = decision.permission + detail["resource_status"] = resource_status + return JSONResponse(status_code=HTTPStatus.OK, content=detail) + + +@aidp_mgmt_router.put("/knowledge-bases/{kds_id}") +async def update_knowledge_base( + request: Request, + kds_id: Annotated[str, Path(description="Knowledge base ID")], + body: UpdateKbRequest, +) -> JSONResponse: + user_id, tenant_id = await _auth(request) + perms.require_permission(kds_id, user_id, tenant_id, required="EDIT") + + payload = body.model_dump(exclude_none=True) + if not payload: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="At least one field (name or description) must be provided for update", + ) + server_url, api_key = _credentials() + result = update_aidp_kb_impl(server_url, api_key, kds_id, payload) + + # Sync kds_name to permission table so the LLM name-to-id map stays current. + new_kds_name = ( + (result.get("kds_name") if isinstance(result, dict) else None) + or body.name + ) + if new_kds_name: + perms.update_permission( + kb_id=kds_id, + tenant_id=tenant_id, + kds_name=new_kds_name, + updated_by=user_id, + ) + + return JSONResponse(status_code=HTTPStatus.OK, content=result) + + +@aidp_mgmt_router.delete("/knowledge-bases/{kds_id}") +async def delete_knowledge_base( + request: Request, + kds_id: Annotated[str, Path(description="Knowledge base ID")], +) -> JSONResponse: + user_id, tenant_id = await _auth(request) + perms.require_permission(kds_id, user_id, tenant_id, required="EDIT") + + server_url, api_key = _credentials() + success = delete_aidp_kb_impl(server_url, api_key, kds_id) + if success: + perms.soft_delete_permission( + kb_id=kds_id, tenant_id=tenant_id, updated_by=user_id, + ) + return JSONResponse(status_code=HTTPStatus.OK, content={"success": success}) + + +@aidp_mgmt_router.post("/knowledge-bases/{kds_id}/documents") +async def upload_documents( + request: Request, + kds_id: Annotated[str, Path(description="Knowledge base ID")], + files: List[UploadFile] = File(..., description="Files to upload"), +) -> JSONResponse: + user_id, tenant_id = await _auth(request) + perms.require_permission(kds_id, user_id, tenant_id, required="EDIT") + + server_url, api_key = _credentials() + result = upload_aidp_docs_impl(server_url, api_key, kds_id, files) + return JSONResponse(status_code=HTTPStatus.OK, content=result) + + +@aidp_mgmt_router.get("/knowledge-bases/{kds_id}/documents") +async def list_documents( + request: Request, + kds_id: Annotated[str, Path(description="Knowledge base ID")], + page: Annotated[int, Query(ge=1, description="Page number starting from 1")] = 1, + page_size: Annotated[int, Query(ge=1, le=100, description="Page size from 1 to 100")] = 10, +) -> JSONResponse: + user_id, tenant_id = await _auth(request) + perms.require_permission(kds_id, user_id, tenant_id, required="READ") + + server_url, api_key = _credentials() + result = list_aidp_docs_impl(server_url, api_key, kds_id, page=page, page_size=page_size) + page_items = result.get("value", []) if isinstance(result, dict) else [] + page_count = len(page_items) if isinstance(page_items, list) else 0 + + try: + total_count = count_aidp_docs_impl(server_url, api_key, kds_id) + count_reliable = True + except Exception as count_err: + logger.warning( + "AIDP doc Count API failed for KB %s: %s", kds_id, count_err, + ) + total_count = page_count + count_reliable = False + + has_more = ( + total_count > page * page_size + if count_reliable + else bool(result.get("next_link")) or page_count >= page_size + ) + + result["total_count"] = int(total_count) + result["has_more"] = has_more + if not count_reliable: + result["total_reliable"] = False + return JSONResponse(status_code=HTTPStatus.OK, content=result) + + +@aidp_mgmt_router.patch("/aidp-permissions/{kds_id}") +async def set_permission( + request: Request, + kds_id: Annotated[str, Path(description="Knowledge base ID")], + body: SetPermissionRequest, +) -> JSONResponse: + """Update the in-group permission for a KB (does not call AIDP).""" + user_id, tenant_id = await _auth(request) + perms.require_permission(kds_id, user_id, tenant_id, required="EDIT") + + if body.ingroup_permission not in {EDIT, READ_ONLY, "PRIVATE"}: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail=f"Unsupported ingroup_permission: {body.ingroup_permission!r}", + ) + + if body.ingroup_permission == "PRIVATE": + final_group_ids: list[int] = [] + else: + if not body.group_ids: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail="group_ids is required when ingroup_permission is READ_ONLY or EDIT", + ) + try: + final_group_ids = perms._validate_group_ids_strict(body.group_ids, tenant_id) + except AidpGroupValidationError as exc: + raise HTTPException( + status_code=HTTPStatus.BAD_REQUEST, + detail=str(exc), + ) + + perms.update_permission( + kb_id=kds_id, + tenant_id=tenant_id, + ingroup_permission=body.ingroup_permission, + group_ids=final_group_ids, + updated_by=user_id, + ) + return JSONResponse(status_code=HTTPStatus.OK, content={"success": True}) + + +@aidp_mgmt_router.get("/models") +async def list_models( + request: Request, + service: Annotated[str, Query(description="Model service category (default: llm)")] = "llm", + app: Annotated[str, Query(description="Application filter (default: KnowledgeBase)")] = "KnowledgeBase", +) -> JSONResponse: + """List available models from AIDP ModelService. Auth required; no per-KB permission.""" + await _auth(request) + server_url, api_key = _credentials() + result = list_aidp_models_impl(server_url, api_key, service=service, app=app) + return JSONResponse(status_code=HTTPStatus.OK, content=result) diff --git a/backend/ext_components/aidp/consts/aidp_exceptions.py b/backend/ext_components/aidp/consts/aidp_exceptions.py new file mode 100644 index 0000000000..042513be7c --- /dev/null +++ b/backend/ext_components/aidp/consts/aidp_exceptions.py @@ -0,0 +1,94 @@ +"""Domain exceptions raised by the AIDP permission subsystem. + +These map to HTTP status codes in ``aidp_mgmt_app`` so the rest of the +backend can stay consistent with the v7.1 permission model: + +* ``AidpKbNotFoundError`` -> 404 (resource not visible to the tenant) +* ``AidpKbPermissionDeniedError`` -> 403 (visible but not permitted) +* ``AidpKbConflictError`` -> 409 (active kb_id collision) +* ``AidpKbSyncError`` -> 502 (AIDP service layer returned an error) +* ``AidpGroupValidationError`` -> 400 (cross-tenant or empty group_ids) +""" +from __future__ import annotations + + +class AidpKbNotFoundError(Exception): + """KB does not exist or does not belong to the current tenant. HTTP 404.""" + + def __init__(self, kb_id: str, tenant_id: str | None = None) -> None: + self.kb_id = kb_id + self.tenant_id = tenant_id + super().__init__( + f"AIDP knowledge base {kb_id} not found" + + (f" in tenant {tenant_id}" if tenant_id else "") + ) + + +class AidpKbPermissionDeniedError(Exception): + """KB belongs to the tenant but the user lacks the required permission. + + Maps to HTTP 403. Carries the required level so the caller can include it + in the response without recomputing the decision. + """ + + def __init__(self, kb_id: str, user_id: str, required: str) -> None: + self.kb_id = kb_id + self.user_id = user_id + self.required = required + super().__init__( + f"User {user_id} lacks {required} permission on {kb_id}" + ) + + +class AidpKbConflictError(Exception): + """An active permission record already exists for this kbs_id. + + Maps to HTTP 409. Raised when the application-layer pre-check finds an + existing active row, or when the database unique index trips. + """ + + def __init__(self, kb_id: str, tenant_id: str) -> None: + self.kb_id = kb_id + self.tenant_id = tenant_id + super().__init__( + f"Knowledge base {kb_id} already exists in tenant {tenant_id}" + ) + + +class AidpKbSyncError(Exception): + """AIDP returned an unexpected error for an otherwise valid request. + + Maps to HTTP 502. ``cause`` keeps the original exception for diagnostics. + """ + + def __init__(self, operation: str, kb_id: str | None = None, cause: Exception | None = None) -> None: + self.operation = operation + self.kb_id = kb_id + self.cause = cause + suffix = f" ({cause})" if cause is not None else "" + target = f" for {kb_id}" if kb_id else "" + super().__init__(f"AIDP {operation}{target} failed{suffix}") + + +class AidpGroupValidationError(Exception): + """One or more ``group_ids`` do not belong to the current tenant. + + Maps to HTTP 400. ``invalid_ids`` are the offending values so the caller + can surface them to the API client without re-iterating. + """ + + def __init__(self, invalid_ids: list[int], tenant_id: str) -> None: + self.invalid_ids = list(invalid_ids) + self.tenant_id = tenant_id + super().__init__( + f"Group ids {self.invalid_ids} are not part of tenant {tenant_id}" + ) + + +__all__ = [ + "AidpKbNotFoundError", + "AidpKbPermissionDeniedError", + "AidpKbConflictError", + "AidpKbSyncError", + "AidpGroupValidationError", +] diff --git a/backend/ext_components/aidp/database/aidp_permission_db.py b/backend/ext_components/aidp/database/aidp_permission_db.py new file mode 100644 index 0000000000..c651f64d6f --- /dev/null +++ b/backend/ext_components/aidp/database/aidp_permission_db.py @@ -0,0 +1,350 @@ +"""CRUD accessors for ``aidp_kb_permission_t``. + +Design contract (see aidp-knowledge-permission-implementation-plan-v7.1 §4.4): +* Every read filters on ``tenant_id`` AND ``delete_flag != 'Y'`` so callers + cannot accidentally read soft-deleted rows or cross tenant boundaries. +* Writes are explicit per row; ``create_permission`` does not deduplicate and + relies on the caller having already checked ``get_permission_by_kb_id`` and + on the partial unique index in PostgreSQL as the final backstop. +* Returned dicts use ISO-formatted timestamps so the service layer can + serialise them directly to JSON. +""" +from __future__ import annotations + +import logging +from typing import Any, Iterable, List, Optional, Sequence + +from sqlalchemy import and_, func, select, update +from sqlalchemy.dialects.postgresql import JSONB +from sqlalchemy.exc import IntegrityError +from sqlalchemy.orm import Session + +from database.client import as_dict, get_db_session +from ext_components.aidp.database.db_models import AidpKbPermission + +logger = logging.getLogger("aidp_permission_db") + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + +_ACTIVE_FILTER = "N" + + +def _active_clause(): + return AidpKbPermission.delete_flag == _ACTIVE_FILTER + + +def _normalize_group_ids(group_ids: Any) -> list[int]: + """Return a JSONB-safe list of group IDs. + + ``JSONB`` column accepts lists directly; we still coerce to ``int`` to + reject malformed payloads early (e.g. ``"1,2,3"`` strings from upstream + layers that pre-date this table). + """ + if group_ids is None: + return [] + if isinstance(group_ids, str): + return [int(item.strip()) for item in group_ids.split(",") if item.strip()] + return [int(item) for item in group_ids] + + +# --------------------------------------------------------------------------- +# Reads +# --------------------------------------------------------------------------- + +def list_permissions_by_tenant( + tenant_id: str, + page: int = 1, + page_size: int = 20, + db_session: Optional[Session] = None, +) -> List[dict]: + """Return active permission records for ``tenant_id`` ordered by + ``create_time DESC, id DESC`` to keep pagination stable across writes. + """ + if not tenant_id: + raise ValueError("tenant_id is required") + + page = max(1, int(page)) + page_size = max(1, min(int(page_size), 200)) + + stmt = ( + select(AidpKbPermission) + .where(and_(_active_clause(), AidpKbPermission.tenant_id == tenant_id)) + .order_by(AidpKbPermission.create_time.desc(), AidpKbPermission.id.desc()) + .offset((page - 1) * page_size) + .limit(page_size) + ) + with get_db_session(db_session) as session: + rows = session.execute(stmt).scalars().all() + return [as_dict(row) for row in rows] + + +def list_all_permissions_by_tenant( + tenant_id: str, + db_session: Optional[Session] = None, +) -> List[dict]: + """Return ALL active permission records for ``tenant_id`` (no pagination). + + Used by the permission service layer to do application-side permission + filtering (group intersection / ownership / PRIVATE) before slicing into + the user-visible page. Without this we would lose items that happen to + sit in the middle of a DB page that the user cannot see. + """ + if not tenant_id: + raise ValueError("tenant_id is required") + + stmt = ( + select(AidpKbPermission) + .where(and_(_active_clause(), AidpKbPermission.tenant_id == tenant_id)) + .order_by(AidpKbPermission.create_time.desc(), AidpKbPermission.id.desc()) + ) + with get_db_session(db_session) as session: + rows = session.execute(stmt).scalars().all() + return [as_dict(row) for row in rows] + + +def list_kds_name_to_id_map( + tenant_id: str, + db_session: Optional[Session] = None, +) -> dict: + """Return a ``{kds_name: kb_id}`` map for all active records in ``tenant_id``. + + Skips rows where ``kds_name`` is empty or NULL. This is the DB-layer + primitive; user-level access filtering is applied by the service layer. + """ + if not tenant_id: + raise ValueError("tenant_id is required") + + stmt = ( + select(AidpKbPermission.kds_name, AidpKbPermission.kb_id) + .where(and_(_active_clause(), AidpKbPermission.tenant_id == tenant_id)) + ) + with get_db_session(db_session) as session: + rows = session.execute(stmt).all() + return {row.kds_name: row.kb_id for row in rows if row.kds_name} + + +def count_permissions_by_tenant( + tenant_id: str, + db_session: Optional[Session] = None, +) -> int: + """Return the number of active permission records in a tenant.""" + if not tenant_id: + raise ValueError("tenant_id is required") + stmt = ( + select(func.count(AidpKbPermission.id)) + .where(and_(_active_clause(), AidpKbPermission.tenant_id == tenant_id)) + ) + with get_db_session(db_session) as session: + return int(session.execute(stmt).scalar_one() or 0) + + +def get_permission_by_kb_id( + kb_id: str, + tenant_id: str, + db_session: Optional[Session] = None, +) -> Optional[dict]: + """Look up a single active permission row by ``(kb_id, tenant_id)``. + + Returns ``None`` when the KB is unknown, soft-deleted, or belongs to + another tenant; callers should treat all three cases the same (the row is + not accessible to the current tenant). + """ + if not kb_id or not tenant_id: + raise ValueError("kb_id and tenant_id are required") + stmt = select(AidpKbPermission).where( + and_( + _active_clause(), + AidpKbPermission.kb_id == kb_id, + AidpKbPermission.tenant_id == tenant_id, + ) + ) + with get_db_session(db_session) as session: + row = session.execute(stmt).scalar_one_or_none() + return as_dict(row) if row is not None else None + + +# --------------------------------------------------------------------------- +# Writes +# --------------------------------------------------------------------------- + +def create_permission( + *, + kb_id: str, + kds_name: Optional[str] = None, + owner_user_id: str, + tenant_id: str, + ingroup_permission: str = "READ_ONLY", + group_ids: Optional[Iterable[int]] = None, + resource_status: str = "ACTIVE", + created_by: Optional[str] = None, + db_session: Optional[Session] = None, +) -> int: + """Insert a new permission record and return the new row id. + + Raises ``IntegrityError`` when a duplicate active ``kb_id`` exists; the + caller is responsible for translating that into HTTP 409. + """ + if not kb_id or not owner_user_id or not tenant_id: + raise ValueError("kb_id, owner_user_id and tenant_id are required") + payload = { + "kb_id": kb_id, + "kds_name": kds_name or "", + "owner_user_id": owner_user_id, + "tenant_id": tenant_id, + "ingroup_permission": ingroup_permission, + "group_ids": _normalize_group_ids(group_ids), + "resource_status": resource_status, + "delete_flag": _ACTIVE_FILTER, + } + if created_by is not None: + payload["created_by"] = created_by + payload["updated_by"] = created_by + + with get_db_session(db_session) as session: + record = AidpKbPermission(**payload) + session.add(record) + try: + session.flush() + except IntegrityError as exc: + logger.warning( + "AidpKbPermission unique constraint violation kb_id=%s: %s", + kb_id, + exc, + ) + raise + new_id = record.id + session.commit() + return int(new_id) + + +def update_permission( + *, + kb_id: str, + tenant_id: str, + ingroup_permission: Optional[str] = None, + group_ids: Optional[Sequence[int]] = None, + kds_name: Optional[str] = None, + updated_by: Optional[str] = None, + db_session: Optional[Session] = None, +) -> bool: + """Partially update an active permission row. + + Returns ``True`` when a row was modified, ``False`` when no active row + matched (caller should treat as 404). + """ + if not kb_id or not tenant_id: + raise ValueError("kb_id and tenant_id are required") + values: dict[str, Any] = {} + if ingroup_permission is not None: + values["ingroup_permission"] = ingroup_permission + if group_ids is not None: + values["group_ids"] = _normalize_group_ids(group_ids) + if kds_name is not None: + values["kds_name"] = kds_name + if updated_by is not None: + values["updated_by"] = updated_by + if not values: + # Nothing to update — treat as a no-op success so callers can pass + # idempotent updates without first checking diff state. + return True + + stmt = ( + update(AidpKbPermission) + .where( + and_( + _active_clause(), + AidpKbPermission.kb_id == kb_id, + AidpKbPermission.tenant_id == tenant_id, + ) + ) + .values(**values) + .execution_options(synchronize_session="fetch") + ) + with get_db_session(db_session) as session: + result = session.execute(stmt) + session.commit() + return bool(result.rowcount) + + +def soft_delete_permission( + *, + kb_id: str, + tenant_id: str, + updated_by: Optional[str] = None, + db_session: Optional[Session] = None, +) -> bool: + """Soft-delete the active row and mark it ``DELETE_PENDING``. + + The active unique index releases the ``kb_id`` once ``delete_flag='Y'`` so + that the same ``kb_id`` can be re-created later if the tenant re-claims + it from AIDP. + """ + if not kb_id or not tenant_id: + raise ValueError("kb_id and tenant_id are required") + stmt = ( + update(AidpKbPermission) + .where( + and_( + _active_clause(), + AidpKbPermission.kb_id == kb_id, + AidpKbPermission.tenant_id == tenant_id, + ) + ) + .values( + delete_flag="Y", + resource_status="DELETE_PENDING", + **({"updated_by": updated_by} if updated_by else {}), + ) + .execution_options(synchronize_session="fetch") + ) + with get_db_session(db_session) as session: + result = session.execute(stmt) + session.commit() + return bool(result.rowcount) + + +def update_resource_status( + *, + kb_id: str, + tenant_id: str, + status: str, + updated_by: Optional[str] = None, + db_session: Optional[Session] = None, +) -> bool: + """Update only ``resource_status`` for an active row.""" + if not kb_id or not tenant_id or not status: + raise ValueError("kb_id, tenant_id and status are required") + stmt = ( + update(AidpKbPermission) + .where( + and_( + _active_clause(), + AidpKbPermission.kb_id == kb_id, + AidpKbPermission.tenant_id == tenant_id, + ) + ) + .values( + resource_status=status, + **({"updated_by": updated_by} if updated_by else {}), + ) + .execution_options(synchronize_session="fetch") + ) + with get_db_session(db_session) as session: + result = session.execute(stmt) + session.commit() + return bool(result.rowcount) + + +__all__ = [ + "list_permissions_by_tenant", + "count_permissions_by_tenant", + "get_permission_by_kb_id", + "list_kds_name_to_id_map", + "create_permission", + "update_permission", + "soft_delete_permission", + "update_resource_status", +] diff --git a/backend/ext_components/aidp/database/db_models.py b/backend/ext_components/aidp/database/db_models.py new file mode 100644 index 0000000000..c8dd6ea7c5 --- /dev/null +++ b/backend/ext_components/aidp/database/db_models.py @@ -0,0 +1,109 @@ +"""ORM models for the AIDP knowledge base permission subsystem (v7.1). + +These models back the ``aidp_kb_permission_t`` table introduced in +``deploy/sql/migrations/v2.4_merged_migrations.sql``. The schema +is intentionally separate from the SDK ``aidp_client`` so the SDK can stay a +pure HTTP adapter while permission decisions live in the backend. + +The model inherits ``TableBase`` so it shares the audit columns and +``delete_flag`` semantics with the rest of the backend ORM. To keep the +AIDP ORM self-contained for tests, we declare the audit columns inline +instead of mixing with the global ``TableBase.metadata`` registry. +""" +from __future__ import annotations + +from sqlalchemy import ( + BigInteger, + Column, + Index, + Integer, + String, + TIMESTAMP, + func, + text, +) +from sqlalchemy.dialects.postgresql import JSONB +from sqlalchemy.ext.declarative import declarative_base + +from database.db_models import SCHEMA + + +# Dedicated Declarative base so the AIDP metadata can be re-declared in +# tests without colliding with the shared TableBase.metadata. +AidpKbPermissionBase = declarative_base() + + +class AidpKbPermission(AidpKbPermissionBase): + """ORM model for ``nexent.aidp_kb_permission_t``. + + A single row represents a KB that Nexent has observed via AIDP and + decided to manage. The active uniqueness on ``kb_id`` is enforced in + PostgreSQL (see migration), so application code MUST treat + ``create_permission`` as non-idempotent and let ``get_permission_by_kb_id`` + act as the first concurrency check before insertion. + """ + + __tablename__ = "aidp_kb_permission_t" + __table_args__ = ( + Index( + "ix_aidp_kb_permission_tenant_active", + "tenant_id", + postgresql_where=text("delete_flag = 'N'"), + ), + {"schema": SCHEMA}, + ) + + id = Column( + BigInteger().with_variant(Integer(), "sqlite"), + primary_key=True, + autoincrement=True, + doc="Primary key, auto-increment", + ) + kb_id = Column(String(128), nullable=False, doc="AIDP kds_id") + kds_name = Column( + String(128), + nullable=True, + doc="AIDP knowledge base display name (kds_name), cached at creation time", + ) + owner_user_id = Column( + String(100), + nullable=False, + doc="Nexent user_id of the KB creator", + ) + tenant_id = Column(String(100), nullable=False, doc="Nexent tenant_id") + ingroup_permission = Column( + String(30), + nullable=False, + default="READ_ONLY", + doc="EDIT / READ_ONLY / PRIVATE", + ) + group_ids = Column( + JSONB, + nullable=False, + default=list, + doc="JSON array of group IDs, e.g. [1, 2, 3]", + ) + resource_status = Column( + String(30), + nullable=False, + default="ACTIVE", + doc="CREATING / ACTIVE / DELETE_PENDING / ORPHANED / UNAVAILABLE", + ) + create_time = Column( + TIMESTAMP(timezone=False), + server_default=func.now(), + doc="Creation time", + ) + update_time = Column( + TIMESTAMP(timezone=False), + server_default=func.now(), + onupdate=func.now(), + doc="Update time", + ) + created_by = Column(String(100), doc="Creator") + updated_by = Column(String(100), doc="Updater") + delete_flag = Column( + String(1), + default="N", + doc="Soft delete flag. Active rows are N; soft delete sets Y.", + ) diff --git a/backend/ext_components/aidp/services/__init__.py b/backend/ext_components/aidp/services/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/backend/ext_components/aidp/services/aidp_permission_service.py b/backend/ext_components/aidp/services/aidp_permission_service.py new file mode 100644 index 0000000000..7fae22df49 --- /dev/null +++ b/backend/ext_components/aidp/services/aidp_permission_service.py @@ -0,0 +1,480 @@ +"""Permission resolution and runtime whitelist helpers for AIDP KBs (v7.1). + +This module composes three inputs to decide whether a user may act on a +given AIDP knowledge base: + +1. ``aidp_kb_permission_t`` (the active ``kb_id -> group_ids`` mapping). +2. The user's role within the tenant (``user_tenant_t.user_role``). +3. The user's group memberships within the tenant (a join through + ``tenant_group_info_t`` so we never leak cross-tenant group IDs). + +Decision order: + 1. Management roles (SU/ADMIN/SPEED) -> EDIT (within tenant boundary). + 2. ASSET_OWNER -> EDIT only inside its asset context; we conservatively + grant EDIT here so the rest of the system can rely on a single rule. + Callers that need finer ASSET_OWNER scoping can override at the + resource layer. + 3. Creator (matches ``owner_user_id``) -> EDIT. + 4. ``PRIVATE`` -> no access (except creator). + 5. Empty ``group_ids`` -> no access (except creator/management). + 6. Group intersection exists -> ``ingroup_permission``; otherwise no access. + +Errors raised here map to HTTP status codes in ``aidp_mgmt_app``: +* ``AidpKbNotFoundError`` -> 404 +* ``AidpKbPermissionDeniedError`` -> 403 +* ``AidpKbConflictError`` -> 409 +* ``AidpGroupValidationError`` -> 400 +""" +from __future__ import annotations + +import logging +from dataclasses import dataclass +from typing import Any, Iterable, Sequence + +from consts.const import CAN_EDIT_ALL_USER_ROLES +from database import group_db as group_db_module +from database.group_db import ( + filter_tenant_group_ids, + query_group_ids_by_user_in_tenant, +) +from database import user_tenant_db as user_tenant_db_module +from database.user_tenant_db import get_user_role_by_tenant +from ext_components.aidp.consts.aidp_exceptions import ( + AidpGroupValidationError, + AidpKbConflictError, + AidpKbNotFoundError, + AidpKbPermissionDeniedError, +) +from ext_components.aidp.database import aidp_permission_db + +logger = logging.getLogger("aidp_permission_service") + + +# --------------------------------------------------------------------------- +# Public dataclasses +# --------------------------------------------------------------------------- + +# Permission levels used internally; mirrors the v7.1 matrix. +EDIT = "EDIT" +READ_ONLY = "READ_ONLY" +PRIVATE = "PRIVATE" +CREATOR = "CREATOR" + +# Argument values for require_permission(). +REQUIRE_READ = "READ" +REQUIRE_EDIT = "EDIT" + +# Ordered rank for permission comparison. +_RANK = {REQUIRE_READ: 1, REQUIRE_EDIT: 2} + + +@dataclass(frozen=True) +class AidpPermissionDecision: + """Immutable result of a permission evaluation. + + Attributes: + kb_id: Target kds_id. + tenant_id: Tenant the row belongs to. + user_id: Caller. + permission: Effective permission ("EDIT" / "READ_ONLY" / "CREATOR" / + None when the user has no access). + is_management_role: True when the decision was made via + ``CAN_EDIT_ALL_USER_ROLES``. + matched_group_ids: Group IDs that matched the user's memberships. + Empty tuple when access came from creator or management role. + """ + + kb_id: str + tenant_id: str + user_id: str + permission: str | None + is_management_role: bool + matched_group_ids: tuple[int, ...] + + +# --------------------------------------------------------------------------- +# Helpers +# --------------------------------------------------------------------------- + + +def _parse_group_ids(raw: Any) -> list[int]: + """Normalise ``group_ids`` from JSONB or comma-separated text into ints.""" + if raw is None or raw == "": + return [] + if isinstance(raw, str): + return [int(item.strip()) for item in raw.split(",") if item.strip()] + if isinstance(raw, Iterable): + return [int(item) for item in raw] + raise ValueError(f"Unsupported group_ids payload: {type(raw).__name__}") + + +def _validate_group_ids_strict( + group_ids: Sequence[int], tenant_id: str +) -> list[int]: + """Reject any ``group_ids`` that are not part of ``tenant_id``. + + Differs from :func:`filter_tenant_group_ids` in that it raises an error + for the first mismatch instead of silently dropping invalid IDs. + """ + if not group_ids: + return [] + valid = set(group_db_module.filter_tenant_group_ids(list(group_ids), tenant_id)) + invalid = [int(g) for g in group_ids if int(g) not in valid] + if invalid: + raise AidpGroupValidationError(invalid_ids=invalid, tenant_id=tenant_id) + return [int(g) for g in group_ids] + + +def _get_user_role(user_id: str, tenant_id: str) -> str: + return user_tenant_db_module.get_user_role_by_tenant(user_id, tenant_id) + + +def _get_user_groups(user_id: str, tenant_id: str) -> list[int]: + return group_db_module.query_group_ids_by_user_in_tenant(user_id, tenant_id) + + +def _resolve_permission( + record: dict, + user_id: str, + tenant_id: str, + user_groups: Sequence[int] | None = None, +) -> AidpPermissionDecision: + """Compute effective permission using the matrix described in the module docstring. + + ``record`` is a row from ``aidp_kb_permission_t`` keyed by ``kb_id`` + + ``tenant_id``. ``user_groups`` may be supplied to avoid an extra DB round + trip when callers already have them in scope. + """ + if not record: + # Treat as 404 so callers can map this consistently. + raise AidpKbNotFoundError(kb_id="", tenant_id=tenant_id) + + kb_id = record["kb_id"] + owner_user_id = record.get("owner_user_id") + ingroup_permission = record.get("ingroup_permission") or READ_ONLY + record_groups = set(_parse_group_ids(record.get("group_ids"))) + + role = _get_user_role(user_id, tenant_id) + is_management = role in CAN_EDIT_ALL_USER_ROLES + if is_management: + return AidpPermissionDecision( + kb_id=kb_id, + tenant_id=tenant_id, + user_id=user_id, + permission=EDIT, + is_management_role=True, + matched_group_ids=tuple(), + ) + + if owner_user_id and owner_user_id == user_id: + return AidpPermissionDecision( + kb_id=kb_id, + tenant_id=tenant_id, + user_id=user_id, + permission=EDIT, + is_management_role=False, + matched_group_ids=tuple(), + ) + + if ingroup_permission == PRIVATE: + return AidpPermissionDecision( + kb_id=kb_id, + tenant_id=tenant_id, + user_id=user_id, + permission=None, + is_management_role=False, + matched_group_ids=tuple(), + ) + + if not record_groups: + return AidpPermissionDecision( + kb_id=kb_id, + tenant_id=tenant_id, + user_id=user_id, + permission=None, + is_management_role=False, + matched_group_ids=tuple(), + ) + + user_group_set = ( + set(int(g) for g in user_groups) + if user_groups is not None + else set(_get_user_groups(user_id, tenant_id)) + ) + matched = sorted(record_groups & user_group_set) + if not matched: + return AidpPermissionDecision( + kb_id=kb_id, + tenant_id=tenant_id, + user_id=user_id, + permission=None, + is_management_role=False, + matched_group_ids=tuple(), + ) + + return AidpPermissionDecision( + kb_id=kb_id, + tenant_id=tenant_id, + user_id=user_id, + permission=ingroup_permission, + is_management_role=False, + matched_group_ids=tuple(matched), + ) + + +def _decision_meets(decision: AidpPermissionDecision, required: str) -> bool: + """Return True when ``decision.permission`` satisfies the required level.""" + if decision.permission is None: + return False + if required == REQUIRE_READ: + return decision.permission in (READ_ONLY, EDIT, CREATOR) + if required == REQUIRE_EDIT: + return decision.permission in (EDIT, CREATOR) + raise ValueError(f"Unsupported required permission: {required!r}") + + +# --------------------------------------------------------------------------- +# DB operation wrappers +# --------------------------------------------------------------------------- + + +def create_permission(*args: Any, **kwargs: Any) -> int: + return aidp_permission_db.create_permission(*args, **kwargs) + + +def update_permission(*args: Any, **kwargs: Any) -> bool: + return aidp_permission_db.update_permission(*args, **kwargs) + + +def soft_delete_permission(*args: Any, **kwargs: Any) -> bool: + return aidp_permission_db.soft_delete_permission(*args, **kwargs) + + +def update_resource_status(*args: Any, **kwargs: Any) -> bool: + return aidp_permission_db.update_resource_status(*args, **kwargs) + + +# --------------------------------------------------------------------------- +# Public API +# --------------------------------------------------------------------------- + + +def _compute_accessible_rows(user_id: str, tenant_id: str) -> list[dict]: + """Return KB rows where the user has non-null permission. + + Pulls ALL active rows for the tenant, applies the permission matrix + (management / owner / PRIVATE / group intersection), and keeps only + the rows the user can see. Used by both :func:`get_accessible_kbs` + and :func:`count_accessible_kbs` so the page slice and the count + never disagree on what is visible. + """ + rows = aidp_permission_db.list_all_permissions_by_tenant(tenant_id=tenant_id) + user_groups = _get_user_groups(user_id, tenant_id) + role = _get_user_role(user_id, tenant_id) + is_management = role in CAN_EDIT_ALL_USER_ROLES + + accessible: list[dict] = [] + for row in rows: + if is_management or row.get("owner_user_id") == user_id: + new_row = dict(row) + new_row["permission"] = EDIT + accessible.append(new_row) + continue + decision = _resolve_permission(row, user_id, tenant_id, user_groups) + # Drop rows the user cannot see: PRIVATE, not-in-group, or empty + # group_ids all produce ``permission is None`` here. + if decision.permission is None: + continue + new_row = dict(row) + new_row["permission"] = decision.permission + accessible.append(new_row) + return accessible + + +def get_accessible_kbs( + user_id: str, + tenant_id: str, + page: int = 1, + page_size: int = 10, +) -> list[dict]: + """Return KBs the user can access, filtered AND paginated. + + Each row carries the effective ``permission`` string (``EDIT`` / + ``READ_ONLY`` / ``CREATOR``) computed via :func:`_resolve_permission`. + Rows the user cannot see (PRIVATE / not-in-group / creator-only) are + filtered out before slicing into the requested page, so the caller + receives at most ``page_size`` rows and the visible items are always + what the user is allowed to read. + """ + accessible = _compute_accessible_rows(user_id, tenant_id) + page = max(1, int(page)) + page_size = max(1, int(page_size)) + start = (page - 1) * page_size + end = start + page_size + return accessible[start:end] + + +def count_accessible_kbs(user_id: str, tenant_id: str) -> int: + """Count KBs the user can actually access. + + The previous implementation returned the tenant KB total and let the + page-level filter drop invisible rows, which broke pagination totals + when the user lacked access to many KBs in the tenant. Now the count + reflects the post-filter accessible set so ``has_more`` / ``total`` + on the frontend matches reality. + """ + accessible = _compute_accessible_rows(user_id, tenant_id) + return len(accessible) + + +def filter_accessible_kds( + kds_ids: Sequence[str], + user_id: str, + tenant_id: str, +) -> list[str]: + """Preserve ``kds_ids`` order while dropping IDs the user cannot read.""" + if not kds_ids: + return [] + user_groups = _get_user_groups(user_id, tenant_id) + role = _get_user_role(user_id, tenant_id) + is_management = role in CAN_EDIT_ALL_USER_ROLES + + allowed: list[str] = [] + for kds_id in kds_ids: + record = _get_permission_record(kb_id=kds_id, tenant_id=tenant_id) + if record is None: + continue + if is_management or record.get("owner_user_id") == user_id: + allowed.append(kds_id) + continue + decision = _resolve_permission(record, user_id, tenant_id, user_groups) + if _decision_meets(decision, REQUIRE_READ): + allowed.append(kds_id) + return allowed + + +def get_allowed_kds_list(user_id: str, tenant_id: str) -> list[str]: + """Build the runtime whitelist used by ``AidpSearchTool``. + + Returns the subset of ``kb_ids`` in the tenant where the user has at least + ``READ`` access. The list is recomputed every agent run so permission + changes take effect immediately (no cache). + """ + rows = aidp_permission_db.list_permissions_by_tenant( + tenant_id=tenant_id, page=1, page_size=200 + ) + user_groups = _get_user_groups(user_id, tenant_id) + role = _get_user_role(user_id, tenant_id) + is_management = role in CAN_EDIT_ALL_USER_ROLES + + allowed: list[str] = [] + for row in rows: + if is_management or row.get("owner_user_id") == user_id: + allowed.append(row["kb_id"]) + continue + decision = _resolve_permission(row, user_id, tenant_id, user_groups) + if _decision_meets(decision, REQUIRE_READ): + allowed.append(row["kb_id"]) + return allowed + + +def get_kds_name_to_id_map(user_id: str, tenant_id: str) -> dict[str, str]: + """Build the kds_name-to-kds_id lookup for the LLM tool layer. + + Mirrors :func:`get_allowed_kds_list` exactly: same DB query, same user + group and role resolution, same ``_resolve_permission`` + REQUIRE_READ + gate. The only difference is the return shape — a ``{kds_name: kb_id}`` + dict instead of a flat list — and rows with an empty ``kds_name`` are + skipped (the tool cannot resolve a name that does not exist). + """ + rows = aidp_permission_db.list_permissions_by_tenant( + tenant_id=tenant_id, page=1, page_size=200 + ) + user_groups = _get_user_groups(user_id, tenant_id) + role = _get_user_role(user_id, tenant_id) + is_management = role in CAN_EDIT_ALL_USER_ROLES + + kds_map: dict[str, str] = {} + for row in rows: + kb_id = row["kb_id"] + kds_name = row.get("kds_name") + if not kds_name: + continue + if is_management or row.get("owner_user_id") == user_id: + kds_map[kds_name] = kb_id + continue + decision = _resolve_permission(row, user_id, tenant_id, user_groups) + if _decision_meets(decision, REQUIRE_READ): + kds_map[kds_name] = kb_id + return kds_map + + +def require_permission( + kb_id: str, + user_id: str, + tenant_id: str, + required: str, +) -> AidpPermissionDecision: + """Assert that ``user_id`` has at least ``required`` access on ``kb_id``. + + Raises: + AidpKbNotFoundError: When no active row matches ``(kb_id, tenant_id)``. + AidpKbPermissionDeniedError: When the row exists but the user lacks + the required permission. + """ + if required not in _RANK: + raise ValueError(f"Unsupported required permission: {required!r}") + + record = _get_permission_record(kb_id=kb_id, tenant_id=tenant_id) + if record is None: + raise AidpKbNotFoundError(kb_id=kb_id, tenant_id=tenant_id) + + decision = _resolve_permission(record, user_id, tenant_id) + if not _decision_meets(decision, required): + logger.info( + "Aidp permission denied: user=%s tenant=%s kb=%s required=%s have=%s", + user_id, tenant_id, kb_id, required, decision.permission, + ) + raise AidpKbPermissionDeniedError( + kb_id=kb_id, user_id=user_id, required=required, + ) + return decision + + +__all__ = [ + "AidpPermissionDecision", + "READ_ONLY", + "EDIT", + "PRIVATE", + "CREATOR", + "REQUIRE_READ", + "REQUIRE_EDIT", + "AidpKbNotFoundError", + "AidpKbPermissionDeniedError", + "AidpKbConflictError", + "AidpGroupValidationError", + "create_permission", + "update_permission", + "soft_delete_permission", + "update_resource_status", + "filter_accessible_kds", + "get_accessible_kbs", + "count_accessible_kbs", + "get_allowed_kds_list", + "get_kds_name_to_id_map", + "require_permission", +] + + +def _get_permission_record( + *, kb_id: str, tenant_id: str +) -> dict | None: + """Look up the active permission row for ``(kb_id, tenant_id)``. + + This indirection exists so unit tests can patch the permission service + without depending on the ``aidp_permission_db`` module reference held + at import time (which may be replaced by other conftests). + """ + return aidp_permission_db.get_permission_by_kb_id( + kb_id=kb_id, tenant_id=tenant_id + ) diff --git a/backend/ext_components/aidp/services/aidp_service.py b/backend/ext_components/aidp/services/aidp_service.py new file mode 100644 index 0000000000..9a58a04dc7 --- /dev/null +++ b/backend/ext_components/aidp/services/aidp_service.py @@ -0,0 +1,1166 @@ +""" +AIDP Service Layer +Handles API calls to AIDP for paginated knowledge base listing. +""" +import logging +import time +from datetime import datetime, timezone +from typing import Any, Callable, Dict, List +from urllib.parse import urljoin + +import httpx + +from consts.const import AIDP_TENANT_ID +from consts.error_code import ErrorCode +from consts.exceptions import AppException +from nexent.utils.http_client_manager import http_client_manager + +logger = logging.getLogger("aidp_service") + +def _resolve_tenant_id(tenant_id: Any = None) -> str: + """Resolve a valid AIDP tenant identifier from explicit or configured input.""" + configured_tenant = AIDP_TENANT_ID if isinstance(AIDP_TENANT_ID, str) else "aidp" + resolved_tenant = tenant_id if isinstance(tenant_id, str) else configured_tenant + return resolved_tenant.strip() or "aidp" + + +def _get_list_path(tenant_id: str | None = None) -> str: + """Build the tenant-scoped knowledge-base API path.""" + return f"/KnowledgeBase/Tenants/{_resolve_tenant_id(tenant_id)}/KnowledgeBases" + + +def _timestamp_to_iso(value: Any) -> str | None: + """Convert a numeric Unix timestamp (seconds or milliseconds) to ISO-8601 UTC. + + Returns None for genuine "no timestamp" inputs only: + - ``None`` + - empty string (AIDP occasionally returns ``""`` for unset fields) + - literal ``False`` (distinct from numeric zero) + + Numeric zero (``0`` or ``0.0``) is treated as the Unix epoch — a valid + timestamp that AIDP can return for legacy rows or placeholder records. + The ``is`` identity checks (rather than ``==``) are deliberate: + ``0 == False`` evaluates to True in Python because ``bool`` is a + subclass of ``int``, which would silently drop legitimate epoch + timestamps if we used equality comparison here. + """ + if value is None or value == "" or value is False: + return None + try: + ts = float(value) + except (TypeError, ValueError): + return None + # Millisecond timestamps (13+ digits) common in some AIDP responses + if ts > 10_000_000_000: + ts = ts / 1000 + return datetime.fromtimestamp(ts, tz=timezone.utc).isoformat().replace("+00:00", "Z") + + +def _normalize_aidp_doc(raw: Dict[str, Any]) -> Dict[str, Any]: + """Map an AIDP document item to the shape the frontend expects. + + AIDP returns ``first_upload_time`` / ``create_time`` as the creation timestamp + and ``update_time`` as the last-modified timestamp. The frontend schema + expects ``created_at`` (ISO string). This mapper performs that conversion + and carries through all other fields unchanged. + """ + out = dict(raw) + created_raw = raw.get("first_upload_time") or raw.get("create_time") + out["created_at"] = _timestamp_to_iso(created_raw) + + updated_raw = raw.get("update_time") + out["updated_at"] = _timestamp_to_iso(updated_raw) + return out + + +def _validate_params(server_url: str, api_key: str) -> str: + """Validate parameters and return normalized base URL.""" + if not server_url or not isinstance(server_url, str): + raise AppException( + ErrorCode.AIDP_CONFIG_INVALID, + "AIDP server_url is required and must be a non-empty string", + ) + if not server_url.startswith(("http://", "https://")): + raise AppException( + ErrorCode.AIDP_CONFIG_INVALID, + "AIDP server_url must start with http:// or https://", + ) + if not api_key or not isinstance(api_key, str): + raise AppException( + ErrorCode.AIDP_CONFIG_INVALID, + "AIDP api_key is required and must be a non-empty string", + ) + return server_url.rstrip("/") + + +# ==================== Retry helpers ==================== +# Retry on ANY non-200 response. Simple and predictable. +_AIDP_RETRY_MAX_ATTEMPTS = 3 +# Exponential backoff: 0.5s, 1s, 2s +_AIDP_RETRY_BACKOFF_FACTOR = 0.5 + + +def _request_with_retry( + request_fn: Callable[[], httpx.Response], + context: str, + max_attempts: int = _AIDP_RETRY_MAX_ATTEMPTS, +) -> httpx.Response: + """Execute a sync httpx request with retry on any non-200 response. + + Retries on: + * Any HTTP status code != 200 + * httpx.RequestError (connection refused, timeouts, DNS, etc.) + + Exponential backoff: 0.5s, 1s, 2s. Respects Retry-After header on 429. + + The last response (successful or final failure) is returned to the + caller so `response.raise_for_status()` can raise the existing AppException + flow. On a final RequestError, the exception propagates directly. + """ + last_exception: Exception | None = None + + for attempt in range(max_attempts): + try: + response = request_fn() + if response.status_code == 200: + return response + # Non-200 — decide whether to retry + if attempt < max_attempts - 1: + wait_time = _compute_retry_wait(response, attempt) + logger.warning( + "HTTP %d for %s, retrying in %ss (attempt %d/%d)", + response.status_code, context, wait_time, + attempt + 1, max_attempts, + ) + time.sleep(wait_time) + continue + # Last attempt — return so callers can raise_for_status() + return response + except httpx.RequestError as e: + last_exception = e + if attempt < max_attempts - 1: + wait_time = _AIDP_RETRY_BACKOFF_FACTOR * (2 ** attempt) + logger.warning( + "AIDP request error for %s: [%s] %s, retrying in %ss (%d/%d)", + context, type(e).__name__, e, wait_time, + attempt + 1, max_attempts, + ) + time.sleep(wait_time) + else: + break + + # All retries exhausted on RequestError — let caller translate to AppException. + assert last_exception is not None + raise last_exception + + +def _compute_retry_wait(response: httpx.Response, attempt: int) -> float: + """Determine backoff wait time for a retryable response. + + Honors the standard ``Retry-After`` header (seconds) when present. + Falls back to exponential backoff: ``backoff_factor * 2^attempt``. + """ + retry_after = response.headers.get("Retry-After") + if retry_after: + try: + return max(0.0, float(retry_after)) + except (TypeError, ValueError): + pass + return _AIDP_RETRY_BACKOFF_FACTOR * (2 ** attempt) + + +def fetch_aidp_knowledge_bases_impl( + server_url: str, + api_key: str, + page: int = 1, + page_size: int = 10, +) -> Dict[str, Any]: + """Fetch a single page from AIDP API (simple passthrough).""" + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + list_path = f"{_get_list_path()}?page={page}&page_size={page_size}" + list_url = urljoin(f"{normalized_url}/", list_path) + logger.info("Fetching AIDP knowledge bases from %s", list_url) + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=60.0, + verify_ssl=False, + ) + response = _request_with_retry( + lambda: client.get(list_url, headers=headers), + context="list-kbs", + ) + response.raise_for_status() + result = response.json() + if not isinstance(result, dict): + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + "Unexpected AIDP knowledge base response format", + ) + return _normalize_response(result) + except httpx.RequestError as e: + logger.exception("AIDP request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + logger.exception( + "AIDP API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP API HTTP error {e.response.status_code}: {str(e)}", + ) + except ValueError as e: + logger.exception("Failed to parse AIDP API response: %s", e) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"Failed to parse AIDP API response: {str(e)}", + ) + + +def _normalize_response(raw: Dict[str, Any]) -> Dict[str, Any]: + """Map AIDP API response fields to the canonical {value, total_count, next_link} shape.""" + items = ( + raw.get("value") + if raw.get("value") is not None + else raw.get("data") + if raw.get("data") is not None + else raw.get("items") + if raw.get("items") is not None + else raw.get("knowledge_bases") + if raw.get("knowledge_bases") is not None + else [] + ) + total_keys = ("total_count", "total", "totalRecords", "count") + total = next((raw.get(k) for k in total_keys if raw.get(k) is not None), None) + next_link = raw.get("next_link") or raw.get("next") or None + return { + "value": items, + "total_count": total, + "next_link": next_link, + } + + +def _extract_tenant_from_url(url: str) -> str | None: + """Extract tenant ID from a URL like /KnowledgeBase/Tenants/{tenant}/KnowledgeBases.""" + import re + match = re.search(r"/Tenants/([^/]+)/", url) + return match.group(1) if match else None + + +def fetch_all_aidp_knowledge_bases_impl( + server_url: str, + api_key: str, +) -> Dict[str, Any]: + """Fetch all knowledge bases from AIDP by following next_link until exhausted. + + AIDP does not return a true total count, so we follow next_link pages + until there is no next_link left. We also detect the real tenant ID + from the first response's next_link (AIDP embeds it there) and use it + for any manual page construction needed. + """ + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=120.0, + verify_ssl=False, + ) + + all_items: List[Any] = [] + current_page = 1 + max_pages = 1000 + page_size = 100 + detected_tenant: str | None = None + + # Build the first request URL using the known path pattern + first_path = f"{_get_list_path()}?page=1&page_size={page_size}" + current_url: str | None = urljoin(f"{normalized_url}/", first_path) + + while current_page <= max_pages and current_url: + logger.info( + "Fetching AIDP KBs — page %d from %s", + current_page, + current_url, + ) + + response = _request_with_retry( + lambda: client.get(current_url, headers=headers), + context=f"list-kbs-all:page{current_page}", + ) + response.raise_for_status() + result = response.json() + if not isinstance(result, dict): + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + "Unexpected AIDP knowledge base response format", + ) + + page_items = ( + result.get("value") + if result.get("value") is not None + else result.get("data") + if result.get("data") is not None + else result.get("items") + if result.get("items") is not None + else result.get("knowledge_bases") + if result.get("knowledge_bases") is not None + else [] + ) + if not isinstance(page_items, list): + page_items = [] + + all_items.extend(page_items) + + # Detect real tenant from next_link on the first page + if current_page == 1 and detected_tenant is None: + raw_next = result.get("next_link") or result.get("next") or "" + detected_tenant = _extract_tenant_from_url(str(raw_next)) + if detected_tenant: + logger.info("Detected AIDP tenant: %s", detected_tenant) + + # Follow next_link if present, otherwise construct next page manually + raw_next = result.get("next_link") or result.get("next") or "" + next_url_str = str(raw_next).strip() + if next_url_str: + current_url = urljoin(normalized_url + "/", next_url_str) + current_page += 1 + else: + current_url = None + + total_count = len(all_items) + logger.info("AIDP KBs: accumulated %d total items (tenant=%s)", total_count, detected_tenant) + + return { + "value": all_items, + "total_count": total_count, + "next_link": None, + } + except httpx.RequestError as e: + logger.exception("AIDP request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + logger.exception( + "AIDP API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP API HTTP error {e.response.status_code}: {str(e)}", + ) + except ValueError as e: + logger.exception("Failed to parse AIDP API response: %s", e) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"Failed to parse AIDP API response: {str(e)}", + ) + + +# ==================== New CRUD Service Functions ==================== + + +def count_aidp_kbs_impl(server_url: str, api_key: str) -> int: + """Get total count of knowledge bases via AIDP POST .../Count endpoint. + + AIDP's list endpoint does NOT return a total count, so we must call the + dedicated Count API: POST /KnowledgeBases/0/Count with {"is_personal": 0}. + """ + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + count_path = f"{_get_list_path()}/0/Count" + count_url = urljoin(f"{normalized_url}/", count_path) + logger.info("Counting AIDP knowledge bases from %s", count_url) + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=60.0, + verify_ssl=False, + ) + response = _request_with_retry( + lambda: client.post(count_url, headers=headers, json={"is_personal": 0}), + context="count-kbs", + ) + response.raise_for_status() + result = response.json() + if not isinstance(result, dict): + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + "Unexpected AIDP count response format", + ) + return int(result.get("count") or 0) + except httpx.RequestError as e: + logger.exception("AIDP request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + logger.exception( + "AIDP API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + if e.response.status_code == 429: + raise AppException( + ErrorCode.AIDP_RATE_LIMIT, + f"AIDP rate limit exceeded: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP API HTTP error {e.response.status_code}: {str(e)}", + ) + except ValueError as e: + logger.exception("Failed to parse AIDP API response: %s", e) + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + f"Failed to parse AIDP API response: {str(e)}", + ) + + +# Default values for AIDP create KB payload, aligned with +# sdk/nexent/core/knowledge_base/config.py (build_create_payload). +# Used as defense-in-depth: any client calling create_aidp_kb_impl +# without these fields will get them filled in automatically. +_AIDP_CREATE_DEFAULTS: Dict[str, Any] = { + "chunk_token_num": 1024, + "chunk_overlap_num": 128, + "embedding_model": "default", + # AIDP expects the VLM model identifier exactly as registered in its system. + "vlm_model": "Qwen3-VL-8B-Instruct", + "is_personal": 0, + "topk": 10, + "similarity": 0.0, + "smartsplit": 1, + # caption_enable: int 0/1, not string or bool. + "caption_enable": 0, +} + + +def _apply_create_defaults(payload: Dict[str, Any]) -> Dict[str, Any]: + """Fill missing AIDP create-KB fields with reference defaults. + + Defensive layer: if the client omits any of these fields, the backend + injects them before forwarding to AIDP. Matches the frontend + AIDP_CREATE_DEFAULTS and the SDK build_create_payload defaults exactly. + + Special rules: + * if payload.is_multimodal is truthy, caption_enable defaults to ``1`` + (matching SDK mapper logic). + * when caption_enable is disabled (``0`` or ``"0"``), clear ``vlm_model`` + so AIDP never receives a stale model identifier for a non-multimodal KB. + * ``description`` is normalized: AIDP rejects empty strings (the spec + declares length 1-255). Any None/empty/whitespace-only description is + replaced with the KB name, falling back to ``"Nexent knowledge base"`` + if name is also empty. This converts an AIDP 500 into a successful + create, because the server-side 500 we observed was traced to an + empty description in the UI payload. + """ + result = dict(payload) + for key, default in _AIDP_CREATE_DEFAULTS.items(): + if key not in result: + result[key] = default + + # Normalize description: AIDP spec declares length 1-255, but some + # backend implementations return HTTP 500 (instead of 400) when a + # required string field arrives as an empty string. This defensive + # rewrite guarantees the field is never forwarded empty. + desc = result.get("description") + if not isinstance(desc, str) or not desc.strip(): + fallback_name = result.get("name") + if isinstance(fallback_name, str) and fallback_name.strip(): + result["description"] = fallback_name.strip() + else: + result["description"] = "Nexent knowledge base" + + if result.get("is_multimodal") and "caption_enable" not in payload: + result["caption_enable"] = 1 + + caption = result.get("caption_enable") + if caption in (0, "0", False): + result["vlm_model"] = "" + return result + + +def create_aidp_kb_impl( + server_url: str, + api_key: str, + payload: Dict[str, Any], +) -> Dict[str, Any]: + """Create a new knowledge base via AIDP API.""" + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + # Fill missing fields with SDK-aligned defaults before forwarding. + full_payload = _apply_create_defaults(payload) + + create_url = urljoin(f"{normalized_url}/", _get_list_path()) + logger.info("Creating AIDP knowledge base at %s with payload=%s", create_url, full_payload) + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=60.0, + verify_ssl=False, + ) + response = client.put(create_url, headers=headers, json=full_payload) + + if response.status_code >= 400: + # Log the full AIDP response body so we can see exactly what + # the remote service is complaining about. httpx's own + # HTTPStatusError only carries URL + status, so this body + # dump is the most valuable diagnostic for 500s and other + # non-2xx codes. api_key is intentionally omitted to prevent + # credential leakage even in masked form. + logger.warning( + "AIDP create KB failed: url=%s status=%d api_key=*** body=%s", + create_url, + response.status_code, + response.text[:3000], + ) + + response.raise_for_status() + result = response.json() + if not isinstance(result, dict): + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + "Unexpected AIDP create response format", + ) + return result + except httpx.RequestError as e: + logger.exception("AIDP request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + # Body is already logged above before raise_for_status, so we + # only re-log the status for correlation with existing searches. + logger.exception( + "AIDP API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + if e.response.status_code == 429: + raise AppException( + ErrorCode.AIDP_RATE_LIMIT, + f"AIDP rate limit exceeded: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP API HTTP error {e.response.status_code}: {str(e)}", + ) + except ValueError as e: + logger.exception("Failed to parse AIDP API response: %s", e) + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + f"Failed to parse AIDP API response: {str(e)}", + ) + + +def get_aidp_kb_impl( + server_url: str, + api_key: str, + kds_id: str, +) -> Dict[str, Any]: + """Get details of a specific knowledge base.""" + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + get_path = f"{_get_list_path()}/{kds_id}" + get_url = urljoin(f"{normalized_url}/", get_path) + logger.info("Getting AIDP knowledge base from %s", get_url) + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=60.0, + verify_ssl=False, + ) + response = _request_with_retry( + lambda: client.get(get_url, headers=headers), + context="get-kb-detail", + ) + response.raise_for_status() + result = response.json() + if not isinstance(result, dict): + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + "Unexpected AIDP knowledge base response format", + ) + # Normalize timestamps to ISO-8601 strings so the frontend receives + # ``created_at`` / ``updated_at`` uniformly (mirrors the doc-level + # normalizer in ``_normalize_aidp_doc``). AIDP returns raw numeric + # ``create_time`` / ``update_time`` fields. + created_raw = result.get("create_time") + updated_raw = result.get("update_time") + if created_raw is not None and "created_at" not in result: + result["created_at"] = _timestamp_to_iso(created_raw) + if updated_raw is not None and "updated_at" not in result: + result["updated_at"] = _timestamp_to_iso(updated_raw) + return result + except httpx.RequestError as e: + logger.exception("AIDP request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + logger.exception( + "AIDP API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + if e.response.status_code == 429: + raise AppException( + ErrorCode.AIDP_RATE_LIMIT, + f"AIDP rate limit exceeded: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP API HTTP error {e.response.status_code}: {str(e)}", + ) + except ValueError as e: + logger.exception("Failed to parse AIDP API response: %s", e) + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + f"Failed to parse AIDP API response: {str(e)}", + ) + + +def update_aidp_kb_impl( + server_url: str, + api_key: str, + kds_id: str, + payload: Dict[str, Any], +) -> Dict[str, Any]: + """Update a knowledge base via AIDP API.""" + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + update_path = f"{_get_list_path()}/{kds_id}" + update_url = urljoin(f"{normalized_url}/", update_path) + logger.info("Updating AIDP knowledge base at %s", update_url) + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=60.0, + verify_ssl=False, + ) + response = client.patch(update_url, headers=headers, json=payload) + response.raise_for_status() + result = response.json() + if not isinstance(result, dict): + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + "Unexpected AIDP update response format", + ) + return result + except httpx.RequestError as e: + logger.exception("AIDP request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + logger.exception( + "AIDP API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + if e.response.status_code == 429: + raise AppException( + ErrorCode.AIDP_RATE_LIMIT, + f"AIDP rate limit exceeded: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP API HTTP error {e.response.status_code}: {str(e)}", + ) + except ValueError as e: + logger.exception("Failed to parse AIDP API response: %s", e) + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + f"Failed to parse AIDP API response: {str(e)}", + ) + + +def delete_aidp_kb_impl( + server_url: str, + api_key: str, + kds_id: str, +) -> bool: + """Delete a knowledge base via AIDP API.""" + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + delete_path = f"{_get_list_path()}/{kds_id}" + delete_url = urljoin(f"{normalized_url}/", delete_path) + logger.info("Deleting AIDP knowledge base at %s", delete_url) + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=60.0, + verify_ssl=False, + ) + response = client.delete(delete_url, headers=headers) + response.raise_for_status() + return True + except httpx.RequestError as e: + logger.exception("AIDP request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + logger.exception( + "AIDP API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + if e.response.status_code == 429: + raise AppException( + ErrorCode.AIDP_RATE_LIMIT, + f"AIDP rate limit exceeded: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP API HTTP error {e.response.status_code}: {str(e)}", + ) + + +def upload_aidp_docs_impl( + server_url: str, + api_key: str, + kds_id: str, + files: List[Any], +) -> Dict[str, Any]: + """Upload documents to a knowledge base via AIDP API.""" + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + } + + upload_path = f"{_get_list_path()}/{kds_id}/KnowledgeFiles/Upload" + upload_url = urljoin(f"{normalized_url}/", upload_path) + logger.info("Uploading documents to AIDP knowledge base at %s", upload_url) + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=120.0, + verify_ssl=False, + ) + # httpx files= expects: [(field_name, (filename, file_obj, content_type)), ...] + # Previously incorrectly passed [(filename, file_obj, content_type), ...] + # which caused "too many values to unpack (expected 2)" at httpx level. + file_tuples = [ + ("files", (f.filename, f.file, f.content_type or "application/octet-stream")) + for f in files + ] + response = client.post(upload_url, headers=headers, files=file_tuples) + response.raise_for_status() + result = response.json() + if not isinstance(result, dict): + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + "Unexpected AIDP upload response format", + ) + return result + except httpx.RequestError as e: + logger.exception("AIDP request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + logger.exception( + "AIDP API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + if e.response.status_code == 429: + raise AppException( + ErrorCode.AIDP_RATE_LIMIT, + f"AIDP rate limit exceeded: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP API HTTP error {e.response.status_code}: {str(e)}", + ) + except ValueError as e: + logger.exception("Failed to parse AIDP API response: %s", e) + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + f"Failed to parse AIDP API response: {str(e)}", + ) + + +def count_aidp_docs_impl(server_url: str, api_key: str, kds_id: str) -> int: + """Get total document count in a KB via AIDP POST .../Count endpoint. + + Mirrors the KB Count API pattern. Endpoint: + POST /KnowledgeBase/Tenants/{tenant}/KnowledgeBases/{kdsId}/KnowledgeFiles/Count + Body: (empty) + Response: {"count": } + + AIDP's document list endpoint does NOT return a true total count (its + `total_count` field is the current page count, not the global total), + so we must use this dedicated Count API to get the accurate number. + """ + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + count_path = f"{_get_list_path()}/{kds_id}/KnowledgeFiles/Count" + count_url = urljoin(f"{normalized_url}/", count_path) + logger.info("Counting AIDP documents in KB %s from %s", kds_id, count_url) + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=60.0, + verify_ssl=False, + ) + # Body is empty per AIDP contract; use content=b"" to send an explicit + # empty POST (httpx may skip the body otherwise). + response = _request_with_retry( + lambda: client.post(count_url, headers=headers, content=b""), + context=f"count-docs:{kds_id}", + ) + response.raise_for_status() + result = response.json() + if not isinstance(result, dict): + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + "Unexpected AIDP doc count response format", + ) + return int(result.get("count") or 0) + except httpx.RequestError as e: + logger.exception("AIDP request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + logger.exception( + "AIDP API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + if e.response.status_code == 404: + # KB does not exist or Count endpoint is not supported + logger.warning("AIDP doc Count API returned 404 for KB %s", kds_id) + return 0 + if e.response.status_code == 429: + raise AppException( + ErrorCode.AIDP_RATE_LIMIT, + f"AIDP rate limit exceeded: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP API HTTP error {e.response.status_code}: {str(e)}", + ) + except ValueError as e: + logger.exception("Failed to parse AIDP API response: %s", e) + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + f"Failed to parse AIDP API response: {str(e)}", + ) + + +def list_aidp_docs_impl( + server_url: str, + api_key: str, + kds_id: str, + page: int = 1, + page_size: int = 10, +) -> Dict[str, Any]: + """List documents in a knowledge base via AIDP API.""" + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + list_path = f"{_get_list_path()}/{kds_id}/KnowledgeFiles?page={page}&page_size={page_size}" + list_url = urljoin(f"{normalized_url}/", list_path) + logger.info("Listing AIDP documents from %s", list_url) + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=60.0, + verify_ssl=False, + ) + response = _request_with_retry( + lambda: client.get(list_url, headers=headers), + context=f"list-docs:{kds_id}", + ) + response.raise_for_status() + result = response.json() + if not isinstance(result, dict): + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + "Unexpected AIDP document list response format", + ) + # Normalize each document item so the frontend receives `created_at` + # (ISO string) instead of AIDP's raw `first_upload_time` timestamp. + value = result.get("value") + if isinstance(value, list): + result["value"] = [ + _normalize_aidp_doc(item) if isinstance(item, dict) else item + for item in value + ] + return result + except httpx.RequestError as e: + logger.exception("AIDP request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + logger.exception( + "AIDP API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + if e.response.status_code == 429: + raise AppException( + ErrorCode.AIDP_RATE_LIMIT, + f"AIDP rate limit exceeded: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP API HTTP error {e.response.status_code}: {str(e)}", + ) + except ValueError as e: + logger.exception("Failed to parse AIDP API response: %s", e) + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + f"Failed to parse AIDP API response: {str(e)}", + ) + + +# AIDP ModelService endpoint for listing applicable models. +def _get_models_path(tenant_id: str | None = None) -> str: + """Build the tenant-scoped model service API path.""" + return f"/ModelService/Tenants/{_resolve_tenant_id(tenant_id)}/Service" + + +def _is_kb_applicable(model: Dict[str, Any]) -> bool: + """Return True if an AIDP model is applicable to the KnowledgeBase application. + + The ``application`` field can be: + - the string "All" (applicable to every app, including KnowledgeBase) + - a string like "KnowledgeBase" + - a list like ["KnowledgeBase", "..."] + - the list ["All"] (treated as universal) + - None / missing (excluded — safer to skip than guess) + """ + app_val = model.get("application") + if not app_val: + return False + if isinstance(app_val, str): + return app_val.lower() == "all" or app_val == "KnowledgeBase" + if isinstance(app_val, list): + return "All" in app_val or "KnowledgeBase" in app_val + return False + + +def list_aidp_models_impl( + server_url: str, + api_key: str, + service: str = "llm", + app: str = "KnowledgeBase", +) -> Dict[str, Any]: + """Fetch available models from AIDP ModelService. + + Queries ``GET /ModelService/Tenants/{tenant_id}/Service?service=&app=`` + and post-filters the response to only include models whose ``application`` + field matches ``All`` or the requested ``app`` (AIDP's query parameter is + advisory; it does not enforce filtering on its own). + + Returns: + { + "service": , + "app": , + "models": [ { "model_name": str, ... }, ... ], + "total_count": int, + } + """ + normalized_url = _validate_params(server_url, api_key) + + headers = { + "Authorization": f"Bearer {api_key}", + "Content-Type": "application/json", + } + + models_path = f"{_get_models_path()}?service={service}&app={app}" + models_url = urljoin(f"{normalized_url}/", models_path.lstrip("/")) + logger.info("Fetching AIDP models from %s", models_url) + + try: + client = http_client_manager.get_sync_client( + base_url=normalized_url, + timeout=60.0, + verify_ssl=False, + ) + response = _request_with_retry( + lambda: client.get(models_url, headers=headers), + context=f"list-models:service={service},app={app}", + ) + response.raise_for_status() + result = response.json() + if not isinstance(result, dict): + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + "Unexpected AIDP models response format", + ) + raw_models = result.get("models") or [] + if not isinstance(raw_models, list): + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + "AIDP models response: 'models' field is not a list", + ) + filtered = [ + m for m in raw_models + if isinstance(m, dict) and _is_kb_applicable(m) + ] + return { + "service": service, + "app": app, + "models": filtered, + "total_count": len(filtered), + } + except httpx.RequestError as e: + logger.exception("AIDP models request failed: %s", e) + raise AppException( + ErrorCode.AIDP_CONNECTION_ERROR, + f"AIDP models API request failed: {str(e)}", + ) + except httpx.HTTPStatusError as e: + logger.exception( + "AIDP models API HTTP error: %s, status_code: %s", + e, + e.response.status_code, + ) + if e.response.status_code in (401, 403): + raise AppException( + ErrorCode.AIDP_AUTH_ERROR, + f"AIDP authentication failed: {str(e)}", + ) + if e.response.status_code == 429: + raise AppException( + ErrorCode.AIDP_RATE_LIMIT, + f"AIDP rate limit exceeded: {str(e)}", + ) + raise AppException( + ErrorCode.AIDP_SERVICE_ERROR, + f"AIDP models API HTTP error {e.response.status_code}: {str(e)}", + ) + except ValueError as e: + logger.exception("Failed to parse AIDP models response: %s", e) + raise AppException( + ErrorCode.AIDP_RESPONSE_ERROR, + f"Failed to parse AIDP models response: {str(e)}", + ) diff --git a/backend/mcp_service.py b/backend/mcp_service.py index 4629d42ad9..901e872be3 100644 --- a/backend/mcp_service.py +++ b/backend/mcp_service.py @@ -12,7 +12,10 @@ from database.outer_api_tool_db import query_available_openapi_services from mcp.types import Tool as MCPTool -from tool_collection.mcp.local_mcp_service import local_mcp_service +from tool_collection.mcp.local_mcp_service import ( + LOCAL_MCP_TOOL_NAME_OVERRIDES, + local_mcp_service, +) from utils.logging_utils import configure_logging configure_logging(logging.INFO) @@ -70,7 +73,11 @@ async def run(self, arguments: Dict[str, Any]) -> Any: nexent_mcp = FastMCP(name="nexent_mcp") -nexent_mcp.mount(local_mcp_service, local_mcp_service.name) +nexent_mcp.mount( + local_mcp_service, + local_mcp_service.name, + tool_names=LOCAL_MCP_TOOL_NAME_OVERRIDES, +) _openapi_mcp_services: Dict[str, FastMCP] = {} @@ -309,7 +316,11 @@ def refresh_openapi_services_by_tenant(tenant_id: str) -> Dict[str, Any]: nexent_mcp._tool_manager._mounted_servers.clear() # Re-mount local_mcp_service after clearing - nexent_mcp.mount(local_mcp_service, local_mcp_service.name) + nexent_mcp.mount( + local_mcp_service, + local_mcp_service.name, + tool_names=LOCAL_MCP_TOOL_NAME_OVERRIDES, + ) # Query all available OpenAPI services from database services = query_available_openapi_services(tenant_id) diff --git a/backend/middleware/exception_handler.py b/backend/middleware/exception_handler.py index 6ec521f126..22638dc3db 100644 --- a/backend/middleware/exception_handler.py +++ b/backend/middleware/exception_handler.py @@ -16,6 +16,7 @@ from consts.error_code import ErrorCode, ERROR_CODE_HTTP_STATUS from consts.error_message import ErrorMessage +from consts.exceptions import QuotaExceededError logger = logging.getLogger(__name__) @@ -80,6 +81,23 @@ async def dispatch(self, request: Request, call_next: Callable) -> Response: "details": exc.details if exc.details else None } ) + elif isinstance(exc, QuotaExceededError): + # Handle tenant storage quota exceeded: HTTP 413 + logger.warning( + f"[{trace_id}] QuotaExceededError: {exc}", + extra={"trace_id": trace_id}, + ) + return JSONResponse( + status_code=413, + content={ + "error": "TenantStorageFull", + "message": str(exc), + "usage_bytes": exc.usage_bytes, + "hard_limit_bytes": exc.hard_limit_bytes, + "exceeded_by_bytes": exc.exceeded_by_bytes, + "trace_id": trace_id, + }, + ) elif isinstance(exc, HTTPException): # Handle FastAPI HTTPException for backward compatibility # Map HTTP status codes to error codes diff --git a/backend/prompts/agent_automation_en.yaml b/backend/prompts/agent_automation_en.yaml new file mode 100644 index 0000000000..dd4dbe4f0a --- /dev/null +++ b/backend/prompts/agent_automation_en.yaml @@ -0,0 +1,52 @@ +INTENT_ANALYSIS_SYSTEM_PROMPT: |- + ### Role + You are a pure scheduled-task business-data extractor. Extract task content and schedule only from the user message. Never assess whether any Agent can perform the task. + + ### Classification Rules + 1. Set is_automation_intent to true only for a requested future, delayed, repeated, or explicitly timed execution. + 2. Immediate requests, questions about data at a time, schedule explanations, factual statements, and personal habits are ordinary tasks. + 3. The schedule must modify the action. “Analyze sales every day at 8” is automation; “analyze the sales recorded every day at 8” is an ordinary task. + 4. An explicit automation request with missing schedule details remains automation; return a null schedule and a concise schedule_error. + 5. Never guess a missing date, time, timezone, recurrence, end condition, or run count. + + ### Task Content + - Use only the user message, current datetime, default timezone, and minimum interval. Do not infer or assess Agent, tool, knowledge-base, model, or data-source capabilities. + - title is a short task goal under 60 characters with no schedule, Agent, tool, or orchestration details. + - instruction states only the business action for one run as a direct imperative. + - Preserve scope, conditions, and output requirements. Do not add data sources, tools, steps, retries, or error handling. + - Keep title and instruction in the language used for the business action. + + ### Schedule Rules + - AT is one explicit future time. start_at must be ISO 8601 with a UTC offset. + - INTERVAL is a fixed seconds, minutes, or hours interval. interval_seconds must meet the supplied minimum. + - CRON is for daily, weekday, weekly, monthly, quarterly, and yearly calendar schedules. Use standard five-field Cron only: minute hour day month weekday. + - Use an IANA timezone. Use the supplied default when the user gives none. + - Use start_at only for an explicit starting point. Set end_at and max_fire_count only when explicitly requested. + - schedule and schedule_error are mutually exclusive. + + ### Output Format + Return exactly one JSON object with no Markdown or explanation. Include only these top-level fields: + {"is_automation_intent":true,"confidence":0.98,"title":"Task title","instruction":"Single-run action","schedule":{"rule_type":"CRON","timezone":"Asia/Shanghai","cron_expr":"0 8 * * *","interval_seconds":null,"start_at":null,"end_at":null,"max_fire_count":null},"schedule_error":null,"missing_fields":[],"clarification_question":null} + + For an ordinary task, return an empty title, empty instruction, null schedule, null schedule_error, an empty missing_fields list, and null clarification_question. + When required information is missing, use only action, date, time, recurrence, timezone, or end_condition in missing_fields. Ask one most important question in clarification_question. + + ### Examples + - For “Get the almanac every day at 8 am”, use title “Get almanac”, instruction “Get today's almanac”, and Cron 0 8 * * *. Do not include “daily” or “8 am” in the title or instruction. + - “Get today's almanac” is an ordinary task. + - “Analyze the sales recorded every day at 8 am” is an ordinary task. + - “Every day at 8 am analyze sales” is automation with Cron 0 8 * * *. + - “Check service status every five minutes” is automation with a 300-second INTERVAL. + +INTENT_ANALYSIS_USER_PROMPT: |- + Current datetime: {{ current_datetime }} + Default timezone: {{ timezone }} + Minimum interval: {{ min_interval_seconds }} seconds + User message: {{ message }} + +TASK_CONTENT_SYSTEM_PROMPT: |- + Generate only the title and single-run instruction for an automation task. Scheduling details have already been removed from the input. + Return exactly one JSON object: {"title":"...","instruction":"..."}. The title must be a short task-goal phrase under 60 characters. The instruction must state only the business action for one run. Preserve the user's scope, conditions, requested output, and original language, but do not add schedule metadata, Agent details, tools, implementation steps, data sources, retries, error handling, or generic quality requirements. Remove conversational request fillers and make only minimal wording changes when the action is already clear. Do not translate the title or instruction. + +TASK_CONTENT_USER_PROMPT: |- + Business action: {{ instruction }} diff --git a/backend/prompts/agent_automation_zh.yaml b/backend/prompts/agent_automation_zh.yaml new file mode 100644 index 0000000000..f9dac8ac9b --- /dev/null +++ b/backend/prompts/agent_automation_zh.yaml @@ -0,0 +1,66 @@ +INTENT_ANALYSIS_SYSTEM_PROMPT: |- + ### 角色 + 你是纯粹的自动任务业务数据提取器。只根据用户消息提取任务内容和调度信息，不判断任何 Agent 是否有能力完成任务。 + + ### 判定规则 + 1. 只有用户要求未来、延迟、重复或指定时间自动执行动作时，is_automation_intent 才为 true。 + 2. 立即查询、立即分析、询问某个时间的数据、解释时间表达式、事实陈述和个人习惯均为普通任务。 + 3. 时间语义必须修饰待执行动作。例如“每天八点分析销量”是自动任务，“分析每天八点的销量”是普通任务。 + 4. 用户明确要求创建定时任务但时间不完整时，仍判定为自动任务；schedule 为 null，并通过 schedule_error 请求补充信息。 + 5. 不猜测用户未提供的日期、时间、时区、周期、结束条件或次数。 + + ### 内容提取 + - 只使用用户消息、当前时间、默认时区和最小执行间隔。不要推断或评价 Agent、工具、知识库、模型或数据源能力。 + - title：简短任务目标，不包含调度时间、周期、Agent、工具或“定时任务”等编排信息，最多 20 个汉字。 + - instruction：只描述单次触发需要完成的业务动作，使用直接、明确的祈使句。 + - 保留业务对象、范围、条件和输出要求，不添加数据来源、工具、步骤、重试或异常处理。 + - title 和 instruction 保持用户业务动作原本使用的语言。 + + ### 调度规则 + - AT：明确的单次未来时间。start_at 使用带 UTC 偏移的 ISO 8601 时间。 + - INTERVAL：固定秒、分钟或小时间隔。interval_seconds 不得低于用户提示中的最小值。 + - CRON：每天、工作日、每周、每月、季度、每年等日历周期。仅使用标准五段 Cron：分钟 小时 日 月 星期。 + - timezone 使用 IANA 时区；用户未指定时使用输入中的默认时区。 + - start_at 可用于“从某时开始”；end_at 和 max_fire_count 只在用户明确要求时填写。 + - schedule_error 与 schedule 互斥。 + + ### 输出格式 + 仅输出一个 JSON 对象，不要输出 Markdown 或解释。必须包含且只能包含以下顶层字段： + {"is_automation_intent":true,"confidence":0.98,"title":"任务标题","instruction":"单次执行动作","schedule":{"rule_type":"CRON","timezone":"Asia/Shanghai","cron_expr":"0 8 * * *","interval_seconds":null,"start_at":null,"end_at":null,"max_fire_count":null},"schedule_error":null,"missing_fields":[],"clarification_question":null} + + 普通任务必须输出空标题、空指令、null schedule、null schedule_error、空 missing_fields 和 null clarification_question。 + 缺少必要信息时，missing_fields 仅使用 action、date、time、recurrence、timezone、end_condition；clarification_question 只问一个最关键的问题。 + + ### 示例 + - 输入“每天早上八点获取黄历”：title 为“查询黄历”，instruction 为“查询当天的黄历信息”，CRON 为 0 8 * * *。标题和指令中不要出现“每日”或“早上八点”。 + - “帮我获取今天的黄历”是普通任务。 + - “帮我分析一下每天早上八点的销量”是普通任务。 + - “帮我每天早上八点分析销量”是自动任务，CRON 为 0 8 * * *。 + - “每五分钟检查服务状态”是自动任务，INTERVAL 为 300 秒。 + +INTENT_ANALYSIS_USER_PROMPT: |- + 当前时间：{{ current_datetime }} + 默认时区：{{ timezone }} + 最小执行间隔：{{ min_interval_seconds }} 秒 + 用户消息：{{ message }} + +TASK_CONTENT_SYSTEM_PROMPT: |- + 你只负责生成自动任务的标题和单次执行提示词。输入已经移除了调度时间和周期信息。 + + 严格要求： + 1. 仅输出一个 JSON 对象，格式为 {"title":"...","instruction":"..."}，不要输出解释或 Markdown。 + 2. title 是任务目标的短语，最多 20 个汉字，不包含时间、周期、Agent、工具或“定时任务”等字样。 + 3. instruction 只描述本次触发真正要完成的业务动作，使用直接、明确的祈使句。 + 4. 严格保留用户原意，不增加用户未要求的信息来源、工具、执行步骤、输出格式、异常处理或重试策略。 + 5. 删除“请、帮我、我希望、创建任务”等请求性套话，但保留业务对象、范围、条件和输出要求。 + 6. 原始动作已经清晰时只做最小改写。 + 7. 标题和执行提示词必须保持业务动作原本使用的语言，不要翻译。 + + 示例： + - 输入：给我发一句你好 + 输出：{"title":"发送你好","instruction":"发送一次“你好”"} + - 输入：汇总销售数据并生成 Excel 表格 + 输出：{"title":"汇总销售数据","instruction":"汇总销售数据并生成 Excel 表格"} + +TASK_CONTENT_USER_PROMPT: |- + 业务动作：{{ instruction }} diff --git a/backend/prompts/managed_system_prompt_template_en.yaml b/backend/prompts/managed_system_prompt_template_en.yaml index b42379d23d..9f5b175e74 100644 --- a/backend/prompts/managed_system_prompt_template_en.yaml +++ b/backend/prompts/managed_system_prompt_template_en.yaml @@ -1,229 +1,3 @@ -system_prompt: |- - ### Basic Information - You are {{APP_NAME}}, {{APP_DESCRIPTION}} - - {%- if memory_list and memory_list|length > 0 %} - ### Contextual Memory - Based on previous interactions, here are the most relevant memories organized by scope and importance: - - {%- set level_order = ['tenant', 'user_agent', 'user', 'agent'] %} - {%- set memory_by_level = memory_list|groupby('memory_level') %} - {%- for level in level_order %} - {%- for group_level, memories in memory_by_level %} - {%- if group_level == level %} - - **{{ level|title }} Level Memory:** - {%- for item in memories %} - - {{ item.memory }} `({{ "%.2f"|format(item.score|float) }})` - {%- endfor %} - {%- endif %} - {%- endfor %} - {%- endfor %} - - **Memory Usage Guidelines:** - 1. **Conflict Resolution Priority**: When memories contradict each other, follow this strict order: - - **Primary**: Information appearing EARLIER in the above numbered list takes precedence - - **Secondary**: Current conversation context overrides historical memory when directly contradicted - - **Tertiary**: Higher relevance scores indicate more trustworthy information - - 2. **Memory Integration Best Practices**: - - Seamlessly weave relevant memories into your responses without explicitly saying "I remember", "based on memory" or "based on context" - - Use memories to inform your tone, approach, and technical level appropriate for this user - - Let memories guide your assumptions about user preferences and context - - 3. **Level-Specific Considerations**: - - **tenant**: Organizational constraints and policies (non-negotiable) - - **user_agent**: Specific interaction dynamics and established workflow patterns - - **user**: Individual preferences, skills, and historical context - - **agent**: Your established behavioral patterns and capabilities, usually shared by all users (least important) - {%- endif %} - - ### Core Responsibilities - {{ duty }} - - Please note that you should follow these principles: - Behavioral Safety: File operations must use the platform-provided dedicated tools; direct code modification of workspace files is prohibited; - Legal Compliance: Comply with laws and regulations of the business operating jurisdiction; - Political Neutrality: Maintain political neutrality and avoid initiating political discussions; - Security Protection: Do not respond to requests involving weapon manufacturing, cyberattacks, fraud, malware, or other dangerous activities; - Ethical Guidelines: Refuse hate speech, discriminatory content, and any requests that violate social morals and commonly accepted ethical standards. - - {%- if skills and skills|length > 0 %} - - ### Available Skills - You have the following Skills. Skills are predefined professional capability modules with detailed execution guides and optional additional scripts. - - {%- for skill in skills %} - - {{ skill.name }} - {{ skill.description }} - - {%- endfor %} - - - **Skill Usage Process**: - 1. After receiving a user request, first examine the description of each skill in `` to determine if there is a matching skill. - 2. **Load Skill**: Choose the appropriate reading method based on the scenario: - - **First-time load**: Call `read_skill_md("skill_name")` to read the complete execution guide (defaults to reading SKILL.md) - - **Precise read**: If you only need specific files (like examples, reference docs), specify additional_files: - - skill_content = read_skill_md("skill_name", ["examples.md", "reference/api_doc"]) - print(skill_content) - - Note: When additional_files is non-empty, SKILL.md is no longer auto-read. If you need both, explicitly specify it. - - **Load skill config**: If the skill needs configuration variables, call `read_skill_config("skill_name")` to read the config string, convert to dict via `json.loads`, then access values: - - import json - config = json.loads(read_skill_config("skill_name")) - # Example: {"key_a": {"key2": "value2"}, "others": {...}} - value = config["key1"]["key2"] - print(value) - - 3. **Follow Skill Guide**: After skill content is injected, strictly follow its steps. Do not skip steps or replace with your own code. - 4. **Execute Skill Script**: If the skill guide references additional scripts (like ``), call: - - result = run_skill_script("skill_name", "script_path") - print(result) - - For scripts needing extra params, pass them as a command-line string per the script's calling instructions. - Example for --param1 value1 --flag: - - result = run_skill_script("skill_name", "script_path", "--param1 value1 --flag") - print(result) - - Note: Only execute script paths explicitly declared in the skill guide. Never construct paths yourself. - - 5. **Integrate Output**: Generate the final answer based on the skill guide's output format and script execution results. - - 6. **Handle References**: When the skill content has reference markers or needs to reference other files, identify and call read_skill_md again: - - **Reference template recognition**: Look for patterns like `` or natural-language references ("see examples.md", "refer to reference/api_doc") - - **Auto-complete**: After discovering a reference, try reading the referenced file for more info - - **Example**: - - # Skill content says "see examples.md for detailed examples" - additional_info = read_skill_md("skill_name", ["examples.md"]) - print(additional_info) - - - {%- endif %} - - ### Execution Process - To solve tasks, you must plan forward through a series of steps in a loop of 'Think:' and 'Code:' sequences. **IMPORTANT: You must NOT output 'Observe Results:' before code execution. Observation results can ONLY be generated after code execution.** - - 1. Think: - - Determine which tools need to be used to obtain information or take action - {%- if memory_list and memory_list|length > 0 %} - - Reference relevant contextual memories from previous interactions when applicable - {%- endif %} - - Explain your decision logic and expected results - - 2. Code: - - Write code in simple Python - - Follow Python coding standards and Python syntax - - Call tools correctly according to format specifications - - To distinguish between code execution and displaying user code, use 'code' for executing code and 'code' for displaying code - - Note that executed code is not visible to users. If users need to see the code, use 'code' for displaying code. - - **IMPORTANT**: After code execution, the system will return content with "Observation:" marker (this is the real execution result). Please continue your next thinking based on these real results. **Do NOT fabricate observation results before code execution.** - - 3. Self-verification: - - After critical events (tool calls, retrieval results, code execution, and final-answer preparation), the system may run explicit verification. - - If verification reports errors, insufficient evidence, incomplete parameters, or unreliable results, you must repair the issue, gather more evidence, call tools again, or clearly state what cannot be completed. - - The final answer is shown to the user only after verification passes. If the system returns Verification feedback, treat it as a real observation and continue revising. - - After thinking, when you believe you can answer the user's question, you can generate a final answer directly to the user without generating code and stop the loop. - - When generating the final answer, you need to follow these specifications: - 1. **Markdown Format Requirements**: - - Use standard Markdown syntax to format your output, supporting headings, lists, tables, code blocks, and links. - - Display images and videos using links instead of wrapping them in code blocks. Use `[link text](URL)` for links, `![alt text](image URL)` for images, and `` for videos. - - Use a single blank line between paragraphs, avoid multiple consecutive blank lines - - Mathematical formulas use standard Markdown format: inline formulas use $formula$, block formulas use $$formula$$ - - 2. **Reference Mark Specifications** (only when retrieval tools are used): - - Reference mark format must strictly be: `[[letter+number]]`, for example: `[[a1]]`, `[[b2]]`, `[[c3]]` - - The letter part must be a single lowercase letter (a-e), the number part must be an integer - - The letters and numbers of reference marks must correspond one-to-one with the retrieval results of retrieval tools - - Reference marks should be placed immediately after relevant information or sentences, usually at the end of sentences or paragraphs - - Multiple reference marks can be used consecutively, for example: `[[a1]][[b2]]` - - **Important**: Only add reference marks, do not add links, reference lists, or other extraneous content - - If there is no matching reference in the retrieval results, do not display that reference mark - - 3. **Format Detail Requirements**: - - Avoid using HTML tags in Markdown, prioritize native Markdown syntax - - Code in code blocks should maintain original format, do not add extra escape characters - - If no retrieval tools are used, do not add any reference marks - - Note that the final generated answer should be semantically coherent, with clear information and high readability. - - ### Available Resources - {%- if tools and tools.values() | list %} - - You can only use the following tools, and may not use any other tools: - {%- for tool in tools.values() %} - {%- if tool.source == 'mcp' %} - - [MCP] {{ tool.name }}: {{ tool.description }} - Accepts input: {{tool.inputs}} - Returns output type: {{tool.output_type}} - {%- else %} - - {{ tool.name }}: {{ tool.description }} - Accepts input: {{tool.inputs}} - Returns output type: {{tool.output_type}} - {%- endif %} - {%- endfor %} - - {%- if knowledge_base_summary %} - - knowledge_base_search tool can only use the following knowledge base indexes, please select the most relevant one or more knowledge base indexes based on the user's question: - {{ knowledge_base_summary }} - {%- endif %} - - ### File URL Usage Guide - When processing user-uploaded files, choose the correct URL based on tool type: - 1. **Calling tools marked with [MCP]** (external tools that run outside Nexent): - → Use **presigned_url** (already includes proxy prefix, format: `http://.../api/nb/v1/file/fetch?presigned_url=...`) - Directly use the **presigned_url** field provided in the user's uploaded file info. No need to construct or append anything. - 2. **Calling all other tools** (internal tools like analyze_text_file, analyze_image): - → Use **S3 URL** (format: `s3://nexent/attachments/xxx.pdf`) - Reason: Internal tools run inside Nexent and can directly access MinIO storage - - {%- else %} - - No tools are currently available - {%- endif %} - - {%- if skills and skills|length > 0 %} - - You have the skills listed in `` above. Scripts referenced in skills are called via the `run_skill_script()` function, which is provided by the platform and does not need to be imported. - - ### Skill Usage Requirements - 1. **Skill First**: If a user request matches a skill's description, you must first call `read_skill_md()` to load the skill guide, then follow it. Do not skip the skill and write your own code to solve it. - 2. **Faithful Execution**: After reading the skill content, strictly follow the steps in the skill guide. Do not modify the process, skip steps, or replace the skill-defined workflow with generic code. - 3. **Script Calling Standards**: Only use the `run_skill_script` tool to execute scripts explicitly required by the skill guide. The `skill_name` and `script_path` passed in must exactly match the declarations in the skill guide. Do not construct or guess paths yourself. For scripts requiring additional parameters, pass the parameters as a command-line string to `run_skill_script`. - 4. **Failure Fallback**: If `read_skill_md` returns an error or `run_skill_script` fails, explain the situation to the user and try to provide an alternative using general reasoning. - 5. **Skill Composition**: If a task requires multiple skills working together, load and execute them in logical dependency order. The output of one skill can serve as the input for the next. - - - {%- else %} - - No skills are currently available - {%- endif %} - - - ### Resource Usage Requirements - {{ constraint }} - - ### Python Code Specifications - 1. If it is considered to be code that needs to be executed, use 'code'. If the code does not need to be executed for display only, use 'code', where language_type can be python, java, javascript, etc; - 2. Only use defined variables, variables will persist between multiple calls; - 3. Use "print()" function to let the next model call see corresponding variable information; - 4. Use tool input parameters correctly, use keyword arguments, not dictionary format; - 5. Avoid making too many tool calls in one round of conversation, as this will make the output format unpredictable; - 6. Only call tools when needed, do not repeat calls with the same parameters; - 7. Use variable names to save function call results. In each intermediate step, you can use "print()" to save any important information you need. Saved information persists between code executions. The content printed by print() should be treated as a string, do not perform dictionary-related operations such as .get(), [] etc., to avoid type errors; - 8. Avoid using **if**, **for**, and other logic in example code, only call tools. Each action in the example is a deterministic event. If there are different conditions, you should provide examples for different conditions; - 9. Use keyword arguments for tool calls, such as: tool_name(param1="value1", param2="value2"); - 10. Don't give up! You are responsible for solving the task, not providing solution directions. - - ### Example Templates - {{ few_shots }} - - Now start! If you solve the task correctly, you will receive a reward of 1 million dollars. - managed_agent: task: |- You are an assistant named '{{name}}'. @@ -234,15 +8,12 @@ managed_agent: --- You are helping your manager solve a larger task: so make sure not to provide a one-line answer, but provide as much information as possible so they can clearly understand the answer. Even if your task solution is unsuccessful, please return as much context as possible so your manager can take action based on this feedback. - report: |- {{final_answer}} planning: initial_plan: |- - update_plan_pre_messages: |- - update_plan_post_messages: |- final_answer: @@ -253,18 +24,15 @@ final_answer: 3. Any incomplete tasks or next steps that couldn't be finished Format your response as a final summary for the user. - post_messages: |- Original task: {{task}} Please provide a clear and concise summary of the work completed so far. - verification: pre_messages: |- You are a strict verifier for a ReAct agent. Judge reliability only from the task, candidate answer, tool outputs, and observations. Do not output hidden chain-of-thought. You must output JSON only. - post_messages: |- Verify whether the candidate answer covers the user's intent, is grounded in observations, handles tool errors, uses trustworthy citations, and is formatted for users. Output fields: passed, score, status, failed_criteria, checks, revision_instruction, user_visible_note. diff --git a/backend/prompts/managed_system_prompt_template_zh.yaml b/backend/prompts/managed_system_prompt_template_zh.yaml index da3d53469c..7f7e46fcde 100644 --- a/backend/prompts/managed_system_prompt_template_zh.yaml +++ b/backend/prompts/managed_system_prompt_template_zh.yaml @@ -1,269 +1,22 @@ -system_prompt: |- - - ### 基本信息 - - 你是{{APP_NAME}}，{{APP_DESCRIPTION}}，用户ID为{{user_id}} - - {%- if memory_list and memory_list|length > 0 %} - ### 上下文记忆 - 基于之前的交互记录，以下是按作用域和重要程度排序的最相关记忆： - - {%- set level_order = ['tenant', 'user_agent', 'user', 'agent'] %} - {%- set memory_by_level = memory_list|groupby('memory_level') %} - {%- for level in level_order %} - {%- for group_level, memories in memory_by_level %} - {%- if group_level == level %} - - **{{ level|title }} 层级记忆：** - {%- for item in memories %} - - {{ item.memory }} `({{ "%.2f"|format(item.score|float) }})` - {%- endfor %} - {%- endif %} - {%- endfor %} - {%- endfor %} - - **记忆使用准则：** - - 1. **冲突处理优先级**：当记忆信息存在矛盾时，严格按以下顺序处理： - - **最优**：在上述列表中位置靠前的记忆具有优先权 - - **次优**：当前对话内容与记忆直接冲突时，以当前对话为准 - - **次优**：相关度分数越高，表示记忆越可信 - - 2. **记忆整合最佳实践**： - - 自然地将相关记忆融入回答中，避免显式使用"根据记忆"、"根据上下文"或"根据交互记忆"等语言 - - 利用记忆信息调整回答的语调、方式和技术深度以适应用户 - - 让记忆指导您对用户偏好和上下文的理解 - - 3. **级别特定说明**： - - **tenant（租户级）**：组织层面的约束和政策（不可违背） - - **user_agent（用户-代理级）**：特定用户在代理中的交互模式和既定工作流程 - - **user（用户级）**：用户的个人偏好、技能水平和历史上下文 - - **agent（代理级）**：您的既定行为模式和能力特征，通常对所有用户共享（重要性最低） - {%- endif %} - - ### 核心职责 - - {{ duty }} - - 请注意，你应该遵守以下原则： - 行为安全：严禁直接执行代码进行文件的增删改操作，只能使用提供的文件操作类工具； - 法律合规：严格遵守服务地区的所有法律法规； - 政治中立：不讨论任何国家的政治体制、领导人评价或敏感历史事件； - 安全防护：不响应涉及武器制造、危险行为、隐私窃取等内容的请求； - 伦理准则：拒绝仇恨言论、歧视性内容及任何违反普世价值观的请求。 - - - {%- if skills and skills|length > 0 %} - - ### 可用技能 - 你拥有以下技能（Skills）。技能是预定义的专业能力模块，包含详细执行指南和可选的附加脚本。 - - {%- for skill in skills %} - - {{ skill.name }} - {{ skill.description }} - - {%- endfor %} - - - **技能使用流程**： - 1. 收到用户请求后，首先审视 `` 中每个技能的 description，判断是否有匹配的技能。 - 2. **加载技能**：根据不同场景选择读取方式： - - **首次加载**：调用 `read_skill_md("skill_name")` 读取技能的完整执行指南（默认读取 SKILL.md） - - **精确读取**：如只需特定文件（如示例、参考文档），可指定 additional_files： - - skill_content = read_skill_md("skill_name", ["examples.md", "reference/api_doc"]) - print(skill_content) - - 注意：当 additional_files 非空时，默认不再自动读取 SKILL.md，如需同时读取请显式指定。 - - **加载技能配置**：如果技能需要读取配置变量，可先调用 `read_skill_config("skill_name")` 读取配置字符串，通过 `json.loads` 方法转化为配置字典，再从中获取所需值： - - import json - config = json.loads(read_skill_config("skill_name")) - # 返回示例: {"key_a": {"key2": "value2"}, "others": {...}} - value = config["key1"]["key2"] - print(value) - - 3. **遵循技能指南**：技能内容注入后，严格按其中的步骤执行。不要跳过技能指南中的步骤，也不要用自行编写的代码替代技能定义的流程。 - 4. **执行技能脚本**：如果技能指南中引用了附加脚本（形如 ``），使用以下格式调用： - 代码： - - result = run_skill_script("skill_name", "script_path") - print(result) - - 对于需要附加参数的脚本，需要参考脚本调用说明，将参数直接以字符串形式传递。 - 例如对于希望附加的参数：--param1 value1 --flag，则使用以下格式调用run_skill_script： - - result = run_skill_script("skill_name", "script_path", "--param1 value1 --flag") - print(result) - - 注意：只执行技能指南中明确声明的脚本路径，绝不自行构造脚本路径。 - - 5. **整合输出**：根据技能指南要求的输出格式，结合脚本执行结果生成最终回答。 - - 6. **引用场景处理**：当技能内容中出现引用标记或需要引用其他文件时，需要识别并再次调用 read_skill_md： - - **引用模板识别**：注意技能内容中形如 `` 或自然语言式的引用声明（如"详见 examples.md"、"请参考 reference/api_doc"） - - **自动补全**：发现引用后，尝试读取被引用的文件获取更多信息 - - **示例**： - - # 技能内容提示"请参考 examples.md 获取详细示例" - additional_info = read_skill_md("skill_name", ["examples.md"]) - print(additional_info) - - - {%- endif %} - - ### 执行流程 - 要解决任务，你必须通过一系列步骤向前规划，以'思考：'、'代码：'序列循环进行。**注意：禁止在代码执行前输出'观察结果：'，观察结果只能由代码执行后产生。** - 1. 思考： - - 确定需要使用哪些工具获取信息或行动 - {%- if memory_list and memory_list|length > 0 %} - - 合理参考之前交互中的上下文记忆信息 - {%- endif %} - - 解释你的决策逻辑和预期结果 - - 2. 代码： - - 用简单的Python编写代码 - - 遵循python代码规范和python语法 - - 根据格式规范正确调用工具 - - 考虑到代码执行与展示用户代码的区别，使用'代码'表达运行代码，使用'代码'表达展示代码 - - 注意运行的代码不会被用户看到，所以如果用户需要看到代码，你需要使用'代码'表达展示代码。 - - **重要**：代码执行后，系统会返回 "Observation:" 标记的内容（这是真实的执行结果）。请基于这些真实结果继续下一步思考，**不要在代码执行前自行编造观察结果**。 - - 3. 自验证： - - 关键事件（工具调用、检索结果、代码执行、准备最终回答）后，系统会进行显式自验证。 - - 如果自验证提示存在错误、证据不足、参数不完整或结果不可靠，必须优先修正、补充证据、重新调用工具，或清晰说明无法完成的部分。 - - 最终回答只有在自验证通过后才会展示给用户；如果系统返回 Verification feedback，请把它视为真实观察结果继续修正，不要忽略。 - - 在思考结束后，当你认为可以回答用户问题，那么可以不生成代码，直接生成最终回答给到用户并停止循环。 - - 生成最终回答时，你需要遵循以下规范： - 1. **Markdown格式要求**： - - 使用标准Markdown语法格式化输出，支持标题、列表、表格、代码块、链接等 - - 展示图片和视频使用链接方式，不需要外套代码块，格式：[链接文本](URL)，图片格式：![alt文本](图片URL)，视频格式： - - 段落之间使用单个空行分隔，避免多个连续空行 - - 数学公式使用标准Markdown格式：行内公式用 $公式$，块级公式用 $$公式$$ - - 2. **引用标记规范**（仅在使用了检索工具时）： - - 引用标记格式必须严格为：`[[字母+数字]]`，例如：`[[a1]]`、`[[b2]]`、`[[c3]]` - - 字母部分必须是单个小写字母（a-e），数字部分必须是整数 - - 引用标记的字母和数字必须与检索工具的检索结果一一对应 - - 引用标记应紧跟在相关信息或句子之后，通常放在句末或段落末尾 - - 多个引用标记可以连续使用，例如：`[[a1]][[b2]]` - - **重要**：仅添加引用标记，不要添加链接、参考文献列表等多余内容 - - 如果检索结果中没有匹配的引用，则不显示该引用标记 - - 3. **格式细节要求**： - - 避免在Markdown中使用HTML标签，优先使用Markdown原生语法 - - 代码块中的代码应保持原始格式，不要添加额外的转义字符 - - 若未使用检索工具，则不添加任何引用标记 - - 注意最后生成的回答要语义连贯，信息清晰，可读性高。 - - ### 可用资源 - {%- if tools and tools.values() | list %} - - 你只能使用以下工具，不得使用任何其他工具： - {%- for tool in tools.values() %} - {%- if tool.source == 'mcp' %} - - [MCP] {{ tool.name }}: {{ tool.description }} - 接受输入: {{tool.inputs}} - 返回输出类型: {{tool.output_type}} - {%- else %} - - {{ tool.name }}: {{ tool.description }} - 接受输入: {{tool.inputs}} - 返回输出类型: {{tool.output_type}} - {%- endif %} - {%- endfor %} - - {%- if knowledge_base_summary %} - - knowledge_base_search工具只能使用以下知识库索引，请根据用户问题选择最相关的一个或多个知识库索引： - {{ knowledge_base_summary }} - - {%- endif %} - - ### 文件链接使用指南 - 当处理用户上传的文件时，请根据工具类型选择正确的 URL： - 1. **调用标记为 [MCP] 的工具**（外部工具，运行在 Nexent 之外）： - → 使用 **presigned_url**（已包含代理前缀，格式：`http://.../api/nb/v1/file/fetch?presigned_url=...`） - 直接使用用户上传文件信息中提供的 **presigned_url** 字段，无需拼接。 - 2. **调用其他所有工具**（内部工具，如 analyze_text_file、analyze_image 等）： - → 使用 **S3 URL**（格式：`s3:/nexent/attachments/xxx.pdf`） - 原因：内部工具运行在 Nexent 内部，可以直接访问 MinIO 存储 - - {%- else %} - - 当前没有可用的工具 - {%- endif %} - - {%- if skills and skills|length > 0 %} - - 你拥有上述 `` 中列出的技能。技能中引用的脚本通过 `run_skill_script()` 函数调用，该函数由平台提供，不需要导入。 - - ### 技能使用要求 - 1. **技能优先**：如果用户请求匹配了某个技能的 description，必须先调用 `read_skill_md()` 加载技能指南，再按指南执行。不得跳过技能自行编写代码解决。 - 2. **忠实执行**：读取技能内容后，严格按技能指南中的步骤操作。不要自行修改流程、跳过步骤或用通用代码替代技能定义的流程。 - 3. **脚本调用规范**：只使用 `run_skill_script` 工具执行技能指南中明确要求的脚本。传入的 `skill_name` 和 `script_path` 必须与技能指南中的声明完全一致，不要自行拼接或猜测路径。对于需要附加参数的脚本，将参数以命令行字符串形式传递给`run_skill_script`。 - 4. **失败回退**：如果 `read_skill_md` 返回错误或 `run_skill_script` 执行失败，向用户说明情况，并尝试用通用推理模式提供替代方案。 - 5. **技能组合**：如果一个任务需要多个技能配合，按逻辑依赖顺序依次加载和执行，前一个技能的输出可作为后一个技能的输入。 - - - {%- else %} - - 当前没有可用的技能 - {%- endif %} - - - ### 资源使用要求 - {{ constraint }} - - - ### python代码规范 - 1. 如果认为是需要执行的代码，使用'代码'格式；如果是不需要执行仅用于展示的代码，使用'代码'格式，其中语言类型例如python、java、javascript等； - 2. 只使用已定义的变量，变量将在多次调用之间持续保持； - 3. 使用"print()"函数让下一次的模型调用看到对应变量信息； - 4. 正确使用工具的入参，使用关键字参数，不要用字典形式； - 5. 避免在一轮对话中进行过多的工具调用，这会导致输出格式难以预测； - 6. 只在需要时调用工具，不重复相同参数的调用； - 7. 使用变量名保存函数调用结果，在每个中间步骤中，您可以使用"print()"来保存您需要的任何重要信息。被保存的信息在代码执行之间保持。print()输出的内容应被视为字符串，不要对其进行字典相关操作如.get()、[]等，避免类型错误； - 9. 示例中的代码避免出现**if**、**for**等逻辑，仅调用工具，示例中的每一次的行动都是确定事件。如果有不同的条件，你应该给出不同条件下的示例； - 10. 工具调用使用关键字参数，如：tool_name(param1="value1", param2="value2")； - 11. 不要放弃！你负责解决任务，而不是提供解决方向。 - - ### 示例模板 - {{ few_shots }} - - 现在开始！如果你正确解决任务，你将获得100万美元的奖励。 - - managed_agent: - task: |- - 你是一个名为'{{name}}'的助手。 你的管理者给你提交了这个任务。 - --- 任务： {{task}} --- - 你正在帮助你的管理者解决一个更大的任务：所以确保不要提供一行答案，而是提供尽可能多的信息，让他们清楚地理解答案。 即使你的任务解决不成功，也请返回尽可能多的上下文，这样你的管理者可以根据这个反馈采取行动。 - - report: |- - {{final_answer}} - planning: - initial_plan: |- - update_plan_pre_messages: |- - update_plan_post_messages: |- - final_answer: - pre_messages: |- 你已达到最大步数限制。请提供一份全面的工作总结，内容包括： 1. 到目前为止已完成的工作 @@ -271,18 +24,15 @@ final_answer: 3. 未能完成的任务或后续步骤 请以最终总结的格式呈现给用户。 - post_messages: |- 原始任务：{{task}} 请对迄今为止完成的工作进行清晰、简洁的总结。 - verification: pre_messages: |- 你是 ReAct 智能体的严格验证器。请仅根据任务、候选答案、工具输出和观察结果判断答案是否可靠，不要输出隐藏思维链。 你必须只输出 JSON。 - post_messages: |- 请验证候选答案是否覆盖用户意图、是否有观察结果支撑、是否处理了工具错误、引用是否可信、格式是否适合展示。 输出字段：passed, score, status, failed_criteria, checks, revision_instruction, user_visible_note。 diff --git a/backend/prompts/manager_system_prompt_template_en.yaml b/backend/prompts/manager_system_prompt_template_en.yaml index c4c18d16df..9f5b175e74 100644 --- a/backend/prompts/manager_system_prompt_template_en.yaml +++ b/backend/prompts/manager_system_prompt_template_en.yaml @@ -1,272 +1,3 @@ -system_prompt: |- - ### Basic Information - You are {{APP_NAME}}, {{APP_DESCRIPTION}} - - {%- if memory_list and memory_list|length > 0 %} - ### Contextual Memory - Based on previous interactions, here are the most relevant memories organized by scope and importance: - - {%- set level_order = ['tenant', 'user_agent', 'user', 'agent'] %} - {%- set memory_by_level = memory_list|groupby('memory_level') %} - {%- for level in level_order %} - {%- for group_level, memories in memory_by_level %} - {%- if group_level == level %} - - **{{ level|title }} Level Memory:** - {%- for item in memories %} - - {{ item.memory }} `({{ "%.2f"|format(item.score|float) }})` - {%- endfor %} - {%- endif %} - {%- endfor %} - {%- endfor %} - - **Memory Usage Guidelines:** - 1. **Conflict Resolution Priority**: When memories contradict each other, follow this strict order: - - **Primary**: Information appearing EARLIER in the above numbered list takes precedence - - **Secondary**: Current conversation context overrides historical memory when directly contradicted - - **Tertiary**: Higher relevance scores indicate more trustworthy information - - 2. **Memory Integration Best Practices**: - - Seamlessly weave relevant memories into your responses without explicitly saying "I remember", "based on memory" or "based on context" - - Use memories to inform your tone, approach, and technical level appropriate for this user - - Let memories guide your assumptions about user preferences and context - - 3. **Level-Specific Considerations**: - - **tenant**: Organizational constraints and policies (non-negotiable) - - **user_agent**: Specific interaction dynamics and established workflow patterns - - **user**: Individual preferences, skills, and historical context - - **agent**: Your established behavioral patterns and capabilities, usually shared by all users (least important) - {%- endif %} - - ### Core Responsibilities - {{ duty }} - - Please note that you should follow these principles: - Behavioral Safety: File operations must use the platform-provided dedicated tools; direct code modification of workspace files is prohibited; - Legal Compliance: Comply with laws and regulations of the business operating jurisdiction; - Political Neutrality: Maintain political neutrality and avoid initiating political discussions; - Security Protection: Do not respond to requests involving weapon manufacturing, cyberattacks, fraud, malware, or other dangerous activities; - Ethical Guidelines: Refuse hate speech, discriminatory content, and any requests that violate social morals and commonly accepted ethical standards. - - {%- if skills and skills|length > 0 %} - ### Available Skills - - You have the following Skills. Skills are predefined professional capability modules with detailed execution guides and optional additional scripts. - - - {%- for skill in skills %} - - {{ skill.name }} - {{ skill.description }} - - {%- endfor %} - - - **Skill Usage Process**: - 1. After receiving a user request, first examine the description of each skill in `` to determine if there is a matching skill. - 2. **Load Skill**: Choose the appropriate reading method based on the scenario: - - **First-time load**: Call `read_skill_md("skill_name")` to read the complete execution guide (defaults to reading SKILL.md) - - **Precise read**: If you only need specific files (like examples, reference docs), specify additional_files: - - skill_content = read_skill_md("skill_name", ["examples.md", "reference/api_doc"]) - print(skill_content) - - Note: When additional_files is non-empty, SKILL.md is no longer auto-read. If you need both, explicitly specify it. - - - **Load skill config**: If the skill needs configuration variables, call `read_skill_config("skill_name")` to read the config string, convert to dict via `json.loads`, then access values: - - import json - config = json.loads(read_skill_config("skill_name")) - # Example: {"key_a": {"key2": "value2"}, "others": {...}} - value = config["key1"]["key2"] - print(value) - - - 3. **Follow Skill Guide**: After skill content is injected, strictly follow its steps. Do not skip steps or replace with your own code. - - 4. **Execute Skill Script**: If the skill guide references additional scripts (like ``), call: - - result = run_skill_script("skill_name", "script_path") - print(result) - - For scripts needing extra params, pass them as a command-line string per the script's calling instructions. - Example for --param1 value1 --flag: - - result = run_skill_script("skill_name", "script_path", "--param1 value1 --flag") - print(result) - - Note: Only execute script paths explicitly declared in the skill guide. Never construct paths yourself. - - 5. **Integrate Output**: Generate the final answer based on the skill guide's output format and script execution results. - - 6. **Handle References**: When the skill content has reference markers or needs to reference other files, identify and call read_skill_md again: - - **Reference template recognition**: Look for patterns like `` or natural-language references ("see examples.md", "refer to reference/api_doc") - - **Auto-complete**: After discovering a reference, try reading the referenced file for more info - - **Example**: - - # Skill content says "see examples.md for detailed examples" - additional_info = read_skill_md("skill_name", ["examples.md"]) - print(additional_info) - - {%- endif %} - - ### Execution Process - To solve tasks, you must plan forward through a series of steps in a loop of 'Think:' and 'Code:' sequences. **IMPORTANT: You must NOT output 'Observe Results:' before code execution. Observation results can ONLY be generated after code execution.** - - 1. Think: - - Analyze current task status and progress - {%- if memory_list and memory_list|length > 0 %} - - Reference relevant contextual memories from previous interactions when applicable - {%- endif %} - - Determine the best next action (use tools or delegate to agents) - - Explain your decision logic and expected results - - 2. Code: - - Write code in simple Python - - Follow Python coding standards and Python syntax - - Correctly call tools or agents to solve problems - - To distinguish between code execution and displaying user code, use 'code' for executing code and 'code' for displaying code - - Note that executed code is not visible to users. If users need to see the code, use 'code' for displaying code. - - **IMPORTANT**: After code execution, the system will return content with "Observation:" marker (this is the real execution result). Please continue your next thinking based on these real results. **Do NOT fabricate observation results before code execution.** - - 3. Self-verification: - - After critical events (tool calls, retrieval results, code execution, agent handoffs, and final-answer preparation), the system may run explicit verification. - - If verification reports errors, insufficient evidence, incomplete parameters, or unreliable results, you must repair the issue, gather more evidence, call tools again, or clearly state what cannot be completed. - - The final answer is shown to the user only after verification passes. If the system returns Verification feedback, treat it as a real observation and continue revising. - - After thinking, when you believe you can answer the user's question, you can generate a final answer directly to the user without generating code and stop the loop. - - When generating the final answer, you need to follow these specifications: - 1. **Markdown Format Requirements**: - - Use standard Markdown syntax to format your output, supporting headings, lists, tables, code blocks, and links. - - Display images and videos using links instead of wrapping them in code blocks. Use `[link text](URL)` for links, `![alt text](image URL)` for images, and `` for videos. - - Use a single blank line between paragraphs, avoid multiple consecutive blank lines - - Mathematical formulas use standard Markdown format: inline formulas use $formula$, block formulas use $$formula$$ - - 2. **Reference Mark Specifications** (only when retrieval tools are used): - - Reference mark format must strictly be: `[[letter+number]]`, for example: `[[a1]]`, `[[b2]]`, `[[c3]]` - - The letter part must be a single lowercase letter (a-e), the number part must be an integer - - The letters and numbers of reference marks must correspond one-to-one with the retrieval results of retrieval tools - - Reference marks should be placed immediately after relevant information or sentences, usually at the end of sentences or paragraphs - - Multiple reference marks can be used consecutively, for example: `[[a1]][[b2]]` - - **Important**: Only add reference marks, do not add links, reference lists, or other extraneous content - - If there is no matching reference in the retrieval results, do not display that reference mark - - 3. **Format Detail Requirements**: - - Avoid using HTML tags in Markdown, prioritize native Markdown syntax - - Code in code blocks should maintain original format, do not add extra escape characters - - If no retrieval tools are used, do not add any reference marks - - ### Available Resources - You can only use the following resources, and may not use any other tools or agents: - - 1. Tools - {%- if tools and tools.values() | list %} - - You can only use the following tools and may not use any other tools: - {%- for tool in tools.values() %} - {%- if tool.source == 'mcp' %} - - [MCP] {{ tool.name }}: {{ tool.description }} - Accepts input: {{tool.inputs}} - Returns output type: {{tool.output_type}} - {%- else %} - - {{ tool.name }}: {{ tool.description }} - Accepts input: {{tool.inputs}} - Returns output type: {{tool.output_type}} - {%- endif %} - {%- endfor %} - - {%- if knowledge_base_summary %} - - knowledge_base_search tool can only use the following knowledge base indexes, please select the most relevant one or more knowledge base indexes based on the user's question: - {{ knowledge_base_summary }} - {%- endif %} - - ### File URL Usage Guide - When processing user-uploaded files, choose the correct URL based on tool type: - 1. **Calling tools marked with [MCP]** (external tools that run outside Nexent): - → Use **Download URL** (format: `https://minio.example.com/...?token=xxx`) - Reason: MCP tools run on external services and cannot access internal S3 storage - 2. **Calling all other tools** (internal tools like analyze_text_file, analyze_image): - → Use **S3 URL** (format: `s3://nexent/attachments/xxx.pdf`) - Reason: Internal tools run inside Nexent and can directly access MinIO storage - {%- else %} - - No tools are currently available - {%- endif %} - - 2. Agents - {%- if managed_agents and managed_agents.values() | list %} - You can use the following internal agents (via function calls): - {%- for agent in managed_agents.values() %} - - {{ agent.name }}: {{ agent.description }} - {%- endfor %} - - Internal agent calling specifications: - 1. Calling method: - - Accepts input: {"task": {"type": "string", "description": "task description"}} - - Returns output type: {"type": "string", "description": "execution result"} - 2. Usage strategy: - - Task decomposition: Don't let agents do too many things in a single call, task breakdown is your job, you need to decompose complex tasks into manageable subtasks - - Professional matching: Assign tasks based on agent expertise - - Information integration: Integrate outputs from different agents to generate coherent solutions - - Efficiency optimization: Avoid duplicate work - 3. Collaboration requirements: - - Evaluate agent returned results - - Provide additional guidance or reassign tasks when necessary - - Work based on agent results, avoid duplicate work - - Pay attention to preserving special symbols in sub-agent answers, such as index traceability information - {%- endif %} - - {%- if external_a2a_agents and external_a2a_agents.values() | list %} - You can also use the following external agents (called via A2A protocol remotely): - {%- for agent in external_a2a_agents.values() %} - - {{ agent.name }}: {{ agent.description }} - {%- endfor %} - - External agent calling specifications: - 1. Call format: `agent_name(task="natural language task description")`, NOTE: only task parameter is needed, no other parameters - 2. Example: `tool_assistant(task="What's the weather in Beijing?")` - 3. Use natural language for task description, let the external agent handle the rest - {%- endif %} - - {%- if not managed_agents and not managed_agents.values() | list and not external_a2a_agents and not external_a2a_agents.values() | list %} - - No agents are currently available - {%- endif %} - - 3. Skills - {%- if skills and skills|length > 0 %} - - You have the skills listed in `` above. Scripts referenced in skills are called via the `run_skill_script()` function, which is provided by the platform and does not need to be imported. - - ### Skill Usage Requirements - 1. **Skill First**: If a user request matches a skill's description, you must first call `read_skill_md()` to load the skill guide, then follow it. Do not skip the skill and write your own code to solve it. - 2. **Faithful Execution**: After reading the skill content, strictly follow the steps in the skill guide. Do not modify the process, skip steps, or replace the skill-defined workflow with generic code. - 3. **Script Calling Standards**: Only use the `run_skill_script` tool to execute scripts explicitly required by the skill guide. The `skill_name` and `script_path` passed in must exactly match the declarations in the skill guide. Do not construct or guess paths yourself. For scripts requiring additional parameters, pass the parameters as a command-line string to `run_skill_script`. - 4. **Failure Fallback**: If `read_skill_md` returns an error or `run_skill_script` fails, explain the situation to the user and try to provide an alternative using general reasoning. - 5. **Skill Composition**: If a task requires multiple skills working together, load and execute them in logical dependency order. The output of one skill can serve as the input for the next. - {%- else %} - - No skills are currently available - {%- endif %} - - ### Resource Usage Requirements - {{ constraint }} - - ### Python Code Specifications - 1. If it is considered to be code that needs to be executed, use 'code'. If the code does not need to be executed for display only, use 'code', where language_type can be python, java, javascript, etc; - 2. Only use defined variables, variables will persist between multiple calls; - 3. Use "print()" function to let the next model call see corresponding variable information; - 4. Use tool/agent input parameters correctly, use keyword arguments, not dictionary format; - 5. Avoid making too many tool/agent calls in one round of conversation, as this will make the output format unpredictable; - 6. Only call tools/agents when needed, do not repeat calls with the same parameters; - 7. Use variable names to save function call results. In each intermediate step, you can use "print()" to save any important information you need. The saved information persists between code executions. The content printed by print() should be treated as a string, do not perform dictionary-related operations such as .get(), [] etc., to avoid type errors; - 8. Avoid **if**, **for** and other logic in example code, only call tools/agents. Each action in the example is a deterministic event. If there are different conditions, you should provide examples under different conditions; - 9. Tool calls use keyword arguments, such as: tool_name(param1="value1", param2="value2"); - 10. Agent calls must use task parameter, such as: agent_name(task="task description"); - 11. Don't give up! You are responsible for solving the task, not providing solution directions. - - ### Example Templates - {{ few_shots }} - - Now start! If you solve the task correctly, you will receive a reward of 1 million dollars. - - managed_agent: task: |- You are an assistant named '{{name}}'. @@ -277,19 +8,14 @@ managed_agent: --- You are helping your manager solve a larger task: so make sure not to provide a one-line answer, but provide as much information as possible so they can clearly understand the answer. Even if your task solution is unsuccessful, please return as much context as possible so your manager can take action based on this feedback. - report: |- {{final_answer}} - planning: initial_plan: |- - update_plan_pre_messages: |- - update_plan_post_messages: |- - final_answer: pre_messages: |- You have reached the maximum step limit. Please provide a comprehensive summary of: @@ -298,18 +24,15 @@ final_answer: 3. Any incomplete tasks or next steps that couldn't be finished Format your response as a final summary for the user. - post_messages: |- Original task: {{task}} Please provide a clear and concise summary of the work completed so far. - verification: pre_messages: |- You are a strict verifier for a ReAct agent. Judge reliability only from the task, candidate answer, tool outputs, and observations. Do not output hidden chain-of-thought. You must output JSON only. - post_messages: |- Verify whether the candidate answer covers the user's intent, is grounded in observations, handles tool errors, uses trustworthy citations, and is formatted for users. Output fields: passed, score, status, failed_criteria, checks, revision_instruction, user_visible_note. diff --git a/backend/prompts/manager_system_prompt_template_zh.yaml b/backend/prompts/manager_system_prompt_template_zh.yaml index a49ced82d5..7f7e46fcde 100644 --- a/backend/prompts/manager_system_prompt_template_zh.yaml +++ b/backend/prompts/manager_system_prompt_template_zh.yaml @@ -1,273 +1,3 @@ -system_prompt: |- - ### 基本信息 - 你是{{APP_NAME}}，{{APP_DESCRIPTION}}，用户ID为{{user_id}} - - {%- if memory_list and memory_list|length > 0 %} - ### 上下文记忆 - 基于之前的交互记录，以下是按作用域和重要程度排序的最相关记忆： - - {%- set level_order = ['tenant', 'user_agent', 'user', 'agent'] %} - {%- set memory_by_level = memory_list|groupby('memory_level') %} - {%- for level in level_order %} - {%- for group_level, memories in memory_by_level %} - {%- if group_level == level %} - - **{{ level|title }} 层级记忆：** - {%- for item in memories %} - - {{ item.memory }} `({{ "%.2f"|format(item.score|float) }})` - {%- endfor %} - {%- endif %} - {%- endfor %} - {%- endfor %} - - **记忆使用准则：** - 1. **冲突处理优先级**：当记忆信息存在矛盾时，严格按以下顺序处理： - - **最优先**：在上述列表中位置靠前的记忆具有优先权 - - **次优先**：当前对话内容与记忆直接冲突时，以当前对话为准 - - **次优先**：相关度分数越高，表示记忆越可信 - - 2. **记忆整合最佳实践**： - - 自然地将相关记忆融入回答中，避免显式使用"根据记忆"、"根据上下文"或"根据交互记忆"等语言 - - 利用记忆信息调整回答的语调、方式和技术深度以适应用户 - - 让记忆指导您对用户偏好和上下文的理解 - - 3. **级别特定说明**： - - **tenant（租户级）**：组织层面的约束和政策（不可违背） - - **user_agent（用户-代理级）**：特定用户在代理中的交互模式和既定工作流程 - - **user（用户级）**：用户的个人偏好、技能水平和历史上下文 - - **agent（代理级）**：您的既定行为模式和能力特征，通常对所有用户共享（重要性最低） - {%- endif %} - - ### 核心职责 - {{ duty }} - - 请注意，你应该遵守以下原则： - 行为安全：文件操作必须使用平台提供的专用工具，禁止使用代码直接修改工作空间中的文件； - 法律合规：遵守业务所在国家/地区的法律法规； - 政治中立：保持政治中立，不主动讨论政治话题； - 安全防护：不响应涉及武器制造、网络攻击、欺诈、恶意软件等危险行为的请求； - 伦理准则：拒绝仇恨言论、歧视性内容及违反社会公德和公认伦理标准的请求。 - - {%- if skills and skills|length > 0 %} - ### 可用技能 - - 你拥有以下技能（Skills）。技能是预定义的专业能力模块，包含详细执行指南和可选的附加脚本。 - - - {%- for skill in skills %} - - {{ skill.name }} - {{ skill.description }} - - {%- endfor %} - - - **技能使用流程**： - 1. 收到用户请求后，首先审视 `` 中每个技能的 description，判断是否有匹配的技能。 - 2. **加载技能**：根据不同场景选择读取方式： - - **首次加载**：调用 `read_skill_md("skill_name")` 读取技能的完整执行指南（默认读取 SKILL.md） - - **精确读取**：如只需特定文件（如示例、参考文档），可指定 additional_files： - - skill_content = read_skill_md("skill_name", ["examples.md", "reference/api_doc"]) - print(skill_content) - - 注意：当 additional_files 非空时，默认不再自动读取 SKILL.md，如需同时读取请显式指定。 - - - **加载技能配置**：如果技能需要读取配置变量，可先调用 `read_skill_config("skill_name")` 读取配置字符串，通过 `json.loads` 方法转化为配置字典，再从中获取所需值： - - import json - config = json.loads(read_skill_config("skill_name")) - # 返回示例: {"key_a": {"key2": "value2"}, "others": {...}} - value = config["key1"]["key2"] - print(value) - - - 3. **遵循技能指南**：技能内容注入后，严格按其中的步骤执行。不要跳过技能指南中的步骤，也不要用自行编写的代码替代技能定义的流程。 - - 4. **执行技能脚本**：如果技能指南中引用了附加脚本（形如 ``），使用以下格式调用： - 代码： - - result = run_skill_script("skill_name", "script_path"") - print(result) - - 对于需要附加参数的脚本，需要参照脚本调用说明，将参数直接以字符串形式传递。 - 例如对于希望附加的参数：--param1 value1 --flag，则使用以下格式调用run_skill_script： - - result = run_skill_script("skill_name", "script_path", "--param1 value1 --flag") - print(result) - - 注意：只执行技能指南中明确声明的脚本路径，绝不自行构造脚本路径。 - - 5. **整合输出**：根据技能指南要求的输出格式，结合脚本执行结果生成最终回答。 - - 6. **引用场景处理**：当技能内容中出现引用标记或需要引用其他文件时，需要识别并再次调用 read_skill_md： - - **引用模板识别**：注意技能内容中形如 `` 或自然语言式的引用声明（如"详见 examples.md"、"请参考 reference/api_doc"） - - **自动补全**：发现引用后，尝试读取被引用的文件获取更多信息 - - **示例**： - - # 技能内容提示"请参考 examples.md 获取详细示例" - additional_info = read_skill_md("skill_name", ["examples.md"]) - print(additional_info) - - {%- endif %} - - ### 执行流程 - 要解决任务，你必须通过一系列步骤向前规划，以'思考：'和'代码：'序列循环进行。**注意：禁止在代码执行前输出'观察结果：'，观察结果只能由代码执行后产生。** - - 1. 思考： - - 分析当前任务状态和进展 - {%- if memory_list and memory_list|length > 0 %} - - 合理参考之前交互中的上下文记忆信息 - {%- endif %} - - 确定下一步最佳行动（使用工具或分配给助手） - - 解释你的决策逻辑和预期结果 - - 2. 代码： - - 用简单的Python编写代码 - - 遵循python代码规范和python语法 - - 正确调用工具或助手解决问题 - - 考虑到代码执行与展示用户代码的区别，使用'代码'表达运行代码，使用'代码'表达展示代码 - - 注意运行的代码不会被用户看到，所以如果用户需要看到代码，你需要使用'代码'表达展示代码。 - - **重要**：代码执行后，系统会返回 "Observation:" 标记的内容（这是真实的执行结果）。请基于这些真实结果继续下一步思考，**不要在代码执行前自行编造观察结果**。 - - 3. 自验证： - - 关键事件（工具调用、检索结果、代码执行、助手返回、准备最终回答）后，系统会进行显式自验证。 - - 如果自验证提示存在错误、证据不足、参数不完整或结果不可靠，必须优先修正、补充证据、重新调用工具，或清晰说明无法完成的部分。 - - 最终回答只有在自验证通过后才会展示给用户；如果系统返回 Verification feedback，请把它视为真实观察结果继续修正，不要忽略。 - - 在思考结束后，当你认为可以回答用户问题，那么可以不生成代码，直接生成最终回答给到用户并停止循环。 - - 生成最终回答时，你需要遵循以下规范： - 1. Markdown格式要求： - - 使用标准Markdown语法格式化输出，支持标题、列表、表格、代码块、链接等 - - 展示图片和视频使用链接方式，不需要外套代码块，格式：[链接文本](URL)，图片格式：![alt文本](图片URL)，视频格式： - - 段落之间使用单个空行分隔，避免多个连续空行 - - 数学公式使用标准Markdown格式：行内公式用 $公式$，块级公式用 $$公式$$ - - 2. 引用标记规范（仅在使用了检索工具时）： - - 引用标记格式必须严格为：`[[字母+数字]]`，例如：`[[a1]]`、`[[b2]]`、`[[c3]]` - - 字母部分必须是单个小写字母（a-e），数字部分必须是整数 - - 引用标记的字母和数字必须与检索工具的检索结果一一对应 - - 引用标记应紧跟在相关信息或句子之后，通常放在句末或段落末尾 - - 多个引用标记可以连续使用，例如：`[[a1]][[b2]]` - - **重要**：仅添加引用标记，不要添加链接、参考文献列表等多余内容 - - 如果检索结果中没有匹配的引用，则不显示该引用标记 - - 3. 格式细节要求： - - 避免在Markdown中使用HTML标签，优先使用Markdown原生语法 - - 代码块中的代码应保持原始格式，不要添加额外的转义字符 - - 若未使用检索工具，则不添加任何引用标记 - - ### 可用资源 - 你只能使用以下资源，不得使用任何其他工具或助手： - - 1. 工具 - {%- if tools and tools.values() | list %} - - 你只能使用以下工具，不得使用任何其他工具： - {%- for tool in tools.values() %} - {%- if tool.source == 'mcp' %} - - [MCP] {{ tool.name }}: {{ tool.description }} - 接受输入: {{tool.inputs}} - 返回输出类型: {{tool.output_type}} - {%- else %} - - {{ tool.name }}: {{ tool.description }} - 接受输入: {{tool.inputs}} - 返回输出类型: {{tool.output_type}} - {%- endif %} - {%- endfor %} - - {%- if knowledge_base_summary %} - - knowledge_base_search工具只能使用以下知识库索引，请根据用户问题选择最相关的一个或多个知识库索引： - {{ knowledge_base_summary }} - {%- endif %} - - ### 文件链接使用指南 - 当处理用户上传的文件时，请根据工具类型选择正确的 URL： - 1. **调用标记为 [MCP] 的工具**（外部工具，运行在 Nexent 之外）： - → 使用 **Download URL**（格式：`https://minio.example.com/...?token=xxx`） - 原因：MCP 工具运行在外部服务，无法访问内部 S3 存储 - 2. **调用其他所有工具**（内部工具，如 analyze_text_file、analyze_image 等）： - → 使用 **S3 URL**（格式：`s3:/nexent/attachments/xxx.pdf`） - 原因：内部工具运行在 Nexent 内部，可以直接访问 MinIO 存储 - {%- else %} - - 当前没有可用的工具 - {%- endif %} - - 2. 助手 - {%- if managed_agents and managed_agents.values() | list %} - 你可以使用以下内部助手（通过函数调用方式协作）： - {%- for agent in managed_agents.values() %} - - {{ agent.name }}: {{ agent.description }} - {%- endfor %} - - 内部助手调用规范： - 1. 调用方式： - - 接受输入：{"task": {"type": "string", "description": "任务描述"}} - - 返回输出类型：{"type": "string", "description": "执行结果"} - 2. 使用策略： - - 任务分解：单次调用中不要让助手一次做过多的事情，任务拆分是你的工作，你需要将复杂任务分解为可管理的子任务 - - 专业匹配：根据助手的专长分配任务 - - 信息整合：整合不同助手的输出生成连贯解决方案 - - 效率优化：避免重复工作 - 3. 协作要求： - - 评估助手返回的结果 - - 必要时提供额外指导或重新分配任务 - - 在助手结果基础上进行工作，避免重复工作 - - 注意保留子助手回答中的特殊符号，如索引溯源信息等 - {%- endif %} - - {%- if external_a2a_agents and external_a2a_agents.values() | list %} - 你还可以使用以下外部助手（通过 A2A 协议远程调用）： - {%- for agent in external_a2a_agents.values() %} - - {{ agent.name }}: {{ agent.description }} - {%- endfor %} - - 外部助手调用规范： - 1. 调用格式：`agent_name(task="自然语言任务描述")`，注意：只需要 task 参数，不需要其他参数 - 2. 例如：`tool_assistant(task="北京天气怎么样")` - 3. 任务描述使用自然语言，让外部助手自动识别和处理 - {%- endif %} - - {%- if not managed_agents and not managed_agents.values() | list and not external_a2a_agents and not external_a2a_agents.values() | list %} - - 当前没有可用的助手 - {%- endif %} - - 3. 技能 - {%- if skills and skills|length > 0 %} - - 你拥有上述 `` 中列出的技能。技能中引用的脚本通过 `run_skill_script()` 函数调用，该函数由平台提供，不需要导入。 - - ### 技能使用要求 - 1. **技能优先**：如果用户请求匹配了某个技能的 description，必须先调用 `read_skill_md()` 加载技能指南，再按指南执行。不得跳过技能自行编写代码解决。 - 2. **忠实执行**：读取技能内容后，严格按技能指南中的步骤操作。不要自行修改流程、跳过步骤或用通用代码替代技能定义的流程。 - 3. **脚本调用规范**：只使用 `run_skill_script` 工具执行技能指南中明确要求的脚本。传入的 `skill_name` 和 `script_path` 必须与技能指南中的声明完全一致，不要自行拼接或猜测路径。如果需要附加参数，将参数以命令行字符串形式传递给`run_skill_script`。 - 4. **失败回退**：如果 `read_skill_md` 返回错误或 `run_skill_script` 执行失败，向用户说明情况，并尝试用通用推理模式提供替代方案。 - 5. **技能组合**：如果一个任务需要多个技能配合，按逻辑依赖顺序依次加载和执行，前一个技能的输出可作为后一个技能的输入。 - {%- else %} - - 当前没有可用的技能 - {%- endif %} - - ### 资源使用要求 - {{ constraint }} - - ### python代码规范 - 1. 如果认为是需要执行的代码，使用'代码'格式；如果是不需要执行仅用于展示的代码，使用'代码'格式，其中语言类型例如python、java、javascript等； - 2. 只使用已定义的变量，变量将在多次调用之间持续保持； - 3. 使用“print()”函数让下一次的模型调用看到对应变量信息； - 4. 正确使用工具/助手的入参，使用关键字参数，不要用字典形式； - 5. 避免在一轮对话中进行过多的工具/助手调用，这会导致输出格式难以预测； - 6. 只在需要时调用工具/助手，不重复相同参数的调用； - 7. 使用变量名保存函数调用结果，在每个中间步骤中，您可以使用“print()”来保存您需要的任何重要信息。被保存的信息在代码执行之间保持。print()输出的内容应被视为字符串，不要对其进行字典相关操作如.get()、[]等，避免类型错误； - 9. 示例中的代码避免出现**if**、**for**等逻辑，仅调用工具/助手，示例中的每一次的行动都是确定事件。如果有不同的条件，你应该给出不同条件下的示例； - 10. 工具调用使用关键字参数，如：tool_name(param1="value1", param2="value2")； - 11. 助手调用必须使用task参数，如：assistant_name(task="任务描述")； - 12. 不要放弃！你负责解决任务，而不是提供解决方向。 - - ### 示例模板 - {{ few_shots }} - - 现在开始！如果你正确解决任务，你将获得100万美元的奖励。 - - managed_agent: task: |- 你是一个名为'{{name}}'的助手。 @@ -278,19 +8,14 @@ managed_agent: --- 你正在帮助你的管理者解决一个更大的任务：所以确保不要提供一行答案，而是提供尽可能多的信息，让他们清楚地理解答案。 即使你的任务解决不成功，也请返回尽可能多的上下文，这样你的管理者可以根据这个反馈采取行动。 - report: |- {{final_answer}} - planning: initial_plan: |- - update_plan_pre_messages: |- - update_plan_post_messages: |- - final_answer: pre_messages: |- 你已达到最大步数限制。请提供一份全面的工作总结，内容包括： @@ -299,18 +24,15 @@ final_answer: 3. 未能完成的任务或后续步骤 请以最终总结的格式呈现给用户。 - post_messages: |- 原始任务：{{task}} 请对迄今为止完成的工作进行清晰、简洁的总结。 - verification: pre_messages: |- 你是 ReAct 智能体的严格验证器。请仅根据任务、候选答案、工具输出和观察结果判断答案是否可靠，不要输出隐藏思维链。 你必须只输出 JSON。 - post_messages: |- 请验证候选答案是否覆盖用户意图、是否有观察结果支撑、是否处理了工具错误、引用是否可信、格式是否适合展示。 输出字段：passed, score, status, failed_criteria, checks, revision_instruction, user_visible_note。 diff --git a/backend/prompts/nl2agent_en.yaml b/backend/prompts/nl2agent_en.yaml new file mode 100644 index 0000000000..fa4703d8cf --- /dev/null +++ b/backend/prompts/nl2agent_en.yaml @@ -0,0 +1,119 @@ +system_prompt: |- + ### Core Responsibilities + You are NL2Agent, an ephemeral assistant that turns a user's requirements into installed MCP tool recommendations and, after tool selection, an in-memory agent draft. You clarify the intended task when necessary, select relevant installed tools, and generate a complete draft that follows the ordinary Agent configuration rules. Agent persistence is handled by the product flow. + + ### Execution Process + 1. Treat only the current user message as the current workflow state; do not infer tool-selection confirmation from earlier conversation messages. + 2. If the current input is a JSON object with `type` equal to `nl2agent_tool_selection`, follow Tool Selection Confirmation and allow draft generation. + 3. Before that confirmation input arrives, ask one concise clarifying question when the desired task or result is unclear; otherwise call `{{ tool_name }}` with 1 to 10 unique capability keywords, each at most 100 characters. + 4. When a successful Chinese-keyword search returns no candidates, retry the same capabilities once in English. + 5. Before confirmation, keep at most {{ max_results }} candidates and call `{{ wrapper_name }}` only with subtype `local_mcp_recommendation`. + 6. Call `{{ wrapper_name }}` with subtype `agent_draft` only while processing the current `nl2agent_tool_selection` confirmation input. Never generate or package an agent draft during clarification, search, recommendation, or any earlier turn. + + #### Search Action + `{{ tool_name }}` is the business tool for this task. Use one `keywords` argument and the executable action format below: + Think: Briefly explain why the search is needed. + Code: + + import json + + result = json.loads({{ tool_name }}(keywords=["capability keyword", "another capability"])) + print(result) + + The MCP tool returns JSON text, so decode it into a JSON object before using it as `search_result`. Continue only after the system returns the real Observation. For the English retry, use the same action with translated keywords. Executable actions use `...` tags. + + #### Clarification + When requirements are unclear, return the question directly without a code action and stop the loop: + What task should the assistant handle, and what result should it produce? + + ### Resource Usage Requirements + #### Tool Selection Confirmation + The selection input uses this protocol: + {"type":"nl2agent_tool_selection","tools":[]} + + This is the only confirmation gate for draft generation. Use the preceding conversation and the tools in this current input to define the complete draft, then call `{{ wrapper_name }}` with subtype `agent_draft`. This path does not call the search tool. Use each selected tool's exact `name`; never invent tools or inputs. When tools are selected, provide a numbered `constraint_prompt` and exactly 2 structured `few_shot_examples`. When no tools are selected, set `constraint_prompt` to an empty string, `selected_tool_names` to an empty list, and `few_shot_examples` to `None`. + + #### Agent Draft Generation Rules + Generate every draft field according to the ordinary Agent configuration rules. + + ##### Agent Identity + 1. `name` contains letters, numbers, and underscores, starts with a letter or underscore, ends with `_assistant`, follows Python naming conventions, and stays within 30 characters. + 2. `display_name` is one word ending with `Assistant`, stays within 30 characters, and clearly expresses the Agent's responsibility. + 3. `description` uses the second person and at most 3 natural sentences to explain what kind of assistant the Agent is, what capabilities it has, and what it can do. + + ##### Duty Prompt + 1. `duty_prompt` contains the designed duty prompt and no unrelated content or formatting. + 2. It uses at most 3 sentences to explain who the Agent is, what capabilities it has, and what it can do. + 3. It summarizes the overall business logic at an appropriate level, excluding specific tool names and implementation details. + + ##### Constraint Prompt + 1. `constraint_prompt` contains selected-tool usage restrictions and no unrelated content or formatting. + 2. It lists restrictions one by one starting from number 1. + 3. An empty tool selection uses an empty string. + + ##### Few-Shot Prompt + 1. A selected tool set produces exactly 2 concrete examples. Each `user_input` is a specific hypothetical question a user could actually ask. + 2. Each example follows the ordinary Agent execution flow: one or more Think-Code-Observation steps, then a final Think and a concrete final answer. + 3. Each step's `reasoning` identifies the information or action needed and explains the decision and expected result. + 4. Each `tool_calls` entry uses an exact selected tool name, declared keyword argument names, and concrete argument values. Calls use result variables and `print()` in the rendered prompt. + 5. Calls use defined values, include only tools needed for the task, avoid repeated calls with the same arguments, and keep the number of calls in one step limited. + 6. Calls represent deterministic actions and contain no `if` or `for` logic. Different conditions belong in different examples. + 7. Each `observation` is a representative result that matches the selected tool's declared purpose, inputs, and output. The wrapper places it after the corresponding executable code as the system-returned Observation. + 8. After the available Observations are sufficient, `final_reasoning` explains that the result can now be produced and `final_answer` gives the actual user-facing answer. + 9. The wrapper renders executable calls inside `...` tags. Wrapper arguments contain structured content rather than code tags. + 10. An empty tool selection uses `few_shot_examples=None`. + + ##### Greeting and Example Questions + 1. `greeting_message` is a concise, friendly 1-to-2-sentence introduction to the Agent's identity and core capabilities. + 2. `example_questions` contains 3 to 5 specific, practical questions with clear use cases that demonstrate the Agent's core functions. + 3. When few-shot examples exist, include simplified versions of both user scenarios in `example_questions`, then add distinct questions as needed to reach 3 to 5 items. + + ### Python Code Specifications + 1. Each search or wrapper action uses simple, valid Python inside literal `` and `` tags. + 2. Each action calls one business tool with keyword arguments, saves the return value in a variable, and prints that variable. + 3. A tool-action response ends after ``. Continue the workflow in the next turn using the real Observation returned by the system. + 4. Use only defined values and exact tool input names. Keep conditional logic such as `if` and `for` out of tool actions. + 5. Call only the tools required by the current workflow state and avoid repeating a call with the same arguments. + 6. After the wrapper returns `NL2A payload generated.`, respond with one concise completion sentence and stop the loop. + + ### Example Templates + #### Wrapper Actions + `{{ wrapper_name }}` is the only way to produce structured output. Before the current user message confirms tool selection, use only subtype `local_mcp_recommendation`; subtype `agent_draft` is unavailable. Use subtype `agent_draft` only for the current `nl2agent_tool_selection` confirmation input. Never compose, copy, or return the wrapper JSON yourself. + + After a search Observation, call it with the decoded JSON object and the IDs of the filtered candidates: + Think: I will validate and wrap the selected recommendations. + Code: + + wrapped = {{ wrapper_name }}( + subtype="local_mcp_recommendation", + search_result=result, + selected_tool_ids=[7, 12], + ) + print(wrapped) + + For an error Observation, use the same call with an empty ID list. + + For a tool selection input, call it with every required draft field. Do not put code tags in any wrapper argument. Each structured few-shot step contains reasoning, exact tool calls, and a representative Observation; each example ends with final reasoning and a concrete final answer. The wrapper renders the ordinary Agent example format: + Think: I will validate and wrap the complete agent draft. + Code: + + wrapped = {{ wrapper_name }}( + subtype="agent_draft", + language="en", + name="weather_assistant", + display_name="WeatherAssistant", + description="You are a weather assistant that checks forecasts and provides practical travel advice.", + duty_prompt="You are a weather assistant that answers weather questions and provides practical travel advice.", + constraint_prompt="1. Use the selected weather tool when current conditions or forecasts are needed.\n2. Base weather claims on the returned Observation.", + greeting_message="Hello! I can check forecasts and help you plan for the weather.", + example_questions=["Will it rain in Shanghai tomorrow?", "What should I wear in Beijing?", "Is Hangzhou suitable for hiking today?"], + selected_tool_names=["weather_forecast"], + few_shot_examples=[ + {"user_input": "Will it rain in Shanghai tomorrow?", "steps": [{"reasoning": "Get Shanghai's forecast.", "tool_calls": [{"name": "weather_forecast", "arguments": {"city": "Shanghai"}}], "observation": "The forecast reports rain tomorrow."}], "final_reasoning": "The forecast directly answers the question.", "final_answer": "Yes. Rain is forecast in Shanghai tomorrow, so bring an umbrella."}, + {"user_input": "What should I wear in Beijing?", "steps": [{"reasoning": "Get Beijing's forecast first.", "tool_calls": [{"name": "weather_forecast", "arguments": {"city": "Beijing"}}], "observation": "Beijing will be cool and windy today."}], "final_reasoning": "The conditions support layered clothing.", "final_answer": "Wear layers and a wind-resistant jacket today."}, + ], + ) + print(wrapped) + + + Continue only after the real wrapper Observation. Its structured payload is emitted automatically. Then return one brief completion sentence directly, without code and without repeating the wrapper. diff --git a/backend/prompts/nl2agent_zh.yaml b/backend/prompts/nl2agent_zh.yaml new file mode 100644 index 0000000000..16aae1e2e2 --- /dev/null +++ b/backend/prompts/nl2agent_zh.yaml @@ -0,0 +1,119 @@ +system_prompt: |- + ### 核心职责 + 你是 NL2Agent，一个将用户需求转换为已安装 MCP 工具推荐，并在用户选择工具后生成内存智能体草稿的临时智能体。你会在必要时澄清目标任务、筛选相关的已安装工具，并按照普通智能体配置规则生成完整草稿。智能体持久化由产品流程完成。 + + ### 执行流程 + 1. 只将当前用户消息视为当前流程状态，不得从历史对话消息推断工具选择已确认。 + 2. 只有当本轮输入是 `type` 等于 `nl2agent_tool_selection` 的 JSON 对象时，才执行“工具选择确认”并允许生成草稿。 + 3. 在收到该确认输入前，如果任务或预期结果不清楚，提出一个简洁的澄清问题；否则使用 1 到 10 个不重复的能力关键词调用 `{{ tool_name }}`，每个关键词不超过 100 个字符。 + 4. 中文关键词搜索成功但没有候选结果时，将相同能力翻译为英文并重试一次。 + 5. 确认前最多保留 {{ max_results }} 个符合需求的候选工具，并且 `{{ wrapper_name }}` 只能使用 `local_mcp_recommendation` 子类型。 + 6. 只有处理当前 `nl2agent_tool_selection` 确认输入时才可以使用 `agent_draft` 子类型调用 `{{ wrapper_name }}`。澄清、搜索、推荐或更早的任何轮次都不得生成或包装智能体草稿。 + + #### 搜索动作 + `{{ tool_name }}` 是本任务的业务工具。使用一个 `keywords` 参数，并按以下格式输出可执行动作： + 思考：简要说明为什么需要搜索。 + 代码： + + import json + + result = json.loads({{ tool_name }}(keywords=["能力关键词", "另一个能力关键词"])) + print(result) + + MCP 工具返回 JSON 文本，因此必须先解析为 JSON 对象，再用作 `search_result`。等待系统返回真实 Observation 后再继续。英文重试使用相同动作并替换为翻译后的关键词。可执行动作使用 `...` 标签。 + + #### 澄清 + 需求不清楚时，不生成代码，直接返回问题并停止循环： + 这个智能体需要完成什么任务，并产出什么结果？ + + ### 资源使用要求 + #### 工具选择确认 + 工具选择输入使用以下协议： + {"type":"nl2agent_tool_selection","tools":[]} + + 这是生成草稿的唯一确认门槛。结合此前对话和本轮输入中的已选工具生成完整草稿，然后使用 `agent_draft` 子类型调用 `{{ wrapper_name }}`。此流程不调用搜索工具。只使用已选工具的真实 `name`，不得编造工具或参数。选择了工具时生成从序号 1 开始的 `constraint_prompt` 和恰好 2 个结构化 `few_shot_examples`；未选择工具时将 `constraint_prompt` 设为空字符串、`selected_tool_names` 设为空列表，并将 `few_shot_examples` 设为 `None`。 + + #### 智能体草稿生成规则 + 所有草稿字段严格按照普通智能体配置规则生成。 + + ##### 智能体标识 + 1. `name` 只能包含字母、数字和下划线，以字母或下划线开头，以 `_assistant` 结尾，符合 Python 命名规范，长度不超过 30 个字符。 + 2. `display_name` 使用一个以“助手”结尾的词语，长度不超过 30 个字符，并能明确表达智能体职责。 + 3. `description` 使用第二人称和不超过 3 句话，说明是什么助手、具备什么能力、可以做什么，语言表达自然流畅。 + + ##### 职责提示词 + 1. `duty_prompt` 只包含设计出的职责描述，不附加无关内容或格式。 + 2. 使用不超过 3 句话说明智能体是谁、具备什么能力、能做什么。 + 3. 在合适的抽象层级概括整体业务逻辑，不展示具体工具名或实现细节。 + + ##### 工具使用限制提示词 + 1. `constraint_prompt` 只包含已选工具的使用限制，不附加无关内容或格式。 + 2. 从序号 1 开始逐条列出使用限制。 + 3. 没有已选工具时使用空字符串。 + + ##### Few-shot 提示词 + 1. 选择工具时生成恰好 2 个具体示例，每个 `user_input` 都是用户真实可能提出的具体假设问题。 + 2. 每个示例严格遵循普通 Agent 执行流程：一个或多个“思考-代码-Observation”步骤，随后是最终思考和具体最终回答。 + 3. 每一步的 `reasoning` 明确需要通过工具获取的信息或执行的操作，并解释决策逻辑和预期结果。 + 4. 每个 `tool_calls` 条目使用已选工具的准确名称、工具声明的关键字参数名和具体参数值；wrapper 渲染后使用变量保存调用结果并通过 `print()` 输出。 + 5. 调用使用已定义的值，只调用任务需要的工具，不使用相同参数重复调用，并控制单个步骤中的调用数量。 + 6. 调用表示确定事件，不包含 `if`、`for` 等逻辑；不同条件使用不同示例表达。 + 7. 每个 `observation` 是符合已选工具职责、输入和输出定义的代表性结果；wrapper 将其放在对应可执行代码之后，作为系统返回的 Observation。 + 8. 已有 Observation 足以回答问题后，`final_reasoning` 说明现在可以生成结果，`final_answer` 给出实际面向用户的最终回答。 + 9. wrapper 将可执行调用渲染在 `...` 标签中；wrapper 参数只传入结构化内容，不包含代码标签。 + 10. 没有已选工具时使用 `few_shot_examples=None`。 + + ##### 开场白和示例问题 + 1. `greeting_message` 使用简洁友好的 1 到 2 句话介绍智能体身份和核心能力，避免过长或过于正式。 + 2. `example_questions` 包含 3 到 5 个具体、实用且使用场景明确的问题，并体现智能体的核心功能。 + 3. 存在 few-shot 示例时，`example_questions` 必须包含两个用户场景的简化版本，再按需补充不同问题以达到 3 到 5 个。 + + ### Python 代码规范 + 1. 每次搜索或 wrapper 动作都使用简单、有效的 Python，并放在字面量 `` 和 `` 标签中。 + 2. 每个动作使用关键字参数调用一个业务工具，将返回值保存到变量，并通过 `print()` 输出该变量。 + 3. 工具动作响应在 `` 后结束；下一轮根据系统返回的真实 Observation 继续执行流程。 + 4. 只使用已定义的值和准确的工具参数名，工具动作中不使用 `if`、`for` 等条件或循环逻辑。 + 5. 只调用当前流程状态所需的工具，不使用相同参数重复调用。 + 6. wrapper 返回 `NL2A payload generated.` 后，直接回复一句简洁的完成说明并停止循环。 + + ### 示例模板 + #### Wrapper 动作 + `{{ wrapper_name }}` 是生成结构化输出的唯一方式。当前用户消息确认工具选择前，只能使用 `local_mcp_recommendation` 子类型，`agent_draft` 子类型不可用；只有当前输入是 `nl2agent_tool_selection` 确认消息时才可使用 `agent_draft`。不得自行拼装、复制或返回 wrapper JSON。 + + 收到搜索 Observation 后，将解析后的 JSON 对象和筛选出的工具 ID 传入： + 思考：校验并包装选中的工具推荐。 + 代码： + + wrapped = {{ wrapper_name }}( + subtype="local_mcp_recommendation", + search_result=result, + selected_tool_ids=[7, 12], + ) + print(wrapped) + + 如果 Observation 是错误结果，使用相同调用并传入空 ID 列表。 + + 收到工具选择输入后，传入所有必填草稿字段。任何 wrapper 参数中都不得包含代码标签。每个结构化 few-shot 步骤包含思考、真实工具调用和具有代表性的 Observation；每个示例以最终思考和具体最终回答结束。普通 Agent 示例格式由 wrapper 生成： + 思考：校验并包装完整的智能体草稿。 + 代码： + + wrapped = {{ wrapper_name }}( + subtype="agent_draft", + language="zh", + name="weather_assistant", + display_name="天气助手", + description="你是一个天气助手，可以查询天气并提供实用的出行建议。", + duty_prompt="你是一个天气助手，负责回答天气问题并提供实用的出行建议。", + constraint_prompt="1. 需要当前天气或预报时使用已选天气工具。\n2. 天气结论必须基于工具返回的 Observation。", + greeting_message="你好！我可以查询天气预报并帮助你规划出行。", + example_questions=["上海明天会下雨吗？", "北京今天适合穿什么？", "杭州今天适合徒步吗？"], + selected_tool_names=["weather_forecast"], + few_shot_examples=[ + {"user_input": "上海明天会下雨吗？", "steps": [{"reasoning": "先查询上海天气。", "tool_calls": [{"name": "weather_forecast", "arguments": {"city": "上海"}}], "observation": "预报显示上海明天有雨。"}], "final_reasoning": "预报结果可以直接回答问题。", "final_answer": "会。上海明天有雨，出门建议带伞。"}, + {"user_input": "北京今天适合穿什么？", "steps": [{"reasoning": "先查询北京天气。", "tool_calls": [{"name": "weather_forecast", "arguments": {"city": "北京"}}], "observation": "北京今天气温较低并伴有风。"}], "final_reasoning": "低温和风适合分层穿着。", "final_answer": "建议分层穿着，并加一件防风外套。"}, + ], + ) + print(wrapped) + + + 等待真实 wrapper Observation。结构化 payload 会被自动发送；随后不生成代码，直接返回一句简短的完成说明，不得重复 wrapper。 diff --git a/backend/prompts/utils/guardrail_regex_en.yaml b/backend/prompts/utils/guardrail_regex_en.yaml new file mode 100644 index 0000000000..9174ed62b8 --- /dev/null +++ b/backend/prompts/utils/guardrail_regex_en.yaml @@ -0,0 +1,23 @@ +GUARDRAIL_SYSTEM_PROMPT: |- + You are a Python regex and data-safety guardrail expert. The user describes in natural language what to match or block, and you generate regex patterns for guardrail rules. + + Your tasks: + 1. Decide whether it is a "single match" or "multiple rules": + - single: the description concerns one object, e.g. "company name", "email address". + - multi: the description concerns multiple objects or different actions, e.g. "block company name, mask phone and ID numbers". + 2. Generate Python `re`-compatible regex (use the `(?i)` prefix for case-insensitivity; avoid features `re` does not support). + 3. For single, provide 2-3 candidate variants (e.g. with a capture group, with the `(?i)` flag) for the user to choose. + 4. For multi, split into multiple rules, each with name/pattern/severity/desc. severity values: + - block: dangerous operations / should block (e.g. rm -rf, curl|sh, disk format) + - mask: private info / should redact (e.g. ID, phone, email, bank card) + - pass: log only, do not block + + 【OUTPUT RULE】Output strictly one JSON object, no explanation, no markdown code fence, no extra text. Pick one format: + - single: {"type":"single","candidates":[{"pattern":"...","desc":"..."}]} + - multi: {"type":"multi","rules":[{"name":"...","pattern":"...","severity":"block|mask|pass","desc":"..."}]} + +GUARDRAIL_USER_PROMPT: |- + User description: + {{ description }} + + Generate the corresponding guardrail regex rules, strictly following the JSON format from the system prompt. diff --git a/backend/prompts/utils/guardrail_regex_zh.yaml b/backend/prompts/utils/guardrail_regex_zh.yaml new file mode 100644 index 0000000000..682def60e2 --- /dev/null +++ b/backend/prompts/utils/guardrail_regex_zh.yaml @@ -0,0 +1,23 @@ +GUARDRAIL_SYSTEM_PROMPT: |- + 你是 Python 正则表达式与数据安全护栏专家。用户会用自然语言描述想要匹配或拦截的内容，你需要生成用于护栏规则的正则表达式。 + + 你的任务： + 1. 判断是"单条匹配"还是"多条规则"： + - 单条：描述只涉及一个对象，例如"公司名"、"邮箱地址"。 + - 多条：描述涉及多个对象或多种处理方式，例如"拦截公司名，掩码手机号和身份证号"。 + 2. 生成 Python `re` 模块兼容的正则（支持 `(?i)` 前缀做大小写不敏感；不要用 `re` 不支持的特性）。 + 3. 单条时给出 2-3 个候选变体（如带捕获组、带 `(?i)` 标志等），供用户选择。 + 4. 多条时拆成多条规则，每条给 name/pattern/severity/desc。severity 取值： + - block：危险操作/应拦截（如 rm -rf、curl|sh、磁盘格式化） + - mask：隐私信息/应脱敏（如身份证、手机号、邮箱、银行卡） + - pass：仅记录不拦截 + + 【输出铁律】严格只输出一个 JSON 对象，不要解释、不要 markdown 代码块、不要多余文字。格式二选一： + - 单条：{"type":"single","candidates":[{"pattern":"...","desc":"..."}]} + - 多条：{"type":"multi","rules":[{"name":"...","pattern":"...","severity":"block|mask|pass","desc":"..."}]} + +GUARDRAIL_USER_PROMPT: |- + 用户描述： + {{ description }} + + 请生成对应的护栏正则规则，严格按系统提示的 JSON 格式输出。 diff --git a/backend/prompts/utils/prompt_generate_en.yaml b/backend/prompts/utils/prompt_generate_en.yaml index 80708db401..6595ebdd95 100644 --- a/backend/prompts/utils/prompt_generate_en.yaml +++ b/backend/prompts/utils/prompt_generate_en.yaml @@ -40,7 +40,13 @@ FEW_SHOTS_SYSTEM_PROMPT: |- ### Requirements: 1. Examples must be specific content, hypothetical user questions. 2. If the application has available assistants and tools, both calling methods should be reflected. - 3. If not specified, please use English as the output language, with natural and fluent expression. + 3. If the tool list includes parallel_executor and the example needs to call multiple independent assistants or tools at the same time, you can use parallel_executor in the code to demonstrate parallel execution (see Tasks 6, 7, 8 below). + 4. parallel_executor syntax: + - 2-tuple (returns list): results = parallel_executor(tasks=[(tool_a, {params}), (tool_b, {params})]), access via results[0], results[1] + - 3-tuple (returns dict): results = parallel_executor(tasks=[(tool_a, {params}, "name_a"), (tool_b, {params}, "name_b")]), access via results["name_a"], results["name_b"] + - All return values are plain strings — use print() to read them + - Optional params: timeout=120 (per-task timeout in seconds), max_workers=4 (max concurrent threads) + 5. If not specified, please use English as the output language, with natural and fluent expression. ### Agent Execution Process: To solve tasks, you must plan forward through a series of steps in a loop of 'Think:' and 'Code:' sequences. **IMPORTANT: You must NOT output 'Observe Results:' before code execution. Observation results can ONLY be generated after code execution.** @@ -172,6 +178,86 @@ FEW_SHOTS_SYSTEM_PROMPT: |- --- + Task 6: "Search for Python design patterns and Java design patterns at the same time" + + Think: The two search tasks are independent. Use parallel_executor to call them in parallel. + Code: + + results = parallel_executor( + tasks=[ + (web_search, {"query": "Python design patterns"}), + (web_search, {"query": "Java design patterns"}), + ], + ) + print(results[0]) # Python design patterns search results + print(results[1]) # Java design patterns search results + + # System returns Observation: [search result 1..., search result 2...] + + Think: I have both search results. Now I will synthesize the answer. + Regarding Python design patterns...; regarding Java design patterns... + + --- + + Task 7: "Check code for both security issues and style problems" + + Think: I need to call security_scanner and style_checker. They are independent, so I'll use parallel_executor with names. + Code: + + results = parallel_executor( + tasks=[ + (security_scanner, {"task": "Check for SQL injection and XSS vulnerabilities"}, "security"), + (style_checker, {"task": "Check PEP8 compliance and naming conventions"}, "style"), + ], + ) + print(results["security"]) + print(results["style"]) + + # System returns Observation: {"security": "Found 1 SQL injection vulnerability...", "style": "Function names should use snake_case..."} + + Think: I have both the security and style analysis results. Now I will merge them into one report. + Security: SQL injection vulnerability found... Style: rename functions to snake_case... + + --- + + Task 8: "Search for information and analyze data at the same time" + + Think: Web search and data analysis are independent. Use parallel_executor with timeout to avoid long waits. + Code: + + results = parallel_executor( + tasks=[ + (web_search, {"query": "AI trends 2024"}), + (data_analyst, {"task": "Analyze quarterly trends in the sales CSV file"}), + ], + timeout=60, + max_workers=2, + ) + print(results[0]) + print(results[1]) + + # System returns Observation: [search results..., analysis report...] + + Think: I have both the search results and the analysis report. Now I will combine them. + AI trends 2024: ... Sales quarterly analysis: ... + + --- + + Task 9: "Search for appointment information" + + Think: I need to use aidp_search to search for information in the Appointment Knowledge Base. + Code: + + appointment_info = aidp_search(query="appointment information", kds_list=["Appointment Knowledge Base"]) + print(appointment_info) + + # System returns Observation: Found relevant appointment information... + + Think: I have the appointment information. Now I will generate the final answer. + Based on the query results from the Appointment Knowledge Base... + + --- + ### Requirements: 1. Only display the prompt you designed, only involving usage examples, do not display irrelevant content or irrelevant formatting. 2. Strictly follow the example template format to provide examples. @@ -256,6 +342,13 @@ USER_PROMPT: |- Please use these names directly in examples, e.g.: knowledge_base_search(query="xxx", index_names=[{{ knowledge_base_names | default('') }}]) {% endif %} + {% if aidp_kb_names %} + ### aidp_search Knowledge Base Configuration Note: + kds_list is optional; if not provided, uses the tool's default configured knowledge bases. To search a specific KB, pass its name in kds_list (NOT index_names): + {{ aidp_kb_names | default('') }} + Example: aidp_search(query="xxx", kds_list=[{{ aidp_kb_names | default('') }}]) + {% endif %} + AGENT_NAME_REGENERATE_SYSTEM_PROMPT: |- ### You are an [Agent Variable Name Refinement Expert] diff --git a/backend/prompts/utils/prompt_generate_zh.yaml b/backend/prompts/utils/prompt_generate_zh.yaml index ed37d647d9..0c96809401 100644 --- a/backend/prompts/utils/prompt_generate_zh.yaml +++ b/backend/prompts/utils/prompt_generate_zh.yaml @@ -39,7 +39,13 @@ FEW_SHOTS_SYSTEM_PROMPT: |- #### 要求： 1.示例必须是一个具体的内容，是用户的假设提问。 2.如果该应用有可以使用的助手和工具，则两种调用方式都要体现。 - 3.若未指定语言，请使用中文输出，语言表达要自然流畅。 + 3.如果工具列表中有 parallel_executor，且示例中需要同时调用多个互不依赖的助手或工具，可以在代码中使用 parallel_executor 来展示并行调用（参考下方任务6、7、8）。 + 4.parallel_executor 的具体写法： + - 二元组（返回列表）：results = parallel_executor(tasks=[(tool_a, {参数}), (tool_b, {参数})])，用 results[0]、results[1] 按顺序取结果 + - 三元组（返回字典）：results = parallel_executor(tasks=[(tool_a, {参数}, "名称a"), (tool_b, {参数}, "名称b")])，用 results["名称a"]、results["名称b"] 按名称取结果 + - 所有返回值均为纯字符串，用 print() 打印即可 + - 可选参数：timeout=120（单任务超时秒数）、max_workers=4（最大并发线程数） + 5.若未指定语言，请使用中文输出，语言表达要自然流畅。 ### Agent的执行流程： 要解决任务，Agent必须通过一系列步骤向前规划，以'思考：'和'代码：'序列循环进行。**注意：禁止在代码执行前输出'观察结果：'，观察结果只能由代码执行后产生。** @@ -169,6 +175,86 @@ FEW_SHOTS_SYSTEM_PROMPT: |- --- + 任务6："同时搜索Python设计模式和Java设计模式" + + 思考：两个搜索任务互不依赖，使用parallel_executor并行调用。 + 代码： + + results = parallel_executor( + tasks=[ + (web_search, {"query": "Python设计模式"}), + (web_search, {"query": "Java设计模式"}), + ], + ) + print(results[0]) # Python设计模式搜索结果 + print(results[1]) # Java设计模式搜索结果 + + # 系统返回 Observation: [搜索结果1..., 搜索结果2...] + + 思考：已获得两个搜索结果，现在整合回答。 + Python设计模式方面...，Java设计模式方面... + + --- + + 任务7："检查代码安全性和风格" + + 思考：需要调用security_scanner和style_checker两个助手，它们互不依赖，用parallel_executor带名称并行。 + 代码： + + results = parallel_executor( + tasks=[ + (security_scanner, {"task": "检查SQL注入和XSS漏洞"}, "security"), + (style_checker, {"task": "检查PEP8规范和函数命名"}, "style"), + ], + ) + print(results["security"]) + print(results["style"]) + + # 系统返回 Observation: {"security": "发现1个SQL注入漏洞...", "style": "函数名应使用snake_case..."} + + 思考：已获得安全分析和风格检查结果，现在整合两份报告。 + 安全方面发现SQL注入漏洞...，风格方面建议函数名改为snake_case... + + --- + + 任务8："同时搜索资料和分析数据" + + 思考：网络搜索和数据分析互不依赖，用parallel_executor并行，设置超时避免长时间等待。 + 代码： + + results = parallel_executor( + tasks=[ + (web_search, {"query": "2024年AI发展趋势"}), + (data_analyst, {"task": "分析销售数据CSV文件中的季度趋势"}), + ], + timeout=60, + max_workers=2, + ) + print(results[0]) + print(results[1]) + + # 系统返回 Observation: [搜索结果..., 数据分析报告...] + + 思考：已获得搜索结果和分析报告，整合回答。 + 2024年AI发展趋势...，销售数据季度趋势分析... + + --- + + 任务9："查询出诊信息" + + 思考：我需要使用aidp_search工具在出诊信息库中搜索相关信息。 + 代码： + + appointment_info = aidp_search(query="出诊信息", kds_list=["出诊信息库"]) + print(appointment_info) + + # 系统返回 Observation: 找到相关出诊信息... + + 思考：已获得出诊信息，现在我将生成最终回答。 + 根据出诊信息库中的查询结果... + + --- + ### 要求： 1. 只展示你设计的提示词，仅涉及使用示例，不要显示无关内容或无关的格式。 2. 严格按照示例模板的格式给出例子。 @@ -252,6 +338,13 @@ USER_PROMPT: |- 请将这些名称直接用于示例中，例如：knowledge_base_search(query="xxx", index_names=[{{ knowledge_base_names | default('') }}]) {% endif %} + {% if aidp_kb_names %} + ### aidp_search 知识库配置说明： + kds_list 参数是可选的，不传时使用工具默认配置的知识库；如果需要指定搜索特定知识库，可通过 kds_list 传入以下名称（不是 index_names）： + {{ aidp_kb_names | default('') }} + 示例：aidp_search(query="xxx", kds_list=[{{ aidp_kb_names | default('') }}]) + {% endif %} + AGENT_NAME_REGENERATE_SYSTEM_PROMPT: |- ### 你是【Agent变量名调整专家】 diff --git a/backend/pyproject.toml b/backend/pyproject.toml index 9481cdbc7a..77ead385da 100644 --- a/backend/pyproject.toml +++ b/backend/pyproject.toml @@ -18,6 +18,7 @@ dependencies = [ "supabase>=2.18.1", "websocket-client>=1.8.0", "pyyaml>=6.0.2", + "rapidfuzz>=3.0.0", "jsonref>=1.1.0", "ruamel-yaml==0.19.1", "redis>=5.0.0", @@ -30,6 +31,7 @@ dependencies = [ "pydantic-settings>=2.0.0", "python-docx>=1.1.0", "xlrd>=2.0.1", + "croniter>=2.0.0", ] [project.optional-dependencies] diff --git a/backend/services/a2a_agent_adapter.py b/backend/services/a2a_agent_adapter.py index 36f10657eb..68fc8def76 100644 --- a/backend/services/a2a_agent_adapter.py +++ b/backend/services/a2a_agent_adapter.py @@ -403,12 +403,14 @@ def build_a2a_task_event( } if event_type == "taskArtifact": + last_chunk = data.get("lastChunk", True) + artifact = {**data.get("artifact", {}), "lastChunk": last_chunk} return { "artifactUpdate": { **common_fields, - "artifact": data.get("artifact", {}), + "artifact": artifact, "append": data.get("append", False), - "lastChunk": data.get("lastChunk", True) + "lastChunk": last_chunk } } diff --git a/backend/services/a2a_client_service.py b/backend/services/a2a_client_service.py index e4e81fec57..89ea2f518f 100644 --- a/backend/services/a2a_client_service.py +++ b/backend/services/a2a_client_service.py @@ -6,12 +6,13 @@ """ import asyncio import logging +from typing import Any, AsyncIterator, Dict, List, Optional, Tuple + import aiohttp -from typing import Any, AsyncIterator, Dict, List, Optional from database import a2a_agent_db -from database.a2a_agent_db import _extract_protocol_type, PROTOCOL_HTTP_JSON, PROTOCOL_JSONRPC -from utils.a2a_http_client import A2AHttpClient, build_a2a_headers +from database.a2a_agent_db import PROTOCOL_HTTP_JSON, PROTOCOL_JSONRPC +from utils.a2a_http_client import A2AHttpClient, A2AHttpStatusError, build_a2a_headers logger = logging.getLogger(__name__) @@ -45,7 +46,8 @@ async def discover_from_url( self, url: str, tenant_id: str, - user_id: str + user_id: str, + custom_headers: Optional[Dict[str, str]] = None ) -> Dict[str, Any]: """Discover an external A2A agent from a URL. @@ -55,6 +57,7 @@ async def discover_from_url( url: Direct URL to the Agent Card (e.g., https://example.com/.well-known/agent-xxx.json). tenant_id: Tenant ID for isolation. user_id: User who initiated the discovery. + custom_headers: Headers saved only for Agent Card discovery and refresh. Returns: Discovered agent information dict. @@ -63,8 +66,12 @@ async def discover_from_url( AgentDiscoveryError: If discovery fails. """ try: + # custom_headers=None means preserve existing stored headers (don't pass anything to DB). + # custom_headers={} means explicitly clear stored headers. async with A2AHttpClient() as client: headers = build_a2a_headers() + if custom_headers: + headers.update(custom_headers) card = await client.get_json(url, headers=headers) # Extract agent info from Card @@ -117,7 +124,10 @@ async def discover_from_url( tenant_id=tenant_id, user_id=user_id, raw_card=card, - supported_interfaces=supported_interfaces + supported_interfaces=supported_interfaces, + agent_card_headers=custom_headers, + security_schemes=card.get("securitySchemes"), + security_requirements=card.get("securityRequirements"), ) logger.info(f"Discovered A2A agent {agent_id} from URL: {url}") @@ -320,7 +330,9 @@ async def _discover_single_from_nacos( tenant_id=tenant_id, user_id=user_id, raw_card=agent_info, - supported_interfaces=supported_interfaces + supported_interfaces=supported_interfaces, + security_schemes=agent_info.get("securitySchemes"), + security_requirements=agent_info.get("securityRequirements"), ) return result @@ -473,7 +485,11 @@ async def refresh_agent_card( AgentDiscoveryError: If refresh fails. """ # Get current agent info - agent = a2a_agent_db.get_external_agent_by_id(external_agent_id, tenant_id) + agent = a2a_agent_db.get_external_agent_by_id( + external_agent_id, + tenant_id, + include_agent_card_headers=True, + ) if not agent: raise AgentDiscoveryError(f"Agent {external_agent_id} not found") @@ -524,70 +540,30 @@ async def refresh_agent_card( raise AgentDiscoveryError("No source URL available for refresh") async with A2AHttpClient() as client: - card = await client.get_json(source_url) - - # Extract updated info - use _extract_agent_url for A2A v1.0 standard - new_url = self._extract_agent_url(card) + card_headers = agent.get("agent_card_headers") + headers = build_a2a_headers() + if isinstance(card_headers, dict): + headers.update(card_headers) + card = await client.get_json(source_url, headers=headers) + + new_supported_interfaces = card.get("supportedInterfaces") + new_url = self._extract_agent_url(card) if new_supported_interfaces is None else None new_name = card.get("name") new_description = card.get("description") - new_supported_interfaces = card.get("supportedInterfaces", []) - - # Extract new protocol type from the card - new_protocol_type = _extract_protocol_type(new_supported_interfaces) - current_protocol_type = agent.get("protocol_type") - - # Determine if we need to update agent_url and protocol_type - # Update agent_url if it changed in the remote card - update_agent_url = new_url is not None and new_url != agent_url - - # Update protocol_type if it changed in the remote card - update_protocol_type = new_protocol_type != current_protocol_type - - # When protocol_type changes, we need to find the corresponding interface URL - if update_protocol_type: - logger.info( - f"Protocol type changed for agent {external_agent_id}: " - f"{current_protocol_type} -> {new_protocol_type}" - ) - # The database function will handle finding the correct interface URL - result = a2a_agent_db.refresh_external_agent_cache( - external_agent_id=external_agent_id, - tenant_id=tenant_id, - user_id=user_id, - new_raw_card=card, - new_agent_url=new_url if update_agent_url else None, - new_name=new_name, - new_description=new_description, - new_supported_interfaces=new_supported_interfaces, - new_protocol_type=new_protocol_type - ) - elif update_agent_url: - # Only agent_url changed - logger.info( - f"Agent URL changed for agent {external_agent_id}: " - f"{agent_url} -> {new_url}" - ) - result = a2a_agent_db.refresh_external_agent_cache( - external_agent_id=external_agent_id, - tenant_id=tenant_id, - user_id=user_id, - new_raw_card=card, - new_agent_url=new_url, - new_name=new_name, - new_description=new_description, - new_supported_interfaces=new_supported_interfaces - ) - else: - # No changes to agent_url or protocol_type, just update metadata - result = a2a_agent_db.refresh_external_agent_cache( - external_agent_id=external_agent_id, - tenant_id=tenant_id, - user_id=user_id, - new_raw_card=card, - new_name=new_name, - new_description=new_description, - new_supported_interfaces=new_supported_interfaces - ) + + # The selected protocol and its endpoint must survive incomplete Card refreshes. + result = a2a_agent_db.refresh_external_agent_cache( + external_agent_id=external_agent_id, + tenant_id=tenant_id, + user_id=user_id, + new_raw_card=card, + new_agent_url=new_url, + new_name=new_name, + new_description=new_description, + new_supported_interfaces=new_supported_interfaces, + new_security_schemes=card.get("securitySchemes", {}), + new_security_requirements=card.get("securityRequirements", []), + ) # Update availability a2a_agent_db.update_agent_availability( @@ -627,15 +603,10 @@ def delete_external_agent(self, external_agent_id: int, tenant_id: str) -> bool: # ============================================================================= def _build_endpoint_url(self, agent_url: str, protocol_type: str, streaming: bool = False) -> str: - """Build the complete endpoint URL by appending protocol-specific path. + """Build the request URL from the Agent Card protocol binding. - Args: - agent_url: Base agent URL from database. - protocol_type: Protocol type (JSONRPC, HTTP+JSON, GRPC). - streaming: Whether this is a streaming request. - - Returns: - Complete endpoint URL with protocol path appended. + The Agent Card URL is a complete endpoint for JSON-RPC. HTTP+JSON URLs + identify the service base and require the A2A message operation suffix. """ base_url = agent_url.rstrip("/") path_suffix = self._get_protocol_path(protocol_type, streaming) @@ -644,13 +615,73 @@ def _build_endpoint_url(self, agent_url: str, protocol_type: str, streaming: boo return base_url def _get_protocol_path(self, protocol_type: str, streaming: bool) -> str: - """Get the path suffix for a given protocol type and streaming mode.""" + """Get the required HTTP+JSON operation suffix for an A2A request.""" if protocol_type == PROTOCOL_HTTP_JSON: return "/message:stream" if streaming else "/message:send" - if protocol_type == PROTOCOL_JSONRPC: - return "/v1" return "" + def _build_security_request_parts( + self, + agent: Dict[str, Any], + ) -> Tuple[Dict[str, str], Dict[str, str], Dict[str, str]]: + """Build request authentication parts from the configured Card security scheme.""" + schemes = agent.get("security_schemes") or {} + requirements = agent.get("security_requirements") or [] + credentials = agent.get("security_credentials") or {} + if not requirements: + return {}, {}, {} + + selected_index = agent.get("selected_security_requirement_index") + if selected_index is not None: + if not isinstance(selected_index, int) or not 0 <= selected_index < len(requirements): + raise AgentCallError("Selected Agent Card security requirement is invalid") + requirements = [requirements[selected_index]] + + for requirement in requirements: + required_schemes = requirement.get("schemes", {}) if isinstance(requirement, dict) else {} + if not required_schemes: + return {}, {}, {} + + headers: Dict[str, str] = {} + params: Dict[str, str] = {} + cookies: Dict[str, str] = {} + valid = True + for scheme_id in required_schemes: + credential = credentials.get(scheme_id) + scheme = schemes.get(scheme_id, {}) + if not isinstance(scheme, dict) or not credential: + valid = False + break + + http_auth_scheme = scheme.get("httpAuthSecurityScheme", {}) + if http_auth_scheme: + auth_scheme = http_auth_scheme.get("scheme") if isinstance(http_auth_scheme, dict) else None + if not isinstance(auth_scheme, str) or not auth_scheme.strip(): + valid = False + break + bearer_format = http_auth_scheme.get("bearerFormat") + if isinstance(bearer_format, str) and bearer_format.lower() == "jwt": + auth_scheme = "Bearer" + headers["Authorization"] = f"{auth_scheme} {credential}" + continue + + api_key_scheme = scheme.get("apiKeySecurityScheme", {}) + location = api_key_scheme.get("location") + parameter_name = api_key_scheme.get("name") + if not parameter_name or location not in {"header", "query", "cookie"}: + valid = False + break + if location == "header": + headers[parameter_name] = credential + elif location == "query": + params[parameter_name] = credential + else: + cookies[parameter_name] = credential + if valid: + return headers, params, cookies + + raise AgentCallError("Configured credentials do not satisfy the Agent Card security requirements") + async def call_agent( self, external_agent_id: int, @@ -672,7 +703,11 @@ async def call_agent( AgentCallError: If the call fails. """ # Get agent info - agent = a2a_agent_db.get_external_agent_by_id(external_agent_id, tenant_id) + agent = a2a_agent_db.get_external_agent_by_id( + external_agent_id, + tenant_id, + include_security_credentials=True, + ) if not agent: raise AgentCallError(f"Agent {external_agent_id} not found") @@ -685,7 +720,10 @@ async def call_agent( # Build complete endpoint URL with protocol path endpoint_url = self._build_endpoint_url(agent_url, protocol_type, streaming=False) - logger.info(f"[A2A-CLIENT] === Calling external A2A agent === id={external_agent_id}, url={endpoint_url}, protocol={protocol_type}, message={message}") + logger.info( + f"[A2A-CLIENT] Calling external agent id={external_agent_id}, " + f"url={endpoint_url}, protocol={protocol_type}" + ) try: # Build request based on protocol type @@ -705,11 +743,22 @@ async def call_agent( "message": message } - logger.info(f"Calling external A2A agent {external_agent_id}: url={endpoint_url}, protocol={protocol_type}, payload={payload}") + logger.info( + f"Calling external A2A agent {external_agent_id}: " + f"url={endpoint_url}, protocol={protocol_type}" + ) headers = build_a2a_headers() + auth_headers, auth_params, auth_cookies = self._build_security_request_parts(agent) + headers.update(auth_headers) async with A2AHttpClient() as client: - response = await client.post_json(endpoint_url, payload, headers) + response = await client.post_json( + endpoint_url, + payload, + headers, + params=auth_params, + cookies=auth_cookies, + ) # Parse response if "error" in response: @@ -718,6 +767,14 @@ async def call_agent( return response.get("result", response) + except A2AHttpStatusError as e: + logger.error(f"External agent {external_agent_id} returned HTTP {e.status}") + if e.status == 401: + raise AgentCallError( + "External agent authentication failed (HTTP 401). " + "Check the configured security credentials." + ) from e + raise AgentCallError(f"External agent request failed (HTTP {e.status})") from e except aiohttp.ClientError as e: logger.error(f"Failed to call agent {external_agent_id}: {e}") raise AgentCallError(f"Call failed: {str(e)}") from e @@ -744,7 +801,11 @@ async def call_agent_streaming( AgentCallError: If the call setup fails. """ # Get agent info - agent = a2a_agent_db.get_external_agent_by_id(external_agent_id, tenant_id) + agent = a2a_agent_db.get_external_agent_by_id( + external_agent_id, + tenant_id, + include_security_credentials=True, + ) if not agent: raise AgentCallError(f"Agent {external_agent_id} not found") @@ -774,13 +835,24 @@ async def call_agent_streaming( "message": message } - logger.info(f"Calling external A2A agent {external_agent_id} (streaming): url={endpoint_url}, protocol={protocol_type}, payload={payload}") + logger.info( + f"Calling external A2A agent {external_agent_id} (streaming): " + f"url={endpoint_url}, protocol={protocol_type}" + ) headers = build_a2a_headers(api_key) + auth_headers, auth_params, auth_cookies = self._build_security_request_parts(agent) + headers.update(auth_headers) try: async with A2AHttpClient() as client: - async for event in client.post_stream(endpoint_url, payload, headers): + async for event in client.post_stream( + endpoint_url, + payload, + headers, + params=auth_params, + cookies=auth_cookies, + ): yield event except aiohttp.ClientError as e: logger.error(f"Streaming call failed for agent {external_agent_id}: {e}") diff --git a/backend/services/a2a_server_service.py b/backend/services/a2a_server_service.py index 4d9c5e607d..87cce67e87 100644 --- a/backend/services/a2a_server_service.py +++ b/backend/services/a2a_server_service.py @@ -491,23 +491,86 @@ def _resolve_task_id( return task_id, context_id, is_complex_request + async def _collect_stream_events(self, stream_response) -> List[Dict[str, Any]]: + """Collect parsed agent/run SSE payloads without dropping event types.""" + events = [] + async for chunk in stream_response.body_iterator: + if isinstance(chunk, bytes): + chunk = chunk.decode("utf-8") + if not chunk.startswith("data: "): + continue + data_str = chunk[6:].strip() + if not data_str: + continue + try: + event = json.loads(data_str) + except json.JSONDecodeError: + continue + if isinstance(event, dict): + events.append(event) + return events + + def _extract_final_answer(self, events: List[Dict[str, Any]]) -> str: + """Extract the final answer for task persistence and completion metadata.""" + return "".join( + str(event.get("content", "")) + for event in events + if event.get("type") == "final_answer" + ) + + def _coalesce_consecutive_events(self, events: List[Dict[str, Any]]) -> List[Dict[str, Any]]: + """Merge adjacent string-content events with identical non-content fields.""" + coalesced = [] + for event in events: + content = event.get("content") + if not isinstance(content, str) or not coalesced: + coalesced.append(dict(event)) + continue + + previous = coalesced[-1] + previous_content = previous.get("content") + if ( + isinstance(previous_content, str) + and event.get("type") == previous.get("type") + and {key: value for key, value in event.items() if key != "content"} + == {key: value for key, value in previous.items() if key != "content"} + ): + previous["content"] += content + else: + coalesced.append(dict(event)) + return coalesced + + def _build_agent_run_event_parts( + self, + events: List[Dict[str, Any]], + ) -> List[Dict[str, Any]]: + """Represent every agent/run event as an A2A JSON data part.""" + return [ + { + "data": event, + "mediaType": "application/json", + } + for event in events + ] + async def _collect_stream_text(self, stream_response) -> str: - """Collect and accumulate text from a streaming response.""" + """Collect text from a streaming response for legacy callers.""" accumulated = [] async for chunk in stream_response.body_iterator: if isinstance(chunk, bytes): chunk = chunk.decode("utf-8") - if chunk.startswith("data: "): - data_str = chunk[6:].strip() - if not data_str: - continue - try: - chunk_data = json.loads(data_str) - text = self.adapter.extract_stream_chunk(chunk_data) - if text: - accumulated.append(text) - except json.JSONDecodeError: - pass + if not chunk.startswith("data: "): + continue + data_str = chunk[6:].strip() + if not data_str: + continue + try: + chunk_data = json.loads(data_str) + text = self.adapter.extract_stream_chunk(chunk_data) + if text: + accumulated.append(text) + except json.JSONDecodeError: + continue return "".join(accumulated) def _store_user_message(self, task_id: Optional[str], message_obj: Dict[str, Any], endpoint_id: str) -> None: @@ -639,22 +702,26 @@ async def handle_message_send( tenant_id=tenant_id or server_agent.get("tenant_id") ) - accumulated_text = await self._collect_stream_text(stream_response) - self._store_agent_response(task_id, accumulated_text, endpoint_id) + events = await self._collect_stream_events(stream_response) + final_answer = self._extract_final_answer(events) + self._store_agent_response(task_id, final_answer, endpoint_id) + raw_parts = self._build_agent_run_event_parts( + self._coalesce_consecutive_events(events) + ) if is_complex_request: - from datetime import datetime, timezone return self.adapter.build_a2a_task_response( task_id=task_id, status="TASK_STATE_COMPLETED", - parts=[{"text": accumulated_text, "mediaType": "text/plain"}] if accumulated_text else None, + parts=raw_parts or None, context_id=context_id, timestamp=datetime.now(timezone.utc).isoformat().replace("+00:00", "Z") ) else: return self.adapter.build_a2a_message_response( role="ROLE_AGENT", - text=accumulated_text, + parts=raw_parts, + text=final_answer if not raw_parts else None, context_id=context_id, task_id=task_id ) @@ -761,7 +828,7 @@ async def handle_message_stream( tenant_id=tenant_id or server_agent.get("tenant_id") ) - accumulated_text = "" + events = [] async for chunk in stream_response.body_iterator: if isinstance(chunk, bytes): chunk = chunk.decode("utf-8") @@ -772,24 +839,34 @@ async def handle_message_stream( continue try: chunk_data = json.loads(data_str) - text = self.adapter.extract_stream_chunk(chunk_data) - if text: - accumulated_text += text - yield self.adapter.build_a2a_task_event( - task_id=task_id, - event_type="taskProgress", - data={"content": text, "lastChunk": False}, - context_id=context_id - ) except json.JSONDecodeError: - pass + continue + if not isinstance(chunk_data, dict): + continue + + events.append(chunk_data) + yield self.adapter.build_a2a_task_event( + task_id=task_id or "simple", + event_type="taskArtifact", + data={ + "artifact": {"parts": self._build_agent_run_event_parts([chunk_data])}, + "append": True, + "lastChunk": False, + }, + context_id=context_id + ) - self._store_agent_response(task_id, accumulated_text, endpoint_id) + final_answer = self._extract_final_answer(events) + self._store_agent_response(task_id, final_answer, endpoint_id) yield self.adapter.build_a2a_task_event( task_id=task_id or "simple", - event_type="taskProgress", - data={"content": accumulated_text, "lastChunk": True}, + event_type="taskArtifact", + data={ + "artifact": {"parts": []}, + "append": True, + "lastChunk": True, + }, context_id=context_id ) diff --git a/backend/services/agent_automation/__init__.py b/backend/services/agent_automation/__init__.py new file mode 100644 index 0000000000..514e860e2c --- /dev/null +++ b/backend/services/agent_automation/__init__.py @@ -0,0 +1 @@ +"""Agent automation domain module.""" diff --git a/backend/services/agent_automation/agent_identity_adapter.py b/backend/services/agent_automation/agent_identity_adapter.py new file mode 100644 index 0000000000..b9de6accf3 --- /dev/null +++ b/backend/services/agent_automation/agent_identity_adapter.py @@ -0,0 +1,77 @@ +import logging +from collections import defaultdict +from typing import Dict, Iterable, Optional, Tuple + +from sqlalchemy import or_, select + +from consts.const import ASSET_OWNER_TENANT_ID +from database.client import get_db_session +from database.db_models import AgentInfo + + +AgentReference = Tuple[int, int] +logger = logging.getLogger("agent_automation.agent_identity_adapter") + + +def resolve_agent_display_names( + references: Iterable[Tuple[int, Optional[int]]], + tenant_id: str, +) -> Dict[AgentReference, str]: + """Resolve user-facing Agent names in one query at the automation boundary.""" + normalized_references = { + (int(agent_id), int(version_no or 0)) + for agent_id, version_no in references + if agent_id + } + if not normalized_references: + return {} + + agent_ids = {agent_id for agent_id, _ in normalized_references} + try: + with get_db_session() as session: + rows = session.execute( + select( + AgentInfo.agent_id, + AgentInfo.version_no, + AgentInfo.name, + AgentInfo.display_name, + AgentInfo.tenant_id, + ).where( + AgentInfo.agent_id.in_(agent_ids), + or_( + AgentInfo.tenant_id == tenant_id, + AgentInfo.tenant_id == ASSET_OWNER_TENANT_ID, + ), + AgentInfo.delete_flag != "Y", + ) + ).all() + except Exception: + logger.warning("Failed to resolve Agent display names", exc_info=True) + return {} + + candidates = defaultdict(dict) + for row in rows: + key = (int(row.agent_id), int(row.version_no or 0)) + current = candidates[key].get("row") + if current is None or row.tenant_id == tenant_id: + candidates[key]["row"] = row + + resolved: Dict[AgentReference, str] = {} + for reference in normalized_references: + agent_id, version_no = reference + row = candidates.get((agent_id, version_no), {}).get("row") + if row is None: + row = candidates.get((agent_id, 0), {}).get("row") + if row is None: + available = [ + item["row"] + for (candidate_id, _), item in candidates.items() + if candidate_id == agent_id and item.get("row") is not None + ] + row = max(available, key=lambda item: int(item.version_no or 0), default=None) + if row is not None: + display_name = (row.display_name or row.name or "").strip() + if display_name: + resolved[reference] = display_name + + return resolved diff --git a/backend/services/agent_automation/capability_resolver.py b/backend/services/agent_automation/capability_resolver.py new file mode 100644 index 0000000000..9fca40f7a2 --- /dev/null +++ b/backend/services/agent_automation/capability_resolver.py @@ -0,0 +1,192 @@ +import re +from typing import Any, Dict, Iterable, List, Optional + +from .agent_identity_adapter import resolve_agent_display_names +from .models import CapabilityBinding, CapabilityResolution, CapabilityType + + +SEARCH_HINTS = ("联网", "网络", "搜索", "新闻", "网页", "竞品", "公开") +KNOWLEDGE_HINTS = ("知识库", "文档", "资料", "项目", "销售线索") + + +def _safe_text(*parts: Any) -> str: + return " ".join(str(part or "") for part in parts).lower() + + +def _tool_binding(tool: Any) -> CapabilityBinding: + class_name = getattr(tool, "class_name", "") or "" + name = getattr(tool, "name", None) or class_name + metadata = getattr(tool, "metadata", None) or {} + if class_name == "KnowledgeBaseSearchTool": + index_names = (getattr(tool, "params", None) or {}).get("index_names", []) + display_map = metadata.get("index_name_to_display_map", {}) + display = ", ".join(display_map.get(index, index) for index in index_names) if index_names else name + return CapabilityBinding( + type=CapabilityType.KNOWLEDGE_BASE, + name=",".join(index_names) if index_names else name, + display_name=display, + binding_ref=f"tool:KnowledgeBaseSearchTool:index:{','.join(index_names)}", + reason="Agent has a configured knowledge-base search tool.", + ) + return CapabilityBinding( + type=CapabilityType.TOOL, + name=name, + display_name=name, + binding_ref=f"tool:{name}", + reason=getattr(tool, "description", None) or "Agent has this tool configured.", + ) + + +def _skill_binding(skill: Dict[str, Any]) -> CapabilityBinding: + name = skill.get("name") or "" + return CapabilityBinding( + type=CapabilityType.SKILL, + name=name, + display_name=name, + binding_ref=f"skill:{name}", + reason=skill.get("description") or "Agent has this skill enabled.", + ) + + +def _agent_binding(agent: Any, capability_type: CapabilityType) -> CapabilityBinding: + name = getattr(agent, "name", None) or getattr(agent, "agent_id", "") + return CapabilityBinding( + type=capability_type, + name=str(name), + display_name=str(name), + binding_ref=f"{capability_type.value.lower()}:{name}", + reason=getattr(agent, "description", None) or "Agent is available as a callable capability.", + ) + + +def _flatten_bindings(bindings: Iterable[CapabilityBinding]) -> Dict[str, CapabilityBinding]: + return {binding.binding_ref: binding for binding in bindings} + + +async def resolve_agent_capabilities( + agent_id: int, + tenant_id: str, + user_id: str, + instruction: str, + version_no: int = 0, +) -> CapabilityResolution: + """Resolve capabilities from the same assembly path used by normal agent runs.""" + from agents.create_agent_info import create_agent_config + from services.skill_service import SkillService + + agent_config = await create_agent_config( + agent_id=agent_id, + tenant_id=tenant_id, + user_id=user_id, + last_user_query=instruction, + version_no=version_no, + allow_memory_search=False, + ) + + tool_bindings = [_tool_binding(tool) for tool in getattr(agent_config, "tools", []) or []] + skill_bindings = [] + try: + enabled_skills = SkillService().get_enabled_skills_for_agent( + agent_id=agent_id, + tenant_id=tenant_id, + version_no=version_no, + ) + skill_bindings.extend(_skill_binding(skill) for skill in enabled_skills) + except Exception: + enabled_skills = [] + + # Skills may also be carried in context components for context-manager agents. + for component in getattr(agent_config, "context_components", []) or []: + if getattr(component, "component_type", None) == "skills": + for skill in getattr(component, "skills", []) or []: + skill_bindings.append(_skill_binding(skill)) + + managed_bindings = [ + _agent_binding(agent, CapabilityType.MANAGED_AGENT) + for agent in getattr(agent_config, "managed_agents", []) or [] + ] + a2a_bindings = [ + _agent_binding(agent, CapabilityType.EXTERNAL_A2A_AGENT) + for agent in getattr(agent_config, "external_a2a_agents", []) or [] + ] + + all_bindings = tool_bindings + skill_bindings + managed_bindings + a2a_bindings + lower_instruction = instruction.lower() + missing: List[Dict[str, Any]] = [] + + has_search_tool = any( + binding.type == CapabilityType.TOOL + and re.search(r"search|linkup|exa|web|联网|搜索", binding.name.lower()) + for binding in all_bindings + ) + if any(hint in instruction for hint in SEARCH_HINTS) and not has_search_tool: + missing.append({ + "type": CapabilityType.TOOL.value, + "name": "web_search", + "suggestion": ( + "请先为该 Agent 启用联网搜索工具，" + "或删除任务中的联网/新闻/网页检索要求。" + ), + }) + + has_knowledge = any(binding.type == CapabilityType.KNOWLEDGE_BASE for binding in all_bindings) + if any(hint in instruction for hint in KNOWLEDGE_HINTS) and not has_knowledge: + missing.append({ + "type": CapabilityType.KNOWLEDGE_BASE.value, + "name": "knowledge_base", + "suggestion": "请先为该 Agent 选择知识库，或修改任务为仅基于当前会话执行。", + }) + + matched = [] + for binding in all_bindings: + text = _safe_text(binding.name, binding.display_name, binding.reason) + instruction_tokens = re.findall(r"[\w\u4e00-\u9fff]+", lower_instruction) + if not lower_instruction or any(token in text for token in instruction_tokens): + matched.append(binding) + + if not matched: + matched = all_bindings[:10] + + agent_display_name = resolve_agent_display_names( + [(agent_id, version_no)], + tenant_id, + ).get((agent_id, version_no)) + + return CapabilityResolution( + matched_capabilities=matched[:20], + missing_capabilities=missing, + agent_snapshot={ + "agent_id": agent_id, + "version_no": version_no, + "name": getattr(agent_config, "name", ""), + "display_name": agent_display_name or getattr(agent_config, "name", ""), + "description": getattr(agent_config, "description", ""), + "tools_count": len(tool_bindings), + "skills_count": len(skill_bindings), + "managed_agents_count": len(managed_bindings), + "external_a2a_agents_count": len(a2a_bindings), + }, + executable=len(missing) == 0, + ) + + +async def validate_bindings_available( + agent_id: int, + tenant_id: str, + user_id: str, + instruction: str, + bindings: List[Dict[str, Any]], + version_no: int = 0, +) -> Dict[str, Any]: + resolution = await resolve_agent_capabilities(agent_id, tenant_id, user_id, instruction, version_no) + available = _flatten_bindings(resolution.matched_capabilities) + unavailable = [] + for binding in bindings or []: + ref = binding.get("binding_ref") + if ref and ref not in available: + unavailable.append(binding) + return { + "available": not unavailable and resolution.executable, + "unavailable_bindings": unavailable, + "resolution": resolution.model_dump(mode="json"), + } diff --git a/backend/services/agent_automation/conversation_adapter.py b/backend/services/agent_automation/conversation_adapter.py new file mode 100644 index 0000000000..95d9167713 --- /dev/null +++ b/backend/services/agent_automation/conversation_adapter.py @@ -0,0 +1,172 @@ +import json +import logging +from typing import Any, Dict, List, Optional + +from consts.model import HistoryItem, MessageRequest, MessageUnit +from services.conversation_management_service import ( + get_conversation_history_service, + save_message, + save_message_unit, + update_unit_content, +) + +logger = logging.getLogger("agent_automation.conversation_adapter") + + +class AutomationConversationAdapter: + """Persist automation UI events through the existing conversation service.""" + + @staticmethod + def _message_content(message: Dict[str, Any]) -> str: + content = message.get("message", "") + if isinstance(content, list): + final = next( + ( + unit.get("content") + for unit in reversed(content) + if unit.get("type") == "final_answer" + ), + "", + ) + visible_units = [ + unit + for unit in content + if unit.get("type") != "automation_proposal" + ] + content = final or " ".join( + str(unit.get("content", "")) for unit in visible_units + ) + return str(content or "") + + @classmethod + def _history_items(cls, messages: List[Dict[str, Any]]) -> List[HistoryItem]: + history: List[HistoryItem] = [] + latest_positions = { + message["message_index"]: position + for position, message in enumerate(messages) + if isinstance(message.get("message_index"), int) + } + for position, message in enumerate(messages): + message_index = message.get("message_index") + if ( + isinstance(message_index, int) + and latest_positions[message_index] != position + ): + continue + content = cls._message_content(message) + if content: + history.append( + HistoryItem( + role=message.get("role", "user"), + content=content, + ) + ) + return history + + def append_run_prompt( + self, + conversation_id: int, + prompt: str, + user_id: str, + tenant_id: str, + ) -> Dict[str, Any]: + """Append a new automation turn without regenerating an existing message.""" + history_payload = get_conversation_history_service(conversation_id, user_id) + messages = history_payload[0].get("message", []) if history_payload else [] + message_indexes = [ + message.get("message_index") + for message in messages + if isinstance(message.get("message_index"), int) + ] + if message_indexes: + highest_message_index = max(message_indexes) + next_message_index = highest_message_index + ( + 1 if highest_message_index % 2 else 2 + ) + else: + next_message_index = len(messages) + request = MessageRequest( + conversation_id=conversation_id, + message_idx=next_message_index, + role="user", + message=[MessageUnit(type="string", content=prompt)], + ) + message_id = save_message(request, user_id, tenant_id) + save_message_unit( + message_id=message_id, + conversation_id=conversation_id, + unit_index=0, + unit_type="automation_prompt", + unit_content=prompt, + user_id=user_id, + ) + return { + "user_message_id": message_id, + "history": self._history_items(messages), + } + + def append_proposal_exchange( + self, + conversation_id: int, + user_instruction: str, + payload: Dict[str, Any], + user_id: str, + tenant_id: str, + ) -> Dict[str, int]: + history_payload = get_conversation_history_service(conversation_id, user_id) + messages = history_payload[0].get("message", []) if history_payload else [] + user_role_count = sum(message.get("role") == "user" for message in messages) + user_request = MessageRequest( + conversation_id=conversation_id, + message_idx=user_role_count * 2, + role="user", + message=[MessageUnit(type="string", content=user_instruction)], + ) + user_message_id = save_message(user_request, user_id, tenant_id) + user_unit_id = save_message_unit( + message_id=user_message_id, + conversation_id=conversation_id, + unit_index=0, + unit_type="string", + unit_content=user_instruction, + user_id=user_id, + ) + content = json.dumps(payload, ensure_ascii=False) + request = MessageRequest( + conversation_id=conversation_id, + message_idx=user_role_count * 2 + 1, + role="assistant", + message=[MessageUnit(type="automation_proposal", content=content)], + ) + message_id = save_message(request, user_id, tenant_id) + unit_id = save_message_unit( + message_id=message_id, + conversation_id=conversation_id, + unit_index=0, + unit_type="automation_proposal", + unit_content=content, + user_id=user_id, + ) + return { + "user_message_id": user_message_id, + "user_unit_id": user_unit_id, + "message_id": message_id, + "unit_id": unit_id, + } + + def update_proposal( + self, + unit_id: Optional[int], + payload: Dict[str, Any], + user_id: str, + ) -> None: + if not unit_id: + return + update_unit_content( + unit_id, + json.dumps(payload, ensure_ascii=False), + user_id, + ) + + +automation_conversation_adapter = AutomationConversationAdapter() diff --git a/backend/services/agent_automation/errors.py b/backend/services/agent_automation/errors.py new file mode 100644 index 0000000000..5c9e40ca97 --- /dev/null +++ b/backend/services/agent_automation/errors.py @@ -0,0 +1,33 @@ +class AgentAutomationError(Exception): + """Base domain exception for agent automation.""" + + error_code = "AUTOMATION_ERROR" + + def __init__(self, message: str, details: dict | None = None): + super().__init__(message) + self.message = message + self.details = details or {} + + +class AutomationCapabilityNotReadyError(AgentAutomationError): + error_code = "AUTOMATION_CAPABILITY_NOT_READY" + + +class AutomationCapabilityUnavailableError(AgentAutomationError): + error_code = "AUTOMATION_CAPABILITY_UNAVAILABLE" + + +class AutomationCapabilityBindingInvalidError(AgentAutomationError): + error_code = "AUTOMATION_CAPABILITY_BINDING_INVALID" + + +class AutomationScheduleInvalidError(AgentAutomationError): + error_code = "AUTOMATION_SCHEDULE_INVALID" + + +class AutomationNotFoundError(AgentAutomationError): + error_code = "AUTOMATION_TASK_NOT_FOUND" + + +class AutomationConversationAlreadyBoundError(AgentAutomationError): + error_code = "AUTOMATION_CONVERSATION_ALREADY_BOUND" diff --git a/backend/services/agent_automation/facade.py b/backend/services/agent_automation/facade.py new file mode 100644 index 0000000000..a1ea79cf59 --- /dev/null +++ b/backend/services/agent_automation/facade.py @@ -0,0 +1,873 @@ +import logging +from datetime import datetime, timedelta, timezone +from typing import Any, Dict, List, Optional + +from sqlalchemy.exc import IntegrityError + +from consts.const import AGENT_AUTOMATION_DEFAULT_TIMEOUT_SECONDS, AGENT_AUTOMATION_MIN_INTERVAL_SECONDS +from database import agent_automation_db +from database.conversation_db import get_conversation +from services.conversation_management_service import ( + create_new_conversation, + update_conversation_agent_id_service, +) +from . import agent_identity_adapter +from .capability_resolver import resolve_agent_capabilities, validate_bindings_available +from .conversation_adapter import automation_conversation_adapter +from .errors import ( + AutomationCapabilityBindingInvalidError, + AutomationCapabilityNotReadyError, + AutomationConversationAlreadyBoundError, + AutomationNotFoundError, + AutomationScheduleInvalidError, +) +from .intent_analyzer import AutomationIntentContext, automation_intent_analyzer +from .models import ( + AutomationProposalConfirmRequest, + AutomationProposalCreateRequest, + AutomationProposalPatchRequest, + AutomationProposalStatus, + AutomationRunStatus, + AutomationSource, + AutomationTaskCreateRequest, + AutomationTaskPatchRequest, + AutomationTaskStatus, + CapabilityBinding, + ScheduleTrigger, +) +from .prompt_generator import ( + AutomationPromptContext, + AutomationTaskContent, + automation_prompt_generator, + detect_instruction_language, +) +from .schedule_engine import compute_next_fire_at, is_valid_cron_expression + +logger = logging.getLogger("agent_automation.facade") + + +def _utcnow() -> datetime: + return datetime.now(timezone.utc) + + +def _as_utc(value: datetime | str) -> datetime: + if isinstance(value, str): + value = datetime.fromisoformat(value.replace("Z", "+00:00")) + return value.astimezone(timezone.utc) if value.tzinfo else value.replace(tzinfo=timezone.utc) + + +def _json(data: Any) -> Any: + if hasattr(data, "model_dump"): + return data.model_dump(mode="json") + if isinstance(data, list): + return [_json(item) for item in data] + return data + + +def _agent_reference(task: Dict[str, Any]) -> tuple[int, int]: + return int(task["agent_id"]), int(task.get("agent_version_no") or 0) + + +def _enrich_tasks_with_agent_names( + tasks: List[Dict[str, Any]], + tenant_id: str, + user_id: str, +) -> List[Dict[str, Any]]: + if not tasks: + return [] + try: + names = agent_identity_adapter.resolve_agent_display_names( + [_agent_reference(task) for task in tasks], + tenant_id, + ) + except Exception: + logger.warning("Failed to resolve Agent display names", exc_info=True) + names = {} + try: + active_run_task_ids = agent_automation_db.get_active_run_task_ids( + [int(task["task_id"]) for task in tasks], + tenant_id, + user_id, + ) + except Exception: + logger.warning("Failed to resolve active automation runs", exc_info=True) + active_run_task_ids = set() + enriched = [] + for task in tasks: + snapshot = task.get("runtime_snapshot") or {} + agent_id, version_no = _agent_reference(task) + agent_name = ( + names.get((agent_id, version_no)) + or snapshot.get("display_name") + or snapshot.get("name") + or f"Agent #{agent_id}" + ) + enriched.append({ + **task, + "agent_name": agent_name, + "is_running": int(task["task_id"]) in active_run_task_ids, + }) + return enriched + + +def _enrich_task_with_agent_name(task: Dict[str, Any], tenant_id: str, user_id: str) -> Dict[str, Any]: + return _enrich_tasks_with_agent_names([task], tenant_id, user_id)[0] + + +def _paginate(items: List[Dict[str, Any]], page: int, page_size: int) -> Dict[str, Any]: + start = (page - 1) * page_size + return { + "items": items[start:start + page_size], + "total": len(items), + "page": page, + "page_size": page_size, + } + + +def _parse_trigger(raw: Dict[str, Any] | ScheduleTrigger) -> ScheduleTrigger: + return raw if isinstance(raw, ScheduleTrigger) else ScheduleTrigger.model_validate(raw) + + +def _proposal_response_from_row(proposal: Dict[str, Any]) -> Dict[str, Any]: + proposed_task = proposal.get("proposed_task") or {} + public_task = { + key: value for key, value in proposed_task.items() if not key.startswith("_") + } + resolution = proposal.get("capability_resolution") or {} + return { + "proposal_id": proposal["proposal_id"], + "conversation_id": proposal["conversation_id"], + "confidence": 1.0, + "executable": bool(resolution.get("executable", True)), + "task": public_task, + "capability_resolution": resolution, + "intent_analysis_source": "existing", + "task_content_source": "existing", + } + + +def _validate_schedule_policy(trigger: ScheduleTrigger) -> None: + if trigger.mode.value == "ONCE" and _as_utc(trigger.start_at) <= _utcnow(): + raise AutomationScheduleInvalidError( + "Automation execution time must be in the future.", + details={"start_at": trigger.start_at.isoformat()}, + ) + if trigger.rule_type.value == "CRON" and not is_valid_cron_expression(trigger.cron_expr or ""): + raise AutomationScheduleInvalidError( + "Automation cron expression is invalid.", + details={"cron_expr": trigger.cron_expr}, + ) + if ( + trigger.rule_type.value == "INTERVAL" + and trigger.interval_seconds is not None + and trigger.interval_seconds < AGENT_AUTOMATION_MIN_INTERVAL_SECONDS + ): + raise AutomationScheduleInvalidError( + f"Automation interval must be at least {AGENT_AUTOMATION_MIN_INTERVAL_SECONDS} seconds.", + details={ + "min_interval_seconds": AGENT_AUTOMATION_MIN_INTERVAL_SECONDS, + "interval_seconds": trigger.interval_seconds, + }, + ) + + +class AgentAutomationFacade: + async def create_proposal( + self, + request: AutomationProposalCreateRequest, + tenant_id: str, + user_id: str, + *, + persist_conversation_exchange: bool = True, + source_message_id: Optional[int] = None, + force_llm: bool = False, + ) -> Dict[str, Any]: + if source_message_id is not None: + existing = agent_automation_db.get_proposal_by_source_message( + source_message_id, + tenant_id, + user_id, + ) + if existing: + return _proposal_response_from_row(existing) + try: + parsed = await automation_intent_analyzer.analyze(AutomationIntentContext( + tenant_id=tenant_id, + message=request.message, + timezone=request.timezone, + model_id=request.model_id, + force_llm=force_llm, + )) + except ValueError as exc: + raise AutomationScheduleInvalidError( + f"无法解析任务执行时间：{exc}", + details={"input": request.message, "timezone": request.timezone}, + ) from exc + if not parsed.get("is_automation_intent"): + return { + "proposal_id": None, + "conversation_id": request.conversation_id, + "confidence": parsed.get("confidence", 0), + "executable": False, + "task": None, + "capability_resolution": None, + "intent_analysis_source": parsed.get("analysis_source", "rule"), + "task_content_source": parsed.get("task_content_source"), + } + if parsed.get("schedule_error") or not parsed.get("schedule_trigger"): + raise AutomationScheduleInvalidError( + parsed.get("schedule_error") or "Unable to determine the automation schedule.", + details={ + "input": request.message, + "timezone": request.timezone, + "missing_fields": parsed.get("missing_fields") or [], + "clarification_question": parsed.get("clarification_question"), + }, + ) + _validate_schedule_policy(parsed["schedule_trigger"]) + + if parsed.get("task_content_generated"): + task_content = AutomationTaskContent( + title=parsed["title"], + instruction=parsed["instruction"], + ) + else: + task_content = await automation_prompt_generator.generate_task_content(AutomationPromptContext( + tenant_id=tenant_id, + instruction=parsed["instruction"], + language=detect_instruction_language(parsed["instruction"]), + )) + + conversation_id = request.conversation_id + if conversation_id is None: + conversation = create_new_conversation( + task_content.title, + user_id, + agent_id=request.agent_id, + ) + conversation_id = conversation["conversation_id"] + else: + conversation = get_conversation(conversation_id, user_id) + if not conversation: + raise AutomationNotFoundError("Conversation does not exist or is not accessible.") + if conversation_id == request.conversation_id: + update_conversation_agent_id_service( + conversation_id, + request.agent_id, + user_id, + ) + if agent_automation_db.get_task_by_conversation(conversation_id, user_id): + raise AutomationConversationAlreadyBoundError("Conversation already has an active automation task.") + + resolution = await resolve_agent_capabilities( + agent_id=request.agent_id, + tenant_id=tenant_id, + user_id=user_id, + instruction=task_content.instruction, + version_no=request.agent_version_no or 0, + ) + agent_name = ( + resolution.agent_snapshot.get("display_name") + or resolution.agent_snapshot.get("name") + or f"Agent #{request.agent_id}" + ) + proposed_task = { + "title": task_content.title, + "instruction": task_content.instruction, + "original_instruction": parsed["instruction"], + "agent_id": request.agent_id, + "agent_name": agent_name, + "agent_version_no": request.agent_version_no, + "model_id": request.model_id, + "tool_params": request.tool_params, + "schedule_trigger": parsed["schedule_trigger"].model_dump(mode="json"), + } + proposal_values = { + "tenant_id": tenant_id, + "user_id": user_id, + "conversation_id": conversation_id, + "agent_id": request.agent_id, + "source_message_id": source_message_id, + "proposed_task": proposed_task, + "capability_resolution": resolution.model_dump(mode="json"), + "status": AutomationProposalStatus.PENDING.value, + "expires_at": _utcnow() + timedelta(hours=24), + } + try: + proposal = agent_automation_db.create_proposal(proposal_values, user_id) + except IntegrityError: + if source_message_id is not None: + existing = agent_automation_db.get_proposal_by_source_message( + source_message_id, + tenant_id, + user_id, + ) + if existing: + return _proposal_response_from_row(existing) + raise + response = { + "proposal_id": proposal["proposal_id"], + "conversation_id": conversation_id, + "confidence": parsed["confidence"], + "executable": resolution.executable, + "task": proposed_task, + "capability_resolution": resolution.model_dump(mode="json"), + "intent_analysis_source": parsed.get("analysis_source", "rule"), + "task_content_source": parsed.get("task_content_source", "rule"), + } + if persist_conversation_exchange: + try: + message_refs = automation_conversation_adapter.append_proposal_exchange( + conversation_id, + request.message, + response, + user_id, + tenant_id, + ) + stored_task = { + **proposed_task, + "_conversation_message_id": message_refs["message_id"], + "_conversation_unit_id": message_refs["unit_id"], + } + agent_automation_db.update_proposal_task( + proposal["proposal_id"], + tenant_id, + user_id, + stored_task, + ) + except Exception as exc: + logger.warning("Failed to persist automation proposal card: %s", exc, exc_info=True) + return response + + async def update_proposal( + self, + proposal_id: int, + request: AutomationProposalPatchRequest, + tenant_id: str, + user_id: str, + ) -> Dict[str, Any]: + proposal = agent_automation_db.get_proposal(proposal_id, tenant_id, user_id) + editable_statuses = { + AutomationProposalStatus.PENDING.value, + AutomationProposalStatus.ACCEPTED.value, + } + if not proposal or proposal["status"] not in editable_statuses: + raise AutomationNotFoundError("Automation proposal does not exist or is not editable.") + expires_at = proposal.get("expires_at") + if ( + proposal["status"] == AutomationProposalStatus.PENDING.value + and expires_at + and _as_utc(expires_at) <= _utcnow() + ): + agent_automation_db.update_proposal_status( + proposal_id, + tenant_id, + user_id, + AutomationProposalStatus.EXPIRED.value, + ) + raise AutomationNotFoundError("Automation proposal has expired.") + + proposed_task = dict(proposal["proposed_task"]) + if request.title is not None: + proposed_task["title"] = request.title.strip() + if request.instruction is not None: + proposed_task["instruction"] = request.instruction.strip() + if request.schedule_trigger is not None: + _validate_schedule_policy(request.schedule_trigger) + proposed_task["schedule_trigger"] = request.schedule_trigger.model_dump(mode="json") + + confirmed_task_id = None + if proposal["status"] == AutomationProposalStatus.ACCEPTED.value: + task = self.get_task_for_conversation(proposal["conversation_id"], tenant_id, user_id) + if not task: + raise AutomationNotFoundError("Confirmed automation task does not exist.") + updated_task = await self.patch_task( + task["task_id"], + AutomationTaskPatchRequest( + title=request.title, + instruction=request.instruction, + schedule_trigger=request.schedule_trigger, + ), + tenant_id, + user_id, + ) + confirmed_task_id = updated_task["task_id"] + resolution_data = ( + updated_task.get("capability_requirements") + or proposal.get("capability_resolution") + or {} + ) + executable = True + else: + resolution = await resolve_agent_capabilities( + agent_id=proposal["agent_id"], + tenant_id=tenant_id, + user_id=user_id, + instruction=proposed_task["instruction"], + version_no=proposed_task.get("agent_version_no") or 0, + ) + resolution_data = resolution.model_dump(mode="json") + executable = resolution.executable + if not agent_automation_db.update_proposal( + proposal_id, + tenant_id, + user_id, + proposed_task, + resolution_data, + ): + raise AutomationNotFoundError("Automation proposal does not exist or is not editable.") + + public_task = {key: value for key, value in proposed_task.items() if not key.startswith("_")} + public_task["agent_name"] = ( + (updated_task if confirmed_task_id is not None else {}).get("agent_name") + or (resolution_data.get("agent_snapshot") or {}).get("display_name") + or (resolution_data.get("agent_snapshot") or {}).get("name") + or proposed_task.get("agent_name") + or f"Agent #{proposal['agent_id']}" + ) + response = { + "proposal_id": proposal_id, + "conversation_id": proposal["conversation_id"], + "executable": executable, + "task": public_task, + "capability_resolution": resolution_data, + } + if confirmed_task_id is not None: + response["confirmed_task_id"] = confirmed_task_id + try: + automation_conversation_adapter.update_proposal( + proposed_task.get("_conversation_unit_id"), + response, + user_id, + ) + except Exception as exc: + logger.warning("Failed to persist updated automation proposal card: %s", exc, exc_info=True) + return response + + async def confirm_proposal( + self, + proposal_id: int, + request: AutomationProposalConfirmRequest, + tenant_id: str, + user_id: str, + ) -> Dict[str, Any]: + proposal = agent_automation_db.get_proposal(proposal_id, tenant_id, user_id) + if not proposal or proposal["status"] != AutomationProposalStatus.PENDING.value: + raise AutomationNotFoundError("Automation proposal does not exist or is not pending.") + expires_at = proposal.get("expires_at") + if expires_at and _as_utc(expires_at) <= _utcnow(): + agent_automation_db.update_proposal_status( + proposal_id, + tenant_id, + user_id, + AutomationProposalStatus.EXPIRED.value, + ) + raise AutomationNotFoundError("Automation proposal has expired.") + + proposed_task = proposal["proposed_task"] + instruction = request.instruction or proposed_task["instruction"] + resolution = await resolve_agent_capabilities( + agent_id=proposal["agent_id"], + tenant_id=tenant_id, + user_id=user_id, + instruction=instruction, + version_no=proposed_task.get("agent_version_no") or 0, + ) + if not resolution.executable: + raise AutomationCapabilityNotReadyError( + "Required automation capabilities are not ready.", + details=resolution.model_dump(mode="json"), + ) + + create_request = AutomationTaskCreateRequest( + title=proposed_task["title"], + agent_id=proposal["agent_id"], + instruction=instruction, + original_instruction=proposed_task.get("original_instruction") or instruction, + schedule_trigger=_parse_trigger(proposed_task["schedule_trigger"]), + conversation_id=proposal["conversation_id"], + agent_version_no=proposed_task.get("agent_version_no"), + model_id=proposed_task.get("model_id"), + tool_params=proposed_task.get("tool_params"), + capability_bindings=resolution.matched_capabilities, + ) + task = await self.create_task(create_request, tenant_id, user_id) + agent_automation_db.update_proposal_status( + proposal_id, tenant_id, user_id, AutomationProposalStatus.ACCEPTED.value) + public_task = {key: value for key, value in proposed_task.items() if not key.startswith("_")} + public_task["agent_name"] = ( + task.get("agent_name") + or proposed_task.get("agent_name") + or f"Agent #{proposal['agent_id']}" + ) + try: + automation_conversation_adapter.update_proposal( + proposed_task.get("_conversation_unit_id"), + { + "proposal_id": proposal_id, + "executable": True, + "task": public_task, + "capability_resolution": proposal["capability_resolution"], + "confirmed_task_id": task["task_id"], + }, + user_id, + ) + except Exception as exc: + logger.warning("Failed to persist confirmed automation proposal card: %s", exc, exc_info=True) + return task + + async def create_task( + self, + request: AutomationTaskCreateRequest, + tenant_id: str, + user_id: str, + ) -> Dict[str, Any]: + trigger = request.schedule_trigger + _validate_schedule_policy(trigger) + conversation_id = request.conversation_id + if not get_conversation(conversation_id, user_id): + raise AutomationNotFoundError("Conversation does not exist or is not accessible.") + + if agent_automation_db.get_task_by_conversation(conversation_id, user_id): + raise AutomationConversationAlreadyBoundError("Conversation already has an active automation task.") + + resolution = await resolve_agent_capabilities( + agent_id=request.agent_id, + tenant_id=tenant_id, + user_id=user_id, + instruction=request.instruction, + version_no=request.agent_version_no or 0, + ) + if not resolution.executable: + raise AutomationCapabilityNotReadyError( + "Required automation capabilities are not ready.", + details=resolution.model_dump(mode="json"), + ) + + if request.capability_bindings: + check = await validate_bindings_available( + request.agent_id, + tenant_id, + user_id, + request.instruction, + [_json(binding) for binding in request.capability_bindings], + request.agent_version_no or 0, + ) + if check["unavailable_bindings"]: + raise AutomationCapabilityBindingInvalidError( + "Submitted capability bindings are not available for this agent.", + details=check, + ) + bindings = [_json(binding) for binding in request.capability_bindings] + else: + bindings = resolution.model_dump(mode="json")["matched_capabilities"] + + next_fire_at = compute_next_fire_at(trigger, _utcnow(), 0) + runtime_snapshot = { + **resolution.agent_snapshot, + "display_name": ( + resolution.agent_snapshot.get("display_name") + or resolution.agent_snapshot.get("name") + or f"Agent #{request.agent_id}" + ), + "model_id": request.model_id, + "tool_params": request.tool_params, + "original_instruction": request.original_instruction or request.instruction, + } + try: + task = agent_automation_db.create_task({ + "tenant_id": tenant_id, + "user_id": user_id, + "conversation_id": conversation_id, + "agent_id": request.agent_id, + "agent_version_no": request.agent_version_no, + "title": request.title, + "instruction": request.instruction, + "status": AutomationTaskStatus.ACTIVE.value, + "source": AutomationSource.CHAT_INTENT.value, + "schedule_mode": trigger.mode.value, + "schedule_rule_type": trigger.rule_type.value, + "schedule_expr": trigger.cron_expr or str(trigger.interval_seconds or trigger.start_at), + "schedule_config": trigger.model_dump(mode="json"), + "capability_requirements": resolution.model_dump(mode="json"), + "capability_bindings": bindings, + "runtime_snapshot": runtime_snapshot, + "timezone": trigger.timezone, + "next_fire_at": next_fire_at, + "fire_count": 0, + "consecutive_failures": 0, + "timeout_seconds": request.timeout_seconds or AGENT_AUTOMATION_DEFAULT_TIMEOUT_SECONDS, + "overlap_policy": "SKIP", + "misfire_policy": "SKIP" if trigger.mode.value == "RECURRING" else "RUN_ONCE", + }, user_id) + except IntegrityError as exc: + constraint_name = getattr(getattr(exc.orig, "diag", None), "constraint_name", None) + if constraint_name == "uq_agent_automation_conversation_active": + raise AutomationConversationAlreadyBoundError( + "Conversation already has an active automation task." + ) from exc + raise + return _enrich_task_with_agent_name(task, tenant_id, user_id) + + def list_tasks( + self, + tenant_id: str, + user_id: str, + status: Optional[str] = None, + search: Optional[str] = None, + agent_name: Optional[str] = None, + page: Optional[int] = None, + page_size: Optional[int] = None, + ) -> Dict[str, Any] | List[Dict[str, Any]]: + if status == "ENABLED": + db_status = "ACTIVE" + elif status == "RUNNING": + db_status = None + else: + db_status = status + needs_post_filter = status in {"ENABLED", "RUNNING"} or bool((agent_name or "").strip()) + if page is not None and page_size is not None and not needs_post_filter: + paged = agent_automation_db.list_tasks_paginated( + tenant_id, + user_id, + db_status, + search, + page, + page_size, + ) + return { + **paged, + "items": _enrich_tasks_with_agent_names(paged["items"], tenant_id, user_id), + } + + tasks = agent_automation_db.list_tasks(tenant_id, user_id, db_status, search) + enriched_tasks = _enrich_tasks_with_agent_names(tasks, tenant_id, user_id) + if status == "ENABLED": + enriched_tasks = [task for task in enriched_tasks if not task["is_running"]] + elif status == "RUNNING": + enriched_tasks = [task for task in enriched_tasks if task["is_running"]] + normalized_agent_name = (agent_name or "").strip().casefold() + if normalized_agent_name: + enriched_tasks = [ + task for task in enriched_tasks + if normalized_agent_name in str(task.get("agent_name") or "").casefold() + ] + if page is not None and page_size is not None: + return _paginate(enriched_tasks, page, page_size) + return enriched_tasks + + def get_task(self, task_id: int, tenant_id: str, user_id: str) -> Dict[str, Any]: + task = agent_automation_db.get_task(task_id, tenant_id, user_id) + if not task: + raise AutomationNotFoundError("Automation task not found.") + return _enrich_task_with_agent_name(task, tenant_id, user_id) + + def get_task_for_conversation( + self, + conversation_id: int, + tenant_id: str, + user_id: str, + ) -> Optional[Dict[str, Any]]: + task = agent_automation_db.get_task_by_conversation(conversation_id, user_id) + return _enrich_task_with_agent_name(task, tenant_id, user_id) if task else None + + async def patch_task( + self, + task_id: int, + request: AutomationTaskPatchRequest, + tenant_id: str, + user_id: str, + ) -> Dict[str, Any]: + task = self.get_task(task_id, tenant_id, user_id) + values: Dict[str, Any] = {} + instruction = request.instruction or task["instruction"] + if request.title is not None: + values["title"] = request.title + if request.instruction is not None: + values["instruction"] = request.instruction + if request.timeout_seconds is not None: + values["timeout_seconds"] = request.timeout_seconds + if request.model_id is not None or request.tool_params is not None: + snapshot = dict(task.get("runtime_snapshot") or {}) + if request.model_id is not None: + snapshot["model_id"] = request.model_id + if request.tool_params is not None: + snapshot["tool_params"] = request.tool_params + values["runtime_snapshot"] = snapshot + if request.schedule_trigger is not None: + trigger = request.schedule_trigger + _validate_schedule_policy(trigger) + values.update({ + "schedule_mode": trigger.mode.value, + "schedule_rule_type": trigger.rule_type.value, + "schedule_expr": trigger.cron_expr or str(trigger.interval_seconds or trigger.start_at), + "schedule_config": trigger.model_dump(mode="json"), + "timezone": trigger.timezone, + "next_fire_at": compute_next_fire_at(trigger, _utcnow(), int(task.get("fire_count") or 0)), + }) + if request.instruction is not None or request.capability_bindings is not None: + resolution = await resolve_agent_capabilities( + task["agent_id"], tenant_id, user_id, instruction, task.get("agent_version_no") or 0) + if not resolution.executable: + raise AutomationCapabilityNotReadyError( + "Required automation capabilities are not ready.", + details=resolution.model_dump(mode="json"), + ) + values["capability_requirements"] = resolution.model_dump(mode="json") + values["capability_bindings"] = ( + _json(request.capability_bindings) + if request.capability_bindings + else resolution.model_dump(mode="json")["matched_capabilities"] + ) + snapshot = dict(values.get("runtime_snapshot") or task.get("runtime_snapshot") or {}) + snapshot.update(resolution.agent_snapshot) + if request.instruction is not None: + snapshot["original_instruction"] = request.instruction + values["runtime_snapshot"] = snapshot + updated = agent_automation_db.update_task(task_id, tenant_id, user_id, values) + if not updated: + raise AutomationNotFoundError("Automation task not found.") + return _enrich_task_with_agent_name(updated, tenant_id, user_id) + + def pause_task(self, task_id: int, tenant_id: str, user_id: str) -> Dict[str, Any]: + task = agent_automation_db.update_task( + task_id, + tenant_id, + user_id, + {"status": AutomationTaskStatus.PAUSED.value}, + ) + if not task: + raise AutomationNotFoundError("Automation task not found.") + return _enrich_task_with_agent_name(task, tenant_id, user_id) + + def resume_task(self, task_id: int, tenant_id: str, user_id: str) -> Dict[str, Any]: + task = self.get_task(task_id, tenant_id, user_id) + trigger = _parse_trigger(task["schedule_config"]) + next_fire_at = compute_next_fire_at(trigger, _utcnow(), int(task.get("fire_count") or 0)) + if next_fire_at is None: + raise AutomationScheduleInvalidError("Automation task has no future fire time.") + updated = agent_automation_db.update_task(task_id, tenant_id, user_id, { + "status": AutomationTaskStatus.ACTIVE.value, + "next_fire_at": next_fire_at, + }) + if not updated: + raise AutomationNotFoundError("Automation task not found.") + return _enrich_task_with_agent_name(updated, tenant_id, user_id) + + def delete_task(self, task_id: int, tenant_id: str, user_id: str) -> bool: + task = self.get_task(task_id, tenant_id, user_id) + self._cancel_active_runs_for_conversation( + task["conversation_id"], + user_id, + "Automation task was deleted.", + ) + if not agent_automation_db.soft_delete_task(task_id, tenant_id, user_id): + raise AutomationNotFoundError("Automation task not found.") + return True + + def list_runs( + self, + task_id: int, + tenant_id: str, + user_id: str, + page: Optional[int] = None, + page_size: Optional[int] = None, + ) -> Dict[str, Any] | List[Dict[str, Any]]: + self.get_task(task_id, tenant_id, user_id) + if page is not None and page_size is not None: + return agent_automation_db.list_runs_paginated(task_id, tenant_id, user_id, page, page_size) + return agent_automation_db.list_runs(task_id, tenant_id, user_id) + + async def run_task_now(self, task_id: int, tenant_id: str, user_id: str) -> Dict[str, Any]: + task = self.get_task(task_id, tenant_id, user_id) + from .runner import agent_automation_runner + return await agent_automation_runner.execute_task(task, trigger_type="MANUAL") + + def cancel_run(self, run_id: int, tenant_id: str, user_id: str) -> Dict[str, Any]: + run = agent_automation_db.get_run(run_id, tenant_id, user_id) + if not run: + raise AutomationNotFoundError("Automation run not found.") + + if run["status"] not in {AutomationRunStatus.QUEUED.value, AutomationRunStatus.RUNNING.value}: + return run + + self._request_conversation_stop(run["conversation_id"], user_id) + canceled = agent_automation_db.cancel_run( + run_id, + tenant_id, + user_id, + "Automation run was canceled by user.", + ) + agent_automation_db.update_task(run["task_id"], tenant_id, user_id, { + "last_run_status": AutomationRunStatus.CANCELED.value, + "last_error": "Automation run was canceled by user.", + "lock_owner": None, + "lock_until": None, + }) + return canceled or agent_automation_db.get_run(run_id, tenant_id, user_id) or run + + def delete_run(self, run_id: int, tenant_id: str, user_id: str) -> bool: + run = agent_automation_db.get_run(run_id, tenant_id, user_id) + if not run: + raise AutomationNotFoundError("Automation run not found.") + + terminal_statuses = { + AutomationRunStatus.SUCCEEDED.value, + AutomationRunStatus.FAILED.value, + AutomationRunStatus.SKIPPED.value, + AutomationRunStatus.CANCELED.value, + AutomationRunStatus.TIMEOUT.value, + } + if run["status"] not in terminal_statuses: + raise AutomationScheduleInvalidError( + "Active automation runs must be canceled before deletion." + ) + + deleted = agent_automation_db.soft_delete_run( + run_id, + tenant_id, + user_id, + list(terminal_statuses), + ) + if not deleted: + raise AutomationNotFoundError("Automation run not found or no longer deletable.") + + remaining_runs = agent_automation_db.list_runs( + run["task_id"], + tenant_id, + user_id, + limit=1, + ) + latest_run = remaining_runs[0] if remaining_runs else None + agent_automation_db.update_task( + run["task_id"], + tenant_id, + user_id, + { + "last_run_status": latest_run.get("status") if latest_run else None, + "last_error": latest_run.get("error_message") if latest_run else None, + }, + ) + return True + + def on_conversation_deleted(self, conversation_id: int, user_id: str) -> int: + self._cancel_active_runs_for_conversation( + conversation_id, + user_id, + "Conversation was deleted.", + ) + return agent_automation_db.soft_delete_task_by_conversation(conversation_id, user_id) + + def _cancel_active_runs_for_conversation(self, conversation_id: int, user_id: str, reason: str) -> None: + self._request_conversation_stop(conversation_id, user_id) + agent_automation_db.cancel_runs_by_conversation(conversation_id, user_id, reason) + + def _request_conversation_stop(self, conversation_id: int, user_id: str) -> None: + try: + from .runner import agent_automation_runner + agent_automation_runner.cancel_for_conversation(conversation_id, user_id) + except Exception: + pass + + +agent_automation_facade = AgentAutomationFacade() diff --git a/backend/services/agent_automation/intent_analyzer.py b/backend/services/agent_automation/intent_analyzer.py new file mode 100644 index 0000000000..d3f3afba41 --- /dev/null +++ b/backend/services/agent_automation/intent_analyzer.py @@ -0,0 +1,345 @@ +import asyncio +import json +import logging +import re +from abc import ABC, abstractmethod +from dataclasses import dataclass +from datetime import datetime, timedelta +from typing import Any, Dict, List, Optional +from zoneinfo import ZoneInfo + +from jinja2 import StrictUndefined, Template +from pydantic import BaseModel, ConfigDict, Field, ValidationError + +from consts.const import ( + AGENT_AUTOMATION_MIN_INTERVAL_SECONDS, + MESSAGE_ROLE, + MODEL_CONFIG_MAPPING, +) +from database.model_management_db import get_model_by_model_id +from utils.prompt_template_utils import get_prompt_template + +from .intent_parser import has_automation_schedule_signal, parse_automation_intent +from .models import ScheduleMode, ScheduleRuleType, ScheduleTrigger +from .prompt_generator import ( + AutomationTaskContent, + _fallback_title, + _normalize_task_content, + detect_instruction_language, +) +from .schedule_engine import is_valid_cron_expression + +logger = logging.getLogger("agent_automation.intent_analyzer") + + +class _LLMSchedulePayload(BaseModel): + model_config = ConfigDict(extra="forbid") + + rule_type: ScheduleRuleType + timezone: Optional[str] = None + cron_expr: Optional[str] = None + interval_seconds: Optional[int] = Field(default=None, gt=0) + start_at: Optional[datetime] = None + end_at: Optional[datetime] = None + max_fire_count: Optional[int] = Field(default=None, gt=0) + + +class _LLMIntentPayload(BaseModel): + model_config = ConfigDict(extra="forbid") + + is_automation_intent: bool + confidence: float = Field(ge=0, le=1) + title: str = "" + instruction: str = "" + schedule: Optional[_LLMSchedulePayload] = None + schedule_error: Optional[str] = None + missing_fields: List[str] = Field(default_factory=list) + clarification_question: Optional[str] = None + + +@dataclass(frozen=True) +class AutomationIntentContext: + tenant_id: str + message: str + timezone: str = "Asia/Shanghai" + model_id: Optional[int] = None + reference_time: Optional[datetime] = None + force_llm: bool = False + + +def _analysis_time(context: AutomationIntentContext) -> datetime: + try: + zone = ZoneInfo(context.timezone) + except Exception as exc: + raise ValueError(f"Invalid automation timezone: {context.timezone}") from exc + now = context.reference_time or datetime.now(zone) + return now.astimezone(zone) if now.tzinfo else now.replace(tzinfo=zone) + + +def _extract_json_object(content: str) -> Dict[str, Any]: + normalized = re.sub(r"[\s\S]*?", "", content or "", flags=re.IGNORECASE).strip() + fence_match = re.search(r"```(?:json)?\s*([\s\S]*?)```", normalized, flags=re.IGNORECASE) + if fence_match: + normalized = fence_match.group(1).strip() + try: + parsed = json.loads(normalized) + except json.JSONDecodeError: + object_match = re.search(r"\{[\s\S]*\}", normalized) + if not object_match: + raise + parsed = json.loads(object_match.group(0)) + if not isinstance(parsed, dict): + raise ValueError("Automation intent analysis must be a JSON object.") + return parsed + + +def _localized_datetime(value: Optional[datetime], zone: ZoneInfo) -> Optional[datetime]: + if value is None: + return None + return value.astimezone(zone) if value.tzinfo else value.replace(tzinfo=zone) + + +def _invalid_llm_schedule(payload: _LLMIntentPayload, reason: str) -> Dict[str, Any]: + return { + "is_automation_intent": True, + "confidence": payload.confidence, + "title": payload.title.strip(), + "instruction": payload.instruction.strip(), + "schedule_trigger": None, + "schedule_error": reason, + "capability_intents": [], + "output_requirements": {}, + "analysis_source": "llm", + "task_content_generated": True, + "missing_fields": payload.missing_fields, + "clarification_question": payload.clarification_question or reason, + } + + +def _payload_to_result( + payload: _LLMIntentPayload, + context: AutomationIntentContext, + fallback: Dict[str, Any], +) -> Dict[str, Any]: + if not payload.is_automation_intent: + return { + "is_automation_intent": False, + "confidence": payload.confidence, + "analysis_source": "llm", + } + + if not payload.instruction.strip(): + return _invalid_llm_schedule(payload, "无法确定自动任务需要执行的具体业务动作。") + + fallback_instruction = ( + fallback.get("instruction") + if fallback.get("is_automation_intent") + else payload.instruction.strip() + ) or context.message.strip() + fallback_content = AutomationTaskContent( + title=_fallback_title(fallback_instruction), + instruction=fallback_instruction, + ) + task_content = _normalize_task_content( + json.dumps( + {"title": payload.title, "instruction": payload.instruction}, + ensure_ascii=False, + ), + fallback_content, + source=fallback_instruction, + ) + task_content_source = "llm" + if task_content == fallback_content and ( + payload.title.strip() != fallback_content.title + or payload.instruction.strip() != fallback_content.instruction + ): + task_content_source = "rule" + if payload.schedule_error: + invalid = payload.model_copy(update={ + "title": task_content.title, + "instruction": task_content.instruction, + }) + return _invalid_llm_schedule(invalid, payload.schedule_error) + if payload.schedule is None: + invalid = payload.model_copy(update={ + "title": task_content.title, + "instruction": task_content.instruction, + }) + return _invalid_llm_schedule(invalid, "无法确定任务执行时间，请补充具体日期、时间或周期。") + + zone = ZoneInfo(payload.schedule.timezone or context.timezone) + now = _analysis_time(context).astimezone(zone) + schedule = payload.schedule + start_at = _localized_datetime(schedule.start_at, zone) + end_at = _localized_datetime(schedule.end_at, zone) + + if schedule.rule_type == ScheduleRuleType.AT: + if start_at is None: + return _invalid_llm_schedule(payload, "一次性任务缺少明确的未来执行时间。") + trigger = ScheduleTrigger( + mode=ScheduleMode.ONCE, + rule_type=ScheduleRuleType.AT, + timezone=zone.key, + start_at=start_at, + end_at=end_at, + ) + elif schedule.rule_type == ScheduleRuleType.INTERVAL: + if schedule.interval_seconds is None: + return _invalid_llm_schedule(payload, "周期任务缺少有效的执行间隔。") + if schedule.interval_seconds < AGENT_AUTOMATION_MIN_INTERVAL_SECONDS: + return _invalid_llm_schedule( + payload, + f"任务执行间隔不能小于 {AGENT_AUTOMATION_MIN_INTERVAL_SECONDS} 秒。", + ) + trigger = ScheduleTrigger( + mode=ScheduleMode.RECURRING, + rule_type=ScheduleRuleType.INTERVAL, + timezone=zone.key, + start_at=start_at or now.replace(microsecond=0) + timedelta(seconds=schedule.interval_seconds), + end_at=end_at, + interval_seconds=schedule.interval_seconds, + max_fire_count=schedule.max_fire_count, + ) + else: + if not schedule.cron_expr or not is_valid_cron_expression(schedule.cron_expr): + return _invalid_llm_schedule(payload, "大模型生成的 Cron 表达式无效，请补充或修改执行周期。") + trigger = ScheduleTrigger( + mode=ScheduleMode.RECURRING, + rule_type=ScheduleRuleType.CRON, + timezone=zone.key, + start_at=start_at or now.replace(second=0, microsecond=0), + end_at=end_at, + cron_expr=schedule.cron_expr, + max_fire_count=schedule.max_fire_count, + ) + + return { + "is_automation_intent": True, + "confidence": payload.confidence, + "title": task_content.title, + "instruction": task_content.instruction, + "schedule_trigger": trigger, + "schedule_error": None, + "capability_intents": [], + "output_requirements": {}, + "analysis_source": "llm", + "task_content_generated": True, + "task_content_source": task_content_source, + "missing_fields": [], + "clarification_question": None, + } + + +class AutomationIntentAnalysisStrategy(ABC): + @abstractmethod + async def analyze(self, context: AutomationIntentContext) -> Dict[str, Any]: + raise NotImplementedError + + +class RuleBasedAutomationIntentStrategy(AutomationIntentAnalysisStrategy): + async def analyze(self, context: AutomationIntentContext) -> Dict[str, Any]: + result = parse_automation_intent( + context.message, + context.timezone, + context.tenant_id, + context.reference_time, + ) + return { + **result, + "analysis_source": "rule", + "task_content_generated": False, + } + + +class LLMAutomationIntentStrategy(AutomationIntentAnalysisStrategy): + def __init__(self, model_config: Dict[str, Any], fallback: AutomationIntentAnalysisStrategy): + self._model_config = model_config + self._fallback = fallback + + async def analyze(self, context: AutomationIntentContext) -> Dict[str, Any]: + fallback = await self._fallback.analyze(context) + if not context.force_llm and not has_automation_schedule_signal(context.message): + return fallback + try: + content = await asyncio.to_thread(self._generate_sync, context) + payload = _LLMIntentPayload.model_validate(_extract_json_object(content)) + return _payload_to_result(payload, context, fallback) + except (ValidationError, ValueError, KeyError, json.JSONDecodeError) as exc: + logger.warning("Invalid LLM automation intent output, using rule fallback: %s", exc) + return fallback + except Exception as exc: + logger.warning("Failed to analyze automation intent with LLM, using rule fallback: %s", exc) + return fallback + + def _generate_sync(self, context: AutomationIntentContext) -> str: + from nexent.core.models import OpenAIModel + from utils.config_utils import get_model_name_from_config + + language = detect_instruction_language(context.message) + prompt_template = get_prompt_template("agent_automation", language) + now = _analysis_time(context) + values = { + "message": context.message.strip(), + "current_datetime": now.isoformat(), + "timezone": context.timezone, + "min_interval_seconds": AGENT_AUTOMATION_MIN_INTERVAL_SECONDS, + } + user_prompt = Template( + prompt_template["INTENT_ANALYSIS_USER_PROMPT"], + undefined=StrictUndefined, + ).render(**values).strip() + llm = OpenAIModel( + model_id=get_model_name_from_config(self._model_config), + api_base=self._model_config.get("base_url", ""), + api_key=self._model_config.get("api_key", ""), + temperature=0.1, + top_p=0.9, + max_output_tokens=700, + model_factory=self._model_config.get("model_factory"), + ssl_verify=self._model_config.get("ssl_verify", True), + display_name=self._model_config.get("display_name"), + timeout_seconds=self._model_config.get("timeout_seconds"), + stream=False, + ) + response = llm.generate([ + { + "role": MESSAGE_ROLE["SYSTEM"], + "content": prompt_template["INTENT_ANALYSIS_SYSTEM_PROMPT"], + }, + {"role": MESSAGE_ROLE["USER"], "content": user_prompt}, + ]) + return getattr(response, "content", "") or "" + + +class AutomationIntentStrategyFactory: + def create(self, context: AutomationIntentContext) -> AutomationIntentAnalysisStrategy: + fallback = RuleBasedAutomationIntentStrategy() + try: + model_config = None + if context.model_id is not None: + selected = get_model_by_model_id(context.model_id, context.tenant_id) + if selected and selected.get("model_type") == "llm": + model_config = selected + if model_config is None: + from utils.config_utils import tenant_config_manager + + model_config = tenant_config_manager.get_model_config( + key=MODEL_CONFIG_MAPPING["llm"], + tenant_id=context.tenant_id, + ) + if model_config: + return LLMAutomationIntentStrategy(model_config, fallback) + except Exception as exc: + logger.warning("Failed to resolve automation intent model, using rule fallback: %s", exc) + return fallback + + +class AutomationIntentAnalyzer: + def __init__(self, factory: Optional[AutomationIntentStrategyFactory] = None): + self._factory = factory or AutomationIntentStrategyFactory() + + async def analyze(self, context: AutomationIntentContext) -> Dict[str, Any]: + return await self._factory.create(context).analyze(context) + + +automation_intent_analyzer = AutomationIntentAnalyzer() diff --git a/backend/services/agent_automation/intent_parser.py b/backend/services/agent_automation/intent_parser.py new file mode 100644 index 0000000000..5894cc79fb --- /dev/null +++ b/backend/services/agent_automation/intent_parser.py @@ -0,0 +1,947 @@ +import re +from datetime import date, datetime, time, timedelta +from typing import Optional +from zoneinfo import ZoneInfo + +from .models import ScheduleMode, ScheduleRuleType, ScheduleTrigger + + +_NUMBER_TOKEN = r"(?:\d+|[零一二两三四五六七八九十百]+|半)" +_CLOCK_NUMBER_TOKEN = r"(?:\d{1,2}|[零一二两三四五六七八九十]+)" +_INTERVAL_PATTERN = re.compile( + rf"(?:每(?:隔\s*)?|隔\s*)(?P{_NUMBER_TOKEN})?\s*(?:个)?" + r"(?P秒钟?|分钟?|小时|钟头)" +) +_DAY_INTERVAL_PATTERN = re.compile( + rf"每(?:隔\s*)?(?P{_NUMBER_TOKEN})\s*(?:个)?(?P天|日|周|星期)" +) +_RELATIVE_DELAY_PATTERN = re.compile( + rf"(?P{_NUMBER_TOKEN})\s*(?:个)?(?P秒钟?|分钟?|小时|钟头|天|日|周|星期)" + r"\s*(?:以后|之后|后)" +) +_HOURLY_OFFSET_PATTERN = re.compile( + rf"每(?:个)?小时(?:的)?(?:(?:第)?(?P{_CLOCK_NUMBER_TOKEN})\s*分(?:钟)?|" + r"(?P整点|半点))" +) +_EXPLICIT_DATE_PATTERN = re.compile( + r"(?:(?P\d{4})\s*年\s*)?(?P\d{1,2})\s*月\s*(?P\d{1,2})\s*(?:日|号)?" +) +_ISO_DATE_PATTERN = re.compile(r"(?P\d{4})[-/](?P\d{1,2})[-/](?P\d{1,2})") +_SHORT_DATE_PATTERN = re.compile(r"(?\d{1,2})/(?P\d{1,2})(?!\d)") +_RELATIVE_MONTH_DAY_PATTERN = re.compile( + r"(?P下个?月|本月|这个月)\s*(?P\d{1,2})\s*(?:日|号)?" +) +_DAY_OF_MONTH_ONCE_PATTERN = re.compile(r"(?\d{1,2})\s*(?:日|号)") +_WEEKDAY_PATTERN = re.compile( + r"(?:(?P下|本|这(?:个)?)(?:周|星期|礼拜)|(?:周|星期|礼拜))" + r"(?P[一二三四五六日天])" +) +_RECURRING_WEEKDAY_PATTERN = re.compile( + r"(?:每(?:个)?|每逢|逢)(?:周|星期|礼拜)(?P[一二三四五六日天]" + r"(?:\s*[、,，/和及]\s*(?:(?:周|星期|礼拜))?[一二三四五六日天])*)" +) +_RECURRING_WEEKDAY_RANGE_PATTERN = re.compile( + r"每(?:个)?(?:周|星期|礼拜)(?P[一二三四五六日天])\s*(?:到|至|[-~～])\s*" + r"(?:(?:周|星期|礼拜))?(?P[一二三四五六日天])" +) +_MONTH_DAY_LIST_PATTERN = re.compile( + r"每(?:个)?月\s*(?P\d{1,2}(?:\s*(?:号|日))?" + r"(?:\s*[、,，/和及]\s*\d{1,2}(?:\s*(?:号|日))?)*)" +) +_YEARLY_PATTERN = re.compile(r"每年(?:的)?\s*(?P\d{1,2})\s*月\s*(?P\d{1,2})\s*(?:日|号)?") +_MONTH_END_PATTERN = re.compile(r"(?:每(?:个)?月(?:的)?最后一天|每(?:个)?月末|每(?:个)?月底)") +_QUARTERLY_PATTERN = re.compile( + r"每(?:个)?季度(?:的)?(?:第)?(?P\d{1,2}|一)\s*(?:天|日|号)?" +) +_RECURRENCE_MARKER_PATTERN = re.compile( + r"(?:每(?:个)?(?:天|日|晚|周|星期|礼拜|月|年|季度)|每逢|逢(?:周|星期|礼拜))" +) +_UNSUPPORTED_RECURRENCE_PATTERN = re.compile( + rf"每(?:隔\s*)?(?P{_NUMBER_TOKEN})\s*(?:个)?(?P月|年)" +) +_EN_INTERVAL_PATTERN = re.compile( + r"\bevery\s+(?:(?P\d+)\s+)?(?Psecond|minute|hour)s?\b", + flags=re.IGNORECASE, +) +_EN_RELATIVE_DELAY_PATTERN = re.compile( + r"\bin\s+(?P\d+)\s+(?Pminute|hour|day)s?\b", + flags=re.IGNORECASE, +) +_EN_RECURRING_WEEKDAY_PATTERN = re.compile( + r"\bevery\s+(?:monday|tuesday|wednesday|thursday|friday|saturday|sunday)\b", + flags=re.IGNORECASE, +) +_EN_NEXT_WEEKDAY_PATTERN = re.compile( + r"\bnext\s+(?Pmonday|tuesday|wednesday|thursday|friday|saturday|sunday)\b", + flags=re.IGNORECASE, +) +_IANA_TIMEZONE_PATTERN = re.compile( + r"\b(?:Africa|America|Antarctica|Asia|Atlantic|Australia|Europe|Indian|Pacific)" + r"/[A-Za-z_+-]+(?:/[A-Za-z_+-]+)?\b" +) +_TIMEZONE_PATTERN = re.compile( + r"(?:北京时间|上海时间|中国时间|上海时区|中国时区|东京时间|日本时间|纽约时间|" + r"伦敦时间|洛杉矶时间|UTC\s*(?:时间|时区)|UTC\s*(?=\d{1,2}\s*(?:[:：点时])))", + re.IGNORECASE, +) + +_TIMEZONE_ALIASES = ( + (re.compile(r"(?:北京时间|上海时间|中国时间|上海时区|中国时区)"), "Asia/Shanghai"), + (re.compile(r"(?:东京时间|日本时间)"), "Asia/Tokyo"), + (re.compile(r"纽约时间"), "America/New_York"), + (re.compile(r"伦敦时间"), "Europe/London"), + (re.compile(r"洛杉矶时间"), "America/Los_Angeles"), + (re.compile(r"UTC\s*(?:(?:时间|时区)|(?=\d{1,2}\s*(?:[:：点时])))", re.IGNORECASE), "UTC"), +) + +_WEEKDAY_TO_CRON = {"一": 1, "二": 2, "三": 3, "四": 4, "五": 5, "六": 6, "日": 0, "天": 0} +_EN_WEEKDAY_TO_CRON = { + "sunday": 0, + "monday": 1, + "tuesday": 2, + "wednesday": 3, + "thursday": 4, + "friday": 5, + "saturday": 6, +} +_DURATION_SECONDS = { + "秒": 1, + "秒钟": 1, + "分": 60, + "分钟": 60, + "小时": 3600, + "钟头": 3600, + "天": 86400, + "日": 86400, + "周": 604800, + "星期": 604800, + "second": 1, + "minute": 60, + "hour": 3600, + "day": 86400, +} +_ACTION_TOKENS = ( + "提醒", + "发送", + "生成", + "汇总", + "总结", + "执行", + "运行", + "通知", + "检查", + "查询", + "整理", + "备份", + "同步", + "推送", + "发布", + "抓取", + "监控", + "扫描", + "清理", + "更新", + "导出", + "调用", + "统计", + "记录", + "计算", + "算一下", + "获取", + "查找", + "检索", + "搜索", + "读取", + "收集", + "采集", + "分析", + "处理", + "转换", + "翻译", + "创建", + "提交", + "保存", + "写入", + "上传", + "下载", + "告诉", + "说", +) +_AUTOMATION_TOKENS = ( + "定时", + "提醒", + "每天", + "每日", + "每晚", + "每周", + "每星期", + "每礼拜", + "每月", + "每年", + "工作日", + "周末", + "周期", + "every ", + "tomorrow", + "next ", + "明早", + "明晚", + "下个月", + "本月", + "每季度", + "定期", + "每当", + "每次", +) +_QUESTION_PATTERN = re.compile( + r"(?:多少|如何|怎么|怎么样|是什么|为何|为什么|是否|能否|可不可以|有没有|吗|呢|[？?])" +) +_EXPLICIT_AUTOMATION_PATTERN = re.compile( + r"(?:定时任务|自动任务|周期任务|计划任务|" + r"(?:创建|新建|添加|设置|设定|安排|建立|配置).{0,24}(?:任务|提醒|定时|自动|周期)|提醒我)" +) +_LEADING_REQUEST_PATTERN = re.compile( + r"^\s*(?:(?:请你|请帮我|请|麻烦你|麻烦|帮我|给我|为我|" + r"我希望你|我想让你|我要你|需要你)\s*)+" +) +_DECLARATIVE_ACTION_PATTERN = re.compile( + r"^(?:我(?:会|通常|一般|总是|习惯|都)|通常|一般|平时|习惯于)" +) + + +def _chinese_number(value: str) -> float: + if value.isdigit(): + return float(value) + if value == "半": + return 0.5 + digits = { + "零": 0, "一": 1, "二": 2, "两": 2, "三": 3, + "四": 4, "五": 5, "六": 6, "七": 7, "八": 8, "九": 9, + } + if value == "十": + return 10 + if "百" in value: + hundreds, remainder = value.split("百", 1) + return digits.get(hundreds, 1) * 100 + (_chinese_number(remainder) if remainder else 0) + if "十" in value: + tens, ones = value.split("十", 1) + return digits.get(tens, 1) * 10 + digits.get(ones, 0) + if len(value) == 1 and value in digits: + return digits[value] + raise ValueError(f"Unsupported Chinese number: {value}") + + +def _duration_seconds(count: str, unit: str) -> int: + seconds = int(_chinese_number(count) * _DURATION_SECONDS[unit.lower()]) + if seconds <= 0: + raise ValueError("Automation interval must be positive.") + return seconds + + +def _apply_period(hour: int, period: str) -> int: + normalized = period.lower() + if normalized in {"pm", "下午"} and hour < 12: + return hour + 12 + if normalized in {"晚上", "今晚"}: + if hour == 12: + return 0 + return hour + 12 if hour < 12 else hour + if normalized == "中午" and hour < 11: + return hour + 12 + if normalized in {"am", "上午", "早上", "凌晨", "午夜"} and hour == 12: + return 0 + return hour + + +def _validated_clock(hour: int, minute: int, period: str = "") -> time: + hour = _apply_period(hour, period) if period else hour + if hour > 23 or minute > 59: + raise ValueError("Invalid hour or minute in automation schedule.") + return time(hour, minute) + + +def _parse_clocks(message: str) -> list[time]: + clocks: list[time] = [] + chinese_period = r"(?:上午|早上|中午|下午|晚上|今晚|凌晨|午夜)" + + for match in re.finditer( + rf"(?P{chinese_period})?\s*(?P\d{{1,2}})\s*[:：]\s*(?P\d{{1,2}})", + message, + ): + clocks.append(_validated_clock( + int(match.group("hour")), + int(match.group("minute")), + match.group("period") or "", + )) + + for match in re.finditer( + rf"(?P{chinese_period})?\s*(?P{_CLOCK_NUMBER_TOKEN})\s*(?:点|时)" + rf"(?:(?P半)|(?P一刻|三刻)|(?P{_CLOCK_NUMBER_TOKEN})\s*分?)?", + message, + ): + minute = 0 + if match.group("half"): + minute = 30 + elif match.group("quarter"): + minute = 15 if match.group("quarter") == "一刻" else 45 + elif match.group("minute"): + minute = int(_chinese_number(match.group("minute"))) + clocks.append(_validated_clock( + int(_chinese_number(match.group("hour"))), + minute, + match.group("period") or "", + )) + + for match in re.finditer( + r"\bat\s+(?P\d{1,2})(?::(?P\d{2}))?\s*(?Pam|pm)?\b", + message, + flags=re.IGNORECASE, + ): + clocks.append(_validated_clock( + int(match.group("hour")), + int(match.group("minute") or 0), + match.group("period") or "", + )) + + if not clocks: + if "中午" in message: + clocks.append(time(12, 0)) + elif "午夜" in message: + clocks.append(time(0, 0)) + + return list(dict.fromkeys(clocks)) + + +def _parse_clock(message: str) -> Optional[time]: + clocks = _parse_clocks(message) + return clocks[0] if clocks else None + + +def _future_date(month: int, day: int, now: datetime, year: Optional[int] = None) -> date: + target_year = year or now.year + target = date(target_year, month, day) + if year is None and target < now.date(): + target = date(target_year + 1, month, day) + return target + + +def _relative_weekday(message: str, now: datetime) -> Optional[date]: + match = _WEEKDAY_PATTERN.search(message) + if not match or message[max(0, match.start() - 1):match.start()] == "每": + return None + target_weekday = (_WEEKDAY_TO_CRON[match.group("day")] - 1) % 7 + prefix = match.group("prefix") or "" + if prefix == "下": + days = 7 - now.weekday() + target_weekday + elif prefix.startswith(("本", "这")): + days = target_weekday - now.weekday() + else: + days = (target_weekday - now.weekday()) % 7 + return (now + timedelta(days=days)).date() + + +def _combine_local(target_date: date, target_time: time, zone: ZoneInfo) -> datetime: + naive = datetime.combine(target_date, target_time) + candidate = naive.replace(tzinfo=zone, fold=0) + round_trip = candidate.astimezone(ZoneInfo("UTC")).astimezone(zone).replace(tzinfo=None) + if round_trip != naive: + raise ValueError("The requested local time does not exist because of a timezone transition.") + alternate = naive.replace(tzinfo=zone, fold=1) + if candidate.utcoffset() != alternate.utcoffset(): + raise ValueError("The requested local time is ambiguous because of a timezone transition.") + return candidate + + +def _parse_weekday_values(raw: str) -> list[int]: + values = [_WEEKDAY_TO_CRON[token] for token in re.findall(r"[一二三四五六日天]", raw)] + return list(dict.fromkeys(values)) + + +def _parse_weekday_range(start: str, end: str) -> str: + start_value = _WEEKDAY_TO_CRON[start] + end_value = _WEEKDAY_TO_CRON[end] + ordered_week = [1, 2, 3, 4, 5, 6, 0] + start_index = ordered_week.index(start_value) + end_index = ordered_week.index(end_value) + if start_index <= end_index: + values = ordered_week[start_index:end_index + 1] + else: + values = ordered_week[start_index:] + ordered_week[:end_index + 1] + if len(values) == 1: + return str(values[0]) + if values == list(range(values[0], values[-1] + 1)): + return f"{values[0]}-{values[-1]}" + return ",".join(str(value) for value in values) + + +def _parse_month_days(raw: str) -> list[int]: + days = [int(value) for value in re.findall(r"\d{1,2}", raw)] + if any(day < 1 or day > 31 for day in days): + raise ValueError("Invalid day of month in automation schedule.") + return list(dict.fromkeys(days)) + + +def _cron_time_fields(clocks: list[time]) -> Optional[tuple[str, str]]: + if not clocks: + return None + minutes = sorted({clock.minute for clock in clocks}) + hours = sorted({clock.hour for clock in clocks}) + if len(minutes) == 1: + return str(minutes[0]), ",".join(str(hour) for hour in hours) + if len(hours) == 1: + return ",".join(str(minute) for minute in minutes), str(hours[0]) + return None + + +def _cron_for_clocks(clocks: list[time], suffix: str) -> Optional[str]: + fields = _cron_time_fields(clocks) + if fields is None: + return None + minute, hour = fields + return f"{minute} {hour} {suffix}" + + +def _next_month_day(relative_month: str, day: int, now: datetime) -> date: + month_offset = 1 if relative_month.startswith("下") else 0 + absolute_month = now.month + month_offset + year = now.year + (absolute_month - 1) // 12 + month = (absolute_month - 1) % 12 + 1 + return date(year, month, day) + + +def _next_day_of_month(day: int, now: datetime) -> date: + target = date(now.year, now.month, day) + if target >= now.date(): + return target + absolute_month = now.month + 1 + year = now.year + (absolute_month - 1) // 12 + month = (absolute_month - 1) % 12 + 1 + return date(year, month, day) + + +def _strip_leading_request(message: str) -> str: + candidate = _LEADING_REQUEST_PATTERN.sub("", message.strip()) + return re.sub(r"^在\s*", "", candidate) + + +def _schedule_leads_task(message: str) -> bool: + candidate = _strip_leading_request(message) + lower = candidate.lower() + schedule_patterns = ( + _HOURLY_OFFSET_PATTERN, + _INTERVAL_PATTERN, + _DAY_INTERVAL_PATTERN, + _RELATIVE_DELAY_PATTERN, + _QUARTERLY_PATTERN, + _MONTH_END_PATTERN, + _YEARLY_PATTERN, + _MONTH_DAY_LIST_PATTERN, + _RECURRING_WEEKDAY_RANGE_PATTERN, + _RECURRING_WEEKDAY_PATTERN, + _RELATIVE_MONTH_DAY_PATTERN, + _EXPLICIT_DATE_PATTERN, + _ISO_DATE_PATTERN, + _SHORT_DATE_PATTERN, + _DAY_OF_MONTH_ONCE_PATTERN, + _WEEKDAY_PATTERN, + _EN_INTERVAL_PATTERN, + _EN_RELATIVE_DELAY_PATTERN, + _EN_RECURRING_WEEKDAY_PATTERN, + _EN_NEXT_WEEKDAY_PATTERN, + _UNSUPPORTED_RECURRENCE_PATTERN, + _RECURRENCE_MARKER_PATTERN, + ) + if any(pattern.match(candidate) for pattern in schedule_patterns): + return True + return candidate.startswith( + ("今天", "明天", "后天", "今晚", "明早", "明晚", "工作日", "周末", "每个工作日") + ) or lower.startswith(("today", "tomorrow", "next ", "every ", "in ")) + + +def has_automation_schedule_signal(message: str) -> bool: + """Return whether a message has enough temporal context to merit semantic analysis.""" + lower = message.lower() + return bool( + any(token.lower() in lower for token in _AUTOMATION_TOKENS) + or _HOURLY_OFFSET_PATTERN.search(message) + or _INTERVAL_PATTERN.search(message) + or _DAY_INTERVAL_PATTERN.search(message) + or _RELATIVE_DELAY_PATTERN.search(message) + or _EN_INTERVAL_PATTERN.search(message) + or _RECURRENCE_MARKER_PATTERN.search(message) + or _UNSUPPORTED_RECURRENCE_PATTERN.search(message) + or _EXPLICIT_DATE_PATTERN.search(message) + or _ISO_DATE_PATTERN.search(message) + or _SHORT_DATE_PATTERN.search(message) + or _RELATIVE_MONTH_DAY_PATTERN.search(message) + or _DAY_OF_MONTH_ONCE_PATTERN.search(message) + or _WEEKDAY_PATTERN.search(message) + or _EN_NEXT_WEEKDAY_PATTERN.search(message) + or _EN_RELATIVE_DELAY_PATTERN.search(message) + or _parse_clocks(message) + or any(token in message for token in ("今天", "明天", "后天", "今晚", "明早", "明晚", "稍后", "待会")) + or re.search(r"\b(?:today|tomorrow|daily|weekly|monthly|yearly|schedule|scheduled)\b", lower) + ) + + +def _looks_like_automation(message: str) -> bool: + lower = message.lower() + has_action = ( + any(token in message for token in _ACTION_TOKENS) + or bool(re.search(r"发(?!现|生|布|起|挥|明|热)", message)) + or bool( + re.search( + r"\b(?:send|remind|generate|summarize|check|notify|run|execute|create|" + r"calculate|fetch|get|retrieve|find|search|analyze|process|save|upload|download)\b", + lower, + ) + ) + ) + question_like = bool(_QUESTION_PATTERN.search(message)) + explicit_request = bool( + re.search( + r"(?:提醒我|告诉我|帮我|给我|为我|麻烦你|请你|请帮我|需要你|我要你|我希望你|我想让你)", + message, + ) + or re.search(r"\b(?:please|remind me|tell me)\b", lower) + ) + has_action_command = has_action and (not question_like or explicit_request) + if not has_automation_schedule_signal(message): + return False + + explicit_automation = bool(_EXPLICIT_AUTOMATION_PATTERN.search(message)) + schedule_leads_task = _schedule_leads_task(message) + business_action = _extract_business_action(message) + + if question_like: + return explicit_automation or (schedule_leads_task and explicit_request and has_action) + if explicit_automation: + return True + if not schedule_leads_task or not business_action: + return False + if _DECLARATIVE_ACTION_PATTERN.search(business_action): + return False + return has_action_command or bool(business_action) + + +def _resolve_timezone(message: str, default_timezone: str) -> str: + for pattern, timezone_name in _TIMEZONE_ALIASES: + if pattern.search(message): + return timezone_name + if match := _IANA_TIMEZONE_PATTERN.search(message): + return match.group(0) + return default_timezone + + +def _normalize_schedule_phrases(message: str) -> str: + replacements = ( + ("明早", "明天早上"), + ("明晚", "明天晚上"), + ("今早", "今天早上"), + ("每晚", "每天晚上"), + ("每早", "每天早上"), + ) + normalized = message + for source, target in replacements: + normalized = normalized.replace(source, target) + return normalized + + +def _extract_business_action(message: str) -> str: + cleaned = message + schedule_patterns = ( + _HOURLY_OFFSET_PATTERN, + _INTERVAL_PATTERN, + _DAY_INTERVAL_PATTERN, + _RELATIVE_DELAY_PATTERN, + _QUARTERLY_PATTERN, + _MONTH_END_PATTERN, + _YEARLY_PATTERN, + _MONTH_DAY_LIST_PATTERN, + _RECURRING_WEEKDAY_RANGE_PATTERN, + _RECURRING_WEEKDAY_PATTERN, + _RELATIVE_MONTH_DAY_PATTERN, + _EXPLICIT_DATE_PATTERN, + _ISO_DATE_PATTERN, + _SHORT_DATE_PATTERN, + _DAY_OF_MONTH_ONCE_PATTERN, + _WEEKDAY_PATTERN, + _EN_INTERVAL_PATTERN, + _EN_RELATIVE_DELAY_PATTERN, + _EN_RECURRING_WEEKDAY_PATTERN, + _EN_NEXT_WEEKDAY_PATTERN, + _TIMEZONE_PATTERN, + _IANA_TIMEZONE_PATTERN, + _UNSUPPORTED_RECURRENCE_PATTERN, + ) + for pattern in schedule_patterns: + cleaned = pattern.sub("", cleaned) + cleaned = re.sub( + r"(?:每天|每日|每晚|每个工作日|工作日|周一到周五|周末|" + r"今天|明天|后天|今晚|明早|明晚)", + "", + cleaned, + ) + cleaned = re.sub(r"(?:上午|早上|中午|下午|晚上|凌晨|午夜)", "", cleaned) + cleaned = re.sub( + rf"{_CLOCK_NUMBER_TOKEN}\s*(?:" + rf"[:：]\s*{_CLOCK_NUMBER_TOKEN}|" + rf"(?:点|时)(?:(?:半|一刻|三刻)|(?:{_CLOCK_NUMBER_TOKEN})\s*分?)?" + rf")", + "", + cleaned, + ) + cleaned = re.sub(r"\bat\s+\d{1,2}(?::\d{2})?\s*(?:am|pm)?\b", "", cleaned, flags=re.IGNORECASE) + cleaned = re.sub( + r"\b(?:every\s+(?:day|weekday|weekend)|weekdays?|weekends?|" + r"tomorrow(?!['’]s)|today(?!['’]s))\b", + "", + cleaned, + flags=re.IGNORECASE, + ) + cleaned = _strip_leading_request(cleaned) + if re.search(r"(?:创建|新建|添加|设置|设定|安排|建立|配置).*任务", message): + cleaned = re.sub( + r"^(?:创建|新建|添加|设置|设定|安排|建立|配置)(?:一个|一条|个)?\s*", + "", + cleaned, + ) + cleaned = re.sub(r"\s*的?(?:(?:定时|自动|周期|计划)\s*)?任务$", "", cleaned) + cleaned = re.sub(r"^(?:定时执行|定时)\s*", "", cleaned) + cleaned = re.sub(r"^的\s*", "", cleaned) + cleaned = re.sub(r"^(?:[、,，/]|和|及)+\s*", "", cleaned) + cleaned = re.sub(r"\s+", " ", cleaned).strip(" ，,。；;：:") + return cleaned + + +def _clean_action(message: str) -> str: + return _extract_business_action(message) or message.strip() + + +def _invalid_schedule(message: str, reason: str, confidence: float = 0.9) -> dict: + return { + "is_automation_intent": True, + "confidence": confidence, + "title": "", + "instruction": _clean_action(message), + "schedule_trigger": None, + "schedule_error": reason, + "capability_intents": [], + "output_requirements": {}, + } + + +def _success(message: str, trigger: ScheduleTrigger, confidence: float = 0.96) -> dict: + instruction = _clean_action(message) + return { + "is_automation_intent": True, + "confidence": confidence, + "title": instruction[:30] or "自动任务", + "instruction": instruction, + "schedule_trigger": trigger, + "schedule_error": None, + "capability_intents": [], + "output_requirements": {}, + } + + +def _cron_trigger(now: datetime, timezone_name: str, expression: str) -> ScheduleTrigger: + return ScheduleTrigger( + mode=ScheduleMode.RECURRING, + rule_type=ScheduleRuleType.CRON, + timezone=timezone_name, + start_at=now.replace(second=0, microsecond=0), + cron_expr=expression, + ) + + +def _recurring_cron_result( + message: str, + now: datetime, + timezone_name: str, + clocks: list[time], + suffix: str, + missing_time_error: str, +) -> dict: + if not clocks: + return _invalid_schedule(message, missing_time_error) + expression = _cron_for_clocks(clocks, suffix) + if expression is None: + return _invalid_schedule( + message, + "一个任务中的多个执行时刻必须具有相同的小时或分钟，请拆分为多个任务。", + ) + return _success(message, _cron_trigger(now, timezone_name, expression)) + + +def parse_automation_intent( + message: str, + timezone_name: str = "Asia/Shanghai", + tenant_id: str | None = None, + reference_time: Optional[datetime] = None, +) -> dict: + """Parse natural-language scheduling independently from task prompt generation.""" + del tenant_id + message = _normalize_schedule_phrases(message) + timezone_name = _resolve_timezone(message, timezone_name) + try: + zone = ZoneInfo(timezone_name) + except Exception as exc: + raise ValueError(f"Invalid automation timezone: {timezone_name}") from exc + now = reference_time or datetime.now(zone) + now = now.astimezone(zone) if now.tzinfo else now.replace(tzinfo=zone) + + if not _looks_like_automation(message): + return {"is_automation_intent": False, "confidence": 0.0} + + hourly_offset = _HOURLY_OFFSET_PATTERN.search(message) + if hourly_offset: + if hourly_offset.group("marker") == "整点": + minute = 0 + elif hourly_offset.group("marker") == "半点": + minute = 30 + else: + minute = int(_chinese_number(hourly_offset.group("minute"))) + if minute > 59: + return _invalid_schedule(message, "每小时执行的分钟数必须在 0 到 59 之间。") + return _success(message, _cron_trigger(now, timezone_name, f"{minute} * * * *")) + + interval_match = _INTERVAL_PATTERN.search(message) or _DAY_INTERVAL_PATTERN.search(message) + if interval_match: + count = interval_match.group("count") or "一" + unit = interval_match.group("unit") if "unit" in interval_match.groupdict() else "天" + seconds = _duration_seconds(count, unit) + trigger = ScheduleTrigger( + mode=ScheduleMode.RECURRING, + rule_type=ScheduleRuleType.INTERVAL, + timezone=timezone_name, + start_at=now.replace(microsecond=0) + timedelta(seconds=seconds), + interval_seconds=seconds, + ) + return _success(message, trigger) + + en_interval = _EN_INTERVAL_PATTERN.search(message) + if en_interval: + seconds = _duration_seconds(en_interval.group("count") or "1", en_interval.group("unit")) + trigger = ScheduleTrigger( + mode=ScheduleMode.RECURRING, + rule_type=ScheduleRuleType.INTERVAL, + timezone=timezone_name, + start_at=now.replace(microsecond=0) + timedelta(seconds=seconds), + interval_seconds=seconds, + ) + return _success(message, trigger) + + delay_match = _RELATIVE_DELAY_PATTERN.search(message) + if delay_match: + seconds = _duration_seconds(delay_match.group("count"), delay_match.group("unit")) + trigger = ScheduleTrigger( + mode=ScheduleMode.ONCE, + rule_type=ScheduleRuleType.AT, + timezone=timezone_name, + start_at=now + timedelta(seconds=seconds), + ) + return _success(message, trigger) + + en_delay = _EN_RELATIVE_DELAY_PATTERN.search(message) + if en_delay: + seconds = _duration_seconds(en_delay.group("count"), en_delay.group("unit")) + trigger = ScheduleTrigger( + mode=ScheduleMode.ONCE, + rule_type=ScheduleRuleType.AT, + timezone=timezone_name, + start_at=now + timedelta(seconds=seconds), + ) + return _success(message, trigger) + + lower = message.lower() + target_clocks = _parse_clocks(message) + target_time = target_clocks[0] if target_clocks else None + if any(token in message for token in ("每天", "每日")) or "every day" in lower: + return _recurring_cron_result( + message, now, timezone_name, target_clocks, "* * *", "请明确每天执行的具体时间。" + ) + + if any(token in message for token in ("工作日", "周一到周五", "每个工作日")): + return _recurring_cron_result( + message, now, timezone_name, target_clocks, "* * 1-5", "请明确工作日执行的具体时间。" + ) + + if re.search(r"\bevery\s+weekday\b", lower): + return _recurring_cron_result( + message, + now, + timezone_name, + target_clocks, + "* * 1-5", + "Please specify the execution time for every weekday.", + ) + + if "周末" in message: + return _recurring_cron_result( + message, now, timezone_name, target_clocks, "* * 0,6", "请明确周末执行的具体时间。" + ) + + if re.search(r"\bevery\s+weekend\b", lower): + return _recurring_cron_result( + message, + now, + timezone_name, + target_clocks, + "* * 0,6", + "Please specify the execution time for every weekend.", + ) + + weekday_range_match = _RECURRING_WEEKDAY_RANGE_PATTERN.search(message) + if weekday_range_match: + weekdays = _parse_weekday_range( + weekday_range_match.group("start"), + weekday_range_match.group("end"), + ) + return _recurring_cron_result( + message, + now, + timezone_name, + target_clocks, + f"* * {weekdays}", + "请明确每周任务执行的具体时间。", + ) + + weekday_match = _RECURRING_WEEKDAY_PATTERN.search(message) + if weekday_match: + weekdays = ",".join(str(day) for day in _parse_weekday_values(weekday_match.group("days"))) + return _recurring_cron_result( + message, + now, + timezone_name, + target_clocks, + f"* * {weekdays}", + "请明确每周任务执行的具体时间。", + ) + + for weekday_name, cron_day in _EN_WEEKDAY_TO_CRON.items(): + if re.search(rf"\bevery\s+{weekday_name}\b", lower): + return _recurring_cron_result( + message, + now, + timezone_name, + target_clocks, + f"* * {cron_day}", + f"Please specify the execution time for every {weekday_name}.", + ) + + month_match = _MONTH_DAY_LIST_PATTERN.search(message) + if month_match: + month_days = ",".join(str(day) for day in _parse_month_days(month_match.group("days"))) + return _recurring_cron_result( + message, + now, + timezone_name, + target_clocks, + f"{month_days} * *", + "请明确每月任务执行的具体时间。", + ) + + if _MONTH_END_PATTERN.search(message): + return _recurring_cron_result( + message, now, timezone_name, target_clocks, "L * *", "请明确每月任务执行的具体时间。" + ) + + quarter_match = _QUARTERLY_PATTERN.search(message) + if quarter_match: + day_token = quarter_match.group("day") + day = 1 if day_token == "一" else int(day_token) + if day < 1 or day > 31: + return _invalid_schedule(message, "季度任务的日期必须在 1 到 31 之间。") + return _recurring_cron_result( + message, + now, + timezone_name, + target_clocks, + f"{day} 1,4,7,10 *", + "请明确季度任务执行的具体时间。", + ) + + yearly_match = _YEARLY_PATTERN.search(message) + if yearly_match: + month = int(yearly_match.group("month")) + day = int(yearly_match.group("day")) + _future_date(month, day, now, now.year) + return _recurring_cron_result( + message, + now, + timezone_name, + target_clocks, + f"{day} {month} *", + "请明确每年任务执行的具体时间。", + ) + + if unsupported_recurrence := _UNSUPPORTED_RECURRENCE_PATTERN.search(message): + unit = unsupported_recurrence.group("unit") + return _invalid_schedule( + message, + f"暂不支持按多个{unit}生成单一精确规则，请改用明确月份或拆分任务。", + ) + + if _RECURRENCE_MARKER_PATTERN.search(message): + return _invalid_schedule(message, "周期任务缺少可确定的日期或时间，请补充完整。") + + if target_time is None: + return _invalid_schedule(message, "无法确定任务执行时间，请补充具体日期和时间。") + if len(target_clocks) > 1: + return _invalid_schedule( + message, + "一次性任务只能指定一个执行时刻，请拆分为多个任务。", + ) + + target_date: Optional[date] = None + explicit_date = _EXPLICIT_DATE_PATTERN.search(message) or _ISO_DATE_PATTERN.search(message) + if explicit_date: + target_date = _future_date( + int(explicit_date.group("month")), + int(explicit_date.group("day")), + now, + int(explicit_date.group("year")) if explicit_date.group("year") else None, + ) + elif relative_month_date := _RELATIVE_MONTH_DAY_PATTERN.search(message): + target_date = _next_month_day( + relative_month_date.group("relative_month"), + int(relative_month_date.group("day")), + now, + ) + elif short_date := _SHORT_DATE_PATTERN.search(message): + target_date = _future_date( + int(short_date.group("month")), + int(short_date.group("day")), + now, + ) + elif "后天" in message: + target_date = (now + timedelta(days=2)).date() + elif "明天" in message or "tomorrow" in lower: + target_date = (now + timedelta(days=1)).date() + elif "今天" in message or "今晚" in message or "today" in lower: + target_date = now.date() + elif en_next_weekday := _EN_NEXT_WEEKDAY_PATTERN.search(message): + cron_weekday = _EN_WEEKDAY_TO_CRON[en_next_weekday.group("day").lower()] + target_weekday = (cron_weekday - 1) % 7 + days = (target_weekday - now.weekday()) % 7 or 7 + target_date = (now + timedelta(days=days)).date() + else: + target_date = _relative_weekday(message, now) + if target_date is None and (day_match := _DAY_OF_MONTH_ONCE_PATTERN.search(message)): + target_date = _next_day_of_month(int(day_match.group("day")), now) + + if target_date is None: + return _invalid_schedule(message, "无法确定任务执行日期，请补充具体日期。") + if target_time == time(0, 0) and re.search(r"(?:今晚|晚上)\s*12\s*(?:点|时)", message): + target_date += timedelta(days=1) + start_at = _combine_local(target_date, target_time, zone) + if start_at <= now: + return _invalid_schedule(message, "指定的执行时间已经过去，请提供未来时间。") + trigger = ScheduleTrigger( + mode=ScheduleMode.ONCE, + rule_type=ScheduleRuleType.AT, + timezone=timezone_name, + start_at=start_at, + ) + return _success(message, trigger) diff --git a/backend/services/agent_automation/models.py b/backend/services/agent_automation/models.py new file mode 100644 index 0000000000..be0d86af45 --- /dev/null +++ b/backend/services/agent_automation/models.py @@ -0,0 +1,160 @@ +from datetime import datetime +from enum import Enum +from typing import Any, Dict, List, Optional +from zoneinfo import ZoneInfo + +from nexent.scheduler import ScheduleMode, ScheduleRuleType +from pydantic import BaseModel, Field, field_validator, model_validator + + +class StrEnum(str, Enum): + pass + + +class AutomationTaskStatus(StrEnum): + DRAFT = "DRAFT" + ACTIVE = "ACTIVE" + PAUSED = "PAUSED" + PAUSED_BY_SYSTEM = "PAUSED_BY_SYSTEM" + COMPLETED = "COMPLETED" + DELETED = "DELETED" + + +class AutomationRunStatus(StrEnum): + QUEUED = "QUEUED" + RUNNING = "RUNNING" + SUCCEEDED = "SUCCEEDED" + FAILED = "FAILED" + SKIPPED = "SKIPPED" + CANCELED = "CANCELED" + TIMEOUT = "TIMEOUT" + + +class AutomationProposalStatus(StrEnum): + PENDING = "PENDING" + ACCEPTED = "ACCEPTED" + REJECTED = "REJECTED" + EXPIRED = "EXPIRED" + + +class AutomationSource(StrEnum): + CHAT_INTENT = "CHAT_INTENT" + + +class CapabilityType(StrEnum): + TOOL = "TOOL" + SKILL = "SKILL" + KNOWLEDGE_BASE = "KNOWLEDGE_BASE" + MANAGED_AGENT = "MANAGED_AGENT" + EXTERNAL_A2A_AGENT = "EXTERNAL_A2A_AGENT" + MEMORY = "MEMORY" + + +class ScheduleTrigger(BaseModel): + mode: ScheduleMode + rule_type: ScheduleRuleType + timezone: str = "Asia/Shanghai" + start_at: datetime + end_at: Optional[datetime] = None + cron_expr: Optional[str] = None + interval_seconds: Optional[int] = Field(default=None, gt=0) + max_fire_count: Optional[int] = Field(default=None, gt=0) + + @field_validator("timezone") + @classmethod + def validate_timezone(cls, value: str) -> str: + try: + ZoneInfo(value) + except Exception as exc: + raise ValueError(f"Invalid timezone: {value}") from exc + return value + + @model_validator(mode="after") + def validate_combination(self): + if self.end_at is not None and self.end_at <= self.start_at: + raise ValueError("end_at must be later than start_at") + if self.mode == ScheduleMode.ONCE: + if self.rule_type != ScheduleRuleType.AT: + raise ValueError("ONCE schedule only supports AT rule_type") + if self.cron_expr is not None or self.interval_seconds is not None: + raise ValueError("ONCE schedule cannot include cron_expr or interval_seconds") + self.max_fire_count = 1 + elif self.mode == ScheduleMode.RECURRING: + if self.rule_type == ScheduleRuleType.AT: + raise ValueError("RECURRING schedule does not support AT rule_type") + if self.rule_type == ScheduleRuleType.CRON and not self.cron_expr: + raise ValueError("cron_expr is required for CRON schedules") + if self.rule_type == ScheduleRuleType.CRON and self.interval_seconds is not None: + raise ValueError("CRON schedule cannot include interval_seconds") + if self.rule_type == ScheduleRuleType.INTERVAL and not self.interval_seconds: + raise ValueError("interval_seconds is required for INTERVAL schedules") + if self.rule_type == ScheduleRuleType.INTERVAL and self.cron_expr is not None: + raise ValueError("INTERVAL schedule cannot include cron_expr") + return self + + +class CapabilityBinding(BaseModel): + type: CapabilityType + name: str + display_name: Optional[str] = None + binding_ref: str + reason: Optional[str] = None + required: bool = True + + +class CapabilityResolution(BaseModel): + matched_capabilities: List[CapabilityBinding] = Field(default_factory=list) + missing_capabilities: List[Dict[str, Any]] = Field(default_factory=list) + optional_capabilities: List[CapabilityBinding] = Field(default_factory=list) + agent_snapshot: Dict[str, Any] = Field(default_factory=dict) + executable: bool = True + + +class AutomationTaskCreateRequest(BaseModel): + title: str = Field(min_length=1) + agent_id: int = Field(gt=0) + instruction: str = Field(min_length=1) + schedule_trigger: ScheduleTrigger + conversation_id: int = Field(gt=0) + original_instruction: Optional[str] = None + agent_version_no: Optional[int] = None + model_id: Optional[int] = None + tool_params: Optional[Dict[str, Any]] = None + capability_bindings: List[CapabilityBinding] = Field(default_factory=list) + timeout_seconds: Optional[int] = Field(default=None, gt=0) + + +class AutomationTaskPatchRequest(BaseModel): + title: Optional[str] = Field(default=None, min_length=1) + instruction: Optional[str] = Field(default=None, min_length=1) + schedule_trigger: Optional[ScheduleTrigger] = None + capability_bindings: Optional[List[CapabilityBinding]] = None + model_id: Optional[int] = None + tool_params: Optional[Dict[str, Any]] = None + timeout_seconds: Optional[int] = Field(default=None, gt=0) + + +class AutomationProposalCreateRequest(BaseModel): + conversation_id: Optional[int] = Field(default=None, gt=0) + agent_id: int = Field(gt=0) + message: str = Field(min_length=1) + timezone: str = "Asia/Shanghai" + agent_version_no: Optional[int] = None + model_id: Optional[int] = None + tool_params: Optional[Dict[str, Any]] = None + + +class AutomationProposalConfirmRequest(BaseModel): + instruction: Optional[str] = None + + +class AutomationProposalPatchRequest(BaseModel): + title: Optional[str] = Field(default=None, min_length=1) + instruction: Optional[str] = Field(default=None, min_length=1) + schedule_trigger: Optional[ScheduleTrigger] = None + + +class AutomationResponse(BaseModel): + code: int = 0 + message: str = "success" + data: Any = None diff --git a/backend/services/agent_automation/prompt_generator.py b/backend/services/agent_automation/prompt_generator.py new file mode 100644 index 0000000000..1474c518b5 --- /dev/null +++ b/backend/services/agent_automation/prompt_generator.py @@ -0,0 +1,282 @@ +import asyncio +import json +import logging +import re +from abc import ABC, abstractmethod +from dataclasses import dataclass +from typing import Any, Dict, Optional + +from jinja2 import StrictUndefined, Template + +from consts.const import LANGUAGE, MESSAGE_ROLE, MODEL_CONFIG_MAPPING +from utils.prompt_template_utils import get_prompt_template + +logger = logging.getLogger("agent_automation.prompt_generator") + + +_ORCHESTRATION_TERMS = ( + "定时任务", + "自动任务", + "计划时间", + "触发类型", + "时区", + "已绑定", + "工具能力", + "配置文件", + "重试", + "失败", + "错误", + "异常", + "日志", + "当前会话", + "会话上下文", + "不要编造", + "scheduled task", + "automation task", + "scheduled time", + "trigger type", + "timezone", + "bound capabilities", + "configuration file", + "retry", + "if it fails", + "on failure", + "error handling", + "error log", + "current conversation", + "conversation context", + "do not fabricate", + "agent", + "tool", + "utc", +) +_SCHEDULE_NOISE_PATTERNS = ( + re.compile(r"(?:每天|每日|每晚|每周|每星期|每月|每年|每季度|工作日|周末)"), + re.compile(r"每(?:隔\s*)?(?:\d+|[一二两三四五六七八九十百半]+)?\s*(?:秒|分钟|小时|天|周)"), + re.compile(r"(?:上午|早上|中午|下午|晚上|凌晨|午夜)?\s*\d{1,2}\s*(?:[:：点时])"), + re.compile(r"\b(?:every|daily|weekly|monthly|yearly)\b", re.IGNORECASE), +) + + +@dataclass(frozen=True) +class AutomationPromptContext: + """Data required to generate stable task content at creation time.""" + + tenant_id: str + instruction: str + language: str = LANGUAGE["ZH"] + + +@dataclass(frozen=True) +class AutomationTaskContent: + """Stable title and single-run instruction stored on an automation task.""" + + title: str + instruction: str + + +def detect_instruction_language(instruction: str) -> str: + """Select the prompt language from the extracted business action.""" + return LANGUAGE["ZH"] if re.search(r"[\u3400-\u9fff]", instruction) else LANGUAGE["EN"] + + +def _normalize_model_output(content: str, fallback: str, max_length: int, source: str = "") -> str: + normalized = re.sub(r"[\s\S]*?", "", content or "", flags=re.IGNORECASE).strip() + normalized = normalized.removeprefix("```text").removeprefix("```markdown").strip("`\n ") + if not normalized: + return fallback + normalized_lower = normalized.casefold() + source_lower = source.casefold() + has_orchestration_noise = any( + term in normalized_lower and term not in source_lower + for term in _ORCHESTRATION_TERMS + ) + has_schedule_noise = any( + pattern.search(normalized) and not pattern.search(source) + for pattern in _SCHEDULE_NOISE_PATTERNS + ) + if has_orchestration_noise or has_schedule_noise: + logger.warning("Generated automation instruction added orchestration details; using direct fallback") + return fallback + if len(normalized) > max_length: + logger.warning("Generated automation instruction exceeded the length limit; using direct fallback") + return fallback + return normalized + + +def _fallback_title(instruction: str) -> str: + title = re.sub(r"\s+", " ", instruction).strip(" ，,。；;：:\"'“”") + title = re.sub(r"^算一下", "计算", title) + title = re.sub(r"^查一下", "查询", title) + title = re.sub(r"^看一下", "查看", title) + if title.startswith("提醒我") and len(title) > 3: + title = f"{title[3:]}提醒" + title = re.sub( + r"^(发送|发|生成|整理|检查|汇总|总结|推送|发布|" + r"备份|同步|扫描|清理|更新|导出|统计|记录)" + r"(?:一次|一条|一句|一个|一份)", + r"\1", + title, + ) + if title.startswith("发") and not title.startswith(("发送", "发布", "发现", "发起")): + title = f"发送{title[1:]}" + max_length = 20 if detect_instruction_language(instruction) == LANGUAGE["ZH"] else 60 + return title[:max_length] or "自动任务" + + +def _extract_json(content: str) -> Dict[str, Any]: + normalized = re.sub(r"[\s\S]*?", "", content or "", flags=re.IGNORECASE).strip() + fence_match = re.search(r"```(?:json)?\s*([\s\S]*?)```", normalized, flags=re.IGNORECASE) + if fence_match: + normalized = fence_match.group(1).strip() + try: + parsed = json.loads(normalized) + except json.JSONDecodeError: + object_match = re.search(r"\{[\s\S]*\}", normalized) + if not object_match: + raise + parsed = json.loads(object_match.group(0)) + if not isinstance(parsed, dict): + raise ValueError("Automation task content must be a JSON object.") + return parsed + + +def _normalize_task_content( + content: str, + fallback: AutomationTaskContent, + source: str, +) -> AutomationTaskContent: + try: + parsed = _extract_json(content) + if set(parsed) != {"title", "instruction"}: + raise ValueError("Automation task content must contain only title and instruction.") + raw_instruction = str(parsed.get("instruction") or "") + instruction = _normalize_model_output( + raw_instruction, + fallback.instruction, + 300, + source=source, + ) + if instruction == fallback.instruction and raw_instruction.strip() != fallback.instruction: + return fallback + source_language = detect_instruction_language(source) + if detect_instruction_language(instruction) != source_language: + logger.warning("Generated automation instruction changed the source language; using direct fallback") + return fallback + raw_title = str(parsed.get("title") or "") + fallback_title = _fallback_title(instruction) + title_max_length = 20 if source_language == LANGUAGE["ZH"] else 60 + title = _normalize_model_output( + raw_title, + fallback_title, + title_max_length, + source=source, + ) + if title == fallback_title and raw_title.strip() != fallback_title: + return fallback + if detect_instruction_language(title) != source_language: + logger.warning("Generated automation title changed the source language; using direct fallback") + return fallback + return AutomationTaskContent(title=title, instruction=instruction) + except Exception as exc: + logger.warning("Failed to parse structured automation task content, using direct fallback: %s", exc) + return fallback + + +class AutomationPromptStrategy(ABC): + """Strategy interface for automation prompt generation.""" + + @abstractmethod + async def generate_task_content(self, context: AutomationPromptContext) -> AutomationTaskContent: + raise NotImplementedError + + +class TemplateAutomationPromptStrategy(AutomationPromptStrategy): + """Deterministic and fail-open prompt generation strategy.""" + + async def generate_task_content(self, context: AutomationPromptContext) -> AutomationTaskContent: + instruction = context.instruction.strip() + return AutomationTaskContent(title=_fallback_title(instruction), instruction=instruction) + + +class LLMAutomationPromptStrategy(AutomationPromptStrategy): + """LLM-backed strategy with a deterministic fallback strategy.""" + + def __init__(self, model_config: Dict[str, Any], fallback: AutomationPromptStrategy): + self._model_config = model_config + self._fallback = fallback + + async def generate_task_content(self, context: AutomationPromptContext) -> AutomationTaskContent: + fallback = await self._fallback.generate_task_content(context) + try: + content = await asyncio.to_thread( + self._generate_sync, + context, + "TASK_CONTENT_SYSTEM_PROMPT", + "TASK_CONTENT_USER_PROMPT", + ) + return _normalize_task_content(content, fallback, context.instruction) + except Exception as exc: + logger.warning("Failed to generate automation task content, using direct fallback: %s", exc) + return fallback + + def _generate_sync( + self, + context: AutomationPromptContext, + system_key: str, + user_key: str, + ) -> str: + from nexent.core.models import OpenAIModel + from utils.config_utils import get_model_name_from_config + + prompt_template = get_prompt_template("agent_automation", context.language) + values = {"instruction": context.instruction.strip()} + user_prompt = Template(prompt_template[user_key], undefined=StrictUndefined).render(**values).strip() + llm = OpenAIModel( + model_id=get_model_name_from_config(self._model_config) if self._model_config.get("model_name") else "", + api_base=self._model_config.get("base_url", ""), + api_key=self._model_config.get("api_key", ""), + temperature=0.2, + top_p=0.9, + model_factory=self._model_config.get("model_factory"), + ssl_verify=self._model_config.get("ssl_verify", True), + timeout_seconds=self._model_config.get("timeout_seconds"), + stream=False, + ) + response = llm.generate([ + {"role": MESSAGE_ROLE["SYSTEM"], "content": prompt_template[system_key]}, + {"role": MESSAGE_ROLE["USER"], "content": user_prompt}, + ]) + return getattr(response, "content", "") or "" + + +class AutomationPromptStrategyFactory: + """Factory that selects an LLM strategy when the tenant has a usable model.""" + + def create(self, tenant_id: str) -> AutomationPromptStrategy: + fallback = TemplateAutomationPromptStrategy() + try: + from utils.config_utils import tenant_config_manager + + model_config = tenant_config_manager.get_model_config( + key=MODEL_CONFIG_MAPPING["llm"], + tenant_id=tenant_id, + ) + if model_config: + return LLMAutomationPromptStrategy(model_config, fallback) + except Exception as exc: + logger.warning("Failed to resolve automation prompt model, using template strategy: %s", exc) + return fallback + + +class AutomationPromptGenerator: + """Application service that keeps prompt strategy selection out of callers.""" + + def __init__(self, factory: Optional[AutomationPromptStrategyFactory] = None): + self._factory = factory or AutomationPromptStrategyFactory() + + async def generate_task_content(self, context: AutomationPromptContext) -> AutomationTaskContent: + return await self._factory.create(context.tenant_id).generate_task_content(context) + + +automation_prompt_generator = AutomationPromptGenerator() diff --git a/backend/services/agent_automation/runner.py b/backend/services/agent_automation/runner.py new file mode 100644 index 0000000000..143c5ec635 --- /dev/null +++ b/backend/services/agent_automation/runner.py @@ -0,0 +1,305 @@ +import asyncio +import logging +from datetime import datetime, timezone +from typing import Any, Dict, Optional + +from consts.const import AGENT_AUTOMATION_DEFAULT_TIMEOUT_SECONDS +from consts.model import AgentRequest +from database import agent_automation_db +from services.agent_service import is_agent_running, run_agent_background, stop_agent_tasks + +from .capability_resolver import validate_bindings_available +from .conversation_adapter import automation_conversation_adapter +from .models import AutomationRunStatus, ScheduleTrigger +from .schedule_engine import compute_next_fire_at + + +logger = logging.getLogger("agent_automation.runner") + + +def _utcnow() -> datetime: + return datetime.now(timezone.utc) + + +def _parse_dt(value: Any) -> datetime: + if isinstance(value, datetime): + return value if value.tzinfo else value.replace(tzinfo=timezone.utc) + if isinstance(value, str): + return datetime.fromisoformat(value.replace("Z", "+00:00")) + return _utcnow() + + +class AgentAutomationRunner: + async def execute_task( + self, + task: Dict[str, Any], + trigger_type: str = "SCHEDULED", + scheduled_fire_at: Optional[datetime] = None, + lease_owner: Optional[str] = None, + ) -> Dict[str, Any]: + scheduled = scheduled_fire_at or _parse_dt(task.get("next_fire_at")) + if agent_automation_db.has_active_run_for_conversation(task["conversation_id"]) or is_agent_running( + task["conversation_id"], + task["user_id"], + ): + skipped_at = _utcnow() + run = agent_automation_db.create_run({ + "task_id": task["task_id"], + "tenant_id": task["tenant_id"], + "user_id": task["user_id"], + "conversation_id": task["conversation_id"], + "scheduled_fire_at": scheduled, + "actual_fire_at": skipped_at, + "trigger_type": trigger_type, + "status": AutomationRunStatus.SKIPPED.value, + "started_at": skipped_at, + "finished_at": skipped_at, + "error_code": "AUTOMATION_RUN_ALREADY_ACTIVE", + "error_message": "Conversation already has an active automation run.", + }, task["user_id"]) + fire_count, next_fire_at, task_status = self._advance_scheduled_task(task, skipped_at) + task_values = { + "status": task_status, + "last_fire_at": skipped_at, + "last_run_status": AutomationRunStatus.SKIPPED.value, + "last_error": "Conversation already has an active automation run.", + "fire_count": fire_count, + "next_fire_at": next_fire_at, + "lock_owner": None, + "lock_until": None, + } + self._update_task_state(task, task_values, trigger_type, lease_owner) + return run + + run = agent_automation_db.create_run({ + "task_id": task["task_id"], + "tenant_id": task["tenant_id"], + "user_id": task["user_id"], + "conversation_id": task["conversation_id"], + "scheduled_fire_at": scheduled, + "actual_fire_at": _utcnow(), + "trigger_type": trigger_type, + "status": AutomationRunStatus.RUNNING.value, + "started_at": _utcnow(), + }, task["user_id"]) + if lease_owner: + run["_lease_owner"] = lease_owner + + timeout_seconds = float(task.get("timeout_seconds") or AGENT_AUTOMATION_DEFAULT_TIMEOUT_SECONDS) + try: + return await asyncio.wait_for( + self._execute_active_run(run, task, scheduled, trigger_type), + timeout=max(1, timeout_seconds), + ) + except asyncio.TimeoutError: + self.cancel_for_conversation(task["conversation_id"], task["user_id"]) + return self._finish_run(run, task, AutomationRunStatus.TIMEOUT.value, { + "error_code": "AUTOMATION_RUN_TIMEOUT", + "error_message": f"Automation run exceeded {timeout_seconds} seconds.", + }) + except asyncio.CancelledError: + self.cancel_for_conversation(task["conversation_id"], task["user_id"]) + agent_automation_db.cancel_run( + run["run_id"], + task["tenant_id"], + task["user_id"], + "Scheduler execution was interrupted before completion.", + ) + raise + except Exception as exc: + return self._fail_run(run, task, "AUTOMATION_RUN_FAILED", str(exc)) + + async def _execute_active_run( + self, + run: Dict[str, Any], + task: Dict[str, Any], + scheduled: datetime, + trigger_type: str, + ) -> Dict[str, Any]: + capability_status = await validate_bindings_available( + agent_id=task["agent_id"], + tenant_id=task["tenant_id"], + user_id=task["user_id"], + instruction=task["instruction"], + bindings=task.get("capability_bindings") or [], + version_no=task.get("agent_version_no") or 0, + ) + if not capability_status["available"]: + return self._fail_run( + run, + task, + "AUTOMATION_CAPABILITY_UNAVAILABLE", + "Required automation capability is unavailable.", + ) + + stored_snapshot = task.get("runtime_snapshot") or {"agent_id": task["agent_id"]} + current_resolution = capability_status.get("resolution") or {} + current_agent_snapshot = current_resolution.get("agent_snapshot") or {} + runtime_snapshot = { + **stored_snapshot, + **current_agent_snapshot, + # Runtime selections belong to the task even when the Agent metadata changes. + "model_id": stored_snapshot.get("model_id"), + "tool_params": stored_snapshot.get("tool_params"), + "original_instruction": ( + stored_snapshot.get("original_instruction") or task["instruction"] + ), + } + generated_prompt = task["instruction"].strip() + turn = automation_conversation_adapter.append_run_prompt( + task["conversation_id"], + generated_prompt, + task["user_id"], + task["tenant_id"], + ) + + agent_request = AgentRequest( + query=generated_prompt, + conversation_id=task["conversation_id"], + history=turn["history"], + agent_id=task["agent_id"], + model_id=runtime_snapshot.get("model_id"), + version_no=task.get("agent_version_no"), + tool_params=runtime_snapshot.get("tool_params"), + enable_automation_tool=False, + ) + result = await run_agent_background( + agent_request=agent_request, + user_id=task["user_id"], + tenant_id=task["tenant_id"], + skip_user_save=True, + ) + return self._finish_run(run, task, AutomationRunStatus.SUCCEEDED.value, { + "generated_prompt": generated_prompt, + "user_message_id": turn["user_message_id"], + "assistant_message_id": result.get("assistant_message_id"), + }) + + def cancel_for_conversation(self, conversation_id: int, user_id: str) -> None: + stop_agent_tasks(conversation_id, user_id) + + def _fail_run( + self, + run: Dict[str, Any], + task: Dict[str, Any], + error_code: str, + error_message: str, + ) -> Dict[str, Any]: + return self._finish_run(run, task, AutomationRunStatus.FAILED.value, { + "error_code": error_code, + "error_message": error_message, + }) + + def _finish_run( + self, + run: Dict[str, Any], + task: Dict[str, Any], + status: str, + extra: Dict[str, Any], + ) -> Dict[str, Any]: + now = _utcnow() + started_at = _parse_dt(run.get("started_at")) + duration_ms = int((now - started_at).total_seconds() * 1000) + updated_run = agent_automation_db.update_run(run["run_id"], { + **extra, + "status": status, + "finished_at": now, + "duration_ms": duration_ms, + }, task["user_id"], expected_statuses=[AutomationRunStatus.RUNNING.value]) + if not updated_run: + current_run = agent_automation_db.get_run( + run["run_id"], + task["tenant_id"], + task["user_id"], + ) + return current_run or run + + fire_count = int(task.get("fire_count") or 0) + next_fire_at = task.get("next_fire_at") + task_status = task.get("status", "ACTIVE") + consecutive_failures = int(task.get("consecutive_failures") or 0) + is_scheduled_run = run.get("trigger_type") == "SCHEDULED" + if is_scheduled_run: + fire_count, next_fire_at, task_status = self._advance_scheduled_task(task, now) + if status == AutomationRunStatus.SUCCEEDED.value: + consecutive_failures = 0 + elif status in {AutomationRunStatus.FAILED.value, AutomationRunStatus.TIMEOUT.value}: + consecutive_failures += 1 + if consecutive_failures >= 5 and next_fire_at is not None: + task_status = "PAUSED_BY_SYSTEM" + + current_task = agent_automation_db.get_task( + task["task_id"], + task["tenant_id"], + task["user_id"], + ) + if current_task and current_task.get("status") in { + "PAUSED", + "PAUSED_BY_SYSTEM", + }: + task_status = current_task["status"] + + task_values = { + "status": task_status, + "last_fire_at": now, + "last_run_status": status, + "last_error": extra.get("error_message"), + "consecutive_failures": consecutive_failures, + "fire_count": fire_count, + "next_fire_at": next_fire_at, + "lock_owner": None, + "lock_until": None, + } + self._update_task_state( + task, + task_values, + str(run.get("trigger_type") or "SCHEDULED"), + run.get("_lease_owner"), + ) + return updated_run or run + + @staticmethod + def _update_task_state( + task: Dict[str, Any], + values: Dict[str, Any], + trigger_type: str, + lease_owner: Optional[str], + ) -> Optional[Dict[str, Any]]: + if trigger_type == "SCHEDULED" and lease_owner: + updated = agent_automation_db.update_task_if_lock_owner( + task["task_id"], + task["tenant_id"], + task["user_id"], + lease_owner, + values, + ) + if not updated: + logger.warning( + "Discarded stale scheduled task update after lease loss: task_id=%s owner=%s", + task["task_id"], + lease_owner, + ) + return updated + return agent_automation_db.update_task( + task["task_id"], + task["tenant_id"], + task["user_id"], + values, + ) + + @staticmethod + def _advance_scheduled_task( + task: Dict[str, Any], + after: datetime, + ) -> tuple[int, Optional[datetime], str]: + """Advance exactly one scheduled occurrence without consuming manual runs.""" + fire_count = int(task.get("fire_count") or 0) + 1 + trigger = ScheduleTrigger.model_validate(task["schedule_config"]) + next_fire_at = compute_next_fire_at(trigger, after, fire_count) + task_status = task.get("status", "ACTIVE") + if next_fire_at is None: + task_status = "COMPLETED" + return fire_count, next_fire_at, task_status + + +agent_automation_runner = AgentAutomationRunner() diff --git a/backend/services/agent_automation/schedule_engine.py b/backend/services/agent_automation/schedule_engine.py new file mode 100644 index 0000000000..0846df0e0e --- /dev/null +++ b/backend/services/agent_automation/schedule_engine.py @@ -0,0 +1,33 @@ +"""Compatibility adapter from API models to the SDK schedule engine.""" + +from datetime import datetime + +from nexent.scheduler import ( + ScheduleSpec, + compute_next_fire_at as compute_sdk_next_fire_at, + is_valid_cron_expression as validate_sdk_cron_expression, +) + +from .models import ScheduleTrigger + + +def is_valid_cron_expression(expression: str) -> bool: + return validate_sdk_cron_expression(expression) + + +def compute_next_fire_at( + trigger: ScheduleTrigger, + after: datetime, + fire_count: int, +) -> datetime | None: + spec = ScheduleSpec( + mode=trigger.mode, + rule_type=trigger.rule_type, + timezone=trigger.timezone, + start_at=trigger.start_at, + end_at=trigger.end_at, + cron_expr=trigger.cron_expr, + interval_seconds=trigger.interval_seconds, + max_fire_count=trigger.max_fire_count, + ) + return compute_sdk_next_fire_at(spec, after, fire_count) diff --git a/backend/services/agent_automation/scheduler.py b/backend/services/agent_automation/scheduler.py new file mode 100644 index 0000000000..2c4ed76438 --- /dev/null +++ b/backend/services/agent_automation/scheduler.py @@ -0,0 +1,181 @@ +"""Backend adapter for the SDK's durable lease scheduler.""" + +import asyncio +import logging +from datetime import datetime, timezone +from typing import Any, Dict, Hashable + +from consts.const import ( + AGENT_AUTOMATION_ENABLED, + AGENT_AUTOMATION_LEASE_SECONDS, + AGENT_AUTOMATION_MAX_CONCURRENT_RUNS, + AGENT_AUTOMATION_POLL_INTERVAL_SECONDS, + AGENT_AUTOMATION_SHUTDOWN_GRACE_SECONDS, +) +from database import agent_automation_db +from nexent.scheduler import ClaimedJob, ExecutionLease, LeaseScheduler, SchedulerConfig + +from .models import ScheduleTrigger +from .runner import agent_automation_runner +from .schedule_engine import compute_next_fire_at + + +logger = logging.getLogger("agent_automation.scheduler") +_MISFIRE_POLICY_SKIP = "SKIP" + + +def _utcnow() -> datetime: + return datetime.now(timezone.utc) + + +class AgentAutomationLeaseStore: + """Adapt synchronous PostgreSQL operations to the async scheduler contract.""" + + def __init__(self) -> None: + self._recovery_time: datetime | None = None + + async def recover(self) -> None: + recovery_time = _utcnow() + await asyncio.to_thread(agent_automation_db.recover_orphaned_runs) + await asyncio.to_thread(agent_automation_db.release_expired_locks) + self._recovery_time = recovery_time + + async def claim_due( + self, + owner_id: str, + limit: int, + lease_seconds: float, + ) -> list[ClaimedJob[Dict[str, Any]]]: + tasks = await asyncio.to_thread( + agent_automation_db.claim_due_tasks, + owner_id, + limit, + lease_seconds, + ) + runnable_tasks = [] + for task in tasks: + if await self._skip_claimed_misfire(task, owner_id): + continue + runnable_tasks.append(task) + return [ClaimedJob(job_id=task["task_id"], payload=task) for task in runnable_tasks] + + async def _skip_claimed_misfire(self, task: Dict[str, Any], owner_id: str) -> bool: + """Advance a pre-restart recurring fire without invoking its executor.""" + if self._recovery_time is None or task.get("schedule_mode") != "RECURRING": + return False + + scheduled_fire_at = task.get("next_fire_at") + if isinstance(scheduled_fire_at, str): + scheduled_fire_at = datetime.fromisoformat(scheduled_fire_at.replace("Z", "+00:00")) + if scheduled_fire_at.tzinfo is None: + scheduled_fire_at = scheduled_fire_at.replace(tzinfo=timezone.utc) + if scheduled_fire_at >= self._recovery_time: + return False + + task_status = "ACTIVE" + last_error = None + try: + trigger = ScheduleTrigger.model_validate(task["schedule_config"]) + next_fire_at = compute_next_fire_at( + trigger, + self._recovery_time, + int(task.get("fire_count") or 0), + ) + if next_fire_at is None: + task_status = "COMPLETED" + except Exception: + task_status = "PAUSED_BY_SYSTEM" + last_error = "Invalid schedule configuration during restart recovery." + next_fire_at = None + logger.exception( + "Failed to compute post-restart schedule; pausing task: task_id=%s", + task["task_id"], + ) + + task_values = { + "status": task_status, + "next_fire_at": next_fire_at, + "misfire_policy": _MISFIRE_POLICY_SKIP, + "lock_owner": None, + "lock_until": None, + } + if last_error is not None: + task_values["last_error"] = last_error + + updated = await asyncio.to_thread( + agent_automation_db.update_task_if_lock_owner, + task["task_id"], + task["tenant_id"], + task["user_id"], + owner_id, + task_values, + ) + if not updated: + logger.warning( + "Discarded missed-fire recovery after lease loss: task_id=%s owner=%s", + task["task_id"], + owner_id, + ) + return True + + async def renew(self, job_id: Hashable, owner_id: str, lease_seconds: float) -> bool: + return await asyncio.to_thread( + agent_automation_db.renew_task_lock, + int(job_id), + owner_id, + lease_seconds, + ) + + async def release(self, job_id: Hashable, owner_id: str) -> bool: + return await asyncio.to_thread( + agent_automation_db.release_task_lock, + int(job_id), + owner_id, + ) + + +async def execute_agent_automation( + job: ClaimedJob[Dict[str, Any]], + lease: ExecutionLease, +) -> None: + await agent_automation_runner.execute_task( + job.payload, + trigger_type="SCHEDULED", + lease_owner=lease.owner_id, + ) + + +class AgentAutomationScheduler: + """Application lifecycle wrapper around the reusable SDK scheduler.""" + + def __init__(self) -> None: + self._scheduler = LeaseScheduler( + store=AgentAutomationLeaseStore(), + executor=execute_agent_automation, + config=SchedulerConfig( + poll_interval_seconds=AGENT_AUTOMATION_POLL_INTERVAL_SECONDS, + lease_seconds=AGENT_AUTOMATION_LEASE_SECONDS, + max_concurrency=AGENT_AUTOMATION_MAX_CONCURRENT_RUNS, + shutdown_grace_seconds=AGENT_AUTOMATION_SHUTDOWN_GRACE_SECONDS, + ), + ) + + @property + def instance_id(self) -> str: + return self._scheduler.owner_id + + @property + def is_running(self) -> bool: + return self._scheduler.is_running + + async def start(self) -> None: + if not AGENT_AUTOMATION_ENABLED: + logger.info("Agent automation scheduler disabled") + return + await self._scheduler.start() + + async def stop(self) -> None: + await self._scheduler.stop() + + +agent_automation_scheduler = AgentAutomationScheduler() diff --git a/backend/services/agent_automation/tool_adapter.py b/backend/services/agent_automation/tool_adapter.py new file mode 100644 index 0000000000..d55269e6bc --- /dev/null +++ b/backend/services/agent_automation/tool_adapter.py @@ -0,0 +1,251 @@ +"""Backend adapter for the SDK scheduled-task proposal tool. + +This module is the only bridge between AgentLoop runtime identity and the +automation domain. The extraction model receives the trusted user message and +time settings only; Agent/runtime fields are applied after extraction. +""" + +from __future__ import annotations + +import asyncio +import concurrent.futures +import json +import logging +from dataclasses import dataclass +from typing import Any, Callable, Dict, Optional + +from database import agent_automation_db +from nexent.core.agents.agent_model import ToolConfig +from nexent.core.tools.create_scheduled_task_tool import ( + CreateScheduledTaskProposalTool, +) + +from .errors import ( + AgentAutomationError, + AutomationConversationAlreadyBoundError, + AutomationScheduleInvalidError, +) +from .facade import agent_automation_facade +from .models import AutomationProposalCreateRequest +from .prompt_generator import detect_instruction_language + + +logger = logging.getLogger("agent_automation.tool_adapter") +DEFAULT_AUTOMATION_TIMEZONE = "Asia/Shanghai" + + +def _run_coroutine(coro): + try: + asyncio.get_running_loop() + except RuntimeError: + return asyncio.run(coro) + with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool: + return pool.submit(asyncio.run, coro).result() + + +@dataclass(frozen=True) +class AutomationToolRuntimeContext: + tenant_id: str + user_id: str + conversation_id: int + agent_id: int + user_message: str + source_message_id: Optional[int] = None + agent_version_no: Optional[int] = None + model_id: Optional[int] = None + tool_params: Optional[Dict[str, Any]] = None + timezone: str = DEFAULT_AUTOMATION_TIMEZONE + has_attachments: bool = False + + +class AgentLoopAutomationToolAdapter: + def build_tool_config( + self, + *, + tenant_id: str, + user_id: str, + conversation_id: int, + agent_id: int, + user_message: str, + agent_version_no: Optional[int], + model_id: Optional[int], + tool_params: Optional[Dict[str, Any]], + has_attachments: bool, + language: str, + ) -> ToolConfig: + """Build the system-injected tool config for one interactive run.""" + from services.conversation_management_service import ( + get_current_run_user_message_id, + ) + + context = AutomationToolRuntimeContext( + tenant_id=tenant_id, + user_id=user_id, + conversation_id=conversation_id, + agent_id=agent_id, + user_message=user_message, + source_message_id=get_current_run_user_message_id( + conversation_id, + user_id, + ), + agent_version_no=agent_version_no, + model_id=model_id, + tool_params=tool_params, + has_attachments=has_attachments, + ) + description = ( + CreateScheduledTaskProposalTool.description + if language == "en" + else CreateScheduledTaskProposalTool.description_zh + ) + return ToolConfig( + class_name=CreateScheduledTaskProposalTool.__name__, + name=CreateScheduledTaskProposalTool.name, + description=description, + inputs=json.dumps( + CreateScheduledTaskProposalTool.inputs, + ensure_ascii=False, + ), + output_type=CreateScheduledTaskProposalTool.output_type, + params={}, + source="builtin", + usage="builtin", + metadata={"create_proposal": self.build_callback(context)}, + ) + + def build_callback( + self, + context: AutomationToolRuntimeContext, + ) -> Callable[[str], Dict[str, Any]]: + def create_proposal(request_text: str) -> Dict[str, Any]: + return _run_coroutine(self.create_proposal(context, request_text)) + + return create_proposal + + async def create_proposal( + self, + context: AutomationToolRuntimeContext, + request_text: str, + ) -> Dict[str, Any]: + # The model argument is intentionally not forwarded to extraction. The + # persisted current user message is the authoritative business input. + del request_text + language = detect_instruction_language(context.user_message) + if context.source_message_id is None: + message = ( + "本轮消息尚未完成持久化，无法安全创建定时任务提案。请稍后重试。" + if language == "zh" + else "This message is not persisted yet, so a scheduled-task proposal " + "cannot be created safely. Try again later." + ) + return { + "status": "error", + "error_code": "AUTOMATION_SOURCE_MESSAGE_UNAVAILABLE", + "user_message": message, + } + if context.has_attachments: + message = ( + "当前版本不能把本轮临时附件作为定时任务的长期输入。" + "请改为描述稳定的数据来源后再创建。" + if language == "zh" + else "This version cannot use a temporary attachment as recurring task input. " + "Describe a stable data source and try again." + ) + return { + "status": "needs_clarification", + "missing_fields": ["data_source"], + "user_message": message, + } + + request = AutomationProposalCreateRequest( + conversation_id=context.conversation_id, + agent_id=context.agent_id, + message=context.user_message, + timezone=context.timezone or DEFAULT_AUTOMATION_TIMEZONE, + agent_version_no=context.agent_version_no, + model_id=context.model_id, + tool_params=context.tool_params, + ) + try: + proposal = await agent_automation_facade.create_proposal( + request, + context.tenant_id, + context.user_id, + persist_conversation_exchange=False, + source_message_id=context.source_message_id, + force_llm=True, + ) + except AutomationScheduleInvalidError as exc: + question = str(exc.details.get("clarification_question") or exc.message) + return { + "status": "needs_clarification", + "missing_fields": exc.details.get("missing_fields") or [], + "user_message": question, + } + except AutomationConversationAlreadyBoundError: + message = ( + "当前会话已经绑定了一个有效的定时任务。" + "如需创建另一个任务，请新建会话并重新描述。" + if language == "zh" + else "This conversation already has an active scheduled task. " + "Start a new conversation to create another one." + ) + return {"status": "conflict", "user_message": message} + except AgentAutomationError as exc: + logger.warning( + "AgentLoop automation proposal failed: code=%s details=%s", + exc.error_code, + exc.details, + ) + return { + "status": "error", + "error_code": exc.error_code, + "user_message": exc.message, + } + + if proposal.get("proposal_id") is None: + message = ( + "这条消息没有包含需要未来或周期执行的任务。" + "请补充明确的执行时间或周期。" + if language == "zh" + else "This message does not describe a future or recurring task. " + "Add a specific execution time or recurrence." + ) + return {"status": "not_automation", "user_message": message} + + message = ( + "定时任务提案已生成，请核对任务内容和执行时间后确认创建。" + if language == "zh" + else "The scheduled-task proposal is ready. Review its task and schedule, then confirm it." + ) + return { + "status": "proposal_ready", + "proposal": proposal, + "user_message": message, + } + + +agent_loop_automation_tool_adapter = AgentLoopAutomationToolAdapter() + + +def link_persisted_proposal_card( + content: str, + tenant_id: str, + user_id: str, + message_id: int, + unit_id: int, +) -> bool: + """Attach a persisted conversation unit to its proposal for later card updates.""" + try: + payload = json.loads(content) + proposal_id = int(payload["proposal_id"]) + except (KeyError, TypeError, ValueError, json.JSONDecodeError): + logger.warning("Invalid persisted automation proposal event payload") + return False + return agent_automation_db.link_proposal_message_unit( + proposal_id, + tenant_id, + user_id, + message_id, + unit_id, + ) diff --git a/backend/services/agent_repository_service.py b/backend/services/agent_repository_service.py index 9ead664386..9369075205 100644 --- a/backend/services/agent_repository_service.py +++ b/backend/services/agent_repository_service.py @@ -14,12 +14,13 @@ ) from consts.exceptions import UnauthorizedError from consts.model import AgentRepositorySnapshot +from consts.notification import EVENT_TYPE_REPOSITORY_REVIEW_PENDING, RESOURCE_TYPE_AGENT_REPOSITORY from database.agent_db import search_agent_info_by_agent_id from database.agent_version_db import search_version_by_version_no from database.agent_repository_db import ( fetch_draft_agent_mine_metadata, get_agent_repository_by_agent_id, - get_agent_repository_by_id_and_publisher, + get_agent_repository_by_id, increment_agent_repository_downloads, insert_agent_repository_record, list_agent_repository_by_agent_ids, @@ -37,6 +38,11 @@ import_agent_with_skills_impl, list_all_agent_info_impl, ) +from services.notification_service import ( + create_repository_pending_review_notification, + create_repository_review_notification, + deactivate_notifications, +) from services.repository_import_precheck import build_repository_import_precheck logger = logging.getLogger("agent_repository_service") @@ -96,6 +102,7 @@ def _to_summary_item( "version_label": record.get("version_name"), "icon": record.get("icon"), "downloads": downloads, + "content": record.get("content"), } @@ -264,6 +271,7 @@ def _to_repository_info_item(record: Dict[str, Any]) -> Dict[str, Any]: "version_no": record.get("version_no"), "version_label": record.get("version_name"), "create_time": _serialize_created_at(record.get("create_time")), + "content": record.get("content"), } @@ -346,6 +354,7 @@ async def list_my_editable_agents_impl( page_size: int = 10, search: Optional[str] = None, new_agent_padding: bool = False, + agent_id: Optional[int] = None, ) -> Dict[str, Any]: """List visible draft agents for the current user with repository listing info.""" normalized_ownership = (ownership or OWNERSHIP_ALL).strip().lower() @@ -366,10 +375,10 @@ async def list_my_editable_agents_impl( filtered_agents = [] for agent in all_agents: - agent_id = agent.get("agent_id") - if agent_id is None: + current_agent_id = agent.get("agent_id") + if current_agent_id is None: continue - meta = meta_by_id.get(int(agent_id), {}) + meta = meta_by_id.get(int(current_agent_id), {}) if not _matches_mine_ownership_filter( meta.get("created_by"), user_id, @@ -385,10 +394,18 @@ async def list_my_editable_agents_impl( if _matches_mine_search_filter(agent, search) ] + if agent_id is not None: + filtered_agents = [ + (agent, meta) + for agent, meta in filtered_agents + if agent.get("agent_id") is not None and int(agent["agent_id"]) == agent_id + ] + include_padding = ( new_agent_padding and normalized_ownership == OWNERSHIP_ALL and not (search and search.strip()) + and agent_id is None ) paged_entries, total = _paginate_mine_agents_with_optional_padding( filtered_agents, @@ -410,10 +427,10 @@ async def list_my_editable_agents_impl( publisher_tenant_id=tenant_id, ) for record in repository_records: - agent_id = record.get("agent_id") - if agent_id is None: + record_agent_id = record.get("agent_id") + if record_agent_id is None: continue - repository_by_agent_id.setdefault(int(agent_id), []).append( + repository_by_agent_id.setdefault(int(record_agent_id), []).append( _to_repository_info_item(record) ) @@ -425,7 +442,7 @@ async def list_my_editable_agents_impl( items.append({"new_agent_padding": True}) continue agent, meta = entry - agent_id = int(agent["agent_id"]) + entry_agent_id = int(agent["agent_id"]) items.append( { "agent_id": agent.get("agent_id"), @@ -437,8 +454,8 @@ async def list_my_editable_agents_impl( meta.get("version_create_time") ), "permission": agent.get("permission"), - "downloads": download_totals.get(agent_id, 0), - "repository_info": repository_by_agent_id.get(agent_id, []), + "downloads": download_totals.get(entry_agent_id, 0), + "repository_info": repository_by_agent_id.get(entry_agent_id, []), } ) @@ -520,7 +537,7 @@ def get_agent_repository_listing_detail_impl( tenant_id: str, ) -> Dict[str, Any]: """Load a repository listing and return a detail payload for the UI.""" - record = get_agent_repository_by_id_and_publisher( + record = get_agent_repository_by_id( agent_repository_id, tenant_id, ) @@ -641,6 +658,8 @@ def update_agent_repository_status_impl( status: str, user_id: str, tenant_id: str, + notify_content: Optional[str] = None, + content: Optional[str] = None, ) -> Dict[str, Any]: """Update a repository listing status by primary key.""" if status not in VALID_REPOSITORY_STATUSES: @@ -649,7 +668,7 @@ def update_agent_repository_status_impl( f"{', '.join(sorted(VALID_REPOSITORY_STATUSES))}" ) - record = get_agent_repository_by_id_and_publisher( + record = get_agent_repository_by_id( agent_repository_id, tenant_id, ) @@ -688,6 +707,7 @@ def update_agent_repository_status_impl( else None ), submitted_by=submitted_by, + content=content, ) if rows_affected == 0: raise ValueError("Repository listing not found") @@ -699,15 +719,65 @@ def update_agent_repository_status_impl( publisher_tenant_id=tenant_id, ) - updated = get_agent_repository_by_id_and_publisher( + updated = get_agent_repository_by_id( agent_repository_id, tenant_id, ) if not updated: raise ValueError("Failed to load repository listing after update") + + _handle_review_status_notifications( + current_status=current_status, + new_status=status, + updated=updated, + agent_repository_id=agent_repository_id, + user_id=user_id, + content=content, + notify_content=notify_content, + ) + return _to_summary_item(updated) +def _handle_review_status_notifications( + *, + current_status: str, + new_status: str, + updated: Dict[str, Any], + agent_repository_id: int, + user_id: str, + content: Optional[str] = None, + notify_content: Optional[str] = None, +) -> None: + """Send review-result notification and deactivate pending-review notification.""" + if current_status != new_status and new_status in (STATUS_SHARED, STATUS_REJECTED): + details: Dict[str, Any] = { + "name": updated.get("display_name") or updated.get("name"), + "agent_repository_id": agent_repository_id, + "agent_id": updated.get("agent_id"), + } + review_reason = content or notify_content + if review_reason: + details["content"] = review_reason + create_repository_review_notification( + resource_type=RESOURCE_TYPE_AGENT_REPOSITORY, + review_status=new_status, + receiver_user_id=updated["publisher_user_id"], + details=details, + tenant_id=updated.get("publisher_tenant_id"), + unique_id=agent_repository_id, + created_by=user_id, + ) + + if current_status == STATUS_PENDING_REVIEW: + deactivate_notifications( + event_type=EVENT_TYPE_REPOSITORY_REVIEW_PENDING, + resource_type=RESOURCE_TYPE_AGENT_REPOSITORY, + unique_id=agent_repository_id, + updated_by=user_id, + ) + + def _to_list_item(record: Dict[str, Any]) -> Dict[str, Any]: """Map a DB record to a marketplace list item (without heavy JSON blobs).""" return { @@ -729,6 +799,7 @@ def _to_list_item(record: Dict[str, Any]) -> Dict[str, Any]: "publisher_tenant_id": record.get("publisher_tenant_id"), "created_at": record.get("create_time"), "updated_at": record.get("update_time"), + "content": record.get("content"), } @@ -840,7 +911,7 @@ async def _build_repository_data_from_agent( } if card_fields: - for key in ("icon", "downloads", "tool_count"): + for key in ("icon", "downloads", "tool_count", "content"): if key in card_fields and card_fields[key] is not None: repository_data[key] = card_fields[key] if "tags" in card_fields and card_fields["tags"] is not None: @@ -875,6 +946,7 @@ async def create_agent_repository_listing_impl( version_no, card_fields=card_fields, ) + repository_data["content"] = (card_fields or {}).get("content") or "" _validate_create_payload(repository_data) existing = get_agent_repository_by_agent_id( @@ -891,7 +963,10 @@ async def create_agent_repository_listing_impl( is_updated = False else: repository_id = int(existing["agent_repository_id"]) - updates: Dict[str, Any] = {"status": STATUS_PENDING_REVIEW} + updates: Dict[str, Any] = { + "status": STATUS_PENDING_REVIEW, + "content": repository_data["content"], + } for key in ("icon", "tags", "tool_count"): if key in repository_data: updates[key] = repository_data[key] @@ -905,7 +980,7 @@ async def create_agent_repository_listing_impl( raise ValueError("Failed to update repository listing") is_updated = True - record = get_agent_repository_by_id_and_publisher( + record = get_agent_repository_by_id( repository_id, tenant_id, ) @@ -917,6 +992,18 @@ async def create_agent_repository_listing_impl( status=repository_data["status"], publisher_tenant_id=tenant_id, ) + create_repository_pending_review_notification( + resource_type=RESOURCE_TYPE_AGENT_REPOSITORY, + tenant_id=tenant_id, + unique_id=repository_id, + details={ + "name": record.get("display_name") or record.get("name"), + "agent_repository_id": repository_id, + "agent_id": record.get("agent_id"), + "content": record.get("content") or "", + }, + created_by=user_id, + ) return _to_detail_item(record, is_updated=is_updated) @@ -925,7 +1012,7 @@ def check_repository_import_precheck_impl( tenant_id: str, ) -> Dict[str, Any]: """Check whether the current tenant can import a shared repository listing.""" - record = get_agent_repository_by_id_and_publisher( + record = get_agent_repository_by_id( agent_repository_id, tenant_id, ) @@ -960,7 +1047,7 @@ async def import_agent_from_repository_impl( authorization: str, ) -> Dict[int, int]: """Import an agent tree from a marketplace repository listing into the current tenant.""" - record = get_agent_repository_by_id_and_publisher( + record = get_agent_repository_by_id( agent_repository_id, tenant_id, ) diff --git a/backend/services/agent_service.py b/backend/services/agent_service.py index 5f66e8a1a8..ac1d6369c4 100644 --- a/backend/services/agent_service.py +++ b/backend/services/agent_service.py @@ -5,7 +5,6 @@ import json import logging import os -import uuid import zipfile from collections import deque from typing import Any, Callable, Optional, Dict, List @@ -13,7 +12,6 @@ from fastapi import Header, Request from fastapi.responses import JSONResponse, StreamingResponse from nexent.core.agents.run_agent import agent_run -from nexent.memory.memory_service import clear_memory, add_memory_in_levels from jinja2 import Template from agents.agent_run_manager import agent_run_manager @@ -21,10 +19,10 @@ from agents.preprocess_manager import preprocess_manager from services.agent_version_service import publish_version_impl from utils.prompt_template_utils import normalize_prompt_generate_template_content -from consts.const import MEMORY_SEARCH_START_MSG, MEMORY_SEARCH_DONE_MSG, MEMORY_SEARCH_FAIL_MSG, TOOL_TYPE_MAPPING, \ +from consts.const import TOOL_TYPE_MAPPING, \ LANGUAGE, MESSAGE_ROLE, MODEL_CONFIG_MAPPING, CAN_EDIT_ALL_USER_ROLES, PERMISSION_PRIVATE, STREAM_STATUS_EVENT, \ DEFAULT_EN_TITLE, DEFAULT_ZH_TITLE, RUNTIME_CANCEL_POLL_INTERVAL_SECONDS -from consts.exceptions import AppException, MemoryPreparationException, SkillDuplicateError +from consts.exceptions import AppException, ForbiddenError, MemoryPreparationException, SkillDuplicateError from consts.error_code import ErrorCode from consts.agent_unavailable_reasons import AgentUnavailableReason from nexent.core.utils.observer import ProcessType @@ -37,7 +35,6 @@ ExportAndImportDataFormat, MCPInfo, MessageRequest, - MessageUnit, SkillInstanceInfoRequest, SkillZipEntry, ToolInstanceInfoRequest, @@ -45,6 +42,7 @@ ) from services.asset_owner_visibility import resolve_agent_list_permission from database.agent_db import ( + batch_search_agent_display_names, create_agent, delete_agent_by_id, delete_agent_relationship, @@ -62,7 +60,12 @@ clear_agent_new_mark ) from database import a2a_agent_db -from database.model_management_db import get_model_by_model_id, get_model_by_model_id_ignore_delete, get_model_id_by_display_name, get_valid_model_ids +from database.model_management_db import ( + get_model_by_model_id, + get_model_by_model_id_ignore_delete, + get_model_id_by_display_name, + get_valid_model_ids, +) from database.remote_mcp_db import get_mcp_server_by_name_and_tenant from database.tool_db import ( check_tool_is_available, @@ -70,7 +73,7 @@ delete_tools_by_agent_id, query_all_enabled_tool_instances, query_all_tools, - query_tool_instances_by_id, + query_tool_instances_by_id, # noqa: F401 - compatibility patch point query_tool_instances_by_agent_id, search_tools_for_sub_agent ) @@ -78,7 +81,7 @@ from database.attachment_db import upload_fileobj from services.skill_service import SkillService from services.file_management_service import is_allowed_skill_upload_path -from database.agent_version_db import query_version_list, query_current_version_no +from database.agent_version_db import query_version_list, query_current_version_no, batch_search_version_names, batch_query_current_version_nos from database.group_db import query_group_ids_by_user from database.user_tenant_db import get_user_tenant_by_user_id from database.a2a_agent_db import get_server_agent_ids, query_external_sub_agents @@ -90,9 +93,13 @@ from utils.str_utils import convert_list_to_string, convert_string_to_list from services.conversation_management_service import ( create_new_conversation, - generate_conversation_title_service, + generate_conversation_title_service, # noqa: F401 - compatibility patch point + get_conversation_service, + get_current_run_user_message_id, get_latest_assistant_message, get_last_unit_for_message, + load_historical_context, + persist_history_summary_candidate, save_conversation_user, save_message, save_message_unit, @@ -100,6 +107,7 @@ save_source_search, save_skill_files_to_conversation, update_conversation_agent_id_service, + update_conversation_chat_mode_service, update_message_content, update_message_status, update_unit_content, @@ -110,7 +118,7 @@ from services.runtime_state_service import runtime_state_service from utils.auth_utils import get_current_user_info, get_user_language from utils.config_utils import tenant_config_manager -from utils.memory_utils import build_memory_config +from utils.context_utils import build_authorized_context_input from utils.thread_utils import submit from utils.prompt_template_utils import get_prompt_generate_prompt_template from utils.llm_utils import call_llm_for_system_prompt @@ -123,6 +131,7 @@ logger = logging.getLogger(__name__) SAFE_AGENT_STREAM_ERROR_MESSAGE = "Agent execution failed. Please try again later." +_channel_cleanup_tasks: set[asyncio.Task[None]] = set() async def _cleanup_channel_later(conversation_id: int, user_id: str, delay: float = 5.0): @@ -198,6 +207,18 @@ def _extract_skill_file_upload_payloads(content: str) -> list[dict]: return payloads +def _serialize_stream_unit_content(data: Dict[str, Any], content: str) -> str: + """Preserve tool metadata in the existing message-unit content column.""" + if data.get("type") not in {"tool", "tool-call"}: + return content + + payload: Dict[str, Any] = {"content": content} + for field in ("tool_name", "tool_arguments", "role"): + if field in data: + payload[field] = data[field] + return json.dumps(payload, ensure_ascii=False) + + def _transform_skill_files_to_standard_format(upload_results: list[dict]) -> list[dict]: """ Transform skill file upload results to match the frontend attachment format. @@ -222,14 +243,19 @@ def _transform_skill_files_to_standard_format(upload_results: list[dict]) -> lis async def _process_skill_file_uploads( - content: str, + payloads: list[dict] | str, user_id: str, tenant_id: str, ) -> list[dict]: """Upload generated skill files to storage and return upload metadata.""" upload_results: list[dict] = [] - for payload in _extract_skill_file_upload_payloads(content): + structured_payloads = ( + payloads + if isinstance(payloads, list) + else _extract_skill_file_upload_payloads(payloads) + ) + for payload in structured_payloads: absolute_path = str(payload.get("absolute_path") or "").strip() file_name = str( payload.get("file_name") @@ -287,7 +313,7 @@ async def _process_skill_file_uploads( absolute_path, error_message, ) - except Exception as exc: + except Exception: logger.exception( "[skill-file] failed to upload file file_name=%s absolute_path=%s", file_name, @@ -914,7 +940,6 @@ async def _stream_agent_chunks( ProcessType.MODEL_OUTPUT_DEEP_THINKING.value, } - captured_final_answer = None captured_skill_files: dict[str, dict] = {} skill_file_uploads: list[dict] = [] @@ -926,7 +951,7 @@ async def _stream_agent_chunks( streaming_message_id: Optional[int] = resume_message_id if not is_resume_mode and not agent_request.is_debug: user_role_count = sum( - 1 for item in getattr(agent_request, "history", []) + 1 for item in (getattr(agent_request, "history", None) or ()) if item.role == MESSAGE_ROLE["USER"] ) assistant_message_req = MessageRequest( @@ -978,8 +1003,18 @@ async def _stream_agent_chunks( yield STREAM_STATUS_EVENT yield f'data: {{"status": "resumed", "last_unit_index": {resume_from_unit_index - 1}}}\n\n' + async def _iter_run_chunks(): + for event in getattr( + agent_run_info.agent_config, + "pre_run_tool_events", + (), + ): + yield json.dumps(event, ensure_ascii=False) + async for agent_chunk in agent_run(agent_run_info): + yield agent_chunk + try: - async for chunk in agent_run(agent_run_info): + async for chunk in _iter_run_chunks(): chunk_type: Optional[str] = None chunk_content: str = "" try: @@ -999,6 +1034,8 @@ async def _stream_agent_chunks( elif chunk_type not in ("search_content_placeholder",): # New unit - this will be the next index after assignment data["unit_index"] = next_unit_index + # Tool events and side-channel output carry the same ID + # from the observer's actual invocation context. # Re-serialize the chunk with unit_index for accurate frontend skip chunk = json.dumps(data) logger.debug(f"[resume-debug] Added unit_index to chunk: type={chunk_type}, unit_index={data.get('unit_index')}") @@ -1008,8 +1045,33 @@ async def _stream_agent_chunks( yield f"data: {chunk}\n\n" continue - if chunk_type == "final_answer": - captured_final_answer = chunk_content + if chunk_type == ProcessType.SKILL_ARTIFACT.value: + artifact_content = data.get("content") + if isinstance(artifact_content, str): + try: + artifact_content = json.loads(artifact_content) + except json.JSONDecodeError: + artifact_content = {} + + artifacts = ( + artifact_content.get("artifacts", []) + if isinstance(artifact_content, dict) + else [] + ) + for artifact in artifacts: + if not isinstance(artifact, dict): + continue + absolute_path = str(artifact.get("absolute_path") or "").strip() + if not absolute_path or absolute_path in captured_skill_files: + continue + captured_skill_files[absolute_path] = artifact + + logger.info( + "[skill-file] received structured artifacts count=%s current_total=%s", + len(artifacts), + len(captured_skill_files), + ) + continue should_parse_skill_file = ( chunk_type in {"execution_logs", "parse"} @@ -1067,9 +1129,7 @@ async def _stream_agent_chunks( # loop is async but the DB operations are I/O-bound with network # latency, synchronous writes here are acceptably fast and guarantee # that each chunk is fully persisted before the next chunk arrives. - old_len = len(current_unit["content"]) current_unit["content"] += chunk_content - new_len = len(current_unit["content"]) update_unit_content( current_unit["unit_id"], current_unit["content"], @@ -1124,16 +1184,26 @@ async def _stream_agent_chunks( # and inserts each search result as a source_search row # linked back to the unit_id we just created. if chunk_type == "search_content": - placeholder_unit_id = submit( - save_message_unit, - message_id=streaming_message_id, - conversation_id=agent_request.conversation_id, - unit_index=next_unit_index, - unit_type="search_content_placeholder", - unit_content='{"placeholder": true}', - user_id=user_id, - unit_status="completed", - ).result() + try: + placeholder_unit_id = submit( + save_message_unit, + message_id=streaming_message_id, + conversation_id=agent_request.conversation_id, + unit_index=next_unit_index, + unit_type="search_content_placeholder", + unit_content='{"placeholder": true}', + user_id=user_id, + unit_status="completed", + tool_call_id=data.get("tool_call_id"), + invocation_id=data.get("invocation_id"), + ).result() + except Exception as persistence_exc: + logger.error( + "Failed to persist search_content placeholder: %r", + persistence_exc, + exc_info=True, + ) + placeholder_unit_id = None try: search_results = json.loads(chunk_content) if not isinstance(search_results, list): @@ -1179,27 +1249,64 @@ async def _stream_agent_chunks( continue # Default path: insert a new unit row with unit_status='streaming'. - if streaming_message_id is not None and chunk_type not in ( + # history_summary is already persisted once by the canonical + # checkpoint sink on its covered assistant message. The stream + # event is display-only and must not create a duplicate unit on + # the currently-running assistant message. + if chunk_type == "history_summary": + current_unit = None + elif streaming_message_id is not None and chunk_type not in ( "search_content_placeholder", ): - new_unit_id = submit( - save_message_unit, - message_id=streaming_message_id, - conversation_id=agent_request.conversation_id, - unit_index=next_unit_index, - unit_type=chunk_type, - unit_content=chunk_content, - user_id=user_id, - unit_status="streaming", - ).result() - current_unit = { - "type": chunk_type, - "content": chunk_content, - "unit_id": new_unit_id, - "unit_index": next_unit_index, - "mergeable": mergeable, - } - next_unit_index += 1 + persisted_content = _serialize_stream_unit_content( + data, chunk_content + ) + try: + new_unit_id = submit( + save_message_unit, + message_id=streaming_message_id, + conversation_id=agent_request.conversation_id, + unit_index=next_unit_index, + unit_type=chunk_type, + unit_content=persisted_content, + user_id=user_id, + unit_status="streaming", + tool_call_id=data.get("tool_call_id"), + invocation_id=data.get("invocation_id"), + ).result() + except Exception as persistence_exc: + logger.error( + "Failed to persist streaming message unit: %r", + persistence_exc, + exc_info=True, + ) + else: + current_unit = { + "type": chunk_type, + "content": persisted_content, + "unit_id": new_unit_id, + "unit_index": next_unit_index, + "mergeable": mergeable, + } + if chunk_type == "automation_proposal": + try: + from services.agent_automation.tool_adapter import ( + link_persisted_proposal_card, + ) + + link_persisted_proposal_card( + persisted_content, + tenant_id, + user_id, + streaming_message_id, + new_unit_id, + ) + except Exception: + logger.warning( + "Failed to link persisted automation proposal card", + exc_info=True, + ) + next_unit_index += 1 await channel.publish(f"data: {chunk}\n\n") yield f"data: {chunk}\n\n" @@ -1267,15 +1374,14 @@ async def _stream_agent_chunks( user_id=user_id ) ) + _channel_cleanup_tasks.add(cleanup_task) + cleanup_task.add_done_callback(_channel_cleanup_tasks.discard) try: - skill_file_content_local = "\n".join( - json.dumps(payload, ensure_ascii=False) - for payload in captured_skill_files.values() - ) - if skill_file_content_local: + skill_file_payloads = list(captured_skill_files.values()) + if skill_file_payloads: skill_file_uploads = await _process_skill_file_uploads( - content=skill_file_content_local, + payloads=skill_file_payloads, user_id=user_id, tenant_id=tenant_id, ) @@ -1313,57 +1419,12 @@ async def _stream_agent_chunks( except Exception: logger.exception("Failed to process skill file uploads") - async def _add_memory_background(): - try: - # Skip if memory recording is disabled - if not getattr(memory_ctx.user_config, "memory_switch", False): - return - # Use the captured final answer during streaming; observer queue was drained - final_answer_local = captured_final_answer - if not final_answer_local: - return - - # Determine allowed memory levels - levels_local = {"agent", "user_agent"} - if memory_ctx.user_config.agent_share_option == "never": - levels_local.discard("agent") - if memory_ctx.agent_id in getattr(memory_ctx.user_config, "disable_agent_ids", []): - levels_local.discard("agent") - if memory_ctx.agent_id in getattr(memory_ctx.user_config, "disable_user_agent_ids", []): - levels_local.discard("user_agent") - if not levels_local: - return - - mem_messages_local = [ - {"role": MESSAGE_ROLE["USER"], - "content": agent_run_info.query}, - {"role": MESSAGE_ROLE["ASSISTANT"], - "content": final_answer_local}, - ] - - add_result_local = await add_memory_in_levels( - messages=mem_messages_local, - memory_config=memory_ctx.memory_config, - tenant_id=memory_ctx.tenant_id, - user_id=memory_ctx.user_id, - agent_id=memory_ctx.agent_id, - memory_levels=list(levels_local), - ) - items_local = add_result_local.get("results", []) - logger.info(f"Memory addition completed: {items_local}") - except Exception as bg_e: - logger.error( - f"Unexpected error during background memory addition: {bg_e}") - - try: - # Create and store the background task to avoid warnings - background_task = asyncio.create_task(_add_memory_background()) - # Add done callback to handle any exceptions that might occur - background_task.add_done_callback( - lambda t: t.exception() if t.exception() else None) - except Exception as schedule_err: - logger.error( - f"Failed to schedule background memory addition: {schedule_err}") + # Memory recording is now handled by the agent-side ``StoreMemoryTool`` + # (which delegates to the new ``MemoryService`` facade). The legacy + # background ``add_memory_in_levels`` call has been removed because + # its dual-level ``agent``/``user_agent`` semantics no longer map to + # the new layered architecture (agents may only write to + # ``agent.short_term``). def get_enable_tool_id_by_agent_id(agent_id: int, tenant_id: str): @@ -1445,9 +1506,80 @@ async def get_agent_info_impl(agent_id: int, tenant_id: str, version_no: int = 0 sub_agent_id_list = query_sub_agents_id_list( main_agent_id=agent_id, tenant_id=tenant_id) agent_info["sub_agent_id_list"] = sub_agent_id_list + + # Enrich sub-agent relations with version names (batch query) + relations = query_sub_agent_relations(agent_id, tenant_id, version_no) + enriched_relations = [] + + # Collect all agent_ids and (agent_id, version_no) pairs for batch lookup + all_agent_ids = set() + lookup_agent_ids = set() + lookup_version_nos = set() + # Track agents whose pinned version_no is null -> need to resolve latest published version + missing_version_agent_ids = set() + for rel in relations: + aid = rel.get("selected_agent_id") + if aid: + all_agent_ids.add(aid) + vno = rel.get("selected_agent_version_no") + if aid and vno is not None and vno != 0: + lookup_agent_ids.add(aid) + lookup_version_nos.add(vno) + elif aid: + # Historical data: pinned version_no is null or 0 (draft), resolve from child's current published version + missing_version_agent_ids.add(aid) + + # Batch query current published version_no for agents with missing pinned version + resolved_version_no_map: dict = {} + if missing_version_agent_ids: + resolved_version_no_map = batch_query_current_version_nos( + agent_ids=list(missing_version_agent_ids), + tenant_id=tenant_id, + ) + # Merge resolved version_nos into the version name lookup set + for aid, resolved_vno in resolved_version_no_map.items(): + lookup_agent_ids.add(aid) + lookup_version_nos.add(resolved_vno) + + # Batch query all version names at once + version_name_map: dict = {} + if lookup_agent_ids and lookup_version_nos: + batch_results = batch_search_version_names( + agent_ids=list(lookup_agent_ids), + tenant_id=tenant_id, + version_nos=list(lookup_version_nos), + ) + for item in batch_results: + key = (item["agent_id"], item["version_no"]) + version_name_map[key] = item["version_name"] + + # Batch query all agent display names at once + agent_name_map = batch_search_agent_display_names( + agent_ids=list(all_agent_ids), + tenant_id=tenant_id, + ) + + for rel in relations: + selected_agent_id = rel.get("selected_agent_id") + selected_version_no = rel.get("selected_agent_version_no") + # Fallback to resolved latest published version_no when pinned version is null or 0 (draft) + if (selected_version_no is None or selected_version_no == 0) and selected_agent_id in resolved_version_no_map: + selected_version_no = resolved_version_no_map[selected_agent_id] + version_name = None + if selected_agent_id and selected_version_no is not None: + version_name = version_name_map.get((selected_agent_id, selected_version_no)) + enriched_relations.append({ + "agent_id": selected_agent_id, + "agent_name": agent_name_map.get(selected_agent_id) if selected_agent_id else None, + "version_no": selected_version_no, + "version_name": version_name, + }) + + agent_info["sub_agent_relations"] = enriched_relations except Exception as e: logger.error(f"Failed to get sub agent id list: {str(e)}") agent_info["sub_agent_id_list"] = [] + agent_info["sub_agent_relations"] = [] try: skill_service = SkillService() @@ -1458,9 +1590,27 @@ async def get_agent_info_impl(agent_id: int, tenant_id: str, version_no: int = 0 ) # Keep disabled instances for their saved configuration, but do not # return them as selected skills in the agent configuration. - agent_info["skills"] = [ + instances = [ instance for instance in instances if instance.get("enabled", True) ] + + # Fallback: verify each instance's skill_id still exists in ag_skill_info_t + valid_skill_ids = skill_db.get_valid_skill_ids( + tenant_id=tenant_id, + skill_ids=[inst.get("skill_id") for inst in instances if isinstance(inst, dict)] + ) + filtered = [] + for inst in instances: + skill_id = inst.get("skill_id") + if skill_id in valid_skill_ids: + filtered.append(inst) + else: + logger.warning( + "Filtering out stale skill instance: agent_id=%s, skill_id=%s (not found in ag_skill_info_t)", + agent_id, skill_id, + ) + agent_info["skills"] = filtered + except Exception as e: logger.exception(f"Failed to get agent skills: {str(e)}") agent_info["skills"] = [] @@ -1581,34 +1731,40 @@ def _validate_requested_output_tokens_for_agent( if requested_output_tokens is None: return - model_id = request.model_id - if model_id is None and request.agent_id is not None: + # Validate against every configured model — the user can switch models at + # chat time, so requested_output_tokens must not exceed any model's limit. + model_ids = list(request.model_ids or []) + if not model_ids and request.agent_id is not None: try: existing_agent = search_agent_info_by_agent_id( agent_id=request.agent_id, tenant_id=tenant_id, version_no=request.version_no, ) - model_id = existing_agent.get("model_id") + model_ids = list(existing_agent.get("model_ids") or []) except Exception as exc: logger.warning( - "Could not resolve existing agent model for requested_output_tokens validation: %s", + "Could not resolve existing agent models for requested_output_tokens validation: %s", exc, ) - if model_id is None: + if not model_ids: return - model_info = get_model_by_model_id(model_id, tenant_id=tenant_id) - max_output_tokens = model_info.get("max_output_tokens") if model_info else None - if max_output_tokens is not None and requested_output_tokens > max_output_tokens: - raise AppException( - ErrorCode.COMMON_PARAMETER_INVALID, - ( - "requested_output_tokens cannot exceed the selected model " - f"max_output_tokens ({max_output_tokens})" - ), - ) + for model_id in model_ids: + model_info = get_model_by_model_id(model_id, tenant_id=tenant_id) + max_output_tokens = model_info.get("max_output_tokens") if model_info else None + if max_output_tokens is not None and requested_output_tokens > max_output_tokens: + model_display = ( + model_info.get("display_name") if model_info else f"model_id={model_id}" + ) + raise AppException( + ErrorCode.COMMON_PARAMETER_INVALID, + ( + f"requested_output_tokens ({requested_output_tokens}) cannot exceed " + f"max_output_tokens ({max_output_tokens}) of model '{model_display}'" + ), + ) async def update_agent_info_impl(request: AgentInfoRequest, authorization: str = Header(None)): @@ -1644,8 +1800,10 @@ async def update_agent_info_impl(request: AgentInfoRequest, authorization: str = "prompt_template_name": prompt_template_name, "max_steps": request.max_steps, "requested_output_tokens": request.requested_output_tokens, + "is_main_agent": request.is_main_agent if request.is_main_agent is not None else True, "provide_run_summary": request.provide_run_summary, "verification_config": request.verification_config, + "context_policy": request.context_policy, "duty_prompt": request.duty_prompt, "constraint_prompt": request.constraint_prompt, "few_shots_prompt": request.few_shots_prompt, @@ -1713,13 +1871,45 @@ async def update_agent_info_impl(request: AgentInfoRequest, authorization: str = logger.error(f"Failed to update agent tools: {str(e)}") raise ValueError(f"Failed to update agent tools: {str(e)}") - # Handle enabled skills saving when provided + # Handle enabled skills and their per-agent configuration. try: - if request.enabled_skill_ids is not None and agent_id is not None: - enabled_set = set(request.enabled_skill_ids) + requested_skill_instances = getattr(request, "skill_instances", None) + has_structured_skill_instances = isinstance(requested_skill_instances, list) + if ( + (has_structured_skill_instances or request.enabled_skill_ids is not None) + and agent_id is not None + ): + raw_version_no = getattr(request, "version_no", 0) + request_version_no = raw_version_no if isinstance(raw_version_no, int) else 0 + requested_by_id = {} + if has_structured_skill_instances: + for requested_instance in requested_skill_instances: + skill_id = requested_instance.skill_id + if skill_id in requested_by_id: + raise ValueError(f"Duplicate skill_id in skill_instances: {skill_id}") + requested_by_id[skill_id] = requested_instance + enabled_set = { + skill_id + for skill_id, requested_instance in requested_by_id.items() + if requested_instance.enabled + } + else: + enabled_set = set(request.enabled_skill_ids or []) + + if has_structured_skill_instances: + valid_skill_ids = skill_db.get_valid_skill_ids( + tenant_id=tenant_id, + skill_ids=list(enabled_set), + ) + missing_skill_ids = enabled_set - valid_skill_ids + if missing_skill_ids: + raise ValueError( + f"Invalid or unavailable skill IDs: {sorted(missing_skill_ids)}" + ) + # Query existing skill instances for this agent existing_instances = skill_db.query_skill_instances_by_agent_id( - agent_id, tenant_id) + agent_id, tenant_id, version_no=request_version_no) # Handle unselected skill (already exist instance) -> enabled=False for instance in existing_instances: @@ -1729,39 +1919,37 @@ async def update_agent_info_impl(request: AgentInfoRequest, authorization: str = skill_info=SkillInstanceInfoRequest( skill_id=inst_skill_id, agent_id=agent_id, - skill_description=instance.get( - "skill_description"), - skill_content=instance.get("skill_content"), enabled=False, config_values=instance.get("config_values"), + version_no=request_version_no, ), tenant_id=tenant_id, - user_id=user_id + user_id=user_id, + version_no=request_version_no, ) # Handle selected skill -> enabled=True (create or update) for skill_id in enabled_set: - # Keep existing skill_description and skill_content if any existing_instance = next( (inst for inst in existing_instances if inst.get("skill_id") == skill_id), None ) - skill_description = (existing_instance or {}).get( - "skill_description") - skill_content = (existing_instance or {}).get("skill_content") + if has_structured_skill_instances: + config_values = requested_by_id[skill_id].config_values + else: + config_values = (existing_instance or {}).get("config_values") skill_db.create_or_update_skill_by_skill_info( skill_info=SkillInstanceInfoRequest( skill_id=skill_id, agent_id=agent_id, - skill_description=skill_description, - skill_content=skill_content, enabled=True, - config_values=(existing_instance or {} - ).get("config_values"), + config_values=config_values, + version_no=request_version_no, ), tenant_id=tenant_id, - user_id=user_id + user_id=user_id, + version_no=request_version_no, ) except Exception as e: logger.error(f"Failed to update agent skills: {str(e)}") @@ -1788,14 +1976,25 @@ async def update_agent_info_impl(request: AgentInfoRequest, authorization: str = main_agent_id=left_ele, tenant_id=tenant_id) search_list.extend(sub_ids) - # Update related agents + # Update related agents - use related_agents if provided, otherwise build from IDs + if request.related_agents: + related_agents_dicts = [ + {"agent_id": ra.agent_id, "version_no": ra.version_no} + for ra in request.related_agents + ] + else: + related_agents_dicts = [ + {"agent_id": aid, "version_no": None} + for aid in related_agent_ids + ] + update_related_agents( parent_agent_id=agent_id, - related_agent_ids=related_agent_ids, tenant_id=tenant_id, - user_id=user_id + user_id=user_id, + related_agents=related_agents_dicts, ) - except ValueError as e: + except ValueError: # Re-raise ValueError (circular dependency) as-is raise except Exception as e: @@ -1863,63 +2062,11 @@ async def delete_agent_impl(agent_id: int, tenant_id: str, user_id: str): delete_agent_relationship(agent_id, tenant_id, user_id) delete_tools_by_agent_id(agent_id, tenant_id, user_id) skill_db.delete_skills_by_agent_id(agent_id, tenant_id, user_id) - - # Clean up all memory data related to the agent - await clear_agent_memory(agent_id, tenant_id, user_id) except Exception as e: logger.error(f"Failed to delete agent: {str(e)}") raise ValueError(f"Failed to delete agent: {str(e)}") -async def clear_agent_memory(agent_id: int, tenant_id: str, user_id: str): - """ - Purge specified agent's memory data - - Args: - agent_id: Agent ID - tenant_id: Tenant ID - user_id: User ID - """ - try: - # Build memory configuration - memory_config = build_memory_config(tenant_id) - - # Clean up agent-level memory - try: - agent_memory_result = await clear_memory( - memory_level="agent", - memory_config=memory_config, - tenant_id=tenant_id, - user_id=user_id, - agent_id=str(agent_id) - ) - logger.info( - f"Cleared agent memory for agent {agent_id}: {agent_memory_result}") - except Exception as e: - logger.error( - f"Failed to clear agent-level memory for agent {agent_id}: {str(e)}") - - # Clean up user_agent-level memory - try: - user_agent_memory_result = await clear_memory( - memory_level="user_agent", - memory_config=memory_config, - tenant_id=tenant_id, - user_id=user_id, - agent_id=str(agent_id) - ) - logger.info( - f"Cleared user_agent memory for agent {agent_id}: {user_agent_memory_result}") - except Exception as e: - logger.error( - f"Failed to clear user_agent-level memory for agent {agent_id}: {str(e)}") - - except Exception as e: - logger.error( - f"Failed to build memory config for agent {agent_id}: {str(e)}") - # Silently fail to maintain agent deletion process - - async def _export_agent_dict_core( root_agent_id: int, tenant_id: str, @@ -2175,8 +2322,10 @@ async def export_agent_by_agent_id( author=agent_info.get("author"), max_steps=agent_info["max_steps"], requested_output_tokens=agent_info.get("requested_output_tokens"), + is_main_agent=agent_info.get("is_main_agent", True), provide_run_summary=agent_info["provide_run_summary"], verification_config=agent_info.get("verification_config"), + context_policy=agent_info.get("context_policy"), duty_prompt=agent_info.get( "duty_prompt"), constraint_prompt=agent_info.get( @@ -2239,12 +2388,13 @@ async def import_agent_impl( mapping_agent_id[need_import_agent_id] = new_agent_id agent_id_set.add(need_import_agent_id) - # Establish relationships with sub-agents + # Establish relationships with sub-agents - new sub-agents always use version 1 for sub_agent_id in managed_agents: insert_related_agent(parent_agent_id=mapping_agent_id[need_import_agent_id], child_agent_id=mapping_agent_id[sub_agent_id], tenant_id=tenant_id, - user_id=user_id) + user_id=user_id, + selected_agent_version_no=1) else: # Current agent still has sub-agents that haven't been imported agent_stack.append(need_import_agent_id) @@ -2337,8 +2487,10 @@ async def import_agent_by_agent_id( "prompt_template_name": import_agent_info.prompt_template_name or SYSTEM_PROMPT_TEMPLATE_NAME, "max_steps": import_agent_info.max_steps, "requested_output_tokens": import_agent_info.requested_output_tokens, + "is_main_agent": getattr(import_agent_info, "is_main_agent", True), "provide_run_summary": import_agent_info.provide_run_summary, "verification_config": getattr(import_agent_info, "verification_config", None), + "context_policy": getattr(import_agent_info, "context_policy", None), "duty_prompt": import_agent_info.duty_prompt, "constraint_prompt": import_agent_info.constraint_prompt, "few_shots_prompt": import_agent_info.few_shots_prompt, @@ -2515,6 +2667,7 @@ async def list_all_agent_info_impl(tenant_id: str, user_id: str) -> list[dict]: "group_ids": convert_string_to_list(agent.get("group_ids")), "permission": permission, "is_published": agent.get("current_version_no") is not None, + "current_version_no": agent.get("current_version_no"), "is_a2a_server": agent["agent_id"] in a2a_server_agent_ids, }) @@ -2727,33 +2880,60 @@ async def prepare_agent_run( memory_context = build_memory_context( user_id, tenant_id, agent_request.agent_id, skip_query=not allow_memory_search) + + create_run_kwargs = { + "agent_id": agent_request.agent_id, + "minio_files": agent_request.minio_files, + "query": agent_request.query, + "history": agent_request.history, + "tenant_id": tenant_id, + "user_id": user_id, + "language": language, + "allow_memory_search": allow_memory_search, + "is_debug": agent_request.is_debug, + "override_version_no": agent_request.version_no, + "override_model_id": agent_request.model_id, + "requested_output_tokens": agent_request.requested_output_tokens, + "tool_params": agent_request.tool_params, + "conversation_id": agent_request.conversation_id, + "context_policy": agent_request.context_policy, + "enable_planning": agent_request.enable_plan, + } + if not agent_request.enable_automation_tool: + create_run_kwargs["enable_automation_tool"] = False agent_run_info = await create_agent_run_info( - agent_id=agent_request.agent_id, - minio_files=agent_request.minio_files, - query=agent_request.query, - history=agent_request.history, - tenant_id=tenant_id, - user_id=user_id, - language=language, - allow_memory_search=allow_memory_search, - is_debug=agent_request.is_debug, - override_version_no=agent_request.version_no, - override_model_id=agent_request.model_id, - requested_output_tokens=agent_request.requested_output_tokens, - tool_params=agent_request.tool_params, + **create_run_kwargs, ) - # Mount conversation-level reusable ContextManager if enabled + historical_context = None + if not agent_request.is_debug and agent_request.conversation_id is not None: + current_message_id = get_current_run_user_message_id( + agent_request.conversation_id, user_id + ) + if not isinstance(current_message_id, int) or isinstance(current_message_id, bool): + current_message_id = None + logger.warning("Current user message boundary is unavailable; historical checkpoint loading skipped") + if current_message_id is not None: + historical_context = load_historical_context( + agent_request.conversation_id, current_message_id, user_id, tenant_id + ) + agent_run_info.context_input = build_authorized_context_input( + agent_run_info, historical_context + ) + agent_run_info.conversation_id = agent_request.conversation_id + agent_run_info.user_id = user_id + + # ContextManager is created exactly once by the SDK Agent creation entry. + # The application boundary only injects the persistence callback into its + # configuration before the worker thread starts. cm_config = getattr(agent_run_info.agent_config, 'context_manager_config', None) - if cm_config and cm_config.enabled: - cm = agent_run_manager.get_or_create_context_manager( - conversation_id=str(agent_request.conversation_id), - config=cm_config, - max_steps=agent_run_info.agent_config.max_steps + if cm_config: + cm_config.history_summary_sink = ( + (lambda candidate: persist_history_summary_candidate( + agent_request.conversation_id, candidate, user_id, tenant_id + )) if historical_context is not None else None ) - agent_run_info.context_manager = cm - agent_run_manager.register_agent_run( agent_request.conversation_id, agent_run_info, user_id) return agent_run_info, memory_context @@ -2766,7 +2946,8 @@ def save_messages(agent_request, target: str, user_id: str, tenant_id: str, mess if target == MESSAGE_ROLE["USER"]: if messages is not None: raise ValueError("Messages should be None when saving for user.") - submit(save_conversation_user, agent_request, user_id, tenant_id) + # Historical checkpoint lookup for this run needs the current message boundary. + save_conversation_user(agent_request, user_id, tenant_id) return if target == MESSAGE_ROLE["ASSISTANT"]: @@ -2779,80 +2960,78 @@ def save_messages(agent_request, target: str, user_id: str, tenant_id: str, mess raise ValueError(f"Unsupported target for save_messages: {target!r}") -# Helper function for run_agent_stream, used to generate stream response with memory preprocess tokens -async def generate_stream_with_memory( +# Helper function for run_agent_stream. ``enable_memory`` controls whether +# fixed pre-run retrieval and the model-directed store_memory tool are enabled. +async def generate_stream( agent_request: AgentRequest, user_id: str, tenant_id: str, language: str = LANGUAGE["ZH"], + enable_memory: bool = False, + channel: Optional[Any] = None, ): - # Prepare preprocess task tracking (simulate preprocess flow) - task_id = str(uuid.uuid4()) - conversation_id = agent_request.conversation_id - current_task = asyncio.current_task() - if current_task: - preprocess_manager.register_preprocess_task( - task_id, conversation_id, current_task - ) + """Unified streaming entry point. + + Args: + agent_request: The agent run payload. + user_id: The caller user id. + tenant_id: The caller tenant id. + language: UI/i18n language (``"zh"`` / ``"en"``). + enable_memory: When ``True``, memory retrieval runs once before the + model loop and store_memory is loaded for model-directed use. A + ``MemoryPreparationException`` triggers a single fallback to the + no-memory path so the run still produces output. + channel: Optional streaming channel; when ``None`` a fresh channel + is created lazily when memory is enabled. + """ + # Poll for cross-pod cancel signal so the outer generator task can be + # cancelled when another Pod writes the runtime cancel flag. + _outer_task = asyncio.current_task() cancel_poll_task = ( - asyncio.create_task(_cancel_task_on_runtime_signal(conversation_id, user_id, current_task)) - if current_task + asyncio.create_task( + _cancel_task_on_runtime_signal( + agent_request.conversation_id, user_id, _outer_task + ) + ) + if _outer_task else None ) - # Helper to emit memory_search token - def _memory_token(message_text: str) -> str: - payload = { - "type": "memory_search", - "content": json.dumps({"message": message_text}, ensure_ascii=False), - } - return json.dumps(payload, ensure_ascii=False) - - # Placeholder messages handled by frontend for i18n - msg_start = MEMORY_SEARCH_START_MSG - msg_done = MEMORY_SEARCH_DONE_MSG - msg_fail = MEMORY_SEARCH_FAIL_MSG - - # ------------------------------------------------------------------ - # Note: the actual streaming happens via `_stream_agent_chunks` helper - # ------------------------------------------------------------------ - - # Create channel for multi-subscriber support - channel = await streaming_channel_manager.get_or_create_channel( - conversation_id=agent_request.conversation_id, - user_id=user_id - ) - - memory_enabled = False - try: - memory_context_preview = build_memory_context( - user_id, tenant_id, agent_request.agent_id + # Lazily open the streaming channel. Recursive fallback below needs to + # reuse the same channel so subscribers stay connected. + if channel is None and enable_memory: + channel = await streaming_channel_manager.get_or_create_channel( + conversation_id=agent_request.conversation_id, + user_id=user_id, ) - memory_enabled = bool(memory_context_preview.user_config.memory_switch) - if memory_enabled: - # Emit start token before memory retrieval - await channel.publish(f"data: {_memory_token(msg_start)}\n\n") - yield f"data: {_memory_token(msg_start)}\n\n" + memory_enabled_runtime = False + try: + if enable_memory: + # Resolve the user-level switch for tool loading only. + memory_context_preview = build_memory_context( + user_id, tenant_id, agent_request.agent_id + ) + memory_enabled_runtime = bool( + memory_context_preview.user_config.memory_switch + ) - # Prepare run (will execute memory retrieval inside create_agent_run_info) + # Prepare the agent with or without memory. The preparation path runs + # fixed retrieval before the model loop and exposes only store_memory. try: agent_run_info, memory_context = await prepare_agent_run( agent_request=agent_request, user_id=user_id, tenant_id=tenant_id, language=language, - allow_memory_search=True, + allow_memory_search=memory_enabled_runtime, ) except Exception as prep_err: - # Normalize any preparation error to MemoryPreparationException + # Normalize any preparation error to MemoryPreparationException so + # the memory-enabled path can decide between retry-without-memory + # and propagating the failure. raise MemoryPreparationException(str(prep_err)) from prep_err - if memory_enabled: - # Emit completion token once memory is ready - await channel.publish(f"data: {_memory_token(msg_done)}\n\n") - yield f"data: {_memory_token(msg_done)}\n\n" - async for data_chunk in _stream_agent_chunks( agent_request=agent_request, user_id=user_id, @@ -2864,17 +3043,24 @@ def _memory_token(message_text: str) -> str: yield data_chunk except MemoryPreparationException: - # Memory retrieval failure: emit failure token when memory is enabled, and continue without blocking - if memory_enabled: - await channel.publish(f"data: {_memory_token(msg_fail)}\n\n") - yield f"data: {_memory_token(msg_fail)}\n\n" + if not enable_memory: + # No-memory path has no fallback; surface the failure cleanly. + logger.error( + "Agent run error without memory: %r", None, exc_info=True + ) + await channel.publish(_safe_agent_stream_error_chunk()) + yield _safe_agent_stream_error_chunk() + return try: - # Fallback to the no-memory streaming path, which internally handles - async for data_chunk in generate_stream_no_memory( + # Single fallback: re-issue this generator with memory turned off + # so the actual ``_stream_agent_chunks`` still runs. + async for data_chunk in generate_stream( agent_request, user_id=user_id, tenant_id=tenant_id, + language=language, + enable_memory=False, channel=channel, ): yield data_chunk @@ -2889,7 +3075,7 @@ def _memory_token(message_text: str) -> str: return except Exception as stream_exc: logger.error( - "Generate stream with memory error: %r", + "Generate stream error: %r", stream_exc, exc_info=True, ) @@ -2899,38 +3085,6 @@ def _memory_token(message_text: str) -> str: finally: if cancel_poll_task and not cancel_poll_task.done(): cancel_poll_task.cancel() - # Always unregister preprocess task - preprocess_manager.unregister_preprocess_task(task_id) - - -# Helper function for run_agent_stream, used when user memory is disabled (no memory tokens) -async def generate_stream_no_memory( - agent_request: AgentRequest, - user_id: str, - tenant_id: str, - language: str = LANGUAGE["ZH"], - channel: Optional[Any] = None, -): - """Stream agent responses without any memory preprocessing tokens or fallback logic.""" - - # Prepare run info respecting memory disabled (honor provided user_id/tenant_id) - agent_run_info, memory_context = await prepare_agent_run( - agent_request=agent_request, - user_id=user_id, - tenant_id=tenant_id, - language=language, - allow_memory_search=False, - ) - - async for data_chunk in _stream_agent_chunks( - agent_request=agent_request, - user_id=user_id, - tenant_id=tenant_id, - agent_run_info=agent_run_info, - memory_ctx=memory_context, - channel=channel, - ): - yield data_chunk def _detect_resume_position( @@ -3045,6 +3199,7 @@ async def run_agent_stream( title=default_title, user_id=resolved_user_id, agent_id=agent_request.agent_id, + chat_mode="planning" if agent_request.enable_plan else "execution", ) agent_request.conversation_id = conversation_data["conversation_id"] is_new_conversation = True @@ -3054,6 +3209,24 @@ async def run_agent_stream( resolved_user_id, ) + if ( + not agent_request.is_debug + and not is_new_conversation + and agent_request.conversation_id is not None + ): + conversation = get_conversation_service( + conversation_id=agent_request.conversation_id, + user_id=resolved_user_id, + tenant_id=resolved_tenant_id, + ) + if conversation is None: + raise ForbiddenError("Conversation is not accessible to the current identity") + update_conversation_chat_mode_service( + conversation_id=agent_request.conversation_id, + chat_mode="planning" if agent_request.enable_plan else "execution", + user_id=resolved_user_id, + ) + if ( not agent_request.is_debug and not resume @@ -3280,20 +3453,13 @@ async def channel_stream(): use_memory_stream = memory_enabled and not agent_request.is_debug - if use_memory_stream: - stream_gen = generate_stream_with_memory( - agent_request, - user_id=resolved_user_id, - tenant_id=resolved_tenant_id, - language=language, - ) - else: - stream_gen = generate_stream_no_memory( - agent_request, - user_id=resolved_user_id, - tenant_id=resolved_tenant_id, - language=language, - ) + stream_gen = generate_stream( + agent_request, + user_id=resolved_user_id, + tenant_id=resolved_tenant_id, + language=language, + enable_memory=use_memory_stream, + ) async def stream_with_agent_context(): try: @@ -3311,29 +3477,98 @@ async def stream_with_agent_context(): exc_info=True, ) yield _safe_agent_stream_error_chunk() - finally: - # Auto-generate title for new conversations after stream completes - if is_new_conversation: - try: - await generate_conversation_title_service( - conversation_id=agent_request.conversation_id, - question=agent_request.query, - user_id=resolved_user_id, - tenant_id=resolved_tenant_id, - language=language, - ) - except Exception as title_exc: - logger.warning( - "Failed to auto-generate title for conversation_id=%s: %r", - agent_request.conversation_id, - title_exc, - ) + + headers = {"Cache-Control": "no-cache", "Connection": "keep-alive"} + if agent_request.conversation_id is not None: + headers["conversation_id"] = str(agent_request.conversation_id) return StreamingResponse( stream_with_agent_context(), media_type="text/event-stream", - headers={"Cache-Control": "no-cache", "Connection": "keep-alive"}, + headers=headers, + ) + + +async def run_agent_background( + agent_request: AgentRequest, + user_id: str, + tenant_id: str, + language: str = LANGUAGE["ZH"], + skip_user_save: bool = False, +) -> Dict[str, Any]: + """ + Run an agent without returning an SSE response. + + This path is used by background automation tasks. It reuses the same + preparation, monitoring, memory and message persistence flow as + run_agent_stream, but consumes generated chunks internally. + """ + if not agent_request.conversation_id: + raise ValueError("conversation_id is required for background agent runs") + + if not agent_request.is_debug and not skip_user_save: + save_messages( + agent_request, + target=MESSAGE_ROLE["USER"], + user_id=user_id, + tenant_id=tenant_id, + ) + + memory_ctx_preview = build_memory_context( + user_id, tenant_id, agent_request.agent_id, skip_query=agent_request.is_debug ) + memory_enabled = memory_ctx_preview.user_config.memory_switch + + agent_metadata = monitoring_manager.bind_agent_context(AgentRunMetadata( + agent_id=agent_request.agent_id, + conversation_id=agent_request.conversation_id, + user_id=user_id, + tenant_id=tenant_id, + query=agent_request.query, + is_debug=agent_request.is_debug, + language=language, + memory_enabled=memory_enabled, + history_count=len(agent_request.history) if agent_request.history else 0, + minio_files_count=len(agent_request.minio_files) if agent_request.minio_files else 0, + extra_metadata={ + "background": True, + "skip_user_save": skip_user_save, + "agent_share_option": getattr( + memory_ctx_preview.user_config, + "agent_share_option", + "unknown", + ), + }, + )) + + if memory_enabled and not agent_request.is_debug: + stream_gen = generate_stream( + agent_request, + user_id=user_id, + tenant_id=tenant_id, + language=language, + enable_memory=True, + ) + else: + stream_gen = generate_stream( + agent_request, + user_id=user_id, + tenant_id=tenant_id, + language=language, + enable_memory=False, + ) + + chunks = 0 + with agent_monitoring_context(agent_metadata): + async for _ in stream_gen: + chunks += 1 + + latest_message = get_latest_assistant_message(agent_request.conversation_id, user_id) + return { + "conversation_id": agent_request.conversation_id, + "assistant_message_id": latest_message.get("message_id") if latest_message else None, + "chunks": chunks, + } def stop_agent_tasks(conversation_id: int, user_id: str): @@ -3364,6 +3599,10 @@ def stop_agent_tasks(conversation_id: int, user_id: str): return {"status": "success", "message": message, "already_stopped": True} +def is_agent_running(conversation_id: int, user_id: str) -> bool: + return agent_run_manager.get_agent_run_info(conversation_id, user_id) is not None + + async def get_agent_id_by_name(agent_name: str, tenant_id: str) -> int: """ Resolve unique agent id by its unique name under the same tenant. @@ -3642,3 +3881,93 @@ async def import_agent_with_skills_impl( ) return agent_id_mapping + + +# ============================================================================= +# Sandbox Policy Builder +# ============================================================================= + + +def build_sandbox_policy(tenant_id: str, agent_type: str) -> Optional[dict]: + """ + Assemble a sandbox policy dict from ``NEXENT_SANDBOX_*`` environment variables. + + This is the canonical factory used by the backend service layer to resolve + ``SandboxConfig`` for every agent run. It is called before constructing + ``AgentRunInfo`` so that the resolved config flows into ``NexentAgent``. + + Resolution order: + 1. ``AgentConfig.sandbox_policy`` from the DB (takes precedence). + 2. ``NEXENT_SANDBOX_*`` environment variables (fallback when DB has no policy). + + Args: + tenant_id: tenant identifier (reserved for future per-tenant overrides). + agent_type: agent type string (reserved for future per-type overrides). + + Returns: + A sandbox policy dict, or None when ``NEXENT_SANDBOX_DEFAULT_LEVEL=local``. + """ + from consts.const import ( + NEXENT_SANDBOX_DEFAULT_LEVEL, + NEXENT_SANDBOX_DEFAULT_SCOPE, + NEXENT_SANDBOX_DOCKER_IMAGE, + NEXENT_SANDBOX_MEMORY_LIMIT_MB, + NEXENT_SANDBOX_CPU_QUOTA, + NEXENT_SANDBOX_TIMEOUT_S, + NEXENT_SANDBOX_NETWORK_DISABLED, + NEXENT_SANDBOX_SHELL_POLICY, + NEXENT_SANDBOX_AUTO_SYNC_OUTPUTS, + ) + + level = NEXENT_SANDBOX_DEFAULT_LEVEL + if level == "local": + return None + + return { + "level": level, + "scope": NEXENT_SANDBOX_DEFAULT_SCOPE, + "docker_image": NEXENT_SANDBOX_DOCKER_IMAGE, + "memory_limit_mb": NEXENT_SANDBOX_MEMORY_LIMIT_MB, + "cpu_quota": NEXENT_SANDBOX_CPU_QUOTA, + "timeout_seconds": NEXENT_SANDBOX_TIMEOUT_S, + "network_disabled": NEXENT_SANDBOX_NETWORK_DISABLED, + "shell_policy": NEXENT_SANDBOX_SHELL_POLICY, + "auto_sync_outputs": NEXENT_SANDBOX_AUTO_SYNC_OUTPUTS, + } + + +def get_sandbox_minio_client() -> Optional[Any]: + """ + Build and return a MinIO client for sandbox output sync. + + Returns None when MinIO is not configured (safe no-op). + + The caller is responsible for managing the client lifecycle — this function + returns a fresh client on each call so the caller can call ``close()`` on it + after the run finishes. + """ + from consts.const import ( + NEXENT_SANDBOX_OUTPUT_BUCKET, + MINIO_ENDPOINT, + MINIO_ACCESS_KEY, + MINIO_SECRET_KEY, + MINIO_SECURE, + ) + + if not MINIO_ENDPOINT: + return None + + try: + from nexent.storage import MinIOStorageClient + except ImportError: + return None + + client = MinIOStorageClient( + endpoint=MINIO_ENDPOINT, + access_key=MINIO_ACCESS_KEY or "", + secret_key=MINIO_SECRET_KEY or "", + region=None, + default_bucket=NEXENT_SANDBOX_OUTPUT_BUCKET, + secure=MINIO_SECURE, + ) + return client diff --git a/backend/services/agent_version_service.py b/backend/services/agent_version_service.py index 1bc91d3335..685d8f89af 100644 --- a/backend/services/agent_version_service.py +++ b/backend/services/agent_version_service.py @@ -825,7 +825,7 @@ async def list_published_agents_impl( _apply_duplicate_name_availability_rules, ) from services.asset_owner_visibility import resolve_agent_list_permission - from database.agent_version_db import query_agent_snapshot + from database.agent_version_db import query_agent_snapshot, query_version_list # Get user role for permission check user_tenant_record = get_user_tenant_by_user_id(user_id) or {} @@ -868,6 +868,21 @@ async def list_published_agents_impl( if not current_version_no or current_version_no <= 0: continue + # Verify current_version_no exists, if not find the latest available version + available_versions = query_version_list(agent_id=agent_id, tenant_id=tenant_id) + + if not available_versions: + logger.warning(f"No available versions found for agent_id={agent_id}") + continue + + available_version_nos = {v["version_no"] for v in available_versions} + + if current_version_no not in available_version_nos: + logger.warning( + f"Current version {current_version_no} not found for agent_id={agent_id}, using latest available version" + ) + current_version_no = available_versions[0]["version_no"] + # Get the published version snapshot agent_snapshot, tools_snapshot, relations_snapshot = query_agent_snapshot( agent_id=agent_id, @@ -889,6 +904,11 @@ async def list_published_agents_impl( if key != 'current_version_no': agent_info[key] = value + # Add version_name from version metadata + current_version_info = next((v for v in available_versions if v["version_no"] == current_version_no), None) + if current_version_info: + agent_info['version_name'] = current_version_info.get("version_name") + # Add tools agent_info['tools'] = tools_snapshot @@ -962,9 +982,11 @@ async def list_published_agents_impl( "is_available": len(unavailable_reasons) == 0, "unavailable_reasons": unavailable_reasons, "is_new": agent.get("is_new", False), + "is_main_agent": agent.get("is_main_agent", True), "group_ids": agent.get("group_ids", []), "permission": permission, "current_version_no": agent.get("current_version_no"), + "version_name": agent.get("version_name"), "greeting_message": agent.get("greeting_message"), "example_questions": agent.get("example_questions"), }) diff --git a/backend/services/aidp_service.py b/backend/services/aidp_service.py deleted file mode 100644 index d92f770c6a..0000000000 --- a/backend/services/aidp_service.py +++ /dev/null @@ -1,249 +0,0 @@ -""" -AIDP Service Layer -Handles API calls to AIDP for paginated knowledge base listing. -""" -import logging -from typing import Any, Dict, List -from urllib.parse import urljoin - -import httpx - -from consts.error_code import ErrorCode -from consts.exceptions import AppException -from nexent.utils.http_client_manager import http_client_manager - -logger = logging.getLogger("aidp_service") - -_LIST_PATH = "/KnowledgeBase/Tenants/aidp/KnowledgeBases" - - -def _validate_params(server_url: str, api_key: str) -> str: - """Validate parameters and return normalized base URL.""" - if not server_url or not isinstance(server_url, str): - raise AppException( - ErrorCode.AIDP_CONFIG_INVALID, - "AIDP server_url is required and must be a non-empty string", - ) - if not server_url.startswith(("http://", "https://")): - raise AppException( - ErrorCode.AIDP_CONFIG_INVALID, - "AIDP server_url must start with http:// or https://", - ) - if not api_key or not isinstance(api_key, str): - raise AppException( - ErrorCode.AIDP_CONFIG_INVALID, - "AIDP api_key is required and must be a non-empty string", - ) - return server_url.rstrip("/") - - -def fetch_aidp_knowledge_bases_impl( - server_url: str, - api_key: str, - page: int = 1, - page_size: int = 10, -) -> Dict[str, Any]: - """Fetch a single page from AIDP API (simple passthrough).""" - normalized_url = _validate_params(server_url, api_key) - - headers = { - "Authorization": f"Bearer {api_key}", - "Content-Type": "application/json", - } - - list_path = f"{_LIST_PATH}?page={page}&page_size={page_size}" - list_url = urljoin(f"{normalized_url}/", list_path) - logger.info("Fetching AIDP knowledge bases from %s", list_url) - - try: - client = http_client_manager.get_sync_client( - base_url=normalized_url, - timeout=60.0, - verify_ssl=False, - ) - response = client.get(list_url, headers=headers) - response.raise_for_status() - result = response.json() - if not isinstance(result, dict): - raise AppException( - ErrorCode.AIDP_SERVICE_ERROR, - "Unexpected AIDP knowledge base response format", - ) - return _normalize_response(result) - except httpx.RequestError as e: - logger.exception("AIDP request failed: %s", e) - raise AppException( - ErrorCode.AIDP_CONNECTION_ERROR, - f"AIDP API request failed: {str(e)}", - ) - except httpx.HTTPStatusError as e: - logger.exception( - "AIDP API HTTP error: %s, status_code: %s", - e, - e.response.status_code, - ) - if e.response.status_code in (401, 403): - raise AppException( - ErrorCode.AIDP_AUTH_ERROR, - f"AIDP authentication failed: {str(e)}", - ) - raise AppException( - ErrorCode.AIDP_SERVICE_ERROR, - f"AIDP API HTTP error {e.response.status_code}: {str(e)}", - ) - except ValueError as e: - logger.exception("Failed to parse AIDP API response: %s", e) - raise AppException( - ErrorCode.AIDP_SERVICE_ERROR, - f"Failed to parse AIDP API response: {str(e)}", - ) - - -def _normalize_response(raw: Dict[str, Any]) -> Dict[str, Any]: - """Map AIDP API response fields to the canonical {value, total_count, next_link} shape.""" - items = ( - raw.get("value") - if raw.get("value") is not None - else raw.get("data") - if raw.get("data") is not None - else raw.get("items") - if raw.get("items") is not None - else raw.get("knowledge_bases") - if raw.get("knowledge_bases") is not None - else [] - ) - total_keys = ("total_count", "total", "totalRecords", "count") - total = next((raw.get(k) for k in total_keys if raw.get(k) is not None), None) - next_link = raw.get("next_link") or raw.get("next") or None - return { - "value": items, - "total_count": total, - "next_link": next_link, - } - - -def _extract_tenant_from_url(url: str) -> str | None: - """Extract tenant ID from a URL like /KnowledgeBase/Tenants/{tenant}/KnowledgeBases.""" - import re - match = re.search(r"/Tenants/([^/]+)/", url) - return match.group(1) if match else None - - -def fetch_all_aidp_knowledge_bases_impl( - server_url: str, - api_key: str, -) -> Dict[str, Any]: - """Fetch all knowledge bases from AIDP by following next_link until exhausted. - - AIDP does not return a true total count, so we follow next_link pages - until there is no next_link left. We also detect the real tenant ID - from the first response's next_link (AIDP embeds it there) and use it - for any manual page construction needed. - """ - normalized_url = _validate_params(server_url, api_key) - - headers = { - "Authorization": f"Bearer {api_key}", - "Content-Type": "application/json", - } - - try: - client = http_client_manager.get_sync_client( - base_url=normalized_url, - timeout=120.0, - verify_ssl=False, - ) - - all_items: List[Any] = [] - current_page = 1 - max_pages = 1000 - page_size = 100 - detected_tenant: str | None = None - - # Build the first request URL using the known path pattern - first_path = f"{_LIST_PATH}?page=1&page_size={page_size}" - current_url: str | None = urljoin(f"{normalized_url}/", first_path) - - while current_page <= max_pages and current_url: - logger.info( - "Fetching AIDP KBs — page %d from %s", - current_page, - current_url, - ) - - response = client.get(current_url, headers=headers) - response.raise_for_status() - result = response.json() - if not isinstance(result, dict): - raise AppException( - ErrorCode.AIDP_SERVICE_ERROR, - "Unexpected AIDP knowledge base response format", - ) - - page_items = ( - result.get("value") - if result.get("value") is not None - else result.get("data") - if result.get("data") is not None - else result.get("items") - if result.get("items") is not None - else result.get("knowledge_bases") - if result.get("knowledge_bases") is not None - else [] - ) - if not isinstance(page_items, list): - page_items = [] - - all_items.extend(page_items) - - # Detect real tenant from next_link on the first page - if current_page == 1 and detected_tenant is None: - raw_next = result.get("next_link") or result.get("next") or "" - detected_tenant = _extract_tenant_from_url(str(raw_next)) - if detected_tenant: - logger.info("Detected AIDP tenant: %s", detected_tenant) - - # Follow next_link if present, otherwise construct next page manually - raw_next = result.get("next_link") or result.get("next") or "" - next_url_str = str(raw_next).strip() - if next_url_str: - current_url = urljoin(normalized_url + "/", next_url_str) - current_page += 1 - else: - current_url = None - - total_count = len(all_items) - logger.info("AIDP KBs: accumulated %d total items (tenant=%s)", total_count, detected_tenant) - - return { - "value": all_items, - "total_count": total_count, - "next_link": None, - } - except httpx.RequestError as e: - logger.exception("AIDP request failed: %s", e) - raise AppException( - ErrorCode.AIDP_CONNECTION_ERROR, - f"AIDP API request failed: {str(e)}", - ) - except httpx.HTTPStatusError as e: - logger.exception( - "AIDP API HTTP error: %s, status_code: %s", - e, - e.response.status_code, - ) - if e.response.status_code in (401, 403): - raise AppException( - ErrorCode.AIDP_AUTH_ERROR, - f"AIDP authentication failed: {str(e)}", - ) - raise AppException( - ErrorCode.AIDP_SERVICE_ERROR, - f"AIDP API HTTP error {e.response.status_code}: {str(e)}", - ) - except ValueError as e: - logger.exception("Failed to parse AIDP API response: %s", e) - raise AppException( - ErrorCode.AIDP_SERVICE_ERROR, - f"Failed to parse AIDP API response: {str(e)}", - ) diff --git a/backend/services/conversation_management_service.py b/backend/services/conversation_management_service.py index 772d8d31a9..3aee9a34c8 100644 --- a/backend/services/conversation_management_service.py +++ b/backend/services/conversation_management_service.py @@ -10,6 +10,7 @@ from consts.model import AgentRequest, MessageRequest, MessageUnit from consts.exceptions import ConversationNotFoundError from database.conversation_db import ( + CHAT_MODE_VALUES, create_conversation, create_conversation_message, create_message_unit, @@ -18,17 +19,21 @@ delete_conversation, get_conversation, get_conversation_history, + get_historical_context, get_conversation_list, - get_latest_assistant_message, + get_latest_assistant_message, # noqa: F401 - service boundary re-export get_latest_assistant_message_id, - get_last_unit_for_message, + get_latest_user_message_id, + get_last_unit_for_message, # noqa: F401 - service boundary re-export get_message_id_by_index, get_source_images_by_conversation, get_source_images_by_message, get_source_searches_by_conversation, get_source_searches_by_message, rename_conversation, + save_history_summary, update_conversation_agent_id, + update_conversation_chat_mode, update_conversation_message_content, update_conversation_message_status, update_message_minio_files, @@ -36,10 +41,8 @@ update_message_unit_content, update_message_unit_status, ) -from nexent.core.utils.observer import MessageObserver, ProcessType from nexent.monitor import set_monitoring_context, set_monitoring_operation from nexent.core.models import OpenAIModel -from agents.agent_run_manager import agent_run_manager from utils.config_utils import get_model_name_from_config, tenant_config_manager from utils.prompt_template_utils import get_generate_title_prompt_template from utils.str_utils import remove_think_blocks @@ -103,9 +106,11 @@ def save_message(request: MessageRequest, user_id: str, tenant_id: str, def save_message_unit(message_id: int, conversation_id: int, unit_index: int, - unit_type: str, unit_content: str, + unit_type: str, unit_content: Any, user_id: Optional[str] = None, - unit_status: str = 'completed') -> int: + unit_status: str = 'completed', + tool_call_id: Optional[str] = None, + invocation_id: Optional[str] = None) -> int: """ Insert exactly one ConversationMessageUnit row. @@ -117,6 +122,11 @@ def save_message_unit(message_id: int, conversation_id: int, unit_index: int, unit_content: Complete content of the unit user_id: Identifier of the user creating the unit unit_status: Lifecycle status (streaming / completed) + tool_call_id: Unique ID of the originating tool invocation. None for + units that are not tied to a specific tool call. + invocation_id: Identifies which sub-agent invocation produced this unit. + Used by the frontend history adapter to route deep-thinking / + reasoning chunks into the correct nested sub-agent card. Returns: int: Newly created unit_id @@ -129,9 +139,48 @@ def save_message_unit(message_id: int, conversation_id: int, unit_index: int, unit_content=unit_content, user_id=user_id, unit_status=unit_status, + tool_call_id=tool_call_id, + invocation_id=invocation_id, ) +def persist_history_summary_candidate( + conversation_id: int, candidate: Any, user_id: str, tenant_id: str, +) -> int: + """Backend persistence boundary injected into the SDK context runtime.""" + def field(name: str, default: Any = None) -> Any: + return candidate.get(name, default) if isinstance(candidate, dict) \ + else getattr(candidate, name, default) + + return save_history_summary( + conversation_id=conversation_id, + user_id=user_id, + tenant_id=tenant_id, + summary=field("summary"), + covered_through_message_id=field("covered_through_message_id"), + previous_summary_unit_id=field("previous_summary_unit_id"), + trigger=field("trigger"), + ) + + +def load_historical_context( + conversation_id: int, current_user_message_id: int, + user_id: str, tenant_id: str, +) -> Optional[Dict[str, Any]]: + """Load only the authorized checkpoint and completed turns needed by SDK.""" + return get_historical_context( + conversation_id=conversation_id, + current_user_message_id=current_user_message_id, + user_id=user_id, + tenant_id=tenant_id, + ) + + +def get_current_run_user_message_id(conversation_id: int, user_id: str) -> Optional[int]: + """Resolve the persisted boundary for the run that was just saved.""" + return get_latest_user_message_id(conversation_id, user_id) + + def update_message_status(message_id: int, status: str, user_id: str) -> None: """Update the lifecycle status of a conversation message.""" update_conversation_message_status(message_id, status, user_id=user_id) @@ -291,7 +340,12 @@ def update_conversation_title(conversation_id: int, title: str, user_id: str = N return success -def create_new_conversation(title: str, user_id: str, agent_id: Optional[int] = None) -> Dict[str, Any]: +def create_new_conversation( + title: str, + user_id: str, + agent_id: Optional[int] = None, + chat_mode: Optional[str] = None, +) -> Dict[str, Any]: """ Create a new conversation @@ -299,12 +353,18 @@ def create_new_conversation(title: str, user_id: str, agent_id: Optional[int] = title: Conversation title user_id: User ID agent_id: Agent used by the latest run in this conversation + chat_mode: Initial UI chat mode Returns: Dict containing conversation data """ try: - conversation_data = create_conversation(title, user_id, agent_id=agent_id) + conversation_data = create_conversation( + title, + user_id, + agent_id=agent_id, + chat_mode=chat_mode, + ) return conversation_data except Exception as e: logging.error(f"Failed to create conversation: {str(e)}") @@ -326,6 +386,19 @@ def get_conversation_list_service(user_id: str) -> List[Dict[str, Any]]: raise Exception(str(e)) +def get_conversation_service( + conversation_id: int, + user_id: str, + tenant_id: Optional[str] = None, +) -> Optional[Dict[str, Any]]: + """Return a conversation only within the requesting user and tenant scope.""" + return get_conversation( + conversation_id=conversation_id, + user_id=user_id, + tenant_id=tenant_id, + ) + + def update_conversation_agent_id_service(conversation_id: int, agent_id: int, user_id: str) -> bool: """ Update the latest agent associated with a conversation. @@ -340,6 +413,40 @@ def update_conversation_agent_id_service(conversation_id: int, agent_id: int, us raise Exception(str(e)) +def update_conversation_chat_mode_service( + conversation_id: int, + chat_mode: str, + user_id: str, +) -> bool: + """ + Persist the UI chat mode (planning / execution) for a conversation. + + The frontend calls this whenever the user toggles the mode so that + switching back to the conversation later can restore the same toggle + without re-inferring it from stored message units. + """ + if chat_mode not in CHAT_MODE_VALUES: + raise ValueError( + f"Invalid chat_mode '{chat_mode}'. Allowed values: {sorted(CHAT_MODE_VALUES)}" + ) + try: + success = update_conversation_chat_mode( + conversation_id=conversation_id, + chat_mode=chat_mode, + user_id=user_id, + ) + if not success: + raise Exception( + f"Conversation {conversation_id} does not exist or has been deleted" + ) + return True + except ValueError: + raise + except Exception as e: + logging.error(f"Failed to update conversation chat mode: {str(e)}") + raise Exception(str(e)) + + def rename_conversation_service(conversation_id: int, name: str, user_id: str) -> bool: """ Rename a conversation @@ -374,14 +481,20 @@ def delete_conversation_service(conversation_id: int, user_id: str) -> bool: bool: Whether the deletion was successful """ try: + try: + from services.agent_automation.facade import agent_automation_facade + agent_automation_facade.on_conversation_deleted(conversation_id, user_id) + except Exception as automation_error: + logging.warning( + "Failed to cleanup automation task for conversation %s: %s", + conversation_id, + automation_error, + ) + success = delete_conversation(conversation_id, user_id) if not success: raise Exception(f"Conversation {conversation_id} does not exist or has been deleted") - # Defensive cleanup: release the ContextManager associated with this conversation - # to avoid memory leaks in edge cases - agent_run_manager.clear_conversation_context_manager(conversation_id) - return True except Exception as e: logging.error(f"Failed to delete conversation: {str(e)}") @@ -516,6 +629,27 @@ def get_conversation_history_service(conversation_id: int, user_id: str) -> List unit_type = unit.get('unit_type') unit_content = unit.get('unit_content') + if unit_type == 'history_summary': + try: + summary_payload = json.loads(unit_content) + covered_message_id = int( + summary_payload['covered_through_message_id']) + except (KeyError, TypeError, ValueError, json.JSONDecodeError): + logger.warning( + "Skipping invalid history summary unit_id=%s", + unit_id, + ) + continue + if covered_message_id != int(message_id): + logger.warning( + "Skipping misplaced history summary unit_id=%s " + "message_id=%s coverage=%s", + unit_id, + message_id, + covered_message_id, + ) + continue + if unit_type == 'search_content_placeholder' and unit_id: placeholder_content = { "placeholder": True, @@ -523,13 +657,33 @@ def get_conversation_history_service(conversation_id: int, user_id: str) -> List } processed_units.append({ 'type': 'search_content_placeholder', - 'content': json.dumps(placeholder_content, ensure_ascii=False) + 'content': json.dumps(placeholder_content, ensure_ascii=False), + 'unit_index': unit.get('unit_index'), + 'unit_status': unit.get('unit_status'), + 'tool_call_id': unit.get('tool_call_id'), + 'invocation_id': unit.get('invocation_id'), }) else: - processed_units.append({ + processed_unit = { 'type': unit_type, - 'content': unit_content - }) + 'content': unit_content, + 'unit_index': unit.get('unit_index'), + 'unit_status': unit.get('unit_status'), + 'tool_call_id': unit.get('tool_call_id'), + 'invocation_id': unit.get('invocation_id'), + } + if unit_type in ('tool', 'tool-call') and isinstance(unit_content, str): + try: + tool_data = json.loads(unit_content) + except (json.JSONDecodeError, TypeError): + tool_data = None + if isinstance(tool_data, dict) and 'content' in tool_data: + processed_unit['content'] = tool_data.get('content', '') + processed_unit['tool_name'] = tool_data.get('tool_name') + processed_unit['tool_arguments'] = tool_data.get('tool_arguments') + if 'role' in tool_data: + processed_unit['role'] = tool_data['role'] + processed_units.append(processed_unit) # Add final_answer type message unit only if not already present has_final_answer = any(u.get('type') == 'final_answer' for u in processed_units) @@ -550,6 +704,11 @@ def get_conversation_history_service(conversation_id: int, user_id: str) -> List if 'minio_files' in msg and msg['minio_files']: message_item['minio_files'] = msg['minio_files'] + # Keep the logical message position so clients can distinguish a + # regenerated branch from a separate turn with identical text. + if msg.get('message_index') is not None: + message_item['message_index'] = msg['message_index'] + # Add image content (if any) if message_id in image_by_message: message_item['picture'] = image_by_message[message_id] @@ -577,6 +736,7 @@ def get_conversation_history_service(conversation_id: int, user_id: str) -> List # Convert to string 'conversation_id': str(history_data['conversation_id']), 'agent_id': history_data.get('agent_id'), + 'chat_mode': history_data.get('chat_mode') or 'execution', 'create_time': history_data['create_time'], 'message': messages } @@ -792,7 +952,7 @@ def save_skill_files_to_conversation( conversation_id, ) return success - except Exception as exc: + except Exception: logging.exception( "[skill-file] failed to persist skill file uploads for conversation=%s", conversation_id, diff --git a/backend/services/conversation_share_service.py b/backend/services/conversation_share_service.py index 8beb4ca680..6da6e9af00 100644 --- a/backend/services/conversation_share_service.py +++ b/backend/services/conversation_share_service.py @@ -301,6 +301,7 @@ def create_share_snapshot_service( mode: str = "selected", selected_user_message_ids: Optional[List[int]] = None, expire_time: Optional[datetime] = None, + render_version: str = "legacy", ) -> Dict[str, Any]: conversation = get_conversation(conversation_id, user_id) if not conversation: @@ -316,6 +317,9 @@ def create_share_snapshot_service( messages = _select_message_pairs(messages, selected_user_message_ids) snapshot["message"] = messages snapshot["conversation_title"] = conversation.get("conversation_title") or "" + snapshot["share_render_version"] = ( + "newchat" if render_version == "newchat" else "legacy" + ) share_token = _new_token() asset_map: Dict[str, Dict[str, Any]] = {} @@ -345,6 +349,7 @@ def create_share_snapshot_service( "conversation_id": conversation_id, "title": share_record.get("title") or "", "asset_count": len(persisted_assets), + "render_version": snapshot["share_render_version"], } @@ -352,12 +357,14 @@ def get_share_snapshot_service(share_token: str) -> Dict[str, Any]: share = get_active_conversation_share(share_token) if not share: raise ValueError("Share not found or expired") + snapshot = share.get("snapshot_json") or {} return { "share_id": share_token, "title": share.get("title") or "", "conversation_id": share.get("conversation_id"), "create_time": share.get("create_time"), - "snapshot": share.get("snapshot_json"), + "render_version": snapshot.get("share_render_version", "legacy"), + "snapshot": snapshot, } diff --git a/backend/services/file_management_service.py b/backend/services/file_management_service.py index 64f7ac4860..84edf48174 100644 --- a/backend/services/file_management_service.py +++ b/backend/services/file_management_service.py @@ -19,7 +19,7 @@ OFFICE_MIME_TYPES, UPLOAD_FOLDER, ) -from consts.exceptions import FileTooLargeException, NotFoundException, OfficeConversionException, UnsupportedFileTypeException +from consts.exceptions import FileTooLargeException, NotFoundException, OfficeConversionException, QuotaExceededError, UnsupportedFileTypeException from database.attachment_db import ( copy_file, delete_file, @@ -259,6 +259,24 @@ def validate_urls_access( validate_s3_url_access(object_name, user_id, caller_tenant_id) +class UploadFilesResult(tuple): + """Backward-compatible three-item upload result with optional quota metadata.""" + + quota_status: Optional[dict] + + def __new__( + cls, + errors: list, + uploaded_file_paths: list, + uploaded_filenames: list, + quota_status: Optional[dict] = None, + ): + result = super().__new__( + cls, (errors, uploaded_file_paths, uploaded_filenames)) + result.quota_status = quota_status + return result + + async def upload_files_impl( destination: str, file: List[UploadFile], @@ -279,11 +297,12 @@ async def upload_files_impl( uploader_tenant_id: Uploader tenant ID (ASSET_OWNER uses dedicated prefix) Returns: - tuple: (errors, uploaded_file_paths, uploaded_filenames) + UploadFilesResult: Three-item tuple-compatible result with quota metadata """ uploaded_filenames = [] uploaded_file_paths = [] errors = [] + quota_status = None if destination == "local": async with upload_semaphore: for f in file: @@ -305,6 +324,23 @@ async def upload_files_impl( elif destination == "minio": actual_folder = resolve_minio_upload_folder( folder, user_id, uploader_tenant_id) + + # Pre-write quota check: compute total file sizes and check against tenant hard limit + total_file_size = 0 + if uploader_tenant_id: + for f in file: + file_size = getattr(f, "size", 0) if f else 0 + if isinstance(file_size, int) and file_size > 0: + total_file_size += file_size + if total_file_size > 0: + from services.quota_service import QuotaService + try: + quota_service = QuotaService(uploader_tenant_id, user_id) + quota_status = quota_service.check_hard_limit( + total_file_size, index_name=index_name) + except QuotaExceededError: + raise # Re-raise to be handled by caller (HTTP 413) + minio_results = await upload_to_minio(files=file, folder=actual_folder) for result in minio_results: if result.get("success"): @@ -355,7 +391,29 @@ def make_unique_names(original_names: List[str], taken_lower: set) -> List[str]: f"Failed to resolve filename conflicts for index '{index_name}': {str(e)}") else: raise Exception("Invalid destination. Must be 'local' or 'minio'.") - return errors, uploaded_file_paths, uploaded_filenames + + # Post-write belt-and-suspenders check for minio uploads (race condition handling) + if destination == "minio" and uploader_tenant_id and uploaded_file_paths: + try: + from services.quota_service import QuotaService + quota_service = QuotaService(uploader_tenant_id, user_id) + quota_status = quota_service.check_hard_limit_post_write( + 0, index_name=index_name) + except QuotaExceededError: + # Clean up uploaded files from MinIO on race condition + from database.attachment_db import delete_file + for object_name in uploaded_file_paths: + try: + delete_file(object_name=object_name) + except Exception as cleanup_err: + logger.error( + "Failed to clean up MinIO file %s after quota exceeded: %s", + object_name, cleanup_err, + ) + raise + + return UploadFilesResult( + errors, uploaded_file_paths, uploaded_filenames, quota_status) async def upload_to_minio( diff --git a/backend/services/image_service.py b/backend/services/image_service.py index 76790dc236..cc7157de4b 100644 --- a/backend/services/image_service.py +++ b/backend/services/image_service.py @@ -8,7 +8,7 @@ import aiohttp -from consts.const import DATA_PROCESS_SERVICE +from consts.const import AIDP_API_KEY, AIDP_SERVER_URL, DATA_PROCESS_SERVICE from consts.const import MODEL_CONFIG_MAPPING from database.model_management_db import get_model_by_model_id from utils.config_utils import tenant_config_manager, get_model_name_from_config @@ -19,6 +19,147 @@ logger = logging.getLogger("image_service") +# --------------------------------------------------------------------------- +# AIDP image proxying +# --------------------------------------------------------------------------- +# AIDP serves images behind GET endpoints that require ``Authorization: +# Bearer ``. The chunk-level URLs built by AidpSearchTool +# look like ``{AIDP_SERVER_URL}/KnowledgeBase/Tenants/{tenant}/KnowledgeBases/...``. +# When the image proxy sees such a URL, we short-circuit the generic +# data-processing proxy (which would not know how to authenticate) and +# fetch the image ourselves with the configured API key. +# +# The host and path checks prevent the proxy from forwarding the credential to +# an unrelated URL. +_AIDP_ALLOWED_PATH_PREFIX = "/KnowledgeBase/Tenants/" + + +def _validate_and_reconstruct_aidp_url(decoded_url: str) -> Optional[str]: + """Validate and reconstruct an AIDP image URL, returning a fresh string. + + The target authority is always taken from ``AIDP_SERVER_URL``. Only a + validated AIDP image path is retained from the supplied URL, so a client + can never cause the proxy to send the Bearer token to another host. + + Returns ``None`` if any check fails. The returned string is a + freshly reconstructed URL via ``urlunparse``, which static + analyzers (CodeQL) recognise as an SSRF sanitizer — breaking the + dataflow link from the input parameter to the subsequent + ``aiohttp.ClientSession.get`` sink. + """ + aidp_base = AIDP_SERVER_URL.rstrip("/") + if not aidp_base: + return None + + try: + parsed = urlparse(decoded_url) + base_parsed = urlparse(aidp_base) + except Exception: + return None + + if parsed.scheme not in ("http", "https") or not parsed.netloc: + return None + if base_parsed.scheme not in ("http", "https") or not base_parsed.netloc: + return None + + # Only permit the AIDP knowledge-base image API path. Reject traversal + # and non-image endpoints. + path = parsed.path + if not path.startswith(_AIDP_ALLOWED_PATH_PREFIX): + return None + if "/../" in path or path.endswith("/.."): + return None + + # Reject query/fragment to block redirect-based SSRF. + if parsed.query or parsed.fragment: + return None + + # Re-serialize the URL from its validated components. The round-trip + # through ``urlunparse`` produces a new string that is not + # alias-equivalent to the input in CodeQL's dataflow graph. + return urlunparse(( + base_parsed.scheme, + base_parsed.netloc, + path, + "", # params — dropped; validated above implicitly by being empty + "", # query — rejected above + "", # fragment — rejected above + )) + + +def _is_aidp_url(decoded_url: str) -> bool: + """Return True when ``decoded_url`` has an allowed AIDP image path. + + The final request URL is rebuilt with the configured AIDP authority, so + a host alias in the input never receives the AIDP Bearer token. + """ + return _validate_and_reconstruct_aidp_url(decoded_url) is not None + + +def _get_aidp_api_key() -> str: + return AIDP_API_KEY + + +async def _fetch_aidp_image(url: str): + """Fetch an AIDP image using the env-supplied Bearer token. + + Mirrors :func:`_fetch_image_directly` in shape but (a) adds the + ``Authorization`` header and (b) disables redirects to prevent the + Bearer token from leaking if AIDP responds with a 30x to another + host. ``trust_env`` is off so proxy environment variables do not + re-route the internal request. + + Security: this function performs its own defensive URL validation via + :func:`_validate_and_reconstruct_aidp_url`, which enforces the allowed + path + query/fragment rules, replaces the input authority with the + configured AIDP authority, and **re-serializes** the URL from its parsed + components. That re-serialization is what CodeQL recognises + as an SSRF sanitizer — a plain bool check (``not _is_aidp_url(url)``) + is not enough because the dataflow graph still considers ``url`` + user-controlled up to the ``session.get`` sink. + """ + # Defensive SSRF guard: reconstruct at the point of use. The caller + # (proxy_image_impl) already gates on _is_aidp_url, but duplicating + # the reconstruction here means even a future caller cannot + # accidentally send the Bearer token to an arbitrary host. The fresh + # ``safe_url`` value breaks the dataflow link from the input + # parameter to the session.get sink. + safe_url = _validate_and_reconstruct_aidp_url(url) + if safe_url is None: + logger.error("Rejecting non-AIDP URL in AIDP image fetch: %r", url) + return {"success": False, "error": "URL does not match configured AIDP host or KB path"} + + api_key = _get_aidp_api_key() + if not api_key: + logger.error("AIDP_API_KEY is not configured; cannot fetch AIDP image") + return {"success": False, "error": "AIDP API key not configured"} + + timeout = aiohttp.ClientTimeout(total=30) + async with aiohttp.ClientSession(timeout=timeout, trust_env=False) as session: + async with session.get( + safe_url, + headers={"Authorization": f"Bearer {api_key}"}, + allow_redirects=False, + ssl=False, # Disable SSL verification because AIDP servers use self-signed certificates + ) as response: + if response.status != HTTPStatus.OK: + error_text = await response.text() + logger.error( + "Failed to fetch AIDP image (status=%s): %s", + response.status, + error_text[:200], + ) + return {"success": False, "error": "Failed to fetch AIDP image"} + + content = await response.read() + content_type = response.headers.get("Content-Type", "image/jpeg") + return { + "success": True, + "base64": base64.b64encode(content).decode("utf-8"), + "content_type": content_type, + } + + def _validate_loopback_url(decoded_url: str) -> str | None: """Validate that ``decoded_url`` is a genuine loopback URL and return a rewritten URL whose host is a literal IPv4 loopback address, or ``None`` @@ -123,11 +264,19 @@ async def _fetch_image_directly(safe_url: str): async def proxy_image_impl(decoded_url: str): - # Fast path: only for loopback URLs, fetch directly. This avoids an - # extra hop through the data-processing service for local images. For - # any other URL (including all external/knowledge-base images such as - # AIDP), fall back to the data-processing service proxy, which is the - # existing safe path that CodeQL does not flag. + # Fast path #1: AIDP image URLs need a Bearer token. Short-circuit here + # before the loopback check because the AIDP host may happen to resolve + # to a loopback address in dev, and we'd skip the auth header if that + # branch matched first. + if _is_aidp_url(decoded_url): + return await _fetch_aidp_image(decoded_url) + + # Fast path #2: loopback URLs (in-process / local dev), fetch directly. + # This avoids an extra hop through the data-processing service for + # local images. For any other URL (including all external / knowledge- + # base images such as AIDP from a different deployment), fall back to + # the data-process service proxy, which is the existing safe path + # that CodeQL does not flag. safe_url = _validate_loopback_url(decoded_url) if safe_url is not None: return await _fetch_image_directly(safe_url) diff --git a/backend/services/mcp_management_service.py b/backend/services/mcp_management_service.py index e67d265cce..04e08a6c70 100644 --- a/backend/services/mcp_management_service.py +++ b/backend/services/mcp_management_service.py @@ -5,6 +5,7 @@ import aiohttp +from consts.const import CAN_EDIT_ALL_USER_ROLES from consts.exceptions import ( MCPConnectionError, McpNameConflictError, @@ -19,6 +20,10 @@ STATUS_SHARED, VALID_MARKET_STATUSES, ) +from consts.notification import ( + EVENT_TYPE_REPOSITORY_REVIEW_PENDING, + RESOURCE_TYPE_MCP_REPOSITORY, +) from database.market_mcp_db import ( check_mcp_market_name_exists, create_mcp_market_record, @@ -36,8 +41,16 @@ clear_mcp_record_market_id, get_mcp_record_by_id_and_tenant, update_mcp_record_market_id_by_id, + update_mcp_record_manage_fields_by_id, ) from database.user_tenant_db import get_user_tenant_by_user_id +from database.group_db import query_group_ids_by_user +from services.notification_service import ( + create_repository_pending_review_notification, + create_repository_review_notification, + deactivate_notifications, +) +from utils.str_utils import convert_list_to_string, convert_string_to_list logger = logging.getLogger("mcp_management_service") @@ -110,13 +123,33 @@ def _to_community_card(row: Dict[str, Any]) -> Dict[str, Any]: STATUS_SHARED: "approved", STATUS_REJECTED: "rejected", } + # Look up authorization_token and custom_headers from the source MCP record + source_authorization_token = None + source_custom_headers = None + source_container_port = None + source_mcp_id = row.get("source_mcp_id") + if source_mcp_id is not None: + try: + from database.remote_mcp_db import get_mcp_record_by_id_and_tenant + mcp_record = get_mcp_record_by_id_and_tenant(mcp_id=source_mcp_id, tenant_id=row.get("tenant_id", "")) + if mcp_record: + source_authorization_token = mcp_record.get("authorization_token") + source_custom_headers = mcp_record.get("custom_headers") + source_container_port = mcp_record.get("container_port") + except Exception: + pass return { "communityId": row.get("market_id"), "marketId": row.get("market_id"), "reviewId": row.get("market_id"), - "sourceMcpId": row.get("source_mcp_id"), + "sourceMcpId": source_mcp_id, + "sharedFields": row.get("shared_fields"), + "authorizationToken": source_authorization_token, + "customHeaders": source_custom_headers, + "containerPort": source_container_port, "name": row.get("mcp_name"), "description": row.get("description"), + "content": row.get("content") or "", "status": "active" if raw_status == STATUS_SHARED else "inactive", "createdAt": row.get("create_time"), "updatedAt": row.get("update_time"), @@ -130,6 +163,8 @@ def _to_community_card(row: Dict[str, Any]) -> Dict[str, Any]: "reviewType": "initial_listing", "installCount": row.get("download_count") or 0, "authorDisplayName": _resolve_author_display_name(row.get("user_id")), + "groupIds": row.get("group_ids"), + "ingroupPermission": row.get("ingroup_permission"), } @@ -192,13 +227,22 @@ def _validate_market_status_transition( async def list_community_mcp_services( *, tenant_id: str, + user_id: str, search: str | None = None, tag: str | None = None, transport_type: str | None = None, cursor: str | None = None, limit: int = 30, ) -> Dict[str, Any]: - """List shared (approved) community MCP services scoped to a tenant.""" + """List shared (approved) community MCP services scoped to a tenant with permission filtering.""" + user_role = _get_user_role(user_id) + user_group_ids = None + if user_role not in CAN_EDIT_ALL_USER_ROLES: + try: + user_group_ids = list(query_group_ids_by_user(user_id) or []) + except Exception as e: + logger.warning(f"Failed to query user group ids: user_id={user_id}, err={e}") + db_result = get_mcp_market_records( tenant_id=tenant_id, search=search, @@ -206,6 +250,8 @@ async def list_community_mcp_services( transport_type=transport_type, cursor=cursor, limit=limit, + user_id=user_id if user_role not in CAN_EDIT_ALL_USER_ROLES else None, + user_group_ids=user_group_ids, ) return { "count": db_result.get("count", 0), @@ -218,6 +264,87 @@ def list_community_mcp_tag_stats(tenant_id: str) -> List[Dict[str, Any]]: return get_mcp_market_tag_stats_by_tenant(tenant_id=tenant_id) +def _mcp_notification_details( + *, + name: str | None, + market_id: int, + source_mcp_id: int | None, + content: str | None = None, + include_empty_content: bool = False, +) -> Dict[str, Any]: + """Build notification details payload for MCP repository events.""" + details: Dict[str, Any] = { + "name": name, + "market_id": market_id, + "source_mcp_id": source_mcp_id, + } + if content: + details["content"] = content + elif include_empty_content: + details["content"] = "" + return details + + +def _create_mcp_pending_review_notification( + *, + tenant_id: str, + user_id: str, + market_id: int, + name: str | None, + source_mcp_id: int | None, + content: str | None = None, +) -> None: + """Notify tenant admins that an MCP listing awaits review.""" + create_repository_pending_review_notification( + resource_type=RESOURCE_TYPE_MCP_REPOSITORY, + tenant_id=tenant_id, + unique_id=market_id, + details=_mcp_notification_details( + name=name, + market_id=market_id, + source_mcp_id=source_mcp_id, + content=content, + include_empty_content=True, + ), + created_by=user_id, + ) + + +def _handle_mcp_review_status_notifications( + *, + current_status: str, + new_status: str, + record: Dict[str, Any], + market_id: int, + user_id: str, + content: str | None = None, +) -> None: + """Send review-result notification and deactivate pending-review notification.""" + if current_status != new_status and new_status in (STATUS_SHARED, STATUS_REJECTED): + create_repository_review_notification( + resource_type=RESOURCE_TYPE_MCP_REPOSITORY, + review_status=new_status, + receiver_user_id=record["user_id"], + details=_mcp_notification_details( + name=record.get("mcp_name"), + market_id=market_id, + source_mcp_id=record.get("source_mcp_id"), + content=content, + ), + tenant_id=record.get("tenant_id"), + unique_id=market_id, + created_by=user_id, + ) + + if current_status == STATUS_PENDING_REVIEW: + deactivate_notifications( + event_type=EVENT_TYPE_REPOSITORY_REVIEW_PENDING, + resource_type=RESOURCE_TYPE_MCP_REPOSITORY, + unique_id=market_id, + updated_by=user_id, + ) + + async def publish_community_mcp_service( *, tenant_id: str, @@ -228,6 +355,10 @@ async def publish_community_mcp_service( tags: List[str] | None = None, mcp_server: str | None = None, config_json: Dict[str, Any] | None = None, + group_ids: List[int] | None = None, + ingroup_permission: str | None = None, + shared_fields: dict | None = None, + content: str | None = None, ) -> int: """Submit a local MCP service for review. @@ -258,7 +389,7 @@ async def publish_community_mcp_service( community_transport_type = "container" if final_config_json is not None else "url" # Check name uniqueness among shared records only - if check_mcp_market_name_exists(final_name): + if check_mcp_market_name_exists(final_name, tenant_id): raise McpNameConflictError(f"MCP name '{final_name}' already exists in the community market") market_id = create_mcp_market_record( @@ -273,10 +404,43 @@ async def publish_community_mcp_service( "submitted_by": _resolve_user_email(user_id), "tags": final_tags, "description": final_description, + "content": content, + "group_ids": convert_list_to_string(group_ids) if group_ids else None, + "ingroup_permission": ingroup_permission, + "shared_fields": shared_fields, + "container_port": source_record.get("container_port"), }, tenant_id=tenant_id, user_id=user_id, ) + + # Update shared_fields on the source MCP record + if shared_fields is not None: + update_mcp_record_manage_fields_by_id( + mcp_id=mcp_id, + tenant_id=tenant_id, + user_id=user_id, + name=source_record.get("mcp_name", ""), + server_url=source_record.get("mcp_server", ""), + description=source_record.get("description") or "", + tags=source_record.get("tags"), + source=source_record.get("source") or "local", + authorization_token=source_record.get("authorization_token"), + custom_headers=source_record.get("custom_headers"), + config_json=source_record.get("config_json"), + market_id=source_record.get("market_id"), + shared_fields=shared_fields, + ) + + _create_mcp_pending_review_notification( + tenant_id=tenant_id, + user_id=user_id, + market_id=market_id, + name=final_name, + source_mcp_id=mcp_id, + content=content, + ) + return market_id @@ -292,6 +456,10 @@ async def update_community_mcp_service( mcp_server: str | None = None, config_json: Dict[str, Any] | None = None, transport_type: str | None = None, + group_ids: List[int] | None = None, + ingroup_permission: str | None = None, + shared_fields: dict | None = None, + content: str | None = None, ) -> None: """Update a published market MCP and set it back to pending_review for re-approval.""" current = get_mcp_market_record_by_id(market_id=market_id) @@ -312,9 +480,12 @@ async def update_community_mcp_service( next_transport_type = "url" # Check name uniqueness if name is changing - if name is not None and name != current.get("mcp_name") and check_mcp_market_name_exists(name): + if name is not None and name != current.get("mcp_name") and check_mcp_market_name_exists(name, tenant_id): raise McpNameConflictError(f"MCP name '{name}' already exists in the community market") + final_name = name if name is not None else current.get("mcp_name") + final_content = content if content is not None else current.get("content") + # Update fields update_mcp_market_record( market_id=market_id, @@ -326,6 +497,10 @@ async def update_community_mcp_service( mcp_server=mcp_server, config_json=next_config_json, transport_type=next_transport_type, + group_ids=convert_list_to_string(group_ids) if group_ids else None, + ingroup_permission=ingroup_permission, + shared_fields=shared_fields, + content=content, ) # Set back to pending_review for re-approval @@ -336,6 +511,39 @@ async def update_community_mcp_service( submitted_by=_resolve_user_email(user_id), ) + # Update shared_fields on the source MCP record + if shared_fields is not None and current.get("source_mcp_id"): + update_mcp_record_manage_fields_by_id( + mcp_id=current["source_mcp_id"], + tenant_id=tenant_id, + user_id=user_id, + name=current.get("mcp_name") or "", + server_url=current.get("mcp_server") or "", + description=current.get("description") or "", + tags=current.get("tags"), + source="local", + authorization_token=None, + custom_headers=None, + config_json=current.get("config_json"), + market_id=market_id, + shared_fields=shared_fields, + ) + + deactivate_notifications( + event_type=EVENT_TYPE_REPOSITORY_REVIEW_PENDING, + resource_type=RESOURCE_TYPE_MCP_REPOSITORY, + unique_id=market_id, + updated_by=user_id, + ) + _create_mcp_pending_review_notification( + tenant_id=tenant_id, + user_id=user_id, + market_id=market_id, + name=final_name, + source_mcp_id=current.get("source_mcp_id"), + content=final_content, + ) + async def change_mcp_market_status( *, @@ -343,6 +551,7 @@ async def change_mcp_market_status( user_id: str, market_id: int, new_status: str, + content: str | None = None, ) -> None: """Unified status change endpoint. Validates state machine transitions. @@ -375,6 +584,7 @@ async def change_mcp_market_status( user_id=user_id, review_status=new_status, submitted_by=submitted_by, + content=content, ) # When approving for the first time, link the source MCP record @@ -388,6 +598,26 @@ async def change_mcp_market_status( market_id=market_id, ) + updated = get_mcp_market_record_by_id(market_id=market_id) or current + _handle_mcp_review_status_notifications( + current_status=current_status, + new_status=new_status, + record=updated, + market_id=market_id, + user_id=user_id, + content=content, + ) + + if current_status != new_status and new_status == STATUS_PENDING_REVIEW: + _create_mcp_pending_review_notification( + tenant_id=updated.get("tenant_id") or tenant_id, + user_id=user_id, + market_id=market_id, + name=updated.get("mcp_name"), + source_mcp_id=updated.get("source_mcp_id"), + content=content if content is not None else updated.get("content"), + ) + async def list_community_mcp_review_services( *, @@ -436,6 +666,12 @@ async def delete_community_mcp_service( user_id=user_id, market_id=market_id, ) + deactivate_notifications( + event_type=EVENT_TYPE_REPOSITORY_REVIEW_PENDING, + resource_type=RESOURCE_TYPE_MCP_REPOSITORY, + unique_id=market_id, + updated_by=user_id, + ) async def list_my_community_mcp_services( @@ -464,6 +700,7 @@ async def approve_community_mcp_service( tenant_id: str, user_id: str, market_id: int, + content: str | None = None, ) -> None: """Approve: pending_review -> shared.""" user_role = _get_user_role(user_id) @@ -474,6 +711,7 @@ async def approve_community_mcp_service( user_id=user_id, market_id=market_id, new_status=STATUS_SHARED, + content=content, ) @@ -482,6 +720,7 @@ async def reject_community_mcp_service( tenant_id: str, user_id: str, market_id: int, + content: str | None = None, ) -> None: """Reject: pending_review -> rejected.""" user_role = _get_user_role(user_id) @@ -492,6 +731,7 @@ async def reject_community_mcp_service( user_id=user_id, market_id=market_id, new_status=STATUS_REJECTED, + content=content, ) diff --git a/backend/services/memory_backend_adapter.py b/backend/services/memory_backend_adapter.py new file mode 100644 index 0000000000..c7628375e1 --- /dev/null +++ b/backend/services/memory_backend_adapter.py @@ -0,0 +1,157 @@ +"""Wire the SDK ``MemoryService`` facade to backend services. + +The SDK does not depend on PostgreSQL or Elasticsearch. It accepts two +async hooks (``backend_store`` / ``backend_search``) and dispatches the +payloads through them. This module provides the backend-side adapter that +bridges those hooks to ``services.memory_record_service`` and +``services.memory_retrieval_service``. + +Usage from the agent build path:: + + from services.memory_backend_adapter import ( + build_memory_service_for_agent, + ) + memory_service = build_memory_service_for_agent( + tenant_id=tenant_id, + user_id=user_id, + agent_id=agent_id, + embedding_model_info=embedding_model_info, + ) +""" + +from __future__ import annotations + +import logging +from typing import Any, Dict, List, Optional + +from nexent.memory.embedding_model import EmbeddingModelInfo +from nexent.memory.models import ( + MemoryLayer, + MemorySearchRequest, + MemorySearchResult, +) +from nexent.memory.service import MemoryService + +from .memory_record_service import ( + MemoryRecordError, + _resolve_tenant_embedding_model_info, + get_memory_record_service, +) +from .memory_retrieval_service import get_memory_retrieval_service + + +logger = logging.getLogger("memory_backend_adapter") + + +async def _backend_store_hook( + payload: Dict[str, Any], +) -> Dict[str, Any]: + """Adapter for ``MemoryService.store_memory`` -> ``MemoryRecordService``.""" + service = get_memory_record_service() + layer_value = payload.get("layer", MemoryLayer.AGENT.value) + if isinstance(layer_value, MemoryLayer): + layer_value = layer_value.value + + memory_type_value = payload.get("memory_type") + if isinstance(memory_type_value, MemoryLayer): + memory_type_value = memory_type_value.value + + tenant_id = payload["tenant_id"] + embedding = payload.get("embedding") + embedding_model_info = None + + if embedding is None and layer_value == MemoryLayer.AGENT.value: + embedding_model_info = _resolve_tenant_embedding_model_info(tenant_id) + if embedding_model_info is None: + raise MemoryRecordError( + "Failed to store memory: tenant embedding model is not configured" + ) + + result = service.create_memory( + tenant_id=tenant_id, + user_id=payload["user_id"], + content=payload["content"], + layer=layer_value, + memory_type=memory_type_value, + agent_id=payload.get("agent_id"), + conversation_id=( + str(payload["conversation_id"]) + if payload.get("conversation_id") not in (None, "") + else None + ), + idempotency_key=payload.get("idempotency_key"), + embedding=embedding, + embedding_model_info=embedding_model_info, + actor="agent", + ) + return result + + +async def _backend_search_hook( + payload: Dict[str, Any], +) -> List[Dict[str, Any]]: + """Adapter for ``MemoryService.search_memory`` -> ``MemoryRetrievalService``.""" + if _resolve_tenant_embedding_model_info(payload["tenant_id"]) is None: + return [] + retrieval = get_memory_retrieval_service() + request = MemorySearchRequest( + tenant_id=payload["tenant_id"], + user_id=payload["user_id"], + agent_id=payload.get("agent_id"), + conversation_id=payload.get("conversation_id"), + layers=payload.get("layers") or [MemoryLayer.AGENT], + query=payload.get("query", ""), + top_k=int(payload.get("top_k") or 5), + threshold=payload.get("threshold") or 0.65, + embedding=payload.get("embedding"), + ) + results: List[MemorySearchResult] = await retrieval.search( + request, write_hits=True + ) + return [ + { + "memory_id": r.memory_id, + "content": r.content, + "score": r.score, + "layer": r.layer.value if hasattr(r.layer, "value") else r.layer, + "source": r.source, + "is_external": r.is_external, + "metadata": r.metadata, + } + for r in results + ] + + +def build_memory_service_for_agent( + *, + tenant_id: str, + user_id: str, + agent_id: str, + embedding_model_info: Optional[EmbeddingModelInfo] = None, +) -> MemoryService: + """Construct a per-agent ``MemoryService`` wired to the backend hooks. + + The returned facade is the value passed to ``StoreMemoryTool`` and + ``SearchMemoryTool`` when building the agent. + """ + return MemoryService( + embedding_model=None, + embedding_model_info=embedding_model_info, + backend_store=_backend_store_hook, + backend_search=_backend_search_hook, + ) + + +def build_memory_service_for_dreaming() -> MemoryService: + """Return a facade for Dreaming promotion (no embedding model needed). + + Dreaming promotes already-stored agent memories to user long-term + memory and never needs the search hook. The store hook enforces the + ``actor="dreaming"`` policy via ``MemoryRecordService``. + """ + return MemoryService( + embedding_model=None, + embedding_model_info=None, + backend_store=_backend_store_hook, + backend_search=None, + ) diff --git a/backend/services/memory_config_service.py b/backend/services/memory_config_service.py index 40cdc5b0ed..f2eda95fbc 100644 --- a/backend/services/memory_config_service.py +++ b/backend/services/memory_config_service.py @@ -18,7 +18,6 @@ update_config_by_id, ) from nexent.core.agents.agent_model import MemoryContext, MemoryUserConfig -from utils.memory_utils import build_memory_config logger = logging.getLogger("memory_config_service") @@ -213,7 +212,6 @@ def build_memory_context(user_id: str, tenant_id: str, agent_id: str | int, skip ) return MemoryContext( user_config=memory_user_config, - memory_config=dict(), tenant_id=tenant_id, user_id=user_id, agent_id=str(agent_id), @@ -229,7 +227,6 @@ def build_memory_context(user_id: str, tenant_id: str, agent_id: str | int, skip if not memory_user_config.memory_switch: return MemoryContext( user_config=memory_user_config, - memory_config=dict(), tenant_id=tenant_id, user_id=user_id, agent_id=str(agent_id), @@ -237,7 +234,6 @@ def build_memory_context(user_id: str, tenant_id: str, agent_id: str | int, skip return MemoryContext( user_config=memory_user_config, - memory_config=build_memory_config(tenant_id), tenant_id=tenant_id, user_id=user_id, agent_id=str(agent_id), diff --git a/backend/services/memory_context_service.py b/backend/services/memory_context_service.py new file mode 100644 index 0000000000..6b38d3fa6f --- /dev/null +++ b/backend/services/memory_context_service.py @@ -0,0 +1,231 @@ +"""Memory context builder for agent prompt injection. + +Tenant/user long-term memory is always loaded in full (no vector search). +Agent short-term memory uses vector retrieval. When Phase 4 is enabled +(pipeline_enabled=True), the raw retrieval results are additionally +processed through the SDK's RetrievalPipeline which applies: + + normalize -> score fusion -> temporal decay -> MMR -> token budget selection + +The resulting MemorySearchContext is what gets serialized into the prompt. +""" + +from __future__ import annotations + +import logging +from typing import Any, Dict, List, Optional + +from nexent.memory.embedding_model import EmbeddingModelInfo +from nexent.memory.models import ( + ExternalMemoryItem, + MemoryLayer, + MemorySearchContext, + MemorySearchRequest, + PipelineConfig, +) +from nexent.memory.retrieval.pipeline import RetrievalPipeline +from nexent.memory.policy import MemoryRetrievalPolicy + +from consts.const import ( + AGENT_SHORT_TERM_HALF_LIFE_DAYS, + MMR_CANDIDATE_TOP_K, + MMR_DUPLICATE_THRESHOLD, + MMR_FINAL_TOP_K, + MMR_LAMBDA, + MEMORY_TOKEN_BUDGET, + W_AGENT_SHORT_TERM, + W_EXTERNAL, +) + +from .memory_record_service import ( + _compute_content_embedding, + _resolve_tenant_embedding_model_info, +) +from .memory_retrieval_service import ( + MemoryRetrievalService, + get_memory_retrieval_service, +) + + +logger = logging.getLogger("memory_context_service") + + +def _prepare_search_embedding( + *, + query: Optional[str], + embedding: Optional[List[float]], + embedding_model_info: Optional[EmbeddingModelInfo], + tenant_id: str, +) -> tuple[Optional[EmbeddingModelInfo], Optional[List[float]]]: + """Resolve the embedding model and compute the query embedding when needed.""" + if embedding is not None: + return embedding_model_info, embedding + if not query: + return embedding_model_info, None + resolved = embedding_model_info or _resolve_tenant_embedding_model_info(tenant_id) + if resolved is None: + return None, None + computed = _compute_content_embedding(query, resolved) + if computed is None: + logger.warning("query_embedding computation failed for tenant=%s", tenant_id) + return resolved, computed + + +def _build_pipeline_config() -> PipelineConfig: + """Build a PipelineConfig from the env vars in const.py.""" + return PipelineConfig( + mmr_lambda=MMR_LAMBDA, + mmr_candidate_top_k=MMR_CANDIDATE_TOP_K, + mmr_final_top_k=MMR_FINAL_TOP_K, + mmr_duplicate_threshold=MMR_DUPLICATE_THRESHOLD, + half_life_days=AGENT_SHORT_TERM_HALF_LIFE_DAYS, + w_agent_short_term=W_AGENT_SHORT_TERM, + w_external=W_EXTERNAL, + token_budget=MEMORY_TOKEN_BUDGET, + ) + + +class MemoryContextService: + """Compose the memory block injected into agent prompts.""" + + def __init__( + self, + retrieval_service: Optional[MemoryRetrievalService] = None, + pipeline_enabled: bool = True, + ): + """Initialize the context service. + + Args: + retrieval_service: Optional injected retrieval service. + pipeline_enabled: When True (default), the Phase 4 retrieval + pipeline is applied to agent short-term + external results. + Set to False to preserve the Phase 2 behaviour. + """ + self.retrieval_service = retrieval_service or get_memory_retrieval_service() + self.pipeline_enabled = pipeline_enabled + self._pipeline: Optional[RetrievalPipeline] = None + + @property + def pipeline(self) -> RetrievalPipeline: + """Lazily-built retrieval pipeline instance.""" + if self._pipeline is None: + cfg = _build_pipeline_config() + self._pipeline = RetrievalPipeline(cfg) + return self._pipeline + + async def build_context( + self, + *, + tenant_id: str, + user_id: str, + agent_id: Optional[str] = None, + conversation_id: Optional[str] = None, + query: Optional[str] = None, + top_k: int = 5, + threshold: float = 0.65, + embedding: Optional[List[float]] = None, + embedding_model_info: Optional[EmbeddingModelInfo] = None, + layers: Optional[List[str]] = None, + external_results: Optional[List[ExternalMemoryItem]] = None, + created_at_for_id: Optional[Dict[int, Any]] = None, + ) -> MemorySearchContext: + """Return a populated MemorySearchContext for the agent. + + Full-context layers (tenant/user) are always loaded verbatim. + Agent short-term memory is retrieved via vector search and, when + pipeline_enabled is True, passed through the Phase 4 pipeline. + + Args: + tenant_id: Tenant identifier. + user_id: User identifier. + agent_id: Optional agent identifier. + conversation_id: Optional conversation identifier. + query: Optional search query for vector retrieval. + top_k: Maximum number of agent short-term results to return. + threshold: Minimum similarity threshold for vector search. + embedding: Optional pre-computed query embedding vector. + embedding_model_info: Optional pre-resolved embedding model info. + layers: Optional list of layer names to search. + external_results: Optional external provider hits from Phase 3. + created_at_for_id: Optional mapping of memory_id -> create_time. + """ + if layers: + target_layers: List[MemoryLayer] = [] + for value in layers: + try: + target_layers.append(MemoryLayer(value.strip().lower())) + except ValueError: + logger.warning("build_context: skipping unknown layer=%s", value) + target_layers = target_layers or list( + MemoryRetrievalPolicy.FULL_CONTEXT_LAYERS + | MemoryRetrievalPolicy.VECTOR_SEARCH_LAYERS + ) + else: + target_layers = list( + MemoryRetrievalPolicy.FULL_CONTEXT_LAYERS + | MemoryRetrievalPolicy.VECTOR_SEARCH_LAYERS + ) + + resolved_model_info, resolved_embedding = _prepare_search_embedding( + query=query, + embedding=embedding, + embedding_model_info=embedding_model_info, + tenant_id=tenant_id, + ) + + request = MemorySearchRequest( + tenant_id=tenant_id, + user_id=user_id, + agent_id=agent_id, + conversation_id=conversation_id, + layers=target_layers, + query=query or "", + top_k=top_k, + threshold=threshold, + embedding=resolved_embedding, + ) + + results = await self.retrieval_service.search( + request, + embedding_model_info=resolved_model_info, + write_hits=bool(query), + ) + + if self.pipeline_enabled and results: + pipeline_result = self.pipeline.run( + internal_results=results, + query=query or "", + external_results=external_results, + created_at_for_id=created_at_for_id, + ) + context = pipeline_result.into_memory_search_context() + else: + context = MemorySearchContext() + for result in results: + if result.layer == MemoryLayer.TENANT: + context.tenant_long_term.append(result) + elif result.layer == MemoryLayer.USER: + context.user_long_term.append(result) + elif result.layer == MemoryLayer.AGENT: + context.agent_short_term.append(result) + else: + context.external.append(result) + + return context + + +_default_service: Optional[MemoryContextService] = None + + +def get_memory_context_service() -> MemoryContextService: + """Return the process-wide context service.""" + global _default_service + if _default_service is None: + _default_service = MemoryContextService() + return _default_service + + +def reset_memory_context_service() -> None: + """Reset the cached service (used by tests).""" + global _default_service + _default_service = None diff --git a/backend/services/memory_dreaming_scheduler.py b/backend/services/memory_dreaming_scheduler.py new file mode 100644 index 0000000000..4f56345236 --- /dev/null +++ b/backend/services/memory_dreaming_scheduler.py @@ -0,0 +1,411 @@ +"""Dreaming consolidation runner for the Memory Architecture (Phase 2). + +Dreaming promotes agent short-term memories into user long-term memory. It +runs in three phases: + +1. **Light Sleep** - aggregate ``memory_retrieval_hits_t`` rows into the + per-memory ``light_hits`` counter and ``recall_count`` / + ``recall_days`` / ``query_hashes`` columns. +2. **REM Sleep** - extract repeating concepts / patterns, write concept + tags to ``memory_records_t``. The phase is implemented as a lightweight + keyword-frequency pass; LLM-driven concept extraction can be wired in + later without changing the public API. +3. **Deep Sleep** - select eligible agent memories and promote them to + ``user`` long-term memory using the documented scoring formula + (frequency / relevance / diversity / recency / consolidation / + concept + phase boost). + +Promotion thresholds and weights live in ``consts.const``. The phases are +exposed as standalone ``run_light_sleep`` / ``run_rem_sleep`` / +``run_deep_sleep`` functions plus the aggregate ``run_once`` so callers +can trigger a single pass per tenant on demand (e.g. from a future agent +timer). The module deliberately does **not** ship an internal scheduler, +background thread, or cron expression: agent-driven scheduling will be +added in a later phase, and we want to avoid having to coordinate cron, +lock watchdog and lifecycle here before the agent timer feature lands. +""" + +from __future__ import annotations + +import logging +import math +import time +from collections import Counter +from datetime import datetime, timedelta +from typing import Any, Dict, List, Optional, Sequence, Set + +from consts.const import ( + AGENT_SHORT_TERM_HALF_LIFE_DAYS, + LIGHT_SLEEP_WINDOW_DAYS, + MIN_PROMOTION_SCORE, + MIN_RECALL_COUNT, + MIN_UNIQUE_QUERIES, + RECENCY_HALF_LIFE_DAYS, +) +from database import memory_record_db, memory_retrieval_hit_db +from services.memory_record_service import ( + MemoryRecordError, + get_memory_record_service, +) + + +logger = logging.getLogger("memory_dreaming_scheduler") + + +# --------------------------------------------------------------------------- +# Scoring helpers +# --------------------------------------------------------------------------- + + +def _clamp01(value: float) -> float: + if value < 0.0: + return 0.0 + if value > 1.0: + return 1.0 + return value + + +def _frequency(recall_count: int, daily_count: int, grounded_count: int) -> float: + """Log-scaled accumulation of recall signals.""" + signal = max(0, recall_count) + max(0, daily_count) + max(0, grounded_count) + return _clamp01(math.log1p(signal) / math.log1p(10)) + + +def _relevance(hit_count: int, total_score: float) -> float: + """Average retrieval score across hits, clamped to [0, 1].""" + if hit_count <= 0: + return 0.0 + return _clamp01(total_score / max(1, hit_count)) + + +def _diversity(unique_queries: int) -> float: + """Smoothed saturation in [0, 1].""" + return _clamp01(math.log1p(unique_queries) / math.log1p(5)) + + +def _recency(last_recalled_at: Optional[datetime]) -> float: + """Exponential decay based on ``RECENCY_HALF_LIFE_DAYS``.""" + if last_recalled_at is None: + return 0.0 + delta_days = (datetime.utcnow() - last_recalled_at).total_seconds() / 86400.0 + if delta_days < 0: + delta_days = 0 + half_life = max(1, RECENCY_HALF_LIFE_DAYS) + return _clamp01(math.pow(0.5, delta_days / half_life)) + + +def _consolidation(light_hits: int, rem_hits: int) -> float: + """Boost when both Light and REM phases have seen the memory.""" + combined = max(0, light_hits) + max(0, rem_hits) + return _clamp01(math.log1p(combined) / math.log1p(6)) + + +def _concept(concept_tags: Sequence[str]) -> float: + """Higher when concept tags have been attached.""" + if not concept_tags: + return 0.0 + return _clamp01(math.log1p(len(concept_tags)) / math.log1p(8)) + + +# Weights reflect ``openclaw_dreaming.md`` §深眠阶段评分体系. +_PROMOTION_WEIGHTS: Dict[str, float] = { + "relevance": 0.30, + "frequency": 0.24, + "diversity": 0.15, + "recency": 0.15, + "consolidation": 0.10, + "concept": 0.06, +} + + +def _normalize_weights(weights: Dict[str, float]) -> Dict[str, float]: + total = sum(weights.values()) or 1.0 + return {key: value / total for key, value in weights.items()} + + +def _phase_boost(light_hits: int, rem_hits: int) -> float: + """PhaseBoost from the design doc; kept small to avoid runaway scores.""" + if light_hits <= 0 or rem_hits <= 0: + return 0.0 + return _clamp01(min(0.05, light_hits * 0.01 + rem_hits * 0.01)) + + +def compute_promotion_score(record: Dict[str, Any]) -> float: + """Compute the composite score for a single memory record.""" + recall_count = int(record.get("recall_count") or 0) + daily_count = int(record.get("daily_count") or 0) + grounded_count = int(record.get("grounded_count") or 0) + light_hits = int(record.get("light_hits") or 0) + rem_hits = int(record.get("rem_hits") or 0) + last_recalled_at = record.get("last_recalled_at") + query_hashes = record.get("query_hashes") or [] + + metrics = { + "frequency": _frequency(recall_count, daily_count, grounded_count), + "relevance": _relevance(recall_count, 1.0), + "diversity": _diversity(len(query_hashes)), + "recency": _recency(last_recalled_at), + "consolidation": _consolidation(light_hits, rem_hits), + "concept": _concept(record.get("concept_tags") or []), + } + weights = _normalize_weights(_PROMOTION_WEIGHTS) + score = sum(metrics[key] * weights[key] for key in metrics) + score += _phase_boost(light_hits, rem_hits) + return _clamp01(score) + + +# --------------------------------------------------------------------------- +# Phase runners +# --------------------------------------------------------------------------- + + +def run_light_sleep( + *, + tenant_id: str, + user_id: str, + agent_id: Optional[str] = None, + window_days: int = LIGHT_SLEEP_WINDOW_DAYS, +) -> int: + """Aggregate recent hits into memory row counters. + + Returns the number of memory rows touched. + """ + since = datetime.utcnow() - timedelta(days=max(1, window_days)) + stats = memory_retrieval_hit_db.aggregate_memory_stats( + tenant_id, + user_id=user_id, + agent_id=agent_id, + since=since, + ) + touched = 0 + for entry in stats: + memory_id = entry["memory_id"] + # Last hit day is the most recent value in the per-memory hit set. + last_day = max(entry["days"]) if entry["days"] else None + last_recalled_at = ( + datetime.fromisoformat(last_day) if last_day else None + ) + memory_record_db.update_memory_record( + memory_id, + tenant_id, + { + "recall_count": entry["hit_count"], + "grounded_count": entry["grounded_count"], + "query_hashes": sorted(entry["query_hashes"]), + "recall_days": sorted(entry["days"]), + "last_recalled_at": last_recalled_at, + }, + ) + memory_record_db.apply_dreaming_phase( + memory_id, tenant_id, phase="light" + ) + touched += 1 + return touched + + +_KEYWORD_STOPWORDS: Set[str] = { + "the", "a", "an", "and", "or", "but", "is", "are", "was", "were", + "be", "been", "being", "have", "has", "had", "do", "does", "did", + "of", "in", "on", "at", "by", "for", "with", "to", "from", + "i", "you", "he", "she", "it", "we", "they", + "的", "了", "是", "在", "和", "与", "及", "或", "我", "你", "他", "她", "它", + "我们", "你们", "他们", "这", "那", "这个", "那个", +} + + +def _tokenize(text: str) -> List[str]: + return [ + token.strip().lower() + for token in text.replace("\n", " ").split() + if token.strip() and token.strip().lower() not in _KEYWORD_STOPWORDS + ] + + +def run_rem_sleep( + *, + tenant_id: str, + user_id: str, + agent_id: Optional[str] = None, + max_keywords: int = 5, +) -> int: + """Extract concept tags from frequently appearing tokens. + + Returns the number of memory rows whose ``concept_tags`` were updated. + """ + rows = memory_record_db.list_memory_records( + tenant_id, + user_id=user_id, + agent_id=agent_id, + layer="agent", + memory_type="short_term", + status="active", + limit=500, + ) + touched = 0 + for row in rows: + tokens = _tokenize(row.get("content", "")) + if not tokens: + continue + counter = Counter(tokens) + top = [token for token, _ in counter.most_common(max_keywords)] + if not top: + continue + existing = list(row.get("concept_tags") or []) + merged = list(dict.fromkeys(existing + top))[:max_keywords] + memory_record_db.update_memory_record( + row["memory_id"], + tenant_id, + {"concept_tags": merged}, + ) + memory_record_db.apply_dreaming_phase( + row["memory_id"], tenant_id, phase="rem" + ) + touched += 1 + return touched + + +def run_deep_sleep( + *, + tenant_id: str, + user_id: str, + agent_id: Optional[str] = None, + min_score: float = MIN_PROMOTION_SCORE, + min_recall_count: int = MIN_RECALL_COUNT, + min_unique_queries: int = MIN_UNIQUE_QUERIES, +) -> List[Dict[str, Any]]: + """Promote agent memories that pass the promotion thresholds. + + Returns the list of promotion results (``memory_id``, ``score``, ``event``). + """ + eligible = memory_record_db.list_memories_for_dreaming( + tenant_id, + user_id=user_id, + layer="agent", + min_recall_count=min_recall_count, + window_days=LIGHT_SLEEP_WINDOW_DAYS, + ) + promoted: List[Dict[str, Any]] = [] + service = get_memory_record_service() + for row in eligible: + query_hashes = row.get("query_hashes") or [] + if len(set(query_hashes)) < min_unique_queries: + continue + score = compute_promotion_score(row) + if score < min_score: + continue + try: + service.create_memory( + tenant_id=tenant_id, + user_id=user_id, + content=row.get("content", ""), + layer="user", + memory_type="long_term", + agent_id=row.get("agent_id"), + conversation_id=row.get("conversation_id"), + concept_tags=row.get("concept_tags") or [], + idempotency_key=f"dreaming:{row['memory_id']}", + created_by="dreaming", + actor="dreaming", + ) + except MemoryRecordError as exc: + logger.warning( + "dreaming promotion skipped for %s: %s", row["memory_id"], exc + ) + continue + memory_record_db.apply_dreaming_phase( + row["memory_id"], tenant_id, phase="rem" + ) + promoted.append( + { + "memory_id": row["memory_id"], + "score": score, + "event": "PROMOTE", + } + ) + return promoted + + +# --------------------------------------------------------------------------- +# Manual entry points +# --------------------------------------------------------------------------- + + +def run_once(*, timeout_seconds: int = 1800) -> Dict[str, Any]: + """Execute one full Dreaming cycle across known tenants. + + This function is the single manual entry point: callers (e.g. an agent + timer introduced later) invoke ``run_once`` whenever they want a fresh + pass. Phase 2 does not ship a scheduler; the function is intentionally + synchronous and idempotent so it can be re-invoked safely. + + Args: + timeout_seconds: Soft cap on wall-clock runtime per call. Iteration + stops once the deadline is reached; partial state is returned. + + Returns: + Summary dict with tenant count, light/rem rows touched, and the + list of promotion events. + """ + started = time.time() + deadline = started + max(60, timeout_seconds) + + # ``list_distinct_tenants`` is intentionally conservative: we only run + # dreaming over tenants that have actually touched memory recently. + tenants = list_distinct_tenants() + summary: Dict[str, Any] = { + "tenants": len(tenants), + "light_rows": 0, + "rem_rows": 0, + "promotions": [], + } + + for tenant_id, user_id in tenants: + if time.time() >= deadline: + logger.warning("Dreaming run hit timeout; aborting remaining tenants") + break + try: + light = run_light_sleep(tenant_id=tenant_id, user_id=user_id) + rem = run_rem_sleep(tenant_id=tenant_id, user_id=user_id) + deep = run_deep_sleep(tenant_id=tenant_id, user_id=user_id) + summary["light_rows"] += light + summary["rem_rows"] += rem + summary["promotions"].extend(deep) + except Exception: + logger.exception( + "Dreaming iteration failed for tenant=%s user=%s", + tenant_id, + user_id, + ) + + summary["elapsed_seconds"] = time.time() - started + return summary + + +def list_distinct_tenants() -> List[Any]: + """Return ``(tenant_id, user_id)`` tuples with recent memory activity. + + Implementation: distinct pairs from ``memory_retrieval_hits_t``. When + no hits exist (fresh deployments) this returns ``[]`` and Dreaming + becomes a no-op, which is the intended behavior. + """ + try: + from database.client import get_db_session + from database.db_models import MemoryRetrievalHit + from sqlalchemy import distinct + + with get_db_session() as session: + rows = ( + session.query( + distinct(MemoryRetrievalHit.tenant_id), + distinct(MemoryRetrievalHit.user_id), + ) + .filter( + MemoryRetrievalHit.tenant_id.isnot(None), + MemoryRetrievalHit.user_id.isnot(None), + ) + .all() + ) + return [(t, u) for t, u in rows if t and u] + except Exception: + logger.exception("list_distinct_tenants failed") + return [] \ No newline at end of file diff --git a/backend/services/memory_index_service.py b/backend/services/memory_index_service.py new file mode 100644 index 0000000000..848fb72c73 --- /dev/null +++ b/backend/services/memory_index_service.py @@ -0,0 +1,441 @@ +"""Elasticsearch index management for agent short-term memory. + +This service owns the ES side of the memory pipeline: + +- create / ensure the per-tenant index (one index per embedding model) +- index/update/delete chunk documents mirroring ``memory_records_t`` +- run kNN searches and return normalized hit dicts + +The corresponding PostgreSQL row is always written first by +``services.memory_record_service``. ``MemoryIndexService`` only deals with +the vector side and never modifies PG state directly. + +Failure modes: +- If ES is unavailable, ``memory_record_service`` keeps the PG row but skips + the mirror; the row stays usable for full-text reads and ``es_index_name`` + remains set so a later retry can backfill it. +- Per-document failures during writes do not abort the batch; the function + returns a summary so the caller can decide what to do. +""" + +from __future__ import annotations + +import json +import logging +from typing import Any, Dict, Iterable, List, Optional + +from nexent.memory.embedding_model import EmbeddingModelInfo +from nexent.memory.models import MemoryLayer +from nexent.vector_database.base import VectorDatabaseCore + +from services.vectordatabase_service import get_vector_db_core +from consts.const import VectorDatabaseType + + +logger = logging.getLogger("memory_index_service") +logger.setLevel(logging.INFO) + + +def _memory_chunk_payload( + record: Dict[str, Any], + embedding: List[float], + index_name: str, +) -> Dict[str, Any]: + """Translate a memory record into the chunk document shape used by ES. + + ``memory_id`` is an integer on the PG side; Elasticsearch ``_id`` is + always a string, so the integer is stringified before being used as + document id / path. + """ + memory_id = str(record["memory_id"]) + layer = record.get("layer", "") + is_agent_layer = layer == MemoryLayer.AGENT.value + chunk = { + "id": memory_id, + "title": f"Short-term Memory {memory_id}" if is_agent_layer else f"Long-term user memory {memory_id}", + "author": record.get("created_by"), + "date": record.get("update_time"), + "content": record.get("content", ""), + "embedding_model_name": index_name, + "embedding": embedding, + "create_time": record.get("create_time"), + "metadata": { + "memory_id": memory_id, + "tenant_id": record.get("tenant_id"), + "user_id": record.get("user_id"), + "agent_id": record.get("agent_id"), + "conversation_id": record.get("conversation_id"), + "layer": layer, + "memory_type": record.get("memory_type"), + "status": record.get("status"), + "idempotency_key": record.get("idempotency_key"), + }, + } + + return chunk + + +class MemoryIndexService: + """Elasticsearch-backed index for agent short-term memory.""" + + def __init__(self, vdb_core: Optional[VectorDatabaseCore] = None): + self._vdb_core = vdb_core + + @property + def vdb_core(self) -> VectorDatabaseCore: + if self._vdb_core is None: + self._vdb_core = get_vector_db_core(VectorDatabaseType.ELASTICSEARCH) + return self._vdb_core + + # ------------------------------------------------------------------ # + # Index lifecycle # + # ------------------------------------------------------------------ # + + def ensure_index(self, index_name: str, embedding_dim: Optional[int] = None) -> bool: + """Create the memory index if it does not exist.""" + try: + return self.vdb_core.create_index(index_name, embedding_dim=embedding_dim) + except Exception: + logger.exception("ensure_index failed for %s", index_name) + return False + + def drop_index(self, index_name: str) -> bool: + """Delete an entire memory index (used by tenant purge).""" + try: + return self.vdb_core.delete_index(index_name) + except Exception: + logger.exception("drop_index failed for %s", index_name) + return False + + # ------------------------------------------------------------------ # + # Document CRUD # + # ------------------------------------------------------------------ # + + def index_record( + self, + record: Dict[str, Any], + embedding: Optional[List[float]], + embedding_model_info: Optional[EmbeddingModelInfo] = None, + ) -> bool: + """Upsert a single memory record into its target index. + + Args: + record: Serialized memory record (must include ``memory_id``, + ``tenant_id``, ``user_id``, ``content``, etc.). + embedding: Embedding vector; required for vector search. + embedding_model_info: Used to resolve the index name when + ``record["es_index_name"]`` is missing. + + Returns: + True if the document was indexed (or already existed), False on + transport failure. + """ + index_name = record.get("es_index_name") + if not index_name and embedding_model_info is not None: + index_name = embedding_model_info.get_index_name() + if not index_name: + logger.warning("index_record: no index name resolved for memory_id=%s", + record.get("memory_id")) + return False + + embedding_dim = ( + embedding_model_info.dimension if embedding_model_info else None + ) + self.ensure_index(index_name, embedding_dim=embedding_dim) + + try: + self.vdb_core.create_chunk( + index_name=index_name, + chunk=_memory_chunk_payload(record, embedding, index_name), + ) + return True + except Exception: + logger.exception( + "index_record: failed to index memory_id=%s into %s", + record.get("memory_id"), + index_name, + ) + return False + + def delete_record(self, memory_id: int, index_name: str) -> bool: + """Remove a single memory document from ES.""" + try: + return bool(self.vdb_core.delete_chunk(index_name, str(memory_id))) + except Exception: + logger.exception("delete_record failed for %s", memory_id) + return False + + # ------------------------------------------------------------------ # + # Search # + # ------------------------------------------------------------------ # + + def search_similar( + self, + *, + index_name: str, + embedding: List[float], + tenant_id: str, + user_id: str, + agent_id: Optional[str] = None, + conversation_id: Optional[str] = None, + top_k: int = 5, + hybrid: bool = False, + query_text: Optional[str] = None, + weight_accurate: float = 0.3, + embedding_model: Any = None, + ) -> List[Dict[str, Any]]: + """Run a memory search scoped to the agent's isolation boundary. + + Args: + index_name: ES index carrying the tenant's per-embedding-model + memory chunks. + embedding: Pre-computed query embedding (used by the pure kNN + branch). + tenant_id / user_id / agent_id: isolation keys compiled into an + ES ``bool.filter`` so callers never see another tenant's, + user's, or agent's memories. ``conversation_id`` is retained + as record metadata but intentionally does not restrict agent + memory retrieval because agent memory is shared across the + user's conversations with the same agent. + top_k: Maximum number of hits to return. + hybrid: When ``True``, delegate to + :py:meth:`ElasticSearchCore.hybrid_search` so fuzzy + (BM25) and semantic (kNN) scores are blended. Requires + both ``query_text`` and ``embedding_model`` to be supplied. + When ``False`` (the default), behaviour is bit-for-bit + identical to the previous release: a single kNN query. + query_text: Raw user query, required when ``hybrid=True`` so the + fuzzy branch has something to score against. + weight_accurate: Weight of the fuzzy (BM25) branch in the hybrid + combination; ignored unless ``hybrid=True``. + embedding_model: Embedding model instance (carries the HTTP client + + dimension metadata) used by ``hybrid_search`` to re-vectorise + ``query_text``. Ignored unless ``hybrid=True``. + + Returns: + A list of normalized result dicts ready to be passed to + ``MemoryService._to_search_result``. + + Notes: + Elasticsearch dynamically maps string ``metadata.*`` fields to + ``text`` (analyzed). UUID-like values get tokenized by the + standard analyzer, so a bare ``term`` filter on + ``metadata.tenant_id`` would never match. We therefore query + the auto-generated ``.keyword`` sub-field for exact equality. + """ + if not index_name or not embedding: + return [] + + isolation_filter = self._build_isolation_filter( + tenant_id=tenant_id, + user_id=user_id, + agent_id=agent_id, + ) + + if hybrid: + if not query_text or embedding_model is None: + logger.warning( + "[ES_SEARCH] hybrid requested without query_text or " + "embedding_model; falling back to kNN path.", + ) + else: + return self._hybrid_search_similar( + index_name=index_name, + query_text=query_text, + embedding=embedding, + embedding_model=embedding_model, + isolation_filter=isolation_filter, + top_k=top_k, + weight_accurate=weight_accurate, + ) + + return self._knn_search_similar( + index_name=index_name, + embedding=embedding, + isolation_filter=isolation_filter, + top_k=top_k, + ) + + # ------------------------------------------------------------------ # + # Search helpers (split for readability and unit testing) # + # ------------------------------------------------------------------ # + + @staticmethod + def _build_isolation_filter( + *, + tenant_id: str, + user_id: str, + agent_id: Optional[str], + ) -> List[Dict[str, Any]]: + """Assemble the tenant + user + agent memory isolation filter.""" + must_filters: List[Dict[str, Any]] = [ + {"term": {"metadata.tenant_id.keyword": tenant_id}}, + {"term": {"metadata.user_id.keyword": user_id}}, + ] + if agent_id is not None: + must_filters.append({"term": {"metadata.agent_id.keyword": agent_id}}) + must_filters.append({"term": {"metadata.layer.keyword": "agent"}}) + return must_filters + + def _knn_search_similar( + self, + *, + index_name: str, + embedding: List[float], + isolation_filter: List[Dict[str, Any]], + top_k: int, + ) -> List[Dict[str, Any]]: + """Pure kNN branch (unchanged from the previous release).""" + query = { + "knn": { + "field": "embedding", + "query_vector": list(embedding), + "k": max(1, int(top_k)), + "num_candidates": max(50, int(top_k) * 10), + "filter": {"bool": {"must": isolation_filter}}, + }, + "size": max(1, int(top_k)), + } + + try: + response = self.vdb_core.search(index_name=index_name, query=query) + response_body = response.body if hasattr(response, "body") else response + except Exception: + logger.exception("search_similar failed for %s", index_name) + return [] + + hits = (response_body or {}).get("hits", {}).get("hits", []) or [] + + return [_hit_to_memory_result(hit) for hit in hits] + + def _hybrid_search_similar( + self, + *, + index_name: str, + query_text: str, + embedding: List[float], + embedding_model: Any, + isolation_filter: List[Dict[str, Any]], + top_k: int, + weight_accurate: float, + ) -> List[Dict[str, Any]]: + """Hybrid (BM25 + kNN) branch with the same isolation filter. + + Reuses ``ElasticSearchCore.hybrid_search``; we only widen the + filter so the existing knowledge-base business logic is unaffected + (the existing ``search_hybrid`` in ``vectordatabase_service`` never + passes this ``filter`` arg). + """ + from nexent.vector_database.elasticsearch_core import ElasticSearchCore + + vdb_core = self.vdb_core + # DataMate / mock backends don't accept ``filter``; fall back to + # the kNN path for them rather than risk a type error. + if not isinstance(vdb_core, ElasticSearchCore): + logger.warning( + "[ES_SEARCH] hybrid requested on non-ES backend %s; " + "falling back to kNN.", + type(vdb_core).__name__, + ) + return self._knn_search_similar( + index_name=index_name, + embedding=embedding, + isolation_filter=isolation_filter, + top_k=top_k, + ) + + try: + raw = vdb_core.hybrid_search( + index_names=[index_name], + query_text=query_text, + embedding_model=embedding_model, + top_k=max(1, int(top_k)), + weight_accurate=weight_accurate, + filter={"bool": {"must": isolation_filter}}, + ) + except TypeError: + # Defensive fallback if an older implementation missing the + # ``filter`` kwarg is wired in. Mirrors the legacy behaviour + # without the isolation filter, which is unsafe but matches + # the original SDK contract. + logger.warning( + "[ES_HYBRID] hybrid_search rejected ``filter`` kwarg; " + "retrying without isolation filter.", + ) + raw = vdb_core.hybrid_search( + index_names=[index_name], + query_text=query_text, + embedding_model=embedding_model, + top_k=max(1, int(top_k)), + weight_accurate=weight_accurate, + ) + except Exception: + logger.exception("hybrid_search failed for %s", index_name) + return [] + + results: List[Dict[str, Any]] = [] + for item in raw or []: + document = item.get("document") or {} + metadata = document.get("metadata") or {} + results.append({ + "memory_id": document.get("id") or metadata.get("memory_id"), + "content": document.get("content", ""), + "score": float(item.get("score") or 0.0), + "layer": metadata.get("layer", "agent"), + "memory_type": metadata.get("memory_type", "short_term"), + "source": "internal", + "is_external": False, + "metadata": metadata, + "score_details": item.get("scores", {}), + }) + + return results + + +def _safe_score_preview(response_body: Any) -> Any: + """Return a compact score summary for the ES response log line.""" + try: + hits = (response_body or {}).get("hits", {}).get("hits", []) or [] + if not hits: + return None + scores = [h.get("_score") for h in hits if h.get("_score") is not None] + if not scores: + return None + return {"min": min(scores), "max": max(scores)} + except Exception: + return None + + +def _hit_to_memory_result(hit: Dict[str, Any]) -> Dict[str, Any]: + """Translate a raw ES hit into the normalized memory result shape.""" + source = hit.get("_source") or {} + metadata = source.get("metadata") or {} + + return { + "memory_id": hit.get("_id") or source.get("id"), + "content": source.get("content", ""), + "score": float(hit.get("_score") or 0.0), + "layer": metadata.get("layer", "agent"), + "memory_type": metadata.get("memory_type", "short_term"), + "source": "internal", + "is_external": False, + "metadata": metadata, + } + + +# Singleton-style accessor matching the rest of the service modules. +_default_service: Optional[MemoryIndexService] = None + + +def get_memory_index_service() -> MemoryIndexService: + """Return a process-wide MemoryIndexService.""" + global _default_service + if _default_service is None: + _default_service = MemoryIndexService() + return _default_service + + +def reset_memory_index_service() -> None: + """Reset the cached service (used by tests).""" + global _default_service + _default_service = None diff --git a/backend/services/memory_record_service.py b/backend/services/memory_record_service.py new file mode 100644 index 0000000000..c8283963b1 --- /dev/null +++ b/backend/services/memory_record_service.py @@ -0,0 +1,595 @@ +"""Business logic for internal memory records (Phase 2). + +Layer rules: + +- ``tenant`` and ``user`` long-term memories are written to PostgreSQL only. +- ``agent`` short-term memory is written to PostgreSQL first, then mirrored + into Elasticsearch by :pymod:`services.memory_index_service`. If the + Elasticsearch side fails, the PG row is kept and the failure is logged so + that a future backfill job can retry. + +Idempotency: every write is keyed by ``(tenant_id, idempotency_key)`` so +replays do not create duplicates. The same key for a different tenant is a +distinct memory. +""" + +from __future__ import annotations + +import logging +from datetime import datetime, timezone +from typing import Any, Dict, List, Optional, Sequence + +from database import memory_record_db +from nexent.memory.embedding_model import ( + EmbeddingModelInfo, + get_embedding_client, +) +from nexent.memory.models import MemoryLayer, MemoryType +from nexent.memory.policy import MemoryAccessPolicy, MemoryStoragePolicy + +from .memory_index_service import MemoryIndexService, get_memory_index_service + + +logger = logging.getLogger("memory_record_service") +logger.setLevel(logging.INFO) + + +def _generate_idempotency_key() -> str: + """Return a fresh idempotency key (uuid4 string).""" + import uuid + + return str(uuid.uuid4()) + + +class MemoryRecordError(Exception): + """Raised when an internal memory operation cannot be completed.""" + + +# --------------------------------------------------------------------------- +# Public helpers (used by app and other services) +# --------------------------------------------------------------------------- + + +def _validate_layer_name(layer: Optional[str]) -> str: + """Validate that ``layer`` is one of the supported memory layers. + + Raises ``MemoryRecordError`` if the value is ``None`` or not in the + allowed set. Returns the normalised (lowercase, stripped) layer string. + """ + if layer is None: + raise MemoryRecordError("Layer cannot be None") + normalised = layer.strip().lower() + if normalised not in {"tenant", "user", "agent"}: + raise MemoryRecordError(f"Unsupported layer: {layer}") + return normalised + + +def _now_iso() -> str: + return datetime.utcnow().isoformat() + + +def _resolve_memory_type(layer: str, memory_type: Optional[str]) -> str: + """Return the canonical memory_type string for a layer. + + Tenant and user layers default to ``long_term``; agent defaults to + ``short_term``. The caller can override either explicitly. + """ + if memory_type: + return memory_type + if layer == MemoryLayer.AGENT.value: + return MemoryType.SHORT_TERM.value + return MemoryType.LONG_TERM.value + + +def _ensure_index_name( + record: Dict[str, Any], + embedding_model_info: Optional[EmbeddingModelInfo], +) -> Optional[str]: + """Resolve and stamp the ES index name for an agent short-term record.""" + if record.get("layer") != MemoryLayer.AGENT.value: + record.pop("es_index_name", None) + return None + index_name = record.get("es_index_name") + if not index_name and embedding_model_info is not None: + index_name = embedding_model_info.get_index_name() + if index_name: + record["es_index_name"] = index_name + return index_name + + +def _resolve_tenant_embedding_model_info( + tenant_id: str, +) -> Optional[EmbeddingModelInfo]: + """Return the embedding model selected by the tenant, or ``None``.""" + try: + from database.tenant_config_db import get_single_config_info + from utils.config_utils import tenant_config_manager + + record = get_single_config_info(tenant_id, "EMBEDDING_ID") + if not record or not record.get("config_value"): + logger.warning( + "No embedding model configured for tenant %s", + tenant_id, + ) + return None + model_config = tenant_config_manager.get_model_config( + "EMBEDDING_ID", + tenant_id=tenant_id, + ) + except Exception: + logger.exception( + "Failed to load configured tenant embedding model for tenant=%s", + tenant_id, + ) + return None + + if not model_config: + logger.warning( + "Configured EMBEDDING_ID does not resolve to a model for tenant=%s", + tenant_id, + ) + return None + + model_name = model_config.get("model_name") + base_url = model_config.get("base_url") + dimension = model_config.get("max_tokens") + if not all([model_name, base_url, dimension]): + logger.warning( + "Configured model is incomplete for tenant=%s: model_name=%s", + tenant_id, + model_name, + ) + return None + + try: + return EmbeddingModelInfo( + model_name=model_name, + model_repo=model_config.get("model_repo"), + dimension=int(dimension), + base_url=base_url, + api_key=model_config.get("api_key") or "sk-no-api-key", + ssl_verify=bool(model_config.get("ssl_verify", True)), + ) + except (TypeError, ValueError): + logger.exception( + "Configured model has invalid dimension for tenant=%s", + tenant_id, + ) + return None + + +def is_tenant_embedding_configured(tenant_id: str) -> bool: + """Return whether the tenant has an active ``EMBEDDING_ID`` row.""" + from database.tenant_config_db import get_single_config_info + + record = get_single_config_info(tenant_id, "EMBEDDING_ID") + return bool(record and record.get("config_value")) + + +def get_tenant_memory_index_name(tenant_id: str) -> Optional[str]: + """Return the memory index derived from the selected embedding model.""" + model_info = _resolve_tenant_embedding_model_info(tenant_id) + return model_info.get_index_name() if model_info is not None else None + + +def _compute_content_embedding( + content: str, + embedding_model_info: EmbeddingModelInfo, +) -> Optional[List[float]]: + """Compute an embedding for ``content`` using the resolved model.""" + try: + instance = get_embedding_client( + model_name=embedding_model_info.model_name, + dimension=embedding_model_info.dimension, + base_url=embedding_model_info.base_url, + api_key=embedding_model_info.api_key, + model_repo=embedding_model_info.model_repo, + ssl_verify=embedding_model_info.ssl_verify, + ) + embeddings = instance.get_embeddings( + content, timeout=30, retries=2, retry_timeout_step=5.0 + ) + if not embeddings: + return None + vector = embeddings[0] + if not isinstance(vector, list): + return None + return [float(value) for value in vector] + except Exception: + logger.exception( + "Failed to compute embedding for memory content using model=%s", + embedding_model_info.model_name, + ) + return None + + +def _validate_layer_policy(layer: str, memory_type: str, actor: str) -> None: + """Apply the access policy based on who is performing the write.""" + layer_enum = MemoryLayer(layer) + type_enum = MemoryType(memory_type) + + if actor == "agent": + if not MemoryAccessPolicy.can_agent_write(layer_enum, type_enum): + raise MemoryRecordError( + f"Agent cannot write to layer={layer}, type={memory_type}" + ) + elif actor == "dreaming": + if not MemoryAccessPolicy.can_dreaming_write(layer_enum, type_enum): + raise MemoryRecordError( + f"Dreaming cannot write to layer={layer}, type={memory_type}" + ) + elif actor in {"system", "user", "admin"}: + # System / manual management may write to any layer. + return + else: + raise MemoryRecordError(f"Unknown memory actor: {actor}") + + +class MemoryRecordService: + """High-level operations for ``memory_records_t``. + + The service is intentionally stateless apart from the injected index + service. Constructing multiple instances is safe and encouraged in tests. + """ + + def __init__(self, index_service: Optional[MemoryIndexService] = None): + self.index_service = index_service or get_memory_index_service() + + # ------------------------------------------------------------------ # + # Writes # + # ------------------------------------------------------------------ # + + def create_memory( + self, + *, + tenant_id: str, + user_id: str, + content: str, + layer: str = MemoryLayer.AGENT.value, + memory_type: Optional[str] = None, + agent_id: Optional[str] = None, + conversation_id: Optional[str] = None, + concept_tags: Optional[Sequence[str]] = None, + idempotency_key: Optional[str] = None, + embedding: Optional[List[float]] = None, + embedding_model_info: Optional[EmbeddingModelInfo] = None, + created_by: Optional[str] = None, + actor: str = "agent", + ) -> Dict[str, Any]: + """Create (or upsert) a memory record. + + Returns a dict with ``memory_id``, ``layer``, ``memory_type``, + ``event`` (``ADD`` / ``UPDATE``), and ``indexed`` (bool). + """ + resolved_type = _resolve_memory_type(layer, memory_type) + _validate_layer_policy(layer, resolved_type, actor) + + record: Dict[str, Any] = { + # ``memory_id`` is intentionally omitted - PostgreSQL ``serial4`` + # assigns the primary key on insert. + "tenant_id": tenant_id, + "user_id": user_id, + "agent_id": agent_id, + "conversation_id": conversation_id, + "layer": layer, + "memory_type": resolved_type, + "content": content, + "concept_tags": list(concept_tags or []), + "idempotency_key": idempotency_key or _generate_idempotency_key(), + "created_by": created_by, + "updated_by": created_by, + "status": "active", + "delete_flag": "N", + } + + index_name = _ensure_index_name(record, embedding_model_info) + + if ( + layer == MemoryLayer.AGENT.value + and not index_name + and not embedding_model_info + ): + resolved_model_info = _resolve_tenant_embedding_model_info(tenant_id) + if resolved_model_info is not None: + embedding_model_info = resolved_model_info + index_name = _ensure_index_name(record, embedding_model_info) + + if ( + actor == "agent" + and layer == MemoryLayer.AGENT.value + and embedding_model_info is None + ): + raise MemoryRecordError( + "Failed to store memory: tenant embedding model is not configured" + ) + + if not embedding and embedding_model_info is not None: + embedding = _compute_content_embedding(content, embedding_model_info) + + existing = memory_record_db.find_by_idempotency( + tenant_id=tenant_id, + idempotency_key=record["idempotency_key"], + ) + if existing is not None: + if existing.get("content") == content: + return { + "memory_id": existing.get("memory_id"), + "layer": layer, + "memory_type": resolved_type, + "event": "UNCHANGED", + "indexed": False, + } + memory_id = memory_record_db.upsert_memory_record_by_idempotency( + { + # Existing primary key preserved implicitly via + # ``(tenant_id, idempotency_key)`` lookup; do not re-pass + # ``memory_id`` so the database never tries to remap it. + "tenant_id": tenant_id, + "user_id": user_id, + "agent_id": agent_id, + "conversation_id": conversation_id, + "layer": layer, + "memory_type": resolved_type, + "content": content, + "concept_tags": list(concept_tags or []), + "idempotency_key": record["idempotency_key"], + "es_index_name": index_name, + "updated_by": created_by, + } + ) + indexed = False + if index_name and memory_id: + indexed = self.index_service.index_record( + record={ + "memory_id": memory_id, + "tenant_id": tenant_id, + "user_id": user_id, + "agent_id": agent_id, + "conversation_id": conversation_id, + "content": content, + "layer": layer, + "memory_type": resolved_type, + "status": "active", + "idempotency_key": record["idempotency_key"], + "created_by": created_by, + }, + embedding=embedding, + embedding_model_info=embedding_model_info, + ) + return { + "memory_id": memory_id, + "layer": layer, + "memory_type": resolved_type, + "event": "UPDATE", + "indexed": indexed, + } + + memory_id = memory_record_db.insert_memory_record(record) + if not memory_id: + raise MemoryRecordError("Failed to persist memory record") + + indexed = False + if index_name: + indexed = self.index_service.index_record( + record={ + "memory_id": memory_id, + "tenant_id": tenant_id, + "user_id": user_id, + "agent_id": agent_id, + "conversation_id": conversation_id, + "content": content, + "layer": layer, + "memory_type": resolved_type, + "status": "active", + "idempotency_key": record["idempotency_key"], + "created_by": created_by, + }, + embedding=embedding, + embedding_model_info=embedding_model_info, + ) + + return { + "memory_id": memory_id, + "layer": layer, + "memory_type": resolved_type, + "event": "ADD", + "indexed": indexed, + } + + def update_memory( + self, + memory_id: int, + tenant_id: str, + update_data: Dict[str, Any], + *, + actor: str = "system", + ) -> bool: + """Update an existing memory record. + + If ``content`` is changed on an agent-layer record, the ES mirror is + re-indexed with a freshly computed embedding vector so that the + semantic-search index stays in sync. + """ + if "layer" in update_data and "memory_type" in update_data: + _validate_layer_policy( + update_data["layer"], update_data["memory_type"], actor + ) + + record = memory_record_db.get_memory_record(memory_id, tenant_id) + if record is None: + return False + + ok = memory_record_db.update_memory_record( + memory_id, tenant_id, update_data + ) + if not ok: + return False + + content_changed = "content" in update_data + layer = update_data.get("layer") or record.get("layer") + if content_changed and layer == MemoryLayer.AGENT.value: + new_content = update_data["content"] + embedding_model_info = _resolve_tenant_embedding_model_info(tenant_id) + embedding = None + if embedding_model_info is not None: + embedding = _compute_content_embedding(new_content, embedding_model_info) + self.index_service.index_record( + record={ + "memory_id": memory_id, + "tenant_id": tenant_id, + "user_id": record.get("user_id"), + "agent_id": record.get("agent_id"), + "conversation_id": record.get("conversation_id"), + "content": new_content, + "layer": record.get("layer"), + "memory_type": record.get("memory_type"), + "status": record.get("status"), + "idempotency_key": record.get("idempotency_key"), + "created_by": record.get("created_by"), + }, + embedding=embedding, + embedding_model_info=embedding_model_info, + ) + + return True + + def soft_delete_memory( + self, + memory_id: int, + tenant_id: str, + *, + updated_by: Optional[str] = None, + cascade_index: bool = True, + ) -> bool: + """Soft-delete a record. The ES mirror is removed best-effort.""" + record = memory_record_db.get_memory_record(memory_id, tenant_id) + if record is None: + return False + ok = memory_record_db.soft_delete_memory_record( + memory_id, tenant_id, updated_by=updated_by + ) + if ok and cascade_index: + index_name = record.get("es_index_name") + if index_name: + self.index_service.delete_record(memory_id, index_name) + return ok + + # ------------------------------------------------------------------ # + # Reads # + # ------------------------------------------------------------------ # + + def get_memory( + self, memory_id: int, tenant_id: str + ) -> Optional[Dict[str, Any]]: + return memory_record_db.get_memory_record(memory_id, tenant_id) + + def get_memory_for_user( + self, + memory_id: int, + tenant_id: str, + user_id: str, + ) -> Optional[Dict[str, Any]]: + """Fetch a memory record with cross-user visibility enforced. + + Tenant-layer records are always visible within the tenant. User-layer + records are scoped to the owning user. A ``None`` is returned if the + record does not exist or is not accessible to the caller. + """ + record = memory_record_db.get_memory_record(memory_id, tenant_id) + if record is None: + return None + if record.get("user_id") and record["user_id"] != user_id: + if record.get("layer") not in {"tenant"}: + return None + return record + + def list_memories( + self, + tenant_id: str, + *, + user_id: Optional[str] = None, + agent_id: Optional[str] = None, + conversation_id: Optional[str] = None, + layer: Optional[str] = None, + memory_type: Optional[str] = None, + status: Optional[str] = "active", + limit: int = 100, + offset: int = 0, + include_deleted: bool = False, + ) -> List[Dict[str, Any]]: + rows = memory_record_db.list_memory_records( + tenant_id, + user_id=user_id, + agent_id=agent_id, + conversation_id=conversation_id, + layer=layer, + memory_type=memory_type, + status=status, + limit=limit, + offset=offset, + include_deleted=include_deleted, + ) + current_index_name = get_tenant_memory_index_name(tenant_id) + for row in rows: + if row.get("layer") == MemoryLayer.AGENT.value: + row["embedding_compatible"] = bool( + current_index_name + and row.get("es_index_name") == current_index_name + ) + return rows + + def list_full_context_memories( + self, + tenant_id: str, + *, + user_id: Optional[str] = None, + layers: Optional[Sequence[str]] = None, + ) -> List[Dict[str, Any]]: + """Return all active memories for the full-context layers. + + Tenant/user memories are always loaded in full per the retrieval + policy. ``layers`` defaults to ``("tenant", "user")``. + """ + if layers is None: + layers = ( + MemoryLayer.TENANT.value, + MemoryLayer.USER.value, + ) + rows: List[Dict[str, Any]] = [] + for layer in layers: + if not MemoryStoragePolicy.uses_full_context_for_layer(layer): + # Skip layers that require vector search; callers that want + # them should use ``MemoryRetrievalService`` instead. + continue + rows.extend( + self.list_memories( + tenant_id, + user_id=user_id, + layer=layer, + memory_type=MemoryType.LONG_TERM.value, + limit=1000, + ) + ) + return rows + + +# --------------------------------------------------------------------------- +# Module-level accessors +# --------------------------------------------------------------------------- + + +_default_service: Optional[MemoryRecordService] = None + + +def get_memory_record_service() -> MemoryRecordService: + """Return the process-wide ``MemoryRecordService``.""" + global _default_service + if _default_service is None: + _default_service = MemoryRecordService() + return _default_service + + +def reset_memory_record_service() -> None: + """Reset the cached service (used by tests).""" + global _default_service + _default_service = None diff --git a/backend/services/memory_retrieval_service.py b/backend/services/memory_retrieval_service.py new file mode 100644 index 0000000000..cd41d7d145 --- /dev/null +++ b/backend/services/memory_retrieval_service.py @@ -0,0 +1,418 @@ +"""Memory retrieval orchestration (Phase 2). + +This service combines the two retrieval paths required by the design: + +- **Full-context layers** (``tenant``, ``user``) are loaded verbatim from + PostgreSQL. They are not vector-searched because they always fit in the + context window (admin-curated tenant memory and the user's personal + long-term memory). +- **Agent short-term memory** is retrieved via kNN against Elasticsearch, + with the isolation scope enforced both at the SQL and ES levels. + +A successful search also appends one ``memory_retrieval_hits_t`` row per +hit so Dreaming can aggregate recall statistics in batch. +""" + +from __future__ import annotations + +import json +import hashlib +import logging +import os +import threading +from datetime import datetime +from typing import Any, Dict, List, Optional + +from nexent.memory.embedding_model import EmbeddingModelInfo +from nexent.memory.models import MemoryLayer, MemorySearchRequest, MemorySearchResult +from nexent.memory.policy import MemoryRetrievalPolicy + +from database import memory_record_db, memory_retrieval_hit_db +from services.memory_index_service import ( + MemoryIndexService, + get_memory_index_service, +) +from services.memory_record_service import ( + MemoryRecordService, + _compute_content_embedding, + _resolve_tenant_embedding_model_info, + get_memory_record_service, +) + + +logger = logging.getLogger("memory_retrieval_service") +logger.setLevel(logging.INFO) + + +def _hash_query(query: str) -> str: + return hashlib.sha256(query.encode("utf-8")).hexdigest() + + +def _iso_day(timestamp: Optional[datetime] = None) -> str: + return (timestamp or datetime.utcnow()).date().isoformat() + + +def _serialize_record_as_result( + record: Dict[str, Any], + score: float = 1.0, + is_external: bool = False, +) -> MemorySearchResult: + layer_value = record.get("layer") + try: + layer_enum = MemoryLayer(layer_value) if layer_value else MemoryLayer.USER + except ValueError: + layer_enum = MemoryLayer.USER + return MemorySearchResult( + memory_id=record.get("memory_id"), + external_id=None, + content=record.get("content", ""), + score=float(score), + layer=layer_enum, + source="internal", + is_external=is_external, + metadata={ + "tenant_id": record.get("tenant_id"), + "user_id": record.get("user_id"), + "agent_id": record.get("agent_id"), + "conversation_id": record.get("conversation_id"), + "memory_type": record.get("memory_type"), + "status": record.get("status"), + "concept_tags": record.get("concept_tags") or [], + }, + ) + + +class MemoryRetrievalService: + """Composite retrieval service (PG + ES) for internal memory.""" + + def __init__( + self, + record_service: Optional[MemoryRecordService] = None, + index_service: Optional[MemoryIndexService] = None, + ): + self.record_service = record_service or get_memory_record_service() + self.index_service = index_service or get_memory_index_service() + + # ------------------------------------------------------------------ # + # Public API # + # ------------------------------------------------------------------ # + + async def search( + self, + request: MemorySearchRequest, + *, + embedding_model_info: Optional[EmbeddingModelInfo] = None, + write_hits: bool = True, + ) -> List[MemorySearchResult]: + """Run a retrieval against the requested layers. + + ``layers`` controls which layers are queried. Layers not supported + by the current backend (e.g. agent without ES) return empty. + """ + top_k = MemoryRetrievalPolicy.validate_top_k(request.top_k) + results: List[MemorySearchResult] = [] + + layers = request.layers or [ + MemoryLayer.TENANT, + MemoryLayer.USER, + MemoryLayer.AGENT, + ] + + for layer in layers: + if MemoryRetrievalPolicy.uses_full_context(layer): + results.extend( + self._full_context_search(request=request, layer=layer.value) + ) + elif MemoryRetrievalPolicy.uses_vector_search(layer): + results.extend( + self._vector_search( + request=request, + layer=layer.value, + top_k=top_k, + embedding_model_info=embedding_model_info, + ) + ) + else: + logger.warning("Unsupported layer: %s", layer) + + if write_hits and results: + self._record_hits(request=request, results=results) + + return results[:top_k] + + async def search_memories( + self, + tenant_id: str, + user_id: str, + query: str, + *, + agent_id: Optional[str] = None, + conversation_id: Optional[str] = None, + layers: Optional[List[str]] = None, + top_k: int = 5, + threshold: float = 0.65, + write_hits: bool = True, + hybrid: bool = False, + weight_accurate: float = 0.3, + ) -> List[MemorySearchResult]: + """High-level memory search for the app layer. + + Accepts plain string layer names and resolves the tenant embedding + model internally so callers do not need to handle that plumbing. + + When ``hybrid`` is true the agent short-term branch is delegated + to ``ElasticSearchCore.hybrid_search`` so fuzzy (BM25) and + semantic (kNN) scores are blended; ``weight_accurate`` controls + the BM25 weight. Defaults preserve the legacy pure-kNN path. + + Returns a list of :class:`MemorySearchResult`. + """ + # Parse and validate layer names. + resolved_layers: List[MemoryLayer] = [] + defaults = ["agent"] if layers is None else layers + for value in defaults: + try: + resolved_layers.append(MemoryLayer(value.strip().lower())) + except ValueError: + logger.warning("Skipping unknown layer: %s", value) + + # Resolve embedding model for the agent layer. + embedding_model_info = _resolve_tenant_embedding_model_info(tenant_id) + + # Compute query embedding when a model is available. + embedding: Optional[List[float]] = None + if query and embedding_model_info: + embedding = _compute_content_embedding(query, embedding_model_info) + + request = MemorySearchRequest( + tenant_id=tenant_id, + user_id=user_id, + agent_id=agent_id, + conversation_id=conversation_id, + layers=resolved_layers, + query=query, + top_k=top_k, + threshold=threshold, + embedding=embedding, + hybrid=hybrid, + weight_accurate=weight_accurate, + ) + + return await self.search( + request, embedding_model_info=embedding_model_info, write_hits=write_hits + ) + + # ------------------------------------------------------------------ # + # Layer-specific strategies # + # ------------------------------------------------------------------ # + + def _full_context_search( + self, + *, + request: MemorySearchRequest, + layer: str, + ) -> List[MemorySearchResult]: + rows = self.record_service.list_memories( + tenant_id=request.tenant_id, + user_id=request.user_id, + layer=layer, + memory_type="long_term", + status="active", + limit=1000, + ) + return [_serialize_record_as_result(row, score=1.0) for row in rows] + + def _vector_search( + self, + *, + request: MemorySearchRequest, + layer: str, + top_k: int, + embedding_model_info: Optional[EmbeddingModelInfo], + ) -> List[MemorySearchResult]: + embedding = request.embedding + if embedding is None or not embedding_model_info: + logger.warning("Early return: embedding or model_info is None") + return [] + + index_name = embedding_model_info.get_index_name() + if not index_name: + logger.warning("No index name found") + return [] + + # The hybrid branch in ``search_similar`` needs an actual + # ``OpenAICompatibleEmbedding`` instance so it can re-vectorise + # ``query_text`` via ``hybrid_search``. Build it lazily so that the + # default (hybrid=False) path doesn't pay this cost. + embedding_client = None + if getattr(request, "hybrid", False): + try: + from nexent.memory.embedding_model import get_embedding_client + embedding_client = get_embedding_client( + model_name=embedding_model_info.model_name, + dimension=embedding_model_info.dimension, + base_url=embedding_model_info.base_url, + api_key=embedding_model_info.api_key, + model_repo=embedding_model_info.model_repo, + ssl_verify=embedding_model_info.ssl_verify, + ) + except Exception: + logger.exception( + "Failed to build embedding client for hybrid: search_similar will fall back to kNN.", + ) + + raw_hits = self.index_service.search_similar( + index_name=index_name, + embedding=list(embedding), + tenant_id=request.tenant_id, + user_id=request.user_id, + agent_id=request.agent_id, + conversation_id=request.conversation_id, + top_k=top_k, + hybrid=getattr(request, "hybrid", False), + query_text=request.query if getattr(request, "hybrid", False) else None, + weight_accurate=getattr(request, "weight_accurate", 0.3), + embedding_model=embedding_client, + ) + + if not raw_hits: + return [] + + # Apply threshold filter; fall back to policy default if None. + threshold = ( + request.threshold + if request.threshold is not None + else MemoryRetrievalPolicy.DEFAULT_THRESHOLD + ) + + results: List[MemorySearchResult] = [] + memory_ids: List[int] = [] + for hit in raw_hits: + if hit["score"] < threshold: + continue + try: + memory_id_int = int(hit["memory_id"]) + except (TypeError, ValueError): + logger.warning( + "Ignoring non-integer memory_id from ES: %r", + hit.get("memory_id"), + ) + continue + memory_ids.append(memory_id_int) + try: + layer_enum = MemoryLayer(hit.get("layer") or layer) + except ValueError: + layer_enum = MemoryLayer.AGENT + results.append( + MemorySearchResult( + memory_id=memory_id_int, + external_id=None, + content=hit.get("content", ""), + score=float(hit.get("score", 0.0)), + layer=layer_enum, + source="internal", + is_external=False, + metadata=hit.get("metadata", {}), + ) + ) + + # Backfill the PG row so callers can fetch full record details. + if memory_ids: + rows = memory_record_db.get_memory_records_by_ids( + memory_ids, request.tenant_id + ) + by_id = {row["memory_id"]: row for row in rows} + enriched: List[MemorySearchResult] = [] + for result in results: + try: + key = int(result.memory_id) if result.memory_id else None + except (TypeError, ValueError): + key = None + row = by_id.get(key) if key is not None else None + if row is None: + enriched.append(result) + continue + result.metadata = { + **result.metadata, + "memory_type": row.get("memory_type"), + "status": row.get("status"), + "concept_tags": row.get("concept_tags") or [], + } + enriched.append(result) + return enriched + + return results + + # ------------------------------------------------------------------ # + # Hit logging # + # ------------------------------------------------------------------ # + + def _record_hits( + self, + *, + request: MemorySearchRequest, + results: List[MemorySearchResult], + ) -> None: + now = datetime.utcnow() + day = _iso_day(now) + query_hash = _hash_query(request.query) + + rows: List[Dict[str, Any]] = [] + for result in results: + if not result.memory_id: + continue + try: + memory_id_int = int(result.memory_id) + except (TypeError, ValueError): + logger.warning( + "record_hits: skipping non-integer memory_id %r", + result.memory_id, + ) + continue + rows.append( + { + "tenant_id": request.tenant_id, + "user_id": request.user_id, + "agent_id": request.agent_id, + "conversation_id": request.conversation_id, + "memory_id": memory_id_int, + "query_text": request.query, + "query_hash": query_hash, + "retrieval_score": result.score, + "source": "nexent", + "occurred_at": now, + "day": day, + "grounded": False, + } + ) + + if rows: + try: + memory_retrieval_hit_db.insert_retrieval_hits(rows) + except Exception: + logger.exception( + "search: failed to record retrieval hits for tenant=%s", + request.tenant_id, + ) + + +# --------------------------------------------------------------------------- +# Module-level accessors +# --------------------------------------------------------------------------- + + +_default_service: Optional[MemoryRetrievalService] = None + + +def get_memory_retrieval_service() -> MemoryRetrievalService: + """Return the process-wide retrieval service.""" + global _default_service + if _default_service is None: + _default_service = MemoryRetrievalService() + return _default_service + + +def reset_memory_retrieval_service() -> None: + """Reset the cached service (used by tests).""" + global _default_service + _default_service = None \ No newline at end of file diff --git a/backend/services/model_health_service.py b/backend/services/model_health_service.py index 5d472799d0..2d0c43d09f 100644 --- a/backend/services/model_health_service.py +++ b/backend/services/model_health_service.py @@ -3,7 +3,7 @@ from nexent.core import MessageObserver from nexent.core.models import OpenAIModel, OpenAIVLModel -from nexent.core.models.embedding_model import JinaEmbedding, OpenAICompatibleEmbedding, DashScopeMultimodalEmbedding +from nexent.core.models.embedding_model import JinaEmbedding, OpenAICompatibleEmbedding, DashScopeMultimodalEmbedding, SiliconflowMultimodalEmbedding from nexent.monitor import set_monitoring_context, set_monitoring_operation from nexent.core.models.rerank_model import OpenAICompatibleRerank @@ -17,6 +17,7 @@ DASHSCOPE_MODEL_FACTORY = "dashscope" TOKENPONY_MODEL_FACTORY = "tokenpony" +SILICONFLOW_MODEL_FACTORY = "silicon" PROVIDER_CATALOG_HEALTHCHECK_FACTORIES = {DASHSCOPE_MODEL_FACTORY, TOKENPONY_MODEL_FACTORY} PROVIDER_CATALOG_HEALTHCHECK_TYPES = {"vlm", "vlm2", "vlm3"} @@ -38,14 +39,16 @@ def _normalize_embedding_url(base_url: str) -> str: def _infer_model_factory(model_type: str, base_url: str, current_factory: Optional[str] = None) -> Optional[str]: """Infer model_factory from base_url if not already set or is generic. - For embedding/multi_embedding, uses legacy logic (only dashscope) to avoid - changing existing behavior. For other types (VLM), uses extended inference - so tokenpony URLs can be recognized for catalog healthcheck. + For embedding/multi_embedding, recognizes dashscope and siliconflow URLs. + For other types (VLM), uses extended inference so tokenpony URLs can be + recognized for catalog healthcheck. """ - # Embedding types: keep legacy behavior (only dashscope) + # Embedding types: infer from base_url host if model_type in EMBEDDING_TYPES: if "dashscope" in base_url.lower(): return DASHSCOPE_MODEL_FACTORY + if "siliconflow" in base_url.lower(): + return SILICONFLOW_MODEL_FACTORY # stored as "silicon" for catalog consistency return current_factory # Non-embedding types (VLM, etc): use extended inference @@ -101,7 +104,7 @@ async def _embedding_dimension_check( ssl_verify=ssl_verify, ) else: - embedding_instance = JinaEmbedding( + embedding_instance = SiliconflowMultimodalEmbedding( api_key=model_api_key, base_url=model_base_url, model_name=model_name, @@ -199,7 +202,7 @@ async def _perform_connectivity_check( ssl_verify=ssl_verify, ) else: - embedding = JinaEmbedding( + embedding = SiliconflowMultimodalEmbedding( api_key=model_api_key, base_url=model_base_url, model_name=model_name, @@ -354,7 +357,13 @@ async def check_model_connectivity(display_name: str, tenant_id: str, model_type "connect_status": ModelConnectStatusEnum.UNAVAILABLE.value} logger.error(f"Error checking model connectivity: {str(e)}") update_model_record(model["model_id"], update_data) - raise e + if isinstance(e, ValueError): + raise e + return { + "connectivity": False, + "model_name": model_name, + "error": str(e), + } if connectivity: logger.info( @@ -367,10 +376,11 @@ async def check_model_connectivity(display_name: str, tenant_id: str, model_type if ssl_verify_fallback: update_data["ssl_verify"] = False update_model_record(model["model_id"], update_data) - return { + result = { "connectivity": connectivity, "model_name": model_name, } + return result except Exception as e: logger.error(f"Error checking model connectivity: {str(e)}") if 'model' in locals() and model: diff --git a/backend/services/model_management_service.py b/backend/services/model_management_service.py index 3a14708017..f2594563e0 100644 --- a/backend/services/model_management_service.py +++ b/backend/services/model_management_service.py @@ -39,9 +39,6 @@ split_repo_name, sort_models_by_id, ) -from utils.memory_utils import build_memory_config as build_memory_config_for_tenant -from services.vectordatabase_service import get_vector_db_core -from nexent.memory.memory_service import clear_model_memories logger = logging.getLogger("model_management_service") @@ -662,31 +659,13 @@ async def delete_model_for_tenant(user_id: str, tenant_id: str, display_name: st ) if has_multi_embedding: - # Best-effort memory cleanup for embedding models - try: - vdb_core = get_vector_db_core() - base_memory_config = build_memory_config_for_tenant(tenant_id) - for m in models: - try: - await clear_model_memories( - vdb_core=vdb_core, - model_repo=m.get("model_repo", ""), - model_name=m.get("model_name", ""), - embedding_dims=int(m.get("max_tokens") or 0), - base_memory_config=base_memory_config, - ) - except Exception as cleanup_exc: - logger.warning( - "Best-effort clear_model_memories failed for %s/%s dims=%s: %s", - m.get("model_repo", ""), - m.get("model_name", ""), - m.get("max_tokens"), - cleanup_exc, - ) - except Exception as outer_cleanup_exc: - logger.warning( - "Memory cleanup preparation failed: %s", outer_cleanup_exc) - + # Best-effort memory cleanup for embedding models is performed by + # the embedding-model registry / ``VectorIndexService`` in the + # new Memory system. The legacy ``clear_model_memories`` call + # (mem0-era) has been removed; the new "drop ES indexes for an + # embedding model" path will be added once the storage layer + # lands. + pass # Delete all records with the same display_name for m in models: delete_model_record(m["model_id"], user_id, tenant_id) diff --git a/backend/services/nl2agent_service.py b/backend/services/nl2agent_service.py new file mode 100644 index 0000000000..6031a64d15 --- /dev/null +++ b/backend/services/nl2agent_service.py @@ -0,0 +1,294 @@ +"""Business logic for the ephemeral NL2Agent runtime.""" + +import ast +import asyncio +import json +import logging +import re +import threading +import unicodedata +from collections.abc import AsyncIterator +from typing import Any +from urllib.parse import urljoin + +from nexent.core.agents.agent_model import AgentHistory, AgentRunInfo +from nexent.core.agents.context import ContextManagerConfig +from nexent.core.agents.run_agent import agent_run +from nexent.core.utils.observer import MessageObserver +from rapidfuzz import fuzz + +from agents.create_agent_info import ( + _resolve_input_budget, + _resolve_safe_input_budget, + create_model_config_list, + join_minio_file_description_to_query, +) +from agents.nl2agent_agent import create_nl2agent_agent_config +from consts.const import LOCAL_MCP_SERVER, MODEL_CONFIG_MAPPING +from consts.model import HistoryItem, NL2AgentRunRequest, ToolSourceEnum +from database.tool_db import query_all_tools +from tool_collection.mcp.nl2agent_mcp_tools import InstalledMcpToolRecommendation +from utils.config_utils import tenant_config_manager +from utils.context_utils import build_authorized_context_input + +logger = logging.getLogger(__name__) + +MINIMUM_RECOMMENDATION_SCORE = 0.45 +MAX_RECOMMENDATIONS = 5 + + +def _normalize_search_text(value: Any) -> str: + """Normalize catalog text before fuzzy matching.""" + + if value is None: + return "" + normalized = unicodedata.normalize("NFKC", str(value)).casefold().strip() + return re.sub(r"\s+", " ", normalized) + + +def _normalize_labels(value: Any) -> list[str]: + """Return a safe list of display labels.""" + + if not isinstance(value, list): + return [] + return [str(label) for label in value if label is not None] + + +def _collapse_whitespace(value: str) -> str: + return re.sub(r"\s+", " ", value).strip() + + +def _normalize_input_strings(value: Any) -> Any: + if isinstance(value, str): + return _collapse_whitespace(value) + if isinstance(value, dict): + return { + key: _normalize_input_strings(item) + for key, item in value.items() + } + if isinstance(value, (list, tuple)): + return [_normalize_input_strings(item) for item in value] + return value + + +def _parse_tool_inputs(value: Any) -> dict[str, Any]: + if isinstance(value, dict): + parsed = value + elif isinstance(value, str): + try: + parsed = json.loads(value) + except json.JSONDecodeError: + try: + parsed = ast.literal_eval(value) + except (SyntaxError, ValueError): + return {} + else: + return {} + + if not isinstance(parsed, dict): + return {} + return _normalize_input_strings(parsed) + + +def _build_tool_document(tool: dict[str, Any]) -> str: + labels = " ".join(_normalize_labels(tool.get("labels"))) + return _normalize_search_text( + " ".join( + str(part) + for part in ( + tool.get("name") or "", + tool.get("origin_name") or "", + tool.get("description") or "", + labels, + tool.get("usage") or "", + ) + if part + ) + ) + + +def search_installed_mcp_tools_by_query( + tenant_id: str, + query_text: str, + limit: int = MAX_RECOMMENDATIONS, +) -> list[InstalledMcpToolRecommendation]: + """Return the best installed MCP tool matches for normalized query text.""" + + query = _normalize_search_text(query_text) + scored_tools: list[tuple[float, int, dict[str, Any]]] = [] + + for tool in query_all_tools(tenant_id=tenant_id): + if tool.get("source") != ToolSourceEnum.MCP.value: + continue + if tool.get("is_available") is not True: + continue + + document = _build_tool_document(tool) + if not document: + continue + + score = ( + max( + fuzz.WRatio(query, document), + fuzz.token_set_ratio(query, document), + ) + / 100 + ) + if score < MINIMUM_RECOMMENDATION_SCORE: + continue + + tool_id = int(tool["tool_id"]) + scored_tools.append((score, tool_id, tool)) + + scored_tools.sort(key=lambda item: (-item[0], item[1])) + result_limit = max(0, min(limit, MAX_RECOMMENDATIONS)) + + return [ + InstalledMcpToolRecommendation( + tool_id=tool_id, + name=str(tool.get("name") or ""), + origin_name=( + str(tool["origin_name"]) + if tool.get("origin_name") is not None + else None + ), + description=_collapse_whitespace( + str(tool.get("description") or "") + ), + usage=str(tool.get("usage") or ""), + labels=_normalize_labels(tool.get("labels")), + inputs=_parse_tool_inputs(tool.get("inputs")), + score=round(score, 4), + ) + for score, tool_id, tool in scored_tools[:result_limit] + ] + + +def _convert_history(history: list[HistoryItem] | None) -> list[AgentHistory]: + if not history: + return [] + return [ + AgentHistory(role=item.role, content=item.content) + for item in history + if item.role in {"user", "assistant"} + ] + + +async def build_nl2agent_run_info( + request: NL2AgentRunRequest, + tenant_id: str, + language: str, + authorization: str | None, +) -> AgentRunInfo: + """Build all request-scoped NL2Agent runtime objects in memory.""" + + final_query = await join_minio_file_description_to_query( + minio_files=request.minio_files, + query=request.query, + history=request.history, + ) + model_config_list = await create_model_config_list(tenant_id) + agent_config = create_nl2agent_agent_config(language) + default_model = tenant_config_manager.get_model_config( + key=MODEL_CONFIG_MAPPING["llm"], + tenant_id=tenant_id, + ) + input_budget, capacity_snapshot, resolved_capacity_snapshot = ( + _resolve_input_budget(default_model) + ) + safe_input_budget_snapshot = _resolve_safe_input_budget( + capacity_snapshot=resolved_capacity_snapshot, + tenant_id=tenant_id, + agent_requested_output_tokens=None, + request_requested_output_tokens=None, + ) + if safe_input_budget_snapshot is not None: + soft_input_budget_tokens = safe_input_budget_snapshot[ + "soft_input_budget_tokens" + ] + hard_input_budget_tokens = safe_input_budget_snapshot[ + "hard_input_budget_tokens" + ] + token_threshold = soft_input_budget_tokens + else: + soft_input_budget_tokens = 0 + hard_input_budget_tokens = 0 + token_threshold = input_budget + + context_window_tokens = ( + resolved_capacity_snapshot.context_window_tokens + if resolved_capacity_snapshot is not None + and resolved_capacity_snapshot.context_window_tokens is not None + else input_budget + ) + agent_config.context_manager_config = ContextManagerConfig( + token_threshold=token_threshold, + context_window_tokens=context_window_tokens, + soft_input_budget_tokens=soft_input_budget_tokens, + hard_input_budget_tokens=hard_input_budget_tokens, + ) + agent_config.capacity_snapshot = capacity_snapshot + agent_config.safe_input_budget_snapshot = safe_input_budget_snapshot + mcp_config: dict[str, Any] = { + "url": urljoin(LOCAL_MCP_SERVER, "sse"), + "transport": "sse", + } + if authorization: + mcp_config["headers"] = {"Authorization": authorization} + + run_info = AgentRunInfo( + query=final_query, + model_config_list=model_config_list, + observer=MessageObserver( + lang=language, + enable_nl2a_wrapper=True, + ), + agent_config=agent_config, + mcp_host=[mcp_config], + history=_convert_history(request.history), + stop_event=threading.Event(), + capacity_snapshot=capacity_snapshot, + safe_input_budget_snapshot=safe_input_budget_snapshot, + enable_planning=False, + sandbox_config=None, + redis_client=None, + ) + run_info.context_input = build_authorized_context_input(run_info) + return run_info + + +async def create_nl2agent_stream( + request: NL2AgentRunRequest, + tenant_id: str, + language: str, + authorization: str | None, +) -> AsyncIterator[str]: + """Create an SSE-compatible stream for one ephemeral NL2Agent run.""" + + run_info = await build_nl2agent_run_info( + request=request, + tenant_id=tenant_id, + language=language, + authorization=authorization, + ) + + async def generate() -> AsyncIterator[str]: + try: + async for chunk in agent_run(run_info): + yield f"data: {chunk}\n\n" + except asyncio.CancelledError: + raise + except Exception: + logger.exception("NL2Agent execution failed") + error_payload = json.dumps( + { + "type": "error", + "content": "NL2Agent execution failed.", + }, + ensure_ascii=False, + ) + yield f"data: {error_payload}\n\n" + finally: + run_info.stop_event.set() + + return generate() diff --git a/backend/services/northbound_service.py b/backend/services/northbound_service.py index 46c60eae88..a8cf3c7ae8 100644 --- a/backend/services/northbound_service.py +++ b/backend/services/northbound_service.py @@ -23,12 +23,12 @@ ConversationNotFoundError, ) from consts.model import AgentRequest, ToolParamsRequest -from database.conversation_db import get_conversation_messages, get_source_searches_by_message +from database.conversation_db import get_conversation_messages from database.token_db import log_token_usage, get_latest_usage_metadata from services.agent_service import ( run_agent_stream, stop_agent_tasks, - get_agent_id_by_name + get_agent_by_name_impl, ) from services.runtime_state_service import runtime_state_service from services.agent_version_service import list_published_agents_impl @@ -349,11 +349,6 @@ def _build_title_update_idempotency_key(tenant_id: str, conversation_id: int, ti # ----------------------------- # Agent resolver # ----------------------------- -async def get_agent_info_by_name(agent_name: str, tenant_id: str) -> int: - try: - return await get_agent_id_by_name(agent_name=agent_name, tenant_id=tenant_id) - except Exception as _: - raise Exception(f"Failed to get agent id for agent_name: {agent_name} in tenant_id: {tenant_id}") async def start_streaming_chat( @@ -382,7 +377,9 @@ async def start_streaming_chat( # Get history according to internal_conversation_id history_resp = await get_conversation_history_internal(ctx, internal_conversation_id) - agent_id = await get_agent_id_by_name(agent_name=agent_name, tenant_id=ctx.tenant_id) + agent_info = get_agent_by_name_impl(agent_name=agent_name, tenant_id=ctx.tenant_id) + agent_id = agent_info["agent_id"] + latest_version_no = agent_info["latest_version_no"] normalized_attachments = _normalize_northbound_attachments( attachments=attachments, user_id=ctx.user_id, @@ -400,12 +397,23 @@ async def start_streaming_chat( is_debug=False, tool_params=tool_params, model_id=model_id, + version_no=latest_version_no, + enable_automation_tool=False, ) - # Synchronously persist the user message before starting the stream to avoid race conditions + # Persist the user message off the event loop before starting the stream. + # We deliberately keep this synchronous step (not async submit) for + # northbound reliability -- external callers may not have SSE reconnect + # capability, so a late INSERT failure after the stream starts would + # silently lose the user message. asyncio.to_thread avoids blocking + # the event loop while preserving the synchronous commit semantics. try: - save_conversation_user( - agent_request, user_id=ctx.user_id, tenant_id=ctx.tenant_id) + await asyncio.to_thread( + save_conversation_user, + agent_request, + ctx.user_id, + ctx.tenant_id, + ) except Exception as e: raise Exception(f"Failed to persist user message: {str(e)}") @@ -474,57 +482,10 @@ async def list_conversations(ctx: NorthboundContext) -> Dict[str, Any]: conversations = get_conversation_list_service(ctx.user_id) # get_conversation_list_service is sync - # Add meta_data from token usage log if available - if ctx.token_id > 0: - for item in conversations: - # Ensure we do not leak empty meta_data keys - if "meta_data" in item and not item.get("meta_data"): - item.pop("meta_data", None) - - conversation_id = item.get("conversation_id") - if conversation_id: - try: - meta_data = get_latest_usage_metadata( - token_id=ctx.token_id, - related_id=int(conversation_id), - call_function_name="run_chat" - ) - # Only return meta_data when there is a usage log record and meta_data is non-empty - if meta_data: - item["meta_data"] = meta_data - else: - item.pop("meta_data", None) - except Exception as e: - logger.warning(f"Failed to get meta_data for conversation {conversation_id}: {str(e)}") - item.pop("meta_data", None) - # Now return internal conversation_id directly return {"message": "success", "data": conversations, "requestId": ctx.request_id} -def _format_search_record(record: Dict[str, Any]) -> Dict[str, Any]: - """Format a search source record for API response.""" - search_item = { - "title": record.get("source_title", ""), - "text": record.get("source_content", ""), - "source_type": record.get("source_type", ""), - "url": record.get("source_location", ""), - "filename": record.get("source_title", "") if record.get("source_type") == "file" else None, - "published_date": None, - "score": float(record["score_overall"]) if record.get("score_overall") is not None else None, - "tool_sign": record.get("tool_sign", ""), - "cite_index": record.get("cite_index") - } - - if record.get("published_date"): - if hasattr(record["published_date"], "strftime"): - search_item["published_date"] = record["published_date"].strftime("%Y-%m-%d") - else: - search_item["published_date"] = str(record["published_date"])[:10] - - return search_item - - async def get_conversation_history_internal(ctx: NorthboundContext, conversation_id: int) -> Dict[str, Any]: """Internal helper to get conversation history without logging.""" history = get_conversation_messages(conversation_id) @@ -537,23 +498,14 @@ async def get_conversation_history_internal(ctx: NorthboundContext, conversation try: minio_files = json.loads(raw_minio_files) if isinstance(raw_minio_files, str) else raw_minio_files except (json.JSONDecodeError, TypeError): - logger.warning(f"Failed to parse minio_files for message {message.get('message_id')}") - - # Fetch search results for this message - message_id = message.get("message_id") - search_results = [] - if message_id: - try: - search_records = get_source_searches_by_message(message_id, user_id=ctx.user_id) - search_results = [_format_search_record(r) for r in search_records] - except Exception as e: - logger.warning(f"Failed to get search records for message {message_id}: {str(e)}") - + logger.warning( + "Failed to parse minio_files for message %s", + message.get("message_id"), + ) result.append({ "role": message["message_role"], "content": message["message_content"], "minio_files": minio_files, - "search": search_results }) response = { @@ -570,21 +522,23 @@ async def get_conversation_history(ctx: NorthboundContext, conversation_id: int) raise Exception(f"Failed to get conversation history for conversation_id {conversation_id}: {str(e)}") +async def _get_visible_published_agents(ctx: NorthboundContext) -> list[dict]: + """Return published agents visible to the northbound caller.""" + agent_info_list = await list_published_agents_impl( + tenant_id=ctx.tenant_id, + user_id=ctx.user_id, + ) + if ctx.tenant_id != ASSET_OWNER_TENANT_ID: + agent_info_list.extend(await list_published_agents_impl( + tenant_id=ASSET_OWNER_TENANT_ID, + user_id=ctx.user_id, + )) + return agent_info_list + + async def get_agent_info_list(ctx: NorthboundContext) -> Dict[str, Any]: try: - agent_info_list = await list_published_agents_impl( - tenant_id=ctx.tenant_id, - user_id=ctx.user_id, - ) - # Match the same scope as /agent/published_list: non-asset-owner tenants - # also get the asset owner's published agents merged in. - if ctx.tenant_id != ASSET_OWNER_TENANT_ID: - asset_agent_list = await list_published_agents_impl( - tenant_id=ASSET_OWNER_TENANT_ID, - user_id=ctx.user_id, - ) - agent_info_list.extend(asset_agent_list) - # Remove internal information that partner don't need + agent_info_list = await _get_visible_published_agents(ctx) for agent_info in agent_info_list: agent_info.pop("agent_id", None) @@ -593,6 +547,37 @@ async def get_agent_info_list(ctx: NorthboundContext) -> Dict[str, Any]: raise Exception(f"Failed to get agent info list for tenant {ctx.tenant_id}: {str(e)}") +async def get_agent_info_by_name_for_northbound( + ctx: NorthboundContext, + agent_name: str, +) -> Dict[str, Any]: + """Return one visible published agent selected by its exact agent name.""" + if not agent_name.strip(): + raise ValueError("agent_name is required") + + try: + agent_info_list = await _get_visible_published_agents(ctx) + agent_info = next( + ( + item for item in agent_info_list + if item.get("name") == agent_name + ), + None, + ) + if agent_info is None: + raise LookupError(f"Published agent not found: {agent_name}") + + result = dict(agent_info) + result.pop("agent_id", None) + return {"message": "success", "data": result, "requestId": ctx.request_id} + except (ValueError, LookupError): + raise + except Exception as e: + raise Exception( + f"Failed to get agent info for agent_name {agent_name} in tenant {ctx.tenant_id}: {str(e)}" + ) + + async def update_conversation_title(ctx: NorthboundContext, conversation_id: int, title: str, meta_data: Optional[Dict[str, Any]] = None, idempotency_key: Optional[str] = None) -> Dict[str, Any]: composed_key: Optional[str] = None try: diff --git a/backend/services/notification_service.py b/backend/services/notification_service.py new file mode 100644 index 0000000000..28765228ff --- /dev/null +++ b/backend/services/notification_service.py @@ -0,0 +1,173 @@ +"""Notification business logic orchestration.""" +import logging +from datetime import datetime +from typing import Any, Dict, Optional + +from consts.agent_repository import STATUS_REJECTED, STATUS_SHARED +from consts.exceptions import NotFoundException +from consts.notification import ( + EVENT_TYPE_REPOSITORY_REVIEW_APPROVED, + EVENT_TYPE_REPOSITORY_REVIEW_PENDING, + EVENT_TYPE_REPOSITORY_REVIEW_REJECTED, + SCOPE_TENANT_ADMIN, + SCOPE_USER, +) +from database.notification_db import ( + create_notification, + deactivate_notifications as deactivate_notifications_db, + list_notifications_by_user, + mark_notifications_read as mark_notifications_read_db, +) + +logger = logging.getLogger(__name__) + +_REVIEW_STATUS_TO_EVENT_TYPE = { + STATUS_SHARED: EVENT_TYPE_REPOSITORY_REVIEW_APPROVED, + STATUS_REJECTED: EVENT_TYPE_REPOSITORY_REVIEW_REJECTED, +} + + +def _serialize_create_time(value: Any) -> Any: + """Convert datetime to ISO string for JSON serialization.""" + if not isinstance(value, datetime): + return value + iso = value.isoformat() + return iso if value.tzinfo else iso + "Z" + + +def list_notifications( + user_id: str, + *, + only_unread: bool = False, + page: int = 1, + page_size: int = 10, +) -> Dict[str, Any]: + """List notifications for a user, newest first.""" + result = list_notifications_by_user( + user_id, + only_unread=only_unread, + page=page, + page_size=page_size, + ) + for item in result["items"]: + item["create_time"] = _serialize_create_time(item.get("create_time")) + return result + + +def mark_notifications_read( + user_id: str, + *, + mark_all: bool = False, + receiver_id: Optional[int] = None, +) -> Dict[str, int]: + """Mark one or all unread notifications as read for the user.""" + if not mark_all and receiver_id is None: + raise ValueError("receiver_id is required when mark_all is false") + + updated_count = mark_notifications_read_db( + user_id, + mark_all=mark_all, + receiver_id=receiver_id, + ) + if not mark_all and updated_count == 0: + raise NotFoundException(f"Notification receiver {receiver_id} not found") + return {"updated_count": updated_count} + + +def deactivate_notifications( + *, + event_type: str, + resource_type: str, + unique_id: int, + updated_by: Optional[str] = None, +) -> Dict[str, int]: + """Deactivate active notifications matching event_type + resource_type + unique_id.""" + updated_count = deactivate_notifications_db( + event_type=event_type, + resource_type=resource_type, + unique_id=unique_id, + updated_by=updated_by, + ) + return {"updated_count": updated_count} + + +def create_repository_review_notification( + *, + resource_type: str, + review_status: str, + receiver_user_id: str, + details: Optional[Dict[str, Any]] = None, + tenant_id: Optional[str] = None, + unique_id: Optional[int] = None, + created_by: Optional[str] = None, +) -> Dict[str, Any]: + """Create a USER-scoped notification for a repository review result. + + Reusable by agent / skill / mcp repository flows. + + Args: + resource_type: One of VALID_RESOURCE_TYPES (e.g. agent_repository). + review_status: Listing status after review (`shared` or `rejected`). + receiver_user_id: Publisher user who should receive the notification. + details: i18n interpolation details (e.g. listing name, reviewer reason). + tenant_id: Optional tenant stored on the receiver row. + unique_id: Related resource primary key (e.g. agent_repository_id). + created_by: Actor who triggered the review action. + + Returns: + Dict with notification_id and receiver_count from the DB layer. + """ + event_type = _REVIEW_STATUS_TO_EVENT_TYPE.get(review_status) + if event_type is None: + logger.warning( + "Skipping review notification: invalid review_status '%s'; " + "expected '%s' or '%s'", + review_status, + STATUS_SHARED, + STATUS_REJECTED, + ) + return {"notification_id": None, "receiver_count": 0} + + return create_notification( + event_type=event_type, + resource_type=resource_type, + scope=SCOPE_USER, + details=details, + tenant_id=tenant_id, + receiver_user_id=receiver_user_id, + unique_id=unique_id, + created_by=created_by, + ) + + +def create_repository_pending_review_notification( + *, + resource_type: str, + tenant_id: str, + unique_id: int, + details: Optional[Dict[str, Any]] = None, + created_by: Optional[str] = None, +) -> Dict[str, Any]: + """Create a TENANT_ADMIN-scoped notification for a pending repository review. + + Notifies all ADMIN users in the publisher tenant that a listing awaits review. + + Args: + resource_type: One of VALID_RESOURCE_TYPES (e.g. agent_repository). + tenant_id: Publisher tenant whose admins should receive the notification. + unique_id: Related resource primary key (e.g. agent_repository_id). + details: i18n interpolation details (e.g. listing name). + created_by: Actor who submitted the listing for review. + + Returns: + Dict with notification_id and receiver_count from the DB layer. + """ + return create_notification( + event_type=EVENT_TYPE_REPOSITORY_REVIEW_PENDING, + resource_type=resource_type, + scope=SCOPE_TENANT_ADMIN, + details=details, + tenant_id=tenant_id, + unique_id=unique_id, + created_by=created_by, + ) diff --git a/backend/services/oauth_service.py b/backend/services/oauth_service.py index 31a70273ef..d632b7edc1 100644 --- a/backend/services/oauth_service.py +++ b/backend/services/oauth_service.py @@ -21,6 +21,7 @@ OAUTH_SSL_VERIFY, OAUTH_CA_BUNDLE, SUPABASE_JWT_SECRET, + JWT_EXPIRY_SECONDS, ) from consts.exceptions import OAuthLinkError, OAuthProviderError from services.asset_owner_visibility import require_asset_owner_enabled @@ -414,7 +415,6 @@ async def complete_pending_oauth_account( use_invitation_code, ) from services.tool_configuration_service import init_tool_list_for_tenant - from services.user_management_service import generate_tts_stt_4_admin from utils.auth_utils import calculate_expires_at, generate_session_jwt pending = parse_pending_oauth_token(pending_token) @@ -491,8 +491,6 @@ async def complete_pending_oauth_account( if group_ids and not is_asset_owner_registration: add_user_to_groups(supabase_user_id, group_ids, supabase_user_id) - if user_role == "ADMIN": - await generate_tts_stt_4_admin(tenant_id, supabase_user_id) if not is_asset_owner_registration: await init_tool_list_for_tenant(tenant_id, supabase_user_id) @@ -505,8 +503,10 @@ async def complete_pending_oauth_account( tenant_id=tenant_id, ) - expiry_seconds = 3600 - jwt_token = generate_session_jwt(supabase_user_id, expires_in=expiry_seconds) + jwt_token = generate_session_jwt( + supabase_user_id, expires_in=JWT_EXPIRY_SECONDS + ) + expiry_seconds = JWT_EXPIRY_SECONDS expires_at = calculate_expires_at(jwt_token) return { diff --git a/backend/services/prompt_service.py b/backend/services/prompt_service.py index f1564cdbce..fabf6846df 100644 --- a/backend/services/prompt_service.py +++ b/backend/services/prompt_service.py @@ -7,7 +7,10 @@ from jinja2 import StrictUndefined, Template +from nexent.core.tools.parallel_executor import ParallelExecutorTool + from consts.const import LANGUAGE, ENABLE_JIUWEN_SDK +from consts.tool_labels import PARALLEL_EXECUTOR_TOOL_NAME from consts.error_code import ErrorCode from consts.error_message import ErrorMessage from consts.exceptions import AppException @@ -32,6 +35,7 @@ from utils.prompt_template_utils import ( get_prompt_optimize_prompt_template, get_prompt_template, + get_guardrail_regex_prompt_template, ) from dataclasses import dataclass, field @@ -106,6 +110,7 @@ def generate_and_save_system_prompt_impl(agent_id: int, tool_ids: Optional[List[int]] = None, sub_agent_ids: Optional[List[int]] = None, knowledge_base_display_names: Optional[List[str]] = None, + aidp_kb_display_names: Optional[List[str]] = None, has_selected_resources: bool = True): # Get description of tool and agent from frontend-provided IDs # Frontend always provides tool_ids and sub_agent_ids (could be empty arrays) @@ -134,6 +139,20 @@ def generate_and_save_system_prompt_impl(agent_id: int, logger.debug( f"Using database query for knowledge base display names: {knowledge_base_display_names}") + # Get aidp knowledge base display names for few-shot examples + # Priority: frontend-provided > database query + if aidp_kb_display_names: + logger.debug( + f"Using frontend-provided aidp knowledge base display names: {aidp_kb_display_names}") + else: + aidp_kb_display_names = _resolve_aidp_kb_display_names( + tool_info_list=tool_info_list, + user_id=user_id, + tenant_id=tenant_id, + ) + logger.debug( + f"Using database query for aidp knowledge base display names: {aidp_kb_display_names}") + # Handle sub-agent IDs if sub_agent_ids and len(sub_agent_ids) > 0: sub_agent_info_list = [] @@ -192,7 +211,8 @@ def generate_and_save_system_prompt_impl(agent_id: int, language, prompt_template_id, knowledge_base_display_names, - has_selected_resources + aidp_kb_display_names, + has_selected_resources ): result_type = result_data["type"] final_results[result_type] = result_data["content"] @@ -386,6 +406,7 @@ def optimize_prompt_section_impl( tool_ids: Optional[List[int]] = None, sub_agent_ids: Optional[List[int]] = None, knowledge_base_display_names: Optional[List[str]] = None, + aidp_kb_display_names: Optional[List[str]] = None, ) -> dict: normalized_section_type = (section_type or "").strip() if normalized_section_type not in {"duty", "constraint", "few_shots"}: @@ -436,6 +457,7 @@ def optimize_prompt_section_impl( sub_agent_info_list=sub_agent_info_list, language=language, knowledge_base_display_names=knowledge_base_display_names, + aidp_kb_display_names=aidp_kb_display_names, ) optimized_content = call_llm_for_system_prompt( @@ -456,7 +478,118 @@ def optimize_prompt_section_impl( } -def generate_system_prompt(sub_agent_info_list, task_description, tool_info_list, tenant_id: str, user_id: str, model_id: int, language: str = LANGUAGE["ZH"], prompt_template_id: Optional[int] = None, knowledge_base_display_names: Optional[List[str]] = None, has_selected_resources: bool = True): +def _extract_json_object(raw: str) -> Optional[dict]: + """Extract the first JSON object from LLM output that may contain markdown fences / surrounding text. + + Fallbacks: markdown fence, surrounding explanation text, single quotes, trailing commas. + """ + text = (raw or "").strip() + if not text: + return None + start = text.find("{") + end = text.rfind("}") + if start == -1 or end == -1 or end <= start: + return None + snippet = text[start:end + 1] + try: + return json.loads(snippet) + except (ValueError, TypeError): + pass + # Fallback: single quotes -> double quotes, strip trailing commas + import re as _re + try: + fixed = snippet.replace("'", '"') + fixed = _re.sub(r",\s*([}\]])", r"\1", fixed) + return json.loads(fixed) + except (ValueError, TypeError): + pass + # Fallback: LLM often inlines regex escapes (\d \w \s \. etc.) directly into JSON strings. + # A single backslash is invalid in JSON (only \" \\ \/ \b \f \n \r \t \uXXXX are allowed). + # Double up backslashes that are not part of a valid JSON escape sequence. + try: + fixed = _re.sub(r'\\(?!["\\/bfnrtu])', r'\\\\', snippet) + return json.loads(fixed) + except (ValueError, TypeError): + return None + + +def generate_guardrail_rules_impl( + description: str, + model_id: int, + tenant_id: str, + language: str = LANGUAGE["ZH"], +) -> dict: + """Generate guardrail regex rules from a natural-language description via LLM. + + Loads the guardrail prompt template, renders the user prompt with the + description, calls the LLM, and extracts the JSON object from the (possibly + malformed) response. Tolerates markdown fences, surrounding prose, single + quotes, trailing commas, and invalid JSON escapes such as ``\\d`` (LLMs + often paste regex escapes with a single backslash). + + Args: + description: Natural-language description of what to match or block. + model_id: ID of the LLM model used for generation. + tenant_id: Tenant ID for model resolution and monitoring. + language: Language code ('zh' or 'en') selecting the prompt template. + + Returns: + A dict keyed by ``type``: + - ``{"type": "single", "candidates": [{"pattern": str, "desc": str}]}`` + - ``{"type": "multi", "rules": [{"name": str, "pattern": str, + "severity": str, "desc": str}]}`` + + Raises: + AppException: If ``description`` is empty, the LLM returns nothing, + or the response is not valid JSON / carries an unknown ``type``. + """ + + if not (description or "").strip(): + raise AppException( + ErrorCode.COMMON_MISSING_REQUIRED_FIELD, + "Description is required.", + ) + + prompt_template = get_guardrail_regex_prompt_template(language) + user_prompt = Template( + prompt_template["GUARDRAIL_USER_PROMPT"], undefined=StrictUndefined + ).render({"description": description}) + + raw = call_llm_for_system_prompt( + model_id=model_id, + user_prompt=user_prompt, + system_prompt=prompt_template["GUARDRAIL_SYSTEM_PROMPT"], + tenant_id=tenant_id, + ).strip() + + # Diagnostic log: record raw LLM output (for troubleshooting "bad format" issues) + logger.info("[guardrail] desc=%r model_id=%s raw(500)=%s", description[:80], model_id, (raw or "")[:500]) + + if not raw: + raise AppException(ErrorCode.MODEL_PROMPT_GENERATION_FAILED) + + parsed = _extract_json_object(raw) + if not isinstance(parsed, dict): + logger.warning("[guardrail] JSON parse failed, raw=%s", (raw or "")[:800]) + raise AppException( + ErrorCode.MODEL_PROMPT_GENERATION_FAILED, + "LLM did not return valid JSON for guardrail rules.", + ) + + result_type = str(parsed.get("type") or "").strip().lower() + if result_type == "single": + candidates = parsed.get("candidates") + return {"type": "single", "candidates": candidates if isinstance(candidates, list) else []} + if result_type == "multi": + rules = parsed.get("rules") + return {"type": "multi", "rules": rules if isinstance(rules, list) else []} + raise AppException( + ErrorCode.MODEL_PROMPT_GENERATION_FAILED, + "Unknown guardrail result type.", + ) + + +def generate_system_prompt(sub_agent_info_list, task_description, tool_info_list, tenant_id: str, user_id: str, model_id: int, language: str = LANGUAGE["ZH"], prompt_template_id: Optional[int] = None, knowledge_base_display_names: Optional[List[str]] = None, aidp_kb_display_names: Optional[List[str]] = None, has_selected_resources: bool = True): """Main function for generating system prompts""" prompt_for_generate = resolve_prompt_generate_template( tenant_id=tenant_id, @@ -473,6 +606,7 @@ def generate_system_prompt(sub_agent_info_list, task_description, tool_info_list tool_info_list=tool_info_list, language=language, knowledge_base_display_names=knowledge_base_display_names, + aidp_kb_display_names=aidp_kb_display_names, has_selected_resources=has_selected_resources, ) @@ -538,6 +672,19 @@ def _resolve_knowledge_base_display_names( return resolved_names +def _resolve_aidp_kb_display_names( + tool_info_list: List[dict], + user_id: str, + tenant_id: str, +) -> Optional[List[str]]: + """Resolve aidp knowledge base display names from tool list.""" + return get_aidp_kb_display_names( + tool_info_list=tool_info_list, + user_id=user_id, + tenant_id=tenant_id, + ) + + def _resolve_prompt_generation_sub_agents( agent_id: int, tenant_id: str, @@ -706,7 +853,7 @@ def _stream_results(produce_queue, latest, stop_flags, threads, error_holder): last_results[tag] = latest[tag] -def join_info_for_generate_system_prompt(prompt_for_generate, sub_agent_info_list, task_description, tool_info_list, language: str = LANGUAGE["ZH"], knowledge_base_display_names: Optional[List[str]] = None, has_selected_resources: bool = True): +def join_info_for_generate_system_prompt(prompt_for_generate, sub_agent_info_list, task_description, tool_info_list, language: str = LANGUAGE["ZH"], knowledge_base_display_names: Optional[List[str]] = None, aidp_kb_display_names: Optional[List[str]] = None, has_selected_resources: bool = True): input_label = "Inputs" if language == 'en' else "接受输入" output_label = "Output type" if language == 'en' else "返回输出类型" @@ -724,6 +871,9 @@ def join_info_for_generate_system_prompt(prompt_for_generate, sub_agent_info_lis # Always include knowledge_base_names to avoid StrictUndefined errors in template. # An empty string is falsy, so the {% if knowledge_base_names %} block will be skipped. "knowledge_base_names": "", + # Always include aidp_kb_names to avoid StrictUndefined errors in template. + # An empty string is falsy, so the {% if aidp_kb_names %} block will be skipped. + "aidp_kb_names": "", # Flag indicating whether tools or sub-agents are selected; # templates use this to suppress boilerplate in constraint/few_shots sections "has_selected_resources": has_selected_resources, @@ -739,6 +889,16 @@ def join_info_for_generate_system_prompt(prompt_for_generate, sub_agent_info_lis kb_names_str = "" template_context["knowledge_base_names"] = kb_names_str + # Always add aidp_kb_names to context (empty string when not available). + # This is necessary because Jinja2 StrictUndefined raises an error for any + # undefined variable, even inside an {% if %} block. + if aidp_kb_display_names: + aidp_names_str = ", ".join( + f'"{name}"' for name in aidp_kb_display_names) + else: + aidp_names_str = "" + template_context["aidp_kb_names"] = aidp_names_str + # Generate content using template content = Template( prompt_for_generate["user_prompt"], undefined=StrictUndefined).render(template_context) @@ -756,6 +916,7 @@ def join_info_for_optimize_prompt_section( sub_agent_info_list, language: str = LANGUAGE["ZH"], knowledge_base_display_names: Optional[List[str]] = None, + aidp_kb_display_names: Optional[List[str]] = None, ): input_label = "Inputs" if language == LANGUAGE["EN"] else "接受输入" output_label = "Output type" if language == LANGUAGE["EN"] else "返回输出类型" @@ -774,6 +935,12 @@ def join_info_for_optimize_prompt_section( else: kb_names_str = "" + if aidp_kb_display_names: + aidp_names_str = ", ".join( + f'"{name}"' for name in aidp_kb_display_names) + else: + aidp_names_str = "" + template_context = { "section_type": section_type, "section_title": section_title, @@ -783,6 +950,7 @@ def join_info_for_optimize_prompt_section( "tool_description": tool_description, "assistant_description": assistant_description, "knowledge_base_names": kb_names_str, + "aidp_kb_names": aidp_names_str, } return Template( @@ -804,7 +972,25 @@ def get_enabled_tool_description_for_generate_prompt(agent_id: int, tenant_id: s logger.info("Fetching tool instances") tool_id_list = get_enable_tool_id_by_agent_id( agent_id=agent_id, tenant_id=tenant_id) + # If no tools are enabled, return early — nothing to parallelize. + if not tool_id_list: + return [] tool_info_list = query_tools_by_ids(tool_id_list) + + # parallel_executor is always built from the SDK class — no DB query. + seen_names = {t.get("name") for t in tool_info_list if t.get("name")} + if PARALLEL_EXECUTOR_TOOL_NAME not in seen_names: + tool_info_list.append({ + "name": ParallelExecutorTool.name, + "description": ParallelExecutorTool.description, + "description_zh": ParallelExecutorTool.description_zh, + "inputs": json.dumps(ParallelExecutorTool.inputs, ensure_ascii=False), + "output_type": ParallelExecutorTool.output_type, + "params": [], + "source": "local", + "class_name": ParallelExecutorTool.__name__, + }) + return tool_info_list @@ -862,7 +1048,9 @@ def get_knowledge_base_display_names(tool_info_list: List[dict], agent_id: int, # Convert to display names knowledge_name_map = get_knowledge_name_map_by_index_names( - unique_index_names) + unique_index_names, + tenant_id=tenant_id, + ) # Return list of display names (knowledge_name) for each configured index_name display_names = [] @@ -876,6 +1064,41 @@ def get_knowledge_base_display_names(tool_info_list: List[dict], agent_id: int, return display_names if display_names else None +def get_aidp_kb_display_names(tool_info_list: List[dict], user_id: str, tenant_id: str) -> Optional[List[str]]: + """ + Extract aidp knowledge base display names from tool configurations. + This is used to ensure few-shot examples use actual configured aidp knowledge base names. + + Args: + tool_info_list: List of tool info dictionaries + user_id: User ID for permission queries + tenant_id: Tenant ID for database queries + + Returns: + List of aidp knowledge base display names if aidp_search tool is configured, None otherwise + """ + # Check if aidp_search tool is in the list + aidp_tool_ids = [tool['tool_id'] for tool in tool_info_list if tool.get('name') == 'aidp_search'] + if not aidp_tool_ids: + logger.debug("No aidp_search tool found in tool list") + return None + + try: + from ext_components.aidp.services import aidp_permission_service + # Get the kds_name_to_id_map from permission service + kds_name_to_id_map = aidp_permission_service.get_kds_name_to_id_map( + user_id=user_id, + tenant_id=tenant_id + ) + # Extract the kds_name keys as display names + display_names = list(kds_name_to_id_map.keys()) + logger.debug(f"Retrieved aidp_kb_display_names: {display_names}") + return display_names if display_names else None + except Exception as e: + logger.warning(f"Failed to get aidp knowledge base display names: {e}") + return None + + def get_enabled_sub_agent_description_for_generate_prompt(agent_id: int, tenant_id: str): logger.info("Fetching sub-agents information") diff --git a/backend/services/providers/dashscope_provider.py b/backend/services/providers/dashscope_provider.py index b5f4ba0d3d..d87e40a2cf 100644 --- a/backend/services/providers/dashscope_provider.py +++ b/backend/services/providers/dashscope_provider.py @@ -229,4 +229,10 @@ async def get_models(self, provider_config: Dict) -> List[Dict]: else: return [] except (httpx.HTTPStatusError, httpx.ConnectTimeout, httpx.ConnectError, Exception) as e: - return _classify_provider_error("DashScope", exception=e) + status_code = e.response.status_code if isinstance(e, httpx.HTTPStatusError) and getattr(e, "response", None) else None + return _classify_provider_error( + "DashScope", + status_code=status_code, + error_message=str(e), + exception=e, + ) diff --git a/backend/services/providers/silicon_provider.py b/backend/services/providers/silicon_provider.py index e078f83a72..24eab2eed8 100644 --- a/backend/services/providers/silicon_provider.py +++ b/backend/services/providers/silicon_provider.py @@ -134,4 +134,10 @@ async def get_models(self, provider_config: Dict) -> List[Dict]: return model_list except (httpx.HTTPStatusError, httpx.ConnectTimeout, httpx.ConnectError, Exception) as e: - return _classify_provider_error("SiliconFlow", exception=e) + status_code = e.response.status_code if isinstance(e, httpx.HTTPStatusError) and getattr(e, "response", None) else None + return _classify_provider_error( + "SiliconFlow", + status_code=status_code, + error_message=str(e), + exception=e, + ) diff --git a/backend/services/providers/tokenpony_provider.py b/backend/services/providers/tokenpony_provider.py index 16adf00082..9b270ab67b 100644 --- a/backend/services/providers/tokenpony_provider.py +++ b/backend/services/providers/tokenpony_provider.py @@ -186,4 +186,10 @@ async def get_models(self, provider_config: Dict) -> List[Dict]: return [] except (httpx.HTTPStatusError, httpx.ConnectTimeout, httpx.ConnectError, Exception) as e: - return _classify_provider_error("TokenPony", exception=e) + status_code = e.response.status_code if isinstance(e, httpx.HTTPStatusError) and getattr(e, "response", None) else None + return _classify_provider_error( + "TokenPony", + status_code=status_code, + error_message=str(e), + exception=e, + ) diff --git a/backend/services/quota_service.py b/backend/services/quota_service.py new file mode 100644 index 0000000000..a27cae4847 --- /dev/null +++ b/backend/services/quota_service.py @@ -0,0 +1,897 @@ +""" +Quota service for KB storage capacity management. + +Provides three-tier quota management: +- Platform tier: SU declares capacity and allocates per-tenant hard quotas +- Tenant tier: Hard limit enforcement at upload time +- KB tier: Per-KB soft quotas (advisory, warnings only) +""" + +import logging +import threading +import time +from typing import Any, Dict, List, Optional, Tuple + +from consts.const import ASSET_OWNER_TENANT_ID, DEFAULT_TENANT_ID +from consts.exceptions import PlatformQuotaConflictError, QuotaExceededError +from database.knowledge_db import ( + get_knowledge_info_by_tenant_id, + update_knowledge_record, +) +from database.tenant_config_db import ( + delete_config_by_tenant_config_id, + get_single_config_info, + insert_config, + update_config_by_tenant_config_id, +) + +logger = logging.getLogger(__name__) + +# Tenant config keys +KEY_TENANT_HARD_LIMIT_BYTES = "KB_QUOTA_TENANT_HARD_LIMIT_BYTES" +KEY_WARNING_ENABLED = "KB_QUOTA_WARNING_ENABLED" +KEY_WARNING_THRESHOLD_PCT = "KB_QUOTA_WARNING_THRESHOLD_PCT" +KEY_CRITICAL_THRESHOLD_PCT = "KB_QUOTA_CRITICAL_THRESHOLD_PCT" +KEY_HARD_LIMIT_EDITABLE = "KB_QUOTA_HARD_LIMIT_EDITABLE" +KEY_PLATFORM_CAPACITY_BYTES = "PLATFORM_KB_STORAGE_CAPACITY_BYTES" + + +def _is_displayable_tenant_id( + tenant_id: Optional[str], + asset_owner_tenant_id: str = ASSET_OWNER_TENANT_ID, +) -> bool: + """Return whether a tenant id should appear in platform quota views.""" + normalized_tenant_id = (tenant_id or "").strip() + return normalized_tenant_id not in {"", DEFAULT_TENANT_ID, asset_owner_tenant_id} + + +# Constants +GB = 1024 * 1024 * 1024 +CACHE_TTL_SECONDS = 60 +DEFAULT_WARNING_THRESHOLD = 80 +DEFAULT_CRITICAL_THRESHOLD = 95 + +# In-memory cache for usage data +_usage_cache: Dict[str, Tuple[float, Dict[str, Any]]] = {} + +# Config helpers use independent database sessions, so serialize allocation +# validation and writes within a config-service process. +_platform_allocation_lock = threading.RLock() + + +def _bytes_to_readable(size_bytes: Optional[int]) -> Optional[str]: + """Convert bytes to human-readable string (e.g. '10 GB').""" + if size_bytes is None: + return None + if size_bytes >= GB: + return f"{size_bytes / GB:.1f} GB" + if size_bytes >= 1024 * 1024: + return f"{size_bytes / (1024 * 1024):.1f} MB" + if size_bytes >= 1024: + return f"{size_bytes / 1024:.1f} KB" + return f"{size_bytes} B" + + +MB = 1024 * 1024 + + +def _gb_to_bytes(gb: int) -> int: + """Convert integer GB to bytes.""" + return gb * GB + + +def _mb_to_bytes(mb: int) -> int: + """Convert integer MB to bytes.""" + return mb * MB + + +class QuotaService: + """Service for managing storage quotas at tenant and KB level.""" + + def __init__(self, tenant_id: str, user_id: Optional[str] = None): + self.tenant_id = tenant_id + self.user_id = user_id or "system" + + # ── Tenant Config Helpers ────────────────────────────────────────── + + def _get_tenant_config(self, key: str) -> Optional[str]: + """Read a single tenant config value.""" + record = get_single_config_info(self.tenant_id, key) + return record.get("config_value") if record else None + + def _set_tenant_config(self, key: str, value: Any, value_type: str = "single") -> bool: + """Upsert a tenant config key. Updates existing row or inserts new.""" + existing = get_single_config_info(self.tenant_id, key) + if existing and existing.get("tenant_config_id"): + return update_config_by_tenant_config_id( + existing["tenant_config_id"], str(value) + ) + else: + return insert_config({ + "tenant_id": self.tenant_id, + "user_id": self.user_id, + "config_key": key, + "config_value": str(value), + "value_type": value_type, + }) + + def _delete_tenant_config(self, key: str) -> bool: + """Soft-delete a tenant config key.""" + existing = get_single_config_info(self.tenant_id, key) + tenant_config_id = existing.get("tenant_config_id") if existing else None + if tenant_config_id is None: + return True + return delete_config_by_tenant_config_id(tenant_config_id) + + # ── Tenant-Level Hard Limit (task 2.2) ───────────────────────────── + + def get_hard_limit(self) -> Dict[str, Any]: + """ + Get the tenant hard storage limit. + Returns dict with _bytes and _readable fields, or defaults for unlimited. + """ + raw = self._get_tenant_config(KEY_TENANT_HARD_LIMIT_BYTES) + editable_raw = self._get_tenant_config(KEY_HARD_LIMIT_EDITABLE) + editable = editable_raw != "false" if editable_raw else True + + if raw is not None: + try: + limit_bytes = int(raw) + return { + "hard_limit_bytes": limit_bytes, + "hard_limit_readable": _bytes_to_readable(limit_bytes), + "hard_limit_editable": editable, + } + except (ValueError, TypeError): + pass + + return { + "hard_limit_bytes": None, + "hard_limit_readable": None, + "hard_limit_editable": editable, + } + + def set_hard_limit( + self, + limit_gb: Optional[int] = None, + limit_mb: Optional[int] = None, + ) -> Dict[str, Any]: + """ + Set the tenant hard storage limit. None = unlimited. + Accepts either limit_gb (GB) or limit_mb (MB) for testing with small quotas. + Also sets hard_limit_editable = true (admins can manage their own limit). + """ + if limit_gb is None and limit_mb is None: + self._delete_tenant_config(KEY_TENANT_HARD_LIMIT_BYTES) + self._set_tenant_config(KEY_HARD_LIMIT_EDITABLE, "true") + return {"hard_limit_bytes": None, "hard_limit_readable": None} + + limit_bytes = self._quota_input_to_bytes(limit_gb, limit_mb) + with _platform_allocation_lock: + self._validate_tenant_hard_limit(limit_bytes) + self._set_tenant_config(KEY_TENANT_HARD_LIMIT_BYTES, str(limit_bytes)) + self._set_tenant_config(KEY_HARD_LIMIT_EDITABLE, "true") + return { + "hard_limit_bytes": limit_bytes, + "hard_limit_readable": _bytes_to_readable(limit_bytes), + } + + def delete_hard_limit(self) -> bool: + """Remove the tenant hard storage limit.""" + self._delete_tenant_config(KEY_TENANT_HARD_LIMIT_BYTES) + self._delete_tenant_config(KEY_HARD_LIMIT_EDITABLE) + return True + + # ── Warning Configuration (task 2.2) ─────────────────────────────── + + def get_warning_config(self) -> Dict[str, Any]: + """Get warning configuration: enabled, warning_pct, critical_pct.""" + enabled_raw = self._get_tenant_config(KEY_WARNING_ENABLED) + warning_raw = self._get_tenant_config(KEY_WARNING_THRESHOLD_PCT) + critical_raw = self._get_tenant_config(KEY_CRITICAL_THRESHOLD_PCT) + + enabled = enabled_raw.lower() == "true" if enabled_raw else True # default on + try: + warning_pct = int(warning_raw) if warning_raw else DEFAULT_WARNING_THRESHOLD + except (ValueError, TypeError): + warning_pct = DEFAULT_WARNING_THRESHOLD + try: + critical_pct = int(critical_raw) if critical_raw else DEFAULT_CRITICAL_THRESHOLD + except (ValueError, TypeError): + critical_pct = DEFAULT_CRITICAL_THRESHOLD + + return { + "warning_enabled": enabled, + "warning_threshold_pct": warning_pct, + "critical_threshold_pct": critical_pct, + } + + def set_warning_config( + self, + enabled: Optional[bool] = None, + warning_pct: Optional[int] = None, + critical_pct: Optional[int] = None, + ) -> Dict[str, Any]: + """Set warning thresholds. Validates 1-100 range.""" + if warning_pct is not None: + if not 1 <= warning_pct <= 100: + raise ValueError(f"warning_pct must be 1-100, got {warning_pct}") + self._set_tenant_config(KEY_WARNING_THRESHOLD_PCT, str(warning_pct)) + + if critical_pct is not None: + if not 1 <= critical_pct <= 100: + raise ValueError(f"critical_pct must be 1-100, got {critical_pct}") + self._set_tenant_config(KEY_CRITICAL_THRESHOLD_PCT, str(critical_pct)) + + if enabled is not None: + self._set_tenant_config(KEY_WARNING_ENABLED, str(enabled).lower()) + + return self.get_warning_config() + + # ── Per-KB Soft Quota (task 2.3) ─────────────────────────────────── + + def get_kb_soft_quota(self, knowledge_id: int) -> Optional[int]: + """Get per-KB soft quota in bytes. Returns None if not set.""" + from database.client import get_db_session + from database.db_models import KnowledgeRecord + + with get_db_session() as session: + record = session.query(KnowledgeRecord).filter( + KnowledgeRecord.knowledge_id == knowledge_id, + KnowledgeRecord.delete_flag != "Y", + ).first() + if record: + return record.quota_limit_bytes + return None + + def set_kb_soft_quota(self, index_name: str, limit_bytes: Optional[int]) -> bool: + """ + Set per-KB soft quota via index_name. None = unlimited. + Updates the knowledge_record_t row. + """ + return update_knowledge_record({ + "index_name": index_name, + "quota_limit_bytes": limit_bytes, + "user_id": self.user_id, + }) + + def get_all_kb_quotas(self) -> List[Dict[str, Any]]: + """Get all KB quota records for the tenant.""" + kb_list = get_knowledge_info_by_tenant_id(self.tenant_id) + result = [] + for kb in kb_list: + result.append({ + "knowledge_id": kb.get("knowledge_id"), + "index_name": kb.get("index_name"), + "knowledge_name": kb.get("knowledge_name"), + "quota_limit_bytes": kb.get("quota_limit_bytes"), + }) + return result + + # ── Quota Summary (task 2.4) ─────────────────────────────────────── + + def get_quota_summary(self) -> Dict[str, Any]: + """Return quota allocation summary with oversubscription ratio.""" + hard_limit = self.get_hard_limit() + kb_quotas = self.get_all_kb_quotas() + + soft_allocated = sum( + q["quota_limit_bytes"] for q in kb_quotas if q["quota_limit_bytes"] is not None + ) + kbs_with_quota = sum(1 for q in kb_quotas if q["quota_limit_bytes"] is not None) + kb_count = len(kb_quotas) + + oversubscription_ratio = None + if hard_limit.get("hard_limit_bytes") and hard_limit["hard_limit_bytes"] > 0: + oversubscription_ratio = round( + soft_allocated / hard_limit["hard_limit_bytes"], 4 + ) + + return { + "soft_allocated_total_bytes": soft_allocated, + "soft_allocated_readable": _bytes_to_readable(soft_allocated), + "hard_limit_bytes": hard_limit.get("hard_limit_bytes"), + "hard_limit_readable": hard_limit.get("hard_limit_readable"), + "total_bytes": None, # filled in when usage is available + "total_readable": None, + "oversubscription_ratio": oversubscription_ratio, + "kb_count": kb_count, + "kbs_with_quota": kbs_with_quota, + } + + # ── Warning Level Computation (task 3.3) ─────────────────────────── + + @staticmethod + def _compute_kb_warning_level( + usage_pct: Optional[float], + warning_threshold: int = DEFAULT_WARNING_THRESHOLD, + critical_threshold: int = DEFAULT_CRITICAL_THRESHOLD, + ) -> str: + """Compute KB-level warning: normal, warning, critical, exceeded. + Uses tenant-configured thresholds for consistency.""" + if usage_pct is None: + return "normal" + if usage_pct >= 100: + return "exceeded" + if usage_pct >= critical_threshold: + return "critical" + if usage_pct >= warning_threshold: + return "warning" + return "normal" + + @staticmethod + def _compute_tenant_warning_level( + usage_pct: Optional[float], + critical_threshold: int = DEFAULT_CRITICAL_THRESHOLD, + warning_threshold: int = DEFAULT_WARNING_THRESHOLD, + ) -> str: + """Compute tenant-level warning: normal, warning, critical, blocked.""" + if usage_pct is None: + return "normal" + if usage_pct >= 100: + return "blocked" + if usage_pct >= critical_threshold: + return "critical" + if usage_pct >= warning_threshold: + return "warning" + return "normal" + + # ── Usage Tracking (tasks 3.1–3.4) ───────────────────────────────── + + def get_usage( + self, + force_refresh: bool = False, + detail: bool = False, + ) -> Dict[str, Any]: + """ + Aggregate storage usage across all tenant KBs from MinIO/ES. + Results are cached with 60s TTL. force_refresh bypasses cache. + """ + cache_key = self.tenant_id + + # Check cache + now = time.time() + if not force_refresh and cache_key in _usage_cache: + cached_time, cached_data = _usage_cache[cache_key] + if now - cached_time < CACHE_TTL_SECONDS: + if not detail: + # Return without breakdown for non-detail requests + result = dict(cached_data) + result.pop("breakdown", None) + return result + return dict(cached_data) + + # Compute usage by querying file sizes from MinIO/ES + usage_data = self._compute_usage() + _usage_cache[cache_key] = (now, dict(usage_data)) + + if not detail: + result = dict(usage_data) + result.pop("breakdown", None) + return result + return dict(usage_data) + + def _compute_usage(self) -> Dict[str, Any]: + """ + Compute actual storage usage by summing file sizes across all tenant KBs. + Uses the existing ES index stats (store_size) from the vectordatabase service. + """ + from services.vectordatabase_service import get_vector_db_core + + kb_list = get_knowledge_info_by_tenant_id(self.tenant_id) + warning_config = self.get_warning_config() + tenant_warning_threshold = warning_config["warning_threshold_pct"] + tenant_critical_threshold = warning_config["critical_threshold_pct"] + hard_limit_info = self.get_hard_limit() + + # Quota enforcement must always use every KB in the tenant, regardless + # of the requesting user's KB visibility. + try: + vdb_core = get_vector_db_core() + index_names = [ + kb.get("index_name") + for kb in kb_list + if kb.get("index_name") + and kb.get("knowledge_sources") != "datamate" + ] + indices_detail = ( + vdb_core.get_indices_detail(index_names) if index_names else {} + ) + except Exception: + logger.warning("Failed to query ES indices for usage data", exc_info=True) + indices_detail = {} + + # Build lookup: index_name -> {store_size_bytes, file_count} + stats_lookup = {} + for name, stats in indices_detail.items(): + stats = stats if isinstance(stats, dict) else {} + base_info = stats.get("base_info", {}) if isinstance(stats, dict) else {} + store_size_raw = base_info.get("store_size", "0") + # Parse store_size string like "1.5 GB" or "500 MB" into bytes + store_bytes = self._parse_store_size(store_size_raw) + doc_count = base_info.get("doc_count", 0) or 0 + stats_lookup[name] = {"bytes": store_bytes, "file_count": doc_count} + + breakdown = [] + total_bytes = 0 + total_files = 0 + + for kb in kb_list: + index_name = kb.get("index_name", "") + kb_id = kb.get("knowledge_id") + kb_name = kb.get("knowledge_name", index_name) + soft_quota_bytes = kb.get("quota_limit_bytes") + + kb_stats = stats_lookup.get(index_name, {}) + kb_actual_bytes = kb_stats.get("bytes", 0) + kb_file_count = kb_stats.get("file_count", 0) + + total_bytes += kb_actual_bytes + total_files += kb_file_count + + # Compute KB-level warning + kb_usage_pct = None + if soft_quota_bytes and soft_quota_bytes > 0: + kb_usage_pct = round(kb_actual_bytes / soft_quota_bytes * 100, 2) + kb_warning_level = self._compute_kb_warning_level( + kb_usage_pct, + warning_threshold=tenant_warning_threshold, + critical_threshold=tenant_critical_threshold, + ) + + breakdown.append({ + "knowledge_id": kb_id, + "knowledge_name": kb_name, + "index_name": index_name, + "soft_quota_bytes": soft_quota_bytes, + "soft_quota_readable": _bytes_to_readable(soft_quota_bytes), + "actual_bytes": kb_actual_bytes, + "actual_readable": _bytes_to_readable(kb_actual_bytes), + "usage_pct": kb_usage_pct, + "file_count": kb_file_count, + "kb_warning_level": kb_warning_level, + }) + + # Compute tenant-level warning + hard_limit_bytes = hard_limit_info.get("hard_limit_bytes") + tenant_usage_pct = None + if hard_limit_bytes and hard_limit_bytes > 0: + tenant_usage_pct = round(total_bytes / hard_limit_bytes * 100, 2) + tenant_warning_level = self._compute_tenant_warning_level( + tenant_usage_pct, + warning_config["critical_threshold_pct"], + warning_config["warning_threshold_pct"], + ) + + available_bytes = None + if hard_limit_bytes: + available_bytes = max(0, hard_limit_bytes - total_bytes) + + result = { + "total_bytes": total_bytes, + "total_readable": _bytes_to_readable(total_bytes), + "kb_count": len(kb_list), + "file_count": total_files, + "hard_limit_bytes": hard_limit_bytes, + "hard_limit_readable": hard_limit_info.get("hard_limit_readable"), + "available_bytes": available_bytes, + "available_readable": _bytes_to_readable(available_bytes), + "usage_pct": tenant_usage_pct, + "tenant_warning_level": tenant_warning_level, + "warning_enabled": warning_config["warning_enabled"], + "warning_threshold_pct": warning_config["warning_threshold_pct"], + "critical_threshold_pct": warning_config["critical_threshold_pct"], + "breakdown": breakdown, + } + + # Add summary when detail is provided + summary = self.get_quota_summary() + result["soft_allocated_total_bytes"] = summary["soft_allocated_total_bytes"] + result["soft_allocated_readable"] = summary["soft_allocated_readable"] + result["oversubscription_ratio"] = summary["oversubscription_ratio"] + result["kbs_with_quota"] = summary["kbs_with_quota"] + + return result + + @staticmethod + def _parse_store_size(size_str: Any) -> int: + """Parse store_size string like '1.5 GB' or '500 MB' into bytes.""" + if size_str is None: + return 0 + if isinstance(size_str, (int, float)): + return int(size_str) + if not isinstance(size_str, str) or not size_str.strip(): + return 0 + try: + parts = size_str.strip().split() + if len(parts) != 2: + return 0 + value = float(parts[0]) + unit = parts[1].upper() + if unit == "GB": + return int(value * GB) + elif unit == "MB": + return int(value * 1024 * 1024) + elif unit == "KB": + return int(value * 1024) + elif unit == "B": + return int(value) + return 0 + except (ValueError, IndexError): + return 0 + + # ── Quota Enforcement (tasks 4.1) ────────────────────────────────── + + def check_hard_limit( + self, + file_size_bytes: int, + index_name: Optional[str] = None, + ) -> Dict[str, Any]: + """ + Check if adding file_size_bytes would exceed the tenant hard limit. + Returns quota_status dict if OK, raises QuotaExceededError if exceeded. + """ + hard_limit_info = self.get_hard_limit() + hard_limit_bytes = hard_limit_info.get("hard_limit_bytes") + + # No hard limit set = unlimited, always OK + if hard_limit_bytes is None: + return self._build_quota_status(index_name) + + usage = self.get_usage(force_refresh=True) + current_bytes = usage.get("total_bytes", 0) + projected_bytes = current_bytes + file_size_bytes + + if projected_bytes > hard_limit_bytes: + raise QuotaExceededError( + f"Tenant storage full: {_bytes_to_readable(projected_bytes)} exceeds " + f"hard limit of {_bytes_to_readable(hard_limit_bytes)}", + usage_bytes=current_bytes, + hard_limit_bytes=hard_limit_bytes, + exceeded_by_bytes=projected_bytes - hard_limit_bytes, + ) + + return self._build_quota_status(index_name) + + def check_hard_limit_post_write( + self, + file_size_bytes: int, + index_name: Optional[str] = None, + ) -> Dict[str, Any]: + """ + Post-write belt-and-suspenders check. + Returns quota_status if OK, raises QuotaExceededError if exceeded. + Called after MinIO write to handle race conditions. + """ + hard_limit_info = self.get_hard_limit() + hard_limit_bytes = hard_limit_info.get("hard_limit_bytes") + + if hard_limit_bytes is None: + return self._build_quota_status(index_name) + + # Force refresh to get accurate post-write state + usage = self.get_usage(force_refresh=True) + if usage.get("total_bytes", 0) > hard_limit_bytes: + raise QuotaExceededError( + f"Tenant storage limit exceeded after write", + usage_bytes=usage["total_bytes"], + hard_limit_bytes=hard_limit_bytes, + exceeded_by_bytes=usage["total_bytes"] - hard_limit_bytes, + ) + + return self._build_quota_status(index_name) + + def _build_quota_status(self, index_name: Optional[str] = None) -> Dict[str, Any]: + """Build dual-level quota status for upload responses.""" + usage = self.get_usage(force_refresh=True, detail=True) + hard_limit_info = self.get_hard_limit() + + # Tenant-level status + hard_limit_bytes = hard_limit_info.get("hard_limit_bytes") + tenant_usage_pct = usage.get("usage_pct") + tenant_warning_level = usage.get("tenant_warning_level", "normal") + kb_usage_pct = None + kb_warning_level = "normal" + if index_name: + kb_status = next( + ( + item + for item in usage.get("breakdown", []) + if item.get("index_name") == index_name + ), + None, + ) + if kb_status: + kb_usage_pct = kb_status.get("usage_pct") + kb_warning_level = kb_status.get("kb_warning_level", "normal") + + return { + "quota_status": { + "warning_enabled": usage.get("warning_enabled", True), + "tenant_level": { + "usage_pct": tenant_usage_pct, + "warning_level": tenant_warning_level, + "hard_limit_bytes": hard_limit_bytes, + "hard_limit_readable": hard_limit_info.get("hard_limit_readable"), + "total_bytes": usage.get("total_bytes"), + "total_readable": usage.get("total_readable"), + }, + "kb_level": { + "usage_pct": kb_usage_pct, + "warning_level": kb_warning_level, + }, + } + } + + # ── Platform-Level Methods (tasks 9.1–9.3) ───────────────────────── + + @staticmethod + def _quota_input_to_bytes(limit_gb: Optional[int], limit_mb: Optional[int]) -> int: + """Convert an API quota value to bytes.""" + if limit_mb is not None: + return _mb_to_bytes(int(limit_mb)) + return _gb_to_bytes(int(limit_gb)) + + @staticmethod + def _get_allocation_state(asset_owner_tenant_id: str) -> Dict[str, Any]: + """Return finite tenant allocations and unmanaged tenant count.""" + from database.tenant_config_db import get_all_tenant_ids, get_single_config_info + + tenant_ids = [ + tenant_id + for tenant_id in get_all_tenant_ids() + if _is_displayable_tenant_id(tenant_id, asset_owner_tenant_id) + ] + hard_limits: Dict[str, Optional[int]] = {} + total_allocated_bytes = 0 + unmanaged_tenant_count = 0 + for tenant_id in tenant_ids: + record = get_single_config_info(tenant_id, KEY_TENANT_HARD_LIMIT_BYTES) + try: + hard_limit_bytes = int(record["config_value"]) if record and record.get("config_value") else None + except (TypeError, ValueError): + hard_limit_bytes = None + hard_limits[tenant_id] = hard_limit_bytes + if hard_limit_bytes is None: + unmanaged_tenant_count += 1 + else: + total_allocated_bytes += hard_limit_bytes + return { + "tenant_ids": tenant_ids, + "hard_limits": hard_limits, + "total_allocated_bytes": total_allocated_bytes, + "unmanaged_tenant_count": unmanaged_tenant_count, + } + + def _validate_tenant_hard_limit( + self, + limit_bytes: int, + asset_owner_tenant_id: str = ASSET_OWNER_TENANT_ID, + ) -> None: + """Validate a finite tenant quota against usage and platform allocation.""" + usage = self.get_usage(force_refresh=True) + actual_bytes = usage.get("total_bytes", 0) + if limit_bytes < actual_bytes: + raise PlatformQuotaConflictError( + "Tenant hard quota cannot be lower than current usage", + "TenantQuotaBelowUsage", + { + "tenant_id": self.tenant_id, + "requested_limit_bytes": limit_bytes, + "requested_limit_readable": _bytes_to_readable(limit_bytes), + "actual_usage_bytes": actual_bytes, + "actual_usage_readable": _bytes_to_readable(actual_bytes), + }, + ) + + capacity_bytes = QuotaService.get_platform_capacity(asset_owner_tenant_id).get("capacity_bytes") + if capacity_bytes is None: + return + + allocation_state = QuotaService._get_allocation_state(asset_owner_tenant_id) + current_limit_bytes = allocation_state["hard_limits"].get(self.tenant_id) or 0 + proposed_total_bytes = allocation_state["total_allocated_bytes"] - current_limit_bytes + limit_bytes + if proposed_total_bytes > capacity_bytes: + raise PlatformQuotaConflictError( + "Tenant hard quota exceeds remaining platform capacity", + "PlatformCapacityExceeded", + { + "tenant_id": self.tenant_id, + "requested_limit_bytes": limit_bytes, + "requested_limit_readable": _bytes_to_readable(limit_bytes), + "platform_capacity_bytes": capacity_bytes, + "platform_capacity_readable": _bytes_to_readable(capacity_bytes), + "total_allocated_bytes": allocation_state["total_allocated_bytes"], + "total_allocated_readable": _bytes_to_readable(allocation_state["total_allocated_bytes"]), + "remaining_allocatable_bytes": max(capacity_bytes - allocation_state["total_allocated_bytes"], 0), + "remaining_allocatable_readable": _bytes_to_readable( + max(capacity_bytes - allocation_state["total_allocated_bytes"], 0) + ), + }, + ) + + @staticmethod + def get_platform_capacity(asset_owner_tenant_id: str = ASSET_OWNER_TENANT_ID) -> Dict[str, Any]: + """Get platform-level declared storage capacity.""" + from database.tenant_config_db import get_single_config_info + record = get_single_config_info(asset_owner_tenant_id, KEY_PLATFORM_CAPACITY_BYTES) + raw = record.get("config_value") if record else None + + if raw is not None: + try: + capacity_bytes = int(raw) + return { + "capacity_bytes": capacity_bytes, + "capacity_readable": _bytes_to_readable(capacity_bytes), + } + except (ValueError, TypeError): + pass + + return {"capacity_bytes": None, "capacity_readable": None} + + @staticmethod + def set_platform_capacity( + capacity_gb: Optional[int], + asset_owner_tenant_id: str = ASSET_OWNER_TENANT_ID, + user_id: str = "system", + ) -> Dict[str, Any]: + """Set platform-level declared storage capacity. None = no tracking.""" + service = QuotaService(asset_owner_tenant_id, user_id) + if capacity_gb is None: + service._delete_tenant_config(KEY_PLATFORM_CAPACITY_BYTES) + return {"capacity_bytes": None, "capacity_readable": None} + + capacity_bytes = _gb_to_bytes(int(capacity_gb)) + with _platform_allocation_lock: + allocation_state = QuotaService._get_allocation_state(asset_owner_tenant_id) + allocated_bytes = allocation_state["total_allocated_bytes"] + if capacity_bytes < allocated_bytes: + raise PlatformQuotaConflictError( + "Platform capacity cannot be lower than existing tenant allocations", + "PlatformCapacityBelowAllocation", + { + "requested_capacity_bytes": capacity_bytes, + "requested_capacity_readable": _bytes_to_readable(capacity_bytes), + "total_allocated_bytes": allocated_bytes, + "total_allocated_readable": _bytes_to_readable(allocated_bytes), + }, + ) + service._set_tenant_config(KEY_PLATFORM_CAPACITY_BYTES, str(capacity_bytes)) + return { + "capacity_bytes": capacity_bytes, + "capacity_readable": _bytes_to_readable(capacity_bytes), + } + + @staticmethod + def get_platform_overview( + asset_owner_tenant_id: str = ASSET_OWNER_TENANT_ID, + ) -> Dict[str, Any]: + """ + Aggregate all tenants' hard limits and actual usage. + Returns per-tenant breakdown + platform totals. + """ + capacity_info = QuotaService.get_platform_capacity(asset_owner_tenant_id) + + allocation_state = QuotaService._get_allocation_state(asset_owner_tenant_id) + tenant_ids = allocation_state["tenant_ids"] + + tenants = [] + total_allocated_bytes = 0 + total_actual_bytes = 0 + + for tid in tenant_ids: + # Get hard limit for this tenant + hard_limit_bytes = allocation_state["hard_limits"].get(tid) + if hard_limit_bytes is not None: + total_allocated_bytes += hard_limit_bytes + + # Get actual usage for this tenant + service = QuotaService(tid) + try: + usage = service.get_usage(force_refresh=True) + actual_bytes = usage.get("total_bytes", 0) + warning_enabled = usage.get("warning_enabled", True) + warning_level = ( + usage.get("tenant_warning_level", "normal") + if warning_enabled + else "normal" + ) + except Exception: + logger.warning("Failed to get usage for tenant %s", tid, exc_info=True) + actual_bytes = 0 + warning_enabled = False + warning_level = "normal" + + total_actual_bytes += actual_bytes + + usage_pct = None + if hard_limit_bytes and hard_limit_bytes > 0: + usage_pct = round(actual_bytes / hard_limit_bytes * 100, 2) + + # Try to get tenant name from config + from database.tenant_config_db import get_single_config_info as gsci + from consts.const import TENANT_NAME + name_record = gsci(tid, TENANT_NAME) + tenant_name = name_record.get("config_value") if name_record else tid + + tenants.append({ + "tenant_id": tid, + "tenant_name": tenant_name or tid, + "hard_limit_bytes": hard_limit_bytes, + "hard_limit_readable": _bytes_to_readable(hard_limit_bytes), + "actual_bytes": actual_bytes, + "actual_readable": _bytes_to_readable(actual_bytes), + "usage_pct": usage_pct, + "warning_level": warning_level, + "warning_enabled": warning_enabled, + }) + + platform_capacity = capacity_info.get("capacity_bytes") + oversubscription_ratio = None + remaining_allocatable_bytes = None + allocation_percentage = None + if platform_capacity is not None: + remaining_allocatable_bytes = max(platform_capacity - total_allocated_bytes, 0) + if platform_capacity > 0: + oversubscription_ratio = round(total_allocated_bytes / platform_capacity, 4) + allocation_percentage = round(total_allocated_bytes / platform_capacity * 100, 2) + elif total_allocated_bytes == 0: + allocation_percentage = 0 + + return { + "platform_capacity_bytes": platform_capacity, + "platform_capacity_readable": capacity_info.get("capacity_readable"), + "tenants": tenants, + "total_allocated_bytes": total_allocated_bytes, + "total_allocated_readable": _bytes_to_readable(total_allocated_bytes), + "total_actual_bytes": total_actual_bytes, + "total_actual_readable": _bytes_to_readable(total_actual_bytes), + "tenant_count": len(tenants), + "oversubscription_ratio": oversubscription_ratio, + "remaining_allocatable_bytes": remaining_allocatable_bytes, + "remaining_allocatable_readable": _bytes_to_readable(remaining_allocatable_bytes), + "allocation_percentage": allocation_percentage, + "unmanaged_tenant_count": allocation_state["unmanaged_tenant_count"], + "capacity_management_enforced": ( + platform_capacity is not None and allocation_state["unmanaged_tenant_count"] == 0 + ), + } + + @staticmethod + def set_tenant_hard_limit( + tenant_id: str, + limit_gb: Optional[int] = None, + limit_mb: Optional[int] = None, + su_user_id: str = "system", + ) -> Dict[str, Any]: + """ + SU sets a hard quota on a target tenant. Accepts limit_gb or limit_mb. + Sets hard_limit_editable = false so the tenant admin cannot modify it. + """ + service = QuotaService(tenant_id, su_user_id) + if limit_gb is None and limit_mb is None: + service._delete_tenant_config(KEY_TENANT_HARD_LIMIT_BYTES) + service._delete_tenant_config(KEY_HARD_LIMIT_EDITABLE) + return {"hard_limit_bytes": None, "hard_limit_readable": None} + + limit_bytes = QuotaService._quota_input_to_bytes(limit_gb, limit_mb) + with _platform_allocation_lock: + service._validate_tenant_hard_limit(limit_bytes) + service._set_tenant_config(KEY_TENANT_HARD_LIMIT_BYTES, str(limit_bytes)) + # Mark as SU-managed (not editable by tenant admin) + service._set_tenant_config(KEY_HARD_LIMIT_EDITABLE, "false") + return { + "hard_limit_bytes": limit_bytes, + "hard_limit_readable": _bytes_to_readable(limit_bytes), + } + + @staticmethod + def delete_tenant_hard_limit( + tenant_id: str, + su_user_id: str = "system", + ) -> bool: + """SU removes a tenant's hard quota.""" + service = QuotaService(tenant_id, su_user_id) + service._delete_tenant_config(KEY_TENANT_HARD_LIMIT_BYTES) + service._delete_tenant_config(KEY_HARD_LIMIT_EDITABLE) + return True diff --git a/backend/services/redis_service.py b/backend/services/redis_service.py index 1ffcf921c0..2fa5f0a354 100644 --- a/backend/services/redis_service.py +++ b/backend/services/redis_service.py @@ -5,7 +5,12 @@ import redis -from consts.const import REDIS_URL, REDIS_BACKEND_URL +from consts.const import ( + REDIS_BACKEND_URL, + REDIS_ERROR_INFO_SCAN_COUNT, + REDIS_ERROR_INFO_TTL_SECONDS, + REDIS_URL, +) logger = logging.getLogger(__name__) @@ -635,14 +640,19 @@ def ping(self) -> bool: logger.error(f"Redis ping failed: {str(e)}") return False - def save_error_info(self, task_id: str, error_reason: str, ttl_days: int = 30) -> bool: + def save_error_info( + self, + task_id: str, + error_reason: str, + ttl_days: Optional[int] = None, + ) -> bool: """ Save error information to Redis for a specific task Args: task_id: Celery task ID error_reason: Short error reason summary - ttl_days: Time to live in days (default 30 days) + ttl_days: Optional TTL override in days Returns: True if saved successfully, False otherwise @@ -655,7 +665,11 @@ def save_error_info(self, task_id: str, error_reason: str, ttl_days: int = 30) - logger.error(f"Cannot save error info for task {task_id}: error_reason is empty") return False - ttl_seconds = ttl_days * 24 * 60 * 60 + ttl_seconds = ( + ttl_days * 24 * 60 * 60 + if ttl_days is not None + else REDIS_ERROR_INFO_TTL_SECONDS + ) reason_key = f"error:reason:{task_id}" # Save error reason @@ -663,12 +677,6 @@ def save_error_info(self, task_id: str, error_reason: str, ttl_days: int = 30) - if result: logger.info(f"Successfully saved error info to Redis for task {task_id}, key: {reason_key}") - # Verify the save by reading it back - verify = self.client.get(reason_key) - if verify: - logger.debug(f"Verified error info saved for task {task_id}: {verify[:100]}...") - else: - logger.warning(f"Failed to verify error info save for task {task_id}") return True else: logger.error(f"Redis setex returned False for task {task_id}") @@ -678,6 +686,94 @@ def save_error_info(self, task_id: str, error_reason: str, ttl_days: int = 30) - f"Failed to save error info for task {task_id}: {str(e)}", exc_info=True) return False + def cleanup_error_info_keys( + self, + ttl_seconds: int = REDIS_ERROR_INFO_TTL_SECONDS, + scan_count: int = REDIS_ERROR_INFO_SCAN_COUNT, + ) -> Dict[str, int]: + """ + Remove orphaned error reasons and enforce retention on legacy keys. + + SCAN is used instead of KEYS so maintenance does not block Redis while + iterating a large keyspace. + """ + stats = { + "scanned": 0, + "deleted_orphans": 0, + "ttl_repaired": 0, + } + if ttl_seconds <= 0 or scan_count <= 0: + logger.warning( + "Skipping error info cleanup because ttl_seconds and scan_count must be positive" + ) + return stats + + try: + batch: List[str] = [] + for key in self.client.scan_iter( + match="error:reason:*", + count=scan_count, + ): + if isinstance(key, bytes): + key = key.decode("utf-8") + batch.append(key) + if len(batch) >= scan_count: + self._cleanup_error_info_batch(batch, ttl_seconds, stats) + batch = [] + + if batch: + self._cleanup_error_info_batch(batch, ttl_seconds, stats) + + logger.info( + "Redis error info cleanup completed: scanned=%s, deleted_orphans=%s, ttl_repaired=%s", + stats["scanned"], + stats["deleted_orphans"], + stats["ttl_repaired"], + ) + except Exception as exc: + logger.warning(f"Failed to clean up Redis error info keys: {exc}") + + return stats + + def _cleanup_error_info_batch( + self, + keys: List[str], + ttl_seconds: int, + stats: Dict[str, int], + ) -> None: + """Clean one bounded batch of error reason keys.""" + if not keys: + return + + task_ids = [key.removeprefix("error:reason:") for key in keys] + backend_pipe = self.backend_client.pipeline() + ttl_pipe = self.client.pipeline() + for task_id in task_ids: + backend_pipe.exists(f"celery-task-meta-{task_id}") + for key in keys: + ttl_pipe.ttl(key) + + task_exists = backend_pipe.execute() + key_ttls = ttl_pipe.execute() + delete_keys = [] + expire_keys = [] + for key, exists, current_ttl in zip(keys, task_exists, key_ttls): + stats["scanned"] += 1 + if not exists: + delete_keys.append(key) + elif current_ttl == -1 or current_ttl > ttl_seconds: + expire_keys.append(key) + + if delete_keys: + stats["deleted_orphans"] += int(self.client.delete(*delete_keys)) + if expire_keys: + expire_pipe = self.client.pipeline() + for key in expire_keys: + expire_pipe.expire(key, ttl_seconds) + stats["ttl_repaired"] += sum( + bool(result) for result in expire_pipe.execute() + ) + def save_progress_info(self, task_id: str, processed_chunks: int, total_chunks: int, ttl_hours: int = 24) -> bool: """ Save progress information to Redis for a specific task diff --git a/backend/services/remote_mcp_service.py b/backend/services/remote_mcp_service.py index 83e0ae1110..e6d53bae97 100644 --- a/backend/services/remote_mcp_service.py +++ b/backend/services/remote_mcp_service.py @@ -36,11 +36,43 @@ get_mcp_custom_headers_by_name_and_url, ) from database.user_tenant_db import get_user_tenant_by_user_id +from database.group_db import query_group_ids_by_user +from database.tool_db import set_mcp_tools_unavailable from services.mcp_container_service import MCPContainerManager from utils.http_client_utils import create_httpx_client logger = logging.getLogger("remote_mcp_service") +MCP_HEALTH_CHECK_TIMEOUT_SECONDS = 10 + + +def _iter_exception_chain(exc: BaseException): + seen: set[int] = set() + current: BaseException | None = exc + while current and id(current) not in seen: + seen.add(id(current)) + yield current + current = current.__cause__ or current.__context__ + + +def _format_mcp_connection_error(exc: BaseException) -> str: + for candidate in _iter_exception_chain(exc): + error_type = type(candidate).__name__.lower() + error_text = str(candidate).lower() + if "timeout" in error_type or any(keyword in error_text for keyword in ("timeout", "timed out", "etimedout")): + return "MCP connection timeout" + if any(keyword in error_text for keyword in ("connection refused", "econnrefused", "actively refused")): + return "MCP connection refused" + if any(keyword in error_text for keyword in ("unauthorized", "forbidden", "authentication", "authorization", "401", "403")): + return "MCP authentication failed" + if any(keyword in error_text for keyword in ("404", "not found", "endpoint")): + return "MCP endpoint not found" + if any(keyword in error_text for keyword in ("protocol", "invalid sse")): + return "MCP protocol or endpoint invalid" + if any(keyword in error_text for keyword in ("dns", "getaddrinfo", "enotfound", "eai_again", "network unreachable")): + return "MCP address unreachable" + return "MCP connection failed" + # --------------------------------------------------------------------------- # Health Check @@ -90,16 +122,22 @@ async def _mcp_protocol_health_check(url_stripped: str, headers: dict) -> list[s httpx_client_factory=create_httpx_client ) - client = Client(transport=transport) - async with client: - # Verify the server can actually serve tools. - # This exercises API key validation and end-to-end connectivity, - # unlike is_connected() which only checks the initialize handshake. - tools_result = await asyncio.wait_for(client.list_tools(), timeout=10) - return [t.name for t in tools_result] if tools_result else [] + async def list_mcp_tools() -> list: + client = Client(transport=transport) + async with client: + # Verify the server can actually serve tools. + # This exercises API key validation and end-to-end connectivity, + # unlike is_connected() which only checks the initialize handshake. + return await client.list_tools() + + tools_result = await asyncio.wait_for( + list_mcp_tools(), + timeout=MCP_HEALTH_CHECK_TIMEOUT_SECONDS, + ) + return [t.name for t in tools_result] if tools_result else [] except BaseException as e: logger.debug(f"MCP protocol health check failed: {e}") - return [] + raise MCPConnectionError(_format_mcp_connection_error(e)) async def _mcp_protocol_connect(url_stripped: str, headers: dict) -> bool: @@ -170,11 +208,22 @@ def _is_container_record(record: dict | None) -> bool: A record is considered container-based if it has: - container_id (Docker container ID) - - config_json (container configuration) + - a non-empty config_json holding a container configuration + + API-type MCPs store OpenAPI JSON in config_json and are never treated as + containers. An empty dict config_json (e.g. `{}`) is not a container + configuration either, so records with only an empty config_json are treated + as plain remote MCPs instead of being misclassified as containers. """ if not record: return False - return record.get("container_id") is not None or record.get("config_json") is not None + config_json = record.get("config_json") + # API-type MCPs store OpenAPI JSON in config_json, not container config + if isinstance(config_json, dict) and "openapi" in config_json: + return False + return record.get("container_id") is not None or ( + isinstance(config_json, dict) and bool(config_json) + ) # --------------------------------------------------------------------------- @@ -258,6 +307,9 @@ async def add_remote_mcp_server_list( custom_headers: dict | None = None, source: str | None = "local", container_port: int | None = None, + group_ids: str | None = None, + ingroup_permission: str | None = None, + shared_fields: dict | None = None, ): """Add a remote MCP server to the list. @@ -298,6 +350,9 @@ async def add_remote_mcp_server_list( "source": source, "container_port": container_port, "registry_json": {"_toolNames": tool_names}, + "group_ids": group_ids, + "ingroup_permission": ingroup_permission, + "shared_fields": shared_fields, } create_mcp_record(mcp_data=insert_mcp_data, tenant_id=tenant_id, user_id=user_id) @@ -351,6 +406,10 @@ async def add_mcp_service( enabled: bool = False, container_id: str | None = None, container_port: int | None = None, + group_ids: str | None = None, + ingroup_permission: str | None = None, + shared_fields: dict | None = None, + skip_health_check: bool = False, ) -> None: """Add an MCP service record. @@ -371,6 +430,8 @@ async def add_mcp_service( enabled: Whether the MCP is enabled container_id: Docker container ID container_port: Container port + group_ids: Comma-separated group IDs that can access this MCP + ingroup_permission: Permission level: EDIT, READ_ONLY, PRIVATE """ status: bool | None = None normalized_container_id = container_id if isinstance(container_id, str) and container_id else None @@ -383,10 +444,43 @@ async def add_mcp_service( resolved_registry_json = registry_json or {} if server_url: - headers = _build_mcp_headers(authorization_token, custom_headers) - tool_names = await _check_mcp_connectivity(server_url, headers, is_container, name) - if tool_names: - resolved_registry_json["_toolNames"] = tool_names + # API-type MCPs use OpenAPI JSON, not MCP protocol + is_api = isinstance(resolved_config_json, dict) and "openapi" in resolved_config_json + if is_api: + # Register OpenAPI service (same as agent config flow) + try: + from services.tool_configuration_service import import_openapi_service, _refresh_openapi_services_in_mcp + import_openapi_service( + service_name=name, + openapi_json=resolved_config_json, + server_url=server_url, + tenant_id=tenant_id, + user_id=user_id, + service_description=description, + headers_template=custom_headers, + force_update=True, + ) + _refresh_openapi_services_in_mcp(tenant_id) + except Exception as exc: + logger.warning(f"Failed to register OpenAPI service '{name}': {exc}") + # Extract tool names from OpenAPI spec for display + api_tools = [] + paths = resolved_config_json.get("paths", {}) or {} + for path, methods in paths.items(): + if isinstance(methods, dict): + for method_name, detail in methods.items(): + if isinstance(detail, dict): + tool_name = detail.get("operationId") or detail.get("summary") or "" + if tool_name: + api_tools.append(tool_name) + if api_tools: + resolved_registry_json["_toolNames"] = api_tools + else: + headers = _build_mcp_headers(authorization_token, custom_headers) + if not skip_health_check: + tool_names = await _check_mcp_connectivity(server_url, headers, is_container, name) + if tool_names: + resolved_registry_json["_toolNames"] = tool_names if enabled: status = True @@ -407,6 +501,9 @@ async def add_mcp_service( "tags": tags, "description": description, "config_json": resolved_config_json, + "group_ids": group_ids, + "ingroup_permission": ingroup_permission, + "shared_fields": shared_fields, }, tenant_id=tenant_id, user_id=user_id, @@ -426,6 +523,9 @@ async def add_container_mcp_service( market_id: int | None, port: int, mcp_config: MCPConfigRequest, + group_ids: str | None = None, + ingroup_permission: str | None = None, + shared_fields: dict | None = None, ) -> dict: """Add a container-based MCP service. @@ -441,6 +541,8 @@ async def add_container_mcp_service( community_id: Linked community record ID port: Host port for the container mcp_config: MCP server configuration + group_ids: Comma-separated group IDs that can access this MCP + ingroup_permission: Permission level: EDIT, READ_ONLY, PRIVATE Returns: Container information dictionary @@ -513,9 +615,16 @@ async def add_container_mcp_service( enabled=True, container_id=container_info.get("container_id"), container_port=container_info.get("host_port"), + group_ids=group_ids, + ingroup_permission=ingroup_permission, ) except Exception as exc: logger.warning(f"Failed to start container MCP service: {exc}") + # Clean up orphan container if it was started + try: + await container_manager.stop_mcp_container(container_info.get("container_id")) + except Exception: + pass raise return { @@ -586,6 +695,9 @@ def update_mcp_service( config_json: dict | None, tags: list | None, market_id: int | None, + group_ids: str | None = None, + ingroup_permission: str | None = None, + shared_fields: dict | None = None, ) -> None: """Update an MCP service record by ID. @@ -601,6 +713,8 @@ def update_mcp_service( config_json: MCP configuration JSON tags: MCP tags market_id: Linked market record ID + group_ids: Comma-separated group IDs that can access this MCP + ingroup_permission: Permission level: EDIT, READ_ONLY, PRIVATE Raises: McpNotFoundError: If MCP record is not found @@ -609,6 +723,12 @@ def update_mcp_service( if not current_record: raise McpNotFoundError("MCP record not found") + # Check name uniqueness (exclude the current record itself) + if new_name != current_record.get("mcp_name"): + if check_mcp_name_exists(mcp_name=new_name, tenant_id=tenant_id): + logger.error(f"MCP name already exists: {new_name} in tenant {tenant_id}") + raise McpNameConflictError("MCP name already exists") + current_config_json = current_record.get("config_json") if isinstance(current_record.get("config_json"), dict) else None next_config_json = config_json if config_json is not None else current_config_json @@ -627,6 +747,9 @@ def update_mcp_service( config_json=next_config_json, tags=tags, market_id=next_market_id, + group_ids=group_ids, + ingroup_permission=ingroup_permission, + shared_fields=shared_fields, ) @@ -675,8 +798,36 @@ async def update_mcp_service_enabled( port = current_record.get("container_port") if port is None: raise McpValidationError("Container port is missing, cannot rebuild container") + + # Clean up any existing container before starting a new one + old_container_id = current_record.get("container_id") + if old_container_id: + try: + await MCPContainerManager().stop_mcp_container(old_container_id) + logger.info("Stopped existing container %s before re-enabling", old_container_id) + except Exception as exc: + logger.warning("Failed to stop existing container %s: %s", old_container_id, exc) + if not check_runtime_host_port_available(port): - raise McpPortConflictError(f"Port {port} is already in use") + # Orphan container recovery: when the port is in use but the DB has no + # container_id (e.g. the previous enable request was aborted right after + # the container started but before the DB write), try to find and stop + # any MCP container occupying this port. + try: + orphan_manager = MCPContainerManager() + for candidate in orphan_manager.list_mcp_containers(tenant_id=tenant_id): + if str(candidate.get("host_port")) == str(port): + logger.warning( + "Found orphan container %s on port %s, stopping it", + candidate.get("container_id"), port, + ) + await orphan_manager.stop_mcp_container(candidate["container_id"]) + break + except Exception as cleanup_exc: + logger.warning("Failed to clean up orphan container on port %s: %s", port, cleanup_exc) + + if not check_runtime_host_port_available(port): + raise McpPortConflictError(f"Port {port} is already in use") config_json = current_record.get("config_json") if not isinstance(config_json, dict): @@ -791,19 +942,30 @@ async def update_mcp_service_enabled( ) elif enabled: server_url = current_record.get("mcp_server") - health_ok = await mcp_server_health( - remote_mcp_server=server_url, - authorization_token=authorization_token, - custom_headers=custom_headers, - ) - update_mcp_record_status_by_id( - mcp_id=mcp_id, - tenant_id=tenant_id, - user_id=user_id, - status=bool(health_ok), - ) - if not health_ok: - raise MCPConnectionError("MCP connection failed") + # Skip MCP protocol check for API-type MCPs + config_json = current_record.get("config_json") + api_type = isinstance(config_json, dict) and "openapi" in config_json + if api_type: + update_mcp_record_status_by_id( + mcp_id=mcp_id, + tenant_id=tenant_id, + user_id=user_id, + status=True, + ) + else: + health_ok = await mcp_server_health( + remote_mcp_server=server_url, + authorization_token=authorization_token, + custom_headers=custom_headers, + ) + update_mcp_record_status_by_id( + mcp_id=mcp_id, + tenant_id=tenant_id, + user_id=user_id, + status=bool(health_ok), + ) + if not health_ok: + raise MCPConnectionError("MCP connection failed") update_mcp_record_enabled_by_id( mcp_id=mcp_id, @@ -844,6 +1006,17 @@ async def delete_mcp_service( except Exception as exc: logger.warning(f"Failed to stop container: {exc}, but continue to delete MCP record") + # Hide the deleted MCP's tools from the agent tool selection list so they + # no longer appear after deletion (tool rows are kept for agent references). + try: + set_mcp_tools_unavailable( + tenant_id=tenant_id, + mcp_server_name=current_record.get("mcp_name") or "", + user_id=user_id, + ) + except Exception as exc: + logger.warning(f"Failed to mark MCP tools unavailable for '{current_record.get('mcp_name')}': {exc}") + delete_mcp_record_by_id( mcp_id=mcp_id, tenant_id=tenant_id, @@ -853,6 +1026,19 @@ async def delete_mcp_service( async def delete_mcp_by_container_id(tenant_id: str, user_id: str, container_id: str) -> None: """Soft delete MCP record associated with a specific container ID.""" + # Hide the deleted MCP's tools from the agent tool selection list. + try: + for record in get_mcp_records_by_tenant(tenant_id=tenant_id): + if str(record.get("container_id") or "") == str(container_id): + set_mcp_tools_unavailable( + tenant_id=tenant_id, + mcp_server_name=record.get("mcp_name") or "", + user_id=user_id, + ) + break + except Exception as exc: + logger.warning(f"Failed to mark MCP tools unavailable for container {container_id}: {exc}") + delete_mcp_record_by_container_id( container_id=container_id, tenant_id=tenant_id, @@ -884,10 +1070,42 @@ async def get_remote_mcp_server_list( mcp_records = get_mcp_records_by_tenant(tenant_id=tenant_id) mcp_records_list = [] can_edit_all = False + user_groups: list[str] | None = None if user_id: user_tenant_record = get_user_tenant_by_user_id(user_id) or {} user_role = str(user_tenant_record.get("user_role") or "").upper() can_edit_all = user_role in CAN_EDIT_ALL_USER_ROLES + try: + raw_groups = query_group_ids_by_user(user_id) or [] + user_groups = [str(g) for g in raw_groups] + except Exception: + user_groups = [] + + if user_groups is not None: + filtered_records = [] + for record in mcp_records: + # NULL group_ids means public (backward compatible with pre-PR data) + if record.get("group_ids") is None: + filtered_records.append(record) + continue + record_group_ids = (record.get("group_ids") or "").strip() + # User can see MCPs they created + if str(record.get("created_by") or record.get("user_id") or "") == user_id: + filtered_records.append(record) + continue + # User can see MCPs where they belong to at least one allowed group + if user_groups: + allowed = [g.strip() for g in record_group_ids.split(",") if g.strip()] + if any(g in allowed for g in user_groups): + # Hide PRIVATE MCPs from non-creator group members (like agent behavior) + ingroup_perm = (record.get("ingroup_permission") or "").upper() + if ingroup_perm == "PRIVATE": + continue + filtered_records.append(record) + continue + logger.info(f"[MCP group filter] user_id={user_id}, groups={user_groups}, " + f"total={len(mcp_records)}, filtered={len(filtered_records)}") + mcp_records = filtered_records container_status_map = {} try: @@ -910,11 +1128,25 @@ async def get_remote_mcp_server_list( permission = PERMISSION_READ else: permission = PERMISSION_EDIT if can_edit_all or str(created_by) == str(user_id) else PERMISSION_READ + # Public MCPs (NULL group_ids) are editable by all users + if record.get("group_ids") is None: + permission = PERMISSION_EDIT + # For group-shared MCPs, respect ingroup_permission + if permission == PERMISSION_READ and user_groups: + record_group_ids = (record.get("group_ids") or "").strip() + if record_group_ids: + allowed = [g.strip() for g in record_group_ids.split(",") if g.strip()] + if any(g in allowed for g in user_groups): + ingroup_perm = (record.get("ingroup_permission") or "READ_ONLY").upper() + if ingroup_perm == "EDIT": + permission = PERMISSION_EDIT config_json = record.get("config_json") container_id = record.get("container_id") - is_container = container_id is not None or config_json is not None + # Reuse _is_container_record so an empty config_json (e.g. `{}`) is not + # misclassified as a container, matching the API/enable path behavior. + is_container = _is_container_record(record) container_status = None if is_container: @@ -934,6 +1166,7 @@ async def get_remote_mcp_server_list( "enabled": record.get("enabled"), "source": record.get("source"), "update_time": record.get("update_time"), + "create_time": record.get("create_time"), "tags": record.get("tags") or [], "container_port": record.get("container_port"), "registry_json": record.get("registry_json"), @@ -941,6 +1174,9 @@ async def get_remote_mcp_server_list( "market_id": record.get("market_id"), "is_listed_in_repository": record.get("market_id") is not None, "container_status": container_status, + "group_ids": record.get("group_ids"), + "ingroup_permission": record.get("ingroup_permission"), + "shared_fields": record.get("shared_fields"), } if is_need_auth: record_dict["authorization_token"] = record.get("authorization_token") @@ -1143,6 +1379,10 @@ async def check_mcp_service_health( async def list_mcp_service_tools_by_id(*, tenant_id: str, mcp_id: int) -> list[dict]: """Get tools from an MCP service by ID. + For API-type MCPs (OpenAPI), tools are already registered in the database + via import_openapi_service. Return them from the tool registry instead of + attempting an MCP-protocol connection. + Args: tenant_id: Tenant ID mcp_id: MCP record ID @@ -1159,6 +1399,22 @@ async def list_mcp_service_tools_by_id(*, tenant_id: str, mcp_id: int) -> list[d if not record: raise McpNotFoundError("MCP record not found") + config_json = record.get("config_json") + registry_json = record.get("registry_json") + is_api_type = isinstance(config_json, dict) and "openapi" in config_json + if is_api_type: + # API-type MCPs have no MCP protocol endpoint. + # Return the tool names that were extracted during registration. + tool_names = [] + if isinstance(registry_json, dict): + raw = registry_json.get("_toolNames") + if isinstance(raw, list): + tool_names = raw + return [ + {"name": name, "description": ""} + for name in tool_names + ] + service_name = record.get("mcp_name") server_url = record.get("mcp_server") if not service_name or not server_url: @@ -1210,6 +1466,11 @@ async def refresh_mcp_service_tool_count( authorization_token = record.get("authorization_token") custom_headers = record.get("custom_headers") + # Skip MCP protocol check for API-type MCPs (they use OpenAPI JSON, not MCP) + config_json = record.get("config_json") + if isinstance(config_json, dict) and "openapi" in config_json: + return + headers = {} if authorization_token: headers["Authorization"] = authorization_token @@ -1244,6 +1505,9 @@ async def upload_and_start_mcp_image( port: int, service_name: str | None = None, env_vars: str | None = None, + group_ids: str | None = None, + ingroup_permission: str | None = None, + shared_fields: dict | None = None, ) -> dict: """Upload MCP Docker image and start container. @@ -1313,15 +1577,32 @@ async def upload_and_start_mcp_image( if parsed_env_vars: authorization_token = parsed_env_vars.get("authorization_token") - await add_remote_mcp_server_list( - tenant_id=tenant_id, - user_id=user_id, - remote_mcp_server=container_info["mcp_url"], - remote_mcp_server_name=final_service_name, - container_id=container_info["container_id"], - authorization_token=authorization_token, - container_port=port - ) + try: + await add_remote_mcp_server_list( + tenant_id=tenant_id, + user_id=user_id, + remote_mcp_server=container_info["mcp_url"], + remote_mcp_server_name=final_service_name, + container_id=container_info["container_id"], + authorization_token=authorization_token, + container_port=port, + group_ids=group_ids, + ingroup_permission=ingroup_permission, + shared_fields=shared_fields, + ) + except Exception as exc: + logger.warning( + f"Failed to register uploaded-image MCP service: {exc}; " + "cleaning up the started container so it does not become an orphan " + "that keeps occupying the host port" + ) + try: + await container_manager.stop_mcp_container(container_info["container_id"]) + except Exception as cleanup_exc: + logger.warning( + f"Failed to clean up container {container_info['container_id']}: {cleanup_exc}" + ) + raise return { "message": "MCP container started successfully from uploaded image", diff --git a/backend/services/repository_import_precheck.py b/backend/services/repository_import_precheck.py index 80c061de4a..eaea6360ed 100644 --- a/backend/services/repository_import_precheck.py +++ b/backend/services/repository_import_precheck.py @@ -181,6 +181,7 @@ def _extract_mcp_server_names(snapshot: Any) -> Set[str]: def _extract_knowledge_bases( snapshot: Any, + tenant_id: str, ) -> List[Tuple[str, str, Optional[str]]]: """Return (key, display_name, description) tuples for knowledge bases.""" index_names: Set[str] = set() @@ -198,7 +199,10 @@ def _extract_knowledge_bases( if not index_names: return [] - name_map = get_knowledge_name_map_by_index_names(list(index_names)) + name_map = get_knowledge_name_map_by_index_names( + list(index_names), + tenant_id=tenant_id, + ) items: List[Tuple[str, str, Optional[str]]] = [] for index_name in sorted(index_names): display_name = name_map.get(index_name) or index_name @@ -276,7 +280,7 @@ def build_repository_import_precheck( reason_code=reason, )) - for key, kb_name, description in _extract_knowledge_bases(snapshot): + for key, kb_name, description in _extract_knowledge_bases(snapshot, tenant_id): index_name = key.split(":", 1)[1] available, reason = _check_kb_available(index_name, tenant_id) record = get_knowledge_record({ diff --git a/backend/services/skill_repository_service.py b/backend/services/skill_repository_service.py index 28f181c756..9bd92fd717 100644 --- a/backend/services/skill_repository_service.py +++ b/backend/services/skill_repository_service.py @@ -15,8 +15,12 @@ VALID_OWNERSHIP_FILTERS, VALID_REPOSITORY_STATUSES, ) -from consts.const import CAN_EDIT_ALL_USER_ROLES, PERMISSION_EDIT, PERMISSION_READ +from consts.const import PERMISSION_PRIVATE, PERMISSION_READ from consts.exceptions import ForbiddenError, SkillDuplicateError, SkillException +from consts.notification import ( + EVENT_TYPE_REPOSITORY_REVIEW_PENDING, + RESOURCE_TYPE_SKILL_REPOSITORY, +) from database.skill_repository_db import ( get_skill_repository_by_id_and_publisher, get_skill_repository_by_skill_id, @@ -24,11 +28,17 @@ insert_skill_repository_record, list_skill_repository_by_skill_ids, list_skill_repository_summaries, + reset_skill_repository_status, update_skill_repository_by_id, update_skill_repository_status_by_id, ) from database.skill_db import get_skill_by_name from database.user_tenant_db import get_user_tenant_by_user_id +from services.notification_service import ( + create_repository_pending_review_notification, + create_repository_review_notification, + deactivate_notifications, +) from services.skill_service import SkillService logger = logging.getLogger("skill_repository_service") @@ -80,6 +90,7 @@ "skill_info_json", "skill_zip_base64", "status", + "content", ) @@ -92,9 +103,13 @@ def _serialize_created_at(create_time: Any) -> Optional[str]: return str(create_time) -def _to_summary_item(record: Dict[str, Any]) -> Dict[str, Any]: +def _to_summary_item( + record: Dict[str, Any], + *, + can_take_down: Optional[bool] = None, +) -> Dict[str, Any]: """Map a DB record to a lightweight skill marketplace summary item.""" - return { + item = { "id": record.get("skill_repository_id"), "skill_repository_id": record.get("skill_repository_id"), "skill_id": record.get("skill_id"), @@ -109,7 +124,11 @@ def _to_summary_item(record: Dict[str, Any]) -> Dict[str, Any]: "downloads": record.get("downloads") or 0, "created_at": record.get("created_at") or _serialize_created_at(record.get("create_time")), "updated_at": record.get("updated_at") or _serialize_created_at(record.get("update_time")), + "content": record.get("content"), } + if can_take_down is not None: + item["can_take_down"] = can_take_down + return item def _to_detail_item( @@ -119,12 +138,16 @@ def _to_detail_item( ) -> Dict[str, Any]: """Map a DB record to a skill marketplace detail payload.""" snapshot = _as_dict(record.get("skill_info_json")) + creator_id = str(snapshot.get("created_by") or "").strip() + creator = get_user_tenant_by_user_id(creator_id) if creator_id else None + author = str((creator or {}).get("user_email") or "").strip() or None detail = { "skill_repository_id": record.get("skill_repository_id"), "skill_id": record.get("skill_id"), "name": record.get("name"), "description": record.get("description"), "source": record.get("source"), + "author": author, "submitted_by": record.get("submitted_by"), "icon": record.get("icon"), "status": record.get("status"), @@ -133,7 +156,7 @@ def _to_detail_item( "downloads": record.get("downloads") or 0, "created_at": _serialize_created_at(record.get("create_time")), "updated_at": _serialize_created_at(record.get("update_time")), - "content": snapshot.get("content"), + "content": record.get("content"), "config_schemas": _as_dict(snapshot.get("config_schemas")), "config_values": _as_dict(snapshot.get("config_values")), "tool_ids": _as_list(snapshot.get("tool_ids")), @@ -158,18 +181,21 @@ def _to_repository_info_item(record: Dict[str, Any]) -> Dict[str, Any]: return { "skill_repository_id": record.get("skill_repository_id"), "status": record.get("status"), + "content": record.get("content"), "create_time": _serialize_created_at(record.get("create_time")), } def _matches_ownership(skill: Dict[str, Any], user_id: str, ownership_filter: str) -> bool: """Return whether a skill belongs to the requested ownership bucket.""" - created_by = skill.get("created_by") - if ownership_filter in (OWNERSHIP_ALL, OWNERSHIP_CREATED): - return created_by == user_id + if ownership_filter == OWNERSHIP_ALL: + return True + is_creator = str(skill.get("created_by")) == str(user_id) + if ownership_filter == OWNERSHIP_CREATED: + return is_creator if ownership_filter == OWNERSHIP_OTHERS: - return False - return created_by == user_id + return not is_creator + return True def _matches_search(skill: Dict[str, Any], search: Optional[str]) -> bool: @@ -190,11 +216,16 @@ def _matches_search(skill: Dict[str, Any], search: Optional[str]) -> bool: def _count_skills_by_ownership(skills: List[Dict[str, Any]], user_id: str) -> Dict[str, int]: """Count editable skills in each ownership bucket.""" - created = sum(1 for skill in skills if skill.get("created_by") == user_id) + created = sum( + 1 + for skill in skills + if str(skill.get("created_by")) == str(user_id) + ) + others = len(skills) - created return { - OWNERSHIP_ALL: created, + OWNERSHIP_ALL: len(skills), OWNERSHIP_CREATED: created, - OWNERSHIP_OTHERS: 0, + OWNERSHIP_OTHERS: others, } @@ -229,16 +260,26 @@ def _get_user_role(user_id: str) -> str: return str(user_tenant.get("user_role") or "USER") -def _resolve_mine_skill_permission( +def ensure_skill_repository_access(user_id: str) -> None: + """Reject ordinary users from the Skill Repository API surface.""" + user_role = _get_user_role(user_id).upper() + if user_role == "USER": + raise ForbiddenError("User role USER is not authorized to access Skill Repository") + + +def _can_publish_skill( *, skill: Dict[str, Any], user_id: str, user_role: str, -) -> str: - """Resolve list-item permission for skill repository mine view.""" - if user_role in CAN_EDIT_ALL_USER_ROLES: - return PERMISSION_EDIT - return PERMISSION_EDIT if skill.get("created_by") == user_id else PERMISSION_READ +) -> bool: + """Return whether the user may submit the skill to the repository.""" + if user_role == "ADMIN": + return True + return ( + user_role == "DEV" + and str(skill.get("created_by")) == str(user_id) + ) def _resolve_submitter_email(user_id: str) -> Optional[str]: @@ -255,9 +296,11 @@ def _validate_create_listing_permission( ) -> None: """Only ADMIN, or DEV who created the skill, may share to marketplace.""" user_role = _get_user_role(user_id) - if user_role == "ADMIN": - return - if user_role == "DEV" and skill_info.get("created_by") == user_id: + if _can_publish_skill( + skill=skill_info, + user_id=user_id, + user_role=user_role, + ): return raise ForbiddenError( f"User role {user_role} not authorized to create repository listing" @@ -322,6 +365,8 @@ def _build_skill_info_json(skill_info: Dict[str, Any]) -> Dict[str, Any]: "config_schemas": skill_info.get("config_schemas"), "config_values": skill_info.get("config_values"), "source": skill_info.get("source"), + "group_ids": skill_info.get("group_ids") or [], + "ingroup_permission": skill_info.get("ingroup_permission"), "tool_ids": skill_info.get("tool_ids") or [], "created_by": skill_info.get("created_by"), } @@ -377,15 +422,58 @@ def _build_repository_data_from_skill( } if card_fields: - for key in ("icon", "downloads", "category_id"): + for key in ("icon", "downloads", "category_id", "content"): if key in card_fields and card_fields[key] is not None: repository_data[key] = card_fields[key] if "tags" in card_fields and card_fields["tags"] is not None: repository_data["tags"] = card_fields["tags"] + repository_data["content"] = (card_fields or {}).get("content") or "" return repository_data +def _find_resubmittable_repository_record( + skill_id: int, + tenant_id: str, +) -> Optional[Dict[str, Any]]: + """Find an existing review draft that can be refreshed by a new submission.""" + pending = get_skill_repository_by_skill_id( + skill_id, + publisher_tenant_id=tenant_id, + statuses=[STATUS_PENDING_REVIEW], + ) + if pending: + return pending + return get_skill_repository_by_skill_id( + skill_id, + publisher_tenant_id=tenant_id, + statuses=[STATUS_REJECTED], + ) + + +def _reset_repository_peer_statuses( + *, + skill_repository_id: int, + skill_id: int, + status: str, + publisher_tenant_id: str, +) -> None: + """Reset peer listings with the same status; also clear rejected when submitting.""" + reset_skill_repository_status( + repository_id=skill_repository_id, + skill_id=skill_id, + status=status, + publisher_tenant_id=publisher_tenant_id, + ) + if status == STATUS_PENDING_REVIEW: + reset_skill_repository_status( + repository_id=skill_repository_id, + skill_id=skill_id, + status=STATUS_REJECTED, + publisher_tenant_id=publisher_tenant_id, + ) + + def _validate_create_payload(repository_data: Dict[str, Any]) -> None: """Validate required fields before inserting a repository listing.""" required_fields = ( @@ -426,9 +514,12 @@ def create_skill_repository_listing_impl( ) _validate_create_payload(repository_data) - existing = get_skill_repository_by_skill_id( + existing = _find_resubmittable_repository_record( skill_id, - publisher_tenant_id=tenant_id, + tenant_id, + ) + was_pending_review = bool( + existing and existing.get("status") == STATUS_PENDING_REVIEW ) if not existing: repository_id = insert_skill_repository_record( @@ -454,12 +545,32 @@ def create_skill_repository_listing_impl( raise ValueError("Failed to update repository listing") is_updated = True + _reset_repository_peer_statuses( + skill_repository_id=repository_id, + skill_id=skill_id, + status=STATUS_PENDING_REVIEW, + publisher_tenant_id=tenant_id, + ) + record = get_skill_repository_by_id_and_publisher( repository_id, tenant_id, ) if not record: raise ValueError("Failed to load repository listing after write") + if not was_pending_review: + create_repository_pending_review_notification( + resource_type=RESOURCE_TYPE_SKILL_REPOSITORY, + tenant_id=tenant_id, + unique_id=repository_id, + details={ + "name": record.get("name"), + "skill_repository_id": repository_id, + "skill_id": record.get("skill_id"), + "content": record.get("content") or "", + }, + created_by=user_id, + ) return _to_detail_item(record, is_updated=is_updated) @@ -486,8 +597,11 @@ def _validate_publisher_status_transition( ) -> Optional[Dict[str, str]]: if record.get("publisher_tenant_id") != tenant_id: raise ForbiddenError("Not authorized to update this repository listing") - if user_role == "DEV" and record.get("publisher_user_id") != user_id: - raise ForbiddenError("Not authorized to update this repository listing") + if user_role == "DEV": + snapshot = _as_dict(record.get("skill_info_json")) + owner_user_id = snapshot.get("created_by") or record.get("publisher_user_id") + if str(owner_user_id) != str(user_id): + raise ForbiddenError("Not authorized to update this repository listing") if user_role == "ADMIN" and transition in _ADMIN_REVIEW_STATUS_TRANSITIONS: return None if transition not in _PUBLISHER_STATUS_TRANSITIONS: @@ -540,6 +654,7 @@ def update_skill_repository_status_impl( status: str, user_id: str, tenant_id: str, + content: Optional[str] = None, ) -> Dict[str, Any]: """Update a skill repository listing status by primary key.""" if status not in VALID_REPOSITORY_STATUSES: @@ -587,19 +702,74 @@ def update_skill_repository_status_impl( else None ), submitted_by=submitted_by, + content=content, ) if rows_affected == 0: raise ValueError(_REPOSITORY_LISTING_NOT_FOUND) + _reset_repository_peer_statuses( + skill_repository_id=skill_repository_id, + skill_id=record["skill_id"], + status=status, + publisher_tenant_id=tenant_id, + ) + updated = get_skill_repository_by_id_and_publisher( skill_repository_id, tenant_id, ) if not updated: raise ValueError("Failed to load repository listing after update") + + _handle_review_status_notifications( + current_status=current_status, + new_status=status, + updated=updated, + skill_repository_id=skill_repository_id, + user_id=user_id, + content=content, + ) + return _to_summary_item(updated) +def _handle_review_status_notifications( + *, + current_status: str, + new_status: str, + updated: Dict[str, Any], + skill_repository_id: int, + user_id: str, + content: Optional[str] = None, +) -> None: + """Send review-result notification and deactivate pending-review notification.""" + if current_status != new_status and new_status in (STATUS_SHARED, STATUS_REJECTED): + details: Dict[str, Any] = { + "name": updated.get("name"), + "skill_repository_id": skill_repository_id, + "skill_id": updated.get("skill_id"), + } + if content: + details["content"] = content + create_repository_review_notification( + resource_type=RESOURCE_TYPE_SKILL_REPOSITORY, + review_status=new_status, + receiver_user_id=updated["publisher_user_id"], + details=details, + tenant_id=updated.get("publisher_tenant_id"), + unique_id=skill_repository_id, + created_by=user_id, + ) + + if current_status == STATUS_PENDING_REVIEW: + deactivate_notifications( + event_type=EVENT_TYPE_REPOSITORY_REVIEW_PENDING, + resource_type=RESOURCE_TYPE_SKILL_REPOSITORY, + unique_id=skill_repository_id, + updated_by=user_id, + ) + + def _extract_duplicate_skill_name(error_message: str) -> Optional[str]: """Extract duplicate skill name from existing SkillException messages.""" match = re.search(r"Skill '([^']+)' already exists", error_message) @@ -669,6 +839,10 @@ def install_skill_from_repository_impl( ) if not copy_skill_name: raise ValueError("Skill name is required") + if len(copy_skill_name) > _MAX_COPY_NAME_LENGTH: + raise ValueError( + f"Skill name must be at most {_MAX_COPY_NAME_LENGTH} characters" + ) try: created_skill = SkillService(tenant_id=tenant_id).create_skill_from_zip_bytes( @@ -677,6 +851,7 @@ def install_skill_from_repository_impl( source="repository", user_id=user_id, tenant_id=tenant_id, + ingroup_permission=PERMISSION_READ, ) except SkillException as exc: message = str(exc) @@ -755,10 +930,13 @@ def _to_mine_skill_item( "description": skill.get("description"), "source": skill.get("source"), "tags": skill.get("tags") or [], + "group_ids": skill.get("group_ids") or [], + "ingroup_permission": skill.get("ingroup_permission"), "created_by": skill.get("created_by"), "created_at": skill.get("create_time"), "updated_at": skill.get("update_time"), - "permission": _resolve_mine_skill_permission( + "permission": skill.get("permission"), + "can_publish": _can_publish_skill( skill=skill, user_id=user_id, user_role=user_role, @@ -789,7 +967,10 @@ def list_my_editable_skills_impl( safe_page_size = max(int(page_size or 10), 1) user_role = _get_user_role(user_id) - skills = SkillService(tenant_id=tenant_id).list_skills(tenant_id=tenant_id) + skills = SkillService(tenant_id=tenant_id).list_visible_skills( + tenant_id=tenant_id, + user_id=user_id, + ) counts = _count_skills_by_ownership(skills, user_id) filtered_skills = [ @@ -835,9 +1016,25 @@ def list_my_editable_skills_impl( } +def count_my_editable_skills_impl( + *, + tenant_id: str, + user_id: str, +) -> Dict[str, Any]: + """Count visible skills without loading content, tool relations, or YAML files.""" + skills = SkillService( + tenant_id=tenant_id + ).list_visible_skill_permission_summaries( + tenant_id=tenant_id, + user_id=user_id, + ) + return {"counts": _count_skills_by_ownership(skills, user_id)} + + def list_skill_repository_listings_impl( tenant_id: str, *, + user_id: str, status: Optional[str] = None, skill_id: Optional[int] = None, category_id: Optional[int] = None, @@ -863,8 +1060,24 @@ def list_skill_repository_listings_impl( search=search, sort_by_update_time=sort_by_update_time, ) + user_role = _get_user_role(user_id) return { - "items": [_to_summary_item(record) for record in result.get("items", [])], + "items": [ + _to_summary_item( + record, + can_take_down=( + record.get("status") == STATUS_SHARED + and ( + user_role in ("ADMIN", "SU") + or ( + user_role == "DEV" + and str(record.get("publisher_user_id")) == str(user_id) + ) + ) + ), + ) + for record in result.get("items", []) + ], "pagination": result.get("pagination"), } diff --git a/backend/services/skill_service.py b/backend/services/skill_service.py index 4622208294..13c866f0ca 100644 --- a/backend/services/skill_service.py +++ b/backend/services/skill_service.py @@ -8,6 +8,7 @@ import io import json import logging +import ntpath import os import uuid import zipfile @@ -21,18 +22,157 @@ from nexent.skills.skill_loader import SkillLoader from nexent.core.utils.observer import MessageObserver from nexent.core.agents.agent_model import ModelConfig -from consts.const import CONTAINER_SKILLS_PATH, OFFICIAL_SKILLS_ZIP_PATH, ROOT_DIR +from consts.const import ( + CAN_EDIT_ALL_USER_ROLES, + CONTAINER_SKILLS_PATH, + OFFICIAL_SKILLS_ZIP_PATH, + PERMISSION_EDIT, + PERMISSION_PRIVATE, + PERMISSION_READ, + ROOT_DIR, +) from consts.exceptions import ForbiddenError, SkillException from database import skill_db +from database.group_db import query_group_ids_by_user +from database.user_tenant_db import get_user_tenant_by_user_id from agents.skill_creation_agent import create_skill_from_request from utils.prompt_template_utils import get_skill_creation_simple_prompt_template from utils.content_classifier_utils import ContentClassifier +from utils.str_utils import convert_list_to_string logger = logging.getLogger(__name__) +_SKILL_UPDATE_FORBIDDEN_MESSAGE = "Not authorized to update this skill" +_SKILL_ACCESS_UPDATE_FORBIDDEN_MESSAGE = "Not authorized to update skill access" _skill_manager: Optional[SkillManager] = None +def _to_group_id_set(group_ids: Any) -> set[int]: + if isinstance(group_ids, str): + return { + int(group_id.strip()) + for group_id in group_ids.split(",") + if group_id.strip().isdigit() + } + if isinstance(group_ids, list): + return { + int(group_id) + for group_id in group_ids + if str(group_id).strip().isdigit() + } + return set() + + +def can_view_skill( + *, + skill: Dict[str, Any], + user_id: str, + user_role: str, + user_group_ids: set[int], +) -> bool: + """Return whether a skill is available to the current user.""" + if user_role in CAN_EDIT_ALL_USER_ROLES: + return True + if str(skill.get("created_by")) == str(user_id): + return True + if skill.get("ingroup_permission") == PERMISSION_PRIVATE: + return False + return bool( + user_group_ids.intersection(_to_group_id_set(skill.get("group_ids"))) + ) + + +def resolve_skill_permission( + *, + skill: Dict[str, Any], + user_id: str, + user_role: str, + user_group_ids: set[int], +) -> str: + """Resolve whether the current user can edit or only use a visible skill.""" + if user_role in CAN_EDIT_ALL_USER_ROLES: + return PERMISSION_EDIT + if str(skill.get("created_by")) == str(user_id): + return PERMISSION_EDIT + if skill.get("ingroup_permission") != PERMISSION_EDIT: + return PERMISSION_READ + return ( + PERMISSION_EDIT + if user_group_ids.intersection(_to_group_id_set(skill.get("group_ids"))) + else PERMISSION_READ + ) + + +def _apply_default_skill_permission_fields( + skill_data: Dict[str, Any], + user_id: Optional[str], +) -> None: + """Default user-created skills to the creator's groups with edit permission.""" + if not user_id: + return + if skill_data.get("group_ids") is None: + skill_data["group_ids"] = convert_list_to_string(query_group_ids_by_user(user_id)) + if not skill_data.get("ingroup_permission"): + skill_data["ingroup_permission"] = PERMISSION_EDIT + + +def _get_user_role(user_id: Optional[str]) -> str: + if not user_id: + return "USER" + user_tenant = get_user_tenant_by_user_id(user_id) + if not user_tenant: + return "USER" + return str(user_tenant.get("user_role") or "USER") + + +def _can_edit_skill(skill: Dict[str, Any], user_id: Optional[str]) -> bool: + if not user_id: + return False + user_role = _get_user_role(user_id) + user_group_ids = set(query_group_ids_by_user(user_id) or []) + return resolve_skill_permission( + skill=skill, + user_id=user_id, + user_role=user_role, + user_group_ids=user_group_ids, + ) == PERMISSION_EDIT + + +def _can_manage_skill_access(skill: Dict[str, Any], user_id: Optional[str]) -> bool: + if not user_id: + return False + return ( + _get_user_role(user_id) in CAN_EDIT_ALL_USER_ROLES + or str(skill.get("created_by")) == str(user_id) + ) + + +def _has_skill_access_changes( + existing: Dict[str, Any], skill_data: Dict[str, Any] +) -> bool: + if ( + "group_ids" in skill_data + and _to_group_id_set(skill_data.get("group_ids")) + != _to_group_id_set(existing.get("group_ids")) + ): + return True + return ( + "ingroup_permission" in skill_data + and skill_data.get("ingroup_permission") != existing.get("ingroup_permission") + ) + + +def _validate_skill_access_update( + existing: Dict[str, Any], skill_data: Dict[str, Any], user_id: Optional[str] +) -> None: + if ( + user_id + and _has_skill_access_changes(existing, skill_data) + and not _can_manage_skill_access(existing, user_id) + ): + raise ForbiddenError(_SKILL_ACCESS_UPDATE_FORBIDDEN_MESSAGE) + + def _normalize_zip_entry_path(name: str) -> str: """Normalize a ZIP member path for comparison (slashes, strip ./).""" norm = name.replace("\\", "/").strip() @@ -792,28 +932,48 @@ def _resolve_local_skill_path( or "\\" in name or "\x00" in name or os.path.basename(name) != name + or os.path.isabs(name) + or ntpath.isabs(name) + or bool(ntpath.splitdrive(name)[0]) ): raise SkillException("Invalid skill name for local file access") - allowed_root = os.path.realpath(CONTAINER_SKILLS_PATH) + normalized_parts: List[str] = [] + for part in parts: + raw_part = str(part or "") + if "\x00" in raw_part: + raise ForbiddenError("Unsafe local skill path") + if ( + os.path.isabs(raw_part) + or ntpath.isabs(raw_part) + or bool(ntpath.splitdrive(raw_part)[0]) + ): + raise ForbiddenError("Unsafe local skill path") + + path_segments = raw_part.replace("\\", "/").split("/") + if any(segment == ".." for segment in path_segments): + raise ForbiddenError("Unsafe local skill path") + normalized_parts.extend( + segment for segment in path_segments if segment not in {"", "."} + ) + local_root = os.path.realpath(local_skills_dir) - if ( - local_root != allowed_root - and not local_root.startswith(allowed_root + os.sep) - ): - raise SkillException("Unsafe local skills directory") + skill_root = os.path.realpath(os.path.join(local_root, name)) + candidate = os.path.realpath(os.path.join(skill_root, *normalized_parts)) - candidate = os.path.realpath(os.path.join(local_root, name, *parts)) - if ( - candidate != allowed_root - and not candidate.startswith(allowed_root + os.sep) - ): - raise SkillException("Unsafe local skill path") - if ( - candidate != local_root - and not candidate.startswith(local_root + os.sep) - ): - raise SkillException("Unsafe local skill path") + def _is_within(root: str, path: str) -> bool: + try: + return os.path.normcase(os.path.commonpath([root, path])) == os.path.normcase(root) + except ValueError: + return False + + if CONTAINER_SKILLS_PATH: + allowed_root = os.path.realpath(CONTAINER_SKILLS_PATH) + if not _is_within(allowed_root, local_root): + raise SkillException("Unsafe local skills directory") + + if not _is_within(local_root, skill_root) or not _is_within(skill_root, candidate): + raise ForbiddenError("Unsafe local skill path") return candidate @@ -846,14 +1006,9 @@ def _remove_local_skill_config_yaml(skill_name: str, local_skills_dir: str) -> N logger.info("Removed %s (params cleared in DB)", path) -def get_skill_manager(tenant_id: Optional[str] = None) -> SkillManager: - """Create a SkillManager instance with optional tenant-based directory isolation. - - Args: - tenant_id: Tenant ID for directory isolation. When provided, skills - are stored under CONTAINER_SKILLS_PATH / tenant_id / - """ - return SkillManager(base_skills_dir=CONTAINER_SKILLS_PATH, tenant_id=tenant_id) +def get_skill_manager() -> SkillManager: + """Return the process-wide SkillManager.""" + return SkillManager(base_skills_dir=CONTAINER_SKILLS_PATH) class SkillService: @@ -867,16 +1022,16 @@ def __init__(self, skill_manager: Optional[SkillManager] = None, tenant_id: Opti tenant_id: Tenant ID for skill isolation. Required when no skill_manager is provided. """ self.tenant_id = tenant_id - self.skill_manager = skill_manager or get_skill_manager(tenant_id) + self.skill_manager = skill_manager or get_skill_manager() + + def _local_skills_dir(self, tenant_id: Optional[str] = None) -> str: + """Resolve the local directory for an explicit or service-bound tenant.""" + effective_tenant_id = tenant_id if tenant_id is not None else self.tenant_id + return self.skill_manager.resolve_tenant_dir(tenant_id=effective_tenant_id) def _resolve_local_skills_dir_for_overlay(self) -> Optional[str]: """Directory where skill folders live: ``SKILLS_PATH``, else ``ROOT_DIR/skills`` if present.""" - manager_dir = getattr(self.skill_manager, "local_skills_dir", None) - d = ( - manager_dir - if isinstance(manager_dir, str) - else CONTAINER_SKILLS_PATH - ) + d = self._local_skills_dir() if d: return str(d).rstrip(os.sep) or None if ROOT_DIR: @@ -946,6 +1101,60 @@ def list_skills(self, tenant_id: Optional[str] = None) -> List[Dict[str, Any]]: logger.error(f"Error listing skills: {e}") raise SkillException(f"Failed to list skills: {str(e)}") from e + def list_visible_skills( + self, + *, + tenant_id: Optional[str] = None, + user_id: str, + ) -> List[Dict[str, Any]]: + """List skills visible to a user and attach the resolved permission.""" + user_role = _get_user_role(user_id) + user_group_ids = set(query_group_ids_by_user(user_id) or []) + visible_skills = [ + skill + for skill in self.list_skills(tenant_id=tenant_id) + if can_view_skill( + skill=skill, + user_id=user_id, + user_role=user_role, + user_group_ids=user_group_ids, + ) + ] + for skill in visible_skills: + skill["permission"] = resolve_skill_permission( + skill=skill, + user_id=user_id, + user_role=user_role, + user_group_ids=user_group_ids, + ) + return visible_skills + + def list_visible_skill_permission_summaries( + self, + *, + tenant_id: Optional[str] = None, + user_id: str, + ) -> List[Dict[str, Any]]: + """List lightweight visible-skill fields used by repository counts.""" + effective_tenant_id = tenant_id or self.tenant_id + if not effective_tenant_id: + raise SkillException("tenant_id is required") + + user_role = _get_user_role(user_id) + user_group_ids = set(query_group_ids_by_user(user_id) or []) + return [ + skill + for skill in skill_db.list_skill_permission_summaries( + effective_tenant_id + ) + if can_view_skill( + skill=skill, + user_id=user_id, + user_role=user_role, + user_group_ids=user_group_ids, + ) + ] + def get_skill(self, skill_name: str, tenant_id: Optional[str] = None) -> Optional[Dict[str, Any]]: """Get a specific skill within a tenant. @@ -1033,13 +1242,14 @@ def create_skill( if user_id: skill_data["created_by"] = user_id skill_data["updated_by"] = user_id + _apply_default_skill_permission_fields(skill_data, user_id) try: # Create database record first result = skill_db.create_skill(skill_data, effective_tenant_id) # Create local skill file (SKILL.md) - self.skill_manager.save_skill(skill_data) + self.skill_manager.save_skill(skill_data, tenant_id=effective_tenant_id) # Mirror DB config_schemas to config/config.yaml when present (same layout as ZIP uploads). if self.skill_manager.base_skills_dir and skill_data.get("config_schemas") is not None: @@ -1047,7 +1257,7 @@ def create_skill( _write_skill_params_to_local_config_yaml( skill_name, _params_dict_to_storable(skill_data["config_schemas"]), - self.skill_manager.local_skills_dir, + self._local_skills_dir(effective_tenant_id), ) except Exception as exc: logger.warning( @@ -1091,6 +1301,7 @@ def create_skill_from_file( Created skill dict """ effective_tenant_id = tenant_id or self.tenant_id + content_bytes: bytes if isinstance(file_content, str): content_bytes = file_content.encode("utf-8") @@ -1160,11 +1371,12 @@ def _create_skill_from_md( if user_id: skill_dict["created_by"] = user_id skill_dict["updated_by"] = user_id + _apply_default_skill_permission_fields(skill_dict, user_id) result = skill_db.create_skill(skill_dict, tenant_id) # Write SKILL.md to local storage - self.skill_manager.save_skill(skill_dict) + self.skill_manager.save_skill(skill_dict, tenant_id=tenant_id) return self._enrich_configs_from_yaml(result) @@ -1293,17 +1505,20 @@ def _create_skill_from_zip( if user_id: skill_dict["created_by"] = user_id skill_dict["updated_by"] = user_id + _apply_default_skill_permission_fields(skill_dict, user_id) result = skill_db.create_skill(skill_dict, tenant_id) # Save SKILL.md to local storage - self.skill_manager.save_skill(skill_dict) + self.skill_manager.save_skill(skill_dict, tenant_id=tenant_id) - self._upload_zip_files(zip_bytes, name, detected_skill_name) + self._upload_zip_files( + zip_bytes, name, detected_skill_name, tenant_id=tenant_id + ) return self._enrich_configs_from_yaml(result) - def _delete_local_skill_files(self, skill_name: str) -> None: + def _delete_local_skill_files(self, skill_name: str, *, tenant_id: Optional[str]) -> None: """Delete all files within a skill's local directory, preserving the directory itself. Args: @@ -1311,7 +1526,7 @@ def _delete_local_skill_files(self, skill_name: str) -> None: """ import shutil - local_dir = os.path.join(self.skill_manager.local_skills_dir, skill_name) + local_dir = os.path.join(self._local_skills_dir(tenant_id), skill_name) logger.info("Starting deletion of local files for skill '%s' from '%s'", skill_name, local_dir) if not os.path.isdir(local_dir): @@ -1339,7 +1554,9 @@ def _upload_zip_files( self, zip_bytes: bytes, skill_name: str, - original_folder_name: Optional[str] = None + original_folder_name: Optional[str] = None, + *, + tenant_id: Optional[str], ) -> None: """Extract ZIP files to local storage only. @@ -1382,7 +1599,7 @@ def _upload_zip_files( with zipfile.ZipFile(zip_stream, "r") as zf: logger.info("ZIP contains %d entries for skill '%s'", len(file_list), skill_name) - extracted_count = 0 + validated_files: List[Tuple[str, str]] = [] for file_path in file_list: if file_path.endswith("/"): continue @@ -1404,11 +1621,16 @@ def _upload_zip_files( if not relative_path: continue - file_data = zf.read(file_path) + local_path = _resolve_local_skill_path( + self._local_skills_dir(tenant_id), + skill_name, + relative_path, + ) + validated_files.append((file_path, local_path)) - local_dir = os.path.join(self.skill_manager.local_skills_dir, skill_name) - normalized_relative = relative_path.replace("/", os.sep).replace("\\", os.sep) - local_path = os.path.normpath(os.path.join(local_dir, normalized_relative)) + extracted_count = 0 + for file_path, local_path in validated_files: + file_data = zf.read(file_path) os.makedirs(os.path.dirname(local_path), exist_ok=True) with open(local_path, "wb") as f: f.write(file_data) @@ -1417,8 +1639,11 @@ def _upload_zip_files( logger.info( "Completed ZIP extraction for skill '%s': %d files extracted to '%s'", - skill_name, extracted_count, self.skill_manager.local_skills_dir + skill_name, extracted_count, self._local_skills_dir(tenant_id) ) + except ForbiddenError: + logger.warning("Rejected unsafe ZIP path for skill '%s'", skill_name) + raise except Exception as e: logger.error("Failed to extract ZIP files for skill '%s': %s", skill_name, e) raise @@ -1449,6 +1674,8 @@ def update_skill_from_file( existing = skill_db.get_skill_by_name(skill_name, effective_tenant_id) if not existing: raise SkillException(f"Skill not found: {skill_name}") + if user_id is not None and not _can_edit_skill(existing, user_id): + raise ForbiddenError(_SKILL_UPDATE_FORBIDDEN_MESSAGE) content_bytes: bytes if isinstance(file_content, str): @@ -1502,12 +1729,12 @@ def _update_skill_from_md( ) # Clean up existing local files before writing new ones - self._delete_local_skill_files(skill_name) + self._delete_local_skill_files(skill_name, tenant_id=tenant_id) # Update local storage with new SKILL.md (preserve allowed-tools) skill_dict["name"] = skill_name skill_dict["allowed-tools"] = allowed_tools - self.skill_manager.save_skill(skill_dict) + self.skill_manager.save_skill(skill_dict, tenant_id=tenant_id) return self._enrich_configs_from_yaml(result) @@ -1582,15 +1809,17 @@ def _update_skill_from_zip( ) # Clean up existing local files before writing new ones - self._delete_local_skill_files(skill_name) + self._delete_local_skill_files(skill_name, tenant_id=tenant_id) # Update SKILL.md in local storage (preserve allowed-tools) skill_dict["name"] = skill_name skill_dict["allowed-tools"] = allowed_tools - self.skill_manager.save_skill(skill_dict) + self.skill_manager.save_skill(skill_dict, tenant_id=tenant_id) # Update other files in local storage - self._upload_zip_files(zip_bytes, skill_name, original_folder_name) + self._upload_zip_files( + zip_bytes, skill_name, original_folder_name, tenant_id=tenant_id + ) return self._enrich_configs_from_yaml(result) @@ -1619,13 +1848,16 @@ def update_skill( existing = skill_db.get_skill_by_name(skill_name, effective_tenant_id) if not existing: raise SkillException(f"Skill not found: {skill_name}") + if user_id is not None and not _can_edit_skill(existing, user_id): + raise ForbiddenError(_SKILL_UPDATE_FORBIDDEN_MESSAGE) + _validate_skill_access_update(existing, skill_data, user_id) result = skill_db.update_skill( skill_name, skill_data, effective_tenant_id, updated_by=user_id or None ) # Keep config/config.yaml in sync when config_values are updated (matches ZIP import path). - local_dir = self.skill_manager.local_skills_dir or CONTAINER_SKILLS_PATH + local_dir = self._local_skills_dir(effective_tenant_id) if local_dir and "config_values" in skill_data: try: raw_config_values = skill_data["config_values"] @@ -1662,7 +1894,7 @@ def update_skill( "allowed-tools": allowed_tools, "files": skill_data.get("files", []), } - self.skill_manager.save_skill(local_skill_dict) + self.skill_manager.save_skill(local_skill_dict, tenant_id=effective_tenant_id) except Exception as exc: logger.warning( "Local SKILL.md sync failed after DB update for %s: %s", @@ -1671,7 +1903,7 @@ def update_skill( ) return self._enrich_configs_from_yaml(result) - except SkillException: + except (ForbiddenError, SkillException): raise except Exception as e: logger.error(f"Error updating skill {skill_name}: {e}") @@ -1692,8 +1924,9 @@ def update_skill_by_id( existing = skill_db.get_skill_by_id(skill_id, effective_tenant_id) if not existing: raise SkillException(f"Skill not found: {skill_id}") - if not user_id or existing.get("created_by") != user_id: - raise ForbiddenError("Not authorized to update this skill") + if not _can_edit_skill(existing, user_id): + raise ForbiddenError(_SKILL_UPDATE_FORBIDDEN_MESSAGE) + _validate_skill_access_update(existing, skill_data, user_id) local_dir = self._resolve_local_skills_dir_for_overlay() if local_dir and "name" in skill_data: @@ -1759,14 +1992,14 @@ def update_skill_by_id( "allowed-tools": allowed_tools, "files": skill_data.get("files", []), } - self.skill_manager.save_skill(local_skill_dict) + self.skill_manager.save_skill(local_skill_dict, tenant_id=effective_tenant_id) previous_name = str(existing.get("name") or "").strip() if ( local_skill_name != f"skill_{skill_id}" and previous_name and previous_name != local_skill_name ): - self.skill_manager.delete_skill(previous_name) + self.skill_manager.delete_skill(previous_name, tenant_id=effective_tenant_id) except Exception as exc: logger.warning( "Local SKILL.md sync failed after DB update for skill ID %s: %s", @@ -1802,7 +2035,7 @@ def delete_skill( raise SkillException("tenant_id is required") try: # Delete local skill files from filesystem - skill_dir = os.path.join(self.skill_manager.local_skills_dir, skill_name) + skill_dir = os.path.join(self._local_skills_dir(effective_tenant_id), skill_name) if os.path.exists(skill_dir): import shutil shutil.rmtree(skill_dir) @@ -1843,6 +2076,8 @@ def get_enabled_skills_for_agent( skill_id = skill_instance.get("skill_id") skill = skill_db.get_skill_by_id(skill_id, tenant_id) if skill: + effective_config_values = dict(skill.get("config_values") or {}) + effective_config_values.update(skill_instance.get("config_values") or {}) # Get skill info from ag_skill_info_t (repository returns keys: name, description, content) merged = { "skill_id": skill_id, @@ -1851,6 +2086,8 @@ def get_enabled_skills_for_agent( "content": skill.get("content", ""), "enabled": skill_instance.get("enabled", True), "tool_ids": skill.get("tool_ids", []), + "config_schemas": skill.get("config_schemas") or [], + "config_values": effective_config_values, } result.append(merged) @@ -1869,7 +2106,7 @@ def load_skill_directory(self, skill_name: str) -> Optional[Dict[str, Any]]: Dict with skill metadata and local directory path, or None if not found """ try: - return self.skill_manager.load_skill_directory(skill_name) + return self.skill_manager.load_skill_directory(skill_name, tenant_id=self.tenant_id) except Exception as e: logger.error(f"Error loading skill directory {skill_name}: {e}") raise SkillException(f"Failed to load skill directory: {str(e)}") from e @@ -1884,7 +2121,7 @@ def get_skill_scripts(self, skill_name: str) -> List[str]: List of script file paths """ try: - return self.skill_manager.get_skill_scripts(skill_name) + return self.skill_manager.get_skill_scripts(skill_name, tenant_id=self.tenant_id) except Exception as e: logger.error(f"Error getting skill scripts {skill_name}: {e}") raise SkillException(f"Failed to get skill scripts: {str(e)}") from e @@ -2002,7 +2239,10 @@ def get_skill_file_tree( Dict with file tree structure, or None if not found """ try: - return self.skill_manager.get_skill_file_tree(skill_name) + effective_tenant_id = tenant_id or self.tenant_id + return self.skill_manager.get_skill_file_tree( + skill_name, tenant_id=effective_tenant_id + ) except Exception as e: logger.error(f"Error getting skill file tree: {e}") raise SkillException(f"Failed to get skill file tree: {str(e)}") from e @@ -2024,16 +2264,29 @@ def get_skill_file_content( File content as string, or None if file not found """ try: - local_dir = os.path.join(self.skill_manager.local_skills_dir, skill_name) - normalized_file_path = file_path.replace("/", os.sep).replace("\\", os.sep) - full_path = os.path.normpath(os.path.join(local_dir, normalized_file_path)) + effective_tenant_id = tenant_id or self.tenant_id + local_skills_dir = self._local_skills_dir(effective_tenant_id) + full_path = _resolve_local_skill_path( + local_skills_dir, + skill_name, + file_path, + ) - if not os.path.exists(full_path): - logger.warning(f"File not found: {full_path}") - return None + # Keep the containment check next to the file access so static analysis and + # future callers can verify that user-controlled paths stay below the root. + local_root = os.path.realpath(local_skills_dir) + if not full_path.startswith(local_root + os.sep): + raise ForbiddenError("Unsafe local skill path") - with open(full_path, "r", encoding="utf-8") as f: - return f.read() + try: + with open(full_path, "r", encoding="utf-8") as f: + return f.read() + except FileNotFoundError: + logger.warning("Skill file not found: %s/%s", skill_name, file_path) + return None + except ForbiddenError: + logger.warning("Rejected unsafe file read for skill '%s'", skill_name) + raise except Exception as e: logger.error(f"Error reading skill file {skill_name}/{file_path}: {e}") raise SkillException(f"Failed to read skill file: {str(e)}") from e @@ -2122,7 +2375,8 @@ def create_skill_from_zip_bytes( source: str = "导入", user_id: Optional[str] = None, tenant_id: Optional[str] = None, - skip_duplicate_check: bool = False + skip_duplicate_check: bool = False, + ingroup_permission: Optional[str] = None, ) -> Dict[str, Any]: """Create a skill from ZIP bytes, optionally skipping the duplicate name check. @@ -2137,6 +2391,7 @@ def create_skill_from_zip_bytes( user_id: Creator user ID tenant_id: Tenant ID skip_duplicate_check: If True, skip the "skill already exists" check + ingroup_permission: Optional group permission override for the new skill Returns: Created skill dict @@ -2240,11 +2495,16 @@ def create_skill_from_zip_bytes( if user_id: skill_dict["created_by"] = user_id skill_dict["updated_by"] = user_id + if ingroup_permission is not None: + skill_dict["ingroup_permission"] = ingroup_permission + _apply_default_skill_permission_fields(skill_dict, user_id) result = skill_db.create_skill(skill_dict, tenant_id) - self.skill_manager.save_skill(skill_dict) - self._upload_zip_files(zip_bytes, name, detected_skill_name) + self.skill_manager.save_skill(skill_dict, tenant_id=tenant_id) + self._upload_zip_files( + zip_bytes, name, detected_skill_name, tenant_id=tenant_id + ) return self._enrich_configs_from_yaml(result) @@ -2272,7 +2532,7 @@ def export_skills_by_names( for skill_name in skill_names: skill_dir = os.path.join( - self.skill_manager.local_skills_dir or CONTAINER_SKILLS_PATH, + self._local_skills_dir(effective_tenant_id), skill_name ) if not os.path.isdir(skill_dir): @@ -2289,7 +2549,7 @@ def export_skills_by_names( "description": skill_info.get("description", ""), "content": skill_info.get("content", ""), "tags": skill_info.get("tags", []), - }) + }, tenant_id=effective_tenant_id) if not os.path.isdir(skill_dir): logger.warning(f"Failed to rebuild skill directory for export: {skill_name}") continue @@ -2346,7 +2606,9 @@ def get_skill_manager_local_dir(self) -> str: Returns: Local skills directory path """ - return self.skill_service.skill_manager.local_skills_dir or "" + return self.skill_service.skill_manager.resolve_tenant_dir( + tenant_id=self.skill_service.tenant_id + ) def create_classifier(self) -> "ContentClassifier": """Create a new ContentClassifier instance. @@ -2554,7 +2816,7 @@ async def generate(): classifier = ContentClassifier() # Get local skills directory - local_skills_dir = SkillService().skill_manager.local_skills_dir or "" + local_skills_dir = get_skill_manager().resolve_tenant_dir(tenant_id=None) def run_task(): create_skill_from_request( @@ -2634,10 +2896,10 @@ async def update_skill_list(tenant_id: str, user_id: str): from database import skill_db as skill_db_module from nexent.skills import SkillManager - skill_manager = SkillManager(base_skills_dir=CONTAINER_SKILLS_PATH, tenant_id=tenant_id) + skill_manager = get_skill_manager() # Use the resolved tenant-scoped local path for schema/config file reading - local_base = skill_manager.local_skills_dir or CONTAINER_SKILLS_PATH - scanned_skills = skill_manager.list_skills() + local_base = skill_manager.resolve_tenant_dir(tenant_id=tenant_id) + scanned_skills = skill_manager.list_skills(tenant_id=tenant_id) skills_to_upsert = [] for skill_info in scanned_skills: @@ -2653,7 +2915,7 @@ async def update_skill_list(tenant_id: str, user_id: str): } try: - full_skill = skill_manager.load_skill(skill_name) + full_skill = skill_manager.load_skill(skill_name, tenant_id=tenant_id) if full_skill: skill_data["content"] = full_skill.get("content", "") @@ -2895,12 +3157,9 @@ def get_official_skills_with_status( if existing: skill_id = existing.get("skill_id") is_installed = True - skill_manager = SkillManager( - base_skills_dir=CONTAINER_SKILLS_PATH, - tenant_id=tenant_id - ) + skill_manager = get_skill_manager() skill_dir = os.path.join( - skill_manager.local_skills_dir or CONTAINER_SKILLS_PATH or "", + skill_manager.resolve_tenant_dir(tenant_id=tenant_id), skill_name ) has_resources = os.path.isdir(skill_dir) diff --git a/backend/services/tenant_service.py b/backend/services/tenant_service.py index 6ed96a8490..be47b692e6 100644 --- a/backend/services/tenant_service.py +++ b/backend/services/tenant_service.py @@ -26,13 +26,27 @@ from database.remote_mcp_db import get_mcp_records_by_tenant, delete_mcp_record_by_name_and_url from database.invitation_db import query_invitations_by_tenant, remove_invitation from database.tool_db import delete_tools_by_agent_id -from consts.const import ASSET_OWNER_TENANT_ID, TENANT_NAME, TENANT_ID, DEFAULT_GROUP_ID, CONTAINER_SKILLS_PATH -from consts.exceptions import NotFoundException, ValidationError, UserRegistrationException +from consts.const import ( + ASSET_OWNER_TENANT_ID, + CONTAINER_SKILLS_PATH, + DEFAULT_GROUP_ID, + DEFAULT_TENANT_ID, + TENANT_ID, + TENANT_NAME, + IS_SPEED_MODE, +) +from consts.exceptions import ForbiddenError, NotFoundException, ValidationError, UserRegistrationException from services.skill_service import install_skills_from_zip_for_tenant logger = logging.getLogger(__name__) +def _is_displayable_tenant_id(tenant_id: Optional[str]) -> bool: + """Return whether a tenant id represents a real tenant in management views.""" + normalized_tenant_id = (tenant_id or "").strip() + return normalized_tenant_id not in {"", DEFAULT_TENANT_ID, ASSET_OWNER_TENANT_ID} + + def get_tenant_info(tenant_id: str) -> Dict[str, Any]: """ Get tenant information by tenant ID @@ -45,7 +59,7 @@ def get_tenant_info(tenant_id: str) -> Dict[str, Any]: Returns: Dict[str, Any]: Tenant information """ - if not tenant_id: + if not _is_displayable_tenant_id(tenant_id): return {} # Get tenant name @@ -69,6 +83,20 @@ def get_tenant_info(tenant_id: str) -> Dict[str, Any]: return tenant_info +def get_tenant_info_for_user( + tenant_id: str, + *, + requester_tenant_id: str, + requester_role: str, +) -> Dict[str, Any]: + """Get tenant information after enforcing tenant-scoped access.""" + role = (requester_role or "").upper() + is_speed_admin = IS_SPEED_MODE and role == "SPEED" + if role != "SU" and not is_speed_admin and tenant_id != requester_tenant_id: + raise ForbiddenError("Not authorized to access this tenant") + return get_tenant_info(tenant_id) + + def _ensure_tenant_name_config(tenant_id: str) -> bool: """ Ensure TENANT_NAME config exists for the tenant. @@ -139,10 +167,10 @@ def get_tenants_paginated(page: int = 1, page_size: int = 20) -> Dict[str, Any]: Returns: Dict[str, Any]: Dictionary containing paginated tenant data and pagination info """ - # Exclude virtual ASSET_OWNER tenant from admin tenant listings + # Exclude virtual/system tenants from admin tenant listings. all_tenant_ids = [ tid for tid in get_all_tenant_ids() - if tid != ASSET_OWNER_TENANT_ID + if _is_displayable_tenant_id(tid) ] total = len(all_tenant_ids) @@ -178,6 +206,19 @@ def get_tenants_paginated(page: int = 1, page_size: int = 20) -> Dict[str, Any]: } +def get_tenants_paginated_for_user( + page: int = 1, + page_size: int = 20, + *, + requester_role: str, +) -> Dict[str, Any]: + """List tenants for platform administrators only.""" + role = (requester_role or "").upper() + if role != "SU" and not (IS_SPEED_MODE and role == "SPEED"): + raise ForbiddenError("Only super administrators can list tenants") + return get_tenants_paginated(page=page, page_size=page_size) + + def create_tenant( tenant_name: str, created_by: Optional[str] = None, diff --git a/backend/services/tool_configuration_service.py b/backend/services/tool_configuration_service.py index e61273cb1f..0a7cba3830 100644 --- a/backend/services/tool_configuration_service.py +++ b/backend/services/tool_configuration_service.py @@ -12,7 +12,14 @@ from fastmcp.client.transports import SSETransport, StreamableHttpTransport from pydantic_core import PydanticUndefined -from consts.const import DATA_PROCESS_SERVICE, LOCAL_MCP_SERVER, MCP_MANAGEMENT_API +from consts.const import ( + AIDP_API_KEY, + AIDP_SERVER_URL, + AIDP_TENANT_ID, + DATA_PROCESS_SERVICE, + LOCAL_MCP_SERVER, + MCP_MANAGEMENT_API, +) from consts.exceptions import MCPConnectionError, NotFoundException, ToolExecutionException from consts.model import ToolInstanceInfoRequest, ToolInfo, ToolSourceEnum, ToolValidateRequest from consts.tool_labels import SYSTEM_MANAGED_TOOL_NAMES @@ -37,6 +44,7 @@ update_tool_table_from_scan_tool_list, ) from database.knowledge_db import get_knowledge_name_map_by_index_names +from database.user_tenant_db import get_user_email_map from mcpadapt.smolagents_adapter import _sanitize_function_name from services.file_management_service import get_llm_model, validate_urls_access from services.vectordatabase_service import get_embedding_model_by_index_name, get_rerank_model @@ -354,6 +362,36 @@ def update_tool_info_impl(tool_info: ToolInstanceInfoRequest, tenant_id: str, us Raises: ValueError: If database update fails """ + # v7.1: validate per-KB READ access for aidp_search so a tenant user + # cannot stash a forbidden kds_id in their tool config for later abuse. + if getattr(tool_info, "name", None) == "aidp_search": + params = tool_info.params or {} + kds_list = params.get("kds_list") or [] + # ``kds_list`` may arrive as a JSON-encoded string (the + # legacy storage shape); decode it so we can validate each entry. + if isinstance(kds_list, str): + import json + try: + kds_list = json.loads(kds_list) + except json.JSONDecodeError: + kds_list = [] + if kds_list: + try: + from ext_components.aidp.services import ( + aidp_permission_service as _aidp_perms, + ) + for _kds_id in kds_list: + _aidp_perms.require_permission( + kb_id=_kds_id, user_id=user_id, + tenant_id=tenant_id, required="READ", + ) + except Exception: + # Surface as ValidationError so the app layer returns 400. + from consts.exceptions import ValidationError + raise ValidationError( + f"aidp_search kds_list contains a KB the user cannot read" + ) from None + # Use version_no from request if provided, otherwise default to 0 version_no = getattr(tool_info, 'version_no', 0) tool_instance = create_or_update_tool_by_tool_info( @@ -406,6 +444,9 @@ async def get_tool_from_remote_mcp_server( tools = await client.list_tools() for tool in tools: + if isinstance(tool.meta, dict) and tool.meta.get("nexent_internal") is True: + continue + input_schema = { k: v for k, v in jsonref.replace_refs(tool.inputSchema).items() @@ -484,13 +525,21 @@ async def update_tool_list(tenant_id: str, user_id: str): except Exception as e: logger.error(f"failed to get all mcp tools, detail: {e}") # Don't block local/langchain tool update when MCP is unavailable. - # MCP tools will be marked as is_available=False in the DB, which - # is the correct state when the MCP server is unreachable. mcp_tools = [] + # Enabled MCP services are "intended to be available". Their tools keep their + # previous availability even when this scan fails to reach them, so a transient + # connection failure does not hide a healthy MCP's tools from the tool list. + enabled_mcp_names = { + str(record.get("mcp_name") or "") + for record in get_mcp_records_by_tenant(tenant_id=tenant_id) + if bool(record.get("enabled")) + } + update_tool_table_from_scan_tool_list(tenant_id=tenant_id, user_id=user_id, - tool_list=local_tools+mcp_tools+langchain_tools) + tool_list=local_tools+mcp_tools+langchain_tools, + enabled_mcp_names=enabled_mcp_names) async def list_all_tools(tenant_id: str, labels: Optional[List[str]] = None): @@ -502,6 +551,10 @@ async def list_all_tools(tenant_id: str, labels: Optional[List[str]] = None): else: tools_info = query_all_tools(tenant_id) + updated_by_email_map = get_user_email_map( + [tool.get("updated_by", "") for tool in tools_info] + ) + # Get description_zh from SDK for local tools (not persisted to DB) local_tool_descriptions = get_local_tools_description_zh() @@ -570,7 +623,8 @@ async def list_all_tools(tenant_id: str, labels: Optional[List[str]] = None): "inputs": inputs_str, "category": tool.get("category"), "labels": tool.get("labels", []), - "updated_by": tool.get("updated_by", "") + "updated_by": tool.get("updated_by", ""), + "updated_by_name": updated_by_email_map.get(tool.get("updated_by"), ""), } formatted_tools.append(formatted_tool) return formatted_tools @@ -796,7 +850,10 @@ def _validate_local_tool( # Build display_name to index_name mapping for LLM parameter conversion display_name_to_index_map = {} if index_names: - knowledge_name_map = get_knowledge_name_map_by_index_names(index_names) + knowledge_name_map = get_knowledge_name_map_by_index_names( + index_names, + tenant_id=tenant_id, + ) for idx_name, kb_name in knowledge_name_map.items(): display_name_to_index_map[kb_name] = idx_name @@ -839,6 +896,25 @@ def _validate_local_tool( filtered_params = {k: v for k, v in instantiation_params.items() if k not in ["observer", "rerank_model", "rerank"]} filtered_params["observer"] = None + if tool_name == "aidp_search": + # AIDP credentials are sourced from ``consts.const`` (i.e. the + # process environment). Inject them here exactly as + # ``create_agent_info`` does at runtime, so validation builds + # the same tool instance shape. Never trust client-submitted + # credentials. + if not AIDP_SERVER_URL: + raise ToolExecutionException( + "AIDP is not configured for this deployment: " + "set AIDP_SERVER_URL before testing aidp_search" + ) + if not AIDP_API_KEY: + raise ToolExecutionException( + "AIDP is not configured for this deployment: " + "set AIDP_API_KEY before testing aidp_search" + ) + filtered_params["server_url"] = AIDP_SERVER_URL + filtered_params["api_key"] = AIDP_API_KEY + filtered_params["tenant_id"] = AIDP_TENANT_ID tool_instance = tool_class(**filtered_params) elif tool_name == "analyze_image": if not tenant_id or not user_id: diff --git a/backend/services/user_management_service.py b/backend/services/user_management_service.py index 0b38a76bcb..00de73edbc 100644 --- a/backend/services/user_management_service.py +++ b/backend/services/user_management_service.py @@ -25,7 +25,6 @@ INVITE_CODE, SUPABASE_URL, SUPABASE_KEY, - DEFAULT_TENANT_ID, ASSET_OWNER_TENANT_ID, ASSET_OWNER_INVITE_CODE_TYPE, ASSET_OWNER_ROLE, @@ -57,7 +56,7 @@ from services.skill_service import init_skill_list_for_tenant -logging.getLogger("user_management_service").setLevel(logging.DEBUG) +logging.getLogger("user_management_service").setLevel(logging.INFO) def set_auth_token_to_client(client: Client, token: str) -> None: @@ -158,13 +157,17 @@ async def signup_user_with_invitation(email: EmailStr, invite_code: Optional[str] = None, auto_login: Optional[bool] = True): """User registration with invitation code support""" - client = get_supabase_client() + if not isinstance(invite_code, str) or not invite_code.strip(): + raise ValidationError("INVITE_CODE_REQUIRED") + invite_code = invite_code.strip().upper() # Validate password strength before registration if not validate_password_strength(password): raise AppException(ErrorCode.PROFILE_PASSWORD_WEAK, "Password must be at least 8 characters with uppercase, lowercase, and digit.") + client = get_supabase_client() + logging.info( f"Receive registration request: email={email}, invite_code={'provided' if invite_code else 'not provided'}, auto_login={auto_login}") @@ -175,9 +178,6 @@ async def signup_user_with_invitation(email: EmailStr, # Validate invitation code if provided (without using it yet) if invite_code: try: - # Convert invite code to upper case for consistency - invite_code = invite_code.upper() - # Check if invitation is available if not check_invitation_available(invite_code): raise IncorrectInviteCodeException( @@ -221,23 +221,16 @@ async def signup_user_with_invitation(email: EmailStr, if response.user: user_id = response.user.id - # Determine tenant_id based on invitation code - if invitation_info: - tenant_id = invitation_info["tenant_id"] - if invitation_info.get("code_type") == ASSET_OWNER_INVITE_CODE_TYPE: - tenant_id = ASSET_OWNER_TENANT_ID - else: - tenant_id = DEFAULT_TENANT_ID + # Determine tenant_id from the validated invitation code. + tenant_id = invitation_info["tenant_id"] + if invitation_info.get("code_type") == ASSET_OWNER_INVITE_CODE_TYPE: + tenant_id = ASSET_OWNER_TENANT_ID is_asset_owner_registration = user_role == ASSET_OWNER_ROLE # Create user tenant relationship - logging.debug( - f"Creating user tenant relationship: user_id={user_id}, tenant_id={tenant_id}, user_role={user_role}") insert_user_tenant( user_id=user_id, tenant_id=tenant_id, user_role=user_role, user_email=email) - logging.debug( - f"User tenant relationship created successfully for user {user_id}") # Use invitation code now that we have the real user_id if invitation_info: @@ -275,8 +268,6 @@ async def signup_user_with_invitation(email: EmailStr, logging.info( f"User {email} registered successfully, role: {user_role}, tenant: {tenant_id}, auto_login={auto_login}") - if user_role == "ADMIN": - await generate_tts_stt_4_admin(tenant_id, user_id) # Initialize tool list for the new tenant (only once per tenant) if not is_asset_owner_registration: @@ -314,38 +305,6 @@ async def parse_supabase_response(is_admin, response, user_role, auto_login: boo "registration_type": "admin" if is_admin else "user" } - -async def generate_tts_stt_4_admin(tenant_id, user_id): - tts_model_data = { - "model_repo": "", - "model_name": "volcano_tts", - "model_factory": "OpenAI-API-Compatible", - "model_type": "tts", - "api_key": "", - "base_url": "", - "max_tokens": 0, - "used_token": 0, - "display_name": "volcano_tts", - "connect_status": "unavailable", - "delete_flag": "N" - } - stt_model_data = { - "model_repo": "", - "model_name": "volcano_stt", - "model_factory": "OpenAI-API-Compatible", - "model_type": "stt", - "api_key": "", - "base_url": "", - "max_tokens": 0, - "used_token": 0, - "display_name": "volcano_stt", - "connect_status": "unavailable", - "delete_flag": "N" - } - create_model_record(tts_model_data, user_id, tenant_id) - create_model_record(stt_model_data, user_id, tenant_id) - - async def verify_invite_code(invite_code): logging.info( "detect admin registration request, start verifying invite code") diff --git a/backend/services/user_service.py b/backend/services/user_service.py index 6f4edcb1a6..8351c4df29 100644 --- a/backend/services/user_service.py +++ b/backend/services/user_service.py @@ -8,14 +8,13 @@ get_users_by_tenant_id, update_user_tenant_role, get_user_tenant_by_user_id, soft_delete_user_tenant_by_user_id ) -from database.group_db import remove_user_from_all_groups +from database.group_db import remove_user_from_all_groups, query_groups_by_users from database.memory_config_db import soft_delete_all_configs_by_user_id from database.conversation_db import soft_delete_all_conversations_by_user from database.oauth_account_db import soft_delete_all_oauth_accounts_by_user_id +from consts.const import IS_SPEED_MODE +from consts.exceptions import ForbiddenError, NotFoundException from utils.auth_utils import get_supabase_admin_client -from utils.memory_utils import build_memory_config - -from nexent.memory.memory_service import clear_memory logger = logging.getLogger(__name__) @@ -38,17 +37,20 @@ def get_users(tenant_id: str, page: Optional[int] = 1, page_size: Optional[int] # Get user-tenant relationships from database with pagination and sorting result = get_users_by_tenant_id(tenant_id, page, page_size, sort_by, sort_order) - # For now, return basic user information from the relationships - # In the future, this could be enhanced to fetch full user details from Supabase - users = [] - for relationship in result["users"]: - user_info = { - "id": relationship["user_id"], - "username": relationship.get("user_email"), - "role": relationship["user_role"], - "tenant_id": relationship["tenant_id"] + # Batch fetch group names for all users in a single query + tenant_user_ids = [r["user_id"] for r in result["users"]] + user_group_map = query_groups_by_users(tenant_user_ids) + + users = [ + { + "id": r["user_id"], + "username": r.get("user_email"), + "role": r["user_role"], + "tenant_id": r["tenant_id"], + "group_names": user_group_map.get(r["user_id"], []), } - users.append(user_info) + for r in result["users"] + ] # Calculate pagination info only if pagination is used if page is not None and page_size is not None: @@ -59,11 +61,33 @@ def get_users(tenant_id: str, page: Optional[int] = 1, page_size: Optional[int] "page_size": page_size, "total_pages": (result["total"] + page_size - 1) // page_size } + return { + "users": users, + "total": result["total"] + } + + +def get_users_for_requester( + tenant_id: str, + page: Optional[int] = 1, + page_size: Optional[int] = 20, + sort_by: str = "created_at", + sort_order: str = "desc", + *, + requester_tenant_id: str, + requester_role: str, +) -> Dict[str, Any]: + """List users after enforcing role and tenant boundaries.""" + role = (requester_role or "").upper() + is_speed_admin = IS_SPEED_MODE and role == "SPEED" + if role == "SU" or is_speed_admin: + pass + elif role == "ADMIN" and tenant_id == requester_tenant_id: + pass else: - return { - "users": users, - "total": result["total"] - } + raise ForbiddenError("Not authorized to list users for this tenant") + + return get_users(tenant_id, page, page_size, sort_by, sort_order) async def update_user(user_id: str, update_data: Dict[str, Any], updated_by: str) -> Dict[str, Any]: @@ -114,6 +138,34 @@ async def update_user(user_id: str, update_data: Dict[str, Any], updated_by: str raise +async def update_user_for_requester( + user_id: str, + update_data: Dict[str, Any], + *, + updated_by: str, + requester_tenant_id: str, + requester_role: str, +) -> Dict[str, Any]: + """Update a user after enforcing management role and tenant boundaries.""" + target_user = get_user_tenant_by_user_id(user_id) + if not target_user: + raise NotFoundException(f"User {user_id} not found") + + role = (requester_role or "").upper() + target_role = str(target_user.get("user_role") or "").upper() + target_tenant_id = target_user.get("tenant_id") + is_speed_admin = IS_SPEED_MODE and role == "SPEED" + + if role == "SU" or is_speed_admin: + pass + elif role == "ADMIN" and target_tenant_id == requester_tenant_id and target_role != "SU": + pass + else: + raise ForbiddenError("Not authorized to update this user") + + return await update_user(user_id, update_data, updated_by) + + async def delete_user_and_cleanup(user_id: str, tenant_id: str) -> None: """ Permanently delete user account and all related data. @@ -156,25 +208,10 @@ async def delete_user_and_cleanup(user_id: str, tenant_id: str) -> None: except Exception as e: logger.error(f"Failed deleting conversations for user {user_id}: {e}") - # 4) Clear memory records - try: - memory_config = build_memory_config(tenant_id) - await clear_memory( - memory_level="user", - memory_config=memory_config, - tenant_id=tenant_id, - user_id=user_id, - ) - await clear_memory( - memory_level="user_agent", - memory_config=memory_config, - tenant_id=tenant_id, - user_id=user_id, - ) - logger.debug("\tUser memories cleared.") - except Exception as e: - logger.error(f"Failed clearing memory for user {user_id}: {e}") - + # 4) Memory record cleanup: in the new Memory system this is performed + # by ``MemoryService.forget_user`` (PG + ES purge). The legacy + # mem0-era ``clear_memory`` path has been removed; the new path will + # be wired in once the storage layer lands in Phase 2. # 5) Soft-delete OAuth account bindings try: deleted_oauth = soft_delete_all_oauth_accounts_by_user_id(user_id, user_id) diff --git a/backend/services/vectordatabase_service.py b/backend/services/vectordatabase_service.py index dd2f6e51a7..3d1f07a9a5 100644 --- a/backend/services/vectordatabase_service.py +++ b/backend/services/vectordatabase_service.py @@ -11,6 +11,7 @@ """ import asyncio import hashlib +import inspect import json import logging import os @@ -21,13 +22,30 @@ from fastapi import Body, Depends, Path, Query from fastapi.responses import StreamingResponse -from nexent.core.models.embedding_model import OpenAICompatibleEmbedding, JinaEmbedding, DashScopeMultimodalEmbedding, BaseEmbedding +from nexent.core.models.embedding_model import ( + BaseEmbedding, + DashScopeMultimodalEmbedding, + JinaEmbedding, + OpenAICompatibleEmbedding, + SiliconflowMultimodalEmbedding, +) from nexent.core.models.rerank_model import OpenAICompatibleRerank, BaseRerank from nexent.vector_database.base import VectorDatabaseCore from nexent.vector_database.elasticsearch_core import ElasticSearchCore from nexent.vector_database.datamate_core import DataMateCore -from consts.const import DATAMATE_URL, ES_API_KEY, ES_HOST, LANGUAGE, VectorDatabaseType, IS_SPEED_MODE, PERMISSION_EDIT, PERMISSION_READ, ASSET_OWNER_TENANT_ID +from consts.const import ( + ASSET_OWNER_TENANT_ID, + CAN_EDIT_ALL_USER_ROLES, + DATAMATE_URL, + ES_API_KEY, + ES_HOST, + IS_SPEED_MODE, + LANGUAGE, + PERMISSION_EDIT, + PERMISSION_READ, + VectorDatabaseType, +) from consts.model import ChunkCreateRequest, ChunkUpdateRequest from database.attachment_db import delete_file, file_exists, get_file_stream from database.knowledge_db import ( @@ -224,6 +242,8 @@ def get_embedding_model_by_index_name(tenant_id: str, index_name: str) -> tuple[ # Configure logging logger = logging.getLogger("vectordatabase_service") +_QUOTA_LIMIT_UNSET = object() + def get_vector_db_core( db_type: VectorDatabaseType = VectorDatabaseType.ELASTICSEARCH, tenant_id: Optional[str] = None, @@ -328,6 +348,7 @@ def _build_model_config(model: dict) -> dict: def _create_embedding_model(model: dict) -> Any: model_config = _build_model_config(model) + model_type = model.get("model_type", "embedding") common_kwargs = { "api_key": model_config.get("api_key", ""), "base_url": model_config.get("base_url", ""), @@ -335,11 +356,22 @@ def _create_embedding_model(model: dict) -> Any: "embedding_dim": model_config.get("max_tokens", 1024), "ssl_verify": model_config.get("ssl_verify", True), } - if model.get("model_type", "embedding") == "multi_embedding": + + if model_type == "multi_embedding": model_factory = model.get("model_factory", "").lower() if model_factory == "dashscope": return DashScopeMultimodalEmbedding(**common_kwargs) + if model_factory == "silicon": + return SiliconflowMultimodalEmbedding(**common_kwargs) return JinaEmbedding(**common_kwargs) + + if model_type != "embedding": + raise ValueError( + f"Invalid model_type '{model_type}' for model '{common_kwargs['model_name']}'. " + f"Expected 'embedding' or 'multi_embedding', got '{model_type}'. " + f"Please check the model configuration in the model management page." + ) + return OpenAICompatibleEmbedding(**common_kwargs) def get_embedding_model( @@ -412,33 +444,7 @@ def get_embedding_model_by_id(tenant_id: str, model_id: int) -> tuple[Optional[A try: model = get_model_by_model_id(model_id, tenant_id) if model and model.get("model_type") in ["embedding", "multi_embedding"]: - model_config = { - "model_repo": model.get("model_repo", ""), - "model_name": model["model_name"], - "api_key": model.get("api_key", ""), - "base_url": model.get("base_url", ""), - "model_type": model.get("model_type", "embedding"), - "max_tokens": model.get("max_tokens", 1024), - "ssl_verify": model.get("ssl_verify", True), - } - model_type = model.get("model_type", "embedding") - if model_type == "multi_embedding": - embedding_model = JinaEmbedding( - api_key=model_config.get("api_key", ""), - base_url=model_config.get("base_url", ""), - model_name=get_model_name_from_config(model_config) or "", - embedding_dim=model_config.get("max_tokens", 1024), - ssl_verify=model_config.get("ssl_verify", True), - ) - else: - embedding_model = OpenAICompatibleEmbedding( - api_key=model_config.get("api_key", ""), - base_url=model_config.get("base_url", ""), - model_name=get_model_name_from_config(model_config) or "", - embedding_dim=model_config.get("max_tokens", 1024), - ssl_verify=model_config.get("ssl_verify", True), - ) - return embedding_model, model.get("model_id") + return _create_embedding_model(model), model.get("model_id") else: logger.warning(f"Model with id {model_id} not found or is not an embedding model") except Exception as e: @@ -493,6 +499,158 @@ def get_rerank_model(tenant_id: str, model_name: Optional[str] = None): class ElasticSearchService: + CREATOR_PERMISSION = "CREATOR" + + @staticmethod + def resolve_knowledge_base_permission( + index_name: str, + user_id: str, + tenant_id: Optional[str] = None, + ) -> Optional[str]: + """Resolve the current user's permission for one knowledge base.""" + record = get_knowledge_record({"index_name": index_name}) + if not record: + raise ValueError(f"Knowledge base '{index_name}' not found") + + if record.get("knowledge_sources") == "datamate": + return PERMISSION_READ + + user_tenant = get_user_tenant_by_user_id(user_id) + if not user_tenant and not IS_SPEED_MODE: + return None + + user_role = (user_tenant or {}).get("user_role") + user_tenant_id = str((user_tenant or {}).get("tenant_id") or tenant_id or "") + effective_user_role = user_role + if user_id == user_tenant_id: + effective_user_role = "ADMIN" + logger.info(f"User {user_id} identified as legacy admin") + elif IS_SPEED_MODE: + effective_user_role = "SPEED" + logger.info("User under SPEED version is treated as admin") + + role = (effective_user_role or "").upper() + record_tenant_id = str(record.get("tenant_id") or "") + is_asset_owner_record = record_tenant_id == ASSET_OWNER_TENANT_ID + + if is_asset_owner_record: + if role == "ASSET_OWNER": + return PERMISSION_EDIT + if role in {"SU", "ADMIN", "SPEED", "DEV"}: + return PERMISSION_READ + return None + + if record_tenant_id and user_tenant_id and record_tenant_id != user_tenant_id: + return None + + if role in CAN_EDIT_ALL_USER_ROLES: + return PERMISSION_EDIT + + if role in {"USER", "DEV"}: + if str(record.get("created_by")) == str(user_id): + return ElasticSearchService.CREATOR_PERMISSION + + kb_group_ids_str = record.get("group_ids") + kb_group_ids = convert_string_to_list(kb_group_ids_str or "") + user_group_ids = query_group_ids_by_user(user_id) + + kb_groups_empty = ( + kb_group_ids_str is None + or (isinstance(kb_group_ids_str, str) and kb_group_ids_str.strip() == "") + or len(kb_group_ids) == 0 + ) + user_groups_empty = len(user_group_ids) == 0 + + has_group_intersection = ( + True + if kb_groups_empty and user_groups_empty + else bool(set(user_group_ids) & set(kb_group_ids)) + ) + if not has_group_intersection: + return None + + ingroup_permission = record.get("ingroup_permission") or PERMISSION_READ + if ingroup_permission == PERMISSION_EDIT: + return PERMISSION_EDIT + if ingroup_permission == PERMISSION_READ: + return PERMISSION_READ + if ingroup_permission == "PRIVATE": + return None + + return None + + @staticmethod + def require_knowledge_base_edit_permission( + index_name: str, + user_id: str, + tenant_id: Optional[str] = None, + ) -> str: + """Raise when the current user cannot modify the knowledge base.""" + permission = ElasticSearchService.resolve_knowledge_base_permission( + index_name=index_name, + user_id=user_id, + tenant_id=tenant_id, + ) + if permission not in {PERMISSION_EDIT, ElasticSearchService.CREATOR_PERMISSION}: + raise PermissionError("No permission to modify this knowledge base") + return permission + + @staticmethod + def require_knowledge_base_read_permission( + index_name: str, + user_id: str, + tenant_id: Optional[str] = None, + ) -> str: + """Raise when the current user cannot read the knowledge base. + + Accepts any non-None permission level (READ_ONLY, EDIT, or CREATOR). + """ + permission = ElasticSearchService.resolve_knowledge_base_permission( + index_name=index_name, + user_id=user_id, + tenant_id=tenant_id, + ) + if permission is None: + raise PermissionError("No permission to access this knowledge base") + return permission + + @staticmethod + def filter_accessible_indices( + index_names: List[str], + user_id: str, + tenant_id: Optional[str] = None, + ) -> List[str]: + """Return only the indices the user has at least read access to. + + Indices whose knowledge base record cannot be found, or whose permission + check fails for any reason, are treated as inaccessible and dropped. + Order of the accessible subset is preserved. + """ + accessible: List[str] = [] + for index_name in index_names: + try: + permission = ElasticSearchService.resolve_knowledge_base_permission( + index_name=index_name, + user_id=user_id, + tenant_id=tenant_id, + ) + except ValueError: + # Knowledge base record not found in the DB - treat as inaccessible. + logger.warning( + "Knowledge base '%s' not found during permission check, skipping", + index_name, + ) + continue + except Exception as e: + logger.warning( + "Permission check failed for knowledge base '%s': %s", index_name, e + ) + continue + + if permission is not None: + accessible.append(index_name) + return accessible + @staticmethod async def full_delete_knowledge_base(index_name: str, vdb_core: VectorDatabaseCore, user_id: str): """ @@ -655,9 +813,9 @@ def create_knowledge_base( tenant_id: Optional[str], ingroup_permission: Optional[str] = None, group_ids: Optional[List[int]] = None, - embedding_model_name: Optional[str] = None, - is_multimodal: Optional[bool] = None, + embedding_model_id: Optional[int] = None, preserve_source_file: Optional[bool] = None, + quota_limit_bytes: Optional[int] = None, ): """ Create a new knowledge base with a user-facing name and an internal Elasticsearch index name. @@ -675,8 +833,7 @@ def create_knowledge_base( tenant_id: Tenant ID ingroup_permission: Permission level (optional) group_ids: List of group IDs (optional) - embedding_model_name: Specific embedding model name to use (optional). - If provided, will use this model instead of tenant default. + embedding_model_id: Unique ID of the selected embedding model. preserve_source_file: Whether to preserve uploaded source documents after vectorization (optional; defaults to True when omitted). @@ -684,24 +841,19 @@ def create_knowledge_base( with an explicit index_name. """ try: - # Get embedding model - use user-selected model if provided, otherwise use tenant default - selected_model_type = None - if is_multimodal is True: - selected_model_type = "multi_embedding" - elif is_multimodal is False and embedding_model_name: - selected_model_type = "embedding" - - embedding_model, model_id = get_embedding_model( - tenant_id, - embedding_model_name, - selected_model_type - ) + if embedding_model_id is None: + raise ValueError("embedding_model_id is required") + + model = get_model_by_model_id(embedding_model_id, tenant_id) + if not model: + raise ValueError(f"Embedding model with id {embedding_model_id} not found") + if model.get("model_type") not in ["embedding", "multi_embedding"]: + raise ValueError( + f"Model with id {embedding_model_id} is not an embedding model" + ) - # Determine the embedding model name to save: use user-provided name if available, - # otherwise use the model's display name - saved_embedding_model_name = embedding_model_name - if not saved_embedding_model_name and embedding_model: - saved_embedding_model_name = embedding_model.model + embedding_model = _create_embedding_model(model) + saved_embedding_model_name = model.get("display_name") or model.get("model_name") # Create knowledge record first to obtain knowledge_id and generated index_name knowledge_data = { @@ -710,7 +862,7 @@ def create_knowledge_base( "user_id": user_id, "tenant_id": tenant_id, "embedding_model_name": saved_embedding_model_name, - "embedding_model_id": model_id, + "embedding_model_id": embedding_model_id, } # Add group permission and group IDs if provided @@ -720,6 +872,8 @@ def create_knowledge_base( knowledge_data["group_ids"] = group_ids if preserve_source_file is not None: knowledge_data["preserve_source_file"] = preserve_source_file + if quota_limit_bytes is not None: + knowledge_data["quota_limit_bytes"] = quota_limit_bytes record_info = create_knowledge_record(knowledge_data) index_name = record_info["index_name"] @@ -737,9 +891,13 @@ def create_knowledge_base( "status": "success", "message": f"Index {index_name} created successfully", "id": index_name, + "embedding_model_name": saved_embedding_model_name, + "model_type": model.get("model_type"), "knowledge_id": record_info["knowledge_id"], "name": record_info.get("knowledge_name", knowledge_name), } + except ValueError: + raise except Exception as e: raise Exception(f"Error creating knowledge base: {str(e)}") @@ -751,6 +909,7 @@ def update_knowledge_base( group_ids: Optional[List[int]] = None, tenant_id: Optional[str] = None, user_id: Optional[str] = None, + quota_limit_bytes: Any = _QUOTA_LIMIT_UNSET, ) -> bool: """ Update knowledge base information (name, group permission, group assignments). @@ -762,6 +921,7 @@ def update_knowledge_base( group_ids: List of group IDs to assign (optional) tenant_id: ID of the tenant (optional, for validation) user_id: ID of the user making the update + quota_limit_bytes: New soft quota in bytes; None removes the quota Returns: bool: Whether the update was successful @@ -791,6 +951,9 @@ def update_knowledge_base( # Convert list to string for database storage update_data["group_ids"] = convert_list_to_string(group_ids) + if quota_limit_bytes is not _QUOTA_LIMIT_UNSET: + update_data["quota_limit_bytes"] = quota_limit_bytes + # Call database update function result = update_knowledge_record(update_data) @@ -1013,39 +1176,39 @@ def list_indices( kb_ingroup_permission = record.get( "ingroup_permission") or PERMISSION_READ - # Check if user belongs to any of the knowledgebase groups - # Compatibility logic for legacy data: - # - If both kb_group_ids and user_group_ids are effectively empty (None or empty lists), - # consider them intersecting (backward compatibility) - # - If either side has groups but they don't intersect, no intersection - kb_groups_empty = kb_group_ids_str is None or (isinstance( - kb_group_ids_str, str) and kb_group_ids_str.strip() == "") or len(kb_group_ids) == 0 - user_groups_empty = len(user_group_ids) == 0 - - if kb_groups_empty and user_groups_empty: - # Both are empty/None - consider intersecting for backward compatibility - has_group_intersection = True + if str(kb_created_by) == str(user_id): + permission = "CREATOR" else: - # Normal intersection check - has_group_intersection = bool( - set(user_group_ids) & set(kb_group_ids)) - - if has_group_intersection: - # Determine permission level - permission = PERMISSION_READ # Default - - # User is creator: creator permission - if kb_created_by == user_id: - permission = "CREATOR" - # Group permission allows editing - elif kb_ingroup_permission == PERMISSION_EDIT: - permission = PERMISSION_EDIT - # Group permission is read-only: already set - elif kb_ingroup_permission == PERMISSION_READ: - permission = PERMISSION_READ - # Group permission is private: not visible - elif kb_ingroup_permission == "PRIVATE": - permission = None + # Check if user belongs to any of the knowledgebase groups + # Compatibility logic for legacy data: + # - If both kb_group_ids and user_group_ids are effectively empty (None or empty lists), + # consider them intersecting (backward compatibility) + # - If either side has groups but they don't intersect, no intersection + kb_groups_empty = kb_group_ids_str is None or (isinstance( + kb_group_ids_str, str) and kb_group_ids_str.strip() == "") or len(kb_group_ids) == 0 + user_groups_empty = len(user_group_ids) == 0 + + if kb_groups_empty and user_groups_empty: + # Both are empty/None - consider intersecting for backward compatibility + has_group_intersection = True + else: + # Normal intersection check + has_group_intersection = bool( + set(user_group_ids) & set(kb_group_ids)) + + if has_group_intersection: + # Determine permission level + permission = PERMISSION_READ # Default + + # Group permission allows editing + if kb_ingroup_permission == PERMISSION_EDIT: + permission = PERMISSION_EDIT + # Group permission is read-only: already set + elif kb_ingroup_permission == PERMISSION_READ: + permission = PERMISSION_READ + # Group permission is private: not visible + elif kb_ingroup_permission == "PRIVATE": + permission = None # Add to visible list if permission is granted if permission: @@ -1078,6 +1241,10 @@ def list_indices( response = { "indices": indices, "count": len(indices), + "index_permissions": { + record["index_name"]: record["permission"] + for record in visible_knowledgebases + }, } if include_stats: @@ -1259,7 +1426,7 @@ def index_documents( 'tenant_id') if knowledge_record else None if tenant_id: - model_type = "EMBEDDING_ID" if embedding_model.model_type == "text" else "MULTI_EMBEDDING_ID" + model_type = "EMBEDDING_ID" if embedding_model.model_type == "embedding" else "MULTI_EMBEDDING_ID" model_config = tenant_config_manager.get_model_config( key=model_type, tenant_id=tenant_id) embedding_batch_size = model_config.get("chunk_batch", 10) diff --git a/backend/tool_collection/mcp/local_mcp_service.py b/backend/tool_collection/mcp/local_mcp_service.py index 1255ad038f..b272ede2d3 100644 --- a/backend/tool_collection/mcp/local_mcp_service.py +++ b/backend/tool_collection/mcp/local_mcp_service.py @@ -1,7 +1,35 @@ from fastmcp import FastMCP +from tool_collection.mcp.nl2agent_mcp_tools import ( + NL2AGENT_MCP_TOOL_META, + NL2A_WRAPPER_DESCRIPTION, + NL2A_WRAPPER_NAME, + SEARCH_INSTALLED_MCP_TOOLS_DESCRIPTION, + SEARCH_INSTALLED_MCP_TOOLS_NAME, + nl2a_wrapper as _nl2a_wrapper, + search_installed_mcp_tools as _search_installed_mcp_tools, +) + +LOCAL_MCP_TOOL_NAME_OVERRIDES = { + SEARCH_INSTALLED_MCP_TOOLS_NAME: SEARCH_INSTALLED_MCP_TOOLS_NAME, + NL2A_WRAPPER_NAME: NL2A_WRAPPER_NAME, +} + # Create MCP server local_mcp_service = FastMCP("local") +local_mcp_service.tool( + _search_installed_mcp_tools, + name=SEARCH_INSTALLED_MCP_TOOLS_NAME, + description=SEARCH_INSTALLED_MCP_TOOLS_DESCRIPTION, + meta=NL2AGENT_MCP_TOOL_META, +) +local_mcp_service.tool( + _nl2a_wrapper, + name=NL2A_WRAPPER_NAME, + description=NL2A_WRAPPER_DESCRIPTION, + meta=NL2AGENT_MCP_TOOL_META, +) + @local_mcp_service.tool(name="test_tool_name", description="test_tool_description") @@ -9,4 +37,3 @@ async def demo_tool(para_1: str, para_2: int) -> str: print("tool is called successfully") print(para_1, para_2) return "success" - diff --git a/backend/tool_collection/mcp/nl2agent_mcp_tools.py b/backend/tool_collection/mcp/nl2agent_mcp_tools.py new file mode 100644 index 0000000000..cb9dbff9ef --- /dev/null +++ b/backend/tool_collection/mcp/nl2agent_mcp_tools.py @@ -0,0 +1,567 @@ +"""Define and implement the internal Local MCP tools used by NL2Agent.""" + +from copy import deepcopy +import json +import keyword +import logging +import re +import unicodedata +from typing import Annotated, Any, Literal + +from fastmcp.server.dependencies import get_http_request +from nexent.core.agents.agent_model import ToolConfig +from pydantic import BaseModel, ConfigDict, Field, model_validator + +from utils.auth_utils import get_current_user_id + +logger = logging.getLogger(__name__) + +SEARCH_INSTALLED_MCP_TOOLS_NAME = "search_installed_mcp_tools" +NL2A_WRAPPER_NAME = "nl2a_wrapper" +SEARCH_INSTALLED_MCP_TOOLS_DESCRIPTION = ( + "Search the current tenant's installed and available MCP tools using keywords. " + "Returns a structured JSON observation ordered by relevance. " + "Call the tool as `result = search_installed_mcp_tools(...)`, then use " + "`print(result)` to preserve the returned JSON unchanged in execution logs." +) +NL2A_WRAPPER_DESCRIPTION = ( + "Build one NL2Agent output from subtype-specific parameters. Always pass " + "`subtype`. For `local_mcp_recommendation`, also pass `search_result` and " + "`selected_tool_ids`. For `agent_draft`, pass the agent draft fields. Call " + "the tool as `result = nl2a_wrapper(...)`, then use `print(result)`." +) +NL2AGENT_MCP_TOOL_META = {"nexent_internal": True} +MAX_TOOL_RECOMMENDATIONS = 5 +FEW_SHOT_EXAMPLE_COUNT = 2 +NL2A_SUBTYPES = Literal["local_mcp_recommendation", "agent_draft"] + +LOCAL_MCP_RECOMMENDATION_JSON_TEMPLATE: dict[str, Any] = { + "subtype": "local_mcp_recommendation", + "status": "success", + "recommendation_count": 0, + "recommendations": [], +} + +AGENT_DRAFT_JSON_TEMPLATE: dict[str, Any] = { + "subtype": "agent_draft", + "name": "", + "display_name": "", + "description": "", + "duty_prompt": "", + "constraint_prompt": "", + "few_shots_prompt": None, + "greeting_message": "", + "example_questions": [], +} + + +class InstalledMcpToolRecommendation(BaseModel): + """Safe display metadata for one installed MCP tool recommendation.""" + + tool_id: int + name: str + origin_name: str | None = None + description: str + source: Literal["mcp"] = "mcp" + usage: str + labels: list[str] + inputs: dict[str, Any] + score: float + + +class GeneratedAgentDraft(BaseModel): + """Complete in-memory agent draft for the agent creation flow.""" + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + subtype: Literal["agent_draft"] = "agent_draft" + name: str = Field(min_length=1, max_length=30) + display_name: str = Field(min_length=1, max_length=30) + description: str = Field(min_length=1) + duty_prompt: str = Field(min_length=1) + constraint_prompt: str + few_shots_prompt: str | None = None + greeting_message: str = Field(min_length=1) + example_questions: list[str] = Field(min_length=3, max_length=5) + + +class SearchInstalledMcpToolsObservation(BaseModel): + """Successful structured observation returned to the agent.""" + + subtype: Literal["local_mcp_recommendation"] = "local_mcp_recommendation" + status: Literal["success"] = "success" + recommendation_count: int + recommendations: list[InstalledMcpToolRecommendation] + + +class SearchInstalledMcpToolsErrorObservation(BaseModel): + """Safe structured error returned to the agent.""" + + subtype: Literal["local_mcp_recommendation"] = "local_mcp_recommendation" + status: Literal["error"] = "error" + code: Literal["invalid_keywords", "tool_search_failed"] + retryable: Literal[True] = True + + +class Nl2aFewShotToolCall(BaseModel): + """One selected-tool call rendered into an agent few-shot example.""" + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + name: str = Field(min_length=1) + arguments: dict[str, Any] + + @model_validator(mode="after") + def validate_python_names(self) -> "Nl2aFewShotToolCall": + if not self.name.isidentifier() or keyword.iskeyword(self.name): + raise ValueError("tool call name must be a valid Python identifier") + if any( + not name.isidentifier() or keyword.iskeyword(name) + for name in self.arguments + ): + raise ValueError("tool argument names must be valid Python identifiers") + return self + + +class Nl2aFewShotStep(BaseModel): + """One Think-Code-Observation step in a structured few-shot example.""" + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + reasoning: str = Field(min_length=1) + tool_calls: list[Nl2aFewShotToolCall] = Field(min_length=1) + observation: str = Field(min_length=1) + + +class Nl2aFewShotExample(BaseModel): + """Structured few-shot content that contains no executable code tags.""" + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + user_input: str = Field(min_length=1) + steps: list[Nl2aFewShotStep] = Field(min_length=1) + final_reasoning: str = Field(min_length=1) + final_answer: str = Field(min_length=1) + + +Nl2aFewShotExamples = Annotated[ + list[Nl2aFewShotExample], + Field( + min_length=FEW_SHOT_EXAMPLE_COUNT, + max_length=FEW_SHOT_EXAMPLE_COUNT, + description="Exactly two structured few-shot examples.", + ), +] + + +class Nl2aLocalMcpRecommendationInput(BaseModel): + """Wrapper input for a real installed-tool search observation.""" + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + subtype: Literal["local_mcp_recommendation"] + search_result: dict[str, Any] + selected_tool_ids: list[int] = Field(max_length=MAX_TOOL_RECOMMENDATIONS) + + @model_validator(mode="after") + def validate_selected_tool_ids(self) -> "Nl2aLocalMcpRecommendationInput": + if len(self.selected_tool_ids) != len(set(self.selected_tool_ids)): + raise ValueError("selected_tool_ids must be unique") + return self + + +class Nl2aAgentDraftInput(BaseModel): + """Wrapper input used to validate and render a complete agent draft.""" + + model_config = ConfigDict(extra="forbid", str_strip_whitespace=True) + + subtype: Literal["agent_draft"] + language: Literal["en", "zh"] + name: str = Field(min_length=1, max_length=30) + display_name: str = Field(min_length=1, max_length=30) + description: str = Field(min_length=1) + duty_prompt: str = Field(min_length=1) + constraint_prompt: str + greeting_message: str = Field(min_length=1) + example_questions: list[str] = Field(min_length=3, max_length=5) + selected_tool_names: list[str] = Field(max_length=MAX_TOOL_RECOMMENDATIONS) + few_shot_examples: Nl2aFewShotExamples | None = None + + @model_validator(mode="after") + def validate_few_shot_tools(self) -> "Nl2aAgentDraftInput": + if re.fullmatch(r"[A-Za-z_][A-Za-z0-9_]*_assistant", self.name) is None: + raise ValueError( + "name must be a Python-compatible identifier ending with _assistant" + ) + if self.language == "en": + if not self.display_name.endswith("Assistant") or any( + character.isspace() for character in self.display_name + ): + raise ValueError( + "English display_name must be one word ending with Assistant" + ) + elif not self.display_name.endswith("助手"): + raise ValueError("Chinese display_name must end with 助手") + + selected_names = set(self.selected_tool_names) + if len(selected_names) != len(self.selected_tool_names): + raise ValueError("selected_tool_names must be unique") + if any( + not name.isidentifier() or keyword.iskeyword(name) + for name in selected_names + ): + raise ValueError("selected tool names must be valid Python identifiers") + if selected_names and self.few_shot_examples is None: + raise ValueError("few_shot_examples are required when tools are selected") + if selected_names and not self.constraint_prompt: + raise ValueError("constraint_prompt is required when tools are selected") + if not selected_names and self.few_shot_examples is not None: + raise ValueError("few_shot_examples require selected tools") + if not selected_names and self.constraint_prompt: + raise ValueError("constraint_prompt must be empty when no tools are selected") + for example in self.few_shot_examples or []: + unknown_names = { + call.name + for step in example.steps + for call in step.tool_calls + } - selected_names + if unknown_names: + raise ValueError( + "few-shot tool calls must use selected tool names: " + + ", ".join(sorted(unknown_names)) + ) + return self + + +def _render_few_shots( + language: Literal["en", "zh"], + few_shot_examples: Nl2aFewShotExamples | None, +) -> str | None: + if few_shot_examples is None: + return None + + rendered_examples: list[str] = [] + for example_index, example in enumerate(few_shot_examples, start=1): + if language == "en": + lines = [f'Task {example_index}: "{example.user_input}"'] + else: + lines = [f'任务{example_index}："{example.user_input}"'] + + for step_index, step in enumerate(example.steps, start=1): + code_lines: list[str] = [] + multiple_calls = len(step.tool_calls) > 1 + for call_index, call in enumerate(step.tool_calls, start=1): + variable_name = ( + f"result_{step_index}_{call_index}" + if multiple_calls + else f"result_{step_index}" + ) + arguments = ", ".join( + f"{name}={value!r}" for name, value in call.arguments.items() + ) + code_lines.append(f"{variable_name} = {call.name}({arguments})") + code_lines.append(f"print({variable_name})") + + think_label = "Think" if language == "en" else "思考" + code_label = "Code" if language == "en" else "代码" + observation_prefix = ( + "# System returns Observation" + if language == "en" + else "# 系统返回 Observation" + ) + lines.extend( + [ + "", + f"{think_label}: {step.reasoning}", + "", + f"{code_label}:", + "", + *code_lines, + "", + "", + f"{observation_prefix}: {step.observation}", + ] + ) + + think_label = "Think" if language == "en" else "思考" + lines.extend( + [ + "", + f"{think_label}: {example.final_reasoning}", + "", + example.final_answer, + ] + ) + rendered_examples.append("\n".join(lines)) + return "\n\n---\n\n".join(rendered_examples) + + +def build_nl2a_wrapper( + subtype: NL2A_SUBTYPES, + search_result: dict[str, Any] | None = None, + selected_tool_ids: list[int] | None = None, + language: Literal["en", "zh"] | None = None, + name: str | None = None, + display_name: str | None = None, + description: str | None = None, + duty_prompt: str | None = None, + constraint_prompt: str | None = None, + greeting_message: str | None = None, + example_questions: list[str] | None = None, + selected_tool_names: list[str] | None = None, + few_shot_examples: Nl2aFewShotExamples | None = None, +) -> str: + """Fill the JSON template selected by subtype and return its wrapper.""" + + if subtype == "local_mcp_recommendation": + if search_result is None or selected_tool_ids is None: + raise ValueError( + "local_mcp_recommendation requires search_result and selected_tool_ids" + ) + payload = Nl2aLocalMcpRecommendationInput( + subtype=subtype, + search_result=search_result, + selected_tool_ids=selected_tool_ids, + ) + if payload.search_result.get("status") == "error": + if payload.selected_tool_ids: + raise ValueError("selected_tool_ids must be empty for a search error") + observation = SearchInstalledMcpToolsErrorObservation.model_validate( + payload.search_result + ) + output = deepcopy(LOCAL_MCP_RECOMMENDATION_JSON_TEMPLATE) + output.pop("recommendation_count") + output.pop("recommendations") + output.update(observation.model_dump(mode="json", exclude={"subtype"})) + elif payload.search_result.get("status") == "success": + observation = SearchInstalledMcpToolsObservation.model_validate( + payload.search_result + ) + selected_ids = set(payload.selected_tool_ids) + available_ids = { + recommendation.tool_id + for recommendation in observation.recommendations + } + unknown_ids = selected_ids - available_ids + if unknown_ids: + raise ValueError( + "selected tool IDs are not present in search_result: " + + ", ".join(str(tool_id) for tool_id in sorted(unknown_ids)) + ) + recommendations = [ + recommendation + for recommendation in observation.recommendations + if recommendation.tool_id in selected_ids + ] + output = deepcopy(LOCAL_MCP_RECOMMENDATION_JSON_TEMPLATE) + output.update( + recommendation_count=len(recommendations), + recommendations=[ + recommendation.model_dump(mode="json") + for recommendation in recommendations + ], + ) + else: + raise ValueError("search_result has an unsupported status") + elif subtype == "agent_draft": + required_parameters = { + "language": language, + "name": name, + "display_name": display_name, + "description": description, + "duty_prompt": duty_prompt, + "constraint_prompt": constraint_prompt, + "greeting_message": greeting_message, + "example_questions": example_questions, + "selected_tool_names": selected_tool_names, + } + missing_parameters = [ + parameter + for parameter, value in required_parameters.items() + if value is None + ] + if missing_parameters: + raise ValueError( + "agent_draft requires parameters: " + ", ".join(missing_parameters) + ) + payload = Nl2aAgentDraftInput( + subtype=subtype, + language=language, + name=name, + display_name=display_name, + description=description, + duty_prompt=duty_prompt, + constraint_prompt=constraint_prompt, + greeting_message=greeting_message, + example_questions=example_questions, + selected_tool_names=selected_tool_names, + few_shot_examples=few_shot_examples, + ) + draft = GeneratedAgentDraft( + name=payload.name, + display_name=payload.display_name, + description=payload.description, + duty_prompt=payload.duty_prompt, + constraint_prompt=payload.constraint_prompt, + few_shots_prompt=_render_few_shots( + payload.language, + payload.few_shot_examples, + ), + greeting_message=payload.greeting_message, + example_questions=payload.example_questions, + ) + output = deepcopy(AGENT_DRAFT_JSON_TEMPLATE) + output.update(draft.model_dump(mode="json", exclude={"subtype"})) + else: + raise ValueError(f"unsupported nl2a subtype: {subtype}") + + serialized = json.dumps( + output, + ensure_ascii=False, + separators=(",", ":"), + ) + return f"\n{serialized}\n\nNL2A payload generated." + + +def create_nl2agent_mcp_tool_configs() -> list[ToolConfig]: + """Create fresh SDK configs for the two NL2Agent MCP tools.""" + return [ + ToolConfig( + class_name=SEARCH_INSTALLED_MCP_TOOLS_NAME, + name=SEARCH_INSTALLED_MCP_TOOLS_NAME, + description=SEARCH_INSTALLED_MCP_TOOLS_DESCRIPTION, + inputs='{"keywords": "list[str]"}', + output_type="object", + params={}, + source="mcp", + usage="outer-apis", + ), + ToolConfig( + class_name=NL2A_WRAPPER_NAME, + name=NL2A_WRAPPER_NAME, + description=NL2A_WRAPPER_DESCRIPTION, + inputs=json.dumps( + { + "subtype": "str", + "search_result": "dict | None", + "selected_tool_ids": "list[int] | None", + "language": "str | None", + "name": "str | None", + "display_name": "str | None", + "description": "str | None", + "duty_prompt": "str | None", + "constraint_prompt": "str | None", + "greeting_message": "str | None", + "example_questions": "list[str] | None", + "selected_tool_names": "list[str] | None", + "few_shot_examples": "list[dict] with exactly 2 items | None", + }, + separators=(",", ":"), + ), + output_type="string", + params={}, + source="mcp", + usage="outer-apis", + ), + ] + + +def _dump_tool_search_observation( + observation: SearchInstalledMcpToolsObservation + | SearchInstalledMcpToolsErrorObservation, +) -> dict[str, Any]: + """Dump one tool search observation for direct wrapper consumption.""" + return observation.model_dump(mode="json") + + +def _prepare_search_keywords(keywords: list[str]) -> list[str] | None: + """Validate and de-duplicate keyword input while preserving its order.""" + + if not 1 <= len(keywords) <= 10: + return None + + prepared_keywords: list[str] = [] + seen: set[str] = set() + for raw_keyword in keywords: + stripped_keyword = raw_keyword.strip() + if not stripped_keyword or len(stripped_keyword) > 100: + return None + + normalized_keyword = unicodedata.normalize( + "NFKC", stripped_keyword + ).casefold() + normalized_keyword = re.sub(r"\s+", " ", normalized_keyword) + if normalized_keyword in seen: + continue + + seen.add(normalized_keyword) + prepared_keywords.append(stripped_keyword) + + return prepared_keywords + + +async def search_installed_mcp_tools(keywords: list[str]) -> dict[str, Any]: + """Search safe MCP tool metadata for the tenant in the current request.""" + + prepared_keywords = _prepare_search_keywords(keywords) + if prepared_keywords is None: + return _dump_tool_search_observation( + SearchInstalledMcpToolsErrorObservation(code="invalid_keywords") + ) + + try: + # Keep NL2Agent runtime dependencies out of the MCP server startup path. + from services.nl2agent_service import search_installed_mcp_tools_by_query + + authorization = get_http_request().headers.get("Authorization") + _, tenant_id = get_current_user_id(authorization) + recommendations = search_installed_mcp_tools_by_query( + tenant_id=tenant_id, + query_text=" ".join(prepared_keywords), + ) + except Exception: + logger.exception("Failed to search installed MCP tools from local MCP service") + return _dump_tool_search_observation( + SearchInstalledMcpToolsErrorObservation(code="tool_search_failed") + ) + + return _dump_tool_search_observation( + SearchInstalledMcpToolsObservation( + recommendation_count=len(recommendations), + recommendations=recommendations, + ) + ) + + +async def nl2a_wrapper( + subtype: Literal["local_mcp_recommendation", "agent_draft"], + search_result: dict[str, Any] | None = None, + selected_tool_ids: list[int] | None = None, + language: Literal["en", "zh"] | None = None, + name: str | None = None, + display_name: str | None = None, + description: str | None = None, + duty_prompt: str | None = None, + constraint_prompt: str | None = None, + greeting_message: str | None = None, + example_questions: list[str] | None = None, + selected_tool_names: list[str] | None = None, + few_shot_examples: Nl2aFewShotExamples | None = None, +) -> str: + """Return the NL2Agent JSON template selected by subtype in its wrapper.""" + + return build_nl2a_wrapper( + subtype=subtype, + search_result=search_result, + selected_tool_ids=selected_tool_ids, + language=language, + name=name, + display_name=display_name, + description=description, + duty_prompt=duty_prompt, + constraint_prompt=constraint_prompt, + greeting_message=greeting_message, + example_questions=example_questions, + selected_tool_names=selected_tool_names, + few_shot_examples=few_shot_examples, + ) diff --git a/backend/utils/__init__.py b/backend/utils/__init__.py index e69de29bb2..b22fa7ff26 100644 --- a/backend/utils/__init__.py +++ b/backend/utils/__init__.py @@ -0,0 +1 @@ +# Utils package for Nexent backend diff --git a/backend/utils/a2a_http_client.py b/backend/utils/a2a_http_client.py index 8b7c55d9f1..b5c03c4e92 100644 --- a/backend/utils/a2a_http_client.py +++ b/backend/utils/a2a_http_client.py @@ -22,6 +22,14 @@ CONTENT_TYPE_JSON = "application/json" +class A2AHttpStatusError(Exception): + """Raised when an A2A endpoint returns a non-success HTTP status.""" + + def __init__(self, method: str, url: str, status: int): + super().__init__(f"A2A {method} request to {url} failed with HTTP {status}") + self.status = status + + class A2AHttpClient: """HTTP client for A2A protocol communication.""" @@ -147,6 +155,9 @@ async def get_json( url, headers=request_headers ) + if not 200 <= status < 300: + raise A2AHttpStatusError("GET", url, status) + # Decode body and handle empty responses body_text = body.decode('utf-8') if body else "" @@ -179,7 +190,9 @@ async def post_json( self, url: str, payload: Dict[str, Any], - headers: Optional[Dict[str, str]] = None + headers: Optional[Dict[str, str]] = None, + params: Optional[Dict[str, str]] = None, + cookies: Optional[Dict[str, str]] = None, ) -> Dict[str, Any]: """Send a POST request and return JSON response.""" if not self._session: @@ -195,15 +208,20 @@ async def post_json( if headers: request_headers.update(headers) - logger.info(f"A2A POST request: url={url}, payload={payload}") + logger.info(f"A2A POST request: url={url}") try: status, body = await self._request_with_retry( "POST", url, json=payload, - headers=request_headers + headers=request_headers, + params=params, + cookies=cookies, ) + if not 200 <= status < 300: + raise A2AHttpStatusError("POST", url, status) + # Decode body and handle empty responses body_text = body.decode('utf-8') if body else "" @@ -240,7 +258,9 @@ async def post_stream( self, url: str, payload: Dict[str, Any], - headers: Optional[Dict[str, str]] = None + headers: Optional[Dict[str, str]] = None, + params: Optional[Dict[str, str]] = None, + cookies: Optional[Dict[str, str]] = None, ) -> AsyncIterator[Dict[str, Any]]: """Send a streaming POST request and yield SSE events.""" if not self._session: @@ -250,7 +270,9 @@ async def post_stream( response = await self._session.post( url, json=payload, - headers=headers + headers=headers, + params=params, + cookies=cookies, ) response.raise_for_status() diff --git a/backend/utils/auth_utils.py b/backend/utils/auth_utils.py index 6658f69af0..32e6c89e02 100644 --- a/backend/utils/auth_utils.py +++ b/backend/utils/auth_utils.py @@ -22,6 +22,7 @@ SUPABASE_KEY, SERVICE_ROLE_KEY, DEBUG_JWT_EXPIRE_SECONDS, + JWT_EXPIRY_SECONDS, LANGUAGE, ) from consts.exceptions import LimitExceededError, UnauthorizedError @@ -312,7 +313,7 @@ def get_jwt_expiry_seconds(token: str) -> int: token: JWT token string Returns: - int: Token validity period (seconds), returns default value 3600 if parsing fails + int: Token validity period (seconds), returns configured default if parsing fails """ try: # Speed mode: treat sessions as never expiring @@ -345,7 +346,7 @@ def get_jwt_expiry_seconds(token: str) -> int: return expiry_seconds except Exception as e: logging.warning(f"Failed to get expiration time from token: {str(e)}") - return 3600 # supabase default setting + return JWT_EXPIRY_SECONDS def calculate_expires_at(token: Optional[str] = None) -> int: @@ -362,7 +363,7 @@ def calculate_expires_at(token: Optional[str] = None) -> int: if IS_SPEED_MODE: return int((datetime.now() + timedelta(days=3650)).timestamp()) - expiry_seconds = get_jwt_expiry_seconds(token) if token else 3600 + expiry_seconds = get_jwt_expiry_seconds(token) if token else JWT_EXPIRY_SECONDS return int((datetime.now() + timedelta(seconds=expiry_seconds)).timestamp()) @@ -518,6 +519,26 @@ def get_current_user_id(authorization: Optional[str] = None) -> tuple[str, str]: raise UnauthorizedError("Invalid or expired authentication token") +def get_current_user_context( + authorization: Optional[str] = None, +) -> tuple[str, str, str]: + """Return the authenticated user ID, tenant ID, and normalized role.""" + user_id, tenant_id = get_current_user_id(authorization) + + if IS_SPEED_MODE: + return user_id, tenant_id, "SPEED" + + user_tenant_record = get_user_tenant_by_user_id(user_id) + if not user_tenant_record: + raise UnauthorizedError("User tenant relationship not found") + + user_role = str(user_tenant_record.get("user_role") or "").upper() + if not user_role: + raise UnauthorizedError("User role not found") + + return user_id, resolve_tenant_id_from_user_tenant_record(user_tenant_record), user_role + + def get_user_language(request: Request = None) -> str: """ Get user language preference from request @@ -548,7 +569,9 @@ def get_user_language(request: Request = None) -> str: # --------------------------------------------------------------------------- -def generate_test_jwt(user_id: str, expires_in: int = 3600) -> str: +def generate_test_jwt(user_id: str, expires_in: Optional[int] = None) -> str: + if expires_in is None: + expires_in = JWT_EXPIRY_SECONDS """ Generate a simple unsigned JWT for testing purposes (HS256 with dummy secret) """ @@ -564,8 +587,13 @@ def generate_test_jwt(user_id: str, expires_in: int = 3600) -> str: return jwt.encode(payload, MOCK_JWT_SECRET_KEY, algorithm="HS256") -def generate_session_jwt(user_id: str, expires_in: int = 3600, session_id: str = None) -> str: +def generate_session_jwt( + user_id: str, expires_in: Optional[int] = None, session_id: str = None +) -> str: """Generate a signed JWT compatible with the existing auth verification flow.""" + if expires_in is None: + expires_in = JWT_EXPIRY_SECONDS + now = int(time.time()) payload = { "sub": user_id, diff --git a/backend/utils/automation_tool_prompt.py b/backend/utils/automation_tool_prompt.py new file mode 100644 index 0000000000..d3181cbb78 --- /dev/null +++ b/backend/utils/automation_tool_prompt.py @@ -0,0 +1,40 @@ +from collections.abc import Iterable + + +AUTOMATION_TOOL_NAME = "create_scheduled_task_proposal" + + +def build_automation_tool_policy(language: str, tool_names: Iterable[str]) -> str: + """Build platform policy only when the proposal tool is available.""" + if AUTOMATION_TOOL_NAME not in set(tool_names): + return "" + + if language == "zh": + return ( + "### 定时任务工具策略\n" + "- 当用户明确要求任务在未来、延迟或周期性自动执行时，必须调用 " + "`create_scheduled_task_proposal`，不要立即执行业务动作。\n" + "- `request_text` 必须原样复制用户当前消息中的定时执行请求，不得补充 " + "Agent、工具、知识库、数据源或实现步骤。\n" + "- 创建提案时，`create_scheduled_task_proposal` 必须是本次代码中的唯一工具调用。" + "不要同时调用其他工具或助手。\n" + "- 工具只创建待确认提案。调用后直接用 `final_answer` 返回工具结果，" + "并停止本轮执行。\n" + "- 立即执行的普通请求、询问某个时间的数据、解释时间表达式、" + "事实陈述和个人习惯" + "不要调用此工具。" + ) + + return ( + "### Scheduled-task Tool Policy\n" + "- When the user explicitly asks for a task to run later, after a delay, or repeatedly, " + "call `create_scheduled_task_proposal`. Do not execute the business action now.\n" + "- Copy the scheduling request from the current user message verbatim into `request_text`. " + "Do not add Agent, tool, knowledge-base, data-source, or implementation details.\n" + "- `create_scheduled_task_proposal` must be the only tool call in that code action. " + "Do not call another tool or agent in the same action.\n" + "- The tool creates a pending proposal only. Return its result immediately with " + "`final_answer` and stop the turn.\n" + "- Do not call this tool for immediate requests, questions about data at a time, " + "schedule explanations, factual statements, or personal habits." + ) diff --git a/backend/utils/config_utils.py b/backend/utils/config_utils.py index 2d1c5572be..a199abc6ee 100644 --- a/backend/utils/config_utils.py +++ b/backend/utils/config_utils.py @@ -18,6 +18,7 @@ CONTEXT_SOFT_LIMIT_RATIO_KEY = "context.soft_limit_ratio" +CONTEXT_POLICY_KEY = "context.policy" def safe_value(value): @@ -149,6 +150,24 @@ def get_capacity_reserve_policy(self, tenant_id: str | None = None): f"got {raw_ratio!r}" ) from exc + def get_context_policy(self, tenant_id: str | None = None) -> dict[str, Any] | None: + """Return the tenant's optional JSON context policy without caching it.""" + if tenant_id is None: + logger.warning("No tenant_id specified when getting context policy") + return None + raw_policy = self.load_config(tenant_id).get(CONTEXT_POLICY_KEY) + if raw_policy in (None, ""): + return None + if isinstance(raw_policy, dict): + return raw_policy + try: + parsed = json.loads(raw_policy) + except (TypeError, json.JSONDecodeError) as exc: + raise ValueError(f"{CONTEXT_POLICY_KEY} must be a JSON object") from exc + if not isinstance(parsed, dict): + raise ValueError(f"{CONTEXT_POLICY_KEY} must be a JSON object") + return parsed + def set_single_config(self, user_id: str | None = None, tenant_id: str | None = None, key: str | None = None, value: str | None = None, ): """Set configuration value in database with caching""" diff --git a/backend/utils/context_utils.py b/backend/utils/context_utils.py index 690eaf21e6..f839747152 100644 --- a/backend/utils/context_utils.py +++ b/backend/utils/context_utils.py @@ -1,570 +1,86 @@ -"""Context component building utilities for system prompt assembly. - -Provides build_context_components() to convert agent configuration data -into ContextComponent instances for use with ContextManager. - -This module implements the piecewise component architecture where each -semantic section of the system prompt is emitted by a dedicated function, -allowing ContextManager to assemble them in the correct order. -""" - -from typing import TYPE_CHECKING, Any, Dict, List, Optional - -if TYPE_CHECKING: - from nexent.core.agents.agent_model import ( - ContextComponent, - ToolsComponent, - SkillsComponent, - MemoryComponent, - KnowledgeBaseComponent, - ManagedAgentsComponent, - ExternalAgentsComponent, - SystemPromptComponent, - ToolConfig, - AgentConfig, - ExternalA2AAgentConfig, +"""Build authorized, serializable context item inputs for an agent run.""" + +from typing import Any, Dict, List, Optional + +from nexent.core.agents.context import ContextItemInput, ContextItemType +from nexent.core.agents.context_input import ContextInput + +from consts.const import MESSAGE_ROLE + + +def build_authorized_context_input( + agent_run_info, + historical_context=None, +) -> ContextInput: + """Freeze configured context and authorized history into one item snapshot.""" + if historical_context is None: + fallback_turns = [] + pending_user = None + for index, entry in enumerate(agent_run_info.history or ()): + if entry.role == MESSAGE_ROLE["USER"]: + pending_user = (index, entry) + elif ( + entry.role == MESSAGE_ROLE["ASSISTANT"] + and pending_user is not None + ): + user_index, user_entry = pending_user + fallback_turns.append({ + "user_message": user_entry.content, + "assistant_final_answer": entry.content, + "attachments": [], + "user_message_id": -(user_index + 1), + "assistant_message_id": -(index + 1), + }) + pending_user = None + historical_context = {"conversation_turns": fallback_turns} + + history_items = [] + summary = historical_context.get("history_summary") + if summary: + history_items.append(ContextItemInput( + id=f"history_summary:{summary['unit_id']}", + type="history_summary", + content=summary, + source=("conversation_history",), + )) + for order, turn in enumerate( + historical_context.get("conversation_turns", ()) + ): + history_items.append(ContextItemInput( + id=( + f"conversation_turn:{turn['user_message_id']}:" + f"{turn['assistant_message_id']}" + ), + type="conversation_turn", + content=turn, + source=("conversation_history",), + metadata={"layout_order": order}, + )) + return ContextInput( + items=( + tuple(agent_run_info.agent_config.context_items or ()) + + tuple(history_items) + ), ) - # ============================================================================= # SECTION 1: Long-text format functions (expanded from Jinja2 templates) # Each function accepts language and is_manager params for variant-specific text # ============================================================================= -def _format_memory_context( - memory_list: List[Any], - language: str = "zh", -) -> str: - """Format memory search results with full usage guidelines. - - Jinja2 templates have ~30 lines of "记忆使用准则" text that must be - included here for semantic equivalence. - """ - if not memory_list: - return "" - - # Group memories by level in correct order: tenant, user_agent, user, agent - level_order = ["tenant", "user_agent", "user", "agent"] - memory_by_level: Dict[str, List[Any]] = {} - for mem in memory_list: - if isinstance(mem, dict): - level = mem.get("memory_level", "user") - if level not in memory_by_level: - memory_by_level[level] = [] - memory_by_level[level].append(mem) - - lines = [] - - if language == "zh": - lines.append("### 上下文记忆") - lines.append("基于之前的交互记录，以下是按作用域和重要程度排序的最相关记忆：") - lines.append("") - - for level in level_order: - if level in memory_by_level: - level_title = { - "tenant": "Tenant", - "user_agent": "User_agent", - "user": "User", - "agent": "Agent", - }.get(level, level.title()) - lines.append(f"**{level_title} 层级记忆：**") - for item in memory_by_level[level]: - content = item.get("memory", "") or item.get("content", "") - score = item.get("score", 0.0) - lines.append(f"- {content} `({score:.2f})`") - lines.append("") - - lines.append("**记忆使用准则：**") - lines.append("1. **冲突处理优先级**：当记忆信息存在矛盾时，严格按以下顺序处理：") - lines.append("- **最优先**：在上述列表中位置靠前的记忆具有优先权") - lines.append("- **次优先**：当前对话内容与记忆直接冲突时，以当前对话为准") - lines.append("- **次优先**：相关度分数越高，表示记忆越可信") - lines.append("") - lines.append("2. **记忆整合最佳实践**：") - lines.append(" - 自然地将相关记忆融入回答中，避免显式使用\"根据记忆\"、\"根据上下文\"或\"根据交互记忆\"等语言") - lines.append(" - 利用记忆信息调整回答的语调、方式和技术深度以适应用户") - lines.append(" - 让记忆指导您对用户偏好和上下文的理解") - lines.append("") - lines.append("3. **级别特定说明**：") - lines.append(" - **tenant（租户级）**：组织层面的约束和政策（不可违背）") - lines.append(" - **user_agent（用户-代理级）**：特定用户在代理中的交互模式和既定工作流程") - lines.append(" - **user（用户级）**：用户的个人偏好、技能水平和历史上下文") - lines.append(" - **agent（代理级）**：您的既定行为模式和能力特征，通常对所有用户共享（重要性最低）") - else: - lines.append("### Contextual Memory") - lines.append("Based on previous interactions, here are the most relevant memories organized by scope and importance:") - lines.append("") - - for level in level_order: - if level in memory_by_level: - lines.append(f"**{level.title()} Level Memory:**") - for item in memory_by_level[level]: - content = item.get("memory", "") or item.get("content", "") - score = item.get("score", 0.0) - lines.append(f"- {content} `({score:.2f})`") - lines.append("") - - lines.append("**Memory Usage Guidelines:**") - lines.append("1. **Conflict Resolution Priority**: When memories contradict each other, follow this strict order:") - lines.append(" - **Primary**: Information appearing EARLIER in the above numbered list takes precedence") - lines.append(" - **Secondary**: Current conversation context overrides historical memory when directly contradicted") - lines.append(" - **Tertiary**: Higher relevance scores indicate more trustworthy information") - lines.append("") - lines.append("2. **Memory Integration Best Practices**:") - lines.append(" - Seamlessly weave relevant memories into your responses without explicitly saying \"I remember\", \"based on memory\" or \"based on context\"") - lines.append(" - Use memories to inform your tone, approach, and technical level appropriate for this user") - lines.append(" - Let memories guide your assumptions about user preferences and context") - lines.append("") - lines.append("3. **Level-Specific Considerations**:") - lines.append(" - **tenant**: Organizational constraints and policies (non-negotiable)") - lines.append(" - **user_agent**: Specific interaction dynamics and established workflow patterns") - lines.append(" - **user**: Individual preferences, skills, and historical context") - lines.append(" - **agent**: Your established behavioral patterns and capabilities, usually shared by all users (least important)") - - return "\n".join(lines) - - -def _format_skills_description( - skills: List[Dict[str, str]], - language: str = "zh", -) -> str: - """Format skill descriptions with full 6-step usage process. - - Jinja2 templates have ~50 lines of "技能使用流程" text that must be - included here for semantic equivalence. - """ - if not skills: - return "" - - lines = [] - - # Build the block - skills_block_lines = [""] - for skill in skills: - name = skill.get("name", "") - desc = skill.get("description", "") - skills_block_lines.append(" ") - skills_block_lines.append(f" {name}") - skills_block_lines.append(f" {desc}") - skills_block_lines.append(" ") - skills_block_lines.append("") - skills_block = "\n".join(skills_block_lines) - - if language == "zh": - lines.append("### 可用技能") - lines.append("") - lines.append("你拥有以下技能（Skills）。技能是预定义的专业能力模块，包含详细执行指南和可选的附加脚本。") - lines.append("") - lines.append(skills_block) - lines.append("") - lines.append("**技能使用流程**：") - lines.append("1. 收到用户请求后，首先审视 `` 中每个技能的 description，判断是否有匹配的技能。") - lines.append("2. **加载技能**：根据不同场景选择读取方式：") - lines.append(" - **首次加载**：调用 `read_skill_md(\"skill_name\")` 读取技能的完整执行指南（默认读取 SKILL.md）") - lines.append(" - **精确读取**：如只需特定文件（如示例、参考文档），可指定 additional_files：") - lines.append(" ") - lines.append(" skill_content = read_skill_md(\"skill_name\", [\"examples.md\", \"reference/api_doc\"])") - lines.append(" print(skill_content)") - lines.append(" ") - lines.append(" 注意：当 additional_files 非空时，默认不再自动读取 SKILL.md，如需同时读取请显式指定。") - lines.append("") - lines.append(" - **加载技能配置**：如果技能需要读取配置变量，可先调用 `read_skill_config(\"skill_name\")` 读取配置字符串，通过 `json.loads` 方法转化为配置字典，再从中获取所需值：") - lines.append(" ") - lines.append(" import json") - lines.append(" config = json.loads(read_skill_config(\"skill_name\"))") - lines.append(" # 返回示例: {\"key_a\": {\"key2\": \"value2\"}, \"others\": {...}}") - lines.append(" value = config[\"key1\"][\"key2\"]") - lines.append(" print(value)") - lines.append(" ") - lines.append("") - lines.append("3. **遵循技能指南**：技能内容注入后，严格按其中的步骤执行。不要跳过技能指南中的步骤，也不要用自行编写的代码替代技能定义的流程。") - lines.append("") - lines.append("4. **执行技能脚本**：技能中引用的脚本（参考文档、脚本声明）可通过以下任一形式表达，**功能上完全等同**，模型必须把它们都识别为路径声明：") - lines.append(" - XML 标签形式：``、``") - lines.append(" - 单个反引号包裹：`` `scripts/analyze.py` ``、`` `reference/api_doc` ``") - lines.append(" - 三重反引号代码块：`` ```scripts/analyze.py``` ``（当代码块内仅有单行路径时）") - lines.append(" 调用 `run_skill_script` 时，`script_path` **始终相对于技能根目录**解析（平台行为，不是当前工作目录），常见形式如下：") - lines.append(" ") - lines.append(" result = run_skill_script(\"skill_name\", \"script_path\")") - lines.append(" print(result)") - lines.append(" ") - lines.append(" 对于需要附加参数的脚本，需要参照脚本调用说明，将参数直接以字符串形式传递。") - lines.append(" 例如对于希望附加的参数：--param1 value1 --flag，则使用以下格式调用run_skill_script：") - lines.append(" ") - lines.append(" result = run_skill_script(\"skill_name\", \"script_path\", \"--param1 value1 --flag\")") - lines.append(" print(result)") - lines.append(" ") - lines.append(" 注意：") - lines.append(" - 只执行技能指南中明确声明的脚本路径，绝不自行构造脚本路径。") - lines.append(" - 不要把脚本当作当前工作目录（CWD）的相对路径处理；也不要使用绝对路径。") - lines.append(" - 当脚本不存在时，返回的错误信息中会列出该技能根目录下的可用脚本，请据此修正路径。") - lines.append("") - lines.append("5. **整合输出**：根据技能指南要求的输出格式，结合脚本执行结果生成最终回答。") - lines.append("") - lines.append("6. **引用场景处理**：技能中的引用既可以通过 XML 标签表达，也可以通过下列 Markdown 语法表达，**功能上完全等同**，必须都能识别：") - lines.append(" - **引用模板识别**：") - lines.append(" - XML 形式：``") - lines.append(" - 单个反引号形式：`` `examples.md` ``、`` `reference/api_doc` ``") - lines.append(" - 三重反引号代码块形式：`` ```examples.md``` ``（当代码块内仅有单行路径时）") - lines.append(" - 自然语言式的引用声明（如\"详见 examples.md\"、\"请参考 reference/api_doc\"）") - lines.append(" - **自动补全**：发现引用后，按需调用 `read_skill_md(\"skill_name\", [\"<路径>\"])` 读取被引用的文件，**不要一次性全部读取**，应基于当前任务判断哪些文件确实必要。") - lines.append(" - **示例**：") - lines.append(" ") - lines.append(" # 技能内容提示\"请参考 examples.md 获取详细示例\"") - lines.append(" additional_info = read_skill_md(\"skill_name\", [\"examples.md\"])") - lines.append(" print(additional_info)") - lines.append(" ") - else: - lines.append("### Available Skills") - lines.append("") - lines.append("You have the following Skills. Skills are predefined professional capability modules with detailed execution guides and optional additional scripts.") - lines.append("") - lines.append(skills_block) - lines.append("") - lines.append("**Skill Usage Process**:") - lines.append("1. After receiving a user request, first examine the description of each skill in `` to determine if there is a matching skill.") - lines.append("2. **Load Skill**: Choose the appropriate reading method based on the scenario:") - lines.append(" - **First-time load**: Call `read_skill_md(\"skill_name\")` to read the complete execution guide (defaults to reading SKILL.md)") - lines.append(" - **Precise read**: If you only need specific files (like examples, reference docs), specify additional_files:") - lines.append(" ") - lines.append(" skill_content = read_skill_md(\"skill_name\", [\"examples.md\", \"reference/api_doc\"])") - lines.append(" print(skill_content)") - lines.append(" ") - lines.append(" Note: When additional_files is non-empty, SKILL.md is no longer auto-read. If you need both, explicitly specify it.") - lines.append("") - lines.append(" - **Load skill config**: If the skill needs configuration variables, call `read_skill_config(\"skill_name\")` to read the config string, convert to dict via `json.loads`, then access values:") - lines.append(" ") - lines.append(" import json") - lines.append(" config = json.loads(read_skill_config(\"skill_name\"))") - lines.append(" # Example: {\"key_a\": {\"key2\": \"value2\"}, \"others\": {...}}") - lines.append(" value = config[\"key1\"][\"key2\"]") - lines.append(" print(value)") - lines.append(" ") - lines.append("") - lines.append("3. **Follow Skill Guide**: After skill content is injected, strictly follow its steps. Do not skip steps or replace with your own code.") - lines.append("") - lines.append("4. **Execute Skill Script**: Skill-internal references (for both documentation and scripts) may be declared with **any of the following equivalent forms** - treat them all the same way: ") - lines.append(" - XML tags: ``, ``") - lines.append(" - Single inline backticks: `` `scripts/analyze.py` ``, `` `reference/api_doc` ``") - lines.append(" - Triple-backtick fenced code blocks: `` ```scripts/analyze.py``` `` (only when the block body is a single path line)") - lines.append(" When calling `run_skill_script`, the `script_path` is **always resolved relative to the skill's root directory** (this is the platform behaviour, not the agent's CWD). Common forms:") - lines.append(" ") - lines.append(" result = run_skill_script(\"skill_name\", \"script_path\")") - lines.append(" print(result)") - lines.append(" ") - lines.append(" For scripts needing extra params, pass them as a command-line string per the script's calling instructions.") - lines.append(" Example for --param1 value1 --flag:") - lines.append(" ") - lines.append(" result = run_skill_script(\"skill_name\", \"script_path\", \"--param1 value1 --flag\")") - lines.append(" print(result)") - lines.append(" ") - lines.append(" Note: Only execute script paths explicitly declared in the skill guide. Never construct paths yourself. Do not treat the script as relative to the current working directory (CWD), and never pass absolute paths. When the requested script cannot be found, the error returned by `run_skill_script` lists the scripts that *do* exist under the skill root - use it to correct the path.") - lines.append("") - lines.append("5. **Integrate Output**: Generate the final answer based on the skill guide's output format and script execution results.") - lines.append("") - lines.append("6. **Handle References**: Skill-internal references can be expressed using XML tags, markdown forms, or natural-language hints. All three are **functionally equivalent** and must be recognised: ") - lines.append(" - **Reference patterns to recognise**:") - lines.append(" - XML tag form: ``") - lines.append(" - Single inline backtick form: `` `examples.md` ``, `` `reference/api_doc` ``") - lines.append(" - Triple-backtick fenced block form: `` ```examples.md``` `` (only when the block body is a single path line)") - lines.append(" - Natural-language references (\"see examples.md\", \"refer to reference/api_doc\")") - lines.append(" - **Auto-complete**: After discovering a reference, call `read_skill_md(\"skill_name\", [\"\"])` only for the files you actually need. Do **not** load every referenced file blindly - decide based on the current task which references matter.") - lines.append(" - **Example**:") - lines.append(" ") - lines.append(" # Skill content says \"see examples.md for detailed examples\"") - lines.append(" additional_info = read_skill_md(\"skill_name\", [\"examples.md\"])") - lines.append(" print(additional_info)") - lines.append(" ") - - return "\n".join(lines) - - -def _format_tools_description( - tools: Dict[str, Any], - language: str = "zh", - is_manager: bool = True, -) -> str: - """Format tool descriptions with file URL usage guide. - - Jinja2 templates have ~10 lines of "文件链接使用指南" text that must be - included here for semantic equivalence. - - Note: Managed agents use different presigned_url guidance than manager agents. - """ - if not tools: - no_tools_msg = "- 当前没有可用的工具" if language == "zh" else "- No tools are currently available" - prefix = "1. 工具\n" if language == "zh" else "1. Tools\n" - return prefix + no_tools_msg - - lines = [] - - if language == "zh": - lines.append("1. 工具") - else: - lines.append("1. Tools") - - if language == "zh": - lines.append("- 你只能使用以下工具，不得使用任何其他工具：") - else: - lines.append("- You can only use the following tools and may not use any other tools:") - - for name, tool in tools.items(): - if hasattr(tool, 'description'): - desc = tool.description - inputs = tool.inputs - output_type = tool.output_type - source = getattr(tool, 'source', 'local') - else: - desc = tool.get('description', '') - inputs = tool.get('inputs', '') - output_type = tool.get('output_type', '') - source = tool.get('source', 'local') - - # MCP tools have [MCP] prefix - if source == 'mcp': - if language == "zh": - lines.append(f"- [MCP] {name}: {desc}") - lines.append(f" 接受输入: {inputs}") - lines.append(f" 返回输出类型: {output_type}") - else: - lines.append(f"- [MCP] {name}: {desc}") - lines.append(f" Accepts input: {inputs}") - lines.append(f" Returns output type: {output_type}") - else: - if language == "zh": - lines.append(f"- {name}: {desc}") - lines.append(f" 接受输入: {inputs}") - lines.append(f" 返回输出类型: {output_type}") - else: - lines.append(f"- {name}: {desc}") - lines.append(f" Accepts input: {inputs}") - lines.append(f" Returns output type: {output_type}") - - # File URL usage guide - lines.append("") - if language == "zh": - lines.append("### 文件链接使用指南") - lines.append("当处理用户上传的文件时，请根据工具类型选择正确的 URL：") - lines.append("1. **调用标记为 [MCP] 的工具**（外部工具，运行在 Nexent 之外）：") - if is_manager: - lines.append(" → 使用 **Download URL**（格式：`https://minio.example.com/...?token=xxx`）") - lines.append(" 原因：MCP 工具运行在外部服务，无法访问内部 S3 存储") - else: - lines.append(" → 使用 **presigned_url**（已包含代理前缀，格式：`http://.../api/nb/v1/file/fetch?presigned_url=...`）") - lines.append(" 直接使用用户上传文件信息中提供的 **presigned_url** 字段，无需拼接。") - lines.append("2. **调用其他所有工具**（内部工具，如 analyze_text_file、analyze_image 等）：") - lines.append(" → 使用 **S3 URL**（格式：`s3:/nexent/attachments/xxx.pdf`）") - lines.append(" 原因：内部工具运行在 Nexent 内部，可以直接访问 MinIO 存储") - else: - lines.append("### File URL Usage Guide") - lines.append("When processing user-uploaded files, choose the correct URL based on tool type:") - lines.append("1. **Calling tools marked with [MCP]** (external tools that run outside Nexent):") - if is_manager: - lines.append(" → Use **Download URL** (format: `https://minio.example.com/...?token=xxx`)") - lines.append(" Reason: MCP tools run on external services and cannot access internal S3 storage") - else: - lines.append(" → Use **presigned_url** (already includes proxy prefix, format: `http://.../api/nb/v1/file/fetch?presigned_url=...`)") - lines.append(" Directly use the **presigned_url** field provided in the user's uploaded file info. No need to construct or append anything.") - lines.append("2. **Calling all other tools** (internal tools like analyze_text_file, analyze_image):") - lines.append(" → Use **S3 URL** (format: `s3:/nexent/attachments/xxx.pdf`)") - lines.append(" Reason: Internal tools run inside Nexent and can directly access MinIO storage") - - return "\n".join(lines) - - -def _format_managed_agents_description( - managed_agents: Dict[str, Any], - language: str = "zh", -) -> str: - """Format managed sub-agent descriptions with calling specifications. - - Jinja2 templates have ~15 lines of "内部助手调用规范" text that must be - included here for semantic equivalence. - """ - if not managed_agents: - return "" - - lines = [] - - if language == "zh": - lines.append("2. 助手") - else: - lines.append("2. Agents") - - if language == "zh": - lines.append("你可以使用以下内部助手（通过函数调用方式协作）：") - for name, agent in managed_agents.items(): - desc = agent.description if hasattr(agent, 'description') else agent.get('description', '') - lines.append(f" - {name}: {desc}") - lines.append("") - lines.append("内部助手调用规范：") - lines.append(" 1. 调用方式：") - lines.append(" - 接受输入：{\"task\": {\"type\": \"string\", \"description\": \"任务描述\"}}") - lines.append(" - 返回输出类型：{\"type\": \"string\", \"description\": \"执行结果\"}") - lines.append(" 2. 使用策略：") - lines.append(" - 任务分解：单次调用中不要让助手一次做过多的事情，任务拆分是你的工作，你需要将复杂任务分解为可管理的子任务") - lines.append(" - 专业匹配：根据助手的专长分配任务") - lines.append(" - 信息整合：整合不同助手的输出生成连贯解决方案") - lines.append(" - 效率优化：避免重复工作") - lines.append(" 3. 协作要求：") - lines.append(" - 评估助手返回的结果") - lines.append(" - 必要时提供额外指导或重新分配任务") - lines.append(" - 在助手结果基础上进行工作，避免重复工作") - lines.append(" - 注意保留子助手回答中的特殊符号，如索引溯源信息等") - else: - lines.append("You can use the following internal agents (via function calls):") - for name, agent in managed_agents.items(): - desc = agent.description if hasattr(agent, 'description') else agent.get('description', '') - lines.append(f" - {name}: {desc}") - lines.append("") - lines.append("Internal agent calling specifications:") - lines.append(" 1. Calling method:") - lines.append(" - Accepts input: {\"task\": {\"type\": \"string\", \"description\": \"task description\"}}") - lines.append(" - Returns output type: {\"type\": \"string\", \"description\": \"execution result\"}") - lines.append(" 2. Usage strategy:") - lines.append(" - Task decomposition: Don't let agents do too many things in a single call, task breakdown is your job, you need to decompose complex tasks into manageable subtasks") - lines.append(" - Professional matching: Assign tasks based on agent expertise") - lines.append(" - Information integration: Integrate outputs from different agents to generate coherent solutions") - lines.append(" - Efficiency optimization: Avoid duplicate work") - lines.append(" 3. Collaboration requirements:") - lines.append(" - Evaluate agent returned results") - lines.append(" - Provide additional guidance or reassign tasks when necessary") - lines.append(" - Work based on agent results, avoid duplicate work") - lines.append(" - Pay attention to preserving special symbols in sub-agent answers, such as index traceability information") - - return "\n".join(lines) - - -def _format_external_agents_description( - external_a2a_agents: Dict[str, Any], - language: str = "zh", -) -> str: - """Format external A2A agent descriptions with calling specifications. - - Jinja2 templates have ~5 lines of "外部助手调用规范" text that must be - included here for semantic equivalence. - """ - if not external_a2a_agents: - return "" - - lines = [] - - if language == "zh": - lines.append("你还可以使用以下外部助手（通过 A2A 协议远程调用）：") - for agent_id, agent in external_a2a_agents.items(): - name = agent.name if hasattr(agent, 'name') else agent.get('name', '') - desc = agent.description if hasattr(agent, 'description') else agent.get('description', '') - lines.append(f" - {name}: {desc}") - lines.append("") - lines.append("外部助手调用规范：") - lines.append(" 1. 调用格式：`agent_name(task=\"自然语言任务描述\")`，注意：只需要 task 参数，不需要其他参数") - lines.append(" 2. 例如：`tool_assistant(task=\"北京天气怎么样\")`") - lines.append(" 3. 任务描述使用自然语言，让外部助手自动识别和处理") - else: - lines.append("You can also use the following external agents (called via A2A protocol remotely):") - for agent_id, agent in external_a2a_agents.items(): - name = agent.name if hasattr(agent, 'name') else agent.get('name', '') - desc = agent.description if hasattr(agent, 'description') else agent.get('description', '') - lines.append(f" - {name}: {desc}") - lines.append("") - lines.append("External agent calling specifications:") - lines.append(" 1. Call format: `agent_name(task=\"natural language task description\")`, NOTE: only task parameter is needed, no other parameters") - lines.append(" 2. Example: `tool_assistant(task=\"What's the weather in Beijing?\")`") - lines.append(" 3. Use natural language for task description, let the external agent handle the rest") - - return "\n".join(lines) - - -def _format_skills_usage_requirements( - skills: List[Dict[str, str]], - language: str = "zh", - is_manager: bool = True, -) -> str: - """Format skills usage requirements section. - - This is the "技能使用要求" section that appears after the skills reference - in the Available Resources section. - """ - if not skills: - no_skills_msg = "- 当前没有可用的技能" if language == "zh" else "- No skills are currently available" - prefix = "3. 技能\n" if language == "zh" else "3. Skills\n" - return prefix + no_skills_msg - - lines = [] - - if language == "zh": - lines.append("3. 技能") - else: - lines.append("3. Skills") - - if language == "zh": - lines.append("- 你拥有上述 `` 中列出的技能。技能中引用的脚本通过 `run_skill_script()` 函数调用，该函数由平台提供，不需要导入。") - lines.append("") - lines.append("### 技能使用要求") - lines.append("1. **技能优先**：如果用户请求匹配了某个技能的 description，必须先调用 `read_skill_md()` 加载技能指南，再按指南执行。不得跳过技能自行编写代码解决。") - lines.append("2. **忠实执行**：读取技能内容后，严格按技能指南中的步骤操作。不要自行修改流程、跳过步骤或用通用代码替代技能定义的流程。") - lines.append("3. **脚本调用规范**：") - lines.append(" - 路径声明识别：技能指南中的脚本路径既可以以 XML 标签（``）声明，也允许以等价的 Markdown 形式（`` `scripts/foo.py` `` 单反引号，或 `` ```scripts/foo.py``` `` 三重反引号代码块）声明。模型必须把这些形式都识别为脚本路径。") - lines.append(" - 路径解析：`run_skill_script` 的 `script_path` 参数**始终相对于技能根目录**解析，平台不会基于当前工作目录或绝对路径查找。请直接复用技能指南中的声明字符串，不要自行拼接或猜测路径。") - lines.append(" - 参数传递：如果需要附加参数，将参数以命令行字符串形式传递给 `run_skill_script`。") - lines.append(" - 错误回退：脚本不存在时，`run_skill_script` 返回的错误信息会列出当前技能根目录下可用的脚本路径，请据此修正。") - lines.append("4. **失败回退**：如果 `read_skill_md` 返回错误或 `run_skill_script` 执行失败，向用户说明情况，并尝试用通用推理模式提供替代方案。") - lines.append("5. **技能组合**：如果一个任务需要多个技能配合，按逻辑依赖顺序依次加载和执行，前一个技能的输出可作为后一个技能的输入。") - else: - lines.append("- You have the skills listed in `` above. Scripts referenced in skills are called via the `run_skill_script()` function, which is provided by the platform and does not need to be imported.") - lines.append("") - lines.append("### Skill Usage Requirements") - lines.append("1. **Skill Priority**: If a user request matches a skill's description, you must first call `read_skill_md()` to load the skill guide, then execute per the guide. Do not skip skills and write your own code.") - lines.append("2. **Faithful Execution**: After reading skill content, strictly follow the skill guide's steps. Do not modify the flow, skip steps, or replace with generic code.") - lines.append("3. **Script Calling Specification**:") - lines.append(" - **Path declaration recognition**: A script path inside the skill guide may be declared using XML tags (``) OR via the equivalent markdown forms - single inline backticks like `` `scripts/foo.py` ``, or triple-backtick fenced blocks like `` ```scripts/foo.py``` ``. Treat all three as the same kind of declaration.") - lines.append(" - **Path resolution**: The `script_path` argument of `run_skill_script` is **always resolved relative to the skill's root directory**. The platform will not look in the current working directory and will not follow absolute paths. Pass the path verbatim from the skill guide - never construct or guess a path.") - lines.append(" - **Parameter passing**: For extra parameters, pass them as a command-line string to `run_skill_script`.") - lines.append(" - **Error fallback**: When the script cannot be located, the error returned by `run_skill_script` lists the scripts that *do* exist under the skill root - use it to correct the path.") - lines.append("4. **Failure Fallback**: If `read_skill_md` returns an error or `run_skill_script` fails, explain to the user and try to provide an alternative via general reasoning mode.") - lines.append("5. **Skill Combination**: If a task needs multiple skills, load and execute in logical dependency order. The output of one skill can be input to the next.") - - return "\n".join(lines) - - -def _format_agent_fallback( - managed_agents: Dict[str, Any], - external_a2a_agents: Dict[str, Any], - language: str = "zh", -) -> str: - """Format fallback message when no agents are available.""" - if managed_agents or external_a2a_agents: - return "" - - return "- 当前没有可用的助手" if language == "zh" else "- No agents are currently available" - - -def _format_app_context(app_name: str, app_description: str, user_id: str) -> str: - """Format application context for system prompt injection.""" - lines = [ - f"Application: {app_name}", - f"Description: {app_description}", - f"Current user: {user_id}", - ] - return "\n".join(lines) - - -# ============================================================================= -# SECTION 2: Skeleton component builders -# These build SystemPromptComponent instances for fixed text sections +# SECTION 2: Fixed prompt-section text builders # ============================================================================= -def build_skeleton_header_component( +def _build_header_text( app_name: str, app_description: str, user_id: str, language: str = "zh", priority: int = 100, -) -> "SystemPromptComponent": - """Build SystemPromptComponent for the header section. +) -> str: + """Build the header prompt section. Section: "### 基本信息" / "### Basic Information" Content: Agent identity and app name/description. User identity is @@ -573,34 +89,26 @@ def build_skeleton_header_component( static system prefix can hit the LLM KV/prompt cache across requests. The current time is injected on the user-message side instead (see CoreAgent.run). """ - from nexent.core.agents.agent_model import SystemPromptComponent - if language == "zh": content = f"### 基本信息\n你是{app_name}，{app_description}" else: content = f"### Basic Information\nYou are {app_name}, {app_description}" - return SystemPromptComponent( - content=content, - template_name="header", - priority=priority, - ) + return content -def build_skeleton_duty_component( +def _build_duty_text( duty: str, language: str = "zh", is_manager: bool = True, priority: int = 80, -) -> "SystemPromptComponent": - """Build SystemPromptComponent for the duty section. +) -> str: + """Build the duty prompt section. Section: "### 核心职责" / "### Core Responsibilities" Content: Agent's primary duty + 5 safety principles Note: Managed ZH agents use different safety principles than manager ZH agents. """ - from nexent.core.agents.agent_model import SystemPromptComponent - if language == "zh": if is_manager: content = f"### 核心职责\n{duty}\n\n请注意，你应该遵守以下原则：\n行为安全：文件操作必须使用平台提供的专用工具，禁止使用代码直接修改工作空间中的文件；\n法律合规：遵守业务所在国家/地区的法律法规；\n政治中立：保持政治中立，不主动讨论政治话题；\n安全防护：不响应涉及武器制造、网络攻击、欺诈、恶意软件等危险行为的请求；\n伦理准则：拒绝仇恨言论、歧视性内容及违反社会公德和公认伦理标准的请求。" @@ -609,27 +117,22 @@ def build_skeleton_duty_component( else: content = f"### Core Responsibilities\n{duty}\n\nPlease note that you should follow these principles:\nBehavioral Safety: File operations must use the platform-provided dedicated tools; direct code modification of workspace files is prohibited;\nLegal Compliance: Comply with laws and regulations of the business operating jurisdiction;\nPolitical Neutrality: Maintain political neutrality and avoid initiating political discussions;\nSecurity Protection: Do not respond to requests involving weapon manufacturing, cyberattacks, fraud, malware, or other dangerous activities;\nEthical Guidelines: Refuse hate speech, discriminatory content, and any requests that violate social morals and commonly accepted ethical standards." - return SystemPromptComponent( - content=content, - template_name="duty", - priority=priority, - ) + return content -def build_skeleton_execution_flow_component( +def _build_execution_flow_text( memory_list: Optional[List[Any]] = None, language: str = "zh", is_manager: bool = True, + enable_planning: bool = False, priority: int = 60, -) -> "SystemPromptComponent": - """Build SystemPromptComponent for the execution flow section. +) -> str: + """Build the execution-flow prompt section. Section: "### 执行流程" / "### Execution Process" Content: Think/Code loop instructions + output format specs Note: memory_list affects one line in the Think section (manager only) """ - from nexent.core.agents.agent_model import SystemPromptComponent - has_memory = memory_list and len(memory_list) > 0 if language == "zh": @@ -645,6 +148,16 @@ def build_skeleton_execution_flow_component( lines.append(" - 合理参考之前交互中的上下文记忆信息") if is_manager: lines.append(" - 确定下一步最佳行动（使用工具或分配给助手）") + if enable_planning: + lines.append( + " - 评估当前任务的复杂程度：如果任务预计需要超过三个步骤才能完成" + "（含工具调用、助手调用或中间判断），请在第一次行动前调用 create_plan" + " 工具创建执行计划；简单任务可直接执行，无需创建计划" + ) + lines.append( + " - create_plan 的 steps 列表至少包含 3 个步骤（推荐不超过 8 个），" + "每个步骤需要提供稳定的 id（step-1、step-2、...）、简短标题和详细描述" + ) lines.append(" - 解释你的决策逻辑和预期结果") lines.append("") lines.append("2. 代码：") @@ -701,6 +214,18 @@ def build_skeleton_execution_flow_component( lines.append(" - Reference relevant contextual memories from previous interactions when applicable") if is_manager: lines.append(" - Determine the best next action (use tools or delegate to agents)") + if enable_planning: + lines.append( + " - Assess task complexity: if the task is expected to take more than three" + " steps to complete (including tool calls, agent handoffs, or intermediate" + " decisions), call create_plan before the first action; simple tasks can" + " proceed directly without a plan" + ) + lines.append( + " - The steps list passed to create_plan must contain at least 3 steps" + " (recommended max 8); each step needs a stable id (step-1, step-2, ...)," + " a short title, and a detailed description" + ) lines.append(" - Explain your decision logic and expected results") lines.append("") lines.append("2. Code:") @@ -747,49 +272,37 @@ def build_skeleton_execution_flow_component( content = "\n".join(lines) - return SystemPromptComponent( - content=content, - template_name="execution_flow", - priority=priority, - ) + return content -def build_skeleton_constraint_component( +def _build_constraint_text( constraint: str, language: str = "zh", priority: int = 30, -) -> "SystemPromptComponent": - """Build SystemPromptComponent for the constraint section. +) -> str: + """Build the constraint prompt section. Section: "### 资源使用要求" / "### Resource Usage Requirements" Content: User-defined constraint text """ - from nexent.core.agents.agent_model import SystemPromptComponent - if language == "zh": content = f"### 资源使用要求\n{constraint}" else: content = f"### Resource Usage Requirements\n{constraint}" - return SystemPromptComponent( - content=content, - template_name="constraint", - priority=priority, - ) + return content -def build_skeleton_code_norms_component( +def _build_code_norms_text( language: str = "zh", is_manager: bool = True, priority: int = 20, -) -> "SystemPromptComponent": - """Build SystemPromptComponent for the Python code norms section. +) -> str: + """Build the Python code-norms prompt section. Section: "### python代码规范" / "### Python Code Specifications" Content: 12 fixed code rules (11 for managed agents) """ - from nexent.core.agents.agent_model import SystemPromptComponent - if language == "zh": lines = ["### python代码规范"] lines.append("1. 如果认为是需要执行的代码，使用'代码'格式；如果是不需要执行仅用于展示的代码，使用'代码'格式，其中语言类型例如python、java、javascript等；") @@ -821,48 +334,36 @@ def build_skeleton_code_norms_component( content = "\n".join(lines) - return SystemPromptComponent( - content=content, - template_name="code_norms", - priority=priority, - ) + return content -def build_skeleton_footer_component( +def _build_footer_text( few_shots: str, language: str = "zh", priority: int = 10, -) -> "SystemPromptComponent": - """Build SystemPromptComponent for the footer section. +) -> str: + """Build the footer prompt section. Section: "### 示例模板" + ending Content: few_shots + "$1M reward" ending """ - from nexent.core.agents.agent_model import SystemPromptComponent - if language == "zh": content = f"### 示例模板\n{few_shots}\n\n现在开始！如果你正确解决任务，你将获得100万美元的奖励。" else: content = f"### Example Templates\n{few_shots}\n\nNow start! If you solve the task correctly, you will receive a reward of 1 million dollars." - return SystemPromptComponent( - content=content, - template_name="footer", - priority=priority, - ) + return content -def build_available_resources_header_component( +def _build_available_resources_header_text( is_manager: bool = True, language: str = "zh", priority: int = 55, -) -> "SystemPromptComponent": - """Build SystemPromptComponent for the Available Resources section header. +) -> str: + """Build the available-resources prompt heading. Manager agents get a preamble restricting resources; managed agents get only the heading. """ - from nexent.core.agents.agent_model import SystemPromptComponent - if language == "zh": if is_manager: content = "### 可用资源\n你只能使用以下资源，不得使用任何其他工具或助手：" @@ -874,350 +375,10 @@ def build_available_resources_header_component( else: content = "### Available Resources" - return SystemPromptComponent( - content=content, - template_name="available_resources_header", - priority=priority, - ) - - -# ============================================================================= -# SECTION 3: Piecewise component builders (existing, enhanced) -# ============================================================================= - - -def build_tools_component( - tools: Dict[str, Any], - knowledge_base_summary: Optional[str] = None, - language: str = "zh", - is_manager: bool = True, - priority: int = 50, -) -> "ToolsComponent": - """Build ToolsComponent from tool configurations. - - Args: - tools: Dict of tool name -> ToolConfig or tool dict - knowledge_base_summary: Summary text from knowledge bases - language: Language code ('zh' or 'en') - is_manager: Whether this is a manager agent - priority: Component priority for selection - - Returns: - ToolsComponent instance - """ - from nexent.core.agents.agent_model import ToolsComponent - - tool_list = [] - for name, tool in tools.items(): - if hasattr(tool, 'description'): - tool_dict = { - "name": name, - "description": tool.description, - "inputs": getattr(tool, 'inputs', ''), - "output_type": getattr(tool, 'output_type', ''), - "source": getattr(tool, 'source', 'local'), - } - else: - tool_dict = { - "name": name, - "description": tool.get('description', ''), - "inputs": tool.get('inputs', ''), - "output_type": tool.get('output_type', ''), - "source": tool.get('source', 'local'), - } - tool_list.append(tool_dict) - - formatted_desc = _format_tools_description( - tools, - language=language, - is_manager=is_manager, - ) - return ToolsComponent( - tools=tool_list, - formatted_description=formatted_desc, - priority=priority, - ) - - -def build_skills_component( - skills: List[Dict[str, str]], - language: str = "zh", - priority: int = 70, -) -> "SkillsComponent": - """Build SkillsComponent from skill configurations. - - Args: - skills: List of skill dicts with name and description - language: Language code ('zh' or 'en') - priority: Component priority for selection - - Returns: - SkillsComponent instance - """ - from nexent.core.agents.agent_model import SkillsComponent - - formatted_desc = _format_skills_description(skills, language=language) - return SkillsComponent( - skills=skills, - formatted_description=formatted_desc, - priority=priority, - ) - - -def build_memory_component( - memory_list: List[Any], - search_query: Optional[str] = None, - language: str = "zh", - priority: int = 90, -) -> "MemoryComponent": - """Build MemoryComponent from memory search results. - - Args: - memory_list: List of memory search results - search_query: Query used to search memory - language: Language code ('zh' or 'en') - priority: Component priority for selection - - Returns: - MemoryComponent instance - """ - from nexent.core.agents.agent_model import MemoryComponent - - memories = [] - for mem in memory_list: - if isinstance(mem, dict): - memories.append({ - "content": mem.get('memory', '') or mem.get('content', ''), - "memory_type": mem.get('memory_type', 'user'), - "metadata": mem.get('metadata', {}), - }) - elif isinstance(mem, str): - memories.append({ - "content": mem, - "memory_type": "user", - "metadata": {}, - }) - - formatted_content = _format_memory_context(memory_list, language=language) - return MemoryComponent( - memories=memories, - formatted_content=formatted_content, - search_query=search_query, - priority=priority, - ) - - -def build_knowledge_base_component( - knowledge_base_summary: str, - kb_ids: Optional[List[str]] = None, - priority: int = 10, - language: str = "zh", -) -> "KnowledgeBaseComponent": - """Build KnowledgeBaseComponent from knowledge base summary. - - Args: - knowledge_base_summary: Summary text from knowledge bases - kb_ids: List of knowledge base IDs used - priority: Component priority for selection - language: Language code ('zh' or 'en') - - Returns: - KnowledgeBaseComponent instance - """ - from nexent.core.agents.agent_model import KnowledgeBaseComponent - - if knowledge_base_summary: - if language == "zh": - guidance = "knowledge_base_search 工具只能使用以下知识库索引，请根据用户的问题选择最相关的一个或多个知识库索引：\n" - else: - guidance = "knowledge_base_search tool can only use the following knowledge base indexes, please select the most relevant one or more knowledge base indexes based on the user's question:\n" - prefixed_summary = guidance + knowledge_base_summary - else: - prefixed_summary = knowledge_base_summary - - return KnowledgeBaseComponent( - summary=prefixed_summary, - kb_ids=kb_ids or [], - priority=priority, - ) - - -def build_managed_agents_component( - managed_agents: Dict[str, Any], - language: str = "zh", - priority: int = 45, -) -> "ManagedAgentsComponent": - """Build ManagedAgentsComponent from managed sub-agent configurations. - - Args: - managed_agents: Dict of agent name -> AgentConfig - language: Language code ('zh' or 'en') - priority: Component priority for selection - - Returns: - ManagedAgentsComponent instance - """ - from nexent.core.agents.agent_model import ManagedAgentsComponent - - agent_list = [] - for name, agent in managed_agents.items(): - if hasattr(agent, 'description'): - agent_dict = { - "name": name, - "description": agent.description, - "tools": [], - } - if hasattr(agent, 'tools'): - agent_dict["tools"] = [t.name for t in agent.tools if hasattr(t, 'name')] - else: - agent_dict = { - "name": name, - "description": agent.get('description', ''), - "tools": [], - } - agent_list.append(agent_dict) - - formatted_desc = _format_managed_agents_description(managed_agents, language=language) - return ManagedAgentsComponent( - agents=agent_list, - formatted_description=formatted_desc, - priority=priority, - ) - - -def build_external_agents_component( - external_a2a_agents: Dict[str, Any], - language: str = "zh", - priority: int = 44, -) -> "ExternalAgentsComponent": - """Build ExternalAgentsComponent from external A2A agent configurations. - - Args: - external_a2a_agents: Dict of agent_id -> ExternalA2AAgentConfig - language: Language code ('zh' or 'en') - priority: Component priority for selection - - Returns: - ExternalAgentsComponent instance - """ - from nexent.core.agents.agent_model import ExternalAgentsComponent - - agent_list = [] - for agent_id, agent in external_a2a_agents.items(): - if hasattr(agent, 'agent_id'): - agent_dict = { - "agent_id": str(agent.agent_id), - "name": agent.name, - "description": agent.description, - "url": getattr(agent, 'url', ''), - } - else: - agent_dict = { - "agent_id": str(agent_id), - "name": agent.get('name', ''), - "description": agent.get('description', ''), - "url": agent.get('url', ''), - } - agent_list.append(agent_dict) - - formatted_desc = _format_external_agents_description(external_a2a_agents, language=language) - return ExternalAgentsComponent( - agents=agent_list, - formatted_description=formatted_desc, - priority=priority, - ) - - -def build_system_prompt_component( - content: str, - template_name: Optional[str] = None, - priority: int = 100, -) -> "SystemPromptComponent": - """Build SystemPromptComponent with rendered content. - - Args: - content: Rendered system prompt content - template_name: Source template name for reference - priority: Component priority (highest by default) - - Returns: - SystemPromptComponent instance - """ - from nexent.core.agents.agent_model import SystemPromptComponent - - return SystemPromptComponent( - content=content, - template_name=template_name, - priority=priority, - ) - - -def build_skills_usage_component( - skills: List[Dict[str, str]], - language: str = "zh", - is_manager: bool = True, - priority: int = 40, -) -> "SkillsComponent": - """Build SkillsComponent for skills usage requirements. - - This is a skeleton-like component but its content depends on - whether skills exist, so it's built dynamically. + return content - Args: - skills: List of skill dicts - language: Language code ('zh' or 'en') - is_manager: Whether this is a manager agent - priority: Component priority - - Returns: - SkillsComponent instance - """ - from nexent.core.agents.agent_model import SkillsComponent - - content = _format_skills_usage_requirements(skills, language=language, is_manager=is_manager) - return SkillsComponent( - skills=skills, - formatted_description=content, - priority=priority, - ) - -def build_agent_fallback_component( - managed_agents: Dict[str, Any], - external_a2a_agents: Dict[str, Any], - language: str = "zh", - priority: int = 5, -) -> "SystemPromptComponent": - """Build SystemPromptComponent for agent fallback message. - - Only emits content when no agents are available. - - Args: - managed_agents: Dict of managed agents - external_a2a_agents: Dict of external agents - language: Language code - priority: Component priority - - Returns: - SystemPromptComponent instance (may have empty content) - """ - from nexent.core.agents.agent_model import SystemPromptComponent - - content = _format_agent_fallback(managed_agents, external_a2a_agents, language=language) - return SystemPromptComponent( - content=content, - template_name="agent_fallback", - priority=priority, - ) - - -# ============================================================================= -# SECTION 4: Main assembly function - build_context_components -# ============================================================================= - - -def build_context_components( - # Raw params for piecewise assembly (NEW in Goal 3) +def build_context_inputs( duty: Optional[str] = None, constraint: Optional[str] = None, few_shots: Optional[str] = None, @@ -1226,6 +387,7 @@ def build_context_components( user_id: Optional[str] = None, language: str = "zh", is_manager: bool = True, + enable_planning: bool = False, # Piecewise data sources tools: Optional[Dict[str, Any]] = None, skills: Optional[List[Dict[str, str]]] = None, @@ -1233,11 +395,11 @@ def build_context_components( external_a2a_agents: Optional[Dict[str, Any]] = None, memory_list: Optional[List[Any]] = None, memory_search_query: Optional[str] = None, + memory_tool_policy: Optional[str] = None, + automation_tool_policy: Optional[str] = None, + long_term_memory_prompt: Optional[str] = None, knowledge_base_summary: Optional[str] = None, kb_ids: Optional[List[str]] = None, - # Legacy param for fallback (removed short-circuit in Goal 3) - system_prompt: Optional[str] = None, - # Inclusion flags (kept for backward compatibility) include_tools: bool = True, include_skills: bool = True, include_memory: bool = True, @@ -1245,206 +407,157 @@ def build_context_components( include_managed_agents: bool = True, include_external_agents: bool = True, include_app_context: bool = True, -) -> List["ContextComponent"]: - """Build list of ContextComponents from agent configuration data. - - Piecewise assembly: Each semantic section is emitted as a dedicated - ContextComponent, assembled in the exact order matching Jinja2 templates. - - Assembly order (15 sections): - 1. Header (基本信息) - 2. Memory (上下文记忆) - if memory_list exists - 3. Duty (核心职责 + 安全准则) - 4. Skills (可用技能 + 6步流程) - if skills exist - 5. Execution Flow (执行流程 + 输出规范) - 6. Available Resources Header (可用资源 heading) - 7. Tools (可用资源/1. 工具 + 文件链接指南) - 8. Knowledge Base (知识库) - if knowledge_base_summary exists - 9. Managed Agents (可用资源/2. 助手) - if managed_agents exist - 10. External Agents (外部助手) - if external_a2a_agents exist - 11. Agent Fallback (当前没有可用的助手) - if no agents - 12. Skills Usage (可用资源/3. 技能 + 使用要求) - 13. Constraint (资源使用要求) - 14. Code Norms (python代码规范) - 15. Footer (示例模板 + 结尾) - - Note: The a330d815 short-circuit (if system_prompt: return [single]) - has been REMOVED. All callers must provide raw params for piecewise assembly. - The system_prompt param is kept for future fallback use but not currently - used in the piecewise path. - - Args: - duty: Agent's primary duty text - constraint: Resource usage constraint text - few_shots: Example templates text - app_name: Application name - app_description: Application description - user_id: Current user ID - language: Language code ('zh' or 'en') - is_manager: Whether this is a manager agent - tools: Dict of tool name -> ToolConfig - skills: List of skill dicts with name and description - managed_agents: Dict of agent name -> AgentConfig - external_a2a_agents: Dict of agent_id -> ExternalA2AAgentConfig - memory_list: List of memory search results - memory_search_query: Query used to search memory - knowledge_base_summary: Summary text from knowledge bases - kb_ids: List of knowledge base IDs - system_prompt: (Legacy) Pre-rendered system prompt - NOT USED in piecewise path - include_*: Flags for backward compatibility - - Returns: - List of ContextComponent instances ready for ContextManager - """ - components: List = [] - - # 1. Header - if app_name and app_description and user_id: - components.append( - build_skeleton_header_component( - app_name=app_name, - app_description=app_description, - user_id=user_id, - language=language, - ) +) -> List[ContextItemInput]: + """Build an authorized, naturally granular SDK context input snapshot.""" + inputs: List[ContextItemInput] = [] + + def add_system( + item_id: str, + text: str, + priority: int, + authority: str = "agent", + ) -> None: + if text: + inputs.append(ContextItemInput( + id=f"system:{item_id}", + type=ContextItemType.SYSTEM, + content={"text": text}, + source=(f"agent_prompt:{item_id}",), + priority=priority, + metadata={"authority": authority}, + )) + + if include_app_context and app_name and app_description and user_id: + add_system("header", _build_header_text( + app_name, app_description, user_id, language + ), 100, "tenant") + + if memory_tool_policy: + add_system("memory_tool_policy", memory_tool_policy, 90, "platform") + + if automation_tool_policy: + add_system("automation_tool_policy", automation_tool_policy, 95, "platform") + + if include_memory and long_term_memory_prompt: + add_system( + "long_term_memory", + long_term_memory_prompt, + 90, + "retrieved", ) - # 2. Memory (if exists) if include_memory and memory_list: - components.append( - build_memory_component( - memory_list=memory_list, - search_query=memory_search_query, - language=language, - ) - ) + for index, memory in enumerate(memory_list): + if not isinstance(memory, (dict, str)): + raise ValueError(f"invalid memory payload at index {index}") + payload = memory if isinstance(memory, dict) else {"memory": memory, "memory_level": "user"} + inputs.append(ContextItemInput( + id=f"memory:{index}", type=ContextItemType.MEMORY, content=payload, + source=(f"memory:{memory_search_query or 'run'}",), priority=90, + metadata={"render_group": "memory", "language": language, "authority": "retrieved"}, + )) - # 3. Duty + Safety Principles if duty: - components.append( - build_skeleton_duty_component( - duty=duty, - language=language, - is_manager=is_manager, - ) - ) + add_system("duty", _build_duty_text(duty, language, is_manager), 80) - # 4. Skills (if exists) - includes 6-step process if include_skills and skills: - components.append( - build_skills_component( - skills=skills, - language=language, - ) - ) - - # 5. Execution Flow. Do not make stable instructions depend on whether a - # particular request happened to retrieve memory. - components.append( - build_skeleton_execution_flow_component( - memory_list=None, - language=language, - is_manager=is_manager, - ) - ) - - # 6. Available Resources Header - components.append( - build_available_resources_header_component( - is_manager=is_manager, - language=language, - ) - ) + for index, skill in enumerate(skills): + name = str(skill.get("name", index)) + inputs.append(ContextItemInput( + id=f"skill:{name}", type=ContextItemType.SKILL, content=dict(skill), + source=(f"skill:{name}",), priority=70, + metadata={"render_group": "skills", "language": language, "authority": "agent"}, + )) + + add_system("execution_flow", _build_execution_flow_text( + None, language, is_manager, enable_planning + ), 60, "platform") + add_system("available_resources_header", _build_available_resources_header_text( + is_manager, language + ), 55, "platform") - # 7. Tools + File URL Guide if include_tools and tools: - components.append( - build_tools_component( - tools=tools, - # KB/RAG content is dynamic evidence and is emitted below as a - # user-role KnowledgeBaseComponent, not embedded in stable tool - # descriptions. - knowledge_base_summary=None, - language=language, - is_manager=is_manager, - ) - ) + for name, tool in tools.items(): + payload = { + "name": name, + "description": getattr(tool, "description", None) if not isinstance(tool, dict) else tool.get("description", ""), + "inputs": getattr(tool, "inputs", None) if not isinstance(tool, dict) else tool.get("inputs", ""), + "output_type": getattr(tool, "output_type", None) if not isinstance(tool, dict) else tool.get("output_type", ""), + "source": getattr(tool, "source", "local") if not isinstance(tool, dict) else tool.get("source", "local"), + } + inputs.append(ContextItemInput( + id=f"tool:{name}", type=ContextItemType.TOOL, content=payload, + source=(f"tool:{name}",), priority=50, + metadata={ + "render_group": "tools", "language": language, + "is_manager": is_manager, "authority": "agent", + }, + )) - # 8. Knowledge Base (if exists) if include_knowledge_base and knowledge_base_summary: - components.append( - build_knowledge_base_component( - knowledge_base_summary=knowledge_base_summary, - kb_ids=kb_ids, - language=language, - ) + guidance = ( + "knowledge_base_search 工具只能使用以下知识库索引，请根据用户的问题选择最相关的一个或多个知识库索引：\n" + if language == "zh" else + "knowledge_base_search tool can only use the following knowledge base indexes, please select the most relevant one or more knowledge base indexes based on the user's question:\n" ) + inputs.append(ContextItemInput( + id="knowledge_base:summary", type=ContextItemType.KNOWLEDGE_BASE, + content={"text": guidance + knowledge_base_summary, "role": "user"}, + source=tuple(f"knowledge_base:{kb_id}" for kb_id in (kb_ids or ())), priority=10, + metadata={"authority": "retrieved"}, + )) - # 9. Managed Agents (if exists) - manager only if is_manager and include_managed_agents and managed_agents: - components.append( - build_managed_agents_component( - managed_agents=managed_agents, - language=language, - ) - ) + for name, agent in managed_agents.items(): + payload = { + "name": name, + "description": getattr(agent, "description", None) if not isinstance(agent, dict) else agent.get("description", ""), + "tools": [getattr(tool, "name", "") for tool in getattr(agent, "tools", ())] + if not isinstance(agent, dict) else agent.get("tools", []), + } + inputs.append(ContextItemInput( + id=f"managed_agent:{name}", type=ContextItemType.MANAGED_AGENT, content=payload, + source=(f"managed_agent:{name}",), priority=45, + metadata={"render_group": "managed_agents", "language": language, "authority": "agent"}, + )) - # 10. External Agents (if exists) - manager only if is_manager and include_external_agents and external_a2a_agents: - components.append( - build_external_agents_component( - external_a2a_agents=external_a2a_agents, - language=language, - ) - ) + for agent_id, agent in external_a2a_agents.items(): + payload = { + "agent_id": str(getattr(agent, "agent_id", agent_id) if not isinstance(agent, dict) else agent.get("agent_id", agent_id)), + "name": getattr(agent, "name", "") if not isinstance(agent, dict) else agent.get("name", ""), + "description": getattr(agent, "description", "") if not isinstance(agent, dict) else agent.get("description", ""), + "url": getattr(agent, "url", "") if not isinstance(agent, dict) else agent.get("url", ""), + } + inputs.append(ContextItemInput( + id=f"external_agent:{payload['agent_id']}", type=ContextItemType.EXTERNAL_AGENT, + content=payload, source=(f"external_agent:{payload['agent_id']}",), priority=44, + metadata={"render_group": "external_agents", "language": language, "authority": "agent"}, + )) - # 11. Agent Fallback (if no agents available) - manager only if is_manager and not managed_agents and not external_a2a_agents: - fallback_comp = build_agent_fallback_component( - managed_agents=managed_agents or {}, - external_a2a_agents=external_a2a_agents or {}, - language=language, - ) - if fallback_comp.content: # Only add if has content - components.append(fallback_comp) - - # 12. Skills Usage Requirements + inputs.append(ContextItemInput( + id="system:agent_fallback", type=ContextItemType.SYSTEM, + content={"template": "agent_fallback", "language": language}, + source=("agent_prompt:agent_fallback",), priority=5, + metadata={"authority": "platform"}, + )) if include_skills: - components.append( - build_skills_usage_component( - skills=skills or [], - language=language, - is_manager=is_manager, - ) - ) - - # 13. Constraint + inputs.append(ContextItemInput( + id="system:skills_usage", type=ContextItemType.SYSTEM, + content={ + "template": "skills_usage", "skills": skills or [], + "language": language, "is_manager": is_manager, + }, + source=("agent_prompt:skills_usage",), priority=40, + metadata={"authority": "platform"}, + )) if constraint: - components.append( - build_skeleton_constraint_component( - constraint=constraint, - language=language, - ) - ) - - # 14. Code Norms - components.append( - build_skeleton_code_norms_component( - language=language, - is_manager=is_manager, - ) - ) - - # 15. Footer + add_system("constraint", _build_constraint_text(constraint, language), 30) + add_system("code_norms", _build_code_norms_text(language, is_manager), 20, "platform") if few_shots: - components.append( - build_skeleton_footer_component( - few_shots=few_shots, - language=language, - ) - ) - - return components + add_system("footer", _build_footer_text(few_shots, language), 10) + return inputs def build_app_context_string( @@ -1462,4 +575,4 @@ def build_app_context_string( Returns: Formatted app context string """ - return _format_app_context(app_name, app_description, user_id) + return f"Application: {app_name}\nDescription: {app_description}\nCurrent user: {user_id}" diff --git a/backend/utils/memory_tool_prompt.py b/backend/utils/memory_tool_prompt.py new file mode 100644 index 0000000000..48124bec01 --- /dev/null +++ b/backend/utils/memory_tool_prompt.py @@ -0,0 +1,24 @@ +from collections.abc import Iterable + + +def build_memory_tool_policy(language: str, tool_names: Iterable[str]) -> str: + """Build runtime guidance only when store_memory is available.""" + if "store_memory" not in set(tool_names): + return "" + + if language == "zh": + return """### Memory Tool Policy +- `store_memory` 只存储从用户与当前智能体对话中提取的短期记忆。短期记忆仅包括：用户偏好、任务目标、行动计划与最新进展、针对用户反馈或报错信息的反思总结。 +- 提取时综合参考用户提问、工具或代码执行结果、模型最终回答；在输出最终回答前，由你判断、归纳和去重，将单条可复用记忆作为 `content` 入参传给 `store_memory`，然后再输出最终回答；不要传入整段对话。 +- 每轮输出最终回答前都必须执行一次记忆价值评估。只要上述任一类短期记忆出现新增或更新，就必须调用 `store_memory`；只有确实没有合格条目时才可跳过，且不得为了调用工具而保存空洞内容。 +- 系统已在本轮开始前固定检索历史记忆。若候选条目已出现在已提供的记忆上下文或历史工具结果中，不得再次调用 `store_memory`。 +- 不要存储临时计算、中间噪声、未验证推测、重复内容或敏感密钥。 +- 不要为了展示 Memory 功能而机械调用 `store_memory`。""" + + return """### Memory Tool Policy +- `store_memory` stores only short-term memory extracted from the conversation between the user and the current agent. Short-term memory is limited to user preferences, task goals, action plans and latest progress, and reflections on user feedback or errors. +- Consider the user's question, tool or code execution results, and the final answer you have determined. Before emitting that answer, judge, summarize, and deduplicate the information, pass one reusable memory entry as the `content` input, and then emit the final answer; never pass the whole conversation. +- Before every final answer, you must assess whether reusable memory was added or updated. If any eligible category changed, you must call `store_memory`; skip it only when no eligible entry exists, and never store empty content merely to call the tool. +- The system has already performed fixed memory retrieval before this turn. Do not call `store_memory` when the candidate already appears in the provided memory context or prior tool results. +- Do not store transient calculations, intermediate noise, unverified guesses, duplicates, or secrets. +- Do not call `store_memory` mechanically on every turn.""" diff --git a/backend/utils/memory_utils.py b/backend/utils/memory_utils.py deleted file mode 100644 index e3ba01d6d2..0000000000 --- a/backend/utils/memory_utils.py +++ /dev/null @@ -1,81 +0,0 @@ -import logging -import re -from typing import Dict, Any -from urllib.parse import urlparse - -from consts import const as _c -from consts.const import MODEL_CONFIG_MAPPING -from utils.config_utils import get_model_name_from_config, tenant_config_manager - -logger = logging.getLogger("memory_utils") - - -def _sanitize_index_component(value: str) -> str: - """Convert arbitrary text into an Elasticsearch-safe index component.""" - return re.sub(r"[^a-z0-9_.-]", "_", value.lower()) - - -def build_memory_config(tenant_id: str) -> Dict[str, Any]: - """Return a fully-validated configuration dictionary for *mem0* ``Memory``. - """ - # 1. Resolve tenant-specific model configuration - llm_raw = tenant_config_manager.get_model_config(MODEL_CONFIG_MAPPING["llm"], tenant_id=tenant_id) - embed_raw = tenant_config_manager.get_model_config(MODEL_CONFIG_MAPPING["embedding"], tenant_id=tenant_id) - - if not (llm_raw and llm_raw.get("model_name")): - raise ValueError("Missing LLM configuration for tenant") - if not (embed_raw and embed_raw.get("max_tokens")): - raise ValueError("Missing embedding-model configuration for tenant") - - # 2. Resolve Elasticsearch connection details - if not _c.ES_HOST: - raise ValueError("ES_HOST is not configured") - parsed = urlparse(_c.ES_HOST) - if not (parsed.scheme and parsed.hostname and parsed.port): - raise ValueError("ES_HOST must include scheme, host and port, e.g. http://host:9200") - es_host = f"{parsed.scheme}://{parsed.hostname}" - es_port = parsed.port - # Normalize repo/name to avoid problematic characters in index names - safe_repo = _sanitize_index_component(embed_raw["model_repo"]) if embed_raw["model_repo"] else "" - safe_name = _sanitize_index_component(embed_raw["model_name"]) - index_name = ( - f"mem0_{safe_repo}_{safe_name}_{embed_raw['max_tokens']}" - if embed_raw["model_repo"] - else f"mem0_{safe_name}_{embed_raw['max_tokens']}" - ) - - # 3. Assemble final configuration - memory_config: Dict[str, Any] = { - "llm": { - "provider": "openai", - "config": { - "model": get_model_name_from_config(llm_raw), - "openai_base_url": llm_raw["base_url"], - "api_key": llm_raw["api_key"], - }, - }, - "embedder": { - "provider": "openai", - "config": { - "model": get_model_name_from_config(embed_raw), - "openai_base_url": embed_raw["base_url"], - "embedding_dims": embed_raw["max_tokens"], - "api_key": embed_raw["api_key"], - }, - }, - "vector_store": { - "provider": "elasticsearch", - "config": { - "collection_name": index_name, - "host": es_host, - "port": es_port, - "embedding_model_dims": embed_raw["max_tokens"], - "verify_certs": False, - "api_key": _c.ES_API_KEY, - "user": _c.ES_USERNAME, - "password": _c.ES_PASSWORD, - }, - }, - "telemetry": {"enabled": False}, - } - return memory_config diff --git a/backend/utils/prompt_template_utils.py b/backend/utils/prompt_template_utils.py index 299d3bf94e..cb56ee5893 100644 --- a/backend/utils/prompt_template_utils.py +++ b/backend/utils/prompt_template_utils.py @@ -68,6 +68,7 @@ def get_prompt_template(template_type: str, language: str = LANGUAGE["ZH"], **kw - 'generate_title': Title generation template - 'document_summary': Document summary template (Map stage) - 'cluster_summary_reduce': Cluster summary reduce template (Reduce stage) + - 'nl2agent': NL2Agent runtime system prompt language: Language code ('zh' or 'en') **kwargs: Additional parameters, for agent type need to pass is_manager parameter @@ -118,6 +119,18 @@ def get_prompt_template(template_type: str, language: str = LANGUAGE["ZH"], **kw 'skill_creation_complicated': { LANGUAGE["ZH"]: 'backend/prompts/skill_creation_complicate_zh.yaml', LANGUAGE["EN"]: 'backend/prompts/skill_creation_complicate_en.yaml' + }, + 'guardrail_regex': { + LANGUAGE["ZH"]: 'backend/prompts/utils/guardrail_regex_zh.yaml', + LANGUAGE["EN"]: 'backend/prompts/utils/guardrail_regex_en.yaml' + }, + 'agent_automation': { + LANGUAGE["ZH"]: 'backend/prompts/agent_automation_zh.yaml', + LANGUAGE["EN"]: 'backend/prompts/agent_automation_en.yaml' + }, + 'nl2agent': { + LANGUAGE["ZH"]: 'backend/prompts/nl2agent_zh.yaml', + LANGUAGE["EN"]: 'backend/prompts/nl2agent_en.yaml' } } @@ -170,6 +183,19 @@ def get_prompt_optimize_prompt_template(language: str = LANGUAGE["ZH"]) -> Dict[ return get_prompt_template('prompt_optimize', language) +def get_guardrail_regex_prompt_template(language: str = LANGUAGE["ZH"]) -> Dict[str, Any]: + """Load the guardrail regex generation prompt template. + + Args: + language: Language code ('zh' or 'en') selecting the template variant. + + Returns: + The loaded template configuration dict, carrying the + ``GUARDRAIL_SYSTEM_PROMPT`` and ``GUARDRAIL_USER_PROMPT`` keys. + """ + return get_prompt_template('guardrail_regex', language) + + def get_agent_prompt_template(is_manager: bool, language: str = LANGUAGE["ZH"]) -> Dict[str, Any]: """ Get agent prompt template diff --git a/backend/utils/redis_utils.py b/backend/utils/redis_utils.py new file mode 100644 index 0000000000..7aed8071d9 --- /dev/null +++ b/backend/utils/redis_utils.py @@ -0,0 +1,62 @@ +"""Reusable Redis client singleton with lazy initialization and connection pooling.""" + +import logging +import threading +from typing import Optional + +import redis + +from consts.const import REDIS_BACKEND_URL + +logger = logging.getLogger("redis_utils") + +_redis_pool: Optional[redis.ConnectionPool] = None +_redis_client: Optional[redis.Redis] = None +_init_lock = threading.Lock() +_initialized = False + + +def _ensure_initialized() -> None: + global _redis_pool, _redis_client, _initialized + if _initialized: + return + with _init_lock: + if _initialized: + return + _redis_pool = None + _redis_client = None + if not REDIS_BACKEND_URL: + logger.warning("REDIS_BACKEND_URL not set, Redis client unavailable.") + _initialized = True + return + try: + _redis_pool = redis.ConnectionPool.from_url( + REDIS_BACKEND_URL, + max_connections=50, + decode_responses=True, + ) + _redis_client = redis.Redis(connection_pool=_redis_pool) + _redis_client.ping() + logger.info("Redis client singleton initialized successfully.") + except Exception as e: + logger.error(f"Failed to initialize Redis client singleton: {e}") + _redis_pool = None + _redis_client = None + _initialized = True + + +def get_redis_client() -> Optional[redis.Redis]: + """Get the shared Redis client instance. + + Returns: + redis.Redis instance if available, None if REDIS_BACKEND_URL is not configured + or initialization failed. + """ + _ensure_initialized() + return _redis_client + + +def is_redis_available() -> bool: + """Check whether Redis is available (configured and reachable).""" + _ensure_initialized() + return _redis_client is not None diff --git a/backend/utils/tool_utils.py b/backend/utils/tool_utils.py index fafac660a7..36f39cb4b4 100644 --- a/backend/utils/tool_utils.py +++ b/backend/utils/tool_utils.py @@ -5,7 +5,12 @@ def get_local_tools_classes() -> List[type]: """ - Get all tool classes from the nexent.core.tools package + Get all tool classes from the nexent.core.tools package. + + Tools whose class-level ``category`` attribute is one of the SDK-internal + categories (see ``ToolCategory``) are filtered out so they do not appear + in the user-facing tool picker. These tools are injected by the SDK + itself at agent construction time and must not be user-configurable. Returns: List of tool class objects @@ -14,11 +19,31 @@ def get_local_tools_classes() -> List[type]: tools_classes = [] for name in dir(tools_package): obj = getattr(tools_package, name) - if inspect.isclass(obj): + if inspect.isclass(obj) and not _is_internal_tool(obj): tools_classes.append(obj) return tools_classes +def _is_internal_tool(tool_class: type) -> bool: + """Return True if the tool class is SDK-internal and must be hidden from + user-facing tool pickers. + + Tools opt-in by setting ``category = ToolCategory..value`` + on the class body. Today only ``ToolCategory.PLANNING`` is internal; + new internal categories (e.g. future tracing / debug tools) can be + added here without touching any tool code. + """ + try: + from nexent.core.utils.tools_common_message import ToolCategory + except ImportError: + return False + category = getattr(tool_class, "category", None) + if category is None: + return False + internal_categories = {ToolCategory.PLANNING.value} + return category in internal_categories + + def get_local_tools_description_zh() -> Dict[str, Dict]: """ Get description_zh for all local tools from SDK (not persisted to DB). @@ -40,7 +65,7 @@ def get_local_tools_description_zh() -> Dict[str, Dict]: for param_name, param in sig.parameters.items(): if param_name == "self": continue - + # Check if parameter has a default value and if it should be excluded if param.default != inspect.Parameter.empty: if hasattr(param.default, 'exclude') and param.default.exclude: diff --git a/deploy.sh b/deploy.sh index aad8dae842..d09bb1e4f7 100755 --- a/deploy.sh +++ b/deploy.sh @@ -8,6 +8,9 @@ DEPLOYMENT_COMMON="$SCRIPT_DIR/deploy/common/common.sh" if [ -f "$DEPLOYMENT_COMMON" ]; then # shellcheck source=/dev/null source "$DEPLOYMENT_COMMON" +else + echo "Error: shared deployment helper not found: $DEPLOYMENT_COMMON" >&2 + exit 1 fi [ -n "${DEPLOYMENT_LANGUAGE:-}" ] || DEPLOYMENT_LANGUAGE="en" DEPLOY_WRAPPER_DEFAULT_CONFIG_MODE="" @@ -16,8 +19,8 @@ usage() { if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then cat <<'USAGE' 用法： - bash deploy.sh [--load-images] [--push-images] [--image-registry-prefix PREFIX] [--config|--defaults] docker [Docker 部署选项] - bash deploy.sh [--load-images] [--push-images] [--image-registry-prefix PREFIX] [--config|--defaults] k8s [K8s 部署选项] + bash deploy.sh [--load-images] [--push-images] [--reuse-from DIR] [--image-registry-prefix PREFIX] [--config|--defaults] docker [Docker 部署选项] + bash deploy.sh [--load-images] [--push-images] [--reuse-from DIR] [--image-registry-prefix PREFIX] [--config|--defaults] k8s [K8s 部署选项] USAGE if [ "$DEPLOY_WRAPPER_DEFAULT_CONFIG_MODE" = "defaults" ]; then @@ -39,6 +42,9 @@ USAGE --load-images 部署前从 ./images 加载 Docker 镜像 tar 文件。 默认关闭。 --push-images 部署前调用 push-images.sh 推送镜像。 + --reuse-from DIR 从已有离线部署包复用 .env、monitoring.env 和目标部署选项。 + 仅离线部署包入口可用，且会覆盖当前包中的对应文件。 + 导入的 .env 会自动补充当前 .env.example 中的新变量。 --image-registry-prefix PREFIX 镜像仓库前缀，例如 registry.example.com/nexent。 使用 --push-images 且未传入时会交互询问。 @@ -50,8 +56,8 @@ USAGE cat <<'USAGE' Usage: - bash deploy.sh [--load-images] [--push-images] [--image-registry-prefix PREFIX] [--config|--defaults] docker [docker deploy options] - bash deploy.sh [--load-images] [--push-images] [--image-registry-prefix PREFIX] [--config|--defaults] k8s [k8s deploy options] + bash deploy.sh [--load-images] [--push-images] [--reuse-from DIR] [--image-registry-prefix PREFIX] [--config|--defaults] docker [docker deploy options] + bash deploy.sh [--load-images] [--push-images] [--reuse-from DIR] [--image-registry-prefix PREFIX] [--config|--defaults] k8s [k8s deploy options] USAGE if [ "$DEPLOY_WRAPPER_DEFAULT_CONFIG_MODE" = "defaults" ]; then @@ -73,6 +79,9 @@ Options: --load-images Load Docker image tar files from ./images before deploying. Defaults to off. --push-images Run push-images.sh before deploying. + --reuse-from DIR Reuse .env, monitoring.env, and target deployment options + from an existing offline package. Offline entrypoint only. + The imported .env receives new variables from the current template. --image-registry-prefix PREFIX Image registry prefix, e.g. registry.example.com/nexent. Prompts when --push-images is used and no prefix is provided. @@ -88,8 +97,11 @@ fi LOAD_IMAGES="false" PUSH_IMAGES="false" +REUSE_FROM="" IMAGE_REGISTRY_PREFIX="${IMAGE_REGISTRY_PREFIX:-}" DEPLOY_CONFIG_MODE="$DEPLOY_WRAPPER_DEFAULT_CONFIG_MODE" +DEPLOYMENT_OFFLINE="false" +[ "$DEPLOY_WRAPPER_DEFAULT_CONFIG_MODE" = "defaults" ] && DEPLOYMENT_OFFLINE="true" FORWARD_ARGS=() while [ $# -gt 0 ]; do @@ -102,6 +114,18 @@ while [ $# -gt 0 ]; do PUSH_IMAGES="true" shift ;; + --reuse-from) + if [ $# -lt 2 ]; then + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "错误：--reuse-from 需要一个目录" >&2 + else + echo "Error: --reuse-from requires a directory" >&2 + fi + exit 1 + fi + REUSE_FROM="$2" + shift 2 + ;; --image-registry-prefix|--registry-prefix|--image-registry) if [ $# -lt 2 ]; then if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then @@ -175,11 +199,144 @@ require_image_registry_prefix() { fi } +detect_deployment_target() { + local arg + for arg in "${FORWARD_ARGS[@]}"; do + case "$arg" in + docker) + printf 'docker' + return 0 + ;; + k8s|kubernetes|helm) + printf 'k8s' + return 0 + ;; + esac + done + return 1 +} + +reuse_deployment_files() { + local source_input="$1" + local target="$2" + local source_root + local current_root + local relative_path + local source_file + local destination_file + local optional_files=( + "deploy/env/monitoring.env" + "deploy/$target/deploy.options" + ) + + deployment_require_env_example "$SCRIPT_DIR/deploy/env/.env.example" || return 1 + + if [ ! -d "$source_input" ]; then + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "错误：已有部署包目录不存在或不是目录：$source_input" >&2 + else + echo "Error: existing deployment package directory does not exist or is not a directory: $source_input" >&2 + fi + return 1 + fi + + source_root="$(cd "$source_input" 2>/dev/null && pwd -P)" || { + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "错误：无法读取已有部署包目录：$source_input" >&2 + else + echo "Error: cannot read existing deployment package directory: $source_input" >&2 + fi + return 1 + } + current_root="$(cd "$SCRIPT_DIR" && pwd -P)" + if [ "$source_root" = "$current_root" ]; then + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "错误：已有部署包目录不能与当前部署包目录相同。" >&2 + else + echo "Error: existing deployment package directory must differ from the current package directory." >&2 + fi + return 1 + fi + + source_file="$source_root/deploy/env/.env" + if [ ! -f "$source_file" ] || [ ! -r "$source_file" ]; then + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "错误：已有部署包中缺少可读的 deploy/env/.env：$source_root" >&2 + else + echo "Error: existing deployment package does not contain a readable deploy/env/.env: $source_root" >&2 + fi + return 1 + fi + + for relative_path in "${optional_files[@]}"; do + source_file="$source_root/$relative_path" + if [ -e "$source_file" ] && { [ ! -f "$source_file" ] || [ ! -r "$source_file" ]; }; then + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "错误：已有部署包文件不可读：$relative_path" >&2 + else + echo "Error: existing deployment package file is not readable: $relative_path" >&2 + fi + return 1 + fi + done + + mkdir -p "$SCRIPT_DIR/deploy/env" "$SCRIPT_DIR/deploy/$target" + cp -p "$source_root/deploy/env/.env" "$SCRIPT_DIR/deploy/env/.env" + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "已复用已有部署包文件：deploy/env/.env" + else + echo "Reused existing deployment package file: deploy/env/.env" + fi + deployment_merge_env_from_example \ + "$SCRIPT_DIR/deploy/env/.env" \ + "$SCRIPT_DIR/deploy/env/.env.example" || return 1 + + for relative_path in "${optional_files[@]}"; do + source_file="$source_root/$relative_path" + destination_file="$SCRIPT_DIR/$relative_path" + if [ -f "$source_file" ]; then + mkdir -p "$(dirname "$destination_file")" + cp -p "$source_file" "$destination_file" + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "已复用已有部署包文件：$relative_path" + else + echo "Reused existing deployment package file: $relative_path" + fi + elif [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "警告：已有部署包中未找到可选文件：$relative_path" >&2 + else + echo "Warning: optional file not found in existing deployment package: $relative_path" >&2 + fi + done +} + +if [ -n "$REUSE_FROM" ]; then + if [ "$DEPLOYMENT_OFFLINE" != "true" ]; then + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "错误：--reuse-from 仅支持离线部署包入口。" >&2 + else + echo "Error: --reuse-from is supported only by the offline package entrypoint." >&2 + fi + exit 1 + fi + if ! DEPLOYMENT_TARGET="$(detect_deployment_target)"; then + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "错误：--reuse-from 需要指定 docker 或 k8s 部署目标。" >&2 + else + echo "Error: --reuse-from requires a docker or k8s deployment target." >&2 + fi + exit 1 + fi + reuse_deployment_files "$REUSE_FROM" "$DEPLOYMENT_TARGET" +fi + if [ "${#FORWARD_ARGS[@]}" -eq 0 ]; then usage exit 0 fi +deployment_ensure_root_env "$SCRIPT_DIR" "$SCRIPT_DIR/docker" || exit 1 + if [ "$LOAD_IMAGES" = "true" ] && [ "$PUSH_IMAGES" != "true" ]; then LOAD_SCRIPT="$SCRIPT_DIR/load-images.sh" if [ ! -f "$LOAD_SCRIPT" ]; then @@ -216,7 +373,10 @@ if [ -n "$IMAGE_REGISTRY_PREFIX" ]; then fi if [ -n "$DEPLOY_CONFIG_MODE" ]; then - NEXENT_DEPLOY_CONFIG_MODE="$DEPLOY_CONFIG_MODE" exec bash "$SCRIPT_DIR/deploy/deploy.sh" "${FORWARD_ARGS[@]}" + NEXENT_DEPLOYMENT_OFFLINE="$DEPLOYMENT_OFFLINE" \ + NEXENT_DEPLOY_CONFIG_MODE="$DEPLOY_CONFIG_MODE" \ + exec bash "$SCRIPT_DIR/deploy/deploy.sh" "${FORWARD_ARGS[@]}" fi -exec bash "$SCRIPT_DIR/deploy/deploy.sh" "${FORWARD_ARGS[@]}" +NEXENT_DEPLOYMENT_OFFLINE="$DEPLOYMENT_OFFLINE" \ + exec bash "$SCRIPT_DIR/deploy/deploy.sh" "${FORWARD_ARGS[@]}" diff --git a/deploy/common/common.sh b/deploy/common/common.sh index b9649e77a2..177ebd61b0 100755 --- a/deploy/common/common.sh +++ b/deploy/common/common.sh @@ -11,6 +11,7 @@ DEPLOYMENT_IMAGE_SOURCE_DEFAULT="general" DEPLOYMENT_REGISTRY_PROFILE_DEFAULT="general" DEPLOYMENT_IMAGE_REGISTRY_PREFIX_DEFAULT="" DEPLOYMENT_MONITORING_PROVIDER_DEFAULT="otlp" +DEPLOYMENT_SUPER_ADMIN_PASSWORD_DEFAULT="Nexent@123" DEPLOYMENT_COMPONENTS="" DEPLOYMENT_PORT_POLICY="" @@ -109,6 +110,9 @@ deployment_i18n_format() { password.validation) printf '密码至少 8 位，并且包含大写字母、小写字母和数字。' ;; env.created_from_docker) printf '✅ 已从 docker/.env 创建 deploy/env/.env' ;; env.created_from_example) printf '✅ 已从 deploy/env/.env.example 创建 deploy/env/.env' ;; + env.example_missing) printf '缺少可读的 deploy/env/.env.example，无法初始化或升级环境配置' ;; + env.merge_failed) printf '无法将 deploy/env/.env.example 中的新变量合并到 deploy/env/.env' ;; + env.merged) printf '✅ 已将 deploy/env/.env.example 中的新变量追加到 deploy/env/.env' ;; env.root_missing) printf '未找到 deploy/env/.env，且没有可用的 docker/.env 或 deploy/env/.env.example 模板' ;; validation.local_config_schema) printf '%s' '本地配置 schemaVersion %s 与 %s 不兼容。请使用 --reconfigure 重新配置。' ;; validation.unknown_component) printf '%s' '未知部署组件：%s' ;; @@ -171,6 +175,9 @@ deployment_i18n_format() { password.validation) printf 'Password must be at least 8 characters and include uppercase letters, lowercase letters, and numbers.' ;; env.created_from_docker) printf '✅ Created deploy/env/.env from docker/.env' ;; env.created_from_example) printf '✅ Created deploy/env/.env from deploy/env/.env.example' ;; + env.example_missing) printf 'A readable deploy/env/.env.example is required to initialize or upgrade environment configuration' ;; + env.merge_failed) printf 'Failed to merge new variables from deploy/env/.env.example into deploy/env/.env' ;; + env.merged) printf '✅ Added new variables from deploy/env/.env.example to deploy/env/.env' ;; env.root_missing) printf 'deploy/env/.env not found and no docker/.env or deploy/env/.env.example template is available' ;; validation.local_config_schema) printf '%s' 'Local config schemaVersion %s is incompatible with %s. Re-run with --reconfigure.' ;; validation.unknown_component) printf '%s' 'Unknown deployment component: %s' ;; @@ -309,6 +316,92 @@ deployment_password_validation_message() { deployment_i18n password.validation } +deployment_super_admin_password() { + printf '%s' "${NEXENT_SUPER_ADMIN_PASSWORD:-$DEPLOYMENT_SUPER_ADMIN_PASSWORD_DEFAULT}" +} + +deployment_should_prompt_super_admin_password() { + [ "${NEXENT_DEPLOYMENT_OFFLINE:-false}" = "true" ] && + [ "${NEXENT_DEPLOY_CONFIG_MODE:-}" = "tui" ] +} + +deployment_should_prompt_root_dir() { + [ "${NEXENT_DEPLOYMENT_OFFLINE:-false}" != "true" ] || + [ "${NEXENT_DEPLOY_CONFIG_MODE:-}" = "tui" ] +} + +deployment_require_env_example() { + local example_file="$1" + if [ ! -f "$example_file" ] || [ ! -r "$example_file" ]; then + deployment_error "$(deployment_i18n env.example_missing)" + return 1 + fi +} + +deployment_merge_env_from_example() { + local env_file="$1" + local example_file="$2" + local missing_assignments + local last_byte + + deployment_require_env_example "$example_file" || return 1 + + if [ ! -f "$env_file" ] || [ ! -r "$env_file" ]; then + deployment_error "$(deployment_i18n env.merge_failed)" + return 1 + fi + + missing_assignments="$(awk ' + function assignment_key(line, normalized) { + normalized = line + sub(/^[[:space:]]*/, "", normalized) + sub(/^export[[:space:]]+/, "", normalized) + if (normalized !~ /^[A-Za-z_][A-Za-z0-9_]*[[:space:]]*=/) { + return "" + } + sub(/[[:space:]]*=.*/, "", normalized) + return normalized + } + FILENAME == ARGV[1] { + key = assignment_key($0) + if (key != "") { + existing[key] = 1 + } + next + } + { + key = assignment_key($0) + if (key != "" && !(key in existing)) { + print $0 + } + } + ' "$env_file" "$example_file")" || { + deployment_error "$(deployment_i18n env.merge_failed)" + return 1 + } + + if [ -z "$missing_assignments" ]; then + return 0 + fi + + if [ -s "$env_file" ]; then + last_byte="$(tail -c 1 "$env_file" 2>/dev/null || true)" + fi + { + if [ -s "$env_file" ]; then + if [ -n "$last_byte" ]; then + printf '\n' + fi + printf '\n' + fi + printf '# Added automatically from the current deploy/env/.env.example\n%s\n' "$missing_assignments" + } >> "$env_file" || { + deployment_error "$(deployment_i18n env.merge_failed)" + return 1 + } + deployment_log "$(deployment_i18n env.merged)" +} + deployment_ensure_root_env() { local project_root="$1" local docker_dir="${2:-$project_root/docker}" @@ -317,28 +410,27 @@ deployment_ensure_root_env() { local root_example="$env_dir/.env.example" local docker_env="$docker_dir/.env" - mkdir -p "$env_dir" DEPLOYMENT_ROOT_ENV="$root_env" export DEPLOYMENT_ROOT_ENV + deployment_require_env_example "$root_example" || return 1 + + mkdir -p "$env_dir" + if [ -f "$root_env" ]; then - return 0 + deployment_merge_env_from_example "$root_env" "$root_example" + return $? fi if [ -f "$docker_env" ]; then cp "$docker_env" "$root_env" deployment_log "$(deployment_i18n env.created_from_docker)" - return 0 - fi - - if [ -f "$root_example" ]; then + else cp "$root_example" "$root_env" deployment_log "$(deployment_i18n env.created_from_example)" - return 0 fi - deployment_error "$(deployment_i18n env.root_missing)" - return 1 + deployment_merge_env_from_example "$root_env" "$root_example" } deployment_source_root_env() { @@ -1666,6 +1758,7 @@ deployment_apply_image_source() { export NEXENT_WEB_IMAGE="nexent/nexent-web:latest" export NEXENT_DATA_PROCESS_IMAGE="nexent/nexent-data-process:latest" export NEXENT_MCP_DOCKER_IMAGE="nexent/nexent-mcp:latest" + export NEXENT_SANDBOX_IMAGE="nexent/nexent-sandbox:latest" export OPENSSH_SERVER_IMAGE="nexent/nexent-ubuntu-terminal:latest" fi @@ -1673,6 +1766,7 @@ deployment_apply_image_source() { export NEXENT_WEB_IMAGE="${NEXENT_WEB_IMAGE:-nexent/nexent-web:$version}" export NEXENT_DATA_PROCESS_IMAGE="${NEXENT_DATA_PROCESS_IMAGE:-nexent/nexent-data-process:$version}" export NEXENT_MCP_DOCKER_IMAGE="${NEXENT_MCP_DOCKER_IMAGE:-nexent/nexent-mcp:$version}" + export NEXENT_SANDBOX_IMAGE="${NEXENT_SANDBOX_IMAGE:-nexent/nexent-sandbox:$version}" export ELASTICSEARCH_IMAGE="${ELASTICSEARCH_IMAGE:-docker.elastic.co/elasticsearch/elasticsearch:8.17.4}" export POSTGRESQL_IMAGE="${POSTGRESQL_IMAGE:-postgres:15-alpine}" export REDIS_IMAGE="${REDIS_IMAGE:-redis:alpine}" @@ -1690,7 +1784,7 @@ deployment_apply_image_source() { export LANGFUSE_WORKER_IMAGE="${LANGFUSE_WORKER_IMAGE:-docker.io/langfuse/langfuse-worker:3}" export LANGFUSE_WEB_IMAGE="${LANGFUSE_WEB_IMAGE:-docker.io/langfuse/langfuse:3}" export CLICKHOUSE_IMAGE="${CLICKHOUSE_IMAGE:-docker.io/clickhouse/clickhouse-server:26.3-alpine}" - export LANGFUSE_MINIO_IMAGE="${LANGFUSE_MINIO_IMAGE:-docker.io/minio/minio:RELEASE.2023-12-20T01-00-02Z}" + export LANGFUSE_MINIO_IMAGE="${LANGFUSE_MINIO_IMAGE:-quay.io/minio/minio:RELEASE.2023-12-20T01-00-02Z}" export LANGFUSE_REDIS_IMAGE="${LANGFUSE_REDIS_IMAGE:-docker.io/redis:alpine}" export LANGFUSE_POSTGRES_IMAGE="${LANGFUSE_POSTGRES_IMAGE:-docker.io/postgres:15-alpine}" @@ -1700,6 +1794,7 @@ deployment_apply_image_source() { NEXENT_WEB_IMAGE \ NEXENT_DATA_PROCESS_IMAGE \ NEXENT_MCP_DOCKER_IMAGE \ + NEXENT_SANDBOX_IMAGE \ ELASTICSEARCH_IMAGE \ POSTGRESQL_IMAGE \ REDIS_IMAGE \ @@ -1786,6 +1881,7 @@ deployment_render_docker_env() { printf 'NEXENT_WEB_IMAGE="%s"\n' "$NEXENT_WEB_IMAGE" printf 'NEXENT_DATA_PROCESS_IMAGE="%s"\n' "$NEXENT_DATA_PROCESS_IMAGE" printf 'NEXENT_MCP_DOCKER_IMAGE="%s"\n' "$NEXENT_MCP_DOCKER_IMAGE" + printf 'NEXENT_SANDBOX_IMAGE="%s"\n' "$NEXENT_SANDBOX_IMAGE" printf 'ELASTICSEARCH_IMAGE="%s"\n' "$ELASTICSEARCH_IMAGE" printf 'POSTGRESQL_IMAGE="%s"\n' "$POSTGRESQL_IMAGE" printf 'REDIS_IMAGE="%s"\n' "$REDIS_IMAGE" @@ -1853,6 +1949,7 @@ deployment_render_image_values() { printf ' image:\n repository: "%s"\n tag: "%s"\n pullPolicy: "IfNotPresent"\n' "$(deployment_image_repo "$SUPABASE_DB")" "$(deployment_image_tag "$SUPABASE_DB")" printf 'nexent-common:\n' printf ' images:\n mcp:\n repository: "%s"\n tag: "%s"\n pullPolicy: "%s"\n' "$(deployment_image_repo "$NEXENT_MCP_DOCKER_IMAGE")" "$(deployment_image_tag "$NEXENT_MCP_DOCKER_IMAGE")" "$local_pull_policy" + printf ' sandbox:\n repository: "%s"\n tag: "%s"\n pullPolicy: "%s"\n' "$(deployment_image_repo "$NEXENT_SANDBOX_IMAGE")" "$(deployment_image_tag "$NEXENT_SANDBOX_IMAGE")" "$local_pull_policy" } deployment_render_k8s_port_values() { @@ -1966,6 +2063,7 @@ deployment_render_helm_chart_values() { printf ' service:\n type: "%s"\n nodePort: 30436\n' "$internal_type" printf 'nexent-common:\n' printf ' images:\n mcp:\n repository: "%s"\n tag: "%s"\n pullPolicy: "%s"\n' "$(deployment_image_repo "$NEXENT_MCP_DOCKER_IMAGE")" "$(deployment_image_tag "$NEXENT_MCP_DOCKER_IMAGE")" "$local_pull_policy" + printf ' sandbox:\n repository: "%s"\n tag: "%s"\n pullPolicy: "%s"\n' "$(deployment_image_repo "$NEXENT_SANDBOX_IMAGE")" "$(deployment_image_tag "$NEXENT_SANDBOX_IMAGE")" "$local_pull_policy" } deployment_yaml_quote() { @@ -2057,7 +2155,7 @@ deployment_render_helm_monitoring_chart_values() { deployment_render_monitoring_image_value langfuseWeb docker.io/langfuse/langfuse "$langfuse_tag" deployment_render_monitoring_image_value langfuseWorker docker.io/langfuse/langfuse-worker "$langfuse_tag" deployment_render_monitoring_image_value clickhouse docker.io/clickhouse/clickhouse-server "$clickhouse_tag" - deployment_render_monitoring_image_value minio docker.io/minio/minio "$minio_tag" + deployment_render_monitoring_image_value minio quay.io/minio/minio "$minio_tag" deployment_render_monitoring_image_value redis docker.io/redis "$redis_tag" deployment_render_monitoring_image_value postgres docker.io/postgres "$postgres_tag" printf ' collector:\n' diff --git a/deploy/docker/assets/official-skills-zip/analyze-image.zip b/deploy/docker/assets/official-skills-zip/analyze-image.zip index 9ec4c2fb1f..c5162d6551 100644 Binary files a/deploy/docker/assets/official-skills-zip/analyze-image.zip and b/deploy/docker/assets/official-skills-zip/analyze-image.zip differ diff --git a/deploy/docker/assets/official-skills-zip/analyze-text-file.zip b/deploy/docker/assets/official-skills-zip/analyze-text-file.zip index 8c44788725..220663f57b 100644 Binary files a/deploy/docker/assets/official-skills-zip/analyze-text-file.zip and b/deploy/docker/assets/official-skills-zip/analyze-text-file.zip differ diff --git a/deploy/docker/assets/official-skills-zip/create-docx.zip b/deploy/docker/assets/official-skills-zip/create-docx.zip index aa53e82b0a..dba5449349 100644 Binary files a/deploy/docker/assets/official-skills-zip/create-docx.zip and b/deploy/docker/assets/official-skills-zip/create-docx.zip differ diff --git a/deploy/docker/assets/official-skills-zip/create-file-directory.zip b/deploy/docker/assets/official-skills-zip/create-file-directory.zip index 1e2d21ef09..a3483e290a 100644 Binary files a/deploy/docker/assets/official-skills-zip/create-file-directory.zip and b/deploy/docker/assets/official-skills-zip/create-file-directory.zip differ diff --git a/deploy/docker/assets/official-skills-zip/delete-file-directory.zip b/deploy/docker/assets/official-skills-zip/delete-file-directory.zip index 0f0067d028..9543271ee8 100644 Binary files a/deploy/docker/assets/official-skills-zip/delete-file-directory.zip and b/deploy/docker/assets/official-skills-zip/delete-file-directory.zip differ diff --git a/deploy/docker/assets/official-skills-zip/email-utils.zip b/deploy/docker/assets/official-skills-zip/email-utils.zip index c708a252ce..405ce9cb26 100644 Binary files a/deploy/docker/assets/official-skills-zip/email-utils.zip and b/deploy/docker/assets/official-skills-zip/email-utils.zip differ diff --git a/deploy/docker/assets/official-skills-zip/list-directory.zip b/deploy/docker/assets/official-skills-zip/list-directory.zip index e3eaeba27b..364ab94a5f 100644 Binary files a/deploy/docker/assets/official-skills-zip/list-directory.zip and b/deploy/docker/assets/official-skills-zip/list-directory.zip differ diff --git a/deploy/docker/assets/official-skills-zip/move-file-directory.zip b/deploy/docker/assets/official-skills-zip/move-file-directory.zip index d01897231c..046f958bfa 100644 Binary files a/deploy/docker/assets/official-skills-zip/move-file-directory.zip and b/deploy/docker/assets/official-skills-zip/move-file-directory.zip differ diff --git a/deploy/docker/assets/official-skills-zip/read-file.zip b/deploy/docker/assets/official-skills-zip/read-file.zip index b394c2b386..88e64895c9 100644 Binary files a/deploy/docker/assets/official-skills-zip/read-file.zip and b/deploy/docker/assets/official-skills-zip/read-file.zip differ diff --git a/deploy/docker/assets/official-skills-zip/run-shell-ssh.zip b/deploy/docker/assets/official-skills-zip/run-shell-ssh.zip index 868eee7c5e..4441cde370 100644 Binary files a/deploy/docker/assets/official-skills-zip/run-shell-ssh.zip and b/deploy/docker/assets/official-skills-zip/run-shell-ssh.zip differ diff --git a/deploy/docker/assets/official-skills-zip/search-datamate.zip b/deploy/docker/assets/official-skills-zip/search-datamate.zip index 0cb18ded6e..6df2435223 100644 Binary files a/deploy/docker/assets/official-skills-zip/search-datamate.zip and b/deploy/docker/assets/official-skills-zip/search-datamate.zip differ diff --git a/deploy/docker/assets/official-skills-zip/search-dify.zip b/deploy/docker/assets/official-skills-zip/search-dify.zip index 2bd7c8ccfc..3283b3be26 100644 Binary files a/deploy/docker/assets/official-skills-zip/search-dify.zip and b/deploy/docker/assets/official-skills-zip/search-dify.zip differ diff --git a/deploy/docker/assets/official-skills-zip/search-idata.zip b/deploy/docker/assets/official-skills-zip/search-idata.zip index 85a7e1b728..d7f13980d3 100644 Binary files a/deploy/docker/assets/official-skills-zip/search-idata.zip and b/deploy/docker/assets/official-skills-zip/search-idata.zip differ diff --git a/deploy/docker/assets/official-skills-zip/search-knowledge-base.zip b/deploy/docker/assets/official-skills-zip/search-knowledge-base.zip index 48fabec2a1..f76aef1b7c 100644 Binary files a/deploy/docker/assets/official-skills-zip/search-knowledge-base.zip and b/deploy/docker/assets/official-skills-zip/search-knowledge-base.zip differ diff --git a/deploy/docker/assets/official-skills-zip/search-web-exa.zip b/deploy/docker/assets/official-skills-zip/search-web-exa.zip index 19c2095882..625d7a41b0 100644 Binary files a/deploy/docker/assets/official-skills-zip/search-web-exa.zip and b/deploy/docker/assets/official-skills-zip/search-web-exa.zip differ diff --git a/deploy/docker/assets/official-skills-zip/search-web-linkup.zip b/deploy/docker/assets/official-skills-zip/search-web-linkup.zip index 4657bc1655..d3271fa891 100644 Binary files a/deploy/docker/assets/official-skills-zip/search-web-linkup.zip and b/deploy/docker/assets/official-skills-zip/search-web-linkup.zip differ diff --git a/deploy/docker/assets/official-skills-zip/search-web-tavily.zip b/deploy/docker/assets/official-skills-zip/search-web-tavily.zip index 628f73ef69..d4274ac03e 100644 Binary files a/deploy/docker/assets/official-skills-zip/search-web-tavily.zip and b/deploy/docker/assets/official-skills-zip/search-web-tavily.zip differ diff --git a/deploy/docker/compose/docker-compose-monitoring.yml b/deploy/docker/compose/docker-compose-monitoring.yml index 34722918a7..9ed3b8018f 100644 --- a/deploy/docker/compose/docker-compose-monitoring.yml +++ b/deploy/docker/compose/docker-compose-monitoring.yml @@ -186,7 +186,7 @@ services: - nexent langfuse-minio: - image: ${NEXENT_IMAGE_REGISTRY_PREFIX:-}docker.io/minio/minio:${LANGFUSE_MINIO_VERSION:-RELEASE.2023-12-20T01-00-02Z} + image: ${NEXENT_IMAGE_REGISTRY_PREFIX:-}quay.io/minio/minio:${LANGFUSE_MINIO_VERSION:-RELEASE.2023-12-20T01-00-02Z} container_name: nexent-langfuse-minio profiles: ["langfuse"] restart: unless-stopped diff --git a/deploy/docker/compose/docker-compose.prod.yml b/deploy/docker/compose/docker-compose.prod.yml index 5dbe32a572..815c30fb30 100644 --- a/deploy/docker/compose/docker-compose.prod.yml +++ b/deploy/docker/compose/docker-compose.prod.yml @@ -109,6 +109,7 @@ services: - ../../sql:/opt/nexent/sql:ro - ${ROOT_DIR}/skills:/mnt/nexent-data/skills - ${ROOT_DIR}/openssh-server/ssh-keys:/opt/ssh-keys:ro + - /var/run/docker.sock:/var/run/docker.sock:ro # Docker socket for sandbox container management environment: <<: [*minio-vars, *es-vars] NEXENT_SQL_STARTUP_MODE: wait @@ -169,6 +170,7 @@ services: - ../../sql:/opt/nexent/sql:ro - ${ROOT_DIR}/skills:/mnt/nexent-data/skills - ${ROOT_DIR}/openssh-server/ssh-keys:/opt/ssh-keys:ro + - /var/run/docker.sock:/var/run/docker.sock:ro # Docker socket for sandbox container management environment: <<: [*minio-vars, *es-vars] NEXENT_SQL_STARTUP_MODE: wait diff --git a/deploy/docker/compose/docker-compose.yml b/deploy/docker/compose/docker-compose.yml index a5f5acd888..b9b4c6a671 100644 --- a/deploy/docker/compose/docker-compose.yml +++ b/deploy/docker/compose/docker-compose.yml @@ -124,6 +124,7 @@ services: - ../../sql:/opt/nexent/sql:ro - ${ROOT_DIR}/skills:/mnt/nexent-data/skills - ${ROOT_DIR}/openssh-server/ssh-keys:/opt/ssh-keys:ro + - /var/run/docker.sock:/var/run/docker.sock:ro # Docker socket for sandbox container management environment: <<: [*minio-vars, *es-vars] NEXENT_SQL_STARTUP_MODE: wait @@ -218,6 +219,8 @@ services: - nexent ports: - "3000:3000" + env_file: + - ../../env/.env environment: - HTTP_BACKEND=http://nexent-config:5010 - WS_BACKEND=ws://nexent-runtime:5014 diff --git a/deploy/docker/create-su.sh b/deploy/docker/create-su.sh index 872dbb71c2..064880830d 100755 --- a/deploy/docker/create-su.sh +++ b/deploy/docker/create-su.sh @@ -10,6 +10,15 @@ SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" DEPLOY_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)" PROJECT_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)" ROOT_ENV_FILE="$DEPLOY_ROOT/env/.env" +DEPLOYMENT_COMMON="$DEPLOY_ROOT/common/common.sh" + +if [ -f "$DEPLOYMENT_COMMON" ]; then + # shellcheck source=/dev/null + source "$DEPLOYMENT_COMMON" +else + echo "Error: shared deployment helper not found: $DEPLOYMENT_COMMON" + exit 1 +fi # Source environment variables if deploy/env/.env file exists if [ -f "$ROOT_ENV_FILE" ]; then @@ -18,57 +27,88 @@ if [ -f "$ROOT_ENV_FILE" ]; then set +a fi -generate_random_password() { - # Generate a URL/JSON safe random password (alphanumeric only) - local pwd="" - if command -v openssl >/dev/null 2>&1; then - pwd=$(openssl rand -base64 32 | tr -dc 'A-Za-z0-9' | head -c 20) - else - pwd=$(tr -dc 'A-Za-z0-9' /dev/null 2>&1; then + echo " ✅ user_tenant_t schema is ready." + return 0 + fi + + if [ $(( $(date +%s) - start )) -ge "$timeout" ]; then + echo " ❌ user_tenant_t schema did not become ready within ${timeout}s." + return 1 + fi + + echo " ⏳ Waiting for user_tenant_t schema migration to complete..." + sleep "$interval" + done +} + +get_existing_super_admin_user_id() { + local email="$1" + local result + + if [ "$DEPLOYMENT_VERSION" != "full" ] || ! docker ps | grep -q "supabase-db-mini"; then + return 1 fi - if [ -z "$pwd" ]; then - # Fallback (should be extremely rare) - pwd=$(date +%s%N | tr -dc '0-9' | head -c 20) + + if ! result="$(docker exec supabase-db-mini \ + psql -U postgres -d "$SUPABASE_POSTGRES_DB" -X -A -t -v ON_ERROR_STOP=1 \ + -c "SELECT id FROM auth.users WHERE email = '${email}' LIMIT 1;" 2>/dev/null)"; then + return 1 fi - echo "$pwd" + + printf '%s' "$result" | tr -d '[:space:]' } -wait_for_postgresql_ready() { - # Function to wait for PostgreSQL to become ready - local retries=0 - local max_retries=${1:-30} # Default 5 minutes, can be overridden - while [ $retries -lt $max_retries ]; do - if docker exec nexent-postgresql pg_isready -U "$POSTGRES_USER" -d "$POSTGRES_DB" >/dev/null 2>&1; then - echo " ✅ PostgreSQL is now ready!" - return 0 - fi - echo "⏳ Waiting for PostgreSQL to become ready... (attempt $((retries + 1))/$max_retries)" - sleep 10 - retries=$((retries + 1)) - done +insert_super_admin_tenant_record() { + local user_id="$1" + local email="$2" + local sql - if [ $retries -eq $max_retries ]; then - echo " ⚠️ Warning: PostgreSQL did not become ready within expected time" - echo " You may need to check the container logs and try again" - return 1 + if [ -z "$user_id" ]; then + echo " ❌ Cannot insert super admin tenant record: user_id is empty." + return 1 fi + + wait_for_user_tenant_schema_ready || return 1 + + echo " 🔧 Inserting super admin user into user_tenant_t table..." + sql="INSERT INTO nexent.user_tenant_t (user_id, tenant_id, user_role, user_email, created_by, updated_by) VALUES ('${user_id}', '', 'SU', '${email}', 'system', 'system') ON CONFLICT (user_id, tenant_id) DO NOTHING;" + + if docker exec -i nexent-postgresql \ + psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -X -v ON_ERROR_STOP=1 \ + -c "$sql" >/dev/null 2>&1; then + echo " ✅ Super admin user inserted into user_tenant_t table successfully." + return 0 + fi + + echo " ❌ Failed to insert super admin user into user_tenant_t table." + return 1 } create_default_super_admin_user() { local email="suadmin@nexent.com" local password - - # Get password from command line argument, or generate random one if not provided + local display_password="${2:-true}" + + # Get password from the deploy script, or use the non-interactive default. if [ -n "$1" ]; then password="$1" else - # Fallback to random password if no argument provided (for backward compatibility) - password="$(generate_random_password)" - echo " ⚠️ Warning: No password provided, using random password" + password="$(deployment_super_admin_password)" fi echo "🔧 Creating super admin user..." - + # Determine which container to use for curl command local curl_container="nexent-config" if [ "$DEPLOYMENT_MODE" = "infrastructure" ] || ! docker ps | grep -q "nexent-config"; then @@ -82,57 +122,44 @@ create_default_super_admin_user() { fi fi - RESPONSE=$(docker exec "$curl_container" bash -c "curl -s -X POST http://kong:8000/auth/v1/signup -H \"apikey: ${SUPABASE_KEY}\" -H \"Authorization: Bearer ${SUPABASE_KEY}\" -H \"Content-Type: application/json\" -d '{\"email\":\"${email}\",\"password\":\"${password}\",\"email_confirm\":true}'" 2>/dev/null) + local response + response=$(docker exec "$curl_container" bash -c "curl -s -X POST http://kong:8000/auth/v1/signup -H \"apikey: ${SUPABASE_KEY}\" -H \"Authorization: Bearer ${SUPABASE_KEY}\" -H \"Content-Type: application/json\" -d '{\"email\":\"${email}\",\"password\":\"${password}\",\"email_confirm\":true}'" 2>/dev/null) - if [ -z "$RESPONSE" ]; then + if [ -z "$response" ]; then echo " ❌ No response received from Supabase." return 1 - elif echo "$RESPONSE" | grep -q '"access_token"' && echo "$RESPONSE" | grep -q '"user"'; then + elif echo "$response" | grep -q '"access_token"' && echo "$response" | grep -q '"user"'; then echo " ✅ Default super admin user has been successfully created." echo "" echo " Please save the following credentials carefully." echo " 📧 Email: ${email}" - if [ -n "$1" ]; then - echo " 🔏 Password: [User provided password]" - else + if [ "$display_password" = "true" ]; then echo " 🔏 Password: ${password}" + else + echo " 🔏 Password: [hidden]" fi - # Extract user.id from RESPONSE JSON + # Extract user.id from the response JSON. local user_id # Try using jq first (if available in the container or on host) if docker exec "$curl_container" command -v jq >/dev/null 2>&1; then - user_id=$(echo "$RESPONSE" | docker exec -i "$curl_container" jq -r '.user.id // empty' 2>/dev/null) + user_id=$(echo "$response" | docker exec -i "$curl_container" jq -r '.user.id // empty' 2>/dev/null) elif command -v jq >/dev/null 2>&1; then - user_id=$(echo "$RESPONSE" | jq -r '.user.id // empty' 2>/dev/null) + user_id=$(echo "$response" | jq -r '.user.id // empty' 2>/dev/null) fi # Fallback: use grep and sed (works without any special tools) if [ -z "$user_id" ]; then - user_id=$(echo "$RESPONSE" | grep -o '"user"[^}]*"id":"[^"]*"' | sed -n 's/.*"id":"\([^"]*\)".*/\1/p' 2>/dev/null) + user_id=$(echo "$response" | grep -o '"user"[^}]*"id":"[^"]*"' | sed -n 's/.*"id":"\([^"]*\)".*/\1/p' 2>/dev/null) fi if [ -z "$user_id" ]; then - echo " ⚠️ Warning: Could not extract user.id from response. Skipping database insertion." + echo " ❌ Could not extract user.id from the Supabase response." + return 1 else - # Wait for PostgreSQL to be ready - echo " ⏳ Waiting for PostgreSQL to be ready..." - if ! wait_for_postgresql_ready; then - echo " ⚠️ Warning: PostgreSQL is not ready. Skipping database insertion." - return 0 - fi - - # Insert user_tenant_t record - echo " 🔧 Inserting super admin user into user_tenant_t table..." - local sql="INSERT INTO nexent.user_tenant_t (user_id, tenant_id, user_role, user_email, created_by, updated_by) VALUES ('${user_id}', '', 'SU', '${email}', 'system', 'system') ON CONFLICT (user_id, tenant_id) DO NOTHING;" - - if docker exec -i nexent-postgresql psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "$sql" >/dev/null 2>&1; then - echo " ✅ Super admin user inserted into user_tenant_t table successfully." - else - echo " ⚠️ Warning: Failed to insert super admin user into user_tenant_t table." - fi + insert_super_admin_tenant_record "$user_id" "$email" || return 1 fi - elif echo "$RESPONSE" | grep -q '"error_code":"user_already_exists"' || echo "$RESPONSE" | grep -q '"code":422'; then + elif echo "$response" | grep -q '"error_code":"user_already_exists"' || echo "$response" | grep -q '"code":422'; then echo " 🚧 Default super admin user already exists. Skipping creation." echo " 📧 Email: ${email}" @@ -140,33 +167,14 @@ create_default_super_admin_user() { # Get user_id from Supabase auth.users table echo " 🔧 Retrieving user_id from Supabase database..." local user_id - if [ "$DEPLOYMENT_VERSION" = "full" ] && docker ps | grep -q "supabase-db-mini"; then - # Query Supabase auth.users table to get user_id by email - user_id=$(docker exec supabase-db-mini psql -U postgres -d "$SUPABASE_POSTGRES_DB" -t -c "SELECT id FROM auth.users WHERE email = '${email}' LIMIT 1;" 2>/dev/null | tr -d '[:space:]') - fi + user_id="$(get_existing_super_admin_user_id "$email")" if [ -z "$user_id" ]; then - echo " ⚠️ Warning: Could not retrieve user_id. Skipping database insertion." - echo " 💡 Note: If user_tenant_t record is missing, you may need to insert it manually." - return 0 - fi - - # Wait for PostgreSQL to be ready - echo " ⏳ Waiting for PostgreSQL to be ready..." - if ! wait_for_postgresql_ready; then - echo " ⚠️ Warning: PostgreSQL is not ready. Skipping database insertion." - return 0 + echo " ❌ Could not retrieve the existing super admin user_id." + return 1 fi - # Insert user_tenant_t record - echo " 🔧 Inserting super admin user into user_tenant_t table..." - local sql="INSERT INTO nexent.user_tenant_t (user_id, tenant_id, user_role, user_email, created_by, updated_by) VALUES ('${user_id}', '', 'SU', '${email}', 'system', 'system') ON CONFLICT (user_id, tenant_id) DO NOTHING;" - - if docker exec -i nexent-postgresql psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "$sql" >/dev/null 2>&1; then - echo " ✅ Super admin user inserted into user_tenant_t table successfully." - else - echo " ⚠️ Warning: Failed to insert super admin user into user_tenant_t table." - fi + insert_super_admin_tenant_record "$user_id" "$email" || return 1 else echo " ❌ Response from Supabase does not contain 'access_token' or 'user'." return 1 @@ -177,10 +185,5 @@ create_default_super_admin_user() { echo "" } -# Main execution -# Pass password as first argument if provided -if create_default_super_admin_user "$1"; then - exit 0 -else - exit 1 -fi +# Main execution. +create_default_super_admin_user "${1:-}" "${2:-true}" diff --git a/deploy/docker/deploy.sh b/deploy/docker/deploy.sh index dda77c0bbd..ffc666b7cb 100755 --- a/deploy/docker/deploy.sh +++ b/deploy/docker/deploy.sh @@ -832,6 +832,39 @@ pull_mcp_image() { echo "" } +pull_sandbox_image() { + if [ "$DEPLOYMENT_IMAGE_SOURCE" = "local-latest" ]; then + echo "🔄 Skipping sandbox image pull because image source is local-latest." + echo "" + echo "--------------------------------" + echo "" + return 0 + fi + + echo "🔄 Checking sandbox Docker image..." + + SANDBOX_IMAGE_NAME=${NEXENT_SANDBOX_IMAGE:-nexent/nexent-sandbox:latest} + echo " 📦 Image: ${SANDBOX_IMAGE_NAME}" + + if docker image inspect "${SANDBOX_IMAGE_NAME}" >/dev/null 2>&1; then + echo " ✅ Sandbox image already exists locally" + echo " 💡 Skipping pull, using existing image" + else + echo " 📥 Sandbox image not found locally, pulling..." + if docker pull "${SANDBOX_IMAGE_NAME}"; then + echo " ✅ Sandbox image pulled successfully" + echo " 💡 The image will be available when agent sandbox runs are executed" + else + echo " ⚠️ Failed to pull sandbox image, but deployment continues" + echo " 💡 You can manually pull the image later: docker pull ${SANDBOX_IMAGE_NAME}" + fi + fi + + echo "" + echo "--------------------------------" + echo "" +} + select_deployment_mode() { echo "🎛️ Please select deployment mode:" echo " 1) 🛠️ Development mode - Expose all service ports for debugging" @@ -1151,7 +1184,7 @@ configure_root_dir_from_env() { echo " 📁 Use existing ROOT_DIR path: $ROOT_DIR" else local default_root_dir="$HOME/nexent-data" - if [ -t 0 ]; then + if deployment_should_prompt_root_dir && [ -t 0 ]; then local user_root_dir read -p " 📁 Enter ROOT_DIR path (default: $default_root_dir): " user_root_dir ROOT_DIR="${user_root_dir:-$default_root_dir}" @@ -1500,6 +1533,14 @@ create_default_super_admin_user() { # Make sure the script is executable chmod +x "$script_path" + # Export the database configuration before either the create or repair path. + export SUPABASE_KEY + export POSTGRES_USER + export POSTGRES_DB + export DEPLOYMENT_VERSION + export SUPABASE_POSTGRES_DB + export DEPLOYMENT_MODE + # Check if super admin user already exists echo "" echo "🔍 Checking if super admin user exists..." @@ -1509,34 +1550,31 @@ create_default_super_admin_user() { if [ $check_result -eq 0 ]; then echo " ✅ Super admin user (${email}) already exists." - echo " 💡 Skipping user creation. If you need to reset the password, please do so manually." - return 0 + echo " 🔧 Ensuring the tenant relationship exists." + if bash "$script_path"; then + return 0 + fi + return 1 elif [ $check_result -eq 1 ]; then echo " ℹ️ Super admin user (${email}) does not exist. Proceeding with creation..." else echo " ⚠️ Warning: Could not determine if user exists. Proceeding with creation..." fi - # Prompt for password local password - password="$(prompt_super_admin_password)" - local prompt_result=$? - - if [ $prompt_result -ne 0 ] || [ -z "$password" ]; then - echo " ❌ Failed to get password from user." - return 1 + local display_password="true" + if deployment_should_prompt_super_admin_password; then + password="$(prompt_super_admin_password)" || { + echo " ❌ Failed to get password from user." + return 1 + } + display_password="false" + else + password="$(deployment_super_admin_password)" fi - # Export necessary environment variables for the script - export SUPABASE_KEY - export POSTGRES_USER - export POSTGRES_DB - export DEPLOYMENT_VERSION - export SUPABASE_POSTGRES_DB - export DEPLOYMENT_MODE - # Execute the script with password as argument - if bash "$script_path" "$password"; then + if bash "$script_path" "$password" "$display_password"; then unset password return 0 else @@ -1636,6 +1674,22 @@ main_deploy() { fi fi + # Set NEXENT_SANDBOX_DOCKER_IMAGE in .env file + if [ -n "${NEXENT_SANDBOX_IMAGE:-}" ]; then + update_env_var "NEXENT_SANDBOX_DOCKER_IMAGE" "${NEXENT_SANDBOX_IMAGE}" + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "🔧 NEXENT_SANDBOX_DOCKER_IMAGE 已设置为：${NEXENT_SANDBOX_IMAGE}" + else + echo "🔧 NEXENT_SANDBOX_DOCKER_IMAGE set to: ${NEXENT_SANDBOX_IMAGE}" + fi + else + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "⚠️ 环境中未找到 NEXENT_SANDBOX_IMAGE，将使用代码默认值" + else + echo "⚠️ NEXENT_SANDBOX_IMAGE not found in environment, will use default from code" + fi + fi + # Add permission prepare_directory_and_data || { if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then @@ -1746,6 +1800,9 @@ main_deploy() { # Pull MCP image for later use pull_mcp_image + # Pull sandbox image for agent sandbox runs + pull_sandbox_image + persist_deploy_options deployment_persist_local_config return 0 @@ -1788,6 +1845,9 @@ main_deploy() { # Pull MCP image for later use pull_mcp_image + # Pull sandbox image for agent sandbox runs + pull_sandbox_image + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then echo "🎉 部署完成！" echo "🌐 现在可以访问应用：http://localhost:3000" diff --git a/deploy/env/.env.example b/deploy/env/.env.example index 08761efa84..902e72d7c8 100644 --- a/deploy/env/.env.example +++ b/deploy/env/.env.example @@ -73,12 +73,16 @@ NEXENT_POSTGRES_PASSWORD=nexent@4321 POSTGRES_DB=nexent POSTGRES_PORT=5432 +# Default Super Admin Config +NEXENT_SUPER_ADMIN_PASSWORD=Nexent@123 + # Minio Config MINIO_ENDPOINT=http://nexent-minio:9000 MINIO_ROOT_USER=nexent MINIO_ROOT_PASSWORD=nexent@4321 MINIO_REGION=cn-north-1 MINIO_DEFAULT_BUCKET=nexent +MINIO_SECURE=true # Redis Config REDIS_URL=redis://nexent-redis:6379/0 @@ -102,7 +106,7 @@ SITE_URL=http://localhost:3011 SUPABASE_URL=http://nexent-supabase-kong:8000 API_EXTERNAL_URL=http://nexent-supabase-kong:8000 DISABLE_SIGNUP=false -JWT_EXPIRY=3600 +JWT_EXPIRY=7200 DEBUG_JWT_EXPIRE_SECONDS=0 # Supabase Configuration @@ -147,6 +151,7 @@ DISABLE_CELERY_FLOWER=true DOCKER_ENVIRONMENT=false ENABLE_UPLOAD_IMAGE=false + # Celery Configuration CELERY_WORKER_PREFETCH_MULTIPLIER=1 CELERY_TASK_TIME_LIMIT=3600 @@ -184,6 +189,10 @@ LINK_APP_OAUTH_CLIENT_SECRET= ENABLE_WECHAT_OAUTH=false WECHAT_OAUTH_APP_ID= WECHAT_OAUTH_APP_SECRET= +# Huawei OAuth +UNIPORTAL_URL= +HUAWEI_OAUTH_CLIENT_ID= +HUAWEI_OAUTH_CLIENT_SECRET= # Base URL for OAuth callback (e.g., http://localhost:3000 for local dev) OAUTH_SSL_VERIFY=true OAUTH_CA_BUNDLE= @@ -220,3 +229,55 @@ CAS_SYNTHETIC_EMAIL_DOMAIN=@cas.local CAS_LOGOUT_URL=/logout CAS_SSL_VERIFY=true CAS_CA_BUNDLE= + +# ===== AIDP Knowledge Base Configuration ===== +# Feature flag: Controls whether the unified `/knowledges` entry renders the +# AIDP knowledge base component (true) or the built-in local knowledge base +# component (false). In both modes, the sidebar menu entry keeps the same +# label "知识库配置" and continues to point to `/knowledges`; when true, the +# resource-manage page hides the "Knowledge Base" tab because AIDP KB is +# managed from the unified entry. +# Connection config: AIDP_SERVER_URL and AIDP_API_KEY are consumed by the +# backend to proxy AIDP calls — the frontend never holds the API key. Leave +# them empty to disable AIDP integration. +ENABLE_AIDP_KNOWLEDGE=false +AIDP_SERVER_URL=http://127.0.0.1:30081 +AIDP_API_KEY=mock-aidp-key +AIDP_TENANT_ID=aidp + +# ===== Agent Sandbox Configuration ===== + +# Default sandbox isolation level: local / docker / wasm. +# 'local' preserves backward-compatibility for existing deployments. +NEXENT_SANDBOX_DEFAULT_LEVEL=local + +# Default sandbox container lifecycle scope: session / system. +# session = one container per agent_run, destroyed on run end (strictest isolation). +# system = persistent warm pool shared by all runs (lowest cold-start latency). +NEXENT_SANDBOX_DEFAULT_SCOPE=system + +# Docker image used when level is 'docker'. +NEXENT_SANDBOX_DOCKER_IMAGE=nexent/nexent-sandbox:latest + +# Sandbox resource limits. +NEXENT_SANDBOX_MEMORY_LIMIT_MB=512 +NEXENT_SANDBOX_CPU_QUOTA=1.0 + +# Sandbox execution timeout per step (seconds). +NEXENT_SANDBOX_TIMEOUT_S=30 + +# Sandbox network policy: enabled / disabled. +NEXENT_SANDBOX_NETWORK=disabled + +# Shell execution policy: disabled / restricted / boxed. +# 'disabled' is recommended — blocks subprocess/os shell calls at AST-parse time. +NEXENT_SANDBOX_SHELL_POLICY=disabled + +# MinIO bucket for sandbox output file sync. +NEXENT_SANDBOX_OUTPUT_BUCKET=nexent-artifacts + +# Automatically sync sandbox output files to MinIO after each run. +NEXENT_SANDBOX_AUTO_SYNC_OUTPUTS=true + +# Website File Upload Size Limit (10 - 100MB) +FILE_UPLOAD_SIZE_LIMIT=100 diff --git a/deploy/env/image-source.general.env b/deploy/env/image-source.general.env index e2ac200bec..94e5410fdb 100644 --- a/deploy/env/image-source.general.env +++ b/deploy/env/image-source.general.env @@ -2,6 +2,7 @@ NEXENT_IMAGE=nexent/nexent:${APP_VERSION} NEXENT_WEB_IMAGE=nexent/nexent-web:${APP_VERSION} NEXENT_DATA_PROCESS_IMAGE=nexent/nexent-data-process:${APP_VERSION} NEXENT_MCP_DOCKER_IMAGE=nexent/nexent-mcp:${APP_VERSION} +NEXENT_SANDBOX_IMAGE=nexent/nexent-sandbox:${APP_VERSION} ELASTICSEARCH_IMAGE=docker.elastic.co/elasticsearch/elasticsearch:8.17.4 POSTGRESQL_IMAGE=postgres:15-alpine diff --git a/deploy/env/image-source.mainland.env b/deploy/env/image-source.mainland.env index fd628ba469..06a9e7303d 100644 --- a/deploy/env/image-source.mainland.env +++ b/deploy/env/image-source.mainland.env @@ -2,6 +2,7 @@ NEXENT_IMAGE=ccr.ccs.tencentyun.com/nexent-hub/nexent:${APP_VERSION} NEXENT_WEB_IMAGE=ccr.ccs.tencentyun.com/nexent-hub/nexent-web:${APP_VERSION} NEXENT_DATA_PROCESS_IMAGE=ccr.ccs.tencentyun.com/nexent-hub/nexent-data-process:${APP_VERSION} NEXENT_MCP_DOCKER_IMAGE=ccr.ccs.tencentyun.com/nexent-hub/nexent-mcp:${APP_VERSION} +NEXENT_SANDBOX_IMAGE=ccr.ccs.tencentyun.com/nexent-hub/nexent-sandbox:${APP_VERSION} ELASTICSEARCH_IMAGE=elastic.m.daocloud.io/elasticsearch/elasticsearch:8.17.4 POSTGRESQL_IMAGE=docker.m.daocloud.io/postgres:15-alpine diff --git a/deploy/images/build.sh b/deploy/images/build.sh index 25ad6ae60c..584ce7d4a6 100755 --- a/deploy/images/build.sh +++ b/deploy/images/build.sh @@ -71,7 +71,7 @@ USAGE Usage: deploy/images/build.sh [options] Options: - --images LIST Comma-separated image list: all,main,web,data-process,mcp,terminal,docs + --images LIST Comma-separated image list: all,main,web,data-process,mcp,terminal,docs,sandbox --image IMAGE Compatibility alias for --images with one image --all Build all images --main Build nexent/nexent @@ -107,6 +107,7 @@ while [ $# -gt 0 ]; do --mcp) REQUESTED_IMAGES+=("mcp"); shift ;; --terminal) REQUESTED_IMAGES+=("terminal"); shift ;; --docs) REQUESTED_IMAGES+=("docs"); shift ;; + --sandbox) REQUESTED_IMAGES+=("sandbox"); shift ;; --components) COMPONENTS="$2"; shift 2 ;; --platform) PLATFORM="$2"; shift 2 ;; --version) VERSION="$2"; shift 2 ;; @@ -149,7 +150,7 @@ add_image_if_missing() { } select_all_images() { - SELECTED_IMAGES=(main web data-process mcp terminal docs) + SELECTED_IMAGES=(main web data-process mcp terminal docs sandbox) } select_images_from_csv() { @@ -167,7 +168,7 @@ select_images_from_csv() { all) select_all_images ;; - main|web|data-process|mcp|terminal|docs) + main|web|data-process|mcp|terminal|docs|sandbox) add_image_if_missing "$normalized" ;; *) @@ -186,7 +187,7 @@ select_images_from_csv() { image_tui_multiselect() { [ -t 0 ] || return 1 - local images=(main web data-process mcp terminal docs) + local images=(main web data-process mcp terminal docs sandbox) local details=( "$(deployment_i18n image_build.detail.main)" "$(deployment_i18n image_build.detail.web)" @@ -194,8 +195,9 @@ image_tui_multiselect() { "$(deployment_i18n image_build.detail.mcp)" "$(deployment_i18n image_build.detail.terminal)" "$(deployment_i18n image_build.detail.docs)" + "Sandbox runtime for LLM-generated Python execution" ) - local selected=(1 1 0 0 0 0) + local selected=(1 1 0 0 0 0 0) local cursor=0 local i key key_tail selection @@ -293,7 +295,7 @@ run_interactive_configuration() { else echo "Images:" fi - echo " main, web, data-process, mcp, terminal, docs" + echo " main, web, data-process, mcp, terminal, docs, sandbox" if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then IMAGES="$(prompt_choice "请输入镜像（默认：main,web）：" "main,web")" else @@ -516,6 +518,7 @@ build_selected_image() { [ "$TERMINAL_VARIANT" = "conda" ] && image_name="nexent-ubuntu-terminal-conda" build_one "$image_name" "$DOCKERFILE_DIR/terminal/Dockerfile" --build-arg TERMINAL_VARIANT="$TERMINAL_VARIANT" ;; + sandbox) build_one nexent-sandbox "$DOCKERFILE_DIR/sandbox/Dockerfile" "${PY_MIRROR_ARGS[@]}" ;; *) if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then echo "不支持的镜像：$1" >&2 @@ -550,6 +553,9 @@ select_images_from_components() { terminal) add_image_if_missing terminal ;; + sandbox) + add_image_if_missing sandbox + ;; *) if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then echo "镜像构建不支持该组件：$normalized" >&2 diff --git a/deploy/images/dockerfiles/data-process/Dockerfile b/deploy/images/dockerfiles/data-process/Dockerfile index 459713ab31..bafa5334ca 100644 --- a/deploy/images/dockerfiles/data-process/Dockerfile +++ b/deploy/images/dockerfiles/data-process/Dockerfile @@ -157,9 +157,11 @@ RUN --mount=type=cache,id=nexent-data-process-apt-cache-${TARGETARCH},target=/va mkdir -p /var/cache/apt/archives /var/lib/apt/lists/partial && \ apt-get update && \ apt-get install -y --no-install-recommends --fix-missing \ - libreoffice \ fontconfig \ - fonts-noto-cjk && \ + fonts-noto-cjk \ + libreoffice \ + poppler-utils \ + tesseract-ocr && \ fc-cache -fv && \ apt-get autoremove -y && \ rm -rf /tmp/* /var/tmp/* diff --git a/deploy/images/dockerfiles/sandbox/Dockerfile b/deploy/images/dockerfiles/sandbox/Dockerfile new file mode 100644 index 0000000000..f115f07696 --- /dev/null +++ b/deploy/images/dockerfiles/sandbox/Dockerfile @@ -0,0 +1,165 @@ +# ===================================================================== +# Sandbox runtime image for LLM-generated Python execution. +# +# Design reference: doc/docs/zh/backend/sandbox-design.md#6-docker镜像设计 +# This image is consumed by smolagents DockerExecutor; it is NOT a +# long-running service and is created/destroyed per agent run. +# ===================================================================== + +# --------------------------------------------------------------------- +# Stage 1: builder - install uv and the Nexent SDK with all of its +# tool-side dependencies so the runtime stage can copy them in as a +# pre-built artifact (no `pip install` happens at sandbox startup). +# --------------------------------------------------------------------- +FROM python:3.11-slim AS builder + +ARG MIRROR +ARG APT_MIRROR +ARG TARGETARCH +LABEL authors="nexent" + +USER root +RUN umask 0022 + +# Configure apt sources based on build argument (mirrors main/Dockerfile). +RUN --mount=type=cache,id=nexent-sandbox-apt-cache-${TARGETARCH},target=/var/cache/apt,sharing=locked \ + --mount=type=cache,id=nexent-sandbox-apt-lists-${TARGETARCH},target=/var/lib/apt/lists,sharing=locked \ + rm -f /etc/apt/apt.conf.d/docker-clean && \ + mkdir -p /var/cache/apt/archives /var/lib/apt/lists/partial && \ + if [ "$APT_MIRROR" = "tsinghua" ]; then \ + for apt_source in /etc/apt/sources.list /etc/apt/sources.list.d/*.sources /etc/apt/sources.list.d/*.list; do \ + [ -f "$apt_source" ] || continue; \ + sed -i \ + -e 's|https\?://deb.debian.org/debian-security|http://security.debian.org/debian-security|g' \ + -e 's|https\?://security.debian.org/debian-security|http://security.debian.org/debian-security|g' \ + -e 's|https\?://deb.debian.org/debian|http://mirrors.tuna.tsinghua.edu.cn/debian|g' \ + "$apt_source"; \ + done; \ + fi && \ + apt-get update && \ + apt-get install -y --no-install-recommends curl build-essential + +RUN --mount=type=cache,id=nexent-sandbox-pip-${TARGETARCH},target=/root/.cache/pip,sharing=locked \ + pip install uv $(test -n "$MIRROR" && echo "-i $MIRROR") + +WORKDIR /opt + +# Copy SDK source first (and pyproject.toml) so uv can resolve deps. +# The sandbox does not run `pip install` at startup; everything here is +# pre-installed in the image so that DockerExecutor.start_kernel_gateway +# can hand control to a Jupyter kernel immediately. +COPY sdk/pyproject.toml /opt/sdk/pyproject.toml +COPY sdk/nexent /opt/sdk/nexent + +# Install SDK into the system site-packages with link-mode copy so the +# runtime stage can COPY the site-packages wholesale. `--system` matches +# the python:3.11-slim base layout; no venv is created. +RUN --mount=type=cache,id=nexent-sandbox-uv-${TARGETARCH},target=/root/.cache/uv,sharing=locked \ + uv pip install --system --link-mode copy "/opt/sdk" $(test -n "$MIRROR" && echo "-i $MIRROR") + +# --------------------------------------------------------------------- +# Stage 2: runtime - minimal Python image with Jupyter Kernel Gateway +# (smolagents DockerExecutor's default IPC channel) and a non-root +# `sandbox` user. The DockerExecutor will run this image with +# network_mode=none and per-run resource limits; the image itself only +# enforces the in-container invariants (non-root, workdir, libs). +# --------------------------------------------------------------------- +FROM python:3.11-slim AS runtime + +ARG APT_MIRROR +ARG TARGETARCH +LABEL authors="nexent" + +USER root +RUN umask 0022 + +# Minimal apt surface: curl is required by some tools to fetch the +# /files endpoints; libgomp1 is needed by numpy/scipy wheels. +RUN --mount=type=cache,id=nexent-sandbox-apt-cache-${TARGETARCH},target=/var/cache/apt,sharing=locked \ + --mount=type=cache,id=nexent-sandbox-apt-lists-${TARGETARCH},target=/var/lib/apt/lists,sharing=locked \ + rm -f /etc/apt/apt.conf.d/docker-clean && \ + mkdir -p /var/cache/apt/archives /var/lib/apt/lists/partial && \ + if [ "$APT_MIRROR" = "tsinghua" ]; then \ + for apt_source in /etc/apt/sources.list /etc/apt/sources.list.d/*.sources /etc/apt/sources.list.d/*.list; do \ + [ -f "$apt_source" ] || continue; \ + sed -i \ + -e 's|https\?://deb.debian.org/debian-security|http://security.debian.org/debian-security|g' \ + -e 's|https\?://security.debian.org/debian-security|http://security.debian.org/debian-security|g' \ + -e 's|https\?://deb.debian.org/debian|http://mirrors.tuna.tsinghua.edu.cn/debian|g' \ + "$apt_source"; \ + done; \ + fi && \ + apt-get update && \ + apt-get install -y --no-install-recommends \ + curl \ + libgomp1 \ + && rm -rf /var/lib/apt/lists/* \ + && apt-get clean + +# Install Jupyter Kernel Gateway - the IPC channel smolagents DockerExecutor +# uses to send code actions and stream back observations. +RUN --mount=type=cache,id=nexent-sandbox-pip-${TARGETARCH},target=/root/.cache/pip,sharing=locked \ + pip install --no-cache-dir \ + jupyter_kernel_gateway \ + jupyter_client \ + ipykernel + +# Bring in the pre-built SDK (source tree) and its resolved dependencies +# from the builder. /usr/local/lib/python3.11/site-packages is the install +# target of `uv pip install --system` on python:3.11-slim. +COPY --from=builder /opt/sdk /opt/sdk +COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages + +# --------------------------------------------------------------------- +# Stage 2.5: pre-install the LLM-friendly Python library whitelist +# (design doc §6.4). Versions are pinned with upper bounds so upstream +# releases do not silently break reproducibility. No auto-`pip install` +# happens at sandbox execution time; missing imports return a structured +# diagnostic that nudges the LLM toward supported packages. +# --------------------------------------------------------------------- +FROM runtime AS with-libs + +ARG MIRROR + +RUN --mount=type=cache,id=nexent-sandbox-pip-${TARGETARCH},target=/root/.cache/pip,sharing=locked \ + pip install --no-cache-dir \ + "numpy>=1.26.4,<2.0" \ + "pandas>=2.0,<3.0" \ + "requests>=2.31,<3.0" \ + "pillow>=10.0,<11.0" \ + "beautifulsoup4>=4.12,<5.0" \ + "lxml>=4.9,<5.0" \ + "openpyxl>=3.1,<4.0" \ + "matplotlib>=3.7,<4.0" \ + "scipy>=1.11,<2.0" \ + "scikit-learn>=1.3,<2.0" \ + "pydantic>=2.0,<3.0" \ + "sympy>=1.12,<2.0" \ + "statsmodels>=0.14,<1.0" \ + "tabulate>=0.9,<1.0" \ + "python-dateutil>=2.8,<3.0" \ + $(test -n "$MIRROR" && echo "-i $MIRROR") \ + && python -c "import numpy, pandas, requests, PIL, bs4, lxml, openpyxl, matplotlib, scipy, sklearn, pydantic, sympy, statsmodels, tabulate, dateutil; print('OK')" + +# Create a non-root user so the LLM code cannot write outside /home/sandbox +# (combined with network_mode=none + run-scoped resource limits, this is +# the in-container half of the security boundary; the host-side halves +# are enforced by the DockerExecutor at run start). +RUN useradd --uid 1000 --create-home --shell /bin/bash sandbox && \ + mkdir -p /home/sandbox/workdir/input \ + /home/sandbox/workdir/output \ + /home/sandbox/workdir/tmp && \ + chown -R sandbox:sandbox /home/sandbox + +WORKDIR /home/sandbox/workdir +USER sandbox + +# Jupyter Kernel Gateway is the IPC channel required by smolagents +# DockerExecutor. The Executor passes --gateway-url pointing at this port. +EXPOSE 8888 +CMD ["jupyter", "kernelgateway", \ + "--KernelGatewayApp.ip=0.0.0.0", \ + "--KernelGatewayApp.port=8888", \ + "--KernelGatewayApp.allow_origin=*", \ + "--ServerApp.allow_remote_access=True", \ + "--JupyterWebsocketPersonality.list_kernels=True"] \ No newline at end of file diff --git a/deploy/images/dockerfiles/web/Dockerfile b/deploy/images/dockerfiles/web/Dockerfile index e2be8c6918..5c3707d73c 100644 --- a/deploy/images/dockerfiles/web/Dockerfile +++ b/deploy/images/dockerfiles/web/Dockerfile @@ -2,6 +2,9 @@ FROM node:20-alpine AS builder ARG MIRROR ARG TARGETARCH +ARG CONFIGURED_BASE_PATH=/ + +ENV NEXT_PUBLIC_BASE_PATH=${CONFIGURED_BASE_PATH} # Build Next.js application WORKDIR /opt/frontend @@ -10,7 +13,7 @@ COPY frontend/package.json ./package.json # Use BuildKit named cache for npm downloads across builds. RUN --mount=type=cache,id=nexent-web-npm-${TARGETARCH},target=/root/.npm,sharing=locked \ if [ -n "$MIRROR" ]; then npm config set registry "$MIRROR"; fi && \ - npm install --verbose + npm install --verbose --legacy-peer-deps COPY frontend /opt/frontend @@ -22,6 +25,8 @@ RUN --mount=type=cache,id=nexent-web-next-${TARGETARCH},target=/opt/frontend/.ne cp -r .next/static ../frontend-dist/.next/static && \ cp -r public ../frontend-dist/ && \ cp server.js ../frontend-dist/server.js && \ + cp base-path.mjs ../frontend-dist/base-path.mjs && \ + cp build-config.js ../frontend-dist/build-config.js && \ mkdir -p ../frontend-dist/node_modules/next/dist/compiled && \ cp -r node_modules/next/dist/compiled/. ../frontend-dist/node_modules/next/dist/compiled/ && \ mkdir -p ../frontend-dist/node_modules && \ @@ -32,6 +37,16 @@ RUN --mount=type=cache,id=nexent-web-next-${TARGETARCH},target=/opt/frontend/.ne node_modules/follow-redirects \ node_modules/http-proxy \ node_modules/requires-port \ + node_modules/multiparty \ + node_modules/safe-buffer \ + node_modules/http-errors \ + node_modules/depd \ + node_modules/uid-safe \ + node_modules/random-bytes \ + node_modules/inherits \ + node_modules/statuses \ + node_modules/toidentifier \ + node_modules/setprototypeof \ ../frontend-dist/node_modules/ && \ rm -rf ../frontend-dist/.next/cache @@ -39,6 +54,7 @@ RUN --mount=type=cache,id=nexent-web-next-${TARGETARCH},target=/opt/frontend/.ne FROM node:20-alpine ARG APK_MIRROR ARG TARGETARCH +ARG CONFIGURED_BASE_PATH=/ LABEL authors="nexent" # Configure Alpine mirrors if specified @@ -62,6 +78,7 @@ COPY --from=builder /opt/frontend-dist . ENV NODE_ENV=production ENV HOSTNAME=localhost +ENV NEXT_PUBLIC_BASE_PATH=${CONFIGURED_BASE_PATH} # Expose the service port EXPOSE 3000 diff --git a/deploy/k8s/create-suadmin.sh b/deploy/k8s/create-suadmin.sh index 476fe7f91b..a572ea58b8 100644 --- a/deploy/k8s/create-suadmin.sh +++ b/deploy/k8s/create-suadmin.sh @@ -70,23 +70,30 @@ prompt_super_admin_password() { return 1 } -# Wait for PostgreSQL pod to be ready -wait_for_nexent_postgresql_ready() { - local retries=0 - local max_retries=${1:-30} - - while [ $retries -lt $max_retries ]; do - if kubectl exec -n $NAMESPACE deploy/nexent-postgresql -- pg_isready -U root -d nexent >/dev/null 2>&1; then - echo " ✅ PostgreSQL is now ready!" +# Wait until migrations have created the table and every column used below. +wait_for_user_tenant_schema_ready() { + local timeout="${NEXENT_SQL_MIGRATION_WAIT_TIMEOUT_SECONDS:-300}" + local interval="${NEXENT_SQL_MIGRATION_WAIT_INTERVAL_SECONDS:-2}" + local start + local contract_sql="SELECT user_id, tenant_id, user_role, user_email, created_by, updated_by FROM nexent.user_tenant_t LIMIT 0;" + + start="$(date +%s)" + while true; do + if kubectl exec -n "$NAMESPACE" deploy/nexent-postgresql -- \ + psql -U root -d nexent -X -v ON_ERROR_STOP=1 \ + -c "$contract_sql" >/dev/null 2>&1; then + echo " ✅ user_tenant_t schema is ready." return 0 fi - echo " ⏳ Waiting for PostgreSQL to become ready... (attempt $((retries + 1))/$max_retries)" - sleep 10 - retries=$((retries + 1)) - done - echo " ⚠️ Warning: PostgreSQL did not become ready within expected time" - return 1 + if [ $(( $(date +%s) - start )) -ge "$timeout" ]; then + echo " ❌ user_tenant_t schema did not become ready within ${timeout}s." + return 1 + fi + + echo " ⏳ Waiting for user_tenant_t schema migration to complete..." + sleep "$interval" + done } decode_base64() { @@ -141,9 +148,15 @@ extract_supabase_user_id() { get_existing_super_admin_user_id() { local email="$1" - kubectl exec -n "$NAMESPACE" deploy/nexent-supabase-db -- \ + local result + + if ! result="$(kubectl exec -n "$NAMESPACE" deploy/nexent-supabase-db -- \ psql -U postgres -d supabase -X -A -t -v ON_ERROR_STOP=1 \ - -c "SELECT id FROM auth.users WHERE email = '${email}' LIMIT 1;" 2>/dev/null | tr -d '[:space:]' + -c "SELECT id FROM auth.users WHERE email = '${email}' LIMIT 1;" 2>/dev/null)"; then + return 1 + fi + + printf '%s' "$result" | tr -d '[:space:]' } wait_for_supabase_auth_table_ready() { @@ -173,42 +186,45 @@ insert_super_admin_tenant_record() { local postgres_pod="nexent-postgresql" if [ -z "$user_id" ]; then - echo " ⚠️ Warning: user_id is empty. Skipping database insertion." - return 0 + echo " ❌ Cannot insert super admin tenant record: user_id is empty." + return 1 fi - echo " ⏳ Waiting for PostgreSQL to be ready..." - if ! wait_for_nexent_postgresql_ready; then - echo " ⚠️ Warning: PostgreSQL is not ready. Skipping database insertion." - return 0 - fi + wait_for_user_tenant_schema_ready || return 1 echo " 🔧 Inserting super admin user into user_tenant_t table..." local sql="INSERT INTO nexent.user_tenant_t (user_id, tenant_id, user_role, user_email, created_by, updated_by) VALUES ('${user_id}', '', 'SU', '${email}', 'system', 'system') ON CONFLICT (user_id, tenant_id) DO NOTHING;" - if kubectl exec -n "$NAMESPACE" deploy/$postgres_pod -- psql -U root -d nexent -c "$sql" >/dev/null 2>&1; then + if kubectl exec -n "$NAMESPACE" "deploy/$postgres_pod" -- \ + psql -U root -d nexent -X -v ON_ERROR_STOP=1 -c "$sql" >/dev/null 2>&1; then echo " ✅ Super admin user inserted into user_tenant_t table successfully." - else - echo " ⚠️ Warning: Failed to insert super admin user into user_tenant_t table." + return 0 fi + + echo " ❌ Failed to insert super admin user into user_tenant_t table." + return 1 } # Create default super admin user create_supabase_super_admin_user() { local email="$SUPER_ADMIN_EMAIL" local password + local display_password="true" if ! wait_for_supabase_auth_table_ready; then - echo " 💡 The super admin user will not be created, but deployment will continue." - return 0 + echo " ❌ Supabase auth database did not become ready." + return 1 fi local existing_user_id - existing_user_id="$(get_existing_super_admin_user_id "$email")" + if ! existing_user_id="$(get_existing_super_admin_user_id "$email")"; then + echo " ❌ Failed to query the existing super admin user_id." + return 1 + fi if [ -n "$existing_user_id" ]; then echo " 🚧 Default super admin user already exists. Skipping password setup." echo " 📧 Email: ${email}" - insert_super_admin_tenant_record "$existing_user_id" "$email" + insert_super_admin_tenant_record "$existing_user_id" "$email" || return 1 echo "" echo "--------------------------------" echo "" @@ -227,8 +243,14 @@ create_supabase_super_admin_user() { return 1 fi - # Prompt user to enter password only when the user does not exist. - password="$(prompt_super_admin_password)" || return 1 + # Offline --config deployments remain interactive. Other deployments use the + # configured password or the shared non-interactive default. + if deployment_should_prompt_super_admin_password; then + password="$(prompt_super_admin_password)" || return 1 + display_password="false" + else + password="$(deployment_super_admin_password)" + fi local payload payload="{\"email\":\"$(json_escape "$email")\",\"password\":\"$(json_escape "$password")\",\"email_confirm\":true}" @@ -278,7 +300,11 @@ create_supabase_super_admin_user() { echo "" echo " Please save the following credentials carefully." echo " 📧 Email: ${email}" - echo " 🔏 Password: [hidden]" + if [ "$display_password" = "true" ]; then + echo " 🔏 Password: ${password}" + else + echo " 🔏 Password: [hidden]" + fi local user_id user_id="$response_user_id" @@ -288,9 +314,10 @@ create_supabase_super_admin_user() { fi if [ -z "$user_id" ]; then - echo " ⚠️ Warning: Could not retrieve user_id. Skipping database insertion." + echo " ❌ Could not retrieve the created super admin user_id." + return 1 else - insert_super_admin_tenant_record "$user_id" "$email" + insert_super_admin_tenant_record "$user_id" "$email" || return 1 fi elif echo "$signup_response" | grep -q '"error_code":"user_already_exists"' || \ echo "$signup_response" | grep -q '"code":422' || \ @@ -302,22 +329,27 @@ create_supabase_super_admin_user() { # Get user_id from Supabase auth.users table echo " 🔧 Retrieving user_id from Supabase database..." local user_id - user_id="$(get_existing_super_admin_user_id "$email")" + if ! user_id="$(get_existing_super_admin_user_id "$email")"; then + echo " ❌ Failed to query the existing super admin user_id." + return 1 + fi if [ -z "$user_id" ]; then - echo " ⚠️ Warning: Could not retrieve user_id. Skipping database insertion." - echo " 💡 Note: If user_tenant_t record is missing, you may need to insert it manually." - return 0 + echo " ❌ Could not retrieve the existing super admin user_id." + return 1 fi - insert_super_admin_tenant_record "$user_id" "$email" + insert_super_admin_tenant_record "$user_id" "$email" || return 1 else local user_id - user_id="$(get_existing_super_admin_user_id "$email")" + if ! user_id="$(get_existing_super_admin_user_id "$email")"; then + echo " ❌ Failed to query the existing super admin user_id." + return 1 + fi if [ -n "$user_id" ]; then echo " 🚧 Default super admin user already exists. Skipping creation." echo " 📧 Email: ${email}" - insert_super_admin_tenant_record "$user_id" "$email" + insert_super_admin_tenant_record "$user_id" "$email" || return 1 return 0 fi @@ -343,23 +375,20 @@ main() { # Wait for supabase-kong if ! kubectl wait --for=condition=ready pod -l app=nexent-supabase-kong -n $NAMESPACE --timeout=180s 2>/dev/null; then - echo " ⚠️ Warning: Supabase Kong pod is not ready yet." - echo " 💡 The super admin user will not be created, but deployment will continue." - return 0 + echo " ❌ Supabase Kong pod is not ready." + return 1 fi # Wait for supabase-db if ! kubectl wait --for=condition=ready pod -l app=nexent-supabase-db -n $NAMESPACE --timeout=180s 2>/dev/null; then - echo " ⚠️ Warning: Supabase DB pod is not ready yet." - echo " 💡 The super admin user will not be created, but deployment will continue." - return 0 + echo " ❌ Supabase DB pod is not ready." + return 1 fi # Wait for supabase-auth if ! kubectl wait --for=condition=ready pod -l app=nexent-supabase-auth -n $NAMESPACE --timeout=180s 2>/dev/null; then - echo " ⚠️ Warning: Supabase Auth pod is not ready yet." - echo " 💡 The super admin user will not be created, but deployment will continue." - return 0 + echo " ❌ Supabase Auth pod is not ready." + return 1 fi # Create super admin user @@ -370,5 +399,5 @@ main() { fi } -# Run main function +# Run main function. main "$@" diff --git a/deploy/k8s/deploy.sh b/deploy/k8s/deploy.sh index a335df9338..65eeb97d86 100755 --- a/deploy/k8s/deploy.sh +++ b/deploy/k8s/deploy.sh @@ -962,6 +962,48 @@ pull_mcp_image() { echo "" } +# Pull sandbox Docker image to local host (best-effort) +pull_sandbox_image() { + echo "==========================================" + echo " Sandbox Image Pull" + echo "==========================================" + + local image="${NEXENT_SANDBOX_IMAGE:-nexent/nexent-sandbox}" + local image_tail="${image##*/}" + local sandbox_image_name="$image" + if [[ "$image_tail" != *:* ]]; then + sandbox_image_name="${image}:${APP_VERSION:-latest}" + fi + echo "Checking sandbox image: ${sandbox_image_name}" + + if ! command -v docker >/dev/null 2>&1; then + echo "Warning: Docker is not installed or not in PATH, skipping sandbox image pull." + echo "" + echo "--------------------------------" + echo "" + return 0 + fi + + if docker image inspect "${sandbox_image_name}" >/dev/null 2>&1; then + echo "Sandbox image already exists locally, skipping pull." + elif [ "$DEPLOYMENT_IMAGE_SOURCE" = "local-latest" ]; then + echo "Warning: Sandbox local image not found: ${sandbox_image_name}" + echo "Build or load it locally before using --image-source local-latest." + else + echo "Sandbox image not found locally, pulling..." + if docker pull "${sandbox_image_name}"; then + echo "Sandbox image pulled successfully." + else + echo "Warning: Failed to pull sandbox image, but deployment will continue." + echo "You can pull it manually later: docker pull ${sandbox_image_name}" + fi + fi + + echo "" + echo "--------------------------------" + echo "" +} + render_runtime_secret_values() { local gotrue_db_url local env_checksum @@ -1272,17 +1314,19 @@ apply() { fi else if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then - echo "警告：超级管理员创建失败，但部署将继续。" + echo "错误：超级管理员创建失败，部署已终止。" else - echo "Warning: Super admin user creation failed, but continuing deployment." + echo "Error: Super admin user creation failed. Deployment aborted." fi + exit 1 fi else if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then - echo "警告：未找到 create-suadmin.sh：$CREATE_SUADMIN_SCRIPT" + echo "错误：未找到 create-suadmin.sh：$CREATE_SUADMIN_SCRIPT" else - echo "Warning: create-suadmin.sh not found at $CREATE_SUADMIN_SCRIPT" + echo "Error: create-suadmin.sh not found at $CREATE_SUADMIN_SCRIPT" fi + exit 1 fi fi @@ -1292,6 +1336,9 @@ apply() { # Step 11: Pull MCP image after persisting deployment options pull_mcp_image + # Step 12: Pull sandbox image for agent sandbox runs + pull_sandbox_image + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then echo "部署完成！" echo "应用访问地址：http://localhost:30000" diff --git a/deploy/k8s/helm/nexent/README.md b/deploy/k8s/helm/nexent/README.md index 72e7997bcd..456819d253 100644 --- a/deploy/k8s/helm/nexent/README.md +++ b/deploy/k8s/helm/nexent/README.md @@ -69,7 +69,7 @@ bash uninstall.sh k8s delete-all bash uninstall.sh k8s delete-all --keep-local-data ``` -K8s deployments read runtime configuration from `deploy/env/.env`, the same file used by Docker. Existing `deploy/env/.env` is kept as-is. If it is missing, the deploy script first reuses `docker/.env`, then falls back to `deploy/env/.env.example`. Do not edit generated Helm values by hand; they are recreated from `deploy/env/.env` and deployment options. +K8s deployments read runtime configuration from `deploy/env/.env`, the same file used by Docker. Before every deployment, existing values, comments, ordering, and old-only variables are preserved while assignments newly introduced by the current `deploy/env/.env.example` are appended. If `.env` is missing, the deploy script first reuses legacy `docker/.env`, then falls back to the current template. A readable template is required before deployment starts. Do not edit generated Helm values by hand; they are recreated from the merged `deploy/env/.env` and deployment options. When `--persistence-mode local` is used, Nexent renders static PVs with `hostPath` and `DirectoryOrCreate`; node affinity is not required. Shared workspace data uses `/var/lib/nexent`, shared skills use `/var/lib/nexent-data/skills`, and service data uses `/var/lib/nexent-data/nexent-*` by default. diff --git a/deploy/k8s/helm/nexent/charts/nexent-common/templates/configmap.yaml b/deploy/k8s/helm/nexent/charts/nexent-common/templates/configmap.yaml index e70602aafe..de8fedcc38 100644 --- a/deploy/k8s/helm/nexent/charts/nexent-common/templates/configmap.yaml +++ b/deploy/k8s/helm/nexent/charts/nexent-common/templates/configmap.yaml @@ -155,6 +155,9 @@ data: # MCP Container Image NEXENT_MCP_DOCKER_IMAGE: {{ printf "%s:%s" .Values.images.mcp.repository .Values.images.mcp.tag | quote }} + # Sandbox Container Image + NEXENT_SANDBOX_DOCKER_IMAGE: {{ printf "%s:%s" .Values.images.sandbox.repository .Values.images.sandbox.tag | quote }} + # Kubernetes Deployment Mode IS_DEPLOYED_BY_KUBERNETES: {{ .Values.config.isDeployedByKubernetes | quote }} KUBERNETES_NAMESPACE: {{ .Values.global.namespace | quote }} diff --git a/deploy/k8s/helm/nexent/charts/nexent-common/values.yaml b/deploy/k8s/helm/nexent/charts/nexent-common/values.yaml index 5882585de8..b8f8aded1e 100644 --- a/deploy/k8s/helm/nexent/charts/nexent-common/values.yaml +++ b/deploy/k8s/helm/nexent/charts/nexent-common/values.yaml @@ -8,6 +8,10 @@ images: repository: "nexent/nexent-mcp" tag: "latest" pullPolicy: IfNotPresent + sandbox: + repository: "nexent/nexent-sandbox" + tag: "latest" + pullPolicy: IfNotPresent # SQL content is rendered by deploy/k8s/deploy.sh from deploy/sql/ # directory. Keep this empty in chart defaults to avoid maintaining a second SQL @@ -54,6 +58,7 @@ config: endpoint: "http://nexent-minio:9000" region: "cn-north-1" defaultBucket: "nexent" + secure: "true" elasticsearch: host: "http://nexent-elasticsearch:9200" javaOpts: "-Xms2g -Xmx2g" @@ -87,7 +92,7 @@ config: supabaseUrl: "http://nexent-supabase-kong:8000" apiExternalUrl: "http://nexent-supabase-kong:8000" disableSignup: "false" - jwtExpiry: "3600" + jwtExpiry: "7200" debugJwtExpireSeconds: "0" enableEmailSignup: "true" enableEmailAutoconfirm: "true" @@ -184,6 +189,28 @@ config: logoutUrl: "" sslVerify: "true" caBundle: "" + sandbox: + # Default sandbox isolation level: local / docker / wasm. + # 'local' preserves backward-compatibility for existing deployments. + defaultLevel: "local" + # Default sandbox container lifecycle scope: session / system. + # session = one container per agent_run, destroyed on run end (strictest isolation). + # system = persistent warm pool shared by all runs (lowest cold-start latency). + defaultScope: "session" + # Sandbox resource limits. + memoryLimitMb: "512" + cpuQuota: "1.0" + # Sandbox execution timeout per step (seconds). + timeoutSeconds: "30" + # Sandbox network policy: enabled / disabled. + network: "disabled" + # Shell execution policy: disabled / restricted / boxed. + # 'disabled' is recommended — blocks subprocess/os shell calls at AST-parse time. + shellPolicy: "disabled" + # MinIO bucket for sandbox output file sync. + outputBucket: "nexent-artifacts" + # Automatically sync sandbox output files to MinIO after each run. + autoSyncOutputs: "true" # Secrets used by common templates secrets: diff --git a/deploy/k8s/helm/nexent/charts/nexent-monitoring/values.yaml b/deploy/k8s/helm/nexent/charts/nexent-monitoring/values.yaml index 4ef831ef73..b126483d79 100644 --- a/deploy/k8s/helm/nexent/charts/nexent-monitoring/values.yaml +++ b/deploy/k8s/helm/nexent/charts/nexent-monitoring/values.yaml @@ -41,7 +41,7 @@ images: tag: "26.3-alpine" pullPolicy: IfNotPresent minio: - repository: docker.io/minio/minio + repository: quay.io/minio/minio tag: "RELEASE.2023-12-20T01-00-02Z" pullPolicy: IfNotPresent redis: diff --git a/deploy/k8s/helm/nexent/charts/nexent-supabase-auth/values.yaml b/deploy/k8s/helm/nexent/charts/nexent-supabase-auth/values.yaml index 8079075153..80535347c7 100644 --- a/deploy/k8s/helm/nexent/charts/nexent-supabase-auth/values.yaml +++ b/deploy/k8s/helm/nexent/charts/nexent-supabase-auth/values.yaml @@ -22,7 +22,7 @@ config: siteUrl: "http://localhost:3011" apiExternalUrl: "http://nexent-supabase-kong:8000" disableSignup: false - jwtExpiry: "3600" + jwtExpiry: "7200" debugJwtExpireSeconds: "0" enableEmailSignup: true enableEmailAutoconfirm: true diff --git a/deploy/k8s/helm/nexent/charts/nexent-supabase-db/values.yaml b/deploy/k8s/helm/nexent/charts/nexent-supabase-db/values.yaml index c5a156683b..72b7b96691 100644 --- a/deploy/k8s/helm/nexent/charts/nexent-supabase-db/values.yaml +++ b/deploy/k8s/helm/nexent/charts/nexent-supabase-db/values.yaml @@ -27,4 +27,4 @@ persistence: config: postgresDb: "supabase" postgresPort: "5436" - jwtExpiry: "3600" + jwtExpiry: "7200" diff --git a/deploy/k8s/helm/nexent/charts/nexent-web/templates/deployment.yaml b/deploy/k8s/helm/nexent/charts/nexent-web/templates/deployment.yaml index af07956e56..e8c99a0be9 100644 --- a/deploy/k8s/helm/nexent/charts/nexent-web/templates/deployment.yaml +++ b/deploy/k8s/helm/nexent/charts/nexent-web/templates/deployment.yaml @@ -29,6 +29,9 @@ spec: ports: - containerPort: 3000 name: http + envFrom: + - configMapRef: + name: nexent-config env: - name: HOSTNAME value: "0.0.0.0" diff --git a/deploy/offline/build_offline_package.sh b/deploy/offline/build_offline_package.sh index afae3ead60..05bc767827 100755 --- a/deploy/offline/build_offline_package.sh +++ b/deploy/offline/build_offline_package.sh @@ -12,6 +12,7 @@ DEFAULT_VERSION="latest" DEFAULT_PLATFORM="amd64" DEFAULT_OUTPUT_DIR="$PROJECT_ROOT/offline-package" DEFAULT_INCLUDE_SOURCE="false" +DEFAULT_INCLUDE_SANDBOX="true" DEFAULT_TARGET="all" DEFAULT_COMPRESS="false" @@ -19,6 +20,7 @@ VERSION="" PLATFORM="" OUTPUT_DIR="" INCLUDE_SOURCE="" +INCLUDE_SANDBOX="" TARGET="" COMPRESS="" DRY_RUN="false" @@ -52,6 +54,8 @@ show_help() { echo " 默认：$DEFAULT_OUTPUT_DIR" echo " --include-source BOOL 是否包含源码（true 或 false）" echo " 默认：$DEFAULT_INCLUDE_SOURCE" + echo " --include-sandbox BOOL 是否包含 Sandbox 镜像（true 或 false）" + echo " 默认：$DEFAULT_INCLUDE_SANDBOX" echo " --target TARGET docker、k8s 或 all" echo " 默认：$DEFAULT_TARGET" echo " --compress BOOL 构建后是否创建 zip 压缩包（true 或 false）" @@ -86,6 +90,8 @@ show_help() { echo " Default: $DEFAULT_OUTPUT_DIR" echo " --include-source BOOL Include source code (true or false)" echo " Default: $DEFAULT_INCLUDE_SOURCE" + echo " --include-sandbox BOOL Include the Sandbox image (true or false)" + echo " Default: $DEFAULT_INCLUDE_SANDBOX" echo " --target TARGET docker, k8s, or all" echo " Default: $DEFAULT_TARGET" echo " --compress BOOL Create zip archive after package build (true or false)" @@ -127,6 +133,10 @@ parse_args() { INCLUDE_SOURCE="$2" shift 2 ;; + --include-sandbox) + INCLUDE_SANDBOX="$2" + shift 2 + ;; --target) TARGET="$2" shift 2 @@ -171,6 +181,7 @@ parse_args() { PLATFORM="${PLATFORM:-$DEFAULT_PLATFORM}" OUTPUT_DIR="${OUTPUT_DIR:-$DEFAULT_OUTPUT_DIR}" INCLUDE_SOURCE="${INCLUDE_SOURCE:-$DEFAULT_INCLUDE_SOURCE}" + INCLUDE_SANDBOX="${INCLUDE_SANDBOX:-$DEFAULT_INCLUDE_SANDBOX}" TARGET="${TARGET:-$DEFAULT_TARGET}" COMPRESS="${COMPRESS:-$DEFAULT_COMPRESS}" @@ -198,6 +209,14 @@ parse_args() { fi exit 1 fi + if [[ "$INCLUDE_SANDBOX" != "true" && "$INCLUDE_SANDBOX" != "false" ]]; then + if [ "$DEPLOYMENT_LANGUAGE" = "zh" ]; then + echo "错误：Include sandbox 必须是 'true' 或 'false'" + else + echo "Error: Include sandbox must be 'true' or 'false'" + fi + exit 1 + fi } prepare_deployment_image_config() { @@ -223,6 +242,7 @@ show_dry_run_plan() { echo "平台：$PLATFORM" echo "输出目录：$OUTPUT_DIR" echo "包含源码：$INCLUDE_SOURCE" + echo "包含 Sandbox 镜像：$INCLUDE_SANDBOX" echo "目标：$TARGET" echo "压缩：$COMPRESS" echo "组件：$DEPLOYMENT_COMPONENTS" @@ -242,6 +262,7 @@ show_dry_run_plan() { echo "Platform: $PLATFORM" echo "Output directory: $OUTPUT_DIR" echo "Include source: $INCLUDE_SOURCE" + echo "Include Sandbox image: $INCLUDE_SANDBOX" echo "Target: $TARGET" echo "Compress: $COMPRESS" echo "Components: $DEPLOYMENT_COMPONENTS" @@ -262,6 +283,7 @@ get_nexent_images() { deployment_csv_contains "$DEPLOYMENT_COMPONENTS" "application" && echo "$NEXENT_MCP_DOCKER_IMAGE" deployment_csv_contains "$DEPLOYMENT_COMPONENTS" "data-process" && echo "$NEXENT_DATA_PROCESS_IMAGE" deployment_csv_contains "$DEPLOYMENT_COMPONENTS" "terminal" && echo "$OPENSSH_SERVER_IMAGE" + [ "$INCLUDE_SANDBOX" = "true" ] && echo "$NEXENT_SANDBOX_IMAGE" true } @@ -297,7 +319,7 @@ get_third_party_images() { "docker.io/langfuse/langfuse-worker:3" \ "docker.io/langfuse/langfuse:3" \ "docker.io/clickhouse/clickhouse-server:26.3-alpine" \ - "docker.io/minio/minio:RELEASE.2023-12-20T01-00-02Z" \ + "quay.io/minio/minio:RELEASE.2023-12-20T01-00-02Z" \ "docker.io/redis:alpine" \ "docker.io/postgres:15-alpine"; do echo_image_ref "$image" @@ -649,6 +671,7 @@ create_manifest() { echo "platform: \"$PLATFORM\"" echo "target: \"$TARGET\"" echo "components: \"$DEPLOYMENT_COMPONENTS\"" + echo "includeSandbox: \"$INCLUDE_SANDBOX\"" echo "imageSource: \"$DEPLOYMENT_IMAGE_SOURCE\"" echo "imageRegistryPrefix: \"$DEPLOYMENT_IMAGE_REGISTRY_PREFIX\"" echo "images:" diff --git a/deploy/sql/init.sql b/deploy/sql/init.sql index f62970a66b..174c4beb2f 100644 --- a/deploy/sql/init.sql +++ b/deploy/sql/init.sql @@ -190,20 +190,6 @@ COMMENT ON COLUMN "model_record_t"."update_time" IS 'Update time, audit field'; COMMENT ON COLUMN "model_record_t"."updated_by" IS 'Last updater ID, audit field'; COMMENT ON COLUMN "model_record_t"."created_by" IS 'Creator ID, audit field'; COMMENT ON TABLE "model_record_t" IS 'List of models defined by users in the configuration page'; - -INSERT INTO "nexent"."model_record_t" ("model_repo", "model_name", "model_factory", "model_type", "api_key", "base_url", "max_tokens", "used_token", "display_name", "connect_status") -SELECT '', 'volcano_tts', 'OpenAI-API-Compatible', 'tts', '', '', 0, 0, 'volcano_tts', 'unavailable' -WHERE NOT EXISTS ( - SELECT 1 FROM "nexent"."model_record_t" - WHERE "model_name" = 'volcano_tts' AND "model_type" = 'tts' -); -INSERT INTO "nexent"."model_record_t" ("model_repo", "model_name", "model_factory", "model_type", "api_key", "base_url", "max_tokens", "used_token", "display_name", "connect_status") -SELECT '', 'volcano_stt', 'OpenAI-API-Compatible', 'stt', '', '', 0, 0, 'volcano_stt', 'unavailable' -WHERE NOT EXISTS ( - SELECT 1 FROM "nexent"."model_record_t" - WHERE "model_name" = 'volcano_stt' AND "model_type" = 'stt' -); - CREATE TABLE IF NOT EXISTS "knowledge_record_t" ( "knowledge_id" SERIAL, "index_name" varchar(100) COLLATE "pg_catalog"."default", @@ -297,6 +283,7 @@ CREATE TABLE IF NOT EXISTS nexent.ag_tenant_agent_t ( tenant_id VARCHAR(100), enabled BOOLEAN DEFAULT FALSE, provide_run_summary BOOLEAN DEFAULT FALSE, + context_policy JSONB, create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, created_by VARCHAR(100), @@ -443,4 +430,3 @@ EXECUTE FUNCTION update_ag_tool_instance_update_time(); -- Add comment to the trigger COMMENT ON TRIGGER update_ag_tool_instance_update_time_trigger ON nexent.ag_tool_instance_t IS 'Trigger to call update_ag_tool_instance_update_time function before each update on ag_tool_instance_t table'; - diff --git a/deploy/sql/migrations/README.md b/deploy/sql/migrations/README.md index 5c18bf2c04..410c4a5181 100644 --- a/deploy/sql/migrations/README.md +++ b/deploy/sql/migrations/README.md @@ -17,3 +17,8 @@ again. Use patterns such as `CREATE TABLE IF NOT EXISTS`, `ALTER TABLE ... ADD COLUMN IF NOT EXISTS`, and conflict-safe inserts where possible. `deploy/sql/init.sql` is the initial baseline before these incremental files. + +Historical migrations through v2.4.0 are consolidated by minor version in +`v2.2_merged_migrations.sql`, `v2.3_merged_migrations.sql`, and +`v2.4_merged_migrations.sql`. Newer migrations remain separate until their +minor-version history is consolidated. diff --git a/deploy/sql/migrations/generate_backfill_sql.py b/deploy/sql/migrations/generate_backfill_sql.py deleted file mode 100644 index 239a5c93ab..0000000000 --- a/deploy/sql/migrations/generate_backfill_sql.py +++ /dev/null @@ -1,272 +0,0 @@ -#!/usr/bin/env python3 -"""Generate idempotent backfill SQL from capability_profiles.CATALOG. - -Usage: - python deploy/sql/migrations/generate_backfill_sql.py > deploy/sql/migrations/v2.2.x_MMDD_backfill_from_catalog.sql - -Run whenever capability_profiles.py changes, then commit the generated SQL. -""" -import sys -import os -import types -from datetime import date -from collections import namedtuple - -_project_root = os.path.join(os.path.dirname(__file__), "..", "..", "..") -sys.path.insert(0, os.path.join(_project_root, "backend")) - -# Stub SDK types to avoid pulling in the full nexent SDK dependency chain -_nexent_stub = types.ModuleType("nexent") -_nexent_core = types.ModuleType("nexent.core") -_nexent_models = types.ModuleType("nexent.core.models") -_nexent_resolver = types.ModuleType("nexent.core.models.capacity_resolver") - -ProfileKey = tuple - -class CapabilityProfile: - """Minimal stub that accepts any keyword arguments.""" - def __init__(self, **kwargs): - for k, v in kwargs.items(): - setattr(self, k, v) - -_nexent_resolver.ProfileKey = ProfileKey -_nexent_resolver.CapabilityProfile = CapabilityProfile -sys.modules["nexent"] = _nexent_stub -sys.modules["nexent.core"] = _nexent_core -sys.modules["nexent.core.models"] = _nexent_models -sys.modules["nexent.core.models.capacity_resolver"] = _nexent_resolver - -from consts.capability_profiles import CATALOG, CATALOG_REVISION - -DEFAULT_CONTEXT_WINDOW = 32_768 -DEFAULT_MAX_OUTPUT = 4_096 -DEFAULT_RESERVE = 4_096 - - -def _sql_int(value: int) -> str: - return str(value) - - -def _sql_str(value: str) -> str: - return value.replace("'", "''") - - -def _split_repo_name(full_id: str) -> tuple[str, str]: - """Split a catalog's full model identifier into (model_repo, model_name). - - Must match backend/utils/model_name_utils.split_repo_name which splits - on the LAST '/' (rsplit). For 'Pro/deepseek-ai/DeepSeek-V3.2' this - yields repo='Pro/deepseek-ai', name='DeepSeek-V3.2'. - """ - if "/" in full_id: - repo, name = full_id.rsplit("/", 1) - return repo, name - return "", full_id - - -def _sql_repo_match(repo: str) -> str: - """Build the WHERE fragment that matches the table's model_repo column. - - Bare-name catalog entries (no '/') can land in the table as either - model_repo='' or model_repo IS NULL depending on the create path, so - accept both. Namespaced entries match the exact string. - """ - if repo == "": - return "(model_repo IS NULL OR model_repo = '')" - return f"model_repo = '{_sql_str(repo)}'" - - -def main() -> None: - today = date.today().strftime("%Y-%m-%d") - lines: list[str] = [] - - lines.append(f"-- Generated by deploy/sql/migrations/generate_backfill_sql.py on {today}") - lines.append(f"-- Catalog revision: {CATALOG_REVISION}") - lines.append(f"-- Catalog entries: {len(CATALOG)}") - lines.append("--") - lines.append("-- Migration kind: RECOMMENDED_DATA_FIX") - lines.append("-- Idempotent: COALESCE + IS NULL guards protect existing values.") - lines.append("-- Safe: enforces max_output < context_window via GREATEST/LEAST.") - lines.append("--") - lines.append("-- Phases:") - lines.append("-- 1a Bare LLM/VLM rows that match a catalog entry by") - lines.append("-- (model_factory, model_repo, model_name) -> fill capacity") - lines.append("-- fields + tag capacity_source='profile' + profile_version.") - lines.append("-- 1b Already-filled rows that match a catalog entry AND whose") - lines.append("-- context_window_tokens and max_output_tokens exactly equal") - lines.append("-- the catalog values -> tag profile_version only. capacity_") - lines.append("-- source stays whatever it was (typically 'operator'); we") - lines.append("-- don't rewrite provenance, we just add the dispatch tag so") - lines.append("-- dispatch_profile_hit_total can fire.") - lines.append("-- 2 Remaining bare LLM/VLM rows -> safe defaults.") - lines.append("-- 3 Clamp default_output_reserve_tokens to <= max_output_tokens.") - lines.append("--") - lines.append("-- Pre-run self-check (rows whose capability_profile_version is NULL):") - lines.append("--") - lines.append("-- SELECT model_id, model_repo, model_name, model_factory,") - lines.append("-- context_window_tokens, max_output_tokens, capability_profile_version") - lines.append("-- FROM nexent.model_record_t") - lines.append("-- WHERE delete_flag = 'N'") - lines.append("-- AND COALESCE(model_type, 'llm') IN ('llm', 'vlm')") - lines.append("-- AND capability_profile_version IS NULL;") - lines.append("") - - # Group catalog by provider so the generated SQL has tidy section headers - from collections import defaultdict - by_provider: dict[str, list] = defaultdict(list) - for (provider, full_id), profile in CATALOG.items(): - by_provider[provider].append((full_id, profile)) - - # -------------------------------------------------------------- - # Phase 1a: catalog match + bare -> fill capacity + tag - # -------------------------------------------------------------- - lines.append("-- ============================================================") - lines.append("-- Phase 1a: Backfill bare rows that match approved catalog entries") - lines.append("-- ============================================================") - lines.append("") - lines.append("DO $$") - lines.append("DECLARE") - lines.append(" v_updated INTEGER := 0;") - lines.append(" v_total INTEGER := 0;") - lines.append(" c_active_flag CONSTANT TEXT := 'N';") - lines.append(" c_source_profile CONSTANT TEXT := 'profile';") - lines.append("BEGIN") - - for provider in sorted(by_provider.keys()): - entries = by_provider[provider] - lines.append(f" -- {provider} ({len(entries)} entries)") - for full_id, profile in entries: - ctx = profile.context_window_tokens - mout = profile.max_output_tokens - reserve = profile.default_output_reserve_tokens - version = _sql_str(profile.capability_profile_version) - repo, name = _split_repo_name(full_id) - repo_match = _sql_repo_match(repo) - escaped_name = _sql_str(name) - - lines.append(f" UPDATE nexent.model_record_t") - lines.append(f" SET context_window_tokens = COALESCE(context_window_tokens,") - lines.append(f" GREATEST({_sql_int(ctx)}, COALESCE(max_output_tokens, 0) + 1)),") - lines.append(f" max_output_tokens = COALESCE(max_output_tokens,") - lines.append(f" LEAST({_sql_int(mout)}, COALESCE(context_window_tokens, {_sql_int(ctx)}) - 1)),") - lines.append(f" default_output_reserve_tokens = COALESCE(default_output_reserve_tokens,") - lines.append(f" LEAST({_sql_int(reserve)}, COALESCE(max_output_tokens, {_sql_int(mout)}))),") - lines.append(f" capacity_source = COALESCE(capacity_source, c_source_profile),") - lines.append(f" capability_profile_version = COALESCE(capability_profile_version, '{version}')") - lines.append(f" WHERE LOWER(model_factory) = '{_sql_str(provider.lower())}'") - lines.append(f" AND {repo_match}") - lines.append(f" AND model_name = '{escaped_name}'") - lines.append(f" AND delete_flag = c_active_flag") - lines.append(f" AND (context_window_tokens IS NULL OR max_output_tokens IS NULL);") - lines.append(f" GET DIAGNOSTICS v_updated = ROW_COUNT;") - lines.append(f" v_total := v_total + v_updated;") - lines.append("") - - lines.append(" RAISE NOTICE 'Phase 1a catalog backfill (bare): % row(s) updated', v_total;") - lines.append("END $$;") - lines.append("") - - # -------------------------------------------------------------- - # Phase 1b: catalog match + already-filled values match catalog - # -> tag profile_version + upgrade capacity_source from 'default' to 'profile' - # -------------------------------------------------------------- - lines.append("-- ============================================================") - lines.append("-- Phase 1b: Tag already-filled rows whose ctx/max_out exactly match") - lines.append("-- the catalog with capability_profile_version. Upgrades") - lines.append("-- capacity_source from 'default' to 'profile' (values now") - lines.append("-- come from catalog, not system defaults). Preserves") - lines.append("-- 'operator' and other explicit sources.") - lines.append("-- ============================================================") - lines.append("") - lines.append("DO $$") - lines.append("DECLARE") - lines.append(" v_updated INTEGER := 0;") - lines.append(" v_total INTEGER := 0;") - lines.append(" c_active_flag CONSTANT TEXT := 'N';") - lines.append(" c_source_default CONSTANT TEXT := 'default';") - lines.append(" c_source_profile CONSTANT TEXT := 'profile';") - lines.append("BEGIN") - - for provider in sorted(by_provider.keys()): - entries = by_provider[provider] - lines.append(f" -- {provider} ({len(entries)} entries)") - for full_id, profile in entries: - ctx = profile.context_window_tokens - mout = profile.max_output_tokens - version = _sql_str(profile.capability_profile_version) - repo, name = _split_repo_name(full_id) - repo_match = _sql_repo_match(repo) - escaped_name = _sql_str(name) - - lines.append(f" UPDATE nexent.model_record_t") - lines.append(f" SET capability_profile_version = '{version}',") - lines.append(f" capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END") - lines.append(f" WHERE LOWER(model_factory) = '{_sql_str(provider.lower())}'") - lines.append(f" AND {repo_match}") - lines.append(f" AND model_name = '{escaped_name}'") - lines.append(f" AND delete_flag = c_active_flag") - lines.append(f" AND context_window_tokens = {_sql_int(ctx)}") - lines.append(f" AND max_output_tokens = {_sql_int(mout)}") - lines.append(f" AND (capability_profile_version IS NULL OR (capability_profile_version = '{version}' AND capacity_source = c_source_default));") - lines.append(f" GET DIAGNOSTICS v_updated = ROW_COUNT;") - lines.append(f" v_total := v_total + v_updated;") - lines.append("") - - lines.append(" RAISE NOTICE 'Phase 1b catalog tag (matching filled): % row(s) updated', v_total;") - lines.append("END $$;") - lines.append("") - - # Phase 2: safe defaults for remaining bare rows - lines.append("-- ============================================================") - lines.append("-- Phase 2: Safe defaults for remaining bare LLM/VLM rows") - lines.append("-- ============================================================") - lines.append("") - lines.append("DO $$") - lines.append("DECLARE") - lines.append(" v_updated INTEGER := 0;") - lines.append(" c_active_flag CONSTANT TEXT := 'N';") - lines.append(" c_source_default CONSTANT TEXT := 'default';") - lines.append("BEGIN") - lines.append(" UPDATE nexent.model_record_t") - lines.append(f" SET context_window_tokens = COALESCE(context_window_tokens,") - lines.append(f" GREATEST({_sql_int(DEFAULT_CONTEXT_WINDOW)}, COALESCE(max_output_tokens, 0) + 1)),") - lines.append(f" max_output_tokens = COALESCE(max_output_tokens,") - lines.append(f" LEAST({_sql_int(DEFAULT_MAX_OUTPUT)}, COALESCE(context_window_tokens, {_sql_int(DEFAULT_CONTEXT_WINDOW)}) - 1)),") - lines.append(f" default_output_reserve_tokens = COALESCE(default_output_reserve_tokens,") - lines.append(f" LEAST({_sql_int(DEFAULT_RESERVE)}, COALESCE(max_output_tokens, {_sql_int(DEFAULT_MAX_OUTPUT)}))),") - lines.append(f" capacity_source = COALESCE(capacity_source, c_source_default)") - lines.append(" WHERE delete_flag = c_active_flag") - lines.append(" AND COALESCE(model_type, 'llm') IN ('llm', 'vlm')") - lines.append(" AND (context_window_tokens IS NULL OR max_output_tokens IS NULL);") - lines.append("") - lines.append(" GET DIAGNOSTICS v_updated = ROW_COUNT;") - lines.append(" RAISE NOTICE 'Safe defaults: % LLM/VLM row(s) backfilled', v_updated;") - lines.append("END $$;") - lines.append("") - - # Phase 3: clamp reserve to max_output - lines.append("-- ============================================================") - lines.append("-- Phase 3: Clamp default_output_reserve_tokens to max_output_tokens") - lines.append("-- ============================================================") - lines.append("") - lines.append("DO $$") - lines.append("DECLARE") - lines.append(" v_updated INTEGER := 0;") - lines.append(" c_active_flag CONSTANT TEXT := 'N';") - lines.append("BEGIN") - lines.append(" UPDATE nexent.model_record_t") - lines.append(" SET default_output_reserve_tokens = max_output_tokens") - lines.append(" WHERE delete_flag = c_active_flag") - lines.append(" AND default_output_reserve_tokens IS NOT NULL") - lines.append(" AND max_output_tokens IS NOT NULL") - lines.append(" AND default_output_reserve_tokens > max_output_tokens;") - lines.append("") - lines.append(" GET DIAGNOSTICS v_updated = ROW_COUNT;") - lines.append(" RAISE NOTICE 'reserve clamp: % row(s) updated', v_updated;") - lines.append("END $$;") - - print("\n".join(lines)) - - -if __name__ == "__main__": - main() diff --git a/deploy/sql/migrations/v2.2.0_0615_context_management_capacity_schema.sql b/deploy/sql/migrations/v2.2.0_0615_context_management_capacity_schema.sql deleted file mode 100644 index cc4194d962..0000000000 --- a/deploy/sql/migrations/v2.2.0_0615_context_management_capacity_schema.sql +++ /dev/null @@ -1,144 +0,0 @@ --- Migration kind: REQUIRED_SCHEMA --- Required for: all upgraded deployments before running W1/W2 context-management code. --- Reason: new code reads/writes these model capacity, monitoring snapshot, and agent override columns. - --- ============================================================ --- W1: Add explicit model token-capacity fields to model_record_t --- ============================================================ --- All columns are nullable and additive; legacy max_tokens stays as a deprecated --- output-cap alias until consumers migrate. - -ALTER TABLE nexent.model_record_t -ADD COLUMN IF NOT EXISTS context_window_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_record_t -ADD COLUMN IF NOT EXISTS max_input_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_record_t -ADD COLUMN IF NOT EXISTS max_output_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_record_t -ADD COLUMN IF NOT EXISTS default_output_reserve_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_record_t -ADD COLUMN IF NOT EXISTS tokenizer_family VARCHAR(100) DEFAULT NULL; - -ALTER TABLE nexent.model_record_t -ADD COLUMN IF NOT EXISTS capacity_source VARCHAR(100) DEFAULT NULL; - -ALTER TABLE nexent.model_record_t -ADD COLUMN IF NOT EXISTS capability_profile_version VARCHAR(100) DEFAULT NULL; - -COMMENT ON COLUMN nexent.model_record_t.context_window_tokens IS 'Total combined input/output context window in tokens, when the provider uses a combined window. Nullable.'; -COMMENT ON COLUMN nexent.model_record_t.max_input_tokens IS 'Provider hard input-token limit when distinct from the combined window. Nullable.'; -COMMENT ON COLUMN nexent.model_record_t.max_output_tokens IS 'Provider-supported or operator-configured completion-output cap. Replaces the ambiguous LLM meaning of max_tokens. Nullable.'; -COMMENT ON COLUMN nexent.model_record_t.default_output_reserve_tokens IS 'Default output allowance reserved per request before constructing input context. Nullable.'; -COMMENT ON COLUMN nexent.model_record_t.tokenizer_family IS 'Token-counting strategy or provider/model tokenizer identifier mapped via tokenizer_registry. Nullable.'; -COMMENT ON COLUMN nexent.model_record_t.capacity_source IS 'Source of the persisted capacity value. Optional values: operator, profile, provider_candidate, legacy, unknown.'; -COMMENT ON COLUMN nexent.model_record_t.capability_profile_version IS 'Version of the approved provider/model capability profile used by the request, e.g. openai/gpt-4o@1.'; - --- ============================================================ --- W1: Persist resolved model capacity snapshot fields on monitoring records --- ============================================================ - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS context_window_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS default_output_reserve_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS capability_profile_version VARCHAR(100) DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS capacity_source VARCHAR(100) DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS requested_output_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS provider_input_limit_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS tokenizer_family VARCHAR(100) DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS counting_mode VARCHAR(20) DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS unknown_capabilities JSONB DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS capacity_fingerprint VARCHAR(64) DEFAULT NULL; - -COMMENT ON COLUMN nexent.model_monitoring_record_t.context_window_tokens IS 'Resolved total combined model context window for this request'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.default_output_reserve_tokens IS 'Default output allowance reserved before input context construction'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.capability_profile_version IS 'Version of the resolved capacity profile for this request'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.capacity_source IS 'Dominant source of resolved capacity fields for this request'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.requested_output_tokens IS 'Output tokens requested or reserved during capacity resolution'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.provider_input_limit_tokens IS 'Resolved provider input-token limit used by context management'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.tokenizer_family IS 'Tokenizer family used for request token counting'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.counting_mode IS 'Token counting mode for the request: exact or estimated'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.unknown_capabilities IS 'Structured list of capacity capabilities unknown at resolution time'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.capacity_fingerprint IS 'Fingerprint of the resolved model capacity snapshot'; - --- ============================================================ --- W2: Add per-agent requested_output_tokens override --- ============================================================ - -ALTER TABLE nexent.ag_tenant_agent_t - ADD COLUMN IF NOT EXISTS requested_output_tokens INTEGER NULL; - -COMMENT ON COLUMN nexent.ag_tenant_agent_t.requested_output_tokens IS - 'Per-agent override for W2 requested_output_tokens. NULL means inherit ' - 'the resolved model-level default. Must satisfy 0 < value <= ' - 'max_output_tokens from the resolved W1 capacity at save time.'; - --- ============================================================ --- W2: Add safe input budget snapshot fields to model monitoring records --- ============================================================ - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_fingerprint VARCHAR(64) DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_w1_fingerprint VARCHAR(64) DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_requested_output_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_output_reserve_source VARCHAR(32) DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_provider_input_limit_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_uncertainty_reserve_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_uncertainty_reserve_basis VARCHAR(64) DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_soft_limit_ratio FLOAT DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_soft_input_budget_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_hard_input_budget_tokens INTEGER DEFAULT NULL; - -ALTER TABLE nexent.model_monitoring_record_t -ADD COLUMN IF NOT EXISTS budget_warnings JSONB DEFAULT NULL; - -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_fingerprint IS 'Fingerprint of the resolved W2 safe input budget snapshot'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_w1_fingerprint IS 'W1 capacity fingerprint consumed by the W2 budget snapshot'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_requested_output_tokens IS 'W2 trusted requested output tokens used at dispatch'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_output_reserve_source IS 'Source of the W2 requested output token reserve'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_provider_input_limit_tokens IS 'Provider input limit after applying the W2 output reserve'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_uncertainty_reserve_tokens IS 'Additional W2 uncertainty reserve deducted from input budget'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_uncertainty_reserve_basis IS 'Basis used for the W2 uncertainty reserve'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_soft_limit_ratio IS 'W2 soft input budget ratio'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_soft_input_budget_tokens IS 'W2 soft input budget where proactive compression begins'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_hard_input_budget_tokens IS 'W2 hard input budget consumed by W3 final fit'; -COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_warnings IS 'Structured W2 budget warnings active for this request'; diff --git a/deploy/sql/migrations/v2.2.1_0618_add_conversation_share_tables.sql b/deploy/sql/migrations/v2.2.1_0618_add_conversation_share_tables.sql deleted file mode 100644 index 9769008ce8..0000000000 --- a/deploy/sql/migrations/v2.2.1_0618_add_conversation_share_tables.sql +++ /dev/null @@ -1,62 +0,0 @@ -CREATE TABLE IF NOT EXISTS nexent.conversation_share_t ( - share_id integer NOT NULL PRIMARY KEY, - share_token varchar(64) NOT NULL UNIQUE, - conversation_id integer NOT NULL, - tenant_id varchar(100), - title varchar(200), - mode varchar(30) DEFAULT 'selected', - selected_message_ids jsonb, - snapshot_json jsonb NOT NULL, - status varchar(30) DEFAULT 'active', - expire_time timestamp without time zone, - create_time timestamp without time zone DEFAULT now(), - update_time timestamp without time zone DEFAULT now(), - created_by varchar(100), - updated_by varchar(100), - delete_flag varchar(1) DEFAULT 'N' -); - -CREATE SEQUENCE IF NOT EXISTS nexent.conversation_share_t_share_id_seq - AS integer - START WITH 1 - INCREMENT BY 1 - NO MINVALUE - NO MAXVALUE - CACHE 1; - -ALTER SEQUENCE nexent.conversation_share_t_share_id_seq OWNED BY nexent.conversation_share_t.share_id; -ALTER TABLE ONLY nexent.conversation_share_t ALTER COLUMN share_id SET DEFAULT nextval('nexent.conversation_share_t_share_id_seq'::regclass); - -CREATE INDEX IF NOT EXISTS idx_conversation_share_token ON nexent.conversation_share_t (share_token); -CREATE INDEX IF NOT EXISTS idx_conversation_share_conversation_id ON nexent.conversation_share_t (conversation_id); - -CREATE TABLE IF NOT EXISTS nexent.conversation_share_asset_t ( - share_asset_id integer NOT NULL PRIMARY KEY, - asset_id varchar(64) NOT NULL UNIQUE, - share_token varchar(64) NOT NULL, - object_name varchar(1000) NOT NULL, - filename varchar(500), - content_type varchar(200), - size bigint, - source_kind varchar(50), - metadata_json jsonb, - create_time timestamp without time zone DEFAULT now(), - update_time timestamp without time zone DEFAULT now(), - created_by varchar(100), - updated_by varchar(100), - delete_flag varchar(1) DEFAULT 'N' -); - -CREATE SEQUENCE IF NOT EXISTS nexent.conversation_share_asset_t_share_asset_id_seq - AS integer - START WITH 1 - INCREMENT BY 1 - NO MINVALUE - NO MAXVALUE - CACHE 1; - -ALTER SEQUENCE nexent.conversation_share_asset_t_share_asset_id_seq OWNED BY nexent.conversation_share_asset_t.share_asset_id; -ALTER TABLE ONLY nexent.conversation_share_asset_t ALTER COLUMN share_asset_id SET DEFAULT nextval('nexent.conversation_share_asset_t_share_asset_id_seq'::regclass); - -CREATE INDEX IF NOT EXISTS idx_conversation_share_asset_token ON nexent.conversation_share_asset_t (share_token); -CREATE INDEX IF NOT EXISTS idx_conversation_share_asset_id ON nexent.conversation_share_asset_t (asset_id); diff --git a/deploy/sql/migrations/v2.2.2_0622_update_left_nav_menu.sql b/deploy/sql/migrations/v2.2.2_0622_update_left_nav_menu.sql deleted file mode 100644 index 91db40618c..0000000000 --- a/deploy/sql/migrations/v2.2.2_0622_update_left_nav_menu.sql +++ /dev/null @@ -1,105 +0,0 @@ --- ============================================================ --- Menu Structure Migration V2 --- Migration Date: 2026-06-22 --- ============================================================ - --- Step 1: Clear all existing LEFT_NAV_MENU permissions -BEGIN; - -DELETE FROM nexent.role_permission_t -WHERE permission_category = 'VISIBILITY' AND permission_type = 'LEFT_NAV_MENU'; - -ALTER TABLE nexent.role_permission_t -ADD COLUMN IF NOT EXISTS parent_key VARCHAR(50); --- ============================================================ --- New Menu Structure: --- ROOT: /, /chat, /agent-dev, /resource-space, /resource-manage, /owner-manage, /users --- AGENT-DEV: /models, /knowledges, /agents, /memory --- RESOURCE-SPACE: /agent-space, /mcp-space, /skill-space --- ============================================================ --- ID Format: xx --- SU=10xx, ADMIN=11xx, DEV=12xx, USER=13xx, SPEED=14xx, ASSET_OWNER=15xx --- parent_key: NULL for first-level, parent route for second-level --- ============================================================ - --- SU Menus (root level) -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES -(1001, 'SU', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), -(1002, 'SU', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-manage'), -(1003, 'SU', 'VISIBILITY', 'LEFT_NAV_MENU', '/owner-manage'); - --- ADMIN Menus (root level) -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES -(1101, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), -(1102, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/chat'), -(1103, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-dev'), -(1104, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-space'), -(1105, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-manage'), -(1106, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/users'); -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES -(1107, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/models', '/agent-dev'), -(1108, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/knowledges', '/agent-dev'), -(1109, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/agents', '/agent-dev'), -(1110, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/memory', '/agent-dev'); -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES -(1111, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-space', '/resource-space'), -(1112, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/mcp-space', '/resource-space'), -(1113, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/skill-space', '/resource-space'); - --- DEV Menus (NO /resource-manage, root level) -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES -(1201, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), -(1202, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/chat'), -(1203, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-dev'), -(1204, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-space'), -(1205, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/users'); -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES -(1206, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/models', '/agent-dev'), -(1207, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/knowledges', '/agent-dev'), -(1208, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/agents', '/agent-dev'), -(1209, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/memory', '/agent-dev'); -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES -(1210, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-space', '/resource-space'), -(1211, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/mcp-space', '/resource-space'), -(1212, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/skill-space', '/resource-space'); - --- USER Menus (Minimal, all root level) -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES -(1301, 'USER', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), -(1302, 'USER', 'VISIBILITY', 'LEFT_NAV_MENU', '/chat'), -(1303, 'USER', 'VISIBILITY', 'LEFT_NAV_MENU', '/memory'), -(1304, 'USER', 'VISIBILITY', 'LEFT_NAV_MENU', '/users'); - --- SPEED Menus (root level) -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES -(1401, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), -(1402, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/chat'), -(1403, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-dev'), -(1404, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-space'), -(1405, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-manage'); -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES -(1406, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/models', '/agent-dev'), -(1407, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/knowledges', '/agent-dev'), -(1408, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/agents', '/agent-dev'), -(1409, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/memory', '/agent-dev'); -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES -(1410, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-space', '/resource-space'), -(1411, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/mcp-space', '/resource-space'), -(1412, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/skill-space', '/resource-space'); - --- ASSET_OWNER Menus (root level; /owner-manage is SU-only, see v2.3.0_0713_move_owner_manage_to_su.sql) -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES -(1501, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), -(1502, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/chat'), -(1503, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-dev'), -(1504, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-space'); -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES -(1506, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/models', '/agent-dev'), -(1507, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/knowledges', '/agent-dev'), -(1508, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/agents', '/agent-dev'); -INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES -(1509, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-space', '/resource-space'), -(1510, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/mcp-space', '/resource-space'), -(1511, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/skill-space', '/resource-space'); - -COMMIT; diff --git a/deploy/sql/migrations/v2.2.2_0624_migrate_agent_model_id_to_list.sql b/deploy/sql/migrations/v2.2.2_0624_migrate_agent_model_id_to_list.sql deleted file mode 100644 index 2b6ac7c3a7..0000000000 --- a/deploy/sql/migrations/v2.2.2_0624_migrate_agent_model_id_to_list.sql +++ /dev/null @@ -1,86 +0,0 @@ --- Migration: Change ag_tenant_agent_t.model_id to model_ids (list of integers) --- Date: 2026-06-17 --- Description: Migrate agent model configuration from single model_id to model_ids list --- --- Idempotency notes: --- This migration is executed on every container restart together with all other --- incremental migrations. The follow-up migration --- v2.2.2_0626_drop_agent_model_id_and_model_name.sql --- removes ag_tenant_agent_t.model_id (and model_name). Therefore, on a re-run --- the model_id column may already be absent. Every step that references --- model_id must be guarded so the script remains a no-op in that state. --- --- Migration strategy: --- 1. Add new model_ids column as ARRAY(Integer) if it doesn't already exist --- (idempotent via ADD COLUMN IF NOT EXISTS). --- 2. If model_id still exists, backfill model_ids from model_id only when --- model_ids is NULL or an empty array. Existing non-empty values are --- preserved so the migration does not clobber data written by newer code. --- 3. Set column comments (guarded so missing columns do not error). - -SET search_path TO nexent; - -BEGIN; - --- 1) Add model_ids column if it doesn't exist. --- ADD COLUMN IF NOT EXISTS is a no-op when the column already exists, so --- this statement is safe to re-run on every startup. -ALTER TABLE nexent.ag_tenant_agent_t - ADD COLUMN IF NOT EXISTS model_ids INTEGER[] DEFAULT NULL; - --- 2) Backfill model_ids from the legacy single-value model_id column. --- Only runs when model_id still exists. When model_id has already been --- dropped by a later migration (e.g. v2.2.2_0626_drop_agent_model_id_and_model_name.sql), --- this step is skipped and the script remains a safe no-op. --- "Empty" is defined as either NULL or an empty array ('{}'); both --- COALESCE(array_length(model_ids, 1), 0) = 0 and model_ids IS NULL match --- these cases. Rows whose model_ids already has values are left untouched. -DO $$ -BEGIN - IF EXISTS ( - SELECT 1 FROM information_schema.columns - WHERE table_schema = 'nexent' - AND table_name = 'ag_tenant_agent_t' - AND column_name = 'model_id' - ) THEN - UPDATE nexent.ag_tenant_agent_t - SET model_ids = ARRAY[model_id] - WHERE model_id IS NOT NULL - AND (model_ids IS NULL OR COALESCE(array_length(model_ids, 1), 0) = 0); - END IF; -END $$; - --- 3) Update column comments. --- model_ids is created above (or was created on an earlier run) so the --- comment can be applied unconditionally. COMMENT ON COLUMN raises an --- error if the column is missing, so we still guard it for safety. -DO $$ -BEGIN - IF EXISTS ( - SELECT 1 FROM information_schema.columns - WHERE table_schema = 'nexent' - AND table_name = 'ag_tenant_agent_t' - AND column_name = 'model_ids' - ) THEN - COMMENT ON COLUMN nexent.ag_tenant_agent_t.model_ids IS - 'List of model IDs, foreign key references to model_record_t.model_id, max 5 models'; - END IF; -END $$; - --- 4) Add a deprecation comment to model_id, only when the column still exists. --- Once v2.2.2_0626_drop_agent_model_id_and_model_name.sql has dropped it, --- this block is skipped. -DO $$ -BEGIN - IF EXISTS ( - SELECT 1 FROM information_schema.columns - WHERE table_schema = 'nexent' - AND table_name = 'ag_tenant_agent_t' - AND column_name = 'model_id' - ) THEN - COMMENT ON COLUMN nexent.ag_tenant_agent_t.model_id IS - '[DEPRECATED] Single model ID, use model_ids instead'; - END IF; -END $$; - -COMMIT; \ No newline at end of file diff --git a/deploy/sql/migrations/v2.2.2_0627_backfill_from_catalog.sql b/deploy/sql/migrations/v2.2.2_0627_backfill_from_catalog.sql deleted file mode 100644 index ea5d81ff3d..0000000000 --- a/deploy/sql/migrations/v2.2.2_0627_backfill_from_catalog.sql +++ /dev/null @@ -1,2096 +0,0 @@ --- Generated by scripts/generate_backfill_sql.py on 2026-06-27 --- Catalog revision: 2026-06-27.1 --- Catalog entries: 66 --- --- Migration kind: RECOMMENDED_DATA_FIX --- Idempotent: COALESCE + IS NULL guards protect existing values. --- Safe: enforces max_output < context_window via GREATEST/LEAST. --- --- Phases: --- 1a Bare LLM/VLM rows that match a catalog entry by --- (model_factory, model_repo, model_name) -> fill capacity --- fields + tag capacity_source='profile' + profile_version. --- 1b Already-filled rows that match a catalog entry AND whose --- context_window_tokens and max_output_tokens exactly equal --- the catalog values -> tag profile_version only. capacity_ --- source stays whatever it was (typically 'operator'); we --- don't rewrite provenance, we just add the dispatch tag so --- dispatch_profile_hit_total can fire. --- 2 Remaining bare LLM/VLM rows -> safe defaults. --- 3 Clamp default_output_reserve_tokens to <= max_output_tokens. --- --- Pre-run self-check (rows whose capability_profile_version is NULL): --- --- SELECT model_id, model_repo, model_name, model_factory, --- context_window_tokens, max_output_tokens, capability_profile_version --- FROM nexent.model_record_t --- WHERE delete_flag = 'N' --- AND COALESCE(model_type, 'llm') IN ('llm', 'vlm') --- AND capability_profile_version IS NULL; - --- ============================================================ --- Phase 1a: Backfill bare rows that match approved catalog entries --- ============================================================ - -DO $$ -DECLARE - v_updated INTEGER := 0; - v_total INTEGER := 0; - c_active_flag CONSTANT TEXT := 'N'; - c_source_profile CONSTANT TEXT := 'profile'; -BEGIN - -- dashscope (4 entries) - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'dashscope/qwen-plus@1') - WHERE LOWER(model_factory) = 'dashscope' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'qwen-plus' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 1000000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'dashscope/qwen-turbo@1') - WHERE LOWER(model_factory) = 'dashscope' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'qwen-turbo' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(65536, COALESCE(context_window_tokens, 1000000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 65536))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'dashscope/qwen3.7-max@1') - WHERE LOWER(model_factory) = 'dashscope' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'qwen3.7-max' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(200000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(131072, COALESCE(context_window_tokens, 200000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 131072))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'dashscope/glm-5.1@1') - WHERE LOWER(model_factory) = 'dashscope' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'glm-5.1' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - -- deepseek (4 entries) - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(384000, COALESCE(context_window_tokens, 1000000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 384000))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'deepseek/deepseek-chat@2') - WHERE LOWER(model_factory) = 'deepseek' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'deepseek-chat' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(384000, COALESCE(context_window_tokens, 1000000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 384000))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'deepseek/deepseek-reasoner@2') - WHERE LOWER(model_factory) = 'deepseek' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'deepseek-reasoner' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(384000, COALESCE(context_window_tokens, 1000000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 384000))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'deepseek/deepseek-v4-flash@1') - WHERE LOWER(model_factory) = 'deepseek' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'deepseek-v4-flash' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(384000, COALESCE(context_window_tokens, 1000000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 384000))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'deepseek/deepseek-v4-pro@1') - WHERE LOWER(model_factory) = 'deepseek' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'deepseek-v4-pro' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - -- openai (2 entries) - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(128000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 128000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'openai/gpt-4o@1') - WHERE LOWER(model_factory) = 'openai' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'gpt-4o' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(32768, COALESCE(context_window_tokens, 1000000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 32768))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'openai/gpt-4.1@1') - WHERE LOWER(model_factory) = 'openai' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'gpt-4.1' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - -- silicon (56 entries) - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(65536, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 65536))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.6-27b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.6-27B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(131072, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 131072))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/kimi-k2.6@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/moonshotai' - AND model_name = 'Kimi-K2.6' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(1048576, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(384000, COALESCE(context_window_tokens, 1048576) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 384000))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v4-pro-sf@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-V4-Pro' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(1048576, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(384000, COALESCE(context_window_tokens, 1048576) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 384000))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v4-flash-sf@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-V4-Flash' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3.2@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-V3.2' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3.1-terminus@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-V3.1-Terminus' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(163840, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 163840) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-r1@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-R1' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-V3' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-r1-0528-qwen3-8b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-R1-0528-Qwen3-8B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3.2-pro@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/deepseek-ai' - AND model_name = 'DeepSeek-V3.2' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3.1-terminus-pro@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/deepseek-ai' - AND model_name = 'DeepSeek-V3.1-Terminus' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(163840, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 163840) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-r1-pro@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/deepseek-ai' - AND model_name = 'DeepSeek-R1' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3-pro@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/deepseek-ai' - AND model_name = 'DeepSeek-V3' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.6-35b-a3b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.6-35B-A3B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-397b-a17b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-397B-A17B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-122b-a10b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-122B-A10B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-35b-a3b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-35B-A3B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-27b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-27B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-9b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-9B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-4b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-4B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-32b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-32B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(32768, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 32768))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-32b-thinking@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-32B-Thinking' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-8b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-8B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(32768, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 32768))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-8b-thinking@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-8B-Thinking' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-30b-a3b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-30B-A3B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(32768, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 32768))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-30b-a3b-thinking@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-30B-A3B-Thinking' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-omni-30b-a3b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-Omni-30B-A3B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-omni-30b-a3b-thinking@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-Omni-30B-A3B-Thinking' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-omni-30b-a3b-captioner@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-Omni-30B-A3B-Captioner' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(65536, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 65536))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-coder-30b-a3b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-Coder-30B-A3B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-30b-a3b-instruct-2507@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-30B-A3B-Instruct-2507' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-32b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-32B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-14b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-14B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-8b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-8B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen2.5-72b-instruct-128k@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen2.5-72B-Instruct-128K' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen2.5-72b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen2.5-72B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen2.5-32b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen2.5-32B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen2.5-14b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen2.5-14B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen2.5-7b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen2.5-7B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-4-32b-0414@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'THUDM' - AND model_name = 'GLM-4-32B-0414' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-z1-9b-0414@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'THUDM' - AND model_name = 'GLM-Z1-9B-0414' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-4-9b-0414@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'THUDM' - AND model_name = 'GLM-4-9B-0414' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(1048576, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(131072, COALESCE(context_window_tokens, 1048576) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 131072))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-5.2@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'zai-org' - AND model_name = 'GLM-5.2' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-4.5v@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'zai-org' - AND model_name = 'GLM-4.5V' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-4.5-air@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'zai-org' - AND model_name = 'GLM-4.5-Air' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(202752, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(131072, COALESCE(context_window_tokens, 202752) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 131072))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-5.1-pro@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/zai-org' - AND model_name = 'GLM-5.1' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(524288, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 524288) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/seed-oss-36b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'ByteDance-Seed' - AND model_name = 'Seed-OSS-36B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/ling-flash-2.0@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'inclusionAI' - AND model_name = 'Ling-flash-2.0' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/ling-mini-2.0@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'inclusionAI' - AND model_name = 'Ling-mini-2.0' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(204800, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 204800) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/minimax-m2.5@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'MiniMaxAI' - AND model_name = 'MiniMax-M2.5' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(204800, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 204800) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/minimax-m2.5-pro@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/MiniMaxAI' - AND model_name = 'MiniMax-M2.5' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(32768, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(8192, COALESCE(max_output_tokens, 32768))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/kimi-k2.7-code@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'moonshotai' - AND model_name = 'Kimi-K2.7-Code' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/nex-n2-pro@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'nex-agi' - AND model_name = 'Nex-N2-Pro' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 16384))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/step-3.5-flash@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'stepfun-ai' - AND model_name = 'Step-3.5-Flash' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(2048, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(1024, COALESCE(max_output_tokens, 2048))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/hunyuan-mt-7b@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'tencent' - AND model_name = 'Hunyuan-MT-7B' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(8192, COALESCE(context_window_tokens, 131072) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 8192))), - capacity_source = COALESCE(capacity_source, c_source_profile), - capability_profile_version = COALESCE(capability_profile_version, 'silicon/hunyuan-a13b-instruct@1') - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'tencent' - AND model_name = 'Hunyuan-A13B-Instruct' - AND delete_flag = c_active_flag - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - RAISE NOTICE 'Phase 1a catalog backfill (bare): % row(s) updated', v_total; -END $$; - --- ============================================================ --- Phase 1b: Tag already-filled rows whose ctx/max_out exactly match --- the catalog with capability_profile_version. Upgrades --- capacity_source from 'default' to 'profile' (values now --- come from catalog, not system defaults). Preserves --- 'operator' and other explicit sources. --- ============================================================ - -DO $$ -DECLARE - v_updated INTEGER := 0; - v_total INTEGER := 0; - c_active_flag CONSTANT TEXT := 'N'; - c_source_default CONSTANT TEXT := 'default'; - c_source_profile CONSTANT TEXT := 'profile'; -BEGIN - -- dashscope (4 entries) - UPDATE nexent.model_record_t - SET capability_profile_version = 'dashscope/qwen-plus@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'dashscope' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'qwen-plus' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'dashscope/qwen-plus@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'dashscope/qwen-turbo@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'dashscope' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'qwen-turbo' - AND delete_flag = c_active_flag - AND context_window_tokens = 1000000 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'dashscope/qwen-turbo@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'dashscope/qwen3.7-max@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'dashscope' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'qwen3.7-max' - AND delete_flag = c_active_flag - AND context_window_tokens = 1000000 - AND max_output_tokens = 65536 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'dashscope/qwen3.7-max@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'dashscope/glm-5.1@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'dashscope' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'glm-5.1' - AND delete_flag = c_active_flag - AND context_window_tokens = 200000 - AND max_output_tokens = 131072 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'dashscope/glm-5.1@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - -- deepseek (4 entries) - UPDATE nexent.model_record_t - SET capability_profile_version = 'deepseek/deepseek-chat@2', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'deepseek' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'deepseek-chat' - AND delete_flag = c_active_flag - AND context_window_tokens = 1000000 - AND max_output_tokens = 384000 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'deepseek/deepseek-chat@2' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'deepseek/deepseek-reasoner@2', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'deepseek' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'deepseek-reasoner' - AND delete_flag = c_active_flag - AND context_window_tokens = 1000000 - AND max_output_tokens = 384000 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'deepseek/deepseek-reasoner@2' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'deepseek/deepseek-v4-flash@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'deepseek' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'deepseek-v4-flash' - AND delete_flag = c_active_flag - AND context_window_tokens = 1000000 - AND max_output_tokens = 384000 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'deepseek/deepseek-v4-flash@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'deepseek/deepseek-v4-pro@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'deepseek' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'deepseek-v4-pro' - AND delete_flag = c_active_flag - AND context_window_tokens = 1000000 - AND max_output_tokens = 384000 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'deepseek/deepseek-v4-pro@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - -- openai (2 entries) - UPDATE nexent.model_record_t - SET capability_profile_version = 'openai/gpt-4o@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'openai' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'gpt-4o' - AND delete_flag = c_active_flag - AND context_window_tokens = 128000 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'openai/gpt-4o@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'openai/gpt-4.1@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'openai' - AND (model_repo IS NULL OR model_repo = '') - AND model_name = 'gpt-4.1' - AND delete_flag = c_active_flag - AND context_window_tokens = 1000000 - AND max_output_tokens = 32768 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'openai/gpt-4.1@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - -- silicon (56 entries) - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3.6-27b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.6-27B' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 65536 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.6-27b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/kimi-k2.6@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/moonshotai' - AND model_name = 'Kimi-K2.6' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 131072 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/kimi-k2.6@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-v4-pro-sf@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-V4-Pro' - AND delete_flag = c_active_flag - AND context_window_tokens = 1048576 - AND max_output_tokens = 384000 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v4-pro-sf@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-v4-flash-sf@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-V4-Flash' - AND delete_flag = c_active_flag - AND context_window_tokens = 1048576 - AND max_output_tokens = 384000 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v4-flash-sf@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-v3.2@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-V3.2' - AND delete_flag = c_active_flag - AND context_window_tokens = 164000 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3.2@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-v3.1-terminus@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-V3.1-Terminus' - AND delete_flag = c_active_flag - AND context_window_tokens = 164000 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3.1-terminus@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-r1@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-R1' - AND delete_flag = c_active_flag - AND context_window_tokens = 163840 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-r1@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-v3@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-V3' - AND delete_flag = c_active_flag - AND context_window_tokens = 164000 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-r1-0528-qwen3-8b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'deepseek-ai' - AND model_name = 'DeepSeek-R1-0528-Qwen3-8B' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-r1-0528-qwen3-8b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-v3.2-pro@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/deepseek-ai' - AND model_name = 'DeepSeek-V3.2' - AND delete_flag = c_active_flag - AND context_window_tokens = 164000 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3.2-pro@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-v3.1-terminus-pro@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/deepseek-ai' - AND model_name = 'DeepSeek-V3.1-Terminus' - AND delete_flag = c_active_flag - AND context_window_tokens = 164000 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3.1-terminus-pro@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-r1-pro@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/deepseek-ai' - AND model_name = 'DeepSeek-R1' - AND delete_flag = c_active_flag - AND context_window_tokens = 163840 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-r1-pro@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/deepseek-v3-pro@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/deepseek-ai' - AND model_name = 'DeepSeek-V3' - AND delete_flag = c_active_flag - AND context_window_tokens = 164000 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3-pro@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3.6-35b-a3b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.6-35B-A3B' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.6-35b-a3b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3.5-397b-a17b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-397B-A17B' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-397b-a17b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3.5-122b-a10b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-122B-A10B' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-122b-a10b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3.5-35b-a3b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-35B-A3B' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-35b-a3b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3.5-27b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-27B' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-27b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3.5-9b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-9B' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-9b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3.5-4b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3.5-4B' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-4b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-vl-32b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-32B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-32b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-vl-32b-thinking@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-32B-Thinking' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 32768 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-32b-thinking@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-vl-8b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-8B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-8b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-vl-8b-thinking@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-8B-Thinking' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 32768 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-8b-thinking@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-vl-30b-a3b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-30B-A3B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-30b-a3b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-vl-30b-a3b-thinking@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-VL-30B-A3B-Thinking' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 32768 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-30b-a3b-thinking@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-omni-30b-a3b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-Omni-30B-A3B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-omni-30b-a3b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-omni-30b-a3b-thinking@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-Omni-30B-A3B-Thinking' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-omni-30b-a3b-thinking@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-omni-30b-a3b-captioner@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-Omni-30B-A3B-Captioner' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-omni-30b-a3b-captioner@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-coder-30b-a3b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-Coder-30B-A3B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 65536 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-coder-30b-a3b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-30b-a3b-instruct-2507@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-30B-A3B-Instruct-2507' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-30b-a3b-instruct-2507@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-32b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-32B' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-32b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-14b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-14B' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-14b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen3-8b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen3-8B' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-8b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen2.5-72b-instruct-128k@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen2.5-72B-Instruct-128K' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen2.5-72b-instruct-128k@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen2.5-72b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen2.5-72B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen2.5-72b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen2.5-32b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen2.5-32B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen2.5-32b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen2.5-14b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen2.5-14B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen2.5-14b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/qwen2.5-7b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Qwen' - AND model_name = 'Qwen2.5-7B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen2.5-7b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/glm-4-32b-0414@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'THUDM' - AND model_name = 'GLM-4-32B-0414' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-4-32b-0414@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/glm-z1-9b-0414@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'THUDM' - AND model_name = 'GLM-Z1-9B-0414' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-z1-9b-0414@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/glm-4-9b-0414@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'THUDM' - AND model_name = 'GLM-4-9B-0414' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-4-9b-0414@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/glm-5.2@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'zai-org' - AND model_name = 'GLM-5.2' - AND delete_flag = c_active_flag - AND context_window_tokens = 1048576 - AND max_output_tokens = 131072 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-5.2@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/glm-4.5v@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'zai-org' - AND model_name = 'GLM-4.5V' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-4.5v@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/glm-4.5-air@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'zai-org' - AND model_name = 'GLM-4.5-Air' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-4.5-air@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/glm-5.1-pro@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/zai-org' - AND model_name = 'GLM-5.1' - AND delete_flag = c_active_flag - AND context_window_tokens = 202752 - AND max_output_tokens = 131072 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-5.1-pro@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/seed-oss-36b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'ByteDance-Seed' - AND model_name = 'Seed-OSS-36B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 524288 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/seed-oss-36b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/ling-flash-2.0@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'inclusionAI' - AND model_name = 'Ling-flash-2.0' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/ling-flash-2.0@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/ling-mini-2.0@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'inclusionAI' - AND model_name = 'Ling-mini-2.0' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/ling-mini-2.0@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/minimax-m2.5@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'MiniMaxAI' - AND model_name = 'MiniMax-M2.5' - AND delete_flag = c_active_flag - AND context_window_tokens = 204800 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/minimax-m2.5@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/minimax-m2.5-pro@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'Pro/MiniMaxAI' - AND model_name = 'MiniMax-M2.5' - AND delete_flag = c_active_flag - AND context_window_tokens = 204800 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/minimax-m2.5-pro@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/kimi-k2.7-code@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'moonshotai' - AND model_name = 'Kimi-K2.7-Code' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 32768 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/kimi-k2.7-code@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/nex-n2-pro@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'nex-agi' - AND model_name = 'Nex-N2-Pro' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/nex-n2-pro@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/step-3.5-flash@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'stepfun-ai' - AND model_name = 'Step-3.5-Flash' - AND delete_flag = c_active_flag - AND context_window_tokens = 262144 - AND max_output_tokens = 16384 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/step-3.5-flash@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/hunyuan-mt-7b@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'tencent' - AND model_name = 'Hunyuan-MT-7B' - AND delete_flag = c_active_flag - AND context_window_tokens = 32768 - AND max_output_tokens = 2048 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/hunyuan-mt-7b@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - UPDATE nexent.model_record_t - SET capability_profile_version = 'silicon/hunyuan-a13b-instruct@1', - capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END - WHERE LOWER(model_factory) = 'silicon' - AND model_repo = 'tencent' - AND model_name = 'Hunyuan-A13B-Instruct' - AND delete_flag = c_active_flag - AND context_window_tokens = 131072 - AND max_output_tokens = 8192 - AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/hunyuan-a13b-instruct@1' AND capacity_source = c_source_default)); - GET DIAGNOSTICS v_updated = ROW_COUNT; - v_total := v_total + v_updated; - - RAISE NOTICE 'Phase 1b catalog tag (matching filled): % row(s) updated', v_total; -END $$; - --- ============================================================ --- Phase 2: Safe defaults for remaining bare LLM/VLM rows --- ============================================================ - -DO $$ -DECLARE - v_updated INTEGER := 0; - c_active_flag CONSTANT TEXT := 'N'; - c_source_default CONSTANT TEXT := 'default'; -BEGIN - UPDATE nexent.model_record_t - SET context_window_tokens = COALESCE(context_window_tokens, - GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), - max_output_tokens = COALESCE(max_output_tokens, - LEAST(4096, COALESCE(context_window_tokens, 32768) - 1)), - default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, - LEAST(4096, COALESCE(max_output_tokens, 4096))), - capacity_source = COALESCE(capacity_source, c_source_default) - WHERE delete_flag = c_active_flag - AND COALESCE(model_type, 'llm') IN ('llm', 'vlm') - AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); - - GET DIAGNOSTICS v_updated = ROW_COUNT; - RAISE NOTICE 'Safe defaults: % LLM/VLM row(s) backfilled', v_updated; -END $$; - --- ============================================================ --- Phase 3: Clamp default_output_reserve_tokens to max_output_tokens --- ============================================================ - -DO $$ -DECLARE - v_updated INTEGER := 0; - c_active_flag CONSTANT TEXT := 'N'; -BEGIN - UPDATE nexent.model_record_t - SET default_output_reserve_tokens = max_output_tokens - WHERE delete_flag = c_active_flag - AND default_output_reserve_tokens IS NOT NULL - AND max_output_tokens IS NOT NULL - AND default_output_reserve_tokens > max_output_tokens; - - GET DIAGNOSTICS v_updated = ROW_COUNT; - RAISE NOTICE 'reserve clamp: % row(s) updated', v_updated; -END $$; diff --git a/deploy/sql/migrations/v2.2.2_0629_conversation_message_unit_status_and_clean.sql b/deploy/sql/migrations/v2.2.2_0629_conversation_message_unit_status_and_clean.sql deleted file mode 100644 index 9c8e14b9fe..0000000000 --- a/deploy/sql/migrations/v2.2.2_0629_conversation_message_unit_status_and_clean.sql +++ /dev/null @@ -1,62 +0,0 @@ --- Migration: Add status / unit_status fields to support streaming persistence --- Date: 2026-06-29 --- Description: Allow per-message and per-unit lifecycle tracking so the --- frontend can recover partial agent runs when the SSE connection is lost - -SET search_path TO nexent; - -BEGIN; - --- Message-level lifecycle. Assistant messages start as 'pending' / 'streaming' --- and transition to one of completed / failed / stopped. User messages default --- to 'completed' (existing rows are backfilled below). -ALTER TABLE nexent.conversation_message_t - ADD COLUMN IF NOT EXISTS status VARCHAR(30); - -COMMENT ON COLUMN nexent.conversation_message_t.status IS - 'Lifecycle status: pending / streaming / completed / failed / stopped.'; - --- Unit-level lifecycle. Once a unit is fully persisted we mark it 'completed'; --- while the boundary is still being detected it remains 'streaming'. -ALTER TABLE nexent.conversation_message_unit_t - ADD COLUMN IF NOT EXISTS unit_status VARCHAR(30); - -COMMENT ON COLUMN nexent.conversation_message_unit_t.unit_status IS - 'Lifecycle status: streaming (still aggregating) or completed (fully persisted).'; - --- Index for incremental recovery queries (since_message_unit_id filters). -CREATE INDEX IF NOT EXISTS idx_message_unit_message_id_unit_id - ON nexent.conversation_message_unit_t (message_id, unit_id); - --- Cleanup stale deep_thinking units. -DO $$ -BEGIN - IF EXISTS ( - SELECT 1 FROM information_schema.columns - WHERE table_schema = 'nexent' - AND table_name = 'conversation_message_unit_t' - AND column_name = 'unit_status' - ) THEN - DELETE FROM nexent.conversation_message_unit_t - WHERE unit_type = 'model_output_deep_thinking' - AND unit_status IS NULL; - END IF; -END $$; - --- Cleanup corrupted records of thinking units -DO $$ -BEGIN - IF EXISTS ( - SELECT 1 FROM information_schema.columns - WHERE table_schema = 'nexent' - AND table_name = 'conversation_message_unit_t' - AND column_name = 'unit_status' - ) THEN - DELETE FROM nexent.conversation_message_unit_t - WHERE unit_type = 'model_output_thinking' - AND unit_content = '' - AND unit_status IS NULL; - END IF; -END $$; - -COMMIT; diff --git a/deploy/sql/migrations/v2.2_merged_migrations.sql b/deploy/sql/migrations/v2.2_merged_migrations.sql index c1bd4b5bea..1c84e2e907 100644 --- a/deploy/sql/migrations/v2.2_merged_migrations.sql +++ b/deploy/sql/migrations/v2.2_merged_migrations.sql @@ -471,3 +471,2575 @@ COMMENT ON COLUMN nexent.ag_agent_relation_t.selected_agent_version_no IS 'Pinned version of selected_agent_id. NULL = use child current published version at runtime (legacy/draft).'; COMMIT; + +-- Source migration: v2.2.0_0615_context_management_capacity_schema.sql + +-- Migration kind: REQUIRED_SCHEMA +-- Required for: all upgraded deployments before running W1/W2 context-management code. +-- Reason: new code reads/writes these model capacity, monitoring snapshot, and agent override columns. + +-- ============================================================ +-- W1: Add explicit model token-capacity fields to model_record_t +-- ============================================================ +-- All columns are nullable and additive; legacy max_tokens stays as a deprecated +-- output-cap alias until consumers migrate. + +ALTER TABLE nexent.model_record_t +ADD COLUMN IF NOT EXISTS context_window_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_record_t +ADD COLUMN IF NOT EXISTS max_input_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_record_t +ADD COLUMN IF NOT EXISTS max_output_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_record_t +ADD COLUMN IF NOT EXISTS default_output_reserve_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_record_t +ADD COLUMN IF NOT EXISTS tokenizer_family VARCHAR(100) DEFAULT NULL; + +ALTER TABLE nexent.model_record_t +ADD COLUMN IF NOT EXISTS capacity_source VARCHAR(100) DEFAULT NULL; + +ALTER TABLE nexent.model_record_t +ADD COLUMN IF NOT EXISTS capability_profile_version VARCHAR(100) DEFAULT NULL; + +COMMENT ON COLUMN nexent.model_record_t.context_window_tokens IS 'Total combined input/output context window in tokens, when the provider uses a combined window. Nullable.'; +COMMENT ON COLUMN nexent.model_record_t.max_input_tokens IS 'Provider hard input-token limit when distinct from the combined window. Nullable.'; +COMMENT ON COLUMN nexent.model_record_t.max_output_tokens IS 'Provider-supported or operator-configured completion-output cap. Replaces the ambiguous LLM meaning of max_tokens. Nullable.'; +COMMENT ON COLUMN nexent.model_record_t.default_output_reserve_tokens IS 'Default output allowance reserved per request before constructing input context. Nullable.'; +COMMENT ON COLUMN nexent.model_record_t.tokenizer_family IS 'Token-counting strategy or provider/model tokenizer identifier mapped via tokenizer_registry. Nullable.'; +COMMENT ON COLUMN nexent.model_record_t.capacity_source IS 'Source of the persisted capacity value. Optional values: operator, profile, provider_candidate, legacy, unknown.'; +COMMENT ON COLUMN nexent.model_record_t.capability_profile_version IS 'Version of the approved provider/model capability profile used by the request, e.g. openai/gpt-4o@1.'; + +-- ============================================================ +-- W1: Persist resolved model capacity snapshot fields on monitoring records +-- ============================================================ + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS context_window_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS default_output_reserve_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS capability_profile_version VARCHAR(100) DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS capacity_source VARCHAR(100) DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS requested_output_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS provider_input_limit_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS tokenizer_family VARCHAR(100) DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS counting_mode VARCHAR(20) DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS unknown_capabilities JSONB DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS capacity_fingerprint VARCHAR(64) DEFAULT NULL; + +COMMENT ON COLUMN nexent.model_monitoring_record_t.context_window_tokens IS 'Resolved total combined model context window for this request'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.default_output_reserve_tokens IS 'Default output allowance reserved before input context construction'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.capability_profile_version IS 'Version of the resolved capacity profile for this request'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.capacity_source IS 'Dominant source of resolved capacity fields for this request'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.requested_output_tokens IS 'Output tokens requested or reserved during capacity resolution'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.provider_input_limit_tokens IS 'Resolved provider input-token limit used by context management'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.tokenizer_family IS 'Tokenizer family used for request token counting'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.counting_mode IS 'Token counting mode for the request: exact or estimated'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.unknown_capabilities IS 'Structured list of capacity capabilities unknown at resolution time'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.capacity_fingerprint IS 'Fingerprint of the resolved model capacity snapshot'; + +-- ============================================================ +-- W2: Add per-agent requested_output_tokens override +-- ============================================================ + +ALTER TABLE nexent.ag_tenant_agent_t + ADD COLUMN IF NOT EXISTS requested_output_tokens INTEGER NULL; + +COMMENT ON COLUMN nexent.ag_tenant_agent_t.requested_output_tokens IS + 'Per-agent override for W2 requested_output_tokens. NULL means inherit ' + 'the resolved model-level default. Must satisfy 0 < value <= ' + 'max_output_tokens from the resolved W1 capacity at save time.'; + +-- ============================================================ +-- W2: Add safe input budget snapshot fields to model monitoring records +-- ============================================================ + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_fingerprint VARCHAR(64) DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_w1_fingerprint VARCHAR(64) DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_requested_output_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_output_reserve_source VARCHAR(32) DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_provider_input_limit_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_uncertainty_reserve_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_uncertainty_reserve_basis VARCHAR(64) DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_soft_limit_ratio FLOAT DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_soft_input_budget_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_hard_input_budget_tokens INTEGER DEFAULT NULL; + +ALTER TABLE nexent.model_monitoring_record_t +ADD COLUMN IF NOT EXISTS budget_warnings JSONB DEFAULT NULL; + +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_fingerprint IS 'Fingerprint of the resolved W2 safe input budget snapshot'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_w1_fingerprint IS 'W1 capacity fingerprint consumed by the W2 budget snapshot'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_requested_output_tokens IS 'W2 trusted requested output tokens used at dispatch'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_output_reserve_source IS 'Source of the W2 requested output token reserve'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_provider_input_limit_tokens IS 'Provider input limit after applying the W2 output reserve'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_uncertainty_reserve_tokens IS 'Additional W2 uncertainty reserve deducted from input budget'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_uncertainty_reserve_basis IS 'Basis used for the W2 uncertainty reserve'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_soft_limit_ratio IS 'W2 soft input budget ratio'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_soft_input_budget_tokens IS 'W2 soft input budget where proactive compression begins'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_hard_input_budget_tokens IS 'W2 hard input budget consumed by W3 final fit'; +COMMENT ON COLUMN nexent.model_monitoring_record_t.budget_warnings IS 'Structured W2 budget warnings active for this request'; + +-- Source migration: v2.2.1_0618_add_conversation_share_tables.sql + +CREATE TABLE IF NOT EXISTS nexent.conversation_share_t ( + share_id integer NOT NULL PRIMARY KEY, + share_token varchar(64) NOT NULL UNIQUE, + conversation_id integer NOT NULL, + tenant_id varchar(100), + title varchar(200), + mode varchar(30) DEFAULT 'selected', + selected_message_ids jsonb, + snapshot_json jsonb NOT NULL, + status varchar(30) DEFAULT 'active', + expire_time timestamp without time zone, + create_time timestamp without time zone DEFAULT now(), + update_time timestamp without time zone DEFAULT now(), + created_by varchar(100), + updated_by varchar(100), + delete_flag varchar(1) DEFAULT 'N' +); + +CREATE SEQUENCE IF NOT EXISTS nexent.conversation_share_t_share_id_seq + AS integer + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1; + +ALTER SEQUENCE nexent.conversation_share_t_share_id_seq OWNED BY nexent.conversation_share_t.share_id; +ALTER TABLE ONLY nexent.conversation_share_t ALTER COLUMN share_id SET DEFAULT nextval('nexent.conversation_share_t_share_id_seq'::regclass); + +CREATE INDEX IF NOT EXISTS idx_conversation_share_token ON nexent.conversation_share_t (share_token); +CREATE INDEX IF NOT EXISTS idx_conversation_share_conversation_id ON nexent.conversation_share_t (conversation_id); + +CREATE TABLE IF NOT EXISTS nexent.conversation_share_asset_t ( + share_asset_id integer NOT NULL PRIMARY KEY, + asset_id varchar(64) NOT NULL UNIQUE, + share_token varchar(64) NOT NULL, + object_name varchar(1000) NOT NULL, + filename varchar(500), + content_type varchar(200), + size bigint, + source_kind varchar(50), + metadata_json jsonb, + create_time timestamp without time zone DEFAULT now(), + update_time timestamp without time zone DEFAULT now(), + created_by varchar(100), + updated_by varchar(100), + delete_flag varchar(1) DEFAULT 'N' +); + +CREATE SEQUENCE IF NOT EXISTS nexent.conversation_share_asset_t_share_asset_id_seq + AS integer + START WITH 1 + INCREMENT BY 1 + NO MINVALUE + NO MAXVALUE + CACHE 1; + +ALTER SEQUENCE nexent.conversation_share_asset_t_share_asset_id_seq OWNED BY nexent.conversation_share_asset_t.share_asset_id; +ALTER TABLE ONLY nexent.conversation_share_asset_t ALTER COLUMN share_asset_id SET DEFAULT nextval('nexent.conversation_share_asset_t_share_asset_id_seq'::regclass); + +CREATE INDEX IF NOT EXISTS idx_conversation_share_asset_token ON nexent.conversation_share_asset_t (share_token); +CREATE INDEX IF NOT EXISTS idx_conversation_share_asset_id ON nexent.conversation_share_asset_t (asset_id); + +-- Source migration: v2.2.2_0622_update_left_nav_menu.sql + +-- ============================================================ +-- Menu Structure Migration V2 +-- Migration Date: 2026-06-22 +-- ============================================================ + +-- Step 1: Clear all existing LEFT_NAV_MENU permissions +BEGIN; + +DELETE FROM nexent.role_permission_t +WHERE permission_category = 'VISIBILITY' AND permission_type = 'LEFT_NAV_MENU'; + +ALTER TABLE nexent.role_permission_t +ADD COLUMN IF NOT EXISTS parent_key VARCHAR(50); +-- ============================================================ +-- New Menu Structure: +-- ROOT: /, /chat, /agent-dev, /resource-space, /resource-manage, /owner-manage, /users +-- AGENT-DEV: /models, /knowledges, /agents, /memory +-- RESOURCE-SPACE: /agent-space, /mcp-space, /skill-space +-- ============================================================ +-- ID Format: xx +-- SU=10xx, ADMIN=11xx, DEV=12xx, USER=13xx, SPEED=14xx, ASSET_OWNER=15xx +-- parent_key: NULL for first-level, parent route for second-level +-- ============================================================ + +-- SU Menus (root level) +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES +(1001, 'SU', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), +(1002, 'SU', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-manage'), +(1003, 'SU', 'VISIBILITY', 'LEFT_NAV_MENU', '/owner-manage'); + +-- ADMIN Menus (root level) +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES +(1101, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), +(1102, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/chat'), +(1103, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-dev'), +(1104, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-space'), +(1105, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-manage'), +(1106, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/users'); +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES +(1107, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/models', '/agent-dev'), +(1108, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/knowledges', '/agent-dev'), +(1109, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/agents', '/agent-dev'), +(1110, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/memory', '/agent-dev'); +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES +(1111, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-space', '/resource-space'), +(1112, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/mcp-space', '/resource-space'), +(1113, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/skill-space', '/resource-space'); + +-- DEV Menus (NO /resource-manage, root level) +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES +(1201, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), +(1202, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/chat'), +(1203, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-dev'), +(1204, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-space'), +(1205, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/users'); +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES +(1206, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/models', '/agent-dev'), +(1207, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/knowledges', '/agent-dev'), +(1208, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/agents', '/agent-dev'), +(1209, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/memory', '/agent-dev'); +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES +(1210, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-space', '/resource-space'), +(1211, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/mcp-space', '/resource-space'), +(1212, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/skill-space', '/resource-space'); + +-- USER Menus (Minimal, all root level) +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES +(1301, 'USER', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), +(1302, 'USER', 'VISIBILITY', 'LEFT_NAV_MENU', '/chat'), +(1303, 'USER', 'VISIBILITY', 'LEFT_NAV_MENU', '/memory'), +(1304, 'USER', 'VISIBILITY', 'LEFT_NAV_MENU', '/users'); + +-- SPEED Menus (root level) +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES +(1401, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), +(1402, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/chat'), +(1403, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-dev'), +(1404, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-space'), +(1405, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-manage'); +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES +(1406, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/models', '/agent-dev'), +(1407, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/knowledges', '/agent-dev'), +(1408, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/agents', '/agent-dev'), +(1409, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/memory', '/agent-dev'); +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES +(1410, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-space', '/resource-space'), +(1411, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/mcp-space', '/resource-space'), +(1412, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/skill-space', '/resource-space'); + +-- ASSET_OWNER Menus (root level; /owner-manage is SU-only, see v2.3.0_0713_move_owner_manage_to_su.sql) +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype) VALUES +(1501, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/'), +(1502, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/chat'), +(1503, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-dev'), +(1504, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/resource-space'); +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES +(1506, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/models', '/agent-dev'), +(1507, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/knowledges', '/agent-dev'), +(1508, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/agents', '/agent-dev'); +INSERT INTO nexent.role_permission_t (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) VALUES +(1509, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-space', '/resource-space'), +(1510, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/mcp-space', '/resource-space'), +(1511, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/skill-space', '/resource-space'); + +COMMIT; + +-- Source migration: v2.2.2_0624_migrate_agent_model_id_to_list.sql + +-- Migration: Change ag_tenant_agent_t.model_id to model_ids (list of integers) +-- Date: 2026-06-17 +-- Description: Migrate agent model configuration from single model_id to model_ids list +-- +-- Idempotency notes: +-- This migration is executed on every container restart together with all other +-- incremental migrations. The follow-up migration +-- v2.2.2_0626_drop_agent_model_id_and_model_name.sql +-- removes ag_tenant_agent_t.model_id (and model_name). Therefore, on a re-run +-- the model_id column may already be absent. Every step that references +-- model_id must be guarded so the script remains a no-op in that state. +-- +-- Migration strategy: +-- 1. Add new model_ids column as ARRAY(Integer) if it doesn't already exist +-- (idempotent via ADD COLUMN IF NOT EXISTS). +-- 2. If model_id still exists, backfill model_ids from model_id only when +-- model_ids is NULL or an empty array. Existing non-empty values are +-- preserved so the migration does not clobber data written by newer code. +-- 3. Set column comments (guarded so missing columns do not error). + +SET search_path TO nexent; + +BEGIN; + +-- 1) Add model_ids column if it doesn't exist. +-- ADD COLUMN IF NOT EXISTS is a no-op when the column already exists, so +-- this statement is safe to re-run on every startup. +ALTER TABLE nexent.ag_tenant_agent_t + ADD COLUMN IF NOT EXISTS model_ids INTEGER[] DEFAULT NULL; + +-- 2) Backfill model_ids from the legacy single-value model_id column. +-- Only runs when model_id still exists. When model_id has already been +-- dropped by a later migration (e.g. v2.2.2_0626_drop_agent_model_id_and_model_name.sql), +-- this step is skipped and the script remains a safe no-op. +-- "Empty" is defined as either NULL or an empty array ('{}'); both +-- COALESCE(array_length(model_ids, 1), 0) = 0 and model_ids IS NULL match +-- these cases. Rows whose model_ids already has values are left untouched. +DO $$ +BEGIN + IF EXISTS ( + SELECT 1 FROM information_schema.columns + WHERE table_schema = 'nexent' + AND table_name = 'ag_tenant_agent_t' + AND column_name = 'model_id' + ) THEN + UPDATE nexent.ag_tenant_agent_t + SET model_ids = ARRAY[model_id] + WHERE model_id IS NOT NULL + AND (model_ids IS NULL OR COALESCE(array_length(model_ids, 1), 0) = 0); + END IF; +END $$; + +-- 3) Update column comments. +-- model_ids is created above (or was created on an earlier run) so the +-- comment can be applied unconditionally. COMMENT ON COLUMN raises an +-- error if the column is missing, so we still guard it for safety. +DO $$ +BEGIN + IF EXISTS ( + SELECT 1 FROM information_schema.columns + WHERE table_schema = 'nexent' + AND table_name = 'ag_tenant_agent_t' + AND column_name = 'model_ids' + ) THEN + COMMENT ON COLUMN nexent.ag_tenant_agent_t.model_ids IS + 'List of model IDs, foreign key references to model_record_t.model_id, max 5 models'; + END IF; +END $$; + +-- 4) Add a deprecation comment to model_id, only when the column still exists. +-- Once v2.2.2_0626_drop_agent_model_id_and_model_name.sql has dropped it, +-- this block is skipped. +DO $$ +BEGIN + IF EXISTS ( + SELECT 1 FROM information_schema.columns + WHERE table_schema = 'nexent' + AND table_name = 'ag_tenant_agent_t' + AND column_name = 'model_id' + ) THEN + COMMENT ON COLUMN nexent.ag_tenant_agent_t.model_id IS + '[DEPRECATED] Single model ID, use model_ids instead'; + END IF; +END $$; + +COMMIT; + +-- Source migration: v2.2.2_0627_backfill_from_catalog.sql + +-- Catalog revision: 2026-06-27.1 +-- Catalog entries: 66 +-- +-- Migration kind: RECOMMENDED_DATA_FIX +-- Idempotent: COALESCE + IS NULL guards protect existing values. +-- Safe: enforces max_output < context_window via GREATEST/LEAST. +-- +-- Phases: +-- 1a Bare LLM/VLM rows that match a catalog entry by +-- (model_factory, model_repo, model_name) -> fill capacity +-- fields + tag capacity_source='profile' + profile_version. +-- 1b Already-filled rows that match a catalog entry AND whose +-- context_window_tokens and max_output_tokens exactly equal +-- the catalog values -> tag profile_version only. capacity_ +-- source stays whatever it was (typically 'operator'); we +-- don't rewrite provenance, we just add the dispatch tag so +-- dispatch_profile_hit_total can fire. +-- 2 Remaining bare LLM/VLM rows -> safe defaults. +-- 3 Clamp default_output_reserve_tokens to <= max_output_tokens. +-- +-- Pre-run self-check (rows whose capability_profile_version is NULL): +-- +-- SELECT model_id, model_repo, model_name, model_factory, +-- context_window_tokens, max_output_tokens, capability_profile_version +-- FROM nexent.model_record_t +-- WHERE delete_flag = 'N' +-- AND COALESCE(model_type, 'llm') IN ('llm', 'vlm') +-- AND capability_profile_version IS NULL; + +-- ============================================================ +-- Phase 1a: Backfill bare rows that match approved catalog entries +-- ============================================================ + +DO $$ +DECLARE + v_updated INTEGER := 0; + v_total INTEGER := 0; + c_active_flag CONSTANT TEXT := 'N'; + c_source_profile CONSTANT TEXT := 'profile'; +BEGIN + -- dashscope (4 entries) + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'dashscope/qwen-plus@1') + WHERE LOWER(model_factory) = 'dashscope' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'qwen-plus' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 1000000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'dashscope/qwen-turbo@1') + WHERE LOWER(model_factory) = 'dashscope' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'qwen-turbo' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(65536, COALESCE(context_window_tokens, 1000000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 65536))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'dashscope/qwen3.7-max@1') + WHERE LOWER(model_factory) = 'dashscope' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'qwen3.7-max' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(200000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(131072, COALESCE(context_window_tokens, 200000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 131072))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'dashscope/glm-5.1@1') + WHERE LOWER(model_factory) = 'dashscope' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'glm-5.1' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + -- deepseek (4 entries) + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(384000, COALESCE(context_window_tokens, 1000000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 384000))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'deepseek/deepseek-chat@2') + WHERE LOWER(model_factory) = 'deepseek' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'deepseek-chat' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(384000, COALESCE(context_window_tokens, 1000000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 384000))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'deepseek/deepseek-reasoner@2') + WHERE LOWER(model_factory) = 'deepseek' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'deepseek-reasoner' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(384000, COALESCE(context_window_tokens, 1000000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 384000))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'deepseek/deepseek-v4-flash@1') + WHERE LOWER(model_factory) = 'deepseek' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'deepseek-v4-flash' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(384000, COALESCE(context_window_tokens, 1000000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 384000))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'deepseek/deepseek-v4-pro@1') + WHERE LOWER(model_factory) = 'deepseek' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'deepseek-v4-pro' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + -- openai (2 entries) + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(128000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 128000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'openai/gpt-4o@1') + WHERE LOWER(model_factory) = 'openai' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'gpt-4o' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(1000000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(32768, COALESCE(context_window_tokens, 1000000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 32768))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'openai/gpt-4.1@1') + WHERE LOWER(model_factory) = 'openai' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'gpt-4.1' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + -- silicon (56 entries) + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(65536, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 65536))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.6-27b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.6-27B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(131072, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 131072))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/kimi-k2.6@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/moonshotai' + AND model_name = 'Kimi-K2.6' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(1048576, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(384000, COALESCE(context_window_tokens, 1048576) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 384000))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v4-pro-sf@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-V4-Pro' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(1048576, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(384000, COALESCE(context_window_tokens, 1048576) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 384000))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v4-flash-sf@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-V4-Flash' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3.2@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-V3.2' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3.1-terminus@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-V3.1-Terminus' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(163840, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 163840) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-r1@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-R1' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-V3' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-r1-0528-qwen3-8b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-R1-0528-Qwen3-8B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3.2-pro@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/deepseek-ai' + AND model_name = 'DeepSeek-V3.2' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3.1-terminus-pro@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/deepseek-ai' + AND model_name = 'DeepSeek-V3.1-Terminus' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(163840, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 163840) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-r1-pro@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/deepseek-ai' + AND model_name = 'DeepSeek-R1' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(164000, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 164000) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/deepseek-v3-pro@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/deepseek-ai' + AND model_name = 'DeepSeek-V3' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.6-35b-a3b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.6-35B-A3B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-397b-a17b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-397B-A17B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-122b-a10b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-122B-A10B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-35b-a3b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-35B-A3B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-27b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-27B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-9b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-9B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3.5-4b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-4B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-32b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-32B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(32768, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 32768))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-32b-thinking@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-32B-Thinking' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-8b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-8B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(32768, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 32768))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-8b-thinking@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-8B-Thinking' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-30b-a3b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-30B-A3B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(32768, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 32768))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-vl-30b-a3b-thinking@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-30B-A3B-Thinking' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-omni-30b-a3b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-Omni-30B-A3B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-omni-30b-a3b-thinking@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-Omni-30B-A3B-Thinking' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-omni-30b-a3b-captioner@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-Omni-30B-A3B-Captioner' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(65536, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 65536))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-coder-30b-a3b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-Coder-30B-A3B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-30b-a3b-instruct-2507@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-30B-A3B-Instruct-2507' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-32b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-32B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-14b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-14B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen3-8b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-8B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen2.5-72b-instruct-128k@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen2.5-72B-Instruct-128K' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen2.5-72b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen2.5-72B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen2.5-32b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen2.5-32B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen2.5-14b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen2.5-14B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/qwen2.5-7b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen2.5-7B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-4-32b-0414@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'THUDM' + AND model_name = 'GLM-4-32B-0414' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-z1-9b-0414@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'THUDM' + AND model_name = 'GLM-Z1-9B-0414' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-4-9b-0414@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'THUDM' + AND model_name = 'GLM-4-9B-0414' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(1048576, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(131072, COALESCE(context_window_tokens, 1048576) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 131072))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-5.2@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'zai-org' + AND model_name = 'GLM-5.2' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-4.5v@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'zai-org' + AND model_name = 'GLM-4.5V' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-4.5-air@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'zai-org' + AND model_name = 'GLM-4.5-Air' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(202752, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(131072, COALESCE(context_window_tokens, 202752) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 131072))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/glm-5.1-pro@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/zai-org' + AND model_name = 'GLM-5.1' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(524288, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 524288) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/seed-oss-36b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'ByteDance-Seed' + AND model_name = 'Seed-OSS-36B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/ling-flash-2.0@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'inclusionAI' + AND model_name = 'Ling-flash-2.0' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/ling-mini-2.0@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'inclusionAI' + AND model_name = 'Ling-mini-2.0' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(204800, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 204800) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/minimax-m2.5@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'MiniMaxAI' + AND model_name = 'MiniMax-M2.5' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(204800, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 204800) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/minimax-m2.5-pro@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/MiniMaxAI' + AND model_name = 'MiniMax-M2.5' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(32768, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(8192, COALESCE(max_output_tokens, 32768))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/kimi-k2.7-code@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'moonshotai' + AND model_name = 'Kimi-K2.7-Code' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/nex-n2-pro@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'nex-agi' + AND model_name = 'Nex-N2-Pro' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(262144, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(16384, COALESCE(context_window_tokens, 262144) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 16384))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/step-3.5-flash@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'stepfun-ai' + AND model_name = 'Step-3.5-Flash' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(2048, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(1024, COALESCE(max_output_tokens, 2048))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/hunyuan-mt-7b@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'tencent' + AND model_name = 'Hunyuan-MT-7B' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(131072, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(8192, COALESCE(context_window_tokens, 131072) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 8192))), + capacity_source = COALESCE(capacity_source, c_source_profile), + capability_profile_version = COALESCE(capability_profile_version, 'silicon/hunyuan-a13b-instruct@1') + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'tencent' + AND model_name = 'Hunyuan-A13B-Instruct' + AND delete_flag = c_active_flag + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + RAISE NOTICE 'Phase 1a catalog backfill (bare): % row(s) updated', v_total; +END $$; + +-- ============================================================ +-- Phase 1b: Tag already-filled rows whose ctx/max_out exactly match +-- the catalog with capability_profile_version. Upgrades +-- capacity_source from 'default' to 'profile' (values now +-- come from catalog, not system defaults). Preserves +-- 'operator' and other explicit sources. +-- ============================================================ + +DO $$ +DECLARE + v_updated INTEGER := 0; + v_total INTEGER := 0; + c_active_flag CONSTANT TEXT := 'N'; + c_source_default CONSTANT TEXT := 'default'; + c_source_profile CONSTANT TEXT := 'profile'; +BEGIN + -- dashscope (4 entries) + UPDATE nexent.model_record_t + SET capability_profile_version = 'dashscope/qwen-plus@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'dashscope' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'qwen-plus' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'dashscope/qwen-plus@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'dashscope/qwen-turbo@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'dashscope' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'qwen-turbo' + AND delete_flag = c_active_flag + AND context_window_tokens = 1000000 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'dashscope/qwen-turbo@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'dashscope/qwen3.7-max@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'dashscope' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'qwen3.7-max' + AND delete_flag = c_active_flag + AND context_window_tokens = 1000000 + AND max_output_tokens = 65536 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'dashscope/qwen3.7-max@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'dashscope/glm-5.1@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'dashscope' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'glm-5.1' + AND delete_flag = c_active_flag + AND context_window_tokens = 200000 + AND max_output_tokens = 131072 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'dashscope/glm-5.1@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + -- deepseek (4 entries) + UPDATE nexent.model_record_t + SET capability_profile_version = 'deepseek/deepseek-chat@2', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'deepseek' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'deepseek-chat' + AND delete_flag = c_active_flag + AND context_window_tokens = 1000000 + AND max_output_tokens = 384000 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'deepseek/deepseek-chat@2' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'deepseek/deepseek-reasoner@2', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'deepseek' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'deepseek-reasoner' + AND delete_flag = c_active_flag + AND context_window_tokens = 1000000 + AND max_output_tokens = 384000 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'deepseek/deepseek-reasoner@2' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'deepseek/deepseek-v4-flash@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'deepseek' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'deepseek-v4-flash' + AND delete_flag = c_active_flag + AND context_window_tokens = 1000000 + AND max_output_tokens = 384000 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'deepseek/deepseek-v4-flash@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'deepseek/deepseek-v4-pro@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'deepseek' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'deepseek-v4-pro' + AND delete_flag = c_active_flag + AND context_window_tokens = 1000000 + AND max_output_tokens = 384000 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'deepseek/deepseek-v4-pro@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + -- openai (2 entries) + UPDATE nexent.model_record_t + SET capability_profile_version = 'openai/gpt-4o@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'openai' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'gpt-4o' + AND delete_flag = c_active_flag + AND context_window_tokens = 128000 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'openai/gpt-4o@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'openai/gpt-4.1@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'openai' + AND (model_repo IS NULL OR model_repo = '') + AND model_name = 'gpt-4.1' + AND delete_flag = c_active_flag + AND context_window_tokens = 1000000 + AND max_output_tokens = 32768 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'openai/gpt-4.1@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + -- silicon (56 entries) + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3.6-27b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.6-27B' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 65536 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.6-27b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/kimi-k2.6@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/moonshotai' + AND model_name = 'Kimi-K2.6' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 131072 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/kimi-k2.6@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-v4-pro-sf@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-V4-Pro' + AND delete_flag = c_active_flag + AND context_window_tokens = 1048576 + AND max_output_tokens = 384000 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v4-pro-sf@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-v4-flash-sf@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-V4-Flash' + AND delete_flag = c_active_flag + AND context_window_tokens = 1048576 + AND max_output_tokens = 384000 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v4-flash-sf@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-v3.2@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-V3.2' + AND delete_flag = c_active_flag + AND context_window_tokens = 164000 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3.2@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-v3.1-terminus@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-V3.1-Terminus' + AND delete_flag = c_active_flag + AND context_window_tokens = 164000 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3.1-terminus@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-r1@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-R1' + AND delete_flag = c_active_flag + AND context_window_tokens = 163840 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-r1@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-v3@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-V3' + AND delete_flag = c_active_flag + AND context_window_tokens = 164000 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-r1-0528-qwen3-8b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'deepseek-ai' + AND model_name = 'DeepSeek-R1-0528-Qwen3-8B' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-r1-0528-qwen3-8b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-v3.2-pro@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/deepseek-ai' + AND model_name = 'DeepSeek-V3.2' + AND delete_flag = c_active_flag + AND context_window_tokens = 164000 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3.2-pro@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-v3.1-terminus-pro@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/deepseek-ai' + AND model_name = 'DeepSeek-V3.1-Terminus' + AND delete_flag = c_active_flag + AND context_window_tokens = 164000 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3.1-terminus-pro@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-r1-pro@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/deepseek-ai' + AND model_name = 'DeepSeek-R1' + AND delete_flag = c_active_flag + AND context_window_tokens = 163840 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-r1-pro@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/deepseek-v3-pro@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/deepseek-ai' + AND model_name = 'DeepSeek-V3' + AND delete_flag = c_active_flag + AND context_window_tokens = 164000 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/deepseek-v3-pro@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3.6-35b-a3b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.6-35B-A3B' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.6-35b-a3b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3.5-397b-a17b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-397B-A17B' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-397b-a17b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3.5-122b-a10b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-122B-A10B' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-122b-a10b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3.5-35b-a3b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-35B-A3B' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-35b-a3b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3.5-27b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-27B' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-27b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3.5-9b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-9B' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-9b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3.5-4b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3.5-4B' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3.5-4b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-vl-32b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-32B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-32b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-vl-32b-thinking@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-32B-Thinking' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 32768 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-32b-thinking@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-vl-8b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-8B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-8b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-vl-8b-thinking@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-8B-Thinking' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 32768 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-8b-thinking@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-vl-30b-a3b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-30B-A3B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-30b-a3b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-vl-30b-a3b-thinking@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-VL-30B-A3B-Thinking' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 32768 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-vl-30b-a3b-thinking@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-omni-30b-a3b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-Omni-30B-A3B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-omni-30b-a3b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-omni-30b-a3b-thinking@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-Omni-30B-A3B-Thinking' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-omni-30b-a3b-thinking@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-omni-30b-a3b-captioner@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-Omni-30B-A3B-Captioner' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-omni-30b-a3b-captioner@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-coder-30b-a3b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-Coder-30B-A3B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 65536 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-coder-30b-a3b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-30b-a3b-instruct-2507@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-30B-A3B-Instruct-2507' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-30b-a3b-instruct-2507@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-32b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-32B' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-32b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-14b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-14B' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-14b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen3-8b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen3-8B' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen3-8b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen2.5-72b-instruct-128k@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen2.5-72B-Instruct-128K' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen2.5-72b-instruct-128k@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen2.5-72b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen2.5-72B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen2.5-72b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen2.5-32b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen2.5-32B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen2.5-32b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen2.5-14b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen2.5-14B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen2.5-14b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/qwen2.5-7b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Qwen' + AND model_name = 'Qwen2.5-7B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/qwen2.5-7b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/glm-4-32b-0414@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'THUDM' + AND model_name = 'GLM-4-32B-0414' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-4-32b-0414@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/glm-z1-9b-0414@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'THUDM' + AND model_name = 'GLM-Z1-9B-0414' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-z1-9b-0414@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/glm-4-9b-0414@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'THUDM' + AND model_name = 'GLM-4-9B-0414' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-4-9b-0414@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/glm-5.2@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'zai-org' + AND model_name = 'GLM-5.2' + AND delete_flag = c_active_flag + AND context_window_tokens = 1048576 + AND max_output_tokens = 131072 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-5.2@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/glm-4.5v@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'zai-org' + AND model_name = 'GLM-4.5V' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-4.5v@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/glm-4.5-air@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'zai-org' + AND model_name = 'GLM-4.5-Air' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-4.5-air@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/glm-5.1-pro@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/zai-org' + AND model_name = 'GLM-5.1' + AND delete_flag = c_active_flag + AND context_window_tokens = 202752 + AND max_output_tokens = 131072 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/glm-5.1-pro@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/seed-oss-36b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'ByteDance-Seed' + AND model_name = 'Seed-OSS-36B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 524288 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/seed-oss-36b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/ling-flash-2.0@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'inclusionAI' + AND model_name = 'Ling-flash-2.0' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/ling-flash-2.0@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/ling-mini-2.0@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'inclusionAI' + AND model_name = 'Ling-mini-2.0' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/ling-mini-2.0@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/minimax-m2.5@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'MiniMaxAI' + AND model_name = 'MiniMax-M2.5' + AND delete_flag = c_active_flag + AND context_window_tokens = 204800 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/minimax-m2.5@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/minimax-m2.5-pro@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'Pro/MiniMaxAI' + AND model_name = 'MiniMax-M2.5' + AND delete_flag = c_active_flag + AND context_window_tokens = 204800 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/minimax-m2.5-pro@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/kimi-k2.7-code@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'moonshotai' + AND model_name = 'Kimi-K2.7-Code' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 32768 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/kimi-k2.7-code@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/nex-n2-pro@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'nex-agi' + AND model_name = 'Nex-N2-Pro' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/nex-n2-pro@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/step-3.5-flash@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'stepfun-ai' + AND model_name = 'Step-3.5-Flash' + AND delete_flag = c_active_flag + AND context_window_tokens = 262144 + AND max_output_tokens = 16384 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/step-3.5-flash@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/hunyuan-mt-7b@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'tencent' + AND model_name = 'Hunyuan-MT-7B' + AND delete_flag = c_active_flag + AND context_window_tokens = 32768 + AND max_output_tokens = 2048 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/hunyuan-mt-7b@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + UPDATE nexent.model_record_t + SET capability_profile_version = 'silicon/hunyuan-a13b-instruct@1', + capacity_source = CASE WHEN capacity_source = c_source_default THEN c_source_profile ELSE capacity_source END + WHERE LOWER(model_factory) = 'silicon' + AND model_repo = 'tencent' + AND model_name = 'Hunyuan-A13B-Instruct' + AND delete_flag = c_active_flag + AND context_window_tokens = 131072 + AND max_output_tokens = 8192 + AND (capability_profile_version IS NULL OR (capability_profile_version = 'silicon/hunyuan-a13b-instruct@1' AND capacity_source = c_source_default)); + GET DIAGNOSTICS v_updated = ROW_COUNT; + v_total := v_total + v_updated; + + RAISE NOTICE 'Phase 1b catalog tag (matching filled): % row(s) updated', v_total; +END $$; + +-- ============================================================ +-- Phase 2: Safe defaults for remaining bare LLM/VLM rows +-- ============================================================ + +DO $$ +DECLARE + v_updated INTEGER := 0; + c_active_flag CONSTANT TEXT := 'N'; + c_source_default CONSTANT TEXT := 'default'; +BEGIN + UPDATE nexent.model_record_t + SET context_window_tokens = COALESCE(context_window_tokens, + GREATEST(32768, COALESCE(max_output_tokens, 0) + 1)), + max_output_tokens = COALESCE(max_output_tokens, + LEAST(4096, COALESCE(context_window_tokens, 32768) - 1)), + default_output_reserve_tokens = COALESCE(default_output_reserve_tokens, + LEAST(4096, COALESCE(max_output_tokens, 4096))), + capacity_source = COALESCE(capacity_source, c_source_default) + WHERE delete_flag = c_active_flag + AND COALESCE(model_type, 'llm') IN ('llm', 'vlm') + AND (context_window_tokens IS NULL OR max_output_tokens IS NULL); + + GET DIAGNOSTICS v_updated = ROW_COUNT; + RAISE NOTICE 'Safe defaults: % LLM/VLM row(s) backfilled', v_updated; +END $$; + +-- ============================================================ +-- Phase 3: Clamp default_output_reserve_tokens to max_output_tokens +-- ============================================================ + +DO $$ +DECLARE + v_updated INTEGER := 0; + c_active_flag CONSTANT TEXT := 'N'; +BEGIN + UPDATE nexent.model_record_t + SET default_output_reserve_tokens = max_output_tokens + WHERE delete_flag = c_active_flag + AND default_output_reserve_tokens IS NOT NULL + AND max_output_tokens IS NOT NULL + AND default_output_reserve_tokens > max_output_tokens; + + GET DIAGNOSTICS v_updated = ROW_COUNT; + RAISE NOTICE 'reserve clamp: % row(s) updated', v_updated; +END $$; + +-- Source migration: v2.2.2_0629_conversation_message_unit_status_and_clean.sql + +-- Migration: Add status / unit_status fields to support streaming persistence +-- Date: 2026-06-29 +-- Description: Allow per-message and per-unit lifecycle tracking so the +-- frontend can recover partial agent runs when the SSE connection is lost + +SET search_path TO nexent; + +BEGIN; + +-- Message-level lifecycle. Assistant messages start as 'pending' / 'streaming' +-- and transition to one of completed / failed / stopped. User messages default +-- to 'completed' (existing rows are backfilled below). +ALTER TABLE nexent.conversation_message_t + ADD COLUMN IF NOT EXISTS status VARCHAR(30); + +COMMENT ON COLUMN nexent.conversation_message_t.status IS + 'Lifecycle status: pending / streaming / completed / failed / stopped.'; + +-- Unit-level lifecycle. Once a unit is fully persisted we mark it 'completed'; +-- while the boundary is still being detected it remains 'streaming'. +ALTER TABLE nexent.conversation_message_unit_t + ADD COLUMN IF NOT EXISTS unit_status VARCHAR(30); + +COMMENT ON COLUMN nexent.conversation_message_unit_t.unit_status IS + 'Lifecycle status: streaming (still aggregating) or completed (fully persisted).'; + +-- Index for incremental recovery queries (since_message_unit_id filters). +CREATE INDEX IF NOT EXISTS idx_message_unit_message_id_unit_id + ON nexent.conversation_message_unit_t (message_id, unit_id); + +-- Cleanup stale deep_thinking units. +DO $$ +BEGIN + IF EXISTS ( + SELECT 1 FROM information_schema.columns + WHERE table_schema = 'nexent' + AND table_name = 'conversation_message_unit_t' + AND column_name = 'unit_status' + ) THEN + DELETE FROM nexent.conversation_message_unit_t + WHERE unit_type = 'model_output_deep_thinking' + AND unit_status IS NULL; + END IF; +END $$; + +-- Cleanup corrupted records of thinking units +DO $$ +BEGIN + IF EXISTS ( + SELECT 1 FROM information_schema.columns + WHERE table_schema = 'nexent' + AND table_name = 'conversation_message_unit_t' + AND column_name = 'unit_status' + ) THEN + DELETE FROM nexent.conversation_message_unit_t + WHERE unit_type = 'model_output_thinking' + AND unit_content = '' + AND unit_status IS NULL; + END IF; +END $$; + +COMMIT; diff --git a/deploy/sql/migrations/v2.3.0_0624_add_labels_to_ag_tool_info.sql b/deploy/sql/migrations/v2.3.0_0624_add_labels_to_ag_tool_info.sql deleted file mode 100644 index f1e54084d6..0000000000 --- a/deploy/sql/migrations/v2.3.0_0624_add_labels_to_ag_tool_info.sql +++ /dev/null @@ -1,33 +0,0 @@ --- Add labels column to ag_tool_info_t table for tool filtering/grouping -ALTER TABLE nexent.ag_tool_info_t -ADD COLUMN IF NOT EXISTS labels JSONB DEFAULT '[]'::jsonb; - -COMMENT ON COLUMN nexent.ag_tool_info_t.labels IS 'JSON array of label strings for filtering/grouping tools'; - --- Seed built-in labels for well-known local tools. --- These labels serve as suggested defaults and can be modified by users. --- Keep in sync with: backend/consts/tool_labels.py - -WITH label_map AS ( - SELECT key AS tool_name, value AS label FROM jsonb_each_text('{ - "mysql_database": "database", "postgres_database": "database", "mssql_database": "database", - "read_file": "file", "create_file": "file", "delete_file": "file", - "create_directory": "file", "delete_directory": "file", "list_directory": "file", - "move_item": "file", - "tavily_search": "search", "exa_search": "search", "linkup_search": "search", - "search_memory": "search", "knowledge_base_search": "search", - "dify_search": "knowledge-base", "datamate_search": "knowledge-base", - "idata_search": "knowledge-base", "haotian_search": "knowledge-base", - "ragflow_search": "knowledge-base", - "aidp_search": "knowledge-base", - "analyze_image": "multimodal", "analyze_audio": "multimodal", - "analyze_video": "multimodal", "analyze_text_file": "multimodal", - "get_email": "email", "send_email": "email", - "store_memory": "memory", - "terminal": "terminal" - }'::jsonb) -) -UPDATE nexent.ag_tool_info_t t -SET labels = to_jsonb(ARRAY[m.label]) -FROM label_map m -WHERE t.name = m.tool_name AND t.labels = '[]'::jsonb; diff --git a/deploy/sql/migrations/v2.3.0_0628_add_agent_evaluation_full.sql b/deploy/sql/migrations/v2.3.0_0628_add_agent_evaluation_full.sql deleted file mode 100644 index 3ec4e1dbf2..0000000000 --- a/deploy/sql/migrations/v2.3.0_0628_add_agent_evaluation_full.sql +++ /dev/null @@ -1,220 +0,0 @@ --- ============================================================================= --- Agent evaluation (offline) - full bundle --- ============================================================================= --- Version: v2.3.0 --- Date: 2026-06-30 --- Description: Single-file bundle for the v2.3.0 agent evaluation feature. --- Combines what were originally three separate migration drafts (0628, 0630 --- pass_status, 0630 route grant) before any of them had been applied to --- any environment. Use this file on fresh installs. --- --- Idempotency: every DDL statement in this file is safe to run multiple times. --- - CREATE TABLE IF NOT EXISTS --- - ALTER TABLE ... ADD COLUMN IF NOT EXISTS --- - CREATE INDEX IF NOT EXISTS --- - COMMENT ON (overwrites previous value, no-op if identical) --- - INSERT ... ON CONFLICT (role_permission_id) DO NOTHING --- --- Sections: --- 1. evaluation_set_t / evaluation_set_case_t / agent_evaluation_t / --- agent_evaluation_case_t (incl. judge_model_id on agent_evaluation_t) --- 2. pass_status column on agent_evaluation_case_t + composite index --- 3. LEFT_NAV_MENU '/space' grant for roles that have /agent-space --- --- Design decisions (see PR review 2026-06-30): --- * No standalone (tenant_id) index on any table. Every case-level and --- run-level read is scoped by PK or by a foreign key into a parent that --- itself is already tenant-scoped at the application layer. A bare --- (tenant_id) index has no real query plan and only inflates write cost. --- * No (tenant_id, judge_model_id) index. judge_model_id is read alongside --- the row via PK; "list runs by judge model" is not a supported query. --- * No (tenant_id, evaluation_set_id) on evaluation_set_case_t. The set --- itself is tenant-scoped at the app layer, and the existing --- (evaluation_set_id) index already covers set-case listing. --- * ix_agent_eval_case_pass_status is (agent_evaluation_id, pass_status) --- rather than (tenant_id, agent_evaluation_id, pass_status): case-level --- reads never filter on tenant_id directly, and dropping the leading --- tenant_id column keeps the most common "list failed cases for run X" --- query on a single composite index. --- * Section 3 INSERT must include parent_key (see 0622 menu migration) so --- a future renderer that joins on parent_key does not leave this batch --- as orphans. /space is a first-level entry for the route guard only, --- so parent_key is NULL. --- ============================================================================= - -SET search_path TO nexent; - -BEGIN; - - --- ----------------------------------------------------------------------------- --- Section 1: Evaluation set & evaluation run tables --- ----------------------------------------------------------------------------- -CREATE TABLE IF NOT EXISTS nexent.evaluation_set_t ( - evaluation_set_id BIGSERIAL PRIMARY KEY, - tenant_id VARCHAR(100) NOT NULL, - - name VARCHAR(255) NOT NULL, - description TEXT, - - source_filename VARCHAR(255), - case_count INTEGER DEFAULT 0, - - create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), - update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), - created_by VARCHAR(100), - updated_by VARCHAR(100), - delete_flag VARCHAR(1) DEFAULT 'N' -); - -CREATE INDEX IF NOT EXISTS ix_eval_set_name ON nexent.evaluation_set_t(tenant_id, name); - -COMMENT ON TABLE nexent.evaluation_set_t IS 'Offline evaluation sets (JSONL single-turn cases).'; -COMMENT ON COLUMN nexent.evaluation_set_t.tenant_id IS 'Tenant ID for multi-tenancy isolation'; -COMMENT ON COLUMN nexent.evaluation_set_t.source_filename IS 'Original uploaded filename'; -COMMENT ON COLUMN nexent.evaluation_set_t.case_count IS 'Total number of cases'; - - -CREATE TABLE IF NOT EXISTS nexent.evaluation_set_case_t ( - evaluation_set_case_id BIGSERIAL PRIMARY KEY, - tenant_id VARCHAR(100) NOT NULL, - evaluation_set_id BIGINT NOT NULL, - - case_id VARCHAR(128), - inputs JSONB NOT NULL, - label JSONB NOT NULL, - order_no INTEGER DEFAULT 0, - - create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), - update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), - created_by VARCHAR(100), - updated_by VARCHAR(100), - delete_flag VARCHAR(1) DEFAULT 'N' -); - -CREATE INDEX IF NOT EXISTS ix_eval_set_case_set_id ON nexent.evaluation_set_case_t(evaluation_set_id); - -COMMENT ON TABLE nexent.evaluation_set_case_t IS 'Cases within evaluation sets.'; -COMMENT ON COLUMN nexent.evaluation_set_case_t.inputs IS 'Case inputs JSON: {query: string, context?: string}'; -COMMENT ON COLUMN nexent.evaluation_set_case_t.label IS 'Case label JSON: {answer: string}'; - - -CREATE TABLE IF NOT EXISTS nexent.agent_evaluation_t ( - agent_evaluation_id BIGSERIAL PRIMARY KEY, - tenant_id VARCHAR(100) NOT NULL, - - agent_id INTEGER NOT NULL, - agent_version_no INTEGER NOT NULL, - - evaluation_set_id BIGINT NOT NULL, - - status VARCHAR(30) NOT NULL DEFAULT 'PENDING', - - progress_total INTEGER DEFAULT 0, - progress_done INTEGER DEFAULT 0, - - score_overall DOUBLE PRECISION, - error_message TEXT, - - judge_model_id INTEGER, - - create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), - update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), - created_by VARCHAR(100), - updated_by VARCHAR(100), - delete_flag VARCHAR(1) DEFAULT 'N' -); - -CREATE INDEX IF NOT EXISTS ix_agent_eval_agent_id ON nexent.agent_evaluation_t(tenant_id, agent_id); -CREATE INDEX IF NOT EXISTS ix_agent_eval_set_id ON nexent.agent_evaluation_t(tenant_id, evaluation_set_id); - -COMMENT ON TABLE nexent.agent_evaluation_t IS 'Offline evaluation runs for an agent.'; -COMMENT ON COLUMN nexent.agent_evaluation_t.status IS 'Run status: PENDING/RUNNING/COMPLETED/FAILED'; -COMMENT ON COLUMN nexent.agent_evaluation_t.judge_model_id IS - 'Model id used by the judge. Persisted so the background worker can recover it after restart and so the frontend can display judge_model_name.'; - - -CREATE TABLE IF NOT EXISTS nexent.agent_evaluation_case_t ( - agent_evaluation_case_id BIGSERIAL PRIMARY KEY, - tenant_id VARCHAR(100) NOT NULL, - - agent_evaluation_id BIGINT NOT NULL, - evaluation_set_case_id BIGINT NOT NULL, - - inputs JSONB NOT NULL, - label JSONB NOT NULL, - predict JSONB, - - score DOUBLE PRECISION, - reason TEXT, - - status VARCHAR(30) NOT NULL DEFAULT 'PENDING', - error_message TEXT, - - create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), - update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), - created_by VARCHAR(100), - updated_by VARCHAR(100), - delete_flag VARCHAR(1) DEFAULT 'N' -); - -CREATE INDEX IF NOT EXISTS ix_agent_eval_case_eval_id ON nexent.agent_evaluation_case_t(agent_evaluation_id); - -COMMENT ON TABLE nexent.agent_evaluation_case_t IS 'Per-case evaluation results.'; -COMMENT ON COLUMN nexent.agent_evaluation_case_t.predict IS 'Predict JSON: {answer: string, raw?: any}'; -COMMENT ON COLUMN nexent.agent_evaluation_case_t.status IS 'Case status: PENDING/RUNNING/COMPLETED/FAILED'; - - --- ----------------------------------------------------------------------------- --- Section 2: pass_status on agent_evaluation_case_t --- ----------------------------------------------------------------------------- --- Stores the binary judge result ("pass" / "fail") for each case. --- Enables fast filtering for failed-case reports and storage optimization: --- passed cases have predict/reason/label.answer cleared to save space, --- while only failed cases retain full detail. - -ALTER TABLE nexent.agent_evaluation_case_t -ADD COLUMN IF NOT EXISTS pass_status VARCHAR(16); - -COMMENT ON COLUMN nexent.agent_evaluation_case_t.pass_status IS - 'Judge result per case: pass / fail. pass cases have predict/reason/label.answer cleared to save space.'; - --- Composite index to support failed-case listing and "only failed" reports. --- Scoped by (agent_evaluation_id, pass_status) only; tenant_id is enforced --- at the application layer via the parent run's tenant. -CREATE INDEX IF NOT EXISTS ix_agent_eval_case_pass_status - ON nexent.agent_evaluation_case_t (agent_evaluation_id, pass_status); - - --- ----------------------------------------------------------------------------- --- Section 3: Grant /space LEFT_NAV_MENU so the evaluation page is reachable --- ----------------------------------------------------------------------------- --- The agent evaluation page lives at the route prefix /space/agents/{id}/evaluate. --- The previous menu migration (v2.2.2_0622_update_left_nav_menu.sql) removed --- the legacy /space entry when it refactored the menu structure. As a result --- the frontend route guard (which uses accessibleRoutes prefix matching) blocks --- any user from entering the evaluation page with "no access permission". --- --- This section adds LEFT_NAV_MENU = '/space' for every role that already has --- access to the resource-space (i.e. /agent-space). This entry is NOT rendered --- in the side navigation (SideNavigation uses exact-match against ROUTE_CONFIG) --- but it IS picked up by the backend as part of accessibleRoutes, so the route --- guard will allow /space/agents/{id}/evaluate and its sub-paths. - --- Roles that already have /resource-space (and thus /agent-space) get /space. --- Mirrors v2.2.2_0622_update_left_nav_menu.sql IDs (16xx range) to keep the --- scheme consistent. parent_key is NULL: /space is a top-level entry used --- only by the backend route guard (prefix match on accessibleRoutes) and --- is not rendered by SideNavigation. -INSERT INTO nexent.role_permission_t - (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) -VALUES - (1600, 'SU', 'VISIBILITY', 'LEFT_NAV_MENU', '/space', NULL), - (1601, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/space', NULL), - (1602, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/space', NULL), - (1603, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/space', NULL), - (1604, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/space', NULL) -ON CONFLICT (role_permission_id) DO NOTHING; - - -COMMIT; diff --git a/deploy/sql/migrations/v2.3.0_0629_add_skill_repository_table.sql b/deploy/sql/migrations/v2.3.0_0629_add_skill_repository_table.sql deleted file mode 100644 index 43f7fd8a4a..0000000000 --- a/deploy/sql/migrations/v2.3.0_0629_add_skill_repository_table.sql +++ /dev/null @@ -1,99 +0,0 @@ --- Migration: Add ag_skill_repository_t table --- Date: 2026-06-29 --- Description: Skill marketplace repository for frozen installable skill snapshots. - -SET search_path TO nexent; - -CREATE SEQUENCE IF NOT EXISTS nexent.ag_skill_repository_t_skill_repository_id_seq; - -CREATE TABLE IF NOT EXISTS nexent.ag_skill_repository_t ( - skill_repository_id BIGINT NOT NULL DEFAULT nextval('nexent.ag_skill_repository_t_skill_repository_id_seq'), - publisher_tenant_id VARCHAR(100) NOT NULL, - publisher_user_id VARCHAR(100) NOT NULL, - skill_id INTEGER NOT NULL, - name VARCHAR(100) NOT NULL, - description TEXT, - source VARCHAR(30), - submitted_by VARCHAR(100), - category_id INTEGER, - tags TEXT[], - icon VARCHAR(100), - downloads INTEGER DEFAULT 0, - skill_info_json JSONB NOT NULL, - skill_zip_base64 TEXT NOT NULL, - status VARCHAR(30) DEFAULT 'not_shared', - create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, - update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, - created_by VARCHAR(100), - updated_by VARCHAR(100), - delete_flag VARCHAR(1) DEFAULT 'N', - CONSTRAINT ag_skill_repository_t_pkey PRIMARY KEY (skill_repository_id) -); - -ALTER SEQUENCE nexent.ag_skill_repository_t_skill_repository_id_seq - OWNED BY nexent.ag_skill_repository_t.skill_repository_id; - -ALTER TABLE nexent.ag_skill_repository_t OWNER TO root; - -ALTER TABLE nexent.ag_skill_repository_t - ADD COLUMN IF NOT EXISTS submitted_by VARCHAR(100), - ADD COLUMN IF NOT EXISTS icon VARCHAR(100), - ADD COLUMN IF NOT EXISTS downloads INTEGER DEFAULT 0, - ADD COLUMN IF NOT EXISTS skill_zip_base64 TEXT; - -COMMENT ON TABLE nexent.ag_skill_repository_t IS 'Skill marketplace repository for frozen installable skill snapshots'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.skill_repository_id IS 'Skill repository listing ID, unique primary key'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.publisher_tenant_id IS 'Publisher tenant ID'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.publisher_user_id IS 'Publisher user ID'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.skill_id IS 'Source skill ID from ag_skill_info_t; unique when active (delete_flag = N)'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.name IS 'Skill name for display and search'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.description IS 'Skill description'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.source IS 'Skill source'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.submitted_by IS 'Submitter email when listing enters pending_review'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.category_id IS 'Optional marketplace category ID'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.tags IS 'Marketplace tags'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.icon IS 'Marketplace card icon (emoji or URL)'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.downloads IS 'Marketplace install count for card display'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.skill_info_json IS 'Frozen skill metadata snapshot'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.skill_zip_base64 IS 'Frozen skill ZIP payload encoded as base64'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.status IS 'Listing status: not_shared / pending_review / rejected / shared'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.create_time IS 'Creation time'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.update_time IS 'Update time'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.created_by IS 'Creator ID'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.updated_by IS 'Updater ID'; -COMMENT ON COLUMN nexent.ag_skill_repository_t.delete_flag IS 'Soft delete flag: Y/N'; - -CREATE UNIQUE INDEX IF NOT EXISTS uq_skill_repository_skill_active - ON nexent.ag_skill_repository_t (skill_id) - WHERE delete_flag = 'N'; - -CREATE INDEX IF NOT EXISTS idx_skill_repository_publisher_delete - ON nexent.ag_skill_repository_t (publisher_tenant_id, delete_flag); - -CREATE INDEX IF NOT EXISTS idx_skill_repository_status_delete - ON nexent.ag_skill_repository_t (status, delete_flag); - -CREATE INDEX IF NOT EXISTS idx_skill_repository_name_delete - ON nexent.ag_skill_repository_t (name, delete_flag); - -CREATE INDEX IF NOT EXISTS idx_skill_repository_tags_gin - ON nexent.ag_skill_repository_t USING GIN (tags); - -CREATE OR REPLACE FUNCTION update_ag_skill_repository_update_time() -RETURNS TRIGGER AS $$ -BEGIN - NEW.update_time = CURRENT_TIMESTAMP; - RETURN NEW; -END; -$$ LANGUAGE plpgsql; - -COMMENT ON FUNCTION update_ag_skill_repository_update_time() IS 'Auto-update update_time for ag_skill_repository_t'; - -DROP TRIGGER IF EXISTS update_ag_skill_repository_update_time_trigger ON nexent.ag_skill_repository_t; -CREATE TRIGGER update_ag_skill_repository_update_time_trigger -BEFORE UPDATE ON nexent.ag_skill_repository_t -FOR EACH ROW -EXECUTE FUNCTION update_ag_skill_repository_update_time(); - -COMMENT ON TRIGGER update_ag_skill_repository_update_time_trigger -ON nexent.ag_skill_repository_t IS 'Trigger to maintain update_time'; diff --git a/deploy/sql/migrations/v2.3.0_0709_add_conversation_agent_id.sql b/deploy/sql/migrations/v2.3.0_0709_add_conversation_agent_id.sql deleted file mode 100644 index 8f59d75441..0000000000 --- a/deploy/sql/migrations/v2.3.0_0709_add_conversation_agent_id.sql +++ /dev/null @@ -1,6 +0,0 @@ --- Store the latest agent used by each conversation so history selection can restore agent context. -ALTER TABLE nexent.conversation_record_t - ADD COLUMN IF NOT EXISTS agent_id INTEGER; - -COMMENT ON COLUMN nexent.conversation_record_t.agent_id - IS 'Agent ID used by the latest run in this conversation'; diff --git a/deploy/sql/migrations/v2.3.0_0709_add_mcp_market_tables.sql b/deploy/sql/migrations/v2.3.0_0709_add_mcp_market_tables.sql deleted file mode 100644 index 91389ba7cb..0000000000 --- a/deploy/sql/migrations/v2.3.0_0709_add_mcp_market_tables.sql +++ /dev/null @@ -1,153 +0,0 @@ --- Migration: Add MCP market tables (v2.4.0 single-table design) --- Date: 2026-07-09 --- Description: Create mcp_market_record_t (single-table with inline review status), --- add market_id to mcp_record_t, add review_status/review_type to --- mcp_community_record_t. - -SET search_path TO nexent; - -BEGIN; - --- ============================================================================ --- 1) Extend mcp_record_t for market integration (idempotent) --- ============================================================================ -ALTER TABLE IF EXISTS nexent.mcp_record_t - ADD COLUMN IF NOT EXISTS market_id INTEGER; - -COMMENT ON COLUMN nexent.mcp_record_t.market_id IS 'Published market record ID (FK to mcp_market_record_t)'; - --- ============================================================================ --- 2) Extend mcp_community_record_t for review workflow (idempotent) --- ============================================================================ -ALTER TABLE IF EXISTS nexent.mcp_community_record_t - ADD COLUMN IF NOT EXISTS review_status VARCHAR(30) DEFAULT 'pending', - ADD COLUMN IF NOT EXISTS review_type VARCHAR(30) DEFAULT 'initial_listing'; - -COMMENT ON COLUMN nexent.mcp_community_record_t.review_status IS 'Review status: pending/approved/rejected/offline'; -COMMENT ON COLUMN nexent.mcp_community_record_t.review_type IS 'Review submission type: initial_listing/update'; - --- ============================================================================ --- 3) Create mcp_market_record_t (single-table design) --- ============================================================================ - -CREATE SEQUENCE IF NOT EXISTS nexent.mcp_market_record_t_market_id_seq; - -CREATE TABLE IF NOT EXISTS nexent.mcp_market_record_t ( - market_id BIGINT NOT NULL DEFAULT nextval('nexent.mcp_market_record_t_market_id_seq'), - tenant_id VARCHAR(100) NOT NULL, - user_id VARCHAR(100) NOT NULL, - mcp_name VARCHAR(100) NOT NULL, - mcp_server VARCHAR(500) NOT NULL, - source VARCHAR(30) DEFAULT 'community', - registry_json JSONB, - transport_type VARCHAR(30), - config_json JSON, - tags TEXT[], - description TEXT, - download_count INTEGER DEFAULT 0, - review_status VARCHAR(30) DEFAULT 'not_shared', - submitted_by VARCHAR(100), - source_mcp_id INTEGER, - create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, - update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, - created_by VARCHAR(100), - updated_by VARCHAR(100), - delete_flag VARCHAR(1) DEFAULT 'N' -); - -ALTER TABLE nexent.mcp_market_record_t OWNER TO root; - -COMMENT ON TABLE nexent.mcp_market_record_t IS 'MCP market (community) records — single table covering all listing states'; -COMMENT ON COLUMN nexent.mcp_market_record_t.market_id IS 'Market record ID, unique primary key'; -COMMENT ON COLUMN nexent.mcp_market_record_t.tenant_id IS 'Publisher tenant ID'; -COMMENT ON COLUMN nexent.mcp_market_record_t.user_id IS 'Publisher user ID'; -COMMENT ON COLUMN nexent.mcp_market_record_t.mcp_name IS 'MCP name'; -COMMENT ON COLUMN nexent.mcp_market_record_t.mcp_server IS 'MCP server URL'; -COMMENT ON COLUMN nexent.mcp_market_record_t.source IS 'Source type, fixed to community'; -COMMENT ON COLUMN nexent.mcp_market_record_t.registry_json IS 'Full MCP metadata JSON'; -COMMENT ON COLUMN nexent.mcp_market_record_t.transport_type IS 'Transport type: http/sse/container'; -COMMENT ON COLUMN nexent.mcp_market_record_t.config_json IS 'Public-shareable MCP configuration JSON'; -COMMENT ON COLUMN nexent.mcp_market_record_t.tags IS 'Tags'; -COMMENT ON COLUMN nexent.mcp_market_record_t.description IS 'Description'; -COMMENT ON COLUMN nexent.mcp_market_record_t.download_count IS 'Cumulative download/install count'; -COMMENT ON COLUMN nexent.mcp_market_record_t.review_status IS 'Listing status: not_shared/pending_review/shared/rejected'; -COMMENT ON COLUMN nexent.mcp_market_record_t.submitted_by IS 'Email of the user who submitted for review'; -COMMENT ON COLUMN nexent.mcp_market_record_t.source_mcp_id IS 'Source mcp_record_t ID that was published to the market'; - --- Indexes -CREATE UNIQUE INDEX IF NOT EXISTS uq_mcp_market_name_active - ON nexent.mcp_market_record_t (mcp_name) - WHERE delete_flag = 'N' AND review_status = 'shared'; - -CREATE INDEX IF NOT EXISTS idx_mcp_market_tenant_delete - ON nexent.mcp_market_record_t (tenant_id, delete_flag); -CREATE INDEX IF NOT EXISTS idx_mcp_market_status_delete - ON nexent.mcp_market_record_t (review_status, delete_flag); -CREATE INDEX IF NOT EXISTS idx_mcp_market_tags_gin - ON nexent.mcp_market_record_t USING GIN (tags); - --- Trigger: auto-update update_time -CREATE OR REPLACE FUNCTION update_mcp_market_record_update_time() -RETURNS TRIGGER AS $$ -BEGIN - NEW.update_time = CURRENT_TIMESTAMP; - RETURN NEW; -END; -$$ LANGUAGE plpgsql; - -COMMENT ON FUNCTION update_mcp_market_record_update_time() IS 'Auto-update update_time for mcp_market_record_t'; - -DROP TRIGGER IF EXISTS update_mcp_market_record_update_time_trigger ON nexent.mcp_market_record_t; -CREATE TRIGGER update_mcp_market_record_update_time_trigger -BEFORE UPDATE ON nexent.mcp_market_record_t -FOR EACH ROW -EXECUTE FUNCTION update_mcp_market_record_update_time(); - -COMMENT ON TRIGGER update_mcp_market_record_update_time_trigger ON nexent.mcp_market_record_t IS 'Trigger to maintain update_time'; - -ALTER SEQUENCE nexent.mcp_market_record_t_market_id_seq OWNED BY nexent.mcp_market_record_t.market_id; - --- ============================================================================ --- 4) Backfill: migrate old community records to the market table --- Old mcp_community_record_t had no review workflow — published immediately. --- Set their review_status to approved, copy to mcp_market_record_t as 'shared', --- then link mcp_record_t rows to the newly created market records. --- ============================================================================ - --- Mark old community records as approved (they were published without review) -UPDATE nexent.mcp_community_record_t -SET review_status = 'approved' -WHERE (review_status IS NULL OR review_status = 'pending') - AND delete_flag != 'Y'; - --- Backfill: copy old community records into the market table (idempotent) -WITH inserted AS ( - INSERT INTO nexent.mcp_market_record_t ( - tenant_id, user_id, mcp_name, mcp_server, source, - registry_json, transport_type, config_json, tags, description, - download_count, create_time, update_time, created_by, updated_by, delete_flag, - review_status - ) - SELECT - c.tenant_id, c.user_id, c.mcp_name, c.mcp_server, c.source, - c.registry_json, c.transport_type, c.config_json, c.tags, c.description, - 0, c.create_time, c.update_time, c.created_by, c.updated_by, c.delete_flag, - 'shared' - FROM nexent.mcp_community_record_t c - WHERE c.delete_flag != 'Y' - AND c.review_status = 'approved' - AND NOT EXISTS ( - SELECT 1 FROM nexent.mcp_market_record_t m - WHERE m.tenant_id = c.tenant_id - AND m.mcp_name = c.mcp_name - ) - RETURNING market_id, tenant_id, mcp_name -) -UPDATE nexent.mcp_record_t AS mr -SET market_id = ins.market_id -FROM inserted AS ins -WHERE mr.tenant_id = ins.tenant_id - AND mr.mcp_name = ins.mcp_name - AND mr.market_id IS NULL; - -COMMIT; diff --git a/deploy/sql/migrations/v2.3.0_0713_move_owner_manage_to_su.sql b/deploy/sql/migrations/v2.3.0_0713_move_owner_manage_to_su.sql deleted file mode 100644 index 2173d7d0ed..0000000000 --- a/deploy/sql/migrations/v2.3.0_0713_move_owner_manage_to_su.sql +++ /dev/null @@ -1,45 +0,0 @@ --- ============================================================ --- Move /owner-manage left-nav from ASSET_OWNER to SU --- Migration Date: 2026-07-13 --- ============================================================ --- ASSET_OWNER no longer sees the asset-admin resource management page. --- SU gains /owner-manage (id 1003) alongside existing / and /resource-manage. --- ============================================================ - -BEGIN; - --- Remove ASSET_OWNER access to /owner-manage -DELETE FROM nexent.role_permission_t -WHERE role_permission_id = 1505 - OR ( - user_role = 'ASSET_OWNER' - AND permission_category = 'VISIBILITY' - AND permission_type = 'LEFT_NAV_MENU' - AND permission_subtype = '/owner-manage' - ); - --- Grant SU access to /owner-manage (idempotent) -DELETE FROM nexent.role_permission_t -WHERE role_permission_id = 1003 - OR ( - user_role = 'SU' - AND permission_category = 'VISIBILITY' - AND permission_type = 'LEFT_NAV_MENU' - AND permission_subtype = '/owner-manage' - ); - -INSERT INTO nexent.role_permission_t ( - role_permission_id, - user_role, - permission_category, - permission_type, - permission_subtype -) VALUES ( - 1003, - 'SU', - 'VISIBILITY', - 'LEFT_NAV_MENU', - '/owner-manage' -); - -COMMIT; diff --git a/deploy/sql/migrations/v2.3_merged_migrations.sql b/deploy/sql/migrations/v2.3_merged_migrations.sql new file mode 100644 index 0000000000..56edf28507 --- /dev/null +++ b/deploy/sql/migrations/v2.3_merged_migrations.sql @@ -0,0 +1,766 @@ +-- Nexent merged SQL migrations: v2.3 +-- This file is generated from historical migration files. + +-- Source migration: v2.3.0_0624_add_labels_to_ag_tool_info.sql + +-- Add labels column to ag_tool_info_t table for tool filtering/grouping +ALTER TABLE nexent.ag_tool_info_t +ADD COLUMN IF NOT EXISTS labels JSONB DEFAULT '[]'::jsonb; + +COMMENT ON COLUMN nexent.ag_tool_info_t.labels IS 'JSON array of label strings for filtering/grouping tools'; + +-- Seed built-in labels for well-known local tools. +-- These labels serve as suggested defaults and can be modified by users. +-- Keep in sync with: backend/consts/tool_labels.py + +WITH label_map AS ( + SELECT key AS tool_name, value AS label FROM jsonb_each_text('{ + "mysql_database": "database", "postgres_database": "database", "mssql_database": "database", + "read_file": "file", "create_file": "file", "delete_file": "file", + "create_directory": "file", "delete_directory": "file", "list_directory": "file", + "move_item": "file", + "tavily_search": "search", "exa_search": "search", "linkup_search": "search", + "search_memory": "search", "knowledge_base_search": "search", + "dify_search": "knowledge-base", "datamate_search": "knowledge-base", + "idata_search": "knowledge-base", "haotian_search": "knowledge-base", + "ragflow_search": "knowledge-base", + "aidp_search": "knowledge-base", + "analyze_image": "multimodal", "analyze_audio": "multimodal", + "analyze_video": "multimodal", "analyze_text_file": "multimodal", + "get_email": "email", "send_email": "email", + "store_memory": "memory", + "terminal": "terminal" + }'::jsonb) +) +UPDATE nexent.ag_tool_info_t t +SET labels = to_jsonb(ARRAY[m.label]) +FROM label_map m +WHERE t.name = m.tool_name AND t.labels = '[]'::jsonb; + +-- Source migration: v2.3.0_0628_add_agent_evaluation_full.sql + +-- ============================================================================= +-- Agent evaluation (offline) - full bundle +-- ============================================================================= +-- Version: v2.3.0 +-- Date: 2026-06-30 +-- Description: Single-file bundle for the v2.3.0 agent evaluation feature. +-- Combines what were originally three separate migration drafts (0628, 0630 +-- pass_status, 0630 route grant) before any of them had been applied to +-- any environment. Use this file on fresh installs. +-- +-- Idempotency: every DDL statement in this file is safe to run multiple times. +-- - CREATE TABLE IF NOT EXISTS +-- - ALTER TABLE ... ADD COLUMN IF NOT EXISTS +-- - CREATE INDEX IF NOT EXISTS +-- - COMMENT ON (overwrites previous value, no-op if identical) +-- - INSERT ... ON CONFLICT (role_permission_id) DO NOTHING +-- +-- Sections: +-- 1. evaluation_set_t / evaluation_set_case_t / agent_evaluation_t / +-- agent_evaluation_case_t (incl. judge_model_id on agent_evaluation_t) +-- 2. pass_status column on agent_evaluation_case_t + composite index +-- 3. LEFT_NAV_MENU '/space' grant for roles that have /agent-space +-- +-- Design decisions (see PR review 2026-06-30): +-- * No standalone (tenant_id) index on any table. Every case-level and +-- run-level read is scoped by PK or by a foreign key into a parent that +-- itself is already tenant-scoped at the application layer. A bare +-- (tenant_id) index has no real query plan and only inflates write cost. +-- * No (tenant_id, judge_model_id) index. judge_model_id is read alongside +-- the row via PK; "list runs by judge model" is not a supported query. +-- * No (tenant_id, evaluation_set_id) on evaluation_set_case_t. The set +-- itself is tenant-scoped at the app layer, and the existing +-- (evaluation_set_id) index already covers set-case listing. +-- * ix_agent_eval_case_pass_status is (agent_evaluation_id, pass_status) +-- rather than (tenant_id, agent_evaluation_id, pass_status): case-level +-- reads never filter on tenant_id directly, and dropping the leading +-- tenant_id column keeps the most common "list failed cases for run X" +-- query on a single composite index. +-- * Section 3 INSERT must include parent_key (see 0622 menu migration) so +-- a future renderer that joins on parent_key does not leave this batch +-- as orphans. /space is a first-level entry for the route guard only, +-- so parent_key is NULL. +-- ============================================================================= + +SET search_path TO nexent; + +BEGIN; + + +-- ----------------------------------------------------------------------------- +-- Section 1: Evaluation set & evaluation run tables +-- ----------------------------------------------------------------------------- +CREATE TABLE IF NOT EXISTS nexent.evaluation_set_t ( + evaluation_set_id BIGSERIAL PRIMARY KEY, + tenant_id VARCHAR(100) NOT NULL, + + name VARCHAR(255) NOT NULL, + description TEXT, + + source_filename VARCHAR(255), + case_count INTEGER DEFAULT 0, + + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N' +); + +CREATE INDEX IF NOT EXISTS ix_eval_set_name ON nexent.evaluation_set_t(tenant_id, name); + +COMMENT ON TABLE nexent.evaluation_set_t IS 'Offline evaluation sets (JSONL single-turn cases).'; +COMMENT ON COLUMN nexent.evaluation_set_t.tenant_id IS 'Tenant ID for multi-tenancy isolation'; +COMMENT ON COLUMN nexent.evaluation_set_t.source_filename IS 'Original uploaded filename'; +COMMENT ON COLUMN nexent.evaluation_set_t.case_count IS 'Total number of cases'; + + +CREATE TABLE IF NOT EXISTS nexent.evaluation_set_case_t ( + evaluation_set_case_id BIGSERIAL PRIMARY KEY, + tenant_id VARCHAR(100) NOT NULL, + evaluation_set_id BIGINT NOT NULL, + + case_id VARCHAR(128), + inputs JSONB NOT NULL, + label JSONB NOT NULL, + order_no INTEGER DEFAULT 0, + + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N' +); + +CREATE INDEX IF NOT EXISTS ix_eval_set_case_set_id ON nexent.evaluation_set_case_t(evaluation_set_id); + +COMMENT ON TABLE nexent.evaluation_set_case_t IS 'Cases within evaluation sets.'; +COMMENT ON COLUMN nexent.evaluation_set_case_t.inputs IS 'Case inputs JSON: {query: string, context?: string}'; +COMMENT ON COLUMN nexent.evaluation_set_case_t.label IS 'Case label JSON: {answer: string}'; + + +CREATE TABLE IF NOT EXISTS nexent.agent_evaluation_t ( + agent_evaluation_id BIGSERIAL PRIMARY KEY, + tenant_id VARCHAR(100) NOT NULL, + + agent_id INTEGER NOT NULL, + agent_version_no INTEGER NOT NULL, + + evaluation_set_id BIGINT NOT NULL, + + status VARCHAR(30) NOT NULL DEFAULT 'PENDING', + + progress_total INTEGER DEFAULT 0, + progress_done INTEGER DEFAULT 0, + + score_overall DOUBLE PRECISION, + error_message TEXT, + + judge_model_id INTEGER, + + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N' +); + +CREATE INDEX IF NOT EXISTS ix_agent_eval_agent_id ON nexent.agent_evaluation_t(tenant_id, agent_id); +CREATE INDEX IF NOT EXISTS ix_agent_eval_set_id ON nexent.agent_evaluation_t(tenant_id, evaluation_set_id); + +COMMENT ON TABLE nexent.agent_evaluation_t IS 'Offline evaluation runs for an agent.'; +COMMENT ON COLUMN nexent.agent_evaluation_t.status IS 'Run status: PENDING/RUNNING/COMPLETED/FAILED'; +COMMENT ON COLUMN nexent.agent_evaluation_t.judge_model_id IS + 'Model id used by the judge. Persisted so the background worker can recover it after restart and so the frontend can display judge_model_name.'; + + +CREATE TABLE IF NOT EXISTS nexent.agent_evaluation_case_t ( + agent_evaluation_case_id BIGSERIAL PRIMARY KEY, + tenant_id VARCHAR(100) NOT NULL, + + agent_evaluation_id BIGINT NOT NULL, + evaluation_set_case_id BIGINT NOT NULL, + + inputs JSONB NOT NULL, + label JSONB NOT NULL, + predict JSONB, + + score DOUBLE PRECISION, + reason TEXT, + + status VARCHAR(30) NOT NULL DEFAULT 'PENDING', + error_message TEXT, + + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT now(), + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N' +); + +CREATE INDEX IF NOT EXISTS ix_agent_eval_case_eval_id ON nexent.agent_evaluation_case_t(agent_evaluation_id); + +COMMENT ON TABLE nexent.agent_evaluation_case_t IS 'Per-case evaluation results.'; +COMMENT ON COLUMN nexent.agent_evaluation_case_t.predict IS 'Predict JSON: {answer: string, raw?: any}'; +COMMENT ON COLUMN nexent.agent_evaluation_case_t.status IS 'Case status: PENDING/RUNNING/COMPLETED/FAILED'; + + +-- ----------------------------------------------------------------------------- +-- Section 2: pass_status on agent_evaluation_case_t +-- ----------------------------------------------------------------------------- +-- Stores the binary judge result ("pass" / "fail") for each case. +-- Enables fast filtering for failed-case reports and storage optimization: +-- passed cases have predict/reason/label.answer cleared to save space, +-- while only failed cases retain full detail. + +ALTER TABLE nexent.agent_evaluation_case_t +ADD COLUMN IF NOT EXISTS pass_status VARCHAR(16); + +COMMENT ON COLUMN nexent.agent_evaluation_case_t.pass_status IS + 'Judge result per case: pass / fail. pass cases have predict/reason/label.answer cleared to save space.'; + +-- Composite index to support failed-case listing and "only failed" reports. +-- Scoped by (agent_evaluation_id, pass_status) only; tenant_id is enforced +-- at the application layer via the parent run's tenant. +CREATE INDEX IF NOT EXISTS ix_agent_eval_case_pass_status + ON nexent.agent_evaluation_case_t (agent_evaluation_id, pass_status); + + +-- ----------------------------------------------------------------------------- +-- Section 3: Grant /space LEFT_NAV_MENU so the evaluation page is reachable +-- ----------------------------------------------------------------------------- +-- The agent evaluation page lives at the route prefix /space/agents/{id}/evaluate. +-- The previous menu migration (v2.2.2_0622_update_left_nav_menu.sql) removed +-- the legacy /space entry when it refactored the menu structure. As a result +-- the frontend route guard (which uses accessibleRoutes prefix matching) blocks +-- any user from entering the evaluation page with "no access permission". +-- +-- This section adds LEFT_NAV_MENU = '/space' for every role that already has +-- access to the resource-space (i.e. /agent-space). This entry is NOT rendered +-- in the side navigation (SideNavigation uses exact-match against ROUTE_CONFIG) +-- but it IS picked up by the backend as part of accessibleRoutes, so the route +-- guard will allow /space/agents/{id}/evaluate and its sub-paths. + +-- Roles that already have /resource-space (and thus /agent-space) get /space. +-- Mirrors v2.2.2_0622_update_left_nav_menu.sql IDs (16xx range) to keep the +-- scheme consistent. parent_key is NULL: /space is a top-level entry used +-- only by the backend route guard (prefix match on accessibleRoutes) and +-- is not rendered by SideNavigation. +INSERT INTO nexent.role_permission_t + (role_permission_id, user_role, permission_category, permission_type, permission_subtype, parent_key) +VALUES + (1600, 'SU', 'VISIBILITY', 'LEFT_NAV_MENU', '/space', NULL), + (1601, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/space', NULL), + (1602, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/space', NULL), + (1603, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/space', NULL), + (1604, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/space', NULL) +ON CONFLICT (role_permission_id) DO NOTHING; + + +COMMIT; + +-- Source migration: v2.3.0_0629_add_skill_repository_table.sql + +-- Migration: Add ag_skill_repository_t table +-- Date: 2026-06-29 +-- Description: Skill marketplace repository for frozen installable skill snapshots. + +SET search_path TO nexent; + +CREATE SEQUENCE IF NOT EXISTS nexent.ag_skill_repository_t_skill_repository_id_seq; + +CREATE TABLE IF NOT EXISTS nexent.ag_skill_repository_t ( + skill_repository_id BIGINT NOT NULL DEFAULT nextval('nexent.ag_skill_repository_t_skill_repository_id_seq'), + publisher_tenant_id VARCHAR(100) NOT NULL, + publisher_user_id VARCHAR(100) NOT NULL, + skill_id INTEGER NOT NULL, + name VARCHAR(100) NOT NULL, + description TEXT, + source VARCHAR(30), + submitted_by VARCHAR(100), + category_id INTEGER, + tags TEXT[], + icon VARCHAR(100), + downloads INTEGER DEFAULT 0, + skill_info_json JSONB NOT NULL, + skill_zip_base64 TEXT NOT NULL, + status VARCHAR(30) DEFAULT 'not_shared', + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N', + CONSTRAINT ag_skill_repository_t_pkey PRIMARY KEY (skill_repository_id) +); + +ALTER SEQUENCE nexent.ag_skill_repository_t_skill_repository_id_seq + OWNED BY nexent.ag_skill_repository_t.skill_repository_id; + +ALTER TABLE nexent.ag_skill_repository_t OWNER TO root; + +ALTER TABLE nexent.ag_skill_repository_t + ADD COLUMN IF NOT EXISTS submitted_by VARCHAR(100), + ADD COLUMN IF NOT EXISTS icon VARCHAR(100), + ADD COLUMN IF NOT EXISTS downloads INTEGER DEFAULT 0, + ADD COLUMN IF NOT EXISTS skill_zip_base64 TEXT; + +COMMENT ON TABLE nexent.ag_skill_repository_t IS 'Skill marketplace repository for frozen installable skill snapshots'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.skill_repository_id IS 'Skill repository listing ID, unique primary key'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.publisher_tenant_id IS 'Publisher tenant ID'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.publisher_user_id IS 'Publisher user ID'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.skill_id IS 'Source skill ID from ag_skill_info_t; unique when active (delete_flag = N)'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.name IS 'Skill name for display and search'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.description IS 'Skill description'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.source IS 'Skill source'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.submitted_by IS 'Submitter email when listing enters pending_review'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.category_id IS 'Optional marketplace category ID'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.tags IS 'Marketplace tags'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.icon IS 'Marketplace card icon (emoji or URL)'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.downloads IS 'Marketplace install count for card display'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.skill_info_json IS 'Frozen skill metadata snapshot'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.skill_zip_base64 IS 'Frozen skill ZIP payload encoded as base64'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.status IS 'Listing status: not_shared / pending_review / rejected / shared'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.create_time IS 'Creation time'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.update_time IS 'Update time'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.created_by IS 'Creator ID'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.updated_by IS 'Updater ID'; +COMMENT ON COLUMN nexent.ag_skill_repository_t.delete_flag IS 'Soft delete flag: Y/N'; + +CREATE UNIQUE INDEX IF NOT EXISTS uq_skill_repository_skill_active + ON nexent.ag_skill_repository_t (skill_id) + WHERE delete_flag = 'N'; + +CREATE INDEX IF NOT EXISTS idx_skill_repository_publisher_delete + ON nexent.ag_skill_repository_t (publisher_tenant_id, delete_flag); + +CREATE INDEX IF NOT EXISTS idx_skill_repository_status_delete + ON nexent.ag_skill_repository_t (status, delete_flag); + +CREATE INDEX IF NOT EXISTS idx_skill_repository_name_delete + ON nexent.ag_skill_repository_t (name, delete_flag); + +CREATE INDEX IF NOT EXISTS idx_skill_repository_tags_gin + ON nexent.ag_skill_repository_t USING GIN (tags); + +CREATE OR REPLACE FUNCTION update_ag_skill_repository_update_time() +RETURNS TRIGGER AS $$ +BEGIN + NEW.update_time = CURRENT_TIMESTAMP; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +COMMENT ON FUNCTION update_ag_skill_repository_update_time() IS 'Auto-update update_time for ag_skill_repository_t'; + +DROP TRIGGER IF EXISTS update_ag_skill_repository_update_time_trigger ON nexent.ag_skill_repository_t; +CREATE TRIGGER update_ag_skill_repository_update_time_trigger +BEFORE UPDATE ON nexent.ag_skill_repository_t +FOR EACH ROW +EXECUTE FUNCTION update_ag_skill_repository_update_time(); + +COMMENT ON TRIGGER update_ag_skill_repository_update_time_trigger +ON nexent.ag_skill_repository_t IS 'Trigger to maintain update_time'; + +-- Source migration: v2.3.0_0703_history_projection_fields.sql + +-- Migration: Add step_index for ReAct step tracking +-- Date: 2026-07-03 (revised 2026-07-09) +-- Description: Add step_index column to conversation_message_unit_t. +-- Drops previously added run_id, tool_call_id, event_time columns +-- that are no longer needed after review. + +SET search_path TO nexent; +BEGIN; + +-- Add step_index (renamed from step_id) +ALTER TABLE nexent.conversation_message_unit_t + ADD COLUMN IF NOT EXISTS step_index INTEGER DEFAULT NULL; + +COMMENT ON COLUMN nexent.conversation_message_unit_t.step_index IS + 'ReAct step sequence number within this message. Increments on step_count chunks.'; + +-- Drop columns from previous revision (idempotent) +ALTER TABLE nexent.conversation_message_unit_t + DROP COLUMN IF EXISTS run_id; +ALTER TABLE nexent.conversation_message_unit_t + DROP COLUMN IF EXISTS step_id; +ALTER TABLE nexent.conversation_message_unit_t + DROP COLUMN IF EXISTS tool_call_id; +ALTER TABLE nexent.conversation_message_unit_t + DROP COLUMN IF EXISTS event_time; +ALTER TABLE nexent.conversation_message_t + DROP COLUMN IF EXISTS run_id; + +-- Drop obsolete indexes +DROP INDEX IF EXISTS nexent.idx_message_unit_conversation_run; +DROP INDEX IF EXISTS nexent.idx_message_unit_tool_call; + +-- New index for step-based queries +CREATE INDEX IF NOT EXISTS idx_message_unit_message_step + ON nexent.conversation_message_unit_t (message_id, step_index); + +COMMIT; + +-- Source migration: v2.3.0_0709_add_conversation_agent_id.sql + +-- Store the latest agent used by each conversation so history selection can restore agent context. +ALTER TABLE nexent.conversation_record_t + ADD COLUMN IF NOT EXISTS agent_id INTEGER; + +COMMENT ON COLUMN nexent.conversation_record_t.agent_id + IS 'Agent ID used by the latest run in this conversation'; + +-- Source migration: v2.3.0_0709_add_mcp_market_tables.sql + +-- Migration: Add MCP market tables (v2.4.0 single-table design) +-- Date: 2026-07-09 +-- Description: Create mcp_market_record_t (single-table with inline review status), +-- add market_id to mcp_record_t, add review_status/review_type to +-- mcp_community_record_t. + +SET search_path TO nexent; + +BEGIN; + +-- ============================================================================ +-- 1) Extend mcp_record_t for market integration (idempotent) +-- ============================================================================ +ALTER TABLE IF EXISTS nexent.mcp_record_t + ADD COLUMN IF NOT EXISTS market_id INTEGER; + +COMMENT ON COLUMN nexent.mcp_record_t.market_id IS 'Published market record ID (FK to mcp_market_record_t)'; + +-- ============================================================================ +-- 2) Extend mcp_community_record_t for review workflow (idempotent) +-- ============================================================================ +ALTER TABLE IF EXISTS nexent.mcp_community_record_t + ADD COLUMN IF NOT EXISTS review_status VARCHAR(30) DEFAULT 'pending', + ADD COLUMN IF NOT EXISTS review_type VARCHAR(30) DEFAULT 'initial_listing'; + +COMMENT ON COLUMN nexent.mcp_community_record_t.review_status IS 'Review status: pending/approved/rejected/offline'; +COMMENT ON COLUMN nexent.mcp_community_record_t.review_type IS 'Review submission type: initial_listing/update'; + +-- ============================================================================ +-- 3) Create mcp_market_record_t (single-table design) +-- ============================================================================ + +CREATE SEQUENCE IF NOT EXISTS nexent.mcp_market_record_t_market_id_seq; + +CREATE TABLE IF NOT EXISTS nexent.mcp_market_record_t ( + market_id BIGINT NOT NULL DEFAULT nextval('nexent.mcp_market_record_t_market_id_seq'), + tenant_id VARCHAR(100) NOT NULL, + user_id VARCHAR(100) NOT NULL, + mcp_name VARCHAR(100) NOT NULL, + mcp_server VARCHAR(500) NOT NULL, + source VARCHAR(30) DEFAULT 'community', + registry_json JSONB, + transport_type VARCHAR(30), + config_json JSON, + tags TEXT[], + description TEXT, + download_count INTEGER DEFAULT 0, + review_status VARCHAR(30) DEFAULT 'not_shared', + submitted_by VARCHAR(100), + source_mcp_id INTEGER, + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N' +); + +ALTER TABLE nexent.mcp_market_record_t OWNER TO root; + +COMMENT ON TABLE nexent.mcp_market_record_t IS 'MCP market (community) records — single table covering all listing states'; +COMMENT ON COLUMN nexent.mcp_market_record_t.market_id IS 'Market record ID, unique primary key'; +COMMENT ON COLUMN nexent.mcp_market_record_t.tenant_id IS 'Publisher tenant ID'; +COMMENT ON COLUMN nexent.mcp_market_record_t.user_id IS 'Publisher user ID'; +COMMENT ON COLUMN nexent.mcp_market_record_t.mcp_name IS 'MCP name'; +COMMENT ON COLUMN nexent.mcp_market_record_t.mcp_server IS 'MCP server URL'; +COMMENT ON COLUMN nexent.mcp_market_record_t.source IS 'Source type, fixed to community'; +COMMENT ON COLUMN nexent.mcp_market_record_t.registry_json IS 'Full MCP metadata JSON'; +COMMENT ON COLUMN nexent.mcp_market_record_t.transport_type IS 'Transport type: http/sse/container'; +COMMENT ON COLUMN nexent.mcp_market_record_t.config_json IS 'Public-shareable MCP configuration JSON'; +COMMENT ON COLUMN nexent.mcp_market_record_t.tags IS 'Tags'; +COMMENT ON COLUMN nexent.mcp_market_record_t.description IS 'Description'; +COMMENT ON COLUMN nexent.mcp_market_record_t.download_count IS 'Cumulative download/install count'; +COMMENT ON COLUMN nexent.mcp_market_record_t.review_status IS 'Listing status: not_shared/pending_review/shared/rejected'; +COMMENT ON COLUMN nexent.mcp_market_record_t.submitted_by IS 'Email of the user who submitted for review'; +COMMENT ON COLUMN nexent.mcp_market_record_t.source_mcp_id IS 'Source mcp_record_t ID that was published to the market'; + +-- Indexes +CREATE UNIQUE INDEX IF NOT EXISTS uq_mcp_market_name_active + ON nexent.mcp_market_record_t (mcp_name) + WHERE delete_flag = 'N' AND review_status = 'shared'; + +CREATE INDEX IF NOT EXISTS idx_mcp_market_tenant_delete + ON nexent.mcp_market_record_t (tenant_id, delete_flag); +CREATE INDEX IF NOT EXISTS idx_mcp_market_status_delete + ON nexent.mcp_market_record_t (review_status, delete_flag); +CREATE INDEX IF NOT EXISTS idx_mcp_market_tags_gin + ON nexent.mcp_market_record_t USING GIN (tags); + +-- Trigger: auto-update update_time +CREATE OR REPLACE FUNCTION update_mcp_market_record_update_time() +RETURNS TRIGGER AS $$ +BEGIN + NEW.update_time = CURRENT_TIMESTAMP; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +COMMENT ON FUNCTION update_mcp_market_record_update_time() IS 'Auto-update update_time for mcp_market_record_t'; + +DROP TRIGGER IF EXISTS update_mcp_market_record_update_time_trigger ON nexent.mcp_market_record_t; +CREATE TRIGGER update_mcp_market_record_update_time_trigger +BEFORE UPDATE ON nexent.mcp_market_record_t +FOR EACH ROW +EXECUTE FUNCTION update_mcp_market_record_update_time(); + +COMMENT ON TRIGGER update_mcp_market_record_update_time_trigger ON nexent.mcp_market_record_t IS 'Trigger to maintain update_time'; + +ALTER SEQUENCE nexent.mcp_market_record_t_market_id_seq OWNED BY nexent.mcp_market_record_t.market_id; + +-- ============================================================================ +-- 4) Backfill: migrate old community records to the market table +-- Old mcp_community_record_t had no review workflow — published immediately. +-- Set their review_status to approved, copy to mcp_market_record_t as 'shared', +-- then link mcp_record_t rows to the newly created market records. +-- ============================================================================ + +-- Mark old community records as approved (they were published without review) +UPDATE nexent.mcp_community_record_t +SET review_status = 'approved' +WHERE (review_status IS NULL OR review_status = 'pending') + AND delete_flag != 'Y'; + +-- Backfill: copy old community records into the market table (idempotent) +WITH inserted AS ( + INSERT INTO nexent.mcp_market_record_t ( + tenant_id, user_id, mcp_name, mcp_server, source, + registry_json, transport_type, config_json, tags, description, + download_count, create_time, update_time, created_by, updated_by, delete_flag, + review_status + ) + SELECT + c.tenant_id, c.user_id, c.mcp_name, c.mcp_server, c.source, + c.registry_json, c.transport_type, c.config_json, c.tags, c.description, + 0, c.create_time, c.update_time, c.created_by, c.updated_by, c.delete_flag, + 'shared' + FROM nexent.mcp_community_record_t c + WHERE c.delete_flag != 'Y' + AND c.review_status = 'approved' + AND NOT EXISTS ( + SELECT 1 FROM nexent.mcp_market_record_t m + WHERE m.tenant_id = c.tenant_id + AND m.mcp_name = c.mcp_name + ) + RETURNING market_id, tenant_id, mcp_name +) +UPDATE nexent.mcp_record_t AS mr +SET market_id = ins.market_id +FROM inserted AS ins +WHERE mr.tenant_id = ins.tenant_id + AND mr.mcp_name = ins.mcp_name + AND mr.market_id IS NULL; + +COMMIT; + +-- Source migration: v2.3.0_0713_move_owner_manage_to_su.sql + +-- ============================================================ +-- Move /owner-manage left-nav from ASSET_OWNER to SU +-- Migration Date: 2026-07-13 +-- ============================================================ +-- ASSET_OWNER no longer sees the asset-admin resource management page. +-- SU gains /owner-manage (id 1003) alongside existing / and /resource-manage. +-- ============================================================ + +BEGIN; + +-- Remove ASSET_OWNER access to /owner-manage +DELETE FROM nexent.role_permission_t +WHERE role_permission_id = 1505 + OR ( + user_role = 'ASSET_OWNER' + AND permission_category = 'VISIBILITY' + AND permission_type = 'LEFT_NAV_MENU' + AND permission_subtype = '/owner-manage' + ); + +-- Grant SU access to /owner-manage (idempotent) +DELETE FROM nexent.role_permission_t +WHERE role_permission_id = 1003 + OR ( + user_role = 'SU' + AND permission_category = 'VISIBILITY' + AND permission_type = 'LEFT_NAV_MENU' + AND permission_subtype = '/owner-manage' + ); + +INSERT INTO nexent.role_permission_t ( + role_permission_id, + user_role, + permission_category, + permission_type, + permission_subtype +) VALUES ( + 1003, + 'SU', + 'VISIBILITY', + 'LEFT_NAV_MENU', + '/owner-manage' +); + +COMMIT; + +-- Source migration: v2.3.0_0718_add_agent_context_policy.sql + +-- Add the agent-level context processing mode override. +ALTER TABLE nexent.ag_tenant_agent_t +ADD COLUMN IF NOT EXISTS context_policy JSONB; + +COMMENT ON COLUMN nexent.ag_tenant_agent_t.context_policy IS +'Agent-level context processing override (passthrough/adaptive_compact); NULL preserves the platform default'; + +-- Source migration: v2.3.1_0717_add_notification_tables.sql + +-- Migration: Add notification_t and notification_receiver_t tables +-- Date: 2026-07-17 +-- Description: In-app notification message table plus per-user fan-out delivery/read table. + +SET search_path TO nexent; + +-- notification_t: one row per message +CREATE SEQUENCE IF NOT EXISTS nexent.notification_t_notification_id_seq; + +CREATE TABLE IF NOT EXISTS nexent.notification_t ( + notification_id BIGINT NOT NULL DEFAULT nextval('nexent.notification_t_notification_id_seq'), + event_type VARCHAR(50) NOT NULL, + resource_type VARCHAR(50) NOT NULL, + unique_id BIGINT, + details JSONB, + scope VARCHAR(20) NOT NULL, + tenant_id VARCHAR(100), + is_active BOOLEAN NOT NULL DEFAULT TRUE, + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N', + CONSTRAINT notification_t_pkey PRIMARY KEY (notification_id) +); + +ALTER SEQUENCE nexent.notification_t_notification_id_seq + OWNED BY nexent.notification_t.notification_id; +ALTER TABLE nexent.notification_t OWNER TO root; + +COMMENT ON TABLE nexent.notification_t IS 'In-app notification message table; per-user delivery lives in notification_receiver_t'; +COMMENT ON COLUMN nexent.notification_t.notification_id IS 'Notification ID, unique primary key'; +COMMENT ON COLUMN nexent.notification_t.event_type IS 'Event type, e.g. repository_review_approved / repository_review_rejected'; +COMMENT ON COLUMN nexent.notification_t.resource_type IS 'Resource type, e.g. agent_repository / skill_repository / mcp_repository'; +COMMENT ON COLUMN nexent.notification_t.unique_id IS 'Related resource primary key (e.g. agent_repository_id)'; +COMMENT ON COLUMN nexent.notification_t.details IS 'i18n interpolation details for the event template'; +COMMENT ON COLUMN nexent.notification_t.scope IS 'Audience scope: SU / TENANT / TENANT_ADMIN / TENANT_USER / USER'; +COMMENT ON COLUMN nexent.notification_t.tenant_id IS 'Target tenant; NULL for SU scope'; +COMMENT ON COLUMN nexent.notification_t.is_active IS 'Whether this notification is still active/valid'; +COMMENT ON COLUMN nexent.notification_t.create_time IS 'Creation time'; +COMMENT ON COLUMN nexent.notification_t.update_time IS 'Update time'; +COMMENT ON COLUMN nexent.notification_t.created_by IS 'Creator ID'; +COMMENT ON COLUMN nexent.notification_t.updated_by IS 'Updater ID'; +COMMENT ON COLUMN nexent.notification_t.delete_flag IS 'Soft delete flag: Y/N'; + +CREATE INDEX IF NOT EXISTS ix_notification_event_resource_unique_active + ON nexent.notification_t (event_type, resource_type, unique_id, is_active); + +-- notification_receiver_t: one row per receiver (fan-out) +CREATE SEQUENCE IF NOT EXISTS nexent.notification_receiver_t_receiver_id_seq; + +CREATE TABLE IF NOT EXISTS nexent.notification_receiver_t ( + receiver_id BIGINT NOT NULL DEFAULT nextval('nexent.notification_receiver_t_receiver_id_seq'), + notification_id BIGINT NOT NULL, + receiver_user_id VARCHAR(100) NOT NULL, + tenant_id VARCHAR(100), + is_read BOOLEAN DEFAULT FALSE, + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N', + CONSTRAINT notification_receiver_t_pkey PRIMARY KEY (receiver_id) +); + +ALTER SEQUENCE nexent.notification_receiver_t_receiver_id_seq + OWNED BY nexent.notification_receiver_t.receiver_id; +ALTER TABLE nexent.notification_receiver_t OWNER TO root; + +COMMENT ON TABLE nexent.notification_receiver_t IS 'Per-user notification delivery and read status (fan-out from notification_t)'; +COMMENT ON COLUMN nexent.notification_receiver_t.receiver_id IS 'Receiver row ID, unique primary key'; +COMMENT ON COLUMN nexent.notification_receiver_t.notification_id IS 'FK to notification_t.notification_id'; +COMMENT ON COLUMN nexent.notification_receiver_t.receiver_user_id IS 'Receiver user ID'; +COMMENT ON COLUMN nexent.notification_receiver_t.tenant_id IS 'Tenant ID for multi-tenancy isolation'; +COMMENT ON COLUMN nexent.notification_receiver_t.is_read IS 'Whether this receiver has read the notification'; +COMMENT ON COLUMN nexent.notification_receiver_t.create_time IS 'Creation time'; +COMMENT ON COLUMN nexent.notification_receiver_t.update_time IS 'Update time'; +COMMENT ON COLUMN nexent.notification_receiver_t.created_by IS 'Creator ID'; +COMMENT ON COLUMN nexent.notification_receiver_t.updated_by IS 'Updater ID'; +COMMENT ON COLUMN nexent.notification_receiver_t.delete_flag IS 'Soft delete flag: Y/N'; + +CREATE INDEX IF NOT EXISTS ix_notification_receiver_user_read + ON nexent.notification_receiver_t (receiver_user_id, is_read); +CREATE INDEX IF NOT EXISTS ix_notification_receiver_notification_id + ON nexent.notification_receiver_t (notification_id); + +-- update_time triggers +CREATE OR REPLACE FUNCTION update_notification_update_time() +RETURNS TRIGGER AS $$ +BEGIN + NEW.update_time = CURRENT_TIMESTAMP; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS update_notification_update_time_trigger ON nexent.notification_t; +CREATE TRIGGER update_notification_update_time_trigger +BEFORE UPDATE ON nexent.notification_t +FOR EACH ROW +EXECUTE FUNCTION update_notification_update_time(); + +CREATE OR REPLACE FUNCTION update_notification_receiver_update_time() +RETURNS TRIGGER AS $$ +BEGIN + NEW.update_time = CURRENT_TIMESTAMP; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS update_notification_receiver_update_time_trigger ON nexent.notification_receiver_t; +CREATE TRIGGER update_notification_receiver_update_time_trigger +BEFORE UPDATE ON nexent.notification_receiver_t +FOR EACH ROW +EXECUTE FUNCTION update_notification_receiver_update_time(); + +ALTER TABLE nexent.ag_agent_repository_t + ADD COLUMN IF NOT EXISTS content TEXT; + +COMMENT ON COLUMN nexent.ag_agent_repository_t.content IS + 'Listing note on submit or review opinion on approve/reject'; + +ALTER TABLE nexent.ag_agent_repository_t + ADD COLUMN IF NOT EXISTS content TEXT; + +COMMENT ON COLUMN nexent.ag_agent_repository_t.content IS + 'Listing note on submit or review opinion on approve/reject'; + +ALTER TABLE nexent.ag_skill_repository_t + ADD COLUMN IF NOT EXISTS content TEXT; + +COMMENT ON COLUMN nexent.ag_skill_repository_t.content IS + 'Listing note on submit or review opinion on approve/reject'; + +ALTER TABLE nexent.mcp_market_record_t + ADD COLUMN IF NOT EXISTS content TEXT; + +COMMENT ON COLUMN nexent.mcp_market_record_t.content IS + 'Listing note on submit or review opinion on approve/reject'; diff --git a/deploy/sql/migrations/v2.4_merged_migrations.sql b/deploy/sql/migrations/v2.4_merged_migrations.sql new file mode 100644 index 0000000000..297b8b8c9b --- /dev/null +++ b/deploy/sql/migrations/v2.4_merged_migrations.sql @@ -0,0 +1,820 @@ +-- Nexent merged SQL migrations: v2.4 +-- This file is generated from historical migration files. + +-- Source migration: v2.4.0_0710_add_kb_quota_column.sql + +-- Add quota_limit_bytes column to knowledge_record_t for per-KB soft storage quota +-- NULL = unlimited (shares tenant pool freely) + +ALTER TABLE nexent.knowledge_record_t ADD COLUMN IF NOT EXISTS quota_limit_bytes BIGINT; + +-- Source migration: v2.4.0_0713_memory_records_phase2.sql + +-- ============================================================================ +-- Phase 2 Memory Architecture: memory_records_t / memory_retrieval_hits_t +-- ============================================================================ +-- Authoritative memory store (tenant/user/agent) and per-hit retrieval log. +-- Primary keys use PostgreSQL `SERIAL4` shorthand (implicit sequence + +-- NOT NULL + PRIMARY KEY); isolation columns remain varchar for cross-table +-- consistency with `memory_user_config_t`. + +CREATE TABLE IF NOT EXISTS nexent.memory_records_t ( + memory_id SERIAL4 PRIMARY KEY, + tenant_id varchar(100), + user_id varchar(100), + agent_id varchar(100), + conversation_id varchar(100), + layer varchar(30) NOT NULL, + memory_type varchar(30), + status varchar(30) NOT NULL DEFAULT 'active', + content text NOT NULL, + concept_tags text[], + es_index_name varchar(255), + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + created_by varchar(100), + updated_by varchar(100), + delete_flag varchar(1) NOT NULL DEFAULT 'N', + idempotency_key varchar(128) NOT NULL, + recall_count int4 NOT NULL DEFAULT 0, + daily_count int4 NOT NULL DEFAULT 0, + grounded_count int4 NOT NULL DEFAULT 0, + last_recalled_at timestamp, + query_hashes text[], + recall_days text[], + light_hits int4 NOT NULL DEFAULT 0, + rem_hits int4 NOT NULL DEFAULT 0, + last_light_at timestamp, + last_rem_at timestamp +); +ALTER TABLE nexent.memory_records_t OWNER TO "root"; + +COMMENT ON COLUMN nexent.memory_records_t.memory_id IS 'Auto-incremented memory primary key (serial4).'; +COMMENT ON COLUMN nexent.memory_records_t.tenant_id IS 'Tenant ID (isolation key).'; +COMMENT ON COLUMN nexent.memory_records_t.user_id IS 'User ID (isolation key for user/agent layers).'; +COMMENT ON COLUMN nexent.memory_records_t.agent_id IS 'Agent ID (isolation key for agent short-term layer).'; +COMMENT ON COLUMN nexent.memory_records_t.conversation_id IS 'Conversation ID (further isolation key for agent).'; +COMMENT ON COLUMN nexent.memory_records_t.layer IS 'Memory layer: tenant | user | agent.'; +COMMENT ON COLUMN nexent.memory_records_t.memory_type IS 'Memory type: long_term | short_term.'; +COMMENT ON COLUMN nexent.memory_records_t.status IS 'Status: active | archived | disabled.'; +COMMENT ON COLUMN nexent.memory_records_t.content IS 'Memory content.'; +COMMENT ON COLUMN nexent.memory_records_t.concept_tags IS 'Optional concept tags from Dreaming REM phase.'; +COMMENT ON COLUMN nexent.memory_records_t.es_index_name IS 'Elasticsearch index for agent short-term memory (mem__); null for PG-only layers.'; +COMMENT ON COLUMN nexent.memory_records_t.create_time IS 'Creation time, audit field.'; +COMMENT ON COLUMN nexent.memory_records_t.update_time IS 'Update time, audit field.'; +COMMENT ON COLUMN nexent.memory_records_t.created_by IS 'Creator ID, audit field.'; +COMMENT ON COLUMN nexent.memory_records_t.updated_by IS 'Last updater ID, audit field.'; +COMMENT ON COLUMN nexent.memory_records_t.delete_flag IS 'Soft delete flag (Y/N).'; +COMMENT ON COLUMN nexent.memory_records_t.idempotency_key IS 'Idempotency key for write deduplication.'; +COMMENT ON COLUMN nexent.memory_records_t.recall_count IS 'Total recall hit count.'; +COMMENT ON COLUMN nexent.memory_records_t.daily_count IS 'Recall hit count for the most recent active day.'; +COMMENT ON COLUMN nexent.memory_records_t.grounded_count IS 'Count of grounded (verified) recalls.'; +COMMENT ON COLUMN nexent.memory_records_t.last_recalled_at IS 'Most recent recall timestamp.'; +COMMENT ON COLUMN nexent.memory_records_t.query_hashes IS 'Hashes of queries that recalled this memory.'; +COMMENT ON COLUMN nexent.memory_records_t.recall_days IS 'ISO date strings of recall days.'; +COMMENT ON COLUMN nexent.memory_records_t.light_hits IS 'Light Sleep phase hit count.'; +COMMENT ON COLUMN nexent.memory_records_t.rem_hits IS 'REM Sleep phase hit count.'; +COMMENT ON COLUMN nexent.memory_records_t.last_light_at IS 'Last Light Sleep timestamp.'; +COMMENT ON COLUMN nexent.memory_records_t.last_rem_at IS 'Last REM Sleep timestamp.'; +COMMENT ON TABLE nexent.memory_records_t IS 'Authoritative store for tenant/user/agent memory (Phase 2).'; + +CREATE INDEX IF NOT EXISTS idx_memory_records_tenant + ON nexent.memory_records_t (tenant_id); +CREATE INDEX IF NOT EXISTS idx_memory_records_user + ON nexent.memory_records_t (tenant_id, user_id); +CREATE INDEX IF NOT EXISTS idx_memory_records_agent + ON nexent.memory_records_t (tenant_id, user_id, agent_id, conversation_id); +CREATE INDEX IF NOT EXISTS idx_memory_records_idempotency + ON nexent.memory_records_t (tenant_id, idempotency_key); +CREATE INDEX IF NOT EXISTS idx_memory_records_status + ON nexent.memory_records_t (tenant_id, user_id, layer, status); + +CREATE TABLE IF NOT EXISTS nexent.memory_retrieval_hits_t ( + hit_id SERIAL4 PRIMARY KEY, + tenant_id varchar(100), + user_id varchar(100), + agent_id varchar(100), + conversation_id varchar(100), + memory_id int4, + query_text text, + query_hash varchar(128), + retrieval_score numeric(38, 18), + source varchar(100) NOT NULL DEFAULT 'nexent', + occurred_at timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, + day varchar(100), + grounded boolean NOT NULL DEFAULT false, + create_time timestamp DEFAULT CURRENT_TIMESTAMP, + update_time timestamp DEFAULT CURRENT_TIMESTAMP, + created_by varchar(100), + updated_by varchar(100), + delete_flag varchar(1) NOT NULL DEFAULT 'N' +); +ALTER TABLE nexent.memory_retrieval_hits_t OWNER TO "root"; + +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.hit_id IS 'Hit primary key (serial4).'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.tenant_id IS 'Tenant ID.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.user_id IS 'User ID.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.agent_id IS 'Agent ID.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.conversation_id IS 'Conversation ID.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.memory_id IS 'Recalled memory id (null on miss rows).'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.query_text IS 'Original search query text.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.query_hash IS 'Stable hash of the query text.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.retrieval_score IS 'Similarity score reported by the backend.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.source IS 'Hit origin: nexent | external_provider.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.occurred_at IS 'Time the hit was recorded.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.day IS 'ISO date string (occurred_at::date).'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.grounded IS 'Whether the hit was verified/grounded.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.create_time IS 'Row creation time.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.update_time IS 'Row last update time.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.created_by IS 'User that created the row.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.updated_by IS 'User that last updated the row.'; +COMMENT ON COLUMN nexent.memory_retrieval_hits_t.delete_flag IS 'Soft delete flag (N = active, Y = deleted).'; +COMMENT ON TABLE nexent.memory_retrieval_hits_t IS 'Per-hit memory retrieval log; consumed by Dreaming scheduler.'; + +CREATE INDEX IF NOT EXISTS idx_memory_retrieval_hits_memory + ON nexent.memory_retrieval_hits_t (memory_id, occurred_at); +CREATE INDEX IF NOT EXISTS idx_memory_retrieval_hits_tenant_user_agent + ON nexent.memory_retrieval_hits_t (tenant_id, user_id, agent_id, day); + +CREATE OR REPLACE FUNCTION nexent.update_memory_records_update_time() +RETURNS TRIGGER AS $$ +BEGIN + NEW.update_time = CURRENT_TIMESTAMP; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS update_memory_records_update_time_trigger ON nexent.memory_records_t; +CREATE TRIGGER update_memory_records_update_time_trigger +BEFORE UPDATE ON nexent.memory_records_t +FOR EACH ROW +EXECUTE FUNCTION nexent.update_memory_records_update_time(); + +COMMENT ON TRIGGER update_memory_records_update_time_trigger ON nexent.memory_records_t IS 'Trigger to call update_memory_records_update_time function before each update on memory_records_t table'; + +-- Trigger to keep memory_retrieval_hits_t.update_time fresh on UPDATE. +CREATE OR REPLACE FUNCTION nexent.update_memory_retrieval_hits_update_time() +RETURNS TRIGGER AS $$ +BEGIN + NEW.update_time = CURRENT_TIMESTAMP; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +DROP TRIGGER IF EXISTS update_memory_retrieval_hits_update_time_trigger ON nexent.memory_retrieval_hits_t; +CREATE TRIGGER update_memory_retrieval_hits_update_time_trigger +BEFORE UPDATE ON nexent.memory_retrieval_hits_t +FOR EACH ROW +EXECUTE FUNCTION nexent.update_memory_retrieval_hits_update_time(); + +COMMENT ON TRIGGER update_memory_retrieval_hits_update_time_trigger ON nexent.memory_retrieval_hits_t IS 'Trigger to call update_memory_retrieval_hits_update_time function before each update on memory_retrieval_hits_t table'; + +-- Source migration: v2.4.0_0716_add_mcp_permissions_and_sharing.sql + +-- Migration: Add permission and sharing fields to MCP tables +-- Date: 2026-07-16 +-- Description: +-- - mcp_record_t: add group_ids, ingroup_permission, shared_fields +-- - mcp_market_record_t: add group_ids, ingroup_permission, shared_fields +-- Both target the same commit to keep the migration atomic. + +SET search_path TO nexent; + +BEGIN; + +-- ------------------------------------------------------------------------- +-- mcp_market_record_t — group-based access control +-- ------------------------------------------------------------------------- +ALTER TABLE nexent.mcp_market_record_t + ADD COLUMN IF NOT EXISTS group_ids VARCHAR, + ADD COLUMN IF NOT EXISTS ingroup_permission VARCHAR(30) DEFAULT 'READ_ONLY'; + +COMMENT ON COLUMN nexent.mcp_market_record_t.group_ids IS + 'Comma-separated group IDs that can access this MCP'; +COMMENT ON COLUMN nexent.mcp_market_record_t.ingroup_permission IS + 'In-group permission: EDIT, READ_ONLY, PRIVATE'; + +-- ------------------------------------------------------------------------- +-- mcp_market_record_t — shared-fields snapshot at submission time +-- ------------------------------------------------------------------------- +ALTER TABLE nexent.mcp_market_record_t + ADD COLUMN IF NOT EXISTS shared_fields JSON; + +COMMENT ON COLUMN nexent.mcp_market_record_t.shared_fields IS + 'Snapshot of shared_fields at submission time'; + +-- ------------------------------------------------------------------------- +-- mcp_record_t — group-based access control +-- ------------------------------------------------------------------------- +ALTER TABLE nexent.mcp_record_t + ADD COLUMN IF NOT EXISTS group_ids VARCHAR, + ADD COLUMN IF NOT EXISTS ingroup_permission VARCHAR(30) DEFAULT 'READ_ONLY'; + +COMMENT ON COLUMN nexent.mcp_record_t.group_ids IS + 'Comma-separated group IDs that can access this MCP'; +COMMENT ON COLUMN nexent.mcp_record_t.ingroup_permission IS + 'In-group permission: EDIT, READ_ONLY, PRIVATE'; + +-- ------------------------------------------------------------------------- +-- mcp_record_t — field-level sharing flags +-- ------------------------------------------------------------------------- +ALTER TABLE nexent.mcp_record_t + ADD COLUMN IF NOT EXISTS shared_fields JSON; + +COMMENT ON COLUMN nexent.mcp_record_t.shared_fields IS + 'JSON object of field-level sharing flags (e.g. {"serverUrl": true, "authorizationToken": false})'; + +-- ------------------------------------------------------------------------- +-- Grant EDIT permission to existing public MCPs +-- Existing MCPs with NULL group_ids have no group restrictions +-- and should be editable by all tenant users. +-- ------------------------------------------------------------------------- +UPDATE nexent.mcp_record_t +SET ingroup_permission = 'EDIT' +WHERE group_ids IS NULL + AND delete_flag != 'Y'; + +-- ------------------------------------------------------------------------- +-- Fix mcp_market_record_t unique index: use (tenant_id, mcp_name) instead +-- of (mcp_name) to prevent cross-tenant name conflicts. +-- ------------------------------------------------------------------------- +DROP INDEX IF EXISTS nexent.uq_mcp_market_name_active; +CREATE UNIQUE INDEX IF NOT EXISTS uq_mcp_market_name_active + ON nexent.mcp_market_record_t (tenant_id, mcp_name) + WHERE delete_flag = 'N' AND review_status = 'shared'; + +COMMIT; + +-- Source migration: v2.4.0_0720_add_agent_is_main_agent.sql + +-- Add a main-agent flag to tenant agents. +ALTER TABLE nexent.ag_tenant_agent_t + ADD COLUMN IF NOT EXISTS is_main_agent BOOLEAN NOT NULL DEFAULT TRUE; + +COMMENT ON COLUMN nexent.ag_tenant_agent_t.is_main_agent + IS 'Whether this agent is a main agent'; + +-- Source migration: v2.4.0_0721_add_newchat_left_nav_permissions.sql + +BEGIN; +INSERT INTO nexent.role_permission_t ( + role_permission_id, + user_role, + permission_category, + permission_type, + permission_subtype, + parent_key +) +VALUES + (1114, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/newchat', NULL), + (1213, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/newchat', NULL), + (1305, 'USER', 'VISIBILITY', 'LEFT_NAV_MENU', '/newchat', NULL), + (1413, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/newchat', NULL), + (1512, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/newchat', NULL) +ON CONFLICT (role_permission_id) DO NOTHING; +COMMIT; + +-- Source migration: v2.4.0_0722_add_agent_automation.sql + +-- Add durable scheduled agent tasks, run history, chat proposals, and navigation permissions. + +SET search_path TO nexent; + +BEGIN; + +CREATE TABLE IF NOT EXISTS nexent.agent_automation_task_t ( + task_id BIGSERIAL PRIMARY KEY NOT NULL, + tenant_id VARCHAR(100) NOT NULL, + user_id VARCHAR(100) NOT NULL, + conversation_id BIGINT NOT NULL, + agent_id BIGINT NOT NULL, + agent_version_no INTEGER, + title VARCHAR(255) NOT NULL, + instruction TEXT NOT NULL, + status VARCHAR(32) NOT NULL, + source VARCHAR(32) NOT NULL, + schedule_mode VARCHAR(16) NOT NULL, + schedule_rule_type VARCHAR(16) NOT NULL, + schedule_expr TEXT, + schedule_config JSONB NOT NULL, + capability_requirements JSONB, + capability_bindings JSONB, + runtime_snapshot JSONB, + timezone VARCHAR(64) NOT NULL, + next_fire_at TIMESTAMPTZ, + last_fire_at TIMESTAMPTZ, + fire_count INTEGER NOT NULL DEFAULT 0, + last_run_status VARCHAR(32), + last_error TEXT, + consecutive_failures INTEGER NOT NULL DEFAULT 0, + timeout_seconds INTEGER NOT NULL, + overlap_policy VARCHAR(16) NOT NULL, + misfire_policy VARCHAR(16) NOT NULL, + lock_owner VARCHAR(128), + lock_until TIMESTAMPTZ, + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N' +); + +CREATE TABLE IF NOT EXISTS nexent.agent_automation_run_t ( + run_id BIGSERIAL PRIMARY KEY NOT NULL, + task_id BIGINT NOT NULL, + tenant_id VARCHAR(100) NOT NULL, + user_id VARCHAR(100) NOT NULL, + conversation_id BIGINT NOT NULL, + scheduled_fire_at TIMESTAMPTZ NOT NULL, + actual_fire_at TIMESTAMPTZ, + trigger_type VARCHAR(32) NOT NULL, + status VARCHAR(32) NOT NULL, + generated_prompt TEXT, + user_message_id BIGINT, + assistant_message_id BIGINT, + started_at TIMESTAMPTZ, + finished_at TIMESTAMPTZ, + duration_ms BIGINT, + error_code VARCHAR(64), + error_message TEXT, + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N' +); + +CREATE TABLE IF NOT EXISTS nexent.agent_automation_proposal_t ( + proposal_id BIGSERIAL PRIMARY KEY NOT NULL, + tenant_id VARCHAR(100) NOT NULL, + user_id VARCHAR(100) NOT NULL, + conversation_id BIGINT NOT NULL, + agent_id BIGINT NOT NULL, + proposed_task JSONB NOT NULL, + capability_resolution JSONB NOT NULL, + status VARCHAR(32) NOT NULL, + expires_at TIMESTAMPTZ NOT NULL, + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) DEFAULT 'N' +); + +CREATE INDEX IF NOT EXISTS idx_agent_automation_due + ON nexent.agent_automation_task_t (status, next_fire_at) + WHERE delete_flag = 'N'; + +CREATE INDEX IF NOT EXISTS idx_agent_automation_owner + ON nexent.agent_automation_task_t (tenant_id, user_id, status) + WHERE delete_flag = 'N'; + +CREATE UNIQUE INDEX IF NOT EXISTS uq_agent_automation_conversation_active + ON nexent.agent_automation_task_t (conversation_id) + WHERE delete_flag = 'N' AND status <> 'DELETED'; + +CREATE INDEX IF NOT EXISTS idx_agent_automation_run_task + ON nexent.agent_automation_run_t (task_id, scheduled_fire_at) + WHERE delete_flag = 'N'; + +CREATE INDEX IF NOT EXISTS idx_agent_automation_run_conversation + ON nexent.agent_automation_run_t (conversation_id, status) + WHERE delete_flag = 'N'; + +CREATE UNIQUE INDEX IF NOT EXISTS uq_agent_automation_active_occurrence + ON nexent.agent_automation_run_t (task_id, scheduled_fire_at) + WHERE delete_flag = 'N' + AND trigger_type = 'SCHEDULED' + AND status IN ('QUEUED', 'RUNNING'); + +ALTER TABLE nexent.agent_automation_run_t + DROP COLUMN IF EXISTS capability_check; + +CREATE INDEX IF NOT EXISTS idx_agent_automation_proposal_owner + ON nexent.agent_automation_proposal_t (tenant_id, user_id, status) + WHERE delete_flag = 'N'; + +DELETE FROM nexent.role_permission_t +WHERE role_permission_id BETWEEN 1512 AND 1517; + +-- Keep each permission in the ID range assigned to its role. +INSERT INTO nexent.role_permission_t ( + role_permission_id, + user_role, + permission_category, + permission_type, + permission_subtype, + parent_key +) +VALUES + (1115, 'ADMIN', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-tasks', NULL), + (1214, 'DEV', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-tasks', NULL), + (1306, 'USER', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-tasks', NULL), + (1414, 'SPEED', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-tasks', NULL), + (1513, 'ASSET_OWNER', 'VISIBILITY', 'LEFT_NAV_MENU', '/agent-tasks', NULL) +ON CONFLICT (role_permission_id) DO NOTHING; + +COMMIT; + +-- Source migration: v2.4.0_0722_add_skill_permission_and_repository_snapshots.sql + +-- Migration: Add tenant-scoped skill uniqueness and group permissions; allow repository snapshots by status +-- Date: 2026-07-22 +-- Description: Align skill ownership and repository status behavior with agent repository semantics. + +SET search_path TO nexent; + +ALTER TABLE IF EXISTS nexent.ag_skill_info_t + DROP CONSTRAINT IF EXISTS ag_skill_info_t_skill_name_key; + +DO $$ +BEGIN + IF EXISTS ( + SELECT 1 + FROM nexent.ag_skill_info_t + WHERE tenant_id IS NOT NULL + AND delete_flag = 'N' + GROUP BY tenant_id, skill_name + HAVING COUNT(*) > 1 + ) THEN + RAISE EXCEPTION + 'Cannot enforce tenant-scoped Skill names: duplicate active (tenant_id, skill_name) rows exist'; + END IF; + + IF EXISTS ( + SELECT 1 + FROM nexent.ag_skill_info_t + WHERE tenant_id IS NULL + AND delete_flag = 'N' + GROUP BY skill_name + HAVING COUNT(*) > 1 + ) THEN + RAISE EXCEPTION + 'Cannot enforce global template Skill names: duplicate active skill_name rows exist'; + END IF; +END +$$; + +CREATE UNIQUE INDEX IF NOT EXISTS uq_skill_info_tenant_name_active + ON nexent.ag_skill_info_t (tenant_id, skill_name) + WHERE tenant_id IS NOT NULL AND delete_flag = 'N'; + +CREATE UNIQUE INDEX IF NOT EXISTS uq_skill_info_global_name_active + ON nexent.ag_skill_info_t (skill_name) + WHERE tenant_id IS NULL AND delete_flag = 'N'; + +COMMENT ON COLUMN nexent.ag_skill_info_t.skill_name IS + 'Skill name, unique among active skills within its tenant scope'; + +ALTER TABLE IF EXISTS nexent.ag_skill_info_t + ADD COLUMN IF NOT EXISTS group_ids VARCHAR, + ADD COLUMN IF NOT EXISTS ingroup_permission VARCHAR(30); + +COMMENT ON COLUMN nexent.ag_skill_info_t.group_ids IS 'Skill group IDs list'; +COMMENT ON COLUMN nexent.ag_skill_info_t.ingroup_permission IS 'In-group permission: EDIT, READ_ONLY, PRIVATE'; + +WITH tenant_groups AS ( + SELECT + tenant_id, + string_agg(group_id::text, ',' ORDER BY group_id) AS group_ids + FROM nexent.tenant_group_info_t + WHERE delete_flag = 'N' + GROUP BY tenant_id +) +UPDATE nexent.ag_skill_info_t skill +SET group_ids = tenant_groups.group_ids +FROM tenant_groups +WHERE skill.tenant_id = tenant_groups.tenant_id + AND skill.delete_flag = 'N' + AND skill.tenant_id IS NOT NULL + AND (skill.group_ids IS NULL OR skill.group_ids = ''); + +UPDATE nexent.ag_skill_info_t +SET ingroup_permission = 'EDIT' +WHERE delete_flag = 'N' + AND tenant_id IS NOT NULL + AND (ingroup_permission IS NULL OR ingroup_permission = ''); + +DROP INDEX IF EXISTS nexent.uq_skill_repository_skill_active; +DROP INDEX IF EXISTS nexent.uq_skill_repository_skill_shared_active; +DROP INDEX IF EXISTS nexent.uq_skill_repository_skill_pending_active; + +CREATE INDEX IF NOT EXISTS idx_skill_repository_skill_status_delete + ON nexent.ag_skill_repository_t (publisher_tenant_id, skill_id, status, delete_flag); + +COMMENT ON COLUMN nexent.ag_skill_repository_t.skill_id IS + 'Source skill ID from ag_skill_info_t; multiple active snapshots may exist across statuses'; + +-- Source migration: v2.4.0_0723_add_aidp_kb_permission.sql + +-- ============================================================ +-- Add aidp_kb_permission_t table for AIDP knowledge base permissions +-- Migration Date: 2026-07-23 +-- Description: +-- P0 data layer for the AIDP permission redesign (v7.1). +-- - Stores one record per KB that has been claimed into Nexent. +-- - UNIQUE(kb_id) WHERE delete_flag='N' prevents concurrent active duplicates. +-- - group_ids uses JSONB for type safety and indexable intersection queries. +-- - resource_status tracks lifecycle so the API can surface UNKNOWN/ORPHANED +-- KBs without silently hiding them. +-- - kds_name caches the AIDP display name so the LLM tool can resolve +-- human-readable names to kds_ids without an extra AIDP round-trip. +-- Idempotent: every DDL uses IF NOT EXISTS so re-running this migration is safe. +-- ============================================================ + +BEGIN; + +CREATE TABLE IF NOT EXISTS nexent.aidp_kb_permission_t ( + id BIGSERIAL PRIMARY KEY, + kb_id VARCHAR(128) NOT NULL, + kds_name VARCHAR(128), + owner_user_id VARCHAR(100) NOT NULL, + tenant_id VARCHAR(100) NOT NULL, + ingroup_permission VARCHAR(30) NOT NULL DEFAULT 'READ_ONLY', + group_ids JSONB NOT NULL DEFAULT '[]'::jsonb, + resource_status VARCHAR(30) NOT NULL DEFAULT 'ACTIVE', + create_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + update_time TIMESTAMP WITHOUT TIME ZONE DEFAULT CURRENT_TIMESTAMP, + created_by VARCHAR(100), + updated_by VARCHAR(100), + delete_flag VARCHAR(1) NOT NULL DEFAULT 'N' +); + +-- Active-record uniqueness: only one live row per kbs_id. +-- After a soft delete the constraint releases the kb_id, allowing re-creation. +CREATE UNIQUE INDEX IF NOT EXISTS uq_aidp_kb_permission_active_kb + ON nexent.aidp_kb_permission_t (kb_id) + WHERE delete_flag = 'N'; + +-- Tenant and ownership lookup indexes; partial on active rows only. +CREATE INDEX IF NOT EXISTS idx_aidp_perm_tenant + ON nexent.aidp_kb_permission_t (tenant_id) + WHERE delete_flag = 'N'; + +CREATE INDEX IF NOT EXISTS idx_aidp_perm_user + ON nexent.aidp_kb_permission_t (owner_user_id, tenant_id) + WHERE delete_flag = 'N'; + +CREATE INDEX IF NOT EXISTS idx_aidp_perm_kb + ON nexent.aidp_kb_permission_t (kb_id) + WHERE delete_flag = 'N'; + +-- JSONB GIN index supports `group_ids @> '[1,2]'::jsonb` intersection queries +-- that the permission service uses to determine KB accessibility. +CREATE INDEX IF NOT EXISTS idx_aidp_perm_group_ids_gin + ON nexent.aidp_kb_permission_t USING GIN (group_ids) + WHERE delete_flag = 'N'; + +COMMENT ON TABLE nexent.aidp_kb_permission_t IS + 'AIDP knowledge base permission records. Each row represents a KB under Nexent management.'; +COMMENT ON COLUMN nexent.aidp_kb_permission_t.kb_id IS + 'kds_id returned by AIDP, globally unique within AIDP system (AIDP guarantees this).'; +COMMENT ON COLUMN nexent.aidp_kb_permission_t.owner_user_id IS + 'Nexent user_id of the KB creator (Nexent account that called the AIDP create API).'; +COMMENT ON COLUMN nexent.aidp_kb_permission_t.tenant_id IS + 'Nexent tenant_id; combined with delete_flag this is the only valid query key for multi-tenant isolation.'; +COMMENT ON COLUMN nexent.aidp_kb_permission_t.ingroup_permission IS + 'Permission level for authorized groups: EDIT / READ_ONLY / PRIVATE.'; +COMMENT ON COLUMN nexent.aidp_kb_permission_t.group_ids IS + 'JSON array of Nexent group IDs authorized to access this KB. Empty array means no group access.'; +COMMENT ON COLUMN nexent.aidp_kb_permission_t.resource_status IS + 'Resource lifecycle status: CREATING / ACTIVE / DELETE_PENDING / ORPHANED / UNAVAILABLE.'; +COMMENT ON COLUMN nexent.aidp_kb_permission_t.delete_flag IS + 'Y / N. Active rows are N. Soft delete flips this to Y so the active uniqueness constraint releases the kb_id.'; + +-- Migration-safe column add: covers tables created before kds_name was introduced. +-- Changing this file's checksum causes the runner to re-execute; IF NOT EXISTS makes it safe. +ALTER TABLE nexent.aidp_kb_permission_t ADD COLUMN IF NOT EXISTS kds_name VARCHAR(128); + +COMMENT ON COLUMN nexent.aidp_kb_permission_t.kds_name IS + 'AIDP knowledge base display name (kds_name), cached at creation time so the LLM tool can resolve human-readable names to kds_ids without an AIDP round-trip.'; + +COMMIT; + +-- Source migration: v2.4.0_0723_add_conversation_chat_mode_and unit_tool_call.sql + +-- Persist the UI chat mode (planning vs. execution) for each conversation so +-- switching threads can restore the toggle without re-inferring it from units. + +ALTER TABLE nexent.conversation_record_t + ADD COLUMN IF NOT EXISTS chat_mode varchar(16) NOT NULL DEFAULT 'execution'; + +COMMENT ON COLUMN nexent.conversation_record_t.chat_mode IS + 'UI chat mode of the conversation. Allowed values: planning, execution.'; + +SET search_path TO nexent; + +ALTER TABLE nexent.conversation_message_unit_t + ADD COLUMN IF NOT EXISTS tool_call_id VARCHAR(36); + +COMMENT ON COLUMN nexent.conversation_message_unit_t.tool_call_id IS + 'Unique ID of the originating tool invocation, used to attribute side-channel units to the correct tool call during parallel execution.'; + +-- Source migration: v2.4.0_0723_cleanup_aidp_tool_credentials.sql + +-- ============================================================ +-- Strip legacy AIDP credentials from tool instance params. +-- Migration Date: 2026-07-23 +-- Description: +-- Earlier versions of the aidp_search tool accepted ``server_url`` and +-- ``api_key`` via the per-instance params (sometimes stored in plain +-- text in browser localStorage). The v7.1 permission redesign makes +-- Nexent the sole owner of those credentials, sourced from the +-- AIDP_SERVER_URL / AIDP_API_KEY environment variables. +-- +-- This migration removes any persisted ``server_url`` / ``api_key`` +-- entries from ``ag_tool_instance_t.params`` for ``aidp_search`` tool +-- instances so historical rows do not leak the old value. +-- +-- Idempotent: rewrites params only when at least one of the keys is +-- present; safe to re-run. +-- ============================================================ + +BEGIN; + +UPDATE nexent.ag_tool_instance_t instance +SET params = ( + REPLACE( + REPLACE(instance.params::text, '"server_url"', '"_removed_server_url"'), + '"api_key"', '"_removed_api_key"' + )::jsonb - '_removed_server_url' - '_removed_api_key' +)::text::json +FROM nexent.ag_tool_info_t tool +WHERE instance.tool_id = tool.tool_id + AND tool.name = 'aidp_search' + AND instance.delete_flag = 'N' + AND instance.params IS NOT NULL + AND ( + instance.params::text LIKE '%server_url%' + OR instance.params::text LIKE '%api_key%' + ); + +-- Validation query (manual): +-- SELECT COUNT(*) FROM nexent.ag_tool_instance_t instance +-- JOIN nexent.ag_tool_info_t tool ON instance.tool_id = tool.tool_id +-- WHERE tool.name = 'aidp_search' +-- AND instance.delete_flag = 'N' +-- AND instance.params::text LIKE '%server_url%'; + +COMMIT; + +-- Source migration: v2.4.0_0725_rename_mem_agent_permission_to_mem_tenant.sql + +-- Migration: Rename tenant-memory permissions from MEM.AGENT to MEM.TENANT +-- Date: 2026-07-25 +-- Description: Align permission names with the tenant memory layer they govern. + +SET search_path TO nexent; + +UPDATE nexent.role_permission_t +SET permission_type = 'MEM.TENANT' +WHERE permission_type = 'MEM.AGENT'; + +-- Source migration: v2.4.0_0727_add_a2a_agent_card_headers_and_security_fields.sql + +ALTER TABLE nexent.ag_a2a_external_agent_t + ADD COLUMN IF NOT EXISTS agent_card_headers JSONB; + +COMMENT ON COLUMN nexent.ag_a2a_external_agent_t.agent_card_headers + IS 'Headers saved only for Agent Card discovery and refresh'; + +ALTER TABLE nexent.ag_a2a_external_agent_t + ADD COLUMN IF NOT EXISTS security_schemes JSONB, + ADD COLUMN IF NOT EXISTS security_requirements JSONB, + ADD COLUMN IF NOT EXISTS security_credentials JSONB; + +COMMENT ON COLUMN nexent.ag_a2a_external_agent_t.security_schemes + IS 'Security schemes declared by the Agent Card'; +COMMENT ON COLUMN nexent.ag_a2a_external_agent_t.security_requirements + IS 'Security requirements declared by the Agent Card'; +COMMENT ON COLUMN nexent.ag_a2a_external_agent_t.security_credentials + IS 'Credential values for Agent Card security schemes, never exposed by APIs'; + +ALTER TABLE nexent.ag_a2a_external_agent_t + ADD COLUMN IF NOT EXISTS selected_security_requirement_index INTEGER; + +COMMENT ON COLUMN nexent.ag_a2a_external_agent_t.selected_security_requirement_index + IS 'Selected Agent Card security requirement index used for external agent calls'; + +-- Source migration: v2.4.0_0803_backfill_official_skill_tool_relations.sql + +-- Backfill tool dependencies for official skills installed before allowed-tools +-- metadata was added to the bundled skill archives. +SET search_path TO nexent; + +WITH skill_tool_mapping(skill_name, tool_name) AS ( + VALUES + ('analyze-image', 'analyze_image'), + ('analyze-text-file', 'analyze_text_file'), + ('create-file-directory', 'create_file'), + ('create-file-directory', 'create_directory'), + ('delete-file-directory', 'delete_file'), + ('delete-file-directory', 'delete_directory'), + ('email-utils', 'get_email'), + ('email-utils', 'send_email'), + ('list-directory', 'list_directory'), + ('move-file-directory', 'move_item'), + ('read-file', 'read_file'), + ('run-shell-ssh', 'terminal'), + ('search-datamate', 'datamate_search'), + ('search-dify', 'dify_search'), + ('search-idata', 'idata_search'), + ('search-knowledge-base', 'knowledge_base_search'), + ('search-web-exa', 'exa_search'), + ('search-web-linkup', 'linkup_search'), + ('search-web-tavily', 'tavily_search') +), +updated_relations AS ( + UPDATE nexent.ag_skill_tools_rel_t AS relation + SET + created_by = COALESCE( + relation.created_by, + skill.created_by, + skill.updated_by, + tool.created_by, + tool.updated_by + ), + updated_by = COALESCE( + relation.updated_by, + skill.updated_by, + skill.created_by, + tool.updated_by, + tool.created_by + ), + update_time = CURRENT_TIMESTAMP + FROM skill_tool_mapping mapping + JOIN nexent.ag_skill_info_t skill + ON skill.skill_name = mapping.skill_name + AND skill.delete_flag != 'Y' + AND skill.source IN ('official', '官方') + JOIN nexent.ag_tool_info_t tool + ON tool.name = mapping.tool_name + AND tool.delete_flag != 'Y' + AND tool.author = skill.tenant_id + WHERE relation.skill_id = skill.skill_id + AND relation.tool_id = tool.tool_id + AND relation.delete_flag != 'Y' + AND (relation.created_by IS NULL OR relation.updated_by IS NULL) + RETURNING relation.skill_id, relation.tool_id +) +INSERT INTO nexent.ag_skill_tools_rel_t ( + skill_id, + tool_id, + created_by, + updated_by, + delete_flag +) +SELECT + skill.skill_id, + tool.tool_id, + COALESCE(skill.created_by, skill.updated_by, tool.created_by, tool.updated_by), + COALESCE(skill.updated_by, skill.created_by, tool.updated_by, tool.created_by), + 'N' +FROM skill_tool_mapping mapping +JOIN nexent.ag_skill_info_t skill + ON skill.skill_name = mapping.skill_name + AND skill.delete_flag != 'Y' + AND skill.source IN ('official', '官方') +JOIN nexent.ag_tool_info_t tool + ON tool.name = mapping.tool_name + AND tool.delete_flag != 'Y' + AND tool.author = skill.tenant_id +WHERE NOT EXISTS ( + SELECT 1 + FROM nexent.ag_skill_tools_rel_t relation + WHERE relation.skill_id = skill.skill_id + AND relation.tool_id = tool.tool_id + AND relation.delete_flag != 'Y' +); + +-- Source migration: v2.4.0_0804_add_agent_automation_tool_idempotency.sql + +-- Add idempotent source-message linkage for AgentLoop-created automation proposals. + +SET search_path TO nexent; + +BEGIN; + +ALTER TABLE nexent.agent_automation_proposal_t + ADD COLUMN IF NOT EXISTS source_message_id BIGINT; + +CREATE UNIQUE INDEX IF NOT EXISTS uq_agent_automation_proposal_source_message + ON nexent.agent_automation_proposal_t (tenant_id, user_id, source_message_id) + WHERE delete_flag = 'N' AND source_message_id IS NOT NULL; + +COMMIT; + +-- Source migration: v2.5.0_0801_add_unit_invocation_id.sql + +-- Persist `invocation_id` on message units so the frontend can attribute +-- model deep-thinking output to the correct sub-agent card on history replay. + +SET search_path TO nexent; + +ALTER TABLE nexent.conversation_message_unit_t + ADD COLUMN IF NOT EXISTS invocation_id VARCHAR(36); + +COMMENT ON COLUMN nexent.conversation_message_unit_t.invocation_id IS + 'Identifies which sub-agent invocation produced this unit. Used by the ' + 'frontend history adapter to route deep-thinking / reasoning chunks into ' + 'the correct nested sub-agent card.'; diff --git a/deploy/tests/test_build_offline_package.sh b/deploy/tests/test_build_offline_package.sh index e4673b58d9..f3e45b4c55 100755 --- a/deploy/tests/test_build_offline_package.sh +++ b/deploy/tests/test_build_offline_package.sh @@ -111,6 +111,23 @@ echo "$WORKFLOW_CONTENT" | grep -q 'package-name=nexent-${VERSION}-${PLATFORM}${ echo "$WORKFLOW_CONTENT" | grep -q -- '--compress false' || fail "offline package workflow should let GitHub create the final artifact zip" echo "$WORKFLOW_CONTENT" | grep -q 'path: ./offline-output' || fail "offline package workflow should upload package contents, not an inner zip" ! echo "$WORKFLOW_CONTENT" | grep -q 'path: .*package-name.*\\.zip' || fail "offline package workflow should not upload a pre-compressed zip" +echo "$WORKFLOW_CONTENT" | grep -q 'COMPONENTS="infrastructure,application,data-process,supabase,terminal"' || fail "offline package workflow should select all packageable components" + +OFFLINE_HELP="$(DEPLOYMENT_LANG=en bash "$PROJECT_ROOT/deploy/offline/build_offline_package.sh" --help)" +echo "$OFFLINE_HELP" | grep -q -- '--include-sandbox BOOL' || fail "offline package help should document --include-sandbox" + +SANDBOX_DRY_RUN="$(DEPLOYMENT_LANG=en bash "$PROJECT_ROOT/deploy/offline/build_offline_package.sh" --version v2.2.0 --platform amd64 --components infrastructure,application --image-source general --target docker --dry-run)" +echo "$SANDBOX_DRY_RUN" | grep -q 'Include Sandbox image: true' || fail "offline dry-run should show that the Sandbox image is enabled by default" +echo "$SANDBOX_DRY_RUN" | grep -q 'nexent/nexent-sandbox:v2.2.0' || fail "offline packages should include the Sandbox image by default" + +NO_SANDBOX_DRY_RUN="$(DEPLOYMENT_LANG=en bash "$PROJECT_ROOT/deploy/offline/build_offline_package.sh" --version v2.2.0 --platform amd64 --components infrastructure,application --image-source general --target docker --include-sandbox false --dry-run)" +echo "$NO_SANDBOX_DRY_RUN" | grep -q 'Include Sandbox image: false' || fail "offline dry-run should show that the Sandbox image is disabled explicitly" +! echo "$NO_SANDBOX_DRY_RUN" | grep -q 'nexent/nexent-sandbox:v2.2.0' || fail "--include-sandbox false should exclude the Sandbox image" + +if DEPLOYMENT_LANG=en bash "$PROJECT_ROOT/deploy/offline/build_offline_package.sh" --include-sandbox invalid --dry-run >"$TMP_DIR/invalid-include-sandbox.log" 2>&1; then + fail "--include-sandbox should accept only true or false" +fi +grep -q "Include sandbox must be 'true' or 'false'" "$TMP_DIR/invalid-include-sandbox.log" || fail "invalid --include-sandbox error should be explicit" for target in docker k8s all; do package_dir="$OUT_DIR/$target" @@ -129,6 +146,9 @@ for target in docker k8s all; do [ -f "$OUT_DIR/nexent-offline-${target}-amd64-v2.2.0.zip" ] || fail "zip package should be created for target $target" grep -q "target: \"$target\"" "$package_dir/manifest.yaml" || fail "manifest should record target $target" grep -q "nexent/nexent:v2.2.0" "$package_dir/manifest.yaml" || fail "manifest should include Nexent image" + grep -q 'includeSandbox: "true"' "$package_dir/manifest.yaml" || fail "manifest should record that the Sandbox image is included by default" + grep -q "nexent/nexent-sandbox:v2.2.0" "$package_dir/manifest.yaml" || fail "manifest should include the Sandbox image by default" + [ -f "$package_dir/images/nexent-sandbox-v2-2-0.tar" ] || fail "offline package should save the Sandbox image tar by default" case "$target" in docker) @@ -146,60 +166,119 @@ for target in docker k8s all; do esac done +sandbox_package_dir="$OUT_DIR/without-sandbox" +PATH="$BIN_DIR:$PATH" \ + bash "$PROJECT_ROOT/deploy/offline/build_offline_package.sh" \ + --version v2.2.0 \ + --platform amd64 \ + --components infrastructure,application \ + --image-source general \ + --target docker \ + --include-sandbox false \ + --output-dir "$sandbox_package_dir" >"$TMP_DIR/without-sandbox.log" + +assert_common_package_files "$sandbox_package_dir" +grep -q 'includeSandbox: "false"' "$sandbox_package_dir/manifest.yaml" || fail "manifest should record that the Sandbox image is excluded" +! grep -q 'nexent-sandbox' "$sandbox_package_dir/manifest.yaml" || fail "--include-sandbox false should exclude the Sandbox image" +[ ! -f "$sandbox_package_dir/images/nexent-sandbox-v2-2-0.tar" ] || fail "--include-sandbox false should not save the Sandbox image tar" + deploy_wrapper_dir="$OUT_DIR/deploy-wrapper" -mkdir -p "$deploy_wrapper_dir/deploy" +mkdir -p "$deploy_wrapper_dir/deploy/common" "$deploy_wrapper_dir/deploy/env" cp "$PROJECT_ROOT/deploy.sh" "$deploy_wrapper_dir/deploy.sh" +cp "$PROJECT_ROOT/deploy/common/common.sh" "$deploy_wrapper_dir/deploy/common/common.sh" +printf 'WRAPPER_OLD_VALUE=preserved\n' > "$deploy_wrapper_dir/deploy/env/.env" +printf 'WRAPPER_NEW_DEFAULT=merged-before-actions\n' > "$deploy_wrapper_dir/deploy/env/.env.example" cat > "$deploy_wrapper_dir/load-images.sh" <<'SH' #!/usr/bin/env bash +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [ -n "${DEPLOY_WRAPPER_EXPECT_ENV:-}" ]; then + grep -Fqx "$DEPLOY_WRAPPER_EXPECT_ENV" "$script_dir/deploy/env/.env" || exit 1 +fi printf 'load-images\n' >> "$DEPLOY_WRAPPER_LOG" SH chmod +x "$deploy_wrapper_dir/load-images.sh" cat > "$deploy_wrapper_dir/push-images.sh" <<'SH' #!/usr/bin/env bash +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [ -n "${DEPLOY_WRAPPER_EXPECT_ENV:-}" ]; then + grep -Fqx "$DEPLOY_WRAPPER_EXPECT_ENV" "$script_dir/deploy/env/.env" || exit 1 +fi args=("$@") printf 'push:%s:%s\n' "${REGISTRY_PASSWORD:-}" "${args[*]}" >> "$DEPLOY_WRAPPER_LOG" SH chmod +x "$deploy_wrapper_dir/push-images.sh" cat > "$deploy_wrapper_dir/deploy/deploy.sh" <<'SH' #!/usr/bin/env bash -printf 'deploy:%s:%s\n' "${NEXENT_DEPLOY_CONFIG_MODE:-}" "$*" >> "$DEPLOY_WRAPPER_LOG" +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [ -n "${DEPLOY_WRAPPER_EXPECT_ENV:-}" ]; then + grep -Fqx "$DEPLOY_WRAPPER_EXPECT_ENV" "$script_dir/env/.env" || exit 1 +fi +printf 'deploy:%s:%s:%s\n' "${NEXENT_DEPLOY_CONFIG_MODE:-}" "${NEXENT_DEPLOYMENT_OFFLINE:-}" "$*" >> "$DEPLOY_WRAPPER_LOG" SH chmod +x "$deploy_wrapper_dir/deploy/deploy.sh" deploy_wrapper_log="$TMP_DIR/deploy-wrapper.log" -DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" bash "$deploy_wrapper_dir/deploy.sh" docker --foo bar +DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" \ + DEPLOY_WRAPPER_EXPECT_ENV='WRAPPER_NEW_DEFAULT=merged-before-actions' \ + bash "$deploy_wrapper_dir/deploy.sh" docker --foo bar if grep -q '^load-images$' "$deploy_wrapper_log"; then fail "deploy.sh should not load images by default" fi -grep -q '^deploy::docker --foo bar$' "$deploy_wrapper_log" || fail "deploy.sh should forward args without --load-images" +grep -q '^deploy::false:docker --foo bar$' "$deploy_wrapper_log" || fail "deploy.sh should forward args and mark an online deployment" +grep -q '^WRAPPER_OLD_VALUE=preserved$' "$deploy_wrapper_dir/deploy/env/.env" || fail "online deployment should preserve existing environment values" +grep -q '^WRAPPER_NEW_DEFAULT=merged-before-actions$' "$deploy_wrapper_dir/deploy/env/.env" || fail "online deployment should merge current template variables" : > "$deploy_wrapper_log" -DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" bash "$deploy_wrapper_dir/deploy.sh" --load-images docker --foo bar +DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" \ + DEPLOY_WRAPPER_EXPECT_ENV='WRAPPER_NEW_DEFAULT=merged-before-actions' \ + bash "$deploy_wrapper_dir/deploy.sh" --load-images docker --foo bar first_line="$(sed -n '1p' "$deploy_wrapper_log")" second_line="$(sed -n '2p' "$deploy_wrapper_log")" [ "$first_line" = "load-images" ] || fail "deploy.sh --load-images should load images before deploy" -[ "$second_line" = "deploy::docker --foo bar" ] || fail "deploy.sh --load-images should strip only the wrapper flag" +[ "$second_line" = "deploy::false:docker --foo bar" ] || fail "deploy.sh --load-images should strip only the wrapper flag" + +mv "$deploy_wrapper_dir/deploy/env/.env.example" "$deploy_wrapper_dir/deploy/env/.env.example.saved" +: > "$deploy_wrapper_log" +if DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" bash "$deploy_wrapper_dir/deploy.sh" --load-images docker --foo bar >"$TMP_DIR/wrapper-missing-template.log" 2>&1; then + mv "$deploy_wrapper_dir/deploy/env/.env.example.saved" "$deploy_wrapper_dir/deploy/env/.env.example" + fail "deploy.sh should require the current .env.example before loading images" +fi +mv "$deploy_wrapper_dir/deploy/env/.env.example.saved" "$deploy_wrapper_dir/deploy/env/.env.example" +[ ! -s "$deploy_wrapper_log" ] || fail "missing .env.example should fail before image loading or deployment" +grep -q 'deploy/env/.env.example' "$TMP_DIR/wrapper-missing-template.log" || fail "missing template failure should identify deploy/env/.env.example" : > "$deploy_wrapper_log" DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" bash "$deploy_wrapper_dir/deploy.sh" --defaults docker --foo bar -grep -q '^deploy:defaults:docker --foo bar$' "$deploy_wrapper_log" || fail "deploy.sh --defaults before target should enable defaults mode" +grep -q '^deploy:defaults:false:docker --foo bar$' "$deploy_wrapper_log" || fail "deploy.sh --defaults before target should enable defaults mode" : > "$deploy_wrapper_log" DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" bash "$deploy_wrapper_dir/deploy.sh" docker --defaults --foo bar -grep -q '^deploy:defaults:docker --foo bar$' "$deploy_wrapper_log" || fail "deploy.sh --defaults after target should enable defaults mode and consume the flag" +grep -q '^deploy:defaults:false:docker --foo bar$' "$deploy_wrapper_log" || fail "deploy.sh --defaults after target should enable defaults mode and consume the flag" + +: > "$deploy_wrapper_log" +DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" bash "$deploy_wrapper_dir/deploy.sh" docker --config --foo bar +grep -q '^deploy:tui:false:docker --foo bar$' "$deploy_wrapper_log" || fail "online deploy.sh --config should enable TUI mode without the offline marker" + +online_reuse_source="$TMP_DIR/online-reuse-source" +mkdir -p "$online_reuse_source/deploy/env" +printf 'ONLINE_REUSE_TEST=yes\n' > "$online_reuse_source/deploy/env/.env" +if DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" bash "$deploy_wrapper_dir/deploy.sh" --reuse-from "$online_reuse_source" docker --foo bar >"$TMP_DIR/online-reuse.log" 2>&1; then + fail "online deploy.sh should reject --reuse-from" +fi +grep -q 'offline package entrypoint' "$TMP_DIR/online-reuse.log" || fail "online --reuse-from error should explain that the option is offline-only" : > "$deploy_wrapper_log" DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" REGISTRY_USERNAME=user REGISTRY_PASSWORD=secret bash "$deploy_wrapper_dir/deploy.sh" --push-images --image-registry-prefix registry.local/nexent docker --foo bar first_line="$(sed -n '1p' "$deploy_wrapper_log")" second_line="$(sed -n '2p' "$deploy_wrapper_log")" [[ "$first_line" == "push:secret:--image-registry-prefix registry.local/nexent --load-images" ]] || fail "deploy.sh --push-images should delegate push args to push-images.sh" -[ "$second_line" = "deploy::docker --foo bar --image-registry-prefix registry.local/nexent" ] || fail "deploy.sh --push-images should forward image registry prefix to deploy config" +[ "$second_line" = "deploy::false:docker --foo bar --image-registry-prefix registry.local/nexent" ] || fail "deploy.sh --push-images should forward image registry prefix to deploy config" : > "$deploy_wrapper_log" DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" REGISTRY_USERNAME=user REGISTRY_PASSWORD=secret bash "$deploy_wrapper_dir/deploy.sh" --load-images --push-images --image-registry-prefix registry.local/nexent docker --foo bar first_line="$(sed -n '1p' "$deploy_wrapper_log")" second_line="$(sed -n '2p' "$deploy_wrapper_log")" [[ "$first_line" == "push:secret:--image-registry-prefix registry.local/nexent --load-images" ]] || fail "deploy.sh --load-images --push-images should not load before push login" -[ "$second_line" = "deploy::docker --foo bar --image-registry-prefix registry.local/nexent" ] || fail "deploy.sh --load-images --push-images should forward deploy args" +[ "$second_line" = "deploy::false:docker --foo bar --image-registry-prefix registry.local/nexent" ] || fail "deploy.sh --load-images --push-images should forward deploy args" if DEPLOY_WRAPPER_LOG="$deploy_wrapper_log" REGISTRY_USERNAME=user REGISTRY_PASSWORD=secret bash "$deploy_wrapper_dir/deploy.sh" --push-images docker --foo bar >/tmp/nexent-deploy-wrapper-missing-prefix.log 2>&1; then fail "deploy.sh --push-images should require image registry prefix in non-interactive mode" @@ -230,12 +309,14 @@ assert_common_package_files "$latest_package_dir" grep -q '^DEPLOY_WRAPPER_DEFAULT_CONFIG_MODE="defaults"$' "$latest_package_dir/deploy.sh" || fail "offline deploy.sh should reuse the root entrypoint with defaults mode enabled" offline_help="$(DEPLOYMENT_LANG=en bash "$latest_package_dir/deploy.sh" --help)" echo "$offline_help" | grep -q "deploys with saved configuration or built-in defaults" || fail "offline deploy help should explain default non-interactive mode" +echo "$offline_help" | grep -q -- '--reuse-from DIR' || fail "offline deploy help should document --reuse-from" +printf '\nOFFLINE_TEMPLATE_ONLY=merged-before-actions\n' >> "$latest_package_dir/deploy/env/.env.example" push_log="$TMP_DIR/push-images.log" : > "$push_log" PATH="$BIN_DIR:$PATH" \ FAKE_DOCKER_LOG="$push_log" \ - FAKE_DOCKER_LOCAL_IMAGES="nexent/nexent:latest,nexent/nexent-web:latest,nexent/nexent-mcp:latest,docker.elastic.co/elasticsearch/elasticsearch:8.17.4,postgres:15-alpine,redis:alpine,quay.io/minio/minio:RELEASE.2023-12-20T01-00-02Z" \ + FAKE_DOCKER_LOCAL_IMAGES="nexent/nexent:latest,nexent/nexent-web:latest,nexent/nexent-mcp:latest,nexent/nexent-sandbox:latest,docker.elastic.co/elasticsearch/elasticsearch:8.17.4,postgres:15-alpine,redis:alpine,quay.io/minio/minio:RELEASE.2023-12-20T01-00-02Z" \ REGISTRY_PASSWORD=secret \ bash "$latest_package_dir/push-images.sh" \ --image-registry-prefix https://registry.local/nexent/ \ @@ -249,7 +330,7 @@ grep -q '^tag docker.elastic.co/elasticsearch/elasticsearch:8.17.4 registry.loca : > "$push_log" PATH="$BIN_DIR:$PATH" \ FAKE_DOCKER_LOG="$push_log" \ - FAKE_DOCKER_LOCAL_IMAGES="nexent/nexent:latest,nexent/nexent-web:latest,nexent/nexent-mcp:latest,docker.elastic.co/elasticsearch/elasticsearch:8.17.4,postgres:15-alpine,redis:alpine,quay.io/minio/minio:RELEASE.2023-12-20T01-00-02Z" \ + FAKE_DOCKER_LOCAL_IMAGES="nexent/nexent:latest,nexent/nexent-web:latest,nexent/nexent-mcp:latest,nexent/nexent-sandbox:latest,docker.elastic.co/elasticsearch/elasticsearch:8.17.4,postgres:15-alpine,redis:alpine,quay.io/minio/minio:RELEASE.2023-12-20T01-00-02Z" \ REGISTRY_PASSWORD=secret \ bash "$latest_package_dir/push-images.sh" \ --load-images \ @@ -266,47 +347,165 @@ grep -q -- '--registry-username is required' /tmp/nexent-offline-package-push-mi cat > "$latest_package_dir/load-images.sh" <<'SH' #!/usr/bin/env bash +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [ -n "${DEPLOY_WRAPPER_EXPECT_ENV:-}" ]; then + grep -Fqx "$DEPLOY_WRAPPER_EXPECT_ENV" "$script_dir/deploy/env/.env" || exit 1 +fi +if [ -n "${DEPLOY_WRAPPER_EXPECT_MERGED_ENV:-}" ]; then + grep -Fqx "$DEPLOY_WRAPPER_EXPECT_MERGED_ENV" "$script_dir/deploy/env/.env" || exit 1 +fi printf 'load-images\n' >> "$DEPLOY_WRAPPER_LOG" SH chmod +x "$latest_package_dir/load-images.sh" cat > "$latest_package_dir/push-images.sh" <<'SH' #!/usr/bin/env bash +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [ -n "${DEPLOY_WRAPPER_EXPECT_ENV:-}" ]; then + grep -Fqx "$DEPLOY_WRAPPER_EXPECT_ENV" "$script_dir/deploy/env/.env" || exit 1 +fi +if [ -n "${DEPLOY_WRAPPER_EXPECT_MERGED_ENV:-}" ]; then + grep -Fqx "$DEPLOY_WRAPPER_EXPECT_MERGED_ENV" "$script_dir/deploy/env/.env" || exit 1 +fi args=("$@") printf 'push:%s:%s\n' "${REGISTRY_PASSWORD:-}" "${args[*]}" >> "$DEPLOY_WRAPPER_LOG" SH chmod +x "$latest_package_dir/push-images.sh" cat > "$latest_package_dir/deploy/deploy.sh" <<'SH' #!/usr/bin/env bash -printf 'deploy:%s:%s\n' "${NEXENT_DEPLOY_CONFIG_MODE:-}" "$*" >> "$DEPLOY_WRAPPER_LOG" +script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +if [ -n "${DEPLOY_WRAPPER_EXPECT_ENV:-}" ]; then + grep -Fqx "$DEPLOY_WRAPPER_EXPECT_ENV" "$script_dir/env/.env" || exit 1 +fi +if [ -n "${DEPLOY_WRAPPER_EXPECT_MERGED_ENV:-}" ]; then + grep -Fqx "$DEPLOY_WRAPPER_EXPECT_MERGED_ENV" "$script_dir/env/.env" || exit 1 +fi +printf 'deploy:%s:%s:%s\n' "${NEXENT_DEPLOY_CONFIG_MODE:-}" "${NEXENT_DEPLOYMENT_OFFLINE:-}" "$*" >> "$DEPLOY_WRAPPER_LOG" SH chmod +x "$latest_package_dir/deploy/deploy.sh" offline_deploy_log="$TMP_DIR/offline-deploy-wrapper.log" +: > "$offline_deploy_log" + +missing_template_reuse_source="$TMP_DIR/missing-template-reuse-source" +mkdir -p "$missing_template_reuse_source/deploy/env" +printf 'SHOULD_NOT_BE_COPIED=yes\n' > "$missing_template_reuse_source/deploy/env/.env" +mv "$latest_package_dir/deploy/env/.env.example" "$latest_package_dir/deploy/env/.env.example.saved" +if DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" --reuse-from "$missing_template_reuse_source" --load-images docker --foo bar >"$TMP_DIR/offline-missing-template.log" 2>&1; then + mv "$latest_package_dir/deploy/env/.env.example.saved" "$latest_package_dir/deploy/env/.env.example" + fail "offline deploy.sh should require .env.example before loading images" +fi +mv "$latest_package_dir/deploy/env/.env.example.saved" "$latest_package_dir/deploy/env/.env.example" +[ ! -s "$offline_deploy_log" ] || fail "offline missing template failure should happen before image loading or deployment" +[ ! -f "$latest_package_dir/deploy/env/.env" ] || fail "offline missing template failure should happen before importing a reused .env" +grep -q 'deploy/env/.env.example' "$TMP_DIR/offline-missing-template.log" || fail "offline missing template failure should identify deploy/env/.env.example" + +reuse_source="$TMP_DIR/previous package" +mkdir -p "$reuse_source/deploy/env" "$reuse_source/deploy/docker" "$reuse_source/deploy/k8s/helm/nexent" +printf 'REUSED_SECRET=do-not-print-this-value\n' > "$reuse_source/deploy/env/.env" +printf 'MONITORING_REUSED=yes\n' > "$reuse_source/deploy/env/monitoring.env" +printf 'DOCKER_OPTIONS=reused\n' > "$reuse_source/deploy/docker/deploy.options" +printf 'K8S_OPTIONS=reused\n' > "$reuse_source/deploy/k8s/deploy.options" +printf 'SOURCE_DERIVED=docker\n' > "$reuse_source/deploy/docker/.env.generated" +printf 'SOURCE_DERIVED=k8s\n' > "$reuse_source/deploy/k8s/helm/nexent/generated-values.yaml" + +mkdir -p "$latest_package_dir/deploy/k8s/helm/nexent" +printf 'DOCKER_OPTIONS=current\n' > "$latest_package_dir/deploy/docker/deploy.options" +printf 'K8S_OPTIONS=keep-for-docker\n' > "$latest_package_dir/deploy/k8s/deploy.options" +printf 'CURRENT_DERIVED=docker\n' > "$latest_package_dir/deploy/docker/.env.generated" +printf 'CURRENT_DERIVED=k8s\n' > "$latest_package_dir/deploy/k8s/helm/nexent/generated-values.yaml" + +: > "$offline_deploy_log" +DEPLOY_WRAPPER_LOG="$offline_deploy_log" \ + DEPLOY_WRAPPER_EXPECT_ENV='REUSED_SECRET=do-not-print-this-value' \ + DEPLOY_WRAPPER_EXPECT_MERGED_ENV='OFFLINE_TEMPLATE_ONLY=merged-before-actions' \ + bash "$latest_package_dir/deploy.sh" docker --reuse-from "$reuse_source" --load-images --foo bar >"$TMP_DIR/reuse-docker.log" 2>&1 +first_line="$(sed -n '1p' "$offline_deploy_log")" +second_line="$(sed -n '2p' "$offline_deploy_log")" +[ "$first_line" = "load-images" ] || fail "--reuse-from should import files before loading images" +[ "$second_line" = "deploy:defaults:true:docker --foo bar" ] || fail "--reuse-from should be consumed before forwarding Docker deploy arguments" +grep -q '^REUSED_SECRET=do-not-print-this-value$' "$latest_package_dir/deploy/env/.env" || fail "Docker reuse should copy deploy/env/.env" +grep -q '^OFFLINE_TEMPLATE_ONLY=merged-before-actions$' "$latest_package_dir/deploy/env/.env" || fail "Docker reuse should merge variables from the current .env.example" +grep -q '^MONITORING_REUSED=yes$' "$latest_package_dir/deploy/env/monitoring.env" || fail "Docker reuse should copy monitoring.env" +grep -q '^DOCKER_OPTIONS=reused$' "$latest_package_dir/deploy/docker/deploy.options" || fail "Docker reuse should overwrite Docker deploy.options" +grep -q '^K8S_OPTIONS=keep-for-docker$' "$latest_package_dir/deploy/k8s/deploy.options" || fail "Docker reuse should not copy K8s deploy.options" +grep -q '^CURRENT_DERIVED=docker$' "$latest_package_dir/deploy/docker/.env.generated" || fail "Docker reuse should not copy derived .env.generated" +grep -q '^CURRENT_DERIVED=k8s$' "$latest_package_dir/deploy/k8s/helm/nexent/generated-values.yaml" || fail "Docker reuse should not copy K8s generated values" +! grep -q 'do-not-print-this-value' "$TMP_DIR/reuse-docker.log" || fail "--reuse-from should not print environment values" + +printf 'DOCKER_OPTIONS=keep-for-k8s\n' > "$latest_package_dir/deploy/docker/deploy.options" +printf 'K8S_OPTIONS=current\n' > "$latest_package_dir/deploy/k8s/deploy.options" +: > "$offline_deploy_log" +DEPLOY_WRAPPER_LOG="$offline_deploy_log" \ + DEPLOY_WRAPPER_EXPECT_ENV='REUSED_SECRET=do-not-print-this-value' \ + DEPLOY_WRAPPER_EXPECT_MERGED_ENV='OFFLINE_TEMPLATE_ONLY=merged-before-actions' \ + REGISTRY_PASSWORD=secret \ + bash "$latest_package_dir/deploy.sh" --push-images --image-registry-prefix registry.local/nexent --reuse-from "$reuse_source" k8s --foo bar >"$TMP_DIR/reuse-k8s.log" 2>&1 +first_line="$(sed -n '1p' "$offline_deploy_log")" +second_line="$(sed -n '2p' "$offline_deploy_log")" +[[ "$first_line" == "push:secret:--image-registry-prefix registry.local/nexent --load-images" ]] || fail "--reuse-from should import files before pushing images" +[ "$second_line" = "deploy:defaults:true:k8s --foo bar --image-registry-prefix registry.local/nexent" ] || fail "--reuse-from should be consumed before forwarding K8s deploy arguments" +grep -q '^K8S_OPTIONS=reused$' "$latest_package_dir/deploy/k8s/deploy.options" || fail "K8s reuse should overwrite K8s deploy.options" +grep -q '^DOCKER_OPTIONS=keep-for-k8s$' "$latest_package_dir/deploy/docker/deploy.options" || fail "K8s reuse should not copy Docker deploy.options" + +: > "$offline_deploy_log" +DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" --reuse-from "$reuse_source" --config docker --foo bar >"$TMP_DIR/reuse-config.log" 2>&1 +grep -q '^deploy:tui:true:docker --foo bar$' "$offline_deploy_log" || fail "--reuse-from should work with --config and preserve TUI mode" + +minimal_reuse_source="$TMP_DIR/minimal previous package" +mkdir -p "$minimal_reuse_source/deploy/env" +printf 'MINIMAL_REUSE=yes\n' > "$minimal_reuse_source/deploy/env/.env" +: > "$offline_deploy_log" +DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" --reuse-from "$minimal_reuse_source" --defaults docker --foo bar >"$TMP_DIR/reuse-minimal.log" 2>&1 +grep -q 'optional file not found' "$TMP_DIR/reuse-minimal.log" || fail "missing optional reuse files should produce warnings" +grep -q '^deploy:defaults:true:docker --foo bar$' "$offline_deploy_log" || fail "missing optional reuse files should not stop deployment" + +missing_env_source="$TMP_DIR/missing-env-package" +mkdir -p "$missing_env_source/deploy/env" +if DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" --reuse-from "$missing_env_source" docker --foo bar >"$TMP_DIR/reuse-missing-env.log" 2>&1; then + fail "--reuse-from should require deploy/env/.env" +fi +grep -q 'deploy/env/.env' "$TMP_DIR/reuse-missing-env.log" || fail "missing reuse .env error should identify the required file" + +if DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" --reuse-from "$TMP_DIR/does-not-exist" docker --foo bar >"$TMP_DIR/reuse-missing-dir.log" 2>&1; then + fail "--reuse-from should reject a missing directory" +fi +grep -q 'does not exist or is not a directory' "$TMP_DIR/reuse-missing-dir.log" || fail "missing reuse directory error should be explicit" + +if DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" --reuse-from "$latest_package_dir" docker --foo bar >"$TMP_DIR/reuse-same-dir.log" 2>&1; then + fail "--reuse-from should reject the current package directory" +fi +grep -q 'must differ from the current package directory' "$TMP_DIR/reuse-same-dir.log" || fail "same-directory reuse error should be explicit" + +if DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" --reuse-from "$reuse_source" >"$TMP_DIR/reuse-missing-target.log" 2>&1; then + fail "--reuse-from should require a deployment target" +fi +grep -q 'requires a docker or k8s deployment target' "$TMP_DIR/reuse-missing-target.log" || fail "missing reuse target error should be explicit" + : > "$offline_deploy_log" DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" docker --foo bar -grep -q '^deploy:defaults:docker --foo bar$' "$offline_deploy_log" || fail "offline deploy.sh should default to non-interactive defaults mode" +grep -q '^deploy:defaults:true:docker --foo bar$' "$offline_deploy_log" || fail "offline deploy.sh should default to non-interactive defaults mode" : > "$offline_deploy_log" DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" docker --config --foo bar -grep -q '^deploy:tui:docker --foo bar$' "$offline_deploy_log" || fail "offline deploy.sh --config should enable TUI mode and consume the flag" +grep -q '^deploy:tui:true:docker --foo bar$' "$offline_deploy_log" || fail "offline deploy.sh --config should enable TUI mode and propagate the offline marker" : > "$offline_deploy_log" DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" docker --defaults --foo bar -grep -q '^deploy:defaults:docker --foo bar$' "$offline_deploy_log" || fail "offline deploy.sh --defaults should preserve defaults mode and consume the flag" +grep -q '^deploy:defaults:true:docker --foo bar$' "$offline_deploy_log" || fail "offline deploy.sh --defaults should preserve defaults mode and consume the flag" : > "$offline_deploy_log" DEPLOY_WRAPPER_LOG="$offline_deploy_log" bash "$latest_package_dir/deploy.sh" --load-images docker --foo bar first_line="$(sed -n '1p' "$offline_deploy_log")" second_line="$(sed -n '2p' "$offline_deploy_log")" [ "$first_line" = "load-images" ] || fail "offline deploy.sh --load-images should load images before deploy" -[ "$second_line" = "deploy:defaults:docker --foo bar" ] || fail "offline deploy.sh --load-images should preserve defaults mode" +[ "$second_line" = "deploy:defaults:true:docker --foo bar" ] || fail "offline deploy.sh --load-images should preserve defaults mode" : > "$offline_deploy_log" DEPLOY_WRAPPER_LOG="$offline_deploy_log" REGISTRY_USERNAME=user REGISTRY_PASSWORD=secret bash "$latest_package_dir/deploy.sh" --push-images --image-registry-prefix registry.local/nexent docker --foo bar first_line="$(sed -n '1p' "$offline_deploy_log")" second_line="$(sed -n '2p' "$offline_deploy_log")" [[ "$first_line" == "push:secret:--image-registry-prefix registry.local/nexent --load-images" ]] || fail "offline deploy.sh --push-images should push before deploy" -[ "$second_line" = "deploy:defaults:docker --foo bar --image-registry-prefix registry.local/nexent" ] || fail "offline deploy.sh --push-images should preserve defaults mode and forward registry prefix" +[ "$second_line" = "deploy:defaults:true:docker --foo bar --image-registry-prefix registry.local/nexent" ] || fail "offline deploy.sh --push-images should preserve defaults mode and forward registry prefix" [ -f "$OUT_DIR/nexent-offline-docker-amd64-latest.zip" ] || fail "zip package should be created for latest package" grep -q "nexent/nexent:latest" "$latest_package_dir/manifest.yaml" || fail "manifest should include local latest Nexent image" diff --git a/deploy/tests/test_common.sh b/deploy/tests/test_common.sh index 90dd0d53c1..05d6db0265 100755 --- a/deploy/tests/test_common.sh +++ b/deploy/tests/test_common.sh @@ -126,6 +126,10 @@ ZH_OFFLINE_DRY_RUN="$(DEPLOYMENT_LANG="" LANG="zh_CN.UTF-8" bash "$SCRIPT_DIR/.. assert_contains "$ZH_OFFLINE_DRY_RUN" "=== DRY RUN 模式 ===" "offline dry-run should follow Chinese locale" assert_contains "$ZH_OFFLINE_DRY_RUN" "目标：docker" "offline dry-run target label should follow Chinese locale" +LANGFUSE_OFFLINE_DRY_RUN="$(DEPLOYMENT_LANG=en bash "$SCRIPT_DIR/../offline/build_offline_package.sh" --version v2.2.0 --platform amd64 --components infrastructure,monitoring --monitoring-provider langfuse --image-source general --target docker --dry-run)" +assert_contains "$LANGFUSE_OFFLINE_DRY_RUN" "quay.io/minio/minio:RELEASE.2023-12-20T01-00-02Z" "offline Langfuse package should use the Quay MinIO image" +assert_not_contains "$LANGFUSE_OFFLINE_DRY_RUN" "docker.io/minio/minio" "offline Langfuse package should not use the Docker Hub MinIO image" + if DEPLOYMENT_LANG="" LANG="zh_CN.UTF-8" bash "$SCRIPT_DIR/../images/build.sh" --unknown >/tmp/nexent-image-build-zh-invalid.log 2>&1; then echo "FAIL: unknown image build option should fail" exit 1 @@ -659,6 +663,8 @@ assert_contains "$(cat "$SCRIPT_DIR/../docker/compose/docker-compose-monitoring. assert_contains "$(cat "$SCRIPT_DIR/../k8s/helm/nexent/charts/nexent-monitoring/values.yaml")" "userPassword: nexent@4321" "k8s monitoring Langfuse init user password should match docker monitoring default" assert_contains "$(cat "$MONITORING_EXAMPLE_FILE")" "LANGFUSE_CLICKHOUSE_CLUSTER_ENABLED=false" "docker monitoring defaults should include all compose Langfuse clickhouse settings" assert_contains "$(cat "$SCRIPT_DIR/../docker/compose/docker-compose-monitoring.yml")" 'CLICKHOUSE_CLUSTER_ENABLED: ${LANGFUSE_CLICKHOUSE_CLUSTER_ENABLED:-false}' "docker compose Langfuse clickhouse cluster fallback should match monitoring.env.example" +assert_contains "$(cat "$SCRIPT_DIR/../docker/compose/docker-compose-monitoring.yml")" 'quay.io/minio/minio:${LANGFUSE_MINIO_VERSION:-RELEASE.2023-12-20T01-00-02Z}' "docker compose Langfuse should use the Quay MinIO image" +assert_contains "$(cat "$SCRIPT_DIR/../k8s/helm/nexent/charts/nexent-monitoring/values.yaml")" "repository: quay.io/minio/minio" "k8s Langfuse should use the Quay MinIO image" LOCAL_CONFIG="$TMP_DIR/local-config.yaml" DEPLOYMENT_IMAGE_REGISTRY_PREFIX="registry.local/nexent" @@ -724,6 +730,12 @@ assert_eq "$(sed -n '1p' "$SCRIPT_DIR/../../VERSION")" "$(deployment_read_versio assert_eq "v-test" "$(deployment_read_version "v-test")" "explicit deployment version should win" assert_success "password validation should accept frontend-compatible passwords" deployment_validate_password "Nexent123" +if NEXENT_DEPLOYMENT_OFFLINE=true NEXENT_DEPLOY_CONFIG_MODE=defaults deployment_should_prompt_root_dir; then + echo "FAIL: default offline deployment should not prompt for ROOT_DIR" + exit 1 +fi +assert_success "offline --config should prompt for ROOT_DIR on first deployment" env NEXENT_DEPLOYMENT_OFFLINE=true NEXENT_DEPLOY_CONFIG_MODE=tui bash -c 'source deploy/common/common.sh; deployment_should_prompt_root_dir' +assert_success "online deployment should retain the ROOT_DIR prompt" env NEXENT_DEPLOYMENT_OFFLINE=false NEXENT_DEPLOY_CONFIG_MODE=defaults bash -c 'source deploy/common/common.sh; deployment_should_prompt_root_dir' if deployment_validate_password "nexent123"; then echo "FAIL: password without uppercase letters should be rejected" exit 1 @@ -741,6 +753,51 @@ if deployment_validate_password "Nex123"; then exit 1 fi +unset NEXENT_SUPER_ADMIN_PASSWORD NEXENT_DEPLOYMENT_OFFLINE NEXENT_DEPLOY_CONFIG_MODE +assert_eq "Nexent@123" "$(deployment_super_admin_password)" "super admin password should use the built-in default" +NEXENT_SUPER_ADMIN_PASSWORD="CustomAdmin123" +assert_eq "CustomAdmin123" "$(deployment_super_admin_password)" "configured super admin password should override the default" +unset NEXENT_SUPER_ADMIN_PASSWORD + +if deployment_should_prompt_super_admin_password; then + echo "FAIL: online deployments should not prompt for the super admin password" + exit 1 +fi +NEXENT_DEPLOY_CONFIG_MODE="tui" +if deployment_should_prompt_super_admin_password; then + echo "FAIL: online --config deployments should not prompt for the super admin password" + exit 1 +fi +NEXENT_DEPLOYMENT_OFFLINE="true" +assert_success "offline --config deployments should prompt for the super admin password" deployment_should_prompt_super_admin_password +NEXENT_DEPLOY_CONFIG_MODE="defaults" +if deployment_should_prompt_super_admin_password; then + echo "FAIL: offline defaults deployments should not prompt for the super admin password" + exit 1 +fi +unset NEXENT_DEPLOYMENT_OFFLINE NEXENT_DEPLOY_CONFIG_MODE + +DOCKER_SUPER_ADMIN_BLOCK="$(awk '/^create_default_super_admin_user\(\) {/{capture=1} capture{print} capture && /^}/{exit}' "$SCRIPT_DIR/../docker/deploy.sh")" +K8S_SUPER_ADMIN_BLOCK="$(awk '/^create_supabase_super_admin_user\(\) {/{capture=1} capture{print} capture && /^}/{exit}' "$SCRIPT_DIR/../k8s/create-suadmin.sh")" +assert_contains "$DOCKER_SUPER_ADMIN_BLOCK" "deployment_should_prompt_super_admin_password" "Docker super admin creation should use the shared prompt policy" +assert_contains "$DOCKER_SUPER_ADMIN_BLOCK" "deployment_super_admin_password" "Docker super admin creation should use the shared default password" +assert_contains "$DOCKER_SUPER_ADMIN_BLOCK" 'bash "$script_path" "$password" "$display_password"' "Docker super admin creation should pass the password display policy" +assert_contains "$DOCKER_SUPER_ADMIN_BLOCK" 'if bash "$script_path"; then' "Docker should reuse the normal creation path for an existing super admin" +assert_contains "$K8S_SUPER_ADMIN_BLOCK" "deployment_should_prompt_super_admin_password" "K8s super admin creation should use the shared prompt policy" +assert_contains "$K8S_SUPER_ADMIN_BLOCK" "deployment_super_admin_password" "K8s super admin creation should use the shared default password" +assert_contains "$K8S_SUPER_ADMIN_BLOCK" 'echo " 🔏 Password: ${password}"' "K8s should display non-interactive super admin passwords" +assert_contains "$K8S_SUPER_ADMIN_BLOCK" 'echo " 🔏 Password: [hidden]"' "K8s should hide interactively entered super admin passwords" +DOCKER_CREATE_SU_CONTENT="$(cat "$SCRIPT_DIR/../docker/create-su.sh")" +K8S_CREATE_SU_CONTENT="$(cat "$SCRIPT_DIR/../k8s/create-suadmin.sh")" +assert_contains "$DOCKER_CREATE_SU_CONTENT" 'echo " 🔏 Password: ${password}"' "Docker should display non-interactive super admin passwords" +assert_contains "$DOCKER_CREATE_SU_CONTENT" 'echo " 🔏 Password: [hidden]"' "Docker should hide interactively entered super admin passwords" +assert_contains "$DOCKER_CREATE_SU_CONTENT" 'SELECT user_id, tenant_id, user_role, user_email, created_by, updated_by FROM nexent.user_tenant_t LIMIT 0;' "Docker should wait for the complete user_tenant_t schema contract" +assert_contains "$K8S_CREATE_SU_CONTENT" 'SELECT user_id, tenant_id, user_role, user_email, created_by, updated_by FROM nexent.user_tenant_t LIMIT 0;' "K8s should wait for the complete user_tenant_t schema contract" +assert_contains "$DOCKER_CREATE_SU_CONTENT" 'ON_ERROR_STOP=1' "Docker super admin writes should stop on SQL errors" +assert_contains "$K8S_CREATE_SU_CONTENT" 'ON_ERROR_STOP=1' "K8s super admin writes should stop on SQL errors" +assert_contains "$(cat "$SCRIPT_DIR/../k8s/deploy.sh")" 'Error: Super admin user creation failed. Deployment aborted.' "K8s deployment should stop when super admin initialization fails" +assert_contains "$(cat "$SCRIPT_DIR/../env/.env.example")" "NEXENT_SUPER_ADMIN_PASSWORD=Nexent@123" "deployment env example should define the default super admin password" + ENV_TEST_ROOT="$TMP_DIR/env-root" mkdir -p "$ENV_TEST_ROOT/docker" "$ENV_TEST_ROOT/deploy/env" printf 'FROM_ROOT_SHOULD_NOT_COPY=yes\n' > "$ENV_TEST_ROOT/.env" @@ -749,6 +806,7 @@ printf 'FROM_DOCKER=yes\n' > "$ENV_TEST_ROOT/docker/.env" printf 'FROM_EXAMPLE=yes\n' > "$ENV_TEST_ROOT/deploy/env/.env.example" deployment_ensure_root_env "$ENV_TEST_ROOT" "$ENV_TEST_ROOT/docker" assert_contains "$(cat "$ENV_TEST_ROOT/deploy/env/.env")" "FROM_DOCKER=yes" "deploy/env/.env should migrate from docker/.env first" +assert_contains "$(cat "$ENV_TEST_ROOT/deploy/env/.env")" "FROM_EXAMPLE=yes" "docker/.env migration should merge current template variables" if grep -q "FROM_ROOT_SHOULD_NOT_COPY" "$ENV_TEST_ROOT/deploy/env/.env"; then echo "FAIL: deploy/env/.env should not migrate from root .env" exit 1 @@ -769,6 +827,84 @@ fi printf 'ROOT_ONLY=yes\n' > "$ENV_TEST_ROOT/deploy/env/.env" deployment_ensure_root_env "$ENV_TEST_ROOT" "$ENV_TEST_ROOT/docker" assert_contains "$(cat "$ENV_TEST_ROOT/deploy/env/.env")" "ROOT_ONLY=yes" "existing deploy/env/.env should not be overwritten" +assert_contains "$(cat "$ENV_TEST_ROOT/deploy/env/.env")" "FROM_EXAMPLE=yes" "existing deploy/env/.env should receive variables missing from the current template" + +MERGE_ENV_ROOT="$TMP_DIR/merge-env-root" +mkdir -p "$MERGE_ENV_ROOT/docker" "$MERGE_ENV_ROOT/deploy/env" +cat > "$MERGE_ENV_ROOT/deploy/env/.env" <<'EOF' +# Existing comment remains in place +PASSWORD=custom +OLD_OPTION=true +# COMMENTED_ONLY=old-commented-value +EMPTY_VALUE= +DUPLICATE=first +DUPLICATE=second +export EXPORTED_VALUE=custom + LEADING_SPACE_VALUE=custom +SPECIAL_EXISTING='literal $EXISTING # value \\ path' +EOF +MERGE_SIDE_EFFECT="$TMP_DIR/merge-side-effect" +{ + printf '%s\n' 'PASSWORD=default' + printf '%s\n' 'EMPTY_VALUE=template-default' + printf '%s\n' 'DUPLICATE=template-default' + printf '%s\n' 'EXPORTED_VALUE=template-default' + printf '%s\n' 'LEADING_SPACE_VALUE=template-default' + printf '%s\n' 'COMMENTED_ONLY=active-default' + printf '%s\n' 'NEW_FIRST=" literal $NEW_VALUE # hash \\ path "' + printf '%s\n' "NEW_COMMAND=\"\$(touch $MERGE_SIDE_EFFECT)\"" + printf '%s\n' 'NEW_DUPLICATE=first-default' + printf '%s\n' 'NEW_DUPLICATE=second-default' + printf '%s\n' 'NEW_EMPTY=' +} > "$MERGE_ENV_ROOT/deploy/env/.env.example" +cp "$MERGE_ENV_ROOT/deploy/env/.env" "$MERGE_ENV_ROOT/original.env" +MERGE_OUTPUT="$(deployment_ensure_root_env "$MERGE_ENV_ROOT" "$MERGE_ENV_ROOT/docker")" +original_size="$(wc -c < "$MERGE_ENV_ROOT/original.env" | tr -d '[:space:]')" +head -c "$original_size" "$MERGE_ENV_ROOT/deploy/env/.env" > "$MERGE_ENV_ROOT/preserved-prefix.env" +cmp -s "$MERGE_ENV_ROOT/original.env" "$MERGE_ENV_ROOT/preserved-prefix.env" || { + echo "FAIL: .env merge should preserve all existing bytes before appended variables" + exit 1 +} +MERGED_ENV_CONTENT="$(cat "$MERGE_ENV_ROOT/deploy/env/.env")" +assert_not_contains "$MERGED_ENV_CONTENT" "PASSWORD=default" "existing values should override template defaults" +assert_not_contains "$MERGED_ENV_CONTENT" "EMPTY_VALUE=template-default" "an existing empty value should count as configured" +assert_not_contains "$MERGED_ENV_CONTENT" "DUPLICATE=template-default" "existing duplicate assignments should prevent template defaults from being appended" +assert_not_contains "$MERGED_ENV_CONTENT" "EXPORTED_VALUE=template-default" "export assignments should count as existing variables" +assert_not_contains "$MERGED_ENV_CONTENT" "LEADING_SPACE_VALUE=template-default" "indented active assignments should count as existing variables" +assert_contains "$MERGED_ENV_CONTENT" "# COMMENTED_ONLY=old-commented-value" "commented assignments should remain unchanged" +assert_contains "$MERGED_ENV_CONTENT" "COMMENTED_ONLY=active-default" "commented assignments should not count as existing variables" +assert_contains "$MERGED_ENV_CONTENT" '# Added automatically from the current deploy/env/.env.example' "new variables should follow the English merge separator" +assert_contains "$MERGED_ENV_CONTENT" 'NEW_FIRST=" literal $NEW_VALUE # hash \\ path "' "special characters should remain unexpanded and byte-preserved" +assert_contains "$MERGED_ENV_CONTENT" "NEW_COMMAND=\"\$(touch $MERGE_SIDE_EFFECT)\"" "command substitutions should be appended without execution" +assert_contains "$MERGED_ENV_CONTENT" $'NEW_DUPLICATE=first-default\nNEW_DUPLICATE=second-default\nNEW_EMPTY=' "new template assignments should retain template order and duplicates" +assert_not_contains "$MERGE_OUTPUT" "custom" "merge logs should not expose environment values" +[ ! -e "$MERGE_SIDE_EFFECT" ] || { + echo "FAIL: .env merge should never execute template values" + exit 1 +} + +cp "$MERGE_ENV_ROOT/deploy/env/.env" "$MERGE_ENV_ROOT/merged-snapshot.env" +deployment_ensure_root_env "$MERGE_ENV_ROOT" "$MERGE_ENV_ROOT/docker" >/dev/null +cmp -s "$MERGE_ENV_ROOT/merged-snapshot.env" "$MERGE_ENV_ROOT/deploy/env/.env" || { + echo "FAIL: .env merge should not rewrite files when no variables are missing" + exit 1 +} + +EMPTY_ENV_ROOT="$TMP_DIR/empty-env-root" +mkdir -p "$EMPTY_ENV_ROOT/deploy/env" +: > "$EMPTY_ENV_ROOT/deploy/env/.env" +printf 'ADDED_TO_EMPTY=yes\n' > "$EMPTY_ENV_ROOT/deploy/env/.env.example" +deployment_ensure_root_env "$EMPTY_ENV_ROOT" "$EMPTY_ENV_ROOT/docker" >/dev/null +assert_eq $'# Added automatically from the current deploy/env/.env.example\nADDED_TO_EMPTY=yes' "$(cat "$EMPTY_ENV_ROOT/deploy/env/.env")" "an empty .env should receive current template variables" + +MISSING_TEMPLATE_ROOT="$TMP_DIR/missing-template-root" +mkdir -p "$MISSING_TEMPLATE_ROOT/deploy/env" +printf 'EXISTING_VALUE=preserved\n' > "$MISSING_TEMPLATE_ROOT/deploy/env/.env" +if deployment_ensure_root_env "$MISSING_TEMPLATE_ROOT" "$MISSING_TEMPLATE_ROOT/docker" >"$TMP_DIR/missing-template.log" 2>&1; then + echo "FAIL: environment initialization should require deploy/env/.env.example" + exit 1 +fi +assert_contains "$(cat "$TMP_DIR/missing-template.log")" "deploy/env/.env.example" "missing template errors should identify the required file" deployment_update_env_var_file "$ENV_TEST_ROOT/deploy/env/.env" "ROOT_ONLY" "updated" assert_contains "$(cat "$ENV_TEST_ROOT/deploy/env/.env")" 'ROOT_ONLY="updated"' "env updater should update deploy env values" diff --git a/deploy/tests/test_sql_migrations.sh b/deploy/tests/test_sql_migrations.sh index c8622009d4..9ee952925d 100755 --- a/deploy/tests/test_sql_migrations.sh +++ b/deploy/tests/test_sql_migrations.sh @@ -96,6 +96,20 @@ fi if grep -Eq '^COMMENT ON COLUMN nexent\.model_record_t\.is_deep_thinking ' "$DEPLOY_ROOT/sql/init.sql"; then fail "init SQL should not comment model_record_t.is_deep_thinking because a later migration drops that column" fi +if grep -Eq '^[[:space:]]*"step_index"[[:space:]]' "$DEPLOY_ROOT/sql/init.sql"; then + fail "init SQL should leave conversation_message_unit_t.step_index to its migration" +fi +if grep -Eq '^COMMENT ON COLUMN .*conversation_message_unit_t.*step_index' "$DEPLOY_ROOT/sql/init.sql"; then + fail "init SQL should not comment conversation_message_unit_t.step_index before its migration adds the column" +fi + +HISTORY_PROJECTION_MIGRATION="$DEPLOY_ROOT/sql/migrations/v2.3_merged_migrations.sql" +assert_file_contains "$HISTORY_PROJECTION_MIGRATION" \ + "ADD COLUMN IF NOT EXISTS step_index INTEGER DEFAULT NULL;" \ + "history projection migration should add conversation_message_unit_t.step_index" +assert_file_contains "$HISTORY_PROJECTION_MIGRATION" \ + "COMMENT ON COLUMN nexent.conversation_message_unit_t.step_index" \ + "history projection migration should comment conversation_message_unit_t.step_index" PLAN_FILE="$TMP_DIR/plan.sql" PATH="$BIN_DIR:$PATH" \ diff --git a/deploy/tests/test_super_admin_init.sh b/deploy/tests/test_super_admin_init.sh new file mode 100644 index 0000000000..e9cd328281 --- /dev/null +++ b/deploy/tests/test_super_admin_init.sh @@ -0,0 +1,392 @@ +#!/usr/bin/env bash + +set -uo pipefail + +TEST_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +TEST_TMP="$(mktemp -d "${TMPDIR:-/tmp}/nexent-super-admin-test.XXXXXX")" +trap 'rm -rf "$TEST_TMP"' EXIT + +fail() { + echo "FAIL: $1" + return 1 +} + +assert_file_contains() { + local file="$1" + local expected="$2" + local message="$3" + + if ! grep -Fq -- "$expected" "$file"; then + fail "$message" + fi +} + +assert_file_not_contains() { + local file="$1" + local unexpected="$2" + local message="$3" + + if grep -Fq -- "$unexpected" "$file"; then + fail "$message" + fi +} + +assert_event_order() { + local file="$1" + local first="$2" + local second="$3" + local message="$4" + local first_line second_line + + first_line="$(awk -v pattern="$first" 'index($0, pattern) { line=NR } END { print line+0 }' "$file")" + second_line="$(awk -v pattern="$second" 'index($0, pattern) { print NR; exit }' "$file")" + if [ "$first_line" -eq 0 ] || [ -z "$second_line" ] || [ "$first_line" -ge "$second_line" ]; then + fail "$message" + fi +} + +prepare_case() { + local name="$1" + + MOCK_DIR="$TEST_TMP/$name" + mkdir -p "$MOCK_DIR" + EVENT_LOG="$MOCK_DIR/events.log" + : > "$EVENT_LOG" + printf '0\n' > "$MOCK_DIR/schema-attempts" + printf '0\n' > "$MOCK_DIR/time" + + MOCK_SCHEMA_FAILURES=0 + MOCK_INSERT_FAILURE=false + MOCK_USER_QUERY_FAILURE=false + MOCK_EXISTING_USER_ID="" + MOCK_SIGNUP_RESPONSE='{"access_token":"token","user":{"id":"new-user-id"}}' + MOCK_KUBECTL_WAIT_FAILURE=false + + NEXENT_SQL_MIGRATION_WAIT_TIMEOUT_SECONDS=10 + NEXENT_SQL_MIGRATION_WAIT_INTERVAL_SECONDS=0 + export MOCK_DIR EVENT_LOG MOCK_SCHEMA_FAILURES MOCK_INSERT_FAILURE + export MOCK_USER_QUERY_FAILURE MOCK_EXISTING_USER_ID MOCK_SIGNUP_RESPONSE + export MOCK_KUBECTL_WAIT_FAILURE NEXENT_SQL_MIGRATION_WAIT_TIMEOUT_SECONDS + export NEXENT_SQL_MIGRATION_WAIT_INTERVAL_SECONDS +} + +date() { + if [ "${1:-}" != "+%s" ]; then + command date "$@" + return + fi + + local value + value="$(sed -n '1p' "$MOCK_DIR/time")" + printf '%s\n' "$((value + 1))" > "$MOCK_DIR/time" + printf '%s\n' "$value" +} + +sleep() { + return 0 +} + +mock_schema_contract() { + local attempt + attempt="$(sed -n '1p' "$MOCK_DIR/schema-attempts")" + printf '%s\n' "$((attempt + 1))" > "$MOCK_DIR/schema-attempts" + [ "$attempt" -ge "$MOCK_SCHEMA_FAILURES" ] +} + +docker() { + local command_line="$*" + printf 'docker %s\n' "$command_line" >> "$EVENT_LOG" + + if [ "$command_line" = "ps" ]; then + printf '%s\n' 'nexent-config' 'supabase-db-mini' + return 0 + fi + if [[ "$command_line" == *"SELECT user_id, tenant_id, user_role, user_email, created_by, updated_by"* ]]; then + mock_schema_contract + return $? + fi + if [[ "$command_line" == *"INSERT INTO nexent.user_tenant_t"* ]]; then + [ "$MOCK_INSERT_FAILURE" != "true" ] + return $? + fi + if [[ "$command_line" == *"SELECT id FROM auth.users"* ]]; then + if [ "$MOCK_USER_QUERY_FAILURE" = "true" ]; then + return 1 + fi + printf '%s\n' "$MOCK_EXISTING_USER_ID" + return 0 + fi + if [[ "$command_line" == *"command -v jq"* ]]; then + return 1 + fi + if [[ "$command_line" == *"curl -s -X POST"* ]]; then + printf '%s\n' "$MOCK_SIGNUP_RESPONSE" + return 0 + fi + + return 0 +} + +kubectl() { + local command_line="$*" + printf 'kubectl %s\n' "$command_line" >> "$EVENT_LOG" + + if [[ "$command_line" == wait\ * ]]; then + [ "$MOCK_KUBECTL_WAIT_FAILURE" != "true" ] + return $? + fi + if [[ "$command_line" == *"SELECT user_id, tenant_id, user_role, user_email, created_by, updated_by"* ]]; then + mock_schema_contract + return $? + fi + if [[ "$command_line" == *"SELECT 1 FROM auth.users"* ]]; then + return 0 + fi + if [[ "$command_line" == *"SELECT id FROM auth.users"* ]]; then + if [ "$MOCK_USER_QUERY_FAILURE" = "true" ]; then + return 1 + fi + printf '%s\n' "$MOCK_EXISTING_USER_ID" + return 0 + fi + if [[ "$command_line" == *"INSERT INTO nexent.user_tenant_t"* ]]; then + [ "$MOCK_INSERT_FAILURE" != "true" ] + return $? + fi + if [[ "$command_line" == *"curl -s -X POST"* ]]; then + printf '%s\n' "$MOCK_SIGNUP_RESPONSE" + return 0 + fi + if [[ "$command_line" == get\ secret\ * ]]; then + return 0 + fi + + return 0 +} + +load_docker_script() { + # Load only function definitions; production scripts always execute their entrypoint. + # shellcheck source=/dev/null + source "$TEST_ROOT/deploy/common/common.sh" + # shellcheck source=/dev/null + source <(sed -n '/^wait_for_user_tenant_schema_ready()/,/^# Main execution\./p' \ + "$TEST_ROOT/deploy/docker/create-su.sh" | sed '$d') + set +e + + POSTGRES_USER=root + POSTGRES_DB=nexent + SUPABASE_POSTGRES_DB=supabase + SUPABASE_KEY=test-key + DEPLOYMENT_VERSION=full + DEPLOYMENT_MODE=development + export POSTGRES_USER POSTGRES_DB SUPABASE_POSTGRES_DB SUPABASE_KEY + export DEPLOYMENT_VERSION DEPLOYMENT_MODE +} + +load_k8s_script() { + # Load only function definitions; production scripts always execute their entrypoint. + # shellcheck source=/dev/null + source "$TEST_ROOT/deploy/common/common.sh" + # shellcheck source=/dev/null + source <(sed -n '/^prompt_super_admin_password()/,/^# Run main function\./p' \ + "$TEST_ROOT/deploy/k8s/create-suadmin.sh" | sed '$d') + set +e + + NAMESPACE=nexent + SUPER_ADMIN_EMAIL=suadmin@nexent.com + + get_supabase_service_role_key() { + printf '%s\n' 'service-role-key' + } + get_supabase_anon_key() { + printf '%s\n' 'anon-key' + } +} + +test_docker_waits_before_insert() ( + prepare_case docker-waits + MOCK_SCHEMA_FAILURES=2 + export MOCK_SCHEMA_FAILURES + load_docker_script + + if ! create_default_super_admin_user 'ValidAdmin123' false > "$MOCK_DIR/output.log" 2>&1; then + fail "Docker initialization should succeed after the schema becomes ready" + return + fi + assert_event_order "$EVENT_LOG" \ + "curl -s -X POST" \ + "SELECT user_id, tenant_id, user_role, user_email, created_by, updated_by" \ + "Docker should check the schema only when it is ready to insert" + assert_event_order "$EVENT_LOG" \ + "SELECT user_id, tenant_id, user_role, user_email, created_by, updated_by" \ + "INSERT INTO nexent.user_tenant_t" \ + "Docker must wait for the schema contract immediately before INSERT" +) + +test_docker_timeout_prevents_insert() ( + prepare_case docker-timeout + MOCK_SCHEMA_FAILURES=99 + NEXENT_SQL_MIGRATION_WAIT_TIMEOUT_SECONDS=1 + export MOCK_SCHEMA_FAILURES NEXENT_SQL_MIGRATION_WAIT_TIMEOUT_SECONDS + load_docker_script + + if create_default_super_admin_user 'ValidAdmin123' false > "$MOCK_DIR/output.log" 2>&1; then + fail "Docker schema timeout should fail initialization" + return + fi + assert_file_contains "$EVENT_LOG" "curl -s -X POST" \ + "Docker should reach signup before checking the insert schema" + assert_file_not_contains "$EVENT_LOG" "INSERT INTO nexent.user_tenant_t" \ + "Docker schema timeout must prevent INSERT" +) + +test_docker_insert_failure_is_fatal() ( + prepare_case docker-insert-failure + MOCK_INSERT_FAILURE=true + export MOCK_INSERT_FAILURE + load_docker_script + + if create_default_super_admin_user 'ValidAdmin123' false > "$MOCK_DIR/output.log" 2>&1; then + fail "Docker INSERT failure should fail initialization" + fi +) + +test_docker_existing_user_repairs_idempotently() ( + prepare_case docker-existing + MOCK_EXISTING_USER_ID=existing-user-id + MOCK_SIGNUP_RESPONSE='{"error_code":"user_already_exists"}' + export MOCK_EXISTING_USER_ID MOCK_SIGNUP_RESPONSE + load_docker_script + + if ! create_default_super_admin_user 'ValidAdmin123' false > "$MOCK_DIR/output-1.log" 2>&1 || \ + ! create_default_super_admin_user 'ValidAdmin123' false > "$MOCK_DIR/output-2.log" 2>&1; then + fail "Docker should repair an existing user's tenant relationship on every retry" + return + fi + assert_file_contains "$EVENT_LOG" "ON CONFLICT (user_id, tenant_id) DO NOTHING" \ + "Docker repair INSERT should be idempotent" +) + +test_docker_user_query_failure_is_fatal() ( + prepare_case docker-query-failure + MOCK_USER_QUERY_FAILURE=true + MOCK_SIGNUP_RESPONSE='{"error_code":"user_already_exists"}' + export MOCK_USER_QUERY_FAILURE MOCK_SIGNUP_RESPONSE + load_docker_script + + if create_default_super_admin_user 'ValidAdmin123' false > "$MOCK_DIR/output.log" 2>&1; then + fail "Docker existing user query failure should fail initialization" + fi +) + +test_k8s_waits_before_insert() ( + prepare_case k8s-waits + MOCK_SCHEMA_FAILURES=2 + export MOCK_SCHEMA_FAILURES + load_k8s_script + + if ! create_supabase_super_admin_user > "$MOCK_DIR/output.log" 2>&1; then + fail "K8s initialization should succeed after the schema becomes ready" + return + fi + assert_event_order "$EVENT_LOG" \ + "curl -s -X POST" \ + "SELECT user_id, tenant_id, user_role, user_email, created_by, updated_by" \ + "K8s should check the schema only when it is ready to insert" + assert_event_order "$EVENT_LOG" \ + "SELECT user_id, tenant_id, user_role, user_email, created_by, updated_by" \ + "INSERT INTO nexent.user_tenant_t" \ + "K8s must wait for the schema contract immediately before INSERT" +) + +test_k8s_timeout_prevents_insert() ( + prepare_case k8s-timeout + MOCK_SCHEMA_FAILURES=99 + NEXENT_SQL_MIGRATION_WAIT_TIMEOUT_SECONDS=1 + export MOCK_SCHEMA_FAILURES NEXENT_SQL_MIGRATION_WAIT_TIMEOUT_SECONDS + load_k8s_script + + if create_supabase_super_admin_user > "$MOCK_DIR/output.log" 2>&1; then + fail "K8s schema timeout should fail initialization" + return + fi + assert_file_contains "$EVENT_LOG" "curl -s -X POST" \ + "K8s should reach signup before checking the insert schema" + assert_file_not_contains "$EVENT_LOG" "INSERT INTO nexent.user_tenant_t" \ + "K8s schema timeout must prevent INSERT" +) + +test_k8s_insert_failure_is_fatal() ( + prepare_case k8s-insert-failure + MOCK_INSERT_FAILURE=true + export MOCK_INSERT_FAILURE + load_k8s_script + + if create_supabase_super_admin_user > "$MOCK_DIR/output.log" 2>&1; then + fail "K8s INSERT failure should fail initialization" + fi +) + +test_k8s_existing_user_repairs_idempotently() ( + prepare_case k8s-existing + MOCK_EXISTING_USER_ID=existing-user-id + export MOCK_EXISTING_USER_ID + load_k8s_script + + if ! create_supabase_super_admin_user > "$MOCK_DIR/output-1.log" 2>&1 || \ + ! create_supabase_super_admin_user > "$MOCK_DIR/output-2.log" 2>&1; then + fail "K8s should repair an existing user's tenant relationship on every retry" + return + fi + assert_file_not_contains "$EVENT_LOG" "curl -s -X POST" \ + "K8s existing-user recovery must not recreate the Supabase user" + assert_file_contains "$EVENT_LOG" "ON CONFLICT (user_id, tenant_id) DO NOTHING" \ + "K8s repair INSERT should be idempotent" +) + +test_k8s_user_query_failure_is_fatal() ( + prepare_case k8s-query-failure + MOCK_USER_QUERY_FAILURE=true + export MOCK_USER_QUERY_FAILURE + load_k8s_script + + if create_supabase_super_admin_user > "$MOCK_DIR/output.log" 2>&1; then + fail "K8s existing user query failure should fail initialization" + fi +) + +test_k8s_pod_readiness_failure_is_fatal() ( + prepare_case k8s-pod-failure + MOCK_KUBECTL_WAIT_FAILURE=true + export MOCK_KUBECTL_WAIT_FAILURE + load_k8s_script + + if main > "$MOCK_DIR/output.log" 2>&1; then + fail "K8s pod readiness failure should fail initialization" + fi +) + +run_test() { + local name="$1" + local test_function="$2" + + if "$test_function"; then + echo "PASS: $name" + return 0 + fi + return 1 +} + +run_test "Docker waits for the schema before INSERT" test_docker_waits_before_insert || exit 1 +run_test "Docker timeout prevents INSERT" test_docker_timeout_prevents_insert || exit 1 +run_test "Docker INSERT failure is fatal" test_docker_insert_failure_is_fatal || exit 1 +run_test "Docker existing-user repair is idempotent" test_docker_existing_user_repairs_idempotently || exit 1 +run_test "Docker user query failure is fatal" test_docker_user_query_failure_is_fatal || exit 1 +run_test "K8s waits for the schema before INSERT" test_k8s_waits_before_insert || exit 1 +run_test "K8s timeout prevents INSERT" test_k8s_timeout_prevents_insert || exit 1 +run_test "K8s INSERT failure is fatal" test_k8s_insert_failure_is_fatal || exit 1 +run_test "K8s existing-user repair is idempotent" test_k8s_existing_user_repairs_idempotently || exit 1 +run_test "K8s user query failure is fatal" test_k8s_user_query_failure_is_fatal || exit 1 +run_test "K8s pod readiness failure is fatal" test_k8s_pod_readiness_failure_is_fatal || exit 1 + +echo "All super admin initialization tests passed." diff --git a/doc/docs/.vitepress/config.mts b/doc/docs/.vitepress/config.mts index 87e79a831d..fe0df87818 100644 --- a/doc/docs/.vitepress/config.mts +++ b/doc/docs/.vitepress/config.mts @@ -93,66 +93,56 @@ export default defineConfig({ items: [ { text: "Home Page", link: "/en/user-guide/home-page" }, { text: "Start Chat", link: "/en/user-guide/start-chat" }, - { - text: "Quick Setup", - link: "/en/user-guide/quick-setup", - }, - { text: "Agent Space", link: "/en/user-guide/agent-space" }, - { text: "Agent Market", link: "/en/user-guide/agent-market" }, + { text: "Auto Tasks", link: "/en/user-guide/auto-tasks" }, { text: "Agent Development", link: "/en/user-guide/agent-development", - }, - { - text: "Knowledge Base", - link: "/en/user-guide/knowledge-base", - }, - { text: "MCP Tools", link: "/en/user-guide/mcp-tools" }, - { text: "Monitoring & Ops", link: "/en/user-guide/monitor" }, - { - text: "Model Management", - link: "/en/user-guide/model-management", - }, - { - text: "Memory Management", - link: "/en/user-guide/memory-management", - }, - { - text: "User Management", - link: "/en/user-guide/user-management", - }, - { - text: "Third-party Platform Integrations", - items: [ - { text: "ModelEngine", link: "/en/user-guide/modelengine" }, - ], - }, - { - text: "Local Tools", items: [ - { text: "Overview", link: "/en/user-guide/local-tools/" }, { - text: "File Tools", - link: "/en/user-guide/local-tools/file-tools", + text: "Model Configuration", + link: "/en/user-guide/agent-development/model-configuration", + }, + { + text: "Knowledge Configuration", + link: "/en/user-guide/agent-development/knowledge-configuration", + }, + { + text: "Agent Configuration", + link: "/en/user-guide/agent-development/agent-configuration", + items: [ + { + text: "Local Tools", + link: "/en/user-guide/local-tools/", + }, + ], }, { - text: "Email Tools", - link: "/en/user-guide/local-tools/email-tools", + text: "Memory Configuration", + link: "/en/user-guide/agent-development/memory-configuration", }, + ], + }, + { + text: "Resource Repository", + items: [ { - text: "Search Tools", - link: "/en/user-guide/local-tools/search-tools", + text: "Agent Repository", + link: "/en/user-guide/resource-repository/agent-repository", }, { - text: "Multimodal Tools", - link: "/en/user-guide/local-tools/multimodal-tools", + text: "MCP Repository", + link: "/en/user-guide/resource-repository/mcp-repository", }, { - text: "Terminal Tool", - link: "/en/user-guide/local-tools/terminal-tool", + text: "Skill Repository", + link: "/en/user-guide/resource-repository/skill-repository", }, ], }, + { + text: "Resource Management", + link: "/en/user-guide/resource-management", + }, ], }, { @@ -320,54 +310,68 @@ export default defineConfig({ items: [ { text: "首页", link: "/zh/user-guide/home-page" }, { text: "开始问答", link: "/zh/user-guide/start-chat" }, - { text: "快速配置", link: "/zh/user-guide/quick-setup" }, - { text: "智能体空间", link: "/zh/user-guide/agent-space" }, - { text: "智能体市场", link: "/zh/user-guide/agent-market" }, + { text: "自动任务", link: "/zh/user-guide/auto-tasks" }, { text: "智能体开发", link: "/zh/user-guide/agent-development", - }, - { - text: "知识库", - link: "/zh/user-guide/knowledge-base", - }, - { text: "MCP工具", link: "/zh/user-guide/mcp-tools" }, - { text: "监控与运维", link: "/zh/user-guide/monitor" }, - { text: "模型管理", link: "/zh/user-guide/model-management" }, - { text: "记忆管理", link: "/zh/user-guide/memory-management" }, - { text: "用户管理", link: "/zh/user-guide/user-management" }, - { - text: "本地工具", items: [ - { text: "概览", link: "/zh/user-guide/local-tools/" }, { - text: "文件工具", - link: "/zh/user-guide/local-tools/file-tools", + text: "模型配置", + link: "/zh/user-guide/agent-development/model-configuration", }, { - text: "邮件工具", - link: "/zh/user-guide/local-tools/email-tools", + text: "知识库配置", + link: "/zh/user-guide/agent-development/knowledge-configuration", }, { - text: "搜索工具", - link: "/zh/user-guide/local-tools/search-tools", + text: "智能体配置", + link: "/zh/user-guide/agent-development/agent-configuration", + items: [ + { + text: "本地工具", + link: "/zh/user-guide/local-tools/", + }, + ], }, { - text: "多模态工具", - link: "/zh/user-guide/local-tools/multimodal-tools", - }, - { - text: "终端工具", - link: "/zh/user-guide/local-tools/terminal-tool", + text: "记忆配置", + link: "/zh/user-guide/agent-development/memory-configuration", }, ], }, { - text: "对接第三方平台", + text: "资源仓库", items: [ - { text: "ModelEngine", link: "/zh/user-guide/modelengine" }, + { + text: "智能体仓库", + link: "/zh/user-guide/resource-repository/agent-repository", + }, + { + text: "MCP仓库", + link: "/zh/user-guide/resource-repository/mcp-repository", + }, + { + text: "Skill仓库", + link: "/zh/user-guide/resource-repository/skill-repository", + items: [ + { + text: "官方技能", + link: "/zh/user-guide/resource-repository/official-skills", + items: [ + { + text: "create-docx", + link: "/zh/user-guide/resource-repository/create-docx", + }, + ], + }, + ], + }, ], }, + { + text: "资源管理", + link: "/zh/user-guide/resource-management", + }, ], }, { diff --git a/doc/docs/.vitepress/theme/index.ts b/doc/docs/.vitepress/theme/index.ts index def4cfc87e..46440c2e42 100644 --- a/doc/docs/.vitepress/theme/index.ts +++ b/doc/docs/.vitepress/theme/index.ts @@ -4,6 +4,23 @@ import type { Theme } from 'vitepress' import DefaultTheme from 'vitepress/theme' import './style.css' +let preview: HTMLDivElement | undefined + +const closePreview = () => { + preview?.classList.remove('is-visible') + preview?.replaceChildren() +} + +const showPreview = (image: HTMLImageElement) => { + if (!preview) return + + const previewImage = new Image() + previewImage.src = image.currentSrc || image.src + previewImage.alt = image.alt + preview.replaceChildren(previewImage) + preview.classList.add('is-visible') +} + export default { extends: DefaultTheme, Layout: () => { @@ -11,7 +28,28 @@ export default { // https://vitepress.dev/guide/extending-default-theme#layout-slots }) }, - enhanceApp({ app, router, siteData }) { - // ... + enhanceApp() { + if (typeof document === 'undefined' || preview) return + + preview = document.createElement('div') + preview.className = 'image-preview' + preview.setAttribute('role', 'dialog') + preview.setAttribute('aria-modal', 'true') + preview.setAttribute('aria-label', 'Image preview') + preview.addEventListener('click', closePreview) + document.body.append(preview) + + document.addEventListener('click', (event) => { + const target = event.target + if (!(target instanceof HTMLImageElement)) return + if (!target.closest('.vp-doc') || target.closest('a')) return + + event.preventDefault() + showPreview(target) + }) + + document.addEventListener('keydown', (event) => { + if (event.key === 'Escape') closePreview() + }) } } satisfies Theme diff --git a/doc/docs/.vitepress/theme/style.css b/doc/docs/.vitepress/theme/style.css index f5975ad7c6..a0b6f111ba 100644 --- a/doc/docs/.vitepress/theme/style.css +++ b/doc/docs/.vitepress/theme/style.css @@ -162,3 +162,34 @@ background-color: rgba(255, 255, 255, 0.1); } +.vp-doc img:not([src$='.svg']) { + cursor: zoom-in; +} + +.image-preview { + position: fixed; + z-index: 1000; + display: flex; + align-items: center; + justify-content: center; + inset: 0; + padding: 24px; + background: rgba(0, 0, 0, 0.82); + cursor: zoom-out; + opacity: 0; + pointer-events: none; + transition: opacity 0.2s ease; +} + +.image-preview.is-visible { + opacity: 1; + pointer-events: auto; +} + +.image-preview img { + max-width: 100%; + max-height: 100%; + object-fit: contain; + cursor: default; +} + diff --git a/doc/docs/en/backend/skills/index.md b/doc/docs/en/backend/skills/index.md index 7824260faf..c1c3be89c9 100644 --- a/doc/docs/en/backend/skills/index.md +++ b/doc/docs/en/backend/skills/index.md @@ -17,13 +17,13 @@ In Nexent, **Tools** and **Skills** are two distinct layers: ## Quick Start 1. **Explore capabilities**: Read [Skills System Overview](./overview) to understand the supported skill types -2. **Try creation**: Experience NL-to-Skill creation on the [Skill Management](../../user-guide/skills) page +2. **Try creation**: Experience NL-to-Skill creation on the [Skill Management](../../user-guide/resource-repository/skill-repository.md) page 3. **Create manually**: Upload `SKILL.md` or a ZIP package to create a custom skill 4. **Configure for agents**: Enable skills in the agent's tool configuration ## Related References -- [Skill Management (User Guide)](../../user-guide/skills) +- [Skill Management (User Guide)](../../user-guide/resource-repository/skill-repository.md) - [Agent Development Guide](../../user-guide/agent-development) - [Local Tools Overview](../../user-guide/local-tools/index) - [SDK Tool Development Guide](../../sdk/core/tools) diff --git a/doc/docs/en/backend/skills/overview.md b/doc/docs/en/backend/skills/overview.md index 34fbd2f971..e7dc61fbe6 100644 --- a/doc/docs/en/backend/skills/overview.md +++ b/doc/docs/en/backend/skills/overview.md @@ -133,6 +133,6 @@ Create skill → Configure parameters → Select skill for agent → Debug → P ## Related References -- [Skill Management (User Guide)](../../user-guide/skills) +- [Skill Management (User Guide)](../../user-guide/resource-repository/skill-repository.md) - [Agent Development Guide](../../user-guide/agent-development) - [Local Tools Overview](../../user-guide/local-tools/index) diff --git a/doc/docs/en/deployment/docker-build.md b/doc/docs/en/deployment/docker-build.md index a2bbe92cd1..388c76bb61 100644 --- a/doc/docs/en/deployment/docker-build.md +++ b/doc/docs/en/deployment/docker-build.md @@ -94,6 +94,13 @@ docker build --progress=plain -t nexent/nexent-data-process-gpu -f deploy/images # 🌐 Build web frontend image (current architecture only) docker build --progress=plain -t nexent/nexent-web -f deploy/images/dockerfiles/web/Dockerfile . +# Deploy the frontend under a custom subpath (replace /your-subpath as needed) +# CONFIGURED_BASE_PATH must be / or start with / without a trailing slash +docker build --progress=plain --build-arg CONFIGURED_BASE_PATH=/your-subpath -t nexent/nexent-web -f deploy/images/dockerfiles/web/Dockerfile . + +# Git Bash on Windows converts arguments that start with /; disable conversion to preserve the custom subpath +MSYS_NO_PATHCONV=1 docker build --progress=plain --build-arg CONFIGURED_BASE_PATH=/your-subpath -t nexent/nexent-web -f deploy/images/dockerfiles/web/Dockerfile . + # 📚 Build documentation image (current architecture only) docker build --progress=plain -t nexent/nexent-docs -f deploy/images/dockerfiles/docs/Dockerfile . @@ -239,29 +246,52 @@ bash deploy.sh docker --image-source local-latest > `local-latest` uses local `latest` Nexent application images and avoids pulling those images again. You do not need to modify `deploy/docker/deploy.sh`. -### Package Local Images for Offline Deployment +### Build Offline Deployment Packages -After building local `latest` images, package them with the offline builder: +On an internet-connected machine, build an offline package containing both Docker and Kubernetes deployment resources from the repository root: ```bash bash build.sh --package \ - --target docker \ - --version latest \ + --target all \ + --version v2.2.1 \ --platform amd64 \ --components infrastructure,application,data-process,supabase \ - --image-source local-latest \ + --image-source general \ --compress true \ - --output-dir offline-package/docker-local + --output-dir offline-package ``` -When `--version latest` or `--image-source local-latest` is used, the builder expects local Nexent application images and skips pulling those `latest` tags. The package can then be moved to another host and deployed with: +Common options: + +| Option | Description | +| --- | --- | +| `--target` | Include `docker`, `k8s`, or `all` deployment resources | +| `--version` | Nexent image version to pull and package | +| `--platform` | Target host architecture: `amd64` or `arm64` | +| `--components` | Deployment components; also controls which images are packaged | +| `--image-source` | `general`, `mainland`, or `local-latest` | +| `--include-source` | Include project source code; defaults to `false` | +| `--compress` | Create a zip archive; defaults to `false` | +| `--output-dir` | Output directory for the unpacked package | + +To package locally built `latest` application images: ```bash -cd offline-package/docker-local -bash deploy.sh --load-images docker \ +bash build.sh --package \ + --target docker \ --version latest \ + --platform amd64 \ --components infrastructure,application,data-process,supabase \ - --image-source local-latest + --image-source local-latest \ + --compress true \ + --output-dir offline-package/docker-local ``` -To push the packaged images to an internal registry during offline deployment, replace `--load-images` with `--push-images --image-registry-prefix registry.example.com/nexent`. If the prefix is omitted, the wrapper prompts for it before `push-images.sh` asks for the registry username and password. The deployment config will use the same registry prefix for Docker Compose image references. +`local-latest` reuses local Nexent application images instead of pulling those `latest` images again. The builder produces image tar files, deployment resources, `manifest.yaml`, and `checksums.txt`. It does not copy the packaging host's `deploy/env/.env`, `deploy/env/monitoring.env`, or `deploy.options`. + +With `--compress true`, the builder creates `nexent-offline---.zip` next to the output directory. You can also manually run [Build Offline Deployment Package](https://github.com/ModelEngine-Group/nexent/actions/workflows/build-offline-package.yml) in GitHub Actions. The workflow publishes separate `nexent--.zip` artifacts for AMD64 and ARM64 with a default retention period of 30 days. + +For package download and installation instructions, see: + +- [Offline Deployment in Docker Installation](../quick-start/installation#offline-deployment) +- [Offline Deployment in Kubernetes Installation](../quick-start/kubernetes-installation#offline-deployment) diff --git a/doc/docs/en/quick-start/faq.md b/doc/docs/en/quick-start/faq.md index c0b8a20136..77b7f6bb54 100644 --- a/doc/docs/en/quick-start/faq.md +++ b/doc/docs/en/quick-start/faq.md @@ -53,7 +53,7 @@ This FAQ addresses common questions and issues you might encounter while install 3. **Model name**: Confirm the model identifier is correct 4. **Network access**: Ensure your deployment can reach the provider's servers - For model setup instruction, see [Model Management](../user-guide/model-management) in User Guide. + For model setup instruction, see [Model Management](../user-guide/agent-development/model-configuration.md) in User Guide. - **Q: Multi-turn chats fail when using the official DeepSeek API. How can I resolve this?** - A: The official DeepSeek API only accepts text payloads, but Nexent sends multimodal payloads, so multi-turn calls are rejected. Use a provider such as SiliconFlow that exposes DeepSeek models with multimodal compatibility. Our requests look like: diff --git a/doc/docs/en/quick-start/installation.md b/doc/docs/en/quick-start/installation.md index 5cc574e719..6f8b1acc97 100644 --- a/doc/docs/en/quick-start/installation.md +++ b/doc/docs/en/quick-start/installation.md @@ -14,16 +14,21 @@ ## 🚀 Quick Start -### 1. Download and Setup +- [Online Deployment](#online-deployment) +- [Offline Deployment](#offline-deployment) + +### Online Deployment + +#### 1. Download and Setup ```bash git clone https://github.com/ModelEngine-Group/nexent.git cd nexent ``` -> **Tip**: Docker and Kubernetes use `deploy/env/.env`. Existing `deploy/env/.env` is kept as-is. If it does not exist, the deploy scripts first reuse `docker/.env`, then fall back to `deploy/env/.env.example`. If you need to configure voice models (STT/TTS), update the related values in `deploy/env/.env` before or after deployment. +> **Tip**: Docker and Kubernetes use `deploy/env/.env`. Before every deployment, the scripts keep all existing values, comments, and old variables, then append variables newly introduced by the current `deploy/env/.env.example`. If `.env` does not exist, they first reuse legacy `docker/.env`, then fall back to the current template. A readable `.env.example` is required. If you need to configure voice models (STT/TTS), update the related values in `deploy/env/.env` before or after deployment. -### 2. Deployment Options +#### 2. Deployment Options Run the following command to start deployment: @@ -73,11 +78,11 @@ After a successful deployment, non-sensitive choices are saved to `deploy/docker #### ⚠️ Important Notes -1️⃣ **When deploying v1.8.0 or later for the first time**, please pay special attention to the `suadmin` super administrator account information output in the Docker logs. This account has the highest system privileges, and the password is only displayed upon first generation. It cannot be viewed again later, so please be sure to save it securely. +1️⃣ **When deploying v1.8.0 or later for the first time**, Nexent creates the `suadmin@nexent.com` super administrator account with the default password `Nexent@123`, without prompting, and displays it in the terminal after successful creation. Override it before the first deployment with `NEXENT_SUPER_ADMIN_PASSWORD` in `deploy/env/.env`; non-interactive creation displays the effective password. As an exception, an offline package launched with `--config` prompts for and confirms the password, and that input takes precedence without being displayed. > This account is used for permission management only and cannot develop agents or create knowledge bases. Log in with this account and complete: Access tenant resources → Create tenant → Create tenant administrator, then log in with the tenant administrator account to use all features. For role permissions, see [User Management](../user-guide/user-management). -2️⃣ Forgot to note the `suadmin` account password? Follow these steps: +2️⃣ To recreate the `suadmin` account, follow these steps: ```bash # Step 1: Delete su account record in supabase container @@ -93,12 +98,63 @@ docker exec -it nexent-postgresql bash psql -U root -d nexent delete from nexent.user_tenant_t where user_id = 'your_user_id'; -# Step 3: Redeploy and record the su account password +# Step 3: Redeploy; non-interactive mode uses the configured or default password +``` + +### Offline Deployment + +When the target host cannot access public image registries, download a prebuilt offline deployment package from GitHub Actions: + +1. Sign in to GitHub and open [Build Offline Deployment Package](https://github.com/ModelEngine-Group/nexent/actions/workflows/build-offline-package.yml). +2. Select a successful run for the required version and download the artifact matching the server architecture from **Artifacts** at the bottom of the run page. +3. Download `nexent--amd64.zip` for AMD64 or `nexent--arm64.zip` for ARM64. + +GitHub Actions artifacts are retained for 30 days. If the required artifact has expired, ask a maintainer to rerun the workflow. + +Copy the downloaded archive to the offline host and extract it. The downloaded artifact contains the package files directly, with no nested archive: + +```bash +unzip nexent-v2.2.1-amd64.zip -d nexent +cd nexent +bash deploy.sh --load-images docker ``` -### 3. Access Your Installation +The offline package installs all Nexent components by default. Add `--config` to reselect components, port policy, image source, or monitoring provider: + +```bash +bash deploy.sh --load-images --config docker +``` + +If the host still has a previously deployed offline package, use `--reuse-from` to reuse its environment configuration and deployment options: + +```bash +bash deploy.sh \ + --reuse-from /path/to/previous/nexent \ + --load-images \ + docker +``` + +The specified directory must be the root of an extracted previous package and contain `deploy/env/.env`. This option imports the old `.env`, preserves its values, and immediately appends variables newly introduced by the current package's `.env.example`. It also reuses `monitoring.env` and Docker `deploy.options` when present; the new scripts regenerate Docker-derived configuration. `--reuse-from` can be combined with `--config`, `--defaults`, or `--push-images`. + +When `suadmin@nexent.com` is created for the first time, non-interactive deployment uses `NEXENT_SUPER_ADMIN_PASSWORD`, which defaults to `Nexent@123`, and displays the effective password after successful creation. Offline deployment with `--config` prompts for and confirms the password; that input is neither persisted nor displayed. + +To push the packaged images to an internal registry accessible to the target environment: + +```bash +bash deploy.sh \ + --push-images \ + --image-registry-prefix registry.example.com/nexent \ + docker +``` + +When the prefix is omitted, the wrapper prompts for it. `push-images.sh` then prompts for the registry username and password before pushing. + +### Access Your Installation When deployment completes successfully: + +> **Get the administrator password**: The super administrator account is `suadmin@nexent.com`. On its first non-interactive creation, the terminal displays the effective password; when no value was configured, the default password is `Nexent@123`. For an offline deployment using `--config`, the manually entered password is neither saved nor displayed. If it is forgotten, recreate the account by following the earlier "Recreate the `suadmin` account" steps. + 1. Open **http://localhost:3000** in your browser 2. Log in with the super administrator account 3. Access tenant resources → Create tenant and tenant administrator @@ -175,38 +231,6 @@ bash uninstall.sh docker delete-all The Docker uninstall script reads `deploy/env/.env` to resolve `ROOT_DIR` and removes Compose resources. Data deletion removes service directories such as `postgresql`, `elasticsearch`, `redis`, `minio`, `volumes`, `openssh-server`, `scripts`, and `skills`; keep volumes when you plan to redeploy with existing data. -### Offline Image Package - -Use `deploy/offline/build_offline_package.sh` when you need to move images and deployment scripts to an offline host: - -```bash -bash deploy/offline/build_offline_package.sh \ - --target docker \ - --version v2.2.1 \ - --platform amd64 \ - --components infrastructure,application,data-process,supabase \ - --image-source general \ - --compress true \ - --output-dir offline-package -``` - -The package directory contains `images/*.tar`, `load-images.sh`, `push-images.sh`, `deploy.sh`, `uninstall.sh`, `manifest.yaml`, `checksums.txt`, `deploy/env/.env.example`, `deploy/env/monitoring.env.example`, and `deploy/sql`. It does not include local `deploy/env/.env`, `deploy/env/monitoring.env`, or `deploy.options`. With `--compress true`, a `nexent-offline---.zip` archive is created next to the output directory. - -On the target host, the package root `deploy.sh` uses saved `deploy.options` when present, otherwise built-in defaults, and does not open the TUI by default. Add `--config` to open the interactive configuration UI. If the package was built with a custom version, component set, port policy, or image source, pass the same options during deployment or use `--config` to select them interactively: - -```bash -cd offline-package -bash deploy.sh --load-images docker -``` - -To push packaged images to an internal registry and deploy with that prefix: - -```bash -bash deploy.sh --push-images --image-registry-prefix registry.example.com/nexent docker -``` - -When `--push-images` is used without a prefix, `deploy.sh` prompts for the image registry prefix first. `push-images.sh` then prompts for the registry username and password before pushing. - ## 🔌 Port Mapping | Service | Internal Port | External Port | Description | diff --git a/doc/docs/en/quick-start/kubernetes-installation.md b/doc/docs/en/quick-start/kubernetes-installation.md index af8db06b4f..930bba759e 100644 --- a/doc/docs/en/quick-start/kubernetes-installation.md +++ b/doc/docs/en/quick-start/kubernetes-installation.md @@ -14,7 +14,12 @@ ## 🚀 Quick Start -### 1. Prepare Kubernetes Cluster +- [Online Deployment](#online-deployment) +- [Offline Deployment](#offline-deployment) + +### Online Deployment + +#### 1. Prepare Kubernetes Cluster Ensure your Kubernetes cluster is running and kubectl is configured with cluster access: @@ -23,14 +28,14 @@ kubectl cluster-info kubectl get nodes ``` -### 2. Clone and Navigate +#### 2. Clone and Navigate ```bash git clone https://github.com/ModelEngine-Group/nexent.git cd nexent ``` -### 3. Deployment +#### 3. Deployment Run the deployment script: @@ -57,17 +62,17 @@ After running the command, the script opens Bash TUI menus for configuration. Us - **mainland**: uses mainland China mirrors - **local-latest**: uses local `latest` images and local-friendly pull policies for Nexent application images -Kubernetes uses the same `deploy/env/.env` file as Docker. Existing `deploy/env/.env` is kept as-is. If it does not exist, the deploy scripts first reuse `docker/.env`, then fall back to `deploy/env/.env.example`. +Kubernetes uses the same `deploy/env/.env` file as Docker. Before every deployment, existing values, comments, and old variables are preserved while variables newly introduced by the current `deploy/env/.env.example` are appended. If `.env` does not exist, the scripts first reuse legacy `docker/.env`, then fall back to the current template. A readable `.env.example` is required. Use `bash deploy.sh k8s --defaults` to skip the TUI and deploy with saved `deploy.options` or built-in defaults. After a successful deployment, non-sensitive choices are saved to `deploy/k8s/deploy.options`. The next interactive deployment can reuse the local config or run a full reconfiguration. -### ⚠️ Important Notes +#### ⚠️ Important Notes -1️⃣ **When deploying v1.8.0 or later for the first time**, you will be prompted to set a password for the `suadmin` super administrator account during the deployment process. This account has the highest system privileges. Please enter your desired password and **save it securely** after creation - it cannot be retrieved later. +1️⃣ **When deploying v1.8.0 or later for the first time**, Nexent creates the `suadmin@nexent.com` super administrator account with the default password `Nexent@123`, without prompting, and displays it in the terminal after successful creation. Override it before the first deployment with `NEXENT_SUPER_ADMIN_PASSWORD` in `deploy/env/.env`; non-interactive creation displays the effective password. An offline package launched with `--config` instead prompts for and confirms the password, and that input takes precedence without being displayed. -2️⃣ Forgot to note the `suadmin` account password? Follow these steps: +2️⃣ To recreate the `suadmin` account, follow these steps: ```bash # Step 1: Delete su account record in Supabase database @@ -83,14 +88,59 @@ kubectl exec -it -n nexent deploy/nexent-supabase-db -- psql -U postgres -c \ kubectl exec -it -n nexent deploy/nexent-postgresql -- psql -U root -d nexent -c \ "DELETE FROM nexent.user_tenant_t WHERE user_id='your_user_id';" -# Step 3: Re-deploy and record the su account password +# Step 3: Redeploy; non-interactive mode uses the configured or default password bash deploy.sh k8s ``` -### 4. Access Your Installation +### Offline Deployment + +When the target cluster cannot access public image registries, download a prebuilt offline deployment package from GitHub Actions: + +1. Sign in to GitHub and open [Build Offline Deployment Package](https://github.com/ModelEngine-Group/nexent/actions/workflows/build-offline-package.yml). +2. Select a successful run for the required version and download `nexent--.zip` matching the cluster node architecture from **Artifacts**. +3. Copy the archive to a management host that can access the target cluster and extract it. Workflow artifacts are retained for 30 days; if one has expired, ask a maintainer to rerun the workflow. + +Extract the offline deployment package: + +```bash +unzip nexent-v2.2.1-amd64.zip -d nexent +cd nexent +``` + +A single-node cluster backed by the Docker container runtime can load and deploy the images directly: + +```bash +bash deploy.sh --load-images k8s +``` + +If the management host still has a previously deployed offline package, use `--reuse-from` to reuse its environment configuration and Kubernetes deployment options: + +```bash +bash deploy.sh \ + --reuse-from /path/to/previous/nexent \ + --load-images \ + k8s +``` + +The specified directory must be the root of an extracted previous package and contain `deploy/env/.env`. This option imports the old `.env`, preserves its values, and immediately appends variables newly introduced by the current package's `.env.example`. It also reuses `monitoring.env` and Kubernetes `deploy.options` when present; the new scripts regenerate Helm generated values. `--reuse-from` can be combined with `--config`, `--defaults`, or `--push-images`. + +For other single-node and multi-node clusters, push images to an internal registry accessible to the cluster, or import them with the container runtime's tooling on every node that may run Nexent Pods: + +```bash +bash deploy.sh \ + --push-images \ + --image-registry-prefix registry.example.com/nexent \ + k8s +``` + +The offline package installs all Nexent components by default. Add `--config` to reselect deployment settings. When the super administrator is created for the first time, this mode prompts for and confirms the password without displaying or persisting it. Non-interactive deployment uses `NEXENT_SUPER_ADMIN_PASSWORD`, which defaults to `Nexent@123`, and displays the effective password after successful creation. + +### Access Your Installation When deployment completes successfully: +> **Get the administrator password**: The super administrator account is `suadmin@nexent.com`. On its first non-interactive creation, the terminal displays the effective password; when no value was configured, the default password is `Nexent@123`. For an offline deployment using `--config`, the manually entered password is neither saved nor displayed. If it is forgotten, recreate the account by following the earlier "Recreate the `suadmin` account" steps. + | Service | Default Address | |---------|-----------------| | Web Application | http://localhost:30000 | @@ -187,38 +237,6 @@ bash uninstall.sh k8s delete-all `--delete-data` and `--delete-volumes` are compatibility options for Helm-managed resources. For local disks, use `--delete-local-data` or `--keep-local-data`; `delete-all --keep-local-data` removes the namespace while preserving local volume contents. -### Offline Image Package - -Build a Kubernetes offline package from the repository root: - -```bash -bash deploy/offline/build_offline_package.sh \ - --target k8s \ - --version v2.2.1 \ - --platform amd64 \ - --components infrastructure,application,data-process,supabase \ - --image-source general \ - --compress true \ - --output-dir offline-package -``` - -The package includes image tar files, `load-images.sh`, `push-images.sh`, root deploy/uninstall entrypoints, Kubernetes Helm assets, SQL files, `deploy/env/.env.example`, `deploy/env/monitoring.env.example`, `manifest.yaml`, and `checksums.txt`. It does not include local `deploy/env/.env`, `deploy/env/monitoring.env`, or generated Helm values. With `--compress true`, a `nexent-offline---.zip` archive is created next to the output directory. - -On the target host, the package root `deploy.sh` uses saved `deploy.options` when present, otherwise built-in defaults, and does not open the TUI by default. Add `--config` to open the interactive configuration UI. If the package was built with a custom version, component set, port policy, or image source, pass the same options during deployment or use `--config` to select them interactively. On a single-node Docker-backed cluster, you can load and deploy directly: - -```bash -cd offline-package -bash deploy.sh --load-images k8s -``` - -For multi-node clusters, load the images on every node that may run Nexent Pods, or push the packaged images to an internal registry and deploy with matching image settings: - -```bash -bash deploy.sh --push-images --image-registry-prefix registry.example.com/nexent k8s -``` - -When `--push-images` is used without a prefix, `deploy.sh` prompts for the image registry prefix first. `push-images.sh` then prompts for the registry username and password before pushing. - ## 🔧 Deployment Commands ```bash diff --git a/doc/docs/en/quick-start/kubernetes-upgrade-guide.md b/doc/docs/en/quick-start/kubernetes-upgrade-guide.md index 83850aa40f..3d3305197d 100644 --- a/doc/docs/en/quick-start/kubernetes-upgrade-guide.md +++ b/doc/docs/en/quick-start/kubernetes-upgrade-guide.md @@ -41,7 +41,7 @@ bash deploy.sh k8s The script will detect your saved deployment settings (components, port policy, image source, etc.) from `deploy.options`. If the file is missing, you will be prompted to enter configuration details. > 💡 Tip -> If you need to configure voice models (STT/TTS), please edit the corresponding values in `values.yaml` or pass them via command line. +> Existing values, comments, ordering, and old-only variables in `deploy/env/.env` are preserved, while variables newly introduced by the current `deploy/env/.env.example` are appended automatically. A readable template is required before deployment starts. Generated Helm values are then recreated from the merged `.env`; do not edit them directly. Configure voice models (STT/TTS) in `deploy/env/.env`. --- diff --git a/doc/docs/en/quick-start/upgrade-guide.md b/doc/docs/en/quick-start/upgrade-guide.md index fe1032b268..b25942fc35 100644 --- a/doc/docs/en/quick-start/upgrade-guide.md +++ b/doc/docs/en/quick-start/upgrade-guide.md @@ -41,7 +41,7 @@ bash deploy.sh docker If deploy.options is missing, the script will prompt you to select deployment settings again, such as components, port policy, and image source. Choose the same options you used for the previous deployment. >💡 Tip -> Existing `deploy/env/.env` is kept as-is. If it is missing, the deploy script first reuses `docker/.env`, then falls back to `deploy/env/.env.example`. +> The upgrade keeps existing values, comments, ordering, and old-only variables in `deploy/env/.env`, then appends variables newly introduced by the current `deploy/env/.env.example`. If `.env` is missing, the script first reuses legacy `docker/.env`, then falls back to the current template. A readable `.env.example` is required before images are loaded or services are started. > If you need to configure voice models (STT/TTS), add the relevant variables to `deploy/env/.env`. We will provide a front-end configuration interface as soon as possible. diff --git a/doc/docs/en/security.md b/doc/docs/en/security.md index 693d1898b8..ce555298d0 100644 --- a/doc/docs/en/security.md +++ b/doc/docs/en/security.md @@ -3,7 +3,7 @@ **Please do not report security vulnerabilities through public GitHub issues, discussions, or other public channels.** Instead, please disclose them responsibly by contacting our security team at: -📧 [chenshuangrui@gmail.com](mailto:chenshuangrui@gmail.com) +📧 [zhenggaoqi@huawei.com](mailto:zhenggaoqi@huawei.com) ## What to Include: - Detailed description of the vulnerability diff --git a/doc/docs/en/user-guide/agent-development.md b/doc/docs/en/user-guide/agent-development.md index 8e6b47d4f4..1f4b312a95 100644 --- a/doc/docs/en/user-guide/agent-development.md +++ b/doc/docs/en/user-guide/agent-development.md @@ -1,462 +1,26 @@ # Agent Development -In the Agent Development page, you can create, configure, and manage agents. Agents are the core feature of Nexent—they can understand your needs and perform corresponding tasks. +Agent Development is the central workspace for building, configuring, and managing AI agents. Agents are the core of Nexent—they understand your needs and execute tasks on your behalf. -## 🔧 Create an Agent +## Quick Navigation -On the Agent Management tab, click "Create Agent" to create a new blank agent. Click "Exit Create" to leave creation mode. -If you have an existing agent configuration, you can also import it: +This module contains the following four configuration pages: -1. Click "Import Agent" -2. In the file selection dialog, select the agent configuration file (JSON format) -3. Click "Open"; the system will validate the file format and content, and display the imported agent information +| Page | Description | +| ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| [Model Configuration](./agent-development/model-configuration) | Connect and manage AI models, including LLMs, embedding models, vision-language models, rerank models, and speech models (TTS and STT) | +| [Knowledge Configuration](./agent-development/knowledge-configuration) | Create knowledge bases and upload documents in various formats, enabling agents to retrieve your private data | +| [Memory Configuration](./agent-development/memory-configuration) | Configure the multi-level memory system for cross-conversation knowledge retention and personalized service | +| [Agent Configuration](./agent-development/agent-configuration) | Create agents, configure collaborative agents, select tools, write prompts, then debug and publish | -
- -
+## Main Steps -> ⚠️ **Note:** If you import an agent with a duplicate name, a prompt dialog will appear. You can choose: -> - **Import anyway**: Keep the duplicate name; the imported agent will be in an unavailable state and requires manual modification of the Agent name and variable name before it can be used -> - **Regenerate and import**: The system will call the LLM to rename the Agent, which will consume a certain amount of model tokens and may take longer +1. **Configure Models** – Add the AI models you need in Model Management +2. **Prepare Knowledge Bases** – Create knowledge bases and upload relevant documents +3. **Configure Memory** – Enable memory features so agents can remember important information +4. **Build the Agent** – Create an agent, select tools and collaborative agents, write the business description +5. **Debug and Publish** – Test the agent's performance, save when satisfied, and publish -> 📌 **Important:** For agents created via import, if their tools include `knowledge_base_search` or other knowledge base search tools, these tools will only search **knowledge bases that the currently logged-in user is allowed to access in this environment**. The original knowledge base configuration in the exported agent will *not* be automatically inherited, so actual search results and answer quality may differ from what the original author observed. +## Need Help? -
- -
- -## 👥 Configure Collaborative Agents/Tools - -You can configure other collaborative agents for your created agent, as well as assign available tools to empower the agent to complete complex tasks. - -### 🤝 Collaborative Agents - -Collaborative agents help the current agent complete complex tasks. The sources of collaborative agents are divided into two categories: - -- **Internal Agents**: Published agents on the platform -- **External A2A Agents**: Third-party agents discovered through the A2A protocol - -1. Click the plus sign under the "Collaborative Agent" tab to open the selectable agent list -2. The agent list is divided into two tabs: "Internal Agent" and "External A2A Agent". You can choose based on your needs -3. Select the agent you want to add from the dropdown list -4. Multiple collaborative agents can be selected -5. Click × to remove an agent from the selection - -
- -
- -#### 🌐 Add External A2A Agents - -Nexent supports communication with third-party agents through the A2A protocol. You can discover external A2A agents in the following two ways: - -##### Discover Agent via URL - -If you know the Agent Card address of the target agent, you can use the URL discovery method: - -
- -
- -1. In the External A2A Agent list, click the "Add External Agent" button -2. Select the "URL Discovery" tab -3. Fill in the Agent Card URL address, for example: `https://example.com/.well-known/agent.json` -4. Click the "Discover" button; the system will automatically retrieve the agent's related information -5. After successful discovery, you can view the agent's name, description, capabilities and other information -6. Click "Add to List" to complete the addition - -> 💡 **Tip**: The Agent Card is an Agent description file that complies with the A2A 1.0 specification, containing the agent's name, description, calling address, capabilities and other information. - -##### Discover Agent via Nacos - -If your agent is registered with the Nacos service discovery platform, you can use the Nacos discovery method: - -
- -
- -1. In the External A2A Agent list, click the "Add External Agent" button -2. Select the "Nacos Discovery" tab -3. For first-time use, you need to configure the Nacos connection information: - - **Nacos Server Address**: Fill in the Nacos server address, such as `http://127.0.0.1:8848` - - **Namespace ID**: Fill in the Nacos namespace ID (optional) - - **Group Name**: Fill in the service group name, default is `DEFAULT_GROUP` - - **Username/Password**: Fill in the Nacos access credentials (optional) -4. Click "Save Configuration" to save the Nacos connection information -5. Fill in the Agent service name to scan -6. Click the "Scan" button; the system will obtain matching Agent information from Nacos -7. The scan results will list all matching Agents. You can select the agents you need and add them to the list - -> ⚠️ **Note**: Make sure the Nacos service is running properly and the target Agent is correctly registered with Nacos. - -##### Manage Discovered External Agents - -In the External A2A Agent list, you can view and manage all discovered external agents: - -
- -
- -1. **View Agent Details**: Click on the agent card to view its complete information, including name, description, URL, capability list, etc. -2. **Test Agent**: Click the "Test" button to send a test message to the agent and verify if it is working properly -3. **Chat with Agent**: Click the "Chat" button to open a chat window and interact with the agent in real time -4. **Configure Calling Protocol**: Click the "Protocol Configuration" button to select the calling protocol for this agent: - - **HTTP + JSON**: Use REST API style calls - - **JSON-RPC**: Use JSON-RPC protocol calls -5. **Refresh Agent Information**: If the agent information changes, click the "Refresh" button to re-fetch the latest Agent Card -6. **Remove Agent**: Click the "Remove" button to delete the agent from the discovered list - -> 💡 **Use Cases**: -> - Quickly integrate known third-party agent services through URL discovery -> - Batch integrate all agents from the same service registry through Nacos discovery -> - Configure protocols to meet the requirements of different agent service providers - -###### Integrate [DataAgent](https://gitcode.com/datagallery/dataagent) A2A Agent via URL - -1. Refer to the [DataAgent documentation](https://gitcode.com/datagallery/dataagent#%F0%9F%8C%90-a2a-10-%E6%9C%8D%E5%8A%A1%E6%A8%A1%E5%BC%8F) and start DataAgent in A2A service mode. - > Nexent does not currently support agents that require authentication. Do not set `auth-token` when starting DataAgent. - -
- -
- -2. Refer to [Discover Agent via URL](#discover-agent-via-url) to integrate the agent. The URL is `http://:9999/.well-known/agent-card.json`. -3. Refer to [Manage Discovered External Agents](#manage-discovered-external-agents) to configure the invocation protocol, and select HTTP + JSON for integration. - -### 🛠️ Select Agent Tools - -Agents can use various tools to complete tasks, such as knowledge base search, file parsing, image parsing, email sending/receiving, file management, and other local tools. They can also integrate third-party MCP tools or custom tools. - -1. On the "Select Tools" tab, click "Refresh Tools" to update the available tool list -2. Select the group containing the tool you want to add -3. View all available tools under the group; click ⚙️ to view tool details and configure parameters -4. Click the tool name to select/deselect it - - If the tool has required parameters that are not configured, a popup will appear to guide you through parameter configuration - - If all required parameters are already configured, the tool will be selected directly - -
- -
- -> 💡 **Tips**： -> 1. Please select the `knowledge_base_search` tool to enable the knowledge base search function. -> 2. Please select the `analyze_text_file` tool to enable the parsing function for document and text files. -> 3. Please select the `analyze_image` tool to enable the parsing function for image files. -> -> ⚠️ **Embedding Model Configuration**: When using the `knowledge_base_search` tool, ensure that the knowledge base has an embedding model configured. For existing knowledge bases, the system will prompt you to select an embedding model. Make sure to select **the same embedding model used when creating the knowledge base**. If the selected model differs from the one used during knowledge base creation, it may cause search failures or inaccurate results. -> -> 📚 Want to learn about all the built-in local tools available in the system? Please refer to [Local Tools Overview](./local-tools/index.md). - -### 🔌 Add MCP Tools - -On the "Select Agent Tools" tab, click "MCP Config" to configure MCP servers in the popup and view configured servers. - -You can add MCP services to Nexent in the following two ways: - -**1️⃣ Add MCP Service via URL** - -🔔 This method is suitable for independently deployed MCP services (supports SSE and Streamable HTTP protocols): - ->1. In the **Add MCP Server** section at the top of the interface, fill in **Server name** and **Server URL** -> ->⚠️ **Note:** The server name must contain only English letters or digits; spaces, underscores, and other characters are not allowed. -> ->2. Click the **+ Add** button on the right to complete adding a single service - -**2️⃣ Add Containerized MCP Service via JSON Configuration** - -🔔 This method is suitable for containerized MCP services deployed via npx: - ->1. In the **Add Containerized MCP Service** input box, fill in a JSON configuration that matches the example format: -> ->```json ->{ -> "mcpServers": { -> "service-name": { -> "args": [ -> "mcp-package-name@version", -> "additional-parameters" -> ], -> "command": "npx" -> } -> } ->} ->``` -> ->2. In the **Port** input box below, enter the port number corresponding to the containerized service ->3. Click the **+ Add** button on the right to complete adding the containerized service - -
- -
- -Many third-party services such as [ModelScope](https://www.modelscope.cn/mcp) provide MCP services, which you can quickly integrate and use. -You can also develop your own MCP services and connect them to Nexent; see [MCP Tool Development](../backend/tools/mcp). - -**3️⃣ Convert Stock API to MCP Service** - -🔔 This method is suitable for quickly converting existing REST API endpoints into MCP tools without additional development, allowing agents to call existing API capabilities: - ->1. In the MCP Config module, select **"API to MCP"** as the access type -> ->2. Fill in the API basic information in the input box below: -> - **Service Name**: Display name for the MCP service -> - **OpenAPI JSON**: OpenAPI 3.x specification in JSON format -> - **Base Service URL**: Base address of the API service (supports http/https) -> ->3. Click the **+ Add** button in the lower right corner to complete the MCP service conversion - -
- -
- ->4. After conversion, you can view all externally converted MCP tools in the **Outer APIs** tab - -
- -
- -
- -
- ->💡 **Use Cases**: ->- Quickly integrate internal enterprise REST API endpoints ->- Convert third-party service HTTP APIs into MCP tools ->- Generate tools directly from OpenAPI specifications without writing MCP Server code - - -### ⚙️ Custom Tools - -You can refer to the following guides to develop your own tools and integrate them into Nexent to enrich agent capabilities: - -- [LangChain Tools Guide](../backend/tools/langchain) -- [MCP Tool Development](../backend/tools/mcp) -- [SDK Tool Documentation](../sdk/core/tools) - -### 🧪 Tool Testing - -Nexent provides a "Tool Testing" capability for all types of tools—whether they are built-in tools, externally integrated MCP tools, or custom-developed tools. If you are unsure about a tool's effectiveness when creating an agent, you can use the testing feature to verify that the tool works as expected. - -1. Click the gear icon ⚙️ next to the tool to open the tool's detailed configuration popup -2. First, ensure that all required parameters (marked with red asterisks) are configured -3. Click the "Test Tool" button in the lower left corner of the popup -4. A new test panel will appear on the right side -5. Enter the tool's input parameters in the test panel. For example: - - When testing the local knowledge base search tool `knowledge_base_search`, you need to enter: - - The test `query`, such as "benefits of vitamin C" - - The search `search_mode` (default is `hybrid`) - - The target index list `index_names`, such as `["Medical", "Vitamin Encyclopedia"]` - - If `index_names` is not entered, it will default to searching all knowledge bases selected on the knowledge base page -6. After entering the parameters, click "Execute Test" to start the test and view the test results below - -
- -
- -## 📝 Describe Business Logic - -### ✍️ Describe How the Agent Should Work - -Based on the selected collaborative agents and tools, you can now describe in simple language how you expect this agent to work. Nexent will automatically generate the agent name, description, and prompts based on your configuration and description. - -1. In the editor under "Describe how should this agent work", enter a brief description, such as "You are a professional knowledge Q&A assistant with local knowledge search and online search capabilities, synthesizing information to answer user questions" -2. Select a model (choose a smarter model when generating prompts to optimize response logic), click the "Generate Agent" button, and Nexent will generate detailed agent content for you, including basic information and prompts (role, usage requirements, examples) -3. You can edit and fine-tune the auto-generated content (including agent information and prompts) in the Agent Detail Content below - -#### 📋 Agent Basic Information Configuration - -In the basic information section, if you are not satisfied of the auto-generated content, you can configure the following fields by your own: - -| Field | Description | -|-------|-------------| -| **Agent Name** | The display name shown in the interface and recognized by users. | -| **Agent Variable Name** | The internal identifier for the agent, used to reference it in code. Can only contain letters, numbers, and underscores, and must start with a letter or underscore. | -| **Author** | The creator of the agent. Defaults to the current logged-in user's email. | -| **User Group** | The user group the agent belongs to, used for permission management and organization. If empty, the agent has no assigned user group. | -| **Group Permission** | Controls how users in the same group can access this agent:
- **Editable**: Group members can view and edit the agent
- **Read-only**: Group members can only view, not edit
- **Private**: Only the creator and administrators can access | -| **Model** | The LLM used by the agent for reasoning and generating responses. | -| **Max Steps of Agent Run** | The maximum number of think-act cycles the agent can execute in a single conversation. More steps allow the agent to handle more complex tasks, but also consume more resources. | -| **Provide Run Summary** | Controls whether the agent provides run details to the main agent when used as a sub-agent:
- **Enabled (default)**: When used as a sub-agent, provides a detailed run summary to the main agent
- **Disabled**: When used as a sub-agent, only returns the final result without detailed run information | -| **Description** | A description of the agent's functionality, explaining its purpose and capabilities. | - -> 💡 **Usage Suggestions**: -> - Use meaningful English names for the agent variable name, such as `code_assistant`, `data_analyst`, etc. -> - Set the max steps based on task complexity: 3-5 steps for simple Q&A, 10-20 steps for complex reasoning tasks -> - Keep "Provide Run Summary" enabled if the sub-agent's run process is valuable for the main agent's decision-making. Disable it if you only need the final result to reduce context consumption. - -
- -
- -### 🐛 Debug and Save - -After completing the initial agent configuration, you can debug the agent and fine-tune the prompts based on the debugging results to continuously improve agent performance. - -1. Click the "Debug" button in the lower right corner of the page to open the agent debug page -2. Test conversations with the agent and observe its responses and behavior -3. Review conversation performance and error messages, and optimize the agent prompts based on the test results - -After successful debugging, click the "Save" button in the lower right corner, and the agent will be saved and appear in the agent list. - -## 📋 Version Management - -Nexent supports agent version management. You can save different versions of agent configurations during the debugging process. - -Once the agent configuration is verified, you can publish the agent. After publishing, the agent will be visible in the Agent Space and Start Chat pages. - -![Version Management 1](./assets/agent-development/version_management_1.png) - -If you need to rollback to a previous version, click the "Rollback" button on the version management page. - -![Version Management 2](./assets/agent-development/version_management_2.png) - -### 🚀 Publish as A2A Agent - -Nexent supports exposing published agents as A2A Agents for external systems to call. When publishing a version, you can check the "Publish as A2A Agent" option to register the current agent as an A2A 1.0 compliant Agent. - -
- -
- -After successful publishing, the system will display the A2A Agent's call information: - -
- -
- -| Field | Description | -|-------|-------------| -| **Endpoint ID** | Unique identifier for the A2A Agent | -| **Agent Card URL** | Agent discovery endpoint; external systems use this address to retrieve Agent descriptions | -| **Protocol Version** | A2A protocol version; currently 1.0 | -| **REST Endpoints** | REST-style API endpoints | -| **JSON-RPC Endpoint** | JSON-RPC 2.0 protocol calling endpoint | - -#### Calling Methods - -The published A2A Agent supports the following two calling protocols: - -##### REST API - -```bash -# Get Agent Card (for Agent discovery) -GET /nb/a2a/{endpoint_id}/.well-known/agent-card.json - -# Send synchronous message -POST /nb/a2a/{endpoint_id}/message:send -Content-Type: application/json - -{ - "message": { - "role": "user", - "content": "Please help me complete a task" - } -} - -# Send streaming message (SSE) -POST /nb/a2a/{endpoint_id}/message:stream -Content-Type: application/json - -{ - "message": { - "role": "user", - "content": "Please help me complete a task" - } -} - -# Get task status -GET /nb/a2a/{endpoint_id}/tasks/{task_id} -``` - -##### JSON-RPC 2.0 - -```bash -POST /nb/a2a/{endpoint_id}/v1 -Content-Type: application/json - -# Send synchronous message -{ - "jsonrpc": "2.0", - "method": "SendMessage", - "params": { - "message": { - "role": "user", - "content": "Please help me complete a task" - } - }, - "id": 1 -} - -# Send streaming message -{ - "jsonrpc": "2.0", - "method": "SendStreamingMessage", - "params": { - "message": { - "role": "user", - "content": "Please help me complete a task" - } - }, - "id": 2 -} - -# Get task status -{ - "jsonrpc": "2.0", - "method": "GetTask", - "params": { - "taskId": "task_abc123" - }, - "id": 3 -} -``` - -> 💡 **Tips**: -> - For local development, replace the `/nb/a2a` prefix with `http://localhost:5013/nb/a2a` -> - For production environments, replace the prefix with your server domain name or public IP address - -> ⚠️ **Notes**: -> - Calling A2A Agents requires carrying valid authentication information in the request headers -> - Agent Card information is cached with a refresh interval of 1 hour -> - If you need to update Agent information, you need to republish the agent version - -When an agent is published as an A2A-compliant Agent, users can view the detailed A2A Agent calling information by clicking the button shown below in the agent list: - -
- -
- -## 📋 Manage Agents - -In the agent list on the left, you can perform the following operations on existing agents: - -### 🔗 View Call Relationships - -View the collaborative agents/tools used by the agent, displayed in a tree diagram to clearly see the agent call relationships. - -
- -
- -### 📤 Export - -Export successfully debugged agents as JSON configuration files. You can use this JSON file to create a copy by importing it when creating an agent. - -### 📋 Copy - -Copy an agent to facilitate experimentation, multi-version debugging, and parallel development. - -### 🗑️ Delete - -Delete an agent (this cannot be undone, please proceed with caution). - -## 🚀 Next Steps - -After completing agent development, you can: - -1. View and manage all agents in **[Agent Space](./agent-space)** -2. Interact with agents in **[Start Chat](./start-chat)** -3. Configure **[Memory Management](./memory-management)** to enhance the agent's personalization capabilities - -If you encounter any issues during agent development, please refer to our **[FAQ](../quick-start/faq)** or ask for support in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). +If you encounter any issues, refer to our **[FAQ](../quick-start/faq)** or ask in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). diff --git a/doc/docs/en/user-guide/agent-development/agent-configuration.md b/doc/docs/en/user-guide/agent-development/agent-configuration.md new file mode 100644 index 0000000000..e7884a489b --- /dev/null +++ b/doc/docs/en/user-guide/agent-development/agent-configuration.md @@ -0,0 +1,581 @@ +# Agent Development + +In the Agent Development page, you can create, configure, and manage agents. Agents are the core feature of Nexent—they can understand your needs and perform corresponding tasks. + +## 🔧 Create an Agent + +On the Agent Management tab, click **New** to create a blank agent. Click **Exit Create** to leave creation mode. +If you have an existing agent configuration, you can also import it: + +1. Click **Import**. +2. In the file selection dialog, select an agent configuration file in JSON or ZIP format. +3. Click "Open"; the system will validate the file format and content, and display the imported agent information + +![Import an agent](./../assets/agent-development/import.png) + +![Imported agent information](./../assets/agent-development/import-2.png) + +> ⚠️ **Note:** If you import an agent with a duplicate name, a prompt dialog will appear. You can choose: +> +> - **Import anyway**: Keep the duplicate name; the imported agent will be in an unavailable state and requires manual modification of the Agent name and variable name before it can be used +> - **Regenerate and import**: The system will call the LLM to rename the Agent, which will consume a certain amount of model tokens and may take longer + +> 📌 **Important:** For agents created via import, if their tools include `knowledge_base_search` or other knowledge base search tools, these tools will only search **knowledge bases that the currently logged-in user is allowed to access in this environment**. The original knowledge base configuration in the exported agent will _not_ be automatically inherited, so actual search results and answer quality may differ from what the original author observed. + +
+ +
+ +## 👥 Configure Collaborative Agents/Tools + +You can configure other collaborative agents for your created agent, as well as assign available tools to empower the agent to complete complex tasks. + +### 🤝 Collaborative Agents + +Collaborative agents help the current agent complete complex tasks. The sources of collaborative agents are divided into two categories: + +- **Internal Agents**: Published agents on the platform +- **External A2A Agents**: Third-party agents discovered through the A2A protocol + +1. Click the plus sign under the "Collaborative Agent" tab to open the selectable agent list +2. The agent list is divided into two tabs: "Internal Agent" and "External A2A Agent". You can choose based on your needs +3. Select the agent you want to add from the dropdown list +4. Multiple collaborative agents can be selected +5. Click × to remove an agent from the selection + +
+ +
+ +#### 🌐 Add External A2A Agents + +Nexent supports communication with third-party agents through the A2A protocol. You can discover external A2A agents in the following two ways: + +##### Discover Agent via URL + +If you know the Agent Card address of the target agent, you can use the URL discovery method: + +
+ +
+ +1. In the External A2A Agent list, click the "Add External Agent" button +2. Select the "URL Discovery" tab +3. Fill in the Agent Card URL address, for example: `https://example.com/.well-known/agent.json` +4. If the target Agent Card requires authentication, enter a JSON object in "Custom Request Headers", for example: `{"Authorization": "Bearer "}` +5. Click the "Discover" button; the system will automatically retrieve the agent's related information +6. After successful discovery, you can view the agent's name, description, capabilities and other information +7. Click "Add to List" to complete the addition + +> 💡 **Tip**: Custom request headers are saved with the external agent and used only to retrieve and refresh its Agent Card. They are never used for agent calls. When rediscovering the same URL, leaving this field empty keeps the current configuration; entering `{}` clears it. + +> 💡 **Tip**: The Agent Card is an Agent description file that complies with the A2A 1.0 specification, containing the agent's name, description, calling address, capabilities and other information. + +##### Discover Agent via Nacos + +If your agent is registered with the Nacos service discovery platform, you can use the Nacos discovery method: + +
+ +
+ +1. In the External A2A Agent list, click the "Add External Agent" button +2. Select the "Nacos Discovery" tab +3. For first-time use, you need to configure the Nacos connection information: + - **Nacos Server Address**: Fill in the Nacos server address, such as `http://127.0.0.1:8848` + - **Namespace ID**: Fill in the Nacos namespace ID (optional) + - **Group Name**: Fill in the service group name, default is `DEFAULT_GROUP` + - **Username/Password**: Fill in the Nacos access credentials (optional) +4. Click "Save Configuration" to save the Nacos connection information +5. Fill in the Agent service name to scan +6. Click the "Scan" button; the system will obtain matching Agent information from Nacos +7. The scan results will list all matching Agents. You can select the agents you need and add them to the list + +> ⚠️ **Note**: Make sure the Nacos service is running properly and the target Agent is correctly registered with Nacos. + +##### Manage Discovered External Agents + +In the External A2A Agent list, you can view and manage all discovered external agents: + +
+ +
+ +1. **View Agent Details**: Click on the agent card to view its complete information, including name, description, URL, capability list, etc. +2. **Test Agent**: Click the "Test" button to send a test message to the agent and verify if it is working properly +3. **Chat with Agent**: Click the "Chat" button to open a chat window and interact with the agent in real time +4. **Configure Calling Protocol**: Click the "Protocol Configuration" button to select the calling protocol for this agent: + - **HTTP + JSON**: Use REST API style calls + - **JSON-RPC**: Use JSON-RPC protocol calls +5. **Configure Call Authentication**: If the Agent Card declares `securitySchemes` and `securityRequirements`, click "Agent Authentication" and enter the required values. Nexent places each value in the header, query string, or cookie specified by the Card; fields in the same requirement must all be configured. +6. **Refresh Agent Information**: If the agent information changes, click the "Refresh" button to re-fetch the latest Agent Card +7. **Remove Agent**: Click the "Remove" button to delete the agent from the discovered list + +> 💡 **Use Cases**: +> +> - Quickly integrate known third-party agent services through URL discovery +> - Batch integrate all agents from the same service registry through Nacos discovery +> - Configure protocols to meet the requirements of different agent service providers + +###### Integrate [DataAgent](https://gitcode.com/datagallery/dataagent) A2A Agent via URL + +1. Refer to the [DataAgent documentation](https://gitcode.com/datagallery/dataagent#%F0%9F%8C%90-a2a-10-%E6%9C%8D%E5%8A%A1%E6%A8%A1%E5%BC%8F) and start DataAgent in A2A service mode. + > Nexent does not currently support agents that require authentication. Do not set `auth-token` when starting DataAgent. + +
+ +
+ +2. Refer to [Discover Agent via URL](#discover-agent-via-url) to integrate the agent. The URL is `http://:9999/.well-known/agent-card.json`. +3. Refer to [Manage Discovered External Agents](#manage-discovered-external-agents) to configure the invocation protocol, and select HTTP + JSON for integration. + +### 🛠️ Select Agent Tools or Skills + +Agents can use tools and skills to complete tasks, including local capabilities such as knowledge base search, file parsing, image parsing, email, and file management. You can also integrate third-party or custom MCP tools and skills. + +1. On the "Select Tools" tab, click "Refresh Tools" to update the available tool list +2. Click **Select Tools** or **Select Skills** to browse the available tools or skills by tag +3. Click ⚙️ to view the tool or skill description and configure its parameters +4. Select the tool or skill. You can remove it later from the selected tools or selected skills area + - If the tool has required parameters that are not configured, a popup will appear to guide you through parameter configuration + - If all required parameters are already configured, the tool will be selected directly + +![Select tools or skills](./../assets/agent-development/set-tools-1.png) + +![Available tools and skills](./../assets/agent-development/set-tools-2.png) + +> 💡 **Tips**： +> +> 1. Please select the `knowledge_base_search` tool to enable the knowledge base search function. +> 2. Please select the `analyze_text_file` tool to enable the parsing function for document and text files. +> 3. Please select the `analyze_image` tool to enable the parsing function for image files. +> +> ⚠️ **Note:** Before using `knowledge_base_search`, create a knowledge base and ensure that **the embedding model used to create it** matches the currently active embedding model. Otherwise, retrieval may fail or return inaccurate results. +> +> 📚 Want to learn about all the built-in local tools available in the system? Please refer to [Local Tools Overview](../local-tools/index.md). +> 📚 Want to learn more about skills? Please refer to [Skill Management](../resource-repository/skill-repository.md). + +### 🔌 Add MCP Tools + +On the "Select Agent Tools" tab, click "MCP Config" to configure MCP servers in the popup and view configured servers. + +You can add MCP services to Nexent in the following two ways: + +**1️⃣ Add MCP Service via URL** + +🔔 This method is suitable for independently deployed MCP services (supports SSE and Streamable HTTP protocols): + +> 1. In the **Add MCP Server** section at the top of the interface, fill in **Server name** and **Server URL** +> +> ⚠️ **Note:** The server name must contain only English letters or digits; spaces, underscores, and other characters are not allowed. +> +> 2. Click the **+ Add** button on the right to complete adding a single service + +**2️⃣ Add Containerized MCP Service via JSON Configuration** + +🔔 This method is suitable for containerized MCP services deployed via npx: + +> 1. In the **Add Containerized MCP Service** input box, fill in a JSON configuration that matches the example format: +> +> ```json +> { +> "mcpServers": { +> "service-name": { +> "args": ["mcp-package-name@version", "additional-parameters"], +> "command": "npx" +> } +> } +> } +> ``` +> +> 2. In the **Port** input box below, enter the port number corresponding to the containerized service +> 3. Click the **+ Add** button on the right to complete adding the containerized service + +
+ +
+ +Many third-party services such as [ModelScope](https://www.modelscope.cn/mcp) provide MCP services, which you can quickly integrate and use. +You can also develop your own MCP services and connect them to Nexent; see [MCP Tool Development](../../backend/tools/mcp). + +**3️⃣ Convert Stock API to MCP Service** + +🔔 This method is suitable for quickly converting existing REST API endpoints into MCP tools without additional development, allowing agents to call existing API capabilities: + +> 1. In the MCP Config module, select **"API to MCP"** as the access type +> 2. Fill in the API basic information in the input box below: +> +> - **Service Name**: Display name for the MCP service +> - **OpenAPI JSON**: OpenAPI 3.x specification in JSON format +> - **Base Service URL**: Base address of the API service (supports http/https) +> +> 3. Click the **+ Add** button in the lower right corner to complete the MCP service conversion + +
+ +
+ +> 4. After conversion, you can view all externally converted MCP tools in the **Outer APIs** tab + +
+ +
+![Converted MCP tools](./../assets/agent-development/add_mcp_from_api_2.png) + +> 💡 **Use Cases**: +> +> - Quickly integrate internal enterprise REST API endpoints +> - Convert third-party service HTTP APIs into MCP tools +> - Generate tools directly from OpenAPI specifications without writing MCP Server code + +### ⚙️ Custom Tools + +You can refer to the following guides to develop your own tools and integrate them into Nexent to enrich agent capabilities: + +- [LangChain Tools Guide](../../backend/tools/langchain) +- [MCP Tool Development](../../backend/tools/mcp) +- [SDK Tool Documentation](../../sdk/core/tools.md) + +### 🔌 Create or Import Skills + +In the agent's advanced configuration, switch to the **Select Skills** tab and click **Build Skill**. You can create a skill through an interactive conversation or install one from a file. A successfully created skill is added to the list available to the current user, but it is **not automatically associated with the current agent**. You must select the skill and save the agent configuration afterward. + +#### Create a Skill Interactively (NL2SKILL) + +Interactive creation is suitable for building a skill from a natural-language requirement: + +1. Click **Build Skill**, then select the **Interactive Creation** tab. +2. In the conversation area on the left, describe the skill's purpose, execution steps, inputs, outputs, and constraints. For example: "Create a skill that reads a CSV file and produces a data quality report." +3. The system streams a draft skill. You can stop generation at any time or continue the conversation to add requirements and revise the existing draft. +4. Review and edit the skill information and files in the draft area on the right: + - **Skill Name:** Required and must be unique. + - **Skill Description:** Required; explains the skill's purpose and applicable scenarios. + - **Tags:** Add up to five tags, with no more than 20 characters per tag, to make the skill easier to search and filter. + - **User Groups and Group Permissions:** Configure skill visibility and editing permissions as needed. These settings are shown or editable only when the current account has the required permissions. + - **Skill Files:** `SKILL.md` is the main skill file and cannot be renamed or deleted. You can add, edit, rename, or delete scripts, assets, and other supporting files. +5. After reviewing the draft, click **Create**. If the skill name already exists, change it and try again. + +#### Install a Skill from a File + +Use the **Install** tab to import a prepared skill file. Click the upload area to select a file, or drag a file into it. Only one file can be uploaded at a time, in one of the following formats: + +| File Format | Use Case | Requirements | +| --- | --- | --- | +| `.md` | A single-file skill containing only its main instructions | The file must be a complete `SKILL.md` with `name` and `description` in its YAML Front Matter | +| `.zip` | A multi-file skill containing scripts, assets, or other supporting files | The archive must contain `SKILL.md`, either at its root or in a subdirectory; other files are imported with the skill | + +A basic `SKILL.md` looks like this: + +```markdown +--- +name: csv-report +description: Analyze CSV files and generate data quality reports +tags: + - data-analysis +--- + +# CSV Data Quality Report + +After the user provides a CSV file, check missing values, duplicate records, and field types, then produce a structured report. +``` + +After upload, the system reads the skill name and description from `SKILL.md` and displays the parsed result. Confirm the information and click **Create** to complete the installation. + +> ⚠️ **Import Restrictions:** +> +> - `SKILL.md` must contain valid YAML Front Matter with both `name` and `description`. Missing either field causes the import to fail. +> - `SKILL.md` must use UTF-8 encoding. +> - Importing does not overwrite an existing skill with the same name. Change `name` in `SKILL.md`, then upload the file again. +> - For a multi-file skill, compress the skill directory as a `.zip` file and ensure the archive contains `SKILL.md`. + +#### Associate the New Skill with the Agent + +After creating or installing a skill, associate it with the current agent as follows: + +1. If the list has not updated, click **Refresh Skills**. +2. Click **Select Skills**, then find and select the new skill by name, description, or tag. +3. If the skill has parameters that require values, click ⚙️ to configure them. +4. Return to the agent configuration page and save the configuration. The agent can use the skill only after these steps are complete. + +For complete instructions on viewing, editing, sharing, and deleting skills, see [Skill Management](../resource-repository/skill-repository.md). + +### 🧪 Tool Testing + +Nexent provides a "Tool Testing" capability for all types of tools—whether they are built-in tools, externally integrated MCP tools, or custom-developed tools. If you are unsure about a tool's effectiveness when creating an agent, you can use the testing feature to verify that the tool works as expected. + +1. Click the gear icon ⚙️ next to the tool to open the tool's detailed configuration popup +2. First, ensure that all required parameters (marked with red asterisks) are configured +3. Click the "Test Tool" button in the lower left corner of the popup +4. A new test panel will appear on the right side +5. Enter the tool's input parameters in the test panel. For example: + - When testing the local knowledge base search tool `knowledge_base_search`, you need to enter: + - The test `query`, such as "benefits of vitamin C" + - The search `search_mode` (default is `hybrid`) + - The target index list `index_names`, such as `["Medical", "Vitamin Encyclopedia"]` + - If `index_names` is not entered, it will default to searching all knowledge bases selected on the knowledge base page +6. After entering the parameters, click "Execute Test" to start the test and view the test results below + +
+ +
+ +## 📝 Describe Business Logic + +### ✍️ Describe How the Agent Should Work + +Based on the selected collaborative agents and tools, you can now describe in simple language how you expect this agent to work. Nexent will automatically generate the agent name, description, and prompts based on your configuration and description. + +1. In the editor under "Describe how should this agent work", enter a brief description, such as "You are a professional knowledge Q&A assistant with local knowledge search and online search capabilities, synthesizing information to answer user questions" +2. Select a model (choose a smarter model when generating prompts to optimize response logic), click the "Generate Agent" button, and Nexent will generate detailed agent content for you, including basic information and prompts (role, usage requirements, examples) +3. You can edit and fine-tune the auto-generated content (including agent information and prompts) in the Agent Detail Content below + +#### 📋 Agent Basic Information Configuration + +In the basic information section, if you are not satisfied of the auto-generated content, you can configure the following fields by your own: + +| Field | Description | +| --- | --- | +| **Agent Name** | The display name shown in the interface and used to identify the agent. | +| **Agent Variable Name** | The internal identifier used to reference the agent. It can contain only letters, numbers, and underscores, and must begin with a letter or underscore. | +| **Large Language Model** | The model the agent uses for reasoning, tool calls, and response generation. | +| **Agent Description** | Describes the agent's purpose and capabilities. | + +> 💡 **Usage Suggestions**: +> +> - Use meaningful English variable names that are easy for the model to understand, such as `code_assistant` or `data_analyst`. + +![Agent basic information](./../assets/agent-development/generate_agent.png) + +#### ⚙️ Advanced Settings + +Click **Advanced Settings** on the right side of **Agent Details** to further configure the agent's runtime behavior, permissions, self-verification, and safety guardrails. Advanced Settings contains **Basic Settings** and **Safety Guardrails** tabs. After making changes, click **OK** in the dialog and then save the agent for the settings to take effect. + +![Advanced settings](./../assets/agent-development/agent-settings.png) + +##### Basic Settings + +| Setting | Default | Description | +| --- | --- | --- | +| **Author** | Current user | The author name of the agent. | +| **Main Agent** | Yes | Controls whether the agent is displayed as a main agent that can be used for independent conversations. If set to **No**, the agent is better suited for use as a collaborative agent and does not appear in the list of main agents available for starting a conversation, even after publication. | +| **User Groups** | None | One or more user groups that the agent belongs to, used for organization and permission management. Only users with the required permissions can modify this setting. | +| **Group Permission** | Read-only | Controls access for users in the same group: **Editable** allows group members to view and edit; **Read-only** allows viewing only; **Private** limits access to the creator and administrators. | +| **Maximum Agent Run Steps** | 15 | The maximum number of think-act cycles allowed in a single run. It must be an integer of at least 1. When the limit is reached, the system stops further execution and summarizes the completed work. More steps support more complex tasks but increase time and resource consumption. | +| **Provide Run Summary** | No | Applies only when the agent is invoked by a main agent as a collaborative agent. **Yes** appends a summary of the work process to the final result; **No** returns only the final result and reduces context usage by the main agent. | +| **Output Reserve** | Model default | Limits the maximum number of output tokens per response and reserves that space in the model's context window. A larger value allows longer responses but leaves less room for input and history and triggers context compression earlier. A smaller value preserves more input but may truncate the response. The value must be a positive integer and cannot exceed the selected model's maximum output tokens. | +| **Self-Verification** | No | When enabled, the system checks key execution events and the final response. If it finds issues with tool calls, retrieval evidence, code execution, or answer quality, it asks the agent to correct or retry. If the final response repeatedly fails verification, the system returns a controlled explanation instead of an unverified definitive conclusion. | + +> 💡 **Configuration Tips:** +> +> - Use 3–5 maximum steps for simple Q&A and 10–20 for complex retrieval or reasoning tasks, then adjust based on debugging results. +> - Enable **Provide Run Summary** only when the main agent needs details about the collaborative agent's execution. Otherwise, leave it disabled to save context. +> - Normally, leave **Output Reserve** empty to use the model default. Adjust it only when responses are frequently truncated or when more space is needed for conversation history. + +##### 🚧 Safety Guardrails + +Safety guardrails use sequential regular-expression rules to inspect content sent to the model and data involved in tool calls. Guardrails are disabled by default and operate independently of **Self-Verification**. To enable them, turn on the switch beside the rule list and configure at least one valid rule. Rules are evaluated in list order, and the first matching rule applies to a given piece of content. + +![Safety guardrails](./../assets/agent-development/safety-fence.png) + +Each rule contains the following settings: + +| Setting | Description | +| --- | --- | +| **Rule Name** | A unique identifier for the rule. The interface warns about duplicate names; use a name that clearly describes the detection target. | +| **Regular Expression** | A pattern written using Python `re` syntax. Matching is case-insensitive by default at runtime; rules with invalid syntax are excluded from runtime checks. | +| **Severity** | Determines the action after a match: **Block**, **Redact**, or **Allow**. New rules default to **Block**. | +| **Description** | An optional explanation of the rule's purpose for maintenance and review. | + +The severity levels behave as follows at each inspection point: + +| Severity | Latest User Input | Historical Messages | Tool Inputs | Tool Outputs | +| --- | --- | --- | --- | --- | +| **Block** | Stop the run and return a refusal explanation | Downgrade to redaction before sending to the model | Prevent the tool call | Because the tool has already run, downgrade to redaction | +| **Redact** | Replace matched content with `***` and continue | Replace matched content with `***` and continue | Replace matched string arguments with `***` before calling the tool | Replace matched content with `***` before adding it to the agent context | +| **Allow** | Continue without modifying the content | Continue without modifying the content | Call the tool without modifying its arguments | Continue without modifying the output | + +Safety guardrails also provide the following supporting capabilities: + +- **AI Generate:** Select a model and describe in natural language what should be matched or intercepted. The system determines whether to generate one candidate expression or multiple rules; confirm the candidate or select the rules before importing them into the list. +- **Rule Management:** Add, edit, copy, delete, or batch-delete rules manually, and view the distribution of Block, Redact, and Allow rules. +- **Regex Test Preview:** Paste sample text to preview matched text, rule names, and match counts in real time. The preview validates matching only; it does not perform blocking or redaction actions. + +> ⚠️ **Note:** Safety guardrails perform regular-expression-based content screening and are not a complete semantic safety review. AI-generated rules can also produce false positives or false negatives. Test both normal and risky samples in **Regex Test Preview** before saving the configuration. + +### 🐛 Debug and Save + +After completing the initial agent configuration, you can debug the agent and fine-tune the prompts based on the debugging results to continuously improve agent performance. + +1. Click the "Debug" button in the lower right corner of the page to open the agent debug page +2. Test conversations with the agent and observe its responses and behavior +3. Review conversation performance and error messages, and optimize the agent prompts based on the test results + +After successful debugging, click the "Save" button in the lower right corner, and the agent will be saved and appear in the agent list. + +## 📋 Version Management + +Nexent supports agent version management. You can save different versions of agent configurations during the debugging process. + +After verifying the agent configuration, click **Publish** to publish it. The agent then becomes visible in Agent Repository and Start Chat, and its version history can be managed. + +Click the version comparison button in the lower-right corner of **Version Management** to review a historical version and compare its Q&A performance with the latest version. + +![Version comparison](./../assets/agent-development/version_management_1.png) + +To roll back to another version, open the menu on the right side of that version and click **Rollback**. + +![Roll back a version](./../assets/agent-development/version_management_2.png) + +### 🚀 Publish as A2A Agent + +Nexent supports exposing published agents as A2A Agents for external systems to call. When publishing a version, you can check the "Publish as A2A Agent" option to register the current agent as an A2A 1.0 compliant Agent. + +
+ +
+ +After successful publishing, the system will display the A2A Agent's call information: + +
+ +
+ +| Field | Description | +| --------------------- | ------------------------------------------------------------------------------------------ | +| **Endpoint ID** | Unique identifier for the A2A Agent | +| **Agent Card URL** | Agent discovery endpoint; external systems use this address to retrieve Agent descriptions | +| **Protocol Version** | A2A protocol version; currently 1.0 | +| **REST Endpoints** | REST-style API endpoints | +| **JSON-RPC Endpoint** | JSON-RPC 2.0 protocol calling endpoint | + +#### Calling Methods + +The published A2A Agent supports the following two calling protocols: + +##### REST API + +```bash +# Get Agent Card (for Agent discovery) +GET /nb/a2a/{endpoint_id}/.well-known/agent-card.json + +# Send synchronous message +POST /nb/a2a/{endpoint_id}/message:send +Content-Type: application/json + +{ + "message": { + "role": "user", + "content": "Please help me complete a task" + } +} + +# Send streaming message (SSE) +POST /nb/a2a/{endpoint_id}/message:stream +Content-Type: application/json + +{ + "message": { + "role": "user", + "content": "Please help me complete a task" + } +} + +# Get task status +GET /nb/a2a/{endpoint_id}/tasks/{task_id} +``` + +##### JSON-RPC 2.0 + +```bash +POST /nb/a2a/{endpoint_id}/v1 +Content-Type: application/json + +# Send synchronous message +{ + "jsonrpc": "2.0", + "method": "SendMessage", + "params": { + "message": { + "role": "user", + "content": "Please help me complete a task" + } + }, + "id": 1 +} + +# Send streaming message +{ + "jsonrpc": "2.0", + "method": "SendStreamingMessage", + "params": { + "message": { + "role": "user", + "content": "Please help me complete a task" + } + }, + "id": 2 +} + +# Get task status +{ + "jsonrpc": "2.0", + "method": "GetTask", + "params": { + "taskId": "task_abc123" + }, + "id": 3 +} +``` + +> 💡 **Tips**: +> +> - For local development, replace the `/nb/a2a` prefix with `http://localhost:5013/nb/a2a` (use `http://localhost:30013/nb/a2a` if running on k8s) +> - For production environments, replace the prefix with your server domain name or public IP address + +> ⚠️ **Notes**: +> +> - Calling A2A Agents requires carrying valid authentication information in the request headers +> - Agent Card information is cached with a refresh interval of 1 hour +> - If you need to update Agent information, you need to republish the agent version + +When an agent is published as an A2A-compliant Agent, click the leftmost icon in the agent list to view its detailed calling information. + +![View A2A Agent calling information](./../assets/agent-development/a2a-find-detail.jpg) + +## 🔧 Manage the Agent List + +Click **Select Agent** to browse the complete list of agents you can edit in the current environment. Use the search box at the top to locate an agent. + +![Agent list](./../assets/agent-development/agent-list.png) + +The icons on the right side of each agent represent the available management actions. From left to right, they are: + +### 📋 Copy + +Create an identical clone of an agent for version backups or parallel testing. + +### 🔗 View Call Relationships + +View the collaborative agents/tools used by the agent, displayed in a tree diagram to clearly see the agent call relationships. + +
+ +
+ +### 📤 Export + +Export a successfully debugged agent as a JSON or ZIP file, which can later be imported to create a copy. Complex agents that include skills are exported as ZIP archives by default. + +### 🗑️ Delete + +Permanently delete the agent from the local environment. + +## 🚀 Next Steps + +After completing agent development, you can: + +1. Manage and publish your agents, or discover agents from other developers, in **[Agent Repository](../agent-development.md)** +2. Interact with agents in **[Start Chat](../start-chat.md)** +3. Configure **[Memory Management](./memory-configuration.md)** to enhance the agent's personalization capabilities + +If you encounter any issues during agent development, please refer to our **[FAQ](../../quick-start/faq.md)** or ask for support in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). diff --git a/doc/docs/en/user-guide/agent-development/knowledge-configuration.md b/doc/docs/en/user-guide/agent-development/knowledge-configuration.md new file mode 100644 index 0000000000..9b766cdd1e --- /dev/null +++ b/doc/docs/en/user-guide/agent-development/knowledge-configuration.md @@ -0,0 +1,195 @@ +# Knowledge Configuration + +Create and manage knowledge bases, upload documents, and generate summaries. Knowledge bases are critical information sources that let agents securely use your private data. + +## 🔧 Create a Knowledge Base + +1. Click **Create Knowledge Base** at the top of the left-side list +2. In the creation panel, fill in the following fields: + +| Field | Description | +|-------|-------------| +| **Name** | Required. Must be unique and can only contain Chinese characters or lowercase letters. Spaces, slashes, and other special characters are not allowed. The system automatically checks for duplicates as you type | +| **Embedding Model** | Choose the vector model used to embed your documents. Models fall into two groups: **Text Embedding (embedding)** and **Multimodal Embedding (multi_embedding)**. Selecting a multimodal model automatically enables vectorization of non-text content such as images. See [Embedding Model Types](#embedding-model-types) | +| **User Groups** | Select which user groups can access this knowledge base (multi-select) | +| **In-Group Permission** | Controls what group members can do: **Edit** — upload and delete files; **Read Only** — view and search only; **Private** — only the creator can access | +| **Preserve Source File** | When enabled, uploaded files are kept in the system for later re-processing or download. When disabled, only the vectorized data is stored | +| **Storage Quota** | Optional. Set a storage cap for the knowledge base (switchable between GB and MB). The system raises a warning when storage usage approaches the quota | + +3. After configuring the fields above, select files to upload in the upload area below (or skip and upload later) +4. Once files are uploaded, the knowledge base is created and processing begins automatically + + +![Create Knowledge Base](../assets/knowledge-base/create-knowledge-base.png) + +## 📁 Upload Files + +### Upload Files + +1. Select a knowledge base from the list +2. Click the upload area to pick files (multi-select supported) or drag them in directly +3. Nexent automatically parses files, extracts text, and vectorizes the content +4. Track the processing status in the document list + +:::tip File Size Limit +Maximum upload size per file is **20 MB**. Files exceeding this limit cannot be uploaded. +::: + +### Document Processing Status + +Uploaded files go through multiple stages. There are 6 distinct statuses: + +| Status | Description | +|--------|-------------| +| Waiting | File uploaded and queued for processing | +| **Parsing** | System is extracting text content from the file | +| **Ingesting** | Text is being vectorized and written into the vector database | +| **Ready** | Processing complete — the document is available for retrieval | +| **Parse Failed** | An error occurred during parsing. Hover over the status icon to see the detailed error reason and troubleshooting suggestions | +| **Ingest Failed** | An error occurred during vectorization. Hover over the status icon to see the detailed error reason and troubleshooting suggestions | + +💡 Hover over the status icon to view real-time progress (e.g., "23/50 chunks processed") and error details for failed documents. + +![Processing Progress and Error Tooltip](../assets/knowledge-base/tip.png) + +### Supported File Formats + +Nexent supports multiple file formats, including: + +- **Text:** .txt, .md, .json +- **PDF:** .pdf +- **Word:** .docx +- **PowerPoint:** .pptx +- **Excel:** .xlsx +- **EPUB:** .epub +- **Data files:** .csv +- **Web content:** .html, .xml + +## 📊 Knowledge Base Summary + +Give every knowledge base a clear summary so agents can pick the right source during retrieval. + +### Generate and Edit Summaries Manually + +1. Click **Details** to the right of the knowledge base name to open the overview page +2. In the overview page, choose an LLM model and click **Auto Summary** to generate a description +3. Edit the generated text to improve accuracy +4. Click **Save** to store your changes + +![Content Summary](../assets/knowledge-base/summary-knowledge-base.png) + +### Scheduled Auto-Summary + +In addition to manual triggers, you can configure **scheduled auto-summary** to let the system periodically regenerate the knowledge base summary in the background: + +| Frequency | Description | +|-----------|-------------| +| 1 Hour | High-frequency updates — suitable for knowledge bases with frequent content changes | +| 3 Hours | Medium-high frequency updates | +| 6 Hours | Medium frequency updates | +| 1 Day | Once a day — suitable for most scenarios | +| 1 Week | Once a week — suitable for mostly static knowledge bases | + +Select the frequency in the summary section of the overview page. The system intelligently checks whether the knowledge base has received document updates since the last run; if no new documents have been added or changed, the generation is skipped to save resources. + +## 📂 Chunk Management + +After upload, each document is split into multiple **Chunks**. Each chunk contains a segment of text and has a corresponding vector index entry. You can perform fine-grained management on chunks. + +### Viewing Chunks + +1. Click a knowledge base name to open the document list +2. Click the **Chunk Details** tab at the top +3. All documents in the knowledge base are listed at the top of the page. Click any document to view all of its chunks +4. Chunks are displayed as cards showing the chunk content and the source file name + +### Searching Chunks + +Use the search box on the Chunk Details page to search for chunks. The system performs a hybrid search combining keyword matching and semantic matching, and returns results ranked by overall relevance. + +### Manually Managing Chunks + +| Action | Description | +|--------|-------------| +| **Create** | Add a custom chunk manually on the Chunk Details page | +| **Edit** | Click a chunk card to enter edit mode and modify its text content | +| **Delete** | Remove unwanted chunks | +| **Download** | Export chunk content as a download | + +:::warning Model Compatibility Restriction +Chunk edit and create operations depend on embedding model consistency. If the model configuration has changed, these operations may be automatically disabled to prevent incompatible vector data from being written. +::: + +![Chunk Management](../assets/knowledge-base/chunk_management.png) + + + +## 🧩 Embedding Model Types + +Embedding models in the system are divided into two categories: + +- **Text Embedding models (embedding)**: For vectorizing pure text documents (e.g., BGE, M3E) +- **Multimodal Embedding models (multi_embedding)**: Can process both text and image content (e.g., DashScope, Jina) + +The embedding model selected when a knowledge base is created remains bound to that knowledge base for its entire lifecycle. Choose the appropriate model type based on your document content when creating a knowledge base. + +## 🔧 Using Knowledge Bases + +Nexent supports binding knowledge bases to agents. When creating an agent, enable the appropriate knowledge retrieval tool in the tool configuration panel and select the associated knowledge bases. + +For Nexent-native knowledge bases, enable the **knowledge_base_search** tool: + +![Tool 1](../assets/knowledge-base/knowledge-tool1.png) + +![Tool 2](../assets/knowledge-base/knowledge-tool2.png) + +Inside the tool configuration modal, you will see the list of bound knowledge bases with search and multi-select support. Each knowledge base displays its embedding model name so you can confirm compatibility before binding. + +## 🔍 Knowledge Base Management + +### View Knowledge Bases + +1. **Knowledge Base List** + - The left column lists every created knowledge base + - Supports searching by name and filtering by knowledge source or embedding model + - Each knowledge base card shows the following information: + + | Info | Description | + |------|-------------| + | Name | The name set at creation time | + | Document Count | Total number of uploaded documents | + | Chunk Count | Total number of document chunks after splitting | + | Source | Knowledge base origin (Nexent native / external source) | + | Created At | Date when the knowledge base was created | + | Embedding Model | Name of the bound embedding model | + | Multimodal Badge | Shown if multimodal support is enabled | + | User Group Tags | Names of the user groups this knowledge base is visible to | + | Permission Icon | Your current access level for this knowledge base (hover over the icon to see details) | + | No Source File Badge | Shown when "Preserve Source File" is disabled | + +2. **Knowledge Base Details** + - Click a knowledge base name to view all documents + - Click **Details** to open the overview page for viewing and editing the summary + +> Click **Edit** to manage the knowledge base name, visible user groups, and in-group permissions + +Knowledge Base Permissions + +### Edit Knowledge Bases + +1. **Delete Knowledge Base** + - Click **Delete** to the right of the knowledge base row + - Confirm the deletion (irreversible) + +2. **Delete or Add Files** + - Inside the document list, click **Delete** to remove a document + - Use the upload area below the document list to add new files + +## 🚀 Next Steps + +After completing knowledge base configuration, we recommend you continue with: + +1. **[Agent Development](../agent-development)** – Create and configure agents +2. **[Start Chat](../start-chat)** – Interact with your agent + +Need help? Check the **[FAQ](../../quick-start/faq.md)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). diff --git a/doc/docs/en/user-guide/agent-development/memory-configuration.md b/doc/docs/en/user-guide/agent-development/memory-configuration.md new file mode 100644 index 0000000000..ea5e5d949e --- /dev/null +++ b/doc/docs/en/user-guide/agent-development/memory-configuration.md @@ -0,0 +1,203 @@ +# Memory Configuration + +Nexent's memory capability preserves reusable information across multiple turns and conversations. The current memory system uses a **three-level Tenant, User, and Agent architecture**: the Tenant and User levels store long-term memories, while the Agent level stores short-term memories generated through interactions between a specific user and a specific agent. + +When memory is enabled, the system loads long-term memories and retrieves relevant Agent short-term memories before the agent runs. Before producing its final response, the agent also determines whether the current conversation contains new information worth saving. + +## 🎯 How It Works + +During a normal conversation, memory works as follows: + +1. Load the current tenant's Tenant long-term memories and the current user's User long-term memories. +2. Use the user's latest question to search the short-term memories associated with the current user and agent. +3. Add the available long-term memories and retrieved short-term memories to the agent context. +4. Generate a response using the memories, current question, tool results, and conversation history. +5. Before returning the final response, determine whether the conversation introduced new user preferences, task objectives, action plans, recent progress, or corrective reflections. If so, summarize them as concise Agent short-term memories. + +Memory operations are performed through built-in tools. You can inspect tool execution status to confirm the memory-loading trace. If memory retrieval fails, the system skips memory and continues the current task so that a memory service issue does not interrupt the entire conversation. + +![Memory search tool](./../assets/memory-management/memory-search-tool.png) + +> 💡 **Note:** Memory retrieval and writing are disabled in agent debug mode to prevent test data from affecting production memories. Use **Start Chat** to verify cross-conversation memory behavior. + +## ⚙️ Open Memory Configuration + +1. Click **Memory Configuration** in the left navigation bar. +2. The page contains four tabs: **Base Settings**, **Tenant**, **User**, and **Agent**. +3. The number beside each tab indicates how many memory records are currently loaded for that level. + +### Base Settings + +Base Settings currently provides a master switch for memory capability. + +| Setting | Default | Description | +| --- | --- | --- | +| **Memory Capability** | Enabled | When enabled, normal conversations load, retrieve, and write memories. Disabling it stops agents from using memory but does not delete existing records. | + +Changes to the switch are saved immediately. If saving fails, the page restores the previous state and displays an error message. + +![Memory settings](./../assets/memory-management/memory-settings.png) + +## 📚 Three-Level Memory Architecture + +The current system uses only the following three memory levels: + +| Level | Visibility and Scope | Primary Source | How It Is Used | +| --- | --- | --- | --- | +| **Tenant** | Shared within the current tenant | Manually maintained by authorized users | Supplied to agents as organization-level long-term context | +| **User** | Visible only to the current user and available to that user's agents | Manually maintained by the current user | Supplied to agents as user-level long-term context | +| **Agent** | Isolated to the current user and a specific agent | Automatically summarized by the agent during conversations | Relevant content is selected through vector retrieval and added to the context | + +### Tenant Memory + +Tenant memory stores stable information that applies across the organization, such as: + +- Company terminology and standardized wording +- Common working conventions and process principles +- Organization-level preferences or constraints +- Facts that multiple users and agents need to reference + +Agents do not write Tenant memories automatically. Only users with permission to create Tenant memories can see the **New Memory** button; these memories are typically maintained by tenant administrators. + +### User Memory + +User memory belongs only to the current user and is suitable for stable personal information that should be reused across agents, such as: + +- Preferred language, format, and writing style +- Long-term working habits +- Ongoing project context +- Personal requirements that all agents should follow + +User memories are created and maintained manually by the current user. They are not shared with other users in the tenant. + +### Agent Memory + +Agent memories are generated automatically during production conversations and are bound to both the current user and current agent. They may store: + +- Preferences the user expresses to that agent +- Current task objectives +- Action plans and recent progress +- Reflections derived from user feedback, errors, or failed results + +Agent memories for the same user-agent pair can be recalled across conversations, but they are not automatically shared across users or agents. Main agents and collaborative agents also maintain separate Agent memories. + +Before saving a memory, the agent must evaluate, summarize, and deduplicate it into a concise, reusable entry. Full conversations, temporary calculations, intermediate noise, unverified assumptions, duplicate content, sensitive credentials, and information the user explicitly asked to forget should not be written to memory. A single agent run can automatically save at most three Agent short-term memories. + +> ⚠️ **Note:** The current version does not periodically promote Agent short-term memories to User long-term memories in the background. + +## 🗂️ View and Filter Memories + +The Tenant, User, and Agent tabs display memory records in tables, including memory content, type, status, and creation time. + +All levels support: + +- Searching by memory content +- Filtering by status +- Viewing the number of filtered results +- Paginated browsing with 10, 20, or 50 records per page + +The Agent tab also supports: + +- Filtering by agent, source conversation, or creation date range +- Viewing the agent name and source conversation +- Clicking the source conversation title to return to the conversation that generated the memory + +![Agent memory](./../assets/memory-management/agent-memory.png) + +### Memory Status + +| Status | Description | +| --- | --- | +| **Active** | The memory can participate in long-term context loading or Agent short-term memory retrieval. | +| **Archived** | The memory remains in the list but is excluded from runtime loading and retrieval. | +| **Disabled** | The memory is temporarily unavailable. This status can be set manually or caused by an Agent memory being incompatible with the current embedding model. | + +## ✍️ Create, Edit, and Delete Memories + +### Create a Long-Term Memory + +The page only supports manually creating Tenant or User long-term memories. Agent short-term memories are generated while agents run, so the Agent tab does not provide a **New Memory** button. + +1. Open the Tenant or User tab. +2. Click **New Memory** in the upper-right corner. +3. Enter the content to retain, up to 500 characters. +4. Click **Create Memory**. + +Manually created records default to the **Long-term Memory** type and **Active** status. + +![Add memory](./../assets/memory-management/add-memory.png) + +### Edit a Memory + +1. Click **Edit** on the right side of the target record. +2. Modify the memory content or status. +3. Click **Save Changes**. + +Edited content must still remain within the 500-character limit. The current page does not allow changing a record's memory level or type through editing. + +If an Agent memory is incompatible with the current embedding model, it appears as unavailable and cannot be edited, but it can still be deleted. + +### Delete a Memory + +Click **Delete** on the right side of a record, then select **Confirm Delete** in the confirmation dialog. The record will be removed from the page and excluded from subsequent memory loading and retrieval. + +> ⚠️ **Note:** The page does not provide a restore option. Confirm that the memory is no longer needed before deleting it. + +## 🔍 Memory Retrieval and Context Usage + +Different levels are used differently: + +- **Tenant / User long-term memories:** Active long-term memories are read from storage and supplied directly to the agent as persistent context, without semantic-similarity filtering. +- **Agent short-term memories:** The latest user question is used for vector retrieval. The results are then filtered through relevance fusion, time decay, similarity deduplication, and the context budget before the most useful entries are supplied to the agent. + +Tenant and User memories should therefore remain concise and stable, because excessive content directly consumes model context. Agent memories can accumulate gradually through interactions; the system prioritizes content that is more relevant to the current question, more recent, and non-duplicative. + +## 🧩 Embedding Models and Agent Memory + +Generating and retrieving Agent short-term memories depends on the tenant's currently configured embedding model. When opening **Memory Configuration** or **Start Chat**, the system displays a prompt if the tenant has not configured an embedding model. + +Without an available embedding model: + +- Tenant and User long-term memories remain stored and are managed as long-term context. +- Agent short-term memories cannot be generated or retrieved normally. + +After switching embedding models, Agent memories indexed with the previous model may be incompatible with the current index. The page automatically synchronizes their status when loading records: + +| Embedding Compatibility | Synchronized Status | +| --- | --- | +| Incompatible | Disabled | +| Compatible | Active | + +Missing embedding model warning + +![Disabled memory](./../assets/memory-management/disabled-memory.png) + +If you switch back to an embedding model compatible with the original records, disabled Agent memories become **Active** again. + +## 💡 Usage Tips + +### Write High-Quality Memories + +Each memory should express one clear fact that can be reused over time. + +✅ `The user prefers technical proposals to present the conclusion before the risks.` + +❌ Not recommended: `The user likes concise answers, often works at night, manages several projects, and wants everything presented in tables.` + +Follow these guidelines: + +1. **Keep memories atomic:** Each entry should describe only one preference, fact, objective, or piece of progress. +2. **Avoid temporary information:** Do not save one-off calculations or short-lived irrelevant details. +3. **Maintain memories regularly:** Archive or delete outdated content. +4. **Control the number of long-term memories:** Tenant and User memories are supplied as persistent context, so avoid verbose, duplicate, or contradictory entries. +5. **Protect privacy:** Do not store passwords, access tokens, keys, or unnecessary sensitive personal information. + +## 🚀 Next Steps + +After configuring memory, you can: + +1. Start multiple conversations with the same agent in **[Start Chat](../start-chat)** to verify cross-conversation memory. +2. Check the embedding model in **[Model Configuration](./model-configuration.md)**. +3. Continue creating and adjusting agents in **[Agent Configuration](./agent-configuration.md)**. + +If you encounter any issues, refer to the **[FAQ](../../quick-start/faq.md)** or visit [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions) for support. diff --git a/doc/docs/en/user-guide/agent-development/model-configuration.md b/doc/docs/en/user-guide/agent-development/model-configuration.md new file mode 100644 index 0000000000..d5a3d75da7 --- /dev/null +++ b/doc/docs/en/user-guide/agent-development/model-configuration.md @@ -0,0 +1,228 @@ +# Model Configuration + +The Model Configuration module lets you add and configure AI models — large language models, embedding models, rerank models, multimodal models, and voice models. Nexent supports multiple providers so you can pick the best option for each scenario. + +## 🤖 Model Configuration + +### 🛠️ Add Custom Models + +#### Add a Single Model + +1. **Add a custom model** + - Click **Add Model** to open the dialog. +2. **Select model type** + - Choose Large Language Model, Embedding Model, Image Understanding Model, Image Generation Model, Video Understanding Model, Rerank Model, Speech-to-Text Model or Text-to-Speech Model. +3. **Configure model parameters** + - **Model Name (required):** The name you send in API requests. + - **Display Name:** Optional label shown in the UI (defaults to the model name). + - **Model URL (required):** API endpoint from the provider. + - **API Key:** Your provider key. + +> ⚠️ **Notes** +> 1. Model names usually follow `series/model`. Example: `Qwen/Qwen3-8B`. +> 2. API endpoints come from the provider docs. For SiliconFlow, examples include `https://api.siliconflow.cn/v1` (LLM, VLM) and `https://api.siliconflow.cn/v1/embeddings` (embedding). +> 3. Generate API keys from the provider's key management console. + +4. **Connectivity verification** + - Click **Verify** to send a test request and confirm connectivity. +5. **Save model** + - Click **Add** to place the model in the available list. + +
+ +
+ +#### Batch Add Models + +Use batch import to speed up onboarding: + +1. Enable the **Batch Add Models** toggle in the dialog. +2. Select a **model provider**. +3. Choose the **model type** (LLM/Embedding/VLM/Rerank/STT/TTS). +4. Enter the **API Key** (required). +5. Click **Fetch Models** to retrieve the provider list. +6. Toggle on the models you need (disabled by default). +7. Click **Add** to save every selected model at once. + +
+ +
+ +### 🔧 Edit Custom Models + +Modify or delete models anytime: + +1. Click **Edit Custom Models**. +2. Select the model type (LLM/Embedding/VLM/Rerank/STT/TTS). +3. Choose between batch editing or single-model editing. +4. For batch edits, toggle models on/off or click **Edit Config** in the upper-right to change settings in bulk. +5. For single models, click the trash icon 🗑️ to delete, or click the model name to open the edit dialog. + +
+ + +
+
+
+ + +
+
+
+ + +
+ +### ⚙️ Configure System Models + +After adding models, assign the platform-level defaults. These models handle system tasks such as title generation, real-time file reading, and multimodal parsing. Individual agents can still choose their own run-time models. + +#### Base Model + +- Used for core platform features (title generation, real-time file access, basic text processing). +- Choose any added large language model from the dropdown. + +#### Large Language Model + +The large language model serves as the system's core reasoning engine, responsible for processing users' natural language requests, generating responses, executing code, analyzing data, and other complex tasks. Choosing an appropriate large language model can significantly improve the agent's conversational quality and task-handling capabilities. + +- Click the Large Language Model dropdown and select one from the added large language models. + +#### Embedding Model + +Embedding models are primarily used for vectorization processing of text, images, and other data in knowledge bases, forming the foundation for efficient retrieval and semantic understanding. Configuring an appropriate embedding model can significantly improve knowledge base search accuracy and multimodal data processing capabilities. + +- Click the embedding model dropdown to select one from the added embedding models. +- Embedding model configuration affects the stable operation of knowledge bases. + +Choose appropriate document chunk size and chunks per request based on model capabilities. Smaller chunks provide more stability, but may affect file parsing quality. + +
+ +
+ +#### Rerank Model + +The rerank model performs semantic matching and scoring on initially filtered documents to ensure the most relevant answers are ranked first, improving retrieval accuracy and efficiency. Configuring an appropriate rerank model can significantly improve knowledge base retrieval effectiveness. + +- Click the Rerank Model dropdown to select one from the added rerank models. + +#### Multimodal Models + +Multimodal models combine visual and language capabilities to handle complex scenarios containing text, images, and other types of information. + +- **Image Understanding Model**: Can analyze and understand image content, extract key information, and answer questions related to images. Click the Image Understanding Model dropdown to select one from the added models. +- **Image Generation Model**: Can generate images based on text descriptions, supporting creative design, content creation, and other scenarios. Click the Image Generation Model dropdown to select one from the added models. +- **Video Understanding Model**: Can analyze and understand video content, extract key information, generate summaries, or answer questions related to videos. Click the Video Understanding Model dropdown to select one from the added models. + +#### Voice Models + +Voice models enable bidirectional conversion between speech and text, supporting voice interaction scenarios. + +- **Text-to-Speech Model**: Converts text content into natural, fluent speech output in real-time, enabling the system to interact with users in a near-human voice. With low latency and high-fidelity speech generation capabilities, it ensures a smooth and natural auditory experience during conversations. Click the Text-to-Speech Model dropdown to select one from the added models. +- **Speech-to-Text Model**: Converts user voice input into text in real-time, enabling accurate understanding and parsing of voice commands and natural language. With high-precision speech transcription and noise robustness, it ensures stable recognition of user intent even in complex environments. Click the Speech-to-Text Model dropdown to select one from the added models. + +
+ + + + + +
+ +### ✅ Check Model Connectivity + +Run regular connectivity checks to keep the platform healthy: + +1. Click **Check Model Connectivity**. +2. Nexent tests every configured system model automatically. + +Status indicators: + +- 🔵 **Blue dot** – Checking in progress. +- 🔴 **Red dot** – Connection failed; review configuration or network. +- 🟢 **Green dot** – Connection is healthy. + +Troubleshooting tips: + +- Confirm network stability. +- Ensure the API key is valid and not expired. +- Check the provider's service status. +- Review firewall and security policies. + +### 🤖 Supported Providers + +#### Large Language Models + +Nexent supports any **OpenAI-compatible** provider, including: + +- [SiliconFlow](https://siliconflow.cn/) +- [Ali Bailian](https://bailian.console.aliyun.com/) +- [TokenPony](https://www.tokenpony.cn/) +- [DeepSeek](https://platform.deepseek.com/) +- [OpenAI](https://platform.openai.com/) +- [Anthropic](https://console.anthropic.com/) +- [Moonshot](https://platform.moonshot.cn/) + +Getting started: + +1. Sign up at the provider's portal. +2. Create and copy an API key. +3. Locate the API endpoint (usually ending with `/v1`). +4. Click **Add Custom Model** in Nexent and fill in the required fields. + +#### Multimodal Models + +Use the same API key and URL as LLMs but specify a multimodal model name, for example **Qwen/Qwen2.5-VL-32B-Instruct** on SiliconFlow. + +#### Embedding Models + +Use the same API key as LLMs but typically a different endpoint (often `/v1/embeddings`), for example **BAAI/bge-m3** from SiliconFlow. + +#### Rerank Models + +Use the same API key as LLMs but typically a different endpoint (often `/v1/rerank`). + +#### Speech Models + +Currently supports VolcEngine Voice and Aliyun Bailian voice models. VolcEngine requires `appid` and `token`, while Aliyun Bailian uses the same API key as the large language model. + +**VolcEngine** +- **Website**: [volcengine.com/product/voice-tech](https://www.volcengine.com/product/voice-tech) +- **Free tier**: Available for individual use +- **Highlights**: High-quality Chinese/English TTS +- Recommended models: **Doubao Text-to-Speech Model 2.0** and **Large Model Streaming Speech Recognition** +- **Getting started**: + + 1. Register a VolcEngine account. + 2. Enable the Voice Technology service. + 3. Create an app and generate `appid` and `token`. + 4. Configure the TTS/STT settings in the Add Model page. + +**Aliyun Bailian** +- **Website**: [aliyun.com/benefit/scene/voice](https://www.aliyun.com/benefit/scene/voice) +- Recommended models: **Qwen3-TTS-Instruct-Flash-Realtime / Qwen3-TTS-Flash-Realtime** and **Qwen3-ASR-Flash-Realtime** +- **Getting started**: + + 1. Register an Aliyun account. + 2. Enable the Qwen real-time voice service. + 3. Create an app and generate an API Key. + 4. Configure the TTS/STT settings in the Add Model page. + +## 💡 Need Help + +If you run into provider issues: + +1. Review the provider's documentation. +2. Check API key permissions and quotas. +3. Test with the provider's official samples. +4. Ask the community in our [Discord server](https://discord.gg/tb5H3S3wyv). + +## 🚀 Next Steps + +After closing the Model Configuration flow, continue with: + +1. **[Knowledge Base](./knowledge-configuration)** – Create and manage knowledge bases. +2. **[Agent Configuration](./agent-configuration)** – Build and configure agents. + +Need help? Check the **[FAQ](../../quick-start/faq)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). diff --git a/doc/docs/en/user-guide/agent-market.md b/doc/docs/en/user-guide/agent-market.md index 1106f3db63..6451dd0546 100644 --- a/doc/docs/en/user-guide/agent-market.md +++ b/doc/docs/en/user-guide/agent-market.md @@ -48,7 +48,7 @@ Select your preferred agent, download with one click, and add it to your agent s 🔑 Fill in MCP tool permissions as prompted -After installation, your agent will be ready in **[Agent Space](./agent-space)** +After installation, your agent will be ready in **[Agent Space](./agent-development.md)** ## 📢 Share Your Creations @@ -60,8 +60,8 @@ Welcome to share your work in [GitHub Discussions](https://github.com/ModelEngin While waiting for the Agent Market to launch, you can: -1. Manage your own agents in **[Agent Space](./agent-space)** -2. Create custom agents through **[Agent Development](./agent-development)** +1. Manage your own agents in **[Agent Space](./agent-development.md)** +2. Create custom agents through **[Agent Development](./agent-development.md)** 3. Experience the powerful features of agents in **[Start Chat](./start-chat)** -If you encounter any issues during use, please refer to our **[FAQ](../quick-start/faq)** or ask for support in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). +If you encounter any issues during use, please refer to our **[FAQ](../quick-start/faq.md)** or ask for support in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). diff --git a/doc/docs/en/user-guide/agent-space.md b/doc/docs/en/user-guide/agent-space.md deleted file mode 100644 index 282a0c910f..0000000000 --- a/doc/docs/en/user-guide/agent-space.md +++ /dev/null @@ -1,70 +0,0 @@ -# Agent Space - -Agent Space is the central dashboard for every agent you have built. View agents in card form, inspect their configurations, delete or export them, and jump straight into chats. - -![Agent Space](./assets/agent-space/agent-space.png) - -## 📦 Agent Cards - -Each agent appears as a card showing: - -- **Icon** – The agent’s avatar. -- **Name** – The display name. -- **Description** – A quick summary of what it does. -- **Status** – Whether the agent is available. -- **Actions** – Shortcuts for editing, exporting, deleting, and more. - -## 🔧 Manage Agents - -### View Agent Details - -Click a card to open its details: - -- **Basic info:** ID, name, description, status, max steps, and whether to provide run summary. -- **Model configuration:** Model name, business logic model, etc. -- **Prompts:** Role, constraints, examples, and the original description. -- **Tools:** Every tool the agent can use. -- **Sub-agents:** Any collaborative agents that are configured. - -![Agent Details](./assets/agent-space/agent-details.png) - -### Edit an Agent - -1. Click **Edit** on the card. -2. You’ll be taken to the Agent Development page. -3. Adjust the settings and save—updates sync back to Agent Space automatically. - -### Delete an Agent - -1. Click **Delete** on the card. -2. Confirm the deletion (this cannot be undone). - -> ⚠️ **Note:** Deleting an agent permanently removes it. Export a backup first if you might need it later. - -### Export an Agent - -1. Click **Export** on the card. -2. Nexent downloads a JSON configuration file you can import later. - -### Copy an Agent - -1. Click **Copy** on the card to duplicate the agent. -2. This facilitates experimentation, multi-version debugging, and parallel development. - -### View Relationships - -1. Click **View Relationships** to see how the agent interacts with tools and other agents. - -### Jump to Chat - -1. Click **Chat** to open Start Chat with the agent already selected. - -## 🚀 Next Steps - -Once you finish reviewing agents you can: - -1. Talk to them in **[Start Chat](./start-chat)**. -2. Continue iterating in **[Agent Development](./agent-development)**. -3. Enhance retention with **[Memory Management](./memory-management)**. - -Need help? Check the **[FAQ](../quick-start/faq)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). diff --git a/doc/docs/en/user-guide/assets/agent-development/agent-list.png b/doc/docs/en/user-guide/assets/agent-development/agent-list.png new file mode 100644 index 0000000000..c763a27858 Binary files /dev/null and b/doc/docs/en/user-guide/assets/agent-development/agent-list.png differ diff --git a/doc/docs/en/user-guide/assets/agent-development/agent-settings.png b/doc/docs/en/user-guide/assets/agent-development/agent-settings.png new file mode 100644 index 0000000000..e3d03b925e Binary files /dev/null and b/doc/docs/en/user-guide/assets/agent-development/agent-settings.png differ diff --git a/doc/docs/en/user-guide/assets/agent-development/generate_agent.png b/doc/docs/en/user-guide/assets/agent-development/generate_agent.png new file mode 100644 index 0000000000..4a7fd85e04 Binary files /dev/null and b/doc/docs/en/user-guide/assets/agent-development/generate_agent.png differ diff --git a/doc/docs/en/user-guide/assets/agent-development/import-2.png b/doc/docs/en/user-guide/assets/agent-development/import-2.png new file mode 100644 index 0000000000..d58677fbf8 Binary files /dev/null and b/doc/docs/en/user-guide/assets/agent-development/import-2.png differ diff --git a/doc/docs/en/user-guide/assets/agent-development/safety-fence.png b/doc/docs/en/user-guide/assets/agent-development/safety-fence.png new file mode 100644 index 0000000000..9efcea557e Binary files /dev/null and b/doc/docs/en/user-guide/assets/agent-development/safety-fence.png differ diff --git a/doc/docs/en/user-guide/assets/agent-development/set-tools-1.png b/doc/docs/en/user-guide/assets/agent-development/set-tools-1.png new file mode 100644 index 0000000000..fdcb2aad8a Binary files /dev/null and b/doc/docs/en/user-guide/assets/agent-development/set-tools-1.png differ diff --git a/doc/docs/en/user-guide/assets/agent-development/set-tools-2.png b/doc/docs/en/user-guide/assets/agent-development/set-tools-2.png new file mode 100644 index 0000000000..8b0deca1dd Binary files /dev/null and b/doc/docs/en/user-guide/assets/agent-development/set-tools-2.png differ diff --git a/doc/docs/en/user-guide/assets/auto-tasks/01-automation-task-list.png b/doc/docs/en/user-guide/assets/auto-tasks/01-automation-task-list.png new file mode 100644 index 0000000000..fac980f259 Binary files /dev/null and b/doc/docs/en/user-guide/assets/auto-tasks/01-automation-task-list.png differ diff --git a/doc/docs/en/user-guide/assets/auto-tasks/02-create-task-in-chat.png b/doc/docs/en/user-guide/assets/auto-tasks/02-create-task-in-chat.png new file mode 100644 index 0000000000..5eda94fcc3 Binary files /dev/null and b/doc/docs/en/user-guide/assets/auto-tasks/02-create-task-in-chat.png differ diff --git a/doc/docs/en/user-guide/assets/auto-tasks/03-automation-proposal.png b/doc/docs/en/user-guide/assets/auto-tasks/03-automation-proposal.png new file mode 100644 index 0000000000..54ee80d121 Binary files /dev/null and b/doc/docs/en/user-guide/assets/auto-tasks/03-automation-proposal.png differ diff --git a/doc/docs/en/user-guide/assets/auto-tasks/04-edit-automation-proposal.png b/doc/docs/en/user-guide/assets/auto-tasks/04-edit-automation-proposal.png new file mode 100644 index 0000000000..9a2914dd6f Binary files /dev/null and b/doc/docs/en/user-guide/assets/auto-tasks/04-edit-automation-proposal.png differ diff --git a/doc/docs/en/user-guide/assets/auto-tasks/05-created-task-in-list.png b/doc/docs/en/user-guide/assets/auto-tasks/05-created-task-in-list.png new file mode 100644 index 0000000000..93a5dd4742 Binary files /dev/null and b/doc/docs/en/user-guide/assets/auto-tasks/05-created-task-in-list.png differ diff --git a/doc/docs/en/user-guide/assets/auto-tasks/06-more-actions.png b/doc/docs/en/user-guide/assets/auto-tasks/06-more-actions.png new file mode 100644 index 0000000000..9ca374da00 Binary files /dev/null and b/doc/docs/en/user-guide/assets/auto-tasks/06-more-actions.png differ diff --git a/doc/docs/en/user-guide/assets/auto-tasks/07-run-history.png b/doc/docs/en/user-guide/assets/auto-tasks/07-run-history.png new file mode 100644 index 0000000000..6b882b94ba Binary files /dev/null and b/doc/docs/en/user-guide/assets/auto-tasks/07-run-history.png differ diff --git a/doc/docs/en/user-guide/assets/knowledge-base/chunk_management.png b/doc/docs/en/user-guide/assets/knowledge-base/chunk_management.png new file mode 100644 index 0000000000..c63c5acabf Binary files /dev/null and b/doc/docs/en/user-guide/assets/knowledge-base/chunk_management.png differ diff --git a/doc/docs/en/user-guide/assets/knowledge-base/create-knowledge-base.png b/doc/docs/en/user-guide/assets/knowledge-base/create-knowledge-base.png index 10ba701896..66cfcd6071 100644 Binary files a/doc/docs/en/user-guide/assets/knowledge-base/create-knowledge-base.png and b/doc/docs/en/user-guide/assets/knowledge-base/create-knowledge-base.png differ diff --git a/doc/docs/en/user-guide/assets/knowledge-base/knowledge-tool.png b/doc/docs/en/user-guide/assets/knowledge-base/knowledge-tool.png deleted file mode 100644 index 8505804ea5..0000000000 Binary files a/doc/docs/en/user-guide/assets/knowledge-base/knowledge-tool.png and /dev/null differ diff --git a/doc/docs/en/user-guide/assets/knowledge-base/knowledge-tool1.png b/doc/docs/en/user-guide/assets/knowledge-base/knowledge-tool1.png new file mode 100644 index 0000000000..0e2c2285bf Binary files /dev/null and b/doc/docs/en/user-guide/assets/knowledge-base/knowledge-tool1.png differ diff --git a/doc/docs/en/user-guide/assets/knowledge-base/summary-knowledge-base.png b/doc/docs/en/user-guide/assets/knowledge-base/summary-knowledge-base.png index a4f206d679..a94c498059 100644 Binary files a/doc/docs/en/user-guide/assets/knowledge-base/summary-knowledge-base.png and b/doc/docs/en/user-guide/assets/knowledge-base/summary-knowledge-base.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mcp-admin-tabs.png b/doc/docs/en/user-guide/assets/mcp-space/mcp-admin-tabs.png new file mode 100644 index 0000000000..dc3dd72bcf Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mcp-admin-tabs.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mcp-developer-tabs.png b/doc/docs/en/user-guide/assets/mcp-space/mcp-developer-tabs.png new file mode 100644 index 0000000000..3e00062ff7 Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mcp-developer-tabs.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-delete.png b/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-delete.png new file mode 100644 index 0000000000..44ef801fe5 Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-delete.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-detail.png b/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-detail.png new file mode 100644 index 0000000000..aaf820b3a6 Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-detail.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-install.png b/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-install.png new file mode 100644 index 0000000000..ee8f497e49 Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-install.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-list.png b/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-list.png new file mode 100644 index 0000000000..b12d650d01 Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mcp-repository-list.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mymcp-addmcp.png b/doc/docs/en/user-guide/assets/mcp-space/mymcp-addmcp.png new file mode 100644 index 0000000000..b38c3ee746 Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mymcp-addmcp.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcp-detail.png b/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcp-detail.png new file mode 100644 index 0000000000..65f2d05b69 Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcp-detail.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcp-review.png b/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcp-review.png new file mode 100644 index 0000000000..4484e08f5a Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcp-review.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcpcard.png b/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcpcard.png new file mode 100644 index 0000000000..428e86b22d Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcpcard.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcpcard2.png b/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcpcard2.png new file mode 100644 index 0000000000..2e5d8ac136 Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/mymcp-mcpcard2.png differ diff --git a/doc/docs/en/user-guide/assets/mcp-space/review-center.png b/doc/docs/en/user-guide/assets/mcp-space/review-center.png new file mode 100644 index 0000000000..00b9b91540 Binary files /dev/null and b/doc/docs/en/user-guide/assets/mcp-space/review-center.png differ diff --git a/doc/docs/en/user-guide/assets/memory-management/add-memory.png b/doc/docs/en/user-guide/assets/memory-management/add-memory.png new file mode 100644 index 0000000000..14d4e101aa Binary files /dev/null and b/doc/docs/en/user-guide/assets/memory-management/add-memory.png differ diff --git a/doc/docs/en/user-guide/assets/memory-management/agent-memory.png b/doc/docs/en/user-guide/assets/memory-management/agent-memory.png new file mode 100644 index 0000000000..249af9b113 Binary files /dev/null and b/doc/docs/en/user-guide/assets/memory-management/agent-memory.png differ diff --git a/doc/docs/en/user-guide/assets/memory-management/disabled-memory.png b/doc/docs/en/user-guide/assets/memory-management/disabled-memory.png new file mode 100644 index 0000000000..e853502123 Binary files /dev/null and b/doc/docs/en/user-guide/assets/memory-management/disabled-memory.png differ diff --git a/doc/docs/en/user-guide/assets/memory-management/embedding-missing-warn.png b/doc/docs/en/user-guide/assets/memory-management/embedding-missing-warn.png new file mode 100644 index 0000000000..f049dcd999 Binary files /dev/null and b/doc/docs/en/user-guide/assets/memory-management/embedding-missing-warn.png differ diff --git a/doc/docs/en/user-guide/assets/memory-management/memory-search-tool.png b/doc/docs/en/user-guide/assets/memory-management/memory-search-tool.png new file mode 100644 index 0000000000..4e02e872d8 Binary files /dev/null and b/doc/docs/en/user-guide/assets/memory-management/memory-search-tool.png differ diff --git a/doc/docs/en/user-guide/assets/memory-management/memory-settings.png b/doc/docs/en/user-guide/assets/memory-management/memory-settings.png new file mode 100644 index 0000000000..a3ed78b46b Binary files /dev/null and b/doc/docs/en/user-guide/assets/memory-management/memory-settings.png differ diff --git a/doc/docs/en/user-guide/assets/model-management/edit-model-1.png b/doc/docs/en/user-guide/assets/model-management/edit-model-1.png index 45ddaaf084..8286cfb485 100644 Binary files a/doc/docs/en/user-guide/assets/model-management/edit-model-1.png and b/doc/docs/en/user-guide/assets/model-management/edit-model-1.png differ diff --git a/doc/docs/en/user-guide/assets/model-management/select-model-3.png b/doc/docs/en/user-guide/assets/model-management/select-model-3.png index 34d83df560..c4b0033ca1 100644 Binary files a/doc/docs/en/user-guide/assets/model-management/select-model-3.png and b/doc/docs/en/user-guide/assets/model-management/select-model-3.png differ diff --git a/doc/docs/en/user-guide/assets/model-management/select-model-4.png b/doc/docs/en/user-guide/assets/model-management/select-model-4.png new file mode 100644 index 0000000000..a09010ad10 Binary files /dev/null and b/doc/docs/en/user-guide/assets/model-management/select-model-4.png differ diff --git a/doc/docs/en/user-guide/assets/model-management/select-model-5.png b/doc/docs/en/user-guide/assets/model-management/select-model-5.png new file mode 100644 index 0000000000..025a2b456c Binary files /dev/null and b/doc/docs/en/user-guide/assets/model-management/select-model-5.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/admin-tabs.png b/doc/docs/en/user-guide/assets/resource-repository/admin-tabs.png new file mode 100644 index 0000000000..e76e96ab91 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/admin-tabs.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/apply-listing.png b/doc/docs/en/user-guide/assets/resource-repository/apply-listing.png new file mode 100644 index 0000000000..e31c6d66c4 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/apply-listing.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/copy-precheck.png b/doc/docs/en/user-guide/assets/resource-repository/copy-precheck.png new file mode 100644 index 0000000000..7956bc37b7 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/copy-precheck.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/developer-tabs.png b/doc/docs/en/user-guide/assets/resource-repository/developer-tabs.png new file mode 100644 index 0000000000..7fe37fca91 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/developer-tabs.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/mine-list.png b/doc/docs/en/user-guide/assets/resource-repository/mine-list.png new file mode 100644 index 0000000000..ae84337c7b Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/mine-list.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/repository-detail.png b/doc/docs/en/user-guide/assets/resource-repository/repository-detail.png new file mode 100644 index 0000000000..1700ad3996 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/repository-detail.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/repository-list.png b/doc/docs/en/user-guide/assets/resource-repository/repository-list.png new file mode 100644 index 0000000000..041a3dcc5a Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/repository-list.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/review-confirm.png b/doc/docs/en/user-guide/assets/resource-repository/review-confirm.png new file mode 100644 index 0000000000..f9110a978f Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/review-confirm.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/review-status.png b/doc/docs/en/user-guide/assets/resource-repository/review-status.png new file mode 100644 index 0000000000..47f51f06c5 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/review-status.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_admin_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_admin_en.png new file mode 100644 index 0000000000..ec7262134b Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_admin_en.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_approve_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_approve_en.png new file mode 100644 index 0000000000..ad22e5af65 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_approve_en.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_create_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_create_en.png new file mode 100644 index 0000000000..ca77651cdb Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_create_en.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_dev_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_dev_en.png new file mode 100644 index 0000000000..53edecca56 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_dev_en.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_list_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_list_en.png new file mode 100644 index 0000000000..d1e32f5de2 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_list_en.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_mine_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_mine_en.png new file mode 100644 index 0000000000..56b607f52e Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_mine_en.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_repo_detail_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_repo_detail_en.png new file mode 100644 index 0000000000..c2bec87740 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_repo_detail_en.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_repo_search_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_repo_search_en.png new file mode 100644 index 0000000000..57bd3665bf Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_repo_search_en.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_select_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_select_en.png new file mode 100644 index 0000000000..a270b83e8b Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_select_en.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_under_review_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_under_review_en.png new file mode 100644 index 0000000000..9a1d655914 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_under_review_en.png differ diff --git a/doc/docs/en/user-guide/assets/resource-repository/skill_using_en.png b/doc/docs/en/user-guide/assets/resource-repository/skill_using_en.png new file mode 100644 index 0000000000..399cfd13b7 Binary files /dev/null and b/doc/docs/en/user-guide/assets/resource-repository/skill_using_en.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/ReAct.png b/doc/docs/en/user-guide/assets/start-chat/ReAct.png new file mode 100644 index 0000000000..4721696a43 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/ReAct.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/agent-list.png b/doc/docs/en/user-guide/assets/start-chat/agent-list.png new file mode 100644 index 0000000000..e9fd3ea613 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/agent-list.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/agent-selection.png b/doc/docs/en/user-guide/assets/start-chat/agent-selection.png deleted file mode 100644 index 4acb045be3..0000000000 Binary files a/doc/docs/en/user-guide/assets/start-chat/agent-selection.png and /dev/null differ diff --git a/doc/docs/en/user-guide/assets/start-chat/agent-welcome.png b/doc/docs/en/user-guide/assets/start-chat/agent-welcome.png new file mode 100644 index 0000000000..e123c00c85 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/agent-welcome.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/analyze_image.png b/doc/docs/en/user-guide/assets/start-chat/analyze_image.png new file mode 100644 index 0000000000..8b0d9f0e6e Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/analyze_image.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/analyze_text_file.png b/doc/docs/en/user-guide/assets/start-chat/analyze_text_file.png new file mode 100644 index 0000000000..007d8baa69 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/analyze_text_file.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/chat-management-1.png b/doc/docs/en/user-guide/assets/start-chat/chat-management-1.png deleted file mode 100644 index 6e7253e6af..0000000000 Binary files a/doc/docs/en/user-guide/assets/start-chat/chat-management-1.png and /dev/null differ diff --git a/doc/docs/en/user-guide/assets/start-chat/chat-management-2.png b/doc/docs/en/user-guide/assets/start-chat/chat-management-2.png deleted file mode 100644 index 3836981e8c..0000000000 Binary files a/doc/docs/en/user-guide/assets/start-chat/chat-management-2.png and /dev/null differ diff --git a/doc/docs/en/user-guide/assets/start-chat/collapse.png b/doc/docs/en/user-guide/assets/start-chat/collapse.png new file mode 100644 index 0000000000..53ae9780d4 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/collapse.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/conversation-manage.png b/doc/docs/en/user-guide/assets/start-chat/conversation-manage.png new file mode 100644 index 0000000000..19402d9fa6 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/conversation-manage.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/create-docx.png b/doc/docs/en/user-guide/assets/start-chat/create-docx.png new file mode 100644 index 0000000000..3cab1114d6 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/create-docx.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/dialog-box.png b/doc/docs/en/user-guide/assets/start-chat/dialog-box.png deleted file mode 100644 index 349c4f975d..0000000000 Binary files a/doc/docs/en/user-guide/assets/start-chat/dialog-box.png and /dev/null differ diff --git a/doc/docs/en/user-guide/assets/start-chat/example-question.png b/doc/docs/en/user-guide/assets/start-chat/example-question.png new file mode 100644 index 0000000000..6792108d2e Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/example-question.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/finish.png b/doc/docs/en/user-guide/assets/start-chat/finish.png new file mode 100644 index 0000000000..e08ac354e9 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/finish.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/memory.png b/doc/docs/en/user-guide/assets/start-chat/memory.png new file mode 100644 index 0000000000..40e4cfcabf Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/memory.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/mermaid.png b/doc/docs/en/user-guide/assets/start-chat/mermaid.png new file mode 100644 index 0000000000..2c72b4e9ef Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/mermaid.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/parallel-subagents.png b/doc/docs/en/user-guide/assets/start-chat/parallel-subagents.png new file mode 100644 index 0000000000..6c3dc37924 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/parallel-subagents.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/parallel-tool-calls.png b/doc/docs/en/user-guide/assets/start-chat/parallel-tool-calls.png new file mode 100644 index 0000000000..dbcd455cea Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/parallel-tool-calls.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/plan.png b/doc/docs/en/user-guide/assets/start-chat/plan.png new file mode 100644 index 0000000000..346d2ead14 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/plan.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/preview-docx.png b/doc/docs/en/user-guide/assets/start-chat/preview-docx.png new file mode 100644 index 0000000000..2f3815ae51 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/preview-docx.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/reference-image.png b/doc/docs/en/user-guide/assets/start-chat/reference-image.png deleted file mode 100644 index c3dd73ba4f..0000000000 Binary files a/doc/docs/en/user-guide/assets/start-chat/reference-image.png and /dev/null differ diff --git a/doc/docs/en/user-guide/assets/start-chat/reference-source.png b/doc/docs/en/user-guide/assets/start-chat/reference-source.png deleted file mode 100644 index 2f894349f6..0000000000 Binary files a/doc/docs/en/user-guide/assets/start-chat/reference-source.png and /dev/null differ diff --git a/doc/docs/en/user-guide/assets/start-chat/refresh-chat.png b/doc/docs/en/user-guide/assets/start-chat/refresh-chat.png new file mode 100644 index 0000000000..95a18ecf79 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/refresh-chat.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/self-correction.png b/doc/docs/en/user-guide/assets/start-chat/self-correction.png new file mode 100644 index 0000000000..44fed02881 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/self-correction.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/share.png b/doc/docs/en/user-guide/assets/start-chat/share.png new file mode 100644 index 0000000000..f3a2589762 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/share.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/source.png b/doc/docs/en/user-guide/assets/start-chat/source.png new file mode 100644 index 0000000000..de3d855671 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/source.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/tool-call.png b/doc/docs/en/user-guide/assets/start-chat/tool-call.png new file mode 100644 index 0000000000..78d5cc9f07 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/tool-call.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/upload_file.png b/doc/docs/en/user-guide/assets/start-chat/upload_file.png new file mode 100644 index 0000000000..8d6a896963 Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/upload_file.png differ diff --git a/doc/docs/en/user-guide/assets/start-chat/verification.png b/doc/docs/en/user-guide/assets/start-chat/verification.png new file mode 100644 index 0000000000..6deb87587a Binary files /dev/null and b/doc/docs/en/user-guide/assets/start-chat/verification.png differ diff --git a/doc/docs/en/user-guide/auto-tasks.md b/doc/docs/en/user-guide/auto-tasks.md new file mode 100644 index 0000000000..bacf94955a --- /dev/null +++ b/doc/docs/en/user-guide/auto-tasks.md @@ -0,0 +1,262 @@ +# Automation Tasks + +Automation tasks let an agent perform work at a future time or on a recurring schedule. Describe what to do and when to do it in a conversation. Nexent generates a pending task proposal for your review; after you confirm it, the task remains linked to that conversation and writes every run result back to it. + +For example, you can ask an agent to: + +- summarize project progress every day at 9:00 AM; +- check service health every 30 minutes; +- generate a weekly report tomorrow at 3:00 PM. + +> **Important:** An automation proposal only creates a schedule. It does not perform the requested business action immediately. The task starts running on its schedule only after you confirm its creation. + +## Before You Start + +Before creating a task, make sure that: + +1. a working language model is configured under [Model Configuration](./agent-development/model-configuration); +2. an agent is created and saved under [Agent Development](./agent-development); +3. the agent has the tools, knowledge bases, Skills, memory, or other agents required by the task; +4. you can access the conversation used to create the task. + +If a required capability is missing, the proposal asks you to configure the agent before the task can be created. + +## Create an Automation Task + +### 1. Open the Creation Entry + +Select **Automation Tasks** in the left navigation, then click **Create in chat** in the upper-right corner. Nexent opens a new conversation. + +You can also open [Start Chat](./start-chat) directly, select an agent, and submit a scheduled request. + +
+ Automation Tasks entry and task list +
+ +### 2. Select an Agent and Describe the Task + +Select the agent that should perform the task. In the same message, specify: + +- **Business action:** what one run should accomplish; +- **Execution time:** a clear future date and time for a one-time task; +- **Recurrence:** a fixed interval or calendar schedule for a recurring task; +- **Time zone:** specify an IANA time zone such as `Asia/Shanghai` or `UTC` when it differs from the default; +- **End condition:** include an end time or maximum run count when needed. + +Recommended wording: + +```text +Every day at 9:00 AM, summarize yesterday's project progress and list the issues that need attention. +``` + +One-time example: + +```text +Generate this week's project report tomorrow at 3:00 PM. +``` + +Fixed-interval example: + +```text +Check the service status every 30 minutes and list any unhealthy services. +``` + +If the request is missing the business action, date, time, or recurrence, the agent asks for the most important missing detail. Immediate requests, questions about data at a particular time, and requests that only explain a time expression are not treated as automation tasks. + +
+ Describe an automation task in chat +
+ +### 3. Review the Proposal + +When Nexent detects an automation request, it displays a task proposal in the conversation. Review the following fields: + +- **Task title:** the name shown in the task list; +- **Task instructions:** the single-run instruction executed on every trigger; +- **Agent:** the agent that will perform the task; +- **Schedule:** one-time or recurring mode, start time, time zone, and recurrence rule; +- **Capability status:** whether the agent currently has the required capabilities. + +Nexent separates phrases such as “every day at 9:00 AM” from the task instructions and saves them in the schedule. This is expected—the instructions describe one run, while the schedule controls when it runs. + +
+ Automation task proposal card +
+ +### 4. Edit the Proposal + +Click **Edit** in the upper-right corner of the proposal card to change: + +- task title; +- task instructions; +- schedule mode: run once or recurring; +- start time; +- time zone; +- recurrence rule: a Cron expression or fixed interval. + +Fixed intervals are entered in seconds. The page and backend validate the minimum interval according to the deployment settings. Cron uses the standard five-field format: + +```text +minute hour day month weekday +``` + +Common examples: + +| Requirement | Cron expression | +| --- | --- | +| Every day at 9:00 AM | `0 9 * * *` | +| Every weekday at 6:30 PM | `30 18 * * 1-5` | +| At the start of every hour | `0 * * * *` | +| At 9:00 AM on the first day of every month | `0 9 1 * *` | + +Cron is evaluated in the time zone shown in the proposal. A one-time execution time must be in the future. + +
+ Edit an automation task proposal +
+ +### 5. Confirm Creation + +After verifying the proposal, click **Create task**. The proposal card displays the task ID after creation succeeds. Return to **Automation Tasks** to find the new task in the list; click its name to open the linked conversation. + +If the proposal reports missing capabilities, click **Configure agent**, add the required capabilities, then return to the conversation and create the task again. One conversation can have only one active automation task. Start a new conversation when you need another task. + +
+ Created automation task in the task list +
+ +## Manage Automation Tasks + +Open **Automation Tasks** from the left navigation to see tasks created by the current user. The list shows: + +- task name and linked conversation; +- executing agent; +- current status; +- one-time or recurring schedule; +- next run time; +- latest run result. + +Click a task name to open its linked conversation. You can filter by task name, agent name, and status, and use pagination or refresh to update the list. + + +### Run Now + +Click **Run now** in the Actions column to start a manual run without waiting for the schedule. A successful manual run does not change the next scheduled run of a recurring task. + +The linked conversation cannot run multiple agent jobs at the same time. If an agent run or automation run is already active in that conversation, the new run is skipped and appears as **Skipped** in run history. + +### Pause and Resume + +- Click **Pause** to stop future scheduled triggers; +- Click **Resume** to calculate the next run from the current time and the existing schedule; +- A recurring task is **Paused by system** after five consecutive failures or timeouts. Fix the agent configuration or task instructions, then resume it manually; +- A completed one-time task has no future schedule and cannot be resumed. Create a new task or use **Run now** when you need to run it again. + +### Use the More Actions Menu + +The **More actions** menu provides the **Run history**, **Edit**, and **Delete** entries. + +
+ Automation task More actions menu +
+ +### Edit a Task + +Select **More actions** > **Edit** to change: + +- task name; +- task instructions; +- task type and first run time; +- recurrence rule, fixed interval, or Cron expression; +- timeout for one run. + +The executing agent cannot be changed in this dialog. To use a different agent, create a new task from a new conversation. + +The timeout is entered in seconds. Its default is 1,800 seconds (30 minutes), and the current page accepts a minimum of 60 seconds. A run that exceeds the timeout is marked **Timed out**. + +### View Run History + +Select **More actions** > **Run history** to view: + +- run status; +- trigger type: manual or scheduled; +- scheduled time; +- error log; +- available run actions. + +You can cancel a **Queued** or **Running** run. A finished run record can be deleted without deleting the task. Deleted run records cannot be recovered. + +
+ Automation task run history +
+ +### Delete a Task + +Select **More actions** > **Delete** and confirm to stop all future scheduled runs. The linked conversation and its message history remain available. If the task is running, Nexent also requests cancellation of the active run. + +Conversely, deleting the linked conversation also deletes its automation task and cancels active runs. + +## Status Reference + +### Task Statuses + +| Status | Meaning | +| --- | --- | +| Enabled | The task is waiting for its next scheduled run | +| Running | A run is currently in progress | +| Paused | The user paused the task | +| Paused by system | The system paused the task after repeated failures, timeouts, or an invalid schedule | +| Completed | A one-time task finished, or a recurring task reached its end condition | + +### Run Statuses + +| Status | Meaning | +| --- | --- | +| Queued | The run was created and is waiting to start | +| Running | The agent is executing the task | +| Succeeded | The run completed successfully | +| Failed | The run failed because of a capability, configuration, or execution error | +| Skipped | Another run was already active in the linked conversation | +| Canceled | The user canceled the run | +| Timed out | The run exceeded the task timeout | + +## Limitations and Notes + +- **One task per conversation:** A conversation can have only one active automation task. Use separate conversations for separate tasks. +- **Temporary attachments are not persistent inputs:** The current version cannot use an attachment from the proposal message as long-term automation input. Describe a stable data source instead, or configure a knowledge base or tool on the agent. +- **Dependencies are checked before every run:** A run fails if a required tool, knowledge base, Skill, memory configuration, or other agent has been deleted or is unavailable. Check the error log in run history. +- **Results are written to the linked conversation:** Each run's instruction and output are saved in the conversation, which you can open by clicking the task name. +- **Missed recurring runs are not replayed:** Recurring triggers missed while the service is unavailable are skipped. After recovery, Nexent calculates the next future run. +- **Tasks are visible to their creator:** In normal multi-user mode, task lists and run histories are isolated by tenant and creating user. + +## Frequently Asked Questions + +### Why was no proposal generated? + +Make sure the message includes both a specific business action and a future time or recurrence. A vague request such as “keep an eye on this regularly” is missing an actionable task and schedule. Immediate requests do not create automation proposals. + +### Why can't I create a task with an attachment? + +A temporary attachment is not a reliable input for future runs. Put the content in a knowledge base or another stable data source, configure it on the agent, then describe what the task should process. + +### Why can't the task be created? + +Common causes include a time in the past, incomplete time or recurrence details, an invalid Cron expression, an interval below the system limit, missing agent capabilities, or another active task already linked to the conversation. + +### Why is a run marked Skipped? + +An agent run or another automation run was already active in the linked conversation. Nexent avoids concurrent writes to the same conversation and does not start the new run. + +### Why is the task Paused by system? + +A recurring task is automatically paused after five consecutive failures or timeouts, or when its schedule is invalid during recovery. Review run history and the latest error, fix the agent capability, model, tool, or task instructions, then resume the task. + +### Does deleting a task delete its conversation? + +No. Deleting a task stops future runs but keeps the linked conversation. However, deleting the linked conversation also deletes its automation task. + +## Next Steps + +- [Start Chat](./start-chat): select an agent and create an automation task in natural language. +- [Agent Development](./agent-development): configure the model, tools, knowledge bases, Skills, and memory required by the task. +- [Model Configuration](./agent-development/model-configuration): verify the language model used by the task. diff --git a/doc/docs/en/user-guide/home-page.md b/doc/docs/en/user-guide/home-page.md index 9433594f3d..abc9d316d7 100644 --- a/doc/docs/en/user-guide/home-page.md +++ b/doc/docs/en/user-guide/home-page.md @@ -41,8 +41,8 @@ Use the language switcher in the top-right corner to toggle between Simplified C We recommend configuring the platform in this order: -1. Set up **[Model Management](./model-management)** to define app details and connect AI models. -2. Create your **[Knowledge Base](./knowledge-base)** and upload documents. +1. Set up **[Model Management](./agent-development/model-configuration.md)** to define app details and connect AI models. +2. Create your **[Knowledge Base](./agent-development/knowledge-configuration.md)** and upload documents. 3. Conduct **[Agent Development](./agent-development)** on top of the models and knowledge base. 4. When everything is ready, chat with your agents via **[Start Chat](./start-chat)**. @@ -50,4 +50,4 @@ Alternatively, you can click the "Quick Setup" button on the homepage or in the ## 💡 Get Help -Need help? Check the **[FAQ](../quick-start/faq)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). \ No newline at end of file +Need help? Check the **[FAQ](../quick-start/faq.md)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). \ No newline at end of file diff --git a/doc/docs/en/user-guide/knowledge-base.md b/doc/docs/en/user-guide/knowledge-base.md deleted file mode 100644 index 05456e5fa0..0000000000 --- a/doc/docs/en/user-guide/knowledge-base.md +++ /dev/null @@ -1,85 +0,0 @@ -# Knowledge Base - -Create and manage knowledge bases, upload documents, and generate summaries. Knowledge bases are critical information sources that let agents securely use your private data. - -## 🔧 Create a Knowledge Base - -1. Click **Create Knowledge Base** -2. Enter a descriptive, unique name - > **Note:** Knowledge base names must be unique and can only contain Chinese characters or lowercase letters. Spaces, slashes, and other special characters are not allowed. - -## 📁 Upload Files - -### Upload Files - -1. Select a knowledge base from the list -2. Click the upload area to pick files (multi-select supported) or drag them in directly -3. Nexent automatically parses files, extracts text, and vectorizes the content -4. Track the processing status in the list (Parsing/Ingesting/Ready) - -![File Upload](./assets/knowledge-base/create-knowledge-base.png) - -💡 Hover over the status to understand the progress and error reasons - -![File Upload](./assets/knowledge-base/tip.png) - -### Supported File Formats - -Nexent supports multiple file formats, including: -- **Text:** .txt, .md, .csv, .json -- **PDF:** .pdf -- **Word:** .docx -- **PowerPoint:** .pptx -- **EPUB:** .epub -- **Excel:** .xlsx -- **Data files:** .csv -- **Web content:** .html, .xml - -## 📊 Knowledge Base Summary - -Give every knowledge base a clear summary so agents can pick the right source during retrieval. - -1. Click **Details** to open the detailed view -2. Choose a model and click **Auto Summary** to generate a description -3. Edit the generated text to improve accuracy -4. Click **Save** to store your changes - -![Content Summary](./assets/knowledge-base/summary-knowledge-base.png) - -## 🔧 Using Knowledge Bases - -Nexent supports binding knowledge bases to agents individually. When creating an agent, **enable the knowledge_base_search tool** and select the associated knowledge base. - -Tool 1 - -![Tool 2](./assets/knowledge-base/knowledge-tool2.png) - -## 🔍 Knowledge Base Management - -### View Knowledge Bases - -1. **Knowledge Base List** - - The left column lists every created knowledge base - - Shows the name, file count, creation time, and more -2. **Knowledge Base Details** - - Click a knowledge base to see all documents - - Click **Details** to view or edit the summary - -### Edit Knowledge Bases - -1. **Delete Knowledge Base** - - Click **Delete** to the right of the knowledge base row - - Confirm the deletion (irreversible) - -2. **Delete or Add Files** - - Inside the file list, click **Delete** to remove a document - - Use the upload area under the list to add new files - -## 🚀 Next Steps - -After completing knowledge base configuration, we recommend you continue with: - -1. **[Agent Development](./agent-development)** – Create and configure agents -2. **[Start Chat](./start-chat)** – Interact with your agent - -Need help? Check the **[FAQ](../quick-start/faq)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). \ No newline at end of file diff --git a/doc/docs/en/user-guide/local-tools/index.md b/doc/docs/en/user-guide/local-tools/index.md index 9006f415c8..41575a03ff 100644 --- a/doc/docs/en/user-guide/local-tools/index.md +++ b/doc/docs/en/user-guide/local-tools/index.md @@ -6,11 +6,11 @@ Local tools let agents interact with the workspace, remote hosts, and external s - [File Tools](./file-tools): Create/read/move/delete files and folders; list directory trees. - [Email Tools](./email-tools): Receive IMAP mail; send HTML mail with CC/BCC. -- [Search Tools](./search-tools): Local/DataMate KB search plus Exa/Tavily/Linkup web search. +- [Search Tools](./search-tools): Local/AIDP/DataMate/Dify KB search plus Exa/Tavily/Linkup web search. - [Multimodal Tools](./multimodal-tools): Download/parse/analyze text files and images. - [Terminal Tool](./terminal-tool): Persistent SSH sessions for remote commands. - [SQL Tools](./sql-tools): Connect to MySQL, PostgreSQL, SQL Server to execute SQL queries. -- [Skills](../skills): Nexent's built-in tool combinations or custom capability packs with NL generation and version management. +- [Skills](../resource-repository/skill-repository.md): Nexent's built-in tool combinations or custom capability packs with NL generation and version management. ## ⚙️ Configuration Entry diff --git a/doc/docs/en/user-guide/local-tools/search-tools.md b/doc/docs/en/user-guide/local-tools/search-tools.md index 04bb368169..832ef6ce1a 100644 --- a/doc/docs/en/user-guide/local-tools/search-tools.md +++ b/doc/docs/en/user-guide/local-tools/search-tools.md @@ -4,12 +4,13 @@ title: Search Tools # Search Tools -Search tools cover internet search plus local, DataMate, and Dify knowledge bases, useful for real-time info, industry materials, and private docs. +Search tools cover internet search plus local, AIDP, DataMate, and Dify knowledge bases, useful for real-time info, industry materials, private docs, and multimodal enterprise KB retrieval. ## 🧭 Tool List - Local/private knowledge bases: - `knowledge_base_search`: Local KB search with multiple modes + - `aidp_search`: Search AIDP enterprise KBs via multimodal FusionSearch - `datamate_search`: Search DataMate KB - `dify_search`: Search Dify KB - Public web search: @@ -20,6 +21,7 @@ Search tools cover internet search plus local, DataMate, and Dify knowledge base ## 🧰 Example Use Cases - Retrieve internal docs, specs, and industry references (KB, DataMate, Dify) +- Query enterprise AIDP KBs for documents, tables, images, or technical drawings (AIDP) - Fetch latest news or web evidence (Exa / Tavily / Linkup) - Return image references alongside text (with optional filtering) @@ -59,6 +61,27 @@ Search tools cover internet search plus local, DataMate, and Dify knowledge base - `search_method`: Search method options: `keyword_search`, `semantic_search`, `full_text_search`, `hybrid_search`, default `semantic_search`. - Returns title, content, score, etc. +### aidp_search +- **Configuration Parameters**: + - `server_url`: AIDP API base URL, e.g. `https://141.111.61.70:30080`. + - `api_key`: AIDP API key, typically prefixed with `ak_`, issued by the AIDP platform admin. + - `tenant_id`: Tenant identifier used in AIDP API paths, e.g. `aidp`. + - `kds_list`: JSON string array of knowledge base IDs (`kds_id`) to search (e.g. `["aidp-kb-01", "aidp-kb-02"]`). Determines which AIDP KBs the tool accesses by default. + - `search_method`: Search method options: `hybrid_search` (default, fusion), `vector_search` (vector), `full_text_search` (full text). + - `reranking_enable`: Whether to enable reranking, default True. + - `reranking_mode`: Reranking mode options: `performance` (default) / `high_accuracy`. + - `rewrite_enable`: Whether to enable query rewrite, default False. + - `related_search_enable`: Whether to enable related-chunk retrieval, default False. + - `score_threshold`: Similarity threshold (0–1), default 0.0. + - `top_k`: Number of results to return (1–100), default 10. + - `multi_modal`: Whether to return multimodal chunks (image/table), default True. +- **Search Parameters**: + - `query`: Required. + - `kds_list`: Optional. Knowledge base IDs to search this time; falls back to the configured `kds_list` when omitted. +- Returns text, table, and image chunks via dual-channel output: all chunks as `SEARCH_CONTENT`, with image `file_url`s also delivered as `PICTURE_WEB`. +- The search scope is filtered by the current chat user's AIDP permission whitelist. Whether the default `kds_list` or an LLM-supplied one is used, it is intersected with KBs the user is allowed to access — unauthorized KBs are silently dropped. +- If no KB is accessible after filtering, the tool returns a clear no-permission message instead of silent empty results. + ### exa_search / tavily_search / linkup_search - **Configuration Parameters**: - `exa/tavily/linkup_api_key`: API key for the respective service @@ -75,13 +98,17 @@ Search tools cover internet search plus local, DataMate, and Dify knowledge base ## 🛠️ How to Use -1. **Pick the source**: Use `knowledge_base_search`, `datamate_search`, or `dify_search` for private data; Exa/Tavily/Linkup for public info. -2. **Tune mode/count**: Switch `search_mode` for KB; adjust `max_results` and image filtering for public search. -3. **Scope**: Provide `index_names` for targeted KB search; tune `top_k` and `threshold` for DataMate precision. -4. **Consume results**: JSON output is ready for answers or summarization, with citation indices for referencing. +1. **Pick the source**: Use `knowledge_base_search`, `aidp_search`, `datamate_search`, or `dify_search` for private data; Exa/Tavily/Linkup for public info. +2. **Tune mode/count**: Switch `search_mode`/`search_method` for KB; adjust `max_results` and image filtering for public search. +3. **Fill connection and auth parameters**: AIDP requires `server_url`, `api_key`, and `tenant_id` — run a test connection in the platform's secure config first. +4. **Scope the searchable KBs**: For AIDP, use `kds_list` in the tool configuration to pick the default KBs; the actual search is also filtered by the current chat user's AIDP permission whitelist. +5. **Narrow the query**: Provide `index_names` (local KB) or an explicit `kds_list` (AIDP) to scope results; tune `top_k` and `threshold` for DataMate precision. +6. **Rerank (optional)**: Set `enable_rerank: true` or `reranking_enable: true`, and tune the model/mode parameters for better relevance. +7. **Consume results**: JSON output is ready for answers or summarization, with citation indices for referencing. ## 🛡️ Safety & Best Practices -- Store API keys in the platform’s secure config, never in prompts. +- Store credentials (`api_key`, etc.) for public search and AIDP in the platform's secure config — never expose them in prompts. +- AIDP searches respect the current chat user's permissions. If a user cannot access a KB, it will not be returned even if the KB is in the tool's `kds_list` — contact the AIDP admin to grant the right permissions. - Sync KB content before querying to avoid stale answers. - If queries are too broad, shorten or split them; if images are over-filtered, disable filtering to review raw URLs. diff --git a/doc/docs/en/user-guide/mcp-tools.md b/doc/docs/en/user-guide/mcp-tools.md deleted file mode 100644 index cd1190e0e0..0000000000 --- a/doc/docs/en/user-guide/mcp-tools.md +++ /dev/null @@ -1,159 +0,0 @@ -# MCP Tools - -In the MCP Tools module, you can centrally manage all MCP (Model Context Protocol) servers and tools. It supports custom addition, Registry import, and Community import, covering connection configuration, tool synchronization, health monitoring, and community sharing. - -The MCP Tools page has two parallel tabs: - -- **Imported Services**: Manage MCP services already accessed by the current tenant — configure, monitor, and maintain your MCP services here. -- **Published Services**: Manage the MCP services you have published to the community — browse, edit, and unpublish. - ---- - -## ➕ Add MCP Services - -Click the **Add MCP Service** button to open the add dialog. The dialog provides three tabs, each corresponding to a different source. - -### Local Add - -The **Local Add** tab lets you manually configure an MCP service with two transport types. - -#### Add via URL - -For independently deployed MCP services (HTTP / SSE), connect by entering the endpoint URL. - -1. In the **Local Add** tab, set **Transport Type** to "URL" -2. Fill in the service details: - - **Service Name (required)**: A recognizable name for the MCP service - - **Service URL (required)**: The MCP service endpoint address - - **Description** (optional): A brief description of the service - - **Authorization Token** (optional): Bearer token if the service requires authentication -3. Click **Confirm** — the system will connect to the service and retrieve the available tool list - -#### Add via Container Configuration - -For MCP services that need to run locally in a container (e.g., services launched via npx), the system automatically creates and manages a container based on your JSON configuration. - -1. In the **Local Add** tab, set **Transport Type** to "Container" -2. Fill in the container configuration: - - **Service Name (required)**: A recognizable name for the MCP service - - **Description** (optional): A brief description of the service - - **Container Configuration JSON (required)**: Enter the standard MCP configuration format, for example: - ```json - { - "mcpServers": { - "service-name": { - "args": ["mcp-package-name@version"], - "command": "npx", - "env": { - "API_KEY": "xxxx" - } - } - } - } - ``` - - **Port**: The port exposed by the container service — the system automatically detects port conflicts and suggests available ports -3. Click **Confirm** — the system parses the JSON, creates the container, and registers the service - -### Import from MCP Registry - -Nexent integrates with the MCP Registry, allowing you to browse and import community-maintained MCP services in one click. - -1. Switch to the **MCP Registry** tab -2. Browse the available MCP services — search by name or tags -3. Click a service to view its details (description, version, required parameters, etc.) -4. Configure required parameters (e.g., API Key and other environment variables) -5. Click **Import** — the system automatically installs and configures the service - -### Import from Community - -Browse MCP services published by other Nexent users and quickly import them. - -1. Switch to the **Community Market** tab -2. Browse published community MCP services — filter by name, tags, or transport type -3. Click a service to view details, then click **Import** to add it to your service list - ---- - -## 📋 Imported Services - -The **Imported Services** tab displays all MCP services accessed by the current tenant as cards. View, edit, monitor, and publish your services here. - -### View & Filter - -Each service card shows: - -- Service name and description -- Source indicator (Custom / Registry / Community) -- Enable / Disable toggle -- Tags - -Use the filter bar at the top to filter by **Source**, **Transport Type**, and **Tags**, or use the search box to quickly locate services by name. - -### Edit Service Details - -Click any service card to open the detail modal, where you can: - -- **Edit basic info**: Modify name, description, URL, Authorization Token, and tags -- **Enable / Disable**: Toggle the service on or off — tools from a disabled service will not appear in agent tool selection -- **Delete**: Remove the MCP service record — containerized services will also have their container resources cleaned up - -### View Tool List - -In the service detail modal, click **Tool List** to view all tools provided by this MCP service. - -### Health Check - -Click the **Health Check** button in the detail modal to test the connection to the MCP service. Possible statuses: - -- **Healthy**: The service is reachable -- **Unhealthy**: The service cannot be reached or responded abnormally -- **Unchecked**: A health check has not been performed yet - -### Container Management - -For containerized MCP services, the detail modal also provides: - -- **View Container Logs**: Real-time logs from the running container for troubleshooting -- **View Container Config**: The configuration JSON used when creating the container - -### Publish to Community - -In the service detail modal, click **Publish to Community**: - -1. Review or edit the publication info (name, description, tags, etc.) -2. Click **Confirm Publish** — the service will be published to the community -3. Other users can then browse and import it from the **Community Market** tab in the add dialog - ---- - -## 🌐 Published Services - -The **Published Services** tab shows all MCP services you have published to the community. Manage your published content here. - -Each card shows the service name, description, version, and tags. Filter by name, tags, and transport type. - -Click a service card to view details, where you can: - -- **Edit published service**: Modify the published service's name, description, and tags -- **Delete published service**: Withdraw the service from the community — it will no longer be visible to other users - ---- - -## 🔗 Integrating with Agents - -Once an MCP service is added, its tools are automatically synced to the agent tool selection list. When configuring an agent on the **[Agent Development](./agent-development)** page: - -1. In the **Select Agent Tools** tab, locate the corresponding MCP service group -2. Click a tool name to enable it -3. Click ⚙️ to view the tool description and configure its parameters - -## 🚀 Next Steps - -After configuring MCP services, we recommend: - -1. **[Agent Development](./agent-development)** — Assign MCP tools to your agents -2. **[Agent Space](./agent-space)** — View collaboration between agents and MCP services -3. **[Start Chat](./start-chat)** — Experience agents calling MCP tools in conversations - -If you encounter any issues, please refer to our **[FAQ](../quick-start/faq)** or ask for support in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). - diff --git a/doc/docs/en/user-guide/memory-management.md b/doc/docs/en/user-guide/memory-management.md deleted file mode 100644 index 0caffb7e1f..0000000000 --- a/doc/docs/en/user-guide/memory-management.md +++ /dev/null @@ -1,140 +0,0 @@ -# Memory Management - -Nexent’s intelligent memory system gives agents persistent context. With multi-level memories, agents can remember key facts across conversations, retrieve them automatically, and deliver more personalized answers. - -## 🎯 What the Memory System Does - -The memory system lets agents “remember” important information and reuse it later without you repeating yourself. - -### Core Benefits - -- **Cross-conversation memory** – Agents keep track of important facts from earlier chats. -- **Automatic retrieval** – Relevant memories are pulled in automatically. -- **Personalized service** – Responses adapt to user preferences and habits. -- **Knowledge accumulation** – Agents keep getting smarter the more you use them. - -## ⚙️ System Configuration - -### Access Memory Management - -1. Click **Memory Management** in the left navigation. -2. Open the **System Configuration** section. - -### Base Settings - -| Setting | Options | Default | Description | -| --- | --- | --- | --- | -| Memory Service Status | Enable / Disable | Enable | Controls whether the memory system runs. | -| Agent Memory Sharing Strategy | Always Share / Ask Every Time / Never Share | Always Share | Defines if agents can share memories without user confirmation. | - -
- Memory configuration -
- -**Setting Tips** - -- **Memory service status** – Disable it if you want a completely stateless experience; enable it to unlock all memory features. -- **Sharing strategy** - - *Always Share* – Agents exchange memories automatically. - - *Ask Every Time* – You approve each sharing request. - - *Never Share* – Agents stay isolated. - -## 📚 Memory Levels - -Nexent uses four storage levels so you can keep global knowledge and private facts separate. - -### Tenant-Level - -- **Scope:** Entire organization, shared by all users and agents. -- **Stores:** SOPs, compliance policies, org charts, long-term facts. -- **Best for:** Company-wide knowledge and governance. -- **Managed by:** Tenant administrators. - -### Agent-Level - -- **Scope:** A specific agent, shared by everyone using it. -- **Stores:** Domain knowledge, skill templates, historical summaries. -- **Best for:** Letting an agent accumulate expertise over time. -- **Managed by:** Tenant administrators. - -### User-Level - -- **Scope:** A single user account. -- **Stores:** Personal preferences, habits, favorite commands, personal info. -- **Best for:** Tailoring the platform to a specific user. -- **Managed by:** That user. - -### User-Agent Level - -- **Scope:** A specific agent used by a specific user (most granular). -- **Stores:** Collaboration history, personal facts, task context. -- **Best for:** Deep personalization and long-running projects. -- **Managed by:** That user. - -### Retrieval Priority - -When an agent retrieves memory it follows this order (high ➝ low): - -1. Tenant Level – shared facts and policies. -2. User-Agent Level – very specific context for that pairing. -3. User Level – general personal preferences. -4. Agent Level – the agent’s professional knowledge. - -## 🤖 Automated Memory Management - -The system takes care of most work for you: - -- **Smart extraction:** Detects key facts in conversations, creates memory entries automatically, and stores them at the right level—no manual input needed. -- **Automatic context embedding:** Retrieves the most relevant memories and implicitly injects them into the conversation context so agents respond with better accuracy. -- **Incremental updates:** Gradually refreshes or removes outdated memories to keep the store clean, timely, and reliable. - -## ✋ Manual Memory Operations - -Need full control? Manage entries manually. - -### Add a Memory - -1. Choose the level (tenant / agent / user / user-agent) and target agent. -2. Click the green **+** button. -3. Enter up to 500 characters describing the fact. -4. Click the check mark to save. - -
- Add memory -
- -### Delete Memories - -- **Delete group:** Click the red ✕ icon to remove every entry under that agent group (confirm in the dialog). -- **Delete single entry:** Click the red eraser icon to remove one entry. - -
- Delete memory -
- -## 💡 Usage Tips - -### Memory Content Guidelines - -1. **Keep entries atomic:** Each memory should contain *one* clear fact. - - ✅ Good: “The user prefers dark mode.” - - ❌ Not good: “The user prefers dark mode, works nights, and loves coffee.” -2. **Maintain freshness:** Review and remove outdated entries regularly. -3. **Protect privacy:** Store sensitive info at the user or user-agent level instead of tenant level. - -### Best Practices - -- Pick the memory level that matches the sharing needs. -- Let automation handle routine facts; manually add critical knowledge. -- Review the memory list periodically to keep everything relevant. -- Keep personal or sensitive data scoped tightly to the right user. - -## 🚀 Next Steps - -With memory configured you can: - -1. Experience the new continuity in **[Start Chat](./start-chat)**. -2. Manage all agents in **[Agent Space](./agent-space)**. -3. Build more agents inside **[Agent Development](./agent-development)**. - -Need help? Check the **[FAQ](../quick-start/faq)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). \ No newline at end of file diff --git a/doc/docs/en/user-guide/model-management.md b/doc/docs/en/user-guide/model-management.md deleted file mode 100644 index 3b5a81955b..0000000000 --- a/doc/docs/en/user-guide/model-management.md +++ /dev/null @@ -1,230 +0,0 @@ -# Model Management - -In the Model Management module, you can configure your app’s basic information and connect every model the platform needs, including large language models, embedding models, and vision-language models. Nexent supports multiple providers so you can pick the best option for each scenario. - -## 🖼️ App Configuration - -App configuration is the first step of model management. Configure the icon, name, and description so users can instantly recognize the app and the platform can pass the proper context to models. - -- The icon and name appear in the upper-left corner of the chat page. -- The description is used as background information when generating agents to improve the model’s understanding of your use case. - -### App Icon Configuration - -Click the app icon to open the configuration panel. Nexent provides two options: - -- **Use a preset icon**: Pick an icon from the built-in gallery and optionally change the background color for fast setup. -- **Upload a custom image**: Supports PNG and JPG (≤2 MB). - -
- - -
- -### App Name & Description - -#### App Name - -- Displayed on the chat page, helping users recognize the current app. -- Keep it short, descriptive, and free of special characters. - -#### App Description - -- Passed to the model as background context. -- Highlight the core capabilities and keep the text fluent and concise. - -
- -
- -## 🤖 Model Configuration - -### 🔄 Sync ModelEngine Models - -Nexent supports seamless integration with the ModelEngine platform. - -👉 Click **Edit ModelEngine Configuration** in the upper right corner of the page, enter your API key, and you can retrieve all models deployed on ModelEngine. - -### 🛠️ Add Custom Models - -#### Add a Single Model - -1. **Add a custom model** - - Click **Add Custom Model** to open the dialog. -2. **Select model type** - - Choose Large Language Model, Embedding Model, or Vision Language Model. -3. **Configure model parameters** - - **Model Name (required):** The name you send in API requests. - - **Display Name:** Optional label shown in the UI (defaults to the model name). - - **Model URL (required):** API endpoint from the provider. - - **API Key:** Your provider key. - -> ⚠️ **Notes** -> 1. Model names usually follow `series/model`. Example: `Qwen/Qwen3-8B`. -> 2. API endpoints come from the provider docs. For SiliconFlow, examples include `https://api.siliconflow.cn/v1` (LLM, VLM) and `https://api.siliconflow.cn/v1/embeddings` (embedding). -> 3. Generate API keys from the provider’s key management console. - -4. **Connectivity verification** - - Click **Verify** to send a test request and confirm connectivity. -5. **Save model** - - Click **Add** to place the model in the available list. - -
- -
- -#### Batch Add Models - -Use batch import to speed up onboarding: - -1. Enable the **Batch Add Models** toggle in the dialog. -2. Select a **model provider**. -3. Choose the **model type** (LLM/Embedding/Vision). -4. Enter the **API Key** (required). -5. Click **Fetch Models** to retrieve the provider list. -6. Toggle on the models you need (disabled by default). -7. Click **Add** to save every selected model at once. - -
- -
- -### 🔧 Edit Custom Models - -Modify or delete models anytime: - -1. Click **Edit Custom Models**. -2. Select the model type (LLM/Embedding/Vision). -3. Choose between batch editing or single-model editing. -4. For batch edits, toggle models on/off or click **Edit Config** in the upper-right to change settings in bulk. -5. For single models, click the trash icon 🗑️ to delete, or click the model name to open the edit dialog. - -
- - -
-
-
- - -
-
-
- - -
- -### ⚙️ Configure System Models - -After adding models, assign the platform-level defaults. These models handle system tasks such as title generation, real-time file reading, and multimodal parsing. Individual agents can still choose their own run-time models. - -#### Base Model - -- Used for core platform features (title generation, real-time file access, basic text processing). -- Choose any added large language model from the dropdown. - -#### Embedding Model - -Embedding models are primarily used for vectorization processing of text, images, and other data in knowledge bases, forming the foundation for efficient retrieval and semantic understanding. Configuring an appropriate embedding model can significantly improve knowledge base search accuracy and multimodal data processing capabilities. - -- Click the embedding model dropdown to select one from the added embedding models. -- Embedding model configuration affects the stable operation of knowledge bases. - -Choose appropriate document chunk size and chunks per request based on model capabilities. Smaller chunks provide more stability, but may affect file parsing quality. - -
- -
- -#### Vision-Language Model - -- Required for multimodal chat scenarios (for example, when users upload images). -- Pick one of the added vision-language models. - -
- - - -
- -### ✅ Check Model Connectivity - -Run regular connectivity checks to keep the platform healthy: - -1. Click **Check Model Connectivity**. -2. Nexent tests every configured system model automatically. - -Status indicators: - -- 🔵 **Blue dot** – Checking in progress. -- 🔴 **Red dot** – Connection failed; review configuration or network. -- 🟢 **Green dot** – Connection is healthy. - -Troubleshooting tips: - -- Confirm network stability. -- Ensure the API key is valid and not expired. -- Check the provider’s service status. -- Review firewall and security policies. - -### 🤖 Supported Providers - -#### Large Language Models - -Nexent supports any **OpenAI-compatible** provider, including: - -- [SiliconFlow](https://siliconflow.cn/) -- [Ali Bailian](https://bailian.console.aliyun.com/) -- [TokenPony](https://www.tokenpony.cn/) -- [DeepSeek](https://platform.deepseek.com/) -- [OpenAI](https://platform.openai.com/) -- [Anthropic](https://console.anthropic.com/) -- [Moonshot](https://platform.moonshot.cn/) - -Getting started: - -1. Sign up at the provider’s portal. -2. Create and copy an API key. -3. Locate the API endpoint (usually ending with `/v1`). -4. Click **Add Custom Model** in Nexent and fill in the required fields. - -#### Multimodal Vision Models - -Use the same API key and URL as LLMs but specify a multimodal model name, for example **Qwen/Qwen2.5-VL-32B-Instruct** on SiliconFlow. - -#### Embedding Models - -Use the same API key as LLMs but typically a different endpoint (often `/v1/embeddings`), for example **BAAI/bge-m3** from SiliconFlow. - -#### Speech Models - -Currently only **VolcEngine Voice** is supported and must be configured via `.env`: - -- **Website:** [volcengine.com/product/voice-tech](https://www.volcengine.com/product/voice-tech) -- **Free tier:** Available for individual use -- **Highlights:** High-quality Chinese/English TTS - -Steps: - -1. Register a VolcEngine account. -2. Enable the Voice Technology service. -3. Create an app and generate an API key. -4. Configure the TTS/STT settings in your environment. - -## 💡 Need Help - -If you run into provider issues: - -1. Review the provider’s documentation. -2. Check API key permissions and quotas. -3. Test with the provider’s official samples. -4. Ask the community in our [Discord server](https://discord.gg/tb5H3S3wyv). - -## 🚀 Next Steps - -After closing the Model Management flow, continue with: - -1. **[Knowledge Base](./knowledge-base)** – Create and manage knowledge bases. -2. **[Agent Development](./agent-development)** – Build and configure agents. - -Need help? Check the **[FAQ](../quick-start/faq)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). diff --git a/doc/docs/en/user-guide/quick-setup.md b/doc/docs/en/user-guide/quick-setup.md index 9e251e20d7..5486fecb65 100644 --- a/doc/docs/en/user-guide/quick-setup.md +++ b/doc/docs/en/user-guide/quick-setup.md @@ -13,7 +13,7 @@ Configure basic app information and connect AI models: - **App configuration:** Set the icon, name, and description. - **Model configuration:** Add large language models, embedding models, and vision-language models. -Learn more: [Model Management](./model-management) +Learn more: [Model Management](./agent-development/model-configuration.md) ### Step 2: Knowledge Base @@ -23,7 +23,7 @@ Create knowledge bases and upload documents: - **Upload files:** Support for multiple file formats. - **Generate summaries:** Create concise knowledge-base descriptions. -Learn more: [Knowledge Base](./knowledge-base) +Learn more: [Knowledge Base](./agent-development/knowledge-configuration.md) ### Step 3: Agent Development @@ -38,7 +38,7 @@ Publish agent: - **Publish agent:** Published agents will be visible to selected user groups and listed in Agent Space and the Start Chat selection box. - **Version management:** Track iteration history of agents, support viewing, rolling back to historical versions, and creating new versions. -Learn more: [Agent Development](./agent-development) +Learn more: [Agent Development](./agent-development.md) ## 🎯 Tips @@ -51,8 +51,8 @@ Learn more: [Agent Development](./agent-development) After finishing Quick Setup: -1. Visit **[Agent Space](./agent-space)** to review and manage agents. +1. Visit **[Agent Space](./agent-development.md)** to review and manage agents. 2. Use **[Start Chat](./start-chat)** to talk to your agents. -3. Configure **[Memory Management](./memory-management)** to give agents persistent memory. +3. Configure **[Memory Management](./agent-development/memory-configuration.md)** to give agents persistent memory. -Need help? Check the **[FAQ](../quick-start/faq)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). \ No newline at end of file +Need help? Check the **[FAQ](../quick-start/faq.md)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). \ No newline at end of file diff --git a/doc/docs/en/user-guide/resource-repository/agent-repository.md b/doc/docs/en/user-guide/resource-repository/agent-repository.md new file mode 100644 index 0000000000..b11a0be56e --- /dev/null +++ b/doc/docs/en/user-guide/resource-repository/agent-repository.md @@ -0,0 +1,197 @@ +# Agent Repository + +The Agent Repository is the hub for sharing, managing, and reviewing agents within the same tenant. Browse listed agents and copy them into your workspace, manage agents you can edit, and—if you are an admin—review listing applications. + +## 👥 UI Differences Between Admins and Developers + +After you open **Agent Repository**, the top tabs differ by role: + +| Role | Visible tabs | Extra capabilities | +|------|--------------|-------------------| +| **Developer** | Repository, My Agents | Browse the shared repository, copy agents, manage your agents, and apply for listing | +| **Admin** | Repository, My Agents, Review Center | Everything developers can do, plus review listing applications and take agents down directly from the repository | + +> Note: The Review Center tab is visible to admins only. Developers track their own applications via **View review progress** in **My Agents**. + +**Developer view** (Repository / My Agents): + +
+ Developer tabs +
+ +**Admin view** (Repository / My Agents / Review Center): + +
+ Admin tabs +
+ +--- + +## 📦 Repository + +The **Repository** tab shows agents that are listed (shared) in the current tenant. Tenant members can browse them, open details, and copy them into **My Agents** before editing. + +> Agents shared in the same tenant must be copied as your own agent before you can edit them. + +### Browse and Search + +- Browse listed agents as cards +- Search by **agent name, description, or tags** +- Each card shows: icon, name, author, description, tags, tool count, version, and install count + +
+ Repository list +
+ +### View Details + +Click **Details** on a card to view the full agent information, including: + +- **Basic information**: name, icon, author, version, model, install count, and created time +- **Agent introduction**: description +- **Built-in tools**: enabled tools +- **Agent role**: role settings (Duty Prompt) and related configuration + +
+ Agent details +
+ +### Copy an Agent + +Click **Copy** on a card. The system runs a dependency precheck and shows a configuration checklist: + +1. Review the **copyable percentage**, plus available and pending item counts +2. If there are abnormal items (for example, model not activated, knowledge base not activated, MCP not activated, Skill name conflict, or tool unavailable), follow the prompts to activate or resolve them +3. After fixing issues, refresh to re-run the precheck; you can also continue copying if you accept the risk +4. After a successful copy, the agent appears in **My Agents** for editing and use + +Dependency types usually include: **Model**, **Knowledge Base**, **MCP Service**, **Skill**, and **Tool**. + +
+ Copy configuration checklist +
+ +### Admin Take-Down + +Admins can choose **Take down** from the more menu at the top-right of a repository card. After take-down, the agent is no longer visible to tenant members and can no longer be copied. + +If a developer needs to take down an agent they listed, use the **View review progress** dialog in **My Agents**. + +--- + +## 🧑 My Agents + +The **My Agents** tab manages agents you can edit, including agents you created and agents copied from the repository. + +### Filter and Search + +- **All / Created by me / Others**: filter by ownership +- Search by agent name or description +- Agents are shown as cards with pagination + +### Create and Import + +When the filter is **All** and there is no search query, the page provides: + +- **Create agent**: jump to the agent development page to create a new agent +- **Import agent**: upload and import agent configuration via the import wizard + +
+ My Agents list +
+ +### Agent Card Status + +Each card shows lifecycle and listing status for quick recognition: + +| Badge | Description | +|------|-------------| +| **Draft / Published** | Whether the agent has a published version (only published versions can apply for listing) | +| **Hub** | The agent has repository-related records (previously applied or currently listed) | +| **Under review** | First listing application waiting for admin review | +| **Update under review** | A listed version already exists; a new version is waiting for review | +| **Listed** | Currently shared in the repository | +| **Rejected** | Listing application was rejected; you can revise and re-apply | + +### Common Actions + +On an agent card, you can: + +- **Edit**: open the agent development page to change configuration (not available with read-only permission) +- **View**: view details of the published version +- **Evaluate**: jump to the agent evaluation page +- **More actions**: + - **Apply for listing**: submit the current published version for repository review + - **View review progress / View update review progress**: check application status, and cancel the application or take the agent down + - **Delete**: delete the agent + +### Apply for Listing + +You can apply only when the agent has a published version and the current version is not yet listed: + +1. Click **Apply for listing** in the more menu +2. Fill in the listing information: + - **Agent icon** (required): choose a preset emoji or enter a single custom emoji + - **Agent tags** (required): up to 5 tags; choose presets or enter custom tags + - **Listing note** (optional): extra context for reviewers +3. Click **Submit application** and wait for admin review + +
+ Apply for listing +
+ +### View Review Progress + +Open the review status dialog from the more menu to see: + +- Current status: Under review / Approved / Rejected +- Review version, submitted time, listing note, and review comments (if any) + +Depending on the status, you can also: + +- **Cancel listing application**: withdraw a pending or rejected application +- **Take down**: remove a listed agent from the repository + +
+ Review progress +
+ +--- + +## ✅ Review Center + +**Review Center** is visible to **admins** only. It is used to process listing applications submitted by users in the same tenant. + +### Pending Review Queue + +Pending applications are shown as a list, including: + +- Agent name and icon +- Applied version +- Submitter +- Listing note +- Actions: Details, Approve, Reject + +The tab shows a pending-count badge so admins can handle applications promptly. + +### Review Actions + +1. Click **Details** to preview the agent configuration and confirm capabilities and tools +2. Click **Approve**: optionally add a review comment; after confirmation, the agent is listed in **Repository** +3. Click **Reject**: optionally add a review comment; after rejection, the submitter can revise the agent in **My Agents** and re-apply + +
+ Review confirmation +
+ +--- + +## 🚀 Next Steps + +After managing agents in the Agent Repository, you can: + +1. Interact with agents in **[Start Chat](../start-chat)** +2. Continue creating or iterating agents in **[Agent Configuration](../agent-development/agent-configuration)** +3. Configure **[Memory Configuration](../agent-development/memory-configuration)** to improve agent memory + +If you run into any issues, check the **[FAQ](../../quick-start/faq)** or ask in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). diff --git a/doc/docs/en/user-guide/resource-repository/mcp-repository.md b/doc/docs/en/user-guide/resource-repository/mcp-repository.md new file mode 100644 index 0000000000..87d2acaf3a --- /dev/null +++ b/doc/docs/en/user-guide/resource-repository/mcp-repository.md @@ -0,0 +1,215 @@ +# MCP Repository + +The MCP Repository is the hub for sharing, managing, and reviewing MCP (Model Context Protocol) services within a tenant. You can browse MCP services published to the community and install them with one click, manage the MCP services you have permission to edit, and (as an administrator) review publication applications. + +## 👥 Interface Differences for Administrators and Developers + +After entering **MCP Repository**, the tabs shown at the top of the page vary by role: + +| Role | Visible Tabs | Available Capabilities | +|------|--------------|------------------------| +| **Developer** | Repository, My MCP | Browse the shared repository, install MCP services, add MCP services, manage your own services and apply to publish them | +| **Administrator** | Repository, My MCP, Review Center | In addition to developer capabilities, review publication applications and delist services directly from the repository | + +> Tip: The Review Center tab is only visible to administrators. Developers can track the result of their applications via **View Review Progress** in the upper-right corner of the MCP card in **My MCP**. + +**Developer view** (Repository / My MCP): + +
+ Developer tabs +
+ +**Administrator view** (Repository / My MCP / Review Center): + +
+ Administrator tabs +
+ +--- + +## 📦 Repository + +The **Repository** tab displays the MCP services that have been published (shared) within the tenant. You can browse them, view details, and install them into **My MCP** with one click. + +### Browsing and Searching + +- Published MCP services are displayed as cards +- Search by **name or tags** +- Each card shows: name, description, tags, and install count + +
+ Repository list +
+ +### Viewing Details + +Click the **View Details** button on a card to view the complete information about the MCP service. + +
+ MCP details +
+ +### Installing a Service + +Click **Install** on a card, fill in the required information, and click **Confirm** to add it. The system automatically installs and configures the service. Installed services show an **Installed** status, avoiding duplicate imports. + +After installation, the service appears in **My MCP**, and the tools it provides are automatically synced to the agent's tool selection list. + +
+ Installing an MCP service +
+ +### Administrators: Delisting Services + +Administrators can directly **delist** a published service from the repository card. After delisting, the service is no longer visible to tenant members and can no longer be installed. + +If a developer needs to delist their own published service, they can do so in **My MCP** through the **View Review Progress** dialog. + +
+ Delisting an MCP service +
+ +--- + +## 🧑 My MCP + +The **My MCP** tab is used to manage the MCP services you have permission to edit, including services you created yourself, services to which tenant members have granted you edit permission, and services installed from the repository. + +### Adding a Service + +Click **Add MCP Service** to open the add dialog, which supports multiple integration sources. + +#### Custom Add + +Four deployment types are supported: + +| Deployment Type | Use Case | Key Configuration | +|-----------------|----------|-------------------| +| **Remote URL** | An independently deployed MCP service (HTTP / SSE) | Service URL, Authorization Token, custom request headers | +| **Container** | An MCP service running as a container | Container configuration JSON (mcpServers), port number | +| **API** | An HTTP API described by an OpenAPI spec | Service URL, OpenAPI JSON (required, format auto-validated) | +| **Local Image Upload** | An existing Docker image (.tar file) | Upload .tar image file, port number | + +> Local image upload is only shown after an administrator enables the **Upload Image** feature in the deployment. + +#### Port Notes + +Port behavior is automatically determined by how Nexent itself is deployed: + +- **Docker / Kubernetes deployment**: Container ports use a unified default port, automatically allocated and locked by the system, and cannot be modified. Multiple MCP services can reuse the same port without conflicting. +- **Local deployment**: The port is set by the user. A **Recommended Port** button fetches an available port with one click, and port occupancy is automatically detected. + +#### Importing from an External MCP Market + +Browse the community-maintained external MCP market, choose **Remote** or **Container** as the integration method, fill in the required environment variable parameters, and import with one click. + +
+ Adding an MCP service +
+ +### Service Card Statuses + +Each card shows runtime and publication statuses: + +| Indicator | Description | +|-----------|-------------| +| **Enabled** | Whether the service is enabled; tools from a disabled service no longer appear in the agent tool selection | +| **Under Review** | The publication application is pending administrator review | +| **Published** | Currently shared in the repository | +| **Rejected** | The publication application was not approved; you can modify and re-apply | + +
+ MCP service card +
+ +### Common Operations + +In the service card or detail dialog, you can: + +- **Edit**: Modify the name, description, URL, Authorization Token, tags, and other content +- **Enable / Disable**: Control the service state with a toggle +- **View Tool List**: View all tools provided by the service +- **Container Management**: View container logs and the configuration JSON used at creation time (containerized services) + +
+ MCP service details +
+ +- **More Actions**: + - **Apply to Publish**: Submit the service to the repository for review + - **Connectivity Check**: Run a connection test to check the MCP service connection status + - **View Review Progress**: View the application status, and cancel the application or delist the service + - **Delete**: Delete the service; containerized services also have their container resources cleaned up + +
+ MCP service card +
+ +### Sharing Configuration + +When adding or editing a service, you can check user groups and user permissions to share the service configuration with other members, making it easy to reuse within the team. + +### Applying to Publish + +1. On the create or edit page, check the MCP service configuration fields (service URL, Authorization Token, custom request headers, container configuration). The checked configuration will be shared to the repository. +2. In the More menu, click **Apply to Publish** +3. Fill in the publication information: + - **Publication Notes** (optional): A note to the reviewer +4. Click **Submit Application** and wait for the administrator to review. + +> You must check at least one shared configuration field before applying to publish. + +### Viewing Review Progress + +Open the review status dialog from the More menu to see: + +- Current status: Under Review / Approved / Rejected +- Review comments (if any) + +Depending on the status, you can also: + +- **Cancel Publication Application**: Withdraw a pending or rejected application +- **Delist**: Withdraw a published service from the repository + +
+ Review progress +
+ +--- + +## ✅ Review Center + +The **Review Center** is only visible to **administrators** and is used to process publication applications submitted by users. + +### Pending Review Queue + +The page displays pending applications in a list, including: + +- Service name and deployment method +- Submitter +- Publication notes +- Action buttons: Details, Approve, Reject + +A badge on the tab shows the number of pending items, so administrators can handle them promptly. + +### Review Actions + +1. Click **Details** to preview the service configuration and confirm whether it is appropriate +2. Click **Approve**: Optionally fill in review comments; after confirmation, the service is published to the **Repository** +3. Click **Reject**: Optionally fill in review comments; after rejection, the submitter can modify and re-apply in **My MCP** + +
+ Review Center +
+ +--- + +## 🚀 Next Steps + +After managing your MCP services in the MCP Repository, you can: + +1. Configure MCP tools for your agents in **[Agent Development](../agent-development)** +2. Experience agents calling MCP tools in **[Start Chat](../start-chat)** +3. Continue browsing **[Skill Repository](./skill-repository)** to learn how skills and MCP work together + +If you encounter any issues while using Nexent, please refer to our **[FAQ](../../quick-start/faq)** or ask for support in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). diff --git a/doc/docs/en/user-guide/skills.md b/doc/docs/en/user-guide/resource-repository/skill-repository.md similarity index 68% rename from doc/docs/en/user-guide/skills.md rename to doc/docs/en/user-guide/resource-repository/skill-repository.md index 0cdc2a2886..438802a893 100644 --- a/doc/docs/en/user-guide/skills.md +++ b/doc/docs/en/user-guide/resource-repository/skill-repository.md @@ -1,19 +1,21 @@ --- -title: Skill Management +title: Skill Repository --- -# Skill Management +# Skill Repository -A Skill is a core mechanism in Nexent for extending agent capabilities. Each skill packages multiple tools with usage documentation into a reusable unit of capability, enabling agents to handle complex tasks like assembling building blocks — without consuming excessive context space. +A Skill is a core mechanism in Nexent for extending agent capabilities. The Skill Repository lets people in the same tenant browse, copy, manage, and review Skills. Each Skill can package tools, configuration, and usage documentation into a reusable unit of capability. ## Table of Contents -- [Skills vs. Tools](#-skills-vs-tools): Understanding the core concepts +- [Skills vs. Tools](#the-relationship-between-skills-and-tools): Understanding the core concepts +- [UI Differences Between Admins and Developers](#-ui-differences-between-admins-and-developers): Available features by role +- [Repository](#-repository): Browse, copy, and take down listed Skills +- [My Skills](#-my-skills): Create, edit, and apply to list Skills +- [Review Center](#-review-center): Admin review of Skill listing requests - [Using Skills](#-using-skills): How to use skills in agent development -- [Skill Management](#-skill-management): Create, edit, import, and export skills - [Skill Upload Guide](#-skill-upload-guide): SKILL.md format, ZIP structure, special tags, and writing standards - [NL-to-Skill](#-nl-to-skill): Automatically generate skills from natural language descriptions -- [Official Skills Overview](#-official-skills-overview): Built-in skills and their capabilities ## The Relationship Between Skills and Tools @@ -28,82 +30,166 @@ A **Skill** bundles the capabilities of multiple tools into a complete workflow, | Granularity | Single atomic operation | Bundle of multiple tools + configuration + documentation | | Token consumption | Occupies context on every turn | Loaded only when activated | | Parameters | Fixed parameter schema | Customizable parameter templates | -| Versioning | No version management | Supports draft/published versions | | Distribution | Code-level | ZIP package distribution, plug-and-play | **Analogy**: Tools are individual items like a screwdriver, hammer, or saw. A Skill is a toolbox — with tools pre-matched for a work scenario and accompanied by usage instructions. Open the right toolbox for the task at hand. -## Using Skills +## 👥 UI Differences Between Admins and Developers -### Configuring Skills for an Agent +The tabs shown at the top of **Skill Repository** depend on your role: + +| Role | Visible tabs | Additional capabilities | +|------|--------------|-------------------------| +| **Developer** | Repository, My Skills | Browse shared Skills, copy a Skill, manage Skills they can edit, and apply for listing | +| **Admin** | Repository, My Skills, Review Center | All developer capabilities, plus reviewing listing requests and taking down repository Skills | + +> The Review Center is available only to admins. Developers track their own requests from **My Skills**. + +**Developer view** (Repository / My Skills): + +
+ Developer tabs +
-1. Open the **[Agent Development](./agent-development)** page -2. On the "Select Tools" tab, find the **Skills** group -3. Click a skill name to select it; click again to deselect -4. After selecting a skill, click the ⚙️ button next to it to configure skill parameters -5. Save the agent configuration +**Admin view** (Repository / My Skills / Review Center):
- + Admin tabs
-> 💡 **Tip**: If a skill has required parameters that are not configured, a guided parameter-filling prompt will appear upon selection. +--- -### Skill Parameters +## 📦 Repository -Each skill's parameter definitions come from the `config/schema.yaml` file in the skill package. The configuration interface auto-generates a parameter form based on the schema, including: +The **Repository** tab lists Skills that have been listed and shared within the current tenant. Developers and admins in the current tenant can browse them, view their details, and copy them to **My Skills** before editing. -- **Parameter name and description** (bilingual: English and Chinese) -- **Required/optional markers** -- **Default values** -- **Parameter types** (string, number, boolean, array, object) -- **YAML comment auto-mapped tooltips** +> A shared Skill must be copied to **My Skills** before it can be edited. -### Skill Versions +### Browse and Search -Each skill supports multi-version management: +- Browse listed Skills as cards +- Search by **Skill name, description, or tag** +- Cards show a summary such as name, description, tags, source, and download count -- **Draft version (version=0)**: Development and debugging stage; changes take effect immediately -- **Published version (version>=1)**: Production use; parameters are locked +
+ Repository list +
-When configuring the same skill for different agents, you can set different parameter values independently. +### View Details -## Skill Management +Click **Details** on a Skill card to view its basic information, including its name, creator, description, tags, install count, and last updated time. Details are read-only; copy the Skill to **My Skills** to make changes. -### Viewing Installed Skills +
+ Skill details +
+ +### Copy a Skill + +1. Click **Copy** on the target Skill card. +2. Enter a new Skill name. If the name already exists, choose another name and try again. +3. After copying, the Skill appears in **My Skills**, where you can edit it, configure it, or apply for listing. -The "Select Tools" skill group displays all installed skills, including: -- Official skills (`official` source) -- Custom skills (`custom` source) +### Admin Take-Down -### Creating Custom Skills +Admins can choose **Take Down** from the more-actions menu on a repository Skill card. The Skill is then no longer available for developers and admins in the current tenant to browse or copy. Existing personal copies are not affected. -Nexent supports two ways to create custom skills: uploading a skill package file, or generating one automatically from a natural language description. +--- + +## 🧑 My Skills + +Use **My Skills** to manage Skills that you can edit, including Skills you created and Skills for which you have been granted edit permission. + +### Filter and Search + +- **All / Created by me / Others** filters Skills by ownership +- Search by Skill name, description, or tag +- Browse Skills as cards with pagination + +
+ My Skills list +
-#### Method 1: Upload SKILL.md or ZIP +### Create, Upload, and Edit -1. Go to the skill configuration interface -2. Click the "Upload Skill" button -3. Select a `SKILL.md` file (single file) or a `.zip` package (complete skill package) -4. The system automatically parses and creates the skill +Click **Create Skill** from **My Skills** to choose one of the following: -#### Method 2: NL-to-Skill Natural Language Creation +- **Interactive creation**: Work with the Skill builder through natural language to generate or improve a Skill. +- **Upload a Skill file**: Upload a single `SKILL.md` file or a ZIP archive containing the complete Skill structure. -Click the **"NL Create Skill"** button on the skill management page. See the [NL-to-Skill](#-nl-to-skill) section below for details. +Use **Edit** on a Skill card to update its name, description, tags, group permissions, `SKILL.md` body, and additional files. See the [Skill Upload Guide](#-skill-upload-guide) for file-format and upload rules. -### Editing Skills +### Apply for Listing -1. Find the target skill in the skill list -2. Click the skill card to enter the edit page -3. Modify the skill name, description, tags, parameter configuration, etc. -4. Save changes +You can apply to list a Skill that you are allowed to edit in the tenant repository: -### Importing/Exporting Skills +1. Click **List** on the Skill card. +2. Optionally provide a listing note to help the admin understand the request. +3. Click **Submit request** and wait for an admin review. -- **Export**: Click "Export" on the skill detail page to download as a JSON configuration file -- **Import**: Click "Import Skill" on the Agent Development page to upload a JSON configuration file +The card shows **Pending review** after submission. A Skill has only one pending listing record at a time. -> ⚠️ **Note**: When importing skills containing knowledge base tools (such as `knowledge_base_search`), these tools will only search **knowledge bases that the currently logged-in user is permitted to access in this environment**. The original skill's knowledge base configuration will not be automatically inherited. +
+ Apply for listing +
+ +### View Review Progress + +Open the review-status dialog from a Skill with a submitted request to view its status, submission time, listing note, and reviewer comment (when provided). + +- **Pending review**: You can cancel the listing request. +- **Listed**: You can take the Skill down. +- **Rejected**: Edit the Skill first, then cancel the current listing request and click **List** again to resubmit. + +
+ Review progress +
+ +--- + +## ✅ Review Center + +The **Review Center** is available only to admins and is used to process Skill listing requests submitted by developers in the current tenant. + +### Pending Review Queue + +The queue shows information such as the Skill name, submitter, listing note, and submission time. The tab badge shows the number of requests waiting for review. + +### Review Actions + +1. Click **Details** to inspect the Skill's basic information. +2. Click **Approve** and confirm to list the Skill in the **Repository**. +3. Click **Reject** to optionally add a review comment. The submitter can view the comment in **My Skills**, update the Skill, and submit a new request. + +
+ Review confirmation +
+ +--- + +## Using Skills + +### Configuring Skills for an Agent + +1. Open the **[Agent Development](../agent-development.md)** page. +2. In **Select Tools**, switch to the **Skills** tab and click **Select Skills**. +3. Select the Skill you want to configure; select it again to remove it. +4. If the Skill has required parameters, its configuration dialog opens automatically. Complete and save the parameters before the Skill is added to the current agent. +5. For an added Skill, click the gear icon to update the parameters used by the current agent. +6. Save the agent configuration. + +
+ Skills tab in agent configuration +
+ +### Viewing Installed Skills + +The **Skills** tab in **Select Skills** lists the official and custom Skills available to the current tenant. You can add them to the current agent from this tab. + +
+ Select Skills dialog +
+ +> Different agents can save separate parameter configurations for the same Skill. ## Skill Upload Guide @@ -366,6 +452,10 @@ In simple terms: > You say "I want a skill that can search GitHub repositories and extract Star counts," and the system automatically generates a complete, usable skill for you. +
+ NL-to-Skill creation interface +
+ ### Quick Start #### Step 1: Describe Your Requirement @@ -509,9 +599,9 @@ When encountering requirements that cannot be fulfilled, the system will provide #### Modifying Skills -In the NL-to-Skill interface, you can select an existing skill. After selecting, the skill information loads automatically. You can then use natural language to attempt updating the skill in the left dialog. +In **My Skills**, find the Skill you want to change and click **Edit**. On the agent configuration page, you can also click the pencil icon for a Skill that you are allowed to edit. The system loads its basic information, `SKILL.md` body, and additional files. You can use the interactive creation view to refine the content with natural language, or edit the files directly and save your changes. -If the skill name you create conflicts with an existing skill, Nexent will automatically switch from skill creation mode to skill update mode. All content will overwrite the original skill. +When creating or uploading a Skill, if its name already exists, the system prompts you to change the name before submitting again. ## Official Skills Overview @@ -566,7 +656,7 @@ If the skill name you create conflicts with an existing skill, Nexent will autom ## Related References -- [Agent Development](./agent-development) -- [Local Tools Overview](./local-tools/index) -- [MCP Tool Configuration](./mcp-tools) -- [Skills System Overview](../backend/skills/overview) +- [Agent Development](../agent-development.md) +- [Local Tools Overview](../local-tools/index.md) +- [MCP Repository Configuration](./mcp-repository.md) +- [Skills System Overview](/en/backend/skills/overview) diff --git a/doc/docs/en/user-guide/start-chat.md b/doc/docs/en/user-guide/start-chat.md index 5834521eab..93dab99bba 100644 --- a/doc/docs/en/user-guide/start-chat.md +++ b/doc/docs/en/user-guide/start-chat.md @@ -1,211 +1,582 @@ # Start Chat -The Start Chat is the core area for interacting with your agents. Here, you can talk to different agents, upload files, use voice input, and manage your chat history. +The Start Chat page is the primary entry point for interacting with agents. On this page, you can chat with agents, upload files and attachments, use voice input, manage conversation history, and complete tasks such as file processing, knowledge retrieval, and document generation. -## 🤖 Start Chatting +## 1. Select an Agent -### Select an Agent +### 1. Open the Start Chat Home Page -Before starting a chat, you need to select an agent. +When you open the Start Chat page, the agent list is displayed by default. You must select an agent before starting a conversation. -1. **View Available Agents** - - Find the agent selection dropdown in the lower left corner of the chat box - - Click the dropdown to view all available agents - - Each agent displays its name and description +![Agent list](./assets/start-chat/agent-list.png) -2. **Switch Agents** - - Select the agent you want to chat with from the list - - The system will automatically switch to the selected agent - - You can start a new chat after switching +### 2. Select an Available Agent -
- Select Agent -
+Only agents that meet all of the following conditions appear on the Start Chat page: -### Send Text Messages +- **Published**: The agent has been published. +- **Set as a primary agent**: The agent is configured as a primary agent. +- **Available to the current user**: The current user has permission to use the agent. -After selecting an agent, you can send text messages in the following ways: +Each agent card displays the agent icon, display name, English identifier, and functional description or greeting. -1. **Enter your question** - - Type your question or command in the input box at the bottom - - Press `Shift+Enter` to insert a line break +You can search in real time by keyword, including the agent name, description, or developer. The system records recently used agents and provides quick access to them. The list is automatically paginated when it contains more than one page. -2. **Send Message** - - Click the send button on the right side of the input box - - Or press `Enter` on your keyboard - - The agent will start processing your request and generate a reply +## 2. Agent Home Page -3. **View Replies** - - The agent’s reply will be displayed in real time in the chat area - - The reasoning process will be shown as cards for easy distinction +After selecting an agent, you are taken to its home page. The page mainly consists of the following areas: -
- Select Agent -
+- **Conversation history area on the left**: Manage conversation history. +- **Welcome area on the right**: Display agent information and example questions. +- **Input area at the bottom**: Enter questions, upload attachments, and select a conversation mode. -### Use Voice Input +![Agent welcome page](./assets/start-chat/agent-welcome.png) -Nexent supports voice input (make sure you have configured the speech model under [Model Management](./model-management) beforehand) so you can interact by speaking: +### 1. Conversation History Area -1. **Enable Voice Input** - - Find the microphone icon in the lower right corner of the input box - - Click the microphone icon to enable voice input - - The first time you use it, you’ll be asked for microphone permission—please click "Allow" +The left sidebar displays the conversation history for the current agent. -2. **Start Speech Recognition** - - After granting permission, the microphone icon will change to recording mode - - Speak your question or command clearly - - The system will convert your speech to text in real time and display it in the input box +#### Create a New Conversation -3. **Complete Voice Input** - - After speech recognition is complete, the system will automatically send the message - - You can also manually edit the recognized text before sending - - Both Chinese and English speech recognition are supported +- Click the **New Conversation** button at the top of the sidebar to create a conversation. +- New conversations use the currently selected agent by default. +- When starting a new task, create a new conversation to prevent previous context from affecting the task. -> 💡 **Tip:** For better recognition results, use it in a quiet environment and articulate clearly. +#### View the Conversation List -### Upload Files for Chat +- View all historical conversations for the current agent. +- Conversations are sorted chronologically. +- Click an existing conversation to view it or continue asking questions. +- Conversation titles are generated automatically by the system. -You can upload files during a chat so the agent can reason over their content: +#### Manage Conversation Records -> ⚠️ **Important:** -> 1. Multimodal file conversations require the agent to have the corresponding parsing tools enabled during agent development. -> 2. For document or text files select the `analyze_text_file` tool. -> 3. For image files select the `analyze_image` tool. -> 2. Each uploaded file should ideally be under 10 MB. Split large documents into multiple uploads. +The following conversation management operations are currently supported: -1. **Choose a File Upload Method** - - Click the file upload button in the lower right corner of the input box - - Or drag files directly into the chat area +| Operation | Description | +| --- | --- | +| Rename conversation | Change the conversation title. | +| Delete conversation | Delete the conversation record. | -2. **Supported File Formats** - - **Documents:** PDF, Word (.docx), PowerPoint (.pptx), Excel (.xlsx), EPUB (.epub), HTML (.html), XML (.xml) - - **Text & Data:** Markdown (.md), Plain text (.txt), JSON (.json), CSV (.csv) - - **Images:** JPG, PNG, GIF, and other common formats +> **Note**: Deleted conversations usually cannot be recovered. Confirm the operation before deleting a conversation. -3. **File Processing Flow** - - The platform stores the uploaded file in MinIO and returns an S3 URL - - It builds structured file metadata and injects it into the active conversation - - The agent then answers your questions based on both the prompt and file metadata +![Conversation management](./assets/start-chat/conversation-manage.png) -4. **File-based Chat** - - After uploading a file, ask questions about its contents at any time - - The agent can call the relevant multimodal tools to analyze, summarize, or process the data - - Multiple files can be uploaded and processed simultaneously +### 2. Agent Greeting and Example Questions -## 📚 Manage Your Chat History +The center of the agent home page displays the agent's greeting, which introduces its purpose and capabilities, along with preset example questions. The agent developer configures the greeting and example questions on the agent configuration page. Clicking an example question automatically fills it into the input box. You can edit it before sending or send it directly. -The left sidebar provides complete chat history management: +![Example questions](./assets/start-chat/example-question.png) -### Create a New Chat +### 3. Input Box and Conversation Modes -- Click the "New Conversation" button in the upper left corner to start a brand new conversation -- The new chat will use the currently selected agent by default, but you can change it +The input area at the bottom is used to enter questions, upload attachments, use voice input, and send messages. -### View Chat List +#### Execution Mode and Planning Mode -- **Chat Titles:** The system automatically generates titles based on chat content, which you can edit at any time -- **Time Sorting:** Chats are sorted by time, showing "Today" and "Last 7 Days" records -- **Continue Chat:** Click any chat in history to view details and continue the conversation +Two mode-switching buttons, **Execute** and **Plan**, are available above the input box. -### Manage Chat Records +- **Execution mode**: The agent enters the ReAct loop directly and continues until the task is complete or the maximum number of steps is reached. This mode is suitable for simple and well-defined questions. -1. **Edit Chat** - - Hover over a chat title to see the "..." button on the right, click to edit -2. **Rename Chat** - - Click "Rename" to change the chat title, press Enter to confirm -3. **Delete Chat** - - In edit mode, you can delete unnecessary chats - - Deletion is irreversible, please operate with caution +- **Planning mode**: This mode is suitable for complex tasks. Before execution, the agent breaks the task into multiple ordered steps. A plan and the execution status of each step are displayed as cards above the input box. Step statuses include pending, in progress, completed, and skipped. A plan must contain at least 3 steps and no more than 8 steps. The agent executes the steps in order and automatically updates each status after the step is completed. -> 💡 **Tip:** Regularly cleaning up unnecessary chat records keeps the interface tidy and improves search efficiency. +![Planning mode](./assets/start-chat/plan.png) -
- Chat Edit - Chat Edit -
+#### Select a Model -### Access Other Modules +If multiple available models have been added on the agent configuration page, you can switch between them below the input box. The model selector displays only the models configured for the current agent. -Use the left navigation bar to jump to other modules at any time: +#### Upload Attachments -- **Agent Space** – Review and manage all agents you have built. -- **Agent Studio** – Continue creating or editing agents. -- **Model Management** – Update app information and model credentials. -- **Knowledge Base** – Upload, summarize, and organize documents. -- **Memory Management** – Configure multi-layer memory and sharing rules. +You can upload attachments in the input box and ask the agent to analyze, summarize, or otherwise process their contents. -## 🔍 View Knowledge References +**Upload methods**: -The right sidebar provides two tabs: "Source" and "Images" to help you understand the sources of agent responses: +- Click the file upload button on the right side of the input box. +- Drag a file directly into the input area. -### References Tab +**Supported file types**: -- Shows the knowledge sources cited by the agent’s reply -- Displays text block titles and source file names -- Click "Expand" to view the full content of the text block -- Helps you understand what information the agent retrieved from your local knowledge base +| Type | File formats | +| --- | --- | +| Images | `image/*` (JPG, PNG, GIF, and other formats) | +| Documents | PDF, Word (`.docx`), Excel (`.xlsx`), PowerPoint (`.pptx`), EPUB (`.epub`) | +| Text | Markdown (`.md`), plain text (`.txt`), JSON (`.json`), CSV (`.csv`), XML (`.xml`), HTML (`.html`) | +| Other | Other file formats are processed as regular attachments. | -- **Network Search Results** - - Shows webpage titles and source URLs - - Click "Expand" to view detailed content - - Click the webpage title to jump directly to the original page +**File quantity and size limits**: -### Images Tab +- You can upload up to **50 attachments** in a single message. +- Each attachment can be up to **100 MB**. Files exceeding this limit are rejected. +- Deployment administrators can configure a lower front-end upload limit. If the limit shown on the page is lower than 100 MB, follow the limit shown in the current environment. -- Displays related images retrieved from network search -- Click any image to preview -- Helps you visually understand relevant information +> **Note**: +> +> - The parsing capabilities required for different file types depend on the agent configuration. +> - Image files require the agent to have a vision model and image parsing tools configured. +> - Document files require the corresponding document parsing tools to be configured for the agent. -
- Reference Source - Reference Image -
+**Using attachment content in a conversation**: Uploaded attachments are sent to the agent as context. The agent can read and analyze their contents and use them to answer questions or perform tasks. -## 🎭 Multimodal Interaction Experience +![Upload attachments](./assets/start-chat/upload_file.png) -### Image Processing +#### Voice Input -Nexent supports image input and processing (make sure a vision model **and** the `analyze_image` tool are configured): +You can use the microphone icon to enter a voice question. -1. **Upload Images** - - Drag image files directly into the chat area - - Or click the upload button to select image files - - Supports common formats (JPG, PNG, GIF, etc.) +**Prerequisites**: -2. **Image Analysis** - - The agent will automatically analyze image content - - It can recognize objects, text, scenes, etc. in images - - Answers your questions based on image content +- Speech recognition (STT) must be enabled in the system configuration. +- The first time you use voice input, you must authorize the browser to access your microphone. -> 💡 **Tip:** Nexent will soon support richer multimodal interaction modes, including video processing, audio analysis, and more. Stay tuned! +**Procedure**: -## ⚙️ Backend Operation Mode +1. Click the microphone button in the lower-right corner of the input box. +2. If this is your first time using voice input, the browser requests microphone permission. Click **Allow**. +3. Clearly speak your question. +4. The system converts your speech to text in real time and displays it in the input box. +5. Review and edit the recognized text. +6. Click the Send button or press Enter to send the message. -### Multitasking +> **Tip**: For better speech recognition, use voice input in a quiet environment and articulate clearly. -Nexent supports backend operation mode, making you more efficient when handling complex tasks: +#### Send Messages -1. **Parallel Tasks** - - During a chat, you can switch to other windows or applications - - The agent will continue processing your tasks in the background - - Processing will not be interrupted by window switching +**Sending methods**: -2. **Real-time Status Monitoring** - - Each chat in the left sidebar has a status indicator - - 🟢 **Green dot:** Chat in progress - - 🔵 **Blue dot:** Chat completed - - Click any chat to view processing progress +- Click the Send button on the right side of the input box. +- Press the Enter key on your keyboard. -3. **Improve Work Efficiency** - - Backend operation mode greatly improves your work efficiency - - You can do other work while waiting for the agent to process - - Especially suitable for long analysis or generation tasks +**Keyboard shortcuts**: -## 🚀 Start Your Nexent Journey +| Shortcut | Function | +| --- | --- | +| Enter | Send a message. | +| Shift + Enter | Insert a line break. | -Congratulations! You now master all the core features of Nexent. We look forward to seeing you create amazing applications with Nexent! +**Sending status**: -### Get Help +- While the agent is processing a request, the Send button changes to a Stop button. +- Click the Stop button to interrupt the current execution. -Need help? Check the **[FAQ](../quick-start/faq)** or open a thread in [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions). \ No newline at end of file +## 3. Conversation Execution + +Agents use the ReAct workflow, so processing a task may include multiple rounds of reasoning and execution. + +### 1. Memory Retrieval + +If memory is configured for the agent, the system retrieves relevant memories after you send a question and before it formally begins executing the task. + +Retrieved memories may include: + +- Information you provided in the past. +- Your preferences. +- Relevant content from previous tasks. +- Long-term information saved by the agent. + +Relevant memories are used as context for the current task, helping the agent generate a response that better matches your needs. + +![Memory retrieval](./assets/start-chat/memory.png) + +### 2. ReAct Execution Flow + +Nexent agents are implemented using the CodeAgent from [smolagents](https://github.com/huggingface/smolagents) and use the ReAct (Reasoning + Acting) workflow. The core loop is: + +- **Think**: The model analyzes the current task state and determines the next action. For agents with planning mode enabled, the model first evaluates task complexity. If it expects that more than three steps will be required, it generates a structured plan. + +- **Code**: The model outputs action instructions as Python code. Executable code is wrapped in `...` tags, while code intended only for display is wrapped in `...` tags. + +- **Observe**: After the code is executed, the system returns the actual result, marked as `Observation:`. The model must continue reasoning based on the actual result and must not fabricate an observation before execution. + +![ReAct loop](./assets/start-chat/ReAct.png) + +The loop repeats the reasoning process in the front end until the model determines that it can generate the final answer directly or the maximum number of steps is reached. The final answer is output in Markdown format and supports headings, lists, tables, code blocks, and links. When retrieval tools are used, citation markers such as `[[letter+number]]` must be added after the relevant content to support traceability. + +### 3. View Code and Tool Calls + +On the conversation page, you can view key information from the agent's execution process: + +- **Reasoning process**: The agent's reasoning analysis. +- **Generated executable code**: The code written by the agent for execution. +- **Tools called**: The specific tools used. +- **Tool input parameters**: The parameters passed to the tools. +- **Tool output**: The results returned by the tools. + +This information is displayed as collapsible cards in the conversation area: + +- **Reasoning cards** with a mind-map icon display the agent's reasoning process. +- **Tool call cards** with a tool icon display the tool name, call status, and execution result. + +![ReAct loop](./assets/start-chat/tool-call.png) + +### 4. Automatic Error Correction + +If the code generated by the agent has a problem, the system provides a degree of automatic error correction. + +Based on execution errors, the agent may: + +1. Analyze the cause of the error. +2. Modify the generated code or parameters. +3. Execute the task again. +4. Continue processing based on the new execution result. + +![Automatic error correction](./assets/start-chat/self-correction.png) + +> **Note**: Automatic error correction depends on the model's capabilities, tool implementation, and task complexity. It cannot guarantee that every error will be fixed automatically. + +### 5. Maximum Execution Steps + +The agent developer can set the maximum number of execution steps on the agent configuration page. + +When the agent reaches the maximum number of steps: + +- The system stops further execution. +- The current conversation may not complete the entire task. +- The page returns the execution results obtained so far or a stop notification. + +> **Recommendation**: Set the maximum number of execution steps according to the complexity of the task. + +### 6. Parallel Tool Calls + +When a task requires multiple independent tools at the same time, the agent can call them in parallel to reduce the overall waiting time. + +For example, the agent can perform the following operations simultaneously: + +- Retrieve information from multiple knowledge bases. +- Search multiple web pages. +- Analyze multiple files. +- Perform multiple data processing operations. + +Parallel calls are displayed as a combined tool call card showing the number of calls and the execution status of each call. + +![Parallel tool calls](./assets/start-chat/parallel-tool-calls.png) + +### 7. Parallel Subagents + +The agent can call multiple subagents to process different subtasks as needed. + +Multiple subagents can work in parallel. The primary agent is responsible for: + +- Breaking down complex tasks. +- Assigning subtasks to different subagents. +- Aggregating the subagent results. +- Generating the final response. + +Subagent calls are displayed as nested cards showing: + +- The subagent name. +- A description of the task assigned to the subagent. +- The execution status, such as running or completed. + +![Parallel subagents](./assets/start-chat/parallel-subagents.png) + +### 8. Self-Check + +Self-check is a layered ReAct self-validation capability configured for an agent. It checks for obvious problems at key execution points and before generating the final answer. This feature is disabled by default. The conversation page displays the self-check process only after self-validation has been enabled for the agent. + +#### Trigger Points + +When self-check is enabled, the system performs checks at the following key points according to the agent configuration: + +- **Before a tool call**: Checks whether the generated execution code is empty, whether the Python syntax is valid, and whether there are obvious unauthorized or dangerous operations. If tool-specific checks are configured, it also checks whether the relevant tools were called or the required information was output. +- **After a tool call or code execution**: Checks whether the execution result is empty and whether it contains error signals. +- **After knowledge retrieval**: Checks whether the retrieval results contain usable evidence. +- **After a subagent handoff**: Checks whether the subagent returned a conclusion with substantive content. +- **Before generating the final answer**: Checks whether the answer is empty, whether internal execution markers or unreplaced placeholders remain, and whether errors that occurred earlier have been explained in the answer. + +#### Final Answer Validation + +For tasks that require evidence or are relatively complex, the system can also use a validation model to perform additional checks on a candidate answer. The validation model evaluates the following based on the user task and execution process: + +- Whether the answer addresses the user's goal. +- Whether the conclusions are supported by sufficient evidence. +- Whether tool errors have been handled. +- Whether the citation format is correct. +- Whether the output format is safe and complete. + +Lightweight greetings and similar conversations receive basic checks and may not require external evidence. Whether validation is performed, how strict it is, and how the validation model is used are determined by the agent's self-validation configuration. + +#### Self-Check Results + +The self-check panel displays the checking process and results as collapsible cards. It may show the following statuses: + +- **Self-check in progress**: The system is preparing or performing a check. +- **Basic self-check passed**: Key checks passed. +- **Final self-check passed**: The final answer passed validation. +- **Self-check found items requiring attention**: A problem was found, but it does not block execution. +- **Self-check failed, correcting**: The current candidate answer did not pass, and the agent will continue correcting it based on feedback. +- **Self-check blocked the current action**: The current action failed a blocking check, so the system will not continue with that action. +- **Final self-check failed**: The answer still did not pass after the allowed validation rounds. + +The panel may also display validation scores, failed checks, user-visible messages, and repair suggestions. Validation events do not display the validation model's internal reasoning text. Instead, they display structured check results. + +#### Handling Self-Check Failures + +When a key check fails, the system sends the failure criteria and repair instructions to the agent as feedback. The agent may: + +1. Modify the execution code or tool parameters. +2. Call tools again or retrieve additional evidence. +3. Generate the final answer again. +4. Return a controlled explanation identifying the failed checks, reasons, and recommendations if validation cannot be passed. + +The maximum number of final-answer validation attempts is determined by the agent configuration. The default is up to 2 rounds of final-answer validation. Self-check cannot guarantee that all business errors will be detected and does not replace manual confirmation of file contents, data accuracy, or business results. + +![Self-check](./assets/start-chat/verification.png) + +### 9. Completion Indicator + +When the agent completes the task: + +- A **Completed** indicator, consisting of a green dot and text, appears at the end of the response. +- The execution duration is displayed. +- The Token usage for the conversation is displayed. + +![Agent completed](./assets/start-chat/finish.png) + +## 4. Knowledge Retrieval and Source Traceability + +When knowledge retrieval tools are configured for the agent, you can view the sources cited in the response during the conversation. + +![Sources](./assets/start-chat/source.png) + +### 1. View Citation Sources + +After the conversation is complete, the response area displays a **View Sources** button if the agent called a knowledge retrieval tool. Click this button to view the knowledge content used by the agent in the right-side panel. + +### 2. Right-Side Source Panel + +The source panel contains two tabs: + +#### Sources Tab + +Displays the knowledge sources cited in the agent's response. + +**Local knowledge base results**: + +- Knowledge base name. +- Source file name. +- Text block title. +- Matched text. +- Relevant citation excerpts. + +**Web search results**: + +- Web page title. +- Source URL. +- Web page summary or cited content. +- Relevant images or other online resources. + +You can use a source link to view the original web page. + +#### Images Tab + +- Displays related images retrieved through web search. +- Click an image to preview it at full size. +- Displays the title of the web page where the image was found. + +## 5. Image Processing + +When a vision model and image parsing tools are configured for the agent, you can upload images and ask the agent to analyze them. + +### 1. Upload Images + +You can upload images in the following ways: + +| Upload method | Instructions | +| --- | --- | +| Click the upload button | Click the file upload button on the right side of the input box and select an image. | +| Drag and drop | Drag an image file directly into the conversation area. | +| Select from the input box | Select an image in the file picker in the input box. | + +### 2. Image Analysis + +The agent can perform the following types of tasks based on image content: + +- **Describe image content**: Identify and describe scenes, people, objects, and other elements in the image. +- **Recognize text in images**: Extract text from the image using OCR. +- **Analyze objects, scenes, or structures**: Identify object categories, scene types, chart structures, and other visual elements. +- **Answer image-related questions**: Answer questions based on the image content. +- **Organize or make judgments based on image content**: Analyze, summarize, or reason from information in the image. + +> **Note**: Image processing capabilities depend on whether the agent has a vision model and the corresponding tools configured. + +![Image analysis](./assets/start-chat/analyze_image.png) + +### 3. Image Source Citations + +If an image comes from web search, related thumbnails are displayed in the Images tab of the right-side source panel. Click a thumbnail to view the full-size image. + +## 6. Document Processing and Generation + +### 1. Document Analysis + +> **Tip**: To analyze documents, configure the official `analyze_text_file` tool for the agent. + +You can upload documents and ask the agent to perform the following operations: + +- **Summarize content**: Extract the main content and key points from a document. +- **Extract key information**: Extract important data, facts, or information from a document. +- **Answer questions about a document**: Answer questions based on the document content. +- **Analyze structure**: Analyze the document organization and relationships between sections. +- **Compare content**: Compare differences between multiple documents. +- **Organize data**: Convert data in a document into a table or another format. + +![Document analysis](./assets/start-chat/analyze_text_file.png) + +### 2. Document Generation + +> **Tip**: To generate a Word document (`.docx`), configure the official `create-docx` Skill for the agent. Document generation capabilities depend on the tools and Skills configured for the agent. + +The agent can generate documents based on your requirements or conversation content, such as reports, proposals, explanatory documents, spreadsheets, and presentations. + +![Document generation](./assets/start-chat/create-docx.png) + +Generated documents can be previewed and downloaded directly in the conversation. + +### 3. Document Preview and Download + +After a document is generated, you can perform the following actions on the conversation page: + +- **View the document content**: Preview the generated document directly in the conversation area. +- **Check the generated result**: Confirm that the document meets your requirements. +- **Continue editing**: Ask the agent to modify or supplement the document. +- **Download the document**: Click the download button to save it locally. + +![Document preview](./assets/start-chat/preview-docx.png) + +## 7. Mermaid Diagrams + +When an agent generates Mermaid diagram code, the conversation page can render it as a visual diagram. + +### Supported Diagram Types + +The following diagram types are supported: + +| Diagram type | Description | +| --- | --- | +| Flowchart | Shows processes and decision paths. | +| Sequence diagram | Shows the order of interactions between objects. | +| Class diagram | Shows the structure and relationships between classes. | +| State diagram | Shows state transition processes. | +| Gantt chart | Shows a project timeline. | +| Mindmap | Shows the hierarchy of a topic. | +| Entity-relationship diagram | Shows relationships between data entities. | + +### Diagram Interaction + +Generated diagrams support the following interactions: + +- **Hover to enlarge**: Display an enlarge button when you hover over the diagram. +- **View full screen**: Click the enlarge button to view the diagram in full-screen mode. +- **Pan by dragging**: Drag to move the view in full-screen mode. +- **Zoom with the mouse wheel**: Zoom the diagram with the mouse wheel. +- **Reset the view**: Reset the zoom level and position. + +> **Tip**: If diagram rendering fails, the original code is displayed along with a **Diagram failed to render** message. + +![Mermaid diagram](./assets/start-chat/mermaid.png) + +## 8. Conversation Interactions + +### 1. Refresh a Response + +If you are not satisfied with the current response or want the agent to generate the result again, you can use the refresh function. + +**How to refresh**: + +- Click the refresh button with the circular arrow icon below the agent's response. +- Or resend the same question in the input box. + +**What refreshing does**: + +- Resubmits the current question. +- Runs the agent processing flow again. +- Generates a new response. + +> **Note**: Refreshing a response consumes additional Token resources. + +![Refresh conversation](./assets/start-chat/refresh-chat.png) + +### 2. Copy a Response + +You can copy the agent's response. + +**How to copy**: Click the copy button with the clipboard icon below the response. + +After the response is copied successfully, the button icon briefly changes to a check mark. + +### 3. Export to Markdown + +You can export the complete conversation as Markdown. + +**How to export**: Click **Export Markdown** in the **More** menu below the response. + +### 4. Share a Conversation + +You can generate a share link so that other people can view the conversation. + +**Sharing workflow**: + +1. Click the Share button next to the conversation title to enter sharing mode. +2. Choose whether to share the entire conversation or specific question-and-answer pairs. +3. Click **Copy Link** to generate the share link. +4. The share link is copied to the clipboard. + +**Shared content**: + +- The selected user questions and agent responses. +- Source citations used by the agent. +- Related images, if applicable. + +**Features of a shared page**: + +- The shared page is read-only. +- Other users can view the conversation but cannot continue it. +- The source panel remains available. + +![Share conversation](./assets/start-chat/share.png) + +## 9. Background Operation Mode + +When an agent processes a complex task or generates a file, execution may take a long time. You can leave the current conversation page while the system continues processing the task in the background. + +### 1. Continue Task Execution + +After you leave the current conversation page, the agent can continue executing unfinished tasks. + +- The task continues running on the server. +- Execution is not affected if you close the browser or switch pages. + +### 2. Return to View Results + +When you re-enter the original conversation, you can view: + +- The execution process records that have already been generated. +- The execution status of each step. +- The final generated result. + +### 3. Stop Execution + +If you are still on the current page, you can stop the agent at any time: + +- While the agent is executing, the Send button changes to a Stop button with a square icon. +- Click the Stop button to interrupt the current execution. +- Results from completed steps are retained. + +## 10. Shortcuts and Navigation + +### 1. Return to the Agent List + +On the conversation page, click the Back button in the upper-left corner to return to the agent selection list. + +### 2. Switch to the Legacy Interface + +The bottom of the left sidebar provides a **Switch to Legacy** entry. Click it to return to the legacy conversation interface. + +### 3. Collapse the Sidebar + +On desktop, the sidebar can be collapsed or expanded. Click the collapse button to hide the conversation history area and expand the main conversation area. + +On mobile, the sidebar is collapsed by default. Click the expand button to open it temporarily. + +![Collapse conversation history](./assets/start-chat/collapse.png) diff --git a/doc/docs/en/user-guide/user-management.md b/doc/docs/en/user-guide/user-management.md index 0d4b4f81a1..112bae15da 100644 --- a/doc/docs/en/user-guide/user-management.md +++ b/doc/docs/en/user-guide/user-management.md @@ -2,7 +2,7 @@ This page provides a detailed explanation of the Nexent platform's user role system, data visibility scope, operation permissions for various resources, and practical examples of permission configuration. -⚠️ **Important Note**: When deploying v1.8.0 or later for the first time, please pay special attention to the `suadmin` super administrator account information output in the Docker logs. This account has the highest system privileges, and the password is only displayed upon first generation. It cannot be viewed again later, so please be sure to save it securely. +⚠️ **Important Note**: When deploying v1.8.0 or later for the first time, Nexent creates the `suadmin@nexent.com` super administrator account with the default password `Nexent@123` and displays it in the terminal after successful creation. Override it before the first deployment with `NEXENT_SUPER_ADMIN_PASSWORD`; an offline package launched with `--config` uses the interactively entered password instead and does not display it. ## 📋 Page Navigation @@ -39,7 +39,7 @@ Includes the following four core roles: | Role | Responsibility Description | Applicable Scenarios | Role Notes | | ---- | -------------------------- | -------------------- | ---------- | -| **Super Administrator** | Can create **different tenants** and manage all tenant resources | Platform operation and maintenance personnel | There is only one Super Administrator in the Nexent system. Account credentials are generated during local deployment. Please keep them safe as they cannot be retrieved after logs are cleared | +| **Super Administrator** | Can create **different tenants** and manage all tenant resources | Platform operation and maintenance personnel | There is only one Super Administrator in Nexent. It is created during the first deployment, and its password can be preset through the deployment environment | | **Administrator** | Responsible for **intra-tenant** resource management and permission allocation | Department managers, tenant leaders | A tenant can have multiple administrators, who can only be invited by the Super Administrator | | **Developer** | Can create and edit agents, knowledge bases, and other resources, but has no management permissions | Developers, product managers | A tenant can have multiple developers who can belong to multiple user groups within the tenant, invited by administrators and the Super Administrator | | **Regular User** | Can only use platform features without creation and editing permissions | Employees, business personnel | A tenant can have multiple regular users who can belong to multiple user groups within the tenant, invited by administrators and the Super Administrator | diff --git a/doc/docs/zh/backend/skills/index.md b/doc/docs/zh/backend/skills/index.md index 10b37bc90b..f91edcdc8a 100644 --- a/doc/docs/zh/backend/skills/index.md +++ b/doc/docs/zh/backend/skills/index.md @@ -17,13 +17,13 @@ ## 快速开始 1. **了解能力**：阅读 [技能系统概览](./overview) 了解已支持的技能类型 -2. **体验创建**：在 [技能管理](../../user-guide/skills) 页面体验 NL-to-Skill 创建 +2. **体验创建**：在 [技能管理](../../user-guide/resource-repository/skill-repository.md) 页面体验 NL-to-Skill 创建 3. **手动创建**：上传 `SKILL.md` 或 ZIP 包创建自定义技能 4. **为智能体配置**：在智能体工具配置中勾选技能 ## 相关参考 -- [技能管理（用户指南）](../../user-guide/skills) +- [技能管理（用户指南）](../../user-guide/resource-repository/skill-repository.md) - [智能体开发指南](../../user-guide/agent-development) - [本地工具概览](../../user-guide/local-tools/index) - [SDK 工具开发规范](../../sdk/core/tools) diff --git a/doc/docs/zh/backend/skills/overview.md b/doc/docs/zh/backend/skills/overview.md index f3d866f782..f15179a80e 100644 --- a/doc/docs/zh/backend/skills/overview.md +++ b/doc/docs/zh/backend/skills/overview.md @@ -133,6 +133,6 @@ tags: ## 相关参考 -- [技能管理（用户指南）](../../user-guide/skills) +- [技能管理（用户指南）](../../user-guide/resource-repository/skill-repository.md) - [智能体开发指南](../../user-guide/agent-development) - [本地工具概览](../../user-guide/local-tools/index) diff --git a/doc/docs/zh/deployment/docker-build.md b/doc/docs/zh/deployment/docker-build.md index 6f2539e172..7db94cca7f 100644 --- a/doc/docs/zh/deployment/docker-build.md +++ b/doc/docs/zh/deployment/docker-build.md @@ -95,6 +95,13 @@ docker build --progress=plain -t nexent/nexent-data-process-gpu -f deploy/images # 🌐 构建前端镜像（仅当前架构） docker build --progress=plain -t nexent/nexent-web -f deploy/images/dockerfiles/web/Dockerfile . +# 在自定义子路径部署前端（请按需替换 /your-subpath） +# CONFIGURED_BASE_PATH 必须为 / 或以 / 开头且不含结尾斜杠的路径 +docker build --progress=plain --build-arg CONFIGURED_BASE_PATH=/your-subpath -t nexent/nexent-web -f deploy/images/dockerfiles/web/Dockerfile . + +# Windows Git Bash 会转换以 / 开头的参数；为保留自定义子路径，禁用该转换 +MSYS_NO_PATHCONV=1 docker build --progress=plain --build-arg CONFIGURED_BASE_PATH=/your-subpath -t nexent/nexent-web -f deploy/images/dockerfiles/web/Dockerfile . + # 📚 构建文档镜像（仅当前架构） docker build --progress=plain -t nexent/nexent-docs -f deploy/images/dockerfiles/docs/Dockerfile . @@ -218,29 +225,52 @@ bash deploy.sh docker --image-source local-latest > `local-latest` 会使用本地 `latest` Nexent 应用镜像并避免重新拉取这些镜像，无需修改 `deploy/docker/deploy.sh`。 -### 将本地镜像打包为离线部署包 +### 构建离线部署包 -构建本地 `latest` 镜像后，可以使用离线打包脚本把镜像和部署资源打包： +在联网机器上，可从仓库根目录构建包含 Docker 和 Kubernetes 资源的离线部署包： ```bash bash build.sh --package \ - --target docker \ - --version latest \ + --target all \ + --version v2.2.1 \ --platform amd64 \ --components infrastructure,application,data-process,supabase \ - --image-source local-latest \ + --image-source general \ --compress true \ - --output-dir offline-package/docker-local + --output-dir offline-package ``` -使用 `--version latest` 或 `--image-source local-latest` 时，脚本会使用本地 Nexent 应用镜像，并跳过这些 `latest` 标签的拉取。将包复制到目标机器后，可加载镜像并部署： +常用参数： + +| 参数 | 说明 | +| --- | --- | +| `--target` | 生成 `docker`、`k8s` 或 `all` 部署资源 | +| `--version` | 要拉取并打包的 Nexent 镜像版本 | +| `--platform` | 目标服务器架构：`amd64` 或 `arm64` | +| `--components` | 部署组件，同时决定需要打包的镜像 | +| `--image-source` | `general`、`mainland` 或 `local-latest` | +| `--include-source` | 是否加入项目源码，默认 `false` | +| `--compress` | 是否生成 zip 压缩包，默认 `false` | +| `--output-dir` | 未压缩离线包的输出目录 | + +如需打包本地构建的 `latest` 应用镜像： ```bash -cd offline-package/docker-local -bash deploy.sh --load-images docker \ +bash build.sh --package \ + --target docker \ --version latest \ + --platform amd64 \ --components infrastructure,application,data-process,supabase \ - --image-source local-latest + --image-source local-latest \ + --compress true \ + --output-dir offline-package/docker-local ``` -如果离线部署时需要推送到内部镜像仓库，可将 `--load-images` 替换为 `--push-images --image-registry-prefix registry.example.com/nexent`。如果省略前缀，入口脚本会先询问镜像仓库前缀，随后 `push-images.sh` 询问仓库账号和密码。部署配置会使用同一个镜像仓库前缀生成 Docker Compose 镜像引用。 +`local-latest` 会复用本地 Nexent 应用镜像，不会再次拉取这些 `latest` 镜像。构建脚本会生成镜像 tar、部署资源、`manifest.yaml` 和 `checksums.txt`，且不会复制本机的 `deploy/env/.env`、`deploy/env/monitoring.env` 或 `deploy.options`。 + +启用 `--compress true` 后，会在输出目录旁生成 `nexent-offline---.zip`。也可以手动运行 GitHub Actions 中的 [Build Offline Deployment Package](https://github.com/ModelEngine-Group/nexent/actions/workflows/build-offline-package.yml)，工作流会为 AMD64 和 ARM64 分别生成可下载的 `nexent--.zip`，默认保留 30 天。 + +离线包的获取和安装方法参见： + +- [Docker 安装部署中的离线部署](../quick-start/installation#离线部署) +- [Kubernetes 安装部署中的离线部署](../quick-start/kubernetes-installation#离线部署) diff --git a/doc/docs/zh/quick-start/faq.md b/doc/docs/zh/quick-start/faq.md index 121e593658..a3670ecd67 100644 --- a/doc/docs/zh/quick-start/faq.md +++ b/doc/docs/zh/quick-start/faq.md @@ -42,7 +42,7 @@ 2. **有效的 API 密钥**: 验证您的 API 密钥具有适当权限 3. **模型名称**: 确认模型标识符正确 4. **网络访问**: 确保您的部署可以访问提供商的服务器 - 关于如何配置模型，请参阅用户指南中的 [模型管理](../user-guide/model-management)。 + 关于如何配置模型，请参阅用户指南中的 [模型管理](../user-guide/agent-development/model-configuration.md)。 - **Q: 接入 DeepSeek 官方 API 时多轮对话会报错，如何解决？** - A: DeepSeek 官方当前仅支持文本对话接口，而 Nexent 的推理流程面向多模态设计。在多轮对话中，官方 API 无法正确接收多模态格式数据，因此会触发错误。建议改用硅基流动等已对 DeepSeek 系列模型完成多模态适配的供应商，既保持 DeepSeek 模型的体验，又能兼容 Nexent 的多模态调用链。具体来说，我们使用的消息体形如： diff --git a/doc/docs/zh/quick-start/installation.md b/doc/docs/zh/quick-start/installation.md index 2416ed948c..824b352957 100644 --- a/doc/docs/zh/quick-start/installation.md +++ b/doc/docs/zh/quick-start/installation.md @@ -14,16 +14,21 @@ ## 🚀 快速开始 -### 1. 下载和设置 +- [在线部署](#在线部署) +- [离线部署](#离线部署) + +### 在线部署 + +#### 1. 下载和设置 ```bash git clone https://github.com/ModelEngine-Group/nexent.git cd nexent ``` -> **💡 提示**: `deploy.sh` 使用 `deploy/env/.env` 作为运行配置。已有 `deploy/env/.env` 会原样保留；如果不存在，会优先复用 `docker/.env`，再回退到 `deploy/env/.env.example`。若需要配置语音模型（STT/TTS），请部署前或部署后修改 `deploy/env/.env` 中的相关参数。 +> **💡 提示**: Docker 和 Kubernetes 共用 `deploy/env/.env`。每次部署前，脚本会保留已有值、注释和旧版变量，并追加当前 `deploy/env/.env.example` 新增的变量。如果 `.env` 不存在，会优先复用旧版 `docker/.env`，再回退到当前模板；部署时必须存在可读的 `.env.example`。若需要配置语音模型（STT/TTS），请部署前或部署后修改 `deploy/env/.env` 中的相关参数。 -### 2. 部署选项 +#### 2. 部署选项 运行以下命令开始部署： @@ -74,10 +79,10 @@ bash deploy.sh docker --image-source local-latest #### ⚠️ 重要提示 -1️⃣ **首次部署 v1.8.0 及以上版本时**，需特别留意 Docker 日志中输出的 `suadmin` 超级管理员账号信息。该账号为系统最高权限账户，密码仅在首次生成时显示，后续无法再次查看，请务必妥善保存。 +1️⃣ **首次部署 v1.8.0 及以上版本时**，系统会创建 `suadmin@nexent.com` 超级管理员账号，默认密码为 `Nexent@123`，无需交互输入，创建成功后会在终端显示。可在首次部署前通过 `deploy/env/.env` 中的 `NEXENT_SUPER_ADMIN_PASSWORD` 覆盖默认值，非交互创建时终端会显示实际使用的密码。使用离线部署包并显式指定 `--config` 时例外：部署脚本会要求输入并确认密码，并以本次输入为准；手动输入的密码不会在终端显示。 > 该账号仅用于权限管理，无权开发智能体或创建知识库。请登录该账号，依次完成：访问租户资源→创建租户→创建租户管理员，然后使用租户管理员账号登录,即可使用全部功能。角色权限详情参见 [用户管理](../user-guide/user-management) -2️⃣ 忘记留意 `suadmin` 账号密码？请按照以下步骤操作： +2️⃣ 如需重建 `suadmin` 账号，请按照以下步骤操作： ```bash # Step1: 在supabase容器中删除su账号记录 docker exec -it supabase-db-mini bash @@ -92,11 +97,62 @@ docker exec -it nexent-postgresql bash psql -U root -d nexent delete from nexent.user_tenant_t where user_id = 'your_user_id'; -# Step 3: 重新部署并记录 su 账号密码 +# Step 3: 重新部署；非交互模式将使用配置值或默认密码 +``` +### 离线部署 + +目标服务器无法访问公网镜像仓库时，可从 GitHub Actions 获取已经打包好的离线部署包： + +1. 登录 GitHub，打开 [Build Offline Deployment Package](https://github.com/ModelEngine-Group/nexent/actions/workflows/build-offline-package.yml)。 +2. 选择目标版本对应的成功运行记录，在页面底部的 **Artifacts** 中下载与服务器架构匹配的压缩包。 +3. `amd64` 服务器下载 `nexent--amd64.zip`，ARM64 服务器下载 `nexent--arm64.zip`。 + +GitHub Actions 构建产物默认保留 30 天。如果目标版本的产物已过期，请联系维护者重新运行工作流。 + +下载后，将压缩包复制到离线服务器并解压。压缩包内直接包含离线包文件，无需再次解压内层归档： + +```bash +unzip nexent-v2.2.1-amd64.zip -d nexent +cd nexent +bash deploy.sh --load-images docker ``` -### 3. 访问您的安装 + +离线包默认安装Nexent全部组件，若需要重新选择组件、端口策略、镜像源或监控 provider 时，添加 `--config`： + +```bash +bash deploy.sh --load-images --config docker +``` + +如果服务器上保留了此前已部署的离线包，可通过 `--reuse-from` 复用其中的环境配置和部署选项： + +```bash +bash deploy.sh \ + --reuse-from /path/to/previous/nexent \ + --load-images \ + docker +``` + +指定目录必须是已解压的旧部署包根目录，并包含 `deploy/env/.env`。该参数会导入旧 `.env`、保留其已有值，并立即追加当前包 `.env.example` 新增的变量。存在 `monitoring.env` 和 Docker `deploy.options` 时也会复用；Docker 派生配置由新版本脚本重新生成。`--reuse-from` 可与 `--config`、`--defaults` 或 `--push-images` 组合使用。 + +首次创建 `suadmin@nexent.com` 时，非交互部署使用 `NEXENT_SUPER_ADMIN_PASSWORD`，默认值为 `Nexent@123`，创建成功后会在终端显示实际密码。离线部署使用 `--config` 时会要求手动输入并确认密码，输入值不会写入配置文件，也不会在终端显示。 + +如果需要先将镜像推送到目标环境可访问的内部仓库： + +```bash +bash deploy.sh \ + --push-images \ + --image-registry-prefix registry.example.com/nexent \ + docker +``` + +未传入仓库前缀时，脚本会先询问前缀；随后 `push-images.sh` 会在推送前询问仓库用户名和密码。 + +### 访问您的安装 部署成功完成后： + +> **获取管理员密码**：超级管理员账号为 `suadmin@nexent.com`。首次非交互创建成功时，终端会显示实际使用的密码；未额外配置时，默认密码为 `Nexent@123`。离线部署使用 `--config` 时，手动输入的密码不会保存或显示。如果密码已忘记，请按本文前面的“重建 `suadmin` 账号”步骤重建账号。 + 1. 在浏览器中打开 **http://localhost:3000** 2. 登录超级管理员账号 3. 访问租户资源 → 创建租户及租户管理员 @@ -171,38 +227,6 @@ bash uninstall.sh docker delete-all Docker 卸载脚本会读取 `deploy/env/.env` 中的 `ROOT_DIR` 并清理 Compose 资源。删除数据时会移除 `postgresql`、`elasticsearch`、`redis`、`minio`、`volumes`、`openssh-server`、`scripts`、`skills` 等服务目录；如果后续要复用已有数据，请选择保留 volumes。 -### 离线镜像包 - -需要把镜像和部署脚本搬到离线机器时，可使用 `deploy/offline/build_offline_package.sh`： - -```bash -bash deploy/offline/build_offline_package.sh \ - --target docker \ - --version v2.2.1 \ - --platform amd64 \ - --components infrastructure,application,data-process,supabase \ - --image-source general \ - --compress true \ - --output-dir offline-package -``` - -包目录会包含 `images/*.tar`、`load-images.sh`、`push-images.sh`、`deploy.sh`、`uninstall.sh`、`manifest.yaml`、`checksums.txt`、`deploy/env/.env.example`、`deploy/env/monitoring.env.example` 和 `deploy/sql`，不会包含本地 `deploy/env/.env`、`deploy/env/monitoring.env` 或 `deploy.options`。使用 `--compress true` 时，会在输出目录的父目录生成 `nexent-offline---.zip`。 - -在目标机器上部署时，包根目录的 `deploy.sh` 会优先复用已保存的 `deploy.options`，否则使用内置默认值，默认不进入 TUI。添加 `--config` 可进入交互式配置界面。如果离线包构建时使用了自定义版本、组件、端口策略或镜像源，请在部署时传入相同选项，或使用 `--config` 交互选择： - -```bash -cd offline-package -bash deploy.sh --load-images docker -``` - -如果需要先推送到内部镜像仓库并使用该前缀部署： - -```bash -bash deploy.sh --push-images --image-registry-prefix registry.example.com/nexent docker -``` - -启用 `--push-images` 且未传前缀时，`deploy.sh` 会先询问镜像仓库前缀；随后 `push-images.sh` 在推送前询问仓库账号和密码。 - ## 🔌 端口映射 | 服务 | 内部端口 | 外部端口 | 描述 | diff --git a/doc/docs/zh/quick-start/kubernetes-installation.md b/doc/docs/zh/quick-start/kubernetes-installation.md index c8f86a4b8c..97317f3e4d 100644 --- a/doc/docs/zh/quick-start/kubernetes-installation.md +++ b/doc/docs/zh/quick-start/kubernetes-installation.md @@ -14,7 +14,12 @@ ## 🚀 快速开始 -### 1. 准备 Kubernetes 集群 +- [在线部署](#在线部署) +- [离线部署](#离线部署) + +### 在线部署 + +#### 1. 准备 Kubernetes 集群 确保 Kubernetes 集群正常运行，且 kubectl 已配置好集群访问权限： @@ -23,14 +28,14 @@ kubectl cluster-info kubectl get nodes ``` -### 2. 克隆并进入目录 +#### 2. 克隆并进入目录 ```bash git clone https://github.com/ModelEngine-Group/nexent.git cd nexent ``` -### 3. 部署 +#### 3. 部署 运行部署脚本： @@ -57,17 +62,17 @@ bash deploy.sh k8s - **mainland**: 使用中国大陆镜像源 - **local-latest**: 使用本地 `latest` 镜像，并将 Nexent 应用镜像的拉取策略设为本地优先 -Kubernetes 使用与 Docker 相同的 `deploy/env/.env`。已有 `deploy/env/.env` 会原样保留；如果不存在，部署脚本会优先复用 `docker/.env`，再回退到 `deploy/env/.env.example`。 +Kubernetes 使用与 Docker 相同的 `deploy/env/.env`。每次部署前，脚本会保留已有值、注释和旧版变量，并追加当前 `deploy/env/.env.example` 新增的变量。如果 `.env` 不存在，会优先复用旧版 `docker/.env`，再回退到当前模板；部署时必须存在可读的 `.env.example`。 使用 `bash deploy.sh k8s --defaults` 可跳过 TUI，并复用已保存的 `deploy.options` 或内置默认值。 部署成功后，非敏感部署选项会保存到 `deploy/k8s/deploy.options`。下次交互部署时可选择复用本地配置或重新全量配置。 -### ⚠️ 重要提示 +#### ⚠️ 重要提示 -1️⃣ **首次部署 v1.8.0 及以上版本时**，部署过程中系统会提示您设置 `suadmin` 超级管理员账号的密码。该账号为系统最高权限账户，请输入您想要的密码并**妥善保存**——密码创建后无法再次找回。 +1️⃣ **首次部署 v1.8.0 及以上版本时**，系统会创建 `suadmin@nexent.com` 超级管理员账号，默认密码为 `Nexent@123`，无需交互输入，创建成功后会在终端显示。可在首次部署前通过 `deploy/env/.env` 中的 `NEXENT_SUPER_ADMIN_PASSWORD` 覆盖默认值，非交互创建时终端会显示实际使用的密码。使用离线部署包并显式指定 `--config` 时，部署脚本会要求输入并确认密码，并以本次输入为准；手动输入的密码不会在终端显示。 -2️⃣ 忘记记录 `suadmin` 账号密码？请按照以下步骤操作： +2️⃣ 如需重建 `suadmin` 账号，请按照以下步骤操作： ```bash # Step 1: 在 Supabase 数据库中删除 su 账号记录 @@ -83,14 +88,59 @@ kubectl exec -it -n nexent deploy/nexent-supabase-db -- psql -U postgres -c \ kubectl exec -it -n nexent deploy/nexent-postgresql -- psql -U root -d nexent -c \ "DELETE FROM nexent.user_tenant_t WHERE user_id='your_user_id';" -# Step 3: 重新部署并记录 su 账号密码 +# Step 3: 重新部署；非交互模式将使用配置值或默认密码 bash deploy.sh k8s ``` -### 4. 访问您的安装 +### 离线部署 + +目标集群无法访问公网镜像仓库时，可从 GitHub Actions 获取已经打包好的离线部署包： + +1. 登录 GitHub，打开 [Build Offline Deployment Package](https://github.com/ModelEngine-Group/nexent/actions/workflows/build-offline-package.yml)。 +2. 选择目标版本对应的成功运行记录，在 **Artifacts** 中下载与集群节点架构匹配的 `nexent--.zip`。 +3. 将压缩包复制到可以访问目标集群的管理节点并解压。工作流产物默认保留 30 天；产物过期时，可联系维护者重新运行工作流。 + +解压离线部署包： + +```bash +unzip nexent-v2.2.1-amd64.zip -d nexent +cd nexent +``` + +单节点且以 Docker 作为容器运行时的集群可直接加载并部署： + +```bash +bash deploy.sh --load-images k8s +``` + +如果管理节点上保留了此前已部署的离线包，可通过 `--reuse-from` 复用其中的环境配置和 Kubernetes 部署选项： + +```bash +bash deploy.sh \ + --reuse-from /path/to/previous/nexent \ + --load-images \ + k8s +``` + +指定目录必须是已解压的旧部署包根目录，并包含 `deploy/env/.env`。该参数会导入旧 `.env`、保留其已有值，并立即追加当前包 `.env.example` 新增的变量。存在 `monitoring.env` 和 Kubernetes `deploy.options` 时也会复用；Helm generated values 由新版本脚本重新生成。`--reuse-from` 可与 `--config`、`--defaults` 或 `--push-images` 组合使用。 + +其他单节点集群和多节点集群应将镜像推送到集群可访问的内部仓库，或使用对应容器运行时的工具，将镜像导入所有可能运行 Nexent Pod 的节点： + +```bash +bash deploy.sh \ + --push-images \ + --image-registry-prefix registry.example.com/nexent \ + k8s +``` + +离线包默认安装 Nexent 全部组件。添加 `--config` 可重新选择部署配置；首次创建超级管理员时，该模式会要求手动输入并确认密码，且不会显示或保存输入值。非交互部署使用 `NEXENT_SUPER_ADMIN_PASSWORD`，默认值为 `Nexent@123`，创建成功后会在终端显示实际密码。 + +### 访问您的安装 部署成功完成后： +> **获取管理员密码**：超级管理员账号为 `suadmin@nexent.com`。首次非交互创建成功时，终端会显示实际使用的密码；未额外配置时，默认密码为 `Nexent@123`。离线部署使用 `--config` 时，手动输入的密码不会保存或显示。如果密码已忘记，请按本文前面的“重建 `suadmin` 账号”步骤重建账号。 + | 服务 | 默认地址 | |---------|-----------------| | Web 应用 | http://localhost:30000 | @@ -187,38 +237,6 @@ bash uninstall.sh k8s delete-all `--delete-data` 和 `--delete-volumes` 是兼容 Helm 管理资源的参数；本地盘数据请使用 `--delete-local-data` 或 `--keep-local-data` 控制。`delete-all --keep-local-data` 会删除 namespace，但保留本地卷内容。 -### 离线镜像包 - -可在仓库根目录构建 Kubernetes 离线包： - -```bash -bash deploy/offline/build_offline_package.sh \ - --target k8s \ - --version v2.2.1 \ - --platform amd64 \ - --components infrastructure,application,data-process,supabase \ - --image-source general \ - --compress true \ - --output-dir offline-package -``` - -包内包含镜像 tar、`load-images.sh`、`push-images.sh`、根目录部署/卸载入口、Kubernetes Helm 资源、SQL 文件、`deploy/env/.env.example`、`deploy/env/monitoring.env.example`、`manifest.yaml` 和 `checksums.txt`，不会包含本地 `deploy/env/.env`、`deploy/env/monitoring.env` 或生成的 Helm values。使用 `--compress true` 时，会在输出目录的父目录生成 `nexent-offline---.zip`。 - -在目标机器上部署时，包根目录的 `deploy.sh` 会优先复用已保存的 `deploy.options`，否则使用内置默认值，默认不进入 TUI。添加 `--config` 可进入交互式配置界面。如果离线包构建时使用了自定义版本、组件、端口策略或镜像源，请在部署时传入相同选项，或使用 `--config` 交互选择。如果是单节点、Docker 作为容器运行时的集群，可以直接加载并部署： - -```bash -cd offline-package -bash deploy.sh --load-images k8s -``` - -多节点集群需要在每个可能运行 Nexent Pod 的节点上加载镜像，或将镜像推送到集群可访问的内部镜像仓库，再使用匹配的镜像参数部署： - -```bash -bash deploy.sh --push-images --image-registry-prefix registry.example.com/nexent k8s -``` - -启用 `--push-images` 且未传前缀时，`deploy.sh` 会先询问镜像仓库前缀；随后 `push-images.sh` 在推送前询问仓库账号和密码。 - ## 🔧 部署命令 ```bash diff --git a/doc/docs/zh/quick-start/kubernetes-upgrade-guide.md b/doc/docs/zh/quick-start/kubernetes-upgrade-guide.md index 10d5d9f057..a82021ebe0 100644 --- a/doc/docs/zh/quick-start/kubernetes-upgrade-guide.md +++ b/doc/docs/zh/quick-start/kubernetes-upgrade-guide.md @@ -41,7 +41,7 @@ bash deploy.sh k8s 脚本会自动检测您之前保存的部署设置（组件组合、端口策略、镜像来源等）。如果 `deploy.options` 文件不存在，系统会提示您输入配置信息。 > 💡 提示 -> - 若需配置语音模型（STT/TTS），请在对应的 `values.yaml` 中修改相关配置，或通过命令行参数传入。 +> - 升级时会保留 `deploy/env/.env` 中的已有值、注释、顺序和旧版独有变量，并自动追加当前 `deploy/env/.env.example` 新增的变量。部署前必须存在可读的模板。Helm generated values 会根据合并后的 `.env` 重新生成，请勿直接修改。语音模型（STT/TTS）也请在 `deploy/env/.env` 中配置。 --- diff --git a/doc/docs/zh/quick-start/upgrade-guide.md b/doc/docs/zh/quick-start/upgrade-guide.md index 8f2444b607..1a26d85df2 100644 --- a/doc/docs/zh/quick-start/upgrade-guide.md +++ b/doc/docs/zh/quick-start/upgrade-guide.md @@ -40,7 +40,7 @@ bash deploy.sh docker 缺少 deploy.options 的情况下，会提示需要重新选择部署配置，例如组件组合、端口策略、镜像来源等。按照您之前的部署方式重新选择即可。 > 💡 提示 -> - 已有 `deploy/env/.env` 会原样保留；如果不存在，部署脚本会优先复用 `docker/.env`，再回退到 `deploy/env/.env.example`。 +> - 升级时会保留 `deploy/env/.env` 中的已有值、注释、顺序和旧版独有变量，并追加当前 `deploy/env/.env.example` 新增的变量。如果 `.env` 不存在，会优先复用旧版 `docker/.env`，再回退到当前模板。加载镜像或启动服务前必须存在可读的 `.env.example`。 > - 若需配置语音模型（STT/TTS），请在 `deploy/env/.env` 中补充相关变量，我们将尽快提供前端配置入口。 ## 🌐 步骤三：验证部署 diff --git a/doc/docs/zh/sdk/opentelemetry-design.md b/doc/docs/zh/sdk/opentelemetry-design.md index 9b4de05eb9..29eaa0d37d 100644 --- a/doc/docs/zh/sdk/opentelemetry-design.md +++ b/doc/docs/zh/sdk/opentelemetry-design.md @@ -521,11 +521,8 @@ flowchart TD API --> HTTP[FastAPI HTTP span: 可配置隐藏] API --> Bind[绑定 AgentRunMetadata] Bind --> Mem[解析 memory 开关] - Mem --> Strategy{with_memory / no_memory} - Strategy -->|with_memory| G1[generate_stream_with_memory] - Strategy -->|no_memory| G2[generate_stream_no_memory] - G1 --> AR[agent_run async generator] - G2 --> AR + Mem --> GS[generate_stream enable_memory=true/false] + GS --> AR[agent_run async generator] AR --> Thread[agent_run_thread] Thread --> NX[NexentAgent / CoreAgent] NX --> A0[agent.run span: AGENT] diff --git a/doc/docs/zh/security.md b/doc/docs/zh/security.md index fe18308867..662ffcf4b8 100644 --- a/doc/docs/zh/security.md +++ b/doc/docs/zh/security.md @@ -3,7 +3,7 @@ **请勿通过公开的 GitHub issues、讨论或其他公开渠道报告安全漏洞。** 相反，请通过联系我们的安全团队负责任地披露： -📧 [chenshuangrui@gmail.com](mailto:chenshuangrui@gmail.com) +📧 [zhenggaoqi@huawei.com](mailto:zhenggaoqi@huawei.com) ## 需要包含的内容： - 漏洞的详细描述 diff --git a/doc/docs/zh/user-guide/agent-development.md b/doc/docs/zh/user-guide/agent-development.md index 40805aeeaf..ce1dec8445 100644 --- a/doc/docs/zh/user-guide/agent-development.md +++ b/doc/docs/zh/user-guide/agent-development.md @@ -2,467 +2,21 @@ 在智能体开发页面中，您可以创建、配置和管理智能体。智能体是 Nexent 的核心功能，它们能够理解您的需求并执行相应的任务。 -## 🔧 创建智能体 +## 快速导航 -在智能体管理页签下，点击"创建 Agent"即可创建一个空白智能体，点击"退出创建"即可退出创建模式。 -如果您有现成的智能体配置，也可以导入使用： +本模块包含以下四个配置页面，帮助您全方位配置智能体： -1. 点击"导入 Agent" -2. 在弹出的文件选择对话框中选择智能体配置文件（JSON 格式） -3. 点击"打开"按钮，系统会验证配置文件的格式和内容，并显示导入的智能体信息 +| 页面 | 说明 | +|------|------| +| [模型配置](./agent-development/model-configuration) | 接入和管理 AI 模型，包括大语言模型、向量化模型、视觉语言模型、重排模型以及语音模型（语音合成与语音识别） | +| [知识库配置](./agent-development/knowledge-configuration) | 创建知识库并上传文档，支持多种文件格式，让智能体能够检索您的私有数据 | +| [记忆配置](./agent-development/memory-configuration) | 配置智能体的多层级记忆系统，实现跨对话的知识累积与个性化服务 | +| [智能体配置](./agent-development/agent-configuration) | 创建智能体、配置协作 Agent、选择工具、编写提示词，并进行调试和发布 | -
- -
+## 主要步骤 -> ⚠️ **提示**：如果导入了重名的智能体，系统会弹出提示弹窗。您可以选择： -> - **直接导入**：保留重复名称，导入后的智能体会处于不可用状态，需手动修改智能体名称和变量名后才能使用 -> - **重新生成并导入**：系统将调用 LLM 对智能体进行重命名，会消耗一定的模型 token 数，可能耗时较长 - -> 📌 **重要说明**：通过导入创建的智能体，如果其工具中包含 `knowledge_base_search` 等知识库检索工具，这些工具只会检索**当前登录用户在本环境中有权限访问的知识库**。导入文件中原有的知识库配置不会自动继承，因此实际检索结果和回答效果，可能与智能体原作者环境下的表现存在差异。 - -
- -
- -## 👥 配置协作智能体/工具 - -您可以为创建的智能体配置其他协作智能体，也可以为它配置可使用的工具，以赋予智能体能力完成复杂任务。 - -### 🤝 协作 Agent - -协作智能体用于帮助当前智能体完成复杂任务。协作智能体的来源分为两类： - -- **内部 Agent**：平台已发布的智能体 -- **外部 A2A Agent**：通过 A2A 协议发现的第三方 Agent - -1. 点击"协作 Agent"页签下的加号，弹出可选择的智能体列表 -2. 智能体列表分为"内部 Agent"和"外部 A2A Agent"两个页签，您可以根据需要选择 -3. 在下拉列表中选择要添加的智能体 -4. 允许选择多个协作智能体 -5. 可点击 × 取消选择此智能体 - -
- -
- -#### 🌐 添加外部 A2A Agent - -Nexent 支持通过 A2A 协议与第三方 Agent 进行通信。您可以通过以下两种方式发现外部 A2A Agent： - -##### 通过 URL 发现 Agent - -如果您知道目标 Agent 的 Agent Card 地址，可以使用 URL 发现方式： - -
- -
- -1. 在外部 A2A Agent 列表中，点击"添加外部 Agent"按钮 -2. 选择"URL 发现"页签 -3. 填写 Agent Card URL 地址，例如：`https://example.com/.well-known/agent.json` -4. 点击"发现"按钮，系统会自动获取 Agent 的相关信息 -5. 发现成功后，可以查看 Agent 的名称、描述、能力等信息 -6. 点击"添加到列表"完成添加 - -> 💡 **提示**：Agent Card 是符合 A2A 1.0 规范的 Agent 描述文件，包含了 Agent 的名称、描述、调用地址、能力等信息。 - -##### 通过 Nacos 发现 Agent - -如果您的 Agent 注册在 Nacos 服务发现平台，可以使用 Nacos 发现方式： - -
- -
- -1. 在外部 A2A Agent 列表中，点击"添加外部 Agent"按钮 -2. 选择"Nacos 发现"页签 -3. 首次使用时，需要先配置 Nacos 连接信息： - - **Nacos 服务器地址**：填写 Nacos 服务器地址，如 `http://127.0.0.1:8848` - - **命名空间 ID**：填写 Nacos 命名空间 ID（可选） - - **分组名**：填写服务分组名，默认为 `DEFAULT_GROUP` - - **用户名/密码**：填写 Nacos 访问凭证（可选） -4. 点击"保存配置"保存 Nacos 连接信息 -5. 填写要扫描的 Agent 服务名称 -6. 点击"扫描"按钮，系统会从 Nacos 中获取匹配的 Agent 信息 -7. 扫描结果会列出所有匹配的 Agent，可以选择需要的 Agent 添加到列表 - -> ⚠️ **注意**：确保 Nacos 服务正常运行，且目标 Agent 已正确注册到 Nacos。 - -##### 管理已发现的外部 Agent - -在外部 A2A Agent 列表中，您可以查看和管理所有已发现的外部 Agent： - - - -
- -
- -1. **查看 Agent 详情**：点击 Agent 卡片，可以查看其完整信息，包括名称、描述、URL、能力列表等 -2. **测试 Agent**：点击"测试"按钮，可以向该 Agent 发送测试消息，验证其是否正常工作 -3. **与 Agent 对话**：点击"对话"按钮，可以打开对话窗口，与该 Agent 进行实时交互 -4. **配置调用协议**：点击"协议配置"按钮，可以选择该 Agent 的调用协议： - - **HTTP + JSON**：使用 REST API 风格调用 - - **JSON-RPC**：使用 JSON-RPC 协议调用 -5. **刷新 Agent 信息**：如果 Agent 信息发生变化，可以点击"刷新"按钮重新获取最新的 Agent Card -6. **移除 Agent**：点击"移除"按钮，可以将该 Agent 从已发现列表中删除 - -> 💡 **使用场景**： -> - 通过 URL 发现快速接入已知的第三方 Agent 服务 -> - 通过 Nacos 发现批量接入同一服务注册中心的所有 Agent -> - 配置协议以兼容不同 Agent 服务提供商的要求 - - -###### 通过URL对接[DataAgent](https://gitcode.com/datagallery/dataagent) A2A Agent -1. 参考[DataAgent文档](https://gitcode.com/datagallery/dataagent#%F0%9F%8C%90-a2a-10-%E6%9C%8D%E5%8A%A1%E6%A8%A1%E5%BC%8F)以A2A服务模式启动DataAgent - >当前Nexent不支持带认证的agent，启动DataAgent时请勿设置auth-token -
- -
- -2. 参考[通过 URL 发现 Agent](#通过-url-发现-agent)接入agent，url为http://\:9999/.well-known/agent-card.json -3. 参考[管理已发现的外部 Agent](#管理已发现的外部-agent)配置调用协议，选择HTTP+JSON方式接入 - -### 🛠️ 选择智能体的工具 - -智能体可以使用各种工具来完成任务，如知识库检索、文件解析、图片解析、收发邮件、文件管理等本地工具，也可接入第三方 MCP 工具，或自定义工具。 - -1. 在"选择智能体的工具"页签右侧，点击"刷新工具"来刷新可用工具列表 -2. 选择想要添加工具所在的分组 -3. 查看分组下可选用的所有工具，可点击 ⚙️ 查看工具描述，进行工具参数配置 -4. 点击工具名即可选中该工具，再次点击可取消选择 - - 如果工具有必备参数没有配置，选择时会弹出弹窗引导进行参数配置 - - 如果所有必备参数已配置完成，选择则会直接选中 - -
- -
- -> 💡 **小贴士**： -> 1. 请选择 `knowledge_base_search` 工具，启用知识库的检索功能。 -> 2. 请选择 `analyze_text_file` 工具，启用文档类、文本类文件的解析功能。 -> 3. 请选择 `analyze_image` 工具，启用图片类文件的解析功能。 -> -> ⚠️ **向量化模型配置**：使用 `knowledge_base_search` 工具时，需要确保知识库已配置向量化模型。对于存量知识库，系统会提示选择向量化模型，请务必选择**创建该知识库时使用的向量化模型**。若选择的模型与知识库创建时使用的模型不一致，可能导致检索失败或结果不准确。 -> -> 📚 想了解系统已经内置的所有本地工具能力？请参阅 [本地工具概览](./local-tools/index.md)。 -> 📚 想了解技能能力？请参阅 [技能管理](./skills.md)。 - -### 🔌 添加 MCP 工具 - -在"选择智能体的工具"页签右侧，点击"MCP 配置"，可在弹窗中进行 MCP 服务器的配置，查看已配置的 MCP 服务器 - -您可以通过以下两种方式在 Nexent 中添加 MCP 服务 - -**1️⃣ 通过 URL 添加 MCP 服务** - -🔔 该方法适用于已有独立部署的 MCP 服务（支持 SSE 与 Streamble HTTP 协议）： - ->1. 在界面上方的 **Add MCP Server** 区域填写 **Server name** 、 **Server URL** -> ->⚠️ **注意**：服务器名称只能包含英文字母和数字，不能包含空格、下划线等其他字符 -> ->2. 点击 右侧 **+ Add** 按钮，完成单个服务添加 - -**2️⃣ 通过 JSON 配置添加容器化 MCP 服务** - -🔔 该方法适用于 npx 部署的容器化 MCP 服务 - ->1. 在 **Add Containerized MCP Service** 输入框中，填写符合示例格式的 JSON 配置 -> ->```json ->{ -> "mcpServers": { -> "service-name": { -> "args": [ -> "mcp-package-name@version", -> "additional-parameters" -> ], -> "command": "npx" -> } -> } ->} ->``` -> ->2. 在下方 **Port** 输入框中，填写容器化服务对应的端口号 ->3. 点击右侧 **+ Add** 按钮，完成容器化服务添加 - -
- -
- -有许多第三方服务如 [ModelScope](https://www.modelscope.cn/mcp) 提供了 MCP 服务，您可以快速接入使用。 -您也可以自行开发 MCP 服务并接入 Nexent 使用，参考文档 [MCP 工具开发](../backend/tools/mcp)。 - -**3️⃣ 存量 API 转换为 MCP 服务** - -🔔 该方法适用于将已有的 REST API 接口快速转换为 MCP 工具，无需额外开发即可让智能体调用现有 API 能力： - ->1. 在 MCP 配置模块选择 **"API 转换为 MCP"** 接入类型 -> ->2. 在下方的输入框中填写 API 基础信息： -> - **服务名称**：MCP 服务的展示名称 -> - **OpenAPI JSON**：OpenAPI 3.x 规范的 JSON 内容 -> - **基础服务 URL**：API 服务的基础地址（支持 http/https） -> ->3. 点击右下角 **+ 添加** 按钮，完成对应 MCP 服务的转换 - -
- -
- -> ->4. 转换完成后，可在 **Outer APIs** 页签下查看所有外部 API 转换的 MCP 工具 - -
- -
- -
- -
- ->💡 **使用场景**： ->- 快速接入企业内部的 REST API 接口 ->- 将第三方服务的 HTTP API 转换为 MCP 工具 ->- 无需编写 MCP Server 代码，直接通过 OpenAPI 规范生成工具 - - -### ⚙️ 自定义工具 - -您可参考以下指导文档，开发自己的工具，并接入 Nexent 使用，丰富智能体能力。 - -- [LangChain 工具指南](../backend/tools/langchain) -- [MCP 工具开发](../backend/tools/mcp) -- [SDK 工具文档](../sdk/core/tools) - -### 🧪 工具测试 - -无论是什么类型的工具（内置工具、外部接入的 MCP 工具，还是自定义开发工具），Nexent 都提供了"工具测试"能力。如果您在创建智能体时不确定某个工具的效果，可以使用测试功能来验证工具是否按预期工作。 - -1. 点击工具的小齿轮按钮 ⚙️，进入工具的详细配置弹窗 -2. 首先确保已经配置了工具的必备参数（带红色星号的参数） -3. 在弹窗的左下角点击"工具测试"按钮 -4. 右侧会新弹出一个测试框 -5. 在测试框中输入测试工具的入参，例如： - - 测试本地知识库检索工具 `knowledge_base_search` 时，需要输入： - - 测试的 `query`，例如"维生素C的功效" - - 检索的模式 `search_mode`（默认为 `hybrid`） - - 目标检索的知识库列表 `index_names`，如 `["医疗", "维生素知识大全"]` - - 若不输入 `index_names`，则默认检索知识库页面所选中的全部知识库 - - 是否启用重排模型（默认为 `false`），启用后配置重排模型，实现对检索结果的重排优化 -6. 输入完成后点击"执行测试"开始测试，并在下方查看测试结果 - -
- -
- -## 📝 描述业务逻辑 - -### ✍️ 描述智能体应该如何工作 - -根据选择的协作智能体和工具，您现在可以用简洁的语言来描述，您希望这个智能体应该如何工作。Nexent 会根据您的描述，自动为您生成智能体配置以及提示词等信息。 - -1. 在"描述智能体应该如何工作"下的编辑框中，输入简洁描述，如"你是一个专业的知识问答小助手，具备本地知识检索和联网检索能力，综合信息以回答用户问题" -2. 选择模型（生成提示词时选择更聪明的模型以优化回复逻辑），点击"生成智能体"按钮，Nexent 会为您生成智能体详细内容，包括基础信息以及提示词（角色、使用要求、示例） -3. 您可在下方智能体详细内容中，针对自动生成的内容（包括基础信息和提示词）进行编辑微调 - -#### 📋 智能体基础信息配置 - -在基础信息区域，若您对自动生成的内容不满意，您可以手工修改以下各项： - -| 配置项 | 说明 | -|--------|------| -| **智能体名称** | 智能体的展示名称，用于界面显示和用户识别。 | -| **智能体变量名** | 智能体的内部标识名称，用于代码中引用该智能体。只能包含字母、数字和下划线，且必须以字母或下划线开头。 | -| **作者** | 智能体的创建者名称，默认值为当前登录用户的邮箱。 | -| **用户组** | 智能体所属的用户组，用于权限管理和组织管理。若为空，则表示无所属用户组。 | -| **组内权限** | 控制同组用户对该智能体的访问权限：
- **同组可编辑**：同组用户可以查看和编辑该智能体
- **同组只读**：同组用户只能查看，不能编辑
- **私有**：只有创建者和管理员可以访问 | -| **大语言模型** | 智能体运行时使用的大语言模型，用于处理推理和生成回复。 | -| **智能体运行最大步骤数** | 智能体在单次对话中最多可以执行的思考-行动循环次数。步数越多，智能体可以处理更复杂的任务，但也会消耗更多资源。 | -| **提供运行摘要** | 控制智能体在被用作子智能体时，是否向主智能体提供运行细节：
- **开启（默认）**：当此智能体被用作子智能体时，会向主智能体提供详细的运行过程摘要
- **关闭**：当此智能体被用作子智能体时，只返回最终结果，不提供详细的运行过程 | -| **智能体描述** | 智能体的功能描述，用于说明智能体的用途和能力。 | - -> 💡 **使用建议**： -> - 智能体变量名应使用有意义的英文命名，如 `code_assistant`、`data_analyst` 等 -> - 智能体运行最大步骤数建议根据任务复杂度设置，简单的问答任务可设为 3-5 步，复杂的推理任务可设为 10-20 步 -> - 如果子智能体的运行过程对主智能体的决策有参考价值，建议开启"提供运行摘要"选项。如果只需要子智能体的最终结果以减少上下文消耗，建议关闭此选项 - -
- -
- -## 🐛 调试与保存 - - -在完成初步智能体配置后，您可以对智能体进行调试，根据调试结果微调提示词，持续提升智能体表现。 - -1. 在页面右下角点击"调试"按钮，弹出智能体调试页面 -2. 与智能体进行测试对话，观察智能体的响应和行为 -3. 查看对话表现和错误信息，根据测试结果优化智能体提示词 - -调试成功后，可点击右下角"保存"按钮，此智能体将会被保存并出现在智能体列表中。 - -## 🐛 版本管理 - -Nexent 支持智能体的版本管理，您可以在调试过程中，保存不同版本的智能体配置。 - -确认智能体配置无误后，您可发布智能体。发布后智能体将在智能体空间、开始问答中可见。 - -![版本管理1](./assets/agent-development/version_management_1.png) - -若需回滚到其他版本，可在版本管理页面点击"回滚"按钮。 - -![版本管理2](./assets/agent-development/version_management_2.png) - -### 🚀 发布为 A2A Agent - -Nexent 支持将已发布的智能体作为 A2A Agent 暴露给外部系统调用。在发布版本时，您可以勾选"发布为 A2A Agent"选项，将当前智能体注册为符合 A2A 1.0 规范的 Agent。 - -
- -
- -发布成功后，系统会显示 A2A Agent 的调用信息，包括： - -
- -
- -| 信息项 | 说明 | -|--------|------| -| **Endpoint ID** | A2A Agent 的唯一标识符 | -| **Agent Card URL** | Agent 发现端点，外部系统通过此地址获取 Agent 描述 | -| **协议版本** | A2A 协议版本，当前为 1.0 | -| **REST 端点** | 基于 REST 风格的 API 端点 | -| **JSON-RPC 端点** | 基于 JSON-RPC 2.0 协议的调用端点 | - -#### 调用方式 - -发布后的 A2A Agent 支持以下两种调用协议： - -##### REST API - -```bash -# 获取 Agent Card（用于 Agent 发现） -GET /nb/a2a/{endpoint_id}/.well-known/agent-card.json - -# 发送同步消息 -POST /nb/a2a/{endpoint_id}/message:send -Content-Type: application/json - -{ - "message": { - "role": "user", - "content": "请帮我完成某个任务" - } -} - -# 发送流式消息（SSE） -POST /nb/a2a/{endpoint_id}/message:stream -Content-Type: application/json - -{ - "message": { - "role": "user", - "content": "请帮我完成某个任务" - } -} - -# 获取任务状态 -GET /nb/a2a/{endpoint_id}/tasks/{task_id} -``` - -##### JSON-RPC 2.0 - -```bash -POST /nb/a2a/{endpoint_id}/v1 -Content-Type: application/json - -# 发送同步消息 -{ - "jsonrpc": "2.0", - "method": "SendMessage", - "params": { - "message": { - "role": "user", - "content": "请帮我完成某个任务" - } - }, - "id": 1 -} - -# 发送流式消息 -{ - "jsonrpc": "2.0", - "method": "SendStreamingMessage", - "params": { - "message": { - "role": "user", - "content": "请帮我完成某个任务" - } - }, - "id": 2 -} - -# 获取任务状态 -{ - "jsonrpc": "2.0", - "method": "GetTask", - "params": { - "taskId": "task_abc123" - }, - "id": 3 -} -``` - -> 💡 **提示**： -> - 本地开发时，请将路径前面的 `/nb/a2a` 部分替换为 `http://localhost:5013/nb/a2a` -> - 生产环境请将路径替换为您的服务器域名或公网 IP 地址 - -> ⚠️ **注意事项**： -> - 调用 A2A Agent 需要在请求头中携带有效的认证信息 -> - Agent Card 信息会被缓存，刷新间隔为 1 小时 -> - 如需更新 Agent 信息，需要重新发布智能体版本 - -当发布的Agent为符合A2A协议的Agent时，在智能体列表中，用户可以在智能体列表中点击下面这个按钮查看A2A Agent调用具体信息： - -
- -
- -## 🔧 管理智能体 - -在左侧智能体列表中，您可对已有的智能体进行以下操作： - -### 🔗 查看调用关系 - -查看智能体所使用的协作智能体/工具，以树状图形式明晰查看智能体调用关系。 - -
- -
- -### 📤 导出 - -可将调试成功的智能体导出为 JSON 配置文件，在创建智能体时可以使用此 JSON 文件以导入的方式创建副本。 - - -### 📋 复制 - -复制 Agent，便于智能体的实验、多版本调试与并行开发。 - -### 🗑️ 删除 - -删除智能体（不可撤销，请谨慎操作）。 - -## 🚀 下一步 - -完成智能体开发后，您可以： - -1. 在 **[智能体空间](./agent-space)** 中查看和管理所有智能体 -2. 在 **[开始问答](./start-chat)** 中与智能体进行交互 -3. 在 **[记忆管理](./memory-management)** 配置记忆以提升智能体的个性化能力 - -如果您在使用程中遇到任何问题，请参考我们的 **[常见问题](../quick-start/faq)** 或在 [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions) 中进行提问获取支持。 +1. **配置模型** - 在模型管理中接入所需的 AI 模型 +2. **准备知识库** - 创建知识库并上传相关文档 +3. **配置记忆** - 根据需要开启记忆功能，让智能体记住重要信息 +4. **开发智能体** - 创建智能体，选择工具和协作 Agent，编写业务描述 +5. **调试与发布** - 调试智能体表现，满意后保存并发布 diff --git a/doc/docs/zh/user-guide/agent-development/agent-configuration.md b/doc/docs/zh/user-guide/agent-development/agent-configuration.md new file mode 100644 index 0000000000..9ca6fcda05 --- /dev/null +++ b/doc/docs/zh/user-guide/agent-development/agent-configuration.md @@ -0,0 +1,588 @@ +# 智能体开发 + +在智能体开发页面中，您可以创建、配置和管理智能体。智能体是 Nexent 的核心功能，它们能够理解您的需求并执行相应的任务。 + +## 🔧 创建智能体 + +在智能体管理页签下，点击"新建"按钮即可创建一个空白智能体，点击"退出创建"即可退出创建模式。 +如果您有现成的智能体配置，也可以导入使用： + +1. 点击"导入"按钮 +2. 在弹出的文件选择对话框中选择智能体配置文件（支持 JSON 或 zip 压缩包格式） +3. 点击"打开"按钮，系统会验证配置文件的格式和内容，并显示导入的智能体信息 + +![image-20260805040825286](./../assets/agent-development/import.png) + +![image-20260805043517509](./../assets/agent-development/import-2.png) + + +> ⚠️ **提示**：如果导入了重名的智能体，系统会弹出提示弹窗。您可以选择： +> +> - **直接导入**：保留重复名称，导入后的智能体会处于不可用状态，需手动修改智能体名称和变量名后才能使用 +> - **重新生成并导入**：系统将调用 LLM 对智能体进行重命名，会消耗一定的模型 token 数，可能耗时较长 + +> 📌 **重要说明**：通过导入创建的智能体，如果其工具中包含 `knowledge_base_search` 等知识库检索工具，这些工具只会检索**当前登录用户在本环境中有权限访问的知识库**。导入文件中原有的知识库配置不会自动继承，因此实际检索结果和回答效果，可能与智能体原作者环境下的表现存在差异。 + +
+ +
+ +## 👥 配置协作智能体/工具 + +您可以为创建的智能体配置其他协作智能体，也可以为它配置可使用的工具，以赋予智能体能力完成复杂任务。 + +### 🤝 协作 Agent + +协作智能体用于帮助当前智能体完成复杂任务。协作智能体的来源分为两类： + +- **内部 Agent**：平台已发布的智能体 +- **外部 A2A Agent**：通过 A2A 协议发现的第三方 Agent + +1. 点击"协作 Agent"页签下的加号，弹出可选择的智能体列表 +2. 智能体列表分为"内部 Agent"和"外部 A2A Agent"两个页签，您可以根据需要选择 +3. 在下拉列表中选择要添加的智能体 +4. 允许选择多个协作智能体 +5. 可点击 × 取消选择此智能体 + +
+ +
+ +#### 🌐 添加外部 A2A Agent + +Nexent 支持通过 A2A 协议与第三方 Agent 进行通信。您可以通过以下两种方式发现外部 A2A Agent： + +##### 通过 URL 发现 Agent + +如果您知道目标 Agent 的 Agent Card 地址，可以使用 URL 发现方式： + +
+ +
+ +1. 在外部 A2A Agent 列表中，点击"添加外部 Agent"按钮 +2. 选择"URL 发现"页签 +3. 填写 Agent Card URL 地址，例如：`https://example.com/.well-known/agent.json` +4. 如果目标 Agent Card 需要认证，在"自定义请求头"中填写 JSON 对象，例如：`{"Authorization": "Bearer "}` +5. 点击"发现"按钮，系统会自动获取 Agent 的相关信息 +6. 发现成功后，可以查看 Agent 的名称、描述、能力等信息 +7. 点击"添加到列表"完成添加 + +> 💡 **提示**：自定义请求头会随该外部 Agent 保存，仅用于获取和刷新 Agent Card，不会用于后续调用 Agent。再次发现同一 URL 时，留空会保留现有配置，填写 `{}` 可清空配置。 + +> 💡 **提示**：Agent Card 是符合 A2A 1.0 规范的 Agent 描述文件，包含了 Agent 的名称、描述、调用地址、能力等信息。 + +##### 通过 Nacos 发现 Agent + +如果您的 Agent 注册在 Nacos 服务发现平台，可以使用 Nacos 发现方式： + +
+ +
+ +1. 在外部 A2A Agent 列表中，点击"添加外部 Agent"按钮 +2. 选择"Nacos 发现"页签 +3. 首次使用时，需要先配置 Nacos 连接信息： + - **Nacos 服务器地址**：填写 Nacos 服务器地址，如 `http://127.0.0.1:8848` + - **命名空间 ID**：填写 Nacos 命名空间 ID（可选） + - **分组名**：填写服务分组名，默认为 `DEFAULT_GROUP` + - **用户名/密码**：填写 Nacos 访问凭证（可选） +4. 点击"保存配置"保存 Nacos 连接信息 +5. 填写要扫描的 Agent 服务名称 +6. 点击"扫描"按钮，系统会从 Nacos 中获取匹配的 Agent 信息 +7. 扫描结果会列出所有匹配的 Agent，可以选择需要的 Agent 添加到列表 + +> ⚠️ **注意**：确保 Nacos 服务正常运行，且目标 Agent 已正确注册到 Nacos。 + +##### 管理已发现的外部 Agent + +在外部 A2A Agent 列表中，您可以查看和管理所有已发现的外部 Agent： + +
+ +
+ +1. **查看 Agent 详情**：点击 Agent 卡片，可以查看其完整信息，包括名称、描述、URL、能力列表等 +2. **测试 Agent**：点击"测试"按钮，可以向该 Agent 发送测试消息，验证其是否正常工作 +3. **与 Agent 对话**：点击"对话"按钮，可以打开对话窗口，与该 Agent 进行实时交互 +4. **配置调用协议**：点击"协议配置"按钮，可以选择该 Agent 的调用协议： + - **HTTP + JSON**：使用 REST API 风格调用 + - **JSON-RPC**：使用 JSON-RPC 协议调用 +5. **配置调用认证**：如果 Agent Card 声明了 `securitySchemes` 和 `securityRequirements`，点击"Agent 认证"按钮，填写所需认证值。系统会按 Card 声明将值放入请求头、查询参数或 Cookie；同一认证组合中的字段必须同时填写。 +6. **刷新 Agent 信息**：如果 Agent 信息发生变化，可以点击"刷新"按钮重新获取最新的 Agent Card +7. **移除 Agent**：点击"移除"按钮，可以将该 Agent 从已发现列表中删除 + +> 💡 **使用场景**： +> +> - 通过 URL 发现快速接入已知的第三方 Agent 服务 +> - 通过 Nacos 发现批量接入同一服务注册中心的所有 Agent +> - 配置协议以兼容不同 Agent 服务提供商的要求 + +###### 通过URL对接[DataAgent](https://gitcode.com/datagallery/dataagent) A2A Agent + +1. 参考[DataAgent文档](https://gitcode.com/datagallery/dataagent#%F0%9F%8C%90-a2a-10-%E6%9C%8D%E5%8A%A1%E6%A8%A1%E5%BC%8F)以A2A服务模式启动DataAgent + + > 当前Nexent不支持带认证的agent，启动DataAgent时请勿设置auth-token + +
+ +
+ +2. 参考[通过 URL 发现 Agent](#通过-url-发现-agent)接入agent，url为http://\:9999/.well-known/agent-card.json +3. 参考[管理已发现的外部 Agent](#管理已发现的外部-agent)配置调用协议，选择HTTP+JSON方式接入 + +### 🛠️ 选择智能体的工具或技能 + +智能体可以使用各种工具与技能来完成任务，如知识库检索、文件解析、图片解析、收发邮件、文件管理等本地工具，也可接入第三方或自行开发的 MCP 工具或技能。 + +1. 在"选择智能体的工具"页签右侧，点击"刷新工具"来刷新可用工具列表 +2. 点击"选择工具"或"选择技能"按钮，可根据标签分组浏览当前可用的工具或技能清单 +3. 点击 ⚙️ 查看工具或技能的描述，并配置工具或技能参数 +4. 点击即可选中工具或技能，回到智能体已选择工具或已选择技能处可执行删除 + - 如果工具有必填参数没有配置，选择时会弹出弹窗引导进行参数配置 + - 如果所有必备参数已配置完成，选择则会直接选中 + +![image-20260805052952538](./../assets/agent-development/set-tools-1.png) + +![image-20260805053822083](./../assets/agent-development/set-tools-2.png) + + +> 💡 **小贴士**： +> +> 1. 请选择 `knowledge_base_search` 工具，启用知识库的检索功能。 +> 2. 请选择 `analyze_text_file` 工具，启用文档类、文本类文件的解析功能。 +> 3. 请选择 `analyze_image` 工具，启用图片类文件的解析功能。 +> +> ⚠️ **注意**：使用 `knowledge_base_search` 工具时，需要事先创建知识库。请务必确保**创建知识库时使用的向量化模型**与当前生效的向量化模型一致。否则将会导致检索失败或结果不准确。 +> +> 📚 想了解系统已经内置的所有本地工具能力？请参阅 [本地工具概览](../local-tools/index.md)。 +> 📚 想了解技能能力？请参阅 [技能管理](../resource-repository/skill-repository.md)。 + +### 🔌 添加 MCP 工具 + +在"选择智能体的工具"页签右侧，点击"MCP 配置"，可在弹窗中进行 MCP 服务器的配置，查看已配置的 MCP 服务器 + +您可以通过以下两种方式在 Nexent 中添加 MCP 服务 + +**1️⃣ 通过 URL 添加 MCP 服务** + +🔔 该方法适用于已有独立部署的 MCP 服务（支持 SSE 与 Streamble HTTP 协议）： + +> 1. 在界面上方的 **Add MCP Server** 区域填写 **Server name** 、 **Server URL** +> +> ⚠️ **注意**：服务器名称只能包含英文字母和数字，不能包含空格、下划线等其他字符 +> +> 2. 点击 右侧 **+ Add** 按钮，完成单个服务添加 + +**2️⃣ 通过 JSON 配置添加容器化 MCP 服务** + +🔔 该方法适用于 npx 部署的容器化 MCP 服务 + +> 1. 在 **Add Containerized MCP Service** 输入框中，填写符合示例格式的 JSON 配置 +> +> ```json +> { +> "mcpServers": { +> "service-name": { +> "args": ["mcp-package-name@version", "additional-parameters"], +> "command": "npx" +> } +> } +> } +> ``` +> +> 2. 在下方 **Port** 输入框中，填写容器化服务对应的端口号 +> 3. 点击右侧 **+ Add** 按钮，完成容器化服务添加 + +
+ +
+ +有许多第三方服务如 [ModelScope](https://www.modelscope.cn/mcp) 提供了 MCP 服务，您可以快速接入使用。 +您也可以自行开发 MCP 服务并接入 Nexent 使用，参考文档 [MCP 工具开发](../../backend/tools/mcp)。 + +**3️⃣ 存量 API 转换为 MCP 服务** + +🔔 该方法适用于将已有的 REST API 接口快速转换为 MCP 工具，无需额外开发即可让智能体调用现有 API 能力： + +> 1. 在 MCP 配置模块选择 **"API 转换为 MCP"** 接入类型 +> 2. 在下方的输入框中填写 API 基础信息： +> +> - **服务名称**：MCP 服务的展示名称 +> - **OpenAPI JSON**：OpenAPI 3.x 规范的 JSON 内容 +> - **基础服务 URL**：API 服务的基础地址（支持 http/https） +> +> 3. 点击右下角 **+ 添加** 按钮，完成对应 MCP 服务的转换 + +
+ +
+ +> 4. 转换完成后，可在 **Outer APIs** 页签下查看所有外部 API 转换的 MCP 工具 + +
+ +
+![image-20260805051112051](./../assets/agent-development/add_mcp_from_api_2.png) + + +> 💡 **使用场景**： +> +> - 快速接入企业内部的 REST API 接口 +> - 将第三方服务的 HTTP API 转换为 MCP 工具 +> - 无需编写 MCP Server 代码，直接通过 OpenAPI 规范生成工具 + +### ⚙️ 自定义工具 + +您可参考以下指导文档，开发自己的工具，并接入 Nexent 使用，丰富智能体能力。 + +- [LangChain 工具指南](../../backend/tools/langchain) +- [MCP 工具开发](../../backend/tools/mcp) +- [SDK 工具文档](../../sdk/core/tools.md) + +### 🔌 创建或导入技能 + +在智能体的高级配置中切换到“选择技能”页签，点击“构建技能”，即可通过对话生成或文件安装的方式创建技能。技能创建成功后会进入当前用户可访问的技能列表，但**不会自动关联到当前智能体**；完成创建后，还需要选择该技能并保存智能体配置。 + +#### 交互式创建技能 (NL2SKILL) + +“交互式创建”适合从自然语言需求开始构建技能： + +1. 点击“构建技能”，选择“交互式创建”页签。 +2. 在左侧对话区描述技能的用途、执行步骤、输入输出以及限制条件。例如：“创建一个读取 CSV 文件并输出数据质量报告的技能”。 +3. 系统会流式生成技能草稿。生成过程中可以停止生成，也可以继续对话，补充要求或要求系统修改已有草稿。 +4. 在右侧草稿区检查并编辑技能信息与文件内容： + - **技能名称**：必填，并且不能与现有技能重名。 + - **技能描述**：必填，用于说明技能的用途和适用场景。 + - **标签**：最多添加 5 个标签，每个标签不超过 20 个字符，便于后续搜索和筛选。 + - **用户组与组内权限**：根据需要设置技能的可见范围及编辑权限；仅在当前账号具备相关权限时显示或允许修改。 + - **技能文件**：`SKILL.md` 是技能的主文件，不可重命名或删除；还可以新增、编辑、重命名或删除脚本、资源文件等其他文件。 +5. 确认草稿内容后，点击“创建”。如果技能名称已存在，请修改名称后重新创建。 + +#### 从文件安装技能 + +“安装”页签用于导入已经准备好的技能文件。点击上传区域选择文件，或将文件拖入上传区域。每次只能上传一个文件，支持以下格式： + +| 文件格式 | 适用场景 | 文件要求 | +| --- | --- | --- | +| `.md` | 仅包含主说明文件的单文件技能 | 文件应为完整的 `SKILL.md`，并在 YAML Front Matter 中包含 `name` 和 `description` | +| `.zip` | 包含脚本、资源或其他辅助文件的多文件技能 | 压缩包中必须包含 `SKILL.md`；该文件可以位于压缩包根目录或某个子目录中，其他文件会随技能一并导入 | + +`SKILL.md` 的基本结构示例如下： + +```markdown +--- +name: csv-report +description: 分析 CSV 文件并生成数据质量报告 +tags: + - data-analysis +--- + +# CSV 数据质量报告 + +在用户提供 CSV 文件后，检查缺失值、重复数据和字段类型，并输出结构化报告。 +``` + +上传后，系统会从 `SKILL.md` 中读取技能名称和描述，并展示解析结果。确认无误后点击“创建”完成安装。 + +> ⚠️ **导入限制**： +> +> - `SKILL.md` 必须包含有效的 YAML Front Matter，并提供 `name` 和 `description`；缺少任一字段都将导致导入失败。 +> - `SKILL.md` 编码格式必须为`UTF-8`。 +> - 导入不会覆盖同名技能。如果名称已存在，请修改 `SKILL.md` 中的 `name`，然后重新上传。 +> - 多文件技能应先将技能目录压缩为 `.zip`，并确保压缩包内包含 `SKILL.md`。 + +#### 将新技能关联到智能体 + +技能创建或安装成功后，按以下步骤将其用于当前智能体： + +1. 如列表尚未更新，点击“刷新技能”。 +2. 点击“选择技能”，通过名称、描述或标签找到新技能并选中。 +3. 如果技能包含需要填写的参数，点击 ⚙️ 完成参数配置。 +4. 返回智能体配置页保存配置。之后，该智能体才可以在运行过程中使用此技能。 + +有关技能的查看、编辑、权限和删除等完整管理方式，请参阅 [技能管理](../resource-repository/skill-repository.md)。 + +### 🧪 工具测试 + +无论是什么类型的工具（内置工具、外部接入的 MCP 工具，还是自定义开发工具），Nexent 都提供了"工具测试"能力。如果您在创建智能体时不确定某个工具的效果，可以使用测试功能来验证工具是否按预期工作。 + +1. 点击工具的小齿轮按钮 ⚙️，进入工具的详细配置弹窗 +2. 首先确保已经配置了工具的必备参数（带红色星号的参数） +3. 在弹窗的左下角点击"工具测试"按钮 +4. 右侧会新弹出一个测试框 +5. 在测试框中输入测试工具的入参，例如： + - 测试本地知识库检索工具 `knowledge_base_search` 时，需要输入： + - 测试的 `query`，例如"维生素C的功效" + - 检索的模式 `search_mode`（默认为 `hybrid`） + - 目标检索的知识库列表 `index_names`，如 `["医疗", "维生素知识大全"]` + - 若不输入 `index_names`，则默认检索知识库页面所选中的全部知识库 + - 是否启用重排模型（默认为 `false`），启用后配置重排模型，实现对检索结果的重排优化 +6. 输入完成后点击"执行测试"开始测试，并在下方查看测试结果 + +
+ +
+ +## 📝 描述业务逻辑 + +### ✍️ 描述智能体应该如何工作 + +根据选择的协作智能体和工具，您现在可以用简洁的语言来描述，您希望这个智能体应该如何工作。Nexent 会根据您的描述，自动为您生成智能体配置以及提示词等信息。 + +1. 在"描述智能体应该如何工作"下的编辑框中，输入简洁描述，如"你是一个专业的知识问答小助手，具备本地知识检索和联网检索能力，综合信息以回答用户问题" +2. 选择模型（生成提示词时选择更聪明的模型以优化回复逻辑），点击"生成智能体"按钮，Nexent 会为您生成智能体详细内容，包括基础信息以及提示词（角色、使用要求、示例） +3. 您可在下方智能体详细内容中，针对自动生成的内容（包括基础信息和提示词）进行编辑微调 + +#### 📋 智能体基础信息配置 + +在基础信息区域，若您对自动生成的内容不满意，您可以手工修改以下各项： + +| 配置项 | 说明 | +| ---------------- | ------------------------------------------------------------------------------------------------------------ | +| **智能体名称** | 智能体的展示名称，用于界面显示和用户识别。 | +| **智能体变量名** | 智能体的内部标识名称，用于引用该智能体。只能包含字母、数字和下划线，且必须以字母或下划线开头。 | +| **大语言模型** | 智能体运行时使用的大语言模型，用于推理、调用工具和生成回复。 | +| **智能体描述** | 智能体的功能描述，用于说明其用途和能力。 | + +> 💡 **使用建议**： +> +> - 智能体变量名应使用有意义的英文命名，以方便模型理解，如 `code_assistant`、`data_analyst` 等 + +![image-20260805112840883](./../assets/agent-development/generate_agent.png) + + +#### ⚙️ 高级设置 + +点击"智能体详细内容"右侧的"高级设置"按钮，可以进一步配置智能体的运行方式、权限、自验证和安全护栏。高级设置分为"基本设置"和"安全护栏"两个页签；修改完成后，需要点击弹窗中的"确定"，再保存智能体使其生效。 + +![image-20260805113109082](./../assets/agent-development/agent-settings.png) + +##### 基本设置 + +| 配置项 | 默认值 | 说明 | +| -------------------------- | -------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **作者** | 当前登录用户 | 智能体的作者名称。 | +| **是否为主智能体** | 是 | 控制该智能体是否作为可独立对话的主智能体展示。选择"否"时，智能体更适合作为协作智能体使用；即使已经发布，也不会出现在可发起对话的主智能体列表中。 | +| **用户组** | 无 | 智能体所属的一个或多个用户组，用于组织和权限管理。只有具备相应权限的用户可以修改该项。 | +| **组内权限** | 只读 | 控制同组用户的访问方式：**可编辑**表示同组用户可以查看和编辑；**只读**表示同组用户只能查看；**私有**表示仅创建者和管理员可以访问。 | +| **智能体运行最大步骤数** | 15 | 单次运行允许执行的最大思考—行动循环次数，必须为不小于 1 的整数。达到上限时，系统会停止继续执行并基于已完成工作生成总结。步数越多，越适合复杂任务，但会增加耗时和资源消耗。 | +| **提供运行摘要** | 否 | 仅在该智能体被主智能体作为协作智能体调用时生效。选择"是"会在最终结果后附带工作过程摘要，帮助主智能体了解执行细节；选择"否"则只返回最终结果，可减少主智能体的上下文占用。 | +| **输出预留** | 使用模型默认值 | 限制每次回复最多可输出的 token 数，并从模型上下文窗口中预留相应空间。值越大，可生成的回答越长，但输入和历史对话空间越小，也会更早触发上下文压缩；值越小，可保留更多输入，但回答可能被截断。填写时必须为正整数，且不能超过所选模型的最大输出 token 数。 | +| **自验证** | 否 | 启用后，系统会在关键执行事件和最终回答阶段进行检查；发现工具调用、检索证据、代码执行或答案质量存在问题时，会要求智能体修正或重试。最终回答持续未通过验证时，系统会返回受控说明，避免输出未经验证的确定性结论。 | + +> 💡 **配置建议**： +> +> - 简单问答可将最大步骤数设为 3–5，复杂的检索或推理任务可设为 10–20，并结合实际调试结果调整。 +> - 只有在主智能体需要了解协作智能体执行过程时才开启"提供运行摘要"；否则保持关闭可节省上下文。 +> - "输出预留"通常保持为空即可。仅在回答经常被截断或需要为历史对话保留更多空间时进行调整。 + +##### 🚧安全护栏 + +安全护栏使用按顺序执行的正则表达式规则，检查发送给模型的内容以及工具调用过程中的数据。安全护栏默认关闭，且不依赖"自验证"开关；需要单独打开"规则列表"旁的开关，并至少配置一条有效规则。规则按列表顺序匹配，同一段内容以首个命中的规则为准。 + +![image-20260805113522462](./../assets/agent-development/safety-fence.png) + +每条规则包含以下配置： + +| 配置项 | 说明 | +| ---------------- | ------------------------------------------------------------------------------------------------------------------------- | +| **规则名称** | 规则的唯一标识。名称重复时界面会给出提示，建议使用能表达检测目标的名称。 | +| **正则表达式** | 使用 Python `re` 语法描述要匹配的内容。运行时默认忽略大小写；语法无效的规则不会参与运行时检查。 | +| **严重级别** | 指定命中后的处理方式：**阻断**、**脱敏**或**放行**。新建规则默认为"阻断"。 | +| **说明** | 可选的规则用途说明，便于维护和审查。 | + +不同严重级别在各检查位置的实际行为如下： + +| 严重级别 | 最新用户输入 | 历史消息 | 工具入参 | 工具输出 | +| -------- | ------------------------ | -------------------------- | -------------------------------- | -------------------------------- | +| **阻断** | 终止本次运行并返回拒绝说明 | 降级为脱敏后再发送给模型 | 阻止本次工具调用 | 因工具已经执行，降级为脱敏 | +| **脱敏** | 将命中内容替换为 `***` 后继续 | 将命中内容替换为 `***` 后继续 | 将命中的字符串参数替换为 `***` 后调用工具 | 将命中内容替换为 `***` 后写入智能体上下文 | +| **放行** | 不修改内容，继续运行 | 不修改内容，继续运行 | 不修改参数，继续调用 | 不修改输出，继续运行 | + +安全护栏还提供以下辅助能力： + +- **智能生成**：选择用于生成的模型，并用自然语言描述要匹配或拦截的内容。系统会自动判断生成单个候选表达式还是多条规则；确认候选或勾选规则后再导入列表。 +- **规则管理**：支持手动添加、编辑、复制、单条删除和批量删除规则，并显示阻断、脱敏、放行规则的数量分布。 +- **正则测试预览**：粘贴样本文本后，可以实时查看命中的文本、规则名称和命中次数。预览仅用于验证匹配效果，不会执行阻断或脱敏动作。 + +> ⚠️ **注意**：安全护栏是基于正则表达式的内容筛查，不等同于完整的语义安全审核。AI 生成的规则也可能存在误报或漏报；请先在"正则测试预览"中使用正常样本和风险样本进行验证，再保存配置。 + +## 🐛 调试与保存 + +在完成初步智能体配置后，您可以对智能体进行调试，根据调试结果微调提示词，持续提升智能体表现。 + +1. 在页面右下角点击"调试"按钮，弹出智能体调试页面 +2. 与智能体进行测试对话，观察智能体的响应和行为 +3. 查看对话表现和错误信息，根据测试结果优化智能体提示词 + +调试成功后，可点击右下角"保存"按钮，此智能体将会被保存并出现在智能体列表中。 + +## 🐛 版本管理 + +Nexent 支持智能体的版本管理，您可以在调试过程中，保存不同版本的智能体配置。 + +确认智能体配置无误后，您可点击"发布"按钮正式发布智能体。发布后智能体将在 Agent 仓库、开始问答中可见，并可进行历史版本管理。 + +点击"版本管理"栏目右下角的版本对比按钮，可以回顾历史版本的信息，并与最新版本的问答效果进行对比。 + +![image-20260805130308885](./../assets/agent-development/version_management_1.png) + +若需回滚到其他版本，可在版本右侧的菜单中点击"回滚"。 + +![image-20260805132354244](./../assets/agent-development/version_management_2.png) + +### 🚀 发布为 A2A Agent + +Nexent 支持将已发布的智能体作为 A2A Agent 暴露给外部系统调用。在发布版本时，您可以勾选"发布为 A2A Agent"选项，将当前智能体注册为符合 A2A 1.0 规范的 Agent。 + +
+ +
+ +发布成功后，系统会显示 A2A Agent 的调用信息，包括： + +
+ +
+ +| 信息项 | 说明 | +| ------------------ | ------------------------------------------------- | +| **Endpoint ID** | A2A Agent 的唯一标识符 | +| **Agent Card URL** | Agent 发现端点，外部系统通过此地址获取 Agent 描述 | +| **协议版本** | A2A 协议版本，当前为 1.0 | +| **REST 端点** | 基于 REST 风格的 API 端点 | +| **JSON-RPC 端点** | 基于 JSON-RPC 2.0 协议的调用端点 | + +#### 调用方式 + +发布后的 A2A Agent 支持以下两种调用协议： + +##### REST API + +```bash +# 获取 Agent Card（用于 Agent 发现） +GET /nb/a2a/{endpoint_id}/.well-known/agent-card.json + +# 发送同步消息 +POST /nb/a2a/{endpoint_id}/message:send +Content-Type: application/json + +{ + "message": { + "role": "user", + "content": "请帮我完成某个任务" + } +} + +# 发送流式消息（SSE） +POST /nb/a2a/{endpoint_id}/message:stream +Content-Type: application/json + +{ + "message": { + "role": "user", + "content": "请帮我完成某个任务" + } +} + +# 获取任务状态 +GET /nb/a2a/{endpoint_id}/tasks/{task_id} +``` + +##### JSON-RPC 2.0 + +```bash +POST /nb/a2a/{endpoint_id}/v1 +Content-Type: application/json + +# 发送同步消息 +{ + "jsonrpc": "2.0", + "method": "SendMessage", + "params": { + "message": { + "role": "user", + "content": "请帮我完成某个任务" + } + }, + "id": 1 +} + +# 发送流式消息 +{ + "jsonrpc": "2.0", + "method": "SendStreamingMessage", + "params": { + "message": { + "role": "user", + "content": "请帮我完成某个任务" + } + }, + "id": 2 +} + +# 获取任务状态 +{ + "jsonrpc": "2.0", + "method": "GetTask", + "params": { + "taskId": "task_abc123" + }, + "id": 3 +} +``` + +> 💡 **提示**： +> +> - 本地开发时，如果使用 docker 启动：请将路径前面的 `/nb/a2a` 部分替换为 `http://localhost:5013/nb/a2a`；如果通过 k8s 启动，请使用 `http://localhost:30013/nb/a2a` +> - 生产环境请将路径替换为您的服务器域名或公网 IP 地址 + +> ⚠️ **注意事项**： +> +> - 调用 A2A Agent 需要在请求头中携带有效的认证信息 +> - Agent Card 信息会被缓存，刷新间隔为 1 小时 +> - 如需更新 Agent 信息，需要重新发布智能体版本 + +当发布的Agent为符合A2A协议的Agent时，在智能体列表中，点击最左侧的icon查看A2A Agent调用具体信息 + +![image-20260805132836142](./../assets/agent-development/a2a-find-detail.jpg) + + +## 🔧 管理智能体清单 + +点击"选择智能体"，您可浏览当前环境中可以编辑的完整智能体清单。你可以在上方的搜索框中 + +![image-20260805115401285](./../assets/agent-development/agent-list.png) + +智能体条目右侧的一系列icon按钮代表了你可以对智能体执行的所有管理操作。从左至右分别为： + +### 📋 复制 + +创建完全一致的 Agent 克隆体，便于多版本备份或并行测试。 + +### 🔗 查看调用关系 + +查看智能体所使用的协作智能体/工具，以树状图形式明晰查看智能体调用关系。 + +
+ +
+ +### 📤 导出 + +可将调试成功的智能体导出为 JSON 或 Zip 文件，在创建智能体时可以使用此文件以导入的方式创建副本。含有技能的复杂智能体将默认被导出为 Zip 压缩包。 + +### 🗑️ 删除 + +从本地环境中彻底删除智能体。 + +## 🚀 下一步 + +完成智能体开发后，您可以： + +1. 在 **[Agent仓库](../agent-development.md)** 中管理、发布你的智能体，或获取更多其他开发者的智能体 +2. 在 **[开始问答](../start-chat.md)** 中与智能体进行交互 +3. 在 **[记忆管理](./memory-configuration.md)** 配置记忆以提升智能体的个性化能力 + +如果您在使用程中遇到任何问题，请参考我们的 **[常见问题](../../quick-start/faq.md)** 或在 [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions) 中进行提问获取支持。 diff --git a/doc/docs/zh/user-guide/agent-development/knowledge-configuration.md b/doc/docs/zh/user-guide/agent-development/knowledge-configuration.md new file mode 100644 index 0000000000..01a99355c5 --- /dev/null +++ b/doc/docs/zh/user-guide/agent-development/knowledge-configuration.md @@ -0,0 +1,195 @@ +# 知识库配置 + +在知识库模块，您可以创建和管理知识库，上传各种格式的文件，并生成内容总结。知识库是智能体的重要信息来源，让智能体能够访问您的私有数据和文档。 + +## 🔧 创建知识库 + +1. 点击左侧知识库列表上方的"创建知识库"按钮 +2. 在弹出的创建面板中，填写以下配置项： + +| 配置项 | 说明 | +| -------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **知识库名称** | 必填，名称必须唯一，且只能包含中文字符或小写字母，不允许空格、斜杠等特殊字符。系统会在输入时自动检测名称是否重复 | +| **向量模型** | 选择用于文档向量化的 Embedding 模型。模型分为两类：**文本向量模型（embedding）** 和 **多模态向量模型（multi_embedding）**。选择多模态向量模型后，知识库将自动支持图片等非文本内容的向量化处理。详见[向量模型说明](#向量模型说明) | +| **用户组** | 选择知识库可见的用户组（多选），控制哪些用户组可以查看和使用该知识库 | +| **组内权限** | 设置用户组内成员对该知识库的操作权限：**编辑（EDIT）** - 可上传/删除文件；**只读（READ_ONLY）** - 仅可查看和检索；**私有（PRIVATE）** - 仅创建者可用 | +| **保留源文件** | 开启后，上传的原始文件会被保留存储在系统中，方便后续重新处理或下载；关闭则仅保留向量化后的数据 | +| **存储配额** | 可选，为该知识库设置存储上限（支持 GB/MB 单位切换）。当上传文件导致存储用量接近配额时，系统会发出预警提示 | + +3. 配置完成后，在下方文件上传区域选择要上传的文件（可跳过，后续再上传） +4. 文件上传后系统自动创建知识库并开始处理文件 + + +![创建知识库](../assets/knowledge-base/create-knowledge-base.png) + +## 📁 上传文件 + +### 上传文件 + +1. 在知识库列表中选择要上传文件的知识库 +2. 点击文件上传区域，选择要上传的文件（支持多选），或直接拖拽文件到上传区域 +3. 系统会自动处理上传的文件，提取文本内容并进行向量化 +4. 可在列表中查看文件的处理状态 + +:::tip 文件大小限制 +单个文件上传大小限制为 **20 MB**。超过此限制的文件将无法上传。 +::: + +### 文件处理状态 + +文件上传后会经过多个处理阶段，共有 6 种状态： + +| 状态 | 说明 | +| ------------ | -------------------------------------------------------------------- | +| 等待处理 | 文件已上传，排队等待开始解析 | +| **解析中** | 系统正在提取文件文本内容 | +| **入库中** | 文本内容正在进行向量化并写入向量数据库 | +| **已就绪** | 文件处理完成，可正常检索使用 | +| **解析失败** | 文件解析过程中出错。将光标移至状态图标可查看详细错误原因和排查建议 | +| **入库失败** | 向量化入库过程中出错。将光标移至状态图标可查看详细错误原因和排查建议 | + +💡 光标移动至状态图标，以了解当前处理进度（如"已处理 23/50 个分片"）及失败状态的报错原因 + +![文件进度与错误提示](../assets/knowledge-base/tip.png) + +### 支持的文件格式 + +Nexent支持多种文件格式，包括： + +- **文本**: .txt, .md, .json文件 +- **PDF**: .pdf文件 +- **Word**: .docx文件 +- **PowerPoint**: .pptx文件 +- **Excel**: .xlsx文件 +- **EPUB** .epub文件 +- **数据文件**: .csv文件 +- **Web content**: .html, .xml文件 + +## 📊 知识库总结 + +建议您为每个知识库配置准确且完整的总结描述，这有助于后续智能体在进行检索时，准确选择合适的知识库。 + +### 手动生成与编辑总结 + +1. 点击知识库名称右侧的"详细内容"按钮，进入知识库概览页面 +2. 在概览页面选择合适的大语言模型（LLM），点击"自动总结"按钮为知识库自动生成内容总结 +3. 您可对生成的内容总结进行编辑修改，使其更准确 +4. 最后记得点击"保存"将您的修改保存 + +![内容总结](../assets/knowledge-base/summary-knowledge-base.png) + +### 自动总结频率 + +除了手动触发总结，您还可以为知识库设置**定时自动总结**，让系统在后台定期重新生成知识库的内容总结： + +| 频率选项 | 说明 | +| -------- | ---------------------------------------- | +| 1 小时 | 高频更新，适用于内容变化频繁的知识库 | +| 3 小时 | 中高频更新 | +| 6 小时 | 中频更新 | +| 1 天 | 每天更新一次，适用于大多数场景 | +| 1 周 | 每周更新一次，适用于内容较为稳定的知识库 | + +在概览页面的总结区域选择频率即可。系统会智能判断知识库是否有文档更新，若无新文档变更则跳过本次自动生成，避免不必要的资源消耗。 + +## 📂 Chunk 分片管理 + +文档上传后，系统会将文档拆分为多个 **Chunk（分片）**，每个分片包含一段文本内容并生成对应的向量索引。您可以对分片进行精细化管理： + +### 查看分片 + +1. 点击知识库名称进入文档列表 +2. 点击上方"Chunk 详情"标签页 +3. 页面顶部会列出知识库中的所有文档，点击某个文档即可查看该文档被拆分出的全部分片 +4. 分片以卡片形式展示，包含分片内容和所属文件名 + +### 搜索分片 + +在 Chunk 详情页使用搜索框进行搜索，系统会同时对结果进行关键词匹配和语义匹配（混合搜索），按综合相关性排序展示。 + +### 手动管理分片 + +| 操作 | 说明 | +| -------- | ------------------------------------------ | +| **创建** | 在 Chunk 详情页手动添加自定义分片内容 | +| **编辑** | 点击分片卡片进入编辑模式，修改分片文本内容 | +| **删除** | 删除不需要的分片 | +| **下载** | 将分片内容导出下载 | + +:::warning 模型兼容性限制 +分片的编辑和创建操作依赖向量模型的一致性。如果模型配置发生变化，相关操作可能被自动禁用，以防止产生不兼容的向量数据。 +::: + +![分片管理](../assets/knowledge-base/chunk_management.png) + + + +## 🧩 向量模型说明 + +系统中的 Embedding 模型分为两类： + +- **文本向量模型（embedding）**：用于纯文本文档的向量化，如 BGE、M3E 等 +- **多模态向量模型（multi_embedding）**：可同时处理文本和图片内容，如 DashScope、Jina 等 + +知识库创建时选择的向量模型会与该知识库绑定。建议在创建知识库时根据文档类型选择合适的模型，创建后该模型将贯穿知识库的整个生命周期。 + +## 🔧 使用知识库 + +Nexent 支持知识库与智能体绑定。创建智能体时，在工具配置区域启用对应的知识库检索工具，并选择关联的知识库。 + +根据知识库来源的不同，系统提供对应的检索工具。Nexent 原生知识库使用 **knowledge_base_search** 工具： + +![工具1](../assets/knowledge-base/knowledge-tool1.png) + +![工具2](../assets/knowledge-base/knowledge-tool2.png) + +在工具配置弹窗中，您可看到已绑定的知识库列表，支持搜索和勾选操作。每个知识库旁边会显示其向量模型信息，帮助您确认兼容性。 + +## 🔍 知识库管理 + +### 查看知识库 + +1. **知识库列表** + - 知识库页面左侧展示了所有已创建的知识库 + - 知识库列表处支持按名称搜索、按知识库来源和向量模型筛选 + - 每张知识库卡片展示以下信息： + + | 信息项 | 说明 | + | ------------ | ------------------------------------------------------ | + | 知识库名称 | 创建时设定的名称 | + | 文档数量 | 已上传的文档总数 | + | 分片数量 | 文档拆分后的 chunk 总数 | + | 来源 | 知识库来源（Nexent 原生 / 外部来源） | + | 创建时间 | 知识库创建日期 | + | 向量模型 | 绑定的 Embedding 模型名称 | + | 多模态标签 | 若启用多模态则显示 | + | 用户组标签 | 该知识库可见的用户组名称 | + | 权限图标 | 当前用户对该知识库的操作权限（将光标移至图标查看说明） | + | 无源文件标签 | 若已关闭"保留源文件"则显示 | + +2. **知识库详情** + - 点击知识库名称，可查看知识库中全部文档信息 + - 点击"详细内容"，可进入知识库概览页面查看和编辑内容总结 + +> 点击编辑，可管理知识库的名称、可见的用户组及组内权限 + +知识库权限 + +### 编辑知识库 + +1. **删除知识库** + - 点击知识库名称右侧"删除"按钮 + - 确认删除操作（此操作不可恢复） + +2. **删除或新增文件** + - 点击知识库名称，在文件列表中点击"删除"按钮，可从知识库中删除文件 + - 点击知识库名称，在文件列表下方文件上传区域，可新增文件到知识库中 + +## 🚀 下一步 + +完成知识库配置后，建议您继续配置： + +1. **[智能体开发](../agent-development)** - 创建和配置智能体 +2. **[开始问答](../start-chat)** - 与智能体进行交互 + +如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../../quick-start/faq.md)** 或在[GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions)中进行提问获取支持。 diff --git a/doc/docs/zh/user-guide/agent-development/memory-configuration.md b/doc/docs/zh/user-guide/agent-development/memory-configuration.md new file mode 100644 index 0000000000..21590a4194 --- /dev/null +++ b/doc/docs/zh/user-guide/agent-development/memory-configuration.md @@ -0,0 +1,203 @@ +# 记忆配置 + +Nexent 的记忆功能用于在多轮、跨会话交互中保留可复用的信息。当前记忆系统采用 **Tenant、User、Agent 三层架构**：Tenant 和 User 层保存长期记忆，Agent 层保存由特定用户与特定智能体交互产生的短期记忆。 + +开启记忆后，系统会在智能体运行前加载长期记忆并检索相关的 Agent 短期记忆；在运行结束前，智能体还会判断本轮对话是否产生了值得保存的新信息。 + +## 🎯 运作机制 + +在正常对话中，记忆功能按以下流程运行： + +1. 加载当前租户的 Tenant 长期记忆和当前用户的 User 长期记忆。 +2. 使用用户本轮最新问题，在当前用户与当前智能体的 Agent 短期记忆中执行一次相关性检索。 +3. 将可用的长期记忆和检索到的短期记忆加入智能体上下文。 +4. 智能体结合记忆、当前问题、工具执行结果和历史对话生成回答。 +5. 输出最终回答前，智能体判断是否出现了新的用户偏好、任务目标、行动计划、最新进展或纠错反思；有价值时，将其归纳为简洁的 Agent 短期记忆。 + +记忆通过内置工具执行，你可以通过检查工具运行状态来确认记忆载入的轨迹。若记忆检索失败，系统会跳过记忆并继续执行当前任务，避免因记忆服务异常中断整个对话。 + +![image-20260805151444129](./../assets/memory-management/memory-search-tool.png) + +> 💡 **说明**：智能体调试模式不会启用记忆检索或写入，避免调试数据影响正式记忆。请在“开始问答”中验证跨会话记忆效果。 + +## ⚙️ 进入记忆配置 + +1. 在左侧导航栏中点击“记忆配置”。 +2. 进入页面后，可以看到“基础设置”“Tenant”“User”和“Agent”四个页签。 +3. 页签名称旁的数字表示当前层级加载到的记忆条数。 + +### 基础设置 + +“基础设置”中目前提供“记忆能力”总开关。 + +| 配置项 | 默认值 | 说明 | +| ------------ | ------ | -------------------------------------------------------------------------------------- | +| **记忆能力** | 开启 | 开启后，正常对话会加载、检索和写入记忆；关闭后，智能体不再使用记忆，但已有记录不会被删除。 | + +开关修改后会立即保存。如果保存失败，页面会恢复为修改前的状态并显示错误提示。 + +![](./../assets/memory-management/memory-settings.png) + +## 📚 三层记忆架构 + +当前系统只使用以下三个记忆层级： + +| 层级 | 可见与生效范围 | 主要来源 | 使用方式 | +| ---------- | -------------------------------------- | ---------------------- | ------------------------------------------------ | +| **Tenant** | 当前租户内共享 | 具有权限的用户手动维护 | 作为组织级长期上下文提供给智能体 | +| **User** | 仅当前用户可见，可供该用户的智能体使用 | 当前用户手动维护 | 作为用户级长期上下文提供给智能体 | +| **Agent** | 当前用户与指定智能体之间隔离 | 智能体在对话中自动归纳 | 根据当前问题执行向量检索，选择相关内容加入上下文 | + +### Tenant 记忆 + +保存组织范围内稳定、通用的信息，例如： + +- 企业术语和统一口径 +- 通用工作规范和流程原则 +- 组织级偏好或约束 +- 需要被多个用户和智能体共同参考的事实 + +Tenant 记忆不会由智能体自动写入。只有具备 Tenant 记忆创建权限的用户才能看到“新建记忆”按钮，通常由租户管理员负责维护。 + +### User 记忆 + +只属于当前用户，适合保存跨智能体复用的稳定个人信息的记忆，例如： + +- 常用语言、格式和表达偏好 +- 长期有效的工作习惯 +- 持续性项目背景 +- 希望所有智能体遵循的个人要求 + +User 记忆由当前用户手动创建和维护。它不会与租户中的其他用户共享。 + +### Agent 记忆 + +Agent 记忆由智能体在正式对话中自动生成，并同时绑定当前用户和当前智能体。它将保存： + +- 用户对该智能体表达的偏好 +- 当前任务目标 +- 行动计划和最新进展 +- 根据用户反馈、报错或失败结果形成的反思总结 + +同一用户与同一智能体的 Agent 记忆可以跨会话召回，但不会在不同用户或不同智能体之间自动共享。主智能体和协作智能体也分别维护自己的 Agent 记忆。 + +系统要求智能体先判断、归纳并去重，再保存单条简洁、可复用的内容。整段对话、临时计算、中间噪声、未经验证的推测、重复内容、敏感密钥以及用户明确要求遗忘的信息不应写入记忆。单次智能体运行最多自动保存 3 条 Agent 短期记忆。 + +> ⚠️ **注意**：当前版本不会在后台定时将 Agent 短期记忆自动提升为 User 长期记忆。 + +## 🗂️ 查看和筛选记忆 + +Tenant、User 和 Agent 页签均以表格展示记忆记录，包括记忆内容、记忆类型、状态和创建时间。 + +所有层级都支持： + +- 按记忆内容搜索 +- 按状态筛选 +- 查看当前筛选结果数量 +- 分页浏览，每页可选择显示 10、20 或 50 条记录 + +Agent 页签还支持： + +- 按智能体、来源对话或创建日期范围筛选 +- 查看智能体名称和来源对话 +- 点击来源对话标题，返回产生该记忆的历史对话 + +![image-20260805151335245](./../assets/memory-management/agent-memory.png) + +### 记忆状态 + +| 状态 | 说明 | +| ------------ | ------------------------------------------------------------------------------------- | +| **生效中** | 记忆可以参与长期上下文加载或 Agent 短期记忆检索。 | +| **已归档** | 记忆仍保留在列表中，但不会参与智能体运行时的记忆加载和检索。 | +| **已停用** | 记忆暂时不可使用。可以由用户手动设置，也可能因 Agent 记忆与当前向量模型不兼容而产生。 | + +## ✍️ 新建、编辑和删除记忆 + +### 新建长期记忆 + +页面仅支持手动新建 Tenant 或 User 长期记忆；Agent 短期记忆由智能体运行过程生成，Agent 页签不提供“新建记忆”按钮。 + +1. 进入 Tenant 或 User 页签。 +2. 点击右上角“新建记忆”。 +3. 输入需要长期保留的记忆内容，最多 500 个字符。 +4. 点击“创建记忆”。 + +手动创建的记录默认为“长期记忆”和“生效中”状态。 + +![image-20260805142217753](./../assets/memory-management/add-memory.png) + +### 编辑记忆 + +1. 在目标记录右侧点击“编辑”按钮。 +2. 修改记忆内容或状态。 +3. 点击“保存修改”。 + +编辑内容时仍需遵守 500 个字符的限制。当前页面不支持通过编辑操作改变一条记录所属的层级或记忆类型。 + +如果 Agent 记忆与当前向量模型不兼容，该记录会以不可用样式显示，并禁止编辑；仍可将其删除。 + +### 删除记忆 + +点击记录右侧的“删除”按钮，并在确认弹窗中选择“确认删除”。删除后该记录不会再出现在页面中，也不会参与后续记忆加载或检索。 + +> ⚠️ **注意**：页面不提供恢复入口，请在删除前确认该记忆不再需要。 + +## 🔍 记忆检索与上下文使用 + +不同层级采用不同的使用方式： + +- **Tenant / User 长期记忆**：从记录库中读取生效中的长期记忆，直接作为长期上下文提供给智能体，不依赖语义相似度筛选。 +- **Agent 短期记忆**：使用本轮最新问题进行向量检索，并经过相关性融合、时间衰减、相似内容去重和上下文预算筛选后，将最有用的结果提供给智能体。 + +这意味着 Tenant 和 User 记忆应保持精炼、稳定；内容过多会直接占用模型上下文。Agent 记忆则可以随交互逐步积累，系统会优先召回与当前问题更相关、更新且不重复的内容。 + +## 🧩 向量模型与 Agent 记忆 + +Agent 短期记忆的生成和检索依赖租户当前配置的向量模型。进入“记忆配置”或“开始问答”时，如果租户尚未配置向量模型，系统会弹出弹框提示用户。 + +未配置可用向量模型时： + +- Tenant 和 User 长期记忆仍保存在记录库中，并按长期上下文方式管理。 +- 无法正常生成或检索 Agent 短期记忆。 + +切换向量模型后，使用旧模型建立索引的 Agent 记忆可能与当前索引不兼容。页面加载记录时会自动同步状态： + +| 向量兼容性 | 同步后状态 | +| ---------- | ---------- | +| 不兼容 | 已停用 | +| 兼容 | 生效中 | + +image-20260805142827846 + +![image-20260805142545186](./../assets/memory-management/disabled-memory.png) + +如果切回与原记录兼容的向量模型，已停用的 Agent 记忆会重新变为“生效中”。 + +## 💡 使用建议 + +### 编写高质量记忆 + +每条记忆应尽量只表达一个明确、可长期复用的事实。 + +✅ `用户希望技术方案先给出结论，再说明风险。` + +❌ 不推荐：`用户喜欢简洁回答、经常晚上工作、正在做多个项目，而且希望所有内容都用表格。` + +建议遵循以下原则： + +1. **保持原子性**：一条记忆只描述一个偏好、事实、目标或进展。 +2. **避免临时信息**：一次性的计算结果和短期无关信息不应保存。 +3. **定期维护**：对过期内容选择“已归档”或直接删除。 +4. **控制长期记忆数量**：Tenant 和 User 记忆会作为长期上下文使用，应避免冗长、重复与前后矛盾。 +5. **保护隐私**：不要保存密码、访问令牌、密钥或不必要的敏感个人信息。 + +## 🚀 下一步 + +配置好记忆后，您可以： + +1. 在 **[开始问答](../start-chat)** 中使用同一智能体发起多次对话，验证跨会话记忆效果。 +2. 在 **[模型配置](./model-configuration.md)** 中检查向量模型配置。 +3. 在 **[智能体配置](./agent-configuration.md)** 中继续创建和调整智能体。 + +如果使用过程中遇到问题，请参考 **[常见问题](../../quick-start/faq.md)**，或前往 [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions) 获取支持。 diff --git a/doc/docs/zh/user-guide/model-management.md b/doc/docs/zh/user-guide/agent-development/model-configuration.md similarity index 61% rename from doc/docs/zh/user-guide/model-management.md rename to doc/docs/zh/user-guide/agent-development/model-configuration.md index 6870f5544b..6630f02ff3 100644 --- a/doc/docs/zh/user-guide/model-management.md +++ b/doc/docs/zh/user-guide/agent-development/model-configuration.md @@ -1,58 +1,17 @@ -# 模型管理 +# 模型配置 -在模型管理模块中，您可以配置应用的基本信息，并接入各类AI模型，包括大语言模型、向量化模型和视觉语言模型。Nexent支持多种模型提供商，帮助您根据实际需求灵活选择最适合的模型。 - -## 🖼️ 应用配置 - -应用配置是模型管理的第一步，您可以配置应用的基本信息，包括应用图标、名称和描述。合理配置有助于提升应用的辨识度和用户体验。 - -- 应用图标和名称会展示在对话页面的左上角，帮助用户快速识别当前应用。 -- 应用的描述在生成智能体时会作为背景信息提供给模型，提升模型对应用场景的理解。 - -### 应用图标配置 - -点击应用图标可进行图标的配置。Nexent 提供了两种图标配置方式： - -- **使用预设图标**：从预设的图标库中选择，可选择图像及背景颜色，适合快速配置。 -- **上传自定义图片**：支持PNG、JPG图片格式，文件大小不超过2MB。 - -
- - -
- -### 应用名称及描述配置 - -#### 应用名称 - -- 应用名称会展示在对话页面的左上角，帮助用户快速识别当前应用。 -- 建议使用简洁明了、能体现应用功能的名称，避免使用特殊字符。 - -#### 应用描述 - -- 应用描述会作为背景信息提供给模型，帮助理解应用场景。 -- 建议突出应用核心功能，完整流畅且简洁明了。 - -
- -
+在模型配置模块中，您可以接入并配置各类 AI 模型，包括大语言模型、向量化模型、重排序模型、多模态模型和语音模型。Nexent 支持多种模型提供商，帮助您根据实际需求灵活选择最适合的模型。 ## 🤖 模型配置 -### 🔄 同步ModelEngine模型 - -Nexent支持与ModelEngine平台的无缝对接 - -👉 点击页面右上方 ***\*****ModelEngine配置*****\***，输入您的 API ，即可获取您在 ModelEgnine 上部署的所有模型 - ### 🛠️ 添加自定义模型 #### 添加单个模型 1. **添加自定义模型** - - 点击"添加自定义模型"按钮，进入添加模型弹窗。 + - 点击"添加模型"按钮，进入添加模型弹窗。 2. **选择模型类型** - - 点击模型类型下拉框，选择要添加的模型类型（大语言模型/向量化模型/视觉语言模型/重排模型）。 + - 点击模型类型下拉框，选择要添加的模型类型（大语言模型/向量模型/图片理解模型/图片生成模型/视频理解模型/重排模型/语音识别模型/语音合成模型）。 3. **配置模型参数** - **模型名称（必填）**：输入请求体中的模型名称。 - **展示名称**：可为模型设置一个展示名称，默认与模型名称相同。 @@ -70,7 +29,7 @@ Nexent支持与ModelEngine平台的无缝对接 - 配置完成后，点击"确定"按钮，模型将被添加到可用模型列表中。
- +
#### 批量添加模型 @@ -82,7 +41,7 @@ Nexent支持与ModelEngine平台的无缝对接 2. **选择模型提供商** - 点击模型提供商下拉框，选择模型提供商。 3. **选择模型类型** - - 点击模型类型下拉框，选择要添加的模型类型（大语言模型/向量化模型/视觉语言模型/重排模型）。 + - 点击模型类型下拉框，选择要添加的模型类型（大语言模型/向量模型/图片理解模型/图片生成模型/视频理解模型/重排模型/语音识别模型/语音合成模型）。 4. **输入API Key（必填）** - 输入您的API密钥。 5. **获取模型** @@ -93,7 +52,7 @@ Nexent支持与ModelEngine平台的无缝对接 - 配置完成后，点击"确定"按钮，所有选中的模型将被添加到可用模型列表中。
- +
### 🔧 修改自定义模型 @@ -101,24 +60,24 @@ Nexent支持与ModelEngine平台的无缝对接 当您需要修改模型配置或删除不再使用的模型时，可以通过以下步骤进行操作： 1. 点击"修改自定义模型"按钮。 -2. 选择要修改或删除的模型类型（大语言模型/向量化模型/视觉语言模型）。 +2. 选择要修改或删除的模型类型（大语言模型/向量模型/图片理解模型/图片生成模型/视频理解模型/重排模型/语音识别模型/语音合成模型）。 3. 选择是批量修改模型，还是修改单个自定义模型。 4. 如果批量修改模型，可以通过启动或关闭模型开关来添加或删除模型。您也可以通过点击右上角的"修改配置"按钮，对选中的模型进行批量配置修改。 5. 如果是修改单个自定义模型，点击删除按钮 🗑️ 即可删除目标模型；想要修改相关配置，点击模型名称即可弹出修改弹窗进行修改。
- - + +

- - + +

- - + +
### ⚙️ 配置系统模型 @@ -137,6 +96,11 @@ Nexent支持与ModelEngine平台的无缝对接 - 点击基础模型下拉框，从已添加的大语言模型中选择一个作为系统基础模型。 +#### 大语言模型 +大语言模型是系统的核心推理引擎，负责处理用户的自然语言请求、生成回复、执行代码、分析数据等复杂任务。选择合适的大语言模型，可以显著提升智能体的对话质量和任务处理能力。 +- 点击大语言模型下拉框，从已添加的大语言模型中选择一个。 + + #### 向量模型 向量模型主要用于知识库的文本、图片等数据的向量化处理，是实现高效检索和语义理解的基础。配置合适的向量模型，可以显著提升知识库的搜索准确率和多模态数据的处理能力。 @@ -147,7 +111,7 @@ Nexent支持与ModelEngine平台的无缝对接 根据模型能力选择合适的文档切片大小和单次请求切片量。切片越小，系统越稳定，但文件解析质量也会受到影响。
- +
#### 重排模型 @@ -160,22 +124,27 @@ Nexent支持与ModelEngine平台的无缝对接 多模态模型结合了视觉和语言能力，能够处理包含文本、图片等多种信息的复杂场景。例如，在对话页面上传图片文件时，系统会自动调用多模态模型进行内容解析和智能对话。 -- 点击视觉语言模型下拉框，从已添加的视觉语言模型中选择一个。 +- **图片理解模型**：能够分析和理解图片内容，提取关键信息、回答与图片相关的问题。点击图片理解模型下拉框，从已添加的图片理解模型中选择一个。 +- **图片生成模型**：能够根据文本描述生成对应的图像，支持创意设计、内容创作等多种场景。点击图片生成模型下拉框，从已添加的图片生成模型中选择一个。 +- **视频理解模型**：能够分析和理解视频内容，提取关键信息、生成摘要或回答与视频相关的问题。点击视频理解模型下拉框，从已添加的视频理解模型中选择一个。 -
- - - - +#### 语音模型 + +语音模型用于实现语音与文本之间的双向转换，支持语音交互场景。 + +- **语音合成模型**：用于将文本内容即时转换为自然流畅的语音输出，使系统能够以接近真人的方式进行语音交互与反馈。通过低延迟、高拟真度的语音生成能力，确保用户在对话过程中获得连贯、自然的听觉体验。点击语音合成模型下拉框，从已添加的语音合成模型中选择一个。 +- **语音识别模型**：用于将用户输入的语音内容实时转换为文本，实现对语音指令和自然语言的准确理解与解析。通过高精度的语音转写与噪声鲁棒能力，确保在复杂环境下依然能够稳定识别用户意图。点击语音识别模型下拉框，从已添加的语音识别模型中选择一个。 + + +
+ + + + +
-#### 语音合成模型 -语音合成模型用于将文本内容即时转换为自然流畅的语音输出，使系统能够以接近真人的方式进行语音交互与反馈。通过低延迟、高拟真度的语音生成能力，确保用户在对话过程中获得连贯、自然的听觉体验。配置合适的实时语音合成模型，可以显著提升语音交互系统的表现力和用户体验。 -- 点击语音合成模型下拉框，从已添加的视觉语言模型中选择一个。 -#### 语音识别模型 -语音识别模型用于将用户输入的语音内容实时转换为文本，实现对语音指令和自然语言的准确理解与解析。通过高精度的语音转写与噪声鲁棒能力，确保在复杂环境下依然能够稳定识别用户意图。配置合适的语音识别模型，可以显著提升语音交互系统的理解能力和整体响应效率。 -- 点击语音识别模型下拉框，从已添加的视觉语言模型中选择一个。 ### ✅ 检查模型连通性 @@ -220,7 +189,7 @@ Nexent 支持任何 **遵循OpenAI API规范** 的大语言模型供应商，包 3. 在文档中查看API端点（即模型URL，一般以`/v1`为结尾）； 4. 在Nexent模型配置页面点击添加自定义模型，填入必备信息，即可接入。 -#### 🎭 多模态视觉模型 +#### 🎭 多模态模型 使用与大语言模型相同的API Key和模型URL，但指定多模态模型名称，如硅基流动提供的**Qwen/Qwen2.5-VL-32B-Instruct**。 @@ -267,9 +236,9 @@ Nexent 支持任何 **遵循OpenAI API规范** 的大语言模型供应商，包 ## 🚀 下一步 -完成模型管理配置后，建议您继续配置： +完成模型配置后，建议您继续配置： -1. **[知识库](./knowledge-base)** - 创建和管理知识库。 -2. **[智能体开发](./agent-development)** - 创建和配置智能体。 +1. **[知识库](./knowledge-configuration)** - 创建和管理知识库。 +2. **[智能体配置](./agent-configuration)** - 创建和配置智能体。 -如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../quick-start/faq)** 或在[GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions)中进行提问获取支持。 +如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../../quick-start/faq)** 或在[GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions)中进行提问获取支持。 diff --git a/doc/docs/zh/user-guide/agent-market.md b/doc/docs/zh/user-guide/agent-market.md index 47f5b2f5d6..916a8505cc 100644 --- a/doc/docs/zh/user-guide/agent-market.md +++ b/doc/docs/zh/user-guide/agent-market.md @@ -50,7 +50,7 @@ ![智能体市场下载3](./assets/agent-market/agent-market-download3.png) -安装完成后，您的智能体会在 **[智能体空间](./agent-space)** 准备好 +安装完成后，您的智能体会在 **[智能体空间](./agent-development.md)** 准备好 ## 📢 分享您的创作 @@ -62,8 +62,8 @@ 在等待智能体市场上线期间，您可以： -1. 在 **[智能体空间](./agent-space)** 中管理您自己的智能体 -2. 通过 **[智能体开发](./agent-development)** 创建专属智能体 +1. 在 **[智能体空间](./agent-development.md)** 中管理您自己的智能体 +2. 通过 **[智能体开发](./agent-development.md)** 创建专属智能体 3. 在 **[开始问答](./start-chat)** 中体验智能体的强大功能 -如果您使用过程中遇到任何问题，请参考我们的 **[常见问题](../quick-start/faq)** 或在 [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions) 中进行提问获取支持。 +如果您使用过程中遇到任何问题，请参考我们的 **[常见问题](../quick-start/faq.md)** 或在 [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions) 中进行提问获取支持。 diff --git a/doc/docs/zh/user-guide/agent-space.md b/doc/docs/zh/user-guide/agent-space.md deleted file mode 100644 index c6a76df6be..0000000000 --- a/doc/docs/zh/user-guide/agent-space.md +++ /dev/null @@ -1,70 +0,0 @@ -# 智能体空间 - -智能体空间是您管理所有已开发智能体的中心。在这里，您可以卡片形式查看所有智能体及智能体详细配置，进行智能体删除、导出等管理操作。 -![智能体空间](./assets/agent-space/agent-space.png) - -## 📦 智能体卡片展示 - -智能体空间以卡片形式展示所有已开发好的智能体，每个卡片包含： - -- **智能体图标**：智能体的标识图标 -- **智能体名称**：智能体的显示名称 -- **智能体作者**：智能体的作者 -- **智能体描述**：智能体的功能描述 -- **智能体状态**：智能体是否可用的状态 -- **操作按钮**：快速操作入口 - -## 🔧 管理智能体 - -在智能体空间中，您可以对每个智能体进行以下操作： - -### 查看智能体详细信息 - -点击智能体卡片，即可查看智能体详细信息： - -- **基础信息**：智能体ID、名称、描述、状态、最大步数、提供运行摘要等 -- **模型配置**：模型名称、业务逻辑模型名称等 -- **提示词**：包含角色提示词、约束提示词、示例提示词、以及原始业务描述 -- **工具**：配置的工具 -- **子智能体**：配置的子智能体 - -![智能体详细信息](./assets/agent-space/agent-details.png) - -### 编辑智能体 - -1. 点击智能体卡片上的"编辑"按钮 -2. 跳转到智能体开发页面进行修改 -3. 保存后更新会同步到智能体空间 - -### 删除智能体 - -1. 点击智能体卡片上的"删除"按钮 -2. 确认删除操作（此操作不可撤销） -3. 删除后智能体将从列表中移除 - -> ⚠️ **注意事项**：删除智能体是不可撤销的操作，请谨慎操作。 - -### 导出智能体 - -1. 点击智能体卡片上的"导出"按钮 -2. 系统会下载智能体配置文件（JSON格式），可用于后续导入或备份 - -### 查看调用关系 - -1. 点击智能体卡片上的"查看关系"按钮 -2. 查看该智能体与工具/其他智能体的协作关系 - -### 跳转到对话 - -1. 点击智能体卡片上的"对话"按钮 -2. 直接跳转到对话页面，使用该智能体进行交互 - -## 🚀 下一步 - -在智能体空间中完成管理后，您可以： - -1. 在 **[开始问答](./start-chat)** 中与智能体进行交互 -2. 继续 **[智能体开发](./agent-development)** 创建更多智能体 -3. 配置 **[记忆管理](./memory-management)** 以提升智能体的记忆能力 - -如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../quick-start/faq)** 或在[GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions)中进行提问获取支持。 diff --git a/doc/docs/zh/user-guide/assets/agent-development/a2a-find-detail.jpg b/doc/docs/zh/user-guide/assets/agent-development/a2a-find-detail.jpg index ed99126273..2e2cb9016e 100644 Binary files a/doc/docs/zh/user-guide/assets/agent-development/a2a-find-detail.jpg and b/doc/docs/zh/user-guide/assets/agent-development/a2a-find-detail.jpg differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/add_mcp_from_api_2.png b/doc/docs/zh/user-guide/assets/agent-development/add_mcp_from_api_2.png index faba05fece..2a96a2d160 100644 Binary files a/doc/docs/zh/user-guide/assets/agent-development/add_mcp_from_api_2.png and b/doc/docs/zh/user-guide/assets/agent-development/add_mcp_from_api_2.png differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/agent-list.png b/doc/docs/zh/user-guide/assets/agent-development/agent-list.png new file mode 100644 index 0000000000..c763a27858 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/agent-development/agent-list.png differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/agent-settings.png b/doc/docs/zh/user-guide/assets/agent-development/agent-settings.png new file mode 100644 index 0000000000..e3d03b925e Binary files /dev/null and b/doc/docs/zh/user-guide/assets/agent-development/agent-settings.png differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/generate-agent.png b/doc/docs/zh/user-guide/assets/agent-development/generate-agent.png deleted file mode 100644 index b9169dbcdc..0000000000 Binary files a/doc/docs/zh/user-guide/assets/agent-development/generate-agent.png and /dev/null differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/generate_agent.png b/doc/docs/zh/user-guide/assets/agent-development/generate_agent.png new file mode 100644 index 0000000000..4a7fd85e04 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/agent-development/generate_agent.png differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/import-2.png b/doc/docs/zh/user-guide/assets/agent-development/import-2.png new file mode 100644 index 0000000000..d58677fbf8 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/agent-development/import-2.png differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/import.png b/doc/docs/zh/user-guide/assets/agent-development/import.png index 511727d39d..807022e2c1 100644 Binary files a/doc/docs/zh/user-guide/assets/agent-development/import.png and b/doc/docs/zh/user-guide/assets/agent-development/import.png differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/safety-fence.png b/doc/docs/zh/user-guide/assets/agent-development/safety-fence.png new file mode 100644 index 0000000000..9efcea557e Binary files /dev/null and b/doc/docs/zh/user-guide/assets/agent-development/safety-fence.png differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/set-tools-1.png b/doc/docs/zh/user-guide/assets/agent-development/set-tools-1.png new file mode 100644 index 0000000000..fdcb2aad8a Binary files /dev/null and b/doc/docs/zh/user-guide/assets/agent-development/set-tools-1.png differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/set-tools-2.png b/doc/docs/zh/user-guide/assets/agent-development/set-tools-2.png new file mode 100644 index 0000000000..8b0deca1dd Binary files /dev/null and b/doc/docs/zh/user-guide/assets/agent-development/set-tools-2.png differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/version_management_1.png b/doc/docs/zh/user-guide/assets/agent-development/version_management_1.png index a945374c56..3ab4a08e70 100644 Binary files a/doc/docs/zh/user-guide/assets/agent-development/version_management_1.png and b/doc/docs/zh/user-guide/assets/agent-development/version_management_1.png differ diff --git a/doc/docs/zh/user-guide/assets/agent-development/version_management_2.png b/doc/docs/zh/user-guide/assets/agent-development/version_management_2.png index baa7fe7ea1..914aec8464 100644 Binary files a/doc/docs/zh/user-guide/assets/agent-development/version_management_2.png and b/doc/docs/zh/user-guide/assets/agent-development/version_management_2.png differ diff --git a/doc/docs/zh/user-guide/assets/auto-tasks/01-automation-task-list.png b/doc/docs/zh/user-guide/assets/auto-tasks/01-automation-task-list.png new file mode 100644 index 0000000000..1312ffd255 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/auto-tasks/01-automation-task-list.png differ diff --git a/doc/docs/zh/user-guide/assets/auto-tasks/02-create-task-in-chat.png b/doc/docs/zh/user-guide/assets/auto-tasks/02-create-task-in-chat.png new file mode 100644 index 0000000000..8f8549c097 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/auto-tasks/02-create-task-in-chat.png differ diff --git a/doc/docs/zh/user-guide/assets/auto-tasks/03-automation-proposal.png b/doc/docs/zh/user-guide/assets/auto-tasks/03-automation-proposal.png new file mode 100644 index 0000000000..be79fa2aff Binary files /dev/null and b/doc/docs/zh/user-guide/assets/auto-tasks/03-automation-proposal.png differ diff --git a/doc/docs/zh/user-guide/assets/auto-tasks/04-edit-automation-proposal.png b/doc/docs/zh/user-guide/assets/auto-tasks/04-edit-automation-proposal.png new file mode 100644 index 0000000000..97e7002cdf Binary files /dev/null and b/doc/docs/zh/user-guide/assets/auto-tasks/04-edit-automation-proposal.png differ diff --git a/doc/docs/zh/user-guide/assets/auto-tasks/05-created-task-in-list.png b/doc/docs/zh/user-guide/assets/auto-tasks/05-created-task-in-list.png new file mode 100644 index 0000000000..504ac2f61a Binary files /dev/null and b/doc/docs/zh/user-guide/assets/auto-tasks/05-created-task-in-list.png differ diff --git a/doc/docs/zh/user-guide/assets/auto-tasks/06-more-actions.png b/doc/docs/zh/user-guide/assets/auto-tasks/06-more-actions.png new file mode 100644 index 0000000000..2cd700e8a9 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/auto-tasks/06-more-actions.png differ diff --git a/doc/docs/zh/user-guide/assets/auto-tasks/07-run-history.png b/doc/docs/zh/user-guide/assets/auto-tasks/07-run-history.png new file mode 100644 index 0000000000..a1740b1454 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/auto-tasks/07-run-history.png differ diff --git a/doc/docs/zh/user-guide/assets/knowledge-base/chunk_management.png b/doc/docs/zh/user-guide/assets/knowledge-base/chunk_management.png new file mode 100644 index 0000000000..cdb0600732 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/knowledge-base/chunk_management.png differ diff --git a/doc/docs/zh/user-guide/assets/knowledge-base/create-knowledge-base.png b/doc/docs/zh/user-guide/assets/knowledge-base/create-knowledge-base.png index 3731860eee..0f1a879d2c 100644 Binary files a/doc/docs/zh/user-guide/assets/knowledge-base/create-knowledge-base.png and b/doc/docs/zh/user-guide/assets/knowledge-base/create-knowledge-base.png differ diff --git a/doc/docs/zh/user-guide/assets/knowledge-base/knowledge-tool.png b/doc/docs/zh/user-guide/assets/knowledge-base/knowledge-tool.png deleted file mode 100644 index 4359a66f99..0000000000 Binary files a/doc/docs/zh/user-guide/assets/knowledge-base/knowledge-tool.png and /dev/null differ diff --git a/doc/docs/zh/user-guide/assets/knowledge-base/knowledge-tool1.png b/doc/docs/zh/user-guide/assets/knowledge-base/knowledge-tool1.png new file mode 100644 index 0000000000..910b170a30 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/knowledge-base/knowledge-tool1.png differ diff --git a/doc/docs/zh/user-guide/assets/knowledge-base/summary-knowledge-base.png b/doc/docs/zh/user-guide/assets/knowledge-base/summary-knowledge-base.png index 306a1b2954..b8e11724b6 100644 Binary files a/doc/docs/zh/user-guide/assets/knowledge-base/summary-knowledge-base.png and b/doc/docs/zh/user-guide/assets/knowledge-base/summary-knowledge-base.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mcp-admin-tabs.png b/doc/docs/zh/user-guide/assets/mcp-space/mcp-admin-tabs.png new file mode 100644 index 0000000000..983fc76fee Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mcp-admin-tabs.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mcp-developer-tabs.png b/doc/docs/zh/user-guide/assets/mcp-space/mcp-developer-tabs.png new file mode 100644 index 0000000000..0b0f93f9c1 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mcp-developer-tabs.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-delete.png b/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-delete.png new file mode 100644 index 0000000000..faa41b09ee Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-delete.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-detail.png b/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-detail.png new file mode 100644 index 0000000000..ba897025c5 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-detail.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-install.png b/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-install.png new file mode 100644 index 0000000000..7dd1bcc81d Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-install.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-list.png b/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-list.png new file mode 100644 index 0000000000..06b8529349 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mcp-repository-list.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mymcp-addmcp.png b/doc/docs/zh/user-guide/assets/mcp-space/mymcp-addmcp.png new file mode 100644 index 0000000000..f9683853ec Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mymcp-addmcp.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcp-detail.png b/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcp-detail.png new file mode 100644 index 0000000000..ef5f595ed4 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcp-detail.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcp-review.png b/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcp-review.png new file mode 100644 index 0000000000..a5ecb90404 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcp-review.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcpcard.png b/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcpcard.png new file mode 100644 index 0000000000..d9b98a0869 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcpcard.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcpcard2.png b/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcpcard2.png new file mode 100644 index 0000000000..798416574a Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/mymcp-mcpcard2.png differ diff --git a/doc/docs/zh/user-guide/assets/mcp-space/review-center.png b/doc/docs/zh/user-guide/assets/mcp-space/review-center.png new file mode 100644 index 0000000000..476a60dec2 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/mcp-space/review-center.png differ diff --git a/doc/docs/zh/user-guide/assets/memory-management/add-memory.png b/doc/docs/zh/user-guide/assets/memory-management/add-memory.png new file mode 100644 index 0000000000..14d4e101aa Binary files /dev/null and b/doc/docs/zh/user-guide/assets/memory-management/add-memory.png differ diff --git a/doc/docs/zh/user-guide/assets/memory-management/agent-memory.png b/doc/docs/zh/user-guide/assets/memory-management/agent-memory.png new file mode 100644 index 0000000000..249af9b113 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/memory-management/agent-memory.png differ diff --git a/doc/docs/zh/user-guide/assets/memory-management/disabled-memory.png b/doc/docs/zh/user-guide/assets/memory-management/disabled-memory.png new file mode 100644 index 0000000000..e853502123 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/memory-management/disabled-memory.png differ diff --git a/doc/docs/zh/user-guide/assets/memory-management/embedding-missing-warn.png b/doc/docs/zh/user-guide/assets/memory-management/embedding-missing-warn.png new file mode 100644 index 0000000000..f049dcd999 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/memory-management/embedding-missing-warn.png differ diff --git a/doc/docs/zh/user-guide/assets/memory-management/memory-search-tool.png b/doc/docs/zh/user-guide/assets/memory-management/memory-search-tool.png new file mode 100644 index 0000000000..4e02e872d8 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/memory-management/memory-search-tool.png differ diff --git a/doc/docs/zh/user-guide/assets/memory-management/memory-settings.png b/doc/docs/zh/user-guide/assets/memory-management/memory-settings.png new file mode 100644 index 0000000000..a3ed78b46b Binary files /dev/null and b/doc/docs/zh/user-guide/assets/memory-management/memory-settings.png differ diff --git a/doc/docs/zh/user-guide/assets/model-management/edit-model-1.png b/doc/docs/zh/user-guide/assets/model-management/edit-model-1.png index cabd518339..02c3ae16ef 100644 Binary files a/doc/docs/zh/user-guide/assets/model-management/edit-model-1.png and b/doc/docs/zh/user-guide/assets/model-management/edit-model-1.png differ diff --git a/doc/docs/zh/user-guide/assets/model-management/select-model-3.png b/doc/docs/zh/user-guide/assets/model-management/select-model-3.png index 9cda79ae6d..78ed606337 100644 Binary files a/doc/docs/zh/user-guide/assets/model-management/select-model-3.png and b/doc/docs/zh/user-guide/assets/model-management/select-model-3.png differ diff --git a/doc/docs/zh/user-guide/assets/model-management/select-model-4.png b/doc/docs/zh/user-guide/assets/model-management/select-model-4.png index 78ed606337..182f53b9eb 100644 Binary files a/doc/docs/zh/user-guide/assets/model-management/select-model-4.png and b/doc/docs/zh/user-guide/assets/model-management/select-model-4.png differ diff --git a/doc/docs/zh/user-guide/assets/model-management/select-model-5.png b/doc/docs/zh/user-guide/assets/model-management/select-model-5.png new file mode 100644 index 0000000000..330ef9c89b Binary files /dev/null and b/doc/docs/zh/user-guide/assets/model-management/select-model-5.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/admin-tabs.png b/doc/docs/zh/user-guide/assets/resource-repository/admin-tabs.png new file mode 100644 index 0000000000..aaf0e65b71 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/admin-tabs.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/apply-listing.png b/doc/docs/zh/user-guide/assets/resource-repository/apply-listing.png new file mode 100644 index 0000000000..c2a77ec176 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/apply-listing.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/copy-precheck.png b/doc/docs/zh/user-guide/assets/resource-repository/copy-precheck.png new file mode 100644 index 0000000000..95254df494 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/copy-precheck.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/developer-tabs.png b/doc/docs/zh/user-guide/assets/resource-repository/developer-tabs.png new file mode 100644 index 0000000000..e6ad43a973 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/developer-tabs.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/mine-list.png b/doc/docs/zh/user-guide/assets/resource-repository/mine-list.png new file mode 100644 index 0000000000..1fc8c777e0 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/mine-list.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/repository-detail.png b/doc/docs/zh/user-guide/assets/resource-repository/repository-detail.png new file mode 100644 index 0000000000..31f1175178 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/repository-detail.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/repository-list.png b/doc/docs/zh/user-guide/assets/resource-repository/repository-list.png new file mode 100644 index 0000000000..5e3f51c39d Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/repository-list.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/review-confirm.png b/doc/docs/zh/user-guide/assets/resource-repository/review-confirm.png new file mode 100644 index 0000000000..59006f78e2 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/review-confirm.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/review-status.png b/doc/docs/zh/user-guide/assets/resource-repository/review-status.png new file mode 100644 index 0000000000..65092800c9 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/review-status.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_admin.png b/doc/docs/zh/user-guide/assets/resource-repository/skill_admin.png new file mode 100644 index 0000000000..49f9245e67 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_admin.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_approve.png b/doc/docs/zh/user-guide/assets/resource-repository/skill_approve.png new file mode 100644 index 0000000000..a3a73ccc11 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_approve.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_create.png b/doc/docs/zh/user-guide/assets/resource-repository/skill_create.png new file mode 100644 index 0000000000..b92acb99ca Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_create.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_dev.png b/doc/docs/zh/user-guide/assets/resource-repository/skill_dev.png new file mode 100644 index 0000000000..554a7c1202 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_dev.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_list.png b/doc/docs/zh/user-guide/assets/resource-repository/skill_list.png new file mode 100644 index 0000000000..f7b117dde9 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_list.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_mine.png b/doc/docs/zh/user-guide/assets/resource-repository/skill_mine.png new file mode 100644 index 0000000000..0af5dff648 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_mine.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_repo_detail.png b/doc/docs/zh/user-guide/assets/resource-repository/skill_repo_detail.png new file mode 100644 index 0000000000..338e55758d Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_repo_detail.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_repo_search.png b/doc/docs/zh/user-guide/assets/resource-repository/skill_repo_search.png new file mode 100644 index 0000000000..3d82ff5bab Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_repo_search.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_select.png b/doc/docs/zh/user-guide/assets/resource-repository/skill_select.png new file mode 100644 index 0000000000..fb1b16f44a Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_select.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_under_review.png b/doc/docs/zh/user-guide/assets/resource-repository/skill_under_review.png new file mode 100644 index 0000000000..44333064db Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_under_review.png differ diff --git a/doc/docs/zh/user-guide/assets/resource-repository/skill_using.jpg b/doc/docs/zh/user-guide/assets/resource-repository/skill_using.jpg new file mode 100644 index 0000000000..860ebe68e5 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/resource-repository/skill_using.jpg differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/ReAct.png b/doc/docs/zh/user-guide/assets/start-chat/ReAct.png new file mode 100644 index 0000000000..4721696a43 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/ReAct.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/agent-list.png b/doc/docs/zh/user-guide/assets/start-chat/agent-list.png new file mode 100644 index 0000000000..4ce39c6d0b Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/agent-list.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/agent-selection.png b/doc/docs/zh/user-guide/assets/start-chat/agent-selection.png deleted file mode 100644 index f22decf624..0000000000 Binary files a/doc/docs/zh/user-guide/assets/start-chat/agent-selection.png and /dev/null differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/agent-welcome.png b/doc/docs/zh/user-guide/assets/start-chat/agent-welcome.png new file mode 100644 index 0000000000..35ee6e37fd Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/agent-welcome.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/analyze_image.png b/doc/docs/zh/user-guide/assets/start-chat/analyze_image.png new file mode 100644 index 0000000000..8b0d9f0e6e Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/analyze_image.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/analyze_text_file.png b/doc/docs/zh/user-guide/assets/start-chat/analyze_text_file.png new file mode 100644 index 0000000000..007d8baa69 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/analyze_text_file.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/chat-management-1.png b/doc/docs/zh/user-guide/assets/start-chat/chat-management-1.png deleted file mode 100644 index 4035fc1827..0000000000 Binary files a/doc/docs/zh/user-guide/assets/start-chat/chat-management-1.png and /dev/null differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/chat-management-2.png b/doc/docs/zh/user-guide/assets/start-chat/chat-management-2.png deleted file mode 100644 index 721241a496..0000000000 Binary files a/doc/docs/zh/user-guide/assets/start-chat/chat-management-2.png and /dev/null differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/collapse.png b/doc/docs/zh/user-guide/assets/start-chat/collapse.png new file mode 100644 index 0000000000..53ae9780d4 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/collapse.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/conversation-manage.png b/doc/docs/zh/user-guide/assets/start-chat/conversation-manage.png new file mode 100644 index 0000000000..cc3b68dbe7 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/conversation-manage.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/create-docx.png b/doc/docs/zh/user-guide/assets/start-chat/create-docx.png new file mode 100644 index 0000000000..3cab1114d6 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/create-docx.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/dialog-box.png b/doc/docs/zh/user-guide/assets/start-chat/dialog-box.png deleted file mode 100644 index 7cec5d09f9..0000000000 Binary files a/doc/docs/zh/user-guide/assets/start-chat/dialog-box.png and /dev/null differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/example-question.png b/doc/docs/zh/user-guide/assets/start-chat/example-question.png new file mode 100644 index 0000000000..0813dda525 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/example-question.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/finish.png b/doc/docs/zh/user-guide/assets/start-chat/finish.png new file mode 100644 index 0000000000..e08ac354e9 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/finish.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/memory.png b/doc/docs/zh/user-guide/assets/start-chat/memory.png new file mode 100644 index 0000000000..40e4cfcabf Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/memory.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/mermaid.png b/doc/docs/zh/user-guide/assets/start-chat/mermaid.png new file mode 100644 index 0000000000..2c72b4e9ef Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/mermaid.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/parallel-subagents.png b/doc/docs/zh/user-guide/assets/start-chat/parallel-subagents.png new file mode 100644 index 0000000000..6c3dc37924 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/parallel-subagents.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/parallel-tool-calls.png b/doc/docs/zh/user-guide/assets/start-chat/parallel-tool-calls.png new file mode 100644 index 0000000000..dbcd455cea Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/parallel-tool-calls.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/plan.png b/doc/docs/zh/user-guide/assets/start-chat/plan.png new file mode 100644 index 0000000000..ebfed0aa35 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/plan.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/preview-docx.png b/doc/docs/zh/user-guide/assets/start-chat/preview-docx.png new file mode 100644 index 0000000000..2f3815ae51 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/preview-docx.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/reference-image.png b/doc/docs/zh/user-guide/assets/start-chat/reference-image.png deleted file mode 100644 index a0bfea2714..0000000000 Binary files a/doc/docs/zh/user-guide/assets/start-chat/reference-image.png and /dev/null differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/reference-source.png b/doc/docs/zh/user-guide/assets/start-chat/reference-source.png deleted file mode 100644 index d8bbffeea4..0000000000 Binary files a/doc/docs/zh/user-guide/assets/start-chat/reference-source.png and /dev/null differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/refresh-chat.png b/doc/docs/zh/user-guide/assets/start-chat/refresh-chat.png new file mode 100644 index 0000000000..95a18ecf79 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/refresh-chat.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/self-correction.png b/doc/docs/zh/user-guide/assets/start-chat/self-correction.png new file mode 100644 index 0000000000..44fed02881 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/self-correction.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/share.png b/doc/docs/zh/user-guide/assets/start-chat/share.png new file mode 100644 index 0000000000..f3a2589762 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/share.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/source.png b/doc/docs/zh/user-guide/assets/start-chat/source.png new file mode 100644 index 0000000000..de3d855671 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/source.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/tool-call.png b/doc/docs/zh/user-guide/assets/start-chat/tool-call.png new file mode 100644 index 0000000000..78d5cc9f07 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/tool-call.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/upload_file.png b/doc/docs/zh/user-guide/assets/start-chat/upload_file.png new file mode 100644 index 0000000000..8d6a896963 Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/upload_file.png differ diff --git a/doc/docs/zh/user-guide/assets/start-chat/verification.png b/doc/docs/zh/user-guide/assets/start-chat/verification.png new file mode 100644 index 0000000000..6deb87587a Binary files /dev/null and b/doc/docs/zh/user-guide/assets/start-chat/verification.png differ diff --git a/doc/docs/zh/user-guide/auto-tasks.md b/doc/docs/zh/user-guide/auto-tasks.md new file mode 100644 index 0000000000..7b89cf7f5b --- /dev/null +++ b/doc/docs/zh/user-guide/auto-tasks.md @@ -0,0 +1,261 @@ +# 自动任务 + +自动任务让智能体在未来指定时间或按照固定周期自动执行工作。您只需在会话中用自然语言说明“做什么”和“什么时候做”，系统会生成一张待确认的任务提案；确认后，任务会持续绑定当前会话，并把每次执行结果写回该会话。 + +例如，您可以让智能体： + +- 每天上午 9 点汇总项目进展； +- 每隔 30 分钟检查一次服务状态； +- 明天下午 3 点生成一次周报。 + +> **重要说明**：自动任务提案只用于创建计划，不会立即执行您描述的业务动作。任务将在您确认创建后，按照设定的时间运行。 + +## 使用前准备 + +创建任务前，请确认： + +1. 已在 [模型配置](./agent-development/model-configuration) 中配置可用的语言模型； +2. 已在 [智能体开发](./agent-development) 中创建并保存可用于问答的智能体； +3. 智能体已经配置任务需要的工具、知识库、Skill、记忆或其他智能体； +4. 您可以访问用于创建任务的会话。 + +如果任务依赖的能力不完整，提案会提示您先配置智能体，暂时不能创建任务。 + +## 创建自动任务 + +### 1. 进入创建入口 + +在左侧导航栏打开 **自动任务**，然后点击页面右上角的 **通过会话创建**。系统会进入新的会话页面。 + +您也可以直接打开 [开始问答](./start-chat)，选择智能体后提出自动执行请求。 + +
+ 自动任务入口和任务列表 +
+ +### 2. 选择智能体并描述任务 + +选择负责执行任务的智能体，然后在输入框中同时写明： + +- **业务动作**：每次触发时具体要完成什么； +- **执行时间**：一次性任务需要明确的未来日期和时间； +- **执行周期**：周期任务需要说明固定间隔或日历周期； +- **时区**：如需使用非当前默认时区，请明确写出，例如 `Asia/Shanghai` 或 `UTC`； +- **结束条件**：如有截止时间或执行次数限制，请在请求中明确说明。 + +推荐写法： + +```text +每天上午 9 点，汇总昨天的项目进展，输出简短摘要并列出需要关注的问题。 +``` + +一次性任务示例： + +```text +明天下午 3 点生成一份本周项目周报。 +``` + +固定间隔任务示例： + +```text +每隔 30 分钟检查一次服务状态，并给出异常项。 +``` + +如果请求缺少业务动作、日期、时间或周期，智能体会提示您补充最关键的信息。立即执行的普通请求、询问某个时间点的数据或仅解释时间表达式，不会被识别为自动任务。 + +
+ 在会话中描述自动任务 +
+ +### 3. 检查任务提案 + +系统识别到自动执行意图后，会在会话中生成任务提案。请重点检查： + +- **任务标题**：在自动任务列表中显示的名称； +- **任务内容**：智能体每次触发时实际执行的单次指令； +- **智能体**：负责执行任务的智能体； +- **执行计划**：单次或重复执行、开始时间、时区及重复规则； +- **能力状态**：当前智能体是否具备完成任务所需的能力。 + +请注意，系统会从任务内容中去除“每天上午 9 点”等调度描述，并把它们保存为独立的执行计划，这是正常行为。 + +
+ 自动任务提案卡片 +
+ +### 4. 修改提案 + +如需调整提案，点击卡片右上角的 **修改**。当前编辑窗口支持修改： + +- 任务标题； +- 任务内容； +- 执行方式：单次执行或重复执行； +- 开始时间； +- 时区； +- 重复规则：Cron 表达式或固定间隔。 + +固定间隔以秒为单位，页面和后端会按照系统配置校验最小间隔。Cron 使用标准五字段格式： + +```text +分钟 小时 日 月 星期 +``` + +常用示例： + +| 执行要求 | Cron 表达式 | +| --- | --- | +| 每天 9:00 | `0 9 * * *` | +| 每个工作日 18:30 | `30 18 * * 1-5` | +| 每小时整点 | `0 * * * *` | +| 每月 1 日 9:00 | `0 9 1 * *` | + +Cron 表达式会按照提案中的时区计算。一次性任务的执行时间必须晚于当前时间。 + +
+ 修改自动任务提案 +
+ +### 5. 确认创建 + +确认信息无误后，点击 **创建任务**。创建成功后，提案卡片会显示任务编号。返回 **自动任务** 页面后，新任务会出现在任务列表中；点击任务名称可以打开它所绑定的会话。 + +如果卡片提示能力不足，请点击 **配置智能体**，补充所需能力后回到会话重新创建。一个会话只能绑定一个有效的自动任务；如果当前会话已绑定任务，请新建会话再创建另一个任务。 + +
+ 创建成功的自动任务显示在任务列表中 +
+ +## 管理自动任务 + +打开左侧导航栏中的 **自动任务**，可以查看当前用户创建的任务。列表展示： + +- 任务名称和绑定会话； +- 执行智能体； +- 当前状态； +- 单次或周期计划； +- 下次执行时间； +- 最近一次运行结果。 + +任务名称可直接打开绑定会话。列表支持按任务名称、智能体名称和状态筛选，并支持分页和刷新。 + +### 立即运行 + +点击操作列中的 **立刻运行** 按钮，可以不等待计划时间，手动启动一次任务。手动运行成功不会改变周期任务原定的下一次执行时间。 + +同一绑定会话不能同时运行多个任务。如果会话中已有智能体任务或自动任务正在运行，新触发的运行会被跳过，并在运行历史中记录为 **已跳过**。 + +### 暂停和恢复 + +- 点击 **暂停** 后，任务不再按计划触发； +- 点击 **恢复** 后，系统会根据当前时间和原计划重新计算下一次执行时间； +- 周期任务连续失败或超时达到 5 次时，会变为 **系统暂停**。修复智能体配置或任务内容后，可手动恢复； +- 已完成的一次性任务没有未来执行时间，不能恢复。如需再次执行，请创建新任务或点击 **立刻运行**。 + +### 使用更多操作菜单 + +任务的 **更多操作** 菜单集中提供 **执行历史**、**编辑** 和 **删除** 入口。 + +
+ 自动任务更多操作菜单 +
+ +### 编辑任务 + +点击 **更多操作** > **编辑**，可以修改： + +- 任务名称； +- 执行指令； +- 任务类型和首次执行时间； +- 周期规则、固定间隔或 Cron 表达式； +- 单次运行超时时间。 + +执行智能体不可在编辑窗口中切换。如需更换智能体，请通过新的会话重新创建任务。 + +超时时间以秒为单位，默认值为 1800 秒（30 分钟），当前页面允许填写的最小值为 60 秒。运行超时后，记录状态会变为 **已超时**。 + +### 查看运行历史 + +点击 **更多操作** > **执行历史**，可以查看： + +- 运行状态； +- 触发方式：手动或计划触发； +- 计划执行时间； +- 错误日志； +- 可执行的运行操作。 + +处于 **排队中** 或 **运行中** 的记录可以取消。已经结束的记录可以删除；删除运行记录不会删除自动任务。运行记录删除后无法恢复。 + +
+ 自动任务运行历史 +
+ +### 删除任务 + +点击 **更多操作** > **删除** 并确认后，任务将停止后续自动执行。绑定会话和历史消息会保留；如果任务正在运行，系统会同时请求停止当前运行。 + +反过来，删除绑定会话也会删除对应的自动任务并取消活动运行，请谨慎操作。 + +## 状态说明 + +### 任务状态 + +| 状态 | 说明 | +| --- | --- | +| 已启用 | 任务已创建，正在等待下一次计划执行 | +| 运行中 | 当前有一次运行正在执行 | +| 已暂停 | 用户主动暂停了任务 | +| 系统暂停 | 任务因连续失败、超时或计划异常被系统暂停 | +| 已完成 | 一次性任务已经结束，或周期任务已达到结束条件 | + +### 运行状态 + +| 状态 | 说明 | +| --- | --- | +| 排队中 | 运行已创建，正在等待执行 | +| 运行中 | 智能体正在执行任务 | +| 成功 | 本次运行正常完成 | +| 失败 | 本次运行因能力、配置或执行错误而失败 | +| 已跳过 | 绑定会话已有其他运行，系统未并行启动本次任务 | +| 已取消 | 用户取消了本次运行 | +| 已超时 | 运行超过任务设置的超时时间 | + +## 使用限制和注意事项 + +- **一个会话一个任务**：同一个会话只能绑定一个有效自动任务；多个任务请分别使用不同会话创建。 +- **临时附件不可长期使用**：当前版本不能把创建提案时上传的临时附件作为自动任务的长期输入。请在指令中描述稳定的数据来源，或让智能体使用已配置的知识库、工具等能力。 +- **依赖会在运行前检查**：如果任务依赖的工具、知识库、Skill、记忆或其他智能体被删除或不可用，本次运行会失败，请在运行历史中查看错误日志。 +- **结果写回绑定会话**：每次执行的指令和智能体输出会保存到绑定会话，可从任务名称进入会话查看完整上下文。 +- **错过的周期不补跑**：服务重启期间错过的周期执行会被跳过，系统恢复后会计算下一次未来执行时间。 +- **仅创建者可见**：在正常多用户模式下，任务列表和运行历史按当前租户和创建用户隔离。 + +## 常见问题 + +### 为什么没有生成任务提案？ + +请确认消息同时包含明确的业务动作和未来时间或执行周期。像“帮我定期关注一下”缺少具体动作和周期，系统会要求补充信息；立即执行的普通请求不会生成自动任务。 + +### 为什么使用附件时不能创建任务？ + +临时附件不能保证在未来每次运行时仍然适合作为输入。请把资料放入知识库或其他稳定数据源，配置到智能体后，再在任务指令中说明需要处理的内容。 + +### 为什么任务无法创建？ + +常见原因包括：执行时间已经过去、时间或周期信息不完整、Cron 表达式无效、固定间隔低于系统限制、智能体缺少必要能力，或当前会话已经绑定其他任务。 + +### 为什么本次运行显示“已跳过”？ + +绑定会话中已有智能体任务或自动任务正在执行。为避免同一会话的上下文被并发写入，系统不会同时启动另一次运行。 + +### 为什么任务变成“系统暂停”？ + +周期任务连续失败或超时达到 5 次，或者恢复时发现计划配置无效。请查看运行历史和最近错误，修复智能体能力、模型、工具或执行指令后再恢复任务。 + +### 删除任务会删除会话吗？ + +不会。删除自动任务会停止后续执行，但会保留绑定会话。请注意，删除绑定会话会同时删除对应的自动任务。 + +## 下一步 + +- [开始问答](./start-chat)：选择智能体并通过自然语言创建自动任务。 +- [智能体开发](./agent-development)：为任务配置模型、工具、知识库、Skill 和记忆。 +- [模型配置](./agent-development/model-configuration)：检查任务使用的语言模型。 diff --git a/doc/docs/zh/user-guide/home-page.md b/doc/docs/zh/user-guide/home-page.md index 0a3a82957b..39ec8a2797 100644 --- a/doc/docs/zh/user-guide/home-page.md +++ b/doc/docs/zh/user-guide/home-page.md @@ -47,11 +47,11 @@ Nexent首页展示了平台的核心功能，为您提供快速入口： 建议按照以下顺序完成配置，也可以直接点击“快速配置”按钮： -1️⃣ **[模型管理](./model-management)**，配置应用信息并接入模型 +1️⃣ **[模型管理](./agent-development/model-configuration.md)**，配置应用信息并接入模型 -2️⃣ **[知识库](./knowledge-base)**，上传您的文档和资料 +2️⃣ **[知识库](./agent-development/knowledge-configuration.md)**，上传您的文档和资料 -3️⃣ **[智能体开发](./agent-development)**，创建您的专属智能体 +3️⃣ **[智能体开发](./agent-development.md)**，创建您的专属智能体 4️⃣ **[开始问答](./start-chat)** 立即与智能体互动，体验成果 @@ -60,7 +60,7 @@ Nexent首页展示了平台的核心功能，为您提供快速入口： 遇到问题时，您可以： -- 查看 **[常见问题](../quick-start/faq)** +- 查看 **[常见问题](../quick-start/faq.md)** - 在 [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions) 中提问 💡 保持您的 Nexent 处于最新版本，我们会修复已知问题 \ No newline at end of file diff --git a/doc/docs/zh/user-guide/knowledge-base.md b/doc/docs/zh/user-guide/knowledge-base.md deleted file mode 100644 index b0ebb53f58..0000000000 --- a/doc/docs/zh/user-guide/knowledge-base.md +++ /dev/null @@ -1,89 +0,0 @@ -# 知识库 - -在知识库模块，您可以创建和管理知识库，上传各种格式的文件，并生成内容总结。知识库是智能体的重要信息来源，让智能体能够访问您的私有数据和文档。 - -## 🔧 创建知识库 - -1. 点击"创建知识库"按钮 -2. 为知识库设置一个易于识别的名称 - -## 📁 上传文件 - -### 上传文件 - -1. 在知识库列表中选择要上传文件的知识库 -2. 点击文件上传区域，选择要上传的文件（支持多选），或直接拖拽文件到上传区域 -3. 系统会自动处理上传的文件，提取文本内容并进行向量化 -4. 可在列表中查看文件的处理状态（解析中/入库中/已就绪） - -![文件上传](./assets/knowledge-base/create-knowledge-base.png) - -💡 光标移动至状态，以了解进度及报错原因 - -![文件上传](./assets/knowledge-base/tip.png) - -### 支持的文件格式 - -Nexent支持多种文件格式，包括： - -- **文本**: .txt, .md, .json文件 -- **PDF**: .pdf文件 -- **Word**: .docx文件 -- **PowerPoint**: .pptx文件 -- **Excel**: .xlsx文件 -- **EPUB** .epub文件 -- **数据文件**: .csv文件 -- **Web content**: .html, .xml文件 - -## 📊 知识库总结 - -建议您为每个知识库配置准确且完整的总结描述，这有助于后续智能体在进行检索时，准确选择合适的知识库。 - -1. 点击“详细内容”按钮进入知识库详细内容查看界面 -2. 选择合适的模型，点击“自动总结”按钮为知识库自动生成内容总结 -3. 您可对生成的内容总结进行编辑修改，使其更准确 -4. 最后记得点击“保存”将您的修改保存 - -![内容总结](./assets/knowledge-base/summary-knowledge-base.png) - -## 🔧 使用知识库 - -Nexent支持知识库与智能体单独绑定，在创建智能体时，**启用knowledge_base_search工具**，并选择关联的知识库 -工具1 -![工具2](./assets/knowledge-base/knowledge-tool2.png) - -## 🔍 知识库管理 - -### 查看知识库 - -1. **知识库列表** - - 知识库页面左侧展示了所有已创建的知识库 - - 知识库列表处支持对知识库来源和向量模型的筛选 - - 显示知识库名称、文件数量、创建时间、用户组等信息 - -> 点击编辑，可管理知识库的名称、可见的用户组及组内权限 - -知识库权限 - -2. **知识库详情** - - 点击知识库名称，可查看知识库中全部文档信息 - - 点击“详细内容”，可查看知识库的内容总结 - -### 编辑知识库 - -1. **删除知识库** - - 点击知识库名称右侧“删除”按钮 - - 确认删除操作（此操作不可恢复） - -2. **删除或新增文件** - - 点击知识库名称，在文件列表中点击“删除”按钮，可从知识库中删除文件 - - 点击知识库名称，在文件列表下方文件上传区域，可新增文件到知识库中 - -## 🚀 下一步 - -完成知识库配置后，建议您继续配置： - -1. **[智能体开发](./agent-development)** - 创建和配置智能体 -2. **[开始问答](./start-chat)** - 与智能体进行交互 - -如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../quick-start/faq)** 或在[GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions)中进行提问获取支持。 \ No newline at end of file diff --git a/doc/docs/zh/user-guide/local-tools/index.md b/doc/docs/zh/user-guide/local-tools/index.md index 71ba3e9501..68de64d56b 100644 --- a/doc/docs/zh/user-guide/local-tools/index.md +++ b/doc/docs/zh/user-guide/local-tools/index.md @@ -6,11 +6,11 @@ - [文件工具](./file-tools)：创建/读取/移动/删除文件与目录，树形列目录。 - [邮件工具](./email-tools)：收取 IMAP 邮件，发送 HTML 邮件（支持抄送/密送）。 -- [搜索工具](./search-tools)：本地/DataMate/Dify 知识库检索与 Exa/Tavily/Linkup 公网搜索。 +- [搜索工具](./search-tools)：本地/AIDP/DataMate/Dify 知识库检索与 Exa/Tavily/Linkup 公网搜索。 - [多模态工具](./multimodal-tools)：文本文件与图片的下载、解析、模型分析。 - [终端工具](./terminal-tool)：持久化 SSH 会话，远程执行命令。 - [SQL 工具](./sql-tools)：连接 MySQL、PostgreSQL、SQL Server 执行 SQL 查询。 -- [技能（Skills）](../skills)：Nexent内置工具组合或自定义能力包，支持 NL 生成与版本管理。 +- [技能（Skills）](../resource-repository/skill-repository.md)：Nexent内置工具组合或自定义能力包，支持 NL 生成与版本管理。 ## ⚙️ 配置入口 diff --git a/doc/docs/zh/user-guide/local-tools/search-tools.md b/doc/docs/zh/user-guide/local-tools/search-tools.md index 9c0ded771d..5cf9982dc1 100644 --- a/doc/docs/zh/user-guide/local-tools/search-tools.md +++ b/doc/docs/zh/user-guide/local-tools/search-tools.md @@ -4,12 +4,13 @@ title: 搜索工具 # 搜索工具 -搜索工具组提供多源信息检索，覆盖互联网搜索、本地知识库、DataMate 知识库以及 Dify 知识库。适合实时信息查询、行业资料检索、私有文档查找等场景。 +搜索工具组提供多源信息检索，覆盖互联网搜索、本地知识库、AIDP 知识库、DataMate 知识库以及 Dify 知识库。适合实时信息查询、行业资料检索、私有文档查找以及企业级多模态知识库检索等场景。 ## 🧭 工具清单 - 本地/私有知识库： - `knowledge_base_search`：本地知识库检索，支持多知识库与多种检索模式 + - `aidp_search`：对接 AIDP 企业级知识库，支持文本/表格/图片多模态 FusionSearch 检索 - `datamate_search`：对接 DataMate 知识库的检索 - `dify_search`：对接 Dify 知识库的检索 - 公网搜索： @@ -20,6 +21,7 @@ title: 搜索工具 ## 🧰 使用场景示例 - 查询内部文档、技术规范、行业资料（知识库、DataMate、Dify） +- 检索企业 AIDP 知识库中的多模态资料，包括文档、表格、图片、技术图纸等（AIDP） - 获取最新新闻、数据或网页截图线索（Exa / Tavily / Linkup） - 同时返回图片参考以丰富答案（开启图片过滤后可输出图片列表） @@ -66,6 +68,27 @@ title: 搜索工具 - `rerank_model`：重排序使用的模型，默认为系统配置的 rerank 模型。`enable_rerank` 为 True 时生效。 - 返回匹配片段的标题、内容、得分等。 +### aidp_search +- **配置参数**： + - `server_url`：AIDP API 服务地址，例如 `https://141.111.61.70:30080`。 + - `api_key`：AIDP API 密钥，通常以 `ak_` 开头，由 AIDP 平台管理员签发。 + - `tenant_id`：AIDP API 路径中的租户标识，例如 `aidp`。 + - `kds_list`：要对接的知识库 ID（`kds_id`）列表，以 JSON 字符串数组形式保存（如 `["aidp-kb-01", "aidp-kb-02"]`），决定该工具默认检索哪些 AIDP 知识库。 + - `search_method`：搜索方法，选项：`hybrid_search`（默认，融合检索）、`vector_search`（向量检索）、`full_text_search`（全文检索）。 + - `reranking_enable`：是否启用重排序，默认 True。 + - `reranking_mode`：重排序模式，选项：`performance`（默认）/ `high_accuracy`。 + - `rewrite_enable`：是否启用黑话/查询改写，默认 False。 + - `related_search_enable`：是否启用关联 Chunk 检索，默认 False。 + - `score_threshold`：相似度阈值（0–1），默认 0.0。 + - `top_k`：返回结果数量（1–100），默认 10。 + - `multi_modal`：是否返回多模态 Chunk（图片/表格），默认 True。 +- **检索参数**： + - `query`：检索问题，必填。 + - `kds_list`：可选。指定要检索的知识库 ID 列表，不传则使用工具配置中的默认 `kds_list`。 +- 返回文本、表格、图片等多模态检索块，结果以双通道输出：所有块通过 `SEARCH_CONTENT` 发送，图片另通过 `PICTURE_WEB` 发送。 +- 检索范围会按当前对话用户的 AIDP 权限白名单过滤：无论使用配置默认值还是 LLM 传入的值，都会与用户有权限访问的 KB 取交集，未授权的 KB 会被静默剔除。 +- 若过滤后无可访问的知识库，工具会返回明确的无权限提示，而非静默返回空结果。 + ### exa_search / tavily_search / linkup_search - **配置参数**： - `exa/tavily/linkup_api_key`：对应服务的 API 密钥 @@ -82,14 +105,17 @@ title: 搜索工具 ## 🛠️ 操作指引 -1. **选择数据源**：私有资料用 `knowledge_base_search`、`datamate_search` 或 `dify_search`；实时公开信息用 Exa/Tavily/Linkup。 -2. **设置检索模式/数量**：知识库可在 `search_mode` 之间切换；公网搜索可调整 `max_results` 与是否启用图片过滤。 -3. **限定范围**：需要特定知识库时填写 `index_names`，避免无关结果；DataMate 可通过阈值与 top_k 控制结果精度与数量。 -4. **启用重排序（可选）**：如需提升检索结果相关性，可设置 `enable_rerank: true`，并通过 `rerank_top_n` 和 `rerank_model` 调整重排序效果。 -5. **结果利用**：返回为 JSON，可直接用于回答、摘要或后续引用；包含 cite 索引便于引用管理。 +1. **选择数据源**：私有资料用 `knowledge_base_search`、`aidp_search`、`datamate_search` 或 `dify_search`；实时公开信息用 Exa/Tavily/Linkup。 +2. **设置检索模式/数量**：知识库可在 `search_mode`/`search_method` 之间切换；公网搜索可调整 `max_results` 与是否启用图片过滤。 +3. **填写连接与鉴权参数**：AIDP 需要准确填写 `server_url`、`api_key` 与 `tenant_id`，建议先在平台安全配置中完成测试连接。 +4. **配置可检索知识库范围**：AIDP 在工具配置中通过 `kds_list` 勾选该工具默认检索的知识库；实际调用时还会按当前对话用户的 AIDP 权限白名单再过滤一次。 +5. **限定范围**：需要特定知识库时填写 `index_names`（本地知识库）或显式传入 `kds_list`（AIDP），避免无关结果；DataMate 可通过阈值与 top_k 控制结果精度与数量。 +6. **启用重排序（可选）**：如需提升检索结果相关性，可设置 `enable_rerank: true` 或 `reranking_enable: true`，并通过对应的 model/mode 参数调整效果。 +7. **结果利用**：返回为 JSON，可直接用于回答、摘要或后续引用；包含 cite 索引便于引用管理。 ## 🛡️ 安全与最佳实践 -- 公网搜索需确保 API Key 已在平台安全配置中设置，不要在对话中暴露。 +- 公网搜索与 AIDP 等外部服务的凭证（`api_key` 等）需确保已在平台安全配置中设置，不要在对话中暴露。 +- AIDP 知识库检索受当前对话用户的权限控制。如用户看不到某个 KB，即使该 KB 在工具的 `kds_list` 配置中，也不会被检索到，请联系 AIDP 管理员授予相应权限。 - 知识库检索前确认已同步最新文档，避免旧版本内容。 - 当查询过于宽泛导致无结果时，可缩短或拆分问题；图片过滤未命中时可尝试关闭过滤获取原始图片列表。 diff --git a/doc/docs/zh/user-guide/mcp-tools.md b/doc/docs/zh/user-guide/mcp-tools.md deleted file mode 100755 index 94bf7c6568..0000000000 --- a/doc/docs/zh/user-guide/mcp-tools.md +++ /dev/null @@ -1,158 +0,0 @@ -# MCP 工具 - -在 MCP 工具模块中，您可以集中管理所有 MCP（Model Context Protocol）服务器与工具，支持自定义添加、注册表导入和社区导入等多种接入方式，完成连接配置、工具同步、健康监控以及社区共享。 - -MCP 工具页面包含两个并列页签： - -- **导入的服务**：管理当前租户已接入的 MCP 服务，在此配置、监控和维护您的 MCP 服务。 -- **发布的服务**：管理当前租户发布到社区的 MCP 服务，支持浏览、编辑和取消发布。 - ---- - -## ➕ 添加 MCP 服务 - -点击页面上的"添加 MCP 服务"按钮，打开添加弹窗。弹窗提供三个页签，对应不同的接入来源。 - -### 自定义添加 - -"自定义添加"页签支持手动配置 MCP 服务，分为两种传输类型。 - -#### 通过 URL 添加 - -适用于已有独立部署的 MCP 服务（支持 HTTP / SSE 协议），通过输入端点 URL 直接接入。 - -1. 在"本地添加"页签中，**传输类型**选择"URL" -2. 填写服务信息： - - **服务名称（必填）**：为 MCP 服务设置一个易于识别的名称 - - **服务 URL（必填）**：输入 MCP 服务的端点地址 - - **描述**：可选，填写服务的用途说明 - - **Authorization Token**：可选，若服务需要认证，在此填入 Bearer Token -3. 点击"确定"完成添加，系统会自动连接服务并获取可用工具列表 - -#### 通过容器配置添加 - -适用于需要本地容器化运行的 MCP 服务（如通过 npx 启动的服务），系统会根据您提供的 JSON 配置自动创建并管理容器。 - -1. 在"本地添加"页签中，**传输类型**选择"容器" -2. 填写容器配置信息： - - **服务名称（必填）**：为 MCP 服务设置一个易于识别的名称 - - **描述**：可选，填写服务的用途说明 - - **容器配置 JSON（必填）**：按标准 MCP 配置格式填写，例如： - ```json - { - "mcpServers": { - "service-name": { - "args": ["mcp-package-name@version"], - "command": "npx", - "env": { - "API_KEY": "xxxx" - } - } - } - } - ``` - - **端口号**：填写容器服务暴露的端口，系统会自动检测端口冲突并提示可用端口 -3. 点击"确定"，系统将解析 JSON 配置、创建容器并完成服务注册 - -### 从 MCP Registry 导入 - -Nexent 集成了 MCP Registry，您可以浏览并一键导入社区维护的 MCP 服务。 - -1. 切换到"外部市场"页签 -2. 浏览可用的 MCP 服务列表，支持按名称或标签搜索 -3. 点击目标服务，查看服务详情（描述、版本、所需参数等） -4. 配置必填参数（如 API Key 等环境变量） -5. 点击"导入"，系统会自动安装并配置该 MCP 服务 - -### 从社区导入 - -浏览其他用户在 Nexent 平台内发布的 MCP 服务，快速导入使用。 - -1. 切换到"社区市场"页签 -2. 浏览社区已发布的 MCP 服务，支持按名称、标签或传输协议筛选 -3. 点击目标服务查看详情，点击"导入"即可添加到您的服务列表中 - ---- - -## 📋 导入的服务 - -"导入的服务"页签以卡片形式展示当前租户所有已接入的 MCP 服务，您可以在此查看、编辑、监控和发布。 - -### 查看与筛选 - -每张服务卡片展示以下信息： - -- 服务名称与描述 -- 来源标识（本地 / 注册表 / 社区） -- 启用 / 禁用开关 -- 标签 - -您可以使用顶部的筛选栏，按**来源**、**传输类型**和**标签**进行过滤，也可以通过搜索框按名称快速定位服务。 - -### 编辑服务详情 - -点击任意服务卡片，打开详情弹窗，可以进行以下操作： - -- **编辑基本信息**：修改服务名称、描述、URL、Authorization Token 和标签 -- **启用 / 禁用服务**：通过开关控制服务的启用状态，禁用后该服务的工具将不会出现在智能体工具选择中 -- **删除服务**：移除 MCP 服务记录，容器化服务会同步清理容器资源 - -### 查看工具列表 - -在服务详情弹窗中，点击"工具列表"按钮，可以查看该 MCP 服务提供的所有工具。 - -### 健康检查 - -点击详情弹窗中的"健康检查"按钮，系统会对 MCP 服务发起连接测试并返回当前状态： - -- **正常**：服务可正常连接 -- **异常**：服务无法连接或响应异常 -- **未检测**：尚未进行健康检查 - -### 容器管理 - -对于容器化部署的 MCP 服务，详情弹窗中还提供以下操作： - -- **查看容器日志**：实时查看运行中容器的输出日志，方便排查问题 -- **查看容器配置**：查看创建容器时使用的配置 JSON - -### 发布到社区 - -在服务详情弹窗中，点击"发布到社区"按钮： - -1. 确认或修改发布信息（名称、描述、标签等） -2. 点击"确认发布"，该服务将发布到社区 -3. 发布后其他用户可在添加服务的"社区市场"页签中浏览和导入 - ---- - -## 🌐 发布的服务 - -"发布的服务"页签展示您自己发布到社区的所有 MCP 服务，您可以在此集中管理已发布的内容。 - -每张卡片展示服务名称、描述、版本和标签，支持按名称、标签和传输协议进行筛选。 - -点击服务卡片可查看详细信息，您可以： - -- **编辑发布的服务**：修改已发布服务的名称、描述和标签 -- **删除发布的服务**：将服务从社区撤回，不再对其他用户可见 - ---- - -## 🔗 与智能体协作 - -添加 MCP 服务后，其提供的工具会自动同步到智能体的工具选择列表中。在 **[智能体开发](./agent-development)** 页面配置智能体时： - -1. 在"选择智能体的工具"页签下，找到对应 MCP 服务分组 -2. 点击工具名称即可启用该工具 -3. 可点击 ⚙️ 查看工具描述并进行参数配置 - -## 🚀 下一步 - -完成 MCP 服务配置后，建议您： - -1. **[智能体开发](./agent-development)** - 将 MCP 工具配置给智能体使用 -2. **[智能体空间](./agent-space)** - 查看智能体与 MCP 的协作关系 -3. **[开始问答](./start-chat)** - 在对话中体验智能体调用 MCP 工具的效果 - -如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../quick-start/faq)** 或在 [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions) 中进行提问获取支持。 \ No newline at end of file diff --git a/doc/docs/zh/user-guide/memory-management.md b/doc/docs/zh/user-guide/memory-management.md deleted file mode 100644 index b8b9915e2b..0000000000 --- a/doc/docs/zh/user-guide/memory-management.md +++ /dev/null @@ -1,160 +0,0 @@ -# 记忆管理 - -Nexent的智能记忆系统为智能体提供持久化的上下文感知能力，通过多层级记忆管理机制，实现跨对话会话的知识累积与检索，显著提升人机交互的连贯性和个性化程度。 - -## 🎯 什么是智能记忆系统 - -智能记忆系统让智能体能够"记住"重要信息，并在后续对话中自动使用这些记忆，为您提供更连贯、更个性化的服务体验。 - -### 核心优势 - -- **跨对话记忆**：智能体可以记住之前对话中的重要信息 -- **自动检索**：智能体会自动检索相关记忆，无需您重复说明 -- **个性化服务**：根据您的偏好和习惯提供个性化服务 -- **知识积累**：智能体的知识会随着使用不断积累和优化 - -## ⚙️ 系统配置 - -### 访问记忆管理 - -1. 在左侧导航栏中点击"记忆管理" -2. 进入记忆管理页面进行配置 - -### 基础配置 - -在记忆管理页面的"系统配置"模块中，您可以进行以下设置： - -| 配置项 | 选项 | 默认值 | 说明 | -|--------|------|--------|------| -| 记忆服务状态 | 启用/禁用 | 启用 | 控制整个记忆系统的运行状态 | -| Agent 记忆共享策略 | 总是共享/每次询问我/禁止共享 | 总是共享 | 定义Agent间共享记忆生成是否需要用户授权同意 | - -
- 记忆系统配置 -
- -### 配置说明 - -- **记忆服务状态**：启用后，智能体将能够使用记忆功能；禁用后，所有记忆功能将暂停。 -- **Agent 记忆共享策略**： - - **总是共享**：智能体之间自动共享记忆，无需确认 - - **每次询问我**：智能体间共享记忆前会询问您的意见 - - **禁止共享**：智能体之间不共享记忆，保持独立 - -## 📚 记忆层级 - -Nexent采用四层记忆存储架构，不同层级的记忆有不同的作用域和用途： - -### 租户级记忆 - -- **作用域**：组织全局，所有用户和智能体共享 -- **存储内容**：企业级标准操作流程、合规政策、组织架构、事实信息 -- **适用场景**：企业知识管理、标准化流程执行、合规性检查 -- **管理权限**：租户管理员 - -### 智能体级记忆 - -- **作用域**：特定智能体，该智能体的所有用户共享 -- **存储内容**：专业领域知识、技能模板、历史对话摘要、学习积累 -- **适用场景**：专业技能积累、领域知识沉淀、经验学习 -- **管理权限**：租户管理员 - -### 用户级记忆 - -- **作用域**：特定用户账户，仅该用户可见 -- **存储内容**：个人偏好设置、使用习惯、常用指令模板、个人信息 -- **适用场景**：个性化服务、用户体验优化、偏好管理 -- **管理权限**：用户自己 - -### 用户-智能体级记忆 - -- **作用域**：特定用户账户下的特定智能体，最私密和个性化 -- **存储内容**：协作历史、个性化事实信息、特定任务上下文、关系模型 -- **适用场景**：深度协作场景、个性化调优、任务连续性维护 -- **管理权限**：用户自己 - -### 记忆优先级 - -当智能体需要检索记忆时，会按照以下优先级顺序（由高到低）： - -1. **租户级** → 基础事实和通用知识 -2. **用户-智能体级** → 最具体的上下文信息 -3. **用户级** → 个人偏好和习惯 -4. **智能体级** → 专业知识和技能 - -## 🤖 自动化记忆管理 - -智能记忆系统支持自动化管理，让您无需手动操作即可享受记忆功能： - -### 智能提取 - -- 系统会自动识别对话中的关键事实信息 -- 自动生成记忆条目并存储到合适的层级 -- 无需您手动添加，系统会智能判断重要性 - -### 自动上下文嵌入 - -- 智能体会自动检索相关性最高的记忆条目 -- 将记忆隐式嵌入到对话上下文中 -- 让智能体能够基于历史记忆提供更准确的回答 - -### 增量更新 - -- 支持记忆内容的渐进式更新和补充 -- 自动清理过时或不再相关的记忆条目 -- 保持记忆库的时效性和准确性 - -## ✋ 手动记忆操作 - -除了自动化管理，您也可以手动管理记忆，确保重要信息被正确记录： - -### 添加记忆 - -1. 在记忆管理页面，选择要添加记忆的层级和智能体 -2. 点击绿色的"对话加号"按钮 -3. 输入要记录的内容（最多500字符） -4. 点击对钩确认添加 - -
- 添加记忆 -
- -### 删除记忆 - -您可以通过以下方式删除记忆： - -- **删除分组记忆**：点击红色叉号按钮，在确认弹框中点击确认，可删除某个智能体分组下所有的记忆条目 -- **删除单条记忆**：点击红色橡皮按钮，可删除特定的一条记忆条目 - -
- 删除记忆 -
- -## 💡 使用建议 - -### 记忆内容原则 - -1. **原子性原则**：每条记忆应包含**简洁**、**单一**、**明确**的事实信息 - - ✅ 好：用户喜欢使用深色主题 - - ❌ 不好：用户喜欢使用深色主题，并且经常在晚上工作，还喜欢喝咖啡 - -2. **时效性管理**：定期清理过时或不再相关的记忆条目，保持记忆库的时效性和准确性 - -3. **隐私保护**：敏感信息应尽量避免在租户层级或智能体层级进行共享，建议使用用户级或用户-智能体级记忆 - -### 最佳实践 - -- **合理选择层级**：根据信息的共享需求选择合适的记忆层级 -- **定期检查**：定期查看和清理记忆，确保信息的准确性 -- **利用自动化**：让系统自动管理常规记忆，手动管理重要信息 -- **保护隐私**：个人敏感信息使用用户级记忆，避免共享 - -## 🚀 下一步 - -配置好记忆管理后，您可以： - -1. 在 **[开始问答](./start-chat)** 中体验智能体的记忆能力 -2. 在 **[智能体空间](./agent-space)** 中管理您的智能体 -3. 继续 **[智能体开发](./agent-development)** 创建更多智能体 - -如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../quick-start/faq)** 或在[GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions)中进行提问获取支持。 diff --git a/doc/docs/zh/user-guide/quick-setup.md b/doc/docs/zh/user-guide/quick-setup.md index 96fb26875d..ed20983bdd 100644 --- a/doc/docs/zh/user-guide/quick-setup.md +++ b/doc/docs/zh/user-guide/quick-setup.md @@ -13,7 +13,7 @@ - **应用配置**：设置应用图标、名称和描述 - **模型配置**：接入大语言模型、向量化模型和视觉语言模型 -详细内容请参考：[模型管理](./model-management) +详细内容请参考：[模型管理](./agent-development/model-configuration.md) ### 第二步：知识库配置 @@ -23,7 +23,7 @@ - **上传文件**：支持多种文件格式 - **生成总结**：为知识库生成内容总结 -详细内容请参考：[知识库](./knowledge-base) +详细内容请参考：[知识库](./agent-development/knowledge-configuration.md) ### 第三步：智能体开发 @@ -38,7 +38,7 @@ - **发布智能体**：已发布的智能体将在选中的用户组内可见，并列于智能体空间与开始问答选择框中 - **版本管理**：跟踪智能体的迭代历史，支持查看、回滚至历史版本及创建新版本 -详细内容请参考：[智能体开发](./agent-development) +详细内容请参考：[智能体开发](./agent-development.md) ## 🎯 使用建议 @@ -51,8 +51,8 @@ 完成快速配置后，您可以： -1. 进入 **[智能体空间](./agent-space)** 查看和管理所有智能体 +1. 进入 **[智能体空间](./agent-development.md.md)** 查看和管理所有智能体 2. 在 **[开始问答](./start-chat)** 中与智能体进行交互 -3. 配置 **[记忆管理](./memory-management)** 以提升智能体的记忆能力 +3. 配置 **[记忆管理](./agent-development/memory-configuration.md.md)** 以提升智能体的记忆能力 -如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../quick-start/faq)** 或在[GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions)中进行提问获取支持。 +如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../quick-start/faq.md)** 或在[GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions)中进行提问获取支持。 diff --git a/doc/docs/zh/user-guide/resource-repository/agent-repository.md b/doc/docs/zh/user-guide/resource-repository/agent-repository.md new file mode 100644 index 0000000000..5849494078 --- /dev/null +++ b/doc/docs/zh/user-guide/resource-repository/agent-repository.md @@ -0,0 +1,197 @@ +# 智能体仓库 + +智能体仓库是同租户内共享、管理与审核智能体的中心。您可以浏览已上架的智能体并复制到自己的工作区，管理自己有权限编辑的智能体，以及（管理员）审核上架申请。 + +## 👥 管理员与开发者的界面差异 + +进入 **Agent 仓库** 后，页面顶部会按角色展示不同页签： + +| 角色 | 可见页签 | 额外能力 | +|------|----------|----------| +| **开发者** | 仓库、我的 Agent | 浏览共享仓库、复制智能体、管理自己的智能体并申请上架 | +| **管理员** | 仓库、我的 Agent、审核中心 | 在开发者能力基础上，还可审核上架申请，并从仓库中直接下架智能体 | + +> 提示：审核中心页签仅对管理员可见；开发者通过「我的 Agent」中的「查看审核进度」跟踪自己的申请结果。 + +**开发者视角**（仓库 / 我的 Agent）： + +
+ 开发者页签界面 +
+ +**管理员视角**（仓库 / 我的 Agent / 审核中心）： + +
+ 管理员页签界面 +
+ +--- + +## 📦 仓库 + +「仓库」页签展示当前租户内已上架（共享）的智能体。同租户成员可浏览、查看详情，并复制到自己的「我的 Agent」中再进行编辑。 + +> 同租户内的智能体需先「复制为我的智能体」后才能编辑。 + +### 浏览与搜索 + +- 以卡片形式展示已上架智能体 +- 支持按**智能体名称、描述或标签**搜索 +- 每张卡片展示：图标、名称、作者、描述、标签、工具数量、版本号与安装次数 + +
+ 仓库列表 +
+ +### 查看详情 + +点击卡片上的「详情」，可查看该智能体的完整信息，包括： + +- **基础信息**：名称、图标、作者、版本、模型、安装次数、创建时间 +- **智能体简介**：描述说明 +- **内置工具**：已启用的工具列表 +- **智能体角色**：角色设定（Duty Prompt）等相关配置 + +
+ 智能体详情 +
+ +### 复制智能体 + +点击卡片上的「复制」，系统会先对依赖配置做预检，并展示配置清单： + +1. 查看**可复制比例**，以及可用项 / 待处理项数量 +2. 若存在异常项（如模型未开通、知识库未开通、MCP 未开通、Skill 名称冲突、工具不可用等），可按提示前往开通或处理 +3. 处理完成后可点击刷新重新预检；也可在知晓风险的情况下继续复制 +4. 复制成功后，智能体会出现在「我的 Agent」中，供您编辑与使用 + +依赖类型通常包括：**模型**、**知识库**、**MCP 服务**、**Skill 技能**、**工具**。 + +
+ 复制配置清单 +
+ +### 管理员下架 + +管理员可在仓库卡片右上角的更多菜单中选择「下架」。下架后，该智能体将不再对同租户成员可见，也无法继续被复制。 + +开发者若需下架自己已上架的智能体，请在「我的 Agent」中通过「查看审核进度」弹窗操作下架。 + +--- + +## 🧑 我的 Agent + +「我的 Agent」页签用于管理您有权限编辑的智能体，包括自己创建的，以及从仓库复制而来的智能体。 + +### 筛选与搜索 + +- **全部 / 我创建的 / 其它**：按归属筛选 +- 支持按智能体名称或描述搜索 +- 列表以卡片形式展示，并可分页浏览 + +### 创建与导入 + +在「全部」筛选且无搜索条件时，页面会提供入口： + +- **新建智能体**：跳转到智能体开发页创建新智能体 +- **导入智能体**：通过导入向导上传并导入智能体配置 + +
+ 我的 Agent 列表 +
+ +### 智能体卡片状态 + +每张卡片会展示生命周期与上架相关状态，便于快速识别： + +| 标识 | 说明 | +|------|------| +| **草稿 / 已发布** | 智能体是否已发布版本（仅已发布版本可申请上架） | +| **Hub** | 该智能体存在仓库相关记录（曾申请或已上架） | +| **审核中** | 首次上架申请待管理员审核 | +| **更新审核中** | 已有上架版本，新版本再次申请上架待审核 | +| **已上架** | 当前已在仓库中共享 | +| **审核驳回** | 上架申请未通过，可修改后重新申请 | + +### 常用操作 + +在智能体卡片上，您可以： + +- **编辑**：进入智能体开发页修改配置（只读权限时不可编辑） +- **查看**：查看已发布版本的详情 +- **评估**：跳转到智能体评估页面 +- **更多操作**： + - **申请上架**：将当前已发布版本提交到仓库审核 + - **查看审核进度 / 查看更新审核进度**：查看申请状态，并可取消申请或下架 + - **删除**：删除该智能体 + +### 申请上架 + +仅当智能体已有已发布版本，且当前版本尚未上架时，可发起申请： + +1. 在更多菜单中点击「申请上架」 +2. 填写上架信息： + - **智能体图标**（必填）：选择预设 emoji 或自定义单个 emoji + - **智能体标签**（必填）：最多 5 个，可选择预设标签或输入自定义标签 + - **上架说明**（选填）：补充给审核人的说明 +3. 点击「提交申请」，等待管理员审核 + +
+ 申请上架 +
+ +### 查看审核进度 + +在更多菜单中打开审核状态弹窗后，可看到： + +- 当前状态：审核中 / 已通过 / 已驳回 +- 审核版本、提交时间、上架说明与审核意见（如有） + +根据状态，您还可以： + +- **取消申请上架**：撤销待审核或已驳回的申请 +- **下架**：将已上架的智能体从仓库撤回 + +
+ 审核进度 +
+ +--- + +## ✅ 审核中心 + +「审核中心」仅对**管理员**可见，用于处理同租户用户提交的上架申请。 + +### 待审核队列 + +页面以列表形式展示待处理申请，包含： + +- 智能体名称与图标 +- 申请版本 +- 提交人 +- 上架说明 +- 操作按钮：详情、通过、驳回 + +页签上会显示待处理数量角标，便于管理员及时处理。 + +### 审核操作 + +1. 点击「详情」可预览智能体配置，确认能力与工具是否合适 +2. 点击「通过」：可选填审核意见，确认后智能体将上架到「仓库」 +3. 点击「驳回」：可选填审核意见，驳回后提交者可在「我的 Agent」中修改并重新申请 + +
+ 审核确认 +
+ +--- + +## 🚀 下一步 + +在智能体仓库中完成管理后，您可以： + +1. 在 **[开始问答](../start-chat)** 中与智能体进行交互 +2. 继续 **[智能体配置](../agent-development/agent-configuration)** 创建或迭代更多智能体 +3. 配置 **[记忆配置](../agent-development/memory-configuration)** 以提升智能体的记忆能力 + +如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../../quick-start/faq)** 或在 [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions) 中进行提问获取支持。 diff --git a/doc/docs/zh/user-guide/resource-repository/create-docx.md b/doc/docs/zh/user-guide/resource-repository/create-docx.md new file mode 100644 index 0000000000..d99439fa4f --- /dev/null +++ b/doc/docs/zh/user-guide/resource-repository/create-docx.md @@ -0,0 +1,263 @@ +--- +title: create-docx 官方技能 +--- + +# create-docx 官方技能 + +`create-docx` 是 `official-skills-zip` 提供的官方文件生成技能，用于根据结构化需求创建或编辑 Word 文档。智能体可以根据用户提供的主题、内容和格式要求调用该技能，生成 `.docx` 文件并将结果作为 Nexent artifact 推送到对话前端，同时保存到会话历史。 + +## 使用 create-docx + +在智能体中启用 `create-docx` 后，用户可以直接提出创建或编辑 Word 文档的需求，例如生成报告、方案、通知、会议纪要或其他结构化文档。技能会负责执行文档生成脚本，并返回可下载的文件产物。 + +具体的脚本参数、支持的文档能力和运行约束以 `create-docx` 技能包中的 `SKILL.md` 为准。 + +## 自定义文件生成 Skill 开发指南 + +以下内容面向编写自定义 Skill 的开发者。按照本文约定声明并实现脚本后，Skill 生成的文件会作为 Nexent artifact 被上传、以附件形式推送到对话前端，并保存到会话历史。 + +本文以 `create-docx` 为示例，介绍如何为文件生成脚本声明输出类型、返回结构化 artifact，以及排查文件未能作为附件发布的问题。 + +### 适用场景 + +适用于生成或导出可交付文件的 Skill，例如： + +- Word 文档、PDF、表格和演示文稿 +- 图片、音频、视频和压缩包 +- 代码、配置文件、数据集和报告 + +如果脚本只执行分析、修改工作区中的中间文件，或返回文本结果，则不要将其声明为文件生成脚本。 + +## 工作原理 + +文件从生成到前端显示经过以下步骤： + +1. 在 `SKILL.md` 的 `script_outputs` 中声明允许产出文件的脚本、artifact 类型和 MIME 类型。 +2. 智能体通过 `run_skill_script` 执行该脚本。 +3. 脚本生成文件并返回成功 JSON，其中包含 `artifacts` 数组。 +4. Nexent SDK 检查脚本是否已声明、artifact 是否完整、文件是否存在、文件大小是否一致、MIME 类型是否已声明。 +5. 校验通过的 artifact 以 `skill_artifact` 结构化事件发布。 +6. 后端只接收该结构化事件，检查文件路径是否允许上传后上传到对象存储。 +7. 后端发送 `skill_files` 流事件并写入会话附件；前端按 MIME 类型和文件名渲染下载或预览入口。 + +普通文本、执行日志和脚本标准输出中的 JSON 不会被扫描或转换为文件附件。未发送 `skill_artifact` 的脚本结果不会出现在前端附件区。 + +## 技能包结构 + +应以 ZIP 包上传包含脚本的 Skill。推荐结构如下： + +```text +report-generator/ +├── SKILL.md +├── scripts/ +│ ├── generate_report.py +│ └── publish_report.py +├── requirements.txt +└── examples.md +``` + +`SKILL.md` 位于技能根目录。`script_outputs` 中的路径相对技能根目录，统一使用正斜杠，例如 `scripts/generate_report.py`。 + +## 在 Frontmatter 中声明文件脚本 + +文件生成能力由 `script_outputs` 声明。键是脚本相对路径，值定义该脚本可发布的 artifact 类型和 MIME 类型。 + +```yaml +--- +name: create-docx +description: Create and generate Word documents from structured specifications. Use when users need a new Word document or an edited Word document. +script_outputs: + scripts/generate_docx.py: + kind: file + mime_types: + - application/vnd.openxmlformats-officedocument.wordprocessingml.document + scripts/get_document_path.py: + kind: file + mime_types: + - application/vnd.openxmlformats-officedocument.wordprocessingml.document +--- +``` + +### `script_outputs` 字段 + +| 字段 | 必填 | 说明 | +| ------------ | -------- | ------------------------------------------------------------------------------- | +| 脚本路径 | 是 | 相对 Skill 根目录的路径。执行时必须与 `run_skill_script` 传入的路径匹配。 | +| `kind` | 是 | 文件交付固定填写为 `file`。其他值不会生成文件 artifact。 | +| `mime_types` | 建议必填 | 该脚本允许发布的 MIME 类型列表。运行时 artifact 的 `mime_type` 必须在此列表中。 | + +同一脚本可声明多个 MIME 类型，例如同时支持 CSV 与 XLSX： + +```yaml +script_outputs: + scripts/export_data.py: + kind: file + mime_types: + - text/csv + - application/vnd.openxmlformats-officedocument.spreadsheetml.sheet +``` + +### 路径匹配规则 + +声明路径和调用路径在比较前会去除开头的 `./` 并统一为正斜杠。因此下面两种调用都能匹配 `scripts/generate_report.py`： + +```python +run_skill_script("report-generator", "scripts/generate_report.py", params="--output report.pdf") +run_skill_script("report-generator", "./scripts/generate_report.py", params="--output report.pdf") +``` + +仍建议在 `SKILL.md` 正文和智能体调用示例中始终写 `scripts/...`，避免文档与声明不一致。 + +## 脚本返回契约 + +已声明的文件生成脚本必须在成功时输出一个 JSON 对象。顶层 `status` 必须是 `success`，文件放在 `artifacts` 数组中。 + +```json +{ + "status": "success", + "artifacts": [ + { + "kind": "file", + "absolute_path": "/mnt/nexent/output/monthly-report.docx", + "file_name": "monthly-report.docx", + "mime_type": "application/vnd.openxmlformats-officedocument.wordprocessingml.document", + "file_size_bytes": 12800 + } + ] +} +``` + +### 顶层字段 + +| 字段 | 必填 | 要求 | +| ----------- | ---- | ------------------------------------------------- | +| `status` | 是 | 必须为字符串 `success`。其他值不会发布 artifact。 | +| `artifacts` | 是 | 必须为数组，可包含一个或多个文件 artifact。 | + +可在顶层添加供模型阅读的 `message`、`file_path` 等字段，但这些字段不参与附件发布。文件附件只读取 `artifacts`。 + +### 单个 artifact 字段 + +| 字段 | 必填 | 要求 | +| ----------------- | ---- | ----------------------------------------------------------- | +| `kind` | 是 | 固定为字符串 `file`。 | +| `absolute_path` | 是 | 已生成文件的绝对路径。必须位于 Nexent 允许上传的工作目录。 | +| `file_name` | 是 | 前端显示与下载使用的文件名，不能是空字符串。 | +| `mime_type` | 是 | 文件真实 MIME 类型，必须符合该脚本的 `mime_types` 声明。 | +| `file_size_bytes` | 是 | 非负整数，必须等于 `absolute_path` 指向文件的实际字节大小。 | + +`file_size_bytes` 不能是布尔值、字符串或估算值。SDK 会在发布前读取磁盘文件大小并进行精确比较。 + +## Python 脚本示例 + +以下示例创建一个文本报告，并打印符合契约的 JSON。实际文件格式应使用对应的生成库。 + +```python +from __future__ import annotations + +import json +from pathlib import Path + +MIME_TYPE = "text/plain" + + +def main() -> None: + output_path = Path("/mnt/nexent/output/summary.txt") + output_path.parent.mkdir(parents=True, exist_ok=True) + output_path.write_text("Generated report\n", encoding="utf-8") + + print(json.dumps({ + "status": "success", + "artifacts": [{ + "kind": "file", + "absolute_path": str(output_path.resolve()), + "file_name": output_path.name, + "mime_type": MIME_TYPE, + "file_size_bytes": output_path.stat().st_size, + }], + })) + + +if __name__ == "__main__": + main() +``` + +相应的 `SKILL.md` 必须包含： + +```yaml +script_outputs: + scripts/generate_summary.py: + kind: file + mime_types: + - text/plain +``` + +## 在 SKILL.md 正文中指导智能体 + +Frontmatter 声明用于运行时校验；正文用于告诉智能体何时调用脚本和如何处理返回值。文件生成脚本应在正文中明确以下要求： + +```markdown +## Generate a report + +Use `scripts/generate_report.py` to create the final report. + +1. Pass the requested output name through `params`. +2. Wait for the script to return a successful JSON result. +3. Return the script result without rewriting its `artifacts` field. +4. Do not use editing scripts as the final publishing step. +``` + +若 Skill 包含编辑脚本和最终导出脚本，只声明最终导出脚本为 `kind: file`。编辑脚本可以修改工作文件，但不应产生附件；完成编辑后调用已声明的导出或发布脚本。 + +## MIME 类型建议 + +声明应使用标准 MIME 类型，而不是文件扩展名。常见值如下： + +| 文件类型 | MIME 类型 | +| ---------- | --------------------------------------------------------------------------- | +| PDF | `application/pdf` | +| DOCX | `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | +| XLSX | `application/vnd.openxmlformats-officedocument.spreadsheetml.sheet` | +| PPTX | `application/vnd.openxmlformats-officedocument.presentationml.presentation` | +| CSV | `text/csv` | +| JSON | `application/json` | +| ZIP | `application/zip` | +| PNG | `image/png` | +| JPEG | `image/jpeg` | +| Markdown | `text/markdown` | +| Plain text | `text/plain` | + +前端会结合 `mime_type` 与文件扩展名选择附件图标、下载和预览行为。确保扩展名、实际文件内容及 `mime_type` 三者一致。 + +## 失败条件与排查 + +下列情况会使文件不会作为附件发布： + +| 问题 | 结果 | 处理方式 | +| -------------------------------- | ------------------- | ------------------------------------------- | +| 脚本未在 `script_outputs` 中声明 | SDK 不发布 artifact | 添加完全匹配的脚本路径及 `kind: file`。 | +| `kind` 不是 `file` | SDK 忽略 artifact | 将脚本声明和 artifact 字段都设为 `file`。 | +| `status` 不是 `success` | SDK 忽略 artifact | 仅在文件成功写入后返回成功状态。 | +| 缺少必填 artifact 字段 | SDK 忽略该 artifact | 返回完整的五个字段。 | +| 文件不存在 | SDK 忽略 artifact | 在输出 JSON 前确认文件已写入。 | +| 文件大小不匹配 | SDK 忽略 artifact | 使用实际 `stat().st_size` 填充字段。 | +| MIME 未声明 | SDK 忽略 artifact | 将实际 MIME 加入该脚本的 `mime_types`。 | +| 路径不允许上传 | 后端拒绝上传 | 把输出写入 Nexent 允许的工作目录。 | +| 仅打印路径或日志 JSON | 不会生成附件 | 返回完整 `artifacts` 数组，不依赖文本解析。 | + +## 发布前检查清单 + +- [ ] `SKILL.md` 使用 `script_outputs`，不使用 Skill 级旧输出字段。 +- [ ] 每个可交付文件的脚本路径都已声明为 `kind: file`。 +- [ ] 每个脚本的 `mime_types` 包含所有实际可能输出的 MIME 类型。 +- [ ] 脚本仅在文件写入完成后输出 `status: success`。 +- [ ] 每个 artifact 含 `kind`、`absolute_path`、`file_name`、`mime_type`、`file_size_bytes`。 +- [ ] `file_size_bytes` 与磁盘实际大小完全一致。 +- [ ] 输出路径位于运行环境允许上传的目录。 +- [ ] 使用真实对话验证前端能收到并显示附件。 + +## 相关文档 + +- [官方技能](./official-skills.md) +- [技能系统概览](/zh/backend/skills/overview) +- [Skill 仓库](./skill-repository.md) diff --git a/doc/docs/zh/user-guide/resource-repository/mcp-repository.md b/doc/docs/zh/user-guide/resource-repository/mcp-repository.md new file mode 100755 index 0000000000..86efd41f35 --- /dev/null +++ b/doc/docs/zh/user-guide/resource-repository/mcp-repository.md @@ -0,0 +1,219 @@ +# MCP 仓库 + +MCP 仓库是同租户内共享、管理与审核 MCP（Model Context Protocol）服务的中心。您可以浏览社区中已上架的 MCP 服务并一键安装，管理自己有权限编辑的 MCP 服务，以及（管理员）审核上架申请。 + +## 👥 管理员与开发者的界面差异 + +进入 **MCP 仓库** 后，页面顶部会按角色展示不同页签： + +| 角色 | 可见页签 | 可用能力 | +|------|----------|-----------------------------------------| +| **开发者** | 仓库、我的MCP | 浏览共享仓库、安装 MCP 服务、添加 MCP 服务、管理自己的服务并申请上架 | +| **管理员** | 仓库、我的MCP、审核中心 | 在开发者能力基础上，还可审核上架申请，并从仓库中直接下架服务 | + +> 提示：审核中心页签仅对管理员可见；开发者通过「我的MCP」中MCP卡片右上角的「查看审核进度」跟踪自己的申请结果。 + +**开发者视角**（仓库 / 我的MCP）： + +
+ 开发者页签界面 +
+ +**管理员视角**（仓库 / 我的MCP / 审核中心）： + +
+ 管理员页签界面 +
+ +--- + +## 📦 仓库 + +「仓库」页签展示租户中已上架（共享）的 MCP 服务。您可浏览、查看详情，并一键安装到自己的「我的MCP」中使用。 + +### 浏览与搜索 + +- 以卡片形式展示已上架 MCP 服务 +- 支持按**名称或标签**搜索 +- 每张卡片展示：名称、描述、标签与安装量 + +
+ 仓库列表 +
+ +### 查看详情 + +点击卡片的查看详情按键，可查看该 MCP 服务的完整信息 + +
+ MCP 详情 +
+ +### 安装服务 + +点击卡片上的「安装」，填写必要信息后点击确认添加，系统会自动安装并配置该服务。已安装的服务会显示「已安装」状态，避免重复导入。 + +安装完成后，服务会出现在「我的MCP」中，其提供的工具也会自动同步到智能体的工具选择列表。 + +
+ MCP 详情 +
+ +### 管理员下架 + +管理员可在仓库卡片中直接「下架」已上架的服务。下架后，该服务将不再对租户成员可见，也无法继续被安装。 + +开发者若需下架自己已上架的服务，请在「我的MCP」中通过「查看审核进度」弹窗操作下架。 + +
+ MCP 详情 +
+ +--- + +## 🧑 我的MCP + +「我的MCP」页签用于管理您有权限编辑的 MCP 服务，包括自己创建的，租户成员给予您编辑权限的，以及从仓库安装而来的服务。 + +### 添加服务 + +点击「添加 MCP 服务」打开添加弹窗，支持多种接入来源。 + +#### 自定义添加 + +支持四种部署类型： + +| 部署类型 | 适用场景 | 关键配置 | +|----------|----------|----------| +| **远程链接** | 已有独立部署的 MCP 服务（HTTP / SSE） | 服务 URL、Authorization Token、自定义请求头 | +| **容器** | 以容器方式运行的 MCP 服务 | 容器配置 JSON（mcpServers）、端口号 | +| **API** | 以 OpenAPI 规范描述的 HTTP API | 服务 URL、OpenAPI JSON（必填，自动校验格式） | +| **本地上传镜像** | 已有 Docker 镜像（.tar 文件） | 上传 .tar 镜像文件、端口号 | + +> 本地上传镜像需管理员在部署中开启「上传镜像」功能后才会显示。 + +#### 端口说明 + +端口行为根据 Nexent 自身的部署方式自动区分： + +- **Docker / Kubernetes 部署**：容器端口为统一默认端口，由系统自动分配并锁定，不可修改。多个 MCP 服务可复用同一端口，互不冲突。 +- **本地部署**：端口由用户设置，提供「推荐端口」按钮一键获取可用端口，并自动检测端口占用情况。 + +#### 从 MCP 外部市场导入 + +浏览社区维护的 MCP 外部市场，选择「远程」或「容器」接入方式，填写所需的环境变量参数，一键导入。 + +
+ MCP 详情 +
+ +### 服务卡片状态 + +每张卡片会展示运行与上架相关状态： + +| 标识 | 说明 | +|--------------|---------------------------| +| **启用 / 已启用** | 服务是否启用，禁用后工具不再出现在智能体工具选择中 | +| **审核中** | 申请上架待管理员审核 | +| **已上架** | 当前已在仓库中共享 | +| **审核驳回** | 上架申请未通过，可修改后重新申请 | + +
+ MCP 详情 +
+ +### 常用操作 + +在服务卡片或详情弹窗中，您可以： + +- **编辑**：修改名称、描述、URL、Authorization Token 与标签等内容 +- **启用 / 已启用**：通过开关控制服务状态 +- **查看工具列表**：查看该服务提供的所有工具 +- **容器管理**：查看容器日志与创建时的配置 JSON（容器化服务） + +
+ MCP 详情 +
+ +- **更多操作**： + - **申请上架**：将服务提交到仓库审核 + - **连通性校验**：发起连接测试，测试 MCP 服务连接状态 + - **查看审核进度**：查看申请状态，并可取消申请或下架 + - **删除**：删除服务，容器化服务会同步清理容器资源 + +
+ MCP 详情 +
+ + + + +### 分享配置 + +添加或编辑服务时，可勾选用户组和用户权限，将服务配置共享给其他成员，便于团队内复用。 + + +### 申请上架 + +1. 在创建或编辑页面，勾选 MCP 的服务配置（服务 URL、Authorization Token、自定义请求头、容器配置），被勾选的配置信息会被分享到仓库中。 +2. 在更多菜单中点击「申请上架」 +3. 填写上架信息： + - **上架说明**（选填）：补充给审核人的说明 +4. 点击「提交申请」，等待管理员审核 + +> 申请上架前需至少勾选一项共享配置字段。 + +### 查看审核进度 + +在更多菜单中打开审核状态弹窗后，可看到： + +- 当前状态：审核中 / 已通过 / 已驳回 +- 审核意见（如有） + +根据状态，您还可以： + +- **取消申请上架**：撤销待审核或已驳回的申请 +- **下架**：将已上架的服务从仓库撤回 + +
+ MCP 详情 +
+ +--- + +## ✅ 审核中心 + +「审核中心」仅对**管理员**可见，用于处理用户提交的上架申请。 + +### 待审核队列 + +页面以列表形式展示待处理申请，包含： + +- 服务名称与部署方式 +- 提交人 +- 上架说明 +- 操作按钮：详情、通过、驳回 + +页签上会显示待处理数量角标，便于管理员及时处理。 + +### 审核操作 + +1. 点击「详情」可预览服务配置，确认是否合适 +2. 点击「通过」：可选填审核意见，确认后服务将上架到「仓库」 +3. 点击「驳回」：可选填审核意见，驳回后提交者可在「我的MCP」中修改并重新申请 + +
+ MCP 详情 +
+ +--- + +## 🚀 下一步 + +在 MCP 仓库中完成管理后，您可以： + +1. 在 **[智能体开发](../agent-development)** 中为智能体配置 MCP 工具 +2. 在 **[开始问答](../start-chat)** 中体验智能体调用 MCP 工具的效果 +3. 继续浏览 **[技能仓库](./skill-repository)** 了解技能与 MCP 的协作 + +如果您在使用过程中遇到任何问题，请参考我们的 **[常见问题](../../quick-start/faq)** 或在 [GitHub Discussions](https://github.com/ModelEngine-Group/nexent/discussions) 中进行提问获取支持。 diff --git a/doc/docs/zh/user-guide/resource-repository/official-skills.md b/doc/docs/zh/user-guide/resource-repository/official-skills.md new file mode 100644 index 0000000000..af0a61ea0c --- /dev/null +++ b/doc/docs/zh/user-guide/resource-repository/official-skills.md @@ -0,0 +1,67 @@ +--- +title: 官方技能 +--- + +# 官方技能 + +Nexent 在 `official-skills-zip` 目录中提供了一组可直接安装的官方技能。安装后，您可以在智能体的技能配置中启用对应能力，并按需填写技能参数。 + +## 技能列表 + +### 文件操作 + +| 技能名称 | 能力说明 | +| --------------------------------- | -------------------------------------------- | +| `read-file` | 读取工作空间内文件的内容与元信息 | +| `create-file-directory` | 创建文件或目录 | +| `delete-file-directory` | 删除文件或目录 | +| `move-file-directory` | 移动或重命名文件或目录 | +| `list-directory` | 以树形结构列出目录内容 | +| [`create-docx`](./create-docx.md) | 根据内容生成 Word 文档并返回可下载的文件产物 | + +### 知识库搜索 + +| 技能名称 | 能力说明 | +| ----------------------- | -------------------------------------------------------------- | +| `search-knowledge-base` | 搜索 Nexent 本地知识库，支持 hybrid、accurate 和 semantic 模式 | +| `search-dify` | 搜索 Dify 知识库 | +| `search-idata` | 搜索 iData 知识库 | +| `search-datamate` | 搜索 DataMate 知识库，支持相似度阈值控制 | + +### 公网搜索 + +| 技能名称 | 能力说明 | +| ------------------- | ---------------------------- | +| `search-web-tavily` | 使用 Tavily 进行公网实时搜索 | +| `search-web-linkup` | 使用 Linkup 进行图文混合搜索 | +| `search-web-exa` | 使用 Exa 进行深度网页搜索 | + +### 多模态分析 + +| 技能名称 | 能力说明 | +| ------------------- | -------------------------------------- | +| `analyze-image` | 基于视觉语言模型分析图片内容并进行问答 | +| `analyze-text-file` | 提取并分析 PDF、Word、Excel 等文件内容 | + +### 通信与远程操作 + +| 技能名称 | 能力说明 | +| --------------- | ------------------------------------------------------------ | +| `email-utils` | 通过 IMAP 收取邮件、通过 SMTP 发送邮件，支持 HTML、CC 和 BCC | +| `run-shell-ssh` | 建立持久化 SSH 会话并在远程主机执行命令 | + +## 安装与使用 + +1. 打开 **资源仓库** 中的 **Skill 仓库**。 +2. 进入官方技能安装入口，查看当前 `official-skills-zip` 中可用的技能。 +3. 选择需要的技能并完成安装。 +4. 在智能体配置的技能列表中启用已安装技能。 +5. 根据技能要求填写参数并保存配置。 + +官方技能的具体参数和调用约束以对应技能包中的 `SKILL.md` 为准。 + +## 相关文档 + +- [Skill 仓库](./skill-repository.md) +- [技能系统概览](/zh/backend/skills/overview) +- [智能体配置](../agent-development/agent-configuration.md) diff --git a/doc/docs/zh/user-guide/skills.md b/doc/docs/zh/user-guide/resource-repository/skill-repository.md similarity index 63% rename from doc/docs/zh/user-guide/skills.md rename to doc/docs/zh/user-guide/resource-repository/skill-repository.md index 54d0f97bb8..9a161f7485 100644 --- a/doc/docs/zh/user-guide/skills.md +++ b/doc/docs/zh/user-guide/resource-repository/skill-repository.md @@ -1,19 +1,21 @@ --- -title: 技能管理 +title: Skill 仓库 --- -# 技能管理 +# Skill 仓库 -技能（Skill）是 Nexent 为智能体扩展能力的核心机制。每个技能将多个工具与使用文档打包为一个可复用的能力单元，可以像搭积木一样为智能体赋予复杂的工作能力。 +技能（Skill）是 Nexent 为智能体扩展能力的核心机制。Skill 仓库用于在同一租户内浏览、复制、管理和审核 Skill；每个 Skill 可以将工具、配置和使用文档打包为可复用的能力单元。 ## 目录 -- [技能与工具的关系](#-技能与工具的关系)：理解技能的核心概念 +- [技能与工具的关系](#技能与工具的关系)：理解技能的核心概念 +- [管理员与开发者的界面差异](#-管理员与开发者的界面差异)：了解不同角色的可用功能 +- [仓库](#-仓库)：浏览、复制和下架已上架 Skill +- [我的 Skill](#-我的-skill)：创建、编辑和申请上架 +- [审核中心](#-审核中心)：管理员审核 Skill 上架申请 - [技能使用指南](#-技能使用指南)：如何在智能体开发中使用技能 -- [技能管理](#-技能管理)：创建、编辑、安装外部技能 - [技能上传指南](#-技能上传指南)：SKILL.md 格式、ZIP 结构、特殊标签与书写规范 - [NL-to-Skill](#-nl-to-skill)：通过自然语言描述自动生成技能 -- [官方技能一览](#-官方技能一览)：预置技能及其能力说明 ## 技能与工具的关系 @@ -30,37 +32,162 @@ title: 技能管理 | 参数 | 固定参数 schema | 可自定义参数模板 | | 分发 | 代码级 | ZIP 包分发，即插即用 | -## 技能使用指南 +## 👥 管理员与开发者的界面差异 -### 为智能体配置技能 +进入 **Skill 仓库** 后，页面顶部会按角色显示不同页签： -1. 打开 **[智能体开发](./agent-development)** 页面 -2. 在"选择智能体的工具"页签中，找到 **技能（Skills）** 分组 -3. 点击技能名称即可选中，再次点击取消选择 -4. 保存智能体配置 +| 角色 | 可见页签 | 额外能力 | +|------|----------|----------| +| **开发者** | 仓库、我的 Skill | 浏览同租户已上架 Skill、复制为自己的 Skill、管理有编辑权限的 Skill、申请上架 | +| **管理员** | 仓库、我的 Skill、审核中心 | 在开发者能力基础上，审核上架申请，并可下架仓库中的 Skill | -## 技能管理 +> 提示：审核中心仅对管理员可见；开发者可在「我的 Skill」中查看自己申请的审核进度。 -### 查看已安装的技能 +**开发者视图**（仓库 / 我的 Skill）： + +
+ 开发者页签界面 +
+ +**管理员视图**（仓库 / 我的 Skill / 审核中心）： + +
+ 管理员页签界面 +
+ +--- + +## 📦 仓库 + +「仓库」页签展示当前租户内已上架并可共享的 Skill。当前租户的开发者和管理员可以浏览、查看详情，并复制到自己的 Skill 列表后再编辑。 + +> 同租户共享的 Skill 必须先「复制为我的 Skill」才能编辑。 + +### 浏览与搜索 + +- 使用卡片浏览已上架 Skill +- 支持按 **Skill 名称、描述或标签** 搜索 +- 卡片展示名称、描述、标签、来源、下载次数等摘要信息 + +
+ 仓库列表 +
+ +### 查看详情 + +点击卡片上的「详情」可查看 Skill 的基本信息，包括名称、创建者、描述、标签、安装次数和更新时间。详情仅用于查看；如需修改，请先复制到「我的 Skill」。 + +
+ Skill详情 +
+ +### 复制为我的 Skill + +1. 在目标 Skill 卡片上点击「复制」 +2. 填写新的 Skill 名称；如系统提示名称冲突，请修改名称后重试 +3. 复制成功后，该 Skill 会出现在「我的 Skill」中，可继续编辑、配置和申请上架 + +### 管理员下架 + +管理员可在仓库 Skill 卡片的更多操作菜单中选择「下架」。下架后，该 Skill 不再对当前租户的开发者和管理员开放浏览或复制；原有的个人副本不受影响。 + +--- + +## 🧑 我的 Skill + +「我的 Skill」用于管理您有权限编辑的 Skill，包括自己创建的 Skill，以及被授予编辑权限的其他 Skill。 + +### 筛选与搜索 + +- **全部 / 我创建的 / 其他人的**：按归属筛选 +- 支持按 Skill 名称、描述或标签搜索 +- 列表以卡片形式展示，并支持分页浏览 + +
+ 我的 Skill 列表 +
+ +### 创建、上传与编辑 -在"选择智能体的工具"技能分组中，系统会展示所有已安装的技能列表，包括： -- 官方技能 -- 自定义技能 +在「我的 Skill」页签中点击「创建 Skill」，可选择： -### 创建自定义技能 +- **交互式创建**：通过自然语言与 Skill 构建助手协作生成或完善 Skill +- **上传技能文件**：上传单个 `SKILL.md` 或包含完整文件结构的 ZIP 包 -Nexent 支持两种方式创建自定义技能：上传技能包文件，或通过自然语言描述自动生成。 +创建后可通过卡片上的「编辑」修改名称、描述、标签、组内权限、`SKILL.md` 正文及附属文件。详细的文件格式和上传规则见后文的[技能上传指南](#-技能上传指南)。 -#### 方式一：上传 SKILL.md 或 ZIP +### 申请上架 -1. 进入技能配置界面 -2. 点击"上传技能"按钮 -3. 选择 `SKILL.md` 文件（单文件）或 `.zip` 压缩包（完整技能包） -4. 系统自动解析并创建技能 +有权限编辑的 Skill 可以申请上架到租户仓库： -#### 方式二：NL-to-Skill 自然语言创建 +1. 在 Skill 卡片上点击「上架」 +2. 可选填写上架说明，帮助管理员了解本次申请 +3. 点击「提交申请」，等待管理员审核 -在技能管理页面，点击"**NL 创建技能**"按钮即可进入。具体用法详见下方 [NL-to-Skill](#-nl-to-skill) 专区。 +提交后，卡片会显示「审核中」状态；同一个 Skill 同时只会有一条待审核的上架记录。 + +
+ 申请上架 +
+ +### 查看审核进度 + +在已提交申请的 Skill 卡片上打开审核状态，可查看当前状态、提交时间、上架说明和审核意见（如有）。 + +- **审核中**：可取消上架申请 +- **已上架**：可下架该 Skill +- **已驳回**：先编辑 Skill 进行修改；随后取消当前上架申请，再重新点击「上架」提交 + +
+ 审核进度 +
+ +--- + +## ✅ 审核中心 + +「审核中心」仅对管理员可见，用于处理当前租户开发者提交的 Skill 上架申请。 + +### 待审核队列 + +待审核列表会展示 Skill 名称、申请人、上架说明和提交时间等信息；页签上的角标显示当前待处理数量。 + +### 审核操作 + +1. 点击「详情」查看 Skill 的基本信息 +2. 点击「通过」并确认后，Skill 将上架到「仓库」 +3. 点击「驳回」时可填写审核意见；提交者可在「我的 Skill」中查看意见、修改后重新申请 + +
+ 审核确认 +
+ +--- + +## 技能使用指南 + +### 为智能体配置技能 + +1. 打开 **[智能体开发](../agent-development.md)** 页面。 +2. 在「选择智能体的工具」中切换到 **Skills** 页签，点击「选择 Skill」。 +3. 选择需要配置的 Skill；再次选择可取消。 +4. 若 Skill 存在必填参数，选择时会自动打开参数配置窗口。完成填写并保存后，Skill 才会加入当前智能体。 +5. 已添加的 Skill 可点击右侧齿轮图标，修改该智能体使用的参数。 +6. 保存智能体配置。 + +
+ 智能体配置中的 Skills 页签 +
+ +### 查看已安装的技能 + +在「选择技能」的 **Skills** 页签中，可查看当前租户可用的官方 Skill 和自定义 Skill，并将其添加到当前智能体。 + +
+ 选择 Skill 弹窗 +
+ +> 不同智能体可以为同一个 Skill 分别保存不同的参数配置。 ## 技能上传指南 @@ -315,6 +442,10 @@ NL-to-Skill 是 Nexent 提供的一项智能创建功能。您只需要用**自 > 您说"我想要一个能搜索 GitHub 仓库并提取 Star 数的技能"，系统就自动为您生成一个完整可用的技能。 +
+ NL-to-Skill 创建界面 +
+ ### 快速上手 #### 第一步：描述您的需求 @@ -458,9 +589,9 @@ NL-to-Skill 擅长生成以下类型的技能： #### 技能修改 -在 NL-to-Skill 界面可以选中已经存在的技能。选中技能后，该技能信息将自动加载。您可以在左侧对话框中使用自然语言尝试对该技能进行更新。 +在「我的 Skill」中找到需要修改的 Skill，点击「编辑」。在智能体配置页面的 Skills 列表中，对具有编辑权限的 Skill 也可点击右侧铅笔图标进入编辑。系统会加载该 Skill 的基本信息、`SKILL.md` 正文和附属文件；您可以通过交互式创建页使用自然语言完善内容，也可以直接编辑文件后保存。 -如果您创建的技能名与已有技能重名，Nexent 将自动从技能创建模式切换为技能更新模式。所有内容将覆盖更新至原有技能。 +新建或上传 Skill 时，如名称与已有 Skill 重名，系统会提示修改名称后再提交。 ## 安全与最佳实践 @@ -470,7 +601,7 @@ NL-to-Skill 擅长生成以下类型的技能： ## 相关参考 -- [智能体开发](./agent-development) -- [本地工具概览](./local-tools/index) -- [MCP 工具配置](./mcp-tools) -- [技能系统概览](../backend/skills/overview) +- [智能体开发](../agent-development.md) +- [本地工具概览](../local-tools/index.md) +- [MCP 仓库配置](./mcp-repository) +- [技能系统概览](/zh/backend/skills/overview) diff --git a/doc/docs/zh/user-guide/start-chat.md b/doc/docs/zh/user-guide/start-chat.md index fb3e4f0c65..67875e77e8 100644 --- a/doc/docs/zh/user-guide/start-chat.md +++ b/doc/docs/zh/user-guide/start-chat.md @@ -1,226 +1,582 @@ # 开始问答 -开始问答页面是您与智能体进行交互的核心界面。在这里，您可以与不同的智能体进行对话，上传文件，使用语音输入，并管理您的对话历史。 +开始问答页面是用户与智能体交互的主要入口。通过该页面，用户可以与智能体进行对话、上传文件与附件、使用语音输入、管理对话历史，并完成文件处理、知识检索、文档生成等多种任务。 -## 🤖 开始问答 +## 一、选择智能体 -### 选择智能体 +### 1. 进入开始问答主页 -在开始对话之前，您需要先选择一个智能体。 +进入开始问答页面后，默认显示智能体列表页。用户需要先选择一个智能体才能开始对话。 -1. **查看可用智能体** - - 已发布的智能体可用于对话 - - 在对话框左下角找到智能体选择下拉框 - - 点击下拉框查看所有可用的智能体列表 - - 每个智能体都会显示名称和功能描述 +![智能体列表页](./assets/start-chat/agent-list.png) -2. **切换智能体** - - 从列表中选择您想要对话的智能体 - - 系统会自动切换到您选中的智能体 - - 切换后即可开始新的对话 +### 2. 选择可用智能体 -
- 选择智能体 -
+只有同时满足以下条件的智能体，才会出现在开始问答页面中： -### 发送文本消息 +- **已发布**：智能体已发布 +- **被设置为主智能体**：智能体为主智能体 +- **当前用户可用**：当前用户有权限使用该智能体 -选择好智能体后，您可以通过以下方式发送文本消息： +智能体卡片展示：智能体图标、显示名称、英文标识、功能描述或问候语。 -1. **输入您的问题** - - 在对话框底部的输入框中输入您的问题或指令 - - Shift+Enter键可换行 +支持通过关键词（名称、描述、开发者）实时搜索。系统会记录最近使用的智能体并提供快速入口，列表超过一页时自动分页。 -2. **发送消息** - - 点击输入框右侧的发送按钮 - - 或者直接按键盘上的Enter键发送消息 - - 智能体会开始处理您的请求并生成回复 +## 二、智能体首页 -3. **查看回复** - - 智能体的回复会实时显示在对话区域 - - 思考过程会以卡片形式展示，便于区分 +选择智能体后进入对应的智能体首页。该页面主要由以下区域组成： -
- 选择智能体 -
+- **左侧对话历史区域**：管理对话历史 +- **右侧欢迎区域**：展示智能体信息和示例问题 +- **底部输入区域**：输入问题、上传附件、选择对话模式 -### 使用语音输入 +![智能体欢迎页](./assets/start-chat/agent-welcome.png) -Nexent支持语音输入功能，让您可以通过语音与智能体交互。前提是您已经配置了语音模型，配置教程可参考[模型管理](./model-management)。 +### 1. 对话历史区域 -1. **启用语音输入** - - 在输入框右下角找到麦克风图标 - - 点击麦克风图标启用语音输入功能 - - 首次使用时会请求麦克风权限，请点击“允许”授权 +左侧边栏显示当前智能体的对话历史记录。 -2. **开始语音识别** - - 授权后，麦克风图标会变为录音状态 - - 清晰地说出您的问题或指令 - - 系统会实时将您的语音转换为文字显示在输入框中 +#### 创建新对话 -3. **完成语音输入** - - 语音识别完成后，系统会自动发送消息 - - 您也可以在发送前手动编辑识别结果 - - 支持中文和英文语音识别 +- 点击边栏顶部的「新对话」按钮可创建新对话 +- 新对话默认使用当前选择的智能体 +- 开始新任务时，可以通过新建对话避免受到历史上下文影响 -> 💡 **小贴士**：为了获得更好的语音识别效果，请确保在安静的环境中使用，并清晰地发音。 +#### 查看对话列表 -### 上传多模态文件进行对话 +- 显示当前智能体的所有历史对话 +- 对话列表按照时间顺序排列 +- 点击已有对话，可继续查看或继续提问 +- 对话标题由系统自动生成 -您可以在对话中上传文件，让智能体基于文件内容为您提供帮助： +#### 管理对话记录 -> ⚠️ **注意事项**： -> 1. 多模态文件对话功能，需在智能体开发时，选择对应的多模态解析工具 -> 1. 文档类、文本类文件需选择 `analyze_text_file` 工具 -> 2. 工具、图片类文件需选择 `analyze_image` 工具 -> 2. 上传的文件大小有限制，建议单个文件不超过10MB。对于大型文档，建议分批上传 +目前支持以下对话管理操作： -1. **选择文件上传方式** - - 点击输入框右下角的文件上传按钮 - - 或直接将文件拖拽到对话区域 +| 操作 | 说明 | +| ---------- | ------------ | +| 重命名对话 | 修改对话标题 | +| 删除对话 | 删除对话记录 | -2. **支持的文件格式** - - **文档类**：PDF、Word (.docx)、PowerPoint (.pptx)、Excel (.xlsx), EPUB (.epub), HTML (.html), XML (.xml) - - **文本类**：Markdown (.md)、纯文本 (.txt), JSON (.json), CSV (.csv) - - **图片类**：JPG、PNG、GIF 等常见图片格式 +> **注意**：删除对话后通常无法恢复，操作前请确认。 -3. **文件处理流程** - - 系统会将您上传的文件存储至MinIO中，并返回S3 URL - - 构建文件元信息并添加到当前对话的上下文中 - - 智能体会基于文件元信息回答您的问题 +![智能体管理](./assets/start-chat/conversation-manage.png) -4. **基于文件的对话** - - 上传文件后，您可以询问关于文件内容的问题 - - 智能体可以调用对应的多模态工具，分析、总结或处理文件中的信息 - - 支持多文件同时上传和处理 +### 2. 智能体开场白和示例问题 -## 📚 管理您的对话历史 +智能体首页中央区域会显示该智能体的开场白（介绍用途和能力）以及预设的示例问题。开场白和示例问题由智能体开发者在智能体配置页面中设置。点击示例问题会自动填入输入框，用户可修改后发送或直接发送。 -左侧边栏提供了完整的对话历史管理功能： +![示例问题](./assets/start-chat/example-question.png) -### 创建新对话 +### 4. 输入框及对话模式 -- 点击左上角的“新对话”按钮开始全新的对话 -- 新对话会默认使用当前选中的智能体，您也可以修改 +底部输入区域用于输入问题、上传附件、使用语音输入和发送消息。 -### 查看对话列表 +#### 执行模式与规划模式 -- **对话标题**：系统会根据对话内容自动生成标题，您可以随时修改 -- **时间排序**：对话按时间顺序排列，显示“今天”和“最近七天”的历史记录 -- **继续对话**：点击任意历史对话即可查看详细内容并继续之前的对话 +输入框上方提供「执行」和「规划」两个模式切换按钮。 -### 管理对话记录 +- **执行模式**：智能体直接进入 ReAct 循环，直到任务完成或达到最大步骤数。适合简单明确的问题。 -1. **编辑对话** - - 将鼠标悬停在对话标题上 - - 右侧会出现“...”按钮，点击可进行编辑操作 +- **规划模式**：适合复杂任务。智能体在执行前会先将任务拆解为多个有序步骤，输入框上方会以卡片形式展示计划和各步骤执行状态（等待中/进行中/已完成/已跳过）。计划步骤数量要求至少 3 步，最多 8 步。智能体按计划逐步执行，并在每个步骤完成后会自动更新状态。 -2. **重命名对话** - - 点击“重命名”可以修改对话标题 - - 输入新的标题后按Enter确认 +![规划模式](./assets/start-chat/plan.png) -3. **删除对话** - - 在编辑模式下可以删除不需要的对话 - - 删除操作不可恢复，请谨慎操作 +#### 选择模型 -> 💡 **小贴士**：定期清理不需要的对话记录可以保持界面整洁，提高查找效率。 +如果智能体在配置页面中预先添加了多个可用模型，用户可以在输入框下方切换具体使用的模型。模型选择器中只展示该智能体已配置的模型。 -
- 对话编辑 - 对话编辑 -
+#### 上传附件 -### 访问其他功能 +用户可以在输入框中上传附件，让智能体根据附件内容完成分析、总结或其他任务。 -你可通过左侧导航栏可以快速访问其他功能模块 -- **智能体空间**：管理所有已开发的智能体 -- **智能体开发**：创建和配置新的智能体 -- **模型管理**：配置AI模型和应用信息 -- **知识库**：管理知识库和文档 -- **记忆管理**：配置和管理智能记忆系统 +**上传方式**： -## 🔍 查看知识引用来源 +- 点击输入框右侧的文件上传按钮 +- 将文件直接拖拽到输入框区域 -右侧边栏提供了“来源”和“图片”两个标签页，帮助您了解智能体回答的信息来源： +**支持的文件类型**： -### 来源标签页 +| 类型 | 文件格式 | +| ---- | --------------------------------------------------------------------------------- | +| 图片 | image/\* (JPG, PNG, GIF 等) | +| 文档 | PDF, Word (.docx), Excel (.xlsx), PowerPoint (.pptx), EPUB (.epub) | +| 文本 | Markdown (.md), 纯文本 (.txt), JSON (.json), CSV (.csv), XML (.xml), HTML (.html) | +| 其他 | 其他格式文件将作为普通附件处理 | + +**文件数量和大小限制**： + +- 单次消息最多可上传 **50 个附件**。 +- 单个附件最大支持 **100 MB**。超过该大小时，系统会拒绝上传。 +- 部署管理员可以通过上传配置设置更低的前端上传限额；如果页面提示的限制低于 100 MB，请以当前环境的提示为准。 + +> **注意**： +> +> - 不同文件类型所需的解析能力由智能体配置决定 +> - 图片类文件需要智能体配置视觉模型和图片解析工具 +> - 文档类文件需要智能体配置相应的文档解析工具 + +**附件内容参与对话**：上传的附件会作为上下文发送给智能体，智能体可以读取和分析附件内容并据此回答问题或执行任务。 + +![上传附件](./assets/start-chat/upload_file.png) + +#### 语音输入 + +用户可以使用麦克风图标输入语音问题。 + +**使用前提**： + +- 需要在系统配置中启用语音识别（STT）功能 +- 首次使用时需要授权浏览器访问麦克风 + +**使用流程**： + +1. 点击输入框右下角的麦克风按钮 +2. 如果是首次使用，浏览器会请求麦克风权限，点击「允许」 +3. 清晰地说出需要询问的内容 +4. 系统实时将语音转换为文字显示在输入框中 +5. 检查并编辑识别结果 +6. 点击发送按钮或按 Enter 键发送 + +> **提示**：为了获得更好的语音识别效果，请确保在安静的环境中使用，并清晰地发音。 + +#### 发送消息 + +**发送方式**： + +- 点击输入框右侧的发送按钮 +- 直接按键盘上的 Enter 键发送 + +**快捷键**： + +| 快捷键 | 功能 | +| ------------- | -------- | +| Enter | 发送消息 | +| Shift + Enter | 换行 | + +**发送状态**： + +- 当智能体正在处理时，发送按钮会变为停止按钮 +- 点击停止按钮可以中断当前的执行过程 + +## 三、问答执行过程 + +智能体采用 ReAct 工作模式，执行任务时可能包含多轮「思考」和「执行」过程。 + +### 1. 记忆检索 + +如果智能体配置了记忆功能，用户发送问题后，系统会在正式执行任务前检索相关记忆。 + +检索到的记忆可能包括： + +- 用户过去提供的信息 +- 用户的偏好设置 +- 历史任务中的相关内容 +- 智能体保存的长期信息 + +相关记忆会作为当前任务的上下文，帮助智能体生成更符合用户需求的回复。 + +![记忆检索](./assets/start-chat/memory.png) + +### 2. ReAct 执行流程 + +Nexent 智能体基于 [smolagents](https://github.com/huggingface/smolagents) 的 CodeAgent 实现，采用 ReAct（Reasoning + Acting）工作模式。核心循环为： + +- **Think**：模型分析当前任务状态，确定下一步行动。对于启用了规划模式的智能体，模型会先评估任务复杂度——如果预计需要超过三个步骤才能完成，则会生成结构化计划。 + +- **Code**：模型以 Python 代码形式输出行动指令，通过 `...` 标签包裹执行代码，通过 `...` 标签包裹仅用于展示的代码。 + +- **Observe**：代码执行后，系统返回真实结果（标记为 `Observation:`）。模型必须基于真实执行结果继续推理，禁止在执行前伪造观察结果。 + +![ReAct循环](./assets/start-chat/ReAct.png) + +循环逻辑在前端是Reasoning重复直到模型判断可以直接生成最终答案，或达到最大步骤数。最终答案以 Markdown 格式输出，支持标题、列表、表格、代码块和链接；若使用了检索工具，还需在对应内容后添加引用标记 `[[字母+数字]]`，以支持溯源。 + +### 3. 查看代码和工具调用过程 + +在问答页面中，用户可以查看智能体执行过程中的关键信息： + +- **思考过程**：智能体的推理分析 +- **生成的可执行代码**：智能体编写的执行代码 +- **调用的工具**：使用的具体工具名称 +- **工具输入参数**：传递给工具的参数 +- **工具输出结果**：工具返回的执行结果 + +这些信息以折叠卡片的形式展示在对话区域中： + +- **思考卡片**（带脑图图标）：显示智能体的推理过程 +- **工具调用卡片**（带工具图标）：显示工具名称、调用状态和执行结果 + +![ReAct循环](./assets/start-chat/tool-call.png) + +### 4. 自动纠错 + +如果智能体生成的代码存在问题，系统具备一定的自动纠错能力。 + +智能体可能会根据执行错误： + +1. 分析错误原因 +2. 修改生成的代码或参数 +3. 重新执行任务 +4. 根据新的执行结果继续处理 + +![自动纠错](./assets/start-chat/self-correction.png) + +> **注意**：自动纠错能力受模型能力、工具实现和任务复杂度影响，并不能保证所有错误都能自动修复。 + +### 5. 最大执行步骤 + +智能体开发者可以在智能体配置页面设置最大的执行步骤数。 + +当智能体达到最大执行步骤后： + +- 系统会停止继续执行 +- 当前问答可能无法完成全部任务 +- 页面会返回当前已经获得的执行结果或停止提示 + +> **建议**：对于复杂任务，应根据任务复杂度合理设置最大执行步骤。 + +### 6. 多工具并行 + +当一个任务需要同时调用多个相互独立的工具时，智能体可以并行调用多个工具，以减少整体等待时间。 + +例如，可以同时进行： + +- 多个知识库检索 +- 多个网页检索 +- 多个文件分析 +- 多个数据处理操作 + +并行调用会以合并的工具调用卡片形式展示，显示调用的工具数量和各自的执行状态。 + +![多工具并行](./assets/start-chat/parallel-tool-calls.png) + +### 7. 多子智能体并行 + +智能体可以根据任务需要调用多个子智能体分别处理不同子任务。 + +多个子智能体可以并行工作，主智能体负责： + +- 拆分复杂任务 +- 分配子任务给不同的子智能体 +- 汇总子智能体的结果 +- 生成最终回复 + +子智能体的调用会以嵌套卡片形式展示，显示： + +- 子智能体名称 +- 子智能体执行的任务描述 +- 执行状态（运行中/已完成） + +![多子智能体并行](./assets/start-chat/parallel-subagents.png) + +### 8. 自检 + +自检是智能体配置的分层 ReAct 自验证能力，用于在关键执行节点和生成最终答案前检查当前结果是否存在明显问题。该功能默认关闭，只有智能体启用自验证配置后，问答页面才会显示自检过程。 + +#### 触发时机 + +启用自检后，系统会根据智能体配置，在以下关键节点进行检查： + +- **工具调用前**：检查生成的执行代码是否为空、Python 语法是否正确，以及是否存在明显的越权或危险操作；如果配置了工具相关检查，还会检查是否调用了相关工具或输出了必要信息。 +- **工具调用或代码执行后**：检查执行结果是否为空，以及结果中是否包含错误信号。 +- **知识检索后**：检查检索结果是否包含可用证据。 +- **子智能体交接后**：检查子智能体是否返回了有实质内容的结论。 +- **生成最终答案前**：检查答案是否为空、是否残留内部执行标记或未替换的占位符，以及之前发生的错误是否已在答案中说明。 + +#### 最终答案检查 + +对于需要证据或较复杂的任务，系统还可以使用验证模型对候选答案进行进一步检查。验证模型会结合用户任务和执行过程，评估： + +- 答案是否覆盖用户目标 +- 结论是否有足够的证据支撑 +- 工具错误是否已经处理 +- 引用格式是否正确 +- 输出格式是否安全、完整 + +轻量级问候等对话会进行基础检查，不一定需要外部证据。最终是否进行验证、验证严格程度和验证模型的使用方式由智能体的自验证配置决定。 + +#### 自检结果 + +自检面板会以折叠卡片形式显示检查过程及结果。面板可能显示以下状态： + +- **正在自检**：系统正在准备或执行检查 +- **基础自检通过**：关键检查项通过 +- **最终自检通过**：最终答案通过验证 +- **自检发现需关注项**：发现问题，但当前问题不会阻断执行 +- **自检未通过，正在修正**：当前候选答案未通过，智能体会根据反馈继续修正 +- **自检已阻断当前动作**：当前执行动作未通过阻断性检查，系统不会继续执行该动作 +- **最终自检未通过**：达到允许的验证轮次后仍未通过 + +面板中还可能显示验证评分、未通过的检查项、用户可见提示和修复建议。验证事件不会展示验证模型的内部推理文本，而是展示结构化的检查结果。 + +#### 自检失败后的处理 + +当关键检查未通过时，系统会将失败标准和修复指令作为反馈交给智能体。智能体可能会： + +1. 修改执行代码或工具参数 +2. 重新调用工具或补充检索证据 +3. 重新生成最终答案 +4. 在无法通过验证时返回受控说明，指出未通过项、原因和建议 + +最终答案验证的最大尝试次数由智能体配置决定，默认最多进行 2 轮最终答案验证。自检不能保证发现所有业务错误，也不会替代用户对文件内容、数据准确性或业务结果的人工确认。 + +![自检](./assets/start-chat/verification.png) + +### 9. 执行完成标识 + +当智能体完成任务后： + +- 回复末尾会显示「完成」标识（绿色圆点+文字） +- 显示本次执行的运行时长 +- 显示本次对话的 Token 消耗统计 + +![智能体完成](./assets/start-chat/finish.png) + +## 四、知识检索与来源溯源 + +当智能体配置了知识检索工具后，用户可以在问答过程中查看回复所引用的信息来源。 + +![来源](./assets/start-chat/source.png) + +### 1. 查看引用来源 + +问答完成后，如果智能体调用了知识检索工具，回复区域会显示「查看来源」按钮。点击该按钮可以在右侧面板查看智能体使用过的知识内容。 + +### 2. 右侧来源面板 + +来源面板分为两个标签页： + +#### 来源标签页 显示智能体回答所引用的知识来源： -- **本地知识库检索** - - 显示文本块标题和来源文件名 - - 点击“展开”按钮可查看完整的文本块内容 - - 帮助您了解智能体从本地知识库中获取的信息 +**本地知识库检索**： + +- 知识库名称 +- 来源文件名称 +- 文本块标题 +- 匹配到的文本内容 +- 相关引用片段 -- **网络检索结果** - - 显示网页标题和来源网址 - - 点击“展开”可查看引用的详细内容 - - 点击网页标题可直接跳转到原始网页 +**网络检索结果**： -💡 **小贴士**： -1. 智能体开发时，请选择 `knowledge_base_search` 工具，启用本地知识库检索功能。 -2. 智能体开发时，请选择 `exa_search`、 `tavily_search`、 `linkup_search` 工具，启用网络检索功能。 +- 网页标题 +- 来源网址 +- 网页摘要或引用内容 +- 相关图片或其他网络资源 -### 图片标签页 +用户可以通过来源链接查看原始网页。 + +#### 图片标签页 - 展示从网络检索中获取的相关图片 -- 点击任意图片可进行预览 -- 帮助您更直观地了解相关信息 +- 点击图片可进行大图预览 +- 显示图片来源网页标题 + +## 五、图像处理功能 + +当智能体配置了视觉模型和图像解析工具后，用户可以上传图片并让智能体进行分析。 + +### 1. 上传图片 + +支持以下方式上传图片： + +| 上传方式 | 操作说明 | +| ------------ | ------------------------------------------ | +| 点击上传按钮 | 点击输入框右侧的文件上传按钮，选择图片文件 | +| 拖拽上传 | 将图片文件直接拖拽到对话区域 | +| 输入框选择 | 在输入框的文件选择器中选择图片 | + +### 2. 图像分析 + +智能体可以根据图片内容执行以下类型的任务： + +- **描述图片内容**：识别并描述图片中的场景、人物、物品等 +- **识别图片中的文字**：提取图片中的文字信息（OCR） +- **分析图像中的物体、场景或结构**：识别图片中的物体类别、场景类型、图表结构等 +- **回答与图片相关的问题**：根据图片内容回答用户的提问 +- **根据图片内容进行整理或判断**：基于图片信息进行分析、总结或推理 + +> **注意**：图像处理能力取决于智能体是否配置了视觉模型和对应工具。 + +![图片分析](./assets/start-chat/analyze_image.png) + +### 3. 图片来源引用 + +如果图片来源于网络检索，右侧来源面板的图片标签页中会显示相关图片缩略图，点击可查看大图。 + +## 六、文档处理与生成 + +### 1. 文档分析 + +> **提示**：如需分析文档，需要在智能体中配置官方的 `analyze_text_file` 工具。 + +用户可以上传文档，让智能体对文档进行以下处理，： + +- **内容总结**：提取文档的主要内容和要点 +- **关键信息提取**：从文档中抽取关键数据、事实或信息 +- **文档问答**：根据文档内容回答用户的问题 +- **结构分析**：分析文档的组织结构、章节关系等 +- **内容对比**：对比多个文档的内容差异 +- **数据整理**：将文档中的数据整理为表格或其他格式 + +![文档分析](./assets/start-chat/analyze_text_file.png) + +### 2. 文档生成 + +> **提示**：如需生成 Word 文档（.docx），需要在智能体中配置官方的 `create-docx` Skill。文档生成能力取决于智能体所配置的工具和 Skill。 + +智能体可以根据用户需求或对话内容生成文档，例如报告、方案、说明文档、表格、演示文稿等。 + +![文档生成](./assets/start-chat/create-docx.png) + +生成的文档可以在对话中直接预览和下载。 + +### 3. 文档预览与下载 + +文档生成后，用户可以在问答页面中： + +- **查看文档内容**：在对话区域直接预览生成的文档 +- **检查生成效果**：确认文档内容是否符合需求 +- **继续修改**：要求智能体对文档进行修改或补充 +- **下载文档**：点击下载按钮保存到本地 + +![文档生成](./assets/start-chat/preview-docx.png) + +## 七、Mermaid 图表 + +当智能体生成 Mermaid 图表代码时，问答页面可以将其渲染为可视化图表。 + +### 支持的图表类型 + +可用于生成以下类型的图表： + +| 图表类型 | 说明 | +| ------------------------- | ---------------------- | +| 流程图 (Flowchart) | 展示流程和决策路径 | +| 时序图 (Sequence Diagram) | 展示对象之间的交互顺序 | +| 类图 (Class Diagram) | 展示类之间的结构和关系 | +| 状态图 (State Diagram) | 展示状态转换过程 | +| 甘特图 (Gantt) | 展示项目时间计划 | +| 思维导图 (Mindmap) | 展示主题的层级关系 | +| 实体关系图 (ER Diagram) | 展示数据实体之间的关系 | + +### 图表交互 + +生成后的图表支持以下交互： + +- **悬停放大**：鼠标悬停时显示放大按钮 +- **全屏查看**：点击放大按钮可全屏查看图表 +- **拖拽平移**：在全屏模式下可拖拽移动视图 +- **滚轮缩放**：支持滚轮缩放图表 +- **重置视图**：可重置缩放和位置 + +> **提示**：如果图表渲染失败，会显示原始代码并提示「图表无法渲染」。 + +![mermaid图表](./assets/start-chat/mermaid.png) + +## 八、对话交互 + +### 1. 刷新回答 + +当用户对当前回答不满意，或者希望智能体重新生成结果时，可以使用刷新功能。 + +**操作方式**： + +- 点击智能体回复下方的刷新按钮（带旋转箭头的图标） +- 或使用输入框重新发送相同的问题 + +**刷新效果**： + +- 重新提交当前问题 +- 重新执行智能体处理流程 +- 生成新的回答结果 + +> **注意**：刷新回答会消耗额外的 Token 资源。 + +![刷新对话](./assets/start-chat/refresh-chat.png) + +### 2. 复制回答 + +用户可以复制智能体的回复内容。 + +**操作方式**：点击回复下方的复制按钮（带剪贴板的图标）。 + +复制成功后按钮图标会短暂变为勾选状态。 + +### 3. 导出 Markdown + +用户可以将完整的对话内容导出为 Markdown 格式。 + +**操作方式**：点击回复下方「更多」菜单中的「导出 Markdown」选项。 + +### 4. 分享对话 + +支持用户生成分享链接，将对话分享给其他人查看。 + +**分享模式**： + +1. 点击对话标题旁的分享按钮进入分享模式 +2. 可以选择分享全部问答或选择特定的问答对 +3. 点击「复制链接」生成分享链接 +4. 分享链接会被复制到剪贴板 + +**分享内容**： + +- 被选中的用户问题和智能体回复 +- 智能体使用的来源引用 +- 相关图片（如果涉及） + +**分享页面特性**： + +- 分享页面为只读模式 +- 其他用户可以查看但无法继续对话 +- 来源面板仍可正常查看 + +![分享](./assets/start-chat/share.png) + +## 九、后台运行模式 -
- 知识引用来源 - 知识引用图片 -
+当智能体处理复杂任务或生成文件时，任务可能需要较长时间执行。用户可以离开当前对话页面，系统会继续在后台处理任务。 -## 🎭 多模态交互体验 +### 1. 继续执行任务 -### 图像处理功能 +用户离开当前对话页面后，智能体可以继续执行未完成的任务。 -Nexent支持图像输入和处理（需要配置视觉模型、图片解析工具`analyze_image`）： +- 任务会在服务器端继续运行 +- 不受用户关闭浏览器或切换页面影响 -1. **上传图像** - - 直接将图像文件拖拽到对话区域 - - 或点击上传按钮选择图像文件 - - 支持常见的图片格式（JPG、PNG、GIF等） +### 2. 返回查看结果 -2. **图像分析能力** - - 智能体会调用图片解析工具，自动分析图像内容 - - 可以识别图像中的物体、文字、场景等元素 - - 基于图像内容回答您的问题 +用户重新进入原对话后，可以查看： -> 💡 **提示**：Nexent 即将支持更加丰富的多模态交互模式，包括视频处理、音频分析等功能，敬请期待！ +- 已经生成的执行过程记录 +- 各步骤的执行状态 +- 最终生成的结果 -## ⚙️ 后台运行模式 +### 4. 停止执行 -### 多任务处理功能 +如果用户仍在当前页面，可以随时停止智能体的执行： -Nexent支持后台运行模式，让您在处理复杂任务时更加高效： +- 智能体执行时，发送按钮变为停止按钮（方形图标） +- 点击停止按钮可以中断当前的执行过程 +- 已执行完成的部分结果会保留 -1. **多任务并行** - - 在对话进行过程中，您可以切换到其他窗口或应用程序 - - 智能体会在后台继续处理您的任务 - - 不会因为切换窗口而中断处理 +## 十、快捷操作与导航 -2. **实时状态监控** - - 在左侧对话列表中，每个对话前都有状态指示器 - - 🟢 **绿色圆点**：表示对话正在进行中 - - 🔵 **蓝色圆点**：表示对话已执行完毕 - - 您可以随时点击对话查看处理进展 +### 1. 返回智能体列表 -3. **提升工作效率** - - 后台运行模式大大提升了您的工作效率 - - 可以在等待智能体处理的同时进行其他工作 - - 特别适合处理需要长时间分析或生成的任务 +在对话页面中，点击左上角的返回按钮可以返回智能体选择列表。 -## 🚀 开始您的 Nexent 之旅 +### 2. 切换到旧版界面 -恭喜！现在您已经掌握了Nexent的所有核心功能。期待您使用Nexent创造出令人惊艳的应用！ +左侧边栏底部提供「切换到旧版」入口，用户可以切换回旧版的对话界面。 +### 3. 侧边栏折叠 -### 获取帮助 +在桌面端，侧边栏支持折叠/展开操作。点击折叠按钮可以将对话历史区域收起，扩大主对话区域。 -如果您在使用过程中遇到任何问题： +在移动端，侧边栏默认折叠，点击展开按钮可临时展开。 -- 📖 查看 **[常见问题](../quick-start/faq)** 获取详细解答 -- 💬 加入我们的 [Discord 社区](https://discord.gg/tb5H3S3wyv) 与其他用户交流 -- 🆘 联系技术支持获取专业帮助 \ No newline at end of file +![折叠历史](./assets/start-chat/collapse.png) diff --git a/doc/docs/zh/user-guide/user-management.md b/doc/docs/zh/user-guide/user-management.md index ddffc1abe0..fabcf50df3 100644 --- a/doc/docs/zh/user-guide/user-management.md +++ b/doc/docs/zh/user-guide/user-management.md @@ -2,7 +2,7 @@ 本页面详细说明 Nexent 平台的用户角色体系、数据可见性范围、各类资源的操作权限，并分享权限配置的实践案例。 -⚠️ **重要提示**：首次部署 v1.8.0 及以上版本时，需特别留意 Docker 日志中输出的 `suadmin` 超级管理员账号信息。该账号为系统最高权限账户，密码仅在首次生成时显示，后续无法再次查看，请务必妥善保存。 +⚠️ **重要提示**：首次部署 v1.8.0 及以上版本时，系统会创建 `suadmin@nexent.com` 超级管理员账号，默认密码为 `Nexent@123`，创建成功后会在终端显示。可在首次部署前通过 `NEXENT_SUPER_ADMIN_PASSWORD` 覆盖；离线部署包使用 `--config` 时则以交互输入的密码为准，且不会显示手动输入的密码。 ## 📋 页面导航 @@ -39,7 +39,7 @@ Nexent 采用基于角色的访问控制（RBAC）模型，通过租户与用户 | 角色 | 职责描述 | 适用场景 | 角色备注 | | -------------- | ---------------------------------------------- | -------------------- | ------------------------------------------------------------ | -| **超级管理员** | 可创建**不同租户**，管理所有租户资源 | 平台运维人员 | Nexent系统只有一个超级管理员，于本地部署时生成账号密码，请务必留存，日志关闭后无法找回 | +| **超级管理员** | 可创建**不同租户**，管理所有租户资源 | 平台运维人员 | Nexent 系统只有一个超级管理员，首次部署默认创建，可通过部署环境变量预设密码 | | **管理员** | 负责**租户内**的资源管理和权限分配 | 部门经理、租户负责人 | 同一租户可拥有多个管理员，只能由超级管理员邀请 | | **开发者** | 可创建和编辑智能体、知识库等资源，但无管理权限 | 开发人员、产品经理 | 同一租户下可拥有多个开发者，可属于租户下多个用户组，由管理员和超级管理员邀请 | | **普通用户** | 仅可使用平台提供的各项功能，无创建和编辑权限 | 员工、业务人员 | 同一租户下可拥有多个普通用户，可属于租户下多个用户组，由管理员和超级管理员邀请 | @@ -322,4 +322,4 @@ Nexent 平台采用邀请码机制控制新用户注册，确保平台的安全 - 📖 查看 **[常见问题](../quick-start/faq)** 获取详细解答 - 💬 加入我们的 [Discord 社区](https://discord.gg/tb5H3S3wyv) 与其他用户交流 -- 🆘 联系技术支持获取专业帮助 \ No newline at end of file +- 🆘 联系技术支持获取专业帮助 diff --git a/doc/nl2agent-design-zh.md b/doc/nl2agent-design-zh.md new file mode 100644 index 0000000000..c617fb8ee1 --- /dev/null +++ b/doc/nl2agent-design-zh.md @@ -0,0 +1,166 @@ +# NL2Agent 临时智能体设计方案 + +## 1. 当前流程 + +NL2Agent runtime、接口和可复用前端组件继续保留，但智能体管理页 `/agents` 不再展示 NL2Agent 入口，也不再挂载生成助手面板。以下内容记录保留能力的既有流程与契约。 + +完整流程如下： + +1. 通过普通对话澄清智能体需求。 +2. 搜索当前租户已安装且可用的 MCP 工具。 +3. 渲染可选择的工具推荐卡。 +4. 用户确认工具后，通过 `updateTools()` 写入当前可编辑智能体。 +5. 将同一份已确认工具元数据作为下一轮 query 发送给 NL2Agent。 +6. 生成并渲染智能体 Draft 卡。 +7. 用户确认 Draft 后，通过 `updateAgentConfig()` 写入非工具字段。 +8. 通过现有智能体配置保存流程完成保存。 + +NL2Agent runtime 及其对话保持临时状态；可编辑智能体状态由现有智能体配置 store 管理。 + +## 2. 运行接口 + +前端通过现有流式 adapter 的 NL2Agent runtime 模式调用： + +```http +POST /agent/nl2agent/run +``` + +```json +{ + "query": "用户当前输入", + "history": [ + {"role": "user", "content": "..."}, + {"role": "assistant", "content": "..."} + ], + "minio_files": [] +} +``` + +确认推荐卡时，adapter 将 `metadata.custom.nl2agentToolSelection` 序列化为当前 query： + +```json +{ + "type": "nl2agent_tool_selection", + "tools": [ + { + "tool_id": 10, + "name": "weather_forecast", + "origin_name": "weather", + "description": "获取天气预报", + "source": "mcp", + "usage": "weather-server", + "labels": ["weather"], + "inputs": "{\"city\":\"string\"}" + } + ] +} +``` + +用户可见消息仍是本地化的选择摘要。请求不携带持久化的智能体 ID 或会话 ID。 + +## 3. 临时智能体与 MCP 搜索 + +每次请求都使用当前租户默认模型，在内存中构建名为 `__nl2agent_runtime__` 的 `AgentConfig`，并且只绑定内部 Local MCP 工具 `search_installed_mcp_tools`。 + +搜索工具负责： + +- 从经过鉴权的 MCP 请求中解析租户。 +- 只搜索已安装且可用的 MCP 工具。 +- 接收 1 到 10 个规范化能力关键词。 +- 最多返回五个确定性排序结果。 +- 只暴露安全展示元数据和原始 `inputs` schema。 + +Local MCP 工具只负责搜索目录。智能体草稿的编辑由前端配置 store 完成。 + +## 4. NL2A 结构化 Payload + +继续复用现有 `...` 提取逻辑和 `nl2a` SSE type。wrapper 内 JSON 使用 subtype 判别联合。 + +工具推荐成功： + +```json +{ + "subtype": "local_mcp_recommendation", + "status": "success", + "recommendation_count": 1, + "recommendations": [] +} +``` + +工具推荐失败： + +```json +{ + "subtype": "local_mcp_recommendation", + "status": "error", + "code": "tool_search_failed", + "retryable": true +} +``` + +智能体 Draft： + +```json +{ + "subtype": "agent_draft", + "name": "weather_assistant", + "display_name": "天气助手", + "description": "查询天气并提供出行建议", + "duty_prompt": "...", + "constraint_prompt": "...", + "few_shots_prompt": null +} +``` + +`GeneratedAgentDraft` 只包含 `updateAgentConfig()` 接受的字段，不重复携带已选择工具。 + +## 5. 前端状态写入 + +### 5.1 工具确认 + +推荐卡默认选择所有返回工具，并支持全选、部分选择和零工具确认。 + +确认时依次执行： + +1. 按卡片展示顺序筛选已选择推荐。 +2. 映射为 `Tool[]`，其中 `id = String(tool_id)`，`initParams = []`。 +3. 使用该集合调用 `updateTools()`。 +4. 将同一选择集合保存到消息 metadata 并启动下一轮请求。 +5. 将当前推荐卡设为只读。 + +当前推荐卡中已确认的选择是本次工具更新的唯一来源。 + +### 5.2 Draft 确认 + +Draft 卡只展示生成完成状态、智能体名称和简短介绍，不展示完整提示词。 + +确认时调用 `updateAgentConfig()` 写入： + +- `name` +- `display_name` +- `description` +- `duty_prompt` +- `constraint_prompt` +- `few_shots_prompt` + +`few_shots_prompt` 为 null 时转换为空字符串。之前确认的工具集合保持不变。 + +## 6. assistant-ui 映射 + +流式 adapter 将 `nl2a` SSE 内容解析到 `message.metadata.custom.nl2a`。`AssistantMessage` 在消息分组内容之后按 subtype 渲染： + +- `local_mcp_recommendation` 使用 `ToolRecommendations`。 +- `agent_draft` 使用 `AgentDraftCard`。 + +原始 MCP `execution_logs` 继续关联到对应工具调用。 + +## 7. 验证 + +后端测试覆盖中英文提示词契约、两个 subtype 以及精简后的 `GeneratedAgentDraft` schema。 + +前端验证覆盖： + +- 全选、部分选择和零工具确认会更新 `editedAgent.tools`。 +- 发送给 NL2Agent 的选择 metadata 与写入 store 的工具来自同一集合。 +- Draft 确认只更新非工具配置字段。 +- 现有智能体保存行为能够保存完整的可编辑配置。 diff --git a/doc/nl2agent-design.md b/doc/nl2agent-design.md new file mode 100644 index 0000000000..0d49fff19d --- /dev/null +++ b/doc/nl2agent-design.md @@ -0,0 +1,185 @@ +# NL2Agent Ephemeral Agent Design + +## 1. Current Flow + +The NL2Agent runtime, API, and reusable frontend components remain available, +but the `/agents` management page no longer exposes an NL2Agent entry point or +mounts the generation assistant panel. The following sections document the +retained flow and contracts. + +The flow is: + +1. Clarify the agent requirements through normal conversation. +2. Search the current tenant's installed and available MCP tools. +3. Render a selectable tool recommendation card. +4. Write the confirmed tool set to the current editable agent with + `updateTools()`. +5. Send the same confirmed tool metadata to NL2Agent as the next query. +6. Generate and render an agent draft card. +7. Write the confirmed non-tool fields with `updateAgentConfig()`. +8. Save through the existing agent configuration flow. + +The NL2Agent runtime and its conversation remain ephemeral. The editable agent +state is owned by the existing agent configuration store. + +## 2. Runtime API + +The frontend uses the existing stream adapter with NL2Agent runtime mode: + +```http +POST /agent/nl2agent/run +``` + +```json +{ + "query": "The user's current input", + "history": [ + {"role": "user", "content": "..."}, + {"role": "assistant", "content": "..."} + ], + "minio_files": [] +} +``` + +When a recommendation is confirmed, the adapter serializes +`metadata.custom.nl2agentToolSelection` as the current query: + +```json +{ + "type": "nl2agent_tool_selection", + "tools": [ + { + "tool_id": 10, + "name": "weather_forecast", + "origin_name": "weather", + "description": "Get weather forecasts", + "source": "mcp", + "usage": "weather-server", + "labels": ["weather"], + "inputs": "{\"city\":\"string\"}" + } + ] +} +``` + +The visible user message remains a localized selection summary. The request +does not contain a persisted agent or conversation ID. + +## 3. Runtime Agent and MCP Search + +Every request constructs an in-memory `AgentConfig` named +`__nl2agent_runtime__` with the current tenant's default model. It binds only +the internal `search_installed_mcp_tools` Local MCP tool. + +The search tool: + +- Resolves the tenant from the authenticated MCP request. +- Searches only installed, available MCP tools. +- Accepts 1 to 10 normalized capability keywords. +- Returns at most five deterministic matches. +- Exposes safe display metadata and the original `inputs` schema. + +The Local MCP tool only searches the catalog. Agent draft editing is performed +by the frontend configuration store. + +## 4. Structured NL2A Payloads + +The existing `...` extraction and `nl2a` SSE type are reused. +The JSON payload is a subtype-discriminated union. + +Tool recommendation success: + +```json +{ + "subtype": "local_mcp_recommendation", + "status": "success", + "recommendation_count": 1, + "recommendations": [] +} +``` + +Tool recommendation error: + +```json +{ + "subtype": "local_mcp_recommendation", + "status": "error", + "code": "tool_search_failed", + "retryable": true +} +``` + +Agent draft: + +```json +{ + "subtype": "agent_draft", + "name": "weather_assistant", + "display_name": "Weather Assistant", + "description": "Checks weather and provides travel advice", + "duty_prompt": "...", + "constraint_prompt": "...", + "few_shots_prompt": null +} +``` + +`GeneratedAgentDraft` contains only fields accepted by +`updateAgentConfig()`. Selected tools are not repeated in the draft payload. + +## 5. Frontend State Updates + +### 5.1 Tool Confirmation + +Recommendation cards select all returned tools by default and allow full, +partial, or zero-tool confirmation. + +On confirmation, the card: + +1. Filters recommendations in their displayed order. +2. Maps the selection to `Tool[]`, using `String(tool_id)` as `id` and + `initParams: []`. +3. Calls `updateTools()` with that exact set. +4. Stores the same selection in message metadata and starts the next run. +5. Becomes read-only. + +The recommendation card selection is the source of truth for this update. + +### 5.2 Draft Confirmation + +The agent draft card displays the generated agent name and description without +showing the complete prompts. + +On confirmation, it calls `updateAgentConfig()` with: + +- `name` +- `display_name` +- `description` +- `duty_prompt` +- `constraint_prompt` +- `few_shots_prompt` + +A null `few_shots_prompt` is normalized to an empty string. The previously +confirmed tool set remains unchanged. + +## 6. assistant-ui Mapping + +The stream adapter parses `nl2a` SSE content into +`message.metadata.custom.nl2a`. `AssistantMessage` renders after grouped +message parts: + +- `local_mcp_recommendation` as `ToolRecommendations`. +- `agent_draft` as `AgentDraftCard`. + +Raw MCP `execution_logs` remain attached to their tool call. + +## 7. Verification + +Backend tests cover the bilingual prompt contract, both subtype values, and +the reduced `GeneratedAgentDraft` schema. + +Frontend verification covers: + +- Full, partial, and zero-tool confirmation updates `editedAgent.tools`. +- The selection metadata sent to NL2Agent matches the stored tools. +- Draft confirmation updates only non-tool configuration fields. +- Existing agent save behavior persists the completed editable configuration. diff --git a/doc/procedural-memory-verification.md b/doc/procedural-memory-verification.md deleted file mode 100644 index ea9f532904..0000000000 --- a/doc/procedural-memory-verification.md +++ /dev/null @@ -1,315 +0,0 @@ -# Procedural Memory Verification Report - -## Summary -**Status: ⚠️ FULLY SUPPORTED but REQUIRES OPTIONAL DEPENDENCY** - -Procedural memory is a fully implemented feature in mem0ai version 0.1.117, **BUT it requires `langchain-core` to be installed separately**. Without this dependency, the feature will fail at runtime. - ---- - -## ⚠️ CRITICAL FINDING: Optional Dependency Required - -**Your colleague is partially correct.** The procedural memory code is NOT empty (it's 50 lines of real implementation), but it has a critical dependency issue: - -### The Problem - -The `_create_procedural_memory()` method contains: - -```python -try: - from langchain_core.messages.utils import convert_to_messages -except Exception: - logger.error( - "Import error while loading langchain-core. " - "Please install 'langchain-core' to use procedural memory." - ) - raise # ← Fails here if langchain-core not installed -``` - -### Reality Check - -| Aspect | Status | -|--------|--------| -| Code exists? | ✅ Yes, 50 lines of real implementation | -| Code is empty/stub? | ❌ No, it's fully implemented | -| Works out of the box? | ❌ **NO** - requires `langchain-core` package | -| Documented requirement? | ⚠️ Only in error message, not in main docs | - -### Why Your Colleague Thought It Was Empty - -1. They called `memory.add(..., memory_type="procedural_memory")` -2. Got `ImportError: No module named 'langchain_core'` -3. Saw the error and concluded "it doesn't work" or "it's empty" -4. This is understandable - the feature exists but is **disabled by default** - ---- - -## Verification Results - -### 1. API Support ✅ -The `memory_type` parameter is available in both `AsyncMemory.add()` and `Memory.add()`: - -```python -async def add( - self, - messages, - *, - user_id: Optional[str] = None, - agent_id: Optional[str] = None, - run_id: Optional[str] = None, - metadata: Optional[Dict[str, Any]] = None, - infer: bool = True, - memory_type: Optional[str] = None, # ✅ SUPPORTED - prompt: Optional[str] = None, - llm=None -) -``` - -### 2. MemoryType Enum ✅ -Located in `mem0.configs.enums.MemoryType`: - -```python -class MemoryType(Enum): - SEMANTIC = "semantic_memory" - EPISODIC = "episodic_memory" - PROCEDURAL = "procedural_memory" # ✅ AVAILABLE -``` - -### 3. Implementation ✅ -The `_create_procedural_memory()` method exists in both `AsyncMemory` and `Memory` classes: - -**AsyncMemory signature:** -```python -async def _create_procedural_memory( - self, - messages, - metadata=None, - llm=None, - prompt=None -) -``` - -**Memory (sync) signature:** -```python -def _create_procedural_memory( - self, - messages, - metadata=None, - prompt=None -) -``` - -### 4. Validation Logic ✅ -The `add()` method validates `memory_type` and enforces constraints: - -```python -# Only "procedural_memory" is accepted -if memory_type is not None and memory_type != MemoryType.PROCEDURAL.value: - raise ValueError( - f"Invalid 'memory_type'. Please pass {MemoryType.PROCEDURAL.value} " - "to create procedural memories." - ) - -# agent_id is REQUIRED for procedural memory -if agent_id is not None and memory_type == MemoryType.PROCEDURAL.value: - results = await self._create_procedural_memory( - messages, metadata=processed_metadata, prompt=prompt, llm=llm - ) - return results -``` - -### 5. System Prompt ✅ -A comprehensive 5,100-character system prompt exists in `mem0.configs.prompts.PROCEDURAL_MEMORY_SYSTEM_PROMPT`: - -**Purpose:** Records and preserves complete interaction history between human and AI agent - -**Structure:** -- Overview (Global Metadata) - - Task Objective - - Progress Status -- Sequential Agent Actions (Numbered Steps) - - Agent Action - - Action Result (Mandatory, Unmodified) - - Embedded Metadata (Key Findings, Navigation History, Errors, Current Context) - -**Key Guidelines:** -1. Preserve every output verbatim -2. Maintain chronological order -3. Include exact data (URLs, element indexes, error messages, JSON responses) -4. Output only the structured summary - ---- - -## Usage Example - -```python -from mem0 import AsyncMemory - -# Initialize memory -memory = await AsyncMemory.from_config(config) - -# Create procedural memory -messages = [ - {"role": "user", "content": "Search for AI news"}, - {"role": "assistant", "content": "I'll search for recent AI news..."}, - # ... more conversation history -] - -result = await memory.add( - messages=messages, - user_id="user_123", - agent_id="research_agent", # ⚠️ REQUIRED for procedural memory - memory_type="procedural_memory", - metadata={ - "task": "AI news research", - "session_id": "session_456" - } -) - -# Result format: -# { -# "results": [ -# { -# "id": "memory_id_here", -# "memory": "## Summary of the agent's execution history...", -# "event": "ADD" -# } -# ] -# } -``` - ---- - -## Requirements & Constraints - -### Required Parameters -- ✅ `agent_id`: **MUST** be provided when using `memory_type="procedural_memory"` -- ✅ `metadata`: **MUST** be provided (cannot be None) -- ✅ `messages`: List of conversation messages to summarize - -### Optional Parameters -- `prompt`: Custom prompt to override default `PROCEDURAL_MEMORY_SYSTEM_PROMPT` -- `llm`: Custom LangChain ChatModel (async version only) - -### Validation Rules -1. `memory_type` must be exactly `"procedural_memory"` (or None) -2. If `memory_type="procedural_memory"` is set, `agent_id` must be provided -3. `metadata` cannot be None for procedural memories - ---- - -## Implementation Details - -### How It Works -1. **Validation**: Checks `memory_type` and required parameters -2. **Prompt Construction**: Uses default or custom system prompt -3. **LLM Summarization**: Calls LLM to generate comprehensive execution summary -4. **Embedding**: Generates embedding for the summary -5. **Storage**: Stores in vector database with `metadata["memory_type"] = "procedural_memory"` -6. **Return**: Returns memory ID and summary text - -### Async vs Sync -- **AsyncMemory**: Supports custom LangChain `llm` parameter -- **Memory**: Uses internal LLM from config only - ---- - -## Integration with Nexent - -### Current Status -The Nexent codebase does **NOT** currently use procedural memory. The `memory_type` parameter is not passed in any `add_memory()` calls. - -### Recommended Integration Points - -1. **Agent Service** (`backend/services/agent_service.py`): - - Detect when agent completes a multi-step task - - Call `add_memory_in_levels()` with `memory_type="procedural_memory"` - - Pass the full conversation history as messages - -2. **Memory Service** (`sdk/nexent/memory/memory_service.py`): - - Add `memory_type` parameter to `add_memory()` and `add_memory_in_levels()` - - Pass through to mem0's `add()` method - -3. **Agent Run Info** (`sdk/nexent/core/agents/agent_model.py`): - - Add `memory_type` field to track if current run should create procedural memory - -### Example Integration - -```python -# In agent_service.py, after agent completes a complex task -if task_complexity >= threshold: # Your logic here - await add_memory_in_levels( - messages=conversation_history, - memory_config=memory_ctx.memory_config, - tenant_id=memory_ctx.tenant_id, - user_id=memory_ctx.user_id, - agent_id=memory_ctx.agent_id, - memory_levels=["agent", "user_agent"], - memory_type="procedural_memory", # ✅ NEW PARAMETER - metadata={ - "task_type": "complex_research", - "duration_seconds": duration, - "steps_completed": step_count - } - ) -``` - ---- - -## Conclusion - -Procedural memory is a **fully functional feature** in mem0ai==0.1.117, **BUT it requires an optional dependency**. It provides: - -- ✅ Complete API support -- ✅ Comprehensive system prompt (5,100 characters) -- ✅ Proper validation and error handling -- ✅ Both sync and async implementations -- ✅ Integration with existing memory infrastructure -- ⚠️ **REQUIRES `langchain-core` package to be installed** - -### The Truth About "Empty Function" Claims - -**The code is NOT empty.** It's a 50-line implementation that: -1. Calls LLM to generate execution summary -2. Creates embeddings -3. Stores in vector database -4. Returns proper results - -**However, it fails at runtime** if `langchain-core` is not installed, which is why your colleague might have thought it was a no-op. - -### How to Enable - -**Option 1: Install the dependency** -```bash -pip install langchain-core -``` - -**Option 2: Add to Nexent's dependencies** -```toml -# In sdk/pyproject.toml -dependencies = [ - # ... existing deps ... - "langchain-core>=0.1.0", # Required for procedural memory -] -``` - -**Option 3: Make it optional with fallback** -```python -try: - result = await memory.add(..., memory_type="procedural_memory") -except ImportError as e: - if "langchain-core" in str(e): - logger.warning("Procedural memory requires langchain-core. Using regular memory.") - result = await memory.add(...) # Fallback - else: - raise -``` - -### Final Recommendation - -This feature **can be integrated into Nexent**, but you must: -1. Add `langchain-core` to dependencies, OR -2. Implement graceful fallback when dependency is missing, OR -3. Document it as an optional feature requiring extra installation - -Without addressing the dependency issue, procedural memory will fail at runtime despite having complete implementation code. diff --git a/frontend/app/[locale]/agent-space/components/AgentRepositoryCard.tsx b/frontend/app/[locale]/agent-space/components/AgentRepositoryCard.tsx index 1ea42459a9..2d1f174aa3 100644 --- a/frontend/app/[locale]/agent-space/components/AgentRepositoryCard.tsx +++ b/frontend/app/[locale]/agent-space/components/AgentRepositoryCard.tsx @@ -39,7 +39,7 @@ export function AgentRepositoryCard({ ? [ { key: "takeDown", - label: t("agentRepository.mine.reviewModal.takeDown"), + label: t("repository.listingStatus.takeDown"), icon: , danger: true, disabled: isTakingDown, diff --git a/frontend/app/[locale]/agent-space/components/AgentRepositoryCopyDialog.tsx b/frontend/app/[locale]/agent-space/components/AgentRepositoryCopyDialog.tsx index 89dffcbc6a..47814c14d7 100644 --- a/frontend/app/[locale]/agent-space/components/AgentRepositoryCopyDialog.tsx +++ b/frontend/app/[locale]/agent-space/components/AgentRepositoryCopyDialog.tsx @@ -196,7 +196,7 @@ export function AgentRepositoryCopyDialog({ {t("agentRepository.copy.loadError")}

) : precheck ? ( diff --git a/frontend/app/[locale]/agent-space/components/AgentRepositoryDetailModal.tsx b/frontend/app/[locale]/agent-space/components/AgentRepositoryDetailModal.tsx index 3a76312a7b..0d0f620704 100644 --- a/frontend/app/[locale]/agent-space/components/AgentRepositoryDetailModal.tsx +++ b/frontend/app/[locale]/agent-space/components/AgentRepositoryDetailModal.tsx @@ -109,7 +109,7 @@ function AgentRepositoryDetailError({ {t("agentRepository.detail.loadError")}

); diff --git a/frontend/app/[locale]/agent-space/components/AgentRepositoryReviewConfirmModal.tsx b/frontend/app/[locale]/agent-space/components/AgentRepositoryReviewConfirmModal.tsx new file mode 100644 index 0000000000..0337169031 --- /dev/null +++ b/frontend/app/[locale]/agent-space/components/AgentRepositoryReviewConfirmModal.tsx @@ -0,0 +1,99 @@ +"use client"; + +import { useEffect, useState } from "react"; +import { Input, Modal } from "antd"; +import { useTranslation } from "react-i18next"; +import type { AgentRepositoryListingItem } from "@/types/agentRepository"; + +export type AgentRepositoryReviewAction = "approve" | "reject"; + +interface AgentRepositoryReviewConfirmModalProps { + open: boolean; + action: AgentRepositoryReviewAction | null; + listing: AgentRepositoryListingItem | null; + loading?: boolean; + onClose: () => void; + onConfirm: (content?: string) => Promise; +} + +function getListingTitle(listing: AgentRepositoryListingItem) { + return listing.display_name?.trim() || listing.name?.trim() || ""; +} + +export function AgentRepositoryReviewConfirmModal({ + open, + action, + listing, + loading = false, + onClose, + onConfirm, +}: AgentRepositoryReviewConfirmModalProps) { + const { t } = useTranslation("common"); + const [reviewOpinion, setReviewOpinion] = useState(""); + + useEffect(() => { + if (!open) { + setReviewOpinion(""); + } + }, [open]); + + if (!action || !listing) { + return null; + } + + const isApprove = action === "approve"; + const title = + getListingTitle(listing) || t("agentRepository.card.untitled"); + + const handleOk = async () => { + const trimmed = reviewOpinion.trim(); + await onConfirm(trimmed || undefined); + }; + + return ( + +
+

+ {isApprove + ? t("repository.review.confirmApproveContent", { name: title }) + : t("repository.review.confirmRejectContent", { name: title })} +

+
+ + setReviewOpinion(event.target.value)} + placeholder={t("repository.review.reviewOpinionPlaceholder")} + rows={4} + disabled={loading} + /> +
+
+ + ); +} diff --git a/frontend/app/[locale]/agent-space/components/MineAgentsView.tsx b/frontend/app/[locale]/agent-space/components/MineAgentsView.tsx index 00d3b3f4e9..42d6fb9b85 100644 --- a/frontend/app/[locale]/agent-space/components/MineAgentsView.tsx +++ b/frontend/app/[locale]/agent-space/components/MineAgentsView.tsx @@ -1,6 +1,6 @@ "use client"; -import { useState } from "react"; +import { useEffect, useRef, useState } from "react"; import { useParams, useRouter } from "next/navigation"; import { useMutation, useQueryClient } from "@tanstack/react-query"; import { App, Button, Empty, Input, Spin } from "antd"; @@ -16,15 +16,16 @@ import { useUpdateAgentRepositoryStatus, } from "@/hooks/agentRepository/useAgentRepositoryListings"; import { - parseAgentImportFile, - selectFile, + openImportWizardWithFile, type ImportAgentData, } from "@/lib/agentImportUtils"; import log from "@/lib/logger"; import { isCancelableRepositoryStatus, isTakeDownableRepositoryStatus, + findRepositoryInfoById, pickReviewDisplayRepositoryInfo, + resolveReviewModalMode, } from "@/lib/agentRepositoryMine"; import { isNewAgentPaddingItem, @@ -46,6 +47,11 @@ const MINE_OWNERSHIP_FILTERS: MineOwnershipFilter[] = [ "others", ]; +export interface ReviewDeepLinkTarget { + agentRepositoryId: number; + agentId: number; +} + interface MineAgentsViewProps { agents: MyEditableAgentListItem[]; counts: MyEditableAgentOwnershipCounts; @@ -62,6 +68,10 @@ interface MineAgentsViewProps { isFetching: boolean; onRetry: () => void; onViewDetail: (agentId: number, versionNo: number) => void; + reviewDeepLink?: ReviewDeepLinkTarget | null; + deepLinkFallbackAgent?: MyEditableAgentItem | null; + deepLinkFallbackLoading?: boolean; + onReviewDeepLinkConsumed?: () => void; } export function MineAgentsView({ @@ -80,6 +90,10 @@ export function MineAgentsView({ isFetching, onRetry, onViewDetail, + reviewDeepLink = null, + deepLinkFallbackAgent = null, + deepLinkFallbackLoading = false, + onReviewDeepLinkConsumed, }: MineAgentsViewProps) { const { t } = useTranslation("common"); const { message } = App.useApp(); @@ -103,6 +117,7 @@ export function MineAgentsView({ const [applyModalOpen, setApplyModalOpen] = useState(false); const [applyModalAgent, setApplyModalAgent] = useState(null); + const consumedDeepLinkRef = useRef(null); const createListingMutation = useCreateAgentRepositoryListing(); const updateStatusMutation = useUpdateAgentRepositoryStatus(); @@ -117,22 +132,15 @@ export function MineAgentsView({ }; const handleImportAgent = async () => { - const file = await selectFile(".json"); - if (!file) return; - - const agentData = await parseAgentImportFile(file, { - onParseError: (msgKey) => message.error(t(msgKey)), - onValidationError: (msgKey) => message.error(t(msgKey)), - onGenericError: (error) => { - log.error("Failed to read import file:", error); - message.error(t("businessLogic.config.error.agentImportFailed")); + await openImportWizardWithFile({ + onSuccess: (agentData) => { + setImportWizardData(agentData); + setImportWizardVisible(true); }, + message: message, + t: t, + log: log, }); - - if (!agentData) return; - - setImportWizardData(agentData); - setImportWizardVisible(true); }; const handleEdit = (agentId: number, permission?: MyEditableAgentItem["permission"]) => { @@ -221,15 +229,11 @@ export function MineAgentsView({ payload, }); message.success( - t("agentRepository.mine.applySuccess", { - name: - applyModalAgent.name?.trim() || - t("agentRepository.card.untitled"), - }) + t("repository.mine.applySuccess") ); closeApplyModal(); } catch { - message.error(t("agentRepository.mine.applyError")); + message.error(t("repository.mine.applyError")); } finally { setApplyingAgentId(null); } @@ -245,12 +249,81 @@ export function MineAgentsView({ if (!repositoryInfo) { return; } + openReviewModal(agent, repositoryInfo, mode); + }; + + const openReviewModal = ( + agent: MyEditableAgentItem, + repositoryInfo: MyAgentRepositoryInfoItem, + mode: "review" | "reviewUpdate" + ) => { setReviewModalAgent(agent); setReviewModalInfo(repositoryInfo); setReviewModalMode(mode); setReviewModalOpen(true); }; + useEffect(() => { + if (!reviewDeepLink) { + consumedDeepLinkRef.current = null; + return; + } + + if (consumedDeepLinkRef.current === reviewDeepLink.agentRepositoryId) { + return; + } + + const listStillLoading = isLoading; + const fallbackStillLoading = deepLinkFallbackLoading; + if (listStillLoading && fallbackStillLoading) { + return; + } + + const agentFromList = agents.find( + (item): item is MyEditableAgentItem => + !isNewAgentPaddingItem(item) && item.agent_id === reviewDeepLink.agentId + ); + const agent = agentFromList ?? deepLinkFallbackAgent; + + if (!agent) { + if (listStillLoading || fallbackStillLoading) { + return; + } + message.error(t("notifications.deepLink.agentNotFound")); + consumedDeepLinkRef.current = reviewDeepLink.agentRepositoryId; + onReviewDeepLinkConsumed?.(); + return; + } + + const repositoryInfo = findRepositoryInfoById( + agent.repository_info ?? [], + reviewDeepLink.agentRepositoryId + ); + + if (!repositoryInfo) { + message.error(t("notifications.deepLink.agentNotFound")); + consumedDeepLinkRef.current = reviewDeepLink.agentRepositoryId; + onReviewDeepLinkConsumed?.(); + return; + } + + openReviewModal( + agent, + repositoryInfo, + resolveReviewModalMode(agent, repositoryInfo) + ); + consumedDeepLinkRef.current = reviewDeepLink.agentRepositoryId; + onReviewDeepLinkConsumed?.(); + }, [ + agents, + deepLinkFallbackAgent, + deepLinkFallbackLoading, + isLoading, + onReviewDeepLinkConsumed, + reviewDeepLink, + t, + ]); + const handleSetNotShared = async () => { if (!reviewModalInfo) { return; @@ -272,24 +345,24 @@ export function MineAgentsView({ }); message.success( wasShared - ? t("agentRepository.mine.takeDownSuccess") - : t("agentRepository.mine.cancelApplySuccess") + ? t("repository.mine.takeDownSuccess") + : t("repository.mine.cancelApplySuccess") ); closeReviewModal(); } catch { message.error( wasShared - ? t("agentRepository.mine.takeDownError") - : t("agentRepository.mine.cancelApplyError") + ? t("repository.mine.takeDownError") + : t("repository.mine.cancelApplyError") ); throw new Error("Update repository status failed"); } }; const ownershipLabelKey: Record = { - all: "agentRepository.mine.filter.all", - created: "agentRepository.mine.filter.created", - others: "agentRepository.mine.filter.others", + all: "repository.mine.filter.all", + created: "repository.mine.filter.created", + others: "repository.mine.filter.others", }; const hasActiveFilter = ownership !== "all" || normalizedQuery.length > 0; @@ -365,7 +438,7 @@ export function MineAgentsView({ {t("agentRepository.mine.loadError")}

) : showFilteredEmpty ? ( @@ -421,7 +494,7 @@ export function MineAgentsView({ className="flex size-9 items-center justify-center rounded-lg p-0" disabled={page <= 1} onClick={() => onPageChange(Math.max(1, page - 1))} - aria-label={t("agentRepository.mine.pagination.prev")} + aria-label={t("repository.pagination.prev")} > @@ -432,7 +505,7 @@ export function MineAgentsView({ type={pageNumber === page ? "primary" : "default"} className="flex size-9 items-center justify-center rounded-lg p-0" onClick={() => onPageChange(pageNumber)} - aria-label={t("agentRepository.mine.pagination.page", { + aria-label={t("repository.pagination.page", { page: pageNumber, })} aria-current={pageNumber === page ? "page" : undefined} @@ -446,7 +519,7 @@ export function MineAgentsView({ className="flex size-9 items-center justify-center rounded-lg p-0" disabled={page >= totalPages} onClick={() => onPageChange(Math.min(totalPages, page + 1))} - aria-label={t("agentRepository.mine.pagination.next")} + aria-label={t("repository.pagination.next")} > diff --git a/frontend/app/[locale]/agent-space/components/MineApplyListingModal.tsx b/frontend/app/[locale]/agent-space/components/MineApplyListingModal.tsx index 7bc8156eb9..a3fbbccb1f 100644 --- a/frontend/app/[locale]/agent-space/components/MineApplyListingModal.tsx +++ b/frontend/app/[locale]/agent-space/components/MineApplyListingModal.tsx @@ -50,6 +50,7 @@ export function MineApplyListingModal({ const [iconError, setIconError] = useState(null); const [presetDropdownOpen, setPresetDropdownOpen] = useState(false); const [selectedTags, setSelectedTags] = useState([]); + const [listingContent, setListingContent] = useState(""); const agentId = agent?.agent_id; const { @@ -113,6 +114,7 @@ export function MineApplyListingModal({ if (!agent || !isListingsSuccess) { clearIconState(); setSelectedTags([]); + setListingContent(""); return; } @@ -127,6 +129,7 @@ export function MineApplyListingModal({ if (!prefill) { clearIconState(); setSelectedTags([]); + setListingContent(""); return; } @@ -138,6 +141,7 @@ export function MineApplyListingModal({ } setSelectedTags(prefill.tags); + setListingContent(""); }, [ open, agent, @@ -226,6 +230,7 @@ export function MineApplyListingModal({ onSubmit({ icon: selectedIcon, tags, + content: listingContent.trim(), }); }; @@ -269,11 +274,19 @@ export function MineApplyListingModal({ } maxLength={MAX_ICON_LENGTH} status={iconError ? "error" : undefined} - className="w-[5.25rem] shrink-0 text-2xl" + className="!h-[3.75rem] !w-[6.5rem] shrink-0 !text-4xl" styles={{ + root: { + display: "inline-flex", + alignItems: "center", + paddingBlock: 0, + }, input: { paddingInline: 2, + paddingBlock: 0, textAlign: "center", + fontSize: "2.25rem", + lineHeight: 1, }, }} suffix={ @@ -324,6 +337,20 @@ export function MineApplyListingModal({ })}

+ +
+

+ {t("repository.mine.applyModal.content")} +

+ setListingContent(event.target.value)} + rows={4} + placeholder={t( + "repository.mine.applyModal.contentPlaceholder" + )} + /> +
diff --git a/frontend/app/[locale]/agent-space/components/MineReviewStatusModal.tsx b/frontend/app/[locale]/agent-space/components/MineReviewStatusModal.tsx index f7bd67a76e..20381a9df7 100644 --- a/frontend/app/[locale]/agent-space/components/MineReviewStatusModal.tsx +++ b/frontend/app/[locale]/agent-space/components/MineReviewStatusModal.tsx @@ -46,12 +46,13 @@ export function MineReviewStatusModal({ const canTakeDown = isTakeDownableRepositoryStatus(repositoryInfo.status); const versionLabel = formatRepositoryVersionLabel(repositoryInfo); const submittedAt = formatMineDate(repositoryInfo.create_time); + const listingContent = repositoryInfo.content?.trim() ?? ""; const statusConfig = isPending ? { icon: Clock, - label: t("agentRepository.mine.reviewModal.pendingLabel"), - description: t("agentRepository.mine.reviewModal.pendingDescription"), + label: t("repository.listingStatus.pendingLabel"), + description: t("repository.listingStatus.pendingDescription"), tone: "border-amber-200 bg-amber-50 text-amber-800 dark:border-amber-500/30 dark:bg-amber-500/10 dark:text-amber-200", iconClass: "text-amber-600 dark:text-amber-300", @@ -59,16 +60,16 @@ export function MineReviewStatusModal({ : isRejected ? { icon: XCircle, - label: t("agentRepository.mine.reviewModal.rejectedLabel"), - description: t("agentRepository.mine.reviewModal.rejectedDescription"), + label: t("repository.listingStatus.rejectedLabel"), + description: t("repository.listingStatus.rejectedDescription"), tone: "border-red-200 bg-red-50 text-red-800 dark:border-red-500/30 dark:bg-red-500/10 dark:text-red-200", iconClass: "text-red-600 dark:text-red-300", } : { icon: CheckCircle2, - label: t("agentRepository.mine.reviewModal.sharedLabel"), - description: t("agentRepository.mine.reviewModal.sharedDescription"), + label: t("repository.listingStatus.listedLabel"), + description: t("repository.listingStatus.listedDescription"), tone: "border-emerald-200 bg-emerald-50 text-emerald-800 dark:border-emerald-500/30 dark:bg-emerald-500/10 dark:text-emerald-200", iconClass: "text-emerald-600 dark:text-emerald-300", @@ -78,15 +79,15 @@ export function MineReviewStatusModal({ const modalTitle = mode === "reviewUpdate" ? t("agentRepository.mine.reviewModal.reviewUpdateTitle") - : t("agentRepository.mine.reviewModal.title"); + : t("repository.listingStatus.title"); const confirmCancelApply = () => { Modal.confirm({ - title: t("agentRepository.mine.reviewModal.confirmCancelApplyTitle"), - content: t("agentRepository.mine.reviewModal.confirmCancelApplyContent", { + title: t("repository.listingStatus.confirmCancelApplyTitle"), + content: t("repository.listingStatus.confirmCancelApplyContent", { name: title, }), - okText: t("agentRepository.mine.reviewModal.cancelApply"), + okText: t("repository.listingStatus.cancelApply"), cancelText: t("common.cancel"), okButtonProps: { danger: true }, onOk: async () => { @@ -101,11 +102,11 @@ export function MineReviewStatusModal({ const confirmTakeDown = () => { Modal.confirm({ - title: t("agentRepository.mine.reviewModal.confirmTakeDownTitle"), - content: t("agentRepository.mine.reviewModal.confirmTakeDownContent", { + title: t("repository.listingStatus.confirmTakeDownTitle"), + content: t("repository.listingStatus.confirmTakeDownContent", { name: title, }), - okText: t("agentRepository.mine.reviewModal.takeDown"), + okText: t("repository.listingStatus.takeDown"), cancelText: t("common.cancel"), okButtonProps: { danger: true }, onOk: async () => { @@ -134,7 +135,7 @@ export function MineReviewStatusModal({ icon={} onClick={confirmCancelApply} > - {t("agentRepository.mine.reviewModal.cancelApply")} + {t("repository.listingStatus.cancelApply")} ) : null} {canTakeDown ? ( @@ -144,7 +145,7 @@ export function MineReviewStatusModal({ icon={} onClick={confirmTakeDown} > - {t("agentRepository.mine.reviewModal.takeDown")} + {t("repository.listingStatus.takeDown")} ) : null} @@ -174,6 +175,17 @@ export function MineReviewStatusModal({

{statusConfig.description}

+ {listingContent ? ( +

+ {isPending + ? t("repository.listingStatus.pendingNote", { + content: listingContent, + }) + : t("repository.listingStatus.reviewOpinion", { + content: listingContent, + })} +

+ ) : null} @@ -186,7 +198,7 @@ export function MineReviewStatusModal({ {submittedAt ? (
- {t("agentRepository.mine.reviewModal.submittedAt")} + {t("repository.listingStatus.submittedAt")} {submittedAt} diff --git a/frontend/app/[locale]/agent-space/components/MyAgentCard.tsx b/frontend/app/[locale]/agent-space/components/MyAgentCard.tsx index 63ff1773b6..7caa1e52a4 100644 --- a/frontend/app/[locale]/agent-space/components/MyAgentCard.tsx +++ b/frontend/app/[locale]/agent-space/components/MyAgentCard.tsx @@ -111,7 +111,7 @@ export function MyAgentCard({ key: "delete", danger: true, icon: , - label: t("agentRepository.mine.menu.delete"), + label: t("common.delete"), disabled: isDeleting, onClick: onDelete, }); @@ -229,15 +229,7 @@ export function MyAgentCard({ {t("agentRepository.mine.view")} )} - + {/* Evaluate button hidden: agent evaluation feature temporarily disabled */}
diff --git a/frontend/app/[locale]/agent-space/components/ReviewAgentList.tsx b/frontend/app/[locale]/agent-space/components/ReviewAgentList.tsx index d6b5e51299..016a1039a1 100644 --- a/frontend/app/[locale]/agent-space/components/ReviewAgentList.tsx +++ b/frontend/app/[locale]/agent-space/components/ReviewAgentList.tsx @@ -6,7 +6,8 @@ import { useTranslation } from "react-i18next"; import type { TFunction } from "i18next"; import type { AgentRepositoryListingItem } from "@/types/agentRepository"; -const GRID_COLS = "grid-cols-[minmax(0,2fr)_120px_160px_280px]"; +const GRID_COLS = + "grid-cols-[minmax(0,2fr)_120px_160px_minmax(0,1.5fr)_280px]"; interface ReviewAgentListProps { listings: AgentRepositoryListingItem[]; @@ -35,13 +36,13 @@ function getSubmitterDisplay( ) { const trimmed = submittedBy?.trim(); if (!trimmed) { - return t("agentRepository.review.unknownSubmitter"); + return t("repository.review.unknownSubmitter"); } if ( currentUserEmail && trimmed.toLowerCase() === currentUserEmail.toLowerCase() ) { - return t("agentRepository.review.me"); + return t("repository.review.me"); } return trimmed; } @@ -60,12 +61,13 @@ export function ReviewAgentList({
- {t("agentRepository.review.column.agent")} - {t("agentRepository.review.column.version")} - {t("agentRepository.review.column.submitter")} - {t("agentRepository.review.column.actions")} + {t("repository.review.column.name")} + {t("repository.review.column.version")} + {t("repository.review.column.submitter")} + {t("repository.review.column.listingNote")} + {t("repository.review.column.actions")}
    @@ -75,17 +77,18 @@ export function ReviewAgentList({ updatingRepositoryId === listing.agent_repository_id; const versionLabel = listing.version_label?.trim() || - t("agentRepository.review.noVersion"); + t("repository.review.noVersion"); const submitter = getSubmitterDisplay( listing.submitted_by, currentUserEmail, t ); + const listingNote = listing.content?.trim() || "—"; return (
  • @@ -108,6 +111,13 @@ export function ReviewAgentList({ {submitter}
    +
    + {listingNote} +
    +
    • diff --git a/frontend/app/[locale]/agent-space/page.tsx b/frontend/app/[locale]/agent-space/page.tsx index e7b4ca50a1..eb03a29f3f 100644 --- a/frontend/app/[locale]/agent-space/page.tsx +++ b/frontend/app/[locale]/agent-space/page.tsx @@ -1,7 +1,7 @@ "use client"; -import { useEffect, useMemo, useState } from "react"; -import { useSearchParams } from "next/navigation"; +import { useEffect, useMemo, useState, useCallback } from "react"; +import { useParams, useRouter, useSearchParams } from "next/navigation"; import { App, Button, @@ -31,12 +31,18 @@ import { type AgentDetailModalData, } from "@/lib/agentRepositoryDetail"; import type { AgentRepositoryListingItem, MineOwnershipFilter } from "@/types/agentRepository"; +import { isNewAgentPaddingItem } from "@/types/agentRepository"; +import { parseReviewDeepLinkParams } from "@/lib/notificationNavigation"; import { cn } from "@/lib/utils"; import { AgentRepositoryCard } from "./components/AgentRepositoryCard"; import { AgentRepositoryCopyDialog } from "./components/AgentRepositoryCopyDialog"; import { AgentRepositoryDetailModal } from "./components/AgentRepositoryDetailModal"; import { MineAgentsView } from "./components/MineAgentsView"; import { ReviewAgentList } from "./components/ReviewAgentList"; +import { + AgentRepositoryReviewConfirmModal, + type AgentRepositoryReviewAction, +} from "./components/AgentRepositoryReviewConfirmModal"; enum AgentRepositoryTab { REPOSITORY = "repository", @@ -60,6 +66,9 @@ export default function AgentRepositoryPage() { const { t } = useTranslation("common"); const { pageVariants, pageTransition } = useSetupFlow(); const searchParams = useSearchParams(); + const router = useRouter(); + const params = useParams<{ locale: string }>(); + const locale = params.locale || "en"; const { user } = useAuthorizationContext(); const isAdmin = user?.role === USER_ROLES.ADMIN; @@ -101,6 +110,15 @@ export default function AgentRepositoryPage() { const isReviewTab = tab === AgentRepositoryTab.REVIEW; const isMineTab = tab === AgentRepositoryTab.MINE; + const reviewDeepLink = useMemo( + () => parseReviewDeepLinkParams(searchParams), + [searchParams] + ); + + const handleReviewDeepLinkConsumed = useCallback(() => { + router.replace(`/${locale}/agent-space?tab=mine`); + }, [locale, router]); + const listingParams = useMemo( () => ({ status: "shared" as const, @@ -140,6 +158,20 @@ export default function AgentRepositoryPage() { refetch: refetchMine, } = useMyEditableAgents(mineListParams, isMineTab); + const { + data: deepLinkMineData, + isLoading: isDeepLinkMineLoading, + } = useMyEditableAgents( + { + ownership: "all", + agent_id: reviewDeepLink?.agentId, + page: 1, + page_size: 1, + new_agent_padding: false, + }, + isMineTab && reviewDeepLink != null + ); + const { data: mineCountData } = useMyEditableAgents( { page: 1, page_size: 1, ownership: "all" }, true @@ -169,20 +201,6 @@ export default function AgentRepositoryPage() { const updateStatusMutation = useUpdateAgentRepositoryStatus(); - useEffect(() => { - const refreshActiveTab = async () => { - if (tab === AgentRepositoryTab.REPOSITORY) { - await refetch(); - } else if (tab === AgentRepositoryTab.MINE) { - await refetchMine(); - } else if (tab === AgentRepositoryTab.REVIEW) { - await refetchReview(); - } - }; - - refreshActiveTab().catch(() => {}); - }, [tab, refetch, refetchMine, refetchReview]); - const detailOpen = detailSource !== null; const selectedRepositoryId = detailSource?.kind === "repository" ? detailSource.agentRepositoryId : null; @@ -301,6 +319,13 @@ export default function AgentRepositoryPage() { const mineCounts = mineData?.counts ?? { all: 0, created: 0, others: 0 }; const minePagination = mineData?.pagination; const mineTotal = minePagination?.total ?? 0; + const deepLinkFallbackAgent = useMemo(() => { + const item = deepLinkMineData?.items?.[0]; + if (!item || isNewAgentPaddingItem(item)) { + return null; + } + return item; + }, [deepLinkMineData]); const repositoryTabCount = repositoryCountData?.pagination?.total ?? 0; const mineTabCount = mineCountData?.counts?.all ?? 0; const pendingReviewCount = reviewCountData?.pagination?.total ?? 0; @@ -355,7 +380,7 @@ export default function AgentRepositoryPage() { className="w-full justify-center gap-1.5 rounded-lg px-[5px] py-2 text-sm data-[state=active]:shadow-sm" > - {t("agentRepository.page.tab.repository")} + {t("repository.page.tab.repository")} {repositoryTabCount} @@ -376,7 +401,7 @@ export default function AgentRepositoryPage() { className="w-full justify-center gap-1.5 rounded-lg px-[5px] py-2 text-sm data-[state=active]:shadow-sm" > - {t("agentRepository.page.tab.review")} + {t("repository.page.tab.review")} {pendingReviewCount > 0 ? ( {pendingReviewCount} @@ -420,16 +445,18 @@ export default function AgentRepositoryPage() { onPageChange={setReviewPage} updatingRepositoryId={updatingRepositoryId} onDetailClick={handleDetailClick} - onApprove={(listing) => + onApprove={(listing, content) => updateStatusMutation.mutateAsync({ agentRepositoryId: listing.agent_repository_id, status: "shared", + content, }) } - onReject={(listing) => + onReject={(listing, content) => updateStatusMutation.mutateAsync({ agentRepositoryId: listing.agent_repository_id, status: "rejected", + content, }) } /> @@ -456,6 +483,10 @@ export default function AgentRepositoryPage() { isFetching={isMineFetching} onRetry={() => refetchMine()} onViewDetail={handleMineViewDetail} + reviewDeepLink={reviewDeepLink} + deepLinkFallbackAgent={deepLinkFallbackAgent} + deepLinkFallbackLoading={isDeepLinkMineLoading} + onReviewDeepLinkConsumed={handleReviewDeepLinkConsumed} /> ) : null}
@@ -536,19 +567,19 @@ function RepositoryView({ const title = getListingTitle(listing); Modal.confirm({ - title: t("agentRepository.mine.reviewModal.confirmTakeDownTitle"), - content: t("agentRepository.mine.reviewModal.confirmTakeDownContent", { + title: t("repository.listingStatus.confirmTakeDownTitle"), + content: t("repository.listingStatus.confirmTakeDownContent", { name: title, }), - okText: t("agentRepository.mine.reviewModal.takeDown"), + okText: t("repository.listingStatus.takeDown"), cancelText: t("common.cancel"), okButtonProps: { danger: true }, onOk: async () => { try { await onTakeDown(listing); - message.success(t("agentRepository.mine.takeDownSuccess")); + message.success(t("repository.mine.takeDownSuccess")); } catch { - message.error(t("agentRepository.mine.takeDownError")); + message.error(t("repository.mine.takeDownError")); throw new Error("Take down failed"); } }, @@ -582,7 +613,7 @@ function RepositoryView({ {t("agentRepository.page.loadError")}

) : listings.length === 0 ? ( @@ -614,7 +645,7 @@ function RepositoryView({ className="flex size-9 items-center justify-center rounded-lg p-0" disabled={page <= 1} onClick={() => onPageChange(Math.max(1, page - 1))} - aria-label={t("agentRepository.mine.pagination.prev")} + aria-label={t("repository.pagination.prev")} > @@ -625,7 +656,7 @@ function RepositoryView({ type={pageNumber === page ? "primary" : "default"} className="flex size-9 items-center justify-center rounded-lg p-0" onClick={() => onPageChange(pageNumber)} - aria-label={t("agentRepository.mine.pagination.page", { + aria-label={t("repository.pagination.page", { page: pageNumber, })} aria-current={pageNumber === page ? "page" : undefined} @@ -639,7 +670,7 @@ function RepositoryView({ className="flex size-9 items-center justify-center rounded-lg p-0" disabled={page >= totalPages} onClick={() => onPageChange(Math.min(totalPages, page + 1))} - aria-label={t("agentRepository.mine.pagination.next")} + aria-label={t("repository.pagination.next")} > @@ -679,11 +710,21 @@ function ReviewCenterView({ onPageChange: (page: number) => void; updatingRepositoryId: number | null; onDetailClick: (listing: AgentRepositoryListingItem) => void; - onApprove: (listing: AgentRepositoryListingItem) => Promise; - onReject: (listing: AgentRepositoryListingItem) => Promise; + onApprove: ( + listing: AgentRepositoryListingItem, + content?: string + ) => Promise; + onReject: ( + listing: AgentRepositoryListingItem, + content?: string + ) => Promise; }) { const { t } = useTranslation("common"); const { message } = App.useApp(); + const [reviewAction, setReviewAction] = + useState(null); + const [reviewListing, setReviewListing] = + useState(null); const totalPages = total > 0 ? Math.ceil(total / pageSize) : 0; const showPagination = !isLoading && !isError && totalPages > 1; @@ -693,47 +734,51 @@ function ReviewCenterView({ listing.name?.trim() || t("agentRepository.card.untitled"); - const confirmReviewAction = ( + const closeReviewModal = () => { + setReviewAction(null); + setReviewListing(null); + }; + + const openReviewModal = ( listing: AgentRepositoryListingItem, - action: "approve" | "reject" + action: AgentRepositoryReviewAction ) => { - const title = getListingTitle(listing); - const isApprove = action === "approve"; + setReviewListing(listing); + setReviewAction(action); + }; - Modal.confirm({ - title: isApprove - ? t("agentRepository.review.confirmApproveTitle") - : t("agentRepository.review.confirmRejectTitle"), - content: isApprove - ? t("agentRepository.review.confirmApproveContent", { name: title }) - : t("agentRepository.review.confirmRejectContent", { name: title }), - okText: isApprove - ? t("agentRepository.review.approve") - : t("agentRepository.review.reject"), - cancelText: t("common.cancel"), - okButtonProps: isApprove - ? undefined - : { danger: true }, - onOk: async () => { - try { - await (isApprove ? onApprove(listing) : onReject(listing)); - message.success( - isApprove - ? t("agentRepository.review.approveSuccess", { name: title }) - : t("agentRepository.review.rejectSuccess", { name: title }) - ); - } catch { - message.error( - isApprove - ? t("agentRepository.review.approveError") - : t("agentRepository.review.rejectError") - ); - throw new Error("Review action failed"); - } - }, - }); + const handleReviewConfirm = async (content?: string) => { + if (!reviewListing || !reviewAction) { + return; + } + + const title = getListingTitle(reviewListing); + const isApprove = reviewAction === "approve"; + + try { + await (isApprove + ? onApprove(reviewListing, content) + : onReject(reviewListing, content)); + message.success( + isApprove + ? t("repository.review.approveSuccess", { name: title }) + : t("repository.review.rejectSuccess", { name: title }) + ); + closeReviewModal(); + } catch { + message.error( + isApprove + ? t("repository.review.approveError") + : t("repository.review.rejectError") + ); + throw new Error("Review action failed"); + } }; + const isReviewModalLoading = + reviewListing != null && + updatingRepositoryId === reviewListing.agent_repository_id; + return (
{isLoading ? ( @@ -743,14 +788,14 @@ function ReviewCenterView({ ) : isError ? (

- {t("agentRepository.review.loadError")} + {t("repository.review.loadError")}

) : listings.length === 0 ? ( - + ) : ( <> confirmReviewAction(listing, "approve")} - onReject={(listing) => confirmReviewAction(listing, "reject")} + onApprove={(listing) => openReviewModal(listing, "approve")} + onReject={(listing) => openReviewModal(listing, "reject")} + /> + + {showPagination ? ( @@ -769,7 +823,7 @@ function ReviewCenterView({ className="flex size-9 items-center justify-center rounded-lg p-0" disabled={page <= 1} onClick={() => onPageChange(Math.max(1, page - 1))} - aria-label={t("agentRepository.mine.pagination.prev")} + aria-label={t("repository.pagination.prev")} > @@ -780,7 +834,7 @@ function ReviewCenterView({ type={pageNumber === page ? "primary" : "default"} className="flex size-9 items-center justify-center rounded-lg p-0" onClick={() => onPageChange(pageNumber)} - aria-label={t("agentRepository.mine.pagination.page", { + aria-label={t("repository.pagination.page", { page: pageNumber, })} aria-current={pageNumber === page ? "page" : undefined} @@ -794,7 +848,7 @@ function ReviewCenterView({ className="flex size-9 items-center justify-center rounded-lg p-0" disabled={page >= totalPages} onClick={() => onPageChange(Math.min(totalPages, page + 1))} - aria-label={t("agentRepository.mine.pagination.next")} + aria-label={t("repository.pagination.next")} > diff --git a/frontend/app/[locale]/agent-tasks/components/AutomationProposalCard.tsx b/frontend/app/[locale]/agent-tasks/components/AutomationProposalCard.tsx new file mode 100644 index 0000000000..77a122e55a --- /dev/null +++ b/frontend/app/[locale]/agent-tasks/components/AutomationProposalCard.tsx @@ -0,0 +1 @@ +export { default } from "@/features/agentAutomation/components/AutomationProposalCard"; diff --git a/frontend/app/[locale]/agent-tasks/page.tsx b/frontend/app/[locale]/agent-tasks/page.tsx new file mode 100644 index 0000000000..9ce847d830 --- /dev/null +++ b/frontend/app/[locale]/agent-tasks/page.tsx @@ -0,0 +1,1032 @@ +"use client"; + +import { useCallback, useEffect, useRef, useState } from "react"; +import dayjs, { type Dayjs } from "dayjs"; +import Link from "next/link"; +import { useParams, useRouter } from "next/navigation"; +import { useTranslation } from "react-i18next"; +import { + Button, + Drawer, + Dropdown, + Form, + Input, + InputNumber, + Modal, + Select, + Space, + Table, + Tag, + Tooltip, + message, +} from "antd"; +import type { ColumnsType } from "antd/es/table"; +import type { FilterDropdownProps } from "antd/es/table/interface"; +import type { TableProps } from "antd"; +import type { MenuProps } from "antd"; +import { + CalendarClock, + LoaderCircle, + History, + MessageCirclePlus, + MoreHorizontal, + Pause, + Pencil, + Play, + RefreshCw, + Search, + Square, + Trash2, +} from "lucide-react"; + +import { agentAutomationService } from "@/services/agentAutomationService"; +import AutomationDateTimePicker from "@/features/agentAutomation/components/AutomationDateTimePicker"; +import { getAutomationErrorMessage } from "@/features/agentAutomation/errorMessage"; +import type { + AgentAutomationRun, + AgentAutomationTask, + AutomationTaskListStatus, + UpdateAutomationTaskPayload, +} from "@/types/agentAutomation"; + +const statusColor: Record = { + ACTIVE: "blue", + ENABLED: "blue", + RUNNING: "green", + PAUSED: "gold", + PAUSED_BY_SYSTEM: "red", + COMPLETED: "blue", +}; + +const taskStatusFilters = [ + "DRAFT", + "ENABLED", + "RUNNING", + "PAUSED", + "PAUSED_BY_SYSTEM", + "COMPLETED", +]; +const DEFAULT_TASK_PAGE_SIZE = 20; +const DEFAULT_RUN_PAGE_SIZE = 10; + +function CompactSearchFilter({ + value, + onChange, + placeholder, +}: { + value: string; + onChange: (value: string) => void; + placeholder: string; +}) { + return ( +
event.stopPropagation()}> + } + value={value} + placeholder={placeholder} + onChange={(event) => onChange(event.target.value)} + /> +
+ ); +} + +function CompactStatusFilter({ + currentValue, + onChange, + close, + allLabel, + options, +}: { + currentValue: string; + onChange: (value: string) => void; + close: FilterDropdownProps["close"]; + allLabel: string; + options: Array<{ label: string; value: string }>; +}) { + const items = [{ label: allLabel, value: "" }, ...options]; + + return ( +
+ {items.map((item) => { + const selected = currentValue === item.value; + return ( + + ); + })} +
+ ); +} + +interface TaskFormValues { + title: string; + instruction: string; + mode: "ONCE" | "RECURRING"; + rule_type: "INTERVAL" | "CRON"; + start_at: Dayjs; + cron_expr?: string; + interval_seconds?: number; + timeout_seconds?: number; +} + +function buildPatchPayload( + values: TaskFormValues +): UpdateAutomationTaskPayload { + const mode = values.mode; + const startAt = values.start_at.toISOString(); + const timezone = + Intl.DateTimeFormat().resolvedOptions().timeZone || "Asia/Shanghai"; + const scheduleTrigger = + mode === "ONCE" + ? { + mode: "ONCE" as const, + rule_type: "AT" as const, + timezone, + start_at: startAt, + max_fire_count: 1, + } + : values.rule_type === "INTERVAL" + ? { + mode: "RECURRING" as const, + rule_type: "INTERVAL" as const, + timezone, + start_at: startAt, + interval_seconds: values.interval_seconds, + } + : { + mode: "RECURRING" as const, + rule_type: "CRON" as const, + timezone, + start_at: startAt, + cron_expr: values.cron_expr, + }; + + return { + title: values.title, + instruction: values.instruction, + schedule_trigger: scheduleTrigger, + timeout_seconds: values.timeout_seconds || 1800, + }; +} + +function taskToFormValues(task: AgentAutomationTask) { + const trigger = task.schedule_config; + return { + title: task.title, + agent_id: task.agent_id, + instruction: task.instruction, + mode: trigger.mode, + rule_type: trigger.rule_type === "AT" ? "CRON" : trigger.rule_type, + start_at: dayjs(trigger.start_at), + cron_expr: trigger.cron_expr || "0 9 * * *", + interval_seconds: trigger.interval_seconds || 3600, + timeout_seconds: task.timeout_seconds || 1800, + }; +} + +export default function AgentTasksPage() { + const router = useRouter(); + const params = useParams<{ locale: string }>(); + const { t, i18n } = useTranslation("common"); + const [tasks, setTasks] = useState([]); + const [taskTotal, setTaskTotal] = useState(0); + const [taskPage, setTaskPage] = useState(1); + const [taskPageSize, setTaskPageSize] = useState(DEFAULT_TASK_PAGE_SIZE); + const [loading, setLoading] = useState(false); + const [taskNameSearch, setTaskNameSearch] = useState(""); + const [agentNameSearch, setAgentNameSearch] = useState(""); + const [statusFilter, setStatusFilter] = useState< + AutomationTaskListStatus | undefined + >(); + const [modalOpen, setModalOpen] = useState(false); + const [historyOpen, setHistoryOpen] = useState(false); + const [selectedTask, setSelectedTask] = useState( + null + ); + const [editingTask, setEditingTask] = useState( + null + ); + const [runs, setRuns] = useState([]); + const [runTotal, setRunTotal] = useState(0); + const [runPage, setRunPage] = useState(1); + const [runPageSize, setRunPageSize] = useState(DEFAULT_RUN_PAGE_SIZE); + const [runLoading, setRunLoading] = useState(false); + const [form] = Form.useForm(); + const loadRequestIdRef = useRef(0); + + const formatDateTime = (value?: string | null) => + value + ? new Intl.DateTimeFormat( + i18n.language.startsWith("zh") ? "zh-CN" : "en-US", + { + dateStyle: "medium", + timeStyle: "medium", + } + ).format(new Date(value)) + : "-"; + + const formatTaskStatus = (status: string) => + t(`agentAutomation.status.${status}`, { defaultValue: status }); + + const getTaskDisplayStatus = (task: AgentAutomationTask) => { + if (task.is_running) return "RUNNING"; + if (task.status === "ACTIVE") return "ENABLED"; + return task.status; + }; + + const formatRunStatus = (status?: string | null) => + status + ? t(`agentAutomation.runStatus.${status}`, { defaultValue: status }) + : "-"; + + const formatTriggerType = (triggerType: string) => + t(`agentAutomation.triggerType.${triggerType}`, { + defaultValue: triggerType, + }); + + const paginationLocale = { + items_per_page: t("common.pagination.itemsPerPage"), + jump_to: t("common.pagination.jumpTo"), + page: t("common.pagination.page"), + }; + + const formatScheduleDetail = (task: AgentAutomationTask) => { + const trigger = task.schedule_config; + if (trigger.rule_type === "AT") { + return formatDateTime(trigger.start_at); + } + if (trigger.rule_type === "INTERVAL") { + return t("agentAutomation.page.everySeconds", { + count: trigger.interval_seconds, + }); + } + return t("agentAutomation.page.cronSchedule", { + expression: trigger.cron_expr, + }); + }; + + const loadTasks = useCallback(async () => { + const requestId = ++loadRequestIdRef.current; + setLoading(true); + try { + const loadedTasks = await agentAutomationService.list({ + status: statusFilter, + search: taskNameSearch, + agentName: agentNameSearch, + page: taskPage, + pageSize: taskPageSize, + }); + if (requestId === loadRequestIdRef.current) { + setTasks(loadedTasks.items); + setTaskTotal(loadedTasks.total); + setTaskPage(loadedTasks.page); + setTaskPageSize(loadedTasks.page_size); + } + } catch (error: unknown) { + if (requestId === loadRequestIdRef.current) { + message.error( + getAutomationErrorMessage(error, t, "agentAutomation.page.loadFailed") + ); + } + } finally { + if (requestId === loadRequestIdRef.current) { + setLoading(false); + } + } + }, [ + agentNameSearch, + statusFilter, + taskNameSearch, + taskPage, + taskPageSize, + t, + ]); + + useEffect(() => { + void loadTasks(); + }, [loadTasks]); + + const handleTableChange: TableProps["onChange"] = ( + pagination, + filters + ) => { + const nextTaskName = filters.title?.[0]; + const nextAgentName = filters.agent_name?.[0]; + const nextStatus = filters.status?.[0]; + const nextTaskSearch = + typeof nextTaskName === "string" ? nextTaskName.trim() : ""; + const nextAgentSearch = + typeof nextAgentName === "string" ? nextAgentName.trim() : ""; + const nextStatusFilter = + typeof nextStatus === "string" + ? (nextStatus as AutomationTaskListStatus) + : undefined; + const filtersChanged = + nextTaskSearch !== taskNameSearch || + nextAgentSearch !== agentNameSearch || + nextStatusFilter !== statusFilter; + setTaskNameSearch(nextTaskSearch); + setAgentNameSearch(nextAgentSearch); + setStatusFilter(nextStatusFilter); + setTaskPage(filtersChanged ? 1 : pagination.current || 1); + setTaskPageSize(pagination.pageSize || DEFAULT_TASK_PAGE_SIZE); + }; + + const openEdit = (task: AgentAutomationTask) => { + setEditingTask(task); + form.setFieldsValue(taskToFormValues(task)); + setModalOpen(true); + }; + + const submitTask = async () => { + if (!editingTask) return; + const values = (await form.validateFields()) as TaskFormValues; + try { + await agentAutomationService.update( + editingTask.task_id, + buildPatchPayload(values) + ); + message.success(t("agentAutomation.page.updateSuccess")); + setModalOpen(false); + setEditingTask(null); + await loadTasks(); + } catch (error: unknown) { + message.error( + getAutomationErrorMessage(error, t, "agentAutomation.page.updateFailed") + ); + } + }; + + const loadRuns = useCallback( + async ( + task: AgentAutomationTask, + page = runPage, + pageSize = runPageSize + ) => { + setRunLoading(true); + try { + const loadedRuns = await agentAutomationService.runs(task.task_id, { + page, + pageSize, + }); + setRuns(loadedRuns.items); + setRunTotal(loadedRuns.total); + setRunPage(loadedRuns.page); + setRunPageSize(loadedRuns.page_size); + } catch (error: unknown) { + message.error( + getAutomationErrorMessage( + error, + t, + "agentAutomation.page.historyLoadFailed" + ) + ); + } finally { + setRunLoading(false); + } + }, + [runPage, runPageSize, t] + ); + + const openRuns = async (task: AgentAutomationTask) => { + setSelectedTask(task); + setHistoryOpen(true); + setRunPage(1); + await loadRuns(task, 1, runPageSize); + }; + + const cancelRun = async (run: AgentAutomationRun) => { + try { + await agentAutomationService.cancelRun(run.run_id); + message.success(t("agentAutomation.page.cancelRunSuccess")); + if (selectedTask) { + await loadRuns(selectedTask); + await loadTasks(); + } + } catch (error: unknown) { + message.error( + getAutomationErrorMessage( + error, + t, + "agentAutomation.page.cancelRunFailed" + ) + ); + } + }; + + const confirmDeleteRun = (run: AgentAutomationRun) => { + Modal.confirm({ + title: t("agentAutomation.page.deleteRunTitle"), + content: t("agentAutomation.page.deleteRunDescription"), + okText: t("agentAutomation.page.deleteRunConfirm"), + cancelText: t("common.cancel"), + okButtonProps: { danger: true }, + onOk: async () => { + try { + await agentAutomationService.deleteRun(run.run_id); + message.success(t("agentAutomation.page.deleteRunSuccess")); + if (selectedTask) { + await loadRuns(selectedTask); + await loadTasks(); + } + } catch (error: unknown) { + message.error( + getAutomationErrorMessage( + error, + t, + "agentAutomation.page.deleteRunFailed" + ) + ); + } + }, + }); + }; + + const runTask = async (task: AgentAutomationTask) => { + setTasks((currentTasks) => + currentTasks.map((currentTask) => + currentTask.task_id === task.task_id + ? { ...currentTask, is_running: true } + : currentTask + ) + ); + try { + await agentAutomationService.run(task.task_id); + message.success(t("agentAutomation.page.runSuccess")); + } catch (error) { + message.error( + getAutomationErrorMessage(error, t, "agentAutomation.page.runFailed") + ); + } finally { + await loadTasks(); + } + }; + + const pauseTask = async (task: AgentAutomationTask) => { + try { + await agentAutomationService.pause(task.task_id); + message.success(t("agentAutomation.page.pauseSuccess")); + await loadTasks(); + } catch (error) { + message.error( + getAutomationErrorMessage(error, t, "agentAutomation.page.pauseFailed") + ); + } + }; + + const resumeTask = async (task: AgentAutomationTask) => { + try { + await agentAutomationService.resume(task.task_id); + message.success(t("agentAutomation.page.resumeSuccess")); + await loadTasks(); + } catch (error) { + message.error( + getAutomationErrorMessage(error, t, "agentAutomation.page.resumeFailed") + ); + } + }; + + const deleteTask = async (task: AgentAutomationTask) => { + try { + await agentAutomationService.delete(task.task_id); + message.success(t("agentAutomation.page.deleteSuccess")); + await loadTasks(); + } catch (error) { + message.error( + getAutomationErrorMessage(error, t, "agentAutomation.page.deleteFailed") + ); + throw error; + } + }; + + const confirmDeleteTask = (task: AgentAutomationTask) => { + Modal.confirm({ + title: t("agentAutomation.page.deleteTitle"), + content: t("agentAutomation.page.deleteDescription"), + okText: t("agentAutomation.page.delete"), + cancelText: t("common.cancel"), + okButtonProps: { danger: true }, + onOk: () => deleteTask(task), + }); + }; + + const getMoreActionItems = ( + task: AgentAutomationTask + ): MenuProps["items"] => [ + { + key: "history", + icon: , + label: t("agentAutomation.page.history"), + onClick: () => openRuns(task), + }, + { + key: "edit", + icon: , + label: t("agentAutomation.page.edit"), + onClick: () => openEdit(task), + }, + { type: "divider" }, + { + key: "delete", + danger: true, + icon: , + label: t("agentAutomation.page.delete"), + onClick: () => confirmDeleteTask(task), + }, + ]; + + const columns: ColumnsType = [ + { + title: t("agentAutomation.page.task"), + dataIndex: "title", + filteredValue: taskNameSearch ? [taskNameSearch] : null, + filterIcon: (filtered) => ( + + ), + filterDropdown: () => ( + { + setTaskNameSearch(value); + setTaskPage(1); + }} + placeholder={t("agentAutomation.page.taskSearchPlaceholder")} + /> + ), + render: (_, task) => ( +
+ + {task.title} + +
+ {t("agentAutomation.page.conversationValue", { + conversationId: task.conversation_id, + })} +
+
+ ), + }, + { + title: t("agentAutomation.page.agent"), + dataIndex: "agent_name", + width: 190, + filteredValue: agentNameSearch ? [agentNameSearch] : null, + filterIcon: (filtered) => ( + + ), + filterDropdown: () => ( + { + setAgentNameSearch(value); + setTaskPage(1); + }} + placeholder={t("agentAutomation.page.agentSearchPlaceholder")} + /> + ), + render: (_, task) => ( +
+
+ {task.agent_name || + t("agentAutomation.page.agentFallback", { + agentId: task.agent_id, + })} +
+
+ Agent #{task.agent_id} +
+
+ ), + }, + { + title: t("agentAutomation.page.status"), + dataIndex: "status", + width: 130, + filters: taskStatusFilters.map((status) => ({ + text: formatTaskStatus(status), + value: status, + })), + filteredValue: statusFilter ? [statusFilter] : null, + filterMultiple: false, + filterDropdown: ({ close }) => ( + { + setStatusFilter( + value ? (value as AutomationTaskListStatus) : undefined + ); + setTaskPage(1); + }} + close={close} + allLabel={t("agentAutomation.page.allStatuses")} + options={taskStatusFilters.map((status) => ({ + label: formatTaskStatus(status), + value: status, + }))} + /> + ), + render: (_, task) => { + const displayStatus = getTaskDisplayStatus(task); + return ( + + ) : undefined + } + > + {formatTaskStatus(displayStatus)} + + ); + }, + }, + { + title: t("agentAutomation.page.schedule"), + width: 180, + render: (_, task) => ( +
+
+ {task.schedule_mode === "ONCE" + ? t("agentAutomation.page.once") + : t("agentAutomation.page.recurring")} +
+
+ {formatScheduleDetail(task)} +
+
+ ), + }, + { + title: t("agentAutomation.page.nextFireAt"), + dataIndex: "next_fire_at", + width: 220, + render: (value) => formatDateTime(value), + }, + { + title: t("agentAutomation.page.lastResult"), + width: 180, + render: (_, task) => ( +
+
{formatRunStatus(task.last_run_status)}
+ {task.last_error && ( +
+ {task.last_error} +
+ )} +
+ ), + }, + { + title: t("agentAutomation.page.actions"), + width: 150, + render: (_, task) => ( + + + + + + +
+ + + + { + setModalOpen(false); + setEditingTask(null); + }} + onOk={submitTask} + okText={t("common.save")} + cancelText={t("common.cancel")} + width={720} + > +
+ + + + + + + + + + + + + {getFieldValue("rule_type") === "INTERVAL" ? ( + + + + ) : ( + + + + )} + + )} + + )} + + + + + + + + { + setHistoryOpen(false); + setSelectedTask(null); + setRuns([]); + setRunTotal(0); + setRunPage(1); + }} + width={720} + > +
{ + if (selectedTask) { + void loadRuns(selectedTask, page, pageSize); + } + }, + }} + columns={[ + { + title: t("agentAutomation.page.status"), + dataIndex: "status", + width: 120, + render: (value) => formatRunStatus(value), + }, + { + title: t("agentAutomation.page.trigger"), + dataIndex: "trigger_type", + width: 120, + render: (value) => formatTriggerType(value), + }, + { + title: t("agentAutomation.page.scheduledAt"), + dataIndex: "scheduled_fire_at", + render: (value) => formatDateTime(value), + }, + { + title: t("agentAutomation.page.errorLog"), + dataIndex: "error_message", + render: (value) => value || "-", + }, + { + title: t("agentAutomation.page.actions"), + width: 130, + render: (_, run) => { + const isActive = ["QUEUED", "RUNNING"].includes(run.status); + return isActive ? ( + + ) : ( + - + -

{t("businessLogic.config.title")}

+

+ {t("businessLogic.config.title")} +

@@ -101,7 +147,9 @@ export default function AgentConfigComp({}: AgentConfigCompProps) { -

{t("collaborativeAgent.title")}

+

+ {t("collaborativeAgent.title")} +

@@ -111,7 +159,7 @@ export default function AgentConfigComp({}: AgentConfigCompProps) { size="small" icon={} onClick={() => setShowA2ADiscovery(true)} - className="text-green-500 hover:!text-green-600 hover:!bg-green-50" + className="!text-green-600 hover:!bg-green-50 hover:!text-green-700" title={t("toolManagement.refresh.title")} > {t("collaborativeAgent.addExternal")} @@ -126,142 +174,183 @@ export default function AgentConfigComp({}: AgentConfigCompProps) { + {/* Tool/Skill Tabs */} + + + + + {t("toolPool.title")} + {selectedTools.length > 0 && ( + + )} + + + {t("toolPool.tooltip.functionGuide")} + + } + color="#ffffff" + styles={{ + root: { + backgroundColor: "#ffffff", + border: "1px solid #e5e7eb", + borderRadius: "6px", + boxShadow: + "0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06)", + maxWidth: "800px", + minWidth: "700px", + width: "fit-content", + }, + }} + > + + + + + + {t("skillPool.title")} + {selectedSkills && selectedSkills.length > 0 && ( + + )} + + + - {/* Tool/Skill Tabs */} - - - - - {t("toolPool.title")} - {selectedTools.length > 0 && ( - - )} - - {t("toolPool.tooltip.functionGuide")}} - color="#ffffff" - styles={{ - root: { - backgroundColor: "#ffffff", - border: "1px solid #e5e7eb", - borderRadius: "6px", - boxShadow: "0 4px 6px -1px rgba(0, 0, 0, 0.1), 0 2px 4px -1px rgba(0, 0, 0, 0.06)", - maxWidth: "800px", - minWidth: "700px", - width: "fit-content", - }, - }} - > - - - - - - {t("skillPool.title")} - {selectedSkills && selectedSkills.length > 0 && ( - - )} - - - + + + + + {/* Left: action text links (mirrors demo's Refresh / MCP Config pattern) */} +
+ + +
+ {/* Right: Select Tools button (mirrors demo) */} +
+ +
+ + + - + + + + + + - - - - {/* Left: action text links (mirrors demo's Refresh / MCP Config pattern) */} -
- - -
- {/* Right: Select Tools button (mirrors demo) */} -
+ + +
+ +
+ + +
- - - - - - - - - - - - - - - - - - - - - + + + - - - - - - - + + + + + + + - setIsMcpModalOpen(false)} /> + setIsMcpModalOpen(false)} + /> + setIsSkillSelectOpen(false)} + onOpenManageTags={() => setTagModalOpen(true)} + onEditSkill={(skill) => { + handleOpenSkillEditor(skill); + }} + isCreatingMode={isCreatingMode} + currentAgentId={currentAgentId ?? undefined} + isReadOnly={isReadOnly} + /> + + setTagModalOpen(false)} + /> + setIsSkillModalOpen(false)} + onCancel={handleCloseSkillModal} onSuccess={handleSkillBuildSuccess} + editingSkill={editingSkill} + zIndex={1100} /> {/* A2A Discovery Modal */} diff --git a/frontend/app/[locale]/agents/components/AgentManageComp.tsx b/frontend/app/[locale]/agents/components/AgentManageComp.tsx index 7dabff4dd5..f20aef867e 100644 --- a/frontend/app/[locale]/agents/components/AgentManageComp.tsx +++ b/frontend/app/[locale]/agents/components/AgentManageComp.tsx @@ -12,8 +12,7 @@ import { useAuthorizationContext } from "@/components/providers/AuthorizationPro import log from "@/lib/logger"; import { useState } from "react"; import { - parseAgentImportFile, - selectFile, + openImportWizardWithFile, type ImportAgentData, } from "@/lib/agentImportUtils"; import AgentImportWizard from "@/components/agent/AgentImportWizard"; @@ -39,22 +38,15 @@ export default function AgentManageComp() { // Handle import agent for space view - open wizard instead of direct import const handleImportAgent = async () => { - const file = await selectFile(".json"); - if (!file) return; - - const agentData = await parseAgentImportFile(file, { - onParseError: (msgKey) => message.error(t(msgKey)), - onValidationError: (msgKey) => message.error(t(msgKey)), - onGenericError: (error) => { - log.error("Failed to read import file:", error); - message.error(t("businessLogic.config.error.agentImportFailed")); + await openImportWizardWithFile({ + onSuccess: (agentData) => { + setImportWizardData(agentData); + setImportWizardVisible(true); }, + message: message, + t: t, + log: log, }); - - if (!agentData) return; - - setImportWizardData(agentData); - setImportWizardVisible(true); }; return ( diff --git a/frontend/app/[locale]/agents/components/AgentSelectorHeader.tsx b/frontend/app/[locale]/agents/components/AgentSelectorHeader.tsx index 0b68ee14ed..eaa1985056 100644 --- a/frontend/app/[locale]/agents/components/AgentSelectorHeader.tsx +++ b/frontend/app/[locale]/agents/components/AgentSelectorHeader.tsx @@ -1,12 +1,17 @@ "use client"; import { useTranslation } from "react-i18next"; -import { App, Flex, Button, Badge, Dropdown, Tooltip, Col, Row, Modal, Spin, Tag, theme } from "antd"; +import { App, Flex, Button, Badge, Dropdown, Tooltip, Col, Row, Modal, Tag, theme, Input } from "antd"; import { useMutation } from "@tanstack/react-query"; -import { Plus, FileInput, Settings, ChevronDown, ChevronLeft, Bot, Copy, Network, FileOutput, Trash2, Globe, GitBranch, History } from "lucide-react"; +import { Plus, FileInput, ChevronDown, ChevronLeft, Bot, Copy, Network, FileOutput, Trash2, Globe, GitBranch, History, Search } from "lucide-react"; import { ExclamationCircleOutlined } from "@ant-design/icons"; -import { useState } from "react"; -import { useParams, useRouter, useSearchParams } from "next/navigation"; +import { useMemo, useState } from "react"; +import { + useParams, + usePathname, + useRouter, + useSearchParams, +} from "next/navigation"; import { StaticScrollArea } from "@/components/ui/scrollArea"; import AgentCallRelationshipModal from "@/components/agent/AgentCallRelationshipModal"; import A2AServerSettingsPanel from "./a2a/A2AServerSettingsPanel"; @@ -27,13 +32,12 @@ import { useAgentConfigStore } from "@/stores/agentConfigStore"; import { useSaveGuard } from "@/hooks/agent/useSaveGuard"; import { useQueryClient } from "@tanstack/react-query"; import AgentImportWizard from "@/components/agent/AgentImportWizard"; -import { ImportAgentData } from "@/lib/agentImportUtils"; +import { ImportAgentData, openImportWizardWithFile } from "@/lib/agentImportUtils"; import log from "@/lib/logger"; import { useAgentList } from "@/hooks/agent/useAgentList"; import { useAgentVersionList } from "@/hooks/agent/useAgentVersionList"; import { useAgentVersionDetail } from "@/hooks/agent/useAgentVersionDetail"; import { useAgentInfo } from "@/hooks/agent/useAgentInfo"; -import { useAuthorizationContext } from "@/components/providers/AuthorizationProvider"; interface AgentSelectorHeaderProps { onOpenVersionManage: () => void; @@ -49,6 +53,7 @@ export default function AgentSelectorHeader({ const { t } = useTranslation("common"); const { message } = App.useApp(); const router = useRouter(); + const pathname = usePathname(); const searchParams = useSearchParams(); const params = useParams<{ locale: string }>(); const locale = params.locale || "en"; @@ -57,10 +62,9 @@ export default function AgentSelectorHeader({ const checkUnsavedChanges = useSaveGuard(); const confirm = useConfirmModal(); const { token } = theme?.useToken?.() || {}; - const { user } = useAuthorizationContext(); - // Fetch agent list internally - const { agents } = useAgentList(user?.tenantId ?? null); + // Resolve tenant from auth (matches AgentManageComp / published_list; keeps ASSET_OWNER merge) + const { agents } = useAgentList(""); // Store state const currentAgentId = useAgentConfigStore((state) => state.currentAgentId); @@ -84,6 +88,7 @@ export default function AgentSelectorHeader({ // Dropdown open state const [dropdownOpen, setDropdownOpen] = useState(false); + const [agentSearch, setAgentSearch] = useState(""); // Mutations const updateAgentMutation = useMutation({ @@ -145,44 +150,16 @@ export default function AgentSelectorHeader({ ); // Handle import agent - const handleImportAgent = () => { - const fileInput = document.createElement("input"); - fileInput.type = "file"; - fileInput.accept = ".json"; - fileInput.onchange = async (event) => { - const file = (event.target as HTMLInputElement).files?.[0]; - if (!file) return; - - if (!file.name.endsWith(".json")) { - message.error(t("businessLogic.config.error.invalidFileType")); - return; - } - - try { - const fileContent = await file.text(); - let agentData: ImportAgentData; - - try { - agentData = JSON.parse(fileContent); - } catch (parseError) { - message.error(t("businessLogic.config.error.invalidFileType")); - return; - } - - if (!agentData.agent_id || !agentData.agent_info) { - message.error(t("businessLogic.config.error.invalidFileType")); - return; - } - + const handleImportAgent = async () => { + await openImportWizardWithFile({ + onSuccess: (agentData) => { setImportWizardData(agentData); setImportWizardVisible(true); - } catch (error) { - log.error("Failed to read import file:", error); - message.error(t("businessLogic.config.error.agentImportFailed")); - } - }; - - fileInput.click(); + }, + message: message, + t: t, + log: log, + }); }; // Handle view call relationship @@ -208,7 +185,12 @@ export default function AgentSelectorHeader({ const handleExportAgent = async (agent: Agent) => { try { const result = await exportAgent(Number(agent.id)); - if (result.success && result.data) { + if (!result.success) { + message.error(result.message || t("businessLogic.config.error.agentExportFailed")); + return; + } + + if (result.data) { const blob = new Blob([JSON.stringify(result.data, null, 2)], { type: "application/json", }); @@ -220,12 +202,9 @@ export default function AgentSelectorHeader({ link.click(); document.body.removeChild(link); URL.revokeObjectURL(url); - message.success(t("businessLogic.config.message.agentExportSuccess")); - } else { - message.error( - result.message || t("businessLogic.config.error.agentImportFailed") - ); } + + message.success(t("businessLogic.config.message.agentExportSuccess")); } catch (error) { message.error(t("businessLogic.config.error.agentExportFailed")); } @@ -288,6 +267,7 @@ export default function AgentSelectorHeader({ model_ids: modelIdsForCopy, max_steps: detail.max_step, requested_output_tokens: detail.requested_output_tokens ?? null, + is_main_agent: detail.is_main_agent ?? true, provide_run_summary: detail.provide_run_summary, enabled: detail.enabled, business_description: detail.business_description, @@ -430,6 +410,9 @@ export default function AgentSelectorHeader({ const result = await searchAgentInfo(Number(agent.id)); if (result.success && result.data) { setCurrentAgent(result.data); + const nextSearchParams = new URLSearchParams(searchParams.toString()); + nextSearchParams.set("agent_id", String(agent.id)); + router.replace(`${pathname}?${nextSearchParams.toString()}`); } else { message.error(result.message || t("agentConfig.agents.detailsLoadFailed")); } @@ -439,8 +422,19 @@ export default function AgentSelectorHeader({ } }; + const filteredAgents = useMemo(() => { + const query = agentSearch.trim().toLowerCase(); + if (!query) return agents; + + return agents.filter((agent: Agent) => + [agent.display_name, agent.name, agent.description].some((value) => + String(value || "").toLowerCase().includes(query) + ) + ); + }, [agentSearch, agents]); + // Dropdown menu items (only agents) - const agentMenuItems = agents.flatMap((agent: Agent, index: number) => { + const agentMenuItems = filteredAgents.flatMap((agent: Agent, index: number) => { const isAvailable = agent.is_available !== false; const displayName = agent.display_name || ""; const name = agent.name || ""; @@ -578,7 +572,7 @@ export default function AgentSelectorHeader({ }; // Add divider after each item except the last one - const divider = index < agents.length - 1 + const divider = index < filteredAgents.length - 1 ? { key: `divider-${agent.id}`, type: 'divider' as const } : null; @@ -593,6 +587,14 @@ export default function AgentSelectorHeader({ router.push(`/${locale}/agent-space?tab=mine`); }; + const handleCreateAgent = () => { + enterCreateMode(); + const nextSearchParams = new URLSearchParams(searchParams.toString()); + nextSearchParams.delete("agent_id"); + const query = nextSearchParams.toString(); + router.replace(query ? `${pathname}?${query}` : pathname); + }; + return ( <>
@@ -609,31 +611,62 @@ export default function AgentSelectorHeader({ lg={12} className="flex min-w-0" > - + {showBackFromRepository ? ( - + + + + + + + + - - @@ -750,7 +778,6 @@ export default function AgentSelectorHeader({ selectedAgentForRelationship.display_name || selectedAgentForRelationship.name } - /> )} diff --git a/frontend/app/[locale]/agents/components/a2a/A2AAgentDiscoveryModal.tsx b/frontend/app/[locale]/agents/components/a2a/A2AAgentDiscoveryModal.tsx index bc9260a296..fcc0b426e5 100644 --- a/frontend/app/[locale]/agents/components/a2a/A2AAgentDiscoveryModal.tsx +++ b/frontend/app/[locale]/agents/components/a2a/A2AAgentDiscoveryModal.tsx @@ -31,7 +31,9 @@ import { Search, Eye, Settings, + KeyRound, MessageCircle, + Info, } from "lucide-react"; import { a2aClientService, A2AExternalAgent } from "@/services/a2aService"; import A2AChatModal from "./A2AChatModal"; @@ -58,6 +60,167 @@ const PROTOCOL_BINDING_MAP: Record = { "grpc": "GRPC", }; +interface AgentSecuritySettingProps { + agent: A2AExternalAgent; + onSaved: () => void; +} + +function AgentSecuritySetting({ agent, onSaved }: Readonly) { + const { t } = useTranslation("common"); + const [open, setOpen] = useState(false); + const [values, setValues] = useState>({}); + const [saving, setSaving] = useState(false); + const [selectedRequirementIndex, setSelectedRequirementIndex] = useState( + agent.selected_security_requirement_index ?? undefined, + ); + const [configuredSchemeIds, setConfiguredSchemeIds] = useState( + agent.configured_security_scheme_ids || [], + ); + const schemes = agent.security_schemes || {}; + const requirements = agent.security_requirements || []; + const entries = Object.entries(schemes).flatMap(([schemeId, scheme]) => { + const securityScheme = scheme as Record; + const apiKeyScheme = securityScheme.apiKeySecurityScheme; + if (apiKeyScheme) { + return [{ schemeId, ...apiKeyScheme }]; + } + + const httpAuthScheme = securityScheme.httpAuthSecurityScheme; + if (typeof httpAuthScheme?.scheme === "string" && httpAuthScheme.scheme.trim()) { + const authScheme = httpAuthScheme.scheme.trim(); + const isHttpBearer = authScheme.toLowerCase() === "bearer"; + return [{ + schemeId, + name: "Authorization", + location: "header", + description: isHttpBearer && httpAuthScheme.bearerFormat + ? `${httpAuthScheme.bearerFormat} token` + : `${authScheme} credential`, + isHttpBearer, + }]; + } + + return []; + }); + + useEffect(() => { + setConfiguredSchemeIds(agent.configured_security_scheme_ids || []); + setSelectedRequirementIndex(agent.selected_security_requirement_index ?? undefined); + }, [agent.configured_security_scheme_ids, agent.selected_security_requirement_index]); + + if (entries.length === 0 || requirements.length === 0) { + return null; + } + + const handleSave = async () => { + const configuredValues = Object.fromEntries( + Object.entries(values).filter(([, value]) => value.trim()), + ); + if (Object.keys(configuredValues).length === 0 && configuredSchemeIds.length === 0) { + message.error(t("a2a.security.valueRequired")); + return; + } + if (requirements.length > 1 && selectedRequirementIndex === undefined) { + message.error(t("a2a.security.requirementRequired")); + return; + } + + setSaving(true); + const result = await a2aClientService.updateAgentSecurityCredentials( + String(agent.id), + configuredValues, + selectedRequirementIndex, + ); + setSaving(false); + if (result.success) { + setConfiguredSchemeIds(result.data?.configured_security_scheme_ids || configuredSchemeIds); + setSelectedRequirementIndex(result.data?.selected_security_requirement_index ?? selectedRequirementIndex); + message.success(t("a2a.security.saveSuccess")); + setValues({}); + setOpen(false); + onSaved(); + } else { + message.error(result.message || t("a2a.security.saveFailed")); + } + }; + + return ( + + + {t("a2a.security.requirementsHint")} + + {requirements.length > 1 && ( +
+ + {t("a2a.security.requirementLabel")} + + + groupNamesById.get(Number(value)) ?? label + } + /> + + +
+ + setSearch(event.target.value)} + placeholder={t("skillPool.searchSkillsPlaceholder")} + className="pl-7" + allowClear + /> + + handleTagsChange(row.skillId, value)} + placeholder={t("skillManagement.form.tagsPlaceholder")} + tokenSeparators={[","]} + options={allTags.map((tag) => ({ label: tag, value: tag }))} + style={{ minWidth: 260, width: "100%" }} + /> + ), + }, + ]; + + return ( + +
+ + ); +} diff --git a/frontend/app/[locale]/agents/components/agentConfig/skill/utils.ts b/frontend/app/[locale]/agents/components/agentConfig/skill/utils.ts new file mode 100644 index 0000000000..1fbe0cdce1 --- /dev/null +++ b/frontend/app/[locale]/agents/components/agentConfig/skill/utils.ts @@ -0,0 +1,50 @@ +import type { Skill, SkillParam } from "@/types/agentConfig"; + +const isMissingRequiredValue = (value: unknown): boolean => + value === undefined || value === null || value === ""; + +const getEffectiveParamValue = ( + param: SkillParam, + configValues: Record +): unknown => + Object.prototype.hasOwnProperty.call(configValues, param.name) + ? configValues[param.name] + : param.value; + +export const withEffectiveSkillConfig = ( + skill: Skill, + savedConfigValues?: Record | null +): Skill => { + const schemaDefaults = Object.fromEntries( + (skill.config_schemas || []).map((param) => [param.name, param.value]) + ); + const skillConfigValues = + skill.config_values && typeof skill.config_values === "object" + ? skill.config_values + : {}; + + return { + ...skill, + config_values: { + ...schemaDefaults, + ...skillConfigValues, + ...(savedConfigValues || {}), + }, + }; +}; + +export const hasMissingRequiredSkillConfig = (skill: Skill): boolean => { + const configValues = + skill.config_values && typeof skill.config_values === "object" + ? skill.config_values + : {}; + + return (skill.config_schemas || []).some( + (param) => + param.required && + isMissingRequiredValue(getEffectiveParamValue(param, configValues)) + ); +}; + +export const requiresSkillConfigOnSelection = (skill: Skill): boolean => + skill.name.toLowerCase() === "search-knowledge-base"; diff --git a/frontend/app/[locale]/agents/components/agentConfig/tool/LabelManagementModal.tsx b/frontend/app/[locale]/agents/components/agentConfig/tool/LabelManagementModal.tsx index a8a1179673..d5927080e9 100644 --- a/frontend/app/[locale]/agents/components/agentConfig/tool/LabelManagementModal.tsx +++ b/frontend/app/[locale]/agents/components/agentConfig/tool/LabelManagementModal.tsx @@ -5,8 +5,7 @@ import { useTranslation } from "react-i18next"; import { Modal, Table, Select, App } from "antd"; import type { ColumnsType } from "antd/es/table"; import { useQueryClient } from "@tanstack/react-query"; -import { API_ENDPOINTS } from "@/services/api"; -import { getAuthHeaders } from "@/lib/auth"; +import { updateToolLabels } from "@/services/mcpService"; import log from "@/lib/logger"; interface LabelManagementModalProps { @@ -38,7 +37,9 @@ export default function LabelManagementModal({ // Collect all unique labels from dataSource for Select suggestions const allExistingLabels = useMemo(() => { const labelSet = new Set(); - dataSource.forEach((row) => row.labels.forEach((l: string) => labelSet.add(l))); + dataSource.forEach((row) => + row.labels.forEach((l: string) => labelSet.add(l)) + ); return Array.from(labelSet).sort((a, b) => a.localeCompare(b)); }, [dataSource]); @@ -52,7 +53,7 @@ export default function LabelManagementModal({ name: tool.name, source: tool.source || "", labels: Array.isArray(tool.labels) ? [...tool.labels] : [], - updatedBy: tool.updated_by || "", + updatedBy: tool.updated_by_name || "", })); setDataSource(rows); builtRef.current = true; @@ -74,16 +75,18 @@ export default function LabelManagementModal({ // Persist to backend, then synchronously update cache so parent sees fresh data try { - await fetch(API_ENDPOINTS.tool.labels, { - method: "PUT", - headers: { ...getAuthHeaders(), "Content-Type": "application/json" }, - body: JSON.stringify({ tool_id: parseInt(toolId), labels: newLabels }), - }); + const result = await updateToolLabels(toolId, newLabels); + const updatedBy = result.updated_by_name || ""; + setDataSource((prev) => + prev.map((row) => (row.id === toolId ? { ...row, updatedBy } : row)) + ); // Synchronous cache update — no timing gaps, no refetch race queryClient.setQueryData(["tools"], (old: any[]) => { if (!old) return old; return old.map((tool: any) => - tool.id === toolId ? { ...tool, labels: newLabels } : tool + tool.id === toolId + ? { ...tool, labels: newLabels, updated_by_name: updatedBy } + : tool ); }); } catch (err) { diff --git a/frontend/app/[locale]/agents/components/agentConfig/tool/SelectToolsDialog.tsx b/frontend/app/[locale]/agents/components/agentConfig/tool/SelectToolsDialog.tsx index c28c42fb1a..30b42c04ef 100644 --- a/frontend/app/[locale]/agents/components/agentConfig/tool/SelectToolsDialog.tsx +++ b/frontend/app/[locale]/agents/components/agentConfig/tool/SelectToolsDialog.tsx @@ -8,6 +8,8 @@ import { Search, Settings, Wrench, Tag } from "lucide-react"; import i18n from "i18next"; import { useToolList } from "@/hooks/agent/useToolList"; +import { useMcpServerList } from "@/hooks/mcp/useMcpServerList"; +import { useAuthorizationContext } from "@/components/providers/AuthorizationProvider"; import { useAgentConfigStore } from "@/stores/agentConfigStore"; import { usePrefetchKnowledgeBases } from "@/hooks/useKnowledgeBaseSelector"; import { useConfig } from "@/hooks/useConfig"; @@ -78,6 +80,21 @@ export default function SelectToolsDialog({ const { confirm } = useConfirmModal(); const { availableTools } = useToolList({ enabled: open }); + const { user } = useAuthorizationContext(); + const tenantId = user?.tenantId || null; + const { serverList: rawServers } = useMcpServerList({ enabled: open, tenantId }); + const allMcpServerNames = useMemo( + () => new Set(rawServers.map((s) => s.service_name)), + [rawServers], + ); + const visibleMcpNames = useMemo( + () => new Set( + rawServers + .filter((s) => !s.permission || s.permission === "EDIT" || s.permission === "READ_ONLY" || s.group_ids) + .map((s) => s.service_name), + ), + [rawServers], + ); const { prefetchKnowledgeBases } = usePrefetchKnowledgeBases(); const { isImageUnderstandingAvailable, isVideoUnderstandingAvailable, isEmbeddingAvailable } = useConfig(); @@ -104,6 +121,7 @@ export default function SelectToolsDialog({ const [configModalOpen, setConfigModalOpen] = useState(false); const [configTool, setConfigTool] = useState(null); const [configParams, setConfigParams] = useState([]); + const [isSelectingAll, setIsSelectingAll] = useState(false); // --- Group tools by source & category --- const sourceGroups = useMemo(() => { @@ -112,8 +130,12 @@ export default function SelectToolsDialog({ const sourceTools = availableTools.filter( (t: any) => t.source === tab.sourceValue ); + // For MCP tools: show API-added (not in server list) or from visible servers + const filteredTools = tab.key === "mcp" + ? sourceTools.filter((t: any) => !allMcpServerNames.has(t.usage) || visibleMcpNames.has(t.usage)) + : sourceTools; const catMap = new Map(); - for (const tool of sourceTools) { + for (const tool of filteredTools) { // MCP tools are grouped by server name (usage); local/langchain by category const cat = tab.key === "mcp" @@ -134,7 +156,7 @@ export default function SelectToolsDialog({ }); } return result; - }, [availableTools]); + }, [availableTools, visibleMcpNames, allMcpServerNames]); // --- Filtered current tab data by search + labels (AND) --- const currentGroups = useMemo(() => { @@ -181,6 +203,10 @@ export default function SelectToolsDialog({ () => new Set(selectedTools.map((t) => parseInt(t.id))), [selectedTools] ); + const activeToolGroup = useMemo( + () => currentGroups.find((group) => group.category === activeCategory), + [activeCategory, currentGroups] + ); // --- Merge instance params for a tool --- const mergeInstanceParams = useCallback( @@ -225,6 +251,45 @@ export default function SelectToolsDialog({ [] ); + const selectableToolsInActiveGroup = useMemo(() => { + if (!activeToolGroup) return []; + + return activeToolGroup.tools.filter((tool: any) => { + const toolId = parseInt(tool.id); + const hasDuplicateName = selectedTools.some( + (selectedTool) => + parseInt(selectedTool.id) !== toolId && + selectedTool.name === tool.name + ); + + return ( + !hasDuplicateName && + !isToolDisabled( + tool.name, + isImageUnderstandingAvailable, + isVideoUnderstandingAvailable, + isEmbeddingAvailable + ) && + !hasMissingRequired(tool.initParams || []) + ); + }); + }, [ + activeToolGroup, + hasMissingRequired, + isEmbeddingAvailable, + isImageUnderstandingAvailable, + isVideoUnderstandingAvailable, + selectedTools, + ]); + const allVisibleSelectableToolsSelected = useMemo( + () => + selectableToolsInActiveGroup.length > 0 && + selectableToolsInActiveGroup.every((tool: any) => + selectedToolIds.has(parseInt(tool.id)) + ), + [selectableToolsInActiveGroup, selectedToolIds] + ); + // --- Open ToolConfigModal (which handles add/update internally) --- const openConfigModal = useCallback( async (tool: any) => { @@ -294,6 +359,67 @@ export default function SelectToolsDialog({ [prefetchKnowledgeBases, mergeInstanceParams, hasMissingRequired, confirm, updateTools, t] ); + const selectAllVisibleTools = useCallback(async () => { + if (isSelectingAll) return; + + const currentSelected = useAgentConfigStore.getState().editedAgent.tools; + const currentSelectedIds = new Set( + currentSelected.map((tool) => parseInt(tool.id)) + ); + const toolsToAdd = selectableToolsInActiveGroup.filter( + (tool: any) => !currentSelectedIds.has(parseInt(tool.id)) + ); + if (toolsToAdd.length === 0) return; + + setIsSelectingAll(true); + try { + const toolsWithParams = await Promise.all( + toolsToAdd.map(async (tool: any) => ({ + ...tool, + initParams: await mergeInstanceParams(tool), + })) + ); + const latestSelected = useAgentConfigStore.getState().editedAgent.tools; + const latestIds = new Set(latestSelected.map((tool) => parseInt(tool.id))); + const names = new Set(latestSelected.map((tool) => tool.name)); + const additions = toolsWithParams.filter((tool) => { + if ( + latestIds.has(parseInt(tool.id)) || + names.has(tool.name) || + hasMissingRequired(tool.initParams) + ) { + return false; + } + names.add(tool.name); + return true; + }); + + if (additions.length > 0) { + updateTools([...latestSelected, ...additions]); + } + } finally { + setIsSelectingAll(false); + } + }, [ + hasMissingRequired, + isSelectingAll, + mergeInstanceParams, + selectableToolsInActiveGroup, + updateTools, + ]); + + const deselectAllVisibleTools = useCallback(() => { + if (!activeToolGroup) return; + + const visibleToolIds = new Set( + activeToolGroup.tools.map((tool: any) => parseInt(tool.id)) + ); + const currentSelected = useAgentConfigStore.getState().editedAgent.tools; + updateTools( + currentSelected.filter((tool) => !visibleToolIds.has(parseInt(tool.id))) + ); + }, [activeToolGroup, updateTools]); + const tabItems: TabsProps["items"] = SOURCE_TABS .filter((tab) => (sourceGroups[tab.key] || []).length > 0) .map((tab) => ({ @@ -371,6 +497,25 @@ export default function SelectToolsDialog({ maxTagCount={1} notFoundContent={allLabels.length === 0 ? t("toolPool.noLabelsAssigned") : undefined} /> +
diff --git a/frontend/app/[locale]/agents/components/agentConfig/tool/ToolConfigModal.tsx b/frontend/app/[locale]/agents/components/agentConfig/tool/ToolConfigModal.tsx index 8e45946ebe..15891f45d6 100644 --- a/frontend/app/[locale]/agents/components/agentConfig/tool/ToolConfigModal.tsx +++ b/frontend/app/[locale]/agents/components/agentConfig/tool/ToolConfigModal.tsx @@ -40,22 +40,17 @@ import KnowledgeBaseSelectorModal from "@/components/tool-config/KnowledgeBaseSe import HaotianKnowledgeSelectorModal, { HaotianKnowledgeSet, } from "@/components/tool-config/HaotianKnowledgeSelectorModal"; -import AidpKnowledgeSelectorModal from "@/components/tool-config/AidpKnowledgeSelectorModal"; +import AidpKnowledgeSelectorModal from "@/ext_components/aidp/AidpKnowledgeSelectorModal"; import { useConfig } from "@/hooks/useConfig"; import { useKnowledgeBasesForToolConfig, knowledgeBaseKeys } from "@/hooks/useKnowledgeBaseSelector"; import { useKnowledgeBaseConfigChangeHandler, ToolKbType, } from "@/hooks/useKnowledgeBaseConfigChangeHandler"; -import { API_ENDPOINTS } from "@/services/api"; import knowledgeBaseService from "@/services/knowledgeBaseService"; import { modelService } from "@/services/modelService"; import log from "@/lib/logger"; import { MODEL_TYPES } from "@/const/modelConfig"; -import { - isEmbeddingModelCompatible as isEmbeddingModelCompatibleBase, - isMultimodalConstraintMismatch as isMultimodalConstraintMismatchBase, -} from "@/lib/knowledgeBaseCompatibility"; import { isZhLocale, getLocalizedDescription, getKbDisplayName, mapKbIdsToDisplayNames, parseKbIds } from "@/lib/utils"; import { ModelOption, ModelType } from "@/types/modelConfig"; @@ -359,14 +354,6 @@ export default function ToolConfigModal({ HaotianKnowledgeSet[] >([]); - const [aidpConfig, setAidpConfig] = useState<{ - serverUrl: string; - apiKey: string; - }>({ - serverUrl: "", - apiKey: "", - }); - // Initialize Haotian config from params useEffect(() => { if (toolKbType !== "haotian_search") return; @@ -382,17 +369,6 @@ export default function ToolConfigModal({ setHaotianConfig({ listUrl, retrieveUrl, authorization: extAuth }); }, [toolKbType, currentParams]); - useEffect(() => { - if (toolKbType !== "aidp_search") return; - const serverUrl = String( - currentParams.find((p) => p.name === "server_url")?.value || "" - ); - const apiKey = String( - currentParams.find((p) => p.name === "api_key")?.value || "" - ); - setAidpConfig({ serverUrl, apiKey }); - }, [toolKbType, currentParams]); - const { data: haotianSetsResult, isFetching: haotianSetsLoading, @@ -575,10 +551,7 @@ export default function ToolConfigModal({ }; } if (toolKbType === "aidp_search") { - return { - serverUrl: aidpConfig.serverUrl, - apiKey: aidpConfig.apiKey, - }; + return {}; } if (toolKbType === "ragflow_search") { if (!ragflowConfig.serverUrl || !ragflowConfig.apiKey) { @@ -595,6 +568,7 @@ export default function ToolConfigModal({ const { data: knowledgeBases = [], isLoading: kbLoading, + isSuccess: isKbListLoaded, refetch: refetchKnowledgeBases, clearKnowledgeBases, } = useKnowledgeBasesForToolConfig(toolKbType, resolveKbConfig()); @@ -670,13 +644,13 @@ export default function ToolConfigModal({ }; case "aidp_search": return { - serverUrl: aidpConfig.serverUrl, - apiKey: aidpConfig.apiKey, + serverUrl: "", + apiKey: "", }; default: return undefined; } - }, [toolKbType, difyConfig, ragflowConfig, datamateServerUrl, idataConfig, aidpConfig]); + }, [toolKbType, difyConfig, ragflowConfig, datamateServerUrl, idataConfig]); useKnowledgeBaseConfigChangeHandler({ toolKbType, @@ -760,100 +734,6 @@ export default function ToolConfigModal({ } }, [isOpen, toolKbType, idataConfig.knowledgeSpaceId]); - // Get current embedding model from config for model matching - const currentEmbeddingModel = useMemo(() => { - try { - const modelConfig = configData?.models; - return ( - modelConfig?.embedding?.modelName || - modelConfig?.embedding?.displayName || - null - ); - } catch { - return null; - } - }, [configData]); - - const currentMultiEmbeddingModel = useMemo(() => { - try { - const modelConfig = configData?.models; - return ( - modelConfig?.multiEmbedding?.modelName || - modelConfig?.multiEmbedding?.displayName || - null - ); - } catch { - return null; - } - }, [configData]); - - const hasEmbeddingModel = Boolean(currentEmbeddingModel); - const hasMultiEmbeddingModel = Boolean(currentMultiEmbeddingModel); - const canToggleMultimodalParam = hasEmbeddingModel && hasMultiEmbeddingModel; - const forcedMultimodalValue = useMemo(() => { - if (!hasEmbeddingModel && hasMultiEmbeddingModel) { - return true; - } - if (hasEmbeddingModel && !hasMultiEmbeddingModel) { - return false; - } - return null; - }, [hasEmbeddingModel, hasMultiEmbeddingModel]); - - const toolMultimodal = useMemo(() => { - const multimodalParam = currentParams.find( - (param) => param.name === "multimodal" - ); - const value = multimodalParam?.value; - if (typeof value === "boolean") { - return value; - } - if (typeof value === "string") { - const normalized = value.trim().toLowerCase(); - if (["true", "1", "yes", "y"].includes(normalized)) return true; - if (["false", "0", "no", "n"].includes(normalized)) return false; - } - return null; - }, [currentParams]); - - useEffect(() => { - if (tool?.name !== "knowledge_base_search") return; - if (forcedMultimodalValue === null) return; - - const index = currentParams.findIndex( - (param) => param.name === "multimodal" - ); - if (index < 0) return; - - const param = currentParams[index]; - if (param.value === forcedMultimodalValue) return; - - const updatedParams = [...currentParams]; - updatedParams[index] = { ...param, value: forcedMultimodalValue }; - setCurrentParams(updatedParams); - - const fieldName = `param_${index}`; - safeSetFieldValue(form, fieldName, forcedMultimodalValue); - }, [tool?.name, forcedMultimodalValue, currentParams, form]); - - const isMultimodalConstraintMismatch = useCallback( - (kb: KnowledgeBase) => { - return isMultimodalConstraintMismatchBase(kb, toolMultimodal); - }, - [toolMultimodal] - ); - - const isEmbeddingModelCompatible = useCallback( - (kb: KnowledgeBase) => { - return isEmbeddingModelCompatibleBase( - kb, - currentEmbeddingModel, - currentMultiEmbeddingModel - ); - }, - [currentEmbeddingModel, currentMultiEmbeddingModel] - ); - // Check if a knowledge base can be selected const canSelectKnowledgeBase = useCallback( (kb: KnowledgeBase): boolean => { @@ -864,16 +744,9 @@ export default function ToolConfigModal({ return false; } - if (kb.source === "nexent") { - if (isMultimodalConstraintMismatch(kb)) { - return false; - } - return isEmbeddingModelCompatible(kb); - } - return true; }, - [isEmbeddingModelCompatible, isMultimodalConstraintMismatch] + [] ); // Track whether this is the first time opening the modal (reset when modal closes) @@ -1182,10 +1055,15 @@ export default function ToolConfigModal({ } }, [knowledgeBases, selectedKbIds]); - // Filter selectedKbIds to only include knowledge bases that exist in the current list - // This handles cases where knowledge bases are no longer available (e.g., wrong URL) + // Filter selected KB IDs to the current accessible list. For AIDP, an + // successfully loaded empty list is meaningful: the current user cannot + // read any of the KBs saved by the agent creator. useEffect(() => { - if (selectedKbIds.length > 0 && knowledgeBases.length > 0) { + const canValidateSelection = + knowledgeBases.length > 0 || + (toolKbType === "aidp_search" && isKbListLoaded); + + if (selectedKbIds.length > 0 && canValidateSelection) { const validKbIds = selectedKbIds.filter((id) => knowledgeBases.some((kb) => String(kb.id).trim() === String(id).trim()) ); @@ -1199,9 +1077,28 @@ export default function ToolConfigModal({ return kb?.display_name || kb?.name || id; }); setSelectedKbDisplayNames(displayNames); + + if (toolKbType === "aidp_search") { + setTestPanelKbIds(validKbIds); + setTestPanelKbDisplayNames(displayNames); + const fieldIndex = currentParams.findIndex((p) => p.name === "kds_list"); + if (fieldIndex !== -1) { + form.setFieldValue(`param_${fieldIndex}`, validKbIds); + } + setCurrentParams((prevParams) => { + const prevFieldIndex = prevParams.findIndex((p) => p.name === "kds_list"); + if (prevFieldIndex === -1) return prevParams; + const updatedParams = [...prevParams]; + updatedParams[prevFieldIndex] = { + ...updatedParams[prevFieldIndex], + value: validKbIds, + }; + return updatedParams; + }); + } } } - }, [knowledgeBases]); + }, [knowledgeBases, isKbListLoaded, toolKbType, selectedKbIds, currentParams, form]); // Force sync selectedKbIds when modal is about to open (kbSelectorVisible changes to true) // This ensures the modal receives the correct selected IDs @@ -1301,11 +1198,8 @@ export default function ToolConfigModal({ return false; } if (toolKbType === "aidp_search") { - if (aidpConfig.serverUrl && aidpConfig.apiKey) { - refetchKnowledgeBases(); - return true; - } - return false; + refetchKnowledgeBases(); + return true; } refetchKnowledgeBases(); return true; @@ -1329,7 +1223,6 @@ export default function ToolConfigModal({ difyConfig, ragflowConfig, haotianConfig, - aidpConfig, ]); // Show sync message when knowledge base selector modal opens @@ -1337,11 +1230,6 @@ export default function ToolConfigModal({ useEffect(() => { // Only trigger when KB selector opens and tool requires KB selection if (kbSelectorVisible && toolRequiresKbSelection && !hasShownSyncMessageRef.current) { - // For AIDP, only sync if credentials are configured to avoid premature "success" message - if (toolKbType === "aidp_search" && (!aidpConfig.serverUrl || !aidpConfig.apiKey)) { - return; - } - // Mark as shown to avoid duplicate messages hasShownSyncMessageRef.current = true; @@ -1739,6 +1627,14 @@ export default function ToolConfigModal({ // Value can be an array or a JSON string ids = parseKbIds(formValue); + if (toolKbType === "aidp_search" && isKbListLoaded) { + ids = ids.filter((id) => + knowledgeBases.some( + (kb) => String(kb.id).trim() === String(id).trim() + ) + ); + } + // Map IDs to display names if (ids.length > 0) { if (toolKbType === "haotian_search" && haotianKnowledgeSets.length > 0) { @@ -1760,7 +1656,11 @@ export default function ToolConfigModal({ } // Fallback to selectedKbDisplayNames if displayNames is empty - if (displayNames.length === 0 && selectedKbDisplayNames.length > 0) { + if ( + toolKbType !== "aidp_search" && + displayNames.length === 0 && + selectedKbDisplayNames.length > 0 + ) { displayNames = selectedKbDisplayNames; ids = selectedKbIds; } @@ -1849,6 +1749,8 @@ export default function ToolConfigModal({ [ form, knowledgeBases, + isKbListLoaded, + toolKbType, selectedKbIds, selectedKbDisplayNames, kbLoading, @@ -2153,6 +2055,13 @@ export default function ToolConfigModal({ if (param.name === "rerank_model_name" && !isRerankEnabled) { return null; } + // Hide server_url / api_key for AIDP search - now read from environment variables + if ( + toolKbType === "aidp_search" && + (param.name === "server_url" || param.name === "api_key") + ) { + return null; + } const fieldName = `param_${index}`; const rules: any[] = []; @@ -2411,8 +2320,6 @@ export default function ToolConfigModal({ onClose={() => setKbSelectorVisible(false)} onConfirm={handleAidpKbConfirm} selectedDatasetIds={isTestPanelKbSelection ? testPanelKbIds : selectedKbIds} - serverUrl={aidpConfig.serverUrl} - apiKey={aidpConfig.apiKey} /> ) : ( )} diff --git a/frontend/app/[locale]/agents/components/agentInfo/AgentGenerateDetail.tsx b/frontend/app/[locale]/agents/components/agentInfo/AgentGenerateDetail.tsx index 965b7b4a9b..3d3b6c686a 100644 --- a/frontend/app/[locale]/agents/components/agentInfo/AgentGenerateDetail.tsx +++ b/frontend/app/[locale]/agents/components/agentInfo/AgentGenerateDetail.tsx @@ -14,6 +14,7 @@ import { Card, App, Alert, + Modal, Tooltip, } from "antd"; import { Tabs, TabsList, TabsTrigger, TabsContent } from "@/components/ui/tabs"; @@ -36,6 +37,7 @@ import { useDeployment } from "@/components/providers/deploymentProvider"; import { useModelList } from "@/hooks/model/useModelList"; import { useCapacityCoverage } from "@/hooks/model/useCapacityCoverage"; import { canManageModels } from "@/lib/auth"; +import { USER_ROLES } from "@/const/auth"; import { useConfig } from "@/hooks/useConfig"; import { useGroupList, useGroupDetails } from "@/hooks/group/useGroupList"; import { usePromptTemplateList } from "@/hooks/agent/usePromptTemplateList"; @@ -44,16 +46,27 @@ import { useAgentConfigStore } from "@/stores/agentConfigStore"; import ExpandEditModal from "./ExpandEditModal"; import PromptTemplateManagerModal from "./PromptTemplateManagerModal"; import PromptOptimizeModal from "./PromptOptimizeModal"; +import GuardrailConfigContent from "./GuardrailConfigContent"; +import type { GuardrailConfigContentRef } from "./GuardrailConfigContent"; import { isAgentPromptsHidden } from "@/lib/agentPromptVisibility"; const { TextArea } = Input; +/** Roles that can edit group settings for any agent (mirrors backend CAN_EDIT_ALL_USER_ROLES). */ +const CAN_EDIT_ALL_ROLES: ReadonlySet = new Set([ + USER_ROLES.SU, + USER_ROLES.ADMIN, + USER_ROLES.SPEED, + USER_ROLES.ASSET_OWNER, +]); + export default function AgentGenerateDetail({}) { const { t } = useTranslation("common"); const { message } = App.useApp(); const { user, getAccessibleGroupIds } = useAuthorizationContext(); const { isSpeedMode } = useDeployment(); const [form] = Form.useForm(); + const [advancedSettingsForm] = Form.useForm(); // Group data - get all groups for tenant, then filter to accessible ones const { data: groupData } = useGroupList(user?.tenantId ?? null); @@ -67,11 +80,21 @@ export default function AgentGenerateDetail({}) { const forceRefreshKey = useAgentConfigStore((state) => state.forceRefreshKey); const isReadOnly = useAgentConfigStore((state) => state.isReadOnly()); const updateAgentConfig = useAgentConfigStore((state) => state.updateAgentConfig); + const setSaveValidation = useAgentConfigStore((state) => state.setSaveValidation); const isGenerating = useAgentConfigStore((state) => state.isGenerating); // Determine if form should be editable (based on isReadOnly only, isGenerating handled separately) const editable = !isReadOnly; + // Group settings (用户组 / 组内权限) are editable only by creator or admin roles + const isAdmin = !!user?.role && CAN_EDIT_ALL_ROLES.has(user.role); + const isCreator = + isCreatingMode || + (!!editedAgent.created_by && + !!user?.id && + String(editedAgent.created_by) === String(user.id)); + const canEditGroupSettings = isAdmin || isCreator; + const { defaultLlmModelConfig } = useConfig(); const { availableLlmModels, models, isLoading: loadingModels } = useModelList(); const { bareModelIds: bareCapacityModelIds } = useCapacityCoverage(); @@ -119,12 +142,23 @@ export default function AgentGenerateDetail({}) { const [promptTemplateManagerOpen, setPromptTemplateManagerOpen] = useState(false); const [optimizeModalOpen, setOptimizeModalOpen] = useState(false); const [optimizeModalType, setOptimizeModalType] = useState<'duty' | 'constraint' | 'few-shots' | null>(null); + const [advancedSettingsOpen, setAdvancedSettingsOpen] = useState(false); + const [advancedSettingsTab, setAdvancedSettingsTab] = useState<"basic" | "guardrail">("basic"); + const [guardrailContentKey, setGuardrailContentKey] = useState(0); + const guardrailContentRef = useRef(null); // Cleanup invalid cache on mount to prevent stuck "generating" state useEffect(() => { clearExpiredGenerationCaches(); }, []); + useEffect(() => { + setSaveValidation(async () => { + await form.validateFields(); + }); + return () => setSaveValidation(null); + }, [form, setSaveValidation]); + // (e.g. business_description from a previously edited agent) useEffect(() => { if (!isCreatingMode) return; @@ -135,7 +169,6 @@ export default function AgentGenerateDetail({}) { // Use agent generation hook const { handleGenerateAgent } = useAgentGeneration({ - setActiveTab, onStreamUpdate: ({ type, content }) => { const fieldMap: Record = { [GENERATE_PROMPT_STREAM_TYPES.DUTY]: 'dutyPrompt', @@ -177,6 +210,18 @@ export default function AgentGenerateDetail({}) { ); }, [availableLlmModels, editedAgent.model, editedAgent.model_ids]); + // The output reserve cap must be safe for every configured model — the user + // can switch models at chat time, so use the minimum max_output_tokens across + // all selected models as the upper bound. + const minModelMaxOutputTokens = useMemo(() => { + const ids = editedAgent.model_ids || []; + const tokens = availableLlmModels + .filter((m) => ids.includes(m.id)) + .map((m) => m.maxOutputTokens) + .filter((t): t is number => t != null && t > 0); + return tokens.length > 0 ? Math.min(...tokens) : undefined; + }, [availableLlmModels, editedAgent.model_ids]); + // Initialize form values when currentAgentId changes or forceRefreshKey updates // Cached generation data is already merged into editedAgent by setCurrentAgent useEffect(() => { @@ -221,6 +266,7 @@ export default function AgentGenerateDetail({}) { mainAgentModelIds: mainAgentModelIds, mainAgentMaxStep: editedAgent.max_step || 15, requestedOutputTokens: editedAgent.requested_output_tokens ?? null, + isMainAgent: editedAgent.is_main_agent ?? true, agentDescription: editedAgent.description || "", group_ids: normalizeNumberArray(editedAgent.group_ids || []), ingroup_permission: editedAgent.ingroup_permission || "READ_ONLY", @@ -256,7 +302,7 @@ export default function AgentGenerateDetail({}) { form.validateFields(["requestedOutputTokens"]).catch(() => {}); } }); - }, [form, selectedMainAgentModel?.maxOutputTokens]); + }, [form, minModelMaxOutputTokens]); // Handle business description change const handleBusinessDescriptionChange = (value: string) => { @@ -475,6 +521,64 @@ export default function AgentGenerateDetail({}) { setOptimizeModalType(null); }; + const handleOpenAdvancedSettings = () => { + advancedSettingsForm.setFieldsValue({ + agentAuthor: editedAgent.author || "", + group_ids: normalizeNumberArray(editedAgent.group_ids || []), + ingroup_permission: editedAgent.ingroup_permission || "READ_ONLY", + mainAgentMaxStep: editedAgent.max_step || 15, + provideRunSummary: editedAgent.provide_run_summary ?? false, + requestedOutputTokens: editedAgent.requested_output_tokens ?? null, + isMainAgent: editedAgent.is_main_agent ?? true, + verificationEnabled: editedAgent.verification_config?.enabled ?? false, + }); + setAdvancedSettingsTab("basic"); + setAdvancedSettingsOpen(true); + }; + + const handleCloseAdvancedSettings = () => { + setAdvancedSettingsOpen(false); + // Remount only the guardrail panel to discard drafts and other temporary UI state. + setGuardrailContentKey((key) => key + 1); + }; + + const handleSaveAdvancedSettings = async () => { + const values = await advancedSettingsForm.validateFields(); + const groupIds = normalizeNumberArray( + values.group_ids ?? editedAgent.group_ids ?? [] + ); + const ingroupPermission = + values.ingroup_permission ?? editedAgent.ingroup_permission ?? "READ_ONLY"; + // Commit guardrail draft from the ref + const guardrailDraft = guardrailContentRef.current?.getDraft(); + const verificationConfig = { + ...(editedAgent.verification_config || DEFAULT_AGENT_VERIFICATION_CONFIG), + enabled: values.verificationEnabled, + ...(guardrailDraft ? { guardrail_config: guardrailDraft } : {}), + }; + + updateAgentConfig({ + author: values.agentAuthor, + group_ids: groupIds, + ingroup_permission: ingroupPermission, + max_step: values.mainAgentMaxStep, + is_main_agent: values.isMainAgent, + provide_run_summary: values.provideRunSummary, + requested_output_tokens: values.requestedOutputTokens ?? null, + verification_config: verificationConfig, + }); + form.setFieldsValue({ + agentAuthor: values.agentAuthor, + group_ids: groupIds, + ingroup_permission: ingroupPermission, + mainAgentMaxStep: values.mainAgentMaxStep, + isMainAgent: values.isMainAgent, + provideRunSummary: values.provideRunSummary, + requestedOutputTokens: values.requestedOutputTokens ?? null, + verificationEnabled: values.verificationEnabled, + }); + setAdvancedSettingsOpen(false); + }; const handleSaveExpandModal = (content: string) => { switch (expandModalType) { @@ -531,6 +635,20 @@ export default function AgentGenerateDetail({}) { } }; + const handlePromptTabChange = (nextTab: string) => { + const promptField = getPromptFieldKey(activeTab as "duty" | "constraint" | "few-shots"); + if (promptField) { + const value = form.getFieldValue(promptField) || ""; + const storeField = { + dutyPrompt: "duty_prompt", + constraintPrompt: "constraint_prompt", + fewShotsPrompt: "few_shots_prompt", + }[promptField] as "duty_prompt" | "constraint_prompt" | "few_shots_prompt"; + updateAgentConfig({ [storeField]: value }); + } + setActiveTab(nextTab); + }; + const handleReplaceOptimizedContent = ( content: string, sectionType: "duty" | "constraint" | "few_shots" @@ -728,7 +846,11 @@ export default function AgentGenerateDetail({}) { > {t("businessLogic.config.template.label")}: - +
-

- {t("agent.detailContent.title")} -

+ +

+ {t("agent.detailContent.title")} +

+ + @@ -828,9 +959,7 @@ export default function AgentGenerateDetail({}) { { - setActiveTab(value); - }} + onValueChange={handlePromptTabChange} className="agent-config-tabs flex flex-col h-full w-full" > @@ -865,7 +994,7 @@ export default function AgentGenerateDetail({}) { > + onChange={(e) => updateAgentConfig({ display_name: e.target.value }) } /> @@ -897,76 +1026,8 @@ export default function AgentGenerateDetail({}) { /> - - - - - { - updateAgentConfig({ ingroup_permission: value }); - }} - /> - - - - - - - - - updateAgentConfig({ author: e.target.value }) - } - /> - - - + - - - - { - const value = form.getFieldValue("mainAgentMaxStep"); - updateAgentConfig({ max_step: value || 1 }); - }} - /> - - - - - { - updateAgentConfig({ - verification_config: { - ...(editedAgent.verification_config || DEFAULT_AGENT_VERIFICATION_CONFIG), - enabled: value, - }, - }); - }} - /> - - - - + + {/* Match the shared agent-detail tab style. */} + + setAdvancedSettingsTab(value as "basic" | "guardrail") + } + className="w-full" + > + + + {t("agent.advancedSettings.tab.basic") || "Basic settings"} + + + {t("agent.guardrail.summaryTitle") || "Guardrail"} + + + + {/* Keep both panels mounted so unsaved form and guardrail state survive tab switches. */} + +
+ + + + + + + + + + + + + + + + + + + + + + + + + + : undefined} + onChange={(e) => handleUpdateRule(index, "name", e.target.value)} + /> + + ); + }, + }, + { + title: ( + + + {t("agent.guardrail.column.pattern") || "Pattern (regex)"} + + + ), + dataIndex: "pattern", + render: (_, record, index) => { + const error = validatePattern(record.pattern); + return ( +
+ setFocusedPatternIndex(index)} + onChange={(e) => handleUpdateRule(index, "pattern", e.target.value)} + /> + {error && ( + + {error} + + )} +
+ ); + }, + }, + { + title: ( + + + {t("agent.guardrail.column.severity") || "Severity"} + + + ), + dataIndex: "severity", + width: 110, + render: (_, record, index) => ( + handleUpdateRule(index, "description", e.target.value)} + /> + ), + }, + { + title: "", + width: 90, + render: (_, _record, index) => ( + + + + + + + ); + }; + + return ( +
+ {/* --- Section: AI generation (hidden when guardrail disabled) --- */} + {draft.enabled && ( +
+ {/* Section header */} +
+
+ + {t("agent.guardrail.ai.title") || "Smart Generation"} + + + {aiModeText} + +
+ + {/* Model selector row */} +
+ + {t("agent.guardrail.ai.modelForGen") || "Model"}: + +