Ai updates

README.md

@@ -48,7 +48,7 @@ Notely is built with Electron + React and is designed for project notes, meeting
 - Preview Mermaid diagrams and rendered Markdown content.
 - Create and edit structured technical diagrams with **Draw.io integration** directly from markdown previews, supporting drag-and-drop import for `.drawio` and `.drawio.xml` files, image export, and offline drawing.
 - Visualize the workspace as an interactive note graph.
-- Use built-in AI features powered by Vercel AI SDK (Gemini, Groq, OpenAI) with a local-first **Embeddings Engine** (offline `BGE-small-en-v1.5` ONNX model), recursive SQLite **Knowledge Graph**, global workspace chat, referred source chips, and a diagnostics panel with tool execution traces.
+- Use built-in AI features powered by Vercel AI SDK (Gemini, Groq, OpenAI) with a 3-Brain Architecture (`WorkspaceBrain`, `ReasoningBrain`, `ActionBrain`), autonomous multi-step Planner, semantic domain tools, local-first Embeddings Engine (`BGE-small-en-v1.5` ONNX model), recursive SQLite Knowledge Graph, strict read-only note immutability safeguards, ReAct self-correction engine (`SelfCorrectionEngine`), and an automated diagnostic evaluation harness (`AgentHarness`).
 - Aggregate tasks across notes with **Open Tasks** and **All Tasks** panels.
   - Open Tasks focuses on unchecked items.
   - All Tasks includes open + closed items with filtering and note grouping.

ai/README.md

@@ -1,113 +1,130 @@
-# Notely AI Subsystem Architecture
+# Notely AI Platform — Comprehensive AI & Agent Subsystem Architecture
 
-This directory contains the codebase for Notely's local-first, modular AI platform. Markdown remains the single source of truth, parsed and indexed into offline-first SQLite databases.
+This directory contains the codebase for Notely's local-first, modular AI platform. Markdown notes remain the single source of truth, parsed and indexed into offline-first SQLite databases (`ai-embeddings.db`, `ai-graph.db`, `ai-memory.db`).
 
 ---
 
-## Subsystem Architecture Diagram
+## AI Platform Overview & Design Philosophy
+
+Notely's AI is engineered as an **intelligent knowledge companion** rather than a generic LLM chatbot wrapper.
+
+### Core Guiding Principles:
+1. **Human-like & Natural**: Speaks like a knowledgeable pair programmer and workspace teammate. Never exposes internal technical mechanics (`"search_notes"`, `"vector similarity"`, `"knowledge graph nodes"`).
+2. **Context-Aware & Grounded**: Proactively retrieves workspace facts before generating answers. All claims are grounded in verified note file links (`[file.md](file:///path)`).
+3. **Multi-Tool Planning & Orchestration**: Dynamically executes parallel retrievals, chains tool outputs, and evaluates evidence confidence before answer synthesis.
+4. **Strict Note Immutability**: Existing notes are **100% read-only**. AI tools cannot update, edit, move, rename, or delete existing user notes under any circumstances.
+5. **Local-First & Provider Agnostic**: Leverages local ONNX embeddings (`BGE-small-en-v1.5`) and background worker processes (`utilityProcess`), while supporting Gemini, Groq, OpenAI, and Local GGUF models.
+
+---
+
+## Complete 3-Brain Subsystem Architecture
 
 ```mermaid
 graph TD
-  %% Frontend Layer
-  subgraph Frontend ["React UI Components"]
-    AIChatPanel["AIChatPanel.jsx (Sidebar Chat)"]
-    AIPalette["AIPalette.jsx (Editor Inline AI)"]
-    AIHealthPage["AIHealthPage.jsx (Diagnostics & Traces)"]
-    EmbeddingsPage["EmbeddingsPage.jsx (Status & Queue)"]
-    AIPersonasManager["AIPersonasManager.jsx (Persona Registry)"]
+  %% Frontend & IPC
+  subgraph Client ["UI & IPC Bridge"]
+    UI["AIChatPanel / AIPalette"]
+    Diagnostics["AIHealthPage.jsx (Diagnostics & Traces)"]
+    IPC["Electron IPC Handlers (aiHandlers.cjs)"]
   end
 
-  %% IPC Bridge
-  subgraph IPC ["Electron IPC Interface"]
-    Preload["preload.cjs (IPC Exposure)"]
-    AIHandlers["aiHandlers.cjs (IPC Main Listeners)"]
+  %% 3-Brain Core
+  subgraph Core ["3-Brain Architectural Triad & Orchestration"]
+    Agent["Agent.js (Central Orchestrator)"]
+    WorkspaceBrain["WorkspaceBrain.js (Factual Retrieval)"]
+    ReasoningBrain["ReasoningBrain.js (Pure Synthesis)"]
+    ActionBrain["ActionBrain.js (Permission Gatekeeper)"]
+    ContextOrchestrator["ContextOrchestrator.js (Multi-Tool Engine)"]
+    Planner["Planner.js (Intent Classifier)"]
+    SelfCorrectionEngine["SelfCorrectionEngine.js (ReAct Validator)"]
   end
 
-  %% Backend AI Core
-  subgraph Backend ["AI Core Backend Subsystem"]
-    AIService["AIService.js (Central Coordinator)"]
-    Agent["Agent.js (Orchestrator)"]
-    QueryExecutor["QueryExecutor.js (Tool Runner Loop)"]
-    ContextEngine["ContextEngine.js (Prompt Assembly)"]
-    EmbeddingService["EmbeddingService.js (Vector Cache)"]
-    GraphRetriever["GraphRetriever.js (Recursive CTE Traversal)"]
-    SemanticRetriever["SemanticRetriever.js (Cosine Search)"]
+  %% Retrieval & Tools
+  subgraph Retrieval ["Context & Retrieval Engine"]
+    ContextEngine["ContextEngine.js (8-Layer Pipeline)"]
+    HybridRetriever["HybridRetriever.js (Reciprocal Rank Fusion)"]
+    SemanticRetriever["SemanticRetriever.js (Vector Cosine Search)"]
+    GraphRetriever["GraphRetriever.js (Recursive CTE Graph Walk)"]
+    SemanticTools["SemanticTools.js (Domain Tool Suite)"]
   end
 
-  %% Storage Layer
-  subgraph Storage ["SQLite Local DBs (journal_mode = WAL)"]
-    EmbedDB["ai-embeddings.db (Note Chunk Vectors)"]
-    GraphDB["ai-graph.db (Entity Relations)"]
+  %% Storage
+  subgraph Storage ["SQLite Storage (WAL Mode)"]
+    EmbedDB["ai-embeddings.db (Chunk Vectors)"]
+    GraphDB["ai-graph.db (Entity Relations & Evidence)"]
     MemoryDB["ai-memory.db (Chats & Traces)"]
   end
 
-  %% Pipelines
-  subgraph Background ["Async Indexing Pipeline"]
-    IndexQueue["IndexQueue.js (Priority Jobs)"]
-    IndexWorker["IndexWorker.js (Non-blocking Thread)"]
-    ONNXEmbedder["ONNXEmbedder.js (Local BGE-small weights)"]
-  end
-
-  %% Frontend to IPC connections
-  AIChatPanel -->|IPC calls| Preload
-  AIHealthPage -->|IPC calls| Preload
-  AIPersonasManager -->|IPC calls| Preload
-  Preload -->|IPC events| AIHandlers
-
-  %% IPC to Coordinator
-  AIHandlers -->|Delegates to| AIService
-  AIService -->|Orchestrates| Agent
-
-  %% Agent Subsystem Routing
-  Agent --> QueryExecutor
-  Agent --> ContextEngine
-
-  %% Context Engine Retrievals
-  ContextEngine --> SemanticRetriever
-  ContextEngine --> GraphRetriever
-
-  %% DB bindings
+  %% Data Flow
+  UI & Diagnostics --> IPC --> Agent
+  Agent --> ContextOrchestrator & WorkspaceBrain & ReasoningBrain & ActionBrain & Planner
+  ContextOrchestrator --> ParallelRetrieval["Promise.allSettled Parallel Tools"]
+  ParallelRetrieval --> SemanticTools & HybridRetriever
+  HybridRetriever --> SemanticRetriever & GraphRetriever
   SemanticRetriever --> EmbedDB
   GraphRetriever --> GraphDB
   Agent --> MemoryDB
-
-  %% Indexing pipeline
-  AIHandlers -->|Triggers rebuild| IndexQueue
-  IndexQueue -->|Pops job| IndexWorker
-  IndexWorker -->|Saves chunks| EmbedDB
-  IndexWorker -->|Calls local vectorizer| ONNXEmbedder
+  ReasoningBrain --> SelfCorrectionEngine
 ```
 
 ---
 
-## Data Locality & Files
+## Subsystem Component Reference
+
+### 1. 3-Brain Architectural Triad & Orchestrator
+
+| Component | File Path | Architectural Responsibility | Key Safeguards & Capabilities |
+|---|---|---|---|
+| **ContextOrchestrator** | [`ai/core/ContextOrchestrator.js`](file:///c:/Users/oksbw/OneDrive/Desktop/Antigravity%20Workspace/Notely/ai/core/ContextOrchestrator.js) | Multi-Tool Planning & Context Aggregation | Runs parallel tool execution (`Promise.allSettled`), output chaining, evidence deduplication, and confidence looping. |
+| **WorkspaceBrain** | [`ai/core/WorkspaceBrain.js`](file:///c:/Users/oksbw/OneDrive/Desktop/Antigravity%20Workspace/Notely/ai/core/WorkspaceBrain.js) | Factual Retrieval & Context Aggregation | Proactively gathers active note text, vector similarity matches, and graph hops for every query. |
+| **ReasoningBrain** | [`ai/core/ReasoningBrain.js`](file:///c:/Users/oksbw/OneDrive/Desktop/Antigravity%20Workspace/Notely/ai/core/ReasoningBrain.js) | Analytical Reasoning & Synthesis | Synthesizes natural human responses. Has **zero direct access to disk or SQLite**. |
+| **ActionBrain** | [`ai/core/ActionBrain.js`](file:///c:/Users/oksbw/OneDrive/Desktop/Antigravity%20Workspace/Notely/ai/core/ActionBrain.js) | Permission Gatekeeper & Execution Safety | Permanently blocks `update_note`, `delete_note`, `move_note`, `rename_note`. Rejects file overwrites on `create_note`. |
+
+### 2. Planning & Tool Ecosystem
+
+| Component | File Path | Responsibility | Capabilities |
+|---|---|---|---|
+| **Planner** | [`ai/core/Planner.js`](file:///c:/Users/oksbw/OneDrive/Desktop/Antigravity%20Workspace/Notely/ai/core/Planner.js) | Intent Classification & Planning | Classifies query intent (`DirectQuery`, `TopicExploration`, `TimelineReconstruction`, `TaskSummary`) and generates internal plan graphs. |
+| **SemanticTools** | [`ai/tools/SemanticTools.js`](file:///c:/Users/oksbw/OneDrive/Desktop/Antigravity%20Workspace/Notely/ai/tools/SemanticTools.js) | High-Level Domain Tools | Exposes `find_discussions`, `find_architecture`, `find_people_and_tasks`, `reconstruct_timeline`, `explore_topic_graph`. |
+
+### 3. Prompting, Persona & Grounding System
+
+| Component | File Path | Responsibility | Features |
+|---|---|---|---|
+| **PromptLibrary** | [`ai/core/PromptLibrary.js`](file:///c:/Users/oksbw/OneDrive/Desktop/Antigravity%20Workspace/Notely/ai/core/PromptLibrary.js) | Modular System Prompts | Assembles base policies, dynamic domain context inference, active persona instructions, and workspace context. |
+| **GroundingEngine** | [`ai/core/GroundingEngine.js`](file:///c:/Users/oksbw/OneDrive/Desktop/Antigravity%20Workspace/Notely/ai/core/GroundingEngine.js) | Citation Link Validator | Audits file link citations (`[label](file:///path)`) against disk and strips broken links before response output. |
+| **SelfCorrectionEngine**| [`ai/core/SelfCorrectionEngine.js`](file:///c:/Users/oksbw/OneDrive/Desktop/Antigravity%20Workspace/Notely/ai/core/SelfCorrectionEngine.js) | ReAct Response Validation Pass | Intercepts draft responses, strips leaked technical tool narration jargon, and validates grounding. |
+
+---
+
+## Multi-Tool Planning & Context Orchestration
 
-All application-wide settings and keys reside in the system Application Data folder (`%AppData%/notely/`), while workspace-scoped indexes reside in the hidden `.notes-app/` folder:
+`ContextOrchestrator.js` implements an autonomous evidence-gathering workflow:
 
-| Scope | Filename | Purpose |
-|---|---|---|
-| **Global** | `ai-config.json` | API keys (encrypted via safeStorage) & active provider configuration |
-| **Global** | `ai-preferences.json` | Feature flags, custom personas, and active embedding provider selections |
-| **Global** | `ai-model/` | Downloaded local ONNX model weights (`BGE-small-en-v1.5`, ~130MB) |
-| **Workspace** | `ai-embeddings.db` | Note chunk text, coordinate line mappings, vectors, and queue states |
-| **Workspace** | `ai-graph.db` | Note node references and entity-relation triples |
-| **Workspace** | `ai-memory.db` | Chat history messages, custom traces metadata, and user patterns log |
+1. **Internal Intent & Plan Generation**: `Planner.js` creates candidate retrieval steps based on query classification (`DirectQuery`, `TopicExploration`, `TimelineReconstruction`, `TaskSummary`). The plan is strictly internal and never exposed to the user.
+2. **Parallel & Chained Tool Execution**: Independent semantic tools (`find_discussions`, `explore_topic_graph`, `find_architecture`) execute concurrently using `Promise.allSettled`. Outputs chain into subsequent retrieval steps.
+3. **Context Aggregation & Consolidation**:
+   - Evidence deduplication across tools and vector chunks.
+   - Relevance score ranking and source file attribution (`[file.md](file:///path)`).
+   - Trace telemetry collection (`executionTrace`) capturing all graph hops, tool parameters, and execution outputs for the UI **AI Health & Diagnostics** page (`AIHealthPage.jsx`).
+4. **Confidence Evaluation Loop**: Measures overall evidence confidence ($0.0 - 1.0$). If confidence $< 0.70$ and iterations $< 3$, performs additional graph expansion or discussion lookups.
+5. **Curated Handoff**: Supplies consolidated evidence payload to `ReasoningBrain.js` for answer synthesis.
 
 ---
 
-## Optimizations
+## Verification & Test Suite Execution
 
-### 1. SQLite WAL Mode
-All database connections are initialized with:
-```sql
-PRAGMA foreign_keys = ON;
-PRAGMA journal_mode = WAL;
-PRAGMA synchronous = NORMAL;
-```
-This reduces disk write overhead and permits simultaneous reads without write blocking.
+All AI subsystem components are covered by Vitest test suites under `tests/ai/`:
 
-### 2. In-Memory Vector Cache
-`EmbeddingService.js` maintains an in-memory cache map for hot embedding vectors. Repeated calls to `generateVector` or `generateEmbedding` for the same text contents return values instantly without calling remote LLM API providers or local ONNX runtime calculations.
+```bash
+node node_modules/vitest/vitest.mjs run tests/ai
+```
 
-### 3. TTL Graph Cache
-`GraphRetriever.js` caches recursive CTE Knowledge Graph queries for 60 seconds (`TTL = 60000ms`), preventing redundant query execution on large workspaces during fast, conversational chat sequences.
+### Test Suite Map (27 Test Files / 72 Tests Passing 100%):
+- `tests/ai/orchestrator.spec.js`: Multi-tool planning, parallel retrieval & evidence aggregation tests.
+- `tests/ai/brainTriad.spec.js`: 3-Brain isolation & note immutability tests.
+- `tests/ai/planner.spec.js`: Intent classification & semantic tools tests.
+- `tests/ai/grounding.spec.js`: Citation link verification & prompt composition tests.
+- `tests/ai/selfCorrection.spec.js`: ReAct validation pass & zero-jargon gate tests.
+- `tests/ai/harness.spec.js`: Evaluation harness metrics tests.
+- `tests/ai/knowledgeGraph.spec.js`: Recursive CTE graph traversal & UTC date matching tests.

ai/context/ContextEngine.js

@@ -77,7 +77,7 @@ class ContextEngine {
 
     log.info(`Context built for conversation=${conversationId} persona=${personaId} msgs=${messages.length}`);
 
-    return { system, messages, tools };
+    return { system, messages, tools, personaId };
   }
 }
 

ai/context/HybridRetriever.js

@@ -90,18 +90,34 @@ class HybridRetriever {
       } else {
         try {
           const fs = require('fs');
-          if (fs.existsSync(notePath)) {
+          if (fs.existsSync(notePath) && fs.statSync(notePath).isFile()) {
             content = fs.readFileSync(notePath, 'utf8');
           }
         } catch (err) {
           log.warn(`Failed to read note during hybrid search fallback: ${err.message}`);
         }
       }
 
+      // Attach graph relations & evidence triples for the matched note
+      let graphTriples = [];
+      if (this.graphRetriever) {
+        try {
+          const rels = this.graphRetriever.traverse(notePath, 1);
+          graphTriples = (rels || []).map(r => {
+            let line = `(${r.from_type || 'Entity'}) ${r.from_name || r.from_path} --[${r.relation}]--> (${r.to_type || 'Entity'}) ${r.to_name || r.to_path}`;
+            if (r.evidence) {
+              line += ` (Evidence: "${r.evidence}")`;
+            }
+            return line;
+          });
+        } catch { /* ignore graph lookup error */ }
+      }
+
       results.push({
         note_path: notePath,
         content: content.slice(0, 4000), // budget preview limit
-        score: score
+        score: score,
+        graph_triples: graphTriples
       });
     }
 
@@ -125,9 +141,13 @@ class HybridRetriever {
       execute: async ({ query, activeNotePath = null, topK = 5 }) => {
         const results = await this.search(query, activeNotePath, topK);
         if (!results.length) return 'No relevant note content found.';
-        return results.map((r, i) =>
-          `[${i + 1}] ${r.note_path} (RRF score: ${r.score.toFixed(4)})\n${r.content}`
-        ).join('\n\n');
+        return results.map((r, i) => {
+          let output = `[${i + 1}] ${r.note_path} (RRF score: ${r.score.toFixed(4)})\n${r.content}`;
+          if (r.graph_triples && r.graph_triples.length) {
+            output += `\n\nKnowledge Graph Connections:\n  * ` + r.graph_triples.slice(0, 10).join('\n  * ');
+          }
+          return output;
+        }).join('\n\n---\n\n');
       }
     };
   }

ai/context/SemanticRetriever.js

@@ -54,12 +54,22 @@ class SemanticRetriever {
       }
     }
 
-    if (!scored.length) return [];
+    if (!scored.length) {
+      return typeof this.embeddingDB.searchTextFallback === 'function'
+        ? this.embeddingDB.searchTextFallback(query, topK)
+        : [];
+    }
+
+    // Sort scored vector items by similarity descending
+    scored.sort((a, b) => b.score - a.score);
 
-    // Filter by similarity score threshold (e.g. >= 0.70)
-    const thresholdFiltered = scored.filter(item => item.score >= 0.70);
+    // Filter by realistic local similarity threshold (>= 0.35 for ONNX BGE embeddings)
+    let thresholdFiltered = scored.filter(item => item.score >= 0.35);
 
-    thresholdFiltered.sort((a, b) => b.score - a.score);
+    // Fallback to keyword search if no vector matches passed threshold
+    if (thresholdFiltered.length === 0 && typeof this.embeddingDB.searchTextFallback === 'function') {
+      return this.embeddingDB.searchTextFallback(query, topK);
+    }
 
     // Deduplicate by note_path to avoid duplicate chunks from the same note
     const seenNotes = new Set();

ai/core/ActionBrain.js

@@ -0,0 +1,70 @@
+/**
+ * ActionBrain - Execution gatekeeper & permission validator for Notely AI
+ * Strictly enforces zero-edit / read-only safety for existing notes.
+ */
+
+const fs = require('fs');
+const path = require('path');
+
+class ActionBrain {
+  constructor(agent) {
+    this.agent = agent;
+    this.forbiddenActions = new Set([
+      'update_note',
+      'edit_note',
+      'delete_note',
+      'move_note',
+      'rename_note',
+      'overwrite_note'
+    ]);
+  }
+
+  /**
+   * Validate action request before execution
+   * @param {string} actionName
+   * @param {object} params
+   * @returns {{ allowed: boolean, reason?: string }}
+   */
+  validateAction(actionName, params = {}) {
+    const normalizedName = String(actionName || '').toLowerCase().trim();
+
+    if (this.forbiddenActions.has(normalizedName)) {
+      return {
+        allowed: false,
+        reason: `Action '${actionName}' is strictly prohibited. AI is restricted from modifying, moving, or deleting existing workspace notes.`
+      };
+    }
+
+    if (normalizedName === 'create_note') {
+      const title = String(params.title || params.note_title || params.name || 'Untitled').trim();
+      const fileName = title.endsWith('.md') ? title : `${title}.md`;
+      const targetDir = params.subfolder ? path.join(this.agent.workspaceRoot, params.subfolder) : this.agent.workspaceRoot;
+      const fullPath = path.join(targetDir, fileName);
+
+      if (fs.existsSync(fullPath)) {
+        return {
+          allowed: false,
+          reason: `Note '${fileName}' already exists. Overwriting existing notes is strictly disabled.`
+        };
+      }
+    }
+
+    return { allowed: true };
+  }
+
+  /**
+   * Execute validated action
+   * @param {string} actionName
+   * @param {object} params
+   * @param {Function} runner
+   */
+  async execute(actionName, params, runner) {
+    const validation = this.validateAction(actionName, params);
+    if (!validation.allowed) {
+      throw new Error(`[ActionBrain Gate Error]: ${validation.reason}`);
+    }
+    return runner(params);
+  }
+}
+
+module.exports = ActionBrain;