Release GraphRAG 2.0.2

CHANGELOG.md

@@ -1,5 +1,23 @@
 # Changelog
 
+## [2.0.2]
+
+### Added
+- **The Migration Assistant reports graph data-integrity health.** Alongside the query check, it now shows how many vertices are missing embeddings (by type) and how many communities have a placeholder or empty summary.
+- **Targeted regeneration from the Migration Assistant.** Missing embeddings can be re-embedded, and communities with placeholder/empty summaries can be re-summarized, for just the affected items — without running a full rebuild. Items whose source content is unusable are reported as needing a rebuild.
+
+### Changed
+- **Agentic tool-calling support is detected at runtime.** Whether the chat model can drive Agentic mode is now confirmed by a lightweight runtime probe (checked once and cached in memory, re-checked after a restart) instead of a fixed model list, so current and future tool-calling models — including new providers — enable Agentic mode automatically, and a model that can't tool-call falls back to the classic engine until its configuration changes.
+- **The planned agent falls back to document search when a structured query returns nothing.** If the planned agent's structured query returns no rows, it now runs a hybrid document search before answering, so an empty structured result no longer produces a non-answer; it follows the same fallback setting as the classic engine.
+- **Clearer message when an answer can't be generated.** When the assistant is unable to produce an answer, it now suggests rephrasing the question to be more specific and contacting an administrator if the problem continues, instead of a bare failure notice.
+- **Token cost rates can be set per service.** LLM Config accepts optional input/output USD-per-1M-token rates for the completion, chatbot, and multimodal services; when set, Est. Cost uses them so cost is accurate for models the pricing library doesn't recognize (otherwise it keeps the library's pricing).
+
+### Fixed
+- **Knowledge-graph rebuild no longer fails on document names containing parentheses.** Document names with `(` or `)` are normalized without dropping the rest of the name, so rebuilds complete instead of erroring while creating chunks.
+- **PDF chart labels and table numbers survive extraction.** Picture-text blocks are kept intact, stacked and comma-grouped chart values are no longer glued together, and pages with unreadable embedded fonts are recovered from a page image — so figures and tables reach the graph with their numbers intact.
+- **Newer Gemini models are available in agentic chat.** Gemini 3.x and later families are recognized as tool-calling models, so agentic mode works with them instead of silently falling back to classic chat.
+- **Chat Stop button behaves correctly.** Stopping a response just as it finishes no longer sends a stray empty message, and the Stop button is the right size in the light theme.
+
 ## [2.0.1]
 
 ### Changed

README.md

@@ -515,7 +515,7 @@ Copy the below code into `configs/server_config.json`. You shouldn’t need to c
 | `retrieval_include_entity` | bool \| null | `null` (auto) | Whether retriever queries include the generic `Entity` vertex alongside domain types. When unset, the server uses `false` if a domain schema exists and `true` otherwise. Set explicitly to override. |
 | `schema_max_sample_files` | int | `5` | Maximum number of sample documents accepted by the *Generate from sample documents* path on the *Initialize Knowledge Graph* dialog. |
 | `schema_max_total_mb` | int | `50` | Combined upload cap (MB) across all sample files for schema extraction. Bounds the content sent to the LLM. A single file may use the full budget; no separate per-file cap. |
-| `enable_router_fallback` | bool | `true` | When the function-call or Cypher path fails after 3 retries, fall back to vector search instead of failing the query. |
+| `enable_router_fallback` | bool | `true` | When a structured query returns no result, fall back to vector search instead of failing the query. Applies to the classic engine (after the function-call or Cypher path fails its retries) and to the planned agent (when a structural query returns no rows). |
 | `chunker_config` | object | `{}` | Chunker-specific settings (see sub-parameters below). All settings are saved regardless of which chunker is selected as default. |
 | ↳ `chunk_size` | int | `2048` | Maximum number of characters per chunk. Used by `character`, `markdown`, `html`, and `recursive` chunkers. Larger values produce fewer, bigger chunks; smaller values produce more, finer-grained chunks. |
 | ↳ `overlap_size` | int | 1/8 of `chunk_size` | Number of overlapping characters between consecutive chunks. Used by `character`, `markdown`, `html`, and `recursive` chunkers. More overlap preserves cross-chunk context but increases total chunk count. Set to `0` for no overlap. |

VERSION

@@ -1 +1 @@
-2.0.1
+2.0.2

common/chunkers/structured.py

@@ -140,15 +140,28 @@ class Element:
 # pymupdf4llm artifacts:
 #  • "==> picture [WxH] intentionally omitted <==" — image dropped (skip line)
 #  • "----- Start of picture text -----" / "----- End of picture text -----"
-#    bracket OCR'd content inside an image; we fold the body into the figure
-#    so chart-internal labels stay with the image chunk.
+#    (markdown) or ``<!-- Start of picture text -->`` (HTML-comment form)
+#    bracket pymupdf4llm picture-text inside a figure; we fold the body into
+#    the figure so chart-internal labels stay with the image chunk.
 _MD_PICTURE_OMITTED = re.compile(r"^\s*\*+\s*==>\s*picture\b.*intentionally omitted\s*<==\s*\*+.*$", re.IGNORECASE)
-_MD_PICTURE_TEXT_START = re.compile(r"^\s*\*+\s*-+\s*Start of picture text\s*-+\s*\*+\s*(<br\s*/?>)?\s*$", re.IGNORECASE)
-_MD_PICTURE_TEXT_END = re.compile(r"^\s*\*+\s*-+\s*End of picture text\s*-+\s*\*+\s*(<br\s*/?>)?\s*$", re.IGNORECASE)
+_MD_PICTURE_TEXT_START = re.compile(
+    r"^\s*(?:\*+\s*-+\s*Start of picture text\s*-+\s*\*+|"
+    r"<!--\s*Start of picture text\s*-->)\s*(?:<br\s*/?>)?\s*$",
+    re.IGNORECASE,
+)
+_MD_PICTURE_TEXT_END = re.compile(
+    r"^\s*(?:\*+\s*-+\s*End of picture text\s*-+\s*\*+|"
+    r"<!--\s*End of picture text\s*-->)\s*(?:<br\s*/?>)?\s*$",
+    re.IGNORECASE,
+)
 # Inline variant of the End marker: the picture-text body can arrive as a
 # single <br>-joined line with the marker on its tail, so it is not always
 # line-anchored. Searched anywhere in a line to terminate the block.
-_MD_PICTURE_TEXT_END_INLINE = re.compile(r"\*+\s*-+\s*End of picture text\s*-+\s*\*+\s*(?:<br\s*/?>)?", re.IGNORECASE)
+_MD_PICTURE_TEXT_END_INLINE = re.compile(
+    r"(?:\*+\s*-+\s*End of picture text\s*-+\s*\*+|"
+    r"<!--\s*End of picture text\s*-->)\s*(?:<br\s*/?>)?",
+    re.IGNORECASE,
+)
 
 
 def _flush_prose(buf: List[str], heading: Optional[str], page: Optional[int], out: List[Element]) -> None:
@@ -312,18 +325,14 @@ def markdown_to_elements(md: str, page: Optional[int] = None) -> List[Element]:
             i += 1
             continue
 
-        # 5b. Other HTML comments (chunk markers etc.) — skip.
-        if _MD_HTML_COMMENT.match(line):
-            i += 1
-            continue
-
         # 5c. pymupdf4llm "==> picture ... intentionally omitted <==" — drop.
         if _MD_PICTURE_OMITTED.match(line):
             i += 1
             continue
 
-        # 5d. pymupdf4llm picture-text block: ----- Start ... End of picture
-        #     text ----- wraps OCR'd content (chart axis labels, legends).
+        # 5d. pymupdf4llm picture-text block (markdown dash markers OR HTML
+        #     comments). Must run before the generic HTML-comment skip so
+        #     ``<!-- Start of picture text -->`` is not dropped.
         #     Fold the body into the immediately preceding figure when
         #     present so chart-internal text travels with the image.
         if _MD_PICTURE_TEXT_START.match(line):
@@ -361,6 +370,12 @@ def markdown_to_elements(md: str, page: Optional[int] = None) -> List[Element]:
                 out.append(Element(kind="figure", text=body, heading=heading, page=page))
             continue
 
+        # 5b. Other HTML comments (chunk markers etc.) — skip.
+        #     Picture-text comments are handled above.
+        if _MD_HTML_COMMENT.match(line):
+            i += 1
+            continue
+
         # 6. Blank line — flush current prose paragraph.
         if not stripped:
             _flush_prose(prose_buf, heading, page, out)

common/config.py

@@ -187,8 +187,13 @@ def resolve_llm_services(llm_cfg: dict) -> dict:
     embedding_provider = embedding.get("embedding_model_service", "").lower()
     completion_provider = completion.get("llm_service", "").lower()
     if embedding_provider and embedding_provider == completion_provider:
-        # Identity/schema keys that belong to the embedding service itself
-        embedding_own_keys = {"embedding_model_service", "model_name", "authentication_configuration", "token_limit"}
+        # Identity/schema keys that belong to the embedding service itself.
+        # Token-cost rates stay per-service too — completion's per-1M rates
+        # must not be applied to embedding usage, which is priced separately.
+        embedding_own_keys = {
+            "embedding_model_service", "model_name", "authentication_configuration",
+            "token_limit", "input_cost_per_1m", "output_cost_per_1m",
+        }
         for k, v in completion.items():
             if k not in embedding_own_keys and k not in embedding:
                 embedding[k] = v

common/db/health.py

@@ -0,0 +1,88 @@
+"""Data-integrity health checks for the Migration Assistant: embedding coverage
+and community-summary completeness.
+
+Deterministic, read-only — NO LLM calls and NO embedding-service init. They only
+run count queries + a schema check over an existing connection, so they are safe
+on a polled/triggered status check. The connection is the synchronous one the
+status endpoint already holds.
+"""
+
+import logging
+
+from common.utils.summary_placeholders import PLACEHOLDER_MARKERS
+
+logger = logging.getLogger(__name__)
+
+_VECTOR_ATTR = "embedding"
+
+
+def _has_vector_attr(conn, v_type: str) -> bool:
+    """True if *v_type* carries the embedding vector attribute, per the live
+    schema. Uses the connection's schema API — no embedding-service init."""
+    # Native vector attributes are exposed under ``EmbeddingAttributes`` in the
+    # schema, not by getVertexAttrs (which omits them).
+    try:
+        for v in conn.getSchema().get("VertexTypes", []):
+            if v.get("Name") == v_type:
+                return any(
+                    e.get("Name") == _VECTOR_ATTR
+                    for e in (v.get("EmbeddingAttributes") or [])
+                )
+        return False
+    except Exception:
+        return False
+
+
+def embeddable_types(conn) -> list[str]:
+    """Vertex types that carry the embedding vector attribute. Schema-detected
+    (not hardcoded) so it stays correct as the embedded set changes."""
+    # Native vector attributes are NOT returned by getVertexAttrs/getVertexTypes;
+    # they live under each vertex type's ``EmbeddingAttributes`` in the schema.
+    try:
+        schema = conn.getSchema()
+    except Exception as e:
+        logger.warning(f"embeddable_types: getSchema failed: {e}")
+        return []
+    out = []
+    for v in schema.get("VertexTypes", []):
+        if any(e.get("Name") == _VECTOR_ATTR for e in (v.get("EmbeddingAttributes") or [])):
+            out.append(v.get("Name"))
+    return out
+
+
+def embedding_coverage(conn, v_type: str) -> dict | None:
+    """``{"total": M, "missing": N}`` for *v_type*, or ``None`` when the type is
+    not embeddable or the ``vertices_have_embedding`` query is unavailable."""
+    try:
+        if not _has_vector_attr(conn, v_type):
+            return None
+        res = conn.runInstalledQuery(
+            "vertices_have_embedding", params={"vertex_type": v_type}
+        )
+        # PRINT order: [0] all_have_embedding, [1] size (missing), [2] total.
+        missing = int(res[1]["size"])
+        total = int(res[2]["total"])
+        return {"total": total, "missing": missing}
+    except Exception as e:
+        logger.warning(f"embedding_coverage({v_type}) failed: {e}")
+        return None
+
+
+def community_summary_health(conn, markers=None) -> dict | None:
+    """``{"total": M, "needs_resummarize": N}`` for Community vertices, or
+    ``None`` when the ``communities_need_resummarize`` query is unavailable.
+    ``needs_resummarize`` counts communities whose description is empty or a
+    known placeholder (old or new)."""
+    markers = markers if markers is not None else PLACEHOLDER_MARKERS
+    try:
+        res = conn.runInstalledQuery(
+            "communities_need_resummarize", params={"markers": markers}
+        )
+        row = res[0]
+        return {
+            "total": int(row["total"]),
+            "needs_resummarize": int(row["needs_resummarize"]),
+        }
+    except Exception as e:
+        logger.warning(f"community_summary_health failed: {e}")
+        return None

common/db/query_sets.py

@@ -37,6 +37,7 @@
     "common/gsql/graphrag/louvain/stream_community",
     "common/gsql/graphrag/get_community_children",
     "common/gsql/graphrag/communities_have_desc",
+    "common/gsql/graphrag/communities_need_resummarize",
     "common/gsql/graphrag/graphrag_delete_all_communities",
     "common/gsql/graphrag/graphrag_stream_entity_community_pairs",
     "common/gsql/graphrag/graphrag_stream_all_ids",
@@ -69,13 +70,21 @@
     "common/gsql/supportai/Check_Nonexistent_Vertices",
 ]
 
+# Data-integrity health-check queries used by the Migration Assistant panel.
+# vertices_have_embedding is also installed by the embedding store; listing it
+# here lets the Migration Assistant verify/repair it too.
+HEALTH_QUERIES = [
+    "common/gsql/vector/vertices_have_embedding",
+]
+
 # What the Migration Assistant verifies for a GraphRAG graph: everything a
 # GraphRAG graph actually installs. Excludes the opt-in ECC-checker queries.
 MIGRATION_QUERIES = (
     GRAPHRAG_REQUIRED_QUERIES
     + GRAPHRAG_COMMUNITY_QUERIES
     + SUPPORTAI_INIT_QUERIES
     + SUPPORTAI_RETRIEVER_QUERIES
+    + HEALTH_QUERIES
 )
 
 

common/embeddings/tigergraph_embedding_store.py

@@ -566,13 +566,14 @@ def retrieve_similar_with_score(self, query_embedding, top_k=10, similarity_thre
             logger.info(f"Fetch {top_k} similar entries from {vertex_types} with filter {filter_expr}")
 
             start_time = time()
+            # GSQL declares STRING expr=""; pyTigerGraph rejects null.
             verts = self.conn.runInstalledQuery(
                 "get_topk_similar",
                 params={
                     "vertex_types": vertex_types,
                     "query_vector": query_embedding,
                     "top_k": top_k*2,
-                    "expr": filter_expr,
+                    "expr": filter_expr or "",
                 }
             )
             end_time = time()

common/gsql/graphrag/communities_need_resummarize.gsql

@@ -0,0 +1,24 @@
+CREATE OR REPLACE DISTRIBUTED QUERY communities_need_resummarize(SET<STRING> markers, BOOL p=False) SYNTAX V2{
+    // A community needs re-summarization when its description is empty or is a
+    // known placeholder marker (old or new). ``markers`` is passed in so the
+    // set of placeholders stays defined in one place in Python.
+    SumAccum<INT> @@needs;
+    SumAccum<INT> @@total;
+    Comms = {Community.*};
+    Comms = SELECT c FROM Comms:c
+            POST-ACCUM
+                @@total += 1,
+                IF length(c.description) == 0 OR (c.description IN markers) THEN
+                    @@needs += 1
+                END;
+
+    PRINT @@needs AS needs_resummarize, @@total AS total;
+
+    // p=true streams the affected community ids + their Louvain layer so the
+    // regenerate action can re-summarize exactly those.
+    IF p THEN
+        bad = SELECT c FROM Comms:c
+              WHERE length(c.description) == 0 OR (c.description IN markers);
+        PRINT bad[bad.iteration AS iteration] AS results;
+    END;
+}

common/gsql/vector/vertices_have_embedding.gsql

@@ -5,7 +5,8 @@ CREATE OR REPLACE DISTRIBUTED QUERY vertices_have_embedding(String vertex_type,
 
     PRINT (results.size() == 0) as all_have_embedding;
     PRINT results.size() as size;
-
+    PRINT vset.size() as total;
+
     IF verbose THEN
         PRINT results[results.id as id] as results; 
     END;

common/llm_services/base_llm.py

@@ -15,6 +15,7 @@
 import os
 import re
 import logging
+from typing import Optional
 from langchain_core.output_parsers import BaseOutputParser, PydanticOutputParser
 from langchain_core.exceptions import OutputParserException
 from langchain_core.prompts import BasePromptTemplate
@@ -126,7 +127,53 @@ def reset_usage_collection():
     _usage_collector.set(None)
 
 
-def _record_usage(caller_name: str, usage_data: dict):
+def _parse_cost_rate(value) -> Optional[float]:
+    """Return a non-negative float rate, or None if unset/invalid."""
+    if value is None or value == "":
+        return None
+    try:
+        rate = float(value)
+    except (TypeError, ValueError):
+        return None
+    if rate < 0:
+        return None
+    return rate
+
+
+def estimate_cost_from_config(
+    config: Optional[dict],
+    input_tokens: int,
+    output_tokens: int,
+) -> Optional[float]:
+    """USD cost from user-configured per-1M rates, or None if not configured.
+
+    Both ``input_cost_per_1m`` and ``output_cost_per_1m`` must be set on
+    ``config`` (USD per 1M tokens). When present they always override
+    LangChain's built-in cost.
+    """
+    if not config:
+        return None
+    inp_rate = _parse_cost_rate(config.get("input_cost_per_1m"))
+    out_rate = _parse_cost_rate(config.get("output_cost_per_1m"))
+    if inp_rate is None or out_rate is None:
+        return None
+    return (
+        max(0, int(input_tokens or 0)) * inp_rate / 1_000_000.0
+        + max(0, int(output_tokens or 0)) * out_rate / 1_000_000.0
+    )
+
+
+def _record_usage(caller_name: str, usage_data: dict, config: Optional[dict] = None):
+    # User-configured rates replace LangChain cost entirely; otherwise keep
+    # whatever LangChain reported (may be 0 for unknown models).
+    user_cost = estimate_cost_from_config(
+        config,
+        usage_data.get("input_tokens", 0),
+        usage_data.get("output_tokens", 0),
+    )
+    if user_cost is not None:
+        usage_data["cost"] = user_cost
+    logger.info(f"{caller_name} usage: {usage_data}")
     bucket = _usage_collector.get()
     if bucket is not None:
         bucket.append({"caller_name": caller_name, **usage_data})
@@ -500,8 +547,7 @@ def invoke_with_parser(
             usage_data["output_tokens"] = cb.completion_tokens
             usage_data["total_tokens"] = cb.total_tokens
             usage_data["cost"] = cb.total_cost
-            logger.info(f"{caller_name} usage: {usage_data}")
-            _record_usage(caller_name, usage_data)
+            _record_usage(caller_name, usage_data, self.config)
 
         raw_text = self._message_text(raw_output)
 
@@ -545,8 +591,7 @@ def invoke_with_tools(
             usage_data["output_tokens"] = cb.completion_tokens
             usage_data["total_tokens"] = cb.total_tokens
             usage_data["cost"] = cb.total_cost
-            logger.info(f"{caller_name} usage: {usage_data}")
-            _record_usage(caller_name, usage_data)
+            _record_usage(caller_name, usage_data, self.config)
         return resp
 
     def invoke_structured(
@@ -579,8 +624,7 @@ def invoke_structured(
             usage_data["output_tokens"] = cb.completion_tokens
             usage_data["total_tokens"] = cb.total_tokens
             usage_data["cost"] = cb.total_cost
-            logger.info(f"{caller_name} usage: {usage_data}")
-            _record_usage(caller_name, usage_data)
+            _record_usage(caller_name, usage_data, self.config)
         return result
 
     async def ainvoke_with_parser(
@@ -608,8 +652,7 @@ async def ainvoke_with_parser(
             usage_data["output_tokens"] = cb.completion_tokens
             usage_data["total_tokens"] = cb.total_tokens
             usage_data["cost"] = cb.total_cost
-            logger.info(f"{caller_name} usage: {usage_data}")
-            _record_usage(caller_name, usage_data)
+            _record_usage(caller_name, usage_data, self.config)
 
         raw_text = self._message_text(raw_output)