From 176d5bccc3bcaffdda35dff2b410564fa4c49260 Mon Sep 17 00:00:00 2001 From: mkreyman Date: Tue, 30 Jun 2026 15:20:31 -0600 Subject: [PATCH] feat(knowledge): novelty-gated write-back (agents' KB #1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Turn the KB from a blind write sink into a curated one: when an agent proposes an article, gate it against the published corpus instead of publishing whatever it sends. The KB does only the MECHANICAL part (embed → cosine → route); the merge decision stays with the consuming agent, which is a step smarter than the KB. Pipeline (POST /api/v1/articles, default ON; force:true bypasses): ProposalGate.assess → embed '{title}\n\n{body}', VectorSearch.nearest, classify: >= 0.97 duplicate → create nothing, 200 + point at the canonical article >= 0.88 overlap → create as a DRAFT, stamp metadata.proposal_novelty (score + nearest ids) for a reviewer/consumer to merge < 0.88 novel → create on the requested path Response carries so the agent can act in-session. Design choices: - Behaviour + config DI (ProposalAssessorBehaviour / MockProposalAssessor), so propose_article/3 is unit-testable and the assessor is swappable. - Resilient: ANY embedding failure (API down, power/internet outage) or a system-scoped (nil-tenant) proposal falls OPEN (:unknown) → never blocks a write. - Non-destructive: flags duplicates, never edits/deletes; a vanished canonical neighbor falls through to create. - Reuses existing machinery — status :draft + list_drafts review queue, the embedding client, VectorSearch (raw-vector path, no row needed). No new status. - Thresholds config-tunable (:knowledge_proposal_{duplicate,overlap}_threshold). Tests (+19): pure classify bands; real-pgvector assess (duplicate/low/novel/ fall-open/system-scope); propose_article routing incl. canonical-vanished + tenant isolation; controller verdict→HTTP rendering + force bypass. Default stub is :novel so all existing create tests stay green. Full gate green (3037 tests, dialyzer, credo). --- config/config.exs | 7 + config/test.exs | 5 + lib/loopctl/knowledge.ex | 115 ++++++++++++ .../knowledge/proposal_assessor_behaviour.ex | 42 +++++ lib/loopctl/knowledge/proposal_gate.ex | 92 ++++++++++ .../controllers/article_controller.ex | 139 +++++++++++---- test/loopctl/knowledge/proposal_gate_test.exs | 99 +++++++++++ .../knowledge/propose_article_test.exs | 168 ++++++++++++++++++ .../controllers/article_controller_test.exs | 86 +++++++++ test/support/data_case.ex | 7 + test/support/mocks.ex | 1 + 11 files changed, 731 insertions(+), 30 deletions(-) create mode 100644 lib/loopctl/knowledge/proposal_assessor_behaviour.ex create mode 100644 lib/loopctl/knowledge/proposal_gate.ex create mode 100644 test/loopctl/knowledge/proposal_gate_test.exs create mode 100644 test/loopctl/knowledge/propose_article_test.exs diff --git a/config/config.exs b/config/config.exs index f83a3ab..2fe7409 100644 --- a/config/config.exs +++ b/config/config.exs @@ -288,6 +288,13 @@ config :loopctl, # These are source COLLECTIONS, not topics, so a per-tag MOC for them is noise. config :loopctl, :knowledge_moc_excluded_tags, ~w(synology-docs synology-netbackup) +# Novelty-gated write-back (ProposalGate): cosine-similarity bands for an agent's +# proposed article vs. the published corpus. >= duplicate → reject in favour of the +# canonical article; >= overlap → route to a draft for the consumer to resolve; +# below → novel, created on the requested path. +config :loopctl, :knowledge_proposal_duplicate_threshold, 0.97 +config :loopctl, :knowledge_proposal_overlap_threshold, 0.88 + # DI: WebAuthn adapter — defaults to Wax (overridden in test env) config :loopctl, :webauthn_adapter, Loopctl.WebAuthn.Wax diff --git a/config/test.exs b/config/test.exs index 802d4ca..1f6201b 100644 --- a/config/test.exs +++ b/config/test.exs @@ -240,6 +240,11 @@ config :loopctl, :secrets_adapter, Loopctl.MockSecrets # overrides it with Mox.expect/3 to inject a deterministic Postgrex.Error. config :loopctl, :knowledge_suggest_links, Loopctl.MockSuggestLinks +# Novelty-gated write-back: swap the proposal assessor for a mock so propose_article +# tests choose the verdict deterministically. DataCase default-stubs it to `:novel` +# (gate is a no-op); ProposalGate's own tests call it directly with MockEmbeddingClient. +config :loopctl, :proposal_assessor, Loopctl.MockProposalAssessor + # DI (US-27.3): the router wrapped by LoopctlWeb.Plugs.DBErrorBackstop. A thin # REAL plug (Loopctl.Test.BackstopRouter) that delegates to LoopctlWeb.Router for # every request — so the production router stays on the hot path with no global diff --git a/lib/loopctl/knowledge.ex b/lib/loopctl/knowledge.ex index 81de06b..04e15ca 100644 --- a/lib/loopctl/knowledge.ex +++ b/lib/loopctl/knowledge.ex @@ -328,6 +328,121 @@ defmodule Loopctl.Knowledge do end end + @doc """ + Novelty-gated write-back. Wraps `create_article/3` with a semantic dedup gate so + an agent proposing knowledge can't silently bloat the corpus with near-duplicates. + + The proposal is assessed against the published corpus (see + `Loopctl.Knowledge.ProposalAssessorBehaviour`); then, by verdict: + + * `:duplicate` — a near-identical article already exists. Nothing is created; + the canonical article is returned so the caller can read/update it instead. + * `:low_novelty` — high overlap with existing knowledge. The article is created + as a **draft** (downgraded from publish if needed) with the near-neighbors + stamped into `metadata.proposal_novelty`, so the smarter consuming agent (or a + human) resolves merge-vs-keep from the drafts review queue. + * `:novel` / `:unknown` (gate fell open) — created on the requested path. + + The gate is mechanical and non-destructive: it never edits or deletes existing + articles, and it falls open (`:unknown`) rather than blocking a write when the + embedding backend is unavailable. + + Returns `{:ok, result}` where `result` is a map: + + %{ + verdict: :created | :gated_to_draft | :duplicate | :deduplicated, + article: %Article{}, # the created article, or the canonical existing one + created: boolean(), # false for :duplicate / :deduplicated + assessment: %{verdict:, score:, neighbors:} + } + + or `{:error, :duplicate_title, %Article{}}` / `{:error, %Ecto.Changeset{}}`, + forwarded unchanged from `create_article/3`. + """ + @spec propose_article(Ecto.UUID.t() | nil, map(), keyword()) :: + {:ok, map()} + | {:error, :duplicate_title, Article.t()} + | {:error, Ecto.Changeset.t()} + def propose_article(tenant_id, attrs, opts \\ []) do + attrs = stringify_top_keys(attrs) + assessment = proposal_assessor().assess(tenant_id, attrs, opts) + gate_proposal(tenant_id, attrs, assessment, opts) + end + + defp proposal_assessor do + Application.get_env(:loopctl, :proposal_assessor, Loopctl.Knowledge.ProposalGate) + end + + defp gate_proposal(tenant_id, attrs, %{verdict: :duplicate} = assessment, opts) do + case canonical_neighbor(tenant_id, assessment, opts) do + {:ok, existing} -> + {:ok, %{verdict: :duplicate, article: existing, created: false, assessment: assessment}} + + # The canonical neighbor vanished (deleted/unpublished) between assess and now — + # there is nothing to dedup against, so create on the normal path. + :error -> + create_proposal(tenant_id, attrs, %{assessment | verdict: :novel}, opts, :created) + end + end + + defp gate_proposal(tenant_id, attrs, %{verdict: :low_novelty} = assessment, opts) do + gated_attrs = + attrs + |> Map.put("status", "draft") + |> stamp_proposal_metadata(assessment) + + create_proposal(tenant_id, gated_attrs, assessment, opts, :gated_to_draft) + end + + # :novel or :unknown (gate fell open) — proceed on the requested path. + defp gate_proposal(tenant_id, attrs, assessment, opts) do + create_proposal(tenant_id, attrs, assessment, opts, :created) + end + + defp create_proposal(tenant_id, attrs, assessment, opts, verdict) do + case create_article(tenant_id, attrs, opts) do + {:ok, article} -> + {:ok, %{verdict: verdict, article: article, created: true, assessment: assessment}} + + {:ok, :deduplicated, article} -> + {:ok, %{verdict: :deduplicated, article: article, created: false, assessment: assessment}} + + {:error, :duplicate_title, existing} -> + {:error, :duplicate_title, existing} + + {:error, %Ecto.Changeset{} = changeset} -> + {:error, changeset} + end + end + + defp canonical_neighbor(tenant_id, %{neighbors: [%{id: id} | _]}, opts) do + case get_article(tenant_id, id, Keyword.take(opts, [:visibility_agent_id])) do + {:ok, article} -> {:ok, article} + _ -> :error + end + end + + defp canonical_neighbor(_tenant_id, _assessment, _opts), do: :error + + defp stamp_proposal_metadata(attrs, %{score: score, neighbors: neighbors}) do + existing = stringify_top_keys(attrs["metadata"] || %{}) + + novelty = %{ + "verdict" => "low_novelty", + "score" => score, + "nearest" => + Enum.map(neighbors, fn n -> + %{"id" => n.id, "title" => n.title, "score" => n.similarity_score} + end) + } + + Map.put(attrs, "metadata", Map.put(existing, "proposal_novelty", novelty)) + end + + defp stringify_top_keys(map) when is_map(map) do + Map.new(map, fn {k, v} -> {to_string(k), v} end) + end + # Make concurrent/retried creates safe on the (tenant_id, title) active unique # index. By the time the insert fails the constraint, the winning transaction # has committed, so the existing row is visible (the recovery SELECT below diff --git a/lib/loopctl/knowledge/proposal_assessor_behaviour.ex b/lib/loopctl/knowledge/proposal_assessor_behaviour.ex new file mode 100644 index 0000000..769786b --- /dev/null +++ b/lib/loopctl/knowledge/proposal_assessor_behaviour.ex @@ -0,0 +1,42 @@ +defmodule Loopctl.Knowledge.ProposalAssessorBehaviour do + @moduledoc """ + Behaviour for assessing the NOVELTY of a *proposed* (not-yet-persisted) knowledge + article against the existing published corpus, so agent write-back can be gated: + a near-identical proposal is rejected in favour of the canonical article, a + high-overlap proposal is routed to a draft for the (smarter) consuming agent to + resolve, and a genuinely novel proposal flows through normally. + + This is **distinct** from the creativity "novelty" endpoint + (`KnowledgeCreativityController`), which measures idea-distance for generation. + Here, novelty == "does this add anything the corpus doesn't already hold?". + + ## Config-based DI + + `Loopctl.Knowledge.propose_article/3` resolves the implementation at runtime via + `Application.get_env(:loopctl, :proposal_assessor, Loopctl.Knowledge.ProposalGate)`. + `config/test.exs` swaps in `Loopctl.MockProposalAssessor`. + """ + + @type neighbor :: %{ + id: Ecto.UUID.t(), + title: String.t() | nil, + similarity_score: float() + } + + @type assessment :: %{ + verdict: :duplicate | :low_novelty | :novel | :unknown, + score: float() | nil, + neighbors: [neighbor()] + } + + @doc """ + Assess a proposal. `attrs` carries at least `"title"`/`"body"` (string or atom + keys). Returns the verdict, the top nearest-neighbor similarity `score` (or `nil` + when nothing crosses the overlap floor), and the `neighbors` list. + + Implementations MUST fall open — on any embedding/search failure, return + `%{verdict: :unknown, score: nil, neighbors: []}` so the gate never blocks a write. + """ + @callback assess(tenant_id :: Ecto.UUID.t() | nil, attrs :: map(), opts :: keyword()) :: + assessment() +end diff --git a/lib/loopctl/knowledge/proposal_gate.ex b/lib/loopctl/knowledge/proposal_gate.ex new file mode 100644 index 0000000..6f3b566 --- /dev/null +++ b/lib/loopctl/knowledge/proposal_gate.ex @@ -0,0 +1,92 @@ +defmodule Loopctl.Knowledge.ProposalGate do + @moduledoc """ + Default `ProposalAssessorBehaviour` — scores a proposed article's novelty against + the tenant's published corpus by embedding the proposal text and finding its + nearest neighbors via pgvector cosine similarity. + + Mechanical only: it embeds, searches, and classifies by threshold. It does NOT + decide merges or edit anything — that judgment belongs to the consuming agent, + which is a step smarter than the KB. The gate just answers "is this novel?" and + surfaces the near-neighbors. + + ## Bands (config-tunable) + + * `score >= :knowledge_proposal_duplicate_threshold` (default `0.97`) → `:duplicate` + * `score >= :knowledge_proposal_overlap_threshold` (default `0.88`) → `:low_novelty` + * otherwise (incl. nothing above the overlap floor) → `:novel` + + ## Resilience + + Embedding requires a network call. On ANY failure — API down, power/internet + outage, system-scoped proposal with no tenant — `assess/3` falls **open**: + `%{verdict: :unknown, ...}`, so write-back is never blocked by the gate. + """ + + @behaviour Loopctl.Knowledge.ProposalAssessorBehaviour + + require Logger + + alias Loopctl.Knowledge.VectorSearch + + @embedding_client Application.compile_env( + :loopctl, + :embedding_client, + Loopctl.Knowledge.EmbeddingClient + ) + + @default_duplicate_threshold 0.97 + @default_overlap_threshold 0.88 + @neighbors_k 5 + @max_text_length 32_000 + + @impl true + def assess(tenant_id, attrs, opts \\ []) + + # System-scoped (no tenant) proposals are superadmin-only and rare — skip the gate. + def assess(nil, _attrs, _opts), do: open_verdict() + + def assess(tenant_id, attrs, opts) when is_binary(tenant_id) do + dup = config(:knowledge_proposal_duplicate_threshold, @default_duplicate_threshold) + overlap = config(:knowledge_proposal_overlap_threshold, @default_overlap_threshold) + + case @embedding_client.generate_embedding(build_text(attrs)) do + {:ok, vector} when is_list(vector) and vector != [] -> + neighbors = + VectorSearch.nearest(tenant_id, vector, @neighbors_k, + threshold: overlap, + visibility_agent_id: Keyword.get(opts, :visibility_agent_id) + ) + + score = neighbors |> List.first() |> neighbor_score() + %{verdict: classify(score, dup, overlap), score: score, neighbors: neighbors} + + other -> + Logger.warning("ProposalGate: embedding failed, falling open: #{inspect(other)}") + open_verdict() + end + end + + @doc """ + Pure threshold classification — the heart of the gate, unit-tested in isolation. + `score` is the top neighbor's `similarity_score` (or `nil` when none cleared the + overlap floor). + """ + @spec classify(float() | nil, float(), float()) :: :duplicate | :low_novelty | :novel + def classify(nil, _dup, _overlap), do: :novel + def classify(score, dup, _overlap) when score >= dup, do: :duplicate + def classify(score, _dup, overlap) when score >= overlap, do: :low_novelty + def classify(_score, _dup, _overlap), do: :novel + + defp neighbor_score(nil), do: nil + defp neighbor_score(%{similarity_score: s}), do: s + + defp build_text(attrs) do + title = attrs["title"] || attrs[:title] || "" + body = attrs["body"] || attrs[:body] || "" + String.slice("#{title}\n\n#{body}", 0, @max_text_length) + end + + defp open_verdict, do: %{verdict: :unknown, score: nil, neighbors: []} + + defp config(key, default), do: Application.get_env(:loopctl, key, default) +end diff --git a/lib/loopctl_web/controllers/article_controller.ex b/lib/loopctl_web/controllers/article_controller.ex index 01837a5..4bd35ea 100644 --- a/lib/loopctl_web/controllers/article_controller.ex +++ b/lib/loopctl_web/controllers/article_controller.ex @@ -249,6 +249,10 @@ defmodule LoopctlWeb.ArticleController do scope = params["scope"] || "tenant" api_key = conn.assigns.current_api_key draft? = draft_requested?(params) + # Novelty-gated write-back is ON by default; `force: true` bypasses it (e.g. the + # caller has already searched and knows the proposal is intentionally near an + # existing article). + gate? = not truthy?(params["force"]) # System articles require superadmin role; everything else is agent+. if scope == "system" and api_key.role != :superadmin do @@ -280,7 +284,7 @@ defmodule LoopctlWeb.ArticleController do # no key identity can't attribute a memory → 403. case bind_agent_identity(api_key, attrs) do {:ok, bound_attrs} -> - create_article(conn, tenant_id, bound_attrs, audit_opts, draft?) + create_article(conn, tenant_id, bound_attrs, audit_opts, draft?, gate?) {:error, :no_agent_identity} -> conn @@ -328,44 +332,119 @@ defmodule LoopctlWeb.ArticleController do Enum.any?(@agent_memory_marker_atoms, &Map.has_key?(metadata, &1)) end - defp create_article(conn, tenant_id, attrs, audit_opts, draft?) do + defp create_article(conn, tenant_id, attrs, audit_opts, draft?, gate?) do # Pass the caller's visibility scope so idempotency dedup can't echo a private # memory the agent can't see (#163). opts = audit_opts ++ Visibility.scope_opts(conn) - case Knowledge.create_article(tenant_id, attrs, opts) do - {:ok, article} -> - conn - |> put_status(:created) - |> json(create_response(article)) + if gate? do + render_proposal(conn, Knowledge.propose_article(tenant_id, attrs, opts), attrs, draft?) + else + render_create(conn, Knowledge.create_article(tenant_id, attrs, opts), attrs, draft?) + end + end - # Idempotent dedup (no-op), 200 with `deduplicated: true` so clients that - # only see a 2xx (e.g. the MCP layer) can tell a dedup from a real create. - {:ok, :deduplicated, existing} -> - conn - |> put_status(:ok) - |> json(dedup_response(existing, attrs, draft?)) + # Novelty-gated path: render by verdict. A near-duplicate is NOT created — the + # caller is pointed at the canonical article. A low-novelty proposal is created as + # a draft with the near-neighbors surfaced so a reviewer/consumer can merge. + defp render_proposal(conn, {:ok, %{verdict: :duplicate} = result}, _attrs, _draft?) do + %{article: existing, assessment: assessment} = result + + conn + |> put_status(:ok) + |> json(%{ + data: %{id: existing.id, title: existing.title, status: to_string(existing.status)}, + deduplicated: true, + gate: gate_meta(:duplicate, assessment), + note: + "A near-duplicate already exists (id #{existing.id}, similarity " <> + "#{format_score(assessment.score)}). Nothing was created — read or update " <> + "the existing article instead, or pass `force: true` to create anyway." + }) + end - {:error, :duplicate_title, existing} -> - conn - |> put_status(:conflict) - |> json(%{ - error: %{ - status: 409, - code: "title_conflict", - message: - "An article titled \"#{existing.title}\" already exists in this tenant " <> - "with different content. Choose a different (more specific) title. " <> - "(Editing the existing article requires role :user via PATCH /articles/:id.)", - details: %{existing_article_id: existing.id} - } - }) + defp render_proposal(conn, {:ok, %{verdict: :gated_to_draft} = result}, _attrs, _draft?) do + %{article: article, assessment: assessment} = result + + conn + |> put_status(:created) + |> json( + article + |> create_response() + |> Map.put(:gate, gate_meta(:gated_to_draft, assessment)) + |> Map.put( + :note, + "High overlap with existing knowledge (similarity " <> + "#{format_score(assessment.score)}) — created as a DRAFT instead of " <> + "publishing, with near-neighbors in metadata.proposal_novelty for review. " <> + "Resolve via merge/publish, or pass `force: true` to publish on create." + ) + ) + end - {:error, %Ecto.Changeset{} = changeset} -> - {:error, changeset} - end + defp render_proposal(conn, {:ok, %{verdict: :deduplicated, article: existing}}, attrs, draft?) do + conn + |> put_status(:ok) + |> json(dedup_response(existing, attrs, draft?)) + end + + # :created (novel, or gate fell open) — normal create response. + defp render_proposal(conn, {:ok, %{article: article}}, _attrs, _draft?) do + conn + |> put_status(:created) + |> json(create_response(article)) + end + + defp render_proposal(conn, error, attrs, draft?), + do: render_create(conn, error, attrs, draft?) + + # Ungated path (force: true) — the original create semantics. + defp render_create(conn, {:ok, article}, _attrs, _draft?) do + conn + |> put_status(:created) + |> json(create_response(article)) + end + + defp render_create(conn, {:ok, :deduplicated, existing}, attrs, draft?) do + conn + |> put_status(:ok) + |> json(dedup_response(existing, attrs, draft?)) end + defp render_create(conn, {:error, :duplicate_title, existing}, _attrs, _draft?) do + conn + |> put_status(:conflict) + |> json(%{ + error: %{ + status: 409, + code: "title_conflict", + message: + "An article titled \"#{existing.title}\" already exists in this tenant " <> + "with different content. Choose a different (more specific) title. " <> + "(Editing the existing article requires role :user via PATCH /articles/:id.)", + details: %{existing_article_id: existing.id} + } + }) + end + + defp render_create(_conn, {:error, %Ecto.Changeset{} = changeset}, _attrs, _draft?), + do: {:error, changeset} + + defp gate_meta(verdict, assessment) do + %{ + verdict: to_string(verdict), + similarity: assessment.score, + nearest: + Enum.map(assessment.neighbors, fn n -> + %{id: n.id, title: n.title, similarity: n.similarity_score} + end) + } + end + + defp format_score(nil), do: "n/a" + defp format_score(score) when is_float(score), do: :erlang.float_to_binary(score, decimals: 3) + defp format_score(score), do: to_string(score) + @doc "GET /api/v1/articles or GET /api/v1/projects/:project_id/articles" def index(conn, params) do tenant_id = conn.assigns.current_api_key.tenant_id diff --git a/test/loopctl/knowledge/proposal_gate_test.exs b/test/loopctl/knowledge/proposal_gate_test.exs new file mode 100644 index 0000000..911ff6f --- /dev/null +++ b/test/loopctl/knowledge/proposal_gate_test.exs @@ -0,0 +1,99 @@ +defmodule Loopctl.Knowledge.ProposalGateTest do + use Loopctl.DataCase, async: true + + import Mox + + setup :verify_on_exit! + + alias Loopctl.Knowledge + alias Loopctl.Knowledge.ProposalGate + + @dims 1536 + + # Sparse-prefix embedding, rest zero-filled. Cosine is magnitude-independent, so + # e([1.0]) vs e([1.0]) = 1.0, e([1.0]) vs e([0.0, 1.0]) = 0.0 (orthogonal), + # e([1.0]) vs e([1.0, 0.426]) ≈ 0.92. + defp e(prefix), do: prefix ++ List.duplicate(0.0, @dims - length(prefix)) + + defp published_with_embedding(tenant_id, title, prefix) do + a = fixture(:article, %{tenant_id: tenant_id, title: title, status: :published}) + {:ok, _} = Knowledge.update_embedding(tenant_id, a.id, e(prefix)) + a + end + + describe "classify/3 (pure threshold logic)" do + test "nil score (nothing above the overlap floor) is novel" do + assert ProposalGate.classify(nil, 0.97, 0.88) == :novel + end + + test "at/above the duplicate threshold is :duplicate" do + assert ProposalGate.classify(0.97, 0.97, 0.88) == :duplicate + assert ProposalGate.classify(0.991, 0.97, 0.88) == :duplicate + end + + test "in the overlap band [overlap, duplicate) is :low_novelty" do + assert ProposalGate.classify(0.88, 0.97, 0.88) == :low_novelty + assert ProposalGate.classify(0.93, 0.97, 0.88) == :low_novelty + end + + test "below the overlap floor is :novel" do + assert ProposalGate.classify(0.8799, 0.97, 0.88) == :novel + assert ProposalGate.classify(0.1, 0.97, 0.88) == :novel + end + end + + describe "assess/3 (real pgvector)" do + setup do + tenant = fixture(:tenant) + existing = published_with_embedding(tenant.id, "Supervisors and OTP", [1.0]) + %{tenant: tenant, existing: existing} + end + + test "an identical-direction proposal is a duplicate", %{tenant: tenant, existing: existing} do + stub(Loopctl.MockEmbeddingClient, :generate_embedding, fn _text -> {:ok, e([1.0])} end) + + assessment = ProposalGate.assess(tenant.id, %{"title" => "Supervisors", "body" => "OTP"}) + + assert assessment.verdict == :duplicate + assert assessment.score >= 0.97 + assert [%{id: id} | _] = assessment.neighbors + assert id == existing.id + end + + test "a high-overlap proposal is low-novelty", %{tenant: tenant} do + # cos(e([1.0]), e([1.0, 0.426])) ≈ 0.92 — inside [0.88, 0.97). + stub(Loopctl.MockEmbeddingClient, :generate_embedding, fn _text -> + {:ok, e([1.0, 0.426])} + end) + + assessment = ProposalGate.assess(tenant.id, %{"title" => "Supervision", "body" => "trees"}) + + assert assessment.verdict == :low_novelty + assert assessment.score >= 0.88 and assessment.score < 0.97 + end + + test "an orthogonal proposal is novel (no neighbor clears the floor)", %{tenant: tenant} do + stub(Loopctl.MockEmbeddingClient, :generate_embedding, fn _text -> {:ok, e([0.0, 1.0])} end) + + assessment = ProposalGate.assess(tenant.id, %{"title" => "Billing", "body" => "invoices"}) + + assert assessment.verdict == :novel + assert assessment.neighbors == [] + end + + test "falls OPEN (:unknown) when embedding fails — never blocks a write", %{tenant: tenant} do + stub(Loopctl.MockEmbeddingClient, :generate_embedding, fn _text -> {:error, :timeout} end) + + assessment = ProposalGate.assess(tenant.id, %{"title" => "X", "body" => "Y"}) + + assert assessment.verdict == :unknown + assert assessment.score == nil + assert assessment.neighbors == [] + end + + test "system scope (nil tenant) skips the gate" do + assert %{verdict: :unknown, neighbors: []} = + ProposalGate.assess(nil, %{"title" => "X", "body" => "Y"}) + end + end +end diff --git a/test/loopctl/knowledge/propose_article_test.exs b/test/loopctl/knowledge/propose_article_test.exs new file mode 100644 index 0000000..03ad68d --- /dev/null +++ b/test/loopctl/knowledge/propose_article_test.exs @@ -0,0 +1,168 @@ +defmodule Loopctl.Knowledge.ProposeArticleTest do + use Loopctl.DataCase, async: true + + import Mox + + setup :verify_on_exit! + + alias Loopctl.Knowledge + + # Drive the (mocked) assessor to a chosen verdict; the gate's routing is what we test. + defp assessor(verdict, neighbors \\ [], score \\ nil) do + stub(Loopctl.MockProposalAssessor, :assess, fn _tenant_id, _attrs, _opts -> + %{verdict: verdict, score: score, neighbors: neighbors} + end) + end + + defp neighbor(article, score), + do: %{id: article.id, title: article.title, similarity_score: score} + + defp count_articles(tenant_id) do + import Ecto.Query + + Loopctl.AdminRepo.aggregate( + from(a in Knowledge.Article, where: a.tenant_id == ^tenant_id), + :count + ) + end + + setup do + %{tenant: fixture(:tenant)} + end + + describe "novel / fell-open" do + test "novel proposal is created on the requested (published) path", %{tenant: tenant} do + assessor(:novel) + + assert {:ok, %{verdict: :created, created: true, article: article}} = + Knowledge.propose_article(tenant.id, %{ + "title" => "Genuinely new", + "body" => "Something the corpus does not hold yet.", + "category" => "finding", + "status" => "published" + }) + + assert article.status == :published + assert article.title == "Genuinely new" + end + + test "unknown verdict (gate fell open) still creates — never blocks", %{tenant: tenant} do + assessor(:unknown) + + assert {:ok, %{verdict: :created, created: true, article: article}} = + Knowledge.propose_article(tenant.id, %{ + "title" => "Created despite outage", + "body" => "Embedding backend was down; gate fell open.", + "category" => "finding", + "status" => "published" + }) + + assert article.status == :published + end + end + + describe "duplicate" do + test "near-duplicate is NOT created; the canonical article is returned", %{tenant: tenant} do + existing = + fixture(:article, %{tenant_id: tenant.id, title: "Canonical", status: :published}) + + assessor(:duplicate, [neighbor(existing, 0.98)], 0.98) + before = count_articles(tenant.id) + + assert {:ok, %{verdict: :duplicate, created: false, article: returned, assessment: a}} = + Knowledge.propose_article(tenant.id, %{ + "title" => "Canonical (reworded)", + "body" => "Same idea, different words.", + "category" => "finding", + "status" => "published" + }) + + assert returned.id == existing.id + assert a.score == 0.98 + # Nothing new persisted. + assert count_articles(tenant.id) == before + end + + test "if the canonical neighbor has vanished, it creates instead", %{tenant: tenant} do + # The assessor reports a near-duplicate, but its id no longer resolves (the + # article was deleted/unpublished between assess and create) — the gate must + # fall through to creating rather than returning a phantom. + ghost = %{id: Ecto.UUID.generate(), title: "Ghost"} + assessor(:duplicate, [%{id: ghost.id, title: ghost.title, similarity_score: 0.99}], 0.99) + + assert {:ok, %{verdict: :created, created: true, article: article}} = + Knowledge.propose_article(tenant.id, %{ + "title" => "Survives a vanished dup target", + "body" => "The near-neighbor was deleted between assess and create.", + "category" => "finding", + "status" => "published" + }) + + assert article.status == :published + end + end + + describe "low novelty" do + test "high-overlap proposal is downgraded to a draft with neighbors stamped", %{ + tenant: tenant + } do + existing = + fixture(:article, %{tenant_id: tenant.id, title: "Adjacent", status: :published}) + + assessor(:low_novelty, [neighbor(existing, 0.91)], 0.91) + + assert {:ok, %{verdict: :gated_to_draft, created: true, article: article}} = + Knowledge.propose_article(tenant.id, %{ + "title" => "Overlaps existing knowledge", + "body" => "Mostly covered by an adjacent article.", + "category" => "finding", + # caller asked to publish — gate downgrades to draft + "status" => "published" + }) + + assert article.status == :draft + novelty = article.metadata["proposal_novelty"] + assert novelty["verdict"] == "low_novelty" + assert novelty["score"] == 0.91 + assert [%{"id" => id}] = novelty["nearest"] + assert id == existing.id + end + + test "preserves caller metadata while stamping novelty", %{tenant: tenant} do + existing = fixture(:article, %{tenant_id: tenant.id, title: "Near", status: :published}) + assessor(:low_novelty, [neighbor(existing, 0.9)], 0.9) + + assert {:ok, %{article: article}} = + Knowledge.propose_article(tenant.id, %{ + "title" => "Has caller metadata", + "body" => "Body.", + "category" => "finding", + "metadata" => %{"memory_type" => "insight"} + }) + + assert article.metadata["memory_type"] == "insight" + assert article.metadata["proposal_novelty"]["score"] == 0.9 + end + end + + describe "tenant isolation" do + test "dedup only sees the caller's own tenant", %{tenant: tenant_a} do + tenant_b = fixture(:tenant) + # Default stub returns :novel regardless; this asserts the gate doesn't leak a + # cross-tenant article as a canonical neighbor (the assessor is tenant-scoped). + assessor(:novel) + + assert {:ok, %{verdict: :created, article: article}} = + Knowledge.propose_article(tenant_a.id, %{ + "title" => "Tenant A only", + "body" => "Body.", + "category" => "finding", + "status" => "published" + }) + + assert article.tenant_id == tenant_a.id + # Tenant B sees none of A's articles. + assert count_articles(tenant_b.id) == 0 + end + end +end diff --git a/test/loopctl_web/controllers/article_controller_test.exs b/test/loopctl_web/controllers/article_controller_test.exs index e57ba83..bb820d1 100644 --- a/test/loopctl_web/controllers/article_controller_test.exs +++ b/test/loopctl_web/controllers/article_controller_test.exs @@ -344,6 +344,92 @@ defmodule LoopctlWeb.ArticleControllerTest do end end + describe "POST /api/v1/articles novelty gate" do + import Mox + + defp gate_verdict(verdict, neighbors, score) do + stub(Loopctl.MockProposalAssessor, :assess, fn _t, _a, _o -> + %{verdict: verdict, score: score, neighbors: neighbors} + end) + end + + test "near-duplicate returns 200, creates nothing, points to the canonical", %{conn: conn} do + tenant = fixture(:tenant) + {raw_key, _} = fixture(:api_key, %{tenant_id: tenant.id, role: :agent}) + + existing = + fixture(:article, %{tenant_id: tenant.id, title: "Canonical", status: :published}) + + gate_verdict( + :duplicate, + [%{id: existing.id, title: existing.title, similarity_score: 0.98}], + 0.98 + ) + + conn = + conn + |> auth_conn(raw_key) + |> post(~p"/api/v1/articles", %{ + "title" => "Canonical reworded", + "body" => "Same idea.", + "category" => "finding" + }) + + body = json_response(conn, 200) + assert body["deduplicated"] == true + assert body["gate"]["verdict"] == "duplicate" + assert body["data"]["id"] == existing.id + assert body["note"] =~ "near-duplicate" + end + + test "high-overlap proposal is created as a draft with gate metadata", %{conn: conn} do + tenant = fixture(:tenant) + {raw_key, _} = fixture(:api_key, %{tenant_id: tenant.id, role: :agent}) + existing = fixture(:article, %{tenant_id: tenant.id, title: "Adjacent", status: :published}) + + gate_verdict( + :low_novelty, + [%{id: existing.id, title: existing.title, similarity_score: 0.91}], + 0.91 + ) + + conn = + conn + |> auth_conn(raw_key) + |> post(~p"/api/v1/articles", %{ + "title" => "Overlapping", + "body" => "Mostly covered already.", + "category" => "finding" + }) + + body = json_response(conn, 201) + assert body["data"]["status"] == "draft" + assert body["gate"]["verdict"] == "gated_to_draft" + assert body["note"] =~ "DRAFT" + end + + test "force: true bypasses the gate and publishes", %{conn: conn} do + tenant = fixture(:tenant) + {raw_key, _} = fixture(:api_key, %{tenant_id: tenant.id, role: :agent}) + # Even if the assessor would call it a duplicate, force skips assessment entirely. + gate_verdict(:duplicate, [], 0.99) + + conn = + conn + |> auth_conn(raw_key) + |> post(~p"/api/v1/articles", %{ + "title" => "Forced through", + "body" => "Intentionally near an existing article.", + "category" => "finding", + "force" => true + }) + + body = json_response(conn, 201) + assert body["data"]["status"] == "published" + assert body["data"]["title"] == "Forced through" + end + end + describe "POST /api/v1/articles idempotency_key (#137)" do test "re-create with the same idempotency_key is a no-op (returns existing, even with a different body)", %{conn: conn} do diff --git a/test/support/data_case.ex b/test/support/data_case.ex index f350486..576aa46 100644 --- a/test/support/data_case.ex +++ b/test/support/data_case.ex @@ -191,6 +191,13 @@ defmodule Loopctl.DataCase do Loopctl.Knowledge.suggest_links_with_meta(tenant_id, article_id, opts) end) + # Novelty-gated write-back: default to `:novel` so the gate is a no-op for the + # existing article-create tests (they assert the normal create path). Tests that + # exercise the gate override this with Mox.expect/3 to return a chosen verdict. + Mox.stub(Loopctl.MockProposalAssessor, :assess, fn _tenant_id, _attrs, _opts -> + %{verdict: :novel, score: nil, neighbors: []} + end) + # US-27.3: the DBErrorBackstop test seam (Loopctl.Test.BackstopRouter) is a # REAL plug wired via config/test.exs that delegates to LoopctlWeb.Router for # every request and only raises when an opt-in `x-test-raise-db-error` header diff --git a/test/support/mocks.ex b/test/support/mocks.ex index 7734d92..9e9704c 100644 --- a/test/support/mocks.ex +++ b/test/support/mocks.ex @@ -10,6 +10,7 @@ Mox.defmock(Loopctl.MockCategoryClassifier, for: Loopctl.Knowledge.ClassifierBeh Mox.defmock(Loopctl.MockWebAuthn, for: Loopctl.WebAuthn.Behaviour) Mox.defmock(Loopctl.MockSecrets, for: Loopctl.Secrets.Behaviour) Mox.defmock(Loopctl.MockSuggestLinks, for: Loopctl.Knowledge.SuggestLinksBehaviour) +Mox.defmock(Loopctl.MockProposalAssessor, for: Loopctl.Knowledge.ProposalAssessorBehaviour) # US-27.15: webhook delivery DI. ScaleAlerts and the webhook worker share the # `:webhook_delivery` key. In :test it resolves to this mock; the DataCase default stub # delegates to Loopctl.Webhooks.ReqDelivery so the existing Req.Test-stub-based webhook