diff --git a/CLAUDE.md b/CLAUDE.md index fe980cae..771d29f0 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -117,6 +117,9 @@ com.bablsoft.accessflow/ │ ├── api/ │ ├── events/ │ └── internal/ # persistence, scheduled (run + timeout jobs), web +├── discovery/ # Automated sensitive-data discovery (AF-623): DiscoveryScanJob samples column data via the engine sampling path, regex+checksum detectors (email, PAN+Luhn, SSN, IBAN, phone) + optional fail-safe AI pass propose classification tags an admin confirms (AF-447 derivation) or dismisses +│ ├── api/ +│ └── internal/ # config, persistence, detect (pure detectors), scheduled, web └── mcp/ # Spring AI stateless MCP server — @Tool callbacks for AI agents ├── api/ └── internal/ @@ -270,6 +273,11 @@ com.bablsoft.accessflow/ | `ACCESSFLOW_LIFECYCLE_REVIEW_TIMEOUT` | ISO-8601 duration. How long an erasure request may sit in `PENDING_REVIEW` before `ErasureReviewTimeoutJob` auto-rejects it (default `PT168H`, AF-519). Binds `accessflow.lifecycle.review-timeout`. | | `ACCESSFLOW_REQUESTGROUPS_RUN_POLL_INTERVAL` | ISO-8601 duration. Cadence at which `ScheduledGroupRunJob` (the `requestgroups` module, AF-501) scans for `APPROVED` grouped requests whose `scheduled_for ≤ now()` and executes their ordered member sequence (default `PT1M`). Binds `accessflow.requestgroups.run-poll-interval`. | | `ACCESSFLOW_REQUESTGROUPS_TIMEOUT_POLL_INTERVAL` | ISO-8601 duration. Cadence at which `GroupTimeoutJob` (the `requestgroups` module, AF-501) scans for grouped requests stuck in `PENDING_REVIEW` past the review timeout and auto-rejects them (`TIMED_OUT`) (default `PT5M`). Binds `accessflow.requestgroups.timeout-poll-interval`. | +| `ACCESSFLOW_DISCOVERY_SCAN_POLL_INTERVAL` | ISO-8601 duration. Cadence at which `DiscoveryScanJob` (the `discovery` module, AF-623) checks for datasources whose sensitive-data discovery scan is due — enabled `discovery_scan_config` rows past their own `scan_interval_hours` (default `PT15M`). Binds `accessflow.discovery.scan-poll-interval`. | +| `ACCESSFLOW_DISCOVERY_SCAN_TIME_BUDGET` | ISO-8601 duration. Wall-clock budget for one discovery scan; tables past the deadline are skipped and the run is flagged partial (default `PT10M`). Binds `accessflow.discovery.scan-time-budget`. | +| `ACCESSFLOW_DISCOVERY_SAMPLE_STATEMENT_TIMEOUT` | ISO-8601 duration. Per-table statement timeout for the discovery scan's bounded sample read (default `PT10S`). Binds `accessflow.discovery.sample-statement-timeout`. | +| `ACCESSFLOW_DISCOVERY_MAX_TABLES_PER_SCAN` | Hard cap on tables sampled in one discovery scan (default `200`). Binds `accessflow.discovery.max-tables-per-scan`. | +| `ACCESSFLOW_DISCOVERY_MAX_AI_TABLES_PER_SCAN` | Hard cap on tables sent through the optional discovery AI pass per scan, bounding provider spend (default `25`). Binds `accessflow.discovery.max-ai-tables-per-scan`. | | `ACCESSFLOW_APIGOV_REVIEW_TIMEOUT` | ISO-8601 duration. How long an API request may sit in `PENDING_REVIEW` before `ApiRequestTimeoutJob` auto-rejects it (default `PT24H`). | | `ACCESSFLOW_APIGOV_MAX_REQUEST_BODY_BYTES` | Cap on the total encoded size of a submitted API request body (#517) — raw text, x-www-form-urlencoded, or the base64-decoded size of form-data / binary file parts (files ride inline as bounded base64 since AccessFlow has no object storage). Exceeding it rejects the submission with HTTP 422. Default `5242880` (5 MiB). Binds `accessflow.apigov.max-request-body-bytes`. | | `ACCESSFLOW_APIGOV_MAX_RESPONSE_BYTES` | System-wide hard ceiling (#521) on a stored — and therefore downloadable — API response body; the absolute backstop above any per-connector `max_response_bytes` (effective cap is the **min** of the two). The body lives in a Postgres `text` column and is read fully into JVM memory during the call. Default `10485760` (10 MiB). Binds `accessflow.apigov.max-response-bytes`. | diff --git a/README.md b/README.md index 325682e6..86eeb44b 100644 --- a/README.md +++ b/README.md @@ -72,6 +72,7 @@ A glance at the day-to-day flows engineers and approvers actually use. - **Dynamic data masking** — per-column masking policies (full, partial last-N, stable hash, email-preserving, format-preserving) with role / group / user **reveal** conditions evaluated per requester. Masking is applied at result-read time before results are serialized or stored, so unmasked values never persist; applied policy ids are recorded in the audit log. Extends the static `restricted_columns` masking. - **Row-level security** — per-table row predicates the proxy injects into the parsed SQL so a scoped user only sees (SELECT) or affects (UPDATE/DELETE) the rows they are authorised for. Admins author a structured `column operator value` predicate where the value is a fixed literal or a `:user.*` variable (built-in id / email / role / groups, or an admin-set per-user attribute). Values are bound as JDBC parameters — never concatenated; predicates that can't be safely applied are rejected, never run unfiltered. Composes with column masking and the schema/table allow-list. - **Data classification tagging** — tag tables and columns as PII, PCI, PHI, GDPR, FINANCIAL, or SENSITIVE right in the schema explorer. Tagging a column auto-applies a masking policy, the AI analyzer raises a query's risk score when it touches a tagged object, and a derivation preview suggests a stricter review posture. Tags are audited and queryable org-wide as the evidence base for compliance reporting. +- **Automated sensitive-data discovery** — an opt-in per-datasource scanner samples column data through the same governed sampling path, detects sensitive values with local regex + checksum detectors (emails, credit-card PANs with Luhn, SSNs, IBANs, phone numbers) and optionally your bound AI analyzer (which only ever sees column names, types, and redacted samples), and **proposes** classification tags in a review worklist. Confirming a finding applies the tag — deriving masking automatically — while dismissing suppresses it permanently; scans and decisions are audited, and an on-demand "Scan now" complements the scheduled cadence. - **User groups** — bundle reviewers (and other users) into groups; groups can be assigned as datasource reviewers, **granted data/API access** (a whole team inherits a datasource or API-connector grant — read/write/DDL/break-glass — instead of a row per member; effective access is the most-permissive union of a user's direct and group grants), and auto-synced from SAML / OAuth2 IdP claims via per-IdP `group_mappings` (admin-managed memberships are preserved across logins). - **AI query analysis** — pluggable adapters for OpenAI, Anthropic Claude, self-hosted Ollama, any OpenAI API–compatible backend (vLLM, LM Studio, Together, Groq, OpenRouter, …) via a custom endpoint, and Hugging Face (Inference Providers router or a local / self-hosted TGI server); risk scoring (0–100), missing-index detection, anti-pattern hints. Per-organization configuration via the admin UI, including an **editable analyzer prompt** per configuration (override the built-in template, or leave it blank to use the default); admin **AI analyses dashboard** charts average risk over time, top issue categories, and most active submitters. A per-organization **rate limit** (requests/minute, default 30) and optional **monthly token budget** cap AI usage so a runaway editor or compromised account can't drain the provider API key — enforced before every analysis call; exceedance returns HTTP 429 in the editor or records a sentinel `CRITICAL` analysis row on the async path. A per-organization **fallback pool** marks configurations (typically a local Ollama) to retry in priority order when the primary provider is unreachable — air-gap-friendly resilience without weakening any failure semantics. - **Multi-model orchestration, voting & guardrails** — an AI configuration can run several models in parallel (the primary plus weighted `{provider, model, weight}` members — e.g. a fast local Ollama pre-score alongside a deep Claude analysis) and combine their verdicts by a configurable voting strategy (**weighted average**, **highest risk**, or **majority**); issues are merged and risk aggregated. **Guardrails** block configured prompt patterns (case-insensitive regex) before any model is called, and the strict output-schema validation rejects malformed responses without failing the query. Per-model **cost (tokens) and latency** are recorded for every analysis and charted on the admin dashboard. diff --git a/backend/src/main/java/com/bablsoft/accessflow/ai/api/DataDiscoveryAiService.java b/backend/src/main/java/com/bablsoft/accessflow/ai/api/DataDiscoveryAiService.java new file mode 100644 index 00000000..6557b507 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/ai/api/DataDiscoveryAiService.java @@ -0,0 +1,45 @@ +package com.bablsoft.accessflow.ai.api; + +import com.bablsoft.accessflow.core.api.DataClassification; + +import java.util.List; +import java.util.UUID; + +/** + * Optional AI pass of the sensitive-data discovery scan (AF-623). Classifies a table's columns + * using the organization's first usable {@code ai_config}, complementing the local regex/checksum + * detectors (e.g. a {@code national_id} column no regex covers). + * + *

Privacy: callers must pass only redacted sample values (format-preserving + * masking) — raw sampled data never reaches the AI provider. + * + *

Fail-safe by contract: returns an empty list when the org has no usable AI + * config, the provider errors, or the response is not the expected JSON — it never throws and + * never blocks the scan (same posture as the UBA anomaly summary, AF-383). + */ +public interface DataDiscoveryAiService { + + List classifyColumns(UUID organizationId, + DiscoveryTableContext context); + + /** One table's worth of column metadata + redacted samples. */ + record DiscoveryTableContext(String tableName, List columns) { + + public DiscoveryTableContext { + columns = columns == null ? List.of() : List.copyOf(columns); + } + } + + /** {@code redactedSamples} must already be redacted by the caller. */ + record DiscoveryColumnContext(String name, String type, List redactedSamples) { + + public DiscoveryColumnContext { + redactedSamples = redactedSamples == null ? List.of() : List.copyOf(redactedSamples); + } + } + + /** A proposed classification for one column; {@code confidence} is clamped to 0–100. */ + record DiscoveryColumnSuggestion(String columnName, DataClassification classification, + int confidence, String rationale) { + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/ai/internal/AiAnalyzerStrategyHolder.java b/backend/src/main/java/com/bablsoft/accessflow/ai/internal/AiAnalyzerStrategyHolder.java index 5e8caa28..18213165 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/ai/internal/AiAnalyzerStrategyHolder.java +++ b/backend/src/main/java/com/bablsoft/accessflow/ai/internal/AiAnalyzerStrategyHolder.java @@ -84,6 +84,18 @@ class AiAnalyzerStrategyHolder implements AiAnalyzerStrategy { database-access pattern looks anomalous relative to the user's historical baseline, and \ what a reviewer should check. Do not use markdown, headings, or bullet points."""; + // AF-623: sensitive-data discovery AI pass. Sample values are redacted (format-preserving) + // before they reach this prompt — the model judges by column name, type, and value shape. + private static final String DISCOVERY_CLASSIFY_PREAMBLE = """ + You are a data-classification analyst. Given a database table with its column names, \ + types, and redacted sample values (digits replaced by *, letters by x — only the shape \ + is preserved), identify columns likely to contain sensitive data. Respond with ONLY a \ + JSON object of the form \ + {"columns":[{"column":"","classification":"PII|PCI|PHI|GDPR|FINANCIAL|SENSITIVE",\ + "confidence":<0-100>,"rationale":""}]}. \ + Propose only columns you genuinely believe are sensitive; when none are, respond \ + {"columns":[]}. No markdown, no code fences, no prose outside the JSON."""; + @Override public AiAnalysisResult analyze(String sql, DbType dbType, String schemaContext, String costEstimateContext, String language, UUID aiConfigId) { @@ -184,22 +196,38 @@ private static void rethrowIfNotFallbackEligible(RuntimeException ex) { * config exists or any provider/parse error occurs — it never throws and never blocks detection. */ Optional summarizeFreeform(UUID organizationId, String userPrompt) { + return completeFreeform(organizationId, ANOMALY_SUMMARY_PREAMBLE, userPrompt, + "anomaly summary"); + } + + /** + * Freeform call for the sensitive-data discovery AI pass (AF-623). Same fail-safe contract as + * {@link #summarizeFreeform}; the preamble demands strict JSON, which the discovery service + * parses leniently and drops on any mismatch. + */ + Optional classifyDiscoveryColumns(UUID organizationId, String userPrompt) { + return completeFreeform(organizationId, DISCOVERY_CLASSIFY_PREAMBLE, userPrompt, + "discovery classification"); + } + + private Optional completeFreeform(UUID organizationId, String systemPreamble, + String userPrompt, String logContext) { if (organizationId == null || userPrompt == null || userPrompt.isBlank()) { return Optional.empty(); } try { var entity = resolveUsableConfig(organizationId).orElse(null); if (entity == null) { - log.debug("No usable ai_config for org {}; skipping anomaly summary", organizationId); + log.debug("No usable ai_config for org {}; skipping {}", organizationId, logContext); return Optional.empty(); } var chatModel = chatModelCache.computeIfAbsent(entity.getId(), key -> buildChatModel(entity)); - var invocation = ChatModelInvoker.invoke(chatModel, ANOMALY_SUMMARY_PREAMBLE, userPrompt, + var invocation = ChatModelInvoker.invoke(chatModel, systemPreamble, userPrompt, entity.getProvider().name()); var text = invocation.text(); return text == null || text.isBlank() ? Optional.empty() : Optional.of(text.strip()); } catch (RuntimeException ex) { - log.warn("Anomaly AI summary failed for org {}: {}", organizationId, ex.getMessage()); + log.warn("AI {} failed for org {}: {}", logContext, organizationId, ex.getMessage()); return Optional.empty(); } } diff --git a/backend/src/main/java/com/bablsoft/accessflow/ai/internal/DefaultDataDiscoveryAiService.java b/backend/src/main/java/com/bablsoft/accessflow/ai/internal/DefaultDataDiscoveryAiService.java new file mode 100644 index 00000000..007aa7b4 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/ai/internal/DefaultDataDiscoveryAiService.java @@ -0,0 +1,134 @@ +package com.bablsoft.accessflow.ai.internal; + +import com.bablsoft.accessflow.ai.api.DataDiscoveryAiService; +import com.bablsoft.accessflow.core.api.DataClassification; +import lombok.RequiredArgsConstructor; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.stereotype.Service; +import tools.jackson.databind.JsonNode; +import tools.jackson.databind.ObjectMapper; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Locale; +import java.util.UUID; + +/** + * Discovery AI classification pass (AF-623). Builds a plain-text prompt from the (already + * redacted) column contexts, calls the org's first usable {@code ai_config} through the holder's + * freeform lane, and leniently parses the demanded strict-JSON response — unknown columns, + * unknown classifications, and malformed payloads are dropped, never thrown. + * + *

The holder is resolved through an {@link ObjectProvider} (lazy, by-name) rather than injected + * by concrete type, matching {@code AnomalySummaryService}: integration tests that + * {@code @MockitoBean AiAnalyzerStrategy} replace the holder bean with a bare interface mock, + * which is not assignable to the concrete field type. + */ +@Service +@RequiredArgsConstructor +class DefaultDataDiscoveryAiService implements DataDiscoveryAiService { + + private static final Logger log = LoggerFactory.getLogger(DefaultDataDiscoveryAiService.class); + private static final int MAX_RATIONALE_LENGTH = 500; + + private final ObjectProvider strategyHolder; + private final ObjectMapper objectMapper; + + @Override + public List classifyColumns(UUID organizationId, + DiscoveryTableContext context) { + if (organizationId == null || context == null || context.columns().isEmpty()) { + return List.of(); + } + var response = strategyHolder.getObject() + .classifyDiscoveryColumns(organizationId, buildPrompt(context)) + .orElse(null); + if (response == null) { + return List.of(); + } + return parse(response, context); + } + + private static String buildPrompt(DiscoveryTableContext context) { + var sb = new StringBuilder(); + sb.append("Table: ").append(context.tableName()).append('\n'); + sb.append("Columns:\n"); + for (var column : context.columns()) { + sb.append("- ").append(column.name()); + if (column.type() != null && !column.type().isBlank()) { + sb.append(" (").append(column.type()).append(')'); + } + if (!column.redactedSamples().isEmpty()) { + sb.append(" samples: ").append(String.join(", ", column.redactedSamples())); + } + sb.append('\n'); + } + return sb.toString(); + } + + private List parse(String response, DiscoveryTableContext context) { + JsonNode root; + try { + root = objectMapper.readTree(stripCodeFences(response)); + } catch (RuntimeException ex) { + log.warn("Discovery AI response was not valid JSON, ignoring: {}", ex.getMessage()); + return List.of(); + } + var columnsNode = root.path("columns"); + if (!columnsNode.isArray()) { + log.warn("Discovery AI response missing 'columns' array, ignoring"); + return List.of(); + } + var knownColumns = new HashSet(); + for (var column : context.columns()) { + knownColumns.add(column.name().toLowerCase(Locale.ROOT)); + } + var suggestions = new ArrayList(); + for (var node : columnsNode) { + var suggestion = toSuggestion(node, knownColumns); + if (suggestion != null) { + suggestions.add(suggestion); + } + } + return List.copyOf(suggestions); + } + + private DiscoveryColumnSuggestion toSuggestion(JsonNode node, HashSet knownColumns) { + var columnName = node.path("column").asString(""); + if (columnName.isBlank() || !knownColumns.contains(columnName.toLowerCase(Locale.ROOT))) { + return null; + } + DataClassification classification; + try { + classification = DataClassification.valueOf( + node.path("classification").asString("").toUpperCase(Locale.ROOT)); + } catch (IllegalArgumentException ex) { + return null; + } + var confidenceNode = node.path("confidence"); + var confidence = confidenceNode.isNumber() + ? Math.clamp(confidenceNode.asInt(), 0, 100) : 0; + var rationale = node.path("rationale").asString(""); + if (rationale.length() > MAX_RATIONALE_LENGTH) { + rationale = rationale.substring(0, MAX_RATIONALE_LENGTH); + } + return new DiscoveryColumnSuggestion(columnName, classification, confidence, + rationale.isBlank() ? null : rationale); + } + + /** Models sometimes wrap JSON in markdown fences despite instructions — tolerate it. */ + private static String stripCodeFences(String response) { + var trimmed = response.strip(); + if (trimmed.startsWith("```")) { + var firstNewline = trimmed.indexOf('\n'); + var lastFence = trimmed.lastIndexOf("```"); + if (firstNewline >= 0 && lastFence > firstNewline) { + return trimmed.substring(firstNewline + 1, lastFence).strip(); + } + } + return trimmed; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditAction.java b/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditAction.java index 392df831..32f817b0 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditAction.java +++ b/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditAction.java @@ -166,5 +166,9 @@ public enum AuditAction { REQUEST_GROUP_EXECUTED, REQUEST_GROUP_PARTIALLY_EXECUTED, REQUEST_GROUP_FAILED, - REQUEST_GROUP_CANCELLED + REQUEST_GROUP_CANCELLED, + + DISCOVERY_SCAN_COMPLETED, + DISCOVERY_FINDING_CONFIRMED, + DISCOVERY_FINDING_DISMISSED } diff --git a/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditResourceType.java b/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditResourceType.java index db1fbf36..104b7afe 100644 --- a/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditResourceType.java +++ b/backend/src/main/java/com/bablsoft/accessflow/audit/api/AuditResourceType.java @@ -46,7 +46,8 @@ public enum AuditResourceType { RETENTION_POLICY("retention_policy"), DELETION_REQUEST("deletion_request"), REQUEST_GROUP("request_group"), - QUERY_TICKET("query_ticket"); + QUERY_TICKET("query_ticket"), + DISCOVERY_FINDING("discovery_finding"); private final String dbValue; diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryConfigService.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryConfigService.java new file mode 100644 index 00000000..0ca58e1a --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryConfigService.java @@ -0,0 +1,17 @@ +package com.bablsoft.accessflow.discovery.api; + +import java.util.UUID; + +/** + * Admin management of a datasource's discovery settings (AF-623). Both methods validate the + * datasource belongs to the organization (404 otherwise). + */ +public interface DiscoveryConfigService { + + /** Returns the persisted config, or a synthesized default-disabled view when none exists. */ + DiscoveryScanConfigView get(UUID datasourceId, UUID organizationId); + + /** Creates or updates the config row; {@code null} command fields keep the current value. */ + DiscoveryScanConfigView upsert(UUID datasourceId, UUID organizationId, + UpsertDiscoveryConfigCommand command); +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryDecision.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryDecision.java new file mode 100644 index 00000000..c0217b21 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryDecision.java @@ -0,0 +1,7 @@ +package com.bablsoft.accessflow.discovery.api; + +/** Admin decision applied to one or more PENDING discovery findings (AF-623). */ +public enum DiscoveryDecision { + CONFIRM, + DISMISS +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryDetector.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryDetector.java new file mode 100644 index 00000000..7e4ad0a8 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryDetector.java @@ -0,0 +1,15 @@ +package com.bablsoft.accessflow.discovery.api; + +/** + * How a discovery finding was produced (AF-623). Mirrors the PostgreSQL enum + * {@code discovery_detector}. The regex/checksum detectors run locally on sampled values; + * {@code AI} rows come from the optional pass through the org's bound AI analyzer. + */ +public enum DiscoveryDetector { + EMAIL, + CREDIT_CARD, + SSN, + IBAN, + PHONE, + AI +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryException.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryException.java new file mode 100644 index 00000000..b3fc711e --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryException.java @@ -0,0 +1,12 @@ +package com.bablsoft.accessflow.discovery.api; + +/** + * Base type for discovery-module domain exceptions. Concrete subclasses are mapped to + * {@code ProblemDetail} responses by the discovery web layer. + */ +public abstract class DiscoveryException extends RuntimeException { + + protected DiscoveryException(String message) { + super(message); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryFindingService.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryFindingService.java new file mode 100644 index 00000000..ae28817e --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryFindingService.java @@ -0,0 +1,45 @@ +package com.bablsoft.accessflow.discovery.api; + +import com.bablsoft.accessflow.core.api.PageRequest; +import com.bablsoft.accessflow.core.api.PageResponse; + +import java.util.List; +import java.util.UUID; + +/** + * Worklist over proposed sensitive-column classifications (AF-623). Confirming a finding applies + * the classification tag through the existing AF-447 service (which auto-derives masking); + * dismissing permanently suppresses the proposal for future scans. + */ +public interface DiscoveryFindingService { + + /** + * Pages the datasource's findings, newest detection first, optionally filtered by status + * ({@code null} = all). + */ + PageResponse find(UUID datasourceId, UUID organizationId, + DiscoveryFindingStatus status, PageRequest page); + + /** + * Applies the decision to each finding independently (partial success — one bad row never + * rolls back the others) and reports a per-finding outcome row. + */ + BulkDecisionOutcome decide(UUID datasourceId, UUID organizationId, UUID actorId, + List findingIds, DiscoveryDecision decision); + + /** Per-finding outcomes of a bulk decision, in the order the ids were submitted. */ + record BulkDecisionOutcome(List results) { + + public BulkDecisionOutcome { + results = results == null ? List.of() : List.copyOf(results); + } + + /** + * {@code newStatus} is the finding's status after the decision and {@code finding} its + * post-decision view (both {@code null} on NOT_FOUND). + */ + public record Row(UUID findingId, DiscoveryRowStatus status, + DiscoveryFindingStatus newStatus, DiscoveryFindingView finding) { + } + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryFindingStatus.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryFindingStatus.java new file mode 100644 index 00000000..4e9a0f9a --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryFindingStatus.java @@ -0,0 +1,12 @@ +package com.bablsoft.accessflow.discovery.api; + +/** + * Worklist state of a discovery finding (AF-623). Mirrors the PostgreSQL enum + * {@code discovery_finding_status}. {@code CONFIRMED} and {@code DISMISSED} rows are never + * touched by rescans — a dismissal permanently suppresses the proposal. + */ +public enum DiscoveryFindingStatus { + PENDING, + CONFIRMED, + DISMISSED +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryFindingView.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryFindingView.java new file mode 100644 index 00000000..fb46fb81 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryFindingView.java @@ -0,0 +1,30 @@ +package com.bablsoft.accessflow.discovery.api; + +import com.bablsoft.accessflow.core.api.DataClassification; + +import java.time.Instant; +import java.util.UUID; + +/** + * A proposed sensitive-column classification awaiting an admin decision (AF-623). + * {@code sampleRedacted} only ever carries a redacted value — raw sampled data never persists. + * {@code rationale} is populated for {@link DiscoveryDetector#AI} findings only. + */ +public record DiscoveryFindingView( + UUID id, + String schemaName, + String tableName, + String columnName, + DataClassification classification, + DiscoveryDetector detector, + int confidence, + String sampleRedacted, + String rationale, + int matchCount, + int sampleCount, + DiscoveryFindingStatus status, + Instant firstDetectedAt, + Instant lastDetectedAt, + UUID decidedBy, + Instant decidedAt) { +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryRowStatus.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryRowStatus.java new file mode 100644 index 00000000..5cd26489 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryRowStatus.java @@ -0,0 +1,14 @@ +package com.bablsoft.accessflow.discovery.api; + +/** + * Per-finding outcome of a bulk decision (AF-623). {@code TAG_CONFLICT} means the classification + * tag already existed (e.g. added manually since the scan) — the finding is still marked + * CONFIRMED so the worklist clears, but no new tag or derived masking was created. + */ +public enum DiscoveryRowStatus { + SUCCESS, + NOT_FOUND, + INVALID_STATE, + TAG_CONFLICT, + ERROR +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryScanAlreadyRunningException.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryScanAlreadyRunningException.java new file mode 100644 index 00000000..1e5009cc --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryScanAlreadyRunningException.java @@ -0,0 +1,18 @@ +package com.bablsoft.accessflow.discovery.api; + +import java.util.UUID; + +/** A discovery scan for the datasource is already in flight on this node (AF-623) — HTTP 409. */ +public class DiscoveryScanAlreadyRunningException extends DiscoveryException { + + private final UUID datasourceId; + + public DiscoveryScanAlreadyRunningException(UUID datasourceId) { + super("Discovery scan already running for datasource " + datasourceId); + this.datasourceId = datasourceId; + } + + public UUID datasourceId() { + return datasourceId; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryScanConfigView.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryScanConfigView.java new file mode 100644 index 00000000..6c43bf6b --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryScanConfigView.java @@ -0,0 +1,19 @@ +package com.bablsoft.accessflow.discovery.api; + +import java.time.Instant; +import java.util.UUID; + +/** + * Per-datasource discovery settings (AF-623). When a datasource has no persisted config row the + * service synthesizes this view with the defaults ({@code enabled=false}, {@code sampleSize=100}, + * {@code scanIntervalHours=24}, {@code aiClassificationEnabled=false}). + */ +public record DiscoveryScanConfigView( + UUID datasourceId, + boolean enabled, + int sampleSize, + int scanIntervalHours, + boolean aiClassificationEnabled, + Instant lastScanAt, + String lastScanError) { +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryScanTriggerService.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryScanTriggerService.java new file mode 100644 index 00000000..771f2b36 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/DiscoveryScanTriggerService.java @@ -0,0 +1,16 @@ +package com.bablsoft.accessflow.discovery.api; + +import java.util.UUID; + +/** + * On-demand "Scan now" trigger (AF-623). The scan runs asynchronously on a virtual thread; + * ad-hoc scans are allowed even when the datasource has not opted into scheduled discovery. + */ +public interface DiscoveryScanTriggerService { + + /** + * @throws DiscoveryScanAlreadyRunningException when a scan for the datasource is already + * in flight on this node (HTTP 409) + */ + void requestScan(UUID datasourceId, UUID organizationId, UUID actorId); +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/UpsertDiscoveryConfigCommand.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/UpsertDiscoveryConfigCommand.java new file mode 100644 index 00000000..3f736cdf --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/UpsertDiscoveryConfigCommand.java @@ -0,0 +1,13 @@ +package com.bablsoft.accessflow.discovery.api; + +/** + * Upsert command for a datasource's discovery settings (AF-623). {@code null} fields keep the + * current (or default) value, mirroring the partial-update convention of + * {@code UpdateDatasourceCommand}. + */ +public record UpsertDiscoveryConfigCommand( + Boolean enabled, + Integer sampleSize, + Integer scanIntervalHours, + Boolean aiClassificationEnabled) { +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/api/package-info.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/package-info.java new file mode 100644 index 00000000..8387e926 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/api/package-info.java @@ -0,0 +1,4 @@ +@NamedInterface +package com.bablsoft.accessflow.discovery.api; + +import org.springframework.modulith.NamedInterface; diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryConfigService.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryConfigService.java new file mode 100644 index 00000000..78be573f --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryConfigService.java @@ -0,0 +1,67 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.core.api.DatasourceAdminService; +import com.bablsoft.accessflow.discovery.api.DiscoveryConfigService; +import com.bablsoft.accessflow.discovery.api.DiscoveryScanConfigView; +import com.bablsoft.accessflow.discovery.api.UpsertDiscoveryConfigCommand; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryScanConfigEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryScanConfigRepository; +import lombok.RequiredArgsConstructor; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.util.UUID; + +@Service +@RequiredArgsConstructor +class DefaultDiscoveryConfigService implements DiscoveryConfigService { + + private final DiscoveryScanConfigRepository configRepository; + private final DatasourceAdminService datasourceAdminService; + + @Override + @Transactional(readOnly = true) + public DiscoveryScanConfigView get(UUID datasourceId, UUID organizationId) { + requireDatasource(datasourceId, organizationId); + return configRepository.findByDatasourceIdAndOrganizationId(datasourceId, organizationId) + .map(DiscoveryViewMapper::toView) + .orElseGet(() -> defaultView(datasourceId)); + } + + @Override + @Transactional + public DiscoveryScanConfigView upsert(UUID datasourceId, UUID organizationId, + UpsertDiscoveryConfigCommand command) { + requireDatasource(datasourceId, organizationId); + var config = configRepository.findByDatasourceIdAndOrganizationId(datasourceId, + organizationId).orElseGet(() -> { + var created = new DiscoveryScanConfigEntity(); + created.setId(UUID.randomUUID()); + created.setOrganizationId(organizationId); + created.setDatasourceId(datasourceId); + return created; + }); + if (command.enabled() != null) { + config.setEnabled(command.enabled()); + } + if (command.sampleSize() != null) { + config.setSampleSize(command.sampleSize()); + } + if (command.scanIntervalHours() != null) { + config.setScanIntervalHours(command.scanIntervalHours()); + } + if (command.aiClassificationEnabled() != null) { + config.setAiClassificationEnabled(command.aiClassificationEnabled()); + } + return DiscoveryViewMapper.toView(configRepository.save(config)); + } + + /** 404 (DatasourceNotFoundException) when the datasource is not in the caller's org. */ + private void requireDatasource(UUID datasourceId, UUID organizationId) { + datasourceAdminService.getForAdmin(datasourceId, organizationId); + } + + private static DiscoveryScanConfigView defaultView(UUID datasourceId) { + return new DiscoveryScanConfigView(datasourceId, false, 100, 24, false, null, null); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryFindingService.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryFindingService.java new file mode 100644 index 00000000..a4dbeb04 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryFindingService.java @@ -0,0 +1,88 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.core.api.PageRequest; +import com.bablsoft.accessflow.core.api.PageResponse; +import com.bablsoft.accessflow.discovery.api.DiscoveryDecision; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingService; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingView; +import com.bablsoft.accessflow.discovery.api.DiscoveryRowStatus; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryFindingRepository; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.springframework.data.domain.Sort; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.util.ArrayList; +import java.util.List; +import java.util.UUID; + +@Service +@RequiredArgsConstructor +@Slf4j +class DefaultDiscoveryFindingService implements DiscoveryFindingService { + + private final DiscoveryFindingRepository findingRepository; + private final DiscoveryFindingStateService stateService; + + @Override + @Transactional(readOnly = true) + public PageResponse find(UUID datasourceId, UUID organizationId, + DiscoveryFindingStatus status, + PageRequest page) { + var pageable = DiscoveryPageAdapter.toSpringPageable(page); + if (pageable.isPaged() && pageable.getSort().isUnsorted()) { + pageable = org.springframework.data.domain.PageRequest.of(pageable.getPageNumber(), + pageable.getPageSize(), Sort.by(Sort.Direction.DESC, "lastDetectedAt")); + } + var result = status == null + ? findingRepository.findAllByDatasourceIdAndOrganizationId(datasourceId, + organizationId, pageable) + : findingRepository.findAllByDatasourceIdAndOrganizationIdAndStatus(datasourceId, + organizationId, status, pageable); + return DiscoveryPageAdapter.toPageResponse(result.map(DiscoveryViewMapper::toView)); + } + + @Override + public BulkDecisionOutcome decide(UUID datasourceId, UUID organizationId, UUID actorId, + List findingIds, DiscoveryDecision decision) { + // Intentionally NOT @Transactional: each row is decided by DiscoveryFindingStateService in + // its own transaction, so one bad row never rolls back a successful peer. + var rows = new ArrayList(findingIds.size()); + for (var findingId : findingIds) { + rows.add(decideOne(datasourceId, organizationId, actorId, findingId, decision)); + } + return new BulkDecisionOutcome(List.copyOf(rows)); + } + + private BulkDecisionOutcome.Row decideOne(UUID datasourceId, UUID organizationId, UUID actorId, + UUID findingId, DiscoveryDecision decision) { + var finding = findingRepository.findByIdAndDatasourceIdAndOrganizationId(findingId, + datasourceId, organizationId).orElse(null); + if (finding == null) { + return new BulkDecisionOutcome.Row(findingId, DiscoveryRowStatus.NOT_FOUND, null, null); + } + if (finding.getStatus() != DiscoveryFindingStatus.PENDING) { + return new BulkDecisionOutcome.Row(findingId, DiscoveryRowStatus.INVALID_STATE, + finding.getStatus(), DiscoveryViewMapper.toView(finding)); + } + try { + if (decision == DiscoveryDecision.DISMISS) { + var dismissed = stateService.dismiss(finding, actorId); + return new BulkDecisionOutcome.Row(findingId, DiscoveryRowStatus.SUCCESS, + DiscoveryFindingStatus.DISMISSED, DiscoveryViewMapper.toView(dismissed)); + } + var outcome = stateService.confirm(finding, actorId); + return new BulkDecisionOutcome.Row(findingId, + outcome.tagConflict() ? DiscoveryRowStatus.TAG_CONFLICT + : DiscoveryRowStatus.SUCCESS, + DiscoveryFindingStatus.CONFIRMED, + DiscoveryViewMapper.toView(outcome.finding())); + } catch (RuntimeException ex) { + log.error("Discovery finding decision failed for finding {}", findingId, ex); + return new BulkDecisionOutcome.Row(findingId, DiscoveryRowStatus.ERROR, + DiscoveryFindingStatus.PENDING, DiscoveryViewMapper.toView(finding)); + } + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryScanTriggerService.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryScanTriggerService.java new file mode 100644 index 00000000..22a6150f --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryScanTriggerService.java @@ -0,0 +1,41 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.core.api.DatasourceAdminService; +import com.bablsoft.accessflow.discovery.api.DiscoveryScanAlreadyRunningException; +import com.bablsoft.accessflow.discovery.api.DiscoveryScanTriggerService; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.springframework.stereotype.Service; + +import java.util.UUID; +import java.util.concurrent.ExecutorService; + +@Service +@RequiredArgsConstructor +@Slf4j +class DefaultDiscoveryScanTriggerService implements DiscoveryScanTriggerService { + + private final DiscoveryScanService scanService; + private final DatasourceAdminService datasourceAdminService; + private final ExecutorService discoveryScanExecutor; + + @Override + public void requestScan(UUID datasourceId, UUID organizationId, UUID actorId) { + // 404 (DatasourceNotFoundException) when the datasource is not in the caller's org. + datasourceAdminService.getForAdmin(datasourceId, organizationId); + if (scanService.isInFlight(datasourceId)) { + throw new DiscoveryScanAlreadyRunningException(datasourceId); + } + discoveryScanExecutor.submit(() -> { + try { + scanService.scan(datasourceId, organizationId, actorId); + } catch (DiscoveryScanAlreadyRunningException ex) { + // Raced another trigger between the pre-check and the scan; the winner proceeds. + log.info("Discovery scan for datasource {} already running; trigger skipped", + datasourceId); + } catch (RuntimeException ex) { + log.error("On-demand discovery scan failed for datasource {}", datasourceId, ex); + } + }); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryFindingStateService.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryFindingStateService.java new file mode 100644 index 00000000..d72de0a9 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryFindingStateService.java @@ -0,0 +1,83 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.core.api.CreateDataClassificationTagCommand; +import com.bablsoft.accessflow.core.api.DataClassificationAdminService; +import com.bablsoft.accessflow.core.api.IllegalDataClassificationTagException; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryFindingEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryFindingRepository; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +import java.time.Clock; +import java.util.List; +import java.util.UUID; + +/** + * Single-finding decision mutations (AF-623), each in its own transaction so one bad row in a + * bulk decision never poisons a successful peer (the attestation bulk pattern). + */ +@Service +@RequiredArgsConstructor +@Slf4j +class DiscoveryFindingStateService { + + private final DiscoveryFindingRepository findingRepository; + private final DataClassificationAdminService dataClassificationAdminService; + private final Clock clock; + + /** Outcome of a single confirm: the finding plus whether the tag already existed. */ + record ConfirmOutcome(DiscoveryFindingEntity finding, boolean tagConflict) { + } + + /** + * Applies the classification tag through the AF-447 service (deriving masking for the column) + * and marks the finding CONFIRMED. A pre-existing tag ({@link + * IllegalDataClassificationTagException}) still confirms the finding — the worklist clears — + * but is reported as a conflict. + */ + @Transactional + ConfirmOutcome confirm(DiscoveryFindingEntity finding, UUID actorId) { + var tagConflict = false; + try { + dataClassificationAdminService.create(finding.getDatasourceId(), + finding.getOrganizationId(), new CreateDataClassificationTagCommand( + qualifiedTable(finding), finding.getColumnName(), + List.of(finding.getClassification()), confirmNote(finding), true)); + } catch (IllegalDataClassificationTagException ex) { + log.info("Classification tag already exists for {}.{} ({}); confirming finding {} without a new tag", + qualifiedTable(finding), finding.getColumnName(), finding.getClassification(), + finding.getId()); + tagConflict = true; + } + decide(finding, DiscoveryFindingStatus.CONFIRMED, actorId); + return new ConfirmOutcome(finding, tagConflict); + } + + @Transactional + DiscoveryFindingEntity dismiss(DiscoveryFindingEntity finding, UUID actorId) { + decide(finding, DiscoveryFindingStatus.DISMISSED, actorId); + return finding; + } + + private void decide(DiscoveryFindingEntity finding, DiscoveryFindingStatus status, + UUID actorId) { + finding.setStatus(status); + finding.setDecidedBy(actorId); + finding.setDecidedAt(clock.instant()); + findingRepository.save(finding); + } + + private static String qualifiedTable(DiscoveryFindingEntity finding) { + return finding.getSchemaName() == null ? finding.getTableName() + : finding.getSchemaName() + "." + finding.getTableName(); + } + + // Stored free-text note (like manual tag notes) — deliberately not localized. + private static String confirmNote(DiscoveryFindingEntity finding) { + return "Confirmed from discovery finding (" + finding.getDetector() + ", " + + finding.getConfidence() + "% confidence)"; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryPageAdapter.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryPageAdapter.java new file mode 100644 index 00000000..df718e11 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryPageAdapter.java @@ -0,0 +1,44 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.core.api.PageRequest; +import com.bablsoft.accessflow.core.api.PageResponse; +import com.bablsoft.accessflow.core.api.SortOrder; +import org.springframework.data.domain.Page; +import org.springframework.data.domain.Pageable; +import org.springframework.data.domain.Sort; + +/** + * Bridges the library-agnostic {@link PageRequest}/{@link PageResponse} types and Spring Data's + * {@code Pageable}/{@code Page} inside the discovery module (core's PageAdapter is module-private). + */ +final class DiscoveryPageAdapter { + + private DiscoveryPageAdapter() { + } + + static Pageable toSpringPageable(PageRequest request) { + if (request == null) { + return Pageable.unpaged(); + } + return org.springframework.data.domain.PageRequest.of(request.page(), request.size(), + toSpringSort(request)); + } + + static PageResponse toPageResponse(Page page) { + return new PageResponse<>(page.getContent(), page.getNumber(), page.getSize(), + page.getTotalElements(), page.getTotalPages()); + } + + private static Sort toSpringSort(PageRequest request) { + if (request.sort().isEmpty()) { + return Sort.unsorted(); + } + return Sort.by(request.sort().stream().map(DiscoveryPageAdapter::toSpringOrder).toList()); + } + + private static Sort.Order toSpringOrder(SortOrder sortOrder) { + var direction = sortOrder.direction() == SortOrder.Direction.ASC + ? Sort.Direction.ASC : Sort.Direction.DESC; + return new Sort.Order(direction, sortOrder.property()); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryScanService.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryScanService.java new file mode 100644 index 00000000..98ce0f51 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryScanService.java @@ -0,0 +1,477 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.ai.api.DataDiscoveryAiService; +import com.bablsoft.accessflow.audit.api.AuditAction; +import com.bablsoft.accessflow.audit.api.AuditEntry; +import com.bablsoft.accessflow.audit.api.AuditLogService; +import com.bablsoft.accessflow.audit.api.AuditResourceType; +import com.bablsoft.accessflow.core.api.ColumnMasker; +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.core.api.DataClassificationQueryService; +import com.bablsoft.accessflow.core.api.DatabaseSchemaView; +import com.bablsoft.accessflow.core.api.DatasourceAdminService; +import com.bablsoft.accessflow.core.api.MaskingPolicyAdminService; +import com.bablsoft.accessflow.core.api.MaskingStrategy; +import com.bablsoft.accessflow.core.api.SampleTableRequest; +import com.bablsoft.accessflow.core.api.SelectExecutionResult; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.api.DiscoveryScanAlreadyRunningException; +import com.bablsoft.accessflow.discovery.internal.config.DiscoveryProperties; +import com.bablsoft.accessflow.discovery.internal.detect.ValueDetector; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryFindingEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryScanConfigEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryFindingRepository; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryScanConfigRepository; +import com.bablsoft.accessflow.proxy.api.QueryExecutor; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.springframework.stereotype.Service; + +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.UUID; +import java.util.concurrent.ConcurrentHashMap; + +/** + * The discovery scan pipeline (AF-623): enumerate tables via system-lane introspection, read a + * bounded raw sample per table through {@link QueryExecutor#sampleTable} (raw values live only on + * this method's stack — findings persist a redacted sample only), run the local + * {@link ValueDetector} pipeline, optionally the AI pass (redacted samples only), and upsert + * PENDING findings. CONFIRMED/DISMISSED rows are never touched — a dismissal permanently + * suppresses the proposal. + * + *

The in-flight guard is per node; cluster races between a "Scan now" and the scheduled job + * are harmless because upserts are idempotent and the natural-key unique index breaks ties. + */ +@Service +@RequiredArgsConstructor +@Slf4j +public class DiscoveryScanService { + + static final int MIN_SAMPLE_COUNT = 5; + static final double MIN_MATCH_RATIO = 0.30; + private static final int MAX_AI_SAMPLES_PER_COLUMN = 5; + private static final int MAX_ERROR_LENGTH = 500; + private static final Map PARTIAL_PARAMS = Map.of("visible_suffix", "4"); + + private final DiscoveryScanConfigRepository configRepository; + private final DiscoveryFindingRepository findingRepository; + private final DatasourceAdminService datasourceAdminService; + private final DataClassificationQueryService dataClassificationQueryService; + private final MaskingPolicyAdminService maskingPolicyAdminService; + private final QueryExecutor queryExecutor; + private final DataDiscoveryAiService dataDiscoveryAiService; + private final AuditLogService auditLogService; + private final DiscoveryProperties properties; + private final Clock clock; + + private final Set inFlight = ConcurrentHashMap.newKeySet(); + + /** Best-effort pre-check for the on-demand trigger; {@link #scan} re-checks atomically. */ + boolean isInFlight(UUID datasourceId) { + return inFlight.contains(datasourceId); + } + + /** + * Runs a full scan of the datasource synchronously. Never throws once started — all failures + * are logged, stamped on the config row, and audited. {@code actorId} is {@code null} for the + * scheduled path. + * + * @throws DiscoveryScanAlreadyRunningException when a scan for the datasource is already in + * flight on this node + */ + public void scan(UUID datasourceId, UUID organizationId, UUID actorId) { + if (!inFlight.add(datasourceId)) { + throw new DiscoveryScanAlreadyRunningException(datasourceId); + } + try { + runScan(datasourceId, organizationId, actorId); + } finally { + inFlight.remove(datasourceId); + } + } + + private void runScan(UUID datasourceId, UUID organizationId, UUID actorId) { + var startedAt = clock.instant(); + var stats = new ScanStats(); + String error = null; + try { + var config = configRepository.findByDatasourceIdAndOrganizationId(datasourceId, + organizationId).orElse(null); + var sampleSize = config == null ? 100 : config.getSampleSize(); + var aiEnabled = config != null && config.isAiClassificationEnabled(); + + var schemaView = datasourceAdminService.introspectSchemaForSystem(datasourceId, + organizationId); + var targets = flattenTargets(schemaView); + if (targets.size() > properties.maxTablesPerScan()) { + stats.tablesSkipped = targets.size() - properties.maxTablesPerScan(); + targets = targets.subList(0, properties.maxTablesPerScan()); + stats.partial = true; + log.info("Discovery scan for datasource {} capped at {} tables ({} skipped)", + datasourceId, properties.maxTablesPerScan(), stats.tablesSkipped); + } + + var taggedKeys = loadTaggedKeys(datasourceId, organizationId); + var maskedColumnRefs = loadMaskedColumnRefs(datasourceId, organizationId); + var deadline = startedAt.plus(properties.scanTimeBudget()); + + for (var i = 0; i < targets.size(); i++) { + var target = targets.get(i); + if (!clock.instant().isBefore(deadline)) { + stats.partial = true; + stats.tablesSkipped += targets.size() - i; + log.warn("Discovery scan for datasource {} hit its time budget after {} tables", + datasourceId, stats.tablesScanned); + break; + } + try { + scanTable(datasourceId, organizationId, target, sampleSize, aiEnabled, + taggedKeys, maskedColumnRefs, stats); + stats.tablesScanned++; + } catch (RuntimeException ex) { + stats.tablesFailed++; + log.error("Discovery scan failed for table {} of datasource {}: {}", + target.qualifiedName(), datasourceId, ex.getMessage()); + } + } + if (stats.tablesFailed > 0 && stats.tablesScanned == 0) { + error = truncate("All " + stats.tablesFailed + " sampled tables failed"); + } + } catch (RuntimeException ex) { + log.error("Discovery scan failed for datasource {}", datasourceId, ex); + error = truncate(ex.getMessage() == null ? ex.getClass().getSimpleName() + : ex.getMessage()); + } + stampConfig(datasourceId, organizationId, error); + recordScanAudit(datasourceId, organizationId, actorId, startedAt, stats, error); + } + + private void scanTable(UUID datasourceId, UUID organizationId, TableTarget target, + int sampleSize, boolean aiEnabled, Set taggedKeys, + Set maskedColumnRefs, ScanStats stats) { + var result = queryExecutor.sampleTable(new SampleTableRequest(datasourceId, + target.schemaName(), target.tableName(), sampleSize, + properties.sampleStatementTimeout())); + if (!(result instanceof SelectExecutionResult select)) { + return; + } + var columnValues = collectStringColumns(select); + var now = clock.instant(); + var aiCandidates = new ArrayList(); + var columnTypes = columnTypesByName(target); + + for (var entry : columnValues.entrySet()) { + var columnName = entry.getKey(); + var values = entry.getValue(); + if (values.size() < MIN_SAMPLE_COUNT + || isMasked(target, columnName, maskedColumnRefs)) { + continue; + } + var matches = detect(values); + var proposed = false; + for (var detectorEntry : matches.entrySet()) { + var detector = detectorEntry.getKey(); + var detectorMatches = detectorEntry.getValue(); + var ratio = detectorMatches.count / (double) values.size(); + if (ratio < MIN_MATCH_RATIO) { + continue; + } + proposed = true; + if (isTagged(taggedKeys, target, columnName, detector.classification())) { + continue; + } + upsertFinding(datasourceId, organizationId, target, columnName, + detector.classification(), detector.type(), + (int) Math.round(100.0 * ratio), + ColumnMasker.apply(MaskingStrategy.PARTIAL, detectorMatches.firstMatch, + PARTIAL_PARAMS), + null, detectorMatches.count, values.size(), now, stats); + } + if (aiEnabled && !proposed + && !isTaggedAnyClassification(taggedKeys, target, columnName)) { + aiCandidates.add(new DataDiscoveryAiService.DiscoveryColumnContext(columnName, + columnTypes.get(columnName.toLowerCase(Locale.ROOT)), + redactForAi(values))); + } + } + + if (aiEnabled && !aiCandidates.isEmpty() + && stats.aiTablesUsed < properties.maxAiTablesPerScan()) { + stats.aiTablesUsed++; + runAiPass(datasourceId, organizationId, target, aiCandidates, columnValues, + taggedKeys, stats); + } + } + + private void runAiPass(UUID datasourceId, UUID organizationId, TableTarget target, + List candidates, + Map> columnValues, Set taggedKeys, + ScanStats stats) { + var suggestions = dataDiscoveryAiService.classifyColumns(organizationId, + new DataDiscoveryAiService.DiscoveryTableContext(target.qualifiedName(), + candidates)); + var now = clock.instant(); + for (var suggestion : suggestions) { + if (isTagged(taggedKeys, target, suggestion.columnName(), + suggestion.classification())) { + continue; + } + var values = columnValues.getOrDefault(suggestion.columnName(), List.of()); + var sample = values.isEmpty() ? null + : ColumnMasker.apply(MaskingStrategy.FORMAT_PRESERVING, values.getFirst(), + Map.of()); + upsertFinding(datasourceId, organizationId, target, suggestion.columnName(), + suggestion.classification(), DiscoveryDetector.AI, suggestion.confidence(), + sample, suggestion.rationale(), 0, values.size(), now, stats); + stats.aiSuggestions++; + } + } + + private void upsertFinding(UUID datasourceId, UUID organizationId, TableTarget target, + String columnName, DataClassification classification, + DiscoveryDetector detector, int confidence, String sampleRedacted, + String rationale, int matchCount, int sampleCount, Instant now, + ScanStats stats) { + var existing = findingRepository.findByNaturalKey(organizationId, datasourceId, + target.schemaName(), target.tableName(), columnName, classification, detector) + .orElse(null); + if (existing == null) { + var entity = new DiscoveryFindingEntity(); + entity.setId(UUID.randomUUID()); + entity.setOrganizationId(organizationId); + entity.setDatasourceId(datasourceId); + entity.setSchemaName(target.schemaName()); + entity.setTableName(target.tableName()); + entity.setColumnName(columnName); + entity.setClassification(classification); + entity.setDetector(detector); + entity.setConfidence(confidence); + entity.setSampleRedacted(sampleRedacted); + entity.setRationale(rationale); + entity.setMatchCount(matchCount); + entity.setSampleCount(sampleCount); + entity.setStatus(DiscoveryFindingStatus.PENDING); + entity.setFirstDetectedAt(now); + entity.setLastDetectedAt(now); + findingRepository.save(entity); + stats.findingsCreated++; + return; + } + if (existing.getStatus() != DiscoveryFindingStatus.PENDING) { + return; + } + existing.setConfidence(confidence); + existing.setSampleRedacted(sampleRedacted); + existing.setRationale(rationale); + existing.setMatchCount(matchCount); + existing.setSampleCount(sampleCount); + existing.setLastDetectedAt(now); + findingRepository.save(existing); + stats.findingsRefreshed++; + } + + /** Non-blank string values per column name, preserving the result's column order. */ + private static Map> collectStringColumns(SelectExecutionResult select) { + var byColumn = new java.util.LinkedHashMap>(); + var columns = select.columns(); + for (var column : columns) { + byColumn.put(column.name(), new ArrayList<>()); + } + for (var row : select.rows()) { + for (var i = 0; i < columns.size() && i < row.size(); i++) { + if (row.get(i) instanceof String s && !s.isBlank()) { + byColumn.get(columns.get(i).name()).add(s); + } + } + } + return byColumn; + } + + /** First-match-wins detector counts over the column's sampled values. */ + private static Map detect(List values) { + var matches = new HashMap(); + for (var value : values) { + for (var detector : ValueDetector.ORDERED) { + if (detector.matches(value)) { + matches.computeIfAbsent(detector, key -> new DetectorMatches(value)).count++; + break; + } + } + } + return matches; + } + + private List redactForAi(List values) { + return values.stream() + .limit(MAX_AI_SAMPLES_PER_COLUMN) + .map(value -> ColumnMasker.apply(MaskingStrategy.FORMAT_PRESERVING, value, Map.of())) + .toList(); + } + + private static List flattenTargets(DatabaseSchemaView schemaView) { + var targets = new ArrayList(); + for (var schema : schemaView.schemas()) { + var schemaName = schema.name() == null || schema.name().isBlank() ? null : schema.name(); + for (var table : schema.tables()) { + targets.add(new TableTarget(schemaName, table.name(), table.columns())); + } + } + return targets; + } + + private Map columnTypesByName(TableTarget target) { + var types = new HashMap(); + for (var column : target.columns()) { + types.put(column.name().toLowerCase(Locale.ROOT), column.type()); + } + return types; + } + + /** + * Existing-tag keys as {@code table|column|classification} (lowercased), with the table part + * both bare and schema-qualified — AF-447 tags store either form. + */ + private Set loadTaggedKeys(UUID datasourceId, UUID organizationId) { + var keys = new HashSet(); + for (var tag : dataClassificationQueryService.findByDatasource(datasourceId, + organizationId)) { + if (tag.columnName() == null) { + continue; + } + keys.add(tagKey(tag.tableName(), tag.columnName(), tag.classification())); + } + return keys; + } + + private boolean isTagged(Set taggedKeys, TableTarget target, String columnName, + DataClassification classification) { + if (taggedKeys.contains(tagKey(target.tableName(), columnName, classification))) { + return true; + } + return target.schemaName() != null && taggedKeys.contains( + tagKey(target.schemaName() + "." + target.tableName(), columnName, classification)); + } + + private boolean isTaggedAnyClassification(Set taggedKeys, TableTarget target, + String columnName) { + for (var classification : DataClassification.values()) { + if (isTagged(taggedKeys, target, columnName, classification)) { + return true; + } + } + return false; + } + + private static String tagKey(String tableName, String columnName, + DataClassification classification) { + return (tableName + "|" + columnName + "|" + classification).toLowerCase(Locale.ROOT); + } + + /** Enabled masking-policy column refs, lowercased — already-masked columns are skipped. */ + private Set loadMaskedColumnRefs(UUID datasourceId, UUID organizationId) { + var refs = new HashSet(); + for (var policy : maskingPolicyAdminService.listForDatasource(datasourceId, + organizationId)) { + if (policy.enabled()) { + refs.add(policy.columnRef().toLowerCase(Locale.ROOT)); + } + } + return refs; + } + + /** Mirrors the executor's mask-matching precedence: schema.table.column, table.column, column. */ + private boolean isMasked(TableTarget target, String columnName, Set maskedColumnRefs) { + var column = columnName.toLowerCase(Locale.ROOT); + var table = target.tableName().toLowerCase(Locale.ROOT); + if (maskedColumnRefs.contains(column) || maskedColumnRefs.contains(table + "." + column)) { + return true; + } + return target.schemaName() != null && maskedColumnRefs.contains( + target.schemaName().toLowerCase(Locale.ROOT) + "." + table + "." + column); + } + + private void stampConfig(UUID datasourceId, UUID organizationId, String error) { + try { + var config = configRepository.findByDatasourceIdAndOrganizationId(datasourceId, + organizationId).orElseGet(() -> { + var created = new DiscoveryScanConfigEntity(); + created.setId(UUID.randomUUID()); + created.setOrganizationId(organizationId); + created.setDatasourceId(datasourceId); + return created; + }); + config.setLastScanAt(clock.instant()); + config.setLastScanError(error); + configRepository.save(config); + } catch (RuntimeException ex) { + log.error("Failed to stamp discovery scan outcome for datasource {}", datasourceId, ex); + } + } + + private void recordScanAudit(UUID datasourceId, UUID organizationId, UUID actorId, + Instant startedAt, ScanStats stats, String error) { + try { + var metadata = new HashMap(); + metadata.put("datasourceId", datasourceId.toString()); + metadata.put("tablesScanned", stats.tablesScanned); + metadata.put("tablesSkipped", stats.tablesSkipped); + metadata.put("tablesFailed", stats.tablesFailed); + metadata.put("findingsCreated", stats.findingsCreated); + metadata.put("findingsRefreshed", stats.findingsRefreshed); + metadata.put("aiSuggestions", stats.aiSuggestions); + metadata.put("durationMs", Duration.between(startedAt, clock.instant()).toMillis()); + metadata.put("partial", stats.partial); + if (error != null) { + metadata.put("error", error); + } + auditLogService.record(new AuditEntry(AuditAction.DISCOVERY_SCAN_COMPLETED, + AuditResourceType.DATASOURCE, datasourceId, organizationId, actorId, + metadata, null, null)); + } catch (RuntimeException ex) { + log.error("Audit write failed for discovery scan of datasource {}", datasourceId, ex); + } + } + + private static String truncate(String message) { + return message.length() > MAX_ERROR_LENGTH + ? message.substring(0, MAX_ERROR_LENGTH) : message; + } + + private record TableTarget(String schemaName, String tableName, + List columns) { + + String qualifiedName() { + return schemaName == null ? tableName : schemaName + "." + tableName; + } + } + + private static final class DetectorMatches { + private final String firstMatch; + private int count; + + private DetectorMatches(String firstMatch) { + this.firstMatch = firstMatch; + } + } + + private static final class ScanStats { + private int tablesScanned; + private int tablesSkipped; + private int tablesFailed; + private int findingsCreated; + private int findingsRefreshed; + private int aiSuggestions; + private int aiTablesUsed; + private boolean partial; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryViewMapper.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryViewMapper.java new file mode 100644 index 00000000..affa0ddc --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/DiscoveryViewMapper.java @@ -0,0 +1,28 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingView; +import com.bablsoft.accessflow.discovery.api.DiscoveryScanConfigView; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryFindingEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryScanConfigEntity; + +final class DiscoveryViewMapper { + + private DiscoveryViewMapper() { + } + + static DiscoveryScanConfigView toView(DiscoveryScanConfigEntity entity) { + return new DiscoveryScanConfigView(entity.getDatasourceId(), entity.isEnabled(), + entity.getSampleSize(), entity.getScanIntervalHours(), + entity.isAiClassificationEnabled(), entity.getLastScanAt(), + entity.getLastScanError()); + } + + static DiscoveryFindingView toView(DiscoveryFindingEntity entity) { + return new DiscoveryFindingView(entity.getId(), entity.getSchemaName(), + entity.getTableName(), entity.getColumnName(), entity.getClassification(), + entity.getDetector(), entity.getConfidence(), entity.getSampleRedacted(), + entity.getRationale(), entity.getMatchCount(), entity.getSampleCount(), + entity.getStatus(), entity.getFirstDetectedAt(), entity.getLastDetectedAt(), + entity.getDecidedBy(), entity.getDecidedAt()); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/config/DiscoveryConfiguration.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/config/DiscoveryConfiguration.java new file mode 100644 index 00000000..ebac5027 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/config/DiscoveryConfiguration.java @@ -0,0 +1,19 @@ +package com.bablsoft.accessflow.discovery.internal.config; + +import org.springframework.boot.context.properties.EnableConfigurationProperties; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; + +@Configuration(proxyBeanMethods = false) +@EnableConfigurationProperties(DiscoveryProperties.class) +class DiscoveryConfiguration { + + /** Runs on-demand "Scan now" requests off the request thread (virtual threads, AF-623). */ + @Bean(destroyMethod = "close") + ExecutorService discoveryScanExecutor() { + return Executors.newVirtualThreadPerTaskExecutor(); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/config/DiscoveryProperties.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/config/DiscoveryProperties.java new file mode 100644 index 00000000..8bc6d2ac --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/config/DiscoveryProperties.java @@ -0,0 +1,45 @@ +package com.bablsoft.accessflow.discovery.internal.config; + +import org.springframework.boot.context.properties.ConfigurationProperties; + +import java.time.Duration; + +/** + * Tunables for the sensitive-data discovery module (AF-623). + * + *

+ */ +@ConfigurationProperties("accessflow.discovery") +public record DiscoveryProperties(Duration scanPollInterval, Duration scanTimeBudget, + Duration sampleStatementTimeout, Integer maxTablesPerScan, + Integer maxAiTablesPerScan) { + + public DiscoveryProperties { + if (scanPollInterval == null) { + scanPollInterval = Duration.ofMinutes(15); + } + if (scanTimeBudget == null || scanTimeBudget.isNegative() || scanTimeBudget.isZero()) { + scanTimeBudget = Duration.ofMinutes(10); + } + if (sampleStatementTimeout == null || sampleStatementTimeout.isNegative() + || sampleStatementTimeout.isZero()) { + sampleStatementTimeout = Duration.ofSeconds(10); + } + if (maxTablesPerScan == null || maxTablesPerScan <= 0) { + maxTablesPerScan = 200; + } + if (maxAiTablesPerScan == null || maxAiTablesPerScan < 0) { + maxAiTablesPerScan = 25; + } + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/CreditCardDetector.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/CreditCardDetector.java new file mode 100644 index 00000000..a08161fc --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/CreditCardDetector.java @@ -0,0 +1,52 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; + +import java.util.regex.Pattern; + +/** + * Credit-card PAN detector: 13–19 digits (spaces/dashes tolerated as grouping separators) that + * pass the Luhn checksum. The checksum keeps arbitrary numeric ids from matching. + */ +final class CreditCardDetector implements ValueDetector { + + private static final Pattern CANDIDATE = Pattern.compile("^[0-9](?:[0-9 -]*[0-9])$"); + + @Override + public DiscoveryDetector type() { + return DiscoveryDetector.CREDIT_CARD; + } + + @Override + public DataClassification classification() { + return DataClassification.PCI; + } + + @Override + public boolean matches(String value) { + var trimmed = value.trim(); + if (trimmed.length() > 30 || !CANDIDATE.matcher(trimmed).matches()) { + return false; + } + var digits = trimmed.replaceAll("[ -]", ""); + return digits.length() >= 13 && digits.length() <= 19 && luhnValid(digits); + } + + private static boolean luhnValid(String digits) { + var sum = 0; + var doubleIt = false; + for (var i = digits.length() - 1; i >= 0; i--) { + var d = digits.charAt(i) - '0'; + if (doubleIt) { + d *= 2; + if (d > 9) { + d -= 9; + } + } + sum += d; + doubleIt = !doubleIt; + } + return sum % 10 == 0; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/EmailDetector.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/EmailDetector.java new file mode 100644 index 00000000..8f29b9c0 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/EmailDetector.java @@ -0,0 +1,28 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; + +import java.util.regex.Pattern; + +/** Pragmatic RFC-lite email address detector — local part, {@code @}, dotted domain with TLD. */ +final class EmailDetector implements ValueDetector { + + private static final Pattern EMAIL = Pattern.compile( + "^[A-Za-z0-9._%+-]+@[A-Za-z0-9-]+(\\.[A-Za-z0-9-]+)*\\.[A-Za-z]{2,}$"); + + @Override + public DiscoveryDetector type() { + return DiscoveryDetector.EMAIL; + } + + @Override + public DataClassification classification() { + return DataClassification.PII; + } + + @Override + public boolean matches(String value) { + return value.length() <= 320 && EMAIL.matcher(value.trim()).matches(); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/IbanDetector.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/IbanDetector.java new file mode 100644 index 00000000..cf4feac2 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/IbanDetector.java @@ -0,0 +1,71 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; + +import java.math.BigInteger; +import java.util.Map; +import java.util.regex.Pattern; + +/** + * IBAN detector — country prefix with the official per-country length, then the ISO 13616 + * mod-97 checksum (rearrange first four chars to the end, letters → 10..35, remainder must be 1). + */ +final class IbanDetector implements ValueDetector { + + private static final Pattern CANDIDATE = + Pattern.compile("^[A-Z]{2}[0-9]{2}[A-Za-z0-9 ]{10,36}$"); + private static final BigInteger NINETY_SEVEN = BigInteger.valueOf(97); + + /** Official IBAN lengths for common registries (subset; unknown countries never match). */ + private static final Map LENGTHS = Map.ofEntries( + Map.entry("AD", 24), Map.entry("AE", 23), Map.entry("AL", 28), Map.entry("AT", 20), + Map.entry("BA", 20), Map.entry("BE", 16), Map.entry("BG", 22), Map.entry("CH", 21), + Map.entry("CY", 28), Map.entry("CZ", 24), Map.entry("DE", 22), Map.entry("DK", 18), + Map.entry("EE", 20), Map.entry("ES", 24), Map.entry("FI", 18), Map.entry("FR", 27), + Map.entry("GB", 22), Map.entry("GE", 22), Map.entry("GR", 27), Map.entry("HR", 21), + Map.entry("HU", 28), Map.entry("IE", 22), Map.entry("IL", 23), Map.entry("IS", 26), + Map.entry("IT", 27), Map.entry("LI", 21), Map.entry("LT", 20), Map.entry("LU", 20), + Map.entry("LV", 21), Map.entry("MC", 27), Map.entry("MD", 24), Map.entry("ME", 22), + Map.entry("MK", 19), Map.entry("MT", 31), Map.entry("NL", 18), Map.entry("NO", 15), + Map.entry("PL", 28), Map.entry("PT", 25), Map.entry("RO", 24), Map.entry("RS", 22), + Map.entry("SA", 24), Map.entry("SE", 24), Map.entry("SI", 19), Map.entry("SK", 24), + Map.entry("SM", 27), Map.entry("TR", 26), Map.entry("UA", 29), Map.entry("XK", 20)); + + @Override + public DiscoveryDetector type() { + return DiscoveryDetector.IBAN; + } + + @Override + public DataClassification classification() { + return DataClassification.FINANCIAL; + } + + @Override + public boolean matches(String value) { + var trimmed = value.trim().toUpperCase(); + if (trimmed.length() > 42 || !CANDIDATE.matcher(trimmed).matches()) { + return false; + } + var compact = trimmed.replace(" ", ""); + var expected = LENGTHS.get(compact.substring(0, 2)); + if (expected == null || compact.length() != expected) { + return false; + } + return mod97(compact.substring(4) + compact.substring(0, 4)) == 1; + } + + private static int mod97(String rearranged) { + var sb = new StringBuilder(rearranged.length() * 2); + for (var i = 0; i < rearranged.length(); i++) { + var c = rearranged.charAt(i); + if (Character.isDigit(c)) { + sb.append(c); + } else { + sb.append(c - 'A' + 10); + } + } + return new BigInteger(sb.toString()).mod(NINETY_SEVEN).intValue(); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/PhoneDetector.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/PhoneDetector.java new file mode 100644 index 00000000..21e7d821 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/PhoneDetector.java @@ -0,0 +1,43 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; + +import java.util.regex.Pattern; + +/** + * Phone-number detector — an optional {@code +} country prefix and 7–15 digits with common + * grouping punctuation ({@code ()-. } and spaces). Runs last in {@link ValueDetector#ORDERED} + * so card numbers and SSNs (which are also digit runs) are claimed by their stricter detectors + * first. A bare unpunctuated digit run must carry the {@code +} prefix to match — otherwise any + * numeric id column would light up as phone numbers. + */ +final class PhoneDetector implements ValueDetector { + + private static final Pattern CANDIDATE = + Pattern.compile("^\\+?[0-9(][0-9() .-]{5,19}[0-9)]$"); + private static final Pattern PUNCTUATED = Pattern.compile(".*[() .-].*"); + + @Override + public DiscoveryDetector type() { + return DiscoveryDetector.PHONE; + } + + @Override + public DataClassification classification() { + return DataClassification.PII; + } + + @Override + public boolean matches(String value) { + var trimmed = value.trim(); + if (!CANDIDATE.matcher(trimmed).matches()) { + return false; + } + if (!trimmed.startsWith("+") && !PUNCTUATED.matcher(trimmed).matches()) { + return false; + } + var digits = trimmed.replaceAll("\\D", ""); + return digits.length() >= 7 && digits.length() <= 15; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/SsnDetector.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/SsnDetector.java new file mode 100644 index 00000000..00fbce17 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/SsnDetector.java @@ -0,0 +1,42 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; + +import java.util.regex.Pattern; + +/** + * US Social Security number detector — {@code AAA-GG-SSSS} (dashes optional but consistent), + * rejecting the never-issued ranges: area 000/666/9xx, group 00, serial 0000. + */ +final class SsnDetector implements ValueDetector { + + private static final Pattern DASHED = Pattern.compile("^(\\d{3})-(\\d{2})-(\\d{4})$"); + private static final Pattern PLAIN = Pattern.compile("^(\\d{3})(\\d{2})(\\d{4})$"); + + @Override + public DiscoveryDetector type() { + return DiscoveryDetector.SSN; + } + + @Override + public DataClassification classification() { + return DataClassification.PII; + } + + @Override + public boolean matches(String value) { + var trimmed = value.trim(); + var matcher = DASHED.matcher(trimmed); + if (!matcher.matches()) { + matcher = PLAIN.matcher(trimmed); + if (!matcher.matches()) { + return false; + } + } + var area = Integer.parseInt(matcher.group(1)); + var group = Integer.parseInt(matcher.group(2)); + var serial = Integer.parseInt(matcher.group(3)); + return area != 0 && area != 666 && area < 900 && group != 0 && serial != 0; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/ValueDetector.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/ValueDetector.java new file mode 100644 index 00000000..a29c3119 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/detect/ValueDetector.java @@ -0,0 +1,26 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; + +import java.util.List; + +/** + * A local sensitive-value detector (AF-623). Implementations are pure (no Spring, no state) and + * examine one sampled string value at a time. {@link #ORDERED} fixes the first-match-wins + * evaluation order: the stricter checksum detectors run before the looser pattern ones so a card + * number is never double-counted as a phone number, and an SSN never counts as a phone. + */ +public interface ValueDetector { + + /** Evaluation order for first-match-wins classification of a single value. */ + List ORDERED = List.of(new CreditCardDetector(), new IbanDetector(), + new EmailDetector(), new SsnDetector(), new PhoneDetector()); + + DiscoveryDetector type(); + + DataClassification classification(); + + /** @param value a non-null, non-blank sampled value */ + boolean matches(String value); +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/entity/DiscoveryFindingEntity.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/entity/DiscoveryFindingEntity.java new file mode 100644 index 00000000..c1b09889 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/entity/DiscoveryFindingEntity.java @@ -0,0 +1,110 @@ +package com.bablsoft.accessflow.discovery.internal.persistence.entity; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import jakarta.persistence.Access; +import jakarta.persistence.AccessType; +import jakarta.persistence.Column; +import jakarta.persistence.Entity; +import jakarta.persistence.EnumType; +import jakarta.persistence.Enumerated; +import jakarta.persistence.Id; +import jakarta.persistence.PreUpdate; +import jakarta.persistence.Table; +import jakarta.persistence.Version; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; +import org.hibernate.annotations.JdbcType; +import org.hibernate.dialect.type.PostgreSQLEnumJdbcType; + +import java.time.Instant; +import java.util.UUID; + +@Entity +@Table(name = "discovery_finding") +@Access(AccessType.FIELD) +@Getter +@Setter +@NoArgsConstructor +public class DiscoveryFindingEntity { + + @Id + private UUID id; + + @Column(name = "organization_id", nullable = false) + private UUID organizationId; + + @Column(name = "datasource_id", nullable = false) + private UUID datasourceId; + + // null for engines without a schema concept (COALESCE'd to '' in the unique index). + @Column(name = "schema_name", columnDefinition = "text") + private String schemaName; + + @Column(name = "table_name", nullable = false, columnDefinition = "text") + private String tableName; + + @Column(name = "column_name", nullable = false, columnDefinition = "text") + private String columnName; + + @Enumerated(EnumType.STRING) + @JdbcType(PostgreSQLEnumJdbcType.class) + @Column(nullable = false, columnDefinition = "data_classification") + private DataClassification classification; + + @Enumerated(EnumType.STRING) + @JdbcType(PostgreSQLEnumJdbcType.class) + @Column(nullable = false, columnDefinition = "discovery_detector") + private DiscoveryDetector detector; + + @Column(name = "confidence", nullable = false) + private int confidence; + + // Redacted evidence only — raw sampled values never persist. + @Column(name = "sample_redacted", columnDefinition = "text") + private String sampleRedacted; + + // AI detector only. + @Column(name = "rationale", columnDefinition = "text") + private String rationale; + + @Column(name = "match_count", nullable = false) + private int matchCount; + + @Column(name = "sample_count", nullable = false) + private int sampleCount; + + @Enumerated(EnumType.STRING) + @JdbcType(PostgreSQLEnumJdbcType.class) + @Column(nullable = false, columnDefinition = "discovery_finding_status") + private DiscoveryFindingStatus status = DiscoveryFindingStatus.PENDING; + + @Column(name = "decided_by") + private UUID decidedBy; + + @Column(name = "decided_at") + private Instant decidedAt; + + @Column(name = "first_detected_at", nullable = false) + private Instant firstDetectedAt; + + @Column(name = "last_detected_at", nullable = false) + private Instant lastDetectedAt; + + @Version + @Column(nullable = false) + private long version; + + @Column(name = "created_at", nullable = false, updatable = false) + private Instant createdAt = Instant.now(); + + @Column(name = "updated_at", nullable = false) + private Instant updatedAt = Instant.now(); + + @PreUpdate + void onUpdate() { + this.updatedAt = Instant.now(); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/entity/DiscoveryScanConfigEntity.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/entity/DiscoveryScanConfigEntity.java new file mode 100644 index 00000000..a0ca567a --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/entity/DiscoveryScanConfigEntity.java @@ -0,0 +1,69 @@ +package com.bablsoft.accessflow.discovery.internal.persistence.entity; + +import jakarta.persistence.Access; +import jakarta.persistence.AccessType; +import jakarta.persistence.Column; +import jakarta.persistence.Entity; +import jakarta.persistence.Id; +import jakarta.persistence.PreUpdate; +import jakarta.persistence.Table; +import jakarta.persistence.Version; +import lombok.Getter; +import lombok.NoArgsConstructor; +import lombok.Setter; + +import java.time.Instant; +import java.util.UUID; + +@Entity +@Table(name = "discovery_scan_config") +@Access(AccessType.FIELD) +@Getter +@Setter +@NoArgsConstructor +public class DiscoveryScanConfigEntity { + + @Id + private UUID id; + + // Bare UUID references — discovery config is datasource-scoped child config; the services + // validate the datasource-in-organization invariant via DatasourceAdminService. + @Column(name = "organization_id", nullable = false) + private UUID organizationId; + + @Column(name = "datasource_id", nullable = false, unique = true) + private UUID datasourceId; + + @Column(name = "enabled", nullable = false) + private boolean enabled = false; + + @Column(name = "sample_size", nullable = false) + private int sampleSize = 100; + + @Column(name = "scan_interval_hours", nullable = false) + private int scanIntervalHours = 24; + + @Column(name = "ai_classification_enabled", nullable = false) + private boolean aiClassificationEnabled = false; + + @Column(name = "last_scan_at") + private Instant lastScanAt; + + @Column(name = "last_scan_error", columnDefinition = "text") + private String lastScanError; + + @Version + @Column(nullable = false) + private long version; + + @Column(name = "created_at", nullable = false, updatable = false) + private Instant createdAt = Instant.now(); + + @Column(name = "updated_at", nullable = false) + private Instant updatedAt = Instant.now(); + + @PreUpdate + void onUpdate() { + this.updatedAt = Instant.now(); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/repo/DiscoveryFindingRepository.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/repo/DiscoveryFindingRepository.java new file mode 100644 index 00000000..ad87b391 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/repo/DiscoveryFindingRepository.java @@ -0,0 +1,55 @@ +package com.bablsoft.accessflow.discovery.internal.persistence.repo; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryFindingEntity; +import org.springframework.data.domain.Page; +import org.springframework.data.domain.Pageable; +import org.springframework.data.jpa.repository.JpaRepository; +import org.springframework.data.jpa.repository.Query; +import org.springframework.data.repository.query.Param; + +import java.util.List; +import java.util.Optional; +import java.util.UUID; + +public interface DiscoveryFindingRepository extends JpaRepository { + + Page findAllByDatasourceIdAndOrganizationId(UUID datasourceId, + UUID organizationId, + Pageable pageable); + + Page findAllByDatasourceIdAndOrganizationIdAndStatus( + UUID datasourceId, UUID organizationId, DiscoveryFindingStatus status, + Pageable pageable); + + long countByDatasourceIdAndOrganizationIdAndStatus(UUID datasourceId, UUID organizationId, + DiscoveryFindingStatus status); + + Optional findByIdAndDatasourceIdAndOrganizationId( + UUID id, UUID datasourceId, UUID organizationId); + + /** All findings of a datasource — the scan's in-memory upsert index (bounded by the caps). */ + List findAllByDatasourceIdAndOrganizationId(UUID datasourceId, + UUID organizationId); + + /** + * Natural-key lookup matching the {@code uq_discovery_finding} unique index. JPQL derived + * queries treat {@code schemaName = null} as no-match, so the NULL case is explicit. + */ + @Query(""" + select f from DiscoveryFindingEntity f + where f.organizationId = :orgId and f.datasourceId = :dsId + and ((:schemaName is null and f.schemaName is null) or f.schemaName = :schemaName) + and f.tableName = :tableName and f.columnName = :columnName + and f.classification = :classification and f.detector = :detector""") + Optional findByNaturalKey(@Param("orgId") UUID organizationId, + @Param("dsId") UUID datasourceId, + @Param("schemaName") String schemaName, + @Param("tableName") String tableName, + @Param("columnName") String columnName, + @Param("classification") + DataClassification classification, + @Param("detector") DiscoveryDetector detector); +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/repo/DiscoveryScanConfigRepository.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/repo/DiscoveryScanConfigRepository.java new file mode 100644 index 00000000..822913ed --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/persistence/repo/DiscoveryScanConfigRepository.java @@ -0,0 +1,20 @@ +package com.bablsoft.accessflow.discovery.internal.persistence.repo; + +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryScanConfigEntity; +import org.springframework.data.jpa.repository.JpaRepository; + +import java.util.List; +import java.util.Optional; +import java.util.UUID; + +public interface DiscoveryScanConfigRepository + extends JpaRepository { + + Optional findByDatasourceIdAndOrganizationId(UUID datasourceId, + UUID organizationId); + + Optional findByDatasourceId(UUID datasourceId); + + /** Drained by {@code DiscoveryScanJob}; due-filtering happens in Java (WeeklyDigest style). */ + List findAllByEnabledTrue(); +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/scheduled/DiscoveryScanJob.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/scheduled/DiscoveryScanJob.java new file mode 100644 index 00000000..2c0d2b5d --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/scheduled/DiscoveryScanJob.java @@ -0,0 +1,69 @@ +package com.bablsoft.accessflow.discovery.internal.scheduled; + +import com.bablsoft.accessflow.discovery.api.DiscoveryScanAlreadyRunningException; +import com.bablsoft.accessflow.discovery.internal.DiscoveryScanService; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryScanConfigEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryScanConfigRepository; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import net.javacrumbs.shedlock.spring.annotation.SchedulerLock; +import org.springframework.scheduling.annotation.Scheduled; +import org.springframework.stereotype.Component; + +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; + +/** + * Drives the scheduled sensitive-data discovery scans (AF-623). Wakes every + * {@code accessflow.discovery.scan-poll-interval} (default 15 minutes) and runs a scan for every + * enabled {@code discovery_scan_config} whose {@code last_scan_at} is older than its own + * {@code scan_interval_hours} (or was never scanned). + * + *

The {@link SchedulerLock} guarantees only one node in a cluster runs per tick (Redis-backed + * {@code LockProvider} in the {@code scheduling} module). Per-datasource failures are logged and + * skipped so one bad datasource never aborts the batch. + */ +@Component +@RequiredArgsConstructor +@Slf4j +public class DiscoveryScanJob { + + private final DiscoveryScanConfigRepository configRepository; + private final DiscoveryScanService scanService; + private final Clock clock; + + @Scheduled(fixedDelayString = "${accessflow.discovery.scan-poll-interval:PT15M}") + @SchedulerLock(name = "discoveryScanJob", lockAtMostFor = "PT30M", lockAtLeastFor = "PT1M") + public void run() { + var now = clock.instant(); + var due = configRepository.findAllByEnabledTrue().stream() + .filter(config -> isDue(config, now)) + .toList(); + if (due.isEmpty()) { + log.debug("No discovery scans due"); + return; + } + var scanned = 0; + for (var config : due) { + try { + scanService.scan(config.getDatasourceId(), config.getOrganizationId(), null); + scanned++; + } catch (DiscoveryScanAlreadyRunningException ex) { + log.info("Skipping discovery scan for datasource {} — already running", + config.getDatasourceId()); + } catch (RuntimeException ex) { + log.error("Discovery scan failed for datasource {}", config.getDatasourceId(), ex); + } + } + log.info("Completed {} discovery scans (due {})", scanned, due.size()); + } + + private boolean isDue(DiscoveryScanConfigEntity config, Instant now) { + if (config.getLastScanAt() == null) { + return true; + } + return !config.getLastScanAt().plus(Duration.ofHours(config.getScanIntervalHours())) + .isAfter(now); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/BulkDiscoveryDecisionRequest.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/BulkDiscoveryDecisionRequest.java new file mode 100644 index 00000000..e1b2e757 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/BulkDiscoveryDecisionRequest.java @@ -0,0 +1,17 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.discovery.api.DiscoveryDecision; +import jakarta.validation.constraints.NotEmpty; +import jakarta.validation.constraints.NotNull; +import jakarta.validation.constraints.Size; + +import java.util.List; +import java.util.UUID; + +public record BulkDiscoveryDecisionRequest( + @NotEmpty(message = "{validation.discovery.finding_ids.required}") + @Size(max = 100, message = "{validation.discovery.finding_ids.max}") + List findingIds, + @NotNull(message = "{validation.discovery.decision.required}") + DiscoveryDecision decision) { +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/BulkDiscoveryDecisionResponse.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/BulkDiscoveryDecisionResponse.java new file mode 100644 index 00000000..138de28e --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/BulkDiscoveryDecisionResponse.java @@ -0,0 +1,22 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingService.BulkDecisionOutcome; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.api.DiscoveryRowStatus; + +import java.util.List; +import java.util.UUID; + +public record BulkDiscoveryDecisionResponse(List results) { + + public record Row(UUID findingId, DiscoveryRowStatus status, + DiscoveryFindingStatus newStatus) { + } + + public static BulkDiscoveryDecisionResponse from(BulkDecisionOutcome outcome) { + var rows = outcome.results().stream() + .map(r -> new Row(r.findingId(), r.status(), r.newStatus())) + .toList(); + return new BulkDiscoveryDecisionResponse(rows); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryAuditWriter.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryAuditWriter.java new file mode 100644 index 00000000..a58fc0b0 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryAuditWriter.java @@ -0,0 +1,40 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.audit.api.AuditAction; +import com.bablsoft.accessflow.audit.api.AuditEntry; +import com.bablsoft.accessflow.audit.api.AuditLogService; +import com.bablsoft.accessflow.audit.api.AuditResourceType; +import com.bablsoft.accessflow.audit.api.RequestAuditContext; +import com.bablsoft.accessflow.security.api.JwtClaims; +import lombok.RequiredArgsConstructor; +import lombok.extern.slf4j.Slf4j; +import org.springframework.stereotype.Component; + +import java.util.Map; +import java.util.UUID; + +/** Synchronous audit writes for HTTP-driven discovery decisions (AF-623). */ +@Component +@RequiredArgsConstructor +@Slf4j +class DiscoveryAuditWriter { + + private final AuditLogService auditLogService; + + void record(AuditAction action, UUID findingId, JwtClaims caller, + Map metadata, RequestAuditContext auditContext) { + try { + auditLogService.record(new AuditEntry( + action, + AuditResourceType.DISCOVERY_FINDING, + findingId, + caller.organizationId(), + caller.userId(), + metadata, + auditContext.ipAddress(), + auditContext.userAgent())); + } catch (RuntimeException ex) { + log.error("Audit write failed for {} on discovery finding {}", action, findingId, ex); + } + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryConfigResponse.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryConfigResponse.java new file mode 100644 index 00000000..4ad2f556 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryConfigResponse.java @@ -0,0 +1,22 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.discovery.api.DiscoveryScanConfigView; + +import java.time.Instant; +import java.util.UUID; + +public record DiscoveryConfigResponse( + UUID datasourceId, + boolean enabled, + int sampleSize, + int scanIntervalHours, + boolean aiClassificationEnabled, + Instant lastScanAt, + String lastScanError) { + + public static DiscoveryConfigResponse from(DiscoveryScanConfigView view) { + return new DiscoveryConfigResponse(view.datasourceId(), view.enabled(), view.sampleSize(), + view.scanIntervalHours(), view.aiClassificationEnabled(), view.lastScanAt(), + view.lastScanError()); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryController.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryController.java new file mode 100644 index 00000000..11662028 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryController.java @@ -0,0 +1,140 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.audit.api.AuditAction; +import com.bablsoft.accessflow.audit.api.RequestAuditContext; +import com.bablsoft.accessflow.discovery.api.DiscoveryConfigService; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingService; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingService.BulkDecisionOutcome; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.api.DiscoveryRowStatus; +import com.bablsoft.accessflow.discovery.api.DiscoveryScanTriggerService; +import com.bablsoft.accessflow.security.api.JwtClaims; +import io.swagger.v3.oas.annotations.Operation; +import io.swagger.v3.oas.annotations.responses.ApiResponse; +import io.swagger.v3.oas.annotations.tags.Tag; +import jakarta.validation.Valid; +import lombok.RequiredArgsConstructor; +import org.springframework.data.domain.Pageable; +import org.springframework.http.HttpStatus; +import org.springframework.http.ResponseEntity; +import org.springframework.security.access.prepost.PreAuthorize; +import org.springframework.security.core.Authentication; +import org.springframework.web.bind.annotation.GetMapping; +import org.springframework.web.bind.annotation.PathVariable; +import org.springframework.web.bind.annotation.PostMapping; +import org.springframework.web.bind.annotation.PutMapping; +import org.springframework.web.bind.annotation.RequestBody; +import org.springframework.web.bind.annotation.RequestMapping; +import org.springframework.web.bind.annotation.RequestParam; +import org.springframework.web.bind.annotation.RestController; + +import java.util.HashMap; +import java.util.UUID; + +@RestController +@RequestMapping("/api/v1/datasources/{datasourceId}/discovery") +@Tag(name = "Sensitive-Data Discovery", + description = "Automated sensitive-data discovery scans and the proposed-classification worklist (AF-623)") +@RequiredArgsConstructor +@PreAuthorize("hasAuthority('PERM_DATA_CLASSIFICATION_MANAGE')") +class DiscoveryController { + + private final DiscoveryConfigService configService; + private final DiscoveryFindingService findingService; + private final DiscoveryScanTriggerService scanTriggerService; + private final DiscoveryAuditWriter auditWriter; + + @GetMapping("/config") + @Operation(summary = "Get the datasource's discovery settings (defaults when never configured)") + @ApiResponse(responseCode = "200", description = "Current or default settings") + @ApiResponse(responseCode = "404", description = "Datasource not found") + DiscoveryConfigResponse getConfig(@PathVariable UUID datasourceId, + Authentication authentication) { + var caller = currentClaims(authentication); + return DiscoveryConfigResponse.from(configService.get(datasourceId, + caller.organizationId())); + } + + @PutMapping("/config") + @Operation(summary = "Create or update the datasource's discovery settings") + @ApiResponse(responseCode = "200", description = "Updated settings") + @ApiResponse(responseCode = "400", description = "Validation error") + @ApiResponse(responseCode = "404", description = "Datasource not found") + DiscoveryConfigResponse updateConfig(@PathVariable UUID datasourceId, + @Valid @RequestBody UpdateDiscoveryConfigRequest body, + Authentication authentication) { + var caller = currentClaims(authentication); + return DiscoveryConfigResponse.from(configService.upsert(datasourceId, + caller.organizationId(), body.toCommand())); + } + + @GetMapping("/findings") + @Operation(summary = "Page the discovery-findings worklist, newest detection first") + @ApiResponse(responseCode = "200", description = "Page of findings") + DiscoveryFindingPageResponse listFindings(@PathVariable UUID datasourceId, + @RequestParam(required = false) + DiscoveryFindingStatus status, + Authentication authentication, Pageable pageable) { + var caller = currentClaims(authentication); + var page = findingService.find(datasourceId, caller.organizationId(), status, + SpringPageableAdapter.toPageRequest(pageable)); + return DiscoveryFindingPageResponse.from(page); + } + + @PostMapping("/findings/bulk-decision") + @Operation(summary = "Confirm or dismiss a batch of findings; rows are decided independently") + @ApiResponse(responseCode = "200", description = "Per-row outcomes (partial success)") + @ApiResponse(responseCode = "400", description = "Validation error (empty/oversized id list)") + BulkDiscoveryDecisionResponse bulkDecision(@PathVariable UUID datasourceId, + @Valid @RequestBody + BulkDiscoveryDecisionRequest body, + Authentication authentication, + RequestAuditContext auditContext) { + var caller = currentClaims(authentication); + var outcome = findingService.decide(datasourceId, caller.organizationId(), + caller.userId(), body.findingIds(), body.decision()); + recordDecisionAudits(datasourceId, outcome, caller, auditContext); + return BulkDiscoveryDecisionResponse.from(outcome); + } + + @PostMapping("/scan") + @Operation(summary = "Trigger an immediate discovery scan (runs asynchronously)") + @ApiResponse(responseCode = "202", description = "Scan accepted") + @ApiResponse(responseCode = "404", description = "Datasource not found") + @ApiResponse(responseCode = "409", description = "A scan for this datasource is already running") + ResponseEntity triggerScan(@PathVariable UUID datasourceId, + Authentication authentication) { + var caller = currentClaims(authentication); + scanTriggerService.requestScan(datasourceId, caller.organizationId(), caller.userId()); + return ResponseEntity.status(HttpStatus.ACCEPTED).build(); + } + + private void recordDecisionAudits(UUID datasourceId, BulkDecisionOutcome outcome, + JwtClaims caller, RequestAuditContext auditContext) { + for (var row : outcome.results()) { + var decided = row.status() == DiscoveryRowStatus.SUCCESS + || row.status() == DiscoveryRowStatus.TAG_CONFLICT; + if (!decided || row.finding() == null) { + continue; + } + var action = row.newStatus() == DiscoveryFindingStatus.CONFIRMED + ? AuditAction.DISCOVERY_FINDING_CONFIRMED + : AuditAction.DISCOVERY_FINDING_DISMISSED; + var finding = row.finding(); + var metadata = new HashMap(); + metadata.put("datasourceId", datasourceId.toString()); + metadata.put("tableName", finding.schemaName() == null ? finding.tableName() + : finding.schemaName() + "." + finding.tableName()); + metadata.put("columnName", finding.columnName()); + metadata.put("classification", finding.classification().name()); + metadata.put("detector", finding.detector().name()); + metadata.put("confidence", finding.confidence()); + metadata.put("tagConflict", row.status() == DiscoveryRowStatus.TAG_CONFLICT); + auditWriter.record(action, row.findingId(), caller, metadata, auditContext); + } + } + + private static JwtClaims currentClaims(Authentication authentication) { + return (JwtClaims) authentication.getPrincipal(); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryExceptionHandler.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryExceptionHandler.java new file mode 100644 index 00000000..25bf9d2b --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryExceptionHandler.java @@ -0,0 +1,34 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.discovery.api.DiscoveryScanAlreadyRunningException; +import lombok.RequiredArgsConstructor; +import org.springframework.context.MessageSource; +import org.springframework.context.i18n.LocaleContextHolder; +import org.springframework.core.Ordered; +import org.springframework.core.annotation.Order; +import org.springframework.http.HttpStatus; +import org.springframework.http.ProblemDetail; +import org.springframework.web.bind.annotation.ExceptionHandler; +import org.springframework.web.bind.annotation.RestControllerAdvice; + +import java.time.Instant; + +// Higher precedence than the security module's GlobalExceptionHandler, whose Exception.class +// catch-all would otherwise win the resolution race for these specific discovery exceptions. +@RestControllerAdvice +@Order(Ordered.HIGHEST_PRECEDENCE) +@RequiredArgsConstructor +class DiscoveryExceptionHandler { + + private final MessageSource messageSource; + + @ExceptionHandler(DiscoveryScanAlreadyRunningException.class) + ProblemDetail handleScanAlreadyRunning(DiscoveryScanAlreadyRunningException ex) { + var pd = ProblemDetail.forStatusAndDetail(HttpStatus.CONFLICT, + messageSource.getMessage("error.discovery_scan_already_running", null, + LocaleContextHolder.getLocale())); + pd.setProperty("error", "DISCOVERY_SCAN_ALREADY_RUNNING"); + pd.setProperty("timestamp", Instant.now().toString()); + return pd; + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryFindingPageResponse.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryFindingPageResponse.java new file mode 100644 index 00000000..b47779d1 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryFindingPageResponse.java @@ -0,0 +1,20 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.core.api.PageResponse; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingView; + +import java.util.List; + +public record DiscoveryFindingPageResponse( + List content, + int page, + int size, + long totalElements, + int totalPages) { + + public static DiscoveryFindingPageResponse from(PageResponse page) { + return new DiscoveryFindingPageResponse( + page.content().stream().map(DiscoveryFindingResponse::from).toList(), + page.page(), page.size(), page.totalElements(), page.totalPages()); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryFindingResponse.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryFindingResponse.java new file mode 100644 index 00000000..db15d36a --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryFindingResponse.java @@ -0,0 +1,36 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingView; + +import java.time.Instant; +import java.util.UUID; + +public record DiscoveryFindingResponse( + UUID id, + String schemaName, + String tableName, + String columnName, + DataClassification classification, + DiscoveryDetector detector, + int confidence, + String sampleRedacted, + String rationale, + int matchCount, + int sampleCount, + DiscoveryFindingStatus status, + Instant firstDetectedAt, + Instant lastDetectedAt, + UUID decidedBy, + Instant decidedAt) { + + public static DiscoveryFindingResponse from(DiscoveryFindingView view) { + return new DiscoveryFindingResponse(view.id(), view.schemaName(), view.tableName(), + view.columnName(), view.classification(), view.detector(), view.confidence(), + view.sampleRedacted(), view.rationale(), view.matchCount(), view.sampleCount(), + view.status(), view.firstDetectedAt(), view.lastDetectedAt(), view.decidedBy(), + view.decidedAt()); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/SpringPageableAdapter.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/SpringPageableAdapter.java new file mode 100644 index 00000000..39f8e4c0 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/SpringPageableAdapter.java @@ -0,0 +1,32 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.core.api.PageRequest; +import com.bablsoft.accessflow.core.api.SortOrder; +import org.springframework.data.domain.Pageable; +import org.springframework.data.domain.Sort; + +/** + * Translates Spring MVC's bound {@link Pageable} into the library-agnostic {@link PageRequest} + * that module {@code api/} services accept. + */ +final class SpringPageableAdapter { + + private SpringPageableAdapter() { + } + + static PageRequest toPageRequest(Pageable pageable) { + if (pageable == null || !pageable.isPaged()) { + return PageRequest.of(0, Integer.MAX_VALUE); + } + var sort = pageable.getSort().stream() + .map(SpringPageableAdapter::toSortOrder) + .toList(); + return new PageRequest(pageable.getPageNumber(), pageable.getPageSize(), sort); + } + + private static SortOrder toSortOrder(Sort.Order order) { + var direction = order.getDirection() == Sort.Direction.ASC + ? SortOrder.Direction.ASC : SortOrder.Direction.DESC; + return new SortOrder(order.getProperty(), direction); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/UpdateDiscoveryConfigRequest.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/UpdateDiscoveryConfigRequest.java new file mode 100644 index 00000000..d2a92c12 --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/internal/web/UpdateDiscoveryConfigRequest.java @@ -0,0 +1,22 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.discovery.api.UpsertDiscoveryConfigCommand; +import jakarta.validation.constraints.Max; +import jakarta.validation.constraints.Min; + +/** Partial update — {@code null} fields keep their current value. */ +public record UpdateDiscoveryConfigRequest( + Boolean enabled, + @Min(value = 10, message = "{validation.discovery.sample_size.range}") + @Max(value = 1000, message = "{validation.discovery.sample_size.range}") + Integer sampleSize, + @Min(value = 1, message = "{validation.discovery.scan_interval.range}") + @Max(value = 720, message = "{validation.discovery.scan_interval.range}") + Integer scanIntervalHours, + Boolean aiClassificationEnabled) { + + public UpsertDiscoveryConfigCommand toCommand() { + return new UpsertDiscoveryConfigCommand(enabled, sampleSize, scanIntervalHours, + aiClassificationEnabled); + } +} diff --git a/backend/src/main/java/com/bablsoft/accessflow/discovery/package-info.java b/backend/src/main/java/com/bablsoft/accessflow/discovery/package-info.java new file mode 100644 index 00000000..4ee36dab --- /dev/null +++ b/backend/src/main/java/com/bablsoft/accessflow/discovery/package-info.java @@ -0,0 +1,4 @@ +@ApplicationModule(displayName = "Discovery") +package com.bablsoft.accessflow.discovery; + +import org.springframework.modulith.ApplicationModule; diff --git a/backend/src/main/resources/db/migration/V129__create_discovery.sql b/backend/src/main/resources/db/migration/V129__create_discovery.sql new file mode 100644 index 00000000..b0a4cc3c --- /dev/null +++ b/backend/src/main/resources/db/migration/V129__create_discovery.sql @@ -0,0 +1,65 @@ +-- AF-623: automated sensitive-data discovery & classification scanning. A scheduled scanner +-- samples column data through the existing per-engine sampling path, detects sensitive data with +-- regex + checksum detectors (and optionally the org's bound AI analyzer), and proposes +-- data-classification tags (AF-447) that an admin confirms or dismisses. + +CREATE TYPE discovery_detector AS ENUM ('EMAIL', 'CREDIT_CARD', 'SSN', 'IBAN', 'PHONE', 'AI'); + +CREATE TYPE discovery_finding_status AS ENUM ('PENDING', 'CONFIRMED', 'DISMISSED'); + +-- Per-datasource opt-in + cadence. The scheduled job drains this table (enabled rows whose +-- last_scan_at is older than scan_interval_hours), so no cross-org datasource enumeration is +-- needed. One row per datasource; absence = discovery disabled with defaults. +CREATE TABLE discovery_scan_config ( + id UUID PRIMARY KEY, + organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE, + datasource_id UUID NOT NULL UNIQUE REFERENCES datasources(id) ON DELETE CASCADE, + enabled BOOLEAN NOT NULL DEFAULT FALSE, + sample_size INT NOT NULL DEFAULT 100, + scan_interval_hours INT NOT NULL DEFAULT 24, + ai_classification_enabled BOOLEAN NOT NULL DEFAULT FALSE, + last_scan_at TIMESTAMPTZ, + last_scan_error TEXT, + version BIGINT NOT NULL DEFAULT 0, + created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +CREATE INDEX idx_dsc_org ON discovery_scan_config (organization_id); + +-- Proposed classification worklist. One row per (column, classification, detector); rescans +-- refresh PENDING rows in place and never touch CONFIRMED/DISMISSED rows (a dismissal is a +-- permanent suppression for future scans). sample_redacted only ever holds a redacted value — +-- raw sampled data never persists. +CREATE TABLE discovery_finding ( + id UUID PRIMARY KEY, + organization_id UUID NOT NULL REFERENCES organizations(id) ON DELETE CASCADE, + datasource_id UUID NOT NULL REFERENCES datasources(id) ON DELETE CASCADE, + schema_name TEXT, + table_name TEXT NOT NULL, + column_name TEXT NOT NULL, + classification data_classification NOT NULL, + detector discovery_detector NOT NULL, + confidence INT NOT NULL, + sample_redacted TEXT, + rationale TEXT, + match_count INT NOT NULL DEFAULT 0, + sample_count INT NOT NULL DEFAULT 0, + status discovery_finding_status NOT NULL DEFAULT 'PENDING', + decided_by UUID REFERENCES users(id) ON DELETE SET NULL, + decided_at TIMESTAMPTZ, + first_detected_at TIMESTAMPTZ NOT NULL, + last_detected_at TIMESTAMPTZ NOT NULL, + version BIGINT NOT NULL DEFAULT 0, + created_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP, + updated_at TIMESTAMPTZ NOT NULL DEFAULT CURRENT_TIMESTAMP +); + +-- Natural key for rescan upserts. schema_name NULL (engines without schemas) would make a plain +-- UNIQUE treat two rows as distinct; COALESCE collapses NULL to '' (same pattern as V90). +CREATE UNIQUE INDEX uq_discovery_finding ON discovery_finding + (organization_id, datasource_id, COALESCE(schema_name, ''), table_name, column_name, + classification, detector); + +-- Worklist scan (status filter) per datasource. +CREATE INDEX idx_df_ds_status ON discovery_finding (datasource_id, status); diff --git a/backend/src/main/resources/i18n/messages.properties b/backend/src/main/resources/i18n/messages.properties index 71c70d9d..d26e90e9 100644 --- a/backend/src/main/resources/i18n/messages.properties +++ b/backend/src/main/resources/i18n/messages.properties @@ -952,3 +952,11 @@ notification.email.subject.query_escalated=[AccessFlow] Query escalated for revi notification.email.query_escalated.title=Query escalated for review notification.email.query_escalated.heading=Query escalated for your review notification.email.query_escalated.intro=A routing policy escalated this query — it requires additional approvals before it can run. + +# AF-623: sensitive-data discovery +error.discovery_scan_already_running=A discovery scan for this datasource is already running +validation.discovery.sample_size.range=Sample size must be between 10 and 1000 +validation.discovery.scan_interval.range=Scan interval must be between 1 and 720 hours +validation.discovery.finding_ids.required=At least one finding id is required +validation.discovery.finding_ids.max=At most 100 findings can be decided per request +validation.discovery.decision.required=Decision is required diff --git a/backend/src/main/resources/i18n/messages_de.properties b/backend/src/main/resources/i18n/messages_de.properties index 53debd7a..287cab49 100644 --- a/backend/src/main/resources/i18n/messages_de.properties +++ b/backend/src/main/resources/i18n/messages_de.properties @@ -931,3 +931,11 @@ notification.email.subject.query_escalated=[AccessFlow] Abfrage zur Prüfung esk notification.email.query_escalated.title=Abfrage zur Prüfung eskaliert notification.email.query_escalated.heading=Abfrage zu Ihrer Prüfung eskaliert notification.email.query_escalated.intro=Eine Routing-Richtlinie hat diese Abfrage eskaliert — sie erfordert zusätzliche Genehmigungen, bevor sie ausgeführt werden kann. + +# AF-623: sensitive-data discovery +error.discovery_scan_already_running=Für diese Datenquelle läuft bereits ein Discovery-Scan +validation.discovery.sample_size.range=Die Stichprobengröße muss zwischen 10 und 1000 liegen +validation.discovery.scan_interval.range=Das Scan-Intervall muss zwischen 1 und 720 Stunden liegen +validation.discovery.finding_ids.required=Mindestens eine Finding-ID ist erforderlich +validation.discovery.finding_ids.max=Pro Anfrage können höchstens 100 Findings entschieden werden +validation.discovery.decision.required=Eine Entscheidung ist erforderlich diff --git a/backend/src/main/resources/i18n/messages_es.properties b/backend/src/main/resources/i18n/messages_es.properties index 4bc4fb86..1eb86eff 100644 --- a/backend/src/main/resources/i18n/messages_es.properties +++ b/backend/src/main/resources/i18n/messages_es.properties @@ -931,3 +931,11 @@ notification.email.subject.query_escalated=[AccessFlow] Consulta escalada para r notification.email.query_escalated.title=Consulta escalada para revisión notification.email.query_escalated.heading=Consulta escalada para su revisión notification.email.query_escalated.intro=Una política de enrutamiento escaló esta consulta — requiere aprobaciones adicionales antes de poder ejecutarse. + +# AF-623: sensitive-data discovery +error.discovery_scan_already_running=Ya se está ejecutando un escaneo de descubrimiento para esta fuente de datos +validation.discovery.sample_size.range=El tamaño de la muestra debe estar entre 10 y 1000 +validation.discovery.scan_interval.range=El intervalo de escaneo debe estar entre 1 y 720 horas +validation.discovery.finding_ids.required=Se requiere al menos un id de hallazgo +validation.discovery.finding_ids.max=Se pueden decidir como máximo 100 hallazgos por solicitud +validation.discovery.decision.required=La decisión es obligatoria diff --git a/backend/src/main/resources/i18n/messages_fr.properties b/backend/src/main/resources/i18n/messages_fr.properties index 29f86512..241d8c7e 100644 --- a/backend/src/main/resources/i18n/messages_fr.properties +++ b/backend/src/main/resources/i18n/messages_fr.properties @@ -934,3 +934,11 @@ notification.email.subject.query_escalated=[AccessFlow] Requête escaladée pour notification.email.query_escalated.title=Requête escaladée pour relecture notification.email.query_escalated.heading=Requête escaladée pour votre relecture notification.email.query_escalated.intro=Une politique de routage a escaladé cette requête — elle nécessite des approbations supplémentaires avant de pouvoir être exécutée. + +# AF-623: sensitive-data discovery +error.discovery_scan_already_running=Une analyse de découverte est déjà en cours pour cette source de données +validation.discovery.sample_size.range=La taille de l'échantillon doit être comprise entre 10 et 1000 +validation.discovery.scan_interval.range=L'intervalle d'analyse doit être compris entre 1 et 720 heures +validation.discovery.finding_ids.required=Au moins un identifiant de découverte est requis +validation.discovery.finding_ids.max=Au plus 100 découvertes peuvent être décidées par requête +validation.discovery.decision.required=La décision est obligatoire diff --git a/backend/src/main/resources/i18n/messages_hy.properties b/backend/src/main/resources/i18n/messages_hy.properties index 18be8df4..f04910cd 100644 --- a/backend/src/main/resources/i18n/messages_hy.properties +++ b/backend/src/main/resources/i18n/messages_hy.properties @@ -931,3 +931,11 @@ notification.email.subject.query_escalated=[AccessFlow] {0}-ում հարցու notification.email.query_escalated.title=Հարցումը էսկալացվել է վերանայման notification.email.query_escalated.heading=Հարցումը էսկալացվել է Ձեր վերանայման notification.email.query_escalated.intro=Երթուղավորման քաղաքականությունը էսկալացրել է այս հարցումը — կատարումից առաջ պահանջվում են լրացուցիչ հաստատումներ։ + +# AF-623: sensitive-data discovery +error.discovery_scan_already_running=Այս տվյալների աղբյուրի համար հայտնաբերման սկանավորումն արդեն ընթացքի մեջ է +validation.discovery.sample_size.range=Նմուշի չափը պետք է լինի 10-ից 1000 միջակայքում +validation.discovery.scan_interval.range=Սկանավորման միջակայքը պետք է լինի 1-ից 720 ժամ +validation.discovery.finding_ids.required=Անհրաժեշտ է առնվազն մեկ հայտնաբերման նույնացուցիչ +validation.discovery.finding_ids.max=Մեկ հարցումով կարելի է որոշել առավելագույնը 100 հայտնաբերում +validation.discovery.decision.required=Որոշումը պարտադիր է diff --git a/backend/src/main/resources/i18n/messages_ru.properties b/backend/src/main/resources/i18n/messages_ru.properties index b94c207d..63b5be9f 100644 --- a/backend/src/main/resources/i18n/messages_ru.properties +++ b/backend/src/main/resources/i18n/messages_ru.properties @@ -931,3 +931,11 @@ notification.email.subject.query_escalated=[AccessFlow] Запрос эскал notification.email.query_escalated.title=Запрос эскалирован на проверку notification.email.query_escalated.heading=Запрос эскалирован на вашу проверку notification.email.query_escalated.intro=Политика маршрутизации эскалировала этот запрос — перед выполнением требуются дополнительные согласования. + +# AF-623: sensitive-data discovery +error.discovery_scan_already_running=Сканирование обнаружения для этого источника данных уже выполняется +validation.discovery.sample_size.range=Размер выборки должен быть от 10 до 1000 +validation.discovery.scan_interval.range=Интервал сканирования должен быть от 1 до 720 часов +validation.discovery.finding_ids.required=Требуется хотя бы один идентификатор находки +validation.discovery.finding_ids.max=За один запрос можно решить не более 100 находок +validation.discovery.decision.required=Решение обязательно diff --git a/backend/src/main/resources/i18n/messages_zh_CN.properties b/backend/src/main/resources/i18n/messages_zh_CN.properties index aee423b4..d3429db2 100644 --- a/backend/src/main/resources/i18n/messages_zh_CN.properties +++ b/backend/src/main/resources/i18n/messages_zh_CN.properties @@ -931,3 +931,11 @@ notification.email.subject.query_escalated=[AccessFlow] {0} 上的查询已升 notification.email.query_escalated.title=查询已升级审核 notification.email.query_escalated.heading=查询已升级至您审核 notification.email.query_escalated.intro=路由策略已将此查询升级 — 执行前需要额外的审批。 + +# AF-623: sensitive-data discovery +error.discovery_scan_already_running=该数据源的发现扫描已在进行中 +validation.discovery.sample_size.range=采样大小必须介于 10 到 1000 之间 +validation.discovery.scan_interval.range=扫描间隔必须介于 1 到 720 小时之间 +validation.discovery.finding_ids.required=至少需要一个发现项 ID +validation.discovery.finding_ids.max=每次请求最多可决定 100 个发现项 +validation.discovery.decision.required=必须提供决定 diff --git a/backend/src/test/java/com/bablsoft/accessflow/ai/internal/DefaultDataDiscoveryAiServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/ai/internal/DefaultDataDiscoveryAiServiceTest.java new file mode 100644 index 00000000..7fbfbc01 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/ai/internal/DefaultDataDiscoveryAiServiceTest.java @@ -0,0 +1,135 @@ +package com.bablsoft.accessflow.ai.internal; + +import com.bablsoft.accessflow.ai.api.DataDiscoveryAiService.DiscoveryColumnContext; +import com.bablsoft.accessflow.ai.api.DataDiscoveryAiService.DiscoveryTableContext; +import com.bablsoft.accessflow.core.api.DataClassification; +import org.junit.jupiter.api.Test; +import org.mockito.ArgumentCaptor; +import org.springframework.beans.factory.ObjectProvider; +import tools.jackson.databind.ObjectMapper; + +import java.util.List; +import java.util.Optional; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.anyString; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.verifyNoInteractions; +import static org.mockito.Mockito.when; + +@SuppressWarnings("unchecked") +class DefaultDataDiscoveryAiServiceTest { + + private final AiAnalyzerStrategyHolder holder = mock(AiAnalyzerStrategyHolder.class); + private final ObjectProvider holderProvider = mock(ObjectProvider.class); + private final DefaultDataDiscoveryAiService service = + new DefaultDataDiscoveryAiService(holderProvider, new ObjectMapper()); + + private final UUID orgId = UUID.randomUUID(); + + DefaultDataDiscoveryAiServiceTest() { + when(holderProvider.getObject()).thenReturn(holder); + } + + private static DiscoveryTableContext context() { + return new DiscoveryTableContext("public.users", List.of( + new DiscoveryColumnContext("national_id", "varchar", List.of("*** ** ****")), + new DiscoveryColumnContext("name", "varchar", List.of("xxxxx")))); + } + + @Test + void returnsEmptyForNullOrgOrEmptyContext() { + assertThat(service.classifyColumns(null, context())).isEmpty(); + assertThat(service.classifyColumns(orgId, null)).isEmpty(); + assertThat(service.classifyColumns(orgId, + new DiscoveryTableContext("t", List.of()))).isEmpty(); + verifyNoInteractions(holder); + } + + @Test + void returnsEmptyWhenHolderYieldsNothing() { + when(holder.classifyDiscoveryColumns(eq(orgId), anyString())).thenReturn(Optional.empty()); + + assertThat(service.classifyColumns(orgId, context())).isEmpty(); + } + + @Test + void parsesWellFormedResponse() { + when(holder.classifyDiscoveryColumns(eq(orgId), anyString())).thenReturn(Optional.of(""" + {"columns":[{"column":"national_id","classification":"PII","confidence":85,\ + "rationale":"Looks like a national identifier."}]}""")); + + var suggestions = service.classifyColumns(orgId, context()); + + assertThat(suggestions).hasSize(1); + var suggestion = suggestions.getFirst(); + assertThat(suggestion.columnName()).isEqualTo("national_id"); + assertThat(suggestion.classification()).isEqualTo(DataClassification.PII); + assertThat(suggestion.confidence()).isEqualTo(85); + assertThat(suggestion.rationale()).isEqualTo("Looks like a national identifier."); + } + + @Test + void toleratesMarkdownCodeFences() { + when(holder.classifyDiscoveryColumns(eq(orgId), anyString())).thenReturn(Optional.of(""" + ```json + {"columns":[{"column":"national_id","classification":"SENSITIVE","confidence":60}]} + ```""")); + + var suggestions = service.classifyColumns(orgId, context()); + + assertThat(suggestions).hasSize(1); + assertThat(suggestions.getFirst().classification()).isEqualTo(DataClassification.SENSITIVE); + assertThat(suggestions.getFirst().rationale()).isNull(); + } + + @Test + void dropsUnknownColumnsAndClassificationsAndClampsConfidence() { + when(holder.classifyDiscoveryColumns(eq(orgId), anyString())).thenReturn(Optional.of(""" + {"columns":[ + {"column":"not_a_column","classification":"PII","confidence":90}, + {"column":"name","classification":"TOP_SECRET","confidence":90}, + {"column":"name","classification":"pii","confidence":250}, + {"column":"","classification":"PII","confidence":10}, + {"column":"national_id","classification":"PHI","confidence":"high"} + ]}""")); + + var suggestions = service.classifyColumns(orgId, context()); + + assertThat(suggestions).hasSize(2); + assertThat(suggestions.get(0).columnName()).isEqualTo("name"); + assertThat(suggestions.get(0).classification()).isEqualTo(DataClassification.PII); + assertThat(suggestions.get(0).confidence()).isEqualTo(100); + assertThat(suggestions.get(1).columnName()).isEqualTo("national_id"); + assertThat(suggestions.get(1).confidence()).isZero(); + } + + @Test + void returnsEmptyOnMalformedJsonOrMissingColumnsArray() { + when(holder.classifyDiscoveryColumns(eq(orgId), anyString())) + .thenReturn(Optional.of("this is not json")); + assertThat(service.classifyColumns(orgId, context())).isEmpty(); + + when(holder.classifyDiscoveryColumns(eq(orgId), anyString())) + .thenReturn(Optional.of("{\"columns\":{\"column\":\"name\"}}")); + assertThat(service.classifyColumns(orgId, context())).isEmpty(); + } + + @Test + void promptCarriesTableColumnsTypesAndRedactedSamples() { + when(holder.classifyDiscoveryColumns(eq(orgId), anyString())).thenReturn(Optional.empty()); + + service.classifyColumns(orgId, context()); + + var prompt = ArgumentCaptor.forClass(String.class); + verify(holder).classifyDiscoveryColumns(eq(orgId), prompt.capture()); + assertThat(prompt.getValue()) + .contains("Table: public.users") + .contains("national_id (varchar) samples: *** ** ****") + .contains("- name (varchar) samples: xxxxx"); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryConfigServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryConfigServiceTest.java new file mode 100644 index 00000000..3329f520 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryConfigServiceTest.java @@ -0,0 +1,130 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.core.api.DatasourceAdminService; +import com.bablsoft.accessflow.core.api.DatasourceNotFoundException; +import com.bablsoft.accessflow.discovery.api.UpsertDiscoveryConfigCommand; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryScanConfigEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryScanConfigRepository; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.time.Instant; +import java.util.Optional; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class DefaultDiscoveryConfigServiceTest { + + @Mock + private DiscoveryScanConfigRepository configRepository; + @Mock + private DatasourceAdminService datasourceAdminService; + + private final UUID dsId = UUID.randomUUID(); + private final UUID orgId = UUID.randomUUID(); + + private DefaultDiscoveryConfigService service() { + return new DefaultDiscoveryConfigService(configRepository, datasourceAdminService); + } + + private DiscoveryScanConfigEntity existingConfig() { + var config = new DiscoveryScanConfigEntity(); + config.setId(UUID.randomUUID()); + config.setOrganizationId(orgId); + config.setDatasourceId(dsId); + config.setEnabled(true); + config.setSampleSize(200); + config.setScanIntervalHours(12); + config.setAiClassificationEnabled(true); + config.setLastScanAt(Instant.parse("2026-07-26T00:00:00Z")); + config.setLastScanError("partial"); + return config; + } + + @Test + void getSynthesizesDefaultsWhenNoRowExists() { + when(configRepository.findByDatasourceIdAndOrganizationId(dsId, orgId)) + .thenReturn(Optional.empty()); + + var view = service().get(dsId, orgId); + + assertThat(view.datasourceId()).isEqualTo(dsId); + assertThat(view.enabled()).isFalse(); + assertThat(view.sampleSize()).isEqualTo(100); + assertThat(view.scanIntervalHours()).isEqualTo(24); + assertThat(view.aiClassificationEnabled()).isFalse(); + assertThat(view.lastScanAt()).isNull(); + assertThat(view.lastScanError()).isNull(); + } + + @Test + void getReturnsPersistedRow() { + when(configRepository.findByDatasourceIdAndOrganizationId(dsId, orgId)) + .thenReturn(Optional.of(existingConfig())); + + var view = service().get(dsId, orgId); + + assertThat(view.enabled()).isTrue(); + assertThat(view.sampleSize()).isEqualTo(200); + assertThat(view.scanIntervalHours()).isEqualTo(12); + assertThat(view.aiClassificationEnabled()).isTrue(); + assertThat(view.lastScanError()).isEqualTo("partial"); + } + + @Test + void getValidatesDatasourceOwnership() { + when(datasourceAdminService.getForAdmin(dsId, orgId)) + .thenThrow(new DatasourceNotFoundException(dsId)); + + assertThatThrownBy(() -> service().get(dsId, orgId)) + .isInstanceOf(DatasourceNotFoundException.class); + verify(configRepository, never()).findByDatasourceIdAndOrganizationId(any(), any()); + } + + @Test + void upsertCreatesRowWithCommandValues() { + when(configRepository.findByDatasourceIdAndOrganizationId(dsId, orgId)) + .thenReturn(Optional.empty()); + when(configRepository.save(any())).thenAnswer(inv -> inv.getArgument(0)); + + var view = service().upsert(dsId, orgId, + new UpsertDiscoveryConfigCommand(true, 300, 48, true)); + + var captor = ArgumentCaptor.forClass(DiscoveryScanConfigEntity.class); + verify(configRepository).save(captor.capture()); + var saved = captor.getValue(); + assertThat(saved.getOrganizationId()).isEqualTo(orgId); + assertThat(saved.getDatasourceId()).isEqualTo(dsId); + assertThat(saved.isEnabled()).isTrue(); + assertThat(saved.getSampleSize()).isEqualTo(300); + assertThat(saved.getScanIntervalHours()).isEqualTo(48); + assertThat(saved.isAiClassificationEnabled()).isTrue(); + assertThat(view.enabled()).isTrue(); + } + + @Test + void upsertKeepsCurrentValuesForNullFields() { + var existing = existingConfig(); + when(configRepository.findByDatasourceIdAndOrganizationId(dsId, orgId)) + .thenReturn(Optional.of(existing)); + when(configRepository.save(any())).thenAnswer(inv -> inv.getArgument(0)); + + var view = service().upsert(dsId, orgId, + new UpsertDiscoveryConfigCommand(false, null, null, null)); + + assertThat(view.enabled()).isFalse(); + assertThat(view.sampleSize()).isEqualTo(200); + assertThat(view.scanIntervalHours()).isEqualTo(12); + assertThat(view.aiClassificationEnabled()).isTrue(); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryFindingServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryFindingServiceTest.java new file mode 100644 index 00000000..0858350d --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryFindingServiceTest.java @@ -0,0 +1,190 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.core.api.PageRequest; +import com.bablsoft.accessflow.discovery.api.DiscoveryDecision; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.api.DiscoveryRowStatus; +import com.bablsoft.accessflow.discovery.internal.DiscoveryFindingStateService.ConfirmOutcome; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryFindingEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryFindingRepository; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; +import org.springframework.data.domain.PageImpl; +import org.springframework.data.domain.Pageable; +import org.springframework.data.domain.Sort; + +import java.time.Instant; +import java.util.List; +import java.util.Optional; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class DefaultDiscoveryFindingServiceTest { + + @Mock + private DiscoveryFindingRepository findingRepository; + @Mock + private DiscoveryFindingStateService stateService; + + private final UUID dsId = UUID.randomUUID(); + private final UUID orgId = UUID.randomUUID(); + private final UUID actorId = UUID.randomUUID(); + + private DefaultDiscoveryFindingService service() { + return new DefaultDiscoveryFindingService(findingRepository, stateService); + } + + private DiscoveryFindingEntity finding(DiscoveryFindingStatus status) { + var finding = new DiscoveryFindingEntity(); + finding.setId(UUID.randomUUID()); + finding.setOrganizationId(orgId); + finding.setDatasourceId(dsId); + finding.setSchemaName("public"); + finding.setTableName("users"); + finding.setColumnName("email"); + finding.setClassification(DataClassification.PII); + finding.setDetector(DiscoveryDetector.EMAIL); + finding.setConfidence(90); + finding.setStatus(status); + finding.setFirstDetectedAt(Instant.EPOCH); + finding.setLastDetectedAt(Instant.EPOCH); + return finding; + } + + @Test + void findWithoutStatusPagesAllAndDefaultsSortToLastDetectedDesc() { + when(findingRepository.findAllByDatasourceIdAndOrganizationId(eq(dsId), eq(orgId), + any(Pageable.class))).thenReturn(new PageImpl<>(List.of(finding( + DiscoveryFindingStatus.PENDING)))); + + var page = service().find(dsId, orgId, null, PageRequest.of(0, 20)); + + assertThat(page.content()).hasSize(1); + var pageable = ArgumentCaptor.forClass(Pageable.class); + verify(findingRepository).findAllByDatasourceIdAndOrganizationId(eq(dsId), eq(orgId), + pageable.capture()); + assertThat(pageable.getValue().getSort()) + .isEqualTo(Sort.by(Sort.Direction.DESC, "lastDetectedAt")); + } + + @Test + void findWithStatusUsesStatusQuery() { + when(findingRepository.findAllByDatasourceIdAndOrganizationIdAndStatus(eq(dsId), eq(orgId), + eq(DiscoveryFindingStatus.PENDING), any(Pageable.class))) + .thenReturn(new PageImpl<>(List.of(), + org.springframework.data.domain.PageRequest.of(0, 20), 0)); + + var page = service().find(dsId, orgId, DiscoveryFindingStatus.PENDING, + PageRequest.of(0, 20)); + + assertThat(page.content()).isEmpty(); + } + + @Test + void decideReportsNotFoundForUnknownId() { + var unknownId = UUID.randomUUID(); + when(findingRepository.findByIdAndDatasourceIdAndOrganizationId(unknownId, dsId, orgId)) + .thenReturn(Optional.empty()); + + var outcome = service().decide(dsId, orgId, actorId, List.of(unknownId), + DiscoveryDecision.CONFIRM); + + assertThat(outcome.results()).hasSize(1); + var row = outcome.results().getFirst(); + assertThat(row.status()).isEqualTo(DiscoveryRowStatus.NOT_FOUND); + assertThat(row.newStatus()).isNull(); + assertThat(row.finding()).isNull(); + } + + @Test + void decideReportsInvalidStateForAlreadyDecidedFinding() { + var decided = finding(DiscoveryFindingStatus.DISMISSED); + when(findingRepository.findByIdAndDatasourceIdAndOrganizationId(decided.getId(), dsId, + orgId)).thenReturn(Optional.of(decided)); + + var outcome = service().decide(dsId, orgId, actorId, List.of(decided.getId()), + DiscoveryDecision.CONFIRM); + + var row = outcome.results().getFirst(); + assertThat(row.status()).isEqualTo(DiscoveryRowStatus.INVALID_STATE); + assertThat(row.newStatus()).isEqualTo(DiscoveryFindingStatus.DISMISSED); + verify(stateService, never()).confirm(any(), any()); + } + + @Test + void confirmSuccessAndTagConflictAreReportedPerRow() { + var success = finding(DiscoveryFindingStatus.PENDING); + var conflicting = finding(DiscoveryFindingStatus.PENDING); + when(findingRepository.findByIdAndDatasourceIdAndOrganizationId(success.getId(), dsId, + orgId)).thenReturn(Optional.of(success)); + when(findingRepository.findByIdAndDatasourceIdAndOrganizationId(conflicting.getId(), dsId, + orgId)).thenReturn(Optional.of(conflicting)); + when(stateService.confirm(success, actorId)) + .thenAnswer(inv -> confirmed(success, false)); + when(stateService.confirm(conflicting, actorId)) + .thenAnswer(inv -> confirmed(conflicting, true)); + + var outcome = service().decide(dsId, orgId, actorId, + List.of(success.getId(), conflicting.getId()), DiscoveryDecision.CONFIRM); + + assertThat(outcome.results()).extracting(r -> r.status()).containsExactly( + DiscoveryRowStatus.SUCCESS, DiscoveryRowStatus.TAG_CONFLICT); + assertThat(outcome.results()).extracting(r -> r.newStatus()).containsOnly( + DiscoveryFindingStatus.CONFIRMED); + } + + @Test + void dismissDelegatesToStateService() { + var pending = finding(DiscoveryFindingStatus.PENDING); + when(findingRepository.findByIdAndDatasourceIdAndOrganizationId(pending.getId(), dsId, + orgId)).thenReturn(Optional.of(pending)); + when(stateService.dismiss(pending, actorId)).thenAnswer(inv -> { + pending.setStatus(DiscoveryFindingStatus.DISMISSED); + return pending; + }); + + var outcome = service().decide(dsId, orgId, actorId, List.of(pending.getId()), + DiscoveryDecision.DISMISS); + + var row = outcome.results().getFirst(); + assertThat(row.status()).isEqualTo(DiscoveryRowStatus.SUCCESS); + assertThat(row.newStatus()).isEqualTo(DiscoveryFindingStatus.DISMISSED); + } + + @Test + void unexpectedErrorYieldsErrorRowAndContinues() { + var failing = finding(DiscoveryFindingStatus.PENDING); + var ok = finding(DiscoveryFindingStatus.PENDING); + when(findingRepository.findByIdAndDatasourceIdAndOrganizationId(failing.getId(), dsId, + orgId)).thenReturn(Optional.of(failing)); + when(findingRepository.findByIdAndDatasourceIdAndOrganizationId(ok.getId(), dsId, orgId)) + .thenReturn(Optional.of(ok)); + when(stateService.confirm(failing, actorId)).thenThrow(new IllegalStateException("boom")); + when(stateService.confirm(ok, actorId)).thenAnswer(inv -> confirmed(ok, false)); + + var outcome = service().decide(dsId, orgId, actorId, + List.of(failing.getId(), ok.getId()), DiscoveryDecision.CONFIRM); + + assertThat(outcome.results()).extracting(r -> r.status()).containsExactly( + DiscoveryRowStatus.ERROR, DiscoveryRowStatus.SUCCESS); + assertThat(outcome.results().getFirst().newStatus()) + .isEqualTo(DiscoveryFindingStatus.PENDING); + } + + private static ConfirmOutcome confirmed(DiscoveryFindingEntity finding, boolean conflict) { + finding.setStatus(DiscoveryFindingStatus.CONFIRMED); + return new ConfirmOutcome(finding, conflict); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryScanTriggerServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryScanTriggerServiceTest.java new file mode 100644 index 00000000..aeb6567b --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DefaultDiscoveryScanTriggerServiceTest.java @@ -0,0 +1,85 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.core.api.DatasourceAdminService; +import com.bablsoft.accessflow.core.api.DatasourceNotFoundException; +import com.bablsoft.accessflow.discovery.api.DiscoveryScanAlreadyRunningException; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.util.UUID; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.TimeUnit; + +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.timeout; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class DefaultDiscoveryScanTriggerServiceTest { + + @Mock + private DiscoveryScanService scanService; + @Mock + private DatasourceAdminService datasourceAdminService; + + private final ExecutorService executor = Executors.newVirtualThreadPerTaskExecutor(); + + private final UUID dsId = UUID.randomUUID(); + private final UUID orgId = UUID.randomUUID(); + private final UUID actorId = UUID.randomUUID(); + + @AfterEach + void shutDown() throws InterruptedException { + executor.shutdown(); + executor.awaitTermination(5, TimeUnit.SECONDS); + } + + private DefaultDiscoveryScanTriggerService service() { + return new DefaultDiscoveryScanTriggerService(scanService, datasourceAdminService, + executor); + } + + @Test + void submitsScanAsynchronously() { + service().requestScan(dsId, orgId, actorId); + + verify(scanService, timeout(5000)).scan(dsId, orgId, actorId); + } + + @Test + void unknownDatasourceRejectsSynchronously() { + when(datasourceAdminService.getForAdmin(dsId, orgId)) + .thenThrow(new DatasourceNotFoundException(dsId)); + + assertThatThrownBy(() -> service().requestScan(dsId, orgId, actorId)) + .isInstanceOf(DatasourceNotFoundException.class); + verify(scanService, never()).scan(any(), any(), any()); + } + + @Test + void inFlightScanRejectsWith409Synchronously() { + when(scanService.isInFlight(dsId)).thenReturn(true); + + assertThatThrownBy(() -> service().requestScan(dsId, orgId, actorId)) + .isInstanceOf(DiscoveryScanAlreadyRunningException.class); + verify(scanService, never()).scan(any(), any(), any()); + } + + @Test + void raceLostInsideTaskIsSwallowed() { + doThrow(new DiscoveryScanAlreadyRunningException(dsId)) + .when(scanService).scan(dsId, orgId, actorId); + + service().requestScan(dsId, orgId, actorId); + + verify(scanService, timeout(5000)).scan(dsId, orgId, actorId); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DiscoveryFindingStateServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DiscoveryFindingStateServiceTest.java new file mode 100644 index 00000000..1b04badb --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DiscoveryFindingStateServiceTest.java @@ -0,0 +1,120 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.core.api.CreateDataClassificationTagCommand; +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.core.api.DataClassificationAdminService; +import com.bablsoft.accessflow.core.api.IllegalDataClassificationTagException; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryFindingEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryFindingRepository; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class DiscoveryFindingStateServiceTest { + + private static final Instant NOW = Instant.parse("2026-07-27T10:00:00Z"); + + @Mock + private DiscoveryFindingRepository findingRepository; + @Mock + private DataClassificationAdminService dataClassificationAdminService; + + private final UUID dsId = UUID.randomUUID(); + private final UUID orgId = UUID.randomUUID(); + private final UUID actorId = UUID.randomUUID(); + + private DiscoveryFindingStateService service() { + return new DiscoveryFindingStateService(findingRepository, + dataClassificationAdminService, Clock.fixed(NOW, ZoneOffset.UTC)); + } + + private DiscoveryFindingEntity finding(String schemaName) { + var finding = new DiscoveryFindingEntity(); + finding.setId(UUID.randomUUID()); + finding.setOrganizationId(orgId); + finding.setDatasourceId(dsId); + finding.setSchemaName(schemaName); + finding.setTableName("users"); + finding.setColumnName("email"); + finding.setClassification(DataClassification.PII); + finding.setDetector(DiscoveryDetector.EMAIL); + finding.setConfidence(96); + finding.setStatus(DiscoveryFindingStatus.PENDING); + return finding; + } + + @Test + void confirmAppliesTagWithSchemaQualifiedTableAndMarksConfirmed() { + var finding = finding("public"); + + var outcome = service().confirm(finding, actorId); + + var captor = ArgumentCaptor.forClass(CreateDataClassificationTagCommand.class); + verify(dataClassificationAdminService).create(eq(dsId), eq(orgId), captor.capture()); + var command = captor.getValue(); + assertThat(command.tableName()).isEqualTo("public.users"); + assertThat(command.columnName()).isEqualTo("email"); + assertThat(command.classifications()).containsExactly(DataClassification.PII); + assertThat(command.applyMasking()).isTrue(); + assertThat(command.note()).contains("EMAIL").contains("96%"); + + assertThat(outcome.tagConflict()).isFalse(); + assertThat(finding.getStatus()).isEqualTo(DiscoveryFindingStatus.CONFIRMED); + assertThat(finding.getDecidedBy()).isEqualTo(actorId); + assertThat(finding.getDecidedAt()).isEqualTo(NOW); + verify(findingRepository).save(finding); + } + + @Test + void confirmUsesBareTableNameWithoutSchema() { + var finding = finding(null); + + service().confirm(finding, actorId); + + var captor = ArgumentCaptor.forClass(CreateDataClassificationTagCommand.class); + verify(dataClassificationAdminService).create(eq(dsId), eq(orgId), captor.capture()); + assertThat(captor.getValue().tableName()).isEqualTo("users"); + } + + @Test + void existingTagStillConfirmsButReportsConflict() { + var finding = finding("public"); + when(dataClassificationAdminService.create(any(), any(), any())) + .thenThrow(new IllegalDataClassificationTagException("duplicate")); + + var outcome = service().confirm(finding, actorId); + + assertThat(outcome.tagConflict()).isTrue(); + assertThat(finding.getStatus()).isEqualTo(DiscoveryFindingStatus.CONFIRMED); + verify(findingRepository).save(finding); + } + + @Test + void dismissMarksDismissedWithoutTouchingTags() { + var finding = finding("public"); + + var dismissed = service().dismiss(finding, actorId); + + assertThat(dismissed.getStatus()).isEqualTo(DiscoveryFindingStatus.DISMISSED); + assertThat(dismissed.getDecidedBy()).isEqualTo(actorId); + assertThat(dismissed.getDecidedAt()).isEqualTo(NOW); + org.mockito.Mockito.verifyNoInteractions(dataClassificationAdminService); + verify(findingRepository).save(finding); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DiscoveryScanServiceTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DiscoveryScanServiceTest.java new file mode 100644 index 00000000..0b0cee5b --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/DiscoveryScanServiceTest.java @@ -0,0 +1,390 @@ +package com.bablsoft.accessflow.discovery.internal; + +import com.bablsoft.accessflow.ai.api.DataDiscoveryAiService; +import com.bablsoft.accessflow.ai.api.DataDiscoveryAiService.DiscoveryColumnSuggestion; +import com.bablsoft.accessflow.audit.api.AuditAction; +import com.bablsoft.accessflow.audit.api.AuditEntry; +import com.bablsoft.accessflow.audit.api.AuditLogService; +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.core.api.DataClassificationQueryService; +import com.bablsoft.accessflow.core.api.DataClassificationTagView; +import com.bablsoft.accessflow.core.api.DatabaseSchemaView; +import com.bablsoft.accessflow.core.api.DatasourceAdminService; +import com.bablsoft.accessflow.core.api.MaskingPolicyAdminService; +import com.bablsoft.accessflow.core.api.MaskingPolicyView; +import com.bablsoft.accessflow.core.api.MaskingStrategy; +import com.bablsoft.accessflow.core.api.ResultColumn; +import com.bablsoft.accessflow.core.api.SampleTableRequest; +import com.bablsoft.accessflow.core.api.SelectExecutionResult; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.api.DiscoveryScanAlreadyRunningException; +import com.bablsoft.accessflow.discovery.internal.config.DiscoveryProperties; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryFindingEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryScanConfigEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryFindingRepository; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryScanConfigRepository; +import com.bablsoft.accessflow.proxy.api.QueryExecutor; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatThrownBy; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.eq; +import static org.mockito.Mockito.lenient; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class DiscoveryScanServiceTest { + + private static final Instant NOW = Instant.parse("2026-07-27T10:00:00Z"); + + @Mock + private DiscoveryScanConfigRepository configRepository; + @Mock + private DiscoveryFindingRepository findingRepository; + @Mock + private DatasourceAdminService datasourceAdminService; + @Mock + private DataClassificationQueryService dataClassificationQueryService; + @Mock + private MaskingPolicyAdminService maskingPolicyAdminService; + @Mock + private QueryExecutor queryExecutor; + @Mock + private DataDiscoveryAiService dataDiscoveryAiService; + @Mock + private AuditLogService auditLogService; + + private final UUID dsId = UUID.randomUUID(); + private final UUID orgId = UUID.randomUUID(); + + private DiscoveryScanService service; + + @BeforeEach + void setUp() { + service = newService(new DiscoveryProperties(null, null, null, null, null)); + } + + private DiscoveryScanService newService(DiscoveryProperties properties) { + return new DiscoveryScanService(configRepository, findingRepository, + datasourceAdminService, dataClassificationQueryService, maskingPolicyAdminService, + queryExecutor, dataDiscoveryAiService, auditLogService, properties, + Clock.fixed(NOW, ZoneOffset.UTC)); + } + + private void stubHappyPath(DatabaseSchemaView schema, SelectExecutionResult result) { + when(configRepository.findByDatasourceIdAndOrganizationId(dsId, orgId)) + .thenReturn(Optional.empty()); + when(datasourceAdminService.introspectSchemaForSystem(dsId, orgId)).thenReturn(schema); + lenient().when(dataClassificationQueryService.findByDatasource(dsId, orgId)) + .thenReturn(List.of()); + lenient().when(maskingPolicyAdminService.listForDatasource(dsId, orgId)) + .thenReturn(List.of()); + lenient().when(queryExecutor.sampleTable(any())).thenReturn(result); + lenient().when(findingRepository.findByNaturalKey(any(), any(), any(), any(), any(), any(), + any())).thenReturn(Optional.empty()); + } + + private static DatabaseSchemaView schemaWithUsersTable() { + return new DatabaseSchemaView(List.of(new DatabaseSchemaView.Schema("public", List.of( + new DatabaseSchemaView.Table("users", List.of( + new DatabaseSchemaView.Column("email", "varchar", false, false), + new DatabaseSchemaView.Column("name", "varchar", false, false)), + List.of()))))); + } + + private static SelectExecutionResult usersSample() { + var columns = List.of(new ResultColumn("email", 12, "varchar"), + new ResultColumn("name", 12, "varchar")); + var rows = List.>of( + List.of("alice@example.com", "Alice"), + List.of("bob@example.com", "Bob"), + List.of("carol@example.com", "Carol"), + List.of("dave@example.com", "Dave"), + List.of("erin@example.com", "Erin")); + return new SelectExecutionResult(columns, rows, rows.size(), false, Duration.ofMillis(5), + null, null, null); + } + + @Test + void createsPendingFindingForEmailColumn() { + stubHappyPath(schemaWithUsersTable(), usersSample()); + + service.scan(dsId, orgId, null); + + var captor = ArgumentCaptor.forClass(DiscoveryFindingEntity.class); + verify(findingRepository).save(captor.capture()); + var finding = captor.getValue(); + assertThat(finding.getSchemaName()).isEqualTo("public"); + assertThat(finding.getTableName()).isEqualTo("users"); + assertThat(finding.getColumnName()).isEqualTo("email"); + assertThat(finding.getClassification()).isEqualTo(DataClassification.PII); + assertThat(finding.getDetector()).isEqualTo(DiscoveryDetector.EMAIL); + assertThat(finding.getConfidence()).isEqualTo(100); + assertThat(finding.getMatchCount()).isEqualTo(5); + assertThat(finding.getSampleCount()).isEqualTo(5); + assertThat(finding.getStatus()).isEqualTo(DiscoveryFindingStatus.PENDING); + assertThat(finding.getFirstDetectedAt()).isEqualTo(NOW); + // PARTIAL(visible_suffix=4) — never the raw value. + assertThat(finding.getSampleRedacted()).endsWith(".com").doesNotContain("alice"); + } + + @Test + void refreshesPendingFindingInPlace() { + stubHappyPath(schemaWithUsersTable(), usersSample()); + var existing = new DiscoveryFindingEntity(); + existing.setId(UUID.randomUUID()); + existing.setStatus(DiscoveryFindingStatus.PENDING); + existing.setFirstDetectedAt(NOW.minus(Duration.ofDays(2))); + existing.setConfidence(40); + when(findingRepository.findByNaturalKey(orgId, dsId, "public", "users", "email", + DataClassification.PII, DiscoveryDetector.EMAIL)).thenReturn(Optional.of(existing)); + + service.scan(dsId, orgId, null); + + verify(findingRepository).save(existing); + assertThat(existing.getConfidence()).isEqualTo(100); + assertThat(existing.getLastDetectedAt()).isEqualTo(NOW); + assertThat(existing.getFirstDetectedAt()).isEqualTo(NOW.minus(Duration.ofDays(2))); + } + + @Test + void neverTouchesConfirmedOrDismissedFindings() { + stubHappyPath(schemaWithUsersTable(), usersSample()); + var dismissed = new DiscoveryFindingEntity(); + dismissed.setStatus(DiscoveryFindingStatus.DISMISSED); + when(findingRepository.findByNaturalKey(orgId, dsId, "public", "users", "email", + DataClassification.PII, DiscoveryDetector.EMAIL)) + .thenReturn(Optional.of(dismissed)); + + service.scan(dsId, orgId, null); + + verify(findingRepository, never()).save(any()); + } + + @Test + void skipsColumnsAlreadyTagged() { + stubHappyPath(schemaWithUsersTable(), usersSample()); + when(dataClassificationQueryService.findByDatasource(dsId, orgId)).thenReturn(List.of( + new DataClassificationTagView(UUID.randomUUID(), dsId, "public.users", "email", + DataClassification.PII, null, NOW, NOW))); + + service.scan(dsId, orgId, null); + + verify(findingRepository, never()).save(any()); + } + + @Test + void skipsColumnsCoveredByEnabledMaskingPolicy() { + stubHappyPath(schemaWithUsersTable(), usersSample()); + when(maskingPolicyAdminService.listForDatasource(dsId, orgId)).thenReturn(List.of( + new MaskingPolicyView(UUID.randomUUID(), dsId, "users.email", + MaskingStrategy.PARTIAL, Map.of(), List.of(), List.of(), List.of(), true, + NOW, NOW))); + + service.scan(dsId, orgId, null); + + verify(findingRepository, never()).save(any()); + } + + @Test + void ignoresColumnsBelowMinimumSampleCount() { + var columns = List.of(new ResultColumn("email", 12, "varchar")); + var rows = List.>of(List.of("alice@example.com"), List.of("bob@example.com")); + stubHappyPath(schemaWithUsersTable(), + new SelectExecutionResult(columns, rows, 2, false, Duration.ofMillis(1), null, + null, null)); + + service.scan(dsId, orgId, null); + + verify(findingRepository, never()).save(any()); + } + + @Test + void belowMatchRatioProducesNoFinding() { + var columns = List.of(new ResultColumn("notes", 12, "varchar")); + var rows = List.>of( + List.of("call alice@example.com"), List.of("plain note"), List.of("another note"), + List.of("more text"), List.of("and more"), List.of("last")); + stubHappyPath(schemaWithUsersTable(), + new SelectExecutionResult(columns, rows, 6, false, Duration.ofMillis(1), null, + null, null)); + + service.scan(dsId, orgId, null); + + verify(findingRepository, never()).save(any()); + } + + @Test + void capsTablesPerScanAndFlagsPartial() { + var tables = new java.util.ArrayList(); + for (var i = 0; i < 3; i++) { + tables.add(new DatabaseSchemaView.Table("t" + i, List.of(), List.of())); + } + var schema = new DatabaseSchemaView(List.of( + new DatabaseSchemaView.Schema("public", tables))); + service = newService(new DiscoveryProperties(null, null, null, 2, null)); + stubHappyPath(schema, new SelectExecutionResult(List.of(), List.of(), 0, false, + Duration.ofMillis(1), null, null, null)); + + service.scan(dsId, orgId, null); + + verify(queryExecutor, org.mockito.Mockito.times(2)).sampleTable(any()); + var audit = ArgumentCaptor.forClass(AuditEntry.class); + verify(auditLogService).record(audit.capture()); + assertThat(audit.getValue().metadata()) + .containsEntry("partial", true) + .containsEntry("tablesSkipped", 1) + .containsEntry("tablesScanned", 2); + } + + @Test + void perTableFailureContinuesScan() { + var schema = new DatabaseSchemaView(List.of(new DatabaseSchemaView.Schema("public", + List.of(new DatabaseSchemaView.Table("bad", List.of(), List.of()), + new DatabaseSchemaView.Table("users", List.of(), List.of()))))); + stubHappyPath(schema, usersSample()); + when(queryExecutor.sampleTable(any())) + .thenThrow(new IllegalStateException("connection refused")) + .thenReturn(usersSample()); + + service.scan(dsId, orgId, null); + + var audit = ArgumentCaptor.forClass(AuditEntry.class); + verify(auditLogService).record(audit.capture()); + assertThat(audit.getValue().metadata()) + .containsEntry("tablesFailed", 1) + .containsEntry("tablesScanned", 1); + } + + @Test + void introspectionFailureStampsErrorAndAudits() { + when(configRepository.findByDatasourceIdAndOrganizationId(dsId, orgId)) + .thenReturn(Optional.empty()); + when(datasourceAdminService.introspectSchemaForSystem(dsId, orgId)) + .thenThrow(new IllegalStateException("unreachable")); + + service.scan(dsId, orgId, null); + + var config = ArgumentCaptor.forClass(DiscoveryScanConfigEntity.class); + verify(configRepository).save(config.capture()); + assertThat(config.getValue().getLastScanError()).isEqualTo("unreachable"); + assertThat(config.getValue().getLastScanAt()).isEqualTo(NOW); + var audit = ArgumentCaptor.forClass(AuditEntry.class); + verify(auditLogService).record(audit.capture()); + assertThat(audit.getValue().action()).isEqualTo(AuditAction.DISCOVERY_SCAN_COMPLETED); + assertThat(audit.getValue().metadata()).containsEntry("error", "unreachable"); + } + + @Test + void aiPassRunsOnlyWhenEnabledAndCreatesAiFindings() { + var config = new DiscoveryScanConfigEntity(); + config.setId(UUID.randomUUID()); + config.setOrganizationId(orgId); + config.setDatasourceId(dsId); + config.setAiClassificationEnabled(true); + config.setSampleSize(50); + var columns = List.of(new ResultColumn("national_id", 12, "varchar")); + var rows = List.>of(List.of("11-22-33"), List.of("44-55-66"), + List.of("77-88-99"), List.of("12-34-56"), List.of("65-43-21")); + stubHappyPath(schemaWithUsersTable(), + new SelectExecutionResult(columns, rows, 5, false, Duration.ofMillis(1), null, + null, null)); + when(configRepository.findByDatasourceIdAndOrganizationId(dsId, orgId)) + .thenReturn(Optional.of(config)); + when(dataDiscoveryAiService.classifyColumns(eq(orgId), any())).thenReturn(List.of( + new DiscoveryColumnSuggestion("national_id", DataClassification.SENSITIVE, 70, + "identifier-like"))); + + service.scan(dsId, orgId, null); + + var context = ArgumentCaptor.forClass( + DataDiscoveryAiService.DiscoveryTableContext.class); + verify(dataDiscoveryAiService).classifyColumns(eq(orgId), context.capture()); + // Redacted (format-preserving) samples only — digits become '*'. + assertThat(context.getValue().columns().getFirst().redactedSamples()) + .allSatisfy(sample -> assertThat(sample).matches("\\*\\*-\\*\\*-\\*\\*")); + + var captor = ArgumentCaptor.forClass(DiscoveryFindingEntity.class); + verify(findingRepository).save(captor.capture()); + var finding = captor.getValue(); + assertThat(finding.getDetector()).isEqualTo(DiscoveryDetector.AI); + assertThat(finding.getClassification()).isEqualTo(DataClassification.SENSITIVE); + assertThat(finding.getConfidence()).isEqualTo(70); + assertThat(finding.getRationale()).isEqualTo("identifier-like"); + assertThat(finding.getMatchCount()).isZero(); + assertThat(finding.getSampleRedacted()).isEqualTo("**-**-**"); + } + + @Test + void aiPassSkippedWhenDisabled() { + stubHappyPath(schemaWithUsersTable(), usersSample()); + + service.scan(dsId, orgId, null); + + org.mockito.Mockito.verifyNoInteractions(dataDiscoveryAiService); + } + + @Test + void secondConcurrentScanIsRejected() throws Exception { + var latch = new java.util.concurrent.CountDownLatch(1); + var release = new java.util.concurrent.CountDownLatch(1); + when(configRepository.findByDatasourceIdAndOrganizationId(dsId, orgId)) + .thenReturn(Optional.empty()); + when(datasourceAdminService.introspectSchemaForSystem(dsId, orgId)).thenAnswer(inv -> { + latch.countDown(); + release.await(); + return new DatabaseSchemaView(List.of()); + }); + + var first = new Thread(() -> service.scan(dsId, orgId, null)); + first.start(); + latch.await(); + try { + assertThatThrownBy(() -> service.scan(dsId, orgId, null)) + .isInstanceOf(DiscoveryScanAlreadyRunningException.class); + } finally { + release.countDown(); + first.join(); + } + } + + @Test + void sampleRequestCarriesConfiguredSampleSizeAndTimeout() { + var config = new DiscoveryScanConfigEntity(); + config.setId(UUID.randomUUID()); + config.setOrganizationId(orgId); + config.setDatasourceId(dsId); + config.setSampleSize(250); + stubHappyPath(schemaWithUsersTable(), usersSample()); + when(configRepository.findByDatasourceIdAndOrganizationId(dsId, orgId)) + .thenReturn(Optional.of(config)); + + service.scan(dsId, orgId, null); + + var request = ArgumentCaptor.forClass(SampleTableRequest.class); + verify(queryExecutor).sampleTable(request.capture()); + assertThat(request.getValue().maxRowsOverride()).isEqualTo(250); + assertThat(request.getValue().statementTimeoutOverride()) + .isEqualTo(Duration.ofSeconds(10)); + assertThat(request.getValue().columnMasks()).isEmpty(); + assertThat(request.getValue().rowSecurityPredicates()).isEmpty(); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/config/DiscoveryPropertiesTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/config/DiscoveryPropertiesTest.java new file mode 100644 index 00000000..ab18bfc6 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/config/DiscoveryPropertiesTest.java @@ -0,0 +1,44 @@ +package com.bablsoft.accessflow.discovery.internal.config; + +import org.junit.jupiter.api.Test; + +import java.time.Duration; + +import static org.assertj.core.api.Assertions.assertThat; + +class DiscoveryPropertiesTest { + + @Test + void nullFieldsFallBackToDefaults() { + var properties = new DiscoveryProperties(null, null, null, null, null); + + assertThat(properties.scanPollInterval()).isEqualTo(Duration.ofMinutes(15)); + assertThat(properties.scanTimeBudget()).isEqualTo(Duration.ofMinutes(10)); + assertThat(properties.sampleStatementTimeout()).isEqualTo(Duration.ofSeconds(10)); + assertThat(properties.maxTablesPerScan()).isEqualTo(200); + assertThat(properties.maxAiTablesPerScan()).isEqualTo(25); + } + + @Test + void nonPositiveDurationsAndCountsFallBackToDefaults() { + var properties = new DiscoveryProperties(Duration.ZERO, Duration.ZERO, + Duration.ofSeconds(-1), 0, -1); + + assertThat(properties.scanTimeBudget()).isEqualTo(Duration.ofMinutes(10)); + assertThat(properties.sampleStatementTimeout()).isEqualTo(Duration.ofSeconds(10)); + assertThat(properties.maxTablesPerScan()).isEqualTo(200); + assertThat(properties.maxAiTablesPerScan()).isEqualTo(25); + } + + @Test + void explicitValuesAreKept() { + var properties = new DiscoveryProperties(Duration.ofMinutes(5), Duration.ofMinutes(2), + Duration.ofSeconds(3), 50, 0); + + assertThat(properties.scanPollInterval()).isEqualTo(Duration.ofMinutes(5)); + assertThat(properties.scanTimeBudget()).isEqualTo(Duration.ofMinutes(2)); + assertThat(properties.sampleStatementTimeout()).isEqualTo(Duration.ofSeconds(3)); + assertThat(properties.maxTablesPerScan()).isEqualTo(50); + assertThat(properties.maxAiTablesPerScan()).isZero(); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/CreditCardDetectorTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/CreditCardDetectorTest.java new file mode 100644 index 00000000..874ba2eb --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/CreditCardDetectorTest.java @@ -0,0 +1,53 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import static org.assertj.core.api.Assertions.assertThat; + +class CreditCardDetectorTest { + + private final CreditCardDetector detector = new CreditCardDetector(); + + @Test + void exposesTypeAndClassification() { + assertThat(detector.type()).isEqualTo(DiscoveryDetector.CREDIT_CARD); + assertThat(detector.classification()).isEqualTo(DataClassification.PCI); + } + + @ParameterizedTest + @ValueSource(strings = { + "4111111111111111", // Visa test PAN + "4111 1111 1111 1111", // spaces + "4111-1111-1111-1111", // dashes + "5500005555555559", // Mastercard test PAN + "340000000000009", // Amex (15 digits) + "6011000990139424", // Discover + "3566002020360505" // JCB + }) + void matchesLuhnValidPans(String value) { + assertThat(detector.matches(value)).isTrue(); + } + + @ParameterizedTest + @ValueSource(strings = { + "4111111111111112", // Luhn-invalid + "1234567890123456", // Luhn-invalid + "411111111111", // 12 digits — too short + "41111111111111111111", // 20 digits — too long + "4111a11111111111", // letter + "hello world", + "079-05-1120" // SSN shape, not a PAN + }) + void rejectsInvalidValues(String value) { + assertThat(detector.matches(value)).isFalse(); + } + + @Test + void rejectsOversizedValue() { + assertThat(detector.matches("4111111111111111" + " 0".repeat(20))).isFalse(); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/EmailDetectorTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/EmailDetectorTest.java new file mode 100644 index 00000000..24164850 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/EmailDetectorTest.java @@ -0,0 +1,52 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import static org.assertj.core.api.Assertions.assertThat; + +class EmailDetectorTest { + + private final EmailDetector detector = new EmailDetector(); + + @Test + void exposesTypeAndClassification() { + assertThat(detector.type()).isEqualTo(DiscoveryDetector.EMAIL); + assertThat(detector.classification()).isEqualTo(DataClassification.PII); + } + + @ParameterizedTest + @ValueSource(strings = { + "alice@example.com", + "bob.smith+tag@sub.example.co.uk", + " padded@example.org ", + "UPPER.CASE@EXAMPLE.COM", + "a_b-c%d@example.io" + }) + void matchesValidEmails(String value) { + assertThat(detector.matches(value)).isTrue(); + } + + @ParameterizedTest + @ValueSource(strings = { + "not-an-email", + "missing-at.example.com", + "two@@example.com", + "no-tld@example", + "spaces in@example.com", + "@example.com", + "user@", + "12345" + }) + void rejectsInvalidValues(String value) { + assertThat(detector.matches(value)).isFalse(); + } + + @Test + void rejectsOversizedValue() { + assertThat(detector.matches("a".repeat(320) + "@example.com")).isFalse(); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/IbanDetectorTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/IbanDetectorTest.java new file mode 100644 index 00000000..ecbffb73 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/IbanDetectorTest.java @@ -0,0 +1,46 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import static org.assertj.core.api.Assertions.assertThat; + +class IbanDetectorTest { + + private final IbanDetector detector = new IbanDetector(); + + @Test + void exposesTypeAndClassification() { + assertThat(detector.type()).isEqualTo(DiscoveryDetector.IBAN); + assertThat(detector.classification()).isEqualTo(DataClassification.FINANCIAL); + } + + @ParameterizedTest + @ValueSource(strings = { + "DE89370400440532013000", + "GB29NWBK60161331926819", + "FR1420041010050500013M02606", + "DE89 3704 0044 0532 0130 00", // grouped with spaces + "nl91abna0417164300" // lower case tolerated + }) + void matchesValidIbans(String value) { + assertThat(detector.matches(value)).isTrue(); + } + + @ParameterizedTest + @ValueSource(strings = { + "DE89370400440532013001", // checksum broken + "DE8937040044053201300", // wrong length for DE + "ZZ89370400440532013000", // unknown country + "D189370400440532013000", // digit in country code + "GB29NWBK6016133192681", // wrong length for GB + "not an iban", + "4111111111111111" + }) + void rejectsInvalidValues(String value) { + assertThat(detector.matches(value)).isFalse(); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/PhoneDetectorTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/PhoneDetectorTest.java new file mode 100644 index 00000000..58aa190a --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/PhoneDetectorTest.java @@ -0,0 +1,54 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import static org.assertj.core.api.Assertions.assertThat; + +class PhoneDetectorTest { + + private final PhoneDetector detector = new PhoneDetector(); + + @Test + void exposesTypeAndClassification() { + assertThat(detector.type()).isEqualTo(DiscoveryDetector.PHONE); + assertThat(detector.classification()).isEqualTo(DataClassification.PII); + } + + @ParameterizedTest + @ValueSource(strings = { + "+14155552671", + "+1 415 555 2671", + "(415) 555-2671", + "415-555-2671", + "415.555.2671", + "+49 30 901820", + "020 7946 0958" + }) + void matchesValidPhoneNumbers(String value) { + assertThat(detector.matches(value)).isTrue(); + } + + @ParameterizedTest + @ValueSource(strings = { + "1234567", // bare digit run without + or punctuation — likely an id + "1234567890123456", // 16 digits — too long + "+123456", // 6 digits — too short + "phone: 415", + "hello", + "415-55", // too few digits + "()-. -" // punctuation only + }) + void rejectsInvalidValues(String value) { + assertThat(detector.matches(value)).isFalse(); + } + + @Test + void orderedListRunsStricterDetectorsFirst() { + var order = ValueDetector.ORDERED.stream().map(d -> d.type().name()).toList(); + assertThat(order).containsExactly("CREDIT_CARD", "IBAN", "EMAIL", "SSN", "PHONE"); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/SsnDetectorTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/SsnDetectorTest.java new file mode 100644 index 00000000..09e71fd0 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/detect/SsnDetectorTest.java @@ -0,0 +1,42 @@ +package com.bablsoft.accessflow.discovery.internal.detect; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import static org.assertj.core.api.Assertions.assertThat; + +class SsnDetectorTest { + + private final SsnDetector detector = new SsnDetector(); + + @Test + void exposesTypeAndClassification() { + assertThat(detector.type()).isEqualTo(DiscoveryDetector.SSN); + assertThat(detector.classification()).isEqualTo(DataClassification.PII); + } + + @ParameterizedTest + @ValueSource(strings = {"079-05-1120", "079051120", " 123-45-6789 ", "899-99-9999"}) + void matchesValidSsns(String value) { + assertThat(detector.matches(value)).isTrue(); + } + + @ParameterizedTest + @ValueSource(strings = { + "000-12-3456", // area 000 never issued + "666-12-3456", // area 666 never issued + "900-12-3456", // area >= 900 never issued + "123-00-3456", // group 00 + "123-45-0000", // serial 0000 + "123-456-789", // wrong grouping + "12-345-6789", + "1234567890", // 10 digits + "abc-de-fghi" + }) + void rejectsInvalidValues(String value) { + assertThat(detector.matches(value)).isFalse(); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/scheduled/DiscoveryScanJobTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/scheduled/DiscoveryScanJobTest.java new file mode 100644 index 00000000..516f2d8b --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/scheduled/DiscoveryScanJobTest.java @@ -0,0 +1,106 @@ +package com.bablsoft.accessflow.discovery.internal.scheduled; + +import com.bablsoft.accessflow.discovery.api.DiscoveryScanAlreadyRunningException; +import com.bablsoft.accessflow.discovery.internal.DiscoveryScanService; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryScanConfigEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryScanConfigRepository; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.time.Clock; +import java.time.Duration; +import java.time.Instant; +import java.time.ZoneOffset; +import java.util.List; +import java.util.UUID; + +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.ArgumentMatchers.isNull; +import static org.mockito.Mockito.doThrow; +import static org.mockito.Mockito.eq; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class DiscoveryScanJobTest { + + private static final Instant NOW = Instant.parse("2026-07-27T10:00:00Z"); + + @Mock + private DiscoveryScanConfigRepository configRepository; + @Mock + private DiscoveryScanService scanService; + + private DiscoveryScanJob job() { + return new DiscoveryScanJob(configRepository, scanService, + Clock.fixed(NOW, ZoneOffset.UTC)); + } + + private static DiscoveryScanConfigEntity config(Instant lastScanAt, int intervalHours) { + var config = new DiscoveryScanConfigEntity(); + config.setId(UUID.randomUUID()); + config.setOrganizationId(UUID.randomUUID()); + config.setDatasourceId(UUID.randomUUID()); + config.setEnabled(true); + config.setScanIntervalHours(intervalHours); + config.setLastScanAt(lastScanAt); + return config; + } + + @Test + void scansNeverScannedAndOverdueConfigsOnly() { + var neverScanned = config(null, 24); + var overdue = config(NOW.minus(Duration.ofHours(25)), 24); + var fresh = config(NOW.minus(Duration.ofHours(1)), 24); + when(configRepository.findAllByEnabledTrue()) + .thenReturn(List.of(neverScanned, overdue, fresh)); + + job().run(); + + verify(scanService).scan(eq(neverScanned.getDatasourceId()), + eq(neverScanned.getOrganizationId()), isNull()); + verify(scanService).scan(eq(overdue.getDatasourceId()), + eq(overdue.getOrganizationId()), isNull()); + verify(scanService, never()).scan(eq(fresh.getDatasourceId()), any(), any()); + } + + @Test + void perDatasourceFailureDoesNotAbortBatch() { + var first = config(null, 24); + var second = config(null, 24); + when(configRepository.findAllByEnabledTrue()).thenReturn(List.of(first, second)); + doThrow(new IllegalStateException("boom")).when(scanService) + .scan(eq(first.getDatasourceId()), any(), isNull()); + + job().run(); + + verify(scanService).scan(eq(second.getDatasourceId()), + eq(second.getOrganizationId()), isNull()); + } + + @Test + void alreadyRunningScanIsSkippedQuietly() { + var first = config(null, 24); + var second = config(null, 24); + when(configRepository.findAllByEnabledTrue()).thenReturn(List.of(first, second)); + doThrow(new DiscoveryScanAlreadyRunningException(first.getDatasourceId())) + .when(scanService).scan(eq(first.getDatasourceId()), any(), isNull()); + + job().run(); + + verify(scanService).scan(eq(second.getDatasourceId()), + eq(second.getOrganizationId()), isNull()); + } + + @Test + void noDueConfigsDoesNothing() { + when(configRepository.findAllByEnabledTrue()).thenReturn(List.of()); + + job().run(); + + verify(scanService, never()).scan(any(), any(), any()); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryAuditWriterTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryAuditWriterTest.java new file mode 100644 index 00000000..8631f571 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryAuditWriterTest.java @@ -0,0 +1,72 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.audit.api.AuditAction; +import com.bablsoft.accessflow.audit.api.AuditEntry; +import com.bablsoft.accessflow.audit.api.AuditLogService; +import com.bablsoft.accessflow.audit.api.AuditResourceType; +import com.bablsoft.accessflow.audit.api.RequestAuditContext; +import com.bablsoft.accessflow.security.api.JwtClaims; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.extension.ExtendWith; +import org.mockito.ArgumentCaptor; +import org.mockito.Mock; +import org.mockito.junit.jupiter.MockitoExtension; + +import java.util.Map; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.assertj.core.api.Assertions.assertThatCode; +import static org.mockito.ArgumentMatchers.any; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; + +@ExtendWith(MockitoExtension.class) +class DiscoveryAuditWriterTest { + + @Mock + private AuditLogService auditLogService; + @Mock + private JwtClaims caller; + @Mock + private RequestAuditContext auditContext; + + private final UUID findingId = UUID.randomUUID(); + private final UUID orgId = UUID.randomUUID(); + private final UUID userId = UUID.randomUUID(); + + @Test + void recordsEntryWithCallerAndContext() { + when(caller.organizationId()).thenReturn(orgId); + when(caller.userId()).thenReturn(userId); + when(auditContext.ipAddress()).thenReturn("10.0.0.1"); + when(auditContext.userAgent()).thenReturn("test-agent"); + + new DiscoveryAuditWriter(auditLogService).record( + AuditAction.DISCOVERY_FINDING_CONFIRMED, findingId, caller, + Map.of("columnName", "email"), auditContext); + + var captor = ArgumentCaptor.forClass(AuditEntry.class); + verify(auditLogService).record(captor.capture()); + var entry = captor.getValue(); + assertThat(entry.action()).isEqualTo(AuditAction.DISCOVERY_FINDING_CONFIRMED); + assertThat(entry.resourceType()).isEqualTo(AuditResourceType.DISCOVERY_FINDING); + assertThat(entry.resourceId()).isEqualTo(findingId); + assertThat(entry.organizationId()).isEqualTo(orgId); + assertThat(entry.actorId()).isEqualTo(userId); + assertThat(entry.metadata()).containsEntry("columnName", "email"); + assertThat(entry.ipAddress()).isEqualTo("10.0.0.1"); + assertThat(entry.userAgent()).isEqualTo("test-agent"); + } + + @Test + void auditFailureIsSwallowed() { + when(caller.organizationId()).thenReturn(orgId); + when(caller.userId()).thenReturn(userId); + when(auditLogService.record(any())).thenThrow(new IllegalStateException("audit down")); + + assertThatCode(() -> new DiscoveryAuditWriter(auditLogService).record( + AuditAction.DISCOVERY_FINDING_DISMISSED, findingId, caller, Map.of(), + auditContext)).doesNotThrowAnyException(); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryControllerIntegrationTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryControllerIntegrationTest.java new file mode 100644 index 00000000..8460eca2 --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryControllerIntegrationTest.java @@ -0,0 +1,361 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.TestcontainersConfig; +import com.bablsoft.accessflow.core.api.AuthProviderType; +import com.bablsoft.accessflow.core.api.CredentialEncryptionService; +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.core.api.DbType; +import com.bablsoft.accessflow.core.api.SslMode; +import com.bablsoft.accessflow.core.api.UserRoleType; +import com.bablsoft.accessflow.core.internal.persistence.entity.DatasourceEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.OrganizationEntity; +import com.bablsoft.accessflow.core.internal.persistence.entity.UserEntity; +import com.bablsoft.accessflow.core.internal.persistence.repo.DataClassificationTagRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.DatasourceRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.MaskingPolicyRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.OrganizationRepository; +import com.bablsoft.accessflow.core.internal.persistence.repo.UserRepository; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.internal.persistence.entity.DiscoveryFindingEntity; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryFindingRepository; +import com.bablsoft.accessflow.discovery.internal.persistence.repo.DiscoveryScanConfigRepository; +import com.bablsoft.accessflow.security.internal.jwt.JwtService; +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.test.context.SpringBootTest; +import org.springframework.boot.testcontainers.context.ImportTestcontainers; +import org.springframework.http.HttpHeaders; +import org.springframework.http.MediaType; +import org.springframework.security.crypto.password.PasswordEncoder; +import org.springframework.test.context.DynamicPropertyRegistry; +import org.springframework.test.context.DynamicPropertySource; +import org.springframework.test.web.servlet.assertj.MockMvcTester; +import org.springframework.web.context.WebApplicationContext; + +import java.security.KeyPairGenerator; +import java.security.interfaces.RSAPrivateCrtKey; +import java.time.Instant; +import java.util.Base64; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; +import static org.springframework.security.test.web.servlet.setup.SecurityMockMvcConfigurers.springSecurity; + +@SpringBootTest +@ImportTestcontainers(TestcontainersConfig.class) +class DiscoveryControllerIntegrationTest { + + @Autowired WebApplicationContext context; + @Autowired UserRepository userRepository; + @Autowired OrganizationRepository organizationRepository; + @Autowired DatasourceRepository datasourceRepository; + @Autowired DiscoveryScanConfigRepository configRepository; + @Autowired DiscoveryFindingRepository findingRepository; + @Autowired DataClassificationTagRepository tagRepository; + @Autowired MaskingPolicyRepository maskingPolicyRepository; + @Autowired PasswordEncoder passwordEncoder; + @Autowired JwtService jwtService; + @Autowired CredentialEncryptionService encryptionService; + + private MockMvcTester mvc; + private OrganizationEntity primaryOrg; + private DatasourceEntity datasource; + private String adminToken; + private String analystToken; + + @DynamicPropertySource + static void securityProperties(DynamicPropertyRegistry registry) throws Exception { + var kpg = KeyPairGenerator.getInstance("RSA"); + kpg.initialize(2048); + var kp = kpg.generateKeyPair(); + var privateKey = (RSAPrivateCrtKey) kp.getPrivate(); + var pem = "-----BEGIN PRIVATE KEY-----\n" + + Base64.getMimeEncoder(64, new byte[]{'\n'}).encodeToString(privateKey.getEncoded()) + + "\n-----END PRIVATE KEY-----"; + registry.add("accessflow.jwt.private-key", () -> pem); + registry.add("accessflow.encryption-key", () -> + "0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef"); + } + + @AfterEach + void cleanup() { + findingRepository.deleteAll(); + configRepository.deleteAll(); + tagRepository.deleteAll(); + maskingPolicyRepository.deleteAll(); + datasourceRepository.deleteAll(); + } + + @BeforeEach + void setUp() { + mvc = MockMvcTester.from(context, builder -> builder.apply(springSecurity()).build()); + + findingRepository.deleteAll(); + configRepository.deleteAll(); + tagRepository.deleteAll(); + maskingPolicyRepository.deleteAll(); + datasourceRepository.deleteAll(); + userRepository.deleteAll(); + organizationRepository.deleteAll(); + + primaryOrg = saveOrg("Primary", "primary-disc"); + var admin = saveUser(primaryOrg, "admin-disc@example.com", "Admin", UserRoleType.ADMIN); + var analyst = saveUser(primaryOrg, "analyst-disc@example.com", "Analyst", + UserRoleType.ANALYST); + datasource = saveDatasource(primaryOrg, "Discovery-DS"); + adminToken = generateToken(admin); + analystToken = generateToken(analyst); + } + + private String base() { + return "/api/v1/datasources/" + datasource.getId() + "/discovery"; + } + + @Test + void getConfigSynthesizesDefaultsWhenUnconfigured() { + var result = mvc.get().uri(base() + "/config") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .exchange(); + + assertThat(result).hasStatus(200); + assertThat(result).bodyJson().extractingPath("$.enabled").asBoolean().isFalse(); + assertThat(result).bodyJson().extractingPath("$.sample_size").asNumber().isEqualTo(100); + assertThat(result).bodyJson().extractingPath("$.scan_interval_hours").asNumber() + .isEqualTo(24); + } + + @Test + void putConfigUpsertsAndPersists() { + var result = mvc.put().uri(base() + "/config") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .contentType(MediaType.APPLICATION_JSON) + .content(""" + {"enabled":true,"sample_size":250,"scan_interval_hours":12, + "ai_classification_enabled":true} + """) + .exchange(); + + assertThat(result).hasStatus(200); + assertThat(result).bodyJson().extractingPath("$.enabled").asBoolean().isTrue(); + assertThat(result).bodyJson().extractingPath("$.sample_size").asNumber().isEqualTo(250); + + var persisted = configRepository.findByDatasourceId(datasource.getId()).orElseThrow(); + assertThat(persisted.isEnabled()).isTrue(); + assertThat(persisted.getSampleSize()).isEqualTo(250); + assertThat(persisted.getScanIntervalHours()).isEqualTo(12); + assertThat(persisted.isAiClassificationEnabled()).isTrue(); + } + + @Test + void putConfigWithOutOfRangeSampleSizeReturns400() { + var result = mvc.put().uri(base() + "/config") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .contentType(MediaType.APPLICATION_JSON) + .content("{\"sample_size\":5}") + .exchange(); + + assertThat(result).hasStatus(400); + } + + @Test + void endpointsRequireDataClassificationManageAuthority() { + var result = mvc.get().uri(base() + "/config") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + analystToken) + .exchange(); + + assertThat(result).hasStatus(403); + } + + @Test + void unknownDatasourceReturns404() { + var result = mvc.get() + .uri("/api/v1/datasources/" + UUID.randomUUID() + "/discovery/config") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .exchange(); + + assertThat(result).hasStatus(404); + } + + @Test + void findingsListFiltersByStatus() { + saveFinding("email", DataClassification.PII, DiscoveryFindingStatus.PENDING); + saveFinding("iban", DataClassification.FINANCIAL, DiscoveryFindingStatus.DISMISSED); + + var result = mvc.get().uri(base() + "/findings?status=PENDING") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .exchange(); + + assertThat(result).hasStatus(200); + assertThat(result).bodyJson().extractingPath("$.total_elements").asNumber().isEqualTo(1); + assertThat(result).bodyJson().extractingPath("$.content[0].column_name").asString() + .isEqualTo("email"); + assertThat(result).bodyJson().extractingPath("$.content[0].detector").asString() + .isEqualTo("EMAIL"); + } + + @Test + void bulkConfirmCreatesTagDerivesMaskingAndMarksConfirmed() { + var finding = saveFinding("email", DataClassification.PII, + DiscoveryFindingStatus.PENDING); + var unknownId = UUID.randomUUID(); + + var result = mvc.post().uri(base() + "/findings/bulk-decision") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .contentType(MediaType.APPLICATION_JSON) + .content("{\"finding_ids\":[\"" + finding.getId() + "\",\"" + unknownId + + "\"],\"decision\":\"CONFIRM\"}") + .exchange(); + + assertThat(result).hasStatus(200); + assertThat(result).bodyJson().extractingPath("$.results[0].status").asString() + .isEqualTo("SUCCESS"); + assertThat(result).bodyJson().extractingPath("$.results[0].new_status").asString() + .isEqualTo("CONFIRMED"); + assertThat(result).bodyJson().extractingPath("$.results[1].status").asString() + .isEqualTo("NOT_FOUND"); + + // The AF-447 tag exists and derived a masking policy for the column. + var tags = tagRepository + .findAllByOrganizationIdAndDatasourceIdOrderByTableNameAscColumnNameAscClassificationAsc( + primaryOrg.getId(), datasource.getId()); + assertThat(tags).hasSize(1); + assertThat(tags.getFirst().getTableName()).isEqualTo("public.users"); + assertThat(tags.getFirst().getColumnName()).isEqualTo("email"); + var policies = maskingPolicyRepository + .findAllByOrganizationIdAndDatasourceIdOrderByCreatedAtAsc(primaryOrg.getId(), + datasource.getId()); + assertThat(policies).hasSize(1); + assertThat(policies.getFirst().getColumnRef()).isEqualTo("public.users.email"); + + var persisted = findingRepository.findById(finding.getId()).orElseThrow(); + assertThat(persisted.getStatus()).isEqualTo(DiscoveryFindingStatus.CONFIRMED); + assertThat(persisted.getDecidedBy()).isNotNull(); + } + + @Test + void bulkDismissSuppressesFinding() { + var finding = saveFinding("phone", DataClassification.PII, + DiscoveryFindingStatus.PENDING); + + var result = mvc.post().uri(base() + "/findings/bulk-decision") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .contentType(MediaType.APPLICATION_JSON) + .content("{\"finding_ids\":[\"" + finding.getId() + + "\"],\"decision\":\"DISMISS\"}") + .exchange(); + + assertThat(result).hasStatus(200); + assertThat(result).bodyJson().extractingPath("$.results[0].new_status").asString() + .isEqualTo("DISMISSED"); + assertThat(findingRepository.findById(finding.getId()).orElseThrow().getStatus()) + .isEqualTo(DiscoveryFindingStatus.DISMISSED); + assertThat(tagRepository.count()).isZero(); + } + + @Test + void bulkDecisionWithEmptyIdsReturns400() { + var result = mvc.post().uri(base() + "/findings/bulk-decision") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .contentType(MediaType.APPLICATION_JSON) + .content("{\"finding_ids\":[],\"decision\":\"CONFIRM\"}") + .exchange(); + + assertThat(result).hasStatus(400); + } + + @Test + void triggerScanReturns202() { + var result = mvc.post().uri(base() + "/scan") + .header(HttpHeaders.AUTHORIZATION, "Bearer " + adminToken) + .exchange(); + + // The async scan itself fails against the unreachable test datasource — by design the + // trigger only validates ownership and accepts. + assertThat(result).hasStatus(202); + } + + private DiscoveryFindingEntity saveFinding(String column, DataClassification classification, + DiscoveryFindingStatus status) { + var finding = new DiscoveryFindingEntity(); + finding.setId(UUID.randomUUID()); + finding.setOrganizationId(primaryOrg.getId()); + finding.setDatasourceId(datasource.getId()); + finding.setSchemaName("public"); + finding.setTableName("users"); + finding.setColumnName(column); + finding.setClassification(classification); + finding.setDetector(column.equals("email") ? DiscoveryDetector.EMAIL + : DiscoveryDetector.PHONE); + finding.setConfidence(90); + finding.setSampleRedacted("****"); + finding.setMatchCount(9); + finding.setSampleCount(10); + finding.setStatus(status); + finding.setFirstDetectedAt(Instant.now()); + finding.setLastDetectedAt(Instant.now()); + return findingRepository.save(finding); + } + + private OrganizationEntity saveOrg(String name, String slug) { + var org = new OrganizationEntity(); + org.setId(UUID.randomUUID()); + org.setName(name); + org.setSlug(slug); + return organizationRepository.save(org); + } + + private UserEntity saveUser(OrganizationEntity org, String email, String displayName, + UserRoleType role) { + var user = new UserEntity(); + user.setId(UUID.randomUUID()); + user.setEmail(email); + user.setDisplayName(displayName); + user.setPasswordHash(passwordEncoder.encode("Password123!")); + user.setRole(role); + user.setAuthProvider(AuthProviderType.LOCAL); + user.setActive(true); + user.setOrganization(org); + return userRepository.save(user); + } + + private DatasourceEntity saveDatasource(OrganizationEntity org, String name) { + var ds = new DatasourceEntity(); + ds.setId(UUID.randomUUID()); + ds.setOrganization(org); + ds.setName(name); + ds.setDbType(DbType.POSTGRESQL); + ds.setHost("nope.invalid"); + ds.setPort(65000); + ds.setDatabaseName("appdb"); + ds.setUsername("svc"); + ds.setPasswordEncrypted(encryptionService.encrypt("seed-password")); + ds.setSslMode(SslMode.DISABLE); + ds.setConnectionPoolSize(10); + ds.setMaxRowsPerQuery(1000); + ds.setRequireReviewReads(false); + ds.setRequireReviewWrites(true); + ds.setAiAnalysisEnabled(false); + ds.setActive(true); + return datasourceRepository.save(ds); + } + + private String generateToken(UserEntity entity) { + var view = new com.bablsoft.accessflow.core.api.UserView( + entity.getId(), + entity.getEmail(), + entity.getDisplayName(), + entity.getRole(), + entity.getOrganization().getId(), + entity.isActive(), + entity.getAuthProvider(), + entity.getPasswordHash(), + entity.getLastLoginAt(), + entity.getPreferredLanguage(), + entity.isTotpEnabled(), + entity.getCreatedAt()); + return jwtService.generateAccessToken(view); + } +} diff --git a/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryWebModelsTest.java b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryWebModelsTest.java new file mode 100644 index 00000000..f2cfc18f --- /dev/null +++ b/backend/src/test/java/com/bablsoft/accessflow/discovery/internal/web/DiscoveryWebModelsTest.java @@ -0,0 +1,104 @@ +package com.bablsoft.accessflow.discovery.internal.web; + +import com.bablsoft.accessflow.core.api.DataClassification; +import com.bablsoft.accessflow.core.api.PageResponse; +import com.bablsoft.accessflow.discovery.api.DiscoveryDecision; +import com.bablsoft.accessflow.discovery.api.DiscoveryDetector; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingService.BulkDecisionOutcome; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingStatus; +import com.bablsoft.accessflow.discovery.api.DiscoveryFindingView; +import com.bablsoft.accessflow.discovery.api.DiscoveryRowStatus; +import com.bablsoft.accessflow.discovery.api.DiscoveryScanConfigView; +import org.junit.jupiter.api.Test; + +import java.time.Instant; +import java.util.List; +import java.util.UUID; + +import static org.assertj.core.api.Assertions.assertThat; + +class DiscoveryWebModelsTest { + + private static final Instant NOW = Instant.parse("2026-07-27T10:00:00Z"); + + @Test + void configResponseMirrorsView() { + var dsId = UUID.randomUUID(); + var response = DiscoveryConfigResponse.from(new DiscoveryScanConfigView(dsId, true, 200, + 12, true, NOW, "partial")); + + assertThat(response.datasourceId()).isEqualTo(dsId); + assertThat(response.enabled()).isTrue(); + assertThat(response.sampleSize()).isEqualTo(200); + assertThat(response.scanIntervalHours()).isEqualTo(12); + assertThat(response.aiClassificationEnabled()).isTrue(); + assertThat(response.lastScanAt()).isEqualTo(NOW); + assertThat(response.lastScanError()).isEqualTo("partial"); + } + + @Test + void updateRequestMapsToCommand() { + var command = new UpdateDiscoveryConfigRequest(true, 300, 48, false).toCommand(); + + assertThat(command.enabled()).isTrue(); + assertThat(command.sampleSize()).isEqualTo(300); + assertThat(command.scanIntervalHours()).isEqualTo(48); + assertThat(command.aiClassificationEnabled()).isFalse(); + } + + @Test + void findingPageResponseMapsContentAndPaging() { + var view = findingView(); + var page = DiscoveryFindingPageResponse.from( + new PageResponse<>(List.of(view), 1, 20, 41, 3)); + + assertThat(page.page()).isEqualTo(1); + assertThat(page.size()).isEqualTo(20); + assertThat(page.totalElements()).isEqualTo(41); + assertThat(page.totalPages()).isEqualTo(3); + var response = page.content().getFirst(); + assertThat(response.id()).isEqualTo(view.id()); + assertThat(response.schemaName()).isEqualTo("public"); + assertThat(response.tableName()).isEqualTo("users"); + assertThat(response.columnName()).isEqualTo("email"); + assertThat(response.classification()).isEqualTo(DataClassification.PII); + assertThat(response.detector()).isEqualTo(DiscoveryDetector.EMAIL); + assertThat(response.confidence()).isEqualTo(96); + assertThat(response.sampleRedacted()).isEqualTo("****@x.com"); + assertThat(response.status()).isEqualTo(DiscoveryFindingStatus.PENDING); + } + + @Test + void bulkResponseMapsRowsWithoutFindingDetails() { + var findingId = UUID.randomUUID(); + var outcome = new BulkDecisionOutcome(List.of( + new BulkDecisionOutcome.Row(findingId, DiscoveryRowStatus.TAG_CONFLICT, + DiscoveryFindingStatus.CONFIRMED, findingView()), + new BulkDecisionOutcome.Row(findingId, DiscoveryRowStatus.NOT_FOUND, null, null))); + + var response = BulkDiscoveryDecisionResponse.from(outcome); + + assertThat(response.results()).hasSize(2); + assertThat(response.results().getFirst().status()) + .isEqualTo(DiscoveryRowStatus.TAG_CONFLICT); + assertThat(response.results().getFirst().newStatus()) + .isEqualTo(DiscoveryFindingStatus.CONFIRMED); + assertThat(response.results().getLast().status()).isEqualTo(DiscoveryRowStatus.NOT_FOUND); + assertThat(response.results().getLast().newStatus()).isNull(); + } + + @Test + void bulkRequestHoldsDecision() { + var request = new BulkDiscoveryDecisionRequest(List.of(UUID.randomUUID()), + DiscoveryDecision.DISMISS); + + assertThat(request.decision()).isEqualTo(DiscoveryDecision.DISMISS); + assertThat(request.findingIds()).hasSize(1); + } + + private static DiscoveryFindingView findingView() { + return new DiscoveryFindingView(UUID.randomUUID(), "public", "users", "email", + DataClassification.PII, DiscoveryDetector.EMAIL, 96, "****@x.com", null, 48, 50, + DiscoveryFindingStatus.PENDING, NOW, NOW, null, null); + } +} diff --git a/docs/03-data-model.md b/docs/03-data-model.md index fc6b2b9a..def671a9 100644 --- a/docs/03-data-model.md +++ b/docs/03-data-model.md @@ -379,6 +379,67 @@ delete the derived masking policy.** Tag changes are audited via `DATA_CLASSIFIC --- +## discovery_scan_config + +Per-datasource settings for automated sensitive-data discovery (AF-623). One row per datasource; +absence means discovery is disabled with defaults. The `DiscoveryScanJob` drains enabled rows whose +`last_scan_at` is older than their own `scan_interval_hours`. Created by +`V129__create_discovery.sql`. + +| Column | Type / Notes | +|--------|-------------| +| `id` | UUID PK | +| `organization_id` | FK → `organizations` (`ON DELETE CASCADE`) | +| `datasource_id` | FK → `datasources` (`ON DELETE CASCADE`), UNIQUE | +| `enabled` | BOOLEAN NOT NULL DEFAULT FALSE — opt-in for scheduled scans | +| `sample_size` | INT NOT NULL DEFAULT 100 — rows sampled per table (10–1000) | +| `scan_interval_hours` | INT NOT NULL DEFAULT 24 — per-datasource cadence (1–720) | +| `ai_classification_enabled` | BOOLEAN NOT NULL DEFAULT FALSE — opt-in AI pass (redacted samples only) | +| `last_scan_at` | TIMESTAMPTZ nullable — stamped after every scan (scheduled or on-demand) | +| `last_scan_error` | TEXT nullable — failure summary / partial-run note of the most recent scan | +| `version` | BIGINT — optimistic lock | +| `created_at` / `updated_at` | TIMESTAMPTZ | + +Indexed by `(organization_id)`. + +--- + +## discovery_finding + +Proposed sensitive-column classification worklist (AF-623). One row per +`(column, classification, detector)`; rescans refresh `PENDING` rows in place and never touch +`CONFIRMED`/`DISMISSED` rows — a dismissal is a permanent suppression. `sample_redacted` only ever +holds a redacted value (raw sampled data never persists). Confirming a finding applies the tag +through the AF-447 service (deriving masking); enums `discovery_detector` +(`EMAIL` | `CREDIT_CARD` | `SSN` | `IBAN` | `PHONE` | `AI`) and `discovery_finding_status` +(`PENDING` | `CONFIRMED` | `DISMISSED`). Created by `V129__create_discovery.sql`. + +| Column | Type / Notes | +|--------|-------------| +| `id` | UUID PK | +| `organization_id` | FK → `organizations` (`ON DELETE CASCADE`) | +| `datasource_id` | FK → `datasources` (`ON DELETE CASCADE`) | +| `schema_name` | TEXT nullable — NULL for engines without a schema concept | +| `table_name` / `column_name` | TEXT NOT NULL | +| `classification` | ENUM `data_classification` — the proposed classification | +| `detector` | ENUM `discovery_detector` — which detector produced the proposal | +| `confidence` | INT (0–100) — match ratio for regex detectors, model confidence for `AI` | +| `sample_redacted` | TEXT nullable — redacted evidence sample | +| `rationale` | TEXT nullable — `AI` findings only | +| `match_count` / `sample_count` | INT — matched / examined values (`match_count` 0 for AI rows) | +| `status` | ENUM `discovery_finding_status` DEFAULT `'PENDING'` | +| `decided_by` | FK → `users` (`ON DELETE SET NULL`) nullable | +| `decided_at` | TIMESTAMPTZ nullable | +| `first_detected_at` / `last_detected_at` | TIMESTAMPTZ NOT NULL | +| `version` | BIGINT — optimistic lock | +| `created_at` / `updated_at` | TIMESTAMPTZ | + +Unique expression index `(organization_id, datasource_id, COALESCE(schema_name, ''), table_name, +column_name, classification, detector)` (the V90 COALESCE pattern) is the rescan upsert key; +`(datasource_id, status)` serves the worklist scan. + +--- + ## review_plans Defines an approval policy. Assigned to datasources. @@ -1271,6 +1332,8 @@ The hash chain (added in V26) is per organization. Inserts are serialized by a P | `MASKING_POLICY_CREATED` / `MASKING_POLICY_UPDATED` / `MASKING_POLICY_DELETED` | Admin creates / updates / deletes a masking policy via the `/datasources/{id}/masking-policies` CRUD endpoints. Resource: `masking_policy`. | | `ROW_SECURITY_POLICY_CREATED` / `ROW_SECURITY_POLICY_UPDATED` / `ROW_SECURITY_POLICY_DELETED` | Admin creates / updates / deletes a row-security policy via the `/datasources/{id}/row-security-policies` CRUD endpoints (AF-380). Resource: `row_security_policy`. Applied row-security policy ids at execute time ride on `QUERY_EXECUTED` metadata (`applied_row_security_policy_ids`), not a separate action. | | `DATA_CLASSIFICATION_TAG_ADDED` / `DATA_CLASSIFICATION_TAG_REMOVED` | Admin tags / untags a datasource table or column via the `/datasources/{id}/classification-tags` endpoints (AF-447). Resource: `data_classification_tag`. Metadata records the table, column, classification, and (on add) whether masking was auto-applied. | +| `DISCOVERY_SCAN_COMPLETED` | A sensitive-data discovery scan finished (AF-623) — scheduled (`actor_id` NULL) or on-demand (the triggering admin). Resource: `datasource`. Metadata: tables scanned/skipped/failed, findings created/refreshed, AI suggestions, duration, `partial` flag, and the error summary when the scan failed. | +| `DISCOVERY_FINDING_CONFIRMED` / `DISCOVERY_FINDING_DISMISSED` | Admin confirms (tag applied via the AF-447 service, masking derived) or dismisses (permanently suppressed) a discovery finding via `/datasources/{id}/discovery/findings/bulk-decision`. Resource: `discovery_finding`. Metadata: table, column, classification, detector, confidence, and `tagConflict` when the tag already existed. | | `COMPLIANCE_REPORT_EXPORTED` | AUDITOR/ADMIN exported a signed compliance report via `GET /admin/compliance/reports/export` (AF-459). Resource: `compliance_report`, no resource id. Metadata captures `report_type`, `format`, `period_from`, `period_to`, optional `datasource_id`, `row_count`, `truncated`, and the export's `content_sha256` + `signature` + `signature_algorithm` — chaining the export's hash into the tamper-evident log. | | `QUERY_COMMENT_ADDED` / `QUERY_COMMENT_REPLIED` / `QUERY_COMMENT_RESOLVED` / `QUERY_COMMENT_REOPENED` | A collaborator opens / replies to / resolves / reopens an inline comment thread on a query in review (AF-441). Resource: `query_comment` (resource id = the comment id). Metadata: `query_id`, `comment_id`. | | `REQUEST_GROUP_SUBMITTED` / `REQUEST_GROUP_APPROVED` / `REQUEST_GROUP_REJECTED` / `REQUEST_GROUP_EXECUTED` / `REQUEST_GROUP_PARTIALLY_EXECUTED` / `REQUEST_GROUP_FAILED` / `REQUEST_GROUP_TIMED_OUT` / `REQUEST_GROUP_CANCELLED` | A grouped request (AF-501) is submitted / approved / rejected / fully executed / stopped mid-sequence / failed / timed out / cancelled. Resource: `request_group` (resource id = the group id). Recorded alongside each member's own query/API audit row, so the group **and** each step are independently auditable. | diff --git a/docs/04-api-spec.md b/docs/04-api-spec.md index e49fffc3..e8c2f408 100644 --- a/docs/04-api-spec.md +++ b/docs/04-api-spec.md @@ -179,6 +179,11 @@ The list is rendered by the `LanguageSwitcher` component in `mode="public"`; sel | `DELETE` | `/datasources/{id}/classification-tags/{tagId}` | ADMIN | Remove a classification tag (keeps the derived masking policy) | | `GET` | `/datasources/{id}/classification-tags/derivation-preview` | ADMIN | Preview the masking + review handling implied by a datasource's tags | | `GET` | `/admin/data-classifications` | ADMIN | List every classification tag in the organization (compliance reporting) | +| `GET` | `/datasources/{id}/discovery/config` | ADMIN | Get the datasource's sensitive-data discovery settings (AF-623) | +| `PUT` | `/datasources/{id}/discovery/config` | ADMIN | Create or update the discovery settings | +| `GET` | `/datasources/{id}/discovery/findings` | ADMIN | Page the discovery-findings worklist (`status`, `page`, `size`) | +| `POST` | `/datasources/{id}/discovery/findings/bulk-decision` | ADMIN | Confirm or dismiss a batch of findings (partial success) | +| `POST` | `/datasources/{id}/discovery/scan` | ADMIN | Trigger an immediate discovery scan (202) | | `POST` | `/datasources/drivers` | ADMIN | Upload a custom JDBC driver JAR (multipart) | | `GET` | `/datasources/drivers` | ADMIN | List the organization's uploaded JDBC drivers | | `GET` | `/datasources/drivers/{id}` | ADMIN | Get details of one uploaded driver | @@ -910,6 +915,120 @@ datasources, the evidence base for compliance reporting. --- +### Sensitive-data discovery (AF-623) + +A scheduled scanner samples column data through the same governance-aware sampling path as +`GET /datasources/{id}/sample-rows`, detects sensitive data with local regex + checksum detectors +(email, credit-card PAN with Luhn, US SSN, IBAN with mod-97, phone) and optionally the org's bound +AI analyzer, then **proposes** classification tags an admin confirms or dismisses. All endpoints +require the `DATA_CLASSIFICATION_MANAGE` permission (same as manual classification tags). + +#### GET /datasources/{id}/discovery/config — Response 200 + +```json +{ + "datasource_id": "9f7c…", + "enabled": false, + "sample_size": 100, + "scan_interval_hours": 24, + "ai_classification_enabled": false, + "last_scan_at": "2026-07-27T02:00:00Z", + "last_scan_error": null +} +``` + +When the datasource has no persisted config, the defaults above are synthesized +(`last_scan_at: null`). `last_scan_error` carries the failure summary of the most recent scan +(`null` on success; a "partial" note when the table cap or time budget truncated the run). + +#### PUT /datasources/{id}/discovery/config — Request Body + +```json +{ + "enabled": true, + "sample_size": 100, + "scan_interval_hours": 24, + "ai_classification_enabled": false +} +``` + +Upserts the config row; omitted (`null`) fields keep their current value. Validation: +`sample_size` 10–1000, `scan_interval_hours` 1–720. Response 200 mirrors the GET shape. The +optional AI pass sends **only column names, types, and redacted samples** (format-preserving +masking) to the org's first usable `ai_config` — raw sampled values never leave the platform. + +#### GET /datasources/{id}/discovery/findings — Response 200 + +Query params: `status` (`PENDING` | `CONFIRMED` | `DISMISSED`, omitted = all), `page`, `size`. +Sorted by `last_detected_at` descending. + +```json +{ + "content": [ + { + "id": "3d2a…", + "schema_name": "public", + "table_name": "customers", + "column_name": "email", + "classification": "PII", + "detector": "EMAIL", + "confidence": 96, + "sample_redacted": "*************.com", + "rationale": null, + "match_count": 48, + "sample_count": 50, + "status": "PENDING", + "first_detected_at": "2026-07-20T02:00:00Z", + "last_detected_at": "2026-07-27T02:00:00Z", + "decided_by": null, + "decided_at": null + } + ], + "page": 0, + "size": 20, + "total_elements": 1, + "total_pages": 1 +} +``` + +`detector` is `EMAIL | CREDIT_CARD | SSN | IBAN | PHONE | AI`; `rationale` is populated for `AI` +findings only. `sample_redacted` is always a redacted value — raw sampled data is never stored. + +#### POST /datasources/{id}/discovery/findings/bulk-decision — Request Body + +```json +{ "finding_ids": ["3d2a…", "77b1…"], "decision": "CONFIRM" } +``` + +`decision` is `CONFIRM` | `DISMISS`; 1–100 ids per call. Each finding is decided independently +(partial success). Confirming applies the classification tag through the AF-447 service — which +auto-derives a masking policy for the column — and marks the finding `CONFIRMED`; dismissing marks +it `DISMISSED`, permanently suppressing the proposal on future scans. Response 200: + +```json +{ + "results": [ + { "finding_id": "3d2a…", "status": "SUCCESS", "new_status": "CONFIRMED" }, + { "finding_id": "77b1…", "status": "TAG_CONFLICT", "new_status": "CONFIRMED" } + ] +} +``` + +Row `status` values: `SUCCESS`, `NOT_FOUND` (unknown id in this datasource/org), +`INVALID_STATE` (finding already decided), `TAG_CONFLICT` (the tag already existed — e.g. added +manually since the scan; the finding is still marked `CONFIRMED` so the worklist clears, but no +new tag or masking is derived), `ERROR` (unexpected failure; the finding stays `PENDING`). + +#### POST /datasources/{id}/discovery/scan — Response 202 + +Triggers an immediate scan on a background virtual thread — allowed even when `enabled` is false +(ad-hoc preview before opting into the schedule). `409 DISCOVERY_SCAN_ALREADY_RUNNING` when a scan +for the datasource is already in flight; `404` for an unknown datasource. Scan runs are audited as +`DISCOVERY_SCAN_COMPLETED`; confirmations/dismissals as `DISCOVERY_FINDING_CONFIRMED` / +`DISCOVERY_FINDING_DISMISSED`. + +--- + ### GET /datasources/{id}/reviewers — Response 200 Admin-only. Returns per-datasource reviewer assignments. An empty list means the datasource falls back to plan approvers; a non-empty list **scopes** reviewer eligibility to the listed users plus the members of the listed groups (intersected with plan-approver rules). diff --git a/docs/05-backend.md b/docs/05-backend.md index 0ebef174..3cc01ec4 100644 --- a/docs/05-backend.md +++ b/docs/05-backend.md @@ -861,6 +861,46 @@ REST surface lives in the `security` module (`DataClassificationTagController`, the level can only rise, never drop below the LLM's verdict. The boosted score/level is what persists and drives the workflow router. +### Automated sensitive-data discovery (AF-623) + +The `discovery` module closes the AF-447 loop: instead of waiting for an admin to know a column is +sensitive, a scanner **finds** the sensitive columns and proposes the classification tags. It +depends only on `core.api`, `proxy.api`, `ai.api`, and `audit.api`. + +- **Scan pipeline** (`DiscoveryScanService.scan`, driven by `DiscoveryScanJob` per due + `discovery_scan_config` row or by the on-demand `POST /datasources/{id}/discovery/scan`): enumerate + tables via `DatasourceAdminService.introspectSchemaForSystem`, read a bounded raw sample per table + through `QueryExecutor.sampleTable` (the AF-443 path, so every engine — JDBC and plugin — is + covered; the per-table statement timeout is the tighter `accessflow.discovery.sample-statement-timeout`), + run the detector pipeline over each column's string values, and upsert findings. Raw sampled values + live only on the scan method's stack — findings persist a **redacted** sample only + (`ColumnMasker` `PARTIAL`, `visible_suffix=4`). Guards: `max-tables-per-scan` (default 200), a + wall-clock `scan-time-budget` (default `PT10M`; exceeding either flags the run `partial`), per-table + failures swallowed, a per-node in-flight set (cluster races are harmless — upserts are idempotent + against the natural-key unique index), and columns already covered by an enabled masking policy or + an existing tag are skipped. +- **Detectors** (`discovery.internal.detect`, pure classes): `EMAIL`→PII, `CREDIT_CARD` (13–19 + digits + Luhn)→PCI, `SSN` (US, never-issued ranges rejected)→PII, `IBAN` (per-country length + + mod-97)→FINANCIAL, `PHONE`→PII. First-match-wins per value in that order (checksum detectors + first, so a PAN never double-counts as a phone). A column needs ≥ 5 non-null string samples and a + ≥ 30 % match ratio to produce a proposal; `confidence` is the match percentage. +- **AI pass (opt-in, fail-safe).** When `ai_classification_enabled`, columns with no regex proposal + are sent — capped at `max-ai-tables-per-scan` (default 25) — to `ai.api.DataDiscoveryAiService`, + which calls the org's first usable `ai_config` through the analyzer holder's freeform lane with a + strict-JSON preamble and parses leniently (unknown columns/classifications dropped, any failure → + empty). **Only column names, types, and `FORMAT_PRESERVING`-redacted samples reach the provider** + — never raw values. Same fail-safe posture as the UBA anomaly summary: the AI pass can never block + or fail a scan. +- **Worklist.** Findings land as `PENDING` rows keyed by `(column, classification, detector)`; + rescans refresh `PENDING` rows in place and never touch decided ones. Confirming (bulk, per-row + independent transactions like the attestation bulk path) applies the tag through + `DataClassificationAdminService.create(..., applyMasking=true)` — deriving masking exactly like a + manual tag — and marks the finding `CONFIRMED` (a pre-existing tag reports `TAG_CONFLICT` but + still clears the worklist). Dismissing marks `DISMISSED`, permanently suppressing the proposal. + Audited as `DISCOVERY_SCAN_COMPLETED` / `DISCOVERY_FINDING_CONFIRMED` / `DISCOVERY_FINDING_DISMISSED`. +- **Limitations (v1).** Detectors examine scalar string cells only (nested NoSQL documents/maps are + skipped); stale `PENDING` findings whose data disappeared are kept for the admin to dismiss. + ### Compliance reporting (AF-459) The `compliance` module produces pre-built compliance reports and signed exports. It is a thin, @@ -1131,6 +1171,7 @@ This makes horizontal scaling safe: when the AccessFlow backend runs as multiple | `ErasureReviewTimeoutJob` | lifecycle | `erasureReviewTimeoutJob` | `accessflow.lifecycle.review-timeout-poll-interval` | `PT5M` | | `ScheduledGroupRunJob` | requestgroups | `scheduledGroupRunJob` | `accessflow.requestgroups.run-poll-interval` | `PT1M` | | `GroupTimeoutJob` | requestgroups | `groupTimeoutJob` | `accessflow.requestgroups.timeout-poll-interval` | `PT5M` | +| `DiscoveryScanJob` | discovery | `discoveryScanJob` | `accessflow.discovery.scan-poll-interval` | `PT15M` | `WeeklyDigestJob` implements the opt-in weekly dashboard digest (AF-498): it scans `dashboard_digest_subscription` for `enabled = true` rows whose `last_sent_at` is null or older than `accessflow.dashboard.weekly-digest.period` (default `P7D`, a partial index backs the scan) and, per row, builds that user's weekly summary, publishes a `dashboard.events.WeeklyDigestReadyEvent`, and stamps `last_sent_at`. The per-row build+publish+stamp runs inside `WeeklyDigestDispatchService.publishDigest` (`@Transactional`) so the event is published within a committed transaction — otherwise the notifications module's AFTER_COMMIT `@ApplicationModuleListener` would silently drop it. Per-row `RuntimeException`s are swallowed (`log.error`) so one bad subscription cannot abort the batch. The `notifications` module consumes the event and fans the summary out over the user's email + chat channels (`WEEKLY_DIGEST`); PagerDuty treats it as not-applicable (never pages). diff --git a/docs/06-frontend.md b/docs/06-frontend.md index fa1a94e1..f6e868dd 100644 --- a/docs/06-frontend.md +++ b/docs/06-frontend.md @@ -303,6 +303,7 @@ immediately. - `GrantAccessModal` includes a `can_break_glass` switch (after the read/write/DDL trio, default off, AF-385) and a `restricted_columns` multi-select populated from the datasource's introspected schema (`flattenSchemaToColumns` in `src/utils/schemaColumns.ts`). The break-glass help text explains it is an **additional** emergency capability that bypasses review but still requires the underlying read/write/DDL capability at submission time; the existing "at least one of read/write/DDL" rule is preserved and is **not** satisfied by break-glass alone. The restricted-columns help text explains that values are masked in results and the AI reviewer is informed but does not auto-reject. Permissions are create-only (revoke + re-grant); there is no edit flow. - **Masking** tab (`components/datasources/MaskingTab.tsx`, AF-381) — a table of dynamic data masking policies (column, strategy, reveal-to summary, enabled) with a create/edit modal. The modal picks a column via an `AutoComplete` from the introspected schema, a strategy `Select` driven by `enumOptions(MASKING_STRATEGIES, maskingStrategyLabel, t)`, a conditional `visible_suffix` field shown only for `PARTIAL`, and reveal-to multi-selects for roles (`enumOptions`), groups, and users. A **live preview** renders the masked output of an editable sample value through `src/utils/maskingPreview.ts` (a pure client-side mirror of the backend `ColumnMasker` strategies; `HASH` shows an illustrative fixed digest since the real SHA-256 is computed server-side). CRUD calls `src/api/maskingPolicies.ts`; validation parity matches the backend DTO (required column ≤ 512 chars, required strategy, `visible_suffix` 1–256). - **Row security** tab (`components/datasources/RowSecurityTab.tsx`, AF-380) — a table of row-level security policies (table, `column operator value` predicate, applies-to summary, enabled) with a create/edit modal. The structured form picks a table and column via `AutoComplete`s from the introspected schema, an operator `Select` (`enumOptions(ROW_SECURITY_OPERATORS, rowSecurityOperatorLabel, t)`), a value-source `Select` (`VARIABLE` | `LITERAL`), and a value field that switches to a `:user.*` variable `AutoComplete` (offering the `:user.id` / `:user.email` / `:user.role` / `:user.groups` built-ins) when the source is `VARIABLE`. Applies-to multi-selects target roles, groups, and users (empty = everyone). CRUD calls `src/api/rowSecurityPolicies.ts`; validation parity matches the backend DTO (required table/column/value ≤ 512 chars, required operator, required value source). +- **Discovery** tab (`components/datasources/DiscoveryTab.tsx`, AF-623) — the sensitive-data discovery worklist. A settings card edits the per-datasource config (`GET`/`PUT /datasources/{id}/discovery/config`: enable switch, sample size 10–1000, interval 1–720 h, AI-pass toggle — validation parity with the backend DTO) with a **Scan now** button (`POST …/discovery/scan`, 202) and the last-scan time / error. Below, a paginated findings table (`GET …/discovery/findings`, status `Segmented` filter defaulting to `PENDING`) shows column, proposed classification, detector (AI rationale as tooltip), confidence with match ratio, redacted sample, and status. Row selection surfaces a sticky bulk bar (the `AttestationWorklistPage` pattern) whose **Confirm/Dismiss selected** call `POST …/discovery/findings/bulk-decision` with partial-success handling — failed rows stay selected with a per-row status tag. Confirming invalidates the classification + masking query keys so the sibling tabs refresh (the tag + derived policy appear there). API module `src/api/discovery.ts` (`discoveryKeys`); the tab badge on `DatasourceSettingsPage` shows the PENDING-findings count. - Review plan assignment and row limit configuration ### AuditLogPage *(ADMIN)* diff --git a/docs/09-deployment.md b/docs/09-deployment.md index 81334fcb..7a853c29 100644 --- a/docs/09-deployment.md +++ b/docs/09-deployment.md @@ -1060,6 +1060,11 @@ Deployment-wide tuning for the `ai` module's `BehaviorAnomalyDetectionJob`, whic | `ACCESSFLOW_LIFECYCLE_REVIEW_TIMEOUT` | Optional | `PT168H` | ISO-8601 duration. How long an erasure request may sit in `PENDING_REVIEW` before `ErasureReviewTimeoutJob` auto-rejects it (AF-519). | | `ACCESSFLOW_REQUESTGROUPS_RUN_POLL_INTERVAL` | Optional | `PT1M` | ISO-8601 duration. Cadence at which `ScheduledGroupRunJob` (the `requestgroups` module, AF-501) executes `APPROVED` grouped requests whose `scheduled_for ≤ now()`, running their ordered member sequence. ShedLock-guarded. | | `ACCESSFLOW_REQUESTGROUPS_TIMEOUT_POLL_INTERVAL` | Optional | `PT5M` | ISO-8601 duration. Cadence at which `GroupTimeoutJob` (the `requestgroups` module, AF-501) auto-rejects grouped requests stuck in `PENDING_REVIEW` past the review timeout (`TIMED_OUT`). ShedLock-guarded. | +| `ACCESSFLOW_DISCOVERY_SCAN_POLL_INTERVAL` | Optional | `PT15M` | ISO-8601 duration. Cadence at which `DiscoveryScanJob` (the `discovery` module, AF-623) checks for datasources whose sensitive-data discovery scan is due (`enabled` config rows past their own `scan_interval_hours`). ShedLock-guarded. | +| `ACCESSFLOW_DISCOVERY_SCAN_TIME_BUDGET` | Optional | `PT10M` | ISO-8601 duration. Wall-clock budget for a single discovery scan; tables past the deadline are skipped and the run is flagged partial. | +| `ACCESSFLOW_DISCOVERY_SAMPLE_STATEMENT_TIMEOUT` | Optional | `PT10S` | ISO-8601 duration. Per-table statement timeout for the discovery scan's bounded sample read (tighter than the general execution timeout, like the AF-624 estimate bound). | +| `ACCESSFLOW_DISCOVERY_MAX_TABLES_PER_SCAN` | Optional | `200` | Hard cap on tables sampled in one discovery scan; beyond it the run is flagged partial. | +| `ACCESSFLOW_DISCOVERY_MAX_AI_TABLES_PER_SCAN` | Optional | `25` | Hard cap on tables sent through the optional discovery AI pass per scan, bounding AI-provider spend. | | `ACCESSFLOW_APIGOV_OAUTH2_TOKEN_CACHE_SKEW` | Optional | `PT30S` | ISO-8601 duration (#506). Safety skew subtracted from a fetched OAuth2 token's `expires_in` before caching it in Redis (`apigov:oauth2:token:`), so it refreshes a little before the upstream actually expires it. | | `ACCESSFLOW_APIGOV_OAUTH2_TOKEN_REQUEST_TIMEOUT` | Optional | `PT10S` | ISO-8601 duration (#506). Connect/read timeout for the `apigovOAuth2RestClient` that posts to a connector's OAuth2 token endpoint. | | `ACCESSFLOW_APIGOV_OAUTH2_TOKEN_FALLBACK_TTL` | Optional | `PT60S` | ISO-8601 duration (#506). Cache TTL used when a token response omits `expires_in`. | diff --git a/docs/12-roadmap.md b/docs/12-roadmap.md index 88d66747..c75654a7 100644 --- a/docs/12-roadmap.md +++ b/docs/12-roadmap.md @@ -181,6 +181,7 @@ - **High-volume proxy performance** — opt-in Redis-backed SELECT result caching keyed over the security-rewritten query (invalidated on any proxied write to a referenced table), JDBC `executeBatch()` for homogeneous INSERT runs inside `BEGIN…COMMIT` envelopes, and multi-endpoint read-replica load balancing with per-node circuit-breaker health checks and primary failover (AF-457) - **Backup / restore / DR tooling & offline AI fallback** — Helm backup CronJob (`pg_dump` to a kept PVC with retention pruning + optional rclone upload to S3/GCS/…) and a one-shot restore Job that preserves the audit-role ownership split; post-restore audit HMAC chain verification via `ACCESSFLOW_AUDIT_VERIFY_CHAIN_ON_STARTUP`; an AI provider **fallback pool** (`ai_config.fallback_priority`) so analysis degrades to a local Ollama model when the primary provider is unreachable; documented DR runbook (AF-458) - **ServiceNow & Jira ticketing integration** — two new ticketing notification-channel types that auto-create a ServiceNow incident / Jira issue when a query is rejected, escalated by a routing policy (new `QUERY_ESCALATED` event), or times out awaiting review; tickets link back onto the query detail page, and signed inbound status webhooks sync the ticket state — optionally mapping a ticket resolution onto an approve/reject of the still-pending query (bi-directional sync) (AF-453) +- **Automated sensitive-data discovery** — an opt-in per-datasource scanner samples column data through the governed sampling path, detects sensitive values with local regex + checksum detectors (email, credit-card PAN with Luhn, US SSN, IBAN, phone) and optionally the org's bound AI analyzer (column names, types, and redacted samples only), and proposes classification tags in an admin worklist — confirming applies the AF-447 tag (deriving masking), dismissing suppresses permanently; scheduled cadence + on-demand "Scan now", fully audited (AF-623) --- diff --git a/e2e/tests/discovery.spec.ts b/e2e/tests/discovery.spec.ts new file mode 100644 index 00000000..72ac4e02 --- /dev/null +++ b/e2e/tests/discovery.spec.ts @@ -0,0 +1,225 @@ +import { randomUUID } from 'node:crypto'; +import { expect, test, type APIRequestContext, type Page } from '@playwright/test'; +import { + acceptInvitationViaApi, + apiBase, + approveQueryViaApi, + createPostgresDatasource, + createReviewPlanViaApi, + deleteDatasource, + deleteReviewPlanViaApi, + executeQueryViaApi, + inviteUserViaApi, + loginViaApi, + purgeMailcrab, + submitQueryViaApi, + waitForInviteToken, + waitForQueryStatus, + type CreatedDatasource, + type CreatedReviewPlan, +} from '../helpers/datasources'; + +const ADMIN_EMAIL = 'e2e@accessflow.test'; +const ADMIN_PASSWORD = 'E2ePassword!123'; +const APPROVER_PASSWORD = 'ApproverPassword!123'; + +// Unique scratch table so re-runs against a long-lived stack DB never collide. +const TABLE = `discovery_e2e_${Date.now()}`; + +interface DiscoveryFindingRow { + id: string; + schema_name: string | null; + table_name: string; + column_name: string; + classification: string; + detector: string; + status: string; +} + +async function loginViaUi(page: Page, email: string, password: string): Promise { + await page.goto('/login'); + await page.locator('#login-email').fill(email); + await page.locator('#login-password').fill(password); + await page.locator('button[type="submit"]').click(); + await page.waitForURL('**/dashboard', { timeout: 15_000 }); +} + +// The scan runs asynchronously after "Scan now"; poll the findings API until the +// seeded email column surfaces as a PENDING EMAIL/PII finding. +async function waitForEmailFinding( + request: APIRequestContext, + token: string, + datasourceId: string, +): Promise { + const deadline = Date.now() + 60_000; + for (;;) { + const res = await request.get( + `${apiBase()}/api/v1/datasources/${datasourceId}/discovery/findings?status=PENDING&size=100`, + { headers: { Authorization: `Bearer ${token}` } }, + ); + if (res.ok()) { + const body = (await res.json()) as { content: DiscoveryFindingRow[] }; + const match = body.content.find( + (f) => f.table_name === TABLE && f.column_name === 'customer_email', + ); + if (match) return match; + } + if (Date.now() > deadline) { + throw new Error(`No EMAIL finding for ${TABLE}.customer_email within 60s`); + } + await new Promise((resolve) => setTimeout(resolve, 2_000)); + } +} + +test.describe.configure({ timeout: 120_000 }); + +test.describe.serial('sensitive-data discovery (AF-623)', () => { + let adminAccessToken = ''; + let approverAccessToken = ''; + let reviewPlan: CreatedReviewPlan | null = null; + let datasource: CreatedDatasource | null = null; + + async function runApproved(request: APIRequestContext, sql: string): Promise { + if (!datasource) throw new Error('datasource not created in beforeAll'); + const submitted = await submitQueryViaApi( + request, + adminAccessToken, + datasource.id, + sql, + 'AF-623 discovery seed data', + ); + await waitForQueryStatus(request, adminAccessToken, submitted.id, 'PENDING_REVIEW'); + await approveQueryViaApi(request, approverAccessToken, submitted.id); + const exec = await executeQueryViaApi(request, adminAccessToken, submitted.id); + expect(exec.status).toBe('EXECUTED'); + } + + test.beforeAll(async ({ request }) => { + adminAccessToken = await loginViaApi(request, ADMIN_EMAIL, ADMIN_PASSWORD); + + // Second user with reviewer authority — self-approval is rejected. + const approverEmail = `approver-${randomUUID()}@e2e.local`; + await purgeMailcrab(request); + await inviteUserViaApi(request, adminAccessToken, approverEmail, 'AF-623 Approver', 'ADMIN'); + const inviteToken = await waitForInviteToken(request, approverEmail); + await acceptInvitationViaApi(request, inviteToken, APPROVER_PASSWORD, 'AF-623 Approver'); + approverAccessToken = await loginViaApi(request, approverEmail, APPROVER_PASSWORD); + + reviewPlan = await createReviewPlanViaApi(request, adminAccessToken, { + name: `E2E Review Plan AF623 ${Date.now()}`, + approvers: [{ role: 'ADMIN', stage: 1 }], + minApprovalsRequired: 1, + }); + + datasource = await createPostgresDatasource(request, adminAccessToken, { + name: `Postgres E2E Discovery ${Date.now()}`, + reviewPlanId: reviewPlan.id, + }); + + // Seed a table whose email column the scan must flag: the detectors need at + // least 5 non-null string samples, so insert 6 rows. + await runApproved( + request, + `CREATE TABLE ${TABLE} (id integer PRIMARY KEY, customer_email text NOT NULL)`, + ); + await runApproved( + request, + `INSERT INTO ${TABLE} (id, customer_email) VALUES ` + + "(1, 'alice@example.com'), (2, 'bob@example.com'), (3, 'carol@example.com'), " + + "(4, 'dave@example.com'), (5, 'erin@example.com'), (6, 'frank@example.com')", + ); + }); + + test.afterAll(async ({ request }) => { + // Best-effort teardown; uniquely-named objects don't break re-runs anyway. + try { + await runApproved(request, `DROP TABLE IF EXISTS ${TABLE}`); + } catch { + // ignore + } + if (datasource) { + await deleteDatasource(request, adminAccessToken, datasource.id); + } + if (reviewPlan) { + try { + await deleteReviewPlanViaApi(request, adminAccessToken, reviewPlan.id); + } catch { + // plan may still be referenced; ignore + } + } + }); + + test('enable discovery, scan now, and see the proposed finding', async ({ page, request }) => { + if (!datasource) throw new Error('datasource not created in beforeAll'); + + await loginViaUi(page, ADMIN_EMAIL, ADMIN_PASSWORD); + await page.goto(`/datasources/${datasource.id}/settings`); + await page.getByRole('tab', { name: /Discovery/ }).click(); + + // Enable scheduled discovery and save. + await expect(page.getByText('Discovery settings')).toBeVisible({ timeout: 15_000 }); + await page.getByRole('switch').first().click(); + await page.getByRole('button', { name: 'Save' }).click(); + await expect(page.getByText('Discovery settings saved')).toBeVisible({ timeout: 10_000 }); + + // Kick an immediate scan and wait (via API) for the seeded column to surface. + await page.getByRole('button', { name: 'Scan now' }).click(); + await expect(page.getByText('Discovery scan started')).toBeVisible({ timeout: 10_000 }); + const finding = await waitForEmailFinding(request, adminAccessToken, datasource.id); + expect(finding.classification).toBe('PII'); + expect(finding.detector).toBe('EMAIL'); + + // The worklist shows the finding after a reload. + await page.goto(`/datasources/${datasource.id}/settings`); + await page.getByRole('tab', { name: /Discovery/ }).click(); + await expect( + page.getByText(`public.${TABLE}.customer_email`).first(), + ).toBeVisible({ timeout: 15_000 }); + }); + + test('bulk-confirming the finding applies the tag and derives masking', async ({ + page, + request, + }) => { + if (!datasource) throw new Error('datasource not created in beforeAll'); + + await loginViaUi(page, ADMIN_EMAIL, ADMIN_PASSWORD); + await page.goto(`/datasources/${datasource.id}/settings`); + await page.getByRole('tab', { name: /Discovery/ }).click(); + + const row = page.getByRole('row', { name: new RegExp(`public\\.${TABLE}\\.customer_email`) }); + await expect(row).toBeVisible({ timeout: 15_000 }); + await row.getByRole('checkbox').check(); + + await page.getByRole('button', { name: 'Confirm selected' }).click(); + await expect(page.getByText('1 finding decided')).toBeVisible({ timeout: 15_000 }); + + // The confirmed finding leaves the PENDING filter. + await expect( + page.getByRole('row', { name: new RegExp(`public\\.${TABLE}\\.customer_email`) }), + ).toHaveCount(0); + + // Confirming applied the AF-447 tag… + await page.getByRole('tab', { name: /Classification/ }).click(); + await expect( + page.getByText(`public.${TABLE}.customer_email`).first(), + ).toBeVisible({ timeout: 15_000 }); + + // …and auto-derived a masking policy for the column. + await page.getByRole('tab', { name: /Masking/ }).click(); + await expect( + page.getByText(`public.${TABLE}.customer_email`).first(), + ).toBeVisible({ timeout: 15_000 }); + + // API cross-check: the finding is now CONFIRMED. + const res = await request.get( + `${apiBase()}/api/v1/datasources/${datasource.id}/discovery/findings?status=CONFIRMED&size=100`, + { headers: { Authorization: `Bearer ${adminAccessToken}` } }, + ); + expect(res.ok()).toBe(true); + const body = (await res.json()) as { content: DiscoveryFindingRow[] }; + expect( + body.content.some((f) => f.table_name === TABLE && f.column_name === 'customer_email'), + ).toBe(true); + }); +}); diff --git a/frontend/src/api/discovery.ts b/frontend/src/api/discovery.ts new file mode 100644 index 00000000..f1e6e79a --- /dev/null +++ b/frontend/src/api/discovery.ts @@ -0,0 +1,72 @@ +import { apiClient } from './client'; +import type { + BulkDiscoveryDecisionInput, + BulkDiscoveryDecisionResult, + DiscoveryFindingPage, + DiscoveryFindingStatus, + DiscoveryScanConfig, + UpdateDiscoveryConfigInput, +} from '@/types/api'; + +const base = (datasourceId: string) => `/api/v1/datasources/${datasourceId}/discovery`; + +export interface DiscoveryFindingFilters { + status?: DiscoveryFindingStatus; + page?: number; + size?: number; +} + +export const discoveryKeys = { + all: ['discovery'] as const, + config: (datasourceId: string) => ['discovery', 'config', datasourceId] as const, + findings: (datasourceId: string, filters: DiscoveryFindingFilters = {}) => + ['discovery', 'findings', datasourceId, filters] as const, +}; + +export async function fetchDiscoveryConfig(datasourceId: string): Promise { + const { data } = await apiClient.get(`${base(datasourceId)}/config`); + return data; +} + +export async function updateDiscoveryConfig( + datasourceId: string, + input: UpdateDiscoveryConfigInput, +): Promise { + const { data } = await apiClient.put(`${base(datasourceId)}/config`, input); + return data; +} + +export async function fetchDiscoveryFindings( + datasourceId: string, + filters: DiscoveryFindingFilters = {}, +): Promise { + const params: Record = {}; + if (filters.status) { + params.status = filters.status; + } + if (typeof filters.page === 'number') { + params.page = filters.page; + } + if (typeof filters.size === 'number') { + params.size = filters.size; + } + const { data } = await apiClient.get(`${base(datasourceId)}/findings`, { + params, + }); + return data; +} + +export async function bulkDecideFindings( + datasourceId: string, + input: BulkDiscoveryDecisionInput, +): Promise { + const { data } = await apiClient.post( + `${base(datasourceId)}/findings/bulk-decision`, + input, + ); + return data; +} + +export async function triggerDiscoveryScan(datasourceId: string): Promise { + await apiClient.post(`${base(datasourceId)}/scan`); +} diff --git a/frontend/src/components/datasources/DiscoveryTab.test.tsx b/frontend/src/components/datasources/DiscoveryTab.test.tsx new file mode 100644 index 00000000..2c407fd9 --- /dev/null +++ b/frontend/src/components/datasources/DiscoveryTab.test.tsx @@ -0,0 +1,179 @@ +import { describe, expect, it, vi, beforeEach } from 'vitest'; +import { fireEvent, render, screen, waitFor } from '@testing-library/react'; +import { App as AntdApp } from 'antd'; +import { QueryClient, QueryClientProvider } from '@tanstack/react-query'; +import '@/i18n'; +import type { DiscoveryFinding, DiscoveryScanConfig } from '@/types/api'; + +const fetchDiscoveryConfig = vi.fn(); +const updateDiscoveryConfig = vi.fn(); +const fetchDiscoveryFindings = vi.fn(); +const bulkDecideFindings = vi.fn(); +const triggerDiscoveryScan = vi.fn(); + +vi.mock('@/api/discovery', async (importOriginal) => { + const original = await importOriginal(); + return { + discoveryKeys: original.discoveryKeys, + fetchDiscoveryConfig: (...args: unknown[]) => fetchDiscoveryConfig(...args), + updateDiscoveryConfig: (...args: unknown[]) => updateDiscoveryConfig(...args), + fetchDiscoveryFindings: (...args: unknown[]) => fetchDiscoveryFindings(...args), + bulkDecideFindings: (...args: unknown[]) => bulkDecideFindings(...args), + triggerDiscoveryScan: (...args: unknown[]) => triggerDiscoveryScan(...args), + }; +}); + +const { DiscoveryTab } = await import('./DiscoveryTab'); + +const config: DiscoveryScanConfig = { + datasource_id: 'ds-1', + enabled: false, + sample_size: 100, + scan_interval_hours: 24, + ai_classification_enabled: false, + last_scan_at: '2026-07-27T02:00:00Z', + last_scan_error: null, +}; + +const finding: DiscoveryFinding = { + id: 'f-1', + schema_name: 'public', + table_name: 'users', + column_name: 'email', + classification: 'PII', + detector: 'EMAIL', + confidence: 96, + sample_redacted: '*************.com', + rationale: null, + match_count: 48, + sample_count: 50, + status: 'PENDING', + first_detected_at: '2026-07-20T02:00:00Z', + last_detected_at: '2026-07-27T02:00:00Z', + decided_by: null, + decided_at: null, +}; + +function page(content: DiscoveryFinding[]) { + return { + content, + page: 0, + size: 20, + total_elements: content.length, + total_pages: 1, + }; +} + +function renderTab() { + const client = new QueryClient({ defaultOptions: { queries: { retry: false } } }); + return render( + + + + + , + ); +} + +describe('DiscoveryTab', () => { + beforeEach(() => { + fetchDiscoveryConfig.mockReset().mockResolvedValue(config); + fetchDiscoveryFindings.mockReset().mockResolvedValue(page([finding])); + updateDiscoveryConfig.mockReset(); + bulkDecideFindings.mockReset(); + triggerDiscoveryScan.mockReset().mockResolvedValue(undefined); + }); + + it('renders the settings card with config values and the findings table', async () => { + renderTab(); + + expect(await screen.findByText('Discovery settings')).toBeInTheDocument(); + expect(await screen.findByText('public.users.email')).toBeInTheDocument(); + expect(screen.getByText('PII')).toBeInTheDocument(); + expect(screen.getByText('*************.com')).toBeInTheDocument(); + expect(screen.getByText(/96%/)).toBeInTheDocument(); + expect(fetchDiscoveryFindings).toHaveBeenCalledWith('ds-1', { + status: 'PENDING', + page: 0, + size: 20, + }); + }); + + it('saves the settings form', async () => { + updateDiscoveryConfig.mockResolvedValue({ ...config, enabled: true }); + renderTab(); + + const saveButton = await screen.findByRole('button', { name: /Save/ }); + fireEvent.click(screen.getAllByRole('switch')[0]!); + fireEvent.click(saveButton); + + await waitFor(() => expect(updateDiscoveryConfig).toHaveBeenCalled()); + expect(updateDiscoveryConfig.mock.calls[0]![0]).toBe('ds-1'); + expect(await screen.findByText('Discovery settings saved')).toBeInTheDocument(); + }); + + it('triggers an on-demand scan', async () => { + renderTab(); + + fireEvent.click(await screen.findByRole('button', { name: /Scan now/ })); + + await waitFor(() => expect(triggerDiscoveryScan).toHaveBeenCalledWith('ds-1')); + expect(await screen.findByText('Discovery scan started')).toBeInTheDocument(); + }); + + it('bulk-confirms selected findings and reports the summary', async () => { + bulkDecideFindings.mockResolvedValue({ + results: [{ finding_id: 'f-1', status: 'SUCCESS', new_status: 'CONFIRMED' }], + }); + renderTab(); + await screen.findByText('public.users.email'); + + // Select the row via its checkbox, then confirm from the bulk bar. + const checkboxes = screen.getAllByRole('checkbox'); + fireEvent.click(checkboxes[checkboxes.length - 1]!); + fireEvent.click(await screen.findByRole('button', { name: /Confirm selected/ })); + + await waitFor(() => expect(bulkDecideFindings).toHaveBeenCalled()); + expect(bulkDecideFindings.mock.calls[0]![0]).toBe('ds-1'); + expect(bulkDecideFindings.mock.calls[0]![1]).toEqual({ + finding_ids: ['f-1'], + decision: 'CONFIRM', + }); + expect(await screen.findByText('1 finding decided')).toBeInTheDocument(); + }); + + it('keeps failed rows selected and shows the partial toast on mixed outcomes', async () => { + const second: DiscoveryFinding = { ...finding, id: 'f-2', column_name: 'phone' }; + fetchDiscoveryFindings.mockResolvedValue(page([finding, second])); + bulkDecideFindings.mockResolvedValue({ + results: [ + { finding_id: 'f-1', status: 'SUCCESS', new_status: 'CONFIRMED' }, + { finding_id: 'f-2', status: 'INVALID_STATE', new_status: 'DISMISSED' }, + ], + }); + renderTab(); + await screen.findByText('public.users.email'); + + // "Select all" header checkbox. + fireEvent.click(screen.getAllByRole('checkbox')[0]!); + fireEvent.click(await screen.findByRole('button', { name: /Confirm selected/ })); + + await waitFor(() => expect(bulkDecideFindings).toHaveBeenCalled()); + expect(await screen.findByText(/1 decided · 1 failed/)).toBeInTheDocument(); + expect(await screen.findByText('Already decided')).toBeInTheDocument(); + }); + + it('shows the last-scan error alert when present', async () => { + fetchDiscoveryConfig.mockResolvedValue({ ...config, last_scan_error: 'partial: budget' }); + renderTab(); + + expect(await screen.findByText(/partial: budget/)).toBeInTheDocument(); + }); + + it('shows the empty state when there are no findings', async () => { + fetchDiscoveryFindings.mockResolvedValue(page([])); + renderTab(); + + expect(await screen.findByText('No findings')).toBeInTheDocument(); + }); +}); diff --git a/frontend/src/components/datasources/DiscoveryTab.tsx b/frontend/src/components/datasources/DiscoveryTab.tsx new file mode 100644 index 00000000..e5e01b00 --- /dev/null +++ b/frontend/src/components/datasources/DiscoveryTab.tsx @@ -0,0 +1,483 @@ +import { useMemo, useState } from 'react'; +import { + Alert, + App, + Button, + Card, + Form, + InputNumber, + Segmented, + Switch, + Table, + Tag, + Tooltip, +} from 'antd'; +import type { TableColumnsType } from 'antd'; +import { CheckOutlined, CloseOutlined, RadarChartOutlined } from '@ant-design/icons'; +import { useMutation, useQuery, useQueryClient } from '@tanstack/react-query'; +import { useTranslation } from 'react-i18next'; +import { EmptyState } from '@/components/common/EmptyState'; +import { + bulkDecideFindings, + discoveryKeys, + fetchDiscoveryConfig, + fetchDiscoveryFindings, + triggerDiscoveryScan, + updateDiscoveryConfig, +} from '@/api/discovery'; +import { dataClassificationKeys } from '@/api/dataClassifications'; +import { maskingPolicyKeys } from '@/api/maskingPolicies'; +import { + dataClassificationLabel, + discoveryDetectorLabel, + discoveryFindingStatusLabel, +} from '@/utils/enumLabels'; +import { fmtDate, timeAgo } from '@/utils/dateFormat'; +import { apiErrorMessage } from '@/utils/apiErrors'; +import { showApiError } from '@/utils/showApiError'; +import type { + DataClassification, + DiscoveryDecision, + DiscoveryDetector, + DiscoveryFinding, + DiscoveryFindingStatus, + DiscoveryRowStatus, + UpdateDiscoveryConfigInput, +} from '@/types/api'; + +interface ConfigFormValues { + enabled: boolean; + sample_size: number; + scan_interval_hours: number; + ai_classification_enabled: boolean; +} + +const PAGE_SIZE = 20; + +export function DiscoveryTab({ dsId }: { dsId: string }) { + const { t } = useTranslation(); + const { message } = App.useApp(); + const queryClient = useQueryClient(); + const [form] = Form.useForm(); + + const [statusFilter, setStatusFilter] = useState('PENDING'); + const [page, setPage] = useState(0); + const [selectedRowKeys, setSelectedRowKeys] = useState([]); + const [rowStatuses, setRowStatuses] = useState>({}); + + const configQuery = useQuery({ + queryKey: discoveryKeys.config(dsId), + queryFn: () => fetchDiscoveryConfig(dsId), + }); + + const findingFilters = { + status: statusFilter === 'ALL' ? undefined : statusFilter, + page, + size: PAGE_SIZE, + }; + const findingsQuery = useQuery({ + queryKey: discoveryKeys.findings(dsId, findingFilters), + queryFn: () => fetchDiscoveryFindings(dsId, findingFilters), + }); + const findings = findingsQuery.data?.content ?? []; + + const invalidateFindings = () => { + void queryClient.invalidateQueries({ queryKey: discoveryKeys.all }); + }; + + const saveConfig = useMutation({ + mutationFn: (input: UpdateDiscoveryConfigInput) => updateDiscoveryConfig(dsId, input), + onSuccess: (config) => { + queryClient.setQueryData(discoveryKeys.config(dsId), config); + message.success(t('datasources.settings.discovery.config_save_success')); + }, + onError: (err) => { + showApiError(message, err, (e) => + apiErrorMessage(e, () => t('datasources.settings.discovery.config_save_error')), + ); + }, + }); + + const scanNow = useMutation({ + mutationFn: () => triggerDiscoveryScan(dsId), + onSuccess: () => { + message.success(t('datasources.settings.discovery.scan_started')); + // The scan runs asynchronously; refresh the config (last_scan_at) and findings shortly. + window.setTimeout(() => { + void queryClient.invalidateQueries({ queryKey: discoveryKeys.all }); + }, 5000); + }, + onError: (err) => { + showApiError(message, err, (e) => + apiErrorMessage(e, () => t('datasources.settings.discovery.scan_error')), + ); + }, + }); + + const bulk = useMutation({ + mutationFn: (decision: DiscoveryDecision) => + bulkDecideFindings(dsId, { finding_ids: selectedRowKeys, decision }), + onSuccess: (response, decision) => { + const failures: Record = {}; + let success = 0; + for (const row of response.results) { + if (row.status === 'SUCCESS' || row.status === 'TAG_CONFLICT') { + success += 1; + } else { + failures[row.finding_id] = row.status; + } + } + const failure = Object.keys(failures).length; + // Keep failed rows selected so the user can retry; drop the successes. + setSelectedRowKeys((prev) => prev.filter((id) => failures[id] !== undefined)); + setRowStatuses((prev) => ({ ...prev, ...failures })); + invalidateFindings(); + if (decision === 'CONFIRM') { + // Confirming applies tags and derives masking — refresh the sibling tabs. + void queryClient.invalidateQueries({ queryKey: dataClassificationKeys.list(dsId) }); + void queryClient.invalidateQueries({ queryKey: dataClassificationKeys.derivation(dsId) }); + void queryClient.invalidateQueries({ queryKey: maskingPolicyKeys.list(dsId) }); + } + if (failure === 0) { + message.success( + t('datasources.settings.discovery.bulk_toast_summary', { count: success }), + ); + } else { + message.warning( + t('datasources.settings.discovery.bulk_toast_partial', { success, failure }), + ); + } + }, + onError: (err) => { + showApiError(message, err, (e) => + apiErrorMessage(e, () => t('datasources.settings.discovery.bulk_error')), + ); + }, + }); + + const rowStatusTag = (status: DiscoveryRowStatus) => { + const labelKey = + status === 'INVALID_STATE' + ? 'datasources.settings.discovery.row_status_invalid_state' + : status === 'NOT_FOUND' + ? 'datasources.settings.discovery.row_status_not_found' + : 'datasources.settings.discovery.row_status_error'; + return {t(labelKey)}; + }; + + const columns: TableColumnsType = useMemo( + () => [ + { + title: t('datasources.settings.discovery.col_column'), + key: 'column', + render: (_: unknown, f: DiscoveryFinding) => ( + + {(f.schema_name ? `${f.schema_name}.` : '') + `${f.table_name}.${f.column_name}`} + + ), + }, + { + title: t('datasources.settings.discovery.col_classification'), + dataIndex: 'classification', + width: 130, + render: (v: DataClassification) => ( + {dataClassificationLabel(t, v)} + ), + }, + { + title: t('datasources.settings.discovery.col_detector'), + dataIndex: 'detector', + width: 130, + render: (v: DiscoveryDetector, f: DiscoveryFinding) => + f.rationale ? ( + + {discoveryDetectorLabel(t, v)} + + ) : ( + {discoveryDetectorLabel(t, v)} + ), + }, + { + title: t('datasources.settings.discovery.col_confidence'), + dataIndex: 'confidence', + width: 120, + render: (v: number, f: DiscoveryFinding) => ( + + {t('datasources.settings.discovery.confidence_value', { confidence: v })} + {f.match_count > 0 && ( + + {' '} + ({f.match_count}/{f.sample_count}) + + )} + + ), + }, + { + title: t('datasources.settings.discovery.col_sample'), + dataIndex: 'sample_redacted', + render: (v: string | null) => + v ? ( + + {v} + + ) : ( + — + ), + }, + { + title: t('datasources.settings.discovery.col_detected'), + dataIndex: 'last_detected_at', + width: 140, + render: (v: string) => ( + + + {timeAgo(v)} + + + ), + }, + { + title: t('datasources.settings.discovery.col_status'), + dataIndex: 'status', + width: 130, + render: (v: DiscoveryFindingStatus, f: DiscoveryFinding) => { + const s = rowStatuses[f.id]; + if (s) return rowStatusTag(s); + const color = v === 'PENDING' ? 'processing' : v === 'CONFIRMED' ? 'success' : 'default'; + return {discoveryFindingStatusLabel(t, v)}; + }, + }, + ], + // eslint-disable-next-line react-hooks/exhaustive-deps + [t, rowStatuses], + ); + + const config = configQuery.data; + const pendingSelectable = statusFilter === 'PENDING' || statusFilter === 'ALL'; + + return ( +

+ +
+ {t('datasources.settings.discovery.settings_description')} +
+ {config && ( + + form={form} + layout="inline" + initialValues={{ + enabled: config.enabled, + sample_size: config.sample_size, + scan_interval_hours: config.scan_interval_hours, + ai_classification_enabled: config.ai_classification_enabled, + }} + onFinish={(values) => saveConfig.mutate(values)} + style={{ rowGap: 12 }} + > + + + + + + + + + + + + + + + + + + + + )} + {config?.last_scan_at && ( +
+ {t('datasources.settings.discovery.last_scan', { + time: fmtDate(config.last_scan_at), + })} +
+ )} + {config?.last_scan_error && ( + + )} + + +
+
+
+
+ {t('datasources.settings.discovery.findings_title')} +
+
+ {t('datasources.settings.discovery.findings_description')} +
+
+ + value={statusFilter} + onChange={(value) => { + setStatusFilter(value); + setPage(0); + setSelectedRowKeys([]); + }} + options={[ + { + value: 'PENDING', + label: discoveryFindingStatusLabel(t, 'PENDING'), + }, + { + value: 'CONFIRMED', + label: discoveryFindingStatusLabel(t, 'CONFIRMED'), + }, + { + value: 'DISMISSED', + label: discoveryFindingStatusLabel(t, 'DISMISSED'), + }, + { value: 'ALL', label: t('datasources.settings.discovery.filter_all') }, + ]} + /> +
+ + {selectedRowKeys.length > 0 && ( +
+ + {t('datasources.settings.discovery.bulk_action_bar_count', { + count: selectedRowKeys.length, + })} + +
+ + + +
+ )} + + {!findingsQuery.isLoading && findings.length === 0 ? ( + + ) : ( + + rowKey="id" + size="middle" + loading={findingsQuery.isLoading} + dataSource={findings} + columns={columns} + scroll={{ x: 'max-content' }} + pagination={{ + current: page + 1, + pageSize: PAGE_SIZE, + total: findingsQuery.data?.total_elements ?? 0, + showSizeChanger: false, + onChange: (nextPage) => setPage(nextPage - 1), + }} + rowSelection={ + pendingSelectable + ? { + selectedRowKeys, + onChange: (keys) => setSelectedRowKeys(keys as string[]), + getCheckboxProps: (f) => ({ disabled: f.status !== 'PENDING' }), + } + : undefined + } + /> + )} +
+
+ ); +} diff --git a/frontend/src/locales/de.json b/frontend/src/locales/de.json index 5f0f0900..ac321575 100644 --- a/frontend/src/locales/de.json +++ b/frontend/src/locales/de.json @@ -1080,6 +1080,7 @@ }, "tab_row_security": "Zeilensicherheit · {{count}}", "tab_classification": "Klassifizierung · {{count}}", + "tab_discovery": "Discovery · {{count}}", "row_security": { "title": "Zeilensicherheitsrichtlinien", "description": "Filtert die Zeilen, die ein Benutzer in einer Tabelle sehen (SELECT) oder ändern (UPDATE/DELETE) darf. Jede Richtlinie vergleicht eine Spalte mit einem benutzerspezifischen Wert; Benutzer im Geltungsbereich sehen nur autorisierte Zeilen.", @@ -1202,7 +1203,50 @@ "cache_enabled_help": "Wiederholte identische SELECTs aus einem Redis-Cache bedienen. Einträge werden bei jedem Schreibzugriff auf eine referenzierte Tabelle invalidiert; Maskierung und Zeilensicherheit gelten weiterhin.", "label_cache_ttl": "Cache-TTL (Sekunden)", "cache_ttl_help": "Wie lange ein zwischengespeichertes Ergebnis gültig bleibt (1–86.400 Sekunden). Leer lassen für den Bereitstellungsstandard.", - "cache_ttl_placeholder": "Bereitstellungsstandard" + "cache_ttl_placeholder": "Bereitstellungsstandard", + "discovery": { + "settings_title": "Discovery-Einstellungen", + "settings_description": "Tastet Spaltendaten regelmäßig über den kontrollierten Sampling-Pfad ab, erkennt sensible Werte mit lokalen Mustererkennern (und optional Ihrem KI-Analyzer) und schlägt Klassifizierungs-Tags zur Prüfung vor. Rohe Stichprobenwerte werden nie gespeichert und nie an den KI-Anbieter gesendet.", + "label_enabled": "Geplante Scans", + "label_sample_size": "Stichprobengröße", + "label_scan_interval": "Intervall (Stunden)", + "label_ai": "KI-Klassifizierung", + "ai_hint": "Bittet zusätzlich den KI-Analyzer der Organisation, Spalten nur anhand von Namen, Typen und geschwärzten Stichproben zu klassifizieren.", + "sample_size_range": "Die Stichprobengröße muss zwischen 10 und 1000 liegen", + "scan_interval_range": "Das Scan-Intervall muss zwischen 1 und 720 Stunden liegen", + "scan_now": "Jetzt scannen", + "scan_started": "Discovery-Scan gestartet", + "scan_error": "Discovery-Scan konnte nicht gestartet werden", + "config_save_success": "Discovery-Einstellungen gespeichert", + "config_save_error": "Discovery-Einstellungen konnten nicht gespeichert werden", + "last_scan": "Letzter Scan: {{time}}", + "last_scan_error": "Problem beim letzten Scan: {{error}}", + "findings_title": "Funde", + "findings_description": "Vorgeschlagene Klassifizierungen sensibler Spalten. Bestätigen wendet das Tag an und leitet eine Maskierungsrichtlinie ab; Verwerfen unterdrückt den Vorschlag dauerhaft.", + "filter_all": "Alle", + "col_column": "Spalte", + "col_classification": "Klassifizierung", + "col_detector": "Detektor", + "col_confidence": "Konfidenz", + "confidence_value": "{{confidence}}%", + "col_sample": "Stichprobe (geschwärzt)", + "col_detected": "Erkannt", + "col_status": "Status", + "empty_title": "Keine Funde", + "empty_description": "Aktivieren Sie Discovery oder starten Sie einen Scan, um Klassifizierungen für sensible Spalten vorzuschlagen.", + "confirm_selected": "Auswahl bestätigen", + "dismiss_selected": "Auswahl verwerfen", + "clear_selection": "Auswahl aufheben", + "bulk_toast_partial": "{{success}} entschieden · {{failure}} fehlgeschlagen", + "bulk_error": "Massenentscheidung fehlgeschlagen", + "row_status_invalid_state": "Bereits entschieden", + "row_status_not_found": "Nicht gefunden", + "row_status_error": "Fehlgeschlagen", + "bulk_action_bar_count_one": "{{count}} ausgewählt", + "bulk_action_bar_count_other": "{{count}} ausgewählt", + "bulk_toast_summary_one": "{{count}} Fund entschieden", + "bulk_toast_summary_other": "{{count}} Funde entschieden" + } } }, "admin": { @@ -2634,6 +2678,19 @@ "HEX": "Hexadezimal", "BASE64": "Base64", "BASE64URL": "Base64 (URL-sicher, ohne Padding)" + }, + "discovery_detector": { + "EMAIL": "E-Mail", + "CREDIT_CARD": "Kreditkarte", + "SSN": "SSN", + "IBAN": "IBAN", + "PHONE": "Telefon", + "AI": "KI" + }, + "discovery_finding_status": { + "PENDING": "Ausstehend", + "CONFIRMED": "Bestätigt", + "DISMISSED": "Verworfen" } }, "access": { diff --git a/frontend/src/locales/en.json b/frontend/src/locales/en.json index 746ec2c8..38e9c113 100644 --- a/frontend/src/locales/en.json +++ b/frontend/src/locales/en.json @@ -1080,6 +1080,7 @@ "back_to_list": "Back to list", "tab_row_security": "Row security · {{count}}", "tab_classification": "Classification · {{count}}", + "tab_discovery": "Discovery · {{count}}", "row_security": { "title": "Row security policies", "description": "Filter the rows a user can see (SELECT) or change (UPDATE/DELETE) on a table. Each policy compares a column against a per-user value; users in scope only see authorised rows.", @@ -1202,7 +1203,50 @@ "cache_enabled_help": "Serve repeated identical SELECTs from a Redis cache. Entries are invalidated on any write to a referenced table; masking and row-security still apply.", "label_cache_ttl": "Cache TTL (seconds)", "cache_ttl_help": "How long a cached result stays valid (1–86,400 seconds). Leave blank for the deployment default.", - "cache_ttl_placeholder": "Deployment default" + "cache_ttl_placeholder": "Deployment default", + "discovery": { + "settings_title": "Discovery settings", + "settings_description": "Periodically samples column data through the governed sampling path, detects sensitive values with local pattern detectors (and optionally your AI analyzer), and proposes classification tags for review. Raw sampled values are never stored and never sent to the AI provider.", + "label_enabled": "Scheduled scans", + "label_sample_size": "Sample size", + "label_scan_interval": "Interval (hours)", + "label_ai": "AI classification", + "ai_hint": "Additionally asks the organization's AI analyzer to classify columns using names, types, and redacted samples only.", + "sample_size_range": "Sample size must be between 10 and 1000", + "scan_interval_range": "Scan interval must be between 1 and 720 hours", + "scan_now": "Scan now", + "scan_started": "Discovery scan started", + "scan_error": "Failed to start discovery scan", + "config_save_success": "Discovery settings saved", + "config_save_error": "Failed to save discovery settings", + "last_scan": "Last scan: {{time}}", + "last_scan_error": "Last scan issue: {{error}}", + "findings_title": "Findings", + "findings_description": "Proposed sensitive-column classifications. Confirming applies the tag and derives a masking policy; dismissing suppresses the proposal permanently.", + "filter_all": "All", + "col_column": "Column", + "col_classification": "Classification", + "col_detector": "Detector", + "col_confidence": "Confidence", + "confidence_value": "{{confidence}}%", + "col_sample": "Sample (redacted)", + "col_detected": "Detected", + "col_status": "Status", + "empty_title": "No findings", + "empty_description": "Enable discovery or run a scan to propose classifications for sensitive columns.", + "confirm_selected": "Confirm selected", + "dismiss_selected": "Dismiss selected", + "clear_selection": "Clear selection", + "bulk_toast_partial": "{{success}} decided · {{failure}} failed", + "bulk_error": "Bulk decision failed", + "row_status_invalid_state": "Already decided", + "row_status_not_found": "Not found", + "row_status_error": "Failed", + "bulk_action_bar_count_one": "{{count}} selected", + "bulk_action_bar_count_other": "{{count}} selected", + "bulk_toast_summary_one": "{{count}} finding decided", + "bulk_toast_summary_other": "{{count}} findings decided" + } } }, "admin": { @@ -2658,6 +2702,19 @@ "HEX": "Hexadecimal", "BASE64": "Base64", "BASE64URL": "Base64 (URL-safe, unpadded)" + }, + "discovery_detector": { + "EMAIL": "Email", + "CREDIT_CARD": "Credit card", + "SSN": "SSN", + "IBAN": "IBAN", + "PHONE": "Phone", + "AI": "AI" + }, + "discovery_finding_status": { + "PENDING": "Pending", + "CONFIRMED": "Confirmed", + "DISMISSED": "Dismissed" } }, "access": { diff --git a/frontend/src/locales/es.json b/frontend/src/locales/es.json index 9ace6fc9..5de18c06 100644 --- a/frontend/src/locales/es.json +++ b/frontend/src/locales/es.json @@ -1080,6 +1080,7 @@ }, "tab_row_security": "Seguridad de filas · {{count}}", "tab_classification": "Clasificación · {{count}}", + "tab_discovery": "Descubrimiento · {{count}}", "row_security": { "title": "Políticas de seguridad de filas", "description": "Filtra las filas que un usuario puede ver (SELECT) o modificar (UPDATE/DELETE) en una tabla. Cada política compara una columna con un valor por usuario; los usuarios en alcance solo ven las filas autorizadas.", @@ -1202,7 +1203,50 @@ "cache_enabled_help": "Servir SELECTs idénticos repetidos desde una caché Redis. Las entradas se invalidan con cada escritura en una tabla referenciada; el enmascaramiento y la seguridad de filas siguen aplicándose.", "label_cache_ttl": "TTL de caché (segundos)", "cache_ttl_help": "Cuánto tiempo permanece válido un resultado en caché (1–86.400 segundos). Déjelo en blanco para el valor por defecto del despliegue.", - "cache_ttl_placeholder": "Valor por defecto del despliegue" + "cache_ttl_placeholder": "Valor por defecto del despliegue", + "discovery": { + "settings_title": "Configuración de descubrimiento", + "settings_description": "Muestrea periódicamente los datos de columnas a través de la ruta de muestreo gobernada, detecta valores sensibles con detectores de patrones locales (y opcionalmente su analizador de IA) y propone etiquetas de clasificación para revisión. Los valores muestreados sin procesar nunca se almacenan ni se envían al proveedor de IA.", + "label_enabled": "Escaneos programados", + "label_sample_size": "Tamaño de muestra", + "label_scan_interval": "Intervalo (horas)", + "label_ai": "Clasificación por IA", + "ai_hint": "Además pide al analizador de IA de la organización que clasifique columnas usando solo nombres, tipos y muestras redactadas.", + "sample_size_range": "El tamaño de la muestra debe estar entre 10 y 1000", + "scan_interval_range": "El intervalo de escaneo debe estar entre 1 y 720 horas", + "scan_now": "Escanear ahora", + "scan_started": "Escaneo de descubrimiento iniciado", + "scan_error": "No se pudo iniciar el escaneo de descubrimiento", + "config_save_success": "Configuración de descubrimiento guardada", + "config_save_error": "No se pudo guardar la configuración de descubrimiento", + "last_scan": "Último escaneo: {{time}}", + "last_scan_error": "Problema en el último escaneo: {{error}}", + "findings_title": "Hallazgos", + "findings_description": "Clasificaciones propuestas de columnas sensibles. Confirmar aplica la etiqueta y deriva una política de enmascaramiento; descartar suprime la propuesta permanentemente.", + "filter_all": "Todos", + "col_column": "Columna", + "col_classification": "Clasificación", + "col_detector": "Detector", + "col_confidence": "Confianza", + "confidence_value": "{{confidence}}%", + "col_sample": "Muestra (redactada)", + "col_detected": "Detectado", + "col_status": "Estado", + "empty_title": "Sin hallazgos", + "empty_description": "Active el descubrimiento o ejecute un escaneo para proponer clasificaciones de columnas sensibles.", + "confirm_selected": "Confirmar seleccionados", + "dismiss_selected": "Descartar seleccionados", + "clear_selection": "Borrar selección", + "bulk_toast_partial": "{{success}} decididos · {{failure}} fallidos", + "bulk_error": "La decisión masiva falló", + "row_status_invalid_state": "Ya decidido", + "row_status_not_found": "No encontrado", + "row_status_error": "Fallido", + "bulk_action_bar_count_one": "{{count}} seleccionado", + "bulk_action_bar_count_other": "{{count}} seleccionados", + "bulk_toast_summary_one": "{{count}} hallazgo decidido", + "bulk_toast_summary_other": "{{count}} hallazgos decididos" + } } }, "admin": { @@ -2634,6 +2678,19 @@ "HEX": "Hexadecimal", "BASE64": "Base64", "BASE64URL": "Base64 (seguro para URL, sin relleno)" + }, + "discovery_detector": { + "EMAIL": "Correo electrónico", + "CREDIT_CARD": "Tarjeta de crédito", + "SSN": "SSN", + "IBAN": "IBAN", + "PHONE": "Teléfono", + "AI": "IA" + }, + "discovery_finding_status": { + "PENDING": "Pendiente", + "CONFIRMED": "Confirmado", + "DISMISSED": "Descartado" } }, "access": { diff --git a/frontend/src/locales/fr.json b/frontend/src/locales/fr.json index 0ac01835..aa5aa847 100644 --- a/frontend/src/locales/fr.json +++ b/frontend/src/locales/fr.json @@ -1080,6 +1080,7 @@ }, "tab_row_security": "Sécurité des lignes · {{count}}", "tab_classification": "Classification · {{count}}", + "tab_discovery": "Découverte · {{count}}", "row_security": { "title": "Politiques de sécurité au niveau des lignes", "description": "Filtre les lignes qu’un utilisateur peut voir (SELECT) ou modifier (UPDATE/DELETE) dans une table. Chaque politique compare une colonne à une valeur par utilisateur ; les utilisateurs concernés ne voient que les lignes autorisées.", @@ -1202,7 +1203,50 @@ "cache_enabled_help": "Servir les SELECT identiques répétés depuis un cache Redis. Les entrées sont invalidées à chaque écriture sur une table référencée ; le masquage et la sécurité des lignes s’appliquent toujours.", "label_cache_ttl": "TTL du cache (secondes)", "cache_ttl_help": "Durée de validité d’un résultat en cache (1–86 400 secondes). Laisser vide pour la valeur par défaut du déploiement.", - "cache_ttl_placeholder": "Valeur par défaut du déploiement" + "cache_ttl_placeholder": "Valeur par défaut du déploiement", + "discovery": { + "settings_title": "Paramètres de découverte", + "settings_description": "Échantillonne périodiquement les données des colonnes via le chemin d'échantillonnage gouverné, détecte les valeurs sensibles avec des détecteurs de motifs locaux (et éventuellement votre analyseur IA) et propose des étiquettes de classification à examiner. Les valeurs brutes échantillonnées ne sont jamais stockées ni envoyées au fournisseur d'IA.", + "label_enabled": "Analyses planifiées", + "label_sample_size": "Taille de l'échantillon", + "label_scan_interval": "Intervalle (heures)", + "label_ai": "Classification par IA", + "ai_hint": "Demande en plus à l'analyseur IA de l'organisation de classifier les colonnes en utilisant uniquement les noms, les types et des échantillons expurgés.", + "sample_size_range": "La taille de l'échantillon doit être comprise entre 10 et 1000", + "scan_interval_range": "L'intervalle d'analyse doit être compris entre 1 et 720 heures", + "scan_now": "Analyser maintenant", + "scan_started": "Analyse de découverte démarrée", + "scan_error": "Impossible de démarrer l'analyse de découverte", + "config_save_success": "Paramètres de découverte enregistrés", + "config_save_error": "Impossible d'enregistrer les paramètres de découverte", + "last_scan": "Dernière analyse : {{time}}", + "last_scan_error": "Problème lors de la dernière analyse : {{error}}", + "findings_title": "Découvertes", + "findings_description": "Classifications proposées de colonnes sensibles. Confirmer applique l'étiquette et dérive une politique de masquage ; rejeter supprime définitivement la proposition.", + "filter_all": "Toutes", + "col_column": "Colonne", + "col_classification": "Classification", + "col_detector": "Détecteur", + "col_confidence": "Confiance", + "confidence_value": "{{confidence}} %", + "col_sample": "Échantillon (expurgé)", + "col_detected": "Détecté", + "col_status": "Statut", + "empty_title": "Aucune découverte", + "empty_description": "Activez la découverte ou lancez une analyse pour proposer des classifications de colonnes sensibles.", + "confirm_selected": "Confirmer la sélection", + "dismiss_selected": "Rejeter la sélection", + "clear_selection": "Effacer la sélection", + "bulk_toast_partial": "{{success}} décidées · {{failure}} en échec", + "bulk_error": "La décision groupée a échoué", + "row_status_invalid_state": "Déjà décidé", + "row_status_not_found": "Introuvable", + "row_status_error": "Échec", + "bulk_action_bar_count_one": "{{count}} sélectionné", + "bulk_action_bar_count_other": "{{count}} sélectionnés", + "bulk_toast_summary_one": "{{count}} découverte décidée", + "bulk_toast_summary_other": "{{count}} découvertes décidées" + } } }, "admin": { @@ -2634,6 +2678,19 @@ "HEX": "Hexadécimal", "BASE64": "Base64", "BASE64URL": "Base64 (URL, sans remplissage)" + }, + "discovery_detector": { + "EMAIL": "E-mail", + "CREDIT_CARD": "Carte bancaire", + "SSN": "SSN", + "IBAN": "IBAN", + "PHONE": "Téléphone", + "AI": "IA" + }, + "discovery_finding_status": { + "PENDING": "En attente", + "CONFIRMED": "Confirmée", + "DISMISSED": "Rejetée" } }, "access": { diff --git a/frontend/src/locales/hy.json b/frontend/src/locales/hy.json index 4181bd04..004d77c5 100644 --- a/frontend/src/locales/hy.json +++ b/frontend/src/locales/hy.json @@ -1080,6 +1080,7 @@ }, "tab_row_security": "Տողերի անվտանգություն · {{count}}", "tab_classification": "Դասակարգում · {{count}}", + "tab_discovery": "Հայտնաբերում · {{count}}", "row_security": { "title": "Տողերի անվտանգության քաղաքականություններ", "description": "Ֆիլտրում է տողերը, որոնք օգտատերն կարող է տեսնել (SELECT) կամ փոփոխել (UPDATE/DELETE) աղյուսակում։ Ամեն քաղաքականություն համեմատում է սյունակը օգտատերական արժեքի հետ:", @@ -1202,7 +1203,50 @@ "cache_enabled_help": "Կրկնվող նույնական SELECT-ները սպասարկել Redis քեշից: Գրանցումներն անվավեր են ճանաչվում հղված աղյուսակի ցանկացած գրառման դեպքում. դիմակավորումը և տողերի անվտանգությունը շարունակում են գործել:", "label_cache_ttl": "Քեշի TTL (վայրկյան)", "cache_ttl_help": "Որքան ժամանակ է քեշավորված արդյունքը վավեր մնում (1–86400 վայրկյան): Թողեք դատարկ՝ տեղակայման լռելյայն արժեքի համար:", - "cache_ttl_placeholder": "Տեղակայման լռելյայն" + "cache_ttl_placeholder": "Տեղակայման լռելյայն", + "discovery": { + "settings_title": "Հայտնաբերման կարգավորումներ", + "settings_description": "Պարբերաբար նմուշառում է սյունակների տվյալները կառավարվող նմուշառման ուղով, հայտնաբերում է զգայուն արժեքները տեղային շաբլոնային դետեկտորներով (և ըստ ցանկության՝ ձեր AI վերլուծիչով) և առաջարկում է դասակարգման պիտակներ վերանայման համար: Հում նմուշառված արժեքները երբեք չեն պահվում և չեն ուղարկվում AI մատակարարին:", + "label_enabled": "Պլանավորված սկանավորումներ", + "label_sample_size": "Նմուշի չափ", + "label_scan_interval": "Միջակայք (ժամ)", + "label_ai": "AI դասակարգում", + "ai_hint": "Լրացուցիչ խնդրում է կազմակերպության AI վերլուծիչին դասակարգել սյունակները՝ օգտագործելով միայն անունները, տիպերը և խմբագրված նմուշները:", + "sample_size_range": "Նմուշի չափը պետք է լինի 10-ից 1000 միջակայքում", + "scan_interval_range": "Սկանավորման միջակայքը պետք է լինի 1-ից 720 ժամ", + "scan_now": "Սկանավորել հիմա", + "scan_started": "Հայտնաբերման սկանավորումը սկսվեց", + "scan_error": "Չհաջողվեց սկսել հայտնաբերման սկանավորումը", + "config_save_success": "Հայտնաբերման կարգավորումները պահպանվեցին", + "config_save_error": "Չհաջողվեց պահպանել հայտնաբերման կարգավորումները", + "last_scan": "Վերջին սկանավորումը՝ {{time}}", + "last_scan_error": "Վերջին սկանավորման խնդիր՝ {{error}}", + "findings_title": "Հայտնաբերումներ", + "findings_description": "Զգայուն սյունակների առաջարկվող դասակարգումներ: Հաստատումը կիրառում է պիտակը և ածանցում է դիմակավորման քաղաքականություն, մերժումը մշտապես ճնշում է առաջարկը:", + "filter_all": "Բոլորը", + "col_column": "Սյունակ", + "col_classification": "Դասակարգում", + "col_detector": "Դետեկտոր", + "col_confidence": "Վստահություն", + "confidence_value": "{{confidence}}%", + "col_sample": "Նմուշ (խմբագրված)", + "col_detected": "Հայտնաբերվել է", + "col_status": "Կարգավիճակ", + "empty_title": "Հայտնաբերումներ չկան", + "empty_description": "Միացրեք հայտնաբերումը կամ գործարկեք սկանավորում՝ զգայուն սյունակների դասակարգումներ առաջարկելու համար:", + "confirm_selected": "Հաստատել ընտրվածները", + "dismiss_selected": "Մերժել ընտրվածները", + "clear_selection": "Մաքրել ընտրությունը", + "bulk_toast_partial": "{{success}} որոշված · {{failure}} ձախողված", + "bulk_error": "Խմբային որոշումը ձախողվեց", + "row_status_invalid_state": "Արդեն որոշված է", + "row_status_not_found": "Չի գտնվել", + "row_status_error": "Ձախողվեց", + "bulk_action_bar_count_one": "{{count}} ընտրված", + "bulk_action_bar_count_other": "{{count}} ընտրված", + "bulk_toast_summary_one": "{{count}} հայտնաբերում որոշված է", + "bulk_toast_summary_other": "{{count}} հայտնաբերում որոշված է" + } } }, "admin": { @@ -2634,6 +2678,19 @@ "HEX": "Տասնվեցական", "BASE64": "Base64", "BASE64URL": "Base64 (URL-անվտանգ, առանց լրացման)" + }, + "discovery_detector": { + "EMAIL": "Էլ. փոստ", + "CREDIT_CARD": "Կրեդիտ քարտ", + "SSN": "SSN", + "IBAN": "IBAN", + "PHONE": "Հեռախոս", + "AI": "AI" + }, + "discovery_finding_status": { + "PENDING": "Սպասվող", + "CONFIRMED": "Հաստատված", + "DISMISSED": "Մերժված" } }, "access": { diff --git a/frontend/src/locales/ru.json b/frontend/src/locales/ru.json index 27bad69b..4718c0e6 100644 --- a/frontend/src/locales/ru.json +++ b/frontend/src/locales/ru.json @@ -1080,6 +1080,7 @@ }, "tab_row_security": "Безопасность строк · {{count}}", "tab_classification": "Классификация · {{count}}", + "tab_discovery": "Обнаружение · {{count}}", "row_security": { "title": "Политики безопасности строк", "description": "Фильтрует строки, которые пользователь может видеть (SELECT) или изменять (UPDATE/DELETE) в таблице. Каждая политика сравнивает столбец с значением для каждого пользователя; пользователи в области действия видят только разрешённые строки.", @@ -1202,7 +1203,50 @@ "cache_enabled_help": "Обслуживать повторяющиеся одинаковые SELECT из кэша Redis. Записи инвалидируются при любой записи в связанную таблицу; маскирование и защита строк продолжают действовать.", "label_cache_ttl": "TTL кэша (секунды)", "cache_ttl_help": "Как долго кэшированный результат остаётся действительным (1–86 400 секунд). Оставьте пустым для значения по умолчанию.", - "cache_ttl_placeholder": "Значение по умолчанию" + "cache_ttl_placeholder": "Значение по умолчанию", + "discovery": { + "settings_title": "Настройки обнаружения", + "settings_description": "Периодически выбирает данные столбцов через управляемый путь выборки, обнаруживает чувствительные значения локальными детекторами шаблонов (и опционально вашим ИИ-анализатором) и предлагает теги классификации для проверки. Сырые значения выборки никогда не сохраняются и не отправляются провайдеру ИИ.", + "label_enabled": "Плановые сканирования", + "label_sample_size": "Размер выборки", + "label_scan_interval": "Интервал (часы)", + "label_ai": "ИИ-классификация", + "ai_hint": "Дополнительно просит ИИ-анализатор организации классифицировать столбцы, используя только имена, типы и отредактированные образцы.", + "sample_size_range": "Размер выборки должен быть от 10 до 1000", + "scan_interval_range": "Интервал сканирования должен быть от 1 до 720 часов", + "scan_now": "Сканировать сейчас", + "scan_started": "Сканирование обнаружения запущено", + "scan_error": "Не удалось запустить сканирование обнаружения", + "config_save_success": "Настройки обнаружения сохранены", + "config_save_error": "Не удалось сохранить настройки обнаружения", + "last_scan": "Последнее сканирование: {{time}}", + "last_scan_error": "Проблема последнего сканирования: {{error}}", + "findings_title": "Находки", + "findings_description": "Предложенные классификации чувствительных столбцов. Подтверждение применяет тег и создаёт политику маскирования; отклонение навсегда подавляет предложение.", + "filter_all": "Все", + "col_column": "Столбец", + "col_classification": "Классификация", + "col_detector": "Детектор", + "col_confidence": "Уверенность", + "confidence_value": "{{confidence}}%", + "col_sample": "Образец (отредактирован)", + "col_detected": "Обнаружено", + "col_status": "Статус", + "empty_title": "Находок нет", + "empty_description": "Включите обнаружение или запустите сканирование, чтобы получить предложения классификации чувствительных столбцов.", + "confirm_selected": "Подтвердить выбранные", + "dismiss_selected": "Отклонить выбранные", + "clear_selection": "Снять выделение", + "bulk_toast_partial": "{{success}} решено · {{failure}} с ошибкой", + "bulk_error": "Массовое решение не удалось", + "row_status_invalid_state": "Уже решено", + "row_status_not_found": "Не найдено", + "row_status_error": "Ошибка", + "bulk_action_bar_count_one": "Выбрано: {{count}}", + "bulk_action_bar_count_other": "Выбрано: {{count}}", + "bulk_toast_summary_one": "Решено находок: {{count}}", + "bulk_toast_summary_other": "Решено находок: {{count}}" + } } }, "admin": { @@ -2634,6 +2678,19 @@ "HEX": "Шестнадцатеричная", "BASE64": "Base64", "BASE64URL": "Base64 (URL-безопасная, без заполнения)" + }, + "discovery_detector": { + "EMAIL": "Эл. почта", + "CREDIT_CARD": "Кредитная карта", + "SSN": "SSN", + "IBAN": "IBAN", + "PHONE": "Телефон", + "AI": "ИИ" + }, + "discovery_finding_status": { + "PENDING": "Ожидает", + "CONFIRMED": "Подтверждено", + "DISMISSED": "Отклонено" } }, "access": { diff --git a/frontend/src/locales/zh-CN.json b/frontend/src/locales/zh-CN.json index f3dbbdc0..42361ea5 100644 --- a/frontend/src/locales/zh-CN.json +++ b/frontend/src/locales/zh-CN.json @@ -1080,6 +1080,7 @@ }, "tab_row_security": "行级安全 · {{count}}", "tab_classification": "分类 · {{count}}", + "tab_discovery": "发现 · {{count}}", "row_security": { "title": "行级安全策略", "description": "过滤用户在表上可以查看（SELECT）或修改（UPDATE/DELETE）的行。每个策略将一列与每用户值进行比较；范围内的用户只能看到被授权的行。", @@ -1202,7 +1203,50 @@ "cache_enabled_help": "从 Redis 缓存中响应重复的相同 SELECT。对被引用表的任何写入都会使条目失效；脱敏和行级安全仍然生效。", "label_cache_ttl": "缓存 TTL（秒）", "cache_ttl_help": "缓存结果的有效时长（1–86,400 秒）。留空则使用部署默认值。", - "cache_ttl_placeholder": "部署默认值" + "cache_ttl_placeholder": "部署默认值", + "discovery": { + "settings_title": "发现设置", + "settings_description": "定期通过受治理的采样路径对列数据进行采样,使用本地模式检测器(以及可选的 AI 分析器)检测敏感值,并提出分类标签供审核。原始采样值绝不会被存储,也绝不会发送给 AI 提供商。", + "label_enabled": "计划扫描", + "label_sample_size": "采样大小", + "label_scan_interval": "间隔(小时)", + "label_ai": "AI 分类", + "ai_hint": "另外请求组织的 AI 分析器仅使用列名、类型和脱敏样本对列进行分类。", + "sample_size_range": "采样大小必须介于 10 到 1000 之间", + "scan_interval_range": "扫描间隔必须介于 1 到 720 小时之间", + "scan_now": "立即扫描", + "scan_started": "发现扫描已启动", + "scan_error": "无法启动发现扫描", + "config_save_success": "发现设置已保存", + "config_save_error": "无法保存发现设置", + "last_scan": "上次扫描:{{time}}", + "last_scan_error": "上次扫描问题:{{error}}", + "findings_title": "发现项", + "findings_description": "建议的敏感列分类。确认将应用标签并派生脱敏策略;忽略将永久抑制该建议。", + "filter_all": "全部", + "col_column": "列", + "col_classification": "分类", + "col_detector": "检测器", + "col_confidence": "置信度", + "confidence_value": "{{confidence}}%", + "col_sample": "样本(已脱敏)", + "col_detected": "检测时间", + "col_status": "状态", + "empty_title": "暂无发现项", + "empty_description": "启用发现或运行扫描,以获取敏感列的分类建议。", + "confirm_selected": "确认所选", + "dismiss_selected": "忽略所选", + "clear_selection": "清除选择", + "bulk_toast_partial": "{{success}} 个已决定 · {{failure}} 个失败", + "bulk_error": "批量决定失败", + "row_status_invalid_state": "已决定", + "row_status_not_found": "未找到", + "row_status_error": "失败", + "bulk_action_bar_count_one": "已选择 {{count}} 项", + "bulk_action_bar_count_other": "已选择 {{count}} 项", + "bulk_toast_summary_one": "已决定 {{count}} 个发现项", + "bulk_toast_summary_other": "已决定 {{count}} 个发现项" + } } }, "admin": { @@ -2634,6 +2678,19 @@ "HEX": "十六进制", "BASE64": "Base64", "BASE64URL": "Base64（URL 安全、无填充）" + }, + "discovery_detector": { + "EMAIL": "电子邮件", + "CREDIT_CARD": "信用卡", + "SSN": "SSN", + "IBAN": "IBAN", + "PHONE": "电话", + "AI": "AI" + }, + "discovery_finding_status": { + "PENDING": "待处理", + "CONFIRMED": "已确认", + "DISMISSED": "已忽略" } }, "access": { diff --git a/frontend/src/pages/datasources/DatasourceSettingsPage.tsx b/frontend/src/pages/datasources/DatasourceSettingsPage.tsx index 0b33b016..2d3270b1 100644 --- a/frontend/src/pages/datasources/DatasourceSettingsPage.tsx +++ b/frontend/src/pages/datasources/DatasourceSettingsPage.tsx @@ -76,6 +76,8 @@ import { rowSecurityPolicyKeys, } from '@/api/rowSecurityPolicies'; import { ClassificationTab } from '@/components/datasources/ClassificationTab'; +import { DiscoveryTab } from '@/components/datasources/DiscoveryTab'; +import { discoveryKeys, fetchDiscoveryFindings } from '@/api/discovery'; import { dataClassificationKeys, listClassificationTags, @@ -131,6 +133,14 @@ export function DatasourceSettingsPage() { enabled: !!id, }); + const discoveryPendingQuery = useQuery({ + queryKey: id + ? discoveryKeys.findings(id, { status: 'PENDING', page: 0, size: 1 }) + : ['discovery', 'findings', 'idle'], + queryFn: () => fetchDiscoveryFindings(id!, { status: 'PENDING', page: 0, size: 1 }), + enabled: !!id, + }); + const testMutation = useMutation({ mutationFn: () => testConnection(id!), onSuccess: (result) => { @@ -201,6 +211,7 @@ export function DatasourceSettingsPage() { const maskingCount = maskingPoliciesQuery.data?.length ?? 0; const rowSecurityCount = rowSecurityPoliciesQuery.data?.length ?? 0; const classificationCount = classificationTagsQuery.data?.length ?? 0; + const discoveryPendingCount = discoveryPendingQuery.data?.total_elements ?? 0; const testIcon = testMutation.isPending ? ( @@ -258,6 +269,10 @@ export function DatasourceSettingsPage() { key: 'classification', label: t('datasources.settings.tab_classification', { count: classificationCount }), }, + { + key: 'discovery', + label: t('datasources.settings.tab_discovery', { count: discoveryPendingCount }), + }, { key: 'er-diagram', label: t('datasources.settings.tab_er_diagram') }, { key: 'activity', label: t('datasources.settings.tab_activity') }, ]} @@ -269,6 +284,7 @@ export function DatasourceSettingsPage() { {tab === 'masking' && } {tab === 'row-security' && } {tab === 'classification' && } + {tab === 'discovery' && } {tab === 'er-diagram' && } {tab === 'activity' && }
diff --git a/frontend/src/types/api.ts b/frontend/src/types/api.ts index bccb99a6..058a4ad8 100644 --- a/frontend/src/types/api.ts +++ b/frontend/src/types/api.ts @@ -3235,3 +3235,69 @@ export interface PermissionCatalogGroup { export interface PermissionCatalog { groups: PermissionCatalogGroup[]; } + +// --- AF-623: sensitive-data discovery --- + +export type DiscoveryDetector = 'EMAIL' | 'CREDIT_CARD' | 'SSN' | 'IBAN' | 'PHONE' | 'AI'; +export type DiscoveryFindingStatus = 'PENDING' | 'CONFIRMED' | 'DISMISSED'; +export type DiscoveryDecision = 'CONFIRM' | 'DISMISS'; +export type DiscoveryRowStatus = + | 'SUCCESS' + | 'NOT_FOUND' + | 'INVALID_STATE' + | 'TAG_CONFLICT' + | 'ERROR'; + +export interface DiscoveryScanConfig { + datasource_id: string; + enabled: boolean; + sample_size: number; + scan_interval_hours: number; + ai_classification_enabled: boolean; + last_scan_at: string | null; + last_scan_error: string | null; +} + +/** Partial update — omitted fields keep their current value. */ +export interface UpdateDiscoveryConfigInput { + enabled?: boolean; + sample_size?: number; + scan_interval_hours?: number; + ai_classification_enabled?: boolean; +} + +export interface DiscoveryFinding { + id: string; + schema_name: string | null; + table_name: string; + column_name: string; + classification: DataClassification; + detector: DiscoveryDetector; + confidence: number; + sample_redacted: string | null; + rationale: string | null; + match_count: number; + sample_count: number; + status: DiscoveryFindingStatus; + first_detected_at: string; + last_detected_at: string; + decided_by: string | null; + decided_at: string | null; +} + +export type DiscoveryFindingPage = PageEnvelope; + +export interface BulkDiscoveryDecisionInput { + finding_ids: string[]; + decision: DiscoveryDecision; +} + +export interface BulkDiscoveryDecisionRow { + finding_id: string; + status: DiscoveryRowStatus; + new_status: DiscoveryFindingStatus | null; +} + +export interface BulkDiscoveryDecisionResult { + results: BulkDiscoveryDecisionRow[]; +} diff --git a/frontend/src/utils/enumLabels.ts b/frontend/src/utils/enumLabels.ts index 404098ad..52ef6f20 100644 --- a/frontend/src/utils/enumLabels.ts +++ b/frontend/src/utils/enumLabels.ts @@ -45,6 +45,8 @@ import type { RoutingConditionOperand, RowSecurityOperator, RowSecurityValueType, + DiscoveryDetector, + DiscoveryFindingStatus, CommentStatus, SslMode, SubmissionReason, @@ -540,3 +542,26 @@ export const REQUEST_GROUP_TARGET_KINDS: readonly RequestGroupTargetKind[] = [ export const targetKindLabel = (t: TFunction, v: RequestGroupTargetKind): string => t(`enums.request_group_target_kind.${v}` as const); + +// ── Sensitive-data discovery (AF-623) ─────────────────────────────────────── + +export const DISCOVERY_DETECTORS: readonly DiscoveryDetector[] = [ + 'EMAIL', + 'CREDIT_CARD', + 'SSN', + 'IBAN', + 'PHONE', + 'AI', +] as const; + +export const discoveryDetectorLabel = (t: TFunction, v: DiscoveryDetector): string => + t(`enums.discovery_detector.${v}` as const); + +export const DISCOVERY_FINDING_STATUSES: readonly DiscoveryFindingStatus[] = [ + 'PENDING', + 'CONFIRMED', + 'DISMISSED', +] as const; + +export const discoveryFindingStatusLabel = (t: TFunction, v: DiscoveryFindingStatus): string => + t(`enums.discovery_finding_status.${v}` as const); diff --git a/website/README.md b/website/README.md index 777b519f..1d736a48 100644 --- a/website/README.md +++ b/website/README.md @@ -43,6 +43,7 @@ the right. | [`docs/05-backend.md`](../docs/05-backend.md) "Dynamic data masking policies", [`docs/07-security.md`](../docs/07-security.md) masking section, [`docs/03-data-model.md`](../docs/03-data-model.md) `masking_policy` | "Dynamic data masking" blurb in the "Full query proxy — SQL and NoSQL" feature tile (homepage) + "Masking policies" paragraph under "Datasources" in [`docs/index.html`](docs/index.html) | | [`docs/05-backend.md`](../docs/05-backend.md) "Row-level security policies", [`docs/07-security.md`](../docs/07-security.md) row-security section, [`docs/03-data-model.md`](../docs/03-data-model.md) `row_security_policy` / `users.attributes`, [`docs/04-api-spec.md`](../docs/04-api-spec.md) `/datasources/{id}/row-security-policies` | "Row-level security" blurb in the "Full query proxy — SQL and NoSQL" feature tile (homepage) + "Row security policies" paragraph under "Datasources" in [`docs/index.html`](docs/index.html) | | [`docs/05-backend.md`](../docs/05-backend.md) data-classification tags & auto-derived masking (AF-518), [`docs/03-data-model.md`](../docs/03-data-model.md) `data_classification`, [`frontend/src/pages/admin/DataClassificationsPage.tsx`](../frontend/src/pages/admin/DataClassificationsPage.tsx) | "Data classification" subsection (`#cfg-data-classifications`) under "Datasources" in [`docs/index.html`](docs/index.html) | +| [`docs/05-backend.md`](../docs/05-backend.md) "Automated sensitive-data discovery" (AF-623), [`docs/03-data-model.md`](../docs/03-data-model.md) `discovery_scan_config` / `discovery_finding`, [`docs/09-deployment.md`](../docs/09-deployment.md) `ACCESSFLOW_DISCOVERY_*` env vars | "Automated sensitive-data discovery" blurb in the "Full query proxy — SQL and NoSQL" feature tile + "Automated sensitive-data discovery" item in the Planned roadmap group (homepage) + "Automated discovery" paragraph under "Data classification" in [`docs/index.html`](docs/index.html) | | [`frontend/src/i18n.ts`](../frontend/src/i18n.ts) `SUPPORTED_LANGUAGES` / `LANGUAGE_DISPLAY_NAMES`, [`frontend/src/pages/admin/LanguagesConfigPage.tsx`](../frontend/src/pages/admin/LanguagesConfigPage.tsx), [`docs/06-frontend.md`](../docs/06-frontend.md) i18n section, [`docs/04-api-spec.md`](../docs/04-api-spec.md) `/admin/localization-config` | "Languages & localization" section (`#cfg-languages`) under Configuration in [`docs/index.html`](docs/index.html) | | [`frontend/src/config/docs.ts`](../frontend/src/config/docs.ts) (`DOCS_BASE_URL` / `DOCS_ANCHORS`), [`frontend/src/components/common/PageHeader.tsx`](../frontend/src/components/common/PageHeader.tsx) `docsAnchor` prop, [`docs/06-frontend.md`](../docs/06-frontend.md) "Contextual docs links" | The anchor `id`s under Configuration in [`docs/index.html`](docs/index.html) — every one is the target of an in-app **View docs** link, so **renaming or removing an anchor breaks that link**. `frontend/src/config/__tests__/docs.test.ts` fails when a declared anchor has no matching `id` | | [`docs/05-backend.md`](../docs/05-backend.md) "Policy-as-code routing engine", [`docs/03-data-model.md`](../docs/03-data-model.md) `routing_policy` / `routing_decision`, [`docs/04-api-spec.md`](../docs/04-api-spec.md) `/admin/routing-policies` | "Policy-as-code routing" blurb in the "Configurable review workflows" feature tile (homepage) + "Routing policies" section under Configuration in [`docs/index.html`](docs/index.html) | diff --git a/website/docs/index.html b/website/docs/index.html index 7a5d977c..c1157998 100644 --- a/website/docs/index.html +++ b/website/docs/index.html @@ -1204,6 +1204,26 @@

Data classification

classifications (/admin/data-classifications) lists every tag across all datasources as the evidence base for compliance reporting.

+

+ Automated discovery. Instead of tagging hundreds of tables by hand, the + datasource Discovery tab opts a datasource into a scheduled scanner that samples + column data through the same governed sampling path, detects sensitive values with local + regex + checksum detectors (emails, credit-card numbers with Luhn, US SSNs, IBANs, phone + numbers) and — optionally — your bound AI analyzer, then proposes the + classification tags in a review worklist. Confirming a finding applies the tag (deriving + masking exactly like a manual tag); dismissing suppresses the proposal permanently. Raw + sampled values never persist (findings store a redacted sample only), and the AI pass + only ever sees column names, types, and redacted samples. Configure the per-datasource + sample size (10–1000 rows) and cadence (1–720 hours), or hit Scan now for an + immediate run; scans and decisions land in the audit log + (DISCOVERY_SCAN_COMPLETED, DISCOVERY_FINDING_CONFIRMED / + _DISMISSED). Operator knobs: + ACCESSFLOW_DISCOVERY_SCAN_POLL_INTERVAL (PT15M), + ACCESSFLOW_DISCOVERY_SCAN_TIME_BUDGET (PT10M), + ACCESSFLOW_DISCOVERY_SAMPLE_STATEMENT_TIMEOUT (PT10S), + ACCESSFLOW_DISCOVERY_MAX_TABLES_PER_SCAN (200), + ACCESSFLOW_DISCOVERY_MAX_AI_TABLES_PER_SCAN (25). +

Tune it. Per-datasource fields above set row caps and review behaviour; these environment variables set the engine-level connection and execution ceilings diff --git a/website/index.html b/website/index.html index ba64182b..cb8bc1a9 100644 --- a/website/index.html +++ b/website/index.html @@ -328,7 +328,7 @@

The middle ground between blanket access and a DBA ticket queue.

Full query proxy — SQL and NoSQL

-

Every query is inspected before it ever reaches the database — parsed, sorted by type (read, write, or schema change), checked against the tables a user is allowed to touch, and routed through your review policy. The same governance covers NoSQL and the cloud data warehouses too: MongoDB, Couchbase, Redis, Cassandra, Elasticsearch, DynamoDB, Neo4j, Snowflake, BigQuery, and Databricks all flow through identical checks, with dangerous or server-side commands blocked up front. Layer on row-level security so users only see the rows they're cleared for, dynamic masking that hides sensitive columns per person, and data-classification tags (PII, PCI, PHI, GDPR) that auto-apply masking and raise a query's risk score. A dry-run preview shows a query's impact before review without changing any data, and every submitted query automatically gets a pre-flight cost estimate — the database's own EXPLAIN plan plus an exact affected-row count for updates and deletes — shown to reviewers, folded into the AI risk analysis, and usable in routing policies (“auto-escalate any full-scan DELETE over 100k rows”). Every executed query is saved as an exact snapshot you can replay in a test environment.

+

Every query is inspected before it ever reaches the database — parsed, sorted by type (read, write, or schema change), checked against the tables a user is allowed to touch, and routed through your review policy. The same governance covers NoSQL and the cloud data warehouses too: MongoDB, Couchbase, Redis, Cassandra, Elasticsearch, DynamoDB, Neo4j, Snowflake, BigQuery, and Databricks all flow through identical checks, with dangerous or server-side commands blocked up front. Layer on row-level security so users only see the rows they're cleared for, dynamic masking that hides sensitive columns per person, and data-classification tags (PII, PCI, PHI, GDPR) that auto-apply masking and raise a query's risk score — with automated sensitive-data discovery that samples column data on a schedule, spots emails, card numbers, SSNs, IBANs, and phone numbers with local checksum-verified detectors (optionally aided by your AI analyzer, which only ever sees redacted samples), and proposes the tags for one-click confirmation. A dry-run preview shows a query's impact before review without changing any data, and every submitted query automatically gets a pre-flight cost estimate — the database's own EXPLAIN plan plus an exact affected-row count for updates and deletes — shown to reviewers, folded into the AI risk analysis, and usable in routing policies (“auto-escalate any full-scan DELETE over 100k rows”). Every executed query is saved as an exact snapshot you can replay in a test environment.

PostgreSQL · MySQL · MariaDB · Oracle · MSSQL · ClickHouse · MongoDB · Couchbase · Redis · Cassandra · ScyllaDB · Elasticsearch · OpenSearch · DynamoDB · Neo4j · Snowflake · BigQuery · Databricks · connector catalog · custom drivers · read-replica load balancing · result caching · dynamic masking · row-level security · data classification @@ -1293,6 +1293,7 @@
Security & ops
  • External secrets managers (Vault / AWS / Azure Key Vault)
  • Backup / restore & DR tooling — Helm hooks + audit-chain verification
  • ServiceNow & Jira ticketing — auto-created tickets + bi-directional status sync
    • +
  • Automated sensitive-data discovery — scheduled column scanning proposes classification tags