diff --git a/CHANGELOG.md b/CHANGELOG.md index afce8dd77..094822f3c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -57,7 +57,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Account-scoped resource trend + derived security-count history: the daily inventory snapshot is now written per account (over the sync's trusted-account set — an unreachable account keeps its last same-day row, a reachable-but-empty account records a genuine 0), the trend API accepts the accounts scope (validated CSV; `__all__` resolves server-side to self + the enabled member accounts — never an unfiltered read, which would sum backfill 'aggregate' rows and offboarded-account history) and returns per-day PER-TYPE account coverage, and the home trend chart, delta table, 7d net change, and cost-impact estimate all follow the account selector — the guards additionally require each compared type's coverage to equal the scope that type can ever reach (host-only SDK-collected types like S3 check against the host account; everything else against the full resolved scope — the sync runs per type, so an account can be silent for one type's run only, even on both compared days), so a silent (account, type) day — or the deploy boundary, before per-account history exists — renders '—' in the KPI and delta table, hides the cost-impact panel, and draws as a line gap in the chart (with a disclosure caption) instead of a fabricated fleet change; a CSV selection narrowed by dropped ids, or the all-accounts fallback when the accounts registry is unreadable, is disclosed under the chart, and the cost-impact panel hides whenever its own fetch degraded or resolved a different scope than the chart's (per-account history accrues from this deploy; snapshots still carry no region dimension, so a narrowed region scope keeps hiding the trend-derived KPIs). The sync also historizes three derived security series — Public S3 Buckets, Open Security Groups, Unencrypted EBS — by counting the just-synced inventory with the Security page's own predicates (lockstep-guarded), shown as delta rows and as the chart's own default-hidden Security-series toggle group (appended after the top real types so they are always reachable) but excluded from the day's total to avoid double-counting; EKS/K8s counts are deliberately not historized (v2 has no batch K8s collection — EKS reads are live on-request; ECS tasks/services already trend as their own synced types). - Inventory/datasources i18n completion: the generic inventory pages (CloudFront/DynamoDB/WAF and every other type) and the datasources hub (form, tab, card dashboard, Explore, log view) now translate their Korean UI strings into en/zh/ja — 17 unregistered literals, the dynamic card/note catalogs (card_catalog titles, datasource-render result notes), the log-view caption pattern, and the donut/chart title patterns were registered; donut titles are composed fully in Korean so one tt() pass translates them (pre-translating the sample suffix produced untranslatable mixed strings); the two English action buttons (Add datasource / Test connection) are localized. A ratchet test pins the static-literal coverage on those surfaces (dynamic strings are covered by the registered catalogs and RULES — the test is a regression guard, not a completeness proof). Column/spec labels deliberately stay English. - Full-fleet aggregates past the 500-row cap: capped inventory pages fetch server-side GROUP BYs (state tiles, distribution donuts, state-filter and facet dropdown options, and the exact total) over the WHOLE scoped fleet — v1 ran its aggregation SQL fleet-wide, and the v2 sample-based counts were silently inaccurate above 500 resources. Coverage is per-dimension: dimensions whose values are client-derived (lambda's runtime and dynamodb's billing donuts, ecs_task cluster/cpu/memory facets, opensearch's encryption-status donut, msk's kafka-version facet) stay sample-based, and an option list with 50+ distinct values falls back to the sample — every sample-based donut now discloses itself with the '(표본 기준)' qualifier (previously unlabeled). Full-fleet donut 'Other' buckets are computed against the fleet total, so they sum correctly regardless of the server's bucket cap. One aggregation call replaces the previous true-total summary call; the table, Top-N bars, and highlight cards deliberately stay sample-based. -- Per-datasource connection settings: the datasource form gains a Settings section — an upstream query timeout (seconds 1–60, default 10; forwarded as the Prometheus/Mimir API timeout param capped under the connector's HTTP timeout, and as ClickHouse max_execution_time) and a ClickHouse default database (identifier-only, validated on both the web tier and the connector before any HTTP call) — persisted per instance — the ClickHouse bound is the CEILING on every path (Explore, service graph, agent/worker — callers can only tighten it; default 10s, connector HTTP timeout aligned above it) while the Prometheus/Mimir bound applies on the Explore path capped at 10s, and the database rejects system/information_schema on both validation layers; v1's result-cache TTL is deliberately not ported (the v2 query path is uncached by design) and the timeout unit changed from ms to seconds, both disclosed in the guide. ClickHouse settings of 56–60s have an effective 55s ceiling under the Lambda wall limit; the guide distinguishes stored values from effective limits. +- Per-datasource connection settings: instance settings distinguish provider execution ceilings for Prometheus/Mimir/ClickHouse (default 10s, enforced inside the connector on every query path; callers can only tighten them) from HTTP response timeouts for other kinds (default 12s, without a server-cancellation guarantee). Stored 1–60s values have an effective 55s maximum and the form shows the applied limit. ClickHouse database identifiers reject system/information_schema on both validation layers; Mimir's API prefix is configurable without duplicating an existing mount. Missing explicit instance credentials fail closed, and upstream success/error bytes are bounded before parsing. Query-access tests run one real bounded query (readiness endpoints no longer pass) and report latency and the verified operation with fixed sentences; Jaeger/Dynatrace/Datadog distinguish malformed responses from empty results, and Jaeger and Loki report reaching the requested limit as potentially partial with requested/returned counts, separately from dropped records. A connector request shares one deadline across connect, headers and body. The default instance's shared credential is stamped with its instance id so a sibling instance can never borrow it; only the no-ID legacy default path remains compatible, and migrated defaults require admin backfill before credential-blind producers roll out. Optional Jaeger/Dynatrace private networking remains default-off. Custom MCP retains admin-managed extensibility with validated-address pinning, original Host/SNI/certificate verification and no redirects; unsupported three-legged OAuth is rejected. The v2 query path remains uncached and timeout units remain seconds rather than v1 milliseconds. - Unified ECS overview page (/inventory/ecs, 'ECS Overview' in the sidebar's ECS subgroup): v1's one-screen posture restored — a summary KPI band (cluster/service counts from the loaded pages ['+' at the 500 cap], task count from the shared summary, and a tasks-below-desired attention tile aggregated only over an untruncated service page — a sample never presents a fleet total), plus the clusters and services tables stacked on one screen; each table links to its full type page for search/facets/detail (the overview is a read-only glance layer), labels a >=500-row page as a sample, shows a stale-data caption under a failed sync, and reads 'not collected yet' pre-sync instead of a fabricated empty fleet. - Dashboard on-demand sync: the header gains an admin-only 'Sync all' action that dispatches one all-types inventory sync (async batch semantics disclosed in place — a queued acknowledgement [enqueue only, not a completion guarantee; already-running types are skipped], admin-only and sync-disabled states rendered distinctly from transient failures; data lands via the normal Refresh minutes later, no optimistic mutation). - S3 security drill-downs: the S3 page gains v1's Bucket Map by Region (block tiles colored Public=red > Versioned=green > Standard=cyan with an explicit Unknown=gray — a bucket whose policy flag is unknown stays Unknown even when versioning is known (a denied policy lookup must not paint a reassuring color), never a confident Standard; block click opens the detail panel; sample-labeled past the 500-row cap), and the bucket detail gains an 'IAM Roles with S3 Access' section (ADMIN-ONLY — non-admins see a permission note; roles whose newly synced attached AWS-managed policies match the checked set [AmazonS3*/AdministratorAccess/PowerUserAccess/ReadOnlyAccess incl. job-function paths — other policies can also grant S3, so the empty state is matched-set-framed], max 30; the last sync run's status gates every conclusion — a failed/partial run shows a stale-data banner and the empty state is only conclusive under a succeeded run within 24h on an untruncated page (a data-as-of timestamp rides the footer); a 500-row sample is labeled; inline/bucket-policy access is explicitly out of scope, and pre-sync rows show a 'not synced' note rather than an empty all-clear — visible after a terraform apply + the next sync; note: a failed policy-list hydrate — an SCP block, or a fleet whose aggregate role count exceeds the sync's rate-limit budget — retries without the optional attached-policy column (other hydrates remain and may still fail) a successful fallback refreshes base iam_role rows and only this section shows 'not synced' (the run reports degraded freshness so readers see the blind spot); the whole-run last-good freeze on this path requires the base query to also fail — the final run status otherwise follows the normal sync lifecycle — the ADR-010 amendment's disclosed semantics). @@ -95,7 +95,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - AI-diagnosis generation UX quick wins (v1 parity): a running diagnosis shows an mm:ss elapsed timer and a per-section checklist grid (completed / pending, in the UI language — driven by a new additive `completed` list the worker streams into the progress JSONB; no per-section spinner on purpose, since concurrent rendering leaves no in-flight telemetry to show; older in-flight rows and a drifted section catalog fall back to the bar-only view), a completed report shows a stats bar (section count · duration · report id — duration comes from a new `finished_at` column stamped at terminal write [one additive migration]; legacy rows without it omit the segment, never fabricated), the empty state previews the full section scope with Deep-tier tags, and completed history rows carry inline MD/DOCX download links (no need to open the report). - AI-diagnosis parity batch (v1 parity): a completed report now renders as collapsible section cards with a sticky table-of-contents sidebar (click scrolls to the section) and a per-section severity icon derived from body keywords (a display heuristic, labeled as such — not a score; reports without section headings keep the continuous view); report generation language is selectable (한국어/English/中文/日本語 — defaults to the UI language, applies to manual runs and schedules; the language is part of the run's dedup key, so a same-hour language switch starts a new run instead of returning the previous language's report; a legacy dedup-key read fallback ships for one release for rolling-deploy compatibility — REMOVE in the release after this one); the auto-diagnosis schedule gains KST detail settings (weekday for weekly/biweekly, day-of-month 1–28 for monthly, run hour) plus a last-run timestamp display — unset fields keep the previous interval-only behavior; and admins can send a test notification to the diagnosis mailing list from the subscribers panel (one SNS publish scoped to the existing diagnosis topic — the web task's first, admin-only `sns:Publish`; a failed send surfaces its error, never a silent success). - Inventory KPI/chart quick-win batch (v1 parity): EC2 gains a running total-vCPU tile (per-instance `cpu_options` cores×threads — actual vCPUs, not the type default); RDS a total allocated-storage tile; Lambda long-timeout (>300s, danger) and average-memory tiles; EBS volumes an encryption-rate tile (100% → accent, <80% → danger); ECS clusters get a dedicated KPI band (ACTIVE count, running tasks, active services, container instances) instead of the generic state tiles; ECR rows gain a Scan on Push column (a missing/malformed scanning config counts as No — the API default); and the CloudWatch alarm-state donut uses fixed semantic colors (green OK / red ALARM / gray INSUFFICIENT_DATA) instead of size-ordered palette colors. -- Datasource Explore/management parity batch: curated example-query and natural-language prompt chips for all 8 connector kinds, a dedicated Loki log-stream viewer (timestamps, label badges, scrollable pane), 7d/30d time-range presets (per-kind API bound: prometheus/mimir 30d with an upstream `timeout` forwarded by their connectors — the connector change ships via `terraform apply`, so run the apply before relying on 30d; Loki capped at 7d; the 5000-point density cap stays), a result metadata bar (rows/series · execution ms · query language · shape), a dismissible "AI generated from …" banner after NL→query drafting, KPI tiles and a manual refresh button on the management tab, an "AI로 진단" deep link on each kind's DEFAULT datasource row (the chat tool path resolves per-kind defaults; supported kinds only) that prefills the assistant composer with a section-pinned prompt (`/assistant?q=`, review-only — never auto-sends), and proportional duration bars on Tempo/Jaeger trace results. +- Datasource Explore/management parity batch: curated example-query and natural-language prompt chips for all 8 connector kinds, a dedicated Loki log-stream viewer (timestamps, label badges, scrollable pane), 7d/30d time-range presets (Prometheus/Mimir/Dynatrace/Datadog up to 30d; Loki/Tempo/Jaeger up to 7d; the 5000-point density cap stays), a result metadata bar (rows/series · execution ms · query language · shape), a dismissible "AI generated from …" banner after NL→query drafting, KPI tiles and a manual refresh button on the management tab, an "AI로 진단" deep link on each kind's DEFAULT datasource row (the chat tool path resolves per-kind defaults; supported kinds only) that prefills the assistant composer with a section-pinned prompt (`/assistant?q=`, review-only — never auto-sends), and proportional duration bars on Tempo/Jaeger trace results. Search windows are forwarded independently of range-tool availability, SQL API ranges are rejected rather than ignored, and effective request bounds/shared datasource scope are explicit. AI drafting does not change the selected window. Datasource status reports verified configuration separately from untested connectivity, without inferring connection from default selection. Datadog connection checks exercise metric-query access for key pairs and bearer tokens; Datadog point caps and Dynatrace nested series caps or advertised compatibility pages remain visibly partial in Explore. Deploy the Lambda producers before the dependent Web changes. - Datasource detail pages gain a pre-built card dashboard: registering a datasource (and each daily index run) derives an expected card set from the cached schema — the queries each card uses are stored ahead of time (new `datasource_dashboard_cards` table + a deterministic `card_catalog` in the index worker; prometheus/mimir 13 cards covering targets, CPU, memory, disk, load, network, containers, and restarts; loki 2, tempo 2, clickhouse 2) — ready Prometheus/Mimir cards are live-validated against the exact datasource before registration, through the same instant/range tool the page will execute (a conclusive PromQL error — body-derived, never a bare HTTP 4xx — disables only that card and is revalidated on the next daily run; a transient connector failure, a failed re-introspection, or a truncated schema that cannot decide a card requirement all preserve the previous card set; the connector metric-metadata tool also gains a definitive `exists` flag and 3-second upstream deadlines) — card building runs inside the existing datasource-index job, so it is gated on `datasource_diagnosis_enabled` (default false; requires `workers_enabled`/`agentcore_enabled`/`integrations_enabled`) like the diag-signal chips — and the page executes the stored queries live on view through the existing read-only query API (stat/timeseries/table cards; unavailable cards render dimmed with what's missing; a failed card shows an inline error, never a silent zero). - Topology infra page gains two columnar map views — a 5-column infra resource map (External | VPC | Subnet | Compute | NAT) and a per-cluster K8s map (Ingress → Service → Pod → Node; host-account, connected clusters only — the in-cluster read path is host-scoped) — rendered as a real graph (fixed-column ReactFlow with edge lines), with click cross-highlighting, search highlighting, and a color legend. Built on existing inventory/EKS reads plus the pre-existing read-only `/api/tgw` live attachment describe (host-account scope only, first 20 TGWs — degradation is surfaced in the UI); the only server-side addition is the read-only `ingresses` in-cluster kind. @@ -726,7 +726,7 @@ First release of the **v2 line** (versioned independently from the v1 1.x line, - 계정별 리소스 추이 + 파생 보안 카운트 이력화: 일별 인벤토리 스냅샷을 계정별로 기록(sync의 신뢰 계정 집합 기준 — 미도달 계정은 당일 기존 행 보존, 도달했지만 0건인 계정은 진짜 0 기록), 추이 API가 accounts 스코프를 수용(검증된 CSV; `__all__`은 서버에서 self+활성 멤버 계정으로 해석 — 필터를 걷어내지 않음: 무필터 조회는 백필의 'aggregate' 행과 오프보딩된 계정 이력까지 합산)하고 일자·타입별 계정 커버리지를 반환, 홈 추이 차트·수량 변화 테이블·7일 순증감·비용 영향 추정이 계정 선택을 따름 — 가드는 비교 두 시점에서 각 타입의 커버리지가 그 타입이 도달 가능한 스코프와 정확히 일치할 것을 추가로 요구(S3 등 호스트 전용 SDK 수집 타입은 호스트 계정 기준, 나머지는 해석된 전체 스코프 기준 — sync는 타입별로 돌므로 한 계정이 특정 타입 run에서만, 두 시점 모두에서도, 침묵 가능)해, 침묵한 (계정, 타입) 일자나 계정별 이력이 없는 배포 경계는 플릿 변화를 지어내지 않고 KPI·변화 테이블은 '—', 비용 영향 패널은 숨김, 차트는 해당 시점을 공백(라인 갭)으로 표시(공지 캡션 포함); 유효하지 않은 ID로 좁혀진 CSV 선택이나 계정 레지스트리 조회 실패 시의 전체 계정 폴백은 차트 아래에 공지되고, 비용 영향 패널은 자체 조회가 폴백됐거나 차트와 다른 스코프로 해석된 경우 숨김(계정별 이력은 본 배포 이후부터 축적; 스냅샷에는 여전히 리전 차원이 없어 리전 스코프를 좁히면 추이 기반 KPI는 계속 숨김). sync가 파생 보안 시리즈 3종(Public S3 Buckets·Open Security Groups·Unencrypted EBS)을 보안 페이지의 판정 술어 그대로(락스텝 가드) 방금 동기화된 인벤토리에서 COUNT해 이력화 — 변화 테이블 및 차트의 기본 숨김 '보안 시리즈' 토글 그룹(상위 실제 타입 뒤에 항상 추가되어 접근 가능)으로 표시되지만 이중 계산 방지를 위해 일별 total에서는 제외. EKS/K8s 카운트는 의도적으로 미이력화(v2에는 K8s 배치 수집이 없음 — EKS는 온디맨드 라이브 조회; ECS tasks/services는 자체 동기화 타입으로 이미 추이 존재). - 인벤토리/데이터소스 i18n 완결: 범용 인벤토리 페이지(CloudFront/DynamoDB/WAF 포함 전 타입)와 데이터소스 허브(폼·탭·카드 대시보드·Explore·로그 뷰)의 한국어 UI 문자열이 en/zh/ja로 번역됨 — 미등록 리터럴 17건, 동적 카드/노트 카탈로그(card_catalog 제목, datasource-render 결과 note), 로그 뷰 캡션 패턴, 도넛/차트 제목 패턴을 등록. 도넛 제목은 완전 한국어로 조합해 tt() 1회로 번역(표본 접미사 선번역은 번역 불가한 혼합 문자열을 만들었음), 영문 액션 버튼 2종(Add datasource/Test connection) 현지화. 래칫 테스트가 정적 리터럴 커버리지를 고정(동적 문자열은 등록된 카탈로그와 RULES로 커버 — 완전성 증명이 아닌 회귀 방지 장치). 컬럼/스펙 라벨은 의도적으로 영어 유지. - 500행 캡 초과 시 전수 집계: 캡에 도달한 인벤토리 페이지가 서버 측 GROUP BY(상태 타일, 분포 도넛, 상태 필터·패싯 드롭다운 옵션, 정확한 총계)를 전체 스코프 플릿 기준으로 조회 — v1은 집계 SQL을 전 플릿에 실행했고 v2의 표본 기반 수치는 500대 초과 시 조용히 부정확했음. 커버리지는 차원별: 클라이언트 파생 값 차원(lambda runtime·dynamodb billing 도넛, ecs_task cluster/cpu/memory 패싯, opensearch 암호화 상태 도넛, msk kafka 버전 패싯)은 표본 유지, 고유값 50개 이상 옵션 목록도 표본 폴백 — 표본 기반 도넛은 모두 '(표본 기준)'으로 자체 공지(종전엔 무표기). 전수 도넛의 '기타'는 플릿 총계 기준으로 계산되어 서버 버킷 캡과 무관하게 합계가 맞음. 집계 호출 1회가 기존 총계용 summary 호출을 대체하며, 테이블·Top-N 바·하이라이트 카드는 의도적으로 표본 유지. -- 데이터소스별 연결 설정: 데이터소스 폼에 Settings 섹션 추가 — 업스트림 쿼리 타임아웃(초 1–60, 기본 10; Prometheus/Mimir는 API timeout 파라미터로 커넥터 HTTP 타임아웃 아래로 캡, ClickHouse는 max_execution_time)과 ClickHouse 기본 database(식별자만, 웹 계층과 커넥터 양쪽에서 HTTP 호출 전 검증) — 인스턴스별로 저장 — ClickHouse 제한은 모든 경로(Explore·서비스 그래프·에이전트/워커)의 상한으로 적용되고(호출자는 더 짧게만 조정 가능; 기본 10초, 커넥터 HTTP 타임아웃을 그 위로 정렬) Prometheus/Mimir 제한은 Explore 경로에 10초 캡으로 적용되며, database는 system/information_schema를 양쪽 검증 계층에서 거부. v1의 결과 캐시 TTL은 의도적으로 미이식(v2 질의 경로는 무캐시 설계), 타임아웃 단위는 ms→초 변경 — 모두 가이드에 공지. ClickHouse의 56–60초 설정은 Lambda 한도 때문에 유효 55초로 단축되며 가이드에서 저장 값과 실제 상한을 구분한다. +- 데이터소스별 연결 설정: 인스턴스 설정에서 Prometheus/Mimir/ClickHouse의 서버 실행 제한(기본 10초, 모든 쿼리 경로의 커넥터에서 강제하며 호출자는 더 짧게만 조정 가능)과 나머지 종류의 HTTP 응답 제한(기본 12초, 서버 쿼리 중단 보장 없음)을 구분합니다. 저장 값 1–60초의 실제 상한은 55초이며 폼에서 적용 제한을 표시합니다. ClickHouse database는 양쪽 검증 계층에서 system/information_schema를 거부하고, Mimir API 접두사는 기존 경로와 중복되지 않게 설정할 수 있습니다. 명시한 인스턴스 자격증명이 없으면 실패하며 상위 성공·오류 응답은 파싱 전에 바이트 제한을 적용합니다. 연결 테스트는 실제 bounded 쿼리 1회를 실행해(readiness 엔드포인트만으로는 통과하지 않음) 지연 시간과 검증한 작업을 고정 문구로 보고하며, Jaeger/Dynatrace/Datadog는 잘못된 응답과 빈 결과를 구분하고, Jaeger와 Loki는 요청 상한 도달을 요청/반환 개수와 함께 ‘부분 가능’으로 표시해 실제 누락과 구분합니다. 커넥터 요청은 연결·헤더·본문이 하나의 제한 시간을 공유합니다. 기본 인스턴스의 공유 자격증명에는 인스턴스 id를 기록해 다른 인스턴스가 빌려 쓸 수 없으며, ID 없는 레거시 기본 경로만 호환되며, 마이그레이션된 기본 인스턴스는 자격증명 정보를 받지 않는 호출 측을 배포하기 전에 관리자가 백필해야 합니다. Jaeger/Dynatrace 내부망 옵션은 기본 OFF입니다. 커스텀 MCP는 관리자 확장성을 유지하면서 검증한 주소로 접속하고 원래 Host/SNI·인증서 검증과 리다이렉트 차단을 유지하며, 지원하지 않는 3-legged OAuth는 거부합니다. v2 쿼리 경로는 계속 무캐시이고 Timeout 단위는 v1의 밀리초가 아닌 초입니다. - ECS 통합 개요 페이지(/inventory/ecs, 사이드바 ECS 서브그룹의 'ECS 개요'): v1의 한 화면 뷰 복원 — 요약 KPI 밴드(클러스터/서비스 수는 로드된 페이지 기준[500 캡 도달 시 '+'], 태스크 수는 공용 summary, Desired 대비 미달 태스크 타일은 비절단 서비스 페이지에서만 집계 — 표본을 전체 합계처럼 제시하지 않음)와 클러스터·서비스 테이블을 한 화면에 세로로 배치. 각 테이블은 검색/패싯/상세가 있는 타입 페이지로 '전체 보기' 링크(개요는 읽기 전용 글랜스 레이어), 500행 이상은 표본 표기, sync 실패 시 오래된 데이터 캡션, 미수집 시 '미수집' 표시(빈 플릿 조작 없음). - 대시보드 온디맨드 동기화: 헤더에 관리자 전용 '전체 동기화' 버튼 추가 — 전체 타입 인벤토리 sync를 1회 dispatch(비동기 배치 시맨틱을 그 자리에서 공지 — 큐 등록 확인일 뿐 완료 보장이 아니며 이미 실행 중인 타입은 건너뜀; 관리자 전용·sync 비활성 상태를 일시 오류와 구분해 표시; 데이터는 수 분 후 일반 Refresh로 반영, 낙관적 갱신 없음). - S3 보안 드릴다운: S3 페이지에 v1의 리전별 버킷 맵(블록 타일 — Public=빨강 > Versioned=초록 > Standard=시안, 플래그 미동기화 버킷은 명시적 Unknown=회색으로 표시해 확정 Standard로 읽히지 않음; 블록 클릭 시 상세 패널, 500행 캡 초과 시 표본 기준 표기; Public 플래그 미상 버킷은 버저닝이 알려져 있어도 Unknown — 거부된 정책 조회가 안심 색으로 가려지지 않음), 버킷 상세에 'S3 접근 권한 보유 IAM Role' 섹션(관리자 전용 — 비관리자에겐 권한 안내 표시; 신규 sync되는 연결 AWS 관리형 정책이 검사 세트[AmazonS3*/AdministratorAccess/PowerUserAccess/ReadOnlyAccess, job-function 경로 포함]에 일치하는 경우 최대 30개 — 다른 정책도 S3를 부여할 수 있어 빈 결과는 검사-세트 한정 문구; 마지막 sync run 상태가 모든 결론을 게이트 — 실패/부분 run에는 오래된 데이터 배너, 빈 결과는 24시간 내 성공·비절단 run에서만 확정(하단에 기준 시각 표기); 500행 표본 표기; 인라인/버킷 정책 경유 접근은 범위 밖임을 명시, 정책 목록 sync 전에는 '미동기화' 안내 — terraform apply + 다음 sync 후 표시; 참고: 정책 목록 하이드레이트 실패(SCP 차단, 또는 전 계정 합산 role 수가 rate-limit 예산 초과) 시 선택적인 연결 정책 열을 제외해 재시도하며(다른 추가 조회는 남아 있어 실패할 수 있음) 기본 iam_role 인벤토리는 유지되고 이 섹션만 '미동기화'로 표시(run은 degraded freshness로 공개) — 이 경로의 run 전체 failed·전 계정 last-good 동결은 기본 쿼리까지 실패한 경우이며 그 외 최종 run 상태는 통상 sync 라이프사이클을 따름 — ADR-010 개정의 공지된 시맨틱) 추가. @@ -763,7 +763,7 @@ First release of the **v2 line** (versioned independently from the v1 1.x line, - AI 진단 생성 UX 퀵윈(v1 패리티): 생성 중 mm:ss 경과 타이머와 섹션별 체크리스트 그리드(완료/대기 2단계, UI 언어로 표시 — 워커가 progress JSONB에 추가로 스트리밍하는 `completed` 목록 기반; 동시 렌더 특성상 진행 중 섹션 텔레메트리가 존재하지 않아 스피너는 의도적으로 제공하지 않음; 이전 형식의 진행 중 행·카탈로그 드리프트 시 기존 진행 바 뷰로 폴백), 완료 리포트에 통계 바(섹션 수 · 소요 · 리포트 ID — 소요는 종료 시점에 기록되는 신규 `finished_at` 컬럼[추가 마이그레이션 1건] 기반; 값이 없는 레거시 행은 소요를 생략하고 절대 임의 산출하지 않음), 빈 상태에 Deep 티어 태그가 붙은 전체 섹션 범위 프리뷰, 완료된 히스토리 행에 인라인 MD/DOCX 다운로드 링크(리포트를 열지 않고 즉시 다운로드) 추가. - AI 진단 패리티 배치(v1 패리티): 완료 리포트를 접을 수 있는 섹션 카드 + 고정 목차 사이드바(클릭 시 해당 섹션으로 스크롤) + 본문 키워드 기반 섹션별 심각도 아이콘(점수가 아닌 표시 휴리스틱임을 명시; 섹션 헤딩이 없는 리포트는 기존 연속 뷰 유지)으로 렌더링; 리포트 생성 언어 선택(한국어/English/中文/日本語 — UI 언어 기본값, 수동 실행·스케줄 모두 적용; 언어가 실행 dedup 키에 포함되어 같은 시간대 언어 전환 시 이전 언어 리포트를 재사용하지 않고 새로 실행; 롤링 배포 호환을 위한 레거시 dedup 키 read 폴백이 이번 릴리스 한정으로 포함 — 다음 릴리스에서 제거); 자동 진단 스케줄에 KST 상세 설정(매주/격주 요일, 매월 1–28일, 실행 시각)과 최근 실행 시각 표시 추가 — 미설정 필드는 기존 주기-간격 동작 유지; 구독자 패널에서 관리자가 테스트 알림을 발송 가능(기존 진단 토픽 한정 SNS publish 1건 — web 태스크 최초의 관리자 전용 `sns:Publish`; 발송 실패는 조용한 성공이 아니라 에러로 표시). - 인벤토리 KPI/차트 퀵윈 배치(v1 패리티): EC2에 실행 중 총 vCPU 타일(인스턴스별 `cpu_options` 코어×스레드 — 타입 기본값이 아닌 실제 vCPU), RDS에 총 할당 스토리지 타일, Lambda에 장기 타임아웃(>300s, danger)·평균 메모리 타일, EBS 볼륨에 암호화율 타일(100% → accent, <80% → danger) 추가; ECS 클러스터는 일반 상태 타일 대신 전용 KPI 밴드(ACTIVE 수·실행 태스크·활성 서비스·컨테이너 인스턴스)를 표시; ECR 행에 Scan on Push 컬럼 추가(스캔 설정 누락/파싱 불가 시 API 기본값인 No로 집계); CloudWatch 알람 상태 도넛은 크기순 팔레트 색 대신 고정 시맨틱 컬러(초록 OK / 빨강 ALARM / 회색 INSUFFICIENT_DATA)를 사용. -- 데이터소스 Explore/관리 패리티 배치: 8개 커넥터 타입 전부에 큐레이트 예제 쿼리·자연어 프롬프트 칩, Loki 전용 로그 스트림 뷰어(타임스탬프·라벨 배지·스크롤 패널), 7d/30d 기간 프리셋(kind별 API 상한: prometheus/mimir는 커넥터가 업스트림 `timeout`을 전달하는 조건으로 30d — 커넥터 변경은 `terraform apply`로 배포되므로 30d 의존 전 apply 필요; Loki는 7d; 5000-포인트 밀도 캡 유지), 결과 메타데이터 바(행/시리즈 수 · 실행 ms · 쿼리 언어 · 형태), NL→쿼리 생성 후 닫을 수 있는 "AI 생성됨" 배너, 관리 탭 KPI 타일과 수동 새로고침 버튼, kind별 DEFAULT 데이터소스 행의 "AI로 진단" 딥링크(챗 도구 경로가 kind별 default를 해석하므로 default 행·지원 kind 한정, 섹션 고정 프롬프트로 `/assistant?q=` 컴포저 프리필 — 검토 전용, 자동 전송 없음), Tempo/Jaeger 트레이스 결과의 비례 duration 바 추가. +- 데이터소스 Explore/관리 패리티 배치: 8개 커넥터 타입 전부에 큐레이트 예제 쿼리·자연어 프롬프트 칩, Loki 전용 로그 스트림 뷰어(타임스탬프·라벨 배지·스크롤 패널), 7d/30d 기간 프리셋(Prometheus/Mimir/Dynatrace/Datadog 최대 30일, Loki/Tempo/Jaeger 최대 7일, 5000-포인트 밀도 상한 유지), 결과 메타데이터 바(행/시리즈 수 · 실행 ms · 쿼리 언어 · 형태), NL→쿼리 생성 후 닫을 수 있는 "AI 생성됨" 배너, 관리 탭 KPI 타일과 수동 새로고침 버튼, kind별 DEFAULT 데이터소스 행의 "AI로 진단" 딥링크(지원 kind 한정, `/assistant?q=` 컴포저 프리필, 검토 전용·자동 전송 없음), Tempo/Jaeger 트레이스 결과의 비례 duration 바를 제공합니다. 검색 기간은 별도 range 도구 유무와 관계없이 전달하고 SQL API 기간은 무시하지 않고 거부하며 실제 요청 기간과 공유 데이터소스 범위를 명시합니다. AI 생성은 선택한 기간을 변경하지 않습니다. 데이터소스 상태는 기본 선택 여부로 연결을 추정하지 않고 확인된 설정과 미확인 연결을 구분합니다. Datadog 연결 테스트는 키 쌍과 bearer 토큰의 메트릭 조회 접근까지 확인하며 Datadog 포인트 상한과 Dynatrace 내부 시리즈 상한·호환 응답의 추가 페이지는 부분 결과로 표시합니다. 의존하는 Web 변경보다 Lambda producer를 먼저 배포해야 합니다. - 데이터소스 상세 페이지에 사전 생성 카드 대시보드 추가: 등록 시(및 일일 인덱스 배치마다) 캐시된 스키마로부터 예상 카드 세트를 도출하고 각 카드가 사용할 쿼리를 미리 저장(신규 `datasource_dashboard_cards` 테이블 + 인덱스 워커의 결정론적 `card_catalog` — prometheus/mimir는 타깃·CPU·메모리·디스크·로드·네트워크·컨테이너·재시작을 포괄하는 13종, loki 2종·tempo 2종·clickhouse 2종) — ready Prometheus/Mimir 카드는 등록 전에 해당 데이터소스에서, 페이지가 실제 실행할 instant/range 툴 그대로 라이브 검증하며(확정 PromQL 오류 — 응답 본문 기반, 단순 HTTP 4xx 아님 — 는 해당 카드만 비활성화하고 다음 일일 실행에서 재검증, 일시적 커넥터 장애·재수집 실패·카드 요구 메트릭을 판정할 수 없는 절단 스키마는 모두 기존 카드 세트 보존; 커넥터 metric-metadata 툴에는 확정 `exists` 플래그와 업스트림 3초 제한 추가) — 카드 빌드는 기존 datasource-index 잡 내부에서 실행되므로 diag-signal 칩과 동일하게 `datasource_diagnosis_enabled` 게이트(기본 false; `workers_enabled`/`agentcore_enabled`/`integrations_enabled` 선행) 하에 동작 — 페이지가 저장된 쿼리를 기존 read-only 쿼리 API로 조회 시점에 라이브 실행해 렌더링(stat/시계열/테이블 카드, 미충족 카드는 누락 항목과 함께 비활성 표시, 실패 카드는 조용한 0이 아니라 인라인 에러로 표시). - 토폴로지 인프라 페이지에 컬럼형 맵 뷰 2종 추가 — 5컬럼 인프라 리소스 맵(External | VPC | Subnet | Compute | NAT)과 클러스터별 K8s 맵(Ingress → Service → Pod → Node — in-cluster 조회 경로가 host 스코프라 host 계정의 connected 클러스터만 대상) — 고정 컬럼 ReactFlow에 실제 엣지 연결선을 그리는 그래프로 렌더링되며, 클릭 교차 하이라이트·검색 하이라이트·색상 범례 포함. 기존 인벤토리/EKS 조회와 기존 read-only `/api/tgw` 라이브 어태치먼트 조회(호스트 계정 스코프 한정·최대 20개 — 미조회 시 UI에 표시)를 사용하며, 서버 측 추가는 read-only `ingresses` in-cluster kind 1종뿐. diff --git a/docs/api-reference.md b/docs/api-reference.md index 3e9bcc036..0cf3b1264 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -379,11 +379,11 @@ reachability. | `/api/cost/detail` | GET | 서비스별 비용 상세 (`?service=` 필수, ≤100자) | verifyUser | | `/api/finops/findings` | GET | ADR-020 FinOps 기본 권장 엔진 — 미해결 findings + 최근 배치 실행(`finops_runs`) 조회, Aurora만 읽음(라이브 AWS 호출 없음). `finops_baseline_enabled=false`면 `{enabled:false, findings:[], lastRun:null}` | verifyUser | | `/api/customization` | GET, POST, PUT | Admin skill/agent catalog CRUD; invalid tool policy → 400; unavailable validation/read → 503 (see below) | verifyUser | -| `/api/datasources` | GET | 데이터소스 인스턴스 목록 — 크리덴셜 미노출 | verifyUser | +| `/api/datasources` | GET | 데이터소스 인스턴스 목록 — 크리덴셜 미노출. 행마다 `configured`(인스턴스 자체의 설정 근거만, 연결 성공 주장 아님)·`connectionStatus`·`scope: 'shared'`(기본 인스턴스는 타입별 공유 연결) 포함 | verifyUser | | `/api/datasources/generate` | POST | 자연어 → 쿼리 초안 생성 (리뷰용 — 절대 실행 안 함) | verifyUser | -| `/api/datasources/manage` | POST, PATCH | 인스턴스 생성/수정 + 크리덴셜 저장 (admin); `settings`(timeoutS 1–60[clickhouse 유효 최대 55]·clickhouse database)는 서버 측 sanitize 후 ds_settings JSONB에 저장 | verifyUser | +| `/api/datasources/manage` | POST, PATCH | 인스턴스 생성/수정 + 크리덴셜 저장 (admin); `settings`(timeoutS 1–60[모든 kind 유효 최대 55 — Prometheus/Mimir/ClickHouse는 서버 실행 제한, 그 외는 HTTP 응답 제한]·clickhouse `database`·mimir `apiPrefix`)는 서버 측 sanitize 후 ds_settings JSONB에 저장 | verifyUser | | `/api/datasources/query` | POST | 인스턴스 대상 read-only 쿼리 실행 (admin 아님 — 탐색용) | verifyUser | -| `/api/datasources/test` | POST | 저장 전 연결 probe — SSRF 가드 (admin) | verifyUser | +| `/api/datasources/test` | POST | 저장 전 연결 테스트 — 실제 bounded 쿼리 1회 실행(readiness 엔드포인트 아님), `{ok, latencyMs, verified, error?}` 반환, SSRF 가드 (admin) | verifyUser | | `/api/datasources/[id]` | DELETE | 인스턴스 삭제 — 스키마 캐시/크리덴셜 cascade, 기본값 재선정 (admin) | verifyUser | | `/api/datasources/[id]/cards` | GET | 사전 생성 대시보드 카드 조회 (read-only, auth) | verifyUser | | `/api/datasources/[id]/default` | POST | kind별 기본 인스턴스 지정 — 트랜잭션으로 기존 기본 해제 (admin) | verifyUser | diff --git a/docs/reference/README.md b/docs/reference/README.md index 1bb48f358..af0649c22 100644 --- a/docs/reference/README.md +++ b/docs/reference/README.md @@ -83,6 +83,12 @@ scoped registered-fleet coverage from incomplete wildcard discovery. **E2E observability — [observability-e2e.md](observability-e2e.md).** The opt-in `/topology?view=e2e` page connects account/region/global-scoped configuration, host service snapshots and explicit NFM queries using source-quality and scoped-identity gates. The reference distinguishes implemented contracts from the broader observability roadmap. +**External connections — [external-connections.md](external-connections.md).** The eight datasource +HTTP adapters and the administrator-managed custom MCP path: shared datasource scope, +explicit instance identity, bounded responses, query-level connection tests, completion +states and pinned custom-MCP transport. Registration, verification and deployment are +distinct states. + **VPC connectivity — [vpc-connectivity.md](vpc-connectivity.md).** On-demand peering and TGW attachment observations on `/inventory/vpc` and the `/topology/infra?view=vpc` tab, scoped to an enabled collecting account, region and diff --git a/docs/reference/external-connections.md b/docs/reference/external-connections.md new file mode 100644 index 000000000..ad6a9280d --- /dev/null +++ b/docs/reference/external-connections.md @@ -0,0 +1,135 @@ +# External observability connection contract + +This is the source-supported contract for the eight datasource HTTP adapters and +the separate administrator-managed MCP path. Registration, credential presence, +query verification, MCP provisioning and live deployment are different states. + +## Datasource scope and identity + +Datasource Explore is available to authenticated users. Create, edit, delete, +set-default, connection test and credential operations require administrator +authority. Built-in chat resolves one shared default per kind; selecting an AWS +account does not select a different datasource or tenant. + +Explore resolves credentials by instance ID and retains the row's endpoint, +authentication method and settings as authoritative. Lambda/worker requests with +an explicit instance ID fail when that instance is not configured; they cannot +borrow another instance's kind-default credential. The no-ID legacy default path +remains available. Secrets Manager's bounded credential cache does not provide +immediate revocation of previously cached credentials. + +Per-account custom integration selection and exposed-tool ceilings are enforced +separately. They do not restrict built-in shared datasource defaults. External +telemetry needs independent account/region evidence before identity correlation. + +## Capabilities and query windows + +| Kind | Query | Time window | Built-in chat | Private-network option | +|---|---|---|---|---| +| Prometheus | PromQL instant/range | Instant or up to 30 days | `external-obs` | `prometheus_vpc_enabled` | +| Mimir | PromQL instant/range | Instant or up to 30 days | `monitoring` | `mimir_vpc_enabled` | +| Loki | LogQL instant/range | Instant or up to seven days | `monitoring` | `loki_vpc_enabled` | +| Tempo | TraceQL search | One-hour default; up to seven days | `monitoring` | `tempo_vpc_enabled` | +| ClickHouse | Read-only SQL | Time predicates belong to SQL | `external-obs` Lambda | `clickhouse_vpc_enabled` | +| Jaeger | Service-anchored trace search | One-hour default; up to seven days | No built-in target | `jaeger_vpc_enabled` | +| Dynatrace | Environment Metrics API v2 metricSelector | One-hour default; up to 30 days | Separate official MCP preset | `dynatrace_vpc_enabled` | +| Datadog | Metrics REST API query | One-hour default; up to 30 days | Separate official MCP preset | Public endpoint | + +`POST /api/datasources/query` accepts `range: false`, legacy `range: true`, +or `{window, step}` in seconds. Search tools receive `start`/`end` independently +of whether a distinct range tool exists. Unsupported SQL API ranges are rejected. +Metadata includes the effective query window and shared scope. Range density +remains bounded; selecting a window does not guarantee complete provider coverage. +Natural-language drafting does not itself change the selected request window. + +Explore support does not establish service-graph support. Metrics-based graph +sources require suitable Prometheus/Mimir schema; trace sources require suitable +Tempo or ClickHouse data. Loki, Jaeger, Dynatrace and Datadog have no bundled +service-graph relationship adapter in the current catalog. + +## Endpoints and credentials + +Endpoints are HTTP(S) datasource bases, optionally including a reverse-proxy +mount. They are validated at registration and DNS-checked/pinned at connection. +Redirects and metadata/loopback/link-local targets are blocked. Datasource HTTP +permits private addresses; reachability still requires operator-managed networks. + +Mimir's `settings.apiPrefix` defaults to `/prometheus`. An empty value selects +`/api/v1` at the endpoint base. A non-empty prefix uses slash-separated letters, +digits, underscores and hyphens, up to 128 characters. If the endpoint already +ends in that prefix it is not appended twice. + +Generic datasource credentials support none/basic/bearer/custom headers and +an optional tenant header. Datadog's datasource API uses its API/application-key +pair; this differs from the official hosted MCP bearer PAT/SAT. Dynatrace's direct +environment adapter uses `Api-Token`; its official hosted MCP uses a bearer +Platform token. Official MCP credentials remain namespaced under `mcp:`. + +Jaeger currently uses its internal `/api/*` JSON compatibility interface, rather +than the stable v3 interface. A provider version change requires query-envelope +and time-unit validation before claiming compatibility. + +## Deadlines and completion + +Stored `timeoutS` remains an integer from 1 to 60 for compatibility. The effective +maximum is 55 seconds within the existing Lambda wall limit. + +- Prometheus/Mimir/ClickHouse apply a provider execution ceiling on query paths. + Default is 10 seconds; callers can tighten the ceiling. Their query HTTP + deadline is aligned above the execution bound. +- Loki/Tempo/Jaeger/Dynatrace/Datadog apply the saved `timeoutS` as an HTTP response + timeout, defaulting to 12 seconds. This does not establish provider-side query + cancellation. Connect, headers and body share that one deadline. +- Schema discovery may use shorter bounded operation deadlines. +- Success bodies are limited to 4 MiB before parsing; error bodies to 64 KiB. + Overflow fails explicitly and is never represented as an empty dataset. + +Connection tests use bounded reads against the actual supported query operation, +with the configured auth and tenant, and report `{ok, verified, latencyMs, error?}` +with fixed sentences (an SSRF policy rejection keeps its own classification). +Verified query access does not establish access to every metric, trace, dataset or +optional tool. + +Query producers distinguish verified complete `ok`/`empty` from `partial`, +`unknown` and `error`. Malformed/unrecognized responses cannot establish absence; +usable valid siblings may be retained with uncertainty. Jaeger and Loki disclose +the requested limit and returned count; reaching the requested limit is +`limitSaturated` and potentially partial even when no additional record is proven, +while `truncated` is reserved for records the connector actually dropped. +Actual HTTP 206 evidence is retained independently of upstream JSON fields and +cannot certify an empty dataset. + +## Private connectivity and MCP governance + +Every VPC option defaults false. Jaeger/Dynatrace's new options require both +AgentCore and integrations. An operator must separately verify private DNS, +subnet routing, target ingress and required AWS API access. No SG ingress or +Kubernetes grant is created by registering a datasource. + +Official MCP gates, exact read-tool allowlists and endpoint acknowledgements +remain governed by ADR-017. Dynatrace's empty official allowlist still authorizes +zero tools. ClickHouse's stdio official MCP remains frozen dark code. + +Custom external MCP registration remains allowed. Its runtime pins a request-local +validated address set with original Host/SNI and certificate verification, +rejects changed origins and redirects, disables proxy routing, and bounds +address fallback without replaying requests after read/write failures. +HTTPX/HTTPcore versions implementing the tested SNI contract are pinned in the +runtime dependencies. + +`oauth_3lo` is not executable: validation rejects it at registration, the form does not +offer it, enablement requires the default-off custom MCP governance gate (which admits +only `api_key`/`oauth_client_credentials`), and a historical row fails closed at connect. +The existing `oauth_client_credentials` transport consumes a previously stored +bearer token; it does not implement issuance or refresh. External write governance, +AWS-mutation freeze and per-account exposed-tool ceilings are unchanged. + +## Rollout order + +Before strict credential-blind producers roll out, complete administrator backfill +and verification for migrated defaults using +[Datasource credential identity and rollout](../runbooks/datasource-credential-isolation.md). +Deploy the helper and Lambda producers before the Web reader/UI. Roll out the +AgentCore image containing the pinned transport separately. Review a saved +Terraform plan before applying optional private-network settings. No source merge +alone proves activation or vendor readiness. diff --git a/web/app/api/datasources/authz.test.ts b/web/app/api/datasources/authz.test.ts index eb40389f2..29f4a0b9c 100644 --- a/web/app/api/datasources/authz.test.ts +++ b/web/app/api/datasources/authz.test.ts @@ -16,7 +16,7 @@ vi.mock('@/lib/datasources', () => ({ createDatasource: vi.fn(), updateDatasource: vi.fn(), })); vi.mock('@/lib/integration-credentials', () => ({ - getConfiguredIds: vi.fn(async () => []), + getConfiguredDatasourceIds: vi.fn(async () => []), setIntegrationCredentialById: vi.fn(), mirrorDefaultCredential: vi.fn(), })); vi.mock('@/lib/mcp-lambda-invoke', () => ({ invokeMcpLambdaTool: vi.fn(), KNOWN_MCP_LAMBDA_KINDS: ['prometheus'] })); diff --git a/web/app/api/datasources/manage/route.test.ts b/web/app/api/datasources/manage/route.test.ts index 0f75a7df2..63eab3e48 100644 --- a/web/app/api/datasources/manage/route.test.ts +++ b/web/app/api/datasources/manage/route.test.ts @@ -59,6 +59,28 @@ beforeEach(() => { }); describe('POST create', () => { + it('legacy Datadog credentials cannot follow a changed endpoint origin', async () => { + getDatasource.mockResolvedValue({ id: 7, name: 'dd', kind: 'datadog', endpoint: 'https://old.example.test', + authType: 'custom_header', isDefault: true, settings: {} }); + getCredentialById.mockResolvedValue({ endpoint: 'https://old.example.test', authType: 'custom_header', + apiKey: 'synthetic-api', appKey: 'synthetic-app' }); + const { PATCH } = await import('./route'); + expect((await PATCH(req({ id: 7, endpoint: 'https://new.example.test' }, 'PATCH'))).status).toBe(200); + const blob = setIntegrationCredentialById.mock.calls.at(-1)![1]; + expect(blob.endpoint).toBe('https://new.example.test'); + expect(blob).not.toHaveProperty('apiKey'); + expect(blob).not.toHaveProperty('appKey'); + }); + it('a settings-only PATCH keeps a Datadog legacy key pair whatever the row authType says', async () => { + getDatasource.mockResolvedValue({ id: 7, name: 'dd', kind: 'datadog', endpoint: 'https://api.datadoghq.com', + authType: 'none', isDefault: false, settings: {} }); + getCredentialById.mockResolvedValue({ endpoint: 'https://api.datadoghq.com', authType: 'none', + apiKey: 'synthetic-api', appKey: 'synthetic-app' }); + const { PATCH } = await import('./route'); + expect((await PATCH(req({ id: 7, settings: { timeoutS: 20 } }, 'PATCH'))).status).toBe(200); + const blob = setIntegrationCredentialById.mock.calls.at(-1)![1]; + expect(blob).toMatchObject({ apiKey: 'synthetic-api', appKey: 'synthetic-app', timeoutS: 20 }); + }); it('403 non-admin, no writes', async () => { isAdmin.mockResolvedValue(false); const { POST } = await import('./route'); diff --git a/web/app/api/datasources/manage/route.ts b/web/app/api/datasources/manage/route.ts index 8dc33c4e0..a58ffc840 100644 --- a/web/app/api/datasources/manage/route.ts +++ b/web/app/api/datasources/manage/route.ts @@ -22,6 +22,9 @@ const AUTH_TYPES = ['none', 'basic', 'bearer', 'custom_header']; // through the creds spread (round-5: creds.endpoint would otherwise override the validated // endpoint in the blob without tripping the host-change guard). const CRED_KEYS = ['username', 'password', 'token', 'headerName', 'headerValue', 'headerName2', 'headerValue2', 'org_id'] as const; +// Legacy server-side Datadog blobs can contain a dedicated key pair. They are not client +// writable here, but must participate in the same origin/auth-change scrub. +const STORED_AUTH_KEYS = [...CRED_KEYS, 'apiKey', 'appKey'] as const; /** True when a present `settings` value is not a plain object — 400, never a silent clear * (round-8: null/array/string/number shapes bypassed the round-6 empty-sanitize guard). */ function settingsShapeInvalid(body: Record): boolean { @@ -87,6 +90,7 @@ export async function POST(request: Request) { // database is ClickHouse-only config — never persist it on the row for other kinds either // (round-12: the blob-side delete alone left stale, admin-visible row config). if (kind !== 'clickhouse') delete settings.database; + if (kind !== 'mimir') delete settings.apiPrefix; if (body.settings && typeof body.settings === 'object' && !Array.isArray(body.settings) && Object.keys(body.settings as object).length > 0 && Object.keys(settings).length === 0) { return json({ error: 'invalid settings (timeoutS 1–60 integer; database identifier, non-system)' }, 400); @@ -139,6 +143,7 @@ export async function PATCH(request: Request) { if (settingsShapeInvalid(body)) return json({ error: 'settings must be an object' }, 400); const settings = body.settings !== undefined ? sanitizeDsSettings(body.settings) : undefined; if (settings !== undefined && ds.kind !== 'clickhouse') delete settings.database; + if (settings !== undefined && ds.kind !== 'mimir') delete settings.apiPrefix; if (body.settings && typeof body.settings === 'object' && !Array.isArray(body.settings) && Object.keys(body.settings as object).length > 0 && settings !== undefined && Object.keys(settings).length === 0) { return json({ error: 'invalid settings (timeoutS 1–60 integer; database identifier, non-system)' }, 400); @@ -185,6 +190,7 @@ export async function PATCH(request: Request) { delete existing.timeoutS; delete existing.database; delete existing.instanceId; + delete existing.apiPrefix; // ORIGIN compare (scheme+host+port — an https→http downgrade must count as a change, or // Basic material would transmit in cleartext); a malformed URL counts as changed. const originOf = (u: string | null | undefined): string | null => { try { return new URL(u ?? '').origin; } catch { return null; } }; @@ -192,21 +198,24 @@ export async function PATCH(request: Request) { // instance's blob — trust it as a merge base only when its endpoint origin matches THIS // row's; otherwise drop the auth keys rather than bind foreign creds to this endpoint. if (ds.isDefault && existing.endpoint && originOf(String(existing.endpoint)) !== originOf(ds.endpoint)) { - for (const k of CRED_KEYS) delete existing[k]; + for (const k of STORED_AUTH_KEYS) delete existing[k]; } const hostChanged = endpoint !== undefined && originOf(endpoint) !== originOf(ds.endpoint); // On a host change, stored auth material is dropped UNCONDITIONALLY (round-5: a partial // creds object like {username} must not carry the stored password to the new origin) — // whatever the client genuinely re-supplied is reinstated by the creds spread below. if (hostChanged) { - for (const k of CRED_KEYS) delete existing[k]; // org_id included — tenant id is host-scoped + for (const k of STORED_AUTH_KEYS) delete existing[k]; // tenant and legacy keys are origin-scoped } const effAuth = authType ?? ds.authType ?? 'none'; const KEEP_BY_AUTH: Record = { none: [], basic: ['username', 'password'], bearer: ['token'], - custom_header: ['headerName', 'headerValue', 'headerName2', 'headerValue2'], + custom_header: ['headerName', 'headerValue', 'headerName2', 'headerValue2', 'apiKey', 'appKey'], }; - for (const k of CRED_KEYS) if (k !== 'org_id' && !(KEEP_BY_AUTH[effAuth] ?? []).includes(k)) delete existing[k]; + // A Datadog instance's legacy key pair is its credential whatever the row's authType says, so an + // unrelated settings-only PATCH never deletes it (only an origin change or an explicit re-auth does). + const keep = ds.kind === 'datadog' ? [...(KEEP_BY_AUTH[effAuth] ?? []), 'apiKey', 'appKey'] : (KEEP_BY_AUTH[effAuth] ?? []); + for (const k of STORED_AUTH_KEYS) if (k !== 'org_id' && !keep.includes(k)) delete existing[k]; // creds is key-allowlisted (pickCredKeys) so nothing here can override the validated // endpoint/authType/settings fields regardless of spread order. const blob: Record = { @@ -218,7 +227,7 @@ export async function PATCH(request: Request) { }; // Prune the FINAL blob too (round-10: {authType:'none', creds:{password}} re-added the // key AFTER the merge-base pruning) — keys outside the effective authType never persist. - for (const k of CRED_KEYS) if (k !== 'org_id' && !(KEEP_BY_AUTH[effAuth] ?? []).includes(k)) delete blob[k]; + for (const k of STORED_AUTH_KEYS) if (k !== 'org_id' && !keep.includes(k)) delete blob[k]; // database is ClickHouse-only config — never persist it for other kinds (inert but stale). if (ds.kind !== 'clickhouse') delete blob.database; await setIntegrationCredentialById(id, blob, client); diff --git a/web/app/api/datasources/query/route.test.ts b/web/app/api/datasources/query/route.test.ts index 6569f59bf..7e19a1e1b 100644 --- a/web/app/api/datasources/query/route.test.ts +++ b/web/app/api/datasources/query/route.test.ts @@ -94,7 +94,7 @@ describe('POST /api/datasources/query', () => { }); // gap L203: per-datasource timeout setting - it('an instance timeoutS rides prometheus args (capped at 10s under the connector HTTP timeout)', async () => { + it('an instance timeoutS rides prometheus args with the Lambda execution ceiling', async () => { getDatasource.mockResolvedValue({ id: 3, kind: 'prometheus', settings: { timeoutS: 5 } }); resolveConnConfig.mockResolvedValue({ endpoint: 'http://prom.internal:9090' }); invokeMcpLambdaTool.mockResolvedValue({ resultType: 'vector', result: [] }); @@ -103,7 +103,7 @@ describe('POST /api/datasources/query', () => { expect(invokeMcpLambdaTool.mock.calls.at(-1)![0].args.timeout).toBe('5s'); getDatasource.mockResolvedValue({ id: 3, kind: 'prometheus', settings: { timeoutS: 60 } }); await POST(req({ id: 3, query: 'up' })); - expect(invokeMcpLambdaTool.mock.calls.at(-1)![0].args.timeout).toBe('10s'); // capped + expect(invokeMcpLambdaTool.mock.calls.at(-1)![0].args.timeout).toBe('55s'); }); it('clickhouse sends NO timeout arg — the bound rides connConfig (the connector defaults/clamps it)', async () => { @@ -201,12 +201,12 @@ describe('POST /api/datasources/query', () => { expect(invokeMcpLambdaTool.mock.calls.at(-1)![0].tool).toBe('prometheus_query'); }); - it('legacy range:true → range tool with NO start/step args (1h connector default)', async () => { + it('legacy range:true retains the one-hour window and default step', async () => { const { POST } = await import('./route'); await POST(req({ slug: 'prometheus', query: 'up', range: true })); const call = invokeMcpLambdaTool.mock.calls.at(-1)![0]; expect(call.tool).toBe('prometheus_query_range'); - expect(call.args.start).toBeUndefined(); + expect(Number(call.args.end) - Number(call.args.start)).toBe(3600); expect(call.args.step).toBeUndefined(); }); }); diff --git a/web/app/api/datasources/query/route.ts b/web/app/api/datasources/query/route.ts index 07d17ec6e..06c9ffc79 100644 --- a/web/app/api/datasources/query/route.ts +++ b/web/app/api/datasources/query/route.ts @@ -10,7 +10,7 @@ import { isDatasourceKind } from '@/lib/integrations-category'; import { assertDatasourceEndpointAllowed } from '@/lib/ssrf-guard'; import { normalizeResult } from '@/lib/datasource-render'; import { readJsonBounded, BodyTooLargeError } from '@/lib/http-body'; -import { TOOL } from '@/lib/datasource-query-tools'; +import { TOOL, EFFECTIVE_TIMEOUT_CAP, EXECUTION_TIMEOUT_KINDS } from '@/lib/datasource-query-tools'; export const dynamic = 'force-dynamic'; @@ -30,12 +30,18 @@ export async function POST(request: Request) { if (e instanceof BodyTooLargeError) return json({ error: 'request body too large' }, 413); return json({ error: 'invalid JSON body' }, 400); } + if (!body || typeof body !== 'object' || Array.isArray(body)) return json({ error: 'body must be an object' }, 400); // Resolve the kind + (for an instance id) the inline conn-config. let kind = ''; let connConfig: ConnConfig | undefined; let dsTimeoutS: number | undefined; // gap L203: per-datasource upstream execution bound const id = Number(body.id); + if (body.id !== undefined && ( + !['number', 'string'].includes(typeof body.id) + || typeof body.id === 'string' && !/^[1-9]\d*$/.test(body.id) + || !Number.isSafeInteger(id) || id <= 0 + )) return json({ error: 'invalid datasource instance id' }, 400); if (Number.isInteger(id) && id > 0) { const ds = await getDatasource(id); if (!ds || !isDatasourceKind(ds.kind)) return json({ error: 'unknown datasource instance' }, 400); @@ -60,22 +66,20 @@ export async function POST(request: Request) { const args: Record = { [spec.arg]: query, ...(spec.extra ?? {}) }; - // Upstream execution bound (review hardening + gap L203 per-datasource setting): - // prometheus/mimir accept a `timeout` API param — the effective value is the datasource's - // own timeoutS (validated 1..60) further capped at 10s so it stays UNDER the connector's - // 12s HTTP timeout (a longer upstream bound than the HTTP client's is dead config). - // clickhouse needs NO arg here: its timeoutS rides the conn config (resolveConnConfig) and - // the connector applies it as the default max_execution_time on EVERY path — Explore, - // service-graph sources, and the agent/worker secret path — aligning its own HTTP timeout - // above the bound. Other kinds see no unknown arg. - if (kind === 'prometheus' || kind === 'mimir') args.timeout = `${Math.min(dsTimeoutS ?? 10, 10)}s`; + // Provider execution ceilings belong to the connector, on every invocation path. + // Keep the argument as an additional ceiling for older deployed PromQL producers. + if (kind === 'prometheus' || kind === 'mimir') args.timeout = `${Math.min(dsTimeoutS ?? 10, EFFECTIVE_TIMEOUT_CAP)}s`; - // Range mode: absent/false = instant; true = legacy 1h range (connector default); - // { window, step } = explicit time range. An object range is validated regardless of kind (so a bad - // window/step is a 400, not a silent instant); start/end are computed from the request clock. + // Search windows are independent of whether the provider exposes a separate range tool. let tool = spec.instant; const r = body.range; - if (r && typeof r === 'object') { + let queriedWindow: { start: string; end: string } | null = null; + const nowSec = Math.floor(Date.now() / 1000); + if (r !== undefined && r !== false && r !== true && (!r || typeof r !== 'object' || Array.isArray(r))) { + return json({ error: 'range must be false, true, or a window object' }, 400); + } + if (r && !spec.range && !spec.timeWindow) return json({ error: 'this datasource requires time filters in its query' }, 400); + if (r && typeof r === 'object' && !Array.isArray(r)) { const window = Number((r as { window?: unknown }).window); const step = Number((r as { step?: unknown }).step); // Upper bound widened to 30d (gap-audit L86, v1 parity) — the point-density cap below still @@ -83,10 +87,8 @@ export async function POST(request: Request) { // `timeout` param (range + instant) and, for Loki, by the tighter 7d window below (Loki accepts // no per-request timeout). // A 30d window at the UI's ~250-point autoStep is ~10368s steps. - // Per-kind upper bound (review): prometheus/mimir get 30d WITH a forwarded upstream `timeout` - // (their connectors clamp it 1..60s on BOTH instant and range paths); Loki has no per-request - // upstream timeout, so its widened bound stays at 7d. - const maxWindow = kind === 'loki' ? 604800 : 2592000; + // Metric queries permit 30d; log/trace searches retain the narrower 7d bound. + const maxWindow = ['loki', 'tempo', 'jaeger'].includes(kind) ? 604800 : 2592000; if (!Number.isInteger(window) || window < 60 || window > maxWindow) { return json({ error: `range.window must be an integer in [60, ${maxWindow}] seconds` }, 400); } @@ -99,22 +101,30 @@ export async function POST(request: Request) { if (Math.ceil(window / step) > 5000) { return json({ error: 'range too dense: ceil(window / step) must be ≤ 5000 points' }, 400); } - if (spec.range) { // kind has a range tool; otherwise the validated window is ignored (instant) - const nowSec = Math.floor(Date.now() / 1000); + args.start = String(nowSec - window); + args.end = String(nowSec); + queriedWindow = { start: args.start as string, end: args.end as string }; + if (spec.range) { tool = spec.range; - args.start = String(nowSec - window); - args.end = String(nowSec); args.step = String(step); } - } else if (r === true && spec.range) { - tool = spec.range; // back-compat: connector applies its 1h / 60s default + } else if (r === true || spec.timeWindow) { + if (spec.range) tool = spec.range; + args.start = String(nowSec - 3600); + args.end = String(nowSec); + queriedWindow = { start: args.start as string, end: args.end as string }; } try { const t0 = Date.now(); const result = await invokeMcpLambdaTool({ kind, tool, args, connConfig }); // Additive metadata for the Explore result bar (gap-audit L88) — existing consumers read only `result`. - return json({ result: normalizeResult(kind, tool, result), metadata: { executionTimeMs: Date.now() - t0, tool } }, 200); + const timeoutMode = EXECUTION_TIMEOUT_KINDS.has(kind) ? 'execution' : 'transport'; + return json({ result: normalizeResult(kind, tool, result), metadata: { + executionTimeMs: Date.now() - t0, tool, window: queriedWindow, scope: 'shared', + // A slug-only legacy call has no row settings available to the BFF. + ...(connConfig ? { timeoutMode, effectiveTimeoutS: Math.min(dsTimeoutS ?? (timeoutMode === 'execution' ? 10 : 12), EFFECTIVE_TIMEOUT_CAP) } : {}), + } }, 200); } catch (e) { return json({ error: e instanceof Error ? e.message : 'query failed' }, 502); } diff --git a/web/app/api/datasources/query/window.test.ts b/web/app/api/datasources/query/window.test.ts new file mode 100644 index 000000000..0725c24e2 --- /dev/null +++ b/web/app/api/datasources/query/window.test.ts @@ -0,0 +1,70 @@ +import { beforeEach, describe, expect, it, vi } from 'vitest'; + +const invoke = vi.fn(); +const getDatasource = vi.fn(); +vi.mock('@/lib/auth', () => ({ verifyUser: async () => ({ sub: 'synthetic-user' }) })); +vi.mock('@/lib/mcp-lambda-invoke', () => ({ invokeMcpLambdaTool: (...args: unknown[]) => invoke(...args) })); +vi.mock('@/lib/datasources', () => ({ + getDatasource: (...args: unknown[]) => getDatasource(...args), + resolveConnConfig: async () => ({ endpoint: 'https://provider.example.test', timeoutS: 35 }), +})); +import { POST } from './route'; + +function request(kind: string, range?: unknown, id?: number) { + return new Request('https://app.example.test/api/datasources/query', { + method: 'POST', headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ slug: kind, query: 'synthetic', range, id }), + }); +} + +beforeEach(() => { + invoke.mockReset().mockResolvedValue({ result: [], traces: [], series: [] }); + getDatasource.mockReset(); +}); + +describe('provider query-window contract', () => { + it.each(['tempo', 'jaeger', 'dynatrace', 'datadog'])('forwards a selected window to %s and returns the effective bounds', async kind => { + const response = await POST(request(kind, { window: 86400, step: 360 })); + expect(response.status).toBe(200); + const call = invoke.mock.calls[0][0]; + expect(Number(call.args.end) - Number(call.args.start)).toBe(86400); + expect(call.args.step).toBeUndefined(); + const body = await response.json(); + expect(body.metadata.window).toEqual({ start: call.args.start, end: call.args.end }); + expect(body.metadata.scope).toBe('shared'); + }); + + it.each(['tempo', 'jaeger', 'dynatrace', 'datadog'])('%s explicitly retains its one-hour default without a selected range', async kind => { + await POST(request(kind)); + const args = invoke.mock.calls[0][0].args; + expect(Number(args.end) - Number(args.start)).toBe(3600); + }); + + it.each([true, { window: 3600, step: 60 }])('rejects a SQL API range instead of ignoring it', async range => { + expect((await POST(request('clickhouse', range))).status).toBe(400); + expect(invoke).not.toHaveBeenCalled(); + }); + + it.each([[], '24h', 42, null])('rejects malformed range shapes', async range => { + expect((await POST(request('tempo', range))).status).toBe(400); + expect(invoke).not.toHaveBeenCalled(); + }); + + it('returns the effective provider or transport ceiling without exposing connection credentials', async () => { + getDatasource.mockResolvedValue({ id: 1, kind: 'prometheus', settings: { timeoutS: 35 } }); + const response = await POST(request('prometheus', false, 1)); + const body = await response.json(); + expect(invoke.mock.calls[0][0].args.timeout).toBe('35s'); + expect(body.metadata).toMatchObject({ timeoutMode: 'execution', effectiveTimeoutS: 35 }); + expect(JSON.stringify(body)).not.toContain('provider.example.test'); + }); + + it.each([false, true, 0, -1, 'missing', '1.5', Number.MAX_SAFE_INTEGER + 1])('a malformed explicit ID cannot select the legacy kind default', async id => { + const response = await POST(new Request('https://app.example.test/api/datasources/query', { + method: 'POST', body: JSON.stringify({ id, slug: 'prometheus', query: 'up' }), + })); + expect(response.status).toBe(400); + expect(invoke).not.toHaveBeenCalled(); + expect(getDatasource).not.toHaveBeenCalled(); + }); +}); diff --git a/web/app/api/datasources/route.test.ts b/web/app/api/datasources/route.test.ts index 6590f67bd..87a03b22a 100644 --- a/web/app/api/datasources/route.test.ts +++ b/web/app/api/datasources/route.test.ts @@ -3,7 +3,7 @@ import { describe, it, expect, vi, beforeEach } from 'vitest'; const verifyUser = vi.fn(); const isAdmin = vi.fn(); const listDatasources = vi.fn(); -const getConfiguredIds = vi.fn(); +const getConfiguredDatasourceIds = vi.fn(); const deleteDatasource = vi.fn(); vi.mock('@/lib/auth', () => ({ verifyUser: (...a: unknown[]) => verifyUser(...a) })); vi.mock('@/lib/admin', () => ({ isAdmin: (...a: unknown[]) => isAdmin(...a) })); @@ -11,20 +11,20 @@ vi.mock('@/lib/datasources', () => ({ listDatasources: (...a: unknown[]) => listDatasources(...a), deleteDatasource: (...a: unknown[]) => deleteDatasource(...a), })); -vi.mock('@/lib/integration-credentials', () => ({ getConfiguredIds: (...a: unknown[]) => getConfiguredIds(...a) })); +vi.mock('@/lib/integration-credentials', () => ({ getConfiguredDatasourceIds: (...a: unknown[]) => getConfiguredDatasourceIds(...a) })); const get = () => new Request('http://x/api/datasources', { headers: { cookie: 'awsops_token=t' } }); const del = () => new Request('http://x/api/datasources/5', { method: 'DELETE', headers: { cookie: 'awsops_token=t' } }); beforeEach(() => { - for (const m of [verifyUser, isAdmin, listDatasources, getConfiguredIds, deleteDatasource]) m.mockReset(); + for (const m of [verifyUser, isAdmin, listDatasources, getConfiguredDatasourceIds, deleteDatasource]) m.mockReset(); verifyUser.mockResolvedValue({ sub: 'u' }); isAdmin.mockResolvedValue(true); listDatasources.mockResolvedValue([ { id: 1, name: 'prod-prom', kind: 'prometheus', endpoint: 'http://p', authType: 'none', isDefault: true, enabled: true }, { id: 2, name: 'stg-prom', kind: 'prometheus', endpoint: 'http://s', authType: 'basic', isDefault: false, enabled: true }, ]); - getConfiguredIds.mockResolvedValue(['2']); + getConfiguredDatasourceIds.mockResolvedValue(['1', '2']); deleteDatasource.mockResolvedValue(undefined); }); @@ -41,7 +41,8 @@ describe('GET /api/datasources (list instances)', () => { const { datasources } = await resp.json(); expect(datasources).toHaveLength(2); // admin (default mock): the registered URL is included (v1 parity for managers) - expect(datasources[0]).toEqual({ id: 1, name: 'prod-prom', kind: 'prometheus', endpoint: 'http://p', authType: 'none', isDefault: true, connected: true }); + expect(datasources[0]).toEqual({ id: 1, name: 'prod-prom', kind: 'prometheus', endpoint: 'http://p', authType: 'none', isDefault: true, + connected: true, configured: true, connectionStatus: 'not_checked', scope: 'shared' }); expect(datasources[1].connected).toBe(true); // id '2' has a credential }); @@ -59,6 +60,14 @@ describe('GET /api/datasources (list instances)', () => { expect(resp.status).toBe(200); expect((await resp.json()).datasources).toEqual([]); }); + it('does not infer a default instance is configured or connected without its auth configuration', async () => { + listDatasources.mockResolvedValue([{ id: 1, name: 'default-missing', kind: 'prometheus', + endpoint: 'https://provider.example.test', authType: 'bearer', isDefault: true }]); + getConfiguredDatasourceIds.mockResolvedValue([]); + const { GET } = await import('./route'); + const { datasources } = await (await GET(get())).json(); + expect(datasources[0]).toMatchObject({ isDefault: true, configured: false, connected: false, connectionStatus: 'not_checked' }); + }); }); describe('DELETE /api/datasources/[id]', () => { diff --git a/web/app/api/datasources/route.ts b/web/app/api/datasources/route.ts index 35bc2cfaa..dae50625c 100644 --- a/web/app/api/datasources/route.ts +++ b/web/app/api/datasources/route.ts @@ -1,10 +1,11 @@ // GET /api/datasources — list configured datasource INSTANCES for the hub + Explore picker. // Authenticated (read-only), NOT admin. Returns {id,name,kind,authType,isDefault,connected} — never -// credentials. Degrade-safe: [] when Aurora is off / on error so the page doesn't 500. +// credentials. Configured is distinct from tested connectivity; connected is a legacy +// configuration alias. Degrade-safe: [] when Aurora is off / on error. import { verifyUser } from '@/lib/auth'; import { listDatasources } from '@/lib/datasources'; import { isAdmin } from '@/lib/admin'; -import { getConfiguredIds } from '@/lib/integration-credentials'; +import { getConfiguredDatasourceIds } from '@/lib/integration-credentials'; export const dynamic = 'force-dynamic'; @@ -17,7 +18,8 @@ export async function GET(request: Request) { if (!user) return json({ error: 'unauthenticated' }, 401); try { - const [rows, configuredIds, admin] = await Promise.all([listDatasources(), getConfiguredIds(), isAdmin(user)]); + const [rows, admin] = await Promise.all([listDatasources(), isAdmin(user)]); + const configuredIds = await getConfiguredDatasourceIds(rows); const idSet = new Set(configuredIds); const datasources = rows.map((r) => ({ id: r.id, @@ -28,8 +30,10 @@ export async function GET(request: Request) { ...(admin ? { endpoint: r.endpoint, settings: r.settings } : {}), authType: r.authType, isDefault: r.isDefault, - // "connected" = a credential is resolvable: the instance id key, or (for migrated defaults) the kind mirror. - connected: idSet.has(String(r.id)) || r.isDefault, + configured: idSet.has(String(r.id)), + connected: idSet.has(String(r.id)), // compatibility alias; not a live connection test + connectionStatus: 'not_checked', + scope: 'shared', })); return json({ datasources }, 200); } catch { diff --git a/web/app/api/datasources/test/route.test.ts b/web/app/api/datasources/test/route.test.ts index 1b5fe782b..6819cd736 100644 --- a/web/app/api/datasources/test/route.test.ts +++ b/web/app/api/datasources/test/route.test.ts @@ -26,6 +26,18 @@ beforeEach(() => { }); describe('POST /api/datasources/test', () => { + it('candidate credentials cannot replace the validated endpoint or auth method', async () => { + const { POST } = await import('./route'); + const response = await POST(req({ + kind: 'mimir', endpoint: 'https://provider.example.test', authType: 'none', + creds: { endpoint: 'http://169.254.169.254/', authType: 'basic', timeoutS: 999 }, + settings: { apiPrefix: '/metrics', timeoutS: 35 }, + })); + expect(response.status).toBe(200); + expect(invokeMcpLambdaTool.mock.calls[0][0].connConfig).toEqual({ + endpoint: 'https://provider.example.test', authType: 'none', apiPrefix: '/metrics', timeoutS: 35, + }); + }); it('401 unauthenticated / 403 non-admin', async () => { verifyUser.mockResolvedValueOnce(null); let { POST } = await import('./route'); @@ -41,7 +53,7 @@ describe('POST /api/datasources/test', () => { const { POST } = await import('./route'); const resp = await POST(req({ kind: 'prometheus', endpoint: 'http://p:9090', authType: 'bearer', creds: { token: 't' } })); expect(resp.status).toBe(200); - expect(await resp.json()).toEqual({ ok: true, latencyMs: 42, error: undefined }); + expect(await resp.json()).toEqual({ ok: true, latencyMs: 42, verified: [] }); const call = invokeMcpLambdaTool.mock.calls[0][0]; expect(call.tool).toBe('prometheus_health'); expect(call.connConfig).toEqual({ endpoint: 'http://p:9090', authType: 'bearer', token: 't' }); diff --git a/web/app/api/datasources/test/route.ts b/web/app/api/datasources/test/route.ts index dc4978260..2f03614a6 100644 --- a/web/app/api/datasources/test/route.ts +++ b/web/app/api/datasources/test/route.ts @@ -1,13 +1,17 @@ // POST /api/datasources/test — probe a datasource connection BEFORE saving (v1 "Test connection"). // Admin-gated. Takes an UNSAVED candidate {kind, endpoint, authType, creds}, SSRF-guards the endpoint, // and invokes the connector's `${kind}_health` tool with an INLINE conn-config (nothing is stored). -// SECURITY: the credential value is never logged or echoed — only {ok, latency_ms, error?} is returned. +// The health tool runs ONE real bounded query (not a readiness endpoint — an unauthenticated /ready +// used to pass), so `verified: ['query']` is query-access proof. +// SECURITY: the credential value is never logged or echoed — only {ok, latencyMs, verified, error?} +// is returned, with fixed connector sentences and never upstream text. import { verifyUser } from '@/lib/auth'; import { isAdmin } from '@/lib/admin'; import { invokeMcpLambdaTool, KNOWN_MCP_LAMBDA_KINDS } from '@/lib/mcp-lambda-invoke'; import { isDatasourceKind } from '@/lib/integrations-category'; import { assertDatasourceEndpointAllowed } from '@/lib/ssrf-guard'; import { readJsonBounded, BodyTooLargeError } from '@/lib/http-body'; +import { sanitizeDsSettings } from '@/lib/datasources'; export const dynamic = 'force-dynamic'; @@ -20,7 +24,7 @@ export async function POST(request: Request) { if (!user) return json({ error: 'unauthenticated' }, 401); if (!(await isAdmin(user))) return json({ error: 'admin access required' }, 403); - let body: { kind?: unknown; endpoint?: unknown; authType?: unknown; creds?: unknown }; + let body: { kind?: unknown; endpoint?: unknown; authType?: unknown; creds?: unknown; settings?: unknown }; try { body = (await readJsonBounded(request)) as typeof body; } catch (e) { if (e instanceof BodyTooLargeError) return json({ error: 'request body too large' }, 413); @@ -37,15 +41,26 @@ export async function POST(request: Request) { catch (e) { return json({ error: (e as Error).message }, 400); } const authType = typeof body.authType === 'string' ? body.authType : 'none'; + if (!['none', 'basic', 'bearer', 'custom_header'].includes(authType)) return json({ error: 'unknown auth method' }, 400); const creds = (body.creds && typeof body.creds === 'object' && !Array.isArray(body.creds)) ? (body.creds as Record) : {}; - const connConfig = { endpoint, authType, ...creds }; // flat blob the connector Lambda understands + const credentialFields = Object.fromEntries( + ['username', 'password', 'token', 'headerName', 'headerValue', 'headerName2', 'headerValue2', 'org_id'] + .filter(key => typeof creds[key] === 'string').map(key => [key, creds[key]]), + ); + const settings = sanitizeDsSettings(body.settings); + if (kind !== 'clickhouse') delete settings.database; + if (kind !== 'mimir') delete settings.apiPrefix; + const connConfig = { ...credentialFields, ...settings, endpoint, authType }; try { + const started = Date.now(); const result = (await invokeMcpLambdaTool({ kind, tool: `${kind}_health`, connConfig, - })) as { ok?: boolean; latency_ms?: number; error?: string }; - return json({ ok: Boolean(result?.ok), latencyMs: result?.latency_ms, error: result?.error }, 200); + })) as { ok?: boolean; latency_ms?: number; error?: string; verified?: unknown }; + return json({ ok: result?.ok === true, latencyMs: result?.latency_ms ?? Date.now() - started, error: result?.error, + verified: Array.isArray(result?.verified) ? result.verified.filter(value => value === 'query') : [], + }, 200); } catch (e) { // message only — connector errors never carry the credential value return json({ ok: false, error: e instanceof Error ? e.message : 'connection test failed' }, 200); diff --git a/web/app/customization/page.tsx b/web/app/customization/page.tsx index 73cf83eaa..1fdb7d9d1 100644 --- a/web/app/customization/page.tsx +++ b/web/app/customization/page.tsx @@ -19,7 +19,9 @@ const AGENT_TYPES = ['generic', 'on_demand', 'triage', 'rca', 'mitigation', 'eva // jaeger/dynatrace) — see web/lib/integration-validation.ts for the source of truth. const INTEG_KINDS_EGRESS = INTEGRATION_KINDS_EGRESS; const INTEG_KINDS_INGRESS = INTEGRATION_KINDS_INGRESS; -const INTEG_TRANSPORTS = INTEGRATION_TRANSPORTS; +// oauth_3lo stays in the source-of-truth constant for historical rows but is not executable by the +// runtime and is rejected by validation, so the form never offers it. +const INTEG_TRANSPORTS = INTEGRATION_TRANSPORTS.filter((t) => t !== 'oauth_3lo'); // NOTE: curated read connectors (Prometheus/Loki/…/Notion credential cards) moved to the Integrations // hub (/integrations) — Datasources tab + Connectors tab. This page keeps Agents/Skills/Agent-Space + // the advanced custom-integration registration. diff --git a/web/app/integrations/datasources/DatasourceForm.tsx b/web/app/integrations/datasources/DatasourceForm.tsx index 937bb5ccc..9f4d94ba7 100644 --- a/web/app/integrations/datasources/DatasourceForm.tsx +++ b/web/app/integrations/datasources/DatasourceForm.tsx @@ -4,6 +4,7 @@ import Input from '@/components/ui/Input'; import Button from '@/components/ui/Button'; import IntegrationIcon from '@/components/datasources/IntegrationIcon'; import { useI18n } from '@/components/shell/LanguageProvider'; +import { EXECUTION_TIMEOUT_KINDS, EFFECTIVE_TIMEOUT_CAP } from '@/lib/datasource-query-tools'; // v1-parity Add/Edit form: multiple instances per type, a name, an explicit auth method (optional auth), // and a Test-before-save probe. POSTs (create) / PATCHes (update) /api/datasources/manage. @@ -39,7 +40,7 @@ export interface DatasourceFormValue { authType: string; isDefault?: boolean; // gap L203: per-datasource connection settings (server-side sanitized; see lib/datasources.ts) - settings?: { timeoutS?: number; database?: string }; + settings?: { timeoutS?: number; database?: string; apiPrefix?: string }; } export default function DatasourceForm({ @@ -55,6 +56,7 @@ export default function DatasourceForm({ // gap L203: settings kept as strings for the inputs; settingsPayload() validates/coerces const [timeoutS, setTimeoutS] = useState(initial?.settings?.timeoutS != null ? String(initial.settings.timeoutS) : ''); const [database, setDatabase] = useState(initial?.settings?.database ?? ''); + const [apiPrefix, setApiPrefix] = useState(initial?.settings?.apiPrefix ?? '/prometheus'); const [test, setTest] = useState<{ ok: boolean; ms?: number; error?: string } | null>(null); const [testing, setTesting] = useState(false); const [saving, setSaving] = useState(false); @@ -77,11 +79,15 @@ export default function DatasourceForm({ const databaseInvalid = kind === 'clickhouse' && dbTrim !== '' && (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(dbTrim) || dbTrim.length > 128 || ['system', 'information_schema'].includes(dbTrim.toLowerCase())); + const prefixInvalid = kind === 'mimir' && (apiPrefix.length > 128 || !/^(?:\/[A-Za-z0-9_-]+)*$/.test(apiPrefix)); + const executionTimeout = EXECUTION_TIMEOUT_KINDS.has(kind); + const effectiveTimeout = Math.min(timeoutS.trim() && !timeoutInvalid ? Number(timeoutS) : executionTimeout ? 10 : 12, EFFECTIVE_TIMEOUT_CAP); const settingsPayload = () => { - const out: { timeoutS?: number; database?: string } = {}; + const out: { timeoutS?: number; database?: string; apiPrefix?: string } = {}; const t = Number(timeoutS); if (timeoutS.trim() !== '' && Number.isInteger(t) && t >= 1 && t <= 60) out.timeoutS = t; if (kind === 'clickhouse' && database.trim()) out.database = database.trim(); + if (kind === 'mimir') out.apiPrefix = apiPrefix; return out; }; @@ -90,7 +96,7 @@ export default function DatasourceForm({ try { const r = await fetch('/api/datasources/test', { method: 'POST', headers: { 'content-type': 'application/json' }, - body: JSON.stringify({ kind, endpoint, authType, creds: credPayload() }), + body: JSON.stringify({ kind, endpoint, authType, creds: credPayload(), settings: settingsPayload() }), }); const b = await r.json(); if (!r.ok) { setErr(b.error || tt(`오류 ${r.status}`)); return; } @@ -138,6 +144,7 @@ export default function DatasourceForm({
setEndpoint(e.target.value)} placeholder={ENDPOINT_PH[kind] ?? 'http://prometheus.internal:9090'} /> +

{tt('사설 주소는 운영자가 네트워크 연결과 DNS를 구성해야 사용할 수 있습니다.')}

@@ -178,8 +185,9 @@ export default function DatasourceForm({ result-cache TTL is deliberately not ported: the v2 query path is uncached by design) */}
- - setTimeoutS(e.target.value)} placeholder={tt('기본 10')} inputMode="numeric" /> + + setTimeoutS(e.target.value)} placeholder={tt(executionTimeout ? '기본 10' : '기본 12')} inputMode="numeric" /> +

{tt('적용 제한')}: {effectiveTimeout}s{!executionTimeout && ` · ${tt('서버 쿼리 중단을 보장하지 않습니다.')}`}

{timeoutInvalid &&

{tt('1–60 사이의 정수를 입력하세요.')}

}
{kind === 'clickhouse' && ( @@ -189,10 +197,18 @@ export default function DatasourceForm({ {databaseInvalid &&

{tt('영문/숫자/밑줄 식별자만 가능하며 system 계열은 사용할 수 없습니다.')}

}
)} + {kind === 'mimir' && ( +
+ + setApiPrefix(e.target.value)} placeholder="/prometheus" /> +

{tt('기본 /prometheus · API가 루트에 있으면 비워 두세요.')}

+ {prefixInvalid &&

{tt('슬래시와 영문/숫자/밑줄/하이픈 경로만 사용할 수 있습니다.')}

} +
+ )}
- {test && ( @@ -205,7 +221,7 @@ export default function DatasourceForm({ {err &&

{err}

}
- +
diff --git a/web/app/integrations/datasources/DatasourcesTab.test.tsx b/web/app/integrations/datasources/DatasourcesTab.test.tsx index e7e4edeaf..dde63d31b 100644 --- a/web/app/integrations/datasources/DatasourcesTab.test.tsx +++ b/web/app/integrations/datasources/DatasourcesTab.test.tsx @@ -17,6 +17,17 @@ beforeEach(() => { afterEach(() => { cleanup(); vi.restoreAllMocks(); }); describe('DatasourcesTab', () => { + it('uses explicit configuration evidence instead of a legacy connected flag', async () => { + global.fetch = vi.fn(async () => ({ + ok: true, status: 200, json: async () => ({ datasources: [{ + id: 1, name: 'unconfirmed-default', kind: 'prometheus', isDefault: true, + connected: true, configured: false, + }] }), + })) as unknown as typeof fetch; + render(); + expect(await screen.findByText('설정 미확인')).toBeTruthy(); + expect(screen.queryByText('● 연결됨')).toBeNull(); + }); it('lists instances (name/type/auth/default) with an Explore link per row', async () => { render(); await waitFor(() => expect(screen.getByText('prod-prom')).toBeTruthy()); @@ -48,7 +59,8 @@ describe('DatasourcesTab', () => { render(); await waitFor(() => expect(screen.getByText('prod-prom')).toBeTruthy()); expect(screen.getByText('총 데이터소스')).toBeTruthy(); - expect(screen.getByText('연결됨')).toBeTruthy(); + expect(screen.getAllByText('설정됨').length).toBeGreaterThan(0); + expect(screen.queryByText('● 연결됨')).toBeNull(); expect(screen.getByText('타입 종류')).toBeTruthy(); expect(screen.getByText('기본 데이터소스')).toBeTruthy(); // single default in this scenario → the tile names it diff --git a/web/app/integrations/datasources/DatasourcesTab.tsx b/web/app/integrations/datasources/DatasourcesTab.tsx index ec36bcada..085912e8d 100644 --- a/web/app/integrations/datasources/DatasourcesTab.tsx +++ b/web/app/integrations/datasources/DatasourcesTab.tsx @@ -10,7 +10,7 @@ import DatasourceForm, { type DatasourceFormValue } from './DatasourceForm'; import { useI18n } from '@/components/shell/LanguageProvider'; interface Instance { - id: number; name: string; kind: string; endpoint?: string | null; authType?: string | null; isDefault?: boolean; connected?: boolean; settings?: { timeoutS?: number; database?: string }; + id: number; name: string; kind: string; endpoint?: string | null; authType?: string | null; isDefault?: boolean; connected?: boolean; configured?: boolean; settings?: { timeoutS?: number; database?: string; apiPrefix?: string }; } // Chat section per datasource kind (the deep-link prompt pins it with a leading /section — @@ -65,7 +65,8 @@ export default function DatasourcesTab({ canManage = false }: { canManage?: bool } // gap-audit L201: KPI roll-up from the already-fetched list — no extra API call. - const connectedN = list.filter((i) => i.connected).length; + const configured = (i: Instance) => i.configured ?? i.connected ?? false; + const configuredN = list.filter(configured).length; const kindN = new Set(list.map((i) => i.kind)).size; // Defaults are PER KIND — one arbitrary name would misrepresent the set, so show the count // (with every kind: name pair in a tooltip) once there's more than one. @@ -76,7 +77,8 @@ export default function DatasourcesTab({ canManage = false }: { canManage?: bool return (
-

{tt('관측성 데이터소스 (Prometheus·Mimir·Loki·Tempo·ClickHouse). 같은 타입 여러 개 등록 가능.')}

+

{tt('관측성 데이터소스 (Prometheus·Mimir·Loki·Tempo·ClickHouse). 같은 타입 여러 개 등록 가능.')} + {' '}{tt('설정 상태는 실제 연결 성공을 의미하지 않습니다.')}

{/* gap-audit L202: re-fetch without a full page reload. */} }
+

{tt('기본 데이터소스는 타입별 공유 연결이며 AWS 계정 선택으로 바뀌지 않습니다.')}

- +
`${d.kind}: ${d.name}`).join(', ')}> @@ -128,7 +131,7 @@ export default function DatasourcesTab({ canManage = false }: { canManage?: bool )} {i.authType ?? 'none'} - {i.connected ? tt('● 연결됨') : tt('○ 미설정')} + {configured(i) ? tt('설정됨') : tt('설정 미확인')} {i.isDefault ? {tt('★ 기본')} : (canManage && )} diff --git a/web/components/datasources/ExplorePanel.tsx b/web/components/datasources/ExplorePanel.tsx index f112d8eab..4abdc95c5 100644 --- a/web/components/datasources/ExplorePanel.tsx +++ b/web/components/datasources/ExplorePanel.tsx @@ -15,6 +15,7 @@ import HBarList from '@/components/charts/HBarList'; import type { NormalizedResult } from '@/lib/datasource-render'; import { cn } from '@/lib/cn'; import { useI18n } from '@/components/shell/LanguageProvider'; +import { TIME_WINDOW_KINDS, TOOL } from '@/lib/datasource-query-tools'; // A datasource INSTANCE (the hub model): identified by bigint id, with a user-given name. export interface DatasourceInstance { @@ -36,7 +37,7 @@ const PH: Record = { dynatrace: 'metricSelector… 예: builtin:host.cpu.usage:avg', datadog: '메트릭 쿼리… 예: avg:system.cpu.user{*}', }; -const RANGE_KINDS = new Set(['prometheus', 'mimir', 'loki']); // kinds with a *_query_range tool +const RANGE_KINDS = new Set(TIME_WINDOW_KINDS); const selectCls = 'rounded-md border border-ink-200 bg-card px-2.5 py-1.5 text-[13px] text-ink-700'; // Explore range presets: label → window seconds (0 = instant snapshot). Step auto-derived for ~250 points. @@ -76,7 +77,10 @@ export default function ExplorePanel({ instanceId }: { instanceId?: number }) { const ds = list.find((d) => d.id === selId) ?? null; const canRange = ds ? RANGE_KINDS.has(ds.kind) : false; // Loki has no per-request upstream timeout — the API caps it at 7d, so don't offer 30d. - const presets = ds?.kind === 'loki' ? RANGE_PRESETS.filter(([, s]) => s <= 604800) : RANGE_PRESETS; + const searchWindow = ds ? TOOL[ds.kind]?.timeWindow === true : false; + const effectiveRangeWindow = searchWindow ? rangeWindow || 3600 : rangeWindow; + const presets = RANGE_PRESETS.filter(([, seconds]) => + (!searchWindow || seconds > 0) && (!ds || !['loki', 'tempo', 'jaeger'].includes(ds.kind) || seconds <= 604800)); const queryIsMultiline = ds?.kind === 'clickhouse'; useEffect(() => { @@ -98,7 +102,7 @@ export default function ExplorePanel({ instanceId }: { instanceId?: number }) { const run = useCallback(async (windowOverride?: number, queryOverride?: string) => { const q = queryOverride ?? query; // a quick-query chip can run its expr without waiting on setQuery if (selId === '' || !q.trim()) return; - const w = windowOverride ?? rangeWindow; + const w = windowOverride ?? effectiveRangeWindow; const kindNow = list.find((d) => d.id === selId)?.kind; // bind kind to THIS query, not the live selection const range = canRange && w > 0 ? { window: w, step: autoStep(w, kindNow) } : false; const queriedKind = kindNow; @@ -116,7 +120,7 @@ export default function ExplorePanel({ instanceId }: { instanceId?: number }) { } catch (e) { setErr(e instanceof Error ? e.message : tt('쿼리 실패')); } finally { setBusy(false); } - }, [selId, query, rangeWindow, canRange, list, tt]); + }, [selId, query, effectiveRangeWindow, canRange, list, tt]); // NL → query (AI drafts, user reviews, then runs). Never auto-runs. const generate = useCallback(async () => { @@ -142,6 +146,7 @@ export default function ExplorePanel({ instanceId }: { instanceId?: number }) { return (
+

{tt('공유 데이터소스입니다. AWS 계정 선택은 이 연결의 데이터 범위를 변경하지 않습니다.')}

{/* Always show the picker (preselected to the scoped instance) — never a dead-end if the scoped id isn't in the list yet. */} {( @@ -157,7 +162,7 @@ export default function ExplorePanel({ instanceId }: { instanceId?: number }) {