feat: release SchemaForge 0.37 with SQL Server and Windows auth - #128
Merged
Conversation
Document the export design: read-vs-export Cedar action split, fail-closed two-level opt-in (@export entity + @exportable field, intersected with field_access), AU-2 audit trail, row caps/rate limits, the DynamicValue flatten policy, and sync-stream vs async-job delivery. Index ADR-0003 in docs/adr/README.md. Docs-only; epic to be filed.
Add the entity-level @export(formats, bundle_files, max_rows) and the field-level @exportable(flatten?) annotations to the schema model, DSL parser, printer, and validation, per ADR-0003. - core: ExportFormat enum (csv/ndjson/xlsx/zip), Annotation::Export, and ExportFlatten with FieldAnnotation::Exportable; Display + kind() + serde. - dsl: parse @export (validating formats against the canonical vocabulary, rejecting an empty formats list, requiring formats and max_rows) and @exportable (optional flatten: json hint). - dsl: fail-closed validation -- a field carrying @exportable on an entity without @export is rejected with a line:col diagnostic; @exportable inside a composite is likewise rejected. - dsl: printer round-trips both annotations via core Display. API-layer enforcement is intentionally out of scope for this slice.
Introduce AccessAction::Export and ActionVerb::Export so bulk export is
authorized independently of Read. The Export verb renders to a distinct
Cedar action UID (Export{Schema}), which the Cedar schema generator now
declares per schema alongside the CRUD actions so strict-mode policy
validation recognizes policies that reference it.
Add a check_export_access helper mirroring check_schema_access(Read), and
unit tests proving the read-vs-export split (ADR-0003): a read grant does
not imply export, a policy can permit read while forbidding export, and an
explicit export permit is action- and role-scoped. No endpoint yet.
Refs ADR-0003.
Map DynamicValue to CSV/XLSX cell strings and lossless NDJSON values under
the single ADR-0003 flatten policy: scalars as-is, relation-one to a resolved
display (CSV) / {id,display} object (NDJSON), relation-many/array/map/composite
as JSON-in-cell (CSV) / native (NDJSON), bytes omitted by default with base64
opt-in (CSV) and always base64 (NDJSON), file fields as URL/metadata only.
Pure, IO-free functions with a RelationDisplay resolver injected as the only
dependency, plus a csv_record helper that owns RFC 4180 quoting via the csv
crate. Covers every field type with unit tests, including the flatten:json
hint and non-finite float guard. Consumed by export items 5-8.
Implement POST /schemas/{schema}/entities/export for the synchronous,
streamable path (item 5 of the export epic, ADR-0003). An export is the
existing query with no page limit, materialized inline, gated by the two
fail-closed controls layered on the unchanged query path.
- core: add SchemaDefinition::export_annotation/export_config/is_exportable
and FieldDefinition::exportable/is_exportable accessors over the @export and
@exportable annotations from item 3.
- acton: new routes/export.rs. Gates on the distinct AccessAction::Export
Cedar action (NOT Read), refuses any schema without @export, refuses a
format outside the declared vocabulary, and resolves the exported column
set as the intersection of @exportable and field_access-readable. Tenant
injection and filter_entity_fields run identically to the query path, so the
file can never contain a row or field the caller could not already read.
- Pure, unit-tested serializers (entities_to_csv / entities_to_ndjson) over the
item-4 serialization core, with a FieldDisplay resolver for relation columns;
CSV is the rectangular projection, NDJSON the lossless dump with id.
- Enforce the @export(max_rows) cap by probing max_rows+1 and rejecting an
over-cap result with 413 (ExportTooLarge); defer non-streamable formats
(xlsx/zip) and explicit async:true with 422 (ExportDeferred), both pointing
at the forthcoming async-job endpoint.
- Emit the AU-2 exfiltration trail: forge.export.initiated/completed/denied via
the audit logger with subject, schema, filter, fields, row count, and format.
- Wire the route in routes/mod.rs; add two new ForgeError variants and their
status/kind/body mappings.
Tests: pure unit tests for column intersection and CSV/NDJSON serialization,
plus in-process-surreal integration tests covering the streamable happy path,
the fail-closed gates (no @export, non-exportable field never leaks, read-only
role denied export), the row-cap 413 defer, and the async/xlsx 422 defer. No
external services.
Refs ADR-0003.
Implement the async delivery path for bulk export (item 6 of the export
epic, ADR-0003). A POST .../entities/export with async:true (or a
non-streamable format) no longer blocks or flatly defers: it registers a
supervised export job and returns its id, and GET .../exports/{job_id}
reports status plus a TTL-bounded presigned download URL once complete.
- export_job: new ExportJobActor (acton-reactive, NOT tokio::spawn). State
is a job_id -> record map; the StartExportJob handler registers the job
(queued) and runs generation in the runtime-owned Reply::pending future,
self-sending MarkRunning / CompleteExportJob / FailExportJob so the brief
mutate_on transitions never block on the long query+serialize+upload. The
pending future is wrapped in SyncFuture to satisfy acton's Send+Sync bound,
mirroring the hook-dispatch actor.
- routes/export: extract the query/tenant/record-filter/field-strip/serialize
pipeline into a shared materialize_export(ExportContext, ...) so the sync
and async paths inherit the exact same fail-closed guarantees (tenant
injection, @exportable ∩ readable intersection, max_rows cap). The async
branch resolves the storage backend, mints a job id + object key, emits the
forge.export.initiated audit event, and hands an ExportJobSpec to the actor;
the actor emits forge.export.completed / .denied on terminal transition.
- storage: add an ExportArtifactStore trait (put + presign_get) as the seam
the actor depends on, impl'd for S3Client (new put_object for server-held
bytes), plus StorageRegistry::single_backend and pick_export_store to choose
the artifact bucket. The trait is mockable so tests use an in-memory store,
never live MinIO.
- get_export_job reuses the distinct Export{Schema} Cedar action and the
@export gate, and scopes a job id to the schema in its URL so one schema's
ids cannot probe another's. No storage configured => 503 (fail-closed).
- Wire the GET route, register ExportJobActor in serve.rs, and update the
item-5 async-defer test to the new 503-without-storage contract.
Tests: actor lifecycle (queued->running->complete) with a mock store asserting
generated CSV bytes + presigned URL; over-cap job fails without uploading; and
HTTP coverage of the no-storage 503, the schema-scoped status gate, the
not-found path, and a completed job's download URL. In-process surreal + mock
storage, no external services.
Refs ADR-0003.
The async export status endpoint (GET /schemas/{schema}/exports/{job_id})
enforced schema-level Export access but not job ownership, so any caller
permitted to export the schema could read another subject's job and mint its
presigned download URL — an IDOR / bulk-data-exfiltration path flagged by
security review.
- Persist owner_sub on ExportJobRecord from the job spec's subject.
- Gate get_export_job on a pure may_read_export_job(owner, caller) predicate;
mismatches return EntityNotFound (not Forbidden) so job-id existence never
leaks.
- Replace the guessable timestamp+counter job id with a v4 UUID (CSPRNG) as
defense-in-depth against enumeration. uuid is already an acton dependency.
- Add unit tests for the ownership predicate and an HTTP-level cross-subject
denial regression test; document the control in ADR-0003.
XLSX is zip-of-XML and cannot truly stream (rust_xlsxwriter buffers the whole workbook), so it is async-only: a POST requesting xlsx always takes the supervised export-job path (item 6) rather than being deferred. Add a pure, dependency-free XlsxCell typing core (to_xlsx_cell) to schema-forge-core that maps a DynamicValue to a typed spreadsheet cell under the same flatten policy as the CSV path, but preserving native cell types where a spreadsheet benefits: int/float -> number, bool -> boolean, datetime -> native datetime; duration/enum stay text; structured values (relation-many/array/map/composite/json) are JSON-in-cell; bytes follow the fail-closed BytesPolicy. The workbook builder (entities_to_xlsx) lives in the acton routes next to the CSV/NDJSON serializers, translating each XlsxCell into a rust_xlsxwriter write and returning the .xlsx bytes via save_to_buffer. Header row is the resolved @exportable-intersect-readable column set; an absent/stripped field renders an empty cell, never a leak. Wire xlsx into materialize_export and allow it through the async-job gate (async_job_supports); zip remains deferred. Unit tests cover the pure cell typing and the workbook builder (read back with calamine); an integration test drives the job actor with a mock store and asserts a well-formed workbook is uploaded with exportable columns only and no leak of the readable-but-not-exportable field.
Implement ZIP-format bundling on the supervised async export-job path (ADR-0003, item 8): the archive carries the rectangular CSV data file and, when the entity's @export(bundle_files: true) opt-in is set, the raw blobs of its @exportable file fields pulled from storage. - add a pure, storage-free export_bundle module: build_zip_bundle (deterministic in-memory ZIP writer, rejects duplicate members), collect_file_blob_refs (derives blob refs from already-stripped rows, Available-only, zip-slip-safe member names), and the member-name policy - extend ExportArtifactStore with get() so the job actor can read file blobs; the seam stays mockable (in-memory) for tests - factor materialize_export's shared query/tenant/record-filter/strip half into prepare_export, reused by the new materialize_zip_bundle so a bundle can never widen what a flat export authorizes - thread the schema's bundle_files flag through ExportJobSpec and route ZIP through async_job_supports + spawn_export_job - tests (mocked storage, mem:// surreal): data-only bundle never embeds a blob, bundle_files embeds Available blobs under files/<field>/<id>/<name>, quarantined blobs are skipped, and zip routes to the async path Bundling file blobs is a larger exfiltration surface, so it stays gated strictly by bundle_files behind the same Export authz + AU-2 audit as the rest of the export path. cargo add zip.
…t rate limit ADR-0003 item 9 (hardening). Layer two operator-tunable, fail-closed bounds on the export path and complete the AU-2 denied-audit trail. - Configurable row cap: new `[schema_forge.export]` config with a server-wide `default_max_rows` ceiling. `resolve_max_rows` intersects the schema's `@export(max_rows)` with it (min), so a schema can override the cap downward but never widen it above the operator's bound. - Per-subject rate limit: new supervised `ExportRateLimiter` actor over a pure fixed-window core (`RateWindow::admit`), keyed by subject (anonymous callers share one bucket). A `max_requests` of 0 is a kill switch. Exceeding the window returns 429 via the new `ForgeError::RateLimited`. - Audit completeness: rate-limit rejection and a wholly-non-exportable `fields` request both emit `forge.export.denied` (reasons `rate_limited` / `no_exportable_fields_requested`) alongside the existing authz/cap/format denials. A partially-valid `fields` request still narrows and succeeds. The admission policy and cap resolution are pure, unit-tested functions; the live rate-limit state lives in a supervised actor (no tokio::spawn). Tests cover the ceiling clamp, tighter-schema-cap precedence, rate-limit rejection and kill switch, and the new denied paths, all on in-process surreal with no external services.
Mirror the existing `entity` verbs with `schemaforge entity export`,
driving the ADR-0003 bulk-export endpoints over the same HTTP client and
auth-token sourcing the other verbs use:
- POST /schemas/{schema}/entities/export with { filter?, fields?, format,
async? }. csv/ndjson under the row cap stream inline; xlsx/zip, an
over-cap result, or --async return an accepted job.
- --async polls GET /schemas/{schema}/exports/{job_id} to completion, then
downloads the time-limited presigned artifact URL.
Args mirror `entity query`'s filter/field grammar (an export is a query
with no page limit) and add --format {csv|ndjson|xlsx|zip}, --out (file,
directory, or - for stdout), and --poll-interval / --poll-timeout. The
download URL is fetched without a Bearer token since it is self-authorizing
and points at the object store, not the API.
Body assembly, the out-path policy, the Content-Disposition filename
parse, and the job-status accessors are pure and unit-tested; clap parsing
is covered too. No new deps.
Bring all user-facing docs and the in-repo schemaforge skill current with the bulk-export capability (ADR-0003), so it is discoverable and documented. Docs only; no functional code changes. - skill SKILL.md: add export to the trigger description, the When-to-Use list, and the reference-file index. - skill dsl-reference.md: document @export(formats, bundle_files, max_rows) and @exportable(flatten?), the @exportable ∩ readable intersection rule, and the DynamicValue -> cell/value flatten policy table. - skill dsl-quickref.md: add @export / @exportable to the annotation tables. - skill rest-api-reference.md: document POST .../entities/export and GET .../exports/{job_id} — request body, sync-stream vs async-job routing, presigned download, audit trail, and the error paths (400/403/413/422/429). - skill cli-reference.md: document the schemaforge entity export subcommand and its flags (--format/--filter/--fields/--out/--async/--poll-*). - skill config-reference.md: document the [schema_forge.export] row ceiling and per-subject rate-limit bounds. - skill export.md: new focused reference covering the full feature — security model, the two annotations, the distinct Cedar Export action, endpoints, flatten policy, hardening bounds, AU-2 audit trail, CLI, and error paths. - docs/entity-cli-reference.md: add the export verb to the verb->method table and an end-to-end example. Refs ADR-0003.
…raversal The entity export download path joined a server-controlled filename (Content-Disposition, or a synthesized name) onto a directory-like --out without sanitization, so a malicious or compromised server — or one reached via --server — could write the artifact outside the chosen directory via '..', a separator, or an absolute path (flagged MEDIUM by security review). - Add a pure, total safe_basename() that accepts only a single non-traversing path component; resolve_out_path() now falls back to the safe default for anything else. Centralized at the join point so both the sync (Content-Disposition) and async (schema-derived) callers are covered. - Add unit + table tests proving '../../etc/passwd', '..', absolute, and separator-bearing names neutralize to the default while clean basenames are honored. Document the control in the export skill reference.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
mainValidation