[gateway] Add the Gateway runtime, common HTTP layer, lifecycle, and test infrastructure - #3963
[gateway] Add the Gateway runtime, common HTTP layer, lifecycle, and test infrastructure#3963beryllw wants to merge 16 commits into
Conversation
fc487d0 to
f6e94e7
Compare
|
@fresh-borzoni This is the first PR for FIP-49 — could you help review it when you have time? Thanks a lot! 🙏 |
There was a problem hiding this comment.
Pull request overview
This PR introduces the initial Rust-based Fluss Gateway module (fluss-gateway/) as a standalone Cargo workspace, providing the runtime/lifecycle foundation, a shared HTTP layer (error envelope, request IDs, limits/timeouts), baseline observability (logging + Prometheus metrics), and integration test infrastructure for future gateway capabilities described in FIP-49 / issue #3958.
Changes:
- Adds gateway runtime + lifecycle (listener binding, readiness/draining, SIGTERM handling, stable exit codes).
- Adds REST foundation with
GET /health,GET /v1/openapi.json, shared error envelope, request IDs, body limits, and request deadlines. - Adds gateway-focused CI jobs (build/test, fmt/clippy/rustdoc, OpenAPI drift + lint, MSRV gate) and supporting test suites.
Reviewed changes
Copilot reviewed 28 out of 29 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| fluss-gateway/tests/support/mod.rs | Shared HTTP + process helpers for gateway integration tests. |
| fluss-gateway/tests/process.rs | End-to-end tests for the compiled binary (exit codes, SIGTERM drain, bind failures). |
| fluss-gateway/tests/http_api.rs | End-to-end HTTP contract tests against a real listener (health, OpenAPI, envelopes, limits). |
| fluss-gateway/tests/e2e_cluster.rs | Feature-gated dockerized cluster smoke test wiring (integration scaffolding). |
| fluss-gateway/src/protocol/rest/openapi.rs | Router-derived OpenAPI generation + serving and drift checks. |
| fluss-gateway/src/protocol/rest/mod.rs | REST router assembly and middleware (request IDs, envelopes, deadlines, body limits, acceptance guard). |
| fluss-gateway/src/protocol/rest/health.rs | GET /health endpoint and schema. |
| fluss-gateway/src/protocol/mod.rs | Protocol module root exporting REST surface. |
| fluss-gateway/src/observability.rs | Logging + Prometheus metrics inventory and runtime/process metric sampling. |
| fluss-gateway/src/main.rs | Binary entrypoint (CLI, config load/validate, init logging, lifecycle run + exit codes). |
| fluss-gateway/src/lifecycle.rs | Listener/task supervision, readiness state machine, graceful shutdown/draining. |
| fluss-gateway/src/lib.rs | Gateway library crate root (internal implementation modules). |
| fluss-gateway/src/error.rs | Error taxonomy + REST error envelope and OpenAPI-friendly schema generation. |
| fluss-gateway/src/config.rs | Gateway configuration loading (CLI/env/YAML precedence), parsing, validation, and warnings. |
| fluss-gateway/rustfmt.toml | Gateway-local rustfmt configuration. |
| fluss-gateway/rust-toolchain.toml | Gateway-local toolchain selection (stable + components). |
| fluss-gateway/openapi.yaml | Checked-in generated OpenAPI document. |
| fluss-gateway/justfile | Developer recipes for build/test/fmt/clippy/doc/openapi/licenses. |
| fluss-gateway/deny.toml | cargo-deny license allowlist/config. |
| fluss-gateway/copyright.txt | License header template for tooling. |
| fluss-gateway/clippy.toml | Clippy configuration (cognitive complexity threshold). |
| fluss-gateway/Cargo.toml | Standalone workspace/package definition, deps, features, tests, MSRV. |
| fluss-gateway/.licenserc.yaml | License header checking configuration (with generated-file exclusions). |
| fluss-gateway/.gitignore | Gateway-local ignore rules. |
| .github/workflows/rust-license-and-format.yml | Extends Rust license/format workflow to include gateway checks. |
| .github/workflows/rust-build-and-test.yml | Adds gateway build/test/OpenAPI/MSRV/E2E jobs and triggers. |
| .github/workflows/license-check.yml | Excludes gateway from the generic license-check workflow (gateway has its own checks). |
| .github/workflows/ci.yaml | Excludes gateway-only changes from the main CI workflow (gateway has dedicated Rust workflows). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Introduces fluss-gateway as an independent Cargo workspace with library and executable entry points: strict configuration with stable exit codes, the shared error envelope, request-id/body-size/deadline middleware, the FIP-49 GET /health endpoint, REST and metrics listeners, task supervision with SIGTERM draining, and the OpenAPI 3.1 document generated from the typed router. Co-authored-by: gstamatakis95 <126914070+gstamatakis95@users.noreply.github.com>
…uster harness Covers the shared HTTP contract over a real listener, the compiled binary's startup, health, SIGTERM draining and exit codes, and a self-test of the fixed-version dockerized Fluss cluster harness that later capabilities reuse for their end-to-end suites. Co-authored-by: gstamatakis95 <126914070+gstamatakis95@users.noreply.github.com>
Runs the gateway workspace's build, unit tests, license headers, formatting, clippy and rustdoc, checks the checked-in OpenAPI document for drift and validates it externally, and runs the end-to-end suite in a job that fails rather than skipping silently when a selected scenario cannot start. Co-authored-by: gstamatakis95 <126914070+gstamatakis95@users.noreply.github.com>
… rename the harness job Co-authored-by: gstamatakis95 <126914070+gstamatakis95@users.noreply.github.com>
… artifact release
…ines against overflow
795c831 to
cbd4ab8
Compare
…e rust workflows Move the gateway jobs out of rust-build-and-test.yml and rust-license-and-format.yml into the dedicated gateway-ci.yml introduced in apache#4002, and install protoc in every compiling job: the gateway now depends on fluss-rs, whose proto codegen needs protoc, which the scaffold-era workflow steps did not install. The rust workflows are restored to their upstream state so gateway changes never build the fluss-rust workspace and vice versa.
… and slim the gateway CI
fresh-borzoni
left a comment
fresh-borzoni
left a comment
There was a problem hiding this comment.
@beryllw Thank you for the PR!
I looked quickly with a limited time I had, left some quick comments, PTAL
fresh-borzoni
left a comment
fresh-borzoni
left a comment
There was a problem hiding this comment.
@beryllw Thank you, but the thing I'd look at again is dropping axum::serve to get at the builder.
auto first checks whether the connection opens with the HTTP/2 preface, and that read has no timer on it, so a socket that sends literally nothing is still parked (half a request head does get closed now). Pulling in server-auto also flips on hyper/http2, so anything that does send that preface now gets an HTTP/2 connection, and if it then just sits there hyper's graceful shutdown waits on it, my SIGTERM sat for the full drain and exited 1. Then the accept arm returns Err, so one EMFILE (tried it with ulimit -n 40) and run() takes the whole process down, where axum just sleeps a second and carries on.
Both the parked socket and the HTTP/2 surface can go away with hyper::server::conn::http1::Builder instead of auto, the timer starts at setup there and there's no HTTP/2 to serve. Needs the non-upgradeable Connection, graceful.watch won't take the other one. For accept, could we copy what axum does in handle_accept_error (serve/listener.rs), skip the connection-level errors and sleep a second on the rest instead of returning?
WDYT?
|
Hey @leekeiabstraction — would you be interested in helping review this PR (or any of the follow-ups in #3957)? It's part of the FIP-49 gateway implementation, which we're targeting for 1.0. The timeline is a bit tight, so any early feedback would really help. Thanks! 🙏 |
fresh-borzoni
left a comment
fresh-borzoni
left a comment
There was a problem hiding this comment.
@beryllw Thank you, I think it's good overall, left some minors, but I approve. Let me know when you wish to merge it.
Connection errors all logged at debug, so a client sending the HTTP/2 preface looked like a dead gateway from both ends; that case now warns and names the misconfiguration. hyper-util's http1 feature gates the GracefulConnection impl that graceful.watch needs, and it was only enabled because axum asks for it, so it is now declared here. The accept backoff moved inside the future the serve loop selects, so a shutdown signal cancels the wait instead of having to outlast it.
HTTP/2 arrives through Cargo feature unification rather than a code change, so deny.toml bans the h2 crate and CI checks it; tokio-util's unused rt feature goes with it. A panicking handler dropped the connection, leaving the caller without an envelope and the access log and counters without the request. A panic guard inside the access-log layer answers 500 instead, logging the payload with the request id and never returning it.
The previous run failed before any code ran: 'Gateway License and Formatting Check' could not download taiki-e/install-action (HTTP 429 after three attempts), 'Java 11 / core' failed in 'Set up job', and 'Java 11 / spark3-scala213' failed in 'Checkout code'. Other branches, including main, hit the same setup failures at that time. Empty commit, no change to the tree.
luoyuxia
left a comment
luoyuxia
left a comment
There was a problem hiding this comment.
The overall foundation is thoughtfully designed, with strong coverage around configuration, the HTTP error contract, OpenAPI generation, and lifecycle behavior. I left two suggestions inline.
| // The non-upgradeable connection is what `graceful.watch` accepts; the gateway has no | ||
| // upgrade-based protocol. | ||
| let connection = graceful.watch(builder.serve_connection(TokioIo::new(socket), service)); | ||
| tokio::spawn(async move { |
There was a problem hiding this comment.
The JoinHandle returned here is discarded, so this connection task is not owned by the process-level JoinSet. When drain_tasks() reaches its deadline and calls abort_all(), it can abort only the listener task running serve(); connection tasks spawned here may keep running independently. As a result, RunningGateway::shutdown() can return while a timed-out handler is still executing in the same runtime. Once write APIs are added, work with backend side effects could continue after shutdown has reported completion.
Could we keep lifecycle ownership of these connection tasks and abort + join them when the drain deadline expires? A regression test with a handler that outlives the drain timeout could also assert that the handler is dropped before shutdown() returns.
|
|
||
| - name: OpenAPI drift check | ||
| run: | | ||
| cargo test --lib protocol::rest::openapi::tests::export_checked_in_document -- --ignored --exact |
There was a problem hiding this comment.
The Cargo validation commands in this workflow currently run without --locked. If Cargo.toml is changed without the corresponding Cargo.lock update, Cargo may rewrite the lockfile in the CI workspace and still pass, and the workflow does not check that diff. Similarly, the checked-in DEPENDENCIES.rust.tsv has a manual generation recipe but no drift check.
Could we run the relevant validation commands with --locked and add an inventory regeneration plus git diff --exit-code, similar to the OpenAPI drift check? That would ensure the committed lockfile and license inventory describe the dependency graph CI actually validated.
Purpose
Linked issue: close #3958
First PR of FIP-49: the Gateway runtime and HTTP foundation. Authentication, the Fluss backend and the data
APIs come in follow-up issues.
Distilled from the FIP-49 PoC; the foundation was written by @gstamatakis95 and is credited via
Co-authored-by.Brief change log
fluss-gateway/as an independent Cargo workspace, library plus executable.gateway.yamlconfiguration: flat dotted keys, env and CLI overrides, validation, stable exit codes.GET /healthreturning{status, uptime_ms}.Tests
66 tests: unit tests, the HTTP contract over a real listener, and the compiled binary's startup, health, SIGTERM and exit codes. A
--features integration_testssuite drives the gateway against a dockerized Fluss cluster on demand; it is not a pull-request gate yet.API and Format
New endpoints
GET /healthandGET /v1/openapi.json. No change to existing APIs or storage formats.Documentation
Module-level rustdoc, enforced by
cargo doc -D warnings.