Repository navigation
Release develop to r10.0 - #400
Merged
hamzahalq merged 408 commits intoOct 29, 2026
Merged
Conversation
A mail server with a self-signed or private-CA certificate couldn't be used: NativePop3Receiver connected only by implicit TLS to 995 and both it and NativeSmtpHandler accepted only certificates the pod already trusts. Found setting up the lab's in-cluster mail server; customers' internal relays are the same case. - NativePop3Receiver: Encryption (implicit, the default, or explicit STLS), Port (995 or 110 when empty) and CertificateThumbprint. There's no unencrypted option: it always signs in with a password. - NativeSmtpHandler: CertificateThumbprint. - A pinned thumbprint trusts exactly that certificate and refuses any other before credentials are sent, as the FTP adapters already do. Unset, nothing changes.
BITWEEN_INSECURE printed the warning, and then the client printed its own.
Every message through one gateway carries the same file name,
{urlName}.json, so NativeFtpUploadHandler and NativeSftpUploadHandler wrote
each upload over the one before: of a stream of orders, only the last was
there to collect. Found by the lab, where bulk orders to the 3PL's SFTP
outbox all landed as bulk_acme-bulk-orders.json.
- New optional FileName setting on both: a Scriban template filled from a
JSON message's fields, as order-{{ orderId }}.json. A non-JSON message
takes it as written. Only the last path segment is kept. Empty, or a
template that renders empty, keeps today's naming: the message's own
name, else the time. FileNamePrefix still applies.
- The "fill only when the message is JSON" rule moved from the SMTP
handler into ScribanJsonHelper.FillIfJson, used by both.
Nothing published the provider adapters, so a Bitween could configure data sources but never run one: GET /api/datasources/Providers listed nothing. - Each provider project (RabbitMQ, SQS, PostgreSQL, MySQL, SQL Server, Oracle) has an adapter.json marking it resident, so bitween adapter build packages it. The projects move to SimplyWorks.Serverless.Sdk 10.1.0, the first that build can describe. - Every CLI release builds the six packages with its version and attaches them beside the binaries, in SHA256SUMS: bitween adapter publish, or an apply configuration's adapters section, installs them. - The chart passes every environmentVariables key through, as the docs said it did; only three were rendered. Nothing changes without new keys. - Staging sets Bitween__BusProvidersEnabled=true. Integration tests (which run the providers) pass on the new SDK: 680.
The exchanges page is rebuilt around how it gets used: find what failed, read why, act on it. - Status tabs with a count for each status under the other filters, and the newest-attempt switch beside them. - One query bar instead of ten inputs: a time chip with presets, typed ranges and custom times; filters as chips that open their own editor; an add-filter menu; and a search box that tells exchange ids, correlation ids, property values and error text apart. - Saved views, built in and your own (kept in this browser), and a link to the current view. - A volume chart by status over the time range; click or drag across it to zoom in. - The open exchange beside the list instead of inside it, with its stage documents, error, retry chain, details and actions; J and K move through the list, X ticks, Esc closes. A link to one exchange opens it. - Rows show the first line of their error; the table narrows while an exchange is open, and becomes cards on a phone. - Paused, the list stays still and counts what has arrived since. Backend, additive only: GET /xchanges/summary (status counts capped like the list total, and a zero-filled histogram when a start time is given), and the filters ErrorContains, IsRetry, HasScheduledRetry and Aggregated, shared by the search and bulk selections. Every existing URL parameter still means what it did.
SSH.NET waits for ever by default. In the lab, an SFTP server that signed in but never opened the SFTP session held each upload for up to 102 minutes, until the server went away. Every held upload kept a place in the processing queue's prefetch, and once those were full nothing else in Bitween was processed: the gateway's synchronous calls all fell back to 202 after 30 seconds. - The SFTP receiver and upload handler take TimeoutSeconds (default 60) for the connection, the SFTP session and each request. A missing or non-positive value is 60, never unlimited. - Transport test: a server that signs in and never answers fails the upload within the limit (it waits indefinitely without the fix).
Bitween stamps the manifest and zips a package again when it takes it, so the hash it lists for a version is never the uploaded file's. Comparing the two warned "already published from a different package" on every plan of an adapters section, whatever the package. The check is gone; the apply test now plans the same package again and expects no such warning (it fails without the fix).
Bitween keeps no information type for an aggregation. A file that gave one was applied once, then read back as a change to a field that can't change, and every apply after failed on it. The plan now says so before anything is made.
The handler failed a delivery with the status's name alone (ServiceUnavailable: ...), so a retry policy written as the docs show, matching "503:", never matched, and a partner outage was never retried. In the lab, 48 orders the warehouse refused with 503 during a 20-minute outage were never delivered. The message is now "503: ServiceUnavailable: ...": the number the docs match, then the name it always carried, so matchers written against either keep working.
…ess as such With an exchange open beside the list, the Pipeline and Route columns used to disappear. They now fold into the columns that remain: the pipeline strip and its phrase under the status, and the subscription and partner on the exchange's second line. A stage underway was a faint grey pulse that read as faded rather than working. It is now a striped bar moving in the focus blue, with a spinner beside "Mapping now" or "Handling now", an "In progress" badge on the open exchange's stage card, and a line saying its document shows when it finishes. Held still for reduced motion. One stage runs at a time: while mapping is underway, handling shows as not reached yet instead of also running.
Bitween reads a budget that leaves maxAttemptsTotal out as zero, which allows no retries at all; it keeps that for stored policies. The lab's policy left it out, and for a day nothing in the lab was retried: every partner outage, timeout and dropped connection lost what it touched. bitween apply now refuses such a budget and says why; 0 written out, or null for no ceiling, is taken as said. The docs say what a missing total means.
Built-in adapters run in Bitween's own process with no limit of their own, so a call that never returned held its place in the processing queue for good. In the lab, SFTP uploads to a server that signed in and then said nothing held every place within minutes, and nothing else in Bitween was processed for two hours. Bitween:NativeAdapterTimeout (default 900 seconds, 0 for none) caps one call. Past it the exchange fails as a timeout, which a retry policy can retry, and the adapter is disposed, closing what it holds open. The call itself can't be stopped, only left, so a retry may deliver twice what a hung partner did receive; the docs say so. Shown on the settings page, read-only like the custom adapter timeout.
Quartz isn't clustered, so on several replicas each aggregation fires on all of them at once. An exchange can be rolled up only once, so no roll-up was ever duplicated, but every run but one failed on the duplicate key, logged an error and recorded nothing. The job now claims the subscription with the running flag receivers already use; the others return quietly. The new test runs two at once and fails without the change.
The lab's phase 6 deletes one mid-traffic, and anything that runs on every replica at once, scheduled jobs and data source leases among them, shows up on staging first.
The staging branch was renamed from releases/r10.0-staging. The deploy and the CLI pre-release still waited for the old name, so a push to the new one built nothing.
Each line of the block is handed to helm --set, so the comment explaining replicaCount=2 became a key with no value and the staging deploy failed. The comment moves above the block.
releases/r10.0-staging was renamed to releases/r10.0-develop, which no job matched, so pushes to develop built nothing (run 142 skipped). Develop now publishes under its own names, so nothing it builds can reach clients: - develop: simplify9/bitween-develop image and the bitween-develop chart (ChartMuseum + GHCR), deployed to playground, no NuGet packages. charts/develop is charts/default symlinked under its own name, since the chart names the image after itself. The playground release stays bitween (nameOverride=bitween), so it upgrades in place: same resource names and selectors. - develop-tests runs the unit and integration tests, which the reusable only runs in its NuGet job. - r10.0: unchanged release (bitween chart to ChartMuseum, simplify9/bitween, NuGet, CLI, SDKs). - One concurrency queue per branch: the two pipelines no longer share a published name, and a shared queue let a develop push cancel a pending release. - Manual runs follow the branch they are started on. CLI pre-releases are -dev.<run>.
ci: split the develop and release pipelines
Since AWSSDK.S3 3.7.410 uploads go aws-chunked with a trailing checksum, which S3-compatible stores that don't know it save into the object: the lab's invoices arrive in its S3 starting "C3;chunk-signature=...". The SDK's two environment variables put back the old behaviour on staging, to show whether that default is the cause before SW-CloudFiles sets it for good.
A sync call whose delivery failed was answered 400 with no body, which tells the caller its request was wrong. In the lab, while a warehouse was down, 1,015 orders were answered 400 and then delivered by retry minutes later, each followed by a confirmation and a customer email for an order the marketplace had been told was rejected. Each gateway now has failureAnswer. Accurate answers 202 with the exchange id and the next attempt when Bitween has scheduled a retry (it has the message: a resend would make a second one), 422 when the message couldn't be mapped, and 502 when the delivery failed and won't be retried. Legacy answers 400 as before. Gateways made before this keep Legacy (the column defaults to it); new gateways answer Accurate unless told otherwise; leaving the field out on update keeps it. The gateway page, the API docs and bitween apply carry it.
The docs and several comments said Quartz was not clustered, so every replica ran every scheduled job. That stopped being true when SimplyWorks.Scheduler 8.1.7 made clustering unconditional on all three persistent stores (SchedulerId AUTO, UseClustering); 10.0.0 is pinned now. These tests start two real schedulers registered as Startup registers them, against one migrated PostgreSQL, and record every fire through a Quartz job listener: the store is clustered with distinct instance ids, no fire runs on two nodes, the survivor carries on when the node running the jobs stops (21 to 30 s, Quartz's idle wait), and a single node runs every fire.
Quartz runs clustered, so each fire already runs on one replica only. The replica section of architecture.md, the scheduling and caveats docs, the Startup comment and the aggregation comments still said otherwise, and caveats still listed aggregations as having no running flag. The run flags stay and their reason is now the real one: each schedule of a subscription and each Receive now or Aggregate now is a Quartz job of its own, so two runs of one subscription can overlap.
A delivery to a partner that has stopped answering holds its place in the
processing queue for its whole timeout. In the lab an ERP answering in 30 s
against a 20 s timeout filled every slot of the shared queue, and orders to a
healthy warehouse went late; hung SFTP uploads once stopped processing
altogether. Work groups isolate queues, but only when someone sets them up.
The built-in HTTP, SMTP, SFTP and FTP upload handlers now share a circuit
breaker per destination (scheme, host and port), per process. Timeouts,
refused or dropped connections, 429 and any 5xx but 501 count; 4xx business
rejections and failures before anything is sent do not. After
Bitween:CircuitBreakerFailures (5) such failures in a row within
Bitween:CircuitBreakerWindowSeconds (60), calls to that destination fail at
once for Bitween:CircuitBreakerOpenSeconds (60) with
"503: ServiceUnavailable: circuit open for {destination} (...)", so retry
policies that retry a 503 retry them. One trial call then goes through:
success closes the circuit, failure opens it again. Opening is logged as a
warning and counted as bitween.circuits.opened; closing is logged.
CircuitBreakerFailures = 0 switches it off. The handlers take the registry
as an optional constructor argument, so code that builds them directly keeps
working without one. With the defaults nothing changes until a destination
keeps failing.
NotifyResult held the call to a notifier's handler, with its quick retries, inline. Alerts that have no exchange behind them need the same call, so it moves to NotifierDelivery and NotifyResult uses it. Nothing changes for exchange results: same payload, same xchangeid property, same file name, same three attempts.
When a partner's system breaks, a carrier moves its folder or a table stops filling, no exchange is created, so nothing fails and nobody is told. Every entry point creates exchanges on a subscription, so that is where the expectation goes. ExpectTraffic is optional and stored as one JSON column: at least one exchange every N minutes (5 or more), on chosen days, between From and To, in an IANA time zone (UTC when empty). TrafficWindow checks it on save with messages that say what to change, stores it in one spelling, and answers when silence counts: only inside the window, and only once the window has been open a whole gap, so last evening's order doesn't make a subscription quiet at 08:01. QuietSince records when it was found quiet; Get and the search return it and an update can't set it. The field is replaced on update like its neighbours (AutoPauseAfterFailures, CustomRetryPolicy), so an update without it removes it, and removing it clears QuietSince. bitween apply manages expectTraffic and compares it whole, ignoring the nulls the API answers, so a plan doesn't show a difference that isn't there. Migration ExpectTraffic adds two nullable columns on all three providers; no subscription changes behaviour until it is given one.
A subscription that expects traffic and gets none still fails nothing. QuietSubscriptionsJob runs every five minutes (QuietSubscriptionsCron) over active, unpaused subscriptions with an expectation. Inside the window, when no exchange started within the expected gap, it sets QuietSince and tells the notifiers that watch the subscription for failed results. When an exchange starts after QuietSince, inside the window or not, it clears it and tells them it is back. Both changes are conditional updates, so each is sent once. The notifiers get the XchangeResultNotification an exchange result sends, through the same helper and retries, with a new optional Event: "gone-quiet" (Success false) or "back" (Success true), and a plain sentence in Exception. It is null for exchange results, so existing handlers see nothing new. There is no exchange, so the id is one made for the alert and the send is logged rather than written as an XchangeNotification. Each question asked of the exchanges is bounded to the recent past, so it stays cheap on an instance that keeps years of them. Quiet subscriptions are also listed in the dashboard insights and the job on the background jobs page.
The overview gets an Expect traffic panel: every N minutes, days, From and To, and a time zone with the browser's list as suggestions. It is part of the page's draft, so it saves with the rest. A subscription the server found quiet shows "Quiet since ..." beside the pause notice. A save sends the whole subscription and the server replaces the expectation like its other fields, so every save carries the stored one over unless the page changed it, as it does for auto-pause.
How a subscription is found quiet and back, what the notifiers receive, the API fields and their rules, QuietSubscriptionsCron, and how bitween apply compares expectTraffic.
fix: prod mirror picker speed, required secrets and user names
…za/feat/ai-retry-suggestions
…ns' into hamza/feat/ai-retry-suggestions
fix: prod mirror reads prod once, paused jobs wait, no credentials copied
…ions Explain failed exchanges with AI
Ask your exchanges in plain English
# Conflicts: # docs/configuration.md
fix: bring from prod no longer stalls after five quiet minutes
Suggest retry policies with AI
WalkthroughWarning Review details and warnings were omitted to fit the comment limit. |
mmalkhatib
approved these changes
Oct 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Brings releases/r10.0-develop into releases/r10.0, with the two merge conflicts (the pipeline file and docs/deployment.md) resolved on develop's side. The files are exactly develop's, plus one pipeline change: production now gets its settings encryption key from the new
PROD_SETTINGSENCRYPTIONKEYsecret.Merging deploys app.bitween.io, since
BITWEEN_PROD_DEPLOYis on. The update runs old and new pods together briefly; if real clients use app.bitween.io, scale it to 0 just before merging.The 7 merge-gate tests passed on develop 570157b on 29 Oct, including an r10.0 → develop upgrade on a 2M-exchange database (34 migrations in about 6 s).