feat(experimentation): publish warehouse connections to Redis and apply delivery status by gagantrivedi · Pull Request #8551 · Flagsmith/flagsmith

gagantrivedi · 2026-09-18T08:41:23Z

Thanks for submitting a PR! Please check the boxes below:

I have read the Contributing Guide.
I have added information to docs/ if required so people know about the feature.
I have filled in the "Changes" section below.
I have filled in the "How did you test this code" section below.

Changes

Contributes to Flagsmith/flagsmith-analytics-pipeline#29
The warehouse-delivery service reads external connections from the ingestion Redis and leaves each connection's outcome there; this is the API's side of that contract.

sync_environment_ingestion publishes the connection to experimentation:environment_warehouses:<client key> before routing events to the external topic, and removes it after unrouting, so the service never sees an event it cannot place. Credentials go in as the same Fernet ciphertext the database holds.
The connection list and detail views read the delivery service's latest outcome for each verified external connection from the experimentation:warehouse_delivery_status hash with one HMGET and, when it failed, show errored with the reason. Nothing is saved, and a connection that failed verification keeps that result. If the ingestion Redis is down or not configured, the stored status is shown.
The connection's ingestion hook also fires on config and credentials changes, so a rotated password reaches Redis.

Deploy before the service starts, or it commits past the backlog with nothing to deliver to. The service must be given the same WAREHOUSE_CREDENTIALS_SECRET the API task definitions already set.

How did you test this code?

Unit tests for the service functions, the task and the hook; make lint and make typecheck pass; events catalogue regenerated.

…ly delivery status The warehouse-delivery service reads each environment's connection from experimentation:environment_warehouses:<client key> and leaves each connection's outcome in the experimentation:warehouse_delivery_status hash. sync_environment_ingestion now writes the connection, with the credentials as the same Fernet ciphertext the database holds, before the destination so the service never sees an event it cannot place, and removes it after. A one-minute task copies the outcomes onto WarehouseConnection.status and status_detail. The update hook also fires on config and credentials changes.

vercel · 2026-09-18T08:41:30Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
docs	Ready	Preview	Sep 18, 2026 10:35am UTC

2 Skipped Deployments

Project	Deployment	Actions	Updated
flagsmith-frontend-preview	Ignored	Preview	Sep 18, 2026 10:35am UTC
flagsmith-frontend-staging	Ignored	Preview	Sep 18, 2026 10:35am UTC

coderabbitai · 2026-09-18T08:41:31Z

📝 Walkthrough

Walkthrough

The change moves EncryptedJSONField and warehouse credential encryption into the experimentation package. It adds a shared cached Redis client and warehouse delivery synchronisation services. Warehouse connections are published, removed, and read with delivery statuses. Tasks, model hooks, and API responses use these services. Warehouse verification calls now use the verification service. Tests and observability references are updated.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: 🟠 High · up to 11bcd

Failed or pending warehouse connections can still receive routed events, causing delivery failures. This should be fixed before merge.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-09-18T08:49:21Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 98.81%. Comparing base (095addd) to head (11bcdc0).
⚠️ Report is 3 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff            @@
##             main    #8551    +/-   ##
========================================
  Coverage   98.81%   98.81%            
========================================
  Files        1620     1628     +8     
  Lines       66186    66394   +208     
========================================
+ Hits        65401    65609   +208     
  Misses        785      785

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

… module encrypt_warehouse_credentials and decrypt_warehouse_credentials live in core/warehouse_credentials.py, named for the secret they are bound to. EncryptedJSONField and the ingestion sync service both call them.

…ryption into the app Both are only about a warehouse connection's credentials. core/fields.py keeps NoSSRFURLField alone. Migration 0010 now imports the field from its new module, so the migration state matches the model and no new migration is needed.

…e fields say it

…t from the ingestion one warehouse_delivery_sync_service.py holds what the warehouse-delivery service reads and writes: publish_warehouse_connection, remove_warehouse_connection, pop_warehouse_delivery_statuses. ingestion_sync_service.py is back to keys and destinations for the ingestion server. Both use the client in ingestion_redis.py. The old warehouse_delivery_service.py, which verifies a connection when it is saved, is renamed warehouse_verification_service.py.

…ying them into Postgres The connection list and detail views read the delivery service's latest outcome for each verified external connection from Redis with one HMGET and, when it failed, show errored with the reason. Nothing is saved; a connection that failed verification keeps that result. Redis being down or unconfigured falls back to the stored status. The one-minute apply_warehouse_delivery_statuses task and the Lua pop go away, and removing a connection also forgets its outcome so nothing stale shows.

github-actions · 2026-09-18T10:39:37Z

Docker builds report

Image	Build Status	Security report
`ghcr.io/flagsmith/flagsmith-e2e:pr-8551`	Finished ✅	Skipped
`ghcr.io/flagsmith/flagsmith-api-test:pr-8551`	Finished ✅	Skipped
`ghcr.io/flagsmith/flagsmith-frontend:pr-8551`	Finished ✅	Results ✅
`ghcr.io/flagsmith/flagsmith-api:pr-8551`	Finished ✅	Results ✅
`ghcr.io/flagsmith/flagsmith:pr-8551`	Finished ✅	Results ✅
`ghcr.io/flagsmith/flagsmith-private-cloud:pr-8551`	Finished ✅	Results ✅

github-actions · 2026-09-18T10:44:35Z

✅ private-cloud · depot-ubuntu-latest-arm-16 — run #20510 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
35.8 seconds
11bcdc0
🔄 Run: #20510 (attempt 1)

🗂️ Previous results

✅ private-cloud · depot-ubuntu-latest-16 — run #20510 (attempt 1)

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

3 passed

Details

3 tests across 3 suites
48.7 seconds
11bcdc0
🔄 Run: #20510 (attempt 1)

✅ oss · depot-ubuntu-latest-arm-16 — run #20510 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

1 passed

Details

1 test across 1 suite
38.2 seconds
11bcdc0
🔄 Run: #20510 (attempt 1)

✅ oss · depot-ubuntu-latest-16 — run #20510 (attempt 1)

Playwright Test Results (oss - depot-ubuntu-latest-16)

1 passed

Details

1 test across 1 suite
32.9 seconds
11bcdc0
🔄 Run: #20510 (attempt 1)

github-actions · 2026-09-18T10:45:30Z

Visual Regression

19 screenshots compared. See report for details.
View full report

coderabbitai

Actionable comments posted: 1

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 0b14d203-44bd-4373-95b3-191001a70348

📥 Commits

Reviewing files that changed from the base of the PR and between 1fd274c and 11bcdc0.

📒 Files selected for processing (26)

api/core/fields.py
api/experimentation/dataclasses.py
api/experimentation/fields.py
api/experimentation/ingestion_redis.py
api/experimentation/ingestion_sync_service.py
api/experimentation/migrations/0010_warehouse_connection_credentials_and_status_detail.py
api/experimentation/models.py
api/experimentation/services.py
api/experimentation/tasks.py
api/experimentation/views.py
api/experimentation/warehouse_credentials.py
api/experimentation/warehouse_delivery_sync_service.py
api/experimentation/warehouse_verification_service.py
api/tests/unit/core/test_fields.py
api/tests/unit/experimentation/conftest.py
api/tests/unit/experimentation/test_fields.py
api/tests/unit/experimentation/test_ingestion_redis.py
api/tests/unit/experimentation/test_ingestion_sync_service.py
api/tests/unit/experimentation/test_models.py
api/tests/unit/experimentation/test_services.py
api/tests/unit/experimentation/test_tasks.py
api/tests/unit/experimentation/test_views.py
api/tests/unit/experimentation/test_warehouse_credentials.py
api/tests/unit/experimentation/test_warehouse_delivery_sync_service.py
api/tests/unit/experimentation/test_warehouse_verification_service.py
docs/docs/deployment-self-hosting/observability/_events-catalogue.md

Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review.

Zaimwa9

Couple of comments but overall looking good

Zaimwa9 · 2026-09-23T11:44:02Z

+            list[bytes | None],
+            get_client().hmget(WAREHOUSE_DELIVERY_STATUS_KEY, fields),
+        )
+    except RedisError:


I don't think this include RedisClusterException, meaning it would raise a 500, I believe we should catch it too

Zaimwa9 · 2026-09-23T11:44:58Z

+            else None
+        ),
+    }
+    get_client().set(redis_key, json.dumps(document))


I think we should also clear connection devlivery status here to avoid serving a stale one

Zaimwa9 · 2026-09-23T12:03:10Z

+def get_client() -> RedisCluster:
+    """The Redis the ingestion server and the warehouse-delivery service read
+    their configuration from and, for the latter, write outcomes to."""
+    return RedisCluster.from_url(  # type: ignore[no-any-return]


Should it also support a single node redis to serve OSS better? Not sure everyone will go for a cluster mode in modest setups

For oss we will drop redis dependency

github-actions Bot added api Issue related to the REST API docs Documentation updates feature New feature or request and removed docs Documentation updates labels Sep 18, 2026

github-actions Bot added docs Documentation updates feature New feature or request and removed feature New feature or request docs Documentation updates labels Sep 18, 2026

vercel Bot deployed to Preview – docs September 18, 2026 08:54 View deployment

github-actions Bot added feature New feature or request and removed feature New feature or request labels Sep 18, 2026

vercel Bot deployed to Preview – docs September 18, 2026 09:59 View deployment

github-actions Bot added the docs Documentation updates label Sep 18, 2026

gagantrivedi added 2 commits September 18, 2026 15:31

docs(experimentation): drop the WarehouseDeliveryStatus docstring, th…

f4a49cc

…e fields say it

github-actions Bot added feature New feature or request and removed feature New feature or request docs Documentation updates labels Sep 18, 2026

vercel Bot deployed to Preview – docs September 18, 2026 10:17 View deployment

github-actions Bot added docs Documentation updates feature New feature or request and removed feature New feature or request docs Documentation updates labels Sep 18, 2026

vercel Bot deployed to Preview – docs September 18, 2026 10:35 View deployment

gagantrivedi marked this pull request as ready for review September 18, 2026 10:39

gagantrivedi requested review from a team as code owners September 18, 2026 10:39

gagantrivedi requested review from matthewelwell and removed request for a team September 18, 2026 10:39

github-actions Bot removed the feature New feature or request label Sep 18, 2026

github-actions Bot added the feature New feature or request label Sep 18, 2026

coderabbitai Bot reviewed Sep 18, 2026

View reviewed changes

Comment thread api/experimentation/tasks.py

Zaimwa9 requested changes Sep 23, 2026

View reviewed changes

Conversation

gagantrivedi commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

How did you test this code?

Uh oh!

vercel Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Uh oh!

codecov Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Docker builds report

Uh oh!

github-actions Bot commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Playwright Test Results (private-cloud - depot-ubuntu-latest-arm-16)

Details

Playwright Test Results (private-cloud - depot-ubuntu-latest-16)

Details

Playwright Test Results (oss - depot-ubuntu-latest-arm-16)

Details

Playwright Test Results (oss - depot-ubuntu-latest-16)

Details

Uh oh!

github-actions Bot commented Sep 18, 2026

Visual Regression

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Zaimwa9 left a comment

Choose a reason for hiding this comment

Uh oh!

Zaimwa9 Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

Zaimwa9 Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

Zaimwa9 Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

gagantrivedi Sep 23, 2026

Choose a reason for hiding this comment

Uh oh!

This branch was successfully deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

gagantrivedi commented Sep 18, 2026 •

edited

Loading

vercel Bot commented Sep 18, 2026 •

edited

Loading

coderabbitai Bot commented Sep 18, 2026 •

edited

Loading

codecov Bot commented Sep 18, 2026 •

edited

Loading

github-actions Bot commented Sep 18, 2026 •

edited

Loading

github-actions Bot commented Sep 18, 2026 •

edited

Loading