Speed up OIDC login by reconciling org/team memberships in bulk - #662
Speed up OIDC login by reconciling org/team memberships in bulk#6622and3makes23 wants to merge 8 commits into
Conversation
Replace the two per-mapping pipeline steps (update_user_orgs, update_user_teams) with a single update_user_org_team_mappings step that follows the LDAP backend's desired-state approach (awx.sso.backends): * compute the desired org/team membership in memory * create missing orgs/teams once via awx.sso.common.create_org_and_teams, * then apply every change through one bulk reconcile_users_org_team_mappings call. This removes the get-or-create plus immediate m2m add/remove that ran per org/team on every login and became slow with large SOCIAL_AUTH_ORGANIZATION_MAP, SOCIAL_AUTH_TEAM_MAP configurations. _update_m2m_from_expression now matches LDAP's _update_m2m_from_groups, evaluating a mapping expression to a tri-state (True/False/None). Kept for backwards compatibility: * update_user_orgs and update_user_teams remain as compatibility wrappers for custom SOCIAL_AUTH_PIPELINE settings that still reference them; The default SOCIAL_AUTH_PIPELINE in awx/settings/defaults.py now runs the merged step only.
…step - Unit test pins the default SOCIAL_AUTH_PIPELINE to the new merged step; update_user_orgs/update_user_teams stay as wrappers for custom pipelines and are documented as full-merge delegates. - Functional tests cover expression matching (incl. start-anchored .match() semantics) and the merged-step org/team membership cases. - Fix create_org_and_teams to scope team existence to the mapped org, so a team whose name already exists in another org is still created where the mapping expects it; add regression and users:/admins: False coverage. - create_org_and_teams returns early for empty maps, skipping full Organization/Team scans on every login. - Assert on direct role memberships where Role.__contains__ ancestor matches would mask a missing member_role. - Add sso functional conftest replicating the DAB resource_registry ServiceID post_migrate hack so these tests run standalone with --nomigrations.
There was a problem hiding this comment.
Pull request overview
This PR refactors the Social Auth (OIDC) login pipeline to reconcile organization/team memberships using a single desired-state bulk operation (mirroring the LDAP backend) to reduce per-login database churn and improve login latency.
Changes:
- Replace per-mapping
get_or_create+ immediate M2M updates with a mergedupdate_user_org_team_mappingsstep that computes desired state in memory and reconciles in bulk. - Improve team creation correctness in
create_org_and_teamsby scoping existence checks by(organization, team_name)(supporting same team name across multiple orgs). - Add unit/functional coverage to ensure defaults use the merged pipeline step and to validate tri-state expression evaluation behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| awx/sso/social_pipeline.py | Introduces merged Social Auth pipeline step and tri-state expression evaluation, delegating legacy wrappers to the merged step. |
| awx/sso/common.py | Adjusts org/team creation logic (team existence scoped by org); adds an early-return guard. |
| awx/settings/defaults.py | Switches default SOCIAL_AUTH_PIPELINE to the merged org/team mapping step. |
| awx/sso/tests/unit/test_pipelines.py | Adds a unit test asserting the default pipeline uses the merged step and not the legacy wrappers. |
| awx/sso/tests/functional/test_social_pipeline.py | Adds functional tests for tri-state expression evaluation and merged reconciliation behavior. |
| awx/sso/tests/functional/conftest.py | Adds a test-only post-migrate hook to create ServiceID so SSO functional tests can run in isolation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
Copilot found a few things, only the first one is really important. Fix that and this should be good to merge. |
- common.py: skip the full Organization-table read when no orgs/teams are supplied, and scope the existing-team lookup to the orgs and names referenced by team_map instead of scanning every team in the DB. - conftest.py: replace inline post_migrate lambdas with a stable, module-level receiver (weak=False, dispatch_uid) so ServiceID creation can't be lost to garbage collection.
- get_orgs_by_ids() now accepts an optional names= filter so create_org_and_teams only queries the orgs referenced by the configured maps instead of scanning the whole Organization table on every login. - Organization.name is unique, so the name__in filter uses the name index rather than a full table scan. - The full-scan path (names=None) is kept for the SAML attribute-based removal flow, which must enumerate every org. - Add a functional test pinning the scoped lookup.
- Replace `user in <role>` asserts with direct `role.members.filter(pk=user.pk).exists()` checks in the merged-pipeline functional tests. Role.__contains__ matches ancestor roles, so an org admin satisfies `user in member_role` and could mask a missing direct member membership. - Pin the "users only" semantics in test_orgs_and_teams_created_from_maps and test_boolean_true_matches_everyone by also asserting the user was not granted the org admin role, and assert the mapped team's direct membership. - No production code is touched; the fix only sharpens regression coverage for the refactored OIDC membership step.
…ry points The merged step (update_user_org_team_mappings) kept the default pipeline fast, but making the legacy entry points delegate to it changed behavior for custom SOCIAL_AUTH_PIPELINEs: org-only pipelines began reconciling teams and team-only pipelines began managing org memberships. This restores the original contract - update_user_orgs touches organizations only, update_user_teams touches teams only - while preserving the performance win: each entry point now uses the same bulk create_org_and_teams + reconcile_users_org_team_mappings helpers as the merged step, so custom pipelines keep the single-query reconcile instead of the old per-mapping get_or_create + m2m loop. The desired-state logic is shared via _compute_org_desired_states and _compute_team_desired_states, keeping the merged default and the legacy entry points in lockstep. The default SOCIAL_AUTH_PIPELINE stays on the merged step, so stock installs still reconcile everything in one pass.
reconcile_users_org_team_mappings looked up every team whose name appears
in the desired team states across all organizations:
Team.objects.filter(name__in=team_names)
Team has no standalone index on name (uniqueness is per
(organization, name)), so this forced a full-table scan on every
login. Rows for organizations outside the desired state were loaded
and then skipped, so the extra work was pure waste.
Scope the query by the organization names being reconciled:
Team.objects.filter(
name__in=team_names,
organization__name__in=desired_states.keys(),
)
This confines the lookup to the composite (organization, name) index
and bounds the result set by the configuration instead of the size of
the Team table. Behavior is unchanged: teams in non-mapped
organizations were already ignored by the role-resolution logic, they
are simply no longer read.
Add test_reconcile_team_query_scoped_to_mapped_orgs to pin that
same-named teams in non-mapped organizations keep their memberships
while the mapped organization's team is reconciled.
|
I tried to incorporate all recommendations, also ran a few more checks and added some more smaller improvements + tests. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.
Suppressed comments (2)
awx/sso/social_pipeline.py:42
- Using type(re.compile('')) inside isinstance() recompiles an empty regex pattern on every call, which adds avoidable overhead in this per-login hot path. Prefer using a cached pattern type (e.g., re.Pattern) for the isinstance checks.
if isinstance(opts, (str, type(re.compile('')))):
opts = [opts]
for expression in opts:
if isinstance(expression, str):
awx/sso/social_pipeline.py:96
- _compute_org_desired_states() resets desired_org_states[organization_name] for every entry. If multiple ORGANIZATION_MAP items resolve to the same organization_name via organization_alias, the later entry overwrites earlier desired states, which can drop previously computed role grants/removals compared to the historical per-entry behavior.
desired_org_states[organization_name] = {}
# Social auth currently supports organization admins and users.
org_roles_and_expressions = {
'admin_role': 'admins',
This aims at improving #661
SUMMARY
Unify the two per-mapping pipeline steps (
update_user_orgs,update_user_teams) into a singleupdate_user_org_team_mappingsstep that follows the LDAP backend's desired-state approach:awx.sso.common.create_org_and_teams, then apply every change via one bulkreconcile_users_org_team_mappingscall. The defaultSOCIAL_AUTH_PIPELINE(
awx/settings/defaults.py) now runs the merged step only.This removes the per-org/per-team
get_or_createplus immediate m2m add/remove that ran on everylogin, eliminating the noticeable login delay once more than a few orgs/teams are mapped.
Changes
_update_m2m_from_expressionnow matches LDAP's_update_m2m_from_groups, evaluating a mappingexpression to a tri-state (
True/False/None)._compute_org_desired_states/_compute_team_desired_states,keeping the merged default and the legacy entry points in lockstep.
create_org_and_teamsreturns early for empty maps and only reads referenced orgs(
get_orgs_by_ids(names=...), using the uniqueOrganization.nameindex); the full scan is keptonly for the SAML attribute-based removal path.
another org is still created where the mapping expects it.
reconcile_users_org_team_mappingsscopes the team lookup by organization name, hitting thecomposite
(organization, name)index instead of scanning theTeamtable.Kept for backwards compatibility
update_user_orgsandupdate_user_teamsremain for customSOCIAL_AUTH_PIPELINEsettings thatstill reference them. They keep their original per-domain behavior - orgs only, teams only - so
custom pipelines don't start managing the other domain, while still using the same bulk helpers
and keeping the performance win.
Tests
SOCIAL_AUTH_PIPELINEto the merged step (legacy entry points not partof it).
match()start-anchored semantics), merged-steporg/team membership cases, per-domain wrapper behavior, and the scoped lookups - using direct
role-membership assertions to avoid
Role.__contains__ancestor masking.resource_registryServiceIDpost_migratehackso the tests run standalone with
--nomigrations.ISSUE TYPE
COMPONENT NAME
ASCENDER VERSION
Tested with 25.5.1
ADDITIONAL INFO
There is more potential to increase performance (e.g. mentioned here), but these changes seemed to be the lowest hanging fruits with a large impact on OIDC login performance.
We experienced a delay of multiple seconds until forwarding to Ascender frontend. With this there is no noticable delay anymore.