Skip to content

Speed up OIDC login by reconciling org/team memberships in bulk - #662

Open
2and3makes23 wants to merge 8 commits into
ctrliq:mainfrom
2and3makes23:speed_up_oidc_login
Open

Speed up OIDC login by reconciling org/team memberships in bulk#662
2and3makes23 wants to merge 8 commits into
ctrliq:mainfrom
2and3makes23:speed_up_oidc_login

Conversation

@2and3makes23

@2and3makes23 2and3makes23 commented Aug 15, 2026

Copy link
Copy Markdown

This aims at improving #661

SUMMARY

Unify the two per-mapping pipeline steps (update_user_orgs, update_user_teams) into a single
update_user_org_team_mappings step that follows the LDAP backend's desired-state approach:

  • compute the desired org/team membership in memory, create missing orgs/teams once via awx.sso.common.create_org_and_teams, then apply every change via one bulk
  • reconcile_users_org_team_mappings call. The default SOCIAL_AUTH_PIPELINE
    (awx/settings/defaults.py) now runs the merged step only.

This removes the per-org/per-team get_or_create plus immediate m2m add/remove that ran on every
login, eliminating the noticeable login delay once more than a few orgs/teams are mapped.

Changes
  • _update_m2m_from_expression now matches LDAP's _update_m2m_from_groups, evaluating a mapping
    expression to a tri-state (True/False/None).
  • Desired-state computation is shared via _compute_org_desired_states / _compute_team_desired_states,
    keeping the merged default and the legacy entry points in lockstep.
  • Per-login lookups are scoped to the configured maps instead of scanning whole tables:
    • create_org_and_teams returns early for empty maps and only reads referenced orgs
      (get_orgs_by_ids(names=...), using the unique Organization.name index); the full scan is kept
      only for the SAML attribute-based removal path.
    • Team-existence checks are scoped to the mapped organization, so a team whose name exists in
      another org is still created where the mapping expects it.
    • reconcile_users_org_team_mappings scopes the team lookup by organization name, hitting the
      composite (organization, name) index instead of scanning the Team table.
Kept for backwards compatibility
  • update_user_orgs and update_user_teams remain for custom SOCIAL_AUTH_PIPELINE settings that
    still reference them. They keep their original per-domain behavior - orgs only, teams only - so
    custom pipelines don't start managing the other domain, while still using the same bulk helpers
    and keeping the performance win.
Tests
  • Unit test pins the default SOCIAL_AUTH_PIPELINE to the merged step (legacy entry points not part
    of it).
  • Functional tests cover expression matching (match() start-anchored semantics), merged-step
    org/team membership cases, per-domain wrapper behavior, and the scoped lookups - using direct
    role-membership assertions to avoid Role.__contains__ ancestor masking.
  • The sso functional conftest replicates the DAB resource_registry ServiceID post_migrate hack
    so the tests run standalone with --nomigrations.
ISSUE TYPE
  • New or Enhanced Feature
COMPONENT NAME
  • Other (SSO login path in awx/sso)
ASCENDER VERSION

Tested with 25.5.1

ADDITIONAL INFO

There is more potential to increase performance (e.g. mentioned here), but these changes seemed to be the lowest hanging fruits with a large impact on OIDC login performance.

We experienced a delay of multiple seconds until forwarding to Ascender frontend. With this there is no noticable delay anymore.

Replace the two per-mapping pipeline steps (update_user_orgs,
update_user_teams) with a single update_user_org_team_mappings step that
follows the LDAP backend's desired-state approach (awx.sso.backends):
* compute the desired org/team membership in memory
* create missing orgs/teams once via awx.sso.common.create_org_and_teams,
* then apply every change through one bulk reconcile_users_org_team_mappings call.

This removes the get-or-create plus immediate m2m add/remove that ran per
org/team on every login and became slow with large
SOCIAL_AUTH_ORGANIZATION_MAP, SOCIAL_AUTH_TEAM_MAP configurations.

_update_m2m_from_expression now matches LDAP's _update_m2m_from_groups,
evaluating a mapping expression to a tri-state (True/False/None).

Kept for backwards compatibility:
* update_user_orgs and update_user_teams remain as compatibility wrappers for
  custom SOCIAL_AUTH_PIPELINE settings that still reference them;

The default SOCIAL_AUTH_PIPELINE in awx/settings/defaults.py now runs the merged step only.
…step

- Unit test pins the default SOCIAL_AUTH_PIPELINE to the new merged step;
  update_user_orgs/update_user_teams stay as wrappers for custom pipelines
  and are documented as full-merge delegates.
- Functional tests cover expression matching (incl. start-anchored .match()
  semantics) and the merged-step org/team membership cases.
- Fix create_org_and_teams to scope team existence to the mapped org, so a
  team whose name already exists in another org is still created where the
  mapping expects it; add regression and users:/admins: False coverage.
- create_org_and_teams returns early for empty maps, skipping full
  Organization/Team scans on every login.
- Assert on direct role memberships where Role.__contains__ ancestor matches
  would mask a missing member_role.
- Add sso functional conftest replicating the DAB resource_registry ServiceID
  post_migrate hack so these tests run standalone with --nomigrations.
@blaipr blaipr mentioned this pull request Aug 17, 2026
@cigamit
cigamit requested a lite review from Copilot August 17, 2026 19:59
@cigamit cigamit self-assigned this Aug 17, 2026
@cigamit cigamit added the bug Something isn't working label Aug 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the Social Auth (OIDC) login pipeline to reconcile organization/team memberships using a single desired-state bulk operation (mirroring the LDAP backend) to reduce per-login database churn and improve login latency.

Changes:

  • Replace per-mapping get_or_create + immediate M2M updates with a merged update_user_org_team_mappings step that computes desired state in memory and reconciles in bulk.
  • Improve team creation correctness in create_org_and_teams by scoping existence checks by (organization, team_name) (supporting same team name across multiple orgs).
  • Add unit/functional coverage to ensure defaults use the merged pipeline step and to validate tri-state expression evaluation behavior.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file
File Description
awx/sso/social_pipeline.py Introduces merged Social Auth pipeline step and tri-state expression evaluation, delegating legacy wrappers to the merged step.
awx/sso/common.py Adjusts org/team creation logic (team existence scoped by org); adds an early-return guard.
awx/settings/defaults.py Switches default SOCIAL_AUTH_PIPELINE to the merged org/team mapping step.
awx/sso/tests/unit/test_pipelines.py Adds a unit test asserting the default pipeline uses the merged step and not the legacy wrappers.
awx/sso/tests/functional/test_social_pipeline.py Adds functional tests for tri-state expression evaluation and merged reconciliation behavior.
awx/sso/tests/functional/conftest.py Adds a test-only post-migrate hook to create ServiceID so SSO functional tests can run in isolation.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread awx/sso/common.py
Comment thread awx/sso/common.py Outdated
Comment thread awx/sso/tests/functional/conftest.py Outdated
@cigamit

cigamit commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Copilot found a few things, only the first one is really important. Fix that and this should be good to merge.

2and3makes23 added 5 commits August 18, 2026 23:05
- common.py: skip the full Organization-table read when no orgs/teams
  are supplied, and scope the existing-team lookup to the orgs and names
  referenced by team_map instead of scanning every team in the DB.
- conftest.py: replace inline post_migrate lambdas with a stable,
  module-level receiver (weak=False, dispatch_uid) so ServiceID creation
  can't be lost to garbage collection.
- get_orgs_by_ids() now accepts an optional names= filter so
  create_org_and_teams only queries the orgs referenced by the configured
  maps instead of scanning the whole Organization table on every login.
- Organization.name is unique, so the name__in filter uses the name index
  rather than a full table scan.
- The full-scan path (names=None) is kept for the SAML attribute-based
  removal flow, which must enumerate every org.
- Add a functional test pinning the scoped lookup.
- Replace `user in <role>` asserts with direct
  `role.members.filter(pk=user.pk).exists()` checks in the merged-pipeline
  functional tests.  Role.__contains__ matches ancestor roles, so an org
  admin satisfies `user in member_role` and could mask a missing direct
  member membership.
- Pin the "users only" semantics in test_orgs_and_teams_created_from_maps
  and test_boolean_true_matches_everyone by also asserting the user was not
  granted the org admin role, and assert the mapped team's direct membership.
- No production code is touched; the fix only sharpens regression coverage
  for the refactored OIDC membership step.
…ry points

The merged step (update_user_org_team_mappings) kept the default pipeline fast,
but making the legacy entry points delegate to it changed behavior for custom
SOCIAL_AUTH_PIPELINEs: org-only pipelines began reconciling teams and team-only
pipelines began managing org memberships.

This restores the original contract - update_user_orgs touches organizations
only, update_user_teams touches teams only - while preserving the performance
win: each entry point now uses the same bulk create_org_and_teams +
reconcile_users_org_team_mappings helpers as the merged step, so custom
pipelines keep the single-query reconcile instead of the old per-mapping
get_or_create + m2m loop.

The desired-state logic is shared via _compute_org_desired_states and
_compute_team_desired_states, keeping the merged default and the legacy entry
points in lockstep.  The default SOCIAL_AUTH_PIPELINE stays on the merged
step, so stock installs still reconcile everything in one pass.
reconcile_users_org_team_mappings looked up every team whose name appears
in the desired team states across all organizations:
    Team.objects.filter(name__in=team_names)

Team has no standalone index on name (uniqueness is per
(organization, name)), so this forced a full-table scan on every
login.  Rows for organizations outside the desired state were loaded
and then skipped, so the extra work was pure waste.

Scope the query by the organization names being reconciled:
    Team.objects.filter(
        name__in=team_names,
        organization__name__in=desired_states.keys(),
    )

This confines the lookup to the composite (organization, name) index
and bounds the result set by the configuration instead of the size of
the Team table.  Behavior is unchanged: teams in non-mapped
organizations were already ignored by the role-resolution logic, they
are simply no longer read.

Add test_reconcile_team_query_scoped_to_mapped_orgs to pin that
same-named teams in non-mapped organizations keep their memberships
while the mapped organization's team is reconciled.
@2and3makes23

Copy link
Copy Markdown
Author

I tried to incorporate all recommendations, also ran a few more checks and added some more smaller improvements + tests.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Suppressed comments (2)

awx/sso/social_pipeline.py:42

  • Using type(re.compile('')) inside isinstance() recompiles an empty regex pattern on every call, which adds avoidable overhead in this per-login hot path. Prefer using a cached pattern type (e.g., re.Pattern) for the isinstance checks.
        if isinstance(opts, (str, type(re.compile('')))):
            opts = [opts]

        for expression in opts:
            if isinstance(expression, str):

awx/sso/social_pipeline.py:96

  • _compute_org_desired_states() resets desired_org_states[organization_name] for every entry. If multiple ORGANIZATION_MAP items resolve to the same organization_name via organization_alias, the later entry overwrites earlier desired states, which can drop previously computed role grants/removals compared to the historical per-entry behavior.
        desired_org_states[organization_name] = {}

        # Social auth currently supports organization admins and users.
        org_roles_and_expressions = {
            'admin_role': 'admins',

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Development

Successfully merging this pull request may close these issues.

3 participants