Skip to content

python-rust-integration - #46851

Draft
dibahlfi wants to merge 145 commits into
mainfrom
users/dibahl/python-sdk-with-rust-driver
Draft

dibahlfi wants to merge 145 commits into
mainfrom
users/dibahl/python-sdk-with-rust-driver

Conversation

@dibahlfi

@dibahlfi dibahlfi commented May 13, 2026 •

Copy link
Copy Markdown
Member

migrating from core-python to rust native

@dibahlfi
dibahlfi requested a review from a team as a code owner May 13, 2026 00:29
Copilot AI review requested due to automatic review settings May 13, 2026 00:29
@dibahlfi
dibahlfi marked this pull request as draft May 13, 2026 00:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR introduces an internal “backend dispatch” layer for azure-cosmos (sync + async) to allow selecting between the existing core-Python implementation and an opt-in Rust/PyO3-backed path, including per-request backend tagging in the User-Agent and a unit-test suite to pin down the wiring/selection rules.

Changes:

  • Add sync/async backend abstractions, constants, and factories with selection precedence _backend kwarg > COSMOS_BACKEND env var > default core-python.
  • Wire backend selection into CosmosClient/async client construction, expose backend instances on client_connection, and add create_item dispatch + per-request backend stamping.
  • Extend CosmosUserAgentPolicy to append backend=<name> and add fast unit tests (including an import-guard) to verify wiring and behavior.

Reviewed changes

Copilot reviewed 15 out of 17 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
sdk/cosmos/azure-cosmos/tests/test_backend_wiring_unit.py New unit tests covering import-guard, backend selection precedence/validation, container dispatch behavior, and UA stamping.
sdk/cosmos/azure-cosmos/azure/cosmos/cosmos_client.py Selects/initializes sync backend(s) at client construction and attaches them to client_connection.
sdk/cosmos/azure-cosmos/azure/cosmos/container.py Adds sync create_item dispatch and stamps the chosen backend into per-request options.
sdk/cosmos/azure-cosmos/azure/cosmos/aio/_cosmos_client.py Selects/initializes async backend(s) at client construction and attaches them to client_connection.
sdk/cosmos/azure-cosmos/azure/cosmos/aio/_container.py Adds async create_item dispatch and stamps the chosen backend into per-request options.
sdk/cosmos/azure-cosmos/azure/cosmos/_user_agent_policy.py Appends a backend=<name> token to the User-Agent based on per-request options.
sdk/cosmos/azure-cosmos/azure/cosmos/_backend/base.py Defines the sync backend ABC and shared immutable request/response dataclasses.
sdk/cosmos/azure-cosmos/azure/cosmos/_backend/constants.py Centralizes backend names, env var, and per-request option key.
sdk/cosmos/azure-cosmos/azure/cosmos/_backend/factory.py Implements backend name resolution + sync backend instance factory.
sdk/cosmos/azure-cosmos/azure/cosmos/_backend/core_python.py Sync “core-python” backend stub (returns None to fall through to existing path).
sdk/cosmos/azure-cosmos/azure/cosmos/_backend/rust.py Sync Rust backend stub with guarded PyO3 import and loud NotImplementedError.
sdk/cosmos/azure-cosmos/azure/cosmos/aio/_backend/base.py Defines the async backend ABC and re-exports shared dataclasses.
sdk/cosmos/azure-cosmos/azure/cosmos/aio/_backend/factory.py Async backend factory mapping resolved backend name to async backend instance.
sdk/cosmos/azure-cosmos/azure/cosmos/aio/_backend/core_python.py Async “core-python” backend stub (returns None to fall through).
sdk/cosmos/azure-cosmos/azure/cosmos/aio/_backend/rust.py Async Rust backend stub with guarded PyO3 import and loud NotImplementedError.
sdk/cosmos/azure-cosmos/azure/cosmos/_backend/init.py Backend package marker/export surface (no diff shown).
sdk/cosmos/azure-cosmos/azure/cosmos/aio/_backend/init.py Async backend package marker/export surface (no diff shown).

@dibahlfi dibahlfi changed the title Python V5 python-rust-integration Jun 10, 2026
@github-actions

This comment has been minimized.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 6953df5e-04ab-4c2d-9a7d-8311008c4d86
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

dibahlfi and others added 2 commits November 12, 2026 20:31
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81f127fc-3491-4932-9a45-0b1cdaea46a1
@github-actions

This comment has been minimized.

dibahlfi and others added 3 commits November 16, 2026 13:38
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 13b35e7d-a3e2-4c8b-957b-de08a854af98
@github-actions

This comment has been minimized.

@github-actions

This comment has been minimized.

dibahlfi and others added 2 commits November 20, 2026 00:33
Rename the shared client context, operation defaults, and context handoff across sync/async paths. Refresh only affected assertion-ledger fingerprints.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 81f127fc-3491-4932-9a45-0b1cdaea46a1
@github-actions

This comment has been minimized.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor
[Pilot] PR Pipeline Failure Analysis

What failed

Build 7101376 (python - cosmos - ci) and the related native-test build failed across multiple jobs: breaking, pylint, mypy, docs/readme verification, and Windows emulator test jobs (Python 3.10–3.14).

  1. Rust toolchain missing for breaking/pylint builds — python -m build -s invokes maturin, which fails:

    error: custom toolchain 'ms-prod-1.97' specified in override file 'sdk/cosmos/azure-cosmos/rust-toolchain.toml' is not installed
    💥 maturin failed
      Caused by: Cargo metadata failed. Does your crate compile with `cargo build`?
    

    This also explains numerous Windows emulator unit-test failures that error with ImportError: cannot import name '_rust' from 'azure.cosmos' — the compiled Rust extension is unavailable in these check environments because the sdist/wheel build never produced it.

  2. mypy — 12+ real type errors in azure-cosmos, e.g.:

    azure/cosmos/_backend/_async_credential_bridge.py:137: error: Argument 1 to "get" of "dict" has incompatible type "tuple[int, AbstractEventLoop] | None"; expected "tuple[int, AbstractEventLoop]"  [arg-type]
    azure/cosmos/_telemetry.py:437: error: Value of type "object" is not indexable  [index]
    azure/cosmos/_query_rust_routing.py:157: error: Argument 1 to "_build_partition_key_from_properties" has incompatible type "Mapping[str, Any]"; expected "dict[str, Any]"  [arg-type]
    azure/cosmos/aio/_cosmos_client_connection_async.py:290: error: Argument "global_endpoint_manager" to "CosmosHttpLoggingPolicy" has incompatible type "_GlobalPartitionEndpointManagerForPerPartitionAutomaticFailoverAsync"; expected "_GlobalEndpointManager | None"  [arg-type]
    samples/tracing_rust_failures_async.py:184: error: Item "AsyncLegacyItemHelper" of "AsyncItemHelper | AsyncLegacyItemHelper" has no attribute "_backend"  [union-attr]
    

    Found 12 errors in 5 source files, 14 more in 3 sample files.

  3. README verification failure (ward scan):

    /sdk/cosmos/azure-cosmos/azure_cosmos_rust/README.md is missing headers with patterns:
     * ^Azure (.+ client library for Python|...)
     * ^Getting started$ / ^Key concepts$ / ^Examples$ / ^Troubleshooting$ / ^Next steps$ / ^Contributing$
    /sdk/cosmos/azure-cosmos/tests/workloads/native_experiments/README.md is missing the same required headers.
    
  4. Windows emulator test failures (cosmos-azure-cosmos-test-junit-whl.xml, all Python versions): tests such as test_shared_async_credential_uses_one_bridge_on_application_loop_rust and test_aad_credentials_rust fail on container teardown with:

    azure.cosmos.exceptions.CosmosHttpResponseError: (Forbidden) Request blocked by Auth localhost : ... cannot be authorized by AAD token in data plane.
    Code: Forbidden, Sub Status: 5300 (RbacOperationNotSupported)
    

    This is an emulator/environment RBAC configuration issue (AAD delete_container not permitted against the local emulator), not a product logic bug.

  5. cosmos-rust/offline unit tests (Python 3.13) — ~90 failures, mostly cascading from the missing _rust extension (ImportError: cannot import name '_rust' from 'azure.cosmos'), plus two independent assertion failures:

    • test_copy_generator_unit.test_exemplars_reproduced_byte_for_byte_rust: generated test-exemplar text no longer matches the committed golden file (tests/create_item/sync/test_none_options.py) — generator/golden drift.
    • test_migration_enforcer_unit.test_every_migrated_test_collects_without_conftest_rust: pytest --collect-only on migrated test files returns exit code 2 (collection errors) instead of 0.
  6. cosmos-toxiproxy (Python 3.13): test_read_connection_reset_rust[reset-reset-sync] — fault-injection test asserts the connection was reset but it timed out instead (Established connection closed without the intended fault); likely infra/timing flakiness in the toxiproxy harness.

Recommended next steps

  • Install/provision the ms-prod-1.97 Rust toolchain (or relax the rust-toolchain.toml override) in the build/breaking/pylint CI images so maturin/cargo can build the sdist; this should also resolve the cascading _rust import failures in the offline unit tests.
  • Fix the 12 mypy errors in _async_credential_bridge.py, client_config.py, _telemetry.py, _query_rust_routing.py, and aio/_cosmos_client_connection_async.py, plus the 14 in the sample files, per the cited file:line errors.
  • Add the required headers (Getting started, Key concepts, Examples, Troubleshooting, Next steps, Contributing, and the Azure ... client library for Python title) to azure_cosmos_rust/README.md and tests/workloads/native_experiments/README.md.
  • Re-sync test_copy_generator_unit's generated exemplar with tests/create_item/sync/test_none_options.py, and investigate the pytest --collect-only errors in test_migration_enforcer_unit.
  • For the RBAC Forbidden/toxiproxy timeout failures, confirm these are emulator/environment configuration issues (not product regressions) and consider retrying; if they persist, the emulator's data-plane AAD authorization for delete_container needs reconfiguration.
  • See https://aka.ms/ci-fix

Automated fix: Fix found, view and apply fix

Generated by Pipeline Analysis Next Steps · copilot · sonnet50 · 88.4 AIC · ⌖ 0.714 AIC · ⊞ 9.1K · ◷

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

7 participants