MAINT: Dep audit + ty hardening + scenario import cleanup#1931
Merged
romanlutz merged 19 commits intoJun 5, 2026
Conversation
Remove `mock-alchemy` plus 9 `types-*` stubs that are no longer needed: - `mock-alchemy`: last importer (UnifiedAlchemyMagicMock in `test_azure_sql_memory.py`) was deleted in PR microsoft#1128 (2025-10-24, "FEAT Breaking: Default Values") when the SQL test suite migrated to in-memory SQLite + real SQLAlchemy sessions. `pyproject.toml` was missed in that cleanup. - `types-cachetools`, `types-decorator`, `types-paramiko`, `types-pycurl`, `types-pytz`, `types-simplejson`, `types-six`, `types-tabulate`, `types-ujson`: added in commit `a5a0dd51` (2026-01-13, "more typing dev deps") as setup for strict mypy (PR microsoft#1310). The underlying packages are not imported anywhere in `pyrit/`, `tests/`, `doc/`, or `build_scripts/`, and PyRIT has since migrated off mypy to `ty`. Type checkers only consult stubs for imports they actually see, so these have zero effect. Keep `types-aiofiles`, `types-PyYAML`, `types-requests`: the underlying packages are imported widely and ship no inline type info (no `py.typed`), so the stubs are the only type information available. Verified with `uv sync --link-mode=copy` (11 packages uninstalled cleanly, including transitive `sqlalchemy-utils` from mock-alchemy) and `uv run pytest -n 4 --dist=loadfile tests/unit` (8382 passed, 120 skipped — no break from mock-alchemy removal). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Promote `[tool.ty.rules].unresolved-import` from `ignore` to `error`
so that newly-added imports of packages not declared in pyproject.toml
fail type-checking (previously they silently fell back to `Unknown`).
Verified with `ty check pyrit tests`: 1253 diagnostics, identical
non-unresolved-import baseline to before this change. Total unresolved-
import errors: 0.
Mitigations applied to surface real issues without churn:
1. Optional-extras files (azure-speech, cv2, playwright, torch, spacy)
are imported inside `try/except ImportError` blocks for graceful
degradation. Added a per-file `[[tool.ty.overrides]]` block in
pyproject.toml scoping `unresolved-import = "ignore"` to those 19
files (10 source + 9 unit-test mirrors).
2. `typing.Self` was imported from `typing` in three registry files
(`pyrit/registry/base.py`, `…/class_registries/base_class_registry.py`,
`…/object_registries/base_instance_registry.py`). `Self` arrived
in `typing` only on Python 3.11; PyRIT's min is 3.10. Real latent
compat bug, only mitigated today by `from __future__ import
annotations` + `TYPE_CHECKING`. Switched to `typing_extensions.Self`
(already a transitive dependency via pydantic et al — no new dep).
3. `ExceptionGroup` is imported via a conditional `try/except
ImportError` fallback (builtin on 3.11+, `exceptiongroup` package
on 3.10). Added `# type: ignore[…,ty:unresolved-import]` to both
branches in `pyrit/scenario/core/scenario.py` and
`tests/unit/scenario/test_scenario.py`.
4. Four scenario tests (`test_foundry.py`, `test_encoding.py`,
`test_leakage_scenario.py`, `test_strategy_validation.py`) and
one partner-integration test (`test_foundry_scenario_contract.py`)
were importing scenario subpackages via the runtime `sys.modules`
aliasing trick set up in `pyrit/scenario/__init__.py`. ty can't
follow that. Updated test imports to the real paths
(`pyrit.scenario.scenarios.{foundry,garak,airt}`). The user-facing
public API in `pyrit/scenario/__init__.py` and in user-facing docs
is unchanged.
5. The `azure.ai.evaluation.red_team*` package isn't declared in
pyproject.toml — it's an optional partner integration. Added
`# type: ignore[ty:unresolved-import]` to the 4 deferred imports
in `tests/partner_integration/azure_ai_evaluation/test_import_smoke.py`.
6. **Real bug fix:** `tests/partner_integration/azure_ai_evaluation/
test_scorer_contract.py` imported `ScorerIdentifier` from
`pyrit.identifiers`, but the symbol was renamed to
`ScorerEvaluationIdentifier`. Fixed the import.
7. The above changes (especially `Self` resolving correctly) exposed
four latent type-system issues in pre-existing code. Suppressed with
`# type: ignore[ty:<rule>]` matching the dominant convention in
this codebase (169 existing usages):
- `base_class_registry.py:153` and `base_instance_registry.py:92`:
`return cls._instances[cls]` returns `BaseRegistry[Any]` from a
class-level dict but the signature is `-> Self`. Type-system
limitation; refactoring is out of scope.
- `test_attack_technique_registry.py:262`: intentionally assigns
to a read-only property to assert it raises `AttributeError`.
- `test_targets_initializer.py:181`: accesses `target._api_key`
where `target` is typed as `PromptTarget` (base class) but the
test knows it's actually an `OpenAIChatTarget`.
8. Converted the `ty-check` pre-commit hook from the third-party
`allganize/ty-pre-commit` repo to a local hook that runs
`uv run --link-mode=copy ty check` against the project venv.
The third-party hook installs `ty` in an isolated env without
the project's runtime deps, so it surfaced spurious
`unresolved-import` errors for declared deps like `tqdm` once
the rule was flipped to `error`. This was harmless when the
rule was `ignore` but is now a real blocker.
Tests: 8382 passed, 120 skipped (matches pre-change baseline).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contract tests should mirror real consumer usage (matches doc/code/setup/2_resiliency.py). Use the documented short path and silence ty's unresolved-import (sys.modules alias) inline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use the documented public import paths (pyrit.scenario.{airt,foundry,garak}) with inline ty ignores, matching the pattern now used in test_foundry_scenario_contract.py and the doc snippets.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Where a symbol is re-exported from the scenario subpackage __init__ (foundry, garak, airt, benchmark), use the documented public short path (e.g. `pyrit.scenario.foundry.RedTeamAgent`). Note: deep submodule paths like `pyrit.scenario.garak.encoding.X` are NOT safe to shorten. The sys.modules alias only covers the top-level subpackage; importing a submodule via the alias re-executes the module under a different name and produces a duplicate class object, breaking isinstance checks. Those keep the on-disk path `pyrit.scenario.scenarios.garak.encoding.X`. Inline `# type: ignore[ty:unresolved-import]` per existing codebase convention because the alias is set up at runtime in `pyrit/scenario/__init__.py` and ty cannot follow that statically. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Mirrors the import edits made in the paired .py files to keep notebooks in sync (see .github/instructions/docs.instructions.md). Surgical string replacements only on the import lines that changed; preserves cell outputs and notebook metadata. Skipped: doc/code/registry/1_class_registry.ipynb:69 (an execution output showing the real fully-qualified class repr) and doc/scanner/garak.ipynb:47 (deep submodule path that the paired .py legitimately keeps as pyrit.scenario.scenarios.garak.encoding). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replace the per-file override block with surgical inline `# type: ignore[ty:unresolved-import]` comments at the 18 optional-extra import sites (azure-cognitiveservices-speech, cv2, playwright, torch, spacy). Test files keep the broader override since they tend to monkey-patch and re-import these modules in ways that would multiply the ignore sites. Rationale: the per-file override was opt-out at the file level, which is broader than needed — it silenced unresolved-import for every line in the file, including unrelated imports. Inline ignores document the intent at the point of need, match the dominant codebase convention (169 existing `# type: ignore[ty:<rule>]` uses), and self-clean: once a developer installs the optional extra locally, the `unused-type-ignore-comment = warn` rule will flag the stale ignore for removal. Verified: same ty error count before/after (1359 baseline diagnostics, same 7 pre-existing diagnostics in the 10 target files — none introduced by this change). All 106 functional unit tests in the affected modules pass (107 require optional extras and are skipped in base venv). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Replaced the 9-file [[tool.ty.overrides]] block that silenced unresolved-import on the optional-extra test modules with 20 surgical inline `# type: ignore[ty:unresolved-import]` comments at the actual import sites. Also added inline ignores for 13 pre-existing baseline diagnostics in the same files (deliberate bad-input tests, monkeypatched-mock assignments, transformers lazy-attr imports, tokenizer-Union narrowing) so removing the override doesn't surface latent noise. Net effect: `ty check tests/...` is now clean on those 9 files and the override block (other than the HuggingFace warn-only rule) is gone. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolved conflicts:
- .pre-commit-config.yaml: kept local ty hook (PR replaces 3rd-party hook)
- doc/code/scenarios/2_custom_scenario_parameters.{py,ipynb}: kept short scenario import, added new initialize_pyrit_async + ScenarioTechniqueInitializer setup from main
- doc/scanner/benchmark.{py,ipynb}: kept short scenario import, added DatasetConfiguration import from main
- tests/partner_integration/azure_ai_evaluation/test_scorer_contract.py: kept ScorerEvaluationIdentifier import (matches usage)
- tests/unit/auxiliary_attacks/gcg/test_generator.py: accepted main's deletion of dead helper functions
- tests/unit/prompt_target/target/test_huggingface_chat_target.py: adopted load_model_and_tokenizer_async rename + kept ty type-ignore
- uv.lock: regenerated via 'uv lock' against merged pyproject.toml
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…it-unused-dependencies
…it-unused-dependencies
…it-unused-dependencies
…it-unused-dependencies
- scenario.py: pass ScenarioRunState.CREATED enum (Pydantic ScenarioResult is strict) - add_image_to_video_converter.py: use cv2.VideoWriter.fourcc (modern API, properly stubbed) - hugging_face_chat_target.py: type:ignore[ty:possibly-missing-import] for AutoModelForCausalLM/AutoTokenizer; type:ignore[ty:unresolved-attribute] for tokenizer.decode - test_scorer_contract.py: import from pyrit.models.identifiers instead of deprecated pyrit.identifiers shim (test_no_internal_callers_of_deprecated_pyrit_identifiers_path) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Resolved conflicts in 7 files where my PR's # type: ignore[ty:unresolved-import] annotations clashed with PR microsoft#1939's noqa audit (which stripped the noqa parts). Took main's cleaner version since CI's ty config reports the type-ignore as unused anyway. Fix: pyrit/models/results/strategy_result.py:12 — add type:ignore for typing.Self import (Python 3.10 doesn't expose typing.Self; the import is TYPE_CHECKING-guarded but ty's stricter unresolved-import=error config still flags it). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- strategy_result.py: use typing_extensions.Self instead of typing.Self (works on 3.10+ without a type:ignore; typing_extensions is a transitive dep already used in 40+ pyrit files) - test_add_image_video_converter.py: VideoWriter_fourcc -> VideoWriter.fourcc (matches the production fix); drop unused type:ignore[ty:unresolved-import] on lazy cv2 imports - style-guide.instructions.md: document typing_extensions convention for Self/override/TypeAlias/etc. that aren't on every supported Python Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Walk each canonical package with pkgutil.walk_packages and register every submodule under the short name too, so the second import returns the same module object. Verified module identity for deep nesting (e.g. pyrit.scenario.adaptive.selectors.epsilon_greedy). Reverts the docs in doc/scanner/garak.{py,ipynb} back to the intended short path.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Picks up 3 commits from main: PR microsoft#1883 (keyword-only init enforcement), microsoft#1934 (dataset metadata), microsoft#1939 (noqa audit), plus microsoft#1900 (content-filter constants) and microsoft#1947 (notebook fixes). Resolved 6 conflicts in lazy-import noqa/type:ignore annotations by taking main's cleaner version (the # type: ignore[ty:unresolved-import] entries are unused warnings once --extra all is installed). Re-doing a previous merge attempt that aborted without setting MERGE_HEAD, so the prior merge commit was a single-parent commit that silently dropped origin/main's recent changes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
hannahwestra25
approved these changes
Jun 5, 2026
Hannah called out that 'import importlib' and 'import pkgutil' were inside the helper function — moved them to the top-level imports block per the style guide. Also added the missing 'canonical_module: ModuleType' annotation that pre-commit ruff (ANN001) flagged in CI. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rlundeen2
added a commit
to rlundeen2/PyRIT
that referenced
this pull request
Jun 5, 2026
- Apply ruff UP007/UP045 (Optional/Union -> X | None) to the relocated serializers.py and storage.py to match the repo-wide modernization (origin/main microsoft#1884) - Add ty pragma for azure readall() str|bytes stub mismatch surfaced by the post-merge ty hardening (microsoft#1931) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
rlundeen2
added a commit
to rlundeen2/PyRIT
that referenced
this pull request
Jun 5, 2026
These 6 type errors are pre-existing on main (third-party stub and SQLAlchemy Base typing mismatches from the recent dep audit, microsoft#1931). They were ungated on main because that PR did not touch these files; relocating their imports to pyrit.memory newly subjects them to the per-file ty gate. Add minimal pragmas consistent with the existing codebase convention: - sqlite_memory/azure_sql_memory: session.get(type(entry), entry.id) - sqlite_memory: MemoryExporter.export_data(list(data), ...) - add_image_text_converter: ImageFont.truetype(str | None, ...) - openai_tts_target: AsyncSpeech.create response_format/speed - azure_content_filter_scorer: ContentSafetyClient credential Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
A maintenance sweep that started as a dependency audit and grew into three loosely related cleanups.
1. Dropped 10 unused dev dependencies. Catalogued every entry in
pyproject.tomlagainst actual usage acrosspyrit/,tests/,doc/, andbuild_scripts/. Verified imports with grep (the prior audit had a gap onconfusables, which turned out to be in use). Removed the ten that were genuinely orphaned.2. Flipped ty's
unresolved-importrule fromwarntoerrorand fixed the fallout. Previously a broad[[tool.ty.overrides]]block silenced the rule across 19 files (10pyrit/source + 9 test modules) so any genuinely broken import would pass unnoticed. Replaced it with 38 surgical inline# type: ignore[ty:unresolved-import]comments at the actual import sites (18 inpyrit/, 20 intests/), each pinned to one line where an optional extra (azure-speech, cv2, playwright, torch, spacy, torch.multiprocessing) is imported behind atry/except ImportErrororTYPE_CHECKINGblock. Also addressed 13 pre-existing baseline diagnostics in the affected test files so the override could come out without leaving silent breakage. Only the HuggingFaceinvalid-argument-type = "warn"override remains.3. Shortened scenario imports. The codebase exposes short public paths like
pyrit.scenario.foundry.RedTeamAgentviasys.modulesaliases inpyrit/scenario/__init__.py. Several tests and the partner-integration contract test had drifted to the long internal path. Restored the short paths where the public API exposes them; kept the long paths where they're necessary (submodule imports that bypass the alias and would breakisinstancechecks via duplicate class objects).Tests and Documentation
uv run --link-mode=copy ty check pyrit/andty check tests/<the 9 files>both clean (was 33 diagnostics on the test files before the inline ignores landed).uv run --link-mode=copy pytest tests/unit/...on the affected modules: 106 pass, 108 skip (skips waiting for optional extras not in the base dev venv).jupytext --syncon the 6 modifieddoc/code/scenarios/*.pyanddoc/scanner/*.pyfiles so their paired.ipynbcompanions match.Notes for reviewers
# type: ignore[ty:unresolved-import] # noqa: F811comments inpyrit/auth/azure_auth.pylook busy but each suppression is doing different work:F811(pre-existing) silences ruff's redefinition warning whereTYPE_CHECKINGshadows a runtime import;ty:unresolved-import(added here) silences ty whenazure-cognitiveservices-speechis not installed.romanlutz/audit-unused-dependenciesundersells the final scope: the audit was the trigger but most of the diff is the ty-hardening and scenario-import cleanup.Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com