[Python] Honor disableCounterMetrics, disableStringSetMetrics and disableBoundedTrieMetrics experiments by anishmehta24 · Pull Request #40165 · apache/beam

anishmehta24 · 2026-09-17T22:06:48Z

Re-lands #38749 (reverted in #38901) so the Python SDK honors the disableCounterMetrics, disableStringSetMetrics and disableBoundedTrieMetrics experiments the way the Java SDK already does, letting high-throughput jobs stop reporting metric kinds that put pressure on the metrics backend.

Why the first attempt broke, and what is different here. #38749 replaced DelegatingCounter.inc (and the adds on DelegatingStringSet / DelegatingBoundedTrie) — which are MetricUpdater instance attributes with signature __call__(value=_DEFAULT) — with plain methods inc(self, n=1). That broke callers using counter.inc(value=...) (thread) and anything relying on those attributes being MetricUpdater instances (pickling via MetricUpdater.__reduce__, isinstance checks). This PR leaves the metric objects untouched:

MetricUpdater.__call__ (metrics/execution.py) drops the update when its cell type is in a module-level _DISABLED_CELL_TYPES set. With no experiment set this is a truthiness check on an empty set, so the hot path is unchanged; with one set it is a single set lookup. Declared in execution.pxd for the Cython build.
MetricsFlag (metrics/metric.py) mirrors the Java Metrics.MetricsFlag: set_default_pipeline_options(options) maps the three experiments onto CounterCell / StringSetCell / BoundedTrieCell, logs which kinds are disabled, and — as in Java — the first call wins so user code on a worker cannot change what the harness started with. reset() exists for tests.
Hooked from Pipeline.__init__ (so the DirectRunner honors it) and sdk_worker_main.create_harness (portable workers), next to the existing FileSystems.set_options calls, same as [Python] Honor disableCounterMetrics, disableStringSetMetrics, and disableBoundedTrieMetrics experiments #38749.

Because the gate is evaluated per call rather than captured at construction, it also covers metric objects created before the options were known (module-level counters, DoFns constructed before Pipeline(...)) and objects unpickled on the worker.

Tests (metric_test.py::MetricsFlagTest, 9 cases): experiments → flags for each kind and combined; first-call-wins; a regression test that inc(), inc(4), inc(value=5), dec(), add(value=...) all still work and counter.inc is still a MetricUpdater; each disabled kind is a no-op for objects created before and after the flag while other kinds keep reporting; process-wide counters; unpickled metrics; and an end-to-end TestPipeline with --experiments=disableCounterMetrics asserting the counter is absent from results while the distribution is intact. Without the implementation the suite fails.

Run locally on Windows (pure-Python build): pytest apache_beam/metrics/ pipeline_test.py runners/worker/sdk_worker_main_test.py runners/worker/statesampler_test.py → 109 passed (one pre-existing pipeline_test failure from a dill version mismatch in my env), ruff check, yapf, pylint clean on touched files, mypy clean for metrics/execution.py and metrics/metric.py. Prepared with AI assistance (Claude Code) and human-verified.

fixes #38746

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

🤖 Generated with Claude Code

github-actions · 2026-09-17T23:12:07Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

anishmehta24 · 2026-09-18T02:05:49Z

Run Yaml_Xlang_Direct PreCommit

github-actions · 2026-09-18T03:18:06Z

Assigning reviewers:

R: @shunping for label python.

This pull request likely touches a core component ("core" label). Please review with scrutiny.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

anishmehta24 · 2026-09-18T20:18:48Z

The five Python Unit Tests (macos-latest, …) failures on the re-run are an upstream packaging issue, not this change: deltalake 1.6.4 was published to PyPI at 20:12 UTC today with wheels for macOS x86_64, Linux and Windows but no macosx_*_arm64 wheel, so pip install on the arm64 macOS runners falls back to building the sdist and the Rust link step fails (error: linking with 'cc' failed / could not compile 'deltalake-python'). The Linux and Windows matrices are green, and master's last Python-tests run (17:45 UTC, before the release) passed all macOS jobs. It should clear as soon as the arm64 wheel lands, or if setup.py pins deltalake below 1.6.4 in the meantime.

…ableBoundedTrieMetrics experiments Re-lands the feature from apache#38749 (reverted in apache#38901) without changing the public surface of the metric objects. The Java SDK lets high throughput jobs turn off metric kinds that pressure the metrics backend via these experiments; the Python SDK now does the same. Instead of replacing DelegatingCounter.inc / DelegatingStringSet.add / DelegatingBoundedTrie.add with methods (which broke callers passing the value as a keyword and code relying on them being MetricUpdater instances), the gate lives in MetricUpdater.__call__ and is keyed by cell type. With no experiment set it is a truthiness check on an empty set, so the hot path is unchanged. MetricsFlag.set_default_pipeline_options mirrors the Java MetricsFlag and is applied when a Pipeline is constructed and at SDK worker harness start-up, first call wins as in Java. Fixes apache#38746 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The process-wide metrics container is shared with every other test in the pytest-xdist worker process, so asserting it is empty fails whenever another test (the GCP suites in the cloud tox env) registered a process-wide counter first.

anishmehta24 · 2026-09-20T10:58:02Z

The Python PreCommit 3.14 failure after the rebase was in this PR's own test: test_disabled_process_wide_counter_is_noop asserted the whole process-wide container is empty, which does not hold under xdist once another test in the same worker (the GCP suites in the cloud env) has registered a process-wide counter. c467401 makes it check only that this test's key never reached the container. Rebased on master for the CHANGES.md conflict as well; no other changes.

anishmehta24 · 2026-09-24T12:14:19Z

@shunping is there anything else you'd like changed here? The earlier macOS failures were the deltalake wheel issue.

shunping · 2026-09-24T12:22:01Z

Adding @Abacn since he made the last revert.

Abacn · 2026-09-25T18:05:10Z

Let me run an internal import

Abacn · 2026-09-25T19:01:02Z

 _LOGGER = logging.getLogger(__name__)


+class MetricsFlag(object):


Besides the method signature change, I remember another breakage introduced in the first attempt was a circular import. For precaution can we move the new logic into a new metrics_flag.py with minimum required import, to avoid potential issues in callers?

Abacn · 2026-09-25T19:02:12Z

+      if debug_options.lookup_experiment(experiment):
+        disabled.add(cell_type)
+        _LOGGER.info('%s metrics are disabled.', kind)
+    execution.set_disabled_cell_types(disabled)


if we put metrics_flag in a separate module, we can query disabled cells from execution.py and no need to import execution here

Abacn · 2026-09-25T19:26:23Z

+    """
+    if cls._initialized:
+      return
+    debug_options = options.view_as(DebugOptions)


consider import debug options in place. Currently no non-test file in apache_beam/metrics/ imported apache_beam.options at runtime, as a pre-caution for circular import risk

github-actions Bot added python runners core labels Sep 17, 2026

github-actions Bot added the Next Action: Reviewers label Sep 18, 2026

anishmehta24 force-pushed the fix/python-disable-user-metrics-experiments branch from 43f69f3 to 050fb02 Compare September 19, 2026 23:33

Abacn reviewed Sep 25, 2026

View reviewed changes

Navigation Menu

[Python] Honor disableCounterMetrics, disableStringSetMetrics and disableBoundedTrieMetrics experiments - #40165

[Python] Honor disableCounterMetrics, disableStringSetMetrics and disableBoundedTrieMetrics experiments#40165
anishmehta24 wants to merge 2 commits into
apache:masterfrom
anishmehta24:fix/python-disable-user-metrics-experiments

anishmehta24 commented Sep 17, 2026

Uh oh!

github-actions Bot commented Sep 17, 2026

Uh oh!

anishmehta24 commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

anishmehta24 commented Sep 18, 2026 •

edited

Loading

Uh oh!

anishmehta24 commented Sep 20, 2026

Uh oh!

anishmehta24 commented Sep 24, 2026

Uh oh!

shunping commented Sep 24, 2026

Uh oh!

Abacn commented Sep 25, 2026

Uh oh!

Abacn Sep 25, 2026

Uh oh!

Abacn Sep 25, 2026

Uh oh!

Abacn Sep 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		_LOGGER = logging.getLogger(__name__)


		class MetricsFlag(object):

Conversation

anishmehta24 commented Sep 17, 2026

Uh oh!

github-actions Bot commented Sep 17, 2026

Uh oh!

anishmehta24 commented Sep 18, 2026

Uh oh!

github-actions Bot commented Sep 18, 2026

Uh oh!

anishmehta24 commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anishmehta24 commented Sep 20, 2026

Uh oh!

anishmehta24 commented Sep 24, 2026

Uh oh!

shunping commented Sep 24, 2026

Uh oh!

Abacn commented Sep 25, 2026

Uh oh!

Abacn Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

Abacn Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

Abacn Sep 25, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

anishmehta24 commented Sep 18, 2026 •

edited

Loading