Reduce GitHub Actions usage (DataFusion is currently the top ASF consumer of GHA minutes)

[alamb]

Is your feature request related to a problem or challenge?

Recently we have been seeing the Merge Queue get stuck:

• CI takes hours to start #25088

We enquired with ASF Infrastructure on INFRA-28374 and they told us the issue is that the ASF's GitHub Action queue is full which is why it is taking a long time to start jobs:

We are aware of a long term usage pattern whereby our limited number of
concurrent GitHub Actions jobs running on GitHub hosted runners is at 100%
utilisation most of the day most of the time. Thus, from time to time we
communicate to projects who utilise the most minutes to review their
workflows for efficiency improvements.

At the moment, datafusion is the top consumer of GitHub Action minutes over
the last 7 days, and 3x the 10th in the list of minutes used.

Note the "datafusion" project as measured by ASF Infra includes all DataFusion subprojects (datafusion, datafusion-comet, datafusion-ballista, datafusion-python, etc.), so reducing usage may require a cross-repository effort.

Here is the report of our current usage (login required):

History

This is the second time we have been contacted by ASF Infra about GHA usage.

The first was in May 2026, when ASF Infra identified DataFusion as a top-5 consumer: #22455. After some tweaks, by June 2026, our usage dropped below the policy thresholds and DataFusion fell out of the top 5.

Useful resources

• ASF GitHub Actions policy (limits: 250,000 minutes / calendar week, 216,000 minutes / any 5-day window): https://infra.apache.org/github-actions-policy.html
• ASF usage report for the datafusion project (login required): https://infra-reports.apache.org/#ghactions&project=datafusion&hours=168&limit=15&group=name
• ASF Infra best-practices wiki page: https://cwiki.apache.org/confluence/spaces/INFRA/pages/430408443/GitHub+Actions+Recommended+Practices
• ASF Slack channel for sharing workflow optimisations across projects: #project-workflow-optimisations

Related issues

• Reduce Github Action Usage #22455 (previous round of this same problem)
• [EPIC] A collection of items to improve developer / CI speed  #13813
• Add Local Scripts to Reproduce Full CI #21048
• CI takes hours to start #25088 (CI takes hours to start — likely a symptom of pool saturation)
• Move Extended tests to the standard ones? #21241 (discussion of when extended tests should run)
• CI: count-distinct byte tests run for hours and hit the 360-minute limit #25011 (example of a runaway CI job: extended tests hitting the 360 minute limit)

[alamb]

CI time breakdown by workflow

Breaking down the report screenshot above, DataFusion is as

Total usage: 336,347 minutes, or 33 full-time runners — over the ASF policy ceiling of 250,000 minutes / 25 full-time runners per calendar week.

Three workflows exceed 5% of total time, and together account for ~87% of all usage:

1. CI — 49.31% (~165,900 min)

• Repo: apache/datafusion-comet
• Source: https://github.com/apache/datafusion-comet/blob/main/.github/workflows/ci.yml
• Comet's top-level CI orchestrator

2. Rust — 27.16% (~91,350 min)

• Repo: apache/datafusion
• Source: https://github.com/apache/datafusion/blob/main/.github/workflows/rust.yml
• The main DataFusion core test suite that runs on every PR push, again in the merge queue, and again on push to main.

3. Datafusion extended tests — 10.85% (~36,500 min)

• Repo: apache/datafusion
• Source: https://github.com/apache/datafusion/blob/main/.github/workflows/extended.yml
• The long-running correctness suite (extended tests, hash-collision tests, sqlite sqllogictests) that runs on every push to main/branch-* and on every push to any PR touching datafusion/physical*, expr*, optimizer, or sql — a large fraction of all PRs.

Everything else (< 2.5% each)

Workflow	Share
Dev	2.38%
Dependencies	2.01%
Ballista Rust	1.73%
Delta Contrib Build Gate	1.47%
Detect breaking changes	0.96%
TPC-H SF10	0.88%
CodeQL	0.76%
TPC-DS SF1	0.47%
PyArrow UDF Tests	0.42%
Docker	0.29%
h2o	0.24%
Security audit	0.22%
(other builds)	0.86%

Observations

• Unlike the May 2026 (#22455) where Comet was ~78% of usage, the core datafusion repo now accounts for ~40%+ of usage itself
• The "Datafusion extended tests" slice (10.85%) is exactly the one the merge-queue gating idea in Move Extended tests to the standard ones? #21241 would help

[alamb]

FYI @blaginin

[alamb]

Among other things the fact we have a merge queue and the jobs timeout likely exacerbates the problem (as each PR has to get put in the queue a few times rerunning CI each time)

[alamb]

I also filed a ticket in Comet

• Reduce CI time (Comet is consuming 50% of DataFusion CI, which is a lot) datafusion-comet#5830

[comphead]

Thanks @alamb for tracking this. Comet got lots of contributions last month and that explains

[comphead]

This ticket may also help DF CI #25092, very frequently the PR removed from MQ because of flaky MinIO tests

[rluvaton]

Is it possible for Apple to chime in for datafusion and datafusion-comet a little?
Are we getting GitHub Actions minutes from GitHub or from Apache? because if this is from Apache, we might be better change to use Hetzner servers (which is what we are doing in my company)

[comphead]

Is it possible for Apple to chime in for datafusion and datafusion-comet a little? Are we getting GitHub Actions minutes from GitHub or from Apache? because if this is from Apache, we might be better change to use Hetzner servers (which is what we are doing in my company)

Organizational part is bumpy atm, we already tried that. Minutes come from GH actions that has been run using ASF shared resources.

[alamb]

Is it possible for Apple to chime in for datafusion and datafusion-comet a little? Are we getting GitHub Actions minutes from GitHub or from Apache? because if this is from Apache, we might be better change to use Hetzner servers (which is what we are doing in my company)

I believe the runners are provided by Github as an in-kind donation to the ASF

We could try and crowdsource additional runner minutes (the way arrow did with ursa computing, etc) but then someone has to manage / pay / coordinate for those resources

I think we shoudl try and reduce runner usage first, and then if we are still having problems figure out a way to get more resources

[adriangb]

Could we have someone (e.g. Apple, probably needs to be someone w/ a big AWS account) run runs-on runners in their AWS account and we coordinate GitHub App installation / auth w/ ASF Infra? Then the AWS account (and it's bill) would be long to whoever is funding it and there would be no need to coordinate payment (which I understand can be the hard part).

[comphead]

Another option which works for Apache Spark and tested in Comet as well is running CI on the contributor forks apache/datafusion-comet#4570

It changes the model and also contributor machines are 20-30% slower than ASF but the benefit is ASF resource pool is being used much more considerate. This might be later option, but taking into account DF ecosystem is growing and take more resources we might think of it