Project: S3779 LZW ColGroup | ASML - #2549
Open
MasterBrain2000 wants to merge 100 commits into
Open
Conversation
…aders Summary This PR introduces new functionality for multimodal learning in Scuro, including a contrastive learning operator, a modality alignment operator, and additional data loaders. Changes Contrastive Learning Operator - Constructs modality pairs via a Cartesian product - Uses a user-defined function to label pairs as positive or negative - Enables dynamic generation of contrastive samples Modality Alignment Operator - Aligns previously unaligned modalities using feature-based similarity (e.g., ORB, perceptual hashing) - Outputs a matching between a primary and secondary modality - Matching is applied after representation learning and before fusion Data Loaders - PDF loader: converts document pages into NumPy arrays for OpenCV processing - Audio loader: converts audio to text using faster-whisper Closes apache#2461
This patch improves the existing HP Tuner for the new unimodal optimizer.
…ed federated partitions This commit fixes the federated reverse instruction to support federation maps with differently sized partitions and adds the corresponding test cases.
This adds a new HOP rewrite rule, RewriteMatrixMultChainWithTransOptimization.java, to find the optimal execution plan for matrix multiplication chains containing transposes. Previously, these chains were optimized using a simple heuristic that just pushes transposes down from t(A %*% B) -> t(B) %*% t(A), which fails to be the optimal plan in some instances especially with large matrices. An example would be R = t(A %*% B) %*% C with dimensions A = [16, 23], B = [23, 22], C = [16, 34] which would be according to the old rewrite class solved with (t(B) %*% t(A)) %*% C -> costs: t(B) -> 23*22 + t(A) -> 16 * 23 + t(B) %*% t(A) -> 22*23*16 + [...] %*% C -> 22*16*34 = 20938 FLOPs Optimal would be simply: t(A %*% B) %*% C - costs: A %*% B -> 16*23*22 + t(A %*% B) -> 16*22 + [...] %*% C -> 22*16*34 = 20416 FLOPs - difference gets larger with higher matrix dimensions. To solve this, we applied a DP Algorithm with a Memo Table containing Plans without transposing and Plans containing Transposing subchains calculating wether an algebraic transpose pushdown or direct transpose operation is cheaper. This also includes 24 automated DML test cases asserting intermediate HOP dimensions to validate optimal parenthesization and transpose placement. = 20938 FLOPs Optimal would be simply: t(A %*% B) %*% C - costs: A %*% B -> 16*23*22 + t(A %*% B) -> 16*22 + [...] %*% C -> 22*16*34 = 20416 FLOPs - difference gets larger with higher matrix dimensions. To solve this, we applied a DP Algorithm with a Memo Table containing Plans without transposing and Plans containing Transposing subchains calculating wether an algebraic transpose pushdown or direct transpose operation is cheaper. This also includes 24 automated DML test cases asserting intermediate HOP dimensions to validate optimal parenthesization and transpose placement. Closes apache#2465.
* Fixes the remaining invalid method names of the new and existing mmchain-opt rewrites * Fixes the handling of stdout streams with default output buffering (which is on in the github actions)
Closes apache#2464.
…rtup Replace fixed Thread.sleep after each federated worker start with TCP port polling that returns as soon as the worker accepts a connection. Add bulk helpers that spawn N workers in parallel and wait once for the slowest to become ready, instead of summing per-worker waits. Cuts the federated CI total by ~7 min (-5%) vs main, with the biggest wins in setup-heavy suites such as transform+fedplanner (-66%) and codegen (-25%). Closes apache#2468.
This commit adds a timeout to surefire to allow the tests to timeout, with a log message and stack trace rather than allowing github Actions to kill the CI job. When Actions kill the job, we do not know which tests are running or blocking, while if we kill surefire we will.
…rations The wrong variable was used, h1 instead of h2.
This commit implements the row-wise sparsity estimator and adds respective test cases. Closes apache#2466.
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 6.0.0 to 6.0.1. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v6.0.0...v6.0.1) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Wire startLocalFedMonitoring through FederatedWorkerUtils.waitForWorker so the monitoring backend's port-bind is polled instead of slept on (fixes flaky FederatedCoordinatorIntegrationCRUDTest), migrate FederatedLogicalTest to the bulk startLocalFedWorkers(int[]) API, and drop the now-unused FED_WORKER_WAIT_S and FED_MONITOR_WAIT constants.
…ts (apache#2472) FedWorkerReadMatrixCompress.verifyRead failed roughly once per ten component-test CI runs because it called FederatedTestUtils.wait(1000) to give the worker time to finish its async compression (kicked off by CompressedMatrixBlockFactory.compressAsync), then asserted that the returned block was a CompressedMatrixBlock. On a contended runner the 1 s sleep was not enough, the subsequent read returned the still- uncompressed block, and the assertion failed. Surefire's rerunFailingTestsCount=2 hid this as a "Flake" rather than a job failure. Add FedWorkerBase.awaitCompressed(long id), which polls getMatrixBlock at 25 ms intervals for up to COMPRESS_TIMEOUT_MS (10 s) and returns as soon as the worker reports the compressed form, or returns the last- observed block on timeout so the caller's assertion still produces a meaningful failure. Convert the three call sites that used the fixed-sleep anti-pattern: - FedWorkerReadMatrixCompress.verifyRead (the actual CI flake) - FedWorkerMatrixCompress.verifySameOrAlsoCompressedAsLocalCompress (polls only when local compresses, so the "do not compress" parametrization stays fast) - FedWorkerMatrixMultiplyWorkload.verifySameOrAlsoCompressedAsLocalCompress Remove the now-unused FederatedTestUtils.wait helper so the anti-pattern is harder to reintroduce.
Fix intermittent ~26 minute hangs in the **.component.c**.** GitHub Actions job. The forks were finishing their test classes but failing to exit cleanly (leaked non-daemon threads from test executors), and Surefire 3.0.0 did not reliably enforce its fork shutdown timeout.
In this patch the video modality is included in the unimodal optimizer pipeline. Since the video modality is sometimes loaded using the chunked data loader support for the chunked representation is added to the optimizer.
This patch includes the optuna hyperparameter tuning library in the Scuro hyperparameter tuner. It adds support for wandb to observe the progress and parameter configurations.
In this patch we improve the runtime efficiency of the window aggregation operator in Scuro. The problem with the latest approach was the iteration over each window instead of vecortized execution. This change neede a couple of adaptions in subsequent representations.
Bumps [codecov/codecov-action](https://github.com/codecov/codecov-action) from 6.0.1 to 7.0.0. - [Release notes](https://github.com/codecov/codecov-action/releases) - [Changelog](https://github.com/codecov/codecov-action/blob/main/CHANGELOG.md) - [Commits](codecov/codecov-action@v6.0.1...v7.0.0) --- updated-dependencies: - dependency-name: codecov/codecov-action dependency-version: 7.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Based on recommended best practices at Apache: https://cwiki.apache.org/confluence/pages/viewpage.action?spaceKey=INFRA&title=GitHub+Actions+Recommended+Practices This commit change the CI jobs to fail-fast based on subsequent comments.
…che#2485) The per-fork timeout existed to kill a hung test fork before the 30 min CI cap, so the offending test is named in the log instead of GitHub cancelling the whole job. At 1380s the timeout fired ~23 min after a fork started, but earlier forks already consumed ~8 min of the run, so the timeout would only trigger after the 30 min cap had cancelled the job. As a result hung forks were never reported. Lowering to 600s keeps a large margin below the cap regardless of when in the run a fork hangs, while staying well above the slowest observed test class (~35s), so legitimate tests are not killed.
… errors CI jobs intermittently fail at startup when Maven Central returns a transient error (e.g. HTTP 403 while resolving the Apache parent POM), which is unrelated to the code under test. Retry test-compile once after a short pause when the failure is a "Could not transfer artifact" download error, while genuine compilation and test failures still fail fast. Also point the test action at the entrypoint script in the mounted workspace so changes to docker/entrypoint.sh take effect immediately without rebuilding and republishing apache/systemds:testing-latest. Failing job that motivated this change: https://github.com/apache/systemds/actions/runs/27167144065/job/80197161169
…2487) CholeskyTest reconstructs A from its Cholesky factor and asserts that the 1x1 residual D = sum(A-B) is approximately zero. The output was read back with dmlOut.keySet().iterator().next(), which assumes at least one cell is present. When the residual is exactly 0.0, the sparse text writer omits the cell entirely, so the result map comes back empty and the iterator throws NoSuchElementException. A perfect reconstruction therefore caused the test to error out instead of pass. This is not data-dependent flakiness: the input matrix is already seeded, so A is identical on every run. The variability comes from the reduction order of sum(A-B), which differs across Spark partitions and CP threads. Because floating-point addition is not associative, the residual lands on either an exact 0.0 (empty output) or a tiny non-zero value depending on execution, which is why only some runs (notably testLargeCholeskyDenseSP) failed. The fix treats an empty output as 0.0, making the assertion robust to both outcomes, and drops the now-unused MatrixValue import.
* Add HashMapIntToInt primitive int-to-int hash map Introduce a specialized open-addressing map storing primitive int keys and values, avoiding boxing for integer-to-integer lookups used by compressed column-group operations. * Add component tests for HashMapIntToInt and HashMapLongInt Cover put/get, putIfAbsent variants, collision chains, resize, iteration, and the primitive -1 absent-sentinel contract. * Fix HashMapIntToInt iterator skipping buckets and add randomized tests The entry iterator advanced bucketId twice per empty bucket (pre-increment in the loop condition plus the post-increment step), skipping entries when buckets were unevenly filled. Dense sequential keys masked it because the first probe always hit a non-null bucket. Add randomized tests that cross-check against java.util.HashMap to cover imbalanced bucket layouts, plus a chained-collision resize test, reaching full branch coverage.
…apache#2480) * Speed up frame-to-matrix conversion and harden number parsing Tightens the hot path that converts FrameBlocks of arbitrary schema into MatrixBlocks, with a defensive fallback for malformed cells. - DoubleParser.parseFloatingPointLiteral: replace the >= 'I' / >= 'a' character guards with a single 0-9 range check on the last char. The previous guards over-matched and pushed too many strings into the slow Double.parseDouble path - DoubleArray.parseDouble: stop wrapping the parse failure as a DMLRuntimeException so callers can distinguish format errors - MatrixBlockFromFrame: - turn the interface into a class with a private constructor so Jacoco can measure it cleanly - on NumberFormatException / DMLRuntimeException during a bulk block convert, log once and fall back to convertSafeCast which writes NaN per offending cell instead of failing the whole job - add convertSafeCast / convertBlockSafeCast helpers * Gate frame-to-matrix NaN cast fallback behind config flag Frame-to-matrix conversion silently fell back to writing NaN for cells that cannot be cast to double, logging only a single warning. This changed behavior for callers that previously failed fast on incompatible data. Make the lenient behavior opt-in. - Add sysds.frame.tomatrix.warncast (default false): when false the conversion fails fast on number format errors; when true it warns once and writes NaN for the incompatible cells - Read the flag once on the calling thread and pass it down, since the thread-local config is not visible to pool workers - Extract convertStrict to share the contiguous/generic dispatch between the strict and warn-only paths - Add tests for the warn-only fallback, the strict fail-fast default, and the tightened DoubleParser trailing-character guard * Add tests for frame-to-matrix warn-cast and double parse error path - Cover the warn-cast success path where a fully valid frame converts without triggering the NaN fallback - Cover the zero-value branch of the safe-cast non-zero count and an all-invalid frame becoming all NaN - Add parallel fail-fast coverage for the strict (default) path - Assert DoubleArray.parseDouble surfaces the raw NumberFormatException instead of a wrapped DMLRuntimeException
Contributor
|
Thanks for your contribution @MasterBrain2000 @Mancer1 @m-ollka. Could you please change your PR to use tabs instead of spaces (reconfigure your IDE) to make the diff more readable? |
Contributor
|
Also, please rebase and open the PR on the main branch |
…nd downstream CLA operations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
16 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Group Project for the AMLS Module