Conversation
Collaborator
|
I did this too, actually. But an easier way to get the benchmarks running is to push your branch, then run ASV benchmarks directly in the Actions tab. That way you don't have to mess with a PR to get them running. |
ASV BenchmarkingBenchmark Comparison ResultsBenchmarks that have improved:
Benchmarks that have stayed the same:
Benchmarks that have got worse:
|
Collaborator
Author
Ah, that makes sense! Does that lead to a nice comment with the benchmark results too? I suppose even if there isn't a comment it's definitely less noisy than making a draft PR |
7 tasks done
Collaborator
|
Nah, you'd have to dig. I've been mostly looking at total time though, so for me it doesn't matter. |
Member
|
Leaving a note here in case it gets solved and disappears in the above benchmark run, we have been seeing a *1000 or so speedup on a test even when nothing changes, it appears as "no change". Very odd: 
This appears in this PR and others as well (at the time of writing). |
Collaborator
Author
|
@cmdupuis3 I've been noticing that the "import uxarray" benchmark is always showing an improvement, and the EDIT: Ah, I agree now I was wrong about the latter… I felt like I've been seeing NeighborhoodDask benchmarks in that category in a few different PRs, but now I can see that it isn't actually always showing up as "got worse", only sometimes! |
Collaborator
|
@Sevans711 If it's around 300M on a memory benchmark, that usually indicates caching behavior. I think the pure import is pretty indicative. It can be hard to measure consistently under caching conditions, so for the other benchmarks, maybe they aren't rigorous enough. The other one seems like a one-off, you can just compare it with previous benchmark runs and if there's no pattern, I wouldn't worry about it much. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not intended to be merged to main, just trying to check how ASV benchmarks look currently when the code is unchanged.
There's probably a better solution than having a draft PR for this, but this is the simplest/fastest way I could think of right now. I already attempted but struggled to generate ASV benchmark runs on my local machine with the same output format as what the run-benchmark label produces via github CI.
I originally created this with the intention of determining whether the reported benchmark performance degradation in #1705 is actually caused by the code changes there or not. Maybe this should be closed soon after that check, or maybe it should be open for a longer-term basis to repeat checks like this in the future? For example, it might have been useful to have something like this available for initially discovering #1605.