Expose ILU0 numeric boost configuration for CUDA and HIP - #466
Conversation
|
I’d appreciate any help with HIP testing |
tamar-dewilde
left a comment
tamar-dewilde
left a comment
There was a problem hiding this comment.
Tested on HIP. All 64 tests and install validation passed.
|
|
||
| LinSolverDirect& getFactorizationSolver(); | ||
| LinSolverDirect& getRefactorizationSolver(); | ||
| LinSolverDirect& getPreconditionerSolver(); |
There was a problem hiding this comment.
The PR description says this getter lets users configure the numeric boost through SystemSolver. How should users call setZeroDiagonal() through it? The getter returns LinSolverDirect&, which does not declare that method, so the direct call does not compile.
I could only call the setter by including LinSolverDirectCuSparseILU0.hpp and casting the returned solver to that concrete type. Is that the intended use, or should LinSolverDirect provide a common way to access this setting? LinSolverDirectCpuILU0 already has the same setter.
There was a problem hiding this comment.
Thank you for pointing this out! I added the BOOST_TOLERANCE and BOOST_VALUE parameter IDs and the corresponding cli param settings so users can now configure the numeric boost through SystemSolver rather than casting it to a concrete type.
I also implemented numeric boost support for HIP since it was needed for the configuration test added to testSysGmres.cpp.
Can you test this again on HIP?
kakeueda
changed the title
|
One remaining inconsistency is that CPU ILU0 still uses |
|
You showed what the iteration number change to, what did it change from? |
shakedregev
left a comment
shakedregev
left a comment
There was a problem hiding this comment.
Looks good, neeed to test.
|
@shakedregev With the existing hard-coded settings ( My Navier–Stokes FEM system also converges poorly with the default |
|
Oh, now I understand. |
|
Funnily my HIP test worked, but CUDA did not. |
|
|
||
| status_rocsparse_ = rocsparse_dcsrilu0_numeric_boost(workspace_->getRocsparseHandle(), | ||
| info_A_, | ||
| 1, |
There was a problem hiding this comment.
Was it intentional to enable numeric boost by default on HIP here? I set this flag to 0 and reran the full HIP suite on Frontier, and all 64 tests still passed.
There was a problem hiding this comment.
I enabled it for consistency with cuda but I agree that we can leave the hip settings as they are since we haven’t encountered the same issue there. Still the same issue could surface on hip as we expand testing to a broader range of systems.
A better approach would be to make numeric boost configurable on both backends (disabled by default on hip and enabled on cuda) so users can turn regularization on or off as needed.
|
@shakedregev Interesting, I couldn’t reproduce that error on my side (RTX 5050 Laptop GPU, CUDA Toolkit 12.9, driver 595.79, GCC 13). Could you try the same command on the develop branch and let me know whether it still occurs? Just a quick note Running the same command but without -t, I got |
How do you set the boost value then? |
|
I get the same error on develop. Let me know if you want me to try and fix this or if you want to try some other way. |
I forgot to mention this - I tested these values by temporarily adding Added command-line options (luckily no option name collisions yet)
ku@kakeru:~/dev/ReSolve$ ./build/examples/sysGmres.exe \
-m /home/ku/tmp/matrixmarket/fidap012.mtx \
-r /home/ku/tmp/matrixmarket/fidap012_rhs1.mtx \
-b cuda \
-i fgmres \
-n yes \
-t 1e-7 \
-v 1e-7
Matrix file: /home/ku/tmp/matrixmarket/fidap012.mtx
RHS file: /home/ku/tmp/matrixmarket/fidap012_rhs1.mtx
sysGmres with CUDA backend
solver.setMatrix returned status: 0
solver.preconditionerSetup returned status: 0
solver.solve returned status: 0
Initial relative residual norm ||b-A*x||/||b|| : 1.0000000000000000e+00
Final relative residual norm ||b-A*x||/||b|| : 9.2849901493455225e-13
Number of iterations : 143
ku@kakeru:~/dev/ReSolve$ ku@kakeru:~/dev/ReSolve$ ./build/examples/sysGmres.exe \
-m /home/ku/tmp/matrixmarket/fidap012.mtx \
-r /home/ku/tmp/matrixmarket/fidap012_rhs1.mtx \
-b cuda \
-i fgmres \
-n yes \
-t 1e-6 \
-v 1e-6
Matrix file: /home/ku/tmp/matrixmarket/fidap012.mtx
RHS file: /home/ku/tmp/matrixmarket/fidap012_rhs1.mtx
sysGmres with CUDA backend
solver.setMatrix returned status: 0
solver.preconditionerSetup returned status: 0
solver.solve returned status: 0
Initial relative residual norm ||b-A*x||/||b|| : 1.0000000000000000e+00
Final relative residual norm ||b-A*x||/||b|| : 9.9392435281287791e-13
Number of iterations : 1121 |
If you have time to look into it that would be really helpful, thank you! |
Thank you for looking into this. As an alternative test case I added a linear system from one of my Navier-Stokes FEM simulations to The system comes from an idealized numerical test simulating two-dimensional blood flow through a channel with an asymmetric stenosis. Solving this system gives a velocity field that looks like this Velocity field
It also shows similar sensitivity to the stabilization parameters. Could you try running If you still encounter the CUDA error, you can also check the parameter sensitivity using the CPU backend with Here are the results I get CUDA,
|
|
I tested both the original My environment is:
This is not the same environment as the system where the CUDA error was observed, so I can only confirm that I was unable to reproduce it on my configuration. I also verified that all ReSolve libraries were being loaded from the current PR build ( For the original CUDA, default settingstdewilde@TamarROG:~/hiop_work/resolve-pr466-cuda$ ./build/examples/sysGmres.exe \
-m ~/tmp/matrixmarket/fidap012.mtx \
-r ~/tmp/matrixmarket/fidap012_rhs1.mtx \
-b cuda \
-i fgmres
Matrix file: /home/tdewilde/tmp/matrixmarket/fidap012.mtx
RHS file: /home/tdewilde/tmp/matrixmarket/fidap012_rhs1.mtx
sysGmres with CUDA backend
solver.setMatrix returned status: 0
solver.preconditionerSetup returned status: 0
solver.solve returned status: 0
Initial relative residual norm ||b-A*x||/||b|| : 1.0000000000000000e+00
Final relative residual norm ||b-A*x||/||b|| : 9.9414743316942403e-13
Number of iterations : 1121CUDA,
|
|
Just to double check, I also retested on my new CUDA machine and got the same passing results for both the fidap012 and Navier-Stokes cases. My environment is: Ubuntu 24.04.4 LTS I was again unable to reproduce the CUDA error. |
* Expose CUDA ILU0 boost configuration * Update CHANGELOG.md * Add CLI parameter support * Split GPU ILU0 numeric boost configuration * Rename numeric boost parameters * Add ILU0 boost options to sysGmres * Add Navier-Stokes FEM system test data
3 participants
Description
The numeric boost tolerance and value in the CUDA ILU0 solver are hard-coded to
1e-6. This PR makes them configurable through the existingsetCliParam()interface.To see the effect of these values I tested the solver with FIDAP012 from the NIST Matrix Market, a nonsymmetric linear system arising from a fluid dynamics problem
The number of iterations changed as follows:
1e-69.9392e-131e-79.2850e-131e-89.2850e-13Some problems are sensitive to this value.
Proposed changes
zero_diagonalparameter to the CPU ILU0 solver through the existingsetCliParam()interface.boost_toleranceandboost_valueparameters to the CUDA/HIP ILU0 solvers through the existingsetCliParam()interface.SystemSolver::getPreconditionerSolver()so users can configure ILU0 numerical stabilization parameters throughSystemSolver.testSysGmresandMatrixFactorizationTeststo configure ILU0 parameters.CHANGELOG.md.Checklist
make testandmake test_installper testing instructions). Code tested on./examples/<your_example>.exe -hto get instructions how to run examples). Code tested on:-Wall -Wpedantic -Wconversion -Wextra.