CUDA and ROCm optimizations - #1239
Conversation
…, cuda/rocm test coverage
(coauthored by lightvector)
|
Agree! Thanks for ur hard work and effort! |
|
@Looong01 thanks. Would be great if you can also test it out, since we currently lack any datapoints for gpus like your gfx1100, and in case there is some error or incompatibility it would also be very good to know that so we can fix it. |
|
Tested on my RX 7900 XTX (gfx1100), everything compiled and ran cleanly with no errors or incompatibilities. Great speedups across the board! Environment: ROCm 7.14 (HIP 7.14.60850) / MIOpen 3.5.2, Intel i7-9700 (8 cores), self-compiled Release build with hipcc for gfx1100 only. Same benchmark settings as yours: 10k visits, 10 positions, threads 32/64/96/128, Models: b11c768tf is the same big transformer from v1.17.1 and b40c768 is the same zhizi s11472M model you tested. b10c384 and b10c512 are the two smaller models shipped in the v1.17.2 release zip, included as extra datapoints. A few observations:
Thanks for putting this together! |
|
Hi, I have just tested on my RX 6800XT(gfx1030): Note that improvements to transformer model is HUGE because without the 1x1 convolution GEMM optimization, the old version fell back to MIOpen's slow convolution path on RDNA, bottlenecking search performance at only ~10 visits/s. However, b11c768, while being smaller, is still slower than b28, so there must still be room for optimization. |
|
Thanks for the tests!
b11c768 transformer is about the same number of parameters as b28c512 convnet, but is expected to be slightly slower because transformers at equal param counts are generally slower than convnets. There's always room for optimization on any model, but it's not the case that you should expect it to reach the same speed. However transformers at equal param counts are far stronger than convnets, b11c768 transformer at equal visits is stronger than b40c768 convnet, and should be considered in terms of model quality to be the same class as the b40c768 zhizi models. So based on your results, b11c768 is already the clear winner now, since it's both stronger and faster. |
linear1 and linearGate read the same normalized input and differ only in weights, so they are one matmul with the weight matrices concatenated along the output axis, split back afterwards. Applied to both the NHWC and NCHW transformer FFN paths. The FFN projection is the widest GEMM in a transformer block (ffnC > numHeads*headDim), so this is the larger of the two available projection fusions; QKV is the other. PR lightvector#1239 fuses the same pair, but its CUTLASS dual-GEMM path is guarded by '#if defined(KATAGO_GPU_CUDA) && defined(USE_CUTLASS_FUSED_FFN)' and needs sm_80+, so on CDNA its FFN stays unfused. This closes that gap on AMD. Split rather than two Slices: one node stating 'two equal halves' leaves a single producer with two consumers, and needs no index initializers. Those initializers were also a correctness problem in the obvious formulation, since addInitializer does not deduplicate and every FFN block would emit the same names; the remaining fused-weight initializer is run through uniq() for that reason. The fused NCHW conv sets kernel_shape explicitly, as buildMatMul does. Not yet measured: the NHWC path is still gated off by the transformer policy bug.
3 participants
I worked on this PR over the last few days with a large number of optimizations to the CUDA and ROCm backends, hopefully speeding them up a bunch.
Thanks to @hzyhhzy and @zsqdx for forwarding along the hint from doomoooo in https://github.com/doomoooo/KataGomo_fork/tree/final-migration/final-migration that such an improvement was possible in the CUDA backend. A lot of the improvements also straightforwardly carry over to the ROCm backend due to how the ROCm backend shares most of its logic with CUDA.
Fortunately, unlike what was initially forwarded to me, I found many of the optimizations were not GPU-dependent. The optimizations implemented in this PR mostly are general speedups that apply across GPUs, although the exact amount that they each help might vary by GPU.
This PR incorporates also part of @Looong01's PR #1237, (not all of it), thanks to them for pointing out this additional optimization on rocm.
Here are measured visits per second on the katago benchmark on the GPUs I rented temporarily and tested with. Benchmark settings were 10k visits per position, 10 positions (the particular 10 that the benchmark chooses if you ask for 10).
oldis master branch ccdec95 andnewis the tip of this PR.mi300xis using the rocm backend. All others are using the cuda backend.numNNServerThreadsPerModel = 2. Interestingly, it seems that this is bad on master branch, but now with this PR it becomes good - the improvements on this branch enable the GPU to efficiently alternate between batches of requests.