Regression test kernel_regress:skx_avx core dumped with a bus error on FreeBSD on i386

[monwarez]

When attempting to build OpenBLAS 0.3.33 on FreeBSD from within the port on a i386 chroot (jails), the last regression test crashes.

./openblas_utest
TEST 1/125 min:smin_negative [OK]
TEST 2/125 min:dmin_positive [OK]
TEST 3/125 min:smin_zero [OK]
...
...
TEST 124/125 potrf:smoketest_trivial [OK]
TEST 125/125 kernel_regress:skx_avx gmake[1]: *** [Makefile:89: run_test] Bus error (core dumped)
gmake[1]: Leaving directory '/wrkdirs/usr/ports/math/openblas/work/OpenBLAS-0.3.33/utest'

See also the original report here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=293087

[agrajag9]

It seems to have been introduced by commit d9bb8f3

TEST 125/125 kernel_regress:skx_avx gmake[1]: *** [Makefile:86: run_test] Bus error (core dumped)
gmake[1]: Leaving directory '/wrkdirs/usr/ports/math/openblas/work/OpenBLAS-0.3.30/utest'
gmake: *** [Makefile:177: tests] Error 2
*** Error code 1

I was able to get debug symbols to build into the binaries by adding DEBUG_FLAGS to FFLAGS in FreeBSD's ports recipes:

diff --git a/Mk/Features/debug.mk b/Mk/Features/debug.mk
index 6e67a2315223..c9587a82277a 100644
--- a/Mk/Features/debug.mk
+++ b/Mk/Features/debug.mk
@@ -18,6 +18,7 @@ STRIP_CMD=    ${TRUE}
 .  endif
 DEBUG_FLAGS?=  -g
 CFLAGS:=               ${CFLAGS:N-O*:N-fno-strict*} ${DEBUG_FLAGS}
+FFLAGS+=               ${DEBUG_FLAGS}
 .  if defined(INSTALL_TARGET)
 INSTALL_TARGET:=       ${INSTALL_TARGET:S/^install-strip$/install/g}
 .  endif

I could then build the port with make -DWITH_DEBUG build and have both C and Fortran debug symbols.

(gdb) bt
#0  0x00f2531b in dlarf1f (side=..., m=100, n=99, v=..., incv=1, tau=1.1416925237086324, c=..., ldc=100, work=..., _side=4) at dlarf1f.f:156
#1  0x00f2f964 in dgebd2 (m=100, n=100, a=..., lda=100, d=..., e=..., tauq=..., taup=..., work=..., info=0) at dgebd2.f:250
#2  0x00f2f078 in dgebrd (m=100, n=100, a=..., lda=100, d=..., e=..., tauq=..., taup=..., work=..., lwork=6400, info=0) at dgebrd.f:351
#3  0x00f1cdd4 in dgesvd (jobu=..., jobvt=..., m=<optimized out>, n=<optimized out>, a=..., lda=<optimized out>, s=..., u=..., ldu=<optimized out>, vt=..., ldvt=<optimized out>, work=..., lwork=<optimized out>, info=<optimized out>, _jobu=1, _jobvt=1) at dgesvd.f:1982
#4  0x00f1b631 in LAPACKE_dgesvd_work (matrix_layout=101, jobu=65 'A', jobvt=65 'A', m=100, n=100, a=0xf9b708 <m>, lda=100, s=0xf4cec8 <s>, u=0xf4d1e8 <u>, ldu=100, vt=0xf60a68 <vt>, ldvt=100, work=0x21751180, lwork=6700) at lapacke_dgesvd_work.c:113
#5  0x00f1b134 in LAPACKE_dgesvd (matrix_layout=101, jobu=65 'A', jobvt=65 'A', m=100, n=100, a=0xf9b708 <m>, lda=100, s=0xf4cec8 <s>, u=0xf4d1e8 <u>, ldu=100, vt=0xf60a68 <vt>, ldvt=100, superb=0xf87b68 <superb>) at lapacke_dgesvd.c:71
#6  0x004842d9 in __ctest_kernel_regress_skx_avx_run () at test_kernel_regress.c:34
#7  0x00477fc2 in ctest_main (argc=1, argv=0xffffdb68) at ./ctest.h:765
#8  0x004782eb in main (argc=1, argv=0xffffdb68) at utest_main.c:45

disas that subroutine to find the offending instruction:

Dump of assembler code for function dlarf1f_:
   0x00f25300 <+0>:     push   %ebp
   0x00f25301 <+1>:     mov    %esp,%ebp
   0x00f25303 <+3>:     push   %edi
   0x00f25304 <+4>:     push   %esi
   0x00f25305 <+5>:     push   %ebx
   0x00f25306 <+6>:     call   0xf0f9f4 <__x86.get_pc_thunk.bx>
   0x00f2530b <+11>:    add    $0x26a49,%ebx
   0x00f25311 <+17>:    sub    $0x4c,%esp
   0x00f25314 <+20>:    mov    0x24(%ebp),%eax
   0x00f25317 <+23>:    movd   (%eax),%xmm0
=> 0x00f2531b <+27>:    movaps %xmm0,-0x58(%ebp)

What do we have in ebp?

(gdb) info reg
eax            0xffffd918          -9960
ecx            0x21760ca0          561384608
edx            0x217514a0          561321120
ebx            0xf4bd54            16039252
esp            0xffffd554          0xffffd554
ebp            0xffffd5ac          0xffffd5ac
esi            0x1                 1
edi            0xfffffff8          -8
eip            0xf2531b            0xf2531b <dlarf1f+27>
eflags         0x210292            [ AF SF IF RF ID ]
cs             0x33                51
ss             0x3b                59
ds             0x3b                59
es             0x3b                59
fs             0x13                19
gs             0x1b                27
fs_base        0x363340            3552064
gs_base        0x21689140          560501056

ebp - 0x58:

(gdb) x (0xffffd5ac  - 0x58)
0xffffd554:     0x2160b816

That is not 16- or 32-bit aligned, which afaik is required for movaps. However, if we go up the stack to frame 4 we DO see a 16-bit aligned value.

(gdb) up
#4  0x00f1b631 in LAPACKE_dgesvd_work (matrix_layout=101, jobu=65 'A', jobvt=65 'A', m=100, n=100, a=0xf9b708 <m>, lda=100, s=0xf4cec8 <s>, u=0xf4d1e8 <u>, ldu=100, vt=0xf60a68 <vt>, ldvt=100, work=0x21751180, lwork=6700) at lapacke_dgesvd_work.c:113
113             LAPACK_dgesvd( &jobu, &jobvt, &m, &n, a_t, &lda_t, s, u_t, &ldu_t, vt_t,
(gdb) print work
$2 = (double *) 0x21751180

Unfortunately this is where I hit a wall. gdb was reporting that a lot of the variables had been "optimized out" and so we couldn't look deeper into them. But then, when I added DEBUG = 1 to OpenBLAS's Makefile.rule and again ran make -DWITH_DEBUG build all the optimizations went away, but so did the SIGBUS error - the test passed.

[martin-frbg]

Unlikely to be d9bb8f3 (which wouldn't affect DGEMM), probably one of the things recently fixed in the 32bit kernels but I can't check right now.
Is this an AMD cpu by any chance ?

[agrajag9]

For me, yes --- 5950X

I do not know if this is the case for others also experiencing this.

[agrajag9]

If it's not d9bb8f3, it is something else in between v0.30 v0.33

[martin-frbg]

I will try to look into it tonight or early tomorrow. Can you check if the problem persists with current develop branch ?
(Guess DGESVD is probably involved in the comparison branch ... the (LAPACK) commit in question changed some DAXPY/DGESV call sequences so maybe there's some roundabout way this is now hitting a movaps somewhere in ancient 32bit GotoBLAS kernels that could just as well be a movups on any remotely modern hardware

[martin-frbg]

Ofc if it is the compiler generating aligned moves from the Fortran source of LARF1F where the input is already unaligned this won't help. But I'm visiting elderly relatives and hopping around a leaky pond I'm expected to fix...