From 2eed3b6342f506da0ebd045b4e05f3c3321bdec0 Mon Sep 17 00:00:00 2001 From: Daniel Lemire Date: Wed, 12 Aug 2026 10:27:46 -0400 Subject: [PATCH 1/2] Optimize AVX2 popcount inner loop and CPU feature detection MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit David Sparks reviewed popcnt_avx2_amd64.s and suggested the improvements implemented here; the credit for all of them is his. The main win is in COUNTBLOCK. VPSADBW computes |a-b| per byte and sums each group of 8, so it can absorb the per-byte add that VPADDB was doing: feeding it the two nibble counts directly yields (B + lo) - (B - hi) = lo + hi, removing VPADDB and its latency from the hot loop. That needs two lookup tables, one biased up by B and one subtracted from B. The bias must satisfy 4 <= B <= 251 so that neither table wraps as unsigned bytes and a >= b always holds, making the absolute value a no-op; B = 15 is free because Ymask already holds 15 in every byte. The rest: - Generic (VEX-encoded) AVX instructions accept an unaligned memory source, so the second input of the And/Or/Xor/Mask loops is read straight out of memory rather than loaded into a register first. Only VPANDN's non-negated operand may come from memory, so that loop loads m into the register and reads s from memory. - Ydata/Yhi/Yc2 and Ylo/Yc1 have disjoint live ranges and now share registers; six architectural registers suffice instead of ten. - lutmask shrinks from 64 to 24 bytes: VBROADCASTI128 duplicates the 16-byte table into both 128-bit lanes and VPBROADCASTB splats the nibble mask from a single byte. - SHRQ and ANDQ already set ZF, so the TESTQ instructions that followed them were redundant. - SETUP moves below the branch that skips the vector loop entirely. - The 64-bit tail loops use POPCNTQ with a memory source and a 32-bit loop counter (32-bit avoids the partial-register merge a DECB needs). - XORL rather than XORQ for zeroing: no REX prefix, and some cores do not recognize XORQ as a zeroing idiom. - _hasAVX2 complements each feature word and TESTs it instead of AND/CMP, saving a large immediate per check, and all three checks now fall through to a single SETEQ that stores ZF into the bool result. Measured on an Intel Xeon Gold 6548N with go1.26.3. Microbenchmarks, count=10: PopcntSlice1024AVX2 205.0n -> 180.0n -12.20% PopcntAndSlice1024AVX2 241.6n -> 225.7n -6.58% Popcount 14.66n -> 13.73n -6.35% And through the public API on dense bitmaps (32 bitmap containers, count=8), which is what the popcount slice routines actually serve: Bitmap.AndCardinality 9.404µ -> 8.655µ -7.96% Bitmap.OrCardinality 128.6µ -> 120.8µ -6.02% The pure-Go fallbacks are unchanged and measure flat, as expected. The assembly text shrinks from 987 to 914 bytes and the rodata blob from 64 to 24. Verified against the Go reference for all five routines over the existing differential test plus 20000 randomized rounds with unaligned sub-slices and all-ones/all-zeros inputs. --- popcnt_avx2_amd64.s | 240 +++++++++++++++++++++++--------------------- 1 file changed, 125 insertions(+), 115 deletions(-) diff --git a/popcnt_avx2_amd64.s b/popcnt_avx2_amd64.s index 1c61a237..0868f2e1 100644 --- a/popcnt_avx2_amd64.s +++ b/popcnt_avx2_amd64.s @@ -13,10 +13,11 @@ // AVX2 has no single "popcount a whole vector" instruction, so each byte's // popcount is taken from a 16-entry lookup table indexed by a 4-bit nibble: // a byte is split into its low and high nibble, each nibble is looked up (one -// VPSHUFB performs all 32 lookups in a 256-bit register at once), the two -// results are added to give a per-byte popcount, and VPSADBW then sums each -// group of 8 byte-counts into a 64-bit lane total that is accumulated. After -// the loop the four lane totals are summed (HSUM) into a scalar register. +// VPSHUFB performs all 32 lookups in a 256-bit register at once), and VPSADBW +// then combines the two results and sums each group of 8 byte-counts into a +// 64-bit lane total that is accumulated (see COUNTBLOCK for how the two +// nibble counts are added by VPSADBW itself rather than a separate VPADDB). +// After the loop the four lane totals are summed (HSUM) into a scalar register. // Each iteration handles 256 bits (4 uint64); a scalar POPCNTQ tail handles // the trailing len%4 words, so any slice length is counted correctly. // @@ -31,84 +32,102 @@ // (e.g. ret+24(FP) for one slice arg, ret+48(FP) for two). // - Every routine is a leaf (makes no calls): NOSPLIT with a $0 local frame. // - Loads/stores use VMOVDQU (unaligned): container slices are only 8-byte -// aligned, not 32. VZEROUPPER precedes every RET to avoid the AVX<->SSE -// transition penalty in any non-VEX SSE code that runs afterwards. +// aligned, not 32. Generic (VEX-encoded) AVX instructions impose no +// alignment requirement on a memory source either, so the second input of +// the two-operand loops is read straight out of memory by VPAND/VPOR/ +// VPXOR/VPANDN instead of being loaded into a register first. +// VZEROUPPER precedes every RET to avoid the AVX<->SSE transition penalty +// in any non-VEX SSE code that runs afterwards. -// lutmask is a 64-byte read-only blob holding two constants used by every +// lutmask is a 24-byte read-only blob holding the two constants used by every // routine: -// bytes 0..31 - the nibble popcount table, i.e. table[i] = number of set -// bits in the 4-bit value i. VPSHUFB indexes within each -// 128-bit lane independently, so the 16-entry table is stored -// twice (once per lane). Read low-byte-first, the first qword -// 0x0302020102010100 is the bytes {0,1,1,2,1,2,2,3} for +// bytes 0..15 - the nibble popcount table, i.e. table[i] = number of set +// bits in the 4-bit value i. Read low-byte-first, the first +// qword 0x0302020102010100 is the bytes {0,1,1,2,1,2,2,3} for // nibbles 0..7, and 0x0403030203020201 is {1,2,2,3,2,3,3,4} -// for nibbles 8..15. -// bytes 32..63 - 0x0F in every byte: a mask that isolates the low nibble of -// each byte. +// for nibbles 8..15. VPSHUFB indexes within each 128-bit lane +// independently and so needs the table in both lanes, but +// VBROADCASTI128 duplicates the 16 bytes at load time; only +// one copy has to be stored. +// byte 16 - 0x0F, the mask that isolates the low nibble of each byte, +// splatted to all 32 bytes by VPBROADCASTB. (It is written as +// a full qword below purely for readability; only the first +// byte is ever read.) // RODATA|NOPTR marks it read-only and pointer-free (so the GC ignores it). DATA lutmask<>+0(SB)/8, $0x0302020102010100 DATA lutmask<>+8(SB)/8, $0x0403030203020201 -DATA lutmask<>+16(SB)/8, $0x0302020102010100 -DATA lutmask<>+24(SB)/8, $0x0403030203020201 -DATA lutmask<>+32(SB)/8, $0x0f0f0f0f0f0f0f0f -DATA lutmask<>+40(SB)/8, $0x0f0f0f0f0f0f0f0f -DATA lutmask<>+48(SB)/8, $0x0f0f0f0f0f0f0f0f -DATA lutmask<>+56(SB)/8, $0x0f0f0f0f0f0f0f0f -GLOBL lutmask<>(SB), RODATA|NOPTR, $64 +DATA lutmask<>+16(SB)/8, $0x0f0f0f0f0f0f0f0f +GLOBL lutmask<>(SB), RODATA|NOPTR, $24 -// Register aliases. Ylut/Ymask/Yzero are constants set up once per call (see -// SETUP); Yacc is the running accumulator of lane totals; Ydata/Yb hold the -// current input vector(s); Ylo/Yhi/Yc1/Yc2 are scratch used by COUNTBLOCK. -#define Ylut Y0 -#define Ymask Y1 -#define Yzero Y2 +// Register aliases. Ylut1/Ylut2/Ymask are constants set up once per call (see +// SETUP); Yacc is the running accumulator of lane totals; Ydata holds the +// current input vector; Ylo/Yhi/Yc1/Yc2 are scratch used by COUNTBLOCK. The +// scratch values have disjoint live ranges, so Yhi/Yc2 reuse Ydata's register +// and Yc1 reuses Ylo's: only six architectural registers are needed. +#define Ylut1 Y0 +#define Ylut2 Y1 +#define Ymask Y2 #define Yacc Y3 #define Ydata Y4 -#define Yb Y5 -#define Ylo Y6 -#define Yhi Y7 -#define Yc1 Y8 -#define Yc2 Y9 +#define Yhi Y4 +#define Yc2 Y4 +#define Ylo Y5 +#define Yc1 Y5 + +// Low 128-bit halves of Yacc and Ylo, used as scratch by HSUM. +#define Xacc X3 +#define Xtmp X5 // COUNTBLOCK folds the popcount of the 32 bytes currently in Ydata into the // accumulator Yacc. Line by line: -// VPAND Ymask,Ydata,Ylo : Ylo = low nibble of every byte -// VPSRLW $4,Ydata,Yhi : shift each 16-bit lane right by 4... -// VPAND Ymask,Yhi,Yhi : ...then mask, leaving the high nibble of each byte -// VPSHUFB Ylo,Ylut,Yc1 : Yc1[b] = popcount(low nibble of byte b) -// VPSHUFB Yhi,Ylut,Yc2 : Yc2[b] = popcount(high nibble of byte b) -// VPADDB Yc2,Yc1,Yc1 : Yc1[b] = popcount(byte b) (0..8 each) -// VPSADBW Yzero,Yc1,Yc1 : sum each group of 8 bytes -> 4 lane totals (0..512) -// VPADDQ Yc1,Yacc,Yacc : add the 4 lane totals into the accumulator +// VPAND Ymask,Ydata,Ylo : Ylo = low nibble of every byte +// VPSRLW $4,Ydata,Yhi : shift each 16-bit lane right by 4... +// VPAND Ymask,Yhi,Yhi : ...then mask, leaving the high nibble of each byte +// VPSHUFB Ylo,Ylut1,Yc1 : Yc1[b] = B + popcount(low nibble of byte b) +// VPSHUFB Yhi,Ylut2,Yc2 : Yc2[b] = B - popcount(high nibble of byte b) +// VPSADBW Yc1,Yc2,Yc1 : sum each group of 8 bytes -> 4 lane totals +// VPADDQ Yc1,Yacc,Yacc : add the 4 lane totals into the accumulator +// +// VPSADBW computes |a-b| per byte and sums each group of 8, so it can do the +// work of the per-byte add as well: feeding it the two nibble counts directly +// yields (B + lo) - (B - hi) = lo + hi, and the separate VPADDB the naive +// version needs (with a zero second VPSADBW operand) disappears along with its +// latency. That is why SETUP builds two tables, one biased up by B and one +// subtracted from B. The bias must satisfy 4 <= B <= 251 so that neither +// table's entries (max nibble popcount is 4) wrap around as unsigned bytes and +// so that a >= b always holds, making the absolute value a no-op; B = 15 is +// used simply because Ymask already holds 15 in every byte. +// // Per-byte counts max at 8 and lane totals at 512, so accumulating across the // whole loop never overflows the 64-bit lanes. #define COUNTBLOCK \ VPAND Ymask, Ydata, Ylo \ VPSRLW $4, Ydata, Yhi \ VPAND Ymask, Yhi, Yhi \ - VPSHUFB Ylo, Ylut, Yc1 \ - VPSHUFB Yhi, Ylut, Yc2 \ - VPADDB Yc2, Yc1, Yc1 \ - VPSADBW Yzero, Yc1, Yc1 \ + VPSHUFB Ylo, Ylut1, Yc1 \ + VPSHUFB Yhi, Ylut2, Yc2 \ + VPSADBW Yc1, Yc2, Yc1 \ VPADDQ Yc1, Yacc, Yacc -// SETUP loads the lookup table and nibble mask and zeroes Yzero (the VPSADBW -// addend) and Yacc (the accumulator). Run once at the top of each routine. +// SETUP builds the two biased lookup tables and the nibble mask, and zeroes +// Yacc (the accumulator). Run once per routine, after the check that the +// vector loop runs at least one iteration. Ylut1 first holds the raw table, so +// the VPSUBB must come before the VPADDB that overwrites it. #define SETUP \ - VMOVDQU lutmask<>+0(SB), Ylut \ - VMOVDQU lutmask<>+32(SB), Ymask \ - VPXOR Yzero, Yzero, Yzero \ - VPXOR Yacc, Yacc, Yacc + VBROADCASTI128 lutmask<>+0(SB), Ylut1 \ + VPBROADCASTB lutmask<>+16(SB), Ymask \ + VPXOR Yacc, Yacc, Yacc \ + VPSUBB Ylut1, Ymask, Ylut2 \ + VPADDB Ylut1, Ymask, Ylut1 -// HSUM reduces Yacc's four 64-bit lane totals to a single sum in AX. X3 is the -// low 128 bits of Yacc (Y3); VEXTRACTI128 pulls the high 128 bits into X5, the -// two halves are added (giving two qwords in X3), and those two qwords are then -// added into AX. +// HSUM reduces Yacc's four 64-bit lane totals to a single sum in AX. +// VEXTRACTI128 pulls Yacc's high 128 bits into Xtmp, the two halves are added +// (giving two qwords in Xacc), and those two qwords are then added into AX. #define HSUM \ - VEXTRACTI128 $1, Yacc, X5 \ - VPADDQ X5, X3, X3 \ - VPEXTRQ $1, X3, DX \ - MOVQ X3, R9 \ + VEXTRACTI128 $1, Yacc, Xtmp \ + VPADDQ Xtmp, Xacc, Xacc \ + VPEXTRQ $1, Xacc, DX \ + MOVQ Xacc, R9 \ ADDQ R9, AX \ ADDQ DX, AX @@ -119,12 +138,11 @@ GLOBL lutmask<>(SB), RODATA|NOPTR, $64 TEXT ·_popcntSliceAVX2(SB), NOSPLIT, $0-32 MOVQ s_base+0(FP), SI // SI = &s[0] MOVQ s_len+8(FP), CX // CX = len(s), in 64-bit words - XORQ AX, AX // AX = running result - SETUP // load table/mask; zero Yzero and Yacc + XORL AX, AX // AX = running result MOVQ CX, R8 SHRQ $2, R8 // R8 = len/4 = number of full 256-bit blocks - TESTQ R8, R8 JZ slicetail // fewer than 4 words: skip the vector loop + SETUP // load tables/mask; zero Yacc sliceloop: VMOVDQU (SI), Ydata // load 4 words (32 bytes) COUNTBLOCK // Yacc += popcount(those 32 bytes) @@ -133,15 +151,13 @@ sliceloop: JNZ sliceloop HSUM // AX += sum of Yacc's lane totals slicetail: - ANDQ $3, CX // CX = len % 4 = leftover words (0..3) - TESTQ CX, CX + ANDL $3, CX // CX = len % 4 = leftover words (0..3) JZ slicedone slicetailloop: - MOVQ (SI), DX - POPCNTQ DX, DX // scalar popcount of one word + POPCNTQ (SI), DX // scalar popcount of one word ADDQ DX, AX ADDQ $8, SI - DECQ CX + DECL CX JNZ slicetailloop slicedone: VZEROUPPER // clear upper YMM state before returning @@ -149,23 +165,21 @@ slicedone: RET // func _popcntAndSliceAVX2(s, m []uint64) uint64 -// Returns the sum of popcount(s[i] & m[i]). Mirrors _popcntSliceAVX2 but loads -// a vector from each of s and m and ANDs them before counting. s and m are +// Returns the sum of popcount(s[i] & m[i]). Mirrors _popcntSliceAVX2 but ANDs +// a vector of s with the matching bytes of m before counting. s and m are // assumed to have equal length. TEXT ·_popcntAndSliceAVX2(SB), NOSPLIT, $0-56 MOVQ s_base+0(FP), SI // SI = &s[0] MOVQ m_base+24(FP), DI // DI = &m[0] MOVQ s_len+8(FP), CX // CX = len - XORQ AX, AX - SETUP + XORL AX, AX MOVQ CX, R8 SHRQ $2, R8 - TESTQ R8, R8 JZ andtail + SETUP andloop: VMOVDQU (SI), Ydata - VMOVDQU (DI), Yb - VPAND Yb, Ydata, Ydata // Ydata = s & m + VPAND (DI), Ydata, Ydata // Ydata = s & m COUNTBLOCK ADDQ $32, SI ADDQ $32, DI @@ -173,8 +187,7 @@ andloop: JNZ andloop HSUM andtail: - ANDQ $3, CX - TESTQ CX, CX + ANDL $3, CX JZ anddone andtailloop: MOVQ (SI), DX @@ -183,7 +196,7 @@ andtailloop: ADDQ DX, AX ADDQ $8, SI ADDQ $8, DI - DECQ CX + DECL CX JNZ andtailloop anddone: VZEROUPPER @@ -197,16 +210,14 @@ TEXT ·_popcntOrSliceAVX2(SB), NOSPLIT, $0-56 MOVQ s_base+0(FP), SI MOVQ m_base+24(FP), DI MOVQ s_len+8(FP), CX - XORQ AX, AX - SETUP + XORL AX, AX MOVQ CX, R8 SHRQ $2, R8 - TESTQ R8, R8 JZ ortail + SETUP orloop: VMOVDQU (SI), Ydata - VMOVDQU (DI), Yb - VPOR Yb, Ydata, Ydata // Ydata = s | m + VPOR (DI), Ydata, Ydata // Ydata = s | m COUNTBLOCK ADDQ $32, SI ADDQ $32, DI @@ -214,8 +225,7 @@ orloop: JNZ orloop HSUM ortail: - ANDQ $3, CX - TESTQ CX, CX + ANDL $3, CX JZ ordone ortailloop: MOVQ (SI), DX @@ -224,7 +234,7 @@ ortailloop: ADDQ DX, AX ADDQ $8, SI ADDQ $8, DI - DECQ CX + DECL CX JNZ ortailloop ordone: VZEROUPPER @@ -238,16 +248,14 @@ TEXT ·_popcntXorSliceAVX2(SB), NOSPLIT, $0-56 MOVQ s_base+0(FP), SI MOVQ m_base+24(FP), DI MOVQ s_len+8(FP), CX - XORQ AX, AX - SETUP + XORL AX, AX MOVQ CX, R8 SHRQ $2, R8 - TESTQ R8, R8 JZ xortail + SETUP xorloop: VMOVDQU (SI), Ydata - VMOVDQU (DI), Yb - VPXOR Yb, Ydata, Ydata // Ydata = s ^ m + VPXOR (DI), Ydata, Ydata // Ydata = s ^ m COUNTBLOCK ADDQ $32, SI ADDQ $32, DI @@ -255,8 +263,7 @@ xorloop: JNZ xorloop HSUM xortail: - ANDQ $3, CX - TESTQ CX, CX + ANDL $3, CX JZ xordone xortailloop: MOVQ (SI), DX @@ -265,7 +272,7 @@ xortailloop: ADDQ DX, AX ADDQ $8, SI ADDQ $8, DI - DECQ CX + DECL CX JNZ xortailloop xordone: VZEROUPPER @@ -275,21 +282,22 @@ xordone: // func _popcntMaskSliceAVX2(s, m []uint64) uint64 // Returns the sum of popcount(s[i] &^ m[i]) == popcount(s & ~m). Same structure // as _popcntAndSliceAVX2; the combine is VPANDN, which computes (NOT first) AND -// second, i.e. VPANDN Ydata, Yb, Ydata -> Ydata = (NOT Yb) AND Ydata = s &^ m. +// second. Only VPANDN's second source may come from memory, and it is the +// operand that is *not* negated, so here it is m that is loaded into a register +// and s that is read straight out of memory: +// "VPANDN (SI), Ydata, Ydata" with Ydata = m gives (NOT m) AND s = s &^ m. TEXT ·_popcntMaskSliceAVX2(SB), NOSPLIT, $0-56 MOVQ s_base+0(FP), SI MOVQ m_base+24(FP), DI MOVQ s_len+8(FP), CX - XORQ AX, AX - SETUP + XORL AX, AX MOVQ CX, R8 SHRQ $2, R8 - TESTQ R8, R8 JZ masktail + SETUP maskloop: - VMOVDQU (SI), Ydata - VMOVDQU (DI), Yb - VPANDN Ydata, Yb, Ydata // Ydata = s &^ m (= (NOT m) AND s) + VMOVDQU (DI), Ydata // Ydata = m + VPANDN (SI), Ydata, Ydata // Ydata = s &^ m (= (NOT m) AND s) COUNTBLOCK ADDQ $32, SI ADDQ $32, DI @@ -297,8 +305,7 @@ maskloop: JNZ maskloop HSUM masktail: - ANDQ $3, CX - TESTQ CX, CX + ANDL $3, CX JZ maskdone masktailloop: MOVQ (DI), R10 @@ -309,7 +316,7 @@ masktailloop: ADDQ DX, AX ADDQ $8, SI ADDQ $8, DI - DECQ CX + DECL CX JNZ masktailloop maskdone: VZEROUPPER @@ -320,33 +327,36 @@ maskdone: // Reports whether the CPU supports AVX2 and the OS has enabled the wide (YMM) // register state. All three checks must pass; otherwise the Go wrappers fall // back to the scalar implementation. Note CPUID clobbers AX/BX/CX/DX. +// +// Each check complements the feature word and then TESTs the bits of interest: +// ZF is set exactly when every required bit was set in the original value. That +// needs only one large immediate instead of the two an AND/CMP pair would +// encode, and it lets all three checks converge on a single SETEQ, which stores +// the final ZF straight into the bool result. Since this runs once per process, +// code size matters more here than the (negligible) speed difference. TEXT ·_hasAVX2(SB), NOSPLIT, $0-1 - // CPUID leaf 1: require OSXSAVE (ECX bit 27) and AVX (ECX bit 28). Both must - // be set, so mask and compare against the combined bit pattern. + // CPUID leaf 1: require OSXSAVE (ECX bit 27) and AVX (ECX bit 28). MOVL $1, AX XORL CX, CX CPUID - ANDL $0x18000000, CX - CMPL CX, $0x18000000 + NOTL CX + TESTL $0x18000000, CX JNE noavx2 // XGETBV(0): the OS must have enabled saving of SSE and AVX/YMM state, i.e. // XCR0 bits 1 and 2. Without this the YMM registers would be corrupted // across a context switch even though the CPU supports the instructions. XORL CX, CX XGETBV - ANDL $0x6, AX - CMPL AX, $0x6 + NOTL AX + TESTL $0x6, AX JNE noavx2 // CPUID leaf 7, sub-leaf 0: require AVX2 itself (EBX bit 5). The sub-leaf is // selected via ECX, which must be 0. MOVL $7, AX XORL CX, CX CPUID - ANDL $0x20, BX - CMPL BX, $0x20 - JNE noavx2 - MOVB $1, ret+0(FP) - RET + NOTL BX + TESTL $0x20, BX noavx2: - MOVB $0, ret+0(FP) + SETEQ ret+0(FP) // ZF is still set by whichever TESTL ran last RET From 1dc82c81429b8e176357be741579b62a07e7f2f1 Mon Sep 17 00:00:00 2001 From: Daniel Lemire Date: Thu, 13 Aug 2026 15:04:23 -0400 Subject: [PATCH 2/2] Address review feedback on AVX2 popcount Four style/codegen changes from review: - Trim the algorithm overview to just point at COUNTBLOCK rather than restating how VPSADBW folds in the per-byte add. - Drop the Yc1/Yc2 register aliases and reuse Ylo/Yhi directly. The old aliases already mapped to Y5/Y4, so this is a pure rename: the emitted code for COUNTBLOCK is byte-for-byte identical, verified by disassembling before and after. - Finish HSUM's reduction in SIMD, replacing VPEXTRQ (2 uops on AMD Zen and Intel) with VPSHUFD $0x4e + VPADDQ (1 uop each) and a single MOVQ. One fewer uop, and it frees R9, which HSUM was the only user of. - Write the 0x0f nibble mask as a single byte via DATA .../1 rather than a full qword, and size the symbol accordingly. The Go assembler takes a /1 size directly, so no BYTE pseudo-op is needed in the data section. Note this is not a size win: the linker aligns the next .rodata symbol to 16 bytes, so the slot is 32 bytes before and after. Clarity only, and the comment no longer claims otherwise. --- popcnt_avx2_amd64.s | 52 ++++++++++++++++++++++----------------------- 1 file changed, 25 insertions(+), 27 deletions(-) diff --git a/popcnt_avx2_amd64.s b/popcnt_avx2_amd64.s index 0868f2e1..52e2c989 100644 --- a/popcnt_avx2_amd64.s +++ b/popcnt_avx2_amd64.s @@ -15,8 +15,7 @@ // a byte is split into its low and high nibble, each nibble is looked up (one // VPSHUFB performs all 32 lookups in a 256-bit register at once), and VPSADBW // then combines the two results and sums each group of 8 byte-counts into a -// 64-bit lane total that is accumulated (see COUNTBLOCK for how the two -// nibble counts are added by VPSADBW itself rather than a separate VPADDB). +// 64-bit lane total that is accumulated (COUNTBLOCK). // After the loop the four lane totals are summed (HSUM) into a scalar register. // Each iteration handles 256 bits (4 uint64); a scalar POPCNTQ tail handles // the trailing len%4 words, so any slice length is counted correctly. @@ -39,8 +38,8 @@ // VZEROUPPER precedes every RET to avoid the AVX<->SSE transition penalty // in any non-VEX SSE code that runs afterwards. -// lutmask is a 24-byte read-only blob holding the two constants used by every -// routine: +// lutmask is a 17-byte read-only blob (the linker pads it out to whatever its +// alignment requires) holding the two constants used by every routine: // bytes 0..15 - the nibble popcount table, i.e. table[i] = number of set // bits in the 4-bit value i. Read low-byte-first, the first // qword 0x0302020102010100 is the bytes {0,1,1,2,1,2,2,3} for @@ -50,29 +49,25 @@ // VBROADCASTI128 duplicates the 16 bytes at load time; only // one copy has to be stored. // byte 16 - 0x0F, the mask that isolates the low nibble of each byte, -// splatted to all 32 bytes by VPBROADCASTB. (It is written as -// a full qword below purely for readability; only the first -// byte is ever read.) +// splatted to all 32 bytes by VPBROADCASTB. // RODATA|NOPTR marks it read-only and pointer-free (so the GC ignores it). DATA lutmask<>+0(SB)/8, $0x0302020102010100 DATA lutmask<>+8(SB)/8, $0x0403030203020201 -DATA lutmask<>+16(SB)/8, $0x0f0f0f0f0f0f0f0f -GLOBL lutmask<>(SB), RODATA|NOPTR, $24 +DATA lutmask<>+16(SB)/1, $0x0f +GLOBL lutmask<>(SB), RODATA|NOPTR, $17 // Register aliases. Ylut1/Ylut2/Ymask are constants set up once per call (see // SETUP); Yacc is the running accumulator of lane totals; Ydata holds the -// current input vector; Ylo/Yhi/Yc1/Yc2 are scratch used by COUNTBLOCK. The -// scratch values have disjoint live ranges, so Yhi/Yc2 reuse Ydata's register -// and Yc1 reuses Ylo's: only six architectural registers are needed. +// current input vector; Ylo/Yhi are scratch used by COUNTBLOCK. Ydata is dead +// once its nibbles have been extracted, so Yhi shares its register: only five +// architectural registers are needed. #define Ylut1 Y0 #define Ylut2 Y1 #define Ymask Y2 #define Yacc Y3 #define Ydata Y4 #define Yhi Y4 -#define Yc2 Y4 #define Ylo Y5 -#define Yc1 Y5 // Low 128-bit halves of Yacc and Ylo, used as scratch by HSUM. #define Xacc X3 @@ -83,10 +78,10 @@ GLOBL lutmask<>(SB), RODATA|NOPTR, $24 // VPAND Ymask,Ydata,Ylo : Ylo = low nibble of every byte // VPSRLW $4,Ydata,Yhi : shift each 16-bit lane right by 4... // VPAND Ymask,Yhi,Yhi : ...then mask, leaving the high nibble of each byte -// VPSHUFB Ylo,Ylut1,Yc1 : Yc1[b] = B + popcount(low nibble of byte b) -// VPSHUFB Yhi,Ylut2,Yc2 : Yc2[b] = B - popcount(high nibble of byte b) -// VPSADBW Yc1,Yc2,Yc1 : sum each group of 8 bytes -> 4 lane totals -// VPADDQ Yc1,Yacc,Yacc : add the 4 lane totals into the accumulator +// VPSHUFB Ylo,Ylut1,Ylo : Ylo[b] = B + popcount(low nibble of byte b) +// VPSHUFB Yhi,Ylut2,Yhi : Yhi[b] = B - popcount(high nibble of byte b) +// VPSADBW Ylo,Yhi,Ylo : sum each group of 8 bytes -> 4 lane totals +// VPADDQ Ylo,Yacc,Yacc : add the 4 lane totals into the accumulator // // VPSADBW computes |a-b| per byte and sums each group of 8, so it can do the // work of the per-byte add as well: feeding it the two nibble counts directly @@ -104,10 +99,10 @@ GLOBL lutmask<>(SB), RODATA|NOPTR, $24 VPAND Ymask, Ydata, Ylo \ VPSRLW $4, Ydata, Yhi \ VPAND Ymask, Yhi, Yhi \ - VPSHUFB Ylo, Ylut1, Yc1 \ - VPSHUFB Yhi, Ylut2, Yc2 \ - VPSADBW Yc1, Yc2, Yc1 \ - VPADDQ Yc1, Yacc, Yacc + VPSHUFB Ylo, Ylut1, Ylo \ + VPSHUFB Yhi, Ylut2, Yhi \ + VPSADBW Ylo, Yhi, Ylo \ + VPADDQ Ylo, Yacc, Yacc // SETUP builds the two biased lookup tables and the nibble mask, and zeroes // Yacc (the accumulator). Run once per routine, after the check that the @@ -121,14 +116,17 @@ GLOBL lutmask<>(SB), RODATA|NOPTR, $24 VPADDB Ylut1, Ymask, Ylut1 // HSUM reduces Yacc's four 64-bit lane totals to a single sum in AX. -// VEXTRACTI128 pulls Yacc's high 128 bits into Xtmp, the two halves are added -// (giving two qwords in Xacc), and those two qwords are then added into AX. +// VEXTRACTI128 pulls Yacc's high 128 bits into Xtmp and the two halves are +// added, leaving two qwords in Xacc; VPSHUFD $0x4e then swaps those two qwords +// so a second VPADDQ puts their total in the low qword, which one MOVQ moves +// out. Finishing the reduction in SIMD avoids VPEXTRQ, which is 2 uops on both +// AMD Zen and Intel, against 1 each for VPSHUFD and VPADDQ. #define HSUM \ VEXTRACTI128 $1, Yacc, Xtmp \ VPADDQ Xtmp, Xacc, Xacc \ - VPEXTRQ $1, Xacc, DX \ - MOVQ Xacc, R9 \ - ADDQ R9, AX \ + VPSHUFD $0x4e, Xacc, Xtmp \ + VPADDQ Xtmp, Xacc, Xacc \ + MOVQ Xacc, DX \ ADDQ DX, AX // func _popcntSliceAVX2(s []uint64) uint64