Compute the TupleHash element length prefix in long arithmetic and refuse a negative length

[Arpan0995]

XofUtils.encode(byte[] in, int inOff, int len) builds the encode_string prefix of NIST SP 800-185 sec. 2.3.3 for a TupleHash element, but computes the bit length in int arithmetic: core/src/main/java/org/bouncycastle/crypto/digests/XofUtils.java:60 and :62 read leftEncode(len * 8) with len an int, so the product wraps before widening to the long parameter. Its only caller is TupleHash.update (TupleHash.java:98). The other call sites (CSHAKEDigest.java:112, KMAC.java:148/:169/:242, TupleHash.java:104, ParallelHash.java:178) already use * 8L.

From 2^28 bytes the product is negative, and leftEncode sizes its output by while ((v >>= 8) != 0) (XofUtils.java:12), an arithmetic shift that converges to -1, so the call never returns; rightEncode has the same loop at :34. From 2^29 bytes the product wraps non-negative and the element is prefixed at the wrong length: it gets 0100, the prefix of an empty element, so two different tuples can absorb the same byte string, which the tuple encoding of sec. 5.3 exists to prevent.

A second argument reaches the same loop: a negative outLen in doFinal(out, off, outLen) enters rightEncode via TupleHash.wrapUp, ParallelHash.wrapUp and KMAC.doFinal, and does not return. Issue #431 raised the negative argument in 2018; the reply fixed the endianness only.

Reproduced on the released bcprov-jdk18on-1.86.jar: MessageDigest.getInstance("TupleHash256-512", "BC").update(new byte[1 << 28]) is RUNNABLE after 20 s with XofUtils.leftEncode on top, where the same object over 1 MiB returns in 8 ms and JCA SHA-256 over that array in 973 ms; at 2^29 bytes two distinct tuples both digest to 7d362b64...dc1aea82. Current origin/main and the 1.87-SNAPSHOT beta (1.87.0.20718) are byte-identical here and reproduce both.

This change:

• widens both multiplies to len * 8L; and
• has leftEncode and rightEncode throw IllegalArgumentException with 'strLen' cannot be negative instead of spinning, so a negative output length is reported rather than hanging; and
• adds XofUtilsTest (encode vectors either side of the int boundary, and the negative cases) and TupleHashLargeInputTest (a 2^28-byte element checked against cSHAKE driven per sec. 5.3), registered in RegressionTest.tests and slowTests. Both hang without the change and pass with it; TupleHashTest, CSHAKETest, KMACTest and ParallelHashTest pass either way.

A digest over an element of 2^29 bytes or more changes, its old prefix not having conformed; below 2^28 the two arithmetics agree. Separately, encode does not check len against in.length, so update(byte[]{0x7f}, 0, 8) digests seven zero bytes Arrays.copyOfRange fabricated, where SHA256Digest throws. Different cause, so it is left out; happy to raise it separately.

Base tree only: one XofUtils.java, no META-INF/versions copy, no module-info or OSGi change. A release-note entry is included, happy to move it to another block.

[dghgit]

Thanks for the patch, merged with revisions, up on https://www.bouncycastle.org/betas