[deftasparagusanaconda]

previously, we stored the 4 integers of a pickled arb in hexadecimal. thats 2 bytes per byte. instead, we use python's native int pickling, achieving near-optimal packing

old:

>>> import math, pickle
>>> from flint import arb, ctx
>>>
>>> ctx.dps = 1_000_000
>>>
>>> original = arb(2).sqrt().mid()
>>> pickled = pickle.dumps(original)
>>> limit = math.ceil(ctx.dps * math.log2(10) / 8)
>>>
>>> print('pickle size      :', len(pickled))
>>> print('theoretical limit:', limit)
>>> print('difference       :', len(pickled) - limit)
>>>
pickle size      : 830583
theoretical limit: 415242
difference       : 415341

new:

>>> import math, pickle
>>> from flint import arb, ctx
>>>
>>> ctx.dps = 1_000_000
>>>
>>> original = arb(2).sqrt().mid()
>>> pickled = pickle.dumps(original)
>>> limit = math.ceil(ctx.dps * math.log2(10) / 8)
>>>
>>> print('pickle size      :', len(pickled))
>>> print('theoretical limit:', limit)
>>> print('difference       :', len(pickled) - limit)
>>>
pickle size      : 415311
theoretical limit: 415242
difference       : 69

69 haha nice

[deftasparagusanaconda]

we can also reduce pickled acbs if we pickle them as

(acb, ((real_mid, real_rad), (imag_mid, imag_rad)) )

instead of

(acb, (self.real, self.imag))

so we avoid including the arb constructor twice. this may bloat the code for acb.__reduce__ though

[oscarbenjamin]

Seems to have failed all the tests.

[deftasparagusanaconda]

test_arb.py/test_pickle() and test_acb.py/test_acb_pickle() test all pickle protocols from 0 to (currently the highest) 5. protocols 0 and 1 cannot pickle a large int properly:

import pickle

n = 10 ** 10000

for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
    try: 
        pickle.dumps(n, protocol=protocol)
        print('protocol', protocol, 'passed')
    except Exception as e:
        print('protocol', protocol, 'failed')
        print(e)

protocol 0 failed
Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
protocol 1 failed
Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
protocol 2 passed
protocol 3 passed
protocol 4 passed
protocol 5 passed

the fix is to replace __reduce__ with __reduce_ex__(protocol) and use a custom algorithm for protocol = 0 and 1.

[deftasparagusanaconda]

@oscarbenjamin i see two options moving forward. which do you prefer?

1. drop support for protocol 0 and 1, and state in the docs that only pickling protocols ≥2 (i.e. python ≥2.3) are supported. also change tests to match accordingly

2. implement support for protocol 0 and 1 using int.to_bytes() and int.from_bytes(), and reintroduce _arb_from_pickle()

i prefer the first option, because python has used DEFAULT_PROTOCOL = 3 since python 3.0, and = 2 since python 2.3. its better to have simple code instead of special-casing for paleolithically old versions. we can always add support if someone asks for it.

[oscarbenjamin]

We can also just not have the change here and use hex. This is why hex as used in the first place.

[deftasparagusanaconda]

i dont favour hex because it takes up twice the storage needed. its not uncommon to store thousands of arbs with millions of digits that take up gigabytes of storage.

i am currently computing a constant, where i'd like to store 284228 arbs at 100100dps. using hex, that should be roughly 2 * 100100 * log2(10)/8 * 284228 / 1024**3 ≈ 22 GiB. using bit-packing halves that, saving me 11GiB. its the main reason im introducing this PR, actually.

(im just trying to prove that the storage savings are significant)

[oscarbenjamin]

Can it not just use to_bytes for all protocol versions? I don't want to overcomplicate anything here.

[deftasparagusanaconda]

👍 sure ill check and see in an hour

[deftasparagusanaconda]

i also found that fmpz has the same issue, because it also uses native int pickling. the tests did not cover this.

import pickle
from flint import fmpz

n = fmpz(10 ** 10000)

for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
    try: 
        pickle.dumps(n, protocol=protocol)
        print('protocol', protocol, 'passed')
    except Exception as e:
        print('protocol', protocol, 'failed')
        print(e)

protocol 0 failed
Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
protocol 1 failed
Exceeds the limit (4300 digits) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit
protocol 2 passed
protocol 3 passed
protocol 4 passed
protocol 5 passed

we will have to fix any type that pickles ints

[deftasparagusanaconda]

this new version may be harder to maintain

[deftasparagusanaconda]

protocols 0 & 1 are still inefficient, because they cant pickle bytes efficiently. i wont improve this

import math, pickle
from flint import arb, ctx

ctx.dps = 1_000_000

for protocol in range(pickle.HIGHEST_PROTOCOL + 1):
    original = arb(2).sqrt().mid()
    pickled = pickle.dumps(original, protocol=protocol)
    roundtrip = pickle.loads(pickled)
    limit = math.ceil(ctx.dps * math.log2(10) / 8)

    assert original == roundtrip

    print('protocol         :', protocol)
    print('pickle size      :', len(pickled))
    print('theoretical limit:', limit)
    print('difference       :', len(pickled) - limit)
    print()

# protocol         : 0
# pickle size      : 455727
# theoretical limit: 415242
# difference       : 40485
#
# protocol         : 1
# pickle size      : 622855
# theoretical limit: 415242
# difference       : 207613
#
# protocol         : 2
# pickle size      : 415294
# theoretical limit: 415242
# difference       : 52
#
# protocol         : 3
# pickle size      : 415294
# theoretical limit: 415242
# difference       : 52
#
# protocol         : 4
# pickle size      : 415311
# theoretical limit: 415242
# difference       : 69
#
# protocol         : 5
# pickle size      : 415311
# theoretical limit: 415242
# difference       : 69