Issue #1144 rechunk projectfile data

[JoerivanEngelen]

Fixes #1144

Description

This PR builts further upon the ground work and research that went into: #1886 . So you can check that for additional context. I decided not to rechunk, as mentioned in the issue, as in earlier testing for #1886 I found out that rechunking still takes quite some time with very large graphs, as the huge task graph is first optimized by dask before rechunking, which doesn't scale well.

Most relevant changes are in imod/formats/idf.py

This PR adds adds a imod.formats.idf.open_chunked_by_time function, which opens a set of IDFs chunked by layer instead of by time. So the task graph is more optimized. A performance test is added for a set of 32_000 idfs (40 layers 800 timesteps). Opening and loading takes 23 seconds, compared to more than 10 minutes just calling imod.idf.open.

The imod.formats.prj.open_data routine calls the new function now .

I haven't added imod.formats.idf.open_chunked_by_time to the public API yet (so included in the docs and under the imod.idf namespace), as I'm not sure about the headers argument in the signature. What do you think? We could do it, and it could even replacereplace imod.idf.open, as it is a lot more performant in most cases. But that is something to be discussed with users.

Boyscouting:

• Add type annotation to idf.py module and enable mypy for this specific module

EDIT:
The 2 issues discovered by Sonartype about the cognitive overload of two functions in imod.formats are not new. This functions were untouched and not part of this PR.

Checklist

• Links to correct issue
• Update changelog, if changes affect users
• PR title starts with Issue #nr, e.g. Issue #737
• Unit tests were added
• If feature added: Added/extended example
• If feature added: Added feature to API documentation
• If pixi.lock was changed: Ran pixi run generate-sbom and committed changes

[ClaireDons]

Looks good, just a small nitpick about docstrings as usual. I think it makes sense to make this public in some way since it might be a significant performance increase in a lot of cases. If we don't want to change the default behaviour then we could add a "chunk_by" flag or so that makes this behaviour optional for users and document when to use it.

[ClaireDons]

Docstrings could be a little more descriptive about why you might want to choose this or its purpose. Especially because there are quite a few "open" methods.

[sonarqubecloud]

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[JoerivanEngelen]

Looks good, just a small nitpick about docstrings as usual. I think it makes sense to make this public in some way since it might be a significant performance increase in a lot of cases. If we don't want to change the default behaviour then we could add a "chunk_by" flag or so that makes this behaviour optional for users and document when to use it.

Right, I've added an option chunk_layout with options "per_idf" and "per_timestep" to the idf.open which does that. The old behavior is preserved by default.