Mf6Splitter writes structured sub-models from a structured parent, so each
sub-model is the bounding box of its part and carries every inactive cell inside
it. A DISV sub-model would hold only the cells it solves, which removes the
padding and, with it, the tension between contiguous partitions and small
sub-models described in #2814.
For a 210,771,902 cell CONUS model, 124,884,583 active, split into 256 parts:
|
allocated cells |
inactive |
fill |
structured, active_only=False |
311,005,557 |
186,120,974 |
40.2% |
structured, active_only=True |
381,616,033 |
256,731,450 |
32.7% |
| DISV |
124,884,583 |
0 |
100% |
The DISV sub-models run in less memory than either structured split, and less
than the model in production use: 53.8 GB per node at 32 ranks, against 56 GB
for active_only=False and more than 229 GB for active_only=True, which was
OOM killed during GWF-GWF exchange setup on nodes of that size.
Nothing about the construction needs the whole model in memory, which is the
other reason to have it in the splitter rather than downstream of it. Building
the 256 DISV sub-models directly from the splitting array took 70 minutes and a
few GB, against 4 h 15 m and 320 GB peak for split_model on the same
partition. The grid of a part is arithmetic on the parent lattice: the cell
vertices are the four corners of each cell, shared corners collapse to about one
vertex per cell, and the exchanges are the cell pairs that share a face between
two parts, which also gives NEXG, CL1, CL2, HWVA and ANGLDEGX directly.
Two details worth recording for anyone implementing it:
- MODFLOW 6 requires the CELL2D vertices in clockwise order. In map coordinates
with row increasing southward that is top left, top right, bottom right,
bottom left. Counterclockwise ordering is accepted by flopy and then rejected
by MODFLOW with Calculated CELL2D area less than zero for every cell.
- The sub-models have no inactive cells, so IDOMAIN is constant 1 and
nodes
equals nodesuser.
Related to #2813 and #2814.
Mf6Splitterwrites structured sub-models from a structured parent, so eachsub-model is the bounding box of its part and carries every inactive cell inside
it. A DISV sub-model would hold only the cells it solves, which removes the
padding and, with it, the tension between contiguous partitions and small
sub-models described in #2814.
For a 210,771,902 cell CONUS model, 124,884,583 active, split into 256 parts:
active_only=Falseactive_only=TrueThe DISV sub-models run in less memory than either structured split, and less
than the model in production use: 53.8 GB per node at 32 ranks, against 56 GB
for
active_only=Falseand more than 229 GB foractive_only=True, which wasOOM killed during GWF-GWF exchange setup on nodes of that size.
Nothing about the construction needs the whole model in memory, which is the
other reason to have it in the splitter rather than downstream of it. Building
the 256 DISV sub-models directly from the splitting array took 70 minutes and a
few GB, against 4 h 15 m and 320 GB peak for
split_modelon the samepartition. The grid of a part is arithmetic on the parent lattice: the cell
vertices are the four corners of each cell, shared corners collapse to about one
vertex per cell, and the exchanges are the cell pairs that share a face between
two parts, which also gives NEXG, CL1, CL2, HWVA and ANGLDEGX directly.
Two details worth recording for anyone implementing it:
with row increasing southward that is top left, top right, bottom right,
bottom left. Counterclockwise ordering is accepted by flopy and then rejected
by MODFLOW with
Calculated CELL2D area less than zerofor every cell.nodesequals
nodesuser.Related to #2813 and #2814.