[DRAFT] feat(compositor): Layers From Bounding Boxes adapter node - #15386
Conversation
Adapter node: IMAGE (batch) + BOUNDING_BOX (+ MASK, + LAYERS) -> LAYERS. One document item per frame, each placed by its own box. Why this is needed. A node that separates an image into elements emits them as an image batch plus a list of boxes. That batch cannot be fed to Create Layered Image with the placement intact, because `expand_item_frames` applies the item's single x/y/w/h/name/z_index to every frame of a batch. Sixteen layers get one placement between them. The current workaround is to have the producer pre-place each layer on a full-size canvas so x=0,y=0 is correct for all of them. That works but is expensive: at 2K with 16 layers it carries ~768MB of layer tensors and ~268MB of masks to encode what cropped layers hold in a fraction of it. This node emits one item per layer instead, so each carries its own placement. It needs no change to the existing compositor path: `document_items` already sorts by z_index, and `expand_item_frames` already handles single-frame items correctly - the shared-placement limitation only bites on batches. Reads `metadata.name`, `metadata.z_index` and `metadata.content_rect` where present. `crop_to_content` trims each frame out of a padded batch and places it by its box, which is what recovers the memory win. Also restores a `_bbox_list` parser. The equivalent (`_bbox_entries`, `layout_bboxes`, `state_from_bboxes`) was removed in 1c4953f along with the rest of the bbox handling when the node moved to the LAYERS document, so there is currently no path from a bounding box into the compositor. Verified: two padded layers with different content sizes, cropped and placed independently, composite to within 0.50/255 of expectation - the 8-bit quantisation floor of the PIL round trip inside the compositor. The 76 existing compositor tests still pass.
|
🎉 Thank you for your contribution, we really appreciate it! 🎉 Like many open source projects, we require contributors to sign our Contributor License Agreement (CLA). A CLA makes the ownership of contributions explicit, so contributors and the project share a clear understanding of how the code can be used. By signing, you:
CLAs are standard practice across major open source projects including those under the Apache Software Foundation and the Linux Foundation. Ours is based on the Apache Software Foundation's CLA. Most importantly, it would enable us to relicense the project under a more permissive license in the future, giving the project and its community greater flexibility. ✍ To sign, please post a new comment on this PR with exactly the following text: ✍ I have read and agree to the Contributor License Agreement You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
2 participants
Adapter node:
IMAGE (batch) + BOUNDING_BOX (+ MASK, + LAYERS) → LAYERS. One document item per frame, each placed by its own box.The gap
A node that separates an image into elements emits them as an image batch plus a list of boxes. That batch cannot drive Create Layered Image with placement intact —
expand_item_framesapplies the item’s singlex/y/w/h/name/z_indexto every frame of a batch. Sixteen layers get one placement between them.The workaround today is for the producer to pre-place each layer on a full-size canvas so
x=0, y=0is correct for all of them. It works, but at 2K with 16 layers it carries ~768MB of layer tensors plus ~268MB of masks to encode what cropped layers hold in a fraction of that — and it leaves minimal-size mode unusable with the compositor.Why this shape
One item per layer, so each carries its own placement. No change to the existing compositor path —
document_itemsalready sorts byz_indexandexpand_item_framesalready handles single-frame items correctly; the shared-placement limitation only bites on batches. Pure addition, so it cannot regress what ships.Reads
metadata.name,metadata.z_indexandmetadata.content_rectwhere present.crop_to_contenttrims each frame out of a padded batch and places it by its box — that is what recovers the memory win.Generalises beyond one producer:
CreateBoundingBoxesand any future layer-producing model get compositor support without each rebuilding the document by hand.Also restores
_bbox_list_bbox_entries,layout_bboxesandstate_from_bboxeswere removed in1c4953ffalong with the rest of the bbox handling when the node moved to the LAYERS document. There is currently no path from a bounding box into the compositor, and #15317’s description still advertises the removedbboxesinput.Verified
Two padded layers with different content sizes, cropped and placed independently:
The 76 existing compositor tests still pass.
Draft for @jtydhr88 — take it, fold it in, or tell me the per-frame-override approach in
expand_item_framesis the better shape. cc @christian-byrne @bigcat88