MINIFICPP-2892 Tensor extensions based on sonos/tract by martinzink · Pull Request #2258 · apache/nifi-minifi-cpp

martinzink · 2026-09-09T10:24:04Z

Thank you for submitting a contribution to Apache NiFi - MiNiFi C++.

In order to streamline the review of the contribution we ask you to ensure the following steps have been taken:

For all changes:

Is there a JIRA ticket associated with this PR? Is it referenced in the commit message?
Does your PR title start with MINIFICPP-XXXX where XXXX is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character.
Has your PR been rebased against the latest commit within the target branch (typically main)?
Is your initial contribution a single, squashed commit?

For code changes:

If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE file?
If applicable, have you updated the NOTICE file?

For documentation related changes:

Have you ensured that format looks appropriate for the output in which it is rendered?

Note:

Please ensure that once the PR is submitted, you check GitHub Actions CI results for build issues and submit an update to your PR as soon as possible.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

This PR introduces a new minifi_tensor Rust extension (based on sonos/tract) that adds image-to-tensor preprocessing and Tract-based inference processors/services, and updates the build packaging to support it.

Changes:

Added minifi_tensor extension with a Tract-backed controller service and processors for classification, detection, tensor preprocessing, and post-processing.
Enhanced build tooling (Dockerfiles + CI build script) to build with all features and leverage GitHub Actions buildx cache.
Updated extension registration macro to support #[cfg(...)] / attributes on declared processors/controllers.

Reviewed changes

Copilot reviewed 41 out of 41 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
minifi_rust/minifi_rs_behave/linux_build.sh	Enables buildx GHA caching for faster CI builds.
minifi_rust/minifi_rs_behave/Dockerfile.debian	Updates Debian base and builds Rust artifacts with `--all-features`.
minifi_rust/minifi_rs_behave/Dockerfile.alpine	Builds Rust artifacts with `--all-features`.
minifi_rust/minifi_rs_behave/Cargo.toml	Removes playground dependency from behave crate.
minifi_rust/minifi_native/src/lib.rs	Extends `declare_minifi_extension!` to allow optional attributes per processor/controller.
minifi_rust/extensions/minifi_tensor/src/utils/tensor_helpers.rs	Adds tensor deserialization helpers and image loading helpers.
minifi_rust/extensions/minifi_tensor/src/utils/score_activation.rs	Adds `ScoreActivation` enum for classification/detection post-processing.
minifi_rust/extensions/minifi_tensor/src/utils/per_channel_f32.rs	Adds per-channel float parsing for mean/std normalization properties.
minifi_rust/extensions/minifi_tensor/src/utils/mod.rs	Wires up the new `utils` module tree.
minifi_rust/extensions/minifi_tensor/src/utils/dimensions.rs	Adds helper struct for image/original/target dimensions.
minifi_rust/extensions/minifi_tensor/src/utils/bounding_box.rs	Adds bounding-box model, NMS/IoU, drawing, and property parsing.
minifi_rust/extensions/minifi_tensor/src/services/tract_model_service/service_definition.rs	Defines properties and definition for `TractModelService`.
minifi_rust/extensions/minifi_tensor/src/services/tract_model_service.rs	Implements Tract model loading and inference execution.
minifi_rust/extensions/minifi_tensor/src/services/mod.rs	Wires up the new `services` module tree.
minifi_rust/extensions/minifi_tensor/src/processors/mod.rs	Wires up the new processor modules.
minifi_rust/extensions/minifi_tensor/src/processors/draw_bounding_box.rs	Adds processor to draw bounding boxes and output an annotated PNG.
minifi_rust/extensions/minifi_tensor/src/processors/detect_object/detect_object_def.rs	Defines DetectObject processor properties/relationships.
minifi_rust/extensions/minifi_tensor/src/processors/detect_object.rs	Implements DetectObject as a composed processor pipeline.
minifi_rust/extensions/minifi_tensor/src/processors/classify_image/classify_object_def.rs	Defines ClassifyImage processor properties/relationships.
minifi_rust/extensions/minifi_tensor/src/processors/classify_image.rs	Implements ClassifyImage as a composed processor pipeline.
minifi_rust/extensions/minifi_tensor/src/low_level_processors/mod.rs	Adds low-level processor modules (some behind a feature flag).
minifi_rust/extensions/minifi_tensor/src/low_level_processors/invoke_tract_model/invoke_tract_model_def.rs	Defines InvokeTractModel processor properties/relationships.
minifi_rust/extensions/minifi_tensor/src/low_level_processors/invoke_tract_model.rs	Implements low-level InvokeTractModel processor.
minifi_rust/extensions/minifi_tensor/src/low_level_processors/image_to_tensor/image_to_tensor_def.rs	Defines ImageToTensor processor properties/relationships.
minifi_rust/extensions/minifi_tensor/src/low_level_processors/image_to_tensor.rs	Implements image preprocessing into model-ready tensors.
minifi_rust/extensions/minifi_tensor/src/low_level_processors/filter_bounding_boxes/filter_bounding_boxes_def.rs	Defines FilterBoundingBoxes processor properties/relationships.
minifi_rust/extensions/minifi_tensor/src/low_level_processors/filter_bounding_boxes.rs	Implements detection post-processing (activation, decode, NMS, output).
minifi_rust/extensions/minifi_tensor/src/low_level_processors/classify_output/classify_output_def.rs	Defines ClassifyOutput processor properties/relationships.
minifi_rust/extensions/minifi_tensor/src/low_level_processors/classify_output.rs	Implements classification post-processing (activation, Top-K, labels).
minifi_rust/extensions/minifi_tensor/src/lib.rs	Registers new processors and controller service in the extension.
minifi_rust/extensions/minifi_tensor/minifi_tensor.md	Adds user-facing documentation for processors/services.
minifi_rust/extensions/minifi_tensor/features/steps/steps.py	Wires behave step imports for the extension’s feature tests.
minifi_rust/extensions/minifi_tensor/features/resources/.gitignore	Prevents committed test-download artifacts.
minifi_rust/extensions/minifi_tensor/features/image_to_tensor.feature	Adds behave tests for ImageToTensor scenarios.
minifi_rust/extensions/minifi_tensor/features/environment.py	Adds behave hook to download models/images/labels on first run.
minifi_rust/extensions/minifi_tensor/features/detection.feature	Adds behave face-detection scenarios (pipeline + DetectObject).
minifi_rust/extensions/minifi_tensor/features/classification.feature	Adds behave classification scenarios (pipeline + ClassifyImage).
minifi_rust/extensions/minifi_tensor/features/basic.feature	Adds behave smoke test for extension registration.
minifi_rust/extensions/minifi_tensor/Cargo.toml	New crate manifest for the `minifi_tensor` extension.
minifi_rust/extensions/minifi_rs_playground/minifi_rs_playground.md	Updates playground docs (adds dummy controller service row).
.github/references/ubuntu_22_04_clang_arm_manifest.json	Updates component manifest reference to include `minifi_tensor`.

Suppressed comments (3)

minifi_rust/extensions/minifi_tensor/src/utils/tensor_helpers.rs:1

deserialize_tensors can panic on &flow_file_contents[cursor..cursor + tensor_len] if attributes are inconsistent with the payload length (truncated content or malformed tensor.{i}.bytes). This should be validated and return a MinifiError instead of panicking: check cursor + tensor_len <= flow_file_contents.len() before slicing (and consider also rejecting extra trailing bytes after the last tensor if that’s not allowed).
minifi_rust/extensions/minifi_tensor/src/utils/tensor_helpers.rs:1
deserialize_tensors introduces new parsing logic (length/shape/dtype attributes + payload slicing) but has no unit tests in this module, while other helpers here are tested. Adding tests for: (1) valid multi-tensor payload, (2) truncated payload (should error, not panic), and (3) invalid dtype/shape strings would prevent regressions.
minifi_rust/extensions/minifi_tensor/src/utils/bounding_box.rs:1
draw_onto casts potentially user-supplied normalized coordinates (via JSON in DrawBoundingBox) directly to u32. Negative values or NaNs will wrap/produce unexpected large coordinates and can lead to invalid rectangles (and as i32 conversions inside drawing) with unpredictable behavior. Clamp/sanitize x_min/y_min/x_max/y_max to finite values and to [0.0, 1.0] (and ensure min <= max) before converting to pixel coordinates.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Pull request overview

Copilot reviewed 42 out of 42 changed files in this pull request and generated 5 comments.

Suppressed comments (3)

minifi_rust/extensions/minifi_tensor/src/utils/tensor_helpers.rs:1

deserialize_tensors can panic on truncated/malformed payloads because it slices flow_file_contents[cursor..cursor + tensor_len] without verifying bounds. Please add an explicit check that cursor + tensor_len <= flow_file_contents.len() (and ideally that the final cursor equals the payload length, or at least does not exceed it) and return a MinifiError instead of panicking.
minifi_rust/extensions/minifi_tensor/src/utils/bounding_box.rs:1
IoU can divide by zero if either box has zero area (or if both areas cancel such that the union is 0), yielding NaN/inf and destabilizing NMS. Add an explicit guard for union_area <= 0.0 (or non-finite) and return 0.0 in that case.
minifi_rust/minifi_rs_behave/linux_build.sh:1
This uses bash arrays (CACHE_ARGS=(), CACHE_ARGS+=(...), and later ${CACHE_ARGS[@]}). If linux_build.sh is executed under sh (or a non-bash shell), it will fail. Ensure the script’s shebang is bash (e.g. #!/usr/bin/env bash) and that CI invokes it accordingly, or rework CACHE_ARGS to be POSIX-shell compatible.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

martinzink · 2026-09-09T10:25:30Z

 mkdir -p "$TARGET_DIR"

 # 3. Build using Docker
+# When running under GitHub Actions, persist the chef layer across runs


Couldnt allow the actions yet because it needs to be enabled in apache org, it still needs this actions

- name: Set up Docker Buildx uses: docker/setup-buildx-action@v3 - name: Expose GitHub Actions cache to buildx uses: crazy-max/ghaction-github-runtime@v3

szaszm

I've reviewed to the best of my ability, but I'm not a domain expert in ML, and there is a lot of logic here, so I couldn't do a thorough review.

fgerlits

I haven't got to the end, yet, but here is a first batch of nitpicks

lordgamez · 2026-09-24T10:49:04Z

These could also be added to PROCESSORS.md

👍 docs update

lordgamez · 2026-09-24T11:11:48Z

Missing license header here and additionally in minifi_rust/extensions/minifi_tensor/src/services/tract_model_service.rs, minifi_rust/extensions/minifi_tensor/src/utils/dimensions.rs, minifi_rust/extensions/minifi_tensor/src/utils/per_channel_f32.rs, minifi_rust/extensions/minifi_tensor/src/utils/score_activation.rs, minifi_rust/extensions/minifi_tensor/src/utils/tensor_helpers.rs, minifi_rust/extensions/minifi_tensor/src/low_level_processors/mod.rs, minifi_rust/extensions/minifi_tensor/src/processors/draw_bounding_box.rs, minifi_rust/extensions/minifi_tensor/src/services/tract_model_service.rs, minifi_rust/extensions/minifi_tensor/src/utils/bounding_box.rs, minifi_rust/extensions/minifi_tensor/src/utils/dimensions.rs, minifi_rust/extensions/minifi_tensor/src/lib.rs files

👍 changed here added missing apache license (and other commits couldnt separate them fully)

lordgamez · 2026-09-24T11:35:36Z

+        .ok()
+        .flatten()
+        .and_then(|raw| raw.parse::<ResizeMode>().ok())
+        .unwrap_or(ResizeMode::Letterbox)


I am not sure if this is intentional, but in image_to_tensor_def.rs we define the default for RESIZE_MODE to ResizeMode::Stretch, but here the default is ResizeMode::Letterbox

👍 it shouldnt matter because its should be present there, but changed it for consistencys sake 8446df5

lordgamez · 2026-09-24T11:42:22Z

+        };
+
+        let mut transformed = TransformedFlowFile::new(&SUCCESS, content)
+            .with_attribute("mime.type", "application/json")


The output attribute description says "Always 'application/json'", but if the output is written into the attribute instead of the flow file content, should that be the case here?

8446df5#diff-62fccab1592a7a96962a31921fe61d7751cc946ee62535444c8d7e834ea72d1bR101

lordgamez · 2026-09-24T11:51:28Z

+    const INPUT_REQUIREMENT: ProcessorInputRequirement = ProcessorInputRequirement::Required;
+    const SUPPORTS_DYNAMIC_PROPERTIES: bool = false;
+    const SUPPORTS_DYNAMIC_RELATIONSHIPS: bool = false;
+    const OUTPUT_ATTRIBUTES: &'static [OutputAttribute] = &[


TENSOR_BYTES_ATTR seems to be missing here

lordgamez · 2026-09-24T12:02:11Z

+        };
+
+        if !box_floats.len().is_multiple_of(4) {
+            return Err(MinifiError::custom(


should we route to failure here instead? Maybe I am misunderstanding, but if this returns an error, wouldn't that just rollback the flow file forever as this would not change? Same for line 246 and 285

good catch, 8446df5#diff-3c22174e4b301eddbaa1696ac94676d946ca047481663ef77f6b32d2509fa7efR245

lordgamez · 2026-09-24T12:03:07Z

+        let score_floats =
+            tensor_as_f32(&tensors, self.score_output_index).route_err_to_failure()?;
+        if score_floats.is_empty() {
+            return Err(MinifiError::custom("Score tensor is empty; nothing to classify").into());


Same as my other comment in filter_bounding_boxes.rs, would this just rollback forever?

8446df5#diff-4daa3621f9a08dfc1054a44196aebd2a6f86396fb7f4a114c5211dc8e8760038R141

…put_attribute to filter_bounding_boxes.rs

-fixed math mismatch between letterbox into/from -added passwords \n delimited instead of password -fixed possible (but inlikely) overflow -added output_attrs to classify_image

…(SSD)

route_to_failure changes,

Co-authored-by: Ferenc Gerlits <fgerlits@users.noreply.github.com>

(also adding missing RunLlamacpp docs)

fgerlits · 2026-09-25T15:38:28Z

+    pub(crate) fn target_from_attributes<Context: GetAttribute>(
+        context: &Context,
+    ) -> Result<Dimensions, MinifiError> {
+        let orig_w = context
+            .get_required_attribute("image.target.width")?
+            .parse::<f32>()?;
+
+        let orig_h = context
+            .get_required_attribute("image.target.height")?
+            .parse::<f32>()?;
+
+        Ok(Dimensions {
+            width: orig_w,
+            height: orig_h,
+        })
+    }


very minor copy-paste issue:

Suggested change

pub(crate) fn target_from_attributes<Context: GetAttribute>(

context: &Context,

) -> Result<Dimensions, MinifiError> {

let orig_w = context

.get_required_attribute("image.target.width")?

.parse::<f32>()?;

let orig_h = context

.get_required_attribute("image.target.height")?

.parse::<f32>()?;

Ok(Dimensions {

width: orig_w,

height: orig_h,

})

}

pub(crate) fn target_from_attributes<Context: GetAttribute>(

context: &Context,

) -> Result<Dimensions, MinifiError> {

let target_w = context

.get_required_attribute("image.target.width")?

.parse::<f32>()?;

let target_h = context

.get_required_attribute("image.target.height")?

.parse::<f32>()?;

Ok(Dimensions {

width: target_w,

height: target_h,

})

}

fgerlits · 2026-09-25T16:05:41Z

+        let divisor = box1_area + box2_area - intersection_area;
+
+        if intersection_area == 0f32 || divisor == 0f32 {
+            return 0f32;
+        }
+
+        intersection_area / divisor


Extreme nitpicking, but it hurts my eyes that this variable is not called union_area:

Suggested change

let divisor = box1_area + box2_area - intersection_area;

if intersection_area == 0f32 || divisor == 0f32 {

return 0f32;

}

intersection_area / divisor

let union_area = box1_area + box2_area - intersection_area;

if intersection_area == 0f32 || union_area == 0f32 {

return 0f32;

}

intersection_area / union_area

martinzink requested a balanced review from Copilot September 9, 2026 10:24

Copilot AI reviewed Sep 9, 2026

View reviewed changes

martinzink force-pushed the minifi_rust_tensor branch from 0977b92 to 7953915 Compare September 9, 2026 10:47

martinzink requested a balanced review from Copilot September 9, 2026 11:49

Copilot AI reviewed Sep 9, 2026

View reviewed changes

Copilot AI reviewed Sep 9, 2026

martinzink added stacked-PR rust labels Sep 9, 2026

martinzink force-pushed the minifi_rust_tensor branch from d554ada to 983ad0f Compare September 9, 2026 12:51

Copilot started reviewing on behalf of martinzink September 9, 2026 12:56 View session

Copilot started reviewing on behalf of martinzink September 9, 2026 14:08 View session

Copilot started reviewing on behalf of martinzink September 9, 2026 14:23 View session

szaszm reviewed Sep 9, 2026

View reviewed changes

martinzink marked this pull request as ready for review September 10, 2026 08:50

martinzink commented Sep 10, 2026

View reviewed changes

martinzink added the priority Review these first label Sep 10, 2026

martinzink force-pushed the minifi_rust_tensor branch from 22d1f09 to 63fab4a Compare September 11, 2026 13:14

szaszm force-pushed the minifi_rust_impr_2 branch from 406ef78 to 54d33fd Compare September 14, 2026 10:43

martinzink force-pushed the minifi_rust_tensor branch from 67eb052 to 9cc165b Compare September 14, 2026 13:54

martinzink changed the base branch from minifi_rust_impr_2 to main September 14, 2026 13:54

martinzink removed the stacked-PR label Sep 14, 2026

szaszm reviewed Sep 23, 2026

View reviewed changes

fgerlits self-requested a review September 24, 2026 08:36

adamdebreceni self-requested a review September 24, 2026 09:04

fgerlits reviewed Sep 24, 2026

View reviewed changes

adamdebreceni reviewed Sep 24, 2026

View reviewed changes

Comment thread minifi_rust/extensions/minifi_tensor/src/processors/draw_bounding_box.rs Outdated

lordgamez reviewed Sep 24, 2026

View reviewed changes

fgerlits reviewed Sep 25, 2026

View reviewed changes

adamdebreceni approved these changes Sep 25, 2026

View reviewed changes

martinzink and others added 28 commits September 25, 2026 18:14

review changes

cb351bb

fix

5ed98d2

fix panic on malformed tensor payload in tensor_helpers.rs

3925821

fix doc drift in descriptions

56900af

fix 0/0 in IoU

02b0095

classify_image.rs Failure description fix

a613332

remove code duplication from InvokeTract model and add result_via_out…

7fa3863

…put_attribute to filter_bounding_boxes.rs

rebase fix

2e23dd8

Cargo.toml doesnt need explicit all features

22afa6f

fix image_to_tensor.rs

fedbf16

respect original image format in draw_bounding_box.rs

76e7278

rebase fix

a35d641

fixing opus 5's findings

d6e39f2

-fixed math mismatch between letterbox into/from -added passwords \n delimited instead of password -fixed possible (but inlikely) overflow -added output_attrs to classify_image

trim labels both direction

64eebb7

adding missing TENSOR_BYTES_ATTR to image_to_tensor_def.rs

0d344a1

"certifi>=2026.7.22"

dea51ec

Feature: Face detection with UltraFace Single Shot MultiBox Detector …

92c3020

…(SSD)

added missing apache license

db3d02c

output attrs changes,

ab06768

route_to_failure changes,

LineColor changes

d06f0c3

labelsproperty

01eb67e

Apply batched suggestions from code review

9bafc8d

Co-authored-by: Ferenc Gerlits <fgerlits@users.noreply.github.com>

#[strum(serialize = "Grayscale")]

890708a

sigmoid tests in classify_output.rs

cd390d7

move test_resources into target folder

afb709d

folded label_index_offset into labels

0a380d8

docs update

281338c

(also adding missing RunLlamacpp docs)

refresh manifest

c1275a6

martinzink force-pushed the minifi_rust_tensor branch from 4df63e5 to c1275a6 Compare September 25, 2026 16:14

fgerlits reviewed Sep 25, 2026

View reviewed changes

Conversation

martinzink commented Sep 9, 2026

For all changes:

For code changes:

For documentation related changes:

Note:

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

szaszm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fgerlits left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment