Inference and preprocessing#
YoloDetector::detect_frame accepts a gstreamer::BufferInfo, converts its pixels to the
model input and maps detections back to the original frame. detect_from_gstreamer and
VideoDetectorPipeline both use this path. detect is the lower-level interface for
callers that already have normalized RGB NCHW floats; its width and height describe the
original image for stretch coordinate mapping.
Frame memory#
The frame metadata must name RGB or RGBA and nonzero width and height. Alpha is ignored.
row_stride_bytes describes the distance between row starts; zero means tightly packed rows.
plane_offset_bytes locates the first pixel in BufferInfo::data. GStreamer extraction
reads these values from video information and uses per-buffer video metadata when present.
The conversion checks row length, offset, available bytes and size arithmetic before reading.
Negative stride, incomplete buffers and zero dimensions return InvalidInputShape;
unsupported pixel formats return InvalidInputType. A padded RGB row must not be treated as
width * 3 bytes, and a buffer of bytes must never be reinterpreted as a float tensor.
Resize and mapping#
YoloConfig::preprocess_mode selects FramePreprocessMode::Stretch (the default) or
FramePreprocessMode::Letterbox. Both sample with nearest-neighbour floor indexing, divide
RGB bytes by 255 and produce [1, 3, input_height, input_width] in NCHW order.
Stretch resizes each axis independently. Letterbox preserves aspect ratio, rounds and clamps
the resized dimensions, then pads with 114/255. Integer padding puts an extra pixel on the
right or bottom. PreprocessedFrame::mapping records the actual per-axis scale and padding;
detect_frame uses it to undo padding and scaling when decoding boxes. Use this frame path
when letterbox coordinate mapping is needed.
The model’s export determines both preprocessing and output layout. For ModelANT’s
DEIM-D-FINE-N choose stretch and YoloOutputLayout::CornersScoreClass: its rows are
x1, y1, x2, y2, score, class in model-input pixels. The legacy
CenterObjectnessScores layout remains the default for callers using that YOLO export.
These configuration choices are explicit; the detector does not infer them from a byte count.
Output and error handling#
OnnxInferenceEngine returns dense FLOAT tensors. It checks the ONNX value kind, element
type, dimensions and data pointer before copying output storage. Integer, byte, boolean,
sequence, map and other unsupported outputs return InvalidOutputType. Invalid dimensions
return InvalidOutputShape; a failed data access returns OutputAllocationFailed.
Single-input and multi-input inference propagate these errors through std::expected.
The Python bindings export the new error names and report failed operations as exceptions.
For asynchronous video, register VideoDetectorPipeline::set_error_callback to receive
failed-frame errors and inspect last_error() for the most recent failure. Callbacks run on
the streaming thread. start() clears the stored error; subsequent successful frames do not.
Without a callback, a changed error is printed to stderr instead of being silently discarded.
Validation#
The commit suite covers padded RGB rows, plane offsets, channel order, normalization,
letterbox mapping, shared ModelANT cases, real videotestsrc frame dimensions and asynchronous
failure reporting. ONNX tests generate unsupported output kinds and types without downloading
model weights.
Run the suite in the family image:
cmake --preset linux-debug-clang
cmake --build --preset linux-debug-clang --target commitTestSuite
ctest --test-dir build --no-tests=error --output-on-failure -R 'FramePreprocessing|YoloDetector|Onnx'
When changing bound errors or inference behavior, also run the bindings suite against a built package, as described in Getting Started.