Skip to content

Feat: add item_transform_pad for detection and segmentation items - #373

Open
DerrickUnleashed wants to merge 51 commits into
mlverse:mainfrom
DerrickUnleashed:feat/padTransform
Open

Feat: add item_transform_pad for detection and segmentation items#373
DerrickUnleashed wants to merge 51 commits into
mlverse:mainfrom
DerrickUnleashed:feat/padTransform

Conversation

@DerrickUnleashed

Copy link
Copy Markdown
Contributor

Added item_transform_pad for detection item, dataset and segmentation item, dataset

Closes #352

DerrickUnleashed and others added 30 commits July 27, 2026 15:08
Both item_transform_hflip.dataset and target_transform_rotate_box.dataset
need to unlock the R6 .getitem binding before reassigning it.
Co-authored-by: cregouby <cregouby@users.noreply.github.com>
- Rename test file to test-item-transforms-geometry.R (plural)
- Rename make_item → make_detection_item() in hflip tests
- Merge same-input tests for efficiency
- Add composition test for item_transform_hflip
- Add item_transform_hflip to main's R/item-transforms-geometry.R
- Resolve all merge conflicts
Co-authored-by: cregouby <cregouby@users.noreply.github.com>
Co-authored-by: cregouby <cregouby@users.noreply.github.com>
@DerrickUnleashed

Copy link
Copy Markdown
Contributor Author
url <- "https://upload.wikimedia.org/wikipedia/commons/b/b6/Felis_catus-cat_on_snow.jpg"

# ========== 1. DETECTION ITEM ==========
img <- base_loader(url) |> transform_to_tensor()
boxes <- torch_tensor(matrix(c(600, 200, 2880, 1860), ncol = 4), dtype = torch_float32())
det_item <- list(x = img, y = list(boxes = boxes, labels = "CAT"))
class(det_item) <- c("image_with_bounding_box", "list")

det_padded <- item_transform_pad(det_item, padding = 100)

p1 <- draw_bounding_boxes(det_item, colors = "blue", width = 10)$to(torch_float())$div(255)
p2 <- draw_bounding_boxes(det_padded, colors = "red", width = 10)$to(torch_float())$div(255)
p1 <- transform_resize(p1, c(300, 300))
p2 <- transform_resize(p2, c(300, 300))
g1 <- vision_make_grid(torch_stack(list(p1, p2)), scale = TRUE)
tensor_image_browse(g1)

# ========== 2. DETECTION DATASET ==========
ds_det <- pascal_detection_dataset(year = "2007", split = "trainval",
                                   transform = transform_to_tensor, download = TRUE)
orig_det <- ds_det[1]
ds_det_padded <- item_transform_pad(ds_det, padding = c(50, 50, 50, 50))
pad_det <- ds_det_padded[1]

p3 <- draw_bounding_boxes(orig_det, colors = "blue", width = 5)$to(torch_float())$div(255)
p4 <- draw_bounding_boxes(pad_det, colors = "red", width = 5)$to(torch_float())$div(255)
p3 <- transform_resize(p3, c(300, 300))
p4 <- transform_resize(p4, c(300, 300))
g2 <- vision_make_grid(torch_stack(list(p3, p4)), scale = TRUE)
tensor_image_browse(g2)

# ========== 3. SEGMENTATION ITEM ==========
img2 <- base_loader(url) |> transform_to_tensor()
h <- img2$shape[2]; w <- img2$shape[3]

mask1 <- torch_zeros(h, w, dtype = torch_bool())
mask1[, 1:150] <- TRUE

mask2 <- torch_zeros(h, w, dtype = torch_bool())
mask2[, 350:500] <- TRUE

masks <- torch_stack(list(mask1, mask2))
seg_item <- list(x = img2, y = list(masks = masks, labels = torch_tensor(c(1L, 2L)),
                                    image_height = h, image_width = w))
class(seg_item) <- c("image_with_segmentation_mask", "list")

seg_padded <- item_transform_pad(seg_item, padding = 100)

p5 <- draw_segmentation_masks(seg_item, alpha = 0.5, colors = c("red", "blue"))$to(torch_float())$div(255)
p6 <- draw_segmentation_masks(seg_padded, alpha = 0.5, colors = c("red", "blue"))$to(torch_float())$div(255)
p5 <- transform_resize(p5, c(300, 300))
p6 <- transform_resize(p6, c(300, 300))
g3 <- vision_make_grid(torch_stack(list(p5, p6)), scale = TRUE)
tensor_image_browse(g3)

# ========== 4. SEGMENTATION DATASET ==========
ds_seg <- pascal_segmentation_dataset(year = "2007", split = "trainval",
                                      transform = transform_to_tensor, download = TRUE)
orig_seg <- ds_seg[1]
ds_seg_padded <- item_transform_pad(ds_seg, padding = c(50, 50, 50, 50))
pad_seg <- ds_seg_padded[1]

p7 <- draw_segmentation_masks(orig_seg, alpha = 0.5)$to(torch_float())$div(255)
p8 <- draw_segmentation_masks(pad_seg, alpha = 0.5)$to(torch_float())$div(255)
p7 <- transform_resize(p7, c(300, 300))
p8 <- transform_resize(p8, c(300, 300))
g4 <- vision_make_grid(torch_stack(list(p7, p8)), scale = TRUE)
tensor_image_browse(g4)

DETECTION ITEM

file46d5a8ced36

DETECTION DATASET

file46d5a29a2ab

SEGMENTATION ITEM

file46d6376b6cb

SEGMENTATION DATASET

file46d6dafc71

@DerrickUnleashed
DerrickUnleashed marked this pull request as ready for review July 31, 2026 20:24

@cregouby cregouby left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Praise Thanks for this !

Comment thread NEWS.md
Comment on lines +23 to +24
* Added `item_transform_center_crop()` for cropping images from the center to a specified size for dataset items, with support for detection and segmentation item types and datasets (@DerrickUnleashed, #370).
* Added `item_transform_crop()` for cropping dataset items at a specified location and size, with support for detection and segmentation item types and datasets (@DerrickUnleashed, #371).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo This cannot be part of #373 as it is already in #370 and #371, and thus prevents the independant merging of this P.R.
suggestion you should derive a new branch from main, not from the previous branch you where working on to keep them independant.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ohh yes that would be better from the next PR onwards I'll cut branches from main

Comment thread NAMESPACE
Comment on lines +12 to +20
S3method(item_transform_center_crop,dataset)
S3method(item_transform_center_crop,default)
S3method(item_transform_center_crop,image_with_bounding_box)
S3method(item_transform_center_crop,image_with_rotated_box)
S3method(item_transform_center_crop,image_with_segmentation_mask)
S3method(item_transform_crop,dataset)
S3method(item_transform_crop,default)
S3method(item_transform_crop,image_with_bounding_box)
S3method(item_transform_crop,image_with_segmentation_mask)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo This cannot be part of #373 as it is already in #370 and #371, and thus prevents the independant merging of this P.R.

Comment on lines +631 to +976

# --- item_transform_center_crop tests ---

test_that("item_transform_center_crop rejects non-item inputs", {
img <- torch_randn(3, 100, 200)
expect_error(
item_transform_center_crop(img, size = 50),
"requires a dataset item"
)
expect_error(
item_transform_center_crop(42, size = 50),
"requires a dataset item"
)
})

test_that("item_transform_center_crop crops detection items and adjusts targets", {
boxes <- matrix(c(120, 70, 180, 130), ncol = 4)
item <- make_detection_item(boxes, image_size = c(200L, 400L))
original_img <- item$x$clone()
original_img_r <- as_array(item$x)
original_class <- class(item)
original_dtype <- item$x$dtype

result <- item_transform_center_crop(item, size = c(100L, 200L))

expect_s3_class(result, "image_with_bounding_box")
expect_tensor_dtype(result$x, original_dtype)
expect_tensor_shape(result$x, c(3, 100, 200))
expect_equal(result$y$image_height, 100L)
expect_equal(result$y$image_width, 200L)
expect_true(torch_equal(result$x, transform_center_crop(original_img, size = c(100L, 200L))))
expect_equal_to_r(result$y$boxes[1, 1], 120 - 99) # x1
expect_equal_to_r(result$y$boxes[1, 3], 180 - 99) # x2
expect_equal_to_r(result$y$boxes[1, 2], 70 - 49) # y1
expect_equal_to_r(result$y$boxes[1, 4], 130 - 49) # y2

# input is not mutated
expect_equal_to_r(item$x, original_img_r)
expect_equal_to_r(item$y$boxes, boxes)
expect_equal(class(item), original_class)

# labels and metadata are preserved
labels <- torch_tensor(c(1L, 2L), dtype = torch_long())
item <- make_detection_item(
matrix(c(120, 70, 180, 130, 210, 80, 280, 120), ncol = 4, byrow = TRUE),
labels = labels,
image_size = c(200L, 400L)
)
result <- item_transform_center_crop(item, size = c(100L, 200L))

expect_equal_to_r(result$y$labels, as.integer(as_array(labels)))
expect_equal(result$y$image_height, 100L)
expect_equal(result$y$image_width, 200L)

# multiple boxes are adjusted and clamped to the crop
boxes <- matrix(c(
120, 70, 180, 130,
150, 60, 280, 140,
110, 80, 140, 120
), ncol = 4, byrow = TRUE)
item <- make_detection_item(boxes, image_size = c(200L, 400L))
result <- item_transform_center_crop(item, size = c(100L, 200L))

expected_boxes <- boxes
expected_boxes[, 1] <- pmax(0, boxes[, 1] - 99)
expected_boxes[, 3] <- pmin(200, boxes[, 3] - 99)
expected_boxes[, 2] <- pmax(0, boxes[, 2] - 49)
expected_boxes[, 4] <- pmin(100, boxes[, 4] - 49)
expect_tensor_shape(result$y$boxes, c(3, 4))
expect_equal_to_r(result$y$boxes[, 1], expected_boxes[, 1])
expect_equal_to_r(result$y$boxes[, 3], expected_boxes[, 3])
expect_equal_to_r(result$y$boxes[, 2], expected_boxes[, 2])
expect_equal_to_r(result$y$boxes[, 4], expected_boxes[, 4])

# empty boxes are preserved
item <- make_detection_item(
boxes = matrix(numeric(0), ncol = 4),
labels = torch_zeros(0L, dtype = torch_long())
)
result <- item_transform_center_crop(item, size = c(100L, 200L))

expect_tensor_shape(result$y$boxes, c(0, 4))
expect_tensor_dtype(result$y$boxes, torch_float())
})

test_that("item_transform_center_crop square crop via single int", {
item <- make_detection_item(matrix(c(120, 70, 180, 130), ncol = 4), image_size = c(200L, 400L))
result <- item_transform_center_crop(item, size = 100)

expect_tensor_shape(result$x, c(3, 100, 100))
expect_equal(result$y$image_height, 100L)
expect_equal(result$y$image_width, 100L)
})

test_that("item_transform_center_crop crops segmentation items", {
item <- make_segmentation_item(image_size = c(200L, 400L), num_masks = 2L)
original_img <- item$x$clone()
original_masks <- item$y$masks$clone()
original_dtype <- item$x$dtype
original_labels <- as.integer(as_array(item$y$labels))

result <- item_transform_center_crop(item, size = c(100L, 200L))

expect_s3_class(result, "image_with_segmentation_mask")
expect_tensor_dtype(result$x, original_dtype)
expect_tensor_shape(result$x, c(3, 100, 200))
expect_equal(result$y$image_height, 100L)
expect_equal(result$y$image_width, 200L)
expect_equal_to_r(result$y$labels, original_labels)

expected_masks <- transform_center_crop(original_masks, size = c(100L, 200L))
expect_tensor_shape(result$y$masks, c(2, 100, 200))
expect_tensor_dtype(result$y$masks, torch_bool())
expect_true(result$y$masks$equal(expected_masks))
})

test_that("item_transform_center_crop pads when crop is larger than image", {
item <- make_detection_item(matrix(c(5, 5, 15, 15), ncol = 4), image_size = c(20L, 30L))
result <- item_transform_center_crop(item, size = c(30L, 40L))

expect_tensor_shape(result$x, c(3, 30, 40))
expect_equal(result$y$image_height, 30L)
expect_equal(result$y$image_width, 40L)
})

test_that("item_transform_center_crop can be composed", {
boxes <- matrix(c(120, 70, 180, 130, 210, 80, 280, 120), ncol = 4, byrow = TRUE)
labels <- torch_tensor(c(1L, 2L), dtype = torch_long())
item <- make_detection_item(boxes, labels = labels, image_size = c(200L, 400L))

result <- item |>
item_transform_hflip() |>
item_transform_center_crop(size = c(100L, 200L))

# after hflip, boxes become (W - x2, y1, W - x1, y2) with W = 400;
# center crop offsets are 99 (width) and 49 (height)
expect_s3_class(result, "image_with_bounding_box")
expect_tensor_shape(result$x, c(3, 100, 200))
expect_equal(result$y$image_height, 100L)
expect_equal(result$y$image_width, 200L)
expect_equal_to_r(result$y$labels, as.integer(as_array(labels)))
expect_equal_to_r(result$y$boxes[1, ], c(220 - 99, 70 - 49, 280 - 99, 130 - 49))
expect_equal_to_r(result$y$boxes[2, ], c(120 - 99, 80 - 49, 190 - 99, 120 - 49))
})

test_that("item_transform_center_crop handles rotated boxes", {
boxes <- matrix(c(120, 70, 180, 130), ncol = 4)
item <- make_detection_item(boxes, image_size = c(200L, 400L))
rotated <- item_transform_rotate(item, angle = 30)

result <- item_transform_center_crop(rotated, size = c(100L, 200L))

expect_s3_class(result, "image_with_rotated_box")
expect_tensor_shape(result$y$boxes, c(1, 5))
expect_tensor_dtype(result$y$boxes, torch_float())
})

# --- item_transform_crop tests ---

test_that("item_transform_crop rejects non-item inputs", {
img <- torch_randn(3, 100, 200)
expect_error(
item_transform_crop(img, top = 1, left = 1, height = 50, width = 100),
"requires a dataset item"
)
})

test_that("item_transform_crop rejects numeric input", {
expect_error(
item_transform_crop(42, top = 1, left = 1, height = 50, width = 100),
"requires a dataset item"
)
})

test_that("item_transform_crop preserves image shape for detection items", {
item <- make_detection_item(matrix(c(10, 20, 50, 60), ncol = 4), image_size = c(100L, 200L))
result <- item_transform_crop(item, top = 11, left = 21, height = 80, width = 160)

expect_tensor_shape(result$x, c(3, 80, 160))
expect_equal(result$y$image_height, 80L)
expect_equal(result$y$image_width, 160L)
})

test_that("item_transform_crop adjusts boxes correctly", {
item <- make_detection_item(matrix(c(10, 20, 50, 60), ncol = 4), image_size = c(100L, 200L))
result <- item_transform_crop(item, top = 11, left = 21, height = 80, width = 160)

# offset_x = left - 1 = 20, offset_y = top - 1 = 10
# new_x1 = max(0, 10 - 20) = 0, new_x2 = min(160, 50 - 20) = 30
# new_y1 = max(0, 20 - 10) = 10, new_y2 = min(80, 60 - 10) = 50
expect_equal_to_r(result$y$boxes[1, 1], 0)
expect_equal_to_r(result$y$boxes[1, 3], 30)
expect_equal_to_r(result$y$boxes[1, 2], 10)
expect_equal_to_r(result$y$boxes[1, 4], 50)
})

test_that("item_transform_crop removes boxes outside crop area", {
# Box at (150, 50, 180, 80) — completely to the right of crop (left = 140, width = 40 → x range [0, 40) with offset 139)
item <- make_detection_item(
matrix(c(150, 50, 180, 80), ncol = 4),
image_size = c(200L, 300L)
)
result <- item_transform_crop(item, top = 1, left = 140, height = 100, width = 40)

# offset_x = 139, new_x1 = 150 - 139 = 11, new_x2 = 180 - 139 = 41
# clipped: new_x1 = max(0, 11) = 11, new_x2 = min(40, 41) = 40
# keep = (40 > 11) & ... = TRUE
expect_tensor_shape(result$y$boxes, c(1, 4))
})

test_that("item_transform_crop removes boxes entirely outside crop", {
# Box at (200, 10, 250, 50) — outside crop left=1, width=100 (x range [0, 100))
# offset_x = 0, new_x1 = 200, new_x2 = 250, clipped: x1=100, x2=100 → zero width
item <- make_detection_item(
matrix(c(200, 10, 250, 50), ncol = 4),
image_size = c(200L, 300L)
)
result <- item_transform_crop(item, top = 1, left = 1, height = 100, width = 100)

expect_tensor_shape(result$y$boxes, c(0, 4))
})

test_that("item_transform_crop preserves labels", {
labels <- torch_tensor(c(1L, 2L), dtype = torch_long())
item <- make_detection_item(
matrix(c(10, 20, 50, 60, 5, 5, 15, 25), ncol = 4, byrow = TRUE),
labels = labels,
image_size = c(100L, 200L)
)
original_labels <- item$y$labels$clone()

result <- item_transform_crop(item, top = 6, left = 6, height = 80, width = 160)

expect_equal(result$y$labels$size(1), result$y$boxes$size(1))
})

test_that("item_transform_crop handles empty boxes", {
item <- make_detection_item(
boxes = matrix(numeric(0), ncol = 4),
labels = torch_zeros(0L, dtype = torch_long())
)
result <- item_transform_crop(item, top = 1, left = 1, height = 50, width = 100)

expect_tensor_shape(result$y$boxes, c(0, 4))
expect_tensor_dtype(result$y$boxes, torch_float())
})

test_that("item_transform_crop handles multiple boxes", {
boxes <- matrix(c(
10, 20, 50, 60,
100, 20, 180, 80
), ncol = 4, byrow = TRUE)
item <- make_detection_item(boxes, image_size = c(100L, 200L))
result <- item_transform_crop(item, top = 11, left = 21, height = 80, width = 160)

expect_tensor_shape(result$y$boxes, c(2, 4))
})

test_that("item_transform_crop does not mutate input", {
boxes <- matrix(c(10, 20, 50, 60), ncol = 4)
item <- make_detection_item(torch_tensor(boxes))
original_img <- as_array(item$x)
original_class <- class(item)

result <- item_transform_crop(item, top = 6, left = 6, height = 50, width = 100)

expect_equal_to_r(item$x, original_img)
expect_equal_to_r(item$y$boxes, boxes)
expect_equal(class(item), original_class)
})

test_that("item_transform_crop preserves class", {
item <- make_detection_item(matrix(c(10, 20, 50, 60), ncol = 4))
result <- item_transform_crop(item, top = 1, left = 1, height = 50, width = 100)

expect_s3_class(result, "image_with_bounding_box")
})

test_that("item_transform_crop actually crops image pixels", {
h <- 100L
w <- 200L
item <- make_detection_item(matrix(c(10, 20, 50, 60), ncol = 4), image_size = c(h, w))
original_img <- item$x$clone()
result <- item_transform_crop(item, top = 11, left = 21, height = 80, width = 160)

expected_img <- transform_crop(original_img, top = 11, left = 21, height = 80, width = 160)
expect_tensor_shape(result$x, c(3, 80, 160))
expect_true(torch_equal(result$x, expected_img))
})

test_that("item_transform_crop image dtype is preserved for detection", {
item <- make_detection_item(matrix(c(10, 20, 50, 60), ncol = 4))
result <- item_transform_crop(item, top = 1, left = 1, height = 50, width = 100)

expect_tensor_dtype(result$x, item$x$dtype)
})

test_that("item_transform_crop preserves image shape for segmentation", {
item <- make_segmentation_item(image_size = c(100L, 200L))
result <- item_transform_crop(item, top = 11, left = 21, height = 80, width = 160)

expect_tensor_shape(result$x, c(3, 80, 160))
})

test_that("item_transform_crop crops masks for segmentation", {
item <- make_segmentation_item(image_size = c(100L, 200L), num_masks = 2L)
original_masks <- item$y$masks$clone()

result <- item_transform_crop(item, top = 11, left = 21, height = 80, width = 160)

expected_masks <- transform_crop(original_masks, top = 11, left = 21, height = 80, width = 160)
expect_tensor_shape(result$y$masks, c(2, 80, 160))
expect_tensor_dtype(result$y$masks, torch_bool())
expect_true(result$y$masks$equal(expected_masks))
})

test_that("item_transform_crop preserves labels for segmentation", {
item <- make_segmentation_item(image_size = c(100L, 200L), num_masks = 2L)
original_labels <- as.integer(as_array(item$y$labels))

result <- item_transform_crop(item, top = 11, left = 21, height = 80, width = 160)

expect_equal_to_r(result$y$labels, original_labels)
})

test_that("item_transform_crop updates image_height and image_width for segmentation", {
item <- make_segmentation_item(image_size = c(100L, 200L))
result <- item_transform_crop(item, top = 11, left = 21, height = 80, width = 160)

expect_equal(result$y$image_height, 80L)
expect_equal(result$y$image_width, 160L)
})

test_that("item_transform_crop preserves class for segmentation", {
item <- make_segmentation_item(image_size = c(100L, 200L))
result <- item_transform_crop(item, top = 11, left = 21, height = 80, width = 160)

expect_s3_class(result, "image_with_segmentation_mask")
})

test_that("item_transform_crop image dtype is preserved for segmentation", {
item <- make_segmentation_item(image_size = c(100L, 200L))
result <- item_transform_crop(item, top = 11, left = 21, height = 80, width = 160)

expect_tensor_dtype(result$x, item$x$dtype)
})

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo This cannot be part of #373 as it is already in #370 and #371, and thus prevents the independant merging of this P.R.

x
}

#' Crop a dataset item

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

todo This cannot be part of #373 as it is already in #370 and #371, and thus prevents the independant merging of this P.R.

@DerrickUnleashed

Copy link
Copy Markdown
Contributor Author

why I cut from existing branches ie (hflip cut from main, vflip cut from hflip, center crop cut from vflip and so on) was because since im making changes in the same file it'll be easier to fix merge conflicts but however it does have this issue of not being able to review PRs simultaneously which I didn't consider sorry about that, can we merge #371 and then on this PR I'll make sure that from the next PR onwards I cut from main

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add item_transform_pad

2 participants