Segmentation fails with custom RealSense D455 data despite correct camera intrinsics

[haoqingzhou]

Hi, thanks for the great work on this project!
Problem:
The demo RGB-D images provided in the repo work perfectly for segmentation. However, when I use my own captured data, the segmentation produces no meaningful results (or fails entirely).
What I've already tried:
I have replaced the camera intrinsics with those of my own camera.
I verified the depth values are in the correct range (objects at ~0.5–0.7 m).
My setup:
Camera: Intel RealSense D455
Resolution: [640×480]
My questions:
Could the D455 be incompatible with this pipeline (e.g., depth scale, coordinate convention, or IR pattern interference)?
What camera(s) did you use to capture the demo data?
Are there any additional preprocessing steps or parameter adjustments required for RealSense depth images?
Any guidance would be greatly appreciated. I can provide sample images if that helps.
Thanks!

[YoungSean]

Hi Haoqing,

Since the segmentation works well on the demo images, it suggests that the model and inference pipeline are functioning correctly. The issue is therefore more likely related to data preprocessing rather than the model itself.

Could you try running the model using only the RGB images (without the depth input)? If the RGB-only results look reasonable, that would indicate the problem is likely in the preprocessing of the depth images (e.g., depth scaling, alignment, normalization, or formatting).

Let me know how the RGB-only test goes, and we can narrow down the issue further.

You may check the function and the confidence score 0.6 here.

Another possible issue is the camera-to-object distance. The model may be sensitive to the capture distance, so you could also try adjusting the distance between the camera and the objects to see if that improves the segmentation results.