Running with RGB-D in test_demo.py

[syhwang1231]

Hi, thanks for a great work!
I'm trying to run test_demo.py and encountered some issues and questions.

1. What do Weights trained with UOAIS dataset mean? There are 2 types for each RGB and RGB-D; OSD and OCID. If I want to evaluate the model on OSD or OCID, are these the weights that I'm supposed to use? I think basically what I want to know is the difference between all the weights you offered, including the link I wrote.
2. I wanted to run test_demo.py on RGB-D, so I set the config file path and weight path like this:

cfg_file_MSMFormer = os.path.join(dirname, '../../MSMFormer/configs/UOAIS_UCN.yaml')
weight_path_MSMFormer = os.path.join(dirname, "../../data/checkpoints/rgbd_pretrain/norm_RGBD_pretrained.pth")

And when I run the code, I get the following log that says depth related weights are loaded but not used:

=================================================
data keys
load the following keys from the pretrained model
The checkpoint state_dict contains keys that are not used by the model:
  pretrained_backbone.fcn_depth.resnet34_8s.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.0.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.0.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.0.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.0.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.1.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.1.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.1.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.1.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.2.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.2.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.2.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer1.2.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.0.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.0.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.0.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.0.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.0.downsample.0.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.0.downsample.1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.1.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.1.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.1.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.1.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.2.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.2.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.2.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.2.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.3.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.3.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.3.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer2.3.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.0.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.0.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.0.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.0.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.0.downsample.0.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.0.downsample.1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.1.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.1.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.1.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.1.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.2.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.2.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.2.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.2.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.3.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.3.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.3.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.3.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.4.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.4.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.4.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.4.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.5.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.5.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.5.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer3.5.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.0.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.0.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.0.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.0.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.0.downsample.0.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.0.downsample.1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.1.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.1.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.1.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.1.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.2.conv1.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.2.bn1.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.2.conv2.weight
  pretrained_backbone.fcn_depth.resnet34_8s.layer4.2.bn2.{bias, num_batches_tracked, running_mean, running_var, weight}
  pretrained_backbone.fcn_depth.resnet34_8s.fc.{bias, weight}

So, I was wondering if you could share the correct pairs to use for every possible cases such as (RGB-D - OCID - trained on UOAIS), (RGB-D - OSD - trained on TOD), (RGB - OCID - trained on TOD), etc. I'm confused about what config and weight to use for each case that I want to test out.
FYI, these are the config and weight pairs I used for each scenario.

### RGB / OSD (trained with UOAIS-Sim)
# cfg_file_MSMFormer = os.path.join(dirname, '../../MSMFormer/configs/UOAIS_ResNet50.yaml')
# weight_path_MSMFormer = os.path.join(dirname, "../../data/checkpoints/OSD_RGB_MSMFormer_UOAIS_SIM.pth")

### RGB / OSD (trained with TOD)
# cfg_file_MSMFormer = os.path.join(dirname, '../../MSMFormer/configs/mixture_ResNet50.yaml')
# weight_path_MSMFormer = os.path.join(dirname, "../../data/checkpoints/rgb_pretrain/norm_RGB_pretrained.pth")

### RGBD / OSD (UOAIS_UCN.yaml or UOAIS_ResNet50.yaml ?)
cfg_file_MSMFormer = os.path.join(dirname, '../../MSMFormer/configs/UOAIS_UCN.yaml')
weight_path_MSMFormer = os.path.join(dirname, "../../data/checkpoints/rgbd_pretrain/norm_RGBD_pretrained.pth")

### RGB / OCID (trained with UOAIS-Sim)
# cfg_file_MSMFormer = os.path.join(dirname, '../../MSMFormer/configs/UOAIS_ResNet50.yaml')
# weight_path_MSMFormer = os.path.join(dirname, "../../data/checkpoints/OCID_MSMFormer_RGB_UOAIS_SIM.pth")

### RGB / OCID (trained with TOD)
# cfg_file_MSMFormer = os.path.join(dirname, '../../MSMFormer/configs/mixture_ResNet50.yaml')
# weight_path_MSMFormer = os.path.join(dirname, "../../data/checkpoints/rgb_pretrain/norm_RGB_pretrained.pth")

### RGBD / OCID (UOAIS_UCN.yaml or UOAIS_ResNet50.yaml ??)
# cfg_file_MSMFormer = os.path.join(dirname, '../../MSMFormer/configs/UOAIS_ResNet50.yaml')
# weight_path_MSMFormer = os.path.join(dirname, "../../data/checkpoints/OCID_RGBD_MSMFormer_UOAIS_SIM.pth")

[YoungSean]

Hi, config files and model weights are independent.

Config files

For RGB case, use "mixture_ResNet50.yaml" since RGB only uses ResNet50 backbone.
For RGBD case, use "MSMFormer/configs/mixture_UCN.yaml" since RGBD includes UCN backbone.
For the second stage models, use "crop_*" yaml files.

Example usage: find the shell file in the ROOT/scripts folder, like experiments/scripts/demo_msmformer_rgbd.sh

Weights

1. These weights are generally what you need. Trained on TOD, evaluating on OCID and OSD: RGBD weights for both datasets and RGB weights for both datasets
2. We use UOAIS-Sim since the UOAIS-Sim dataset with superior quality RGB images in comparison to the TableTop dataset. Trained on UOAIS-Sim, find the following weights here

• Evaluating on OCID, RGBD case: OCID_RGBD_MSMFormer_UOAIS_SIM.pth
• Evaluating on OCID, RGB case: OCID_RGB_MSMFormer_UOAIS_SIM.pth
• Evaluating on OSD, RGBD case: OSD_RGBD_MSMFormer_UOAIS_SIM.pth
• Evaluating on OSD, RGBD case: OSD_RGB_MSMFormer_UOAIS_SIM.pth

Unused Weight

In test_demo.py, if input_image is set to "RGBD_ADD", it should use the depth related weights. By default, it only use RGB input and weights. Check these lines:

if input_image == "RGBD_ADD":
    cfg.MODEL.USE_DEPTH = True
else:
    cfg.MODEL.USE_DEPTH = False

Hope the above can help you.

[syhwang1231]

Thank you for your kind reply! I recently started studying deep learning, so I'm not familiar with handling codes and stuff. 😅

1. I just want to be sure, so the difference between, say, norm_RGBD_pretrained.pth and OCID_MSMFormer_UOAIS_SIM.pth is that the former is the weight trained on TOD and the latter is the weight trained on UOAIS-Sim? Then, when does weights like norm_RGBD_pretrained.pth and crop_RGB_pretrained.pth get used?
2. Kind of related to my first question, I'm still confused, why do you have different weight for OCID and OSD? (OCID_MSMFormer_UOAIS_Sim.pth and OSD_MSMFormer_UOAIS_Sim.pth) Are these weights fine-tuned on each corresponding dataset? If so, why is that?
3. Also, the server that I'm using doesn't support GUI related libraries. So, I rewrote the code that uses open3d using a library called pypcd4. Do you think this might lead to differences in evaluation?
4. Lastly, the link for the TOD dataset in your ReadMe file is not valid. Could you please check?

[YoungSean]

1. Differences between the pretrained weight files are as follows:

• The difference between norm_RGBD_pretrained.pth and OCID_MSMFormer_UOAIS_SIM.pth is that the former is trained on the TOD dataset, while the latter is trained on UOAIS-Sim. Yes, this understanding is correct.
• norm_RGBD_pretrained.pth is trained on TOD and is used for the first-stage model in the RGB-D setting.
• crop_RGB_pretrained.pth is trained on TOD and is used for the second-stage model in the RGB-only setting.
• The crop_ prefix indicates that the weight file is intended for the second-stage model.

2. No fine-tuning is performed on OCID or OSD. When using UOAIS as the training dataset, optimal performance on OCID and OSD is achieved with different numbers of training epochs. However, when using TOD for training, a single training epoch setting works well for both evaluation datasets.
3. I am not sure about this part. I have not explicitly verified this. In principle, this change should not affect the evaluation results. You may first try to reproduce the prediction results.
4. Contact me via email: yangxiao.lu@utdallas.edu

[syhwang1231]

Following your instructions, I was able to reproduce evaluation results in RGB setting. However, I still get the same error when trying to run the model in RGBD setting.
Below is the cfg file and weight path I used:

### RGBD - 1st stage model 
cfg_file_MSMFormer = os.path.join(dirname, '../../MSMFormer/configs/mixture_UCN.yaml')
weight_path_MSMFormer = os.path.join(dirname, "../../data/checkpoints/rgbd_pretrain/norm_RGBD_pretrained.pth")

### RGBD - 2nd-stage model
cfg_file_MSMFormer_crop = os.path.join(dirname, "../../MSMFormer/configs/crop_mixture_UCN.yaml")
weight_path_MSMFormer_crop = os.path.join(dirname, "../../data/checkpoints/rgbd_pretrain/crop_RGBD_pretrained.pth")

And I still get the same log that I mentioned in the first place, that basically says (from my understanding) norm_RGBD_pretrained.pth is built upon ResNet34 (which includes depth layers) but mixture_UCN.yaml has ResNet50.
By referring to your replies, I'm pretty sure I used correct configs and weights. And I also set input_image = "RGBD_ADD":

    predictor, cfg = get_predictor(cfg_file=cfg_file_MSMFormer,
                                   weight_path=weight_path_MSMFormer,
                                   input_image = "RGBD_ADD"  # NOTE: RGBD or COLOR
                                   )
    predictor_crop, cfg_crop = get_predictor_crop(cfg_file=cfg_file_MSMFormer_crop,
                                                  weight_path=weight_path_MSMFormer_crop,
                                                  input_image="RGBD_ADD")

# And then ran test dataset like this
test_dataset_crop(cfg, osd_dataset, predictor, predictor_crop, visualization=False, topk=False, confident_score=0.7)

What am I doing wrong here? I would really appreciate your help.

[YoungSean]

Hi, did you load the following weights into the model:

Download the pretrained backbone checkpoints from UCN. They are seg_resnet34_8s_embedding_cosine_rgbd_add_sampling_epoch_16.checkpoint.pth and seg_resnet34_8s_embedding_cosine_rgbd_add_crop_sampling_epoch_16.checkpoint.pth. Then move the checkpoint files into $ROOT/data/checkpoints.

[syhwang1231]

Yes, I downloaded those checkpoints and put them in the right directory.
Below is the screenshot of the checkpoints that I have right now.

[YoungSean]

Then this is weird. It’s likely that something is wrong with how the weights are being loaded or how the model inputs are being configured.

[syhwang1231]

OK so.. do you have anything that comes up to your mind about this issue? What I found weird is that mixture_UCN.yaml has Base-COCO-InstanceSegmentation.yaml as its base config. And when I open this "Base" file, the value of MODEL.RESNETS.DEPTH is set to 50, which means I guess that the config file has the model of ResNet-50 (and does not have depth-related layers).
However, I think the weight(norm_RGBD_pretrained.pth) has ResNet-34 weights and depth layers weights. So I think the config and the weight are not based on the same model.
I would like to hear your thoughts about this, and maybe some recommendations to fix this..

[YoungSean]

What datasets are you evaluating? Did you try both OCID and OSD? You may ignore the warning about loading the weights since several pretrained weights are loaded.

[syhwang1231]

Yes, I'm evaluating on OCID and OSD and tried them both. No other datasets are used.
Also, I sent you an email about getting the TOD dataset.

[YoungSean]

Something unknown happens. I did not receive the email.

[syhwang1231]

I sent an email with title "About TOD dataset (from MSMFormer github issue)" to yangxiao.lu@utdallas.edu. Is the email address wrong?
And so, maybe there isn't a solution to this issue..?

[YoungSean]

The edu email address is correct. You may try another email address: yangxiaolu95@gmail.com. If it still does not work, check your email settings.

I am afraid that you may have to debug to find the solution.

[syhwang1231]

I sent a new email to your gmail address.
Then could you recommend me where I should take a look on an abstract level? (Like to take a look at the part where the weights are loaded, or config is re-written... )