lifebeyondexpectations / eccv22-pointmixer Goto Github PK

Hello author, I would like to ask you about the role of this module, I have collected a lot of information, read the principle of it, but do not understand its role in your model is what?

Hi, dear authors.
According to the script run_s3dis_PointMixer.sh, I train the PointMixer on the S3DIS and evaluate it in the setting of npts=800k. The mIoU of the evaluation only reach 69.11, which has a dominant gap with the reported results in the paper. Did I miss something and bring about this gap?

Val result: mIoU/mAcc/allAcc 0.6911/0.7513/0.8989.
Val1 result: mIoU/mAcc/allAcc 0.6911/0.7513/0.8989.
Class_0 Result: iou/accuracy 0.9263/0.9576, name: ceiling.
Class_1 Result: iou/accuracy 0.9798/0.9844, name: floor.
Class_2 Result: iou/accuracy 0.8433/0.9619, name: wall.
Class_3 Result: iou/accuracy 0.0000/0.0000, name: beam.
Class_4 Result: iou/accuracy 0.3351/0.3756, name: column.
Class_5 Result: iou/accuracy 0.6059/0.6212, name: window.
Class_6 Result: iou/accuracy 0.7650/0.9073, name: door.
Class_7 Result: iou/accuracy 0.8343/0.8907, name: chair.
Class_8 Result: iou/accuracy 0.8986/0.9528, name: table.
Class_9 Result: iou/accuracy 0.7628/0.7992, name: bookcase.
Class_10 Result: iou/accuracy 0.7230/0.7979, name: sofa.
Class_11 Result: iou/accuracy 0.7349/0.7707, name: board.
Class_12 Result: iou/accuracy 0.5753/0.7477, name: clutter.
<<<<<<<<<<<<<<<<< End Evaluation <<<<<<<<<<<<<<<<<
Shutting down background jobs, please wait a moment...
Done!
Waiting for the remaining 82 operations to synchronize with Neptune. Do not kill this process.
All 82 operations synced, thanks for waiting!
Explore the metadata in the Neptune app:
https://app.neptune.ai/jaesung.choe/ECCV22-PointMixer-SemSeg/e/EC-182

Hi, thank you for your open codes.
Why do we need an offset when searching the k-nearest neighbours of a given central point?

Traceback (most recent call last):
File "/home/Prjs/ECCV22-PointMixer-main/sem_seg/train_pl.py", line 157, in
cli_main()
File "/home/Prjs/ECCV22-PointMixer-main/sem_seg/train_pl.py", line 150, in cli_main
trainer.fit(model, train_loader, val_loader)
File "/home/anaconda3/envs/pointmixer/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 738, in fit
self._call_and_handle_interrupt(
File "/home/anaconda3/envs/pointmixer/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 683, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/home/anaconda3/envs/pointmixer/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 773, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/home/anaconda3/envs/pointmixer/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1195, in _run
self._dispatch()
.
.
.
.

raise RuntimeError("Default process group has not been initialized, "
RuntimeError: Default process group has not been initialized, please make sure to call init_process_group.

Professor, thank you for your work. While running your code recently, I found that I ran into a few issues when configuring the environment with conda:
In the way of conda, the .yml has a pip way to install some packages, and what is pointops2? pip can not find and install it.

Hello, thank you for your excellent code! I want to know whether this code can reproduce the Point Transformer's result on ModelNet40, which is 93.7 in the paper. Looking forward to your reply! Thanks!

教授您好，百忙之中打扰您，我在复现您的代码时出现pointops2_cuda报错这一问题，我想请教一下是什么原因

What's the purpose of using 'torch.nn.Bilinear'?
The formulation of a bilinear transformation is y= x_1^T A x_2 + b, and the formulation of a linear transformation is y=xA^T+b .
It seems that a bilinear layer just apply a slighter sophisticated linear transformation than the linear layer?

Thank you for your open codes!
It sounds reasonable that spreading point features between different sets via inter-set mixing to enlarge receptive field. However, when I look into the codes of class PointMixerInterSetLayer , I don't find any inverse KNN operation which is described in the paper. It seems that PointMixerInterSetLayer just groups the neighboring features, which is similar as what PointMixerIntraSetLayer have done.
Maybe I misunderstand the code logic of PointMixerInterSetLayer, if so, please help me figure out it. Thank you very much!

# PointMixerInterSetLayerV3
class PointMixerInterSetLayer(nn.Module):

    def __init__(self, in_planes, share_planes, nsample):
        super().__init__()
        self.share_planes = share_planes
        self.nsample = nsample

        self.linear = nn.Sequential(
            nn.Linear(in_planes+in_planes, in_planes//share_planes), # [N*K, C] 
            nn.ReLU(inplace=True))
        self.linear_x = nn.Sequential(
            nn.Linear(in_planes, in_planes//share_planes), # [N*K, C]
            nn.ReLU(inplace=True))
        self.linear_p = nn.Sequential( # [N*K, C]
            nn.Linear(3, 3, bias=False),
            nn.BatchNorm1d(3),
            nn.ReLU(inplace=True), 
            nn.Linear(3, in_planes))

   def forward(self, input):
        x, x_knn, knn_idx, p_r = input
        # p_r: (n, nsample, 3) # point relative (p_r) 
        # x_knn: (n, nsample, c) 
        # knn_idx: (n, nsample)
        N = x_knn.shape[0]

        with torch.no_grad():
            knn_idx_flatten = rearrange(knn_idx, 'n k -> (n k) 1')
        p_r_flatten = rearrange(p_r, 'n k c -> (n k) c')
        p_embed_flatten = self.linear_p(p_r_flatten)
        x_knn_flatten = rearrange(x_knn, 'n k c -> (n k) c')
        x_knn_flatten_shrink = self.linear(
            torch.cat([p_embed_flatten, x_knn_flatten], dim=1))

        x_knn_prob_flatten_shrink = \
            scatter_softmax(x_knn_flatten_shrink, knn_idx_flatten, dim=0) # (n*nsample, c')
        x_v_knn_flatten = self.linear_x(x_knn_flatten) # (n*nsample, c')
        x_knn_weighted_flatten = x_v_knn_flatten * x_knn_prob_flatten_shrink # (n*nsample, c')

        residual = scatter_sum(x_knn_weighted_flatten, knn_idx_flatten, dim=0, dim_size=N) # (n, c')
        residual = repeat(residual, 'n c -> n (repeat c)', repeat=self.share_planes)
        return x + residual