youngskkim / crn Goto Github PK

am currently pursuing my Master's in Artificial Intelligence, and I've utilized your research paper as a fundamental reference for my project implementation. While working on the code, I encountered an error due to the absence of nuscenes_infos_train.pkl and nuscenes_infos_val.pkl files in the directory specified by the code. These files seem crucial for the code's functionality. Could you provide more details or guidance on where to find or generate these files? Your prompt response would greatly assist me as I'm dedicating significant effort to this project and seeking to resolve this issue urgently. Thank you.

Aoa dear sir please tell me about the size of the dataset that you implemented in this project. Please point out from the attached pictures. Also share me screen of the directory in which you save the data.
Please reply me as soon as possible.
Thank you.

Hello,

I am trying to run this code, but I found the error below:
ModuleNotFoundError: No module named 'mmcv._ext'

I thought the reason maybe that my cuda version(12.2) is not compatible with my mmcv version(1.6.0, which is designated in the installation part).

Since I am sharing the cuda environment with other people, it is hard to reinstall a lower version(like 11.1 in the installation part).
So I wonder if the source code can run perfectly with a later version of dependencies? Not only cuda but also others such as torch/ torch vision/ pytorch-lightning/ mmcv/ mmsegmentation/ mmdet/ etc. ...

Looking forward to your reply.
Best wishes.

hello, as mentioned in the paper, the number of queries is reduced by confidence, where can I find the implementation?

Hello! I met this problem after training several epoches. What is the problem and how can I solve it? Thanks!

Hello, I would like to ask how to resume training from a checkpoint. I previously trained for 20 epochs and obtained a checkpoint. When I wanted to continue training to complete 24 epochs, I found that simply setting the 'ckpt_path' to the path of the 20-epoch checkpoint doesn't seem to be the correct method. This is because when I set it up this way, the resumed training still starts from epoch 0.
Looking forward to your reply, thank you very much!

Aoa dear sir Please tell me about the size of data that you implemented in this project. Please tell me from the attached picture.



Also send me screen shot of the directory in which you saved the data.
Please reply me as soon as possible.
Thankyou.

Hi，Could you please provide an estimate of the training duration and the approximate GPU memory usage when the batch size is set to the default value of 32? Thanks！

Hello, I encountered this error while evaluating model on validation set with the checkpoint I trained myself. How can I resolve this?

Hello, I replicated the experiment following the experimental setup in your paper's appendix and your code, but did not achieve the results mentioned in your code comments. Is there any additional configuration, such as Test Time Augmentation (TTA)? Or what could be the reason for this discrepancy?
Looking forward to your reply, thank you very much!

Hi,
I fail to understand in script gen_rada_pv.py https://github.com/youngskkim/CRN/blob/main/scripts/gen_radar_pv.py#L96
why are we using LidarPointcloud but not radar pointcloud?

Hello :)
First of all congrats to your paper + code. It looks super cool.
However I tried using your Model with a Resnet 101 as image backbone with a final image resolution close to the original value. I also tried to detect object within 100m and beyond. Unfortunately this model does not seem to converge...

Thur purple line in the picture is your resnet50 with resolution 256X704 (I didn't change the config) while the Orange line is my (described above) network.
He is also the config of the experiment. Please note the orange line was created using the the optimizer
elf.optimizer_config = dict(type="AdamW", lr=2e-4, weight_decay=1e-4)
The change optimizer didn't changed a thing here:
`class CRNLightningModel(BEVDepthLightningModel):
def init(self, *args, **kwargs) -> None:
self.return_image = True
self.return_depth = True
self.return_radar_pv = True
################################################
# self.optimizer_config = dict(type="AdamW", lr=2e-4, weight_decay=1e-4) ### org
self.optimizer_config = dict(type="AdamW", lr=2e-6, weight_decay=1e-5)
################################################
x_y_bound = [-51.2 * 2, 51.2 * 2]
final_dim = [896, 1600]
self.backbone_img_conf = {
# "x_bound": [-51.2, 51.2, 0.8],
# "y_bound": [-51.2, 51.2, 0.8],
"x_bound": x_y_bound + [0.8], # voxel x bounds
"y_bound": x_y_bound + [0.8], # voxel y bounds
"z_bound": [-5, 3, 8],
"d_bound": [
2.0,
117.2,
0.8,
], # frsutum depth bounds min,max, depth_step_size
# "d_bound": [2.0, 58.0, 0.8],
# "final_dim": (256, 704),
"final_dim": tuple(final_dim),
"downsample_factor": 32,
"img_backbone_conf": dict(
type="ResNet",
depth=101,
frozen_stages=0,
out_indices=[0, 1, 2, 3],
dilations=(2, 2, 1, 1),
#dilations=(1, 1, 1, 1),
strides=(2, 2, 2, 2),
#strides=(1, 1, 1, 1),
norm_eval=False,
init_cfg=dict(type="Pretrained", checkpoint="torchvision://resnet101"),
),
"img_neck_conf": dict(
type="SECONDFPN",
in_channels=[256, 512, 1024, 2048],
upsample_strides=[0.25, 0.50, 1, 2],
out_channels=[128, 128, 128, 128],
# out_channels=[256, 256, 256, 256],
),
"depth_net_conf": dict(in_channels=512, mid_channels=256),
"radar_view_transform": True,
"camera_aware": False,
# "camera_aware": True, # Model camera intrinsic into DepthNet
#"output_channels": 80,
"output_channels": 128,
}
################################################
# point_cloud_range_backbone=[0, 2.0, 0, 704, 58.0, 2], #[x_min, y_min, z_min, x_max, y_max, z_max]
point_cloud_range_backbone = [
0,
2.0,
0,
final_dim[1],
self.backbone_img_conf["d_bound"][1],
2,
] # [x_min, y_min, z_min, x_max, y_max, z_max]
self.backbone_pts_conf = {
"pts_voxel_layer": dict(
max_num_points=8,
voxel_size=[8, 0.4, 2],
point_cloud_range=point_cloud_range_backbone, # [x_min, y_min, z_min, x_max, y_max, z_max]
max_voxels=(768, 1024),
),
"pts_voxel_encoder": dict(
type="PillarFeatureNet",
in_channels=5,
feat_channels=[32, 64],
with_distance=False,
with_cluster_center=False,
with_voxel_center=True,
voxel_size=[8, 0.4, 2],
point_cloud_range=point_cloud_range_backbone,
norm_cfg=dict(type="BN1d", eps=1e-3, momentum=0.01),
legacy=True,
),
"pts_middle_encoder": dict(
# type="PointPillarsScatter", in_channels=64, output_shape=(140, 88)
type="PointPillarsScatter",
in_channels=64,
output_shape=(288, 100)
# type="PointPillarsScatter",
# in_channels=64,
# output_shape=(288, 200),
),
"pts_backbone": dict(
type="SECOND",
in_channels=64,
out_channels=[64, 128, 256],
layer_nums=[3, 5, 5],
layer_strides=[1, 2, 2],
norm_cfg=dict(type="BN", eps=1e-3, momentum=0.01),
conv_cfg=dict(type="Conv2d", bias=True, padding_mode="reflect"),
),
"pts_neck": dict(
type="SECONDFPN",
in_channels=[64, 128, 256],
out_channels=[128, 128, 128],
upsample_strides=[0.5, 1, 2],
norm_cfg=dict(type="BN", eps=1e-3, momentum=0.01),
upsample_cfg=dict(type="deconv", bias=False),
use_conv_for_no_stride=True,
),
"occupancy_init": 0.01,
# "out_channels_pts": 80,
"out_channels_pts": 128,
}
################################################
self.fuser_conf = {
#"img_dims": 80,
#"pts_dims": 80,
"embed_dims": 128,
# "num_layers": 6,
# "num_heads": 4,
# "bev_shape": (128, 128),
"img_dims":128,
"pts_dims":128,
#"embed_dims":256,
"num_layers": 6,
"num_heads": 4,
"bev_shape": (256, 256),
}
################################################
out_size_factor = 2

    # point_cloud_range_head = [-51.2, -51.2, -5, 51.2, 51.2, 3]
    point_cloud_range_head = [
        x_y_bound[0],
        x_y_bound[0],
        -5,
        x_y_bound[1],
        x_y_bound[1],
        3,
    ]
    # post_center_range = [-61.2, -61.2, -10.0, 61.2, 61.2, 10.0]
    post_center_range = [-61.2 * 2, -61.2 * 2, -10.0, 61.2 * 2, 61.2 * 2, 10.0]

    self.head_conf = {
        "bev_backbone_conf": dict(
            type="ResNet",
            #in_channels=256,
            in_channels=128,
            depth=18,
            num_stages=3,
            strides=(1, 2, 2),
            dilations=(1, 1, 1),
            out_indices=[0, 1, 2],
            norm_eval=False,
            base_channels=160
            # base_channels=360,
        ),
        "bev_neck_conf": dict(
            type="SECONDFPN",
            in_channels=[128, 160, 320, 640],
            #upsample_strides=[1, 2, 4, 8],
            #in_channels=[256, 160, 320, 640],
            #in_channels=[256, 360, 720, 1440],
            upsample_strides=[1, 2, 4, 8],
            out_channels=[64, 64, 64, 64],
        ),
        "tasks": [
            dict(num_class=1, class_names=["car"]),
            dict(num_class=2, class_names=["truck", "construction_vehicle"]),
            dict(num_class=2, class_names=["bus", "trailer"]),
            dict(num_class=1, class_names=["barrier"]),
            dict(num_class=2, class_names=["motorcycle", "bicycle"]),
            dict(num_class=2, class_names=["pedestrian", "traffic_cone"]),
        ],
        "common_heads": dict(
            reg=(2, 2), height=(1, 2), dim=(3, 2), rot=(2, 2), vel=(2, 2)
        ),
        "bbox_coder": dict(
            type="CenterPointBBoxCoder",
            # post_center_range=[-61.2, -61.2, -10.0, 61.2, 61.2, 10.0],
            post_center_range=post_center_range,
            max_num=500,
            score_threshold=0.01,
            out_size_factor=out_size_factor,
            voxel_size=[0.2, 0.2, 8],
            pc_range=point_cloud_range_head,
            code_size=9,
        ),
        "train_cfg": dict(
            point_cloud_range=point_cloud_range_head,
            grid_size=[512, 512, 1],
            voxel_size=[0.2, 0.2, 8],
            out_size_factor=out_size_factor,
            dense_reg=1,
            gaussian_overlap=0.1,
            max_objs=500,
            min_radius=2,
            code_weights=[1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0],
        ),
        "test_cfg": dict(
            post_center_limit_range=post_center_range,
            max_per_img=500,
            max_pool_nms=False,
            min_radius=[4, 12, 10, 1, 0.85, 0.175],
            score_threshold=0.01,
            out_size_factor=out_size_factor,
            voxel_size=[0.2, 0.2, 8],
            nms_type="circle",
            pre_max_size=1000,
            post_max_size=200,
            nms_thr=0.2,
        ),
        "in_channels": 256,  # Equal to bev_neck output_channels.
        "loss_cls": dict(type="GaussianFocalLoss", reduction="mean"),
        "loss_bbox": dict(type="L1Loss", reduction="mean", loss_weight=0.25),
        "gaussian_overlap": 0.1,
        "min_radius": 2,
    }
    ################################################
    self.ida_aug_conf = {
        "resize_lim": (1, 1),
        #"resize_lim": (0.386, 0.55),
        # "final_dim": (256, 704),
        "final_dim": (896, 1600),
        "rot_lim": (0.0, 0.0),
        "H": 900,
        "W": 1600,
        "rand_flip": True,
        "bot_pct_lim": (0.0, 0.0),
        "cams": [
            "CAM_FRONT_LEFT",
            "CAM_FRONT",
            "CAM_FRONT_RIGHT",
            "CAM_BACK_LEFT",
            "CAM_BACK",
            "CAM_BACK_RIGHT",
        ],
        "Ncams": 6,
    }
    self.bda_aug_conf = {
        "rot_ratio": 1.0,
        "rot_lim": (-22.5, 22.5),
        "scale_lim": (0.9, 1.1),
        "flip_dx_ratio": 0.5,
        "flip_dy_ratio": 0.5,
    }

    self.rda_aug_conf = {
        "N_sweeps": 6,
        "N_use": 5,
        "drop_ratio": 0.1,
    }
    ################################################
    super().__init__(
        backbone_img_conf=self.backbone_img_conf,
        head_conf=self.head_conf,
        ida_aug_conf=self.ida_aug_conf,
        bda_aug_conf=self.bda_aug_conf,
        rda_aug_conf=self.rda_aug_conf,
        return_image=self.return_image,
        return_depth=self.return_depth,
        return_radar_pv=self.return_radar_pv,
        optimizer_config=self.optimizer_config,
        *args,
        **kwargs
    )
    ################################################
    self.key_idxes = [-2, -4, -6]
    self.model = CameraRadarNetDet(
        self.backbone_img_conf,
        self.backbone_pts_conf,
        self.fuser_conf,
        self.head_conf,
    )

`
Does anybody have an idea why it's not converging?

Kind regards

Stefan

Can you also provide the visualization script as shown in video please?

I run python exps/det/CRN_r18_256x704_128x128_4key.py --amp_backend native -b 1 --gpus 1 and have below errors

/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:240: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the num_workers argument(try 40 which is the number of cpus on this machine) in theDataLoaderinit to improve performance. rank_zero_warn( Epoch 0: 0%| | 0/29634 [00:00<?, ?it/s]Traceback (most recent call last): File "exps/det/CRN_r18_256x704_128x128_4key.py", line 359, in <module> run_cli(CRNLightningModel, File "/work/data06/git_/CRN/exps/base_cli.py", line 85, in run_cli trainer.fit(model) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 771, in fit self._call_and_handle_interrupt( File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 722, in _call_and_handle_interrupt return self.strategy.launcher.launch(trainer_fn, *args, trainer=self, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch return function(*args, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 812, in _fit_impl results = self._run(model, ckpt_path=self.ckpt_path) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1237, in _run results = self._run_stage() File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1324, in _run_stage return self._run_train() File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1354, in _run_train self.fit_loop.run() File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/fit_loop.py", line 269, in advance self._outputs = self.epoch_loop.run(self._data_fetcher) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/epoch/training_epoch_loop.py", line 208, in advance batch_output = self.batch_loop.run(batch, batch_idx) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/batch/training_batch_loop.py", line 88, in advance outputs = self.optimizer_loop.run(split_batch, optimizers, batch_idx) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/base.py", line 204, in run self.advance(*args, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 203, in advance result = self._run_optimization( File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 256, in _run_optimization self._optimizer_step(optimizer, opt_idx, batch_idx, closure) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 369, in _optimizer_step self.trainer._call_lightning_module_hook( File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1596, in _call_lightning_module_hook output = fn(*args, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/core/lightning.py", line 1625, in optimizer_step optimizer.step(closure=optimizer_closure) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/core/optimizer.py", line 168, in step step_output = self._strategy.optimizer_step(self._optimizer, self._optimizer_idx, closure, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 278, in optimizer_step optimizer_output = super().optimizer_step(optimizer, opt_idx, closure, model, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/strategies/strategy.py", line 193, in optimizer_step return self.precision_plugin.optimizer_step(model, optimizer, opt_idx, closure, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/plugins/precision/native_amp.py", line 85, in optimizer_step closure_result = closure() File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 148, in __call__ self._result = self.closure(*args, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 134, in closure step_output = self._step_fn() File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/loops/optimization/optimizer_loop.py", line 427, in _training_step training_step_output = self.trainer._call_strategy_hook("training_step", *step_kwargs.values()) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1766, in _call_strategy_hook output = fn(*args, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/strategies/ddp.py", line 344, in training_step return self.model(*args, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 799, in forward output = self.module(*inputs[0], **kwargs[0]) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/pytorch_lightning/overrides/base.py", line 82, in forward output = self.module.training_step(*inputs, **kwargs) File "exps/det/CRN_r18_256x704_128x128_4key.py", line 298, in training_step preds, depth_preds = self(sweep_imgs, mats, File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "exps/det/CRN_r18_256x704_128x128_4key.py", line 281, in forward return self.model(sweep_imgs, mats, sweep_ptss=inputs['pts_pv'], is_train=is_train) File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/work/data06/git_/CRN/models/camera_radar_net_det.py", line 90, in forward feats, depth, _ = self.backbone_img(sweep_imgs, File "/home/iot/miniconda3/envs/CRN/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl return forward_call(*input, **kwargs) File "/work/data06/git_/CRN/layers/backbones/rvt_lss_fpn.py", line 397, in forward key_frame_res = self._forward_single_sweep( File "/work/data06/git_/CRN/layers/backbones/rvt_lss_fpn.py", line 304, in _forward_single_sweep geom_xyz, geom_xyz_valid = self.get_geometry_collapsed( File "/work/data06/git_/CRN/layers/backbones/rvt_lss_fpn.py", line 195, in get_geometry_collapsed points = ida_mat.inverse().matmul(points.unsqueeze(-1)).double() RuntimeError: CUDA error: operation not supported when callingcusparseCreate(handle)`
Epoch 0: 0%| | 0/29634 [00:11<?, ?it/s]