I notice that you reproduce the result of MAE with ViT-S by using official codes, but there is no decoder design in the official codes.
I wonder that how many layers you use in decoder of ViT-S, also decoder_dim and num_head?

@keyu-tian Hi, could you please share the visualization code? Thanks.

Great work, but when will the full code be released? Thanks

我这边把2d操作修改成高度为1的，为了适配1d时序信号，之前试过了vit1dmae，可以预训练完成，且能够微调，效果也不错，目前的疑问是：这边的loss情况要降低到什么程度才可以呢，我修改成mse，loss发现是0,0002的时候，拿权重去做可视化预测，效果很糟糕，我观察到你代码中设置min_loss设置为1e-9，可以分享下最终loss收敛到什么一个地步可以停止训练吗，谢谢

Thanks for your great work! however, in downstream_imagenet, I found an error.
I used the code '~/SparK/downstream_imagenet$ bash ./main.sh exp1 --data_path=/home/users/data --model=resnet50 --resume_from=/home/users/SparK/pretrain/output/resnet50_1kpretrained.pth --bs=32'

and the following error happened:
[05-17 12:28:48] (nstream_imagenet/main.py, line 49)=> [FT start] from ep0
[05-17 12:28:48] (nstream_imagenet/main.py, line 58)=> [loader_train.sampler.set_epoch(0)]
Traceback (most recent call last):
File "/home/users/SparK/downstream_imagenet/main.py", line 189, in
main_ft()
File "/home/users/SparK/downstream_imagenet/main.py", line 60, in main_ft
train_loss, train_acc = fine_tune_one_epoch(ep, args, tb_lg, loader_train, iters_train, criterion, mixup_fn, model, model_ema, optimizer, params_req_grad)
File "/home/users/SparK/downstream_imagenet/main.py", line 129, in fine_tune_one_epoch
inp, tar = mixup_fn(inp, tar)
ValueError: too many values to unpack (expected 2)
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=3 encoding='UTF-8'>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 3006233) of binary: /usr/bin/python3

Thanks for your great work! Could you please provide the integrated pre-trained weights of Resnet with its decoder?

the link to pretrained imagenet model is broken?

Hi, I am very interested in your research and want to use it for 3D dataset. I want to know where this code and file needs to be modified to adapt to the 3D network. I would thanks for your reply.

Thanks for your wonderful work! The tutorial is fairly friendly for reproduction, but I also wonder if it is possible for us to get the fine-tuned weights on imagenet?

Thanks for your great work! Does these contrastive learning methods' performance on ImageNet refer to finetuning results? If so, since those papers only report linear evaluation results, where did you get the finetuning score?

I installed Tap via pip3 install tap, but some trouble happen.

from tap import Tap
ImportError: cannot import name 'Tap' from 'tap' 

This project is very cool, but it would be nice if you provided a simple tutorial to support more model architectures :)

Dear authors,

with regards to U2 from the reviewer 71p6: "The sparse convolution seems cannot largely speed up the training process like MAE for ViT and the model requires 300 epoch fine-tuning that is the same as the configuration of training from scratch."

I think that the reviewer had in mind that for MAE, the authors only fine-tuned for 100 (ViT-B) and 50 (ViT-L/H) epochs (MAE paper, A.1, Table 9). The supervised models that were trained from scratch in the MAE paper were trained for 300 epochs (MAE paper, A.2, Table 11).

SparK was fine-tuned for 300 epochs, which is the same configuration as training supervised ViTs from scratch. MAE thus achieved comparable performance with 3-6x less fine-tuning epochs.

Can you report your results also with the same fine-tuning configuration (50/100 epochs) as used in MAE? Only then the comparison can be fair.

Comment from: https://openreview.net/forum?id=NRxydtWup1S

Hi,

Have you implemented cosine learning rate annealing as talked in the paper?

Thank you!

Thanks for your work! I wondering GPU time for pre-training 1600 epochs using ConvNext-B.

I notice that you set the finetuning epochs as 200 or 400.

However,

• the standard supervised training only runs for 300 epochs (w/o 1600 or 800 pretraining epochs).
• in MIM works (e.g. MAE and ConvNextV2), they typically finetune 100 epochs.

Did you try 100 epoch schedule? Can you also kindly share the result under such a setting?

Could it be trained on a single GPU?

Hi, thanks for your nice work! It seems it only supports mask patch size of 32x32,if i want to try other size like 16x16/8x8,how should i change the code?

Hi, thanks for your work. Whenever I try to pretrain with my own dataset, following error is happening:

torchrun --data_path=/home/user/augdata --exp_name=ptaugdata --exp_dir=/home/user/models
usage: torchrun [-h] [--nnodes NNODES] [--nproc-per-node NPROC_PER_NODE] [--rdzv-backend RDZV_BACKEND] [--rdzv-endpoint RDZV_ENDPOINT] [--rdzv-id RDZV_ID] [--rdzv-conf RDZV_CONF] [--standalone]
[--max-restarts MAX_RESTARTS] [--monitor-interval MONITOR_INTERVAL] [--start-method {spawn,fork,forkserver}] [--role ROLE] [-m] [--no-python] [--run-path] [--log-dir LOG_DIR]
[-r REDIRECTS] [-t TEE] [--node-rank NODE_RANK] [--master-addr MASTER_ADDR] [--master-port MASTER_PORT] [--local-addr LOCAL_ADDR]
training_script ...
torchrun: error: the following arguments are required: training_script, training_script_args

do I need to specify training_script and training_script_args?

Thanks for your excellent work! I noticed that LR layer decay is not used in ImageNet fine-tuning, but it is used in detection. Why not use layer decay like in transformer fine-tuning? What influence will it have if layer decay is adopted in ImageNet fine-tuning?

Hi @keyu-tian! Thank you for proposing the amazing work. I have a question about data leakage. Here is my situation.
I have a unlabled dataset about 200k images. Can I use the whole dataset to train a backbone, then use the backbone to do detection job on the same dataset (but small part of the dataset with label)? Will it casue a data leakage issue?since during SparK training may contain the image in valid set.
And my final goal is to count objects in the whole dataset(200k) by object detection.

Hi,

Thanks so much for your great work!
I am confused about how you adjust the lr and weight_decay. In pretrain/utils/lr_control.py/lr_wd_annealing, I think lr and wd are only acquired and not changed.

Thank you!

These links are broken. Not found in repo

After finishing pretrain resnet50, we can get a resnet50 weight file in sparse type. I'd like to use resnet50 (dense type) as my backbone in my other projects. But how to convert sparse model to dense model? Is there any convenient function like SparseEncoder.dense_model_to_sparse in encoder.py ?

十分感谢TechBeat上的视频分享，文章思路非常清晰，PPT也很漂亮，想请教下，ppt中的网络结构动图是通过什么画的呀

I do not understand the performance gains of sparse convolution compared to zero-outing (Table 5.). If your implementation is correct (i.e. zero-outing both weight and bias terms), the two implementations should be numerically identical.

Hi,

Thanks so much for your great work!
I wonder if you can publish the pretrained weights on ResNet18 for restricted memory?

Thnak you!

this is more of question than issue.

have you tried to implement sp_conv with scale? I believe this may "fix" the disparity in statistics between full conv (which you actually use after pre-train) and sp_conv

hi, wonderful work! I wonder that will it work well while in_channel == group in nn.Conv2D?

I was running downstream_imagenet code like this:
bash ./main.sh exp1 --data_path=/home/users/datacopy --model=resnet50 --resume_from=/home/users/SparK/pretrain/output_pretraining/resnet50_1kpretrained.pth --bs=16

and the error happened like this:[05-17 02:23:25] (nstream_imagenet/main.py, line 48)=> [FT start] ep_eval=[0, 5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90, 95, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299]
[05-17 02:23:25] (nstream_imagenet/main.py, line 49)=> [FT start] from ep0
[05-17 02:23:25] (nstream_imagenet/main.py, line 58)=> [loader_train.sampler.set_epoch(0)]
[05-17 02:23:44] (nstream_imagenet/main.py, line 165)=> [ep0 it 3/375] L: 0.7013 Acc: 0.00 lr: 6.8e-088.2e-07 Remain: 0:28:38
[05-17 02:24:14] (nstream_imagenet/main.py, line 165)=> [ep0 it187/375] L: 0.6930 Acc: 0.00 lr: 1.1e-061.3e-05 Remain: 0:00:48
Traceback (most recent call last):
File "/home//downstream_imagenet/main.py", line 189, in
main_ft()
File "/home/downstream_imagenet/main.py", line 60, in main_ft
train_loss, train_acc = fine_tune_one_epoch(ep, args, tb_lg, loader_train, iters_train, criterion, mixup_fn, model, model_ema, optimizer, params_req_grad)
File "/home/users/SparK/downstream_imagenet/main.py", line 129, in fine_tune_one_epoch
inp, tar = mixup_fn(inp, tar)
File "/home/users/.local/lib/python3.10/site-packages/timm/data/mixup.py", line 210, in call
assert len(x) % 2 == 0, 'Batch size should be even when using this'
AssertionError: Batch size should be even when using this
sys:1: ResourceWarning: unclosed file <_io.TextIOWrapper name=3 encoding='UTF-8'>
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 2986007) of binary: /usr/bin/python3
Traceback (most recent call last):
File "/home/users/.local/bin/torchrun", line 8, in
sys.exit(main())
File "/home/users/.local/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
return f(*args, **kwargs)
File "/home/users/.local/lib/python3.10/site-packages/torch/distributed/run.py", line 794, in main
run(args)
File "/home/users/.local/lib/python3.10/site-packages/torch/distributed/run.py", line 785, in run
elastic_launch(
File "/home/users/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 134, in call
return launch_agent(self._config, self._entrypoint, list(args))
File "/home/users/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:

============================================================
main.py FAILED

Thank you for your excellent work. I want to do some work on audio classification task, and I found it is impossible to keep input shape as a square. What should i do when input tensor is a rectangle，like (128, 600).

Current implement only support the square input size, it would be great if this repo can support the unsquare input size

Thanks for your wonderful job! I find the time of training resnet50 on V100 is very log. Can you offer the log of training resnet50, and offer the time of training resnet50. Thanks!

Is there any loss curve for that? thanks

Thank you for your excellent work. Replacing the transformer with CNN does make deployment more friendly. Furthermore, I'm wondering if using a smaller model such as Mobilenet v2 for pre-training and then fine-tuning downstream would be effective?

Hi,

Thanks for your great work!

I trained with your code but always got the above exception after training for 1 or 2 hours.
I searched and found that was because the terminal window was closed even the model was training in the background with nohup.

Have you also met the same problem? and I just wonder how did you train the models in the background?

Thank you!

Thanks for your work, we compared SparK with MAE (convnext-base v.s. swin-base), the training time of SparK is about 6.5 times that of MAE. Is there any way to improve training efficiency? Is the lengthy training time caused by insufficient hardware optimization of sparse convolution?

Hello, I don't know if you has any research involving NLP, I want to ask if there is any better pre-training model now, or how to improve Bert

What an outstanding and admirable work! Due to performance limitations, this experiment cannot be trained on a PC. But I want to try to see the effect of image restoration using different models. I would like to ask if you can provide more pretrain model pth files including decoder.

Just like the pth model in https://github.com/keyu-tian/SparK/blob/main/pretrain/viz_reconstruction.ipynb step 3 .

The hyperparameter settings (batch size and learning rate) in the paper seem inconsistent with the code. Which could reproduce the performance (80.6 acc) reported in the paper?

您好，看代码后发现sp_conv_forward的 returning_active_ex=True，也就是说在卷积部分没有进行稀疏计算，而是在batchnorm和layernorm上进行稀疏计算吗？
感谢您的工作！

请问有在训练过程中出现这个问题吗？

Hi, thanks for your work. In general, for smaller batch size, a smaller learning rate is needed. Why SparK needs a larger learning rate with smaller batch size while pretraining on 384x384?

hi~I find I can pre-train resnet in my own dataset by replacing the function [build_dataset_to_pretrain] and running /pretrain/main.py ; also I can finetune resnet in Imagenet by running /downstream-imagenet/main.py; but how can I finetune resnet in my own dataset? when I try to run /pretrain/main.py with args --resume_from=/mypath-to-res50_withdecoder_1kpretrained_spark_style.pth, I will get a error like
File "/workspace/user_code/SparK/pretrain/utils/misc.py", line 180, in load_checkpoint
missing, unexpected = model_without_ddp.load_state_dict(checkpoint['module'], strict=False)
KeyError: 'module'

File "E:/code/SparK-main/SparK-main-latest/pretrain/vis_reconstruction.py", line 55, in build_spark
assert len(missing) == 0, f'load_state_dict missing keys: {missing}'
AssertionError: load_state_dict missing keys: ['imn_m', 'imn_s', 'norm_black', 'sparse_encoder.sp_cnn.conv1.weight', 'sparse_encoder.sp_cnn.bn1.weight', 'sparse_encoder.sp_cnn.bn1.bias', 'sparse_encoder.sp_cnn.bn1.running_mean', 'sparse_encoder.sp_cnn.bn1.running_var', 'sparse_encoder.sp_cnn.layer1.0.conv1.weight', 'sparse_encoder.sp_cnn.layer1.0.bn1.weight', 'sparse_encoder.sp_cnn.layer1.0.bn1.bias', 'sparse_encoder.sp_cnn.layer1.0.bn1.running_mean', 'sparse_encoder.sp_cnn.layer1.0.bn1.running_var', 'sparse_encoder.sp_cnn.layer1.0.conv2.weight', 'sparse_encoder.sp_cnn.layer1.0.bn2.weight', 'sparse_encoder.sp_cnn.layer1.0.bn2.bias', 'sparse_encoder.sp_cnn.layer1.0.bn2.running_mean', 'sparse_encoder.sp_cnn.layer1.0.bn2.running_var', 'sparse_encoder.sp_cnn.layer1.0.conv3.weight', 'sparse_encoder.sp_cnn.layer1.0.bn3.weight', 'sparse_encoder.sp_cnn.layer1.0.bn3.bias', 'sparse_encoder.sp_cnn.layer1.0.bn3.running_mean', 'sparse_encoder.sp_cnn.layer1.0.bn3.running_var', 'sparse_encoder.sp_cnn.layer1.0.downsample.0.weight', 'sparse_encoder.sp_cnn.layer1.0.downsample.1.weight', 'sparse_encoder.sp_cnn.layer1.0.downsample.1.bias', 'sparse_encoder.sp_cnn.layer1.0.downsample.1.running_mean', 'sparse_encoder.sp_cnn.layer1.0.downsample.1.running_var', 'sparse_encoder.sp_cnn.layer1.1.conv1.weight', 'sparse_encoder.sp_cnn.layer1.1.bn1.weight', 'sparse_encoder.sp_cnn.layer1.1.bn1.bias', 'sparse_encoder.sp_cnn.layer1.1.bn1.running_mean', 'sparse_encoder.sp_cnn.layer1.1.bn1.running_var', 'sparse_encoder.sp_cnn.layer1.1.conv2.weight', 'sparse_encoder.sp_cnn.layer1.1.bn2.weight', 'sparse_encoder.sp_cnn.layer1.1.bn2.bias', 'sparse_encoder.sp_cnn.layer1.1.bn2.running_mean', 'sparse_encoder.sp_cnn.layer1.1.bn2.running_var', 'sparse_encoder.sp_cnn.layer1.1.conv3.weight', 'sparse_encoder.sp_cnn.layer1.1.bn3.weight', 'sparse_encoder.sp_cnn.layer1.1.bn3.bias', 'sparse_encoder.sp_cnn.layer1.1.bn3.running_mean', 'sparse_encoder.sp_cnn.layer1.1.bn3.running_var', 'sparse_encoder.sp_cnn.layer1.2.conv1.weight', 'sparse_encoder.sp_cnn.layer1.2.bn1.weight', 'sparse_encoder.sp_cnn.layer1.2.bn1.bias', 'sparse_encoder.sp_cnn.layer1.2.bn1.running_mean', 'sparse_encoder.sp_cnn.layer1.2.bn1.running_var', 'sparse_encoder.sp_cnn.layer1.2.conv2.weight', 'sparse_encoder.sp_cnn.layer1.2.bn2.weight', 'sparse_encoder.sp_cnn.layer1.2.bn2.bias', 'sparse_encoder.sp_cnn.layer1.2.bn2.running_mean', 'sparse_encoder.sp_cnn.layer1.2.bn2.running_var', 'sparse_encoder.sp_cnn.layer1.2.conv3.weight', 'sparse_encoder.sp_cnn.layer1.2.bn3.weight', 'sparse_encoder.sp_cnn.layer1.2.bn3.bias', 'sparse_encoder.sp_cnn.layer1.2.bn3.running_mean', 'sparse_encoder.sp_cnn.layer1.2.bn3.running_var', 'sparse_encoder.sp_cnn.layer2.0.conv1.weight', 'sparse_encoder.sp_cnn.layer2.0.bn1.weight', 'sparse_encoder.sp_cnn.layer2.0.bn1.bias', 'sparse_encoder.sp_cnn.layer2.0.bn1.running_mean', 'sparse_encoder.sp_cnn.layer2.0.bn1.running_var', 'sparse_encoder.sp_cnn.layer2.0.conv2.weight', 'sparse_encoder.sp_cnn.layer2.0.bn2.weight', 'sparse_encoder.sp_cnn.layer2.0.bn2.bias', 'sparse_encoder.sp_cnn.layer2.0.bn2.running_mean', 'sparse_encoder.sp_cnn.layer2.0.bn2.running_var', 'sparse_encoder.sp_cnn.layer2.0.conv3.weight', 'sparse_encoder.sp_cnn.layer2.0.bn3.weight', 'sparse_encoder.sp_cnn.layer2.0.bn3.bias', 'sparse_encoder.sp_cnn.layer2.0.bn3.running_mean', 'sparse_encoder.sp_cnn.layer2.0.bn3.running_var', 'sparse_encoder.sp_cnn.layer2.0.downsample.0.weight', 'sparse_encoder.sp_cnn.layer2.0.downsample.1.weight', 'sparse_encoder.sp_cnn.layer2.0.downsample.1.bias', 'sparse_encoder.sp_cnn.layer2.0.downsample.1.running_mean', 'sparse_encoder.sp_cnn.layer2.0.downsample.1.running_var', 'sparse_encoder.sp_cnn.layer2.1.conv1.weight', 'sparse_encoder.sp_cnn.layer2.1.bn1.weight', 'sparse_encoder.sp_cnn.layer2.1.bn1.bias', 'sparse_encoder.sp_cnn.layer2.1.bn1.running_mean', 'sparse_encoder.sp_cnn.layer2.1.bn1.running_var', 'sparse_encoder.sp_cnn.layer2.1.conv2.weight', 'sparse_encoder.sp_cnn.layer2.1.bn2.weight', 'sparse_encoder.sp_cnn.layer2.1.bn2.bias', 'sparse_encoder.sp_cnn.layer2.1.bn2.running_mean', 'sparse_encoder.sp_cnn.layer2.1.bn2.running_var', 'sparse_encoder.sp_cnn.layer2.1.conv3.weight', 'sparse_encoder.sp_cnn.layer2.1.bn3.weight', 'sparse_encoder.sp_cnn.layer2.1.bn3.bias', 'sparse_encoder.sp_cnn.layer2.1.bn3.running_mean', 'sparse_encoder.sp_cnn.layer2.1.bn3.running_var', 'sparse_encoder.sp_cnn.layer2.2.conv1.weight', 'sparse_encoder.sp_cnn.layer2.2.bn1.weight', 'sparse_encoder.sp_cnn.layer2.2.bn1.bias', 'sparse_encoder.sp_cnn.layer2.2.bn1.running_mean', 'sparse_encoder.sp_cnn.layer2.2.bn1.running_var', 'sparse_encoder.sp_cnn.layer2.2.conv2.weight', 'sparse_encoder.sp_cnn.layer2.2.bn2.weight', 'sparse_encoder.sp_cnn.layer2.2.bn2.bias', 'sparse_encoder.sp_cnn.layer2.2.bn2.running_mean', 'sparse_encoder.sp_cnn.layer2.2.bn2.running_var', 'sparse_encoder.sp_cnn.layer2.2.conv3.weight', 'sparse_encoder.sp_cnn.layer2.2.bn3.weight', 'sparse_encoder.sp_cnn.layer2.2.bn3.bias', 'sparse_encoder.sp_cnn.layer2.2.bn3.running_mean', 'sparse_encoder.sp_cnn.layer2.2.bn3.running_var', 'sparse_encoder.sp_cnn.layer2.3.conv1.weight', 'sparse_encoder.sp_cnn.layer2.3.bn1.weight', 'sparse_encoder.sp_cnn.layer2.3.bn1.bias', 'sparse_encoder.sp_cnn.layer2.3.bn1.running_mean', 'sparse_encoder.sp_cnn.layer2.3.bn1.running_var', 'sparse_encoder.sp_cnn.layer2.3.conv2.weight', 'sparse_encoder.sp_cnn.layer2.3.bn2.weight', 'sparse_encoder.sp_cnn.layer2.3.bn2.bias', 'sparse_encoder.sp_cnn.layer2.3.bn2.running_mean', 'sparse_encoder.sp_cnn.layer2.3.bn2.running_var', 'sparse_encoder.sp_cnn.layer2.3.conv3.weight', 'sparse_encoder.sp_cnn.layer2.3.bn3.weight', 'sparse_encoder.sp_cnn.layer2.3.bn3.bias', 'sparse_encoder.sp_cnn.layer2.3.bn3.running_mean', 'sparse_encoder.sp_cnn.layer2.3.bn3.running_var', 'sparse_encoder.sp_cnn.layer3.0.conv1.weight', 'sparse_encoder.sp_cnn.layer3.0.bn1.weight', 'sparse_encoder.sp_cnn.layer3.0.bn1.bias', 'sparse_encoder.sp_cnn.layer3.0.bn1.running_mean', 'sparse_encoder.sp_cnn.layer3.0.bn1.running_var', 'sparse_encoder.sp_cnn.layer3.0.conv2.weight', 'sparse_encoder.sp_cnn.layer3.0.bn2.weight', 'sparse_encoder.sp_cnn.layer3.0.bn2.bias', 'sparse_encoder.sp_cnn.layer3.0.bn2.running_mean', 'sparse_encoder.sp_cnn.layer3.0.bn2.running_var', 'sparse_encoder.sp_cnn.layer3.0.conv3.weight', 'sparse_encoder.sp_cnn.layer3.0.bn3.weight', 'sparse_encoder.sp_cnn.layer3.0.bn3.bias', 'sparse_encoder.sp_cnn.layer3.0.bn3.running_mean', 'sparse_encoder.sp_cnn.layer3.0.bn3.running_var', 'sparse_encoder.sp_cnn.layer3.0.downsample.0.weight', 'sparse_encoder.sp_cnn.layer3.0.downsample.1.weight', 'sparse_encoder.sp_cnn.layer3.0.downsample.1.bias', 'sparse_encoder.sp_cnn.layer3.0.downsample.1.running_mean', 'sparse_encoder.sp_cnn.layer3.0.downsample.1.running_var', 'sparse_encoder.sp_cnn.layer3.1.conv1.weight', 'sparse_encoder.sp_cnn.layer3.1.bn1.weight', 'sparse_encoder.sp_cnn.layer3.1.bn1.bias', 'sparse_encoder.sp_cnn.layer3.1.bn1.running_mean', 'sparse_encoder.sp_cnn.layer3.1.bn1.running_var', 'sparse_encoder.sp_cnn.layer3.1.conv2.weight', 'sparse_encoder.sp_cnn.layer3.1.bn2.weight', 'sparse_encoder.sp_cnn.layer3.1.bn2.bias', 'sparse_encoder.sp_cnn.layer3.1.bn2.running_mean', 'sparse_encoder.sp_cnn.layer3.1.bn2.running_var', 'sparse_encoder.sp_cnn.layer3.1.conv3.weight', 'sparse_encoder.sp_cnn.layer3.1.bn3.weight', 'sparse_encoder.sp_cnn.layer3.1.bn3.bias', 'sparse_encoder.sp_cnn.layer3.1.bn3.running_mean', 'sparse_encoder.sp_cnn.layer3.1.bn3.running_var', 'sparse_encoder.sp_cnn.layer3.2.conv1.weight', 'sparse_encoder.sp_cnn.layer3.2.bn1.weight', 'sparse_encoder.sp_cnn.layer3.2.bn1.bias', 'sparse_encoder.sp_cnn.layer3.2.bn1.running_mean', 'sparse_encoder.sp_cnn.layer3.2.bn1.running_var', 'sparse_encoder.sp_cnn.layer3.2.conv2.weight', 'sparse_encoder.sp_cnn.layer3.2.bn2.weight', 'sparse_encoder.sp_cnn.layer3.2.bn2.bias', 'sparse_encoder.sp_cnn.layer3.2.bn2.running_mean', 'sparse_encoder.sp_cnn.layer3.2.bn2.running_var', 'sparse_encoder.sp_cnn.layer3.2.conv3.weight', 'sparse_encoder.sp_cnn.layer3.2.bn3.weight', 'sparse_encoder.sp_cnn.layer3.2.bn3.bias', 'sparse_encoder.sp_cnn.layer3.2.bn3.running_mean', 'sparse_encoder.sp_cnn.layer3.2.bn3.running_var', 'sparse_encoder.sp_cnn.layer3.3.conv1.weight', 'sparse_encoder.sp_cnn.layer3.3.bn1.weight', 'sparse_encoder.sp_cnn.layer3.3.bn1.bias', 'sparse_encoder.sp_cnn.layer3.3.bn1.running_mean', 'sparse_encoder.sp_cnn.layer3.3.bn1.running_var', 'sparse_encoder.sp_cnn.layer3.3.conv2.weight', 'sparse_encoder.sp_cnn.layer3.3.bn2.weight', 'sparse_encoder.sp_cnn.layer3.3.bn2.bias', 'sparse_encoder.sp_cnn.layer3.3.bn2.running_mean', 'sparse_encoder.sp_cnn.layer3.3.bn2.running_var', 'sparse_encoder.sp_cnn.layer3.3.conv3.weight', 'sparse_encoder.sp_cnn.layer3.3.bn3.weight', 'sparse_encoder.sp_cnn.layer3.3.bn3.bias', 'sparse_encoder.sp_cnn.layer3.3.bn3.running_mean', 'sparse_encoder.sp_cnn.layer3.3.bn3.running_var', 'sparse_encoder.sp_cnn.layer3.4.conv1.weight', 'sparse_encoder.sp_cnn.layer3.4.bn1.weight', 'sparse_encoder.sp_cnn.layer3.4.bn1.bias', 'sparse_encoder.sp_cnn.layer3.4.bn1.running_mean', 'sparse_encoder.sp_cnn.layer3.4.bn1.running_var', 'sparse_encoder.sp_cnn.layer3.4.conv2.weight', 'sparse_encoder.sp_cnn.layer3.4.bn2.weight', 'sparse_encoder.sp_cnn.layer3.4.bn2.bias', 'sparse_encoder.sp_cnn.layer3.4.bn2.running_mean', 'sparse_encoder.sp_cnn.layer3.4.bn2.running_var', 'sparse_encoder.sp_cnn.layer3.4.conv3.weight', 'sparse_encoder.sp_cnn.layer3.4.bn3.weight', 'sparse_encoder.sp_cnn.layer3.4.bn3.bias', 'sparse_encoder.sp_cnn.layer3.4.bn3.running_mean', 'sparse_encoder.sp_cnn.layer3.4.bn3.running_var', 'sparse_encoder.sp_cnn.layer3.5.conv1.weight', 'sparse_encoder.sp_cnn.layer3.5.bn1.weight', 'sparse_encoder.sp_cnn.layer3.5.bn1.bias', 'sparse_encoder.sp_cnn.layer3.5.bn1.running_mean', 'sparse_encoder.sp_cnn.layer3.5.bn1.running_var', 'sparse_encoder.sp_cnn.layer3.5.conv2.weight', 'sparse_encoder.sp_cnn.layer3.5.bn2.weight', 'sparse_encoder.sp_cnn.layer3.5.bn2.bias', 'sparse_encoder.sp_cnn.layer3.5.bn2.running_mean', 'sparse_encoder.sp_cnn.layer3.5.bn2.running_var', 'sparse_encoder.sp_cnn.layer3.5.conv3.weight', 'sparse_encoder.sp_cnn.layer3.5.bn3.weight', 'sparse_encoder.sp_cnn.layer3.5.bn3.bias', 'sparse_encoder.sp_cnn.layer3.5.bn3.running_mean', 'sparse_encoder.sp_cnn.layer3.5.bn3.running_var', 'sparse_encoder.sp_cnn.layer4.0.conv1.weight', 'sparse_encoder.sp_cnn.layer4.0.bn1.weight', 'sparse_encoder.sp_cnn.layer4.0.bn1.bias', 'sparse_encoder.sp_cnn.layer4.0.bn1.running_mean', 'sparse_encoder.sp_cnn.layer4.0.bn1.running_var', 'sparse_encoder.sp_cnn.layer4.0.conv2.weight', 'sparse_encoder.sp_cnn.layer4.0.bn2.weight', 'sparse_encoder.sp_cnn.layer4.0.bn2.bias', 'sparse_encoder.sp_cnn.layer4.0.bn2.running_mean', 'sparse_encoder.sp_cnn.layer4.0.bn2.running_var', 'sparse_encoder.sp_cnn.layer4.0.conv3.weight', 'sparse_encoder.sp_cnn.layer4.0.bn3.weight', 'sparse_encoder.sp_cnn.layer4.0.bn3.bias', 'sparse_encoder.sp_cnn.layer4.0.bn3.running_mean', 'sparse_encoder.sp_cnn.layer4.0.bn3.running_var', 'sparse_encoder.sp_cnn.layer4.0.downsample.0.weight', 'sparse_encoder.sp_cnn.layer4.0.downsample.1.weight', 'sparse_encoder.sp_cnn.layer4.0.downsample.1.bias', 'sparse_encoder.sp_cnn.layer4.0.downsample.1.running_mean', 'sparse_encoder.sp_cnn.layer4.0.downsample.1.running_var', 'sparse_encoder.sp_cnn.layer4.1.conv1.weight', 'sparse_encoder.sp_cnn.layer4.1.bn1.weight', 'sparse_encoder.sp_cnn.layer4.1.bn1.bias', 'sparse_encoder.sp_cnn.layer4.1.bn1.running_mean', 'sparse_encoder.sp_cnn.layer4.1.bn1.running_var', 'sparse_encoder.sp_cnn.layer4.1.conv2.weight', 'sparse_encoder.sp_cnn.layer4.1.bn2.weight', 'sparse_encoder.sp_cnn.layer4.1.bn2.bias', 'sparse_encoder.sp_cnn.layer4.1.bn2.running_mean', 'sparse_encoder.sp_cnn.layer4.1.bn2.running_var', 'sparse_encoder.sp_cnn.layer4.1.conv3.weight', 'sparse_encoder.sp_cnn.layer4.1.bn3.weight', 'sparse_encoder.sp_cnn.layer4.1.bn3.bias', 'sparse_encoder.sp_cnn.layer4.1.bn3.running_mean', 'sparse_encoder.sp_cnn.layer4.1.bn3.running_var', 'sparse_encoder.sp_cnn.layer4.2.conv1.weight', 'sparse_encoder.sp_cnn.layer4.2.bn1.weight', 'sparse_encoder.sp_cnn.layer4.2.bn1.bias', 'sparse_encoder.sp_cnn.layer4.2.bn1.running_mean', 'sparse_encoder.sp_cnn.layer4.2.bn1.running_var', 'sparse_encoder.sp_cnn.layer4.2.conv2.weight', 'sparse_encoder.sp_cnn.layer4.2.bn2.weight', 'sparse_encoder.sp_cnn.layer4.2.bn2.bias', 'sparse_encoder.sp_cnn.layer4.2.bn2.running_mean', 'sparse_encoder.sp_cnn.layer4.2.bn2.running_var', 'sparse_encoder.sp_cnn.layer4.2.conv3.weight', 'sparse_encoder.sp_cnn.layer4.2.bn3.weight', 'sparse_encoder.sp_cnn.layer4.2.bn3.bias', 'sparse_encoder.sp_cnn.layer4.2.bn3.running_mean', 'sparse_encoder.sp_cnn.layer4.2.bn3.running_var', 'dense_decoder.dec.0.up_sample.weight', 'dense_decoder.dec.0.up_sample.bias', 'dense_decoder.dec.0.conv.0.weight', 'dense_decoder.dec.0.conv.1.weight', 'dense_decoder.dec.0.conv.1.bias', 'dense_decoder.dec.0.conv.1.running_mean', 'dense_decoder.dec.0.conv.1.running_var', 'dense_decoder.dec.0.conv.3.weight', 'dense_decoder.dec.0.conv.4.weight', 'dense_decoder.dec.0.conv.4.bias', 'dense_decoder.dec.0.conv.4.running_mean', 'dense_decoder.dec.0.conv.4.running_var', 'dense_decoder.dec.1.up_sample.weight', 'dense_decoder.dec.1.up_sample.bias', 'dense_decoder.dec.1.conv.0.weight', 'dense_decoder.dec.1.conv.1.weight', 'dense_decoder.dec.1.conv.1.bias', 'dense_decoder.dec.1.conv.1.running_mean', 'dense_decoder.dec.1.conv.1.running_var', 'dense_decoder.dec.1.conv.3.weight', 'dense_decoder.dec.1.conv.4.weight', 'dense_decoder.dec.1.conv.4.bias', 'dense_decoder.dec.1.conv.4.running_mean', 'dense_decoder.dec.1.conv.4.running_var', 'dense_decoder.dec.2.up_sample.weight', 'dense_decoder.dec.2.up_sample.bias', 'dense_decoder.dec.2.conv.0.weight', 'dense_decoder.dec.2.conv.1.weight', 'dense_decoder.dec.2.conv.1.bias', 'dense_decoder.dec.2.conv.1.running_mean', 'dense_decoder.dec.2.conv.1.running_var', 'dense_decoder.dec.2.conv.3.weight', 'dense_decoder.dec.2.conv.4.weight', 'dense_decoder.dec.2.conv.4.bias', 'dense_decoder.dec.2.conv.4.running_mean', 'dense_decoder.dec.2.conv.4.running_var', 'dense_decoder.dec.3.up_sample.weight', 'dense_decoder.dec.3.up_sample.bias', 'dense_decoder.dec.3.conv.0.weight', 'dense_decoder.dec.3.conv.1.weight', 'dense_decoder.dec.3.conv.1.bias', 'dense_decoder.dec.3.conv.1.running_mean', 'dense_decoder.dec.3.conv.1.running_var', 'dense_decoder.dec.3.conv.3.weight', 'dense_decoder.dec.3.conv.4.weight', 'dense_decoder.dec.3.conv.4.bias', 'dense_decoder.dec.3.conv.4.running_mean', 'dense_decoder.dec.3.conv.4.running_var', 'dense_decoder.dec.4.up_sample.weight', 'dense_decoder.dec.4.up_sample.bias', 'dense_decoder.dec.4.conv.0.weight', 'dense_decoder.dec.4.conv.1.weight', 'dense_decoder.dec.4.conv.1.bias', 'dense_decoder.dec.4.conv.1.running_mean', 'dense_decoder.dec.4.conv.1.running_var', 'dense_decoder.dec.4.conv.3.weight', 'dense_decoder.dec.4.conv.4.weight', 'dense_decoder.dec.4.conv.4.bias', 'dense_decoder.dec.4.conv.4.running_mean', 'dense_decoder.dec.4.conv.4.running_var', 'dense_decoder.proj.weight', 'dense_decoder.proj.bias', 'densify_norms.0.weight', 'densify_norms.0.bias', 'densify_norms.1.weight', 'densify_norms.1.bias', 'densify_norms.2.weight', 'densify_norms.2.bias', 'densify_norms.3.weight', 'densify_norms.3.bias', 'densify_projs.0.weight', 'densify_projs.0.bias', 'densify_projs.1.weight', 'densify_projs.1.bias', 'densify_projs.2.weight', 'densify_projs.2.bias', 'densify_projs.3.weight', 'densify_projs.3.bias', 'mask_tokens.0', 'mask_tokens.1', 'mask_tokens.2', 'mask_tokens.3']

when running the command:
torchrun --nproc_per_node=4 --nnodes=1 --node_rank=0 --master_addr=localhost --master_port=0 ./pretrain/main.py --data_path=/data/ --exp_name=loftr_backbone1 --exp_dir=/Logs --model=loftr_backbone --bs=4
I come across the following error

Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.

fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).
fatal: not a git repository (or any parent up to mount point /)
Stopping at filesystem boundary (GIT_DISCOVERY_ACROSS_FILESYSTEM not set).

Im not using gitclone to download this repo but using download zip, I guess the error has something to do with it?
Do you konw how to fix this? Thanks!

This repo is using masked dense convolutions because it is optimized in torch. However, would this implementation speed things up or too complicated getting this work here?

Can I replace imagenet1k with another dataset?