Hello,

thank you very much for your very nice work!

I was wondering why not evaluate your model on some speech-related tasks, e.g ASR and speaker verification ?
Or make a comparision with Hubert or WavLM?

Thank you!

For pretraining, I ran main_pretrain.py but it needed train_all_video.json for processing. Would you let me know where I can find train_all_video.json?

Thanks for the nice work. I have several questions that I would really appreciate your input on them.

1. I tried to reproduce the results of ESC-50 dataset using your shared pre-trained weights and the provided file for finetuning but I could get only 89.5% while the reported results are above 94%, any idea why the performance is less?
   Also, I put the pre-trained weights on the AST framework for finetuning and I got 89% accuracy.

2. Why there is a huge performance jump in the case of SID despite not using external speech dataset?

3. Lastly, The proposed paper is quite similar to MAE-AST, which is essentially using MAE for Audio nevertheless there is a huge gap in performance between MAE-AST and your reported results, what are your thoughts in that?

Thanks a lot,

Hello,

Thanks for the great repo.

I am trying to reproduce the visualization results in the paper for the reconstructed spectrograms. Following the demo notebook and using the pretrained ViT-B checkpoint, the results I got (see attached) are notably worse than those reported in the paper.

I note that the visualizations in the paper are based on the larger ViT-L model. Is it possible for you to share the pretrained checkpoint?

Additionally, can you confirm whether the model configuration used in the notebook is correct?

Thanks in advance!

(Masked with a ratio of 0.3)

(Reconstructed patches)

Thank you meta for your hard work on the audioMAE implementation.
I want to train with 2M data, but in fact, audioset is only releasing features, so I couldn't get the data. I was finally able to get 20k data from another website. Where do I download the 2M data and I can't find pretrain_audioset2M.sh. Check please.

Can't download any ckpt files in github.

I wish to fine-tune pre-trained model on a downstream sound similarity task, I wish to apply self-supervied model without the influence of annotation.
Could u release the self-supervised weights without fine-tuning on audioset?

Thanks for the great work.

When I reproduce the finetuning result on Audioset-20k, the best I can get is 35.4 while the reported result is 37.0.

I use the example script in this repo.

In addition, I vary the value of the learning rate from [-1e-4, -1e-2] and the value of batch size from [4, 64], the best result remains 35.4.

Could you provide some ideas to have better results?

Hi, I couldn't find it, Maybe I get something wrong?
Thank you !!

When will you release the code?

I try finetune on my datasets output this error

i think audio-clip or audio-flamingo will coming soon !!!!!

Hi there! Are there plans to open source the models? There's an existing FaceBook org at Hugging Face with over 300 models where the model could be hosted.

The Hub offers free hosting of over 100K models, and it would make your work more accessible and visible to the rest of the ML community. Creating the repos and adding new models should be relatively straightforward if you've used Git before. This is a step-by-step guide explaining the process in case you're interested. The models could have model cards with metadata and metrics for discoverability. Please let us know if you would be interested, and if you have any questions, we're happy to assist.

Happy to hear your thoughts!

cc @NimaBoscarino

Hello,

I am working with the AudioMAE codebase and have a question about the implementation of the _roll_mag_aug function.

I've noticed that the index for rolling the waveform is determined by np.random.randint(len(waveform)).

However, for a waveform with the shape (1, 16000), this approach seems to always return 0, leading to no actual shift in the time axis of the waveform.

Could you please clarify if this is the intended behavior? If the goal is to roll along the time axis, would it be more appropriate to use np.random.randint(waveform.shape[1]) to ensure variability in the rolling process?

Any insights would be greatly appreciated, as I aim to understand and utilize this function correctly in my project.

Thank you for your time and the amazing work on AudioMAE !

Hi there,

I am wondering what does fbank really give us in the dataloader? I went to torchaudio doc and did not find much info about what it is. Does anyone have a link to its explanation?

Thank you,

Hello, thank you for your great work.

I recently started research about MAE and I want to use AudioMAE with VIT-S based pretrained model.

Could you provide the one in github?

Thanks in advance.

Hi! Great work!

I was trying to reproduce the demo in this notebook. While loading model weights from a pre-trained checkpoint using:

chkpt_dir = join(repo_path, "external/AudioMAE/checkpoints", "pretrained.pth")
assert os.path.exists(chkpt_dir), f"Checkpoint does not exist at {chkpt_dir}"

model = prepare_model(chkpt_dir, 'mae_vit_base_patch16')
print('Model loaded.')

I get the following warning message

_IncompatibleKeys(missing_keys=[], unexpected_keys=['decoder_blocks.8.attn.tau', 'decoder_blocks.8.attn.qkv.weight', 'decoder_blocks.8.attn.qkv.bias', 'decoder_blocks.8.attn.proj.weight', 'decoder_blocks.8.attn.proj.bias', 'decoder_blocks.8.attn.meta_mlp.fc1.weight', 'decoder_blocks.8.attn.meta_mlp.fc1.bias', 'decoder_blocks.8.attn.meta_mlp.fc2.weight', 'decoder_blocks.8.attn.meta_mlp.fc2.bias', 'decoder_blocks.8.norm1.weight', 'decoder_blocks.8.norm1.bias', 'decoder_blocks.8.mlp.fc1.weight', 'decoder_blocks.8.mlp.fc1.bias', 'decoder_blocks.8.mlp.fc2.weight', 'decoder_blocks.8.mlp.fc2.bias', 'decoder_blocks.8.norm2.weight', 'decoder_blocks.8.norm2.bias', 'decoder_blocks.9.attn.tau', 'decoder_blocks.9.attn.qkv.weight', 'decoder_blocks.9.attn.qkv.bias', 'decoder_blocks.9.attn.proj.weight', 'decoder_blocks.9.attn.proj.bias'.....

I believe it isn't loading the decoder weights correctly. Could you please help me fix this? @berniebear

Thanks!

I try finetune on my data output error

python3 main_finetune_as.py
--model $model
--dataset $dataset
--data_train $audioset_train_all_json
--data_eval $audioset_eval_json
--label_csv $audioset_label
--finetune $ckpt
--blr $blr
--epochs 30
--warmup_epochs 2
--first_eval_ep 2
--dist_eval
--batch_size 8
--roll_mag_aug True
--nb_classes 9 \

Hello!
Thank you for the great work.

I am trying to reproduce the inference results. But just run "inf.sh" with your released checkpoints, I have the following results:
Inf with pretrained.pth, I have 0.006102 mAP, which totally makes sense to me.
Inf with finetuned.pth, I have 0.3388 mAP, which is much lower than your reported mAP of 0.4729.

The only difference I think should be the audioset data only.

Could you please give me any hint on the much lower inference mAP numbers?

Thanks a lot!

Hello, I would like to use the speechcommands dataset for pretraining, but I have encounter an error.

The command is as follows:

python main_pretrain.py
--dataset='speechcommand'
--data_train='./speechcommand_train_data.json'
--data_eval='./datafiles/speechcommand_eval_data.json'
--label_csv='./speechcommands_class_labels_indices.csv' \

The following error occurs:

File "../miniconda3/envs/mae/lib/python3.9/site-packages/timm/models/swin_transformer.py", line 330, in _shifted_window_attn
x = x.view(B, H, W, C)
RuntimeError: shape '[16, 64, 8, 512]' is invalid for input of size 524288

I modify this line of code:
line 180 ( main_pretrain.py )
target_length = {'audioset':1024, 'esc50':512, 'speechcommands':128}

If I change " 'speechcommands':128 " to 1024, it runs smoothly,
but I want to execute it with 128.

Could you please help me understand where I went wrong? Thank you!

Hi, thank you for your wonderful work. I'm confuse about where is the weight_train.csv? and weight_train_all.csv?

#2

Excuse me, I am a beginner in the audio field, and I would like to ask you how to apply the pre-trained model to the esc-50 dataset.
The audio length of the esc50 data set is 5 seconds, but in the visualize demo provided by the author, the input of the pre-training model seems to be 10s.When I use this model to reconstruct the audio in esc50, the regular spectrogram is full of noise points.
Can you tell me how to use the pre-trained model to reconstruct and visualize the audio data in esc-50?

Thanks for your nice work！
I wish to inference the finetuned model with dataset ’AudioSet20K'. Could u release the self-supervised weights without fine-tuning on audioset?