Hi! First off, excellent work with the module. It's showing great results so far in my

FSD50K - validating on eval data about passt HOT 5 CLOSED

kkoutini commented on May 27, 2024

FSD50K - validating on eval data

from passt.

Comments (5)

kkoutini commented on May 27, 2024

Hi! thank you!
The evaluate_only command get the validation dataset here, you can edit this line to load any dataloader

val_loader = get_my_loader()

If you want to validate on another dataset, you can create for exmaple a custom loader (like this one )

get_my_loader = ex.datasets.eval.iter(DataLoader, static_args=dict(worker_init_fn=worker_init_fn),
                                        validate=False, batch_size=10, num_workers=16,
                                        dataset=CMD("/basedataset.get_my_eval_set"))

you need however to define the function that gets the dataset /basedataset.get_my_eval_set which can be similar to this

@dataset.command
def get_my_baseeval_set(my_eval_hdf5="path_to_my_hdf.hdf", variable_eval=None):
    if variable_eval:
        print("Variable length eval!!")
        ds = AudioSetDataset(my_eval_hdf5, clip_length=None)
    else:
        ds = AudioSetDataset(my_eval_hdf5)
    return ds

@dataset.command
def get_my_eval_set(normalize):
    ds = get_my_baseeval_set()
    if normalize:
        print("normalized test!")
        fill_norms()
        ds = PreprocessDataset(ds, norm_func)
    return ds

from passt.

Ludvig-Joborn commented on May 27, 2024

First off; thank you a lot for answering! Second; forgive me, I made a silly mistake and wrote the wrong command in my earlier post. The command I used was

python ex_fsd50k.py with  passt_s_swa_p16_s16_128_ap473

and it makes use of the evaluation dataset for validation, which confuses me. I found this by changing the dataset length as described earlier. Is there something I'm misunderstanding?

from passt.

kkoutini commented on May 27, 2024

Hi! yes the default behaviour is that the model is evaluated on both datasets.
You can change that here by setting validate=False
or adding datasets.eval.validate=False to your command:

python ex_fsd50k.py with  passt_s_swa_p16_s16_128_ap473 datasets.eval.validate=False

from passt.

Ludvig-Joborn commented on May 27, 2024

Hi again. When I try to run the program as described above, I get this error:

ERROR - fsd50k - Failed after 0:03:31!
Traceback (most recent calls WITHOUT Sacred internals):
  File "ex_fsd50k.py", line 391, in evaluate_only
    res = trainer.validate(modul, val_dataloaders=val_loader)
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 883, in validate
    results = self.fit(model)
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 474, in fit
    self.dispatch()
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 513, in dispatch
    self.accelerator.start_evaluating(self)
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/accelerators/accelerator.py", line 95, in start_evaluating
    self.training_type_plugin.start_evaluating(trainer)
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 139, in start_evaluating
    self._results = trainer.run_evaluate()
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 745, in run_evaluate
    eval_loop_results, _ = self.run_evaluation()
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/trainer.py", line 700, in run_evaluation
    deprecated_eval_results = self.evaluation_loop.evaluation_epoch_end()
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/evaluation_loop.py", line 187, in evaluation_epoch_end
    deprecated_results = self.__run_eval_epoch_end(self.num_dataloaders)
  File "/home/ludvigj/Desktop/PaSST/src/pytorch-lightning/pytorch_lightning/trainer/evaluation_loop.py", line 222, in __run_eval_epoch_end
    eval_results = model.validation_epoch_end(eval_results)
  File "ex_fsd50k.py", line 216, in validation_epoch_end
    avg_loss = torch.stack([x[net_name + 'val_loss'] for x in one_outputs]).mean()
  File "ex_fsd50k.py", line 216, in <listcomp>
    avg_loss = torch.stack([x[net_name + 'val_loss'] for x in one_outputs]).mean()
TypeError: string indices must be integers

I've spent quite some time trying to make it work but can't get it right. There seems to be some trouble in validation_epoch_end. You don't happen to know what needs to be changed?

from passt.

Ludvig-Joborn commented on May 27, 2024

I think I've got it working now by copying the code for validation_epoch_end from ex_audioset.py, so everything should be settled now!

from passt.

FSD50K - validating on eval data about passt HOT 5 CLOSED

Comments (5)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent