• Sponsors
• Attention
• About
• Key Points
• Dependencies
• Running experiments
• Results
• Loading Pretrained Models
• References

leaf-pytorch implementation is now officially a part of SpeechBrain, with a sample recipe on SpeechCommands-v2 here. I would recommend folks trying to work with LEAF use SpeechBrain implementation instead, because of the overall ecosystem as well as better documentation. Thanks for your interest!

This work would not be possible without cloud resources provided by Google's TPU Research Cloud (TRC) program. I also thank the TRC support team for quickly resolving whatever issues I had: you're awesome!

This is a PyTorch implementation of the LEAF audio frontend [1], made using the official tensorflow implementation as a direct reference.
This implementation supports training on TPUs using torch-xla.

• Will be evaluated on AudioSet, SpeechCommands and Voxceleb1 datasets, and pretrained weights will be made available.
• Currently, torch-xla has some issues with certain complex64 operations: torch.view_as_real(comp), comp.real, comp.imag as highlighted in #Issue 3070. These are used primarily for generating gabor impulse responses. To bypass this shortcoming, an alternate implementation using manual complex number operations is provided.
• Matched performance on SpeechCommands, experiments on other datasets ongoing

torch >= 1.9.0
torchaudio >= 0.9.0
torch-audiomentations==0.9.0
SoundFile==0.10.3.post1
msgpack
msgpack-numpy
wandb
transformers
lmdb
[Optional] torch_xla == 1.9

Additional dependencies include

## needed for augmentations
[WavAugment](https://github.com/facebookresearch/WavAugment)

• The only thing cfgs (such as the efficientnet-b0 default cfg) need is a path to the "meta_root" under data section. Meta dir needs to have file manifest for each split as well as a lbl_map. A sample meta dir for SpeechCommands can be found here

To train a model on speechcommands, run the following:

python train.py --cfg_file cfgs/speechcommands/efficientnet-b0-leaf-default.cfg --expdir ./exps/scv2/efficientnet-b0_default_leaf_bs1x256_adam_warmupcosine_wd_1e-4_rs8881 --epochs 100 --num_workers 8 --log_steps 50 --random_seed 8881 --no_wandb

To evaluate the trained model, do

python test.py --test_csv_name ./speechcommands_v2_meta/test.csv --exp_dir ./exps/scv2/efficientnet-b0_default_leaf_bs1x256_adam_warmupcosine_wd_1e-4_rs8881 --meta_dir ./speechcommands_v2_meta

All experiments on VoxCeleb1 and SpeechCommands were repeated at least 5 times, and 95% ci are reported.

• ResNet-18 likely works better for VoxCeleb1 simply because it's a more difficult task than SpeechCommands and ResNet-18 has more parameters.

To evaluate how non-Mel initialization schemes for complex_conv work, experiments were repeated on xavier_normal, kaiming_normal and randn init schemes on the SpeechCommands dataset.

• download and extract desired ckpt from Results.

import os
import torch
import pickle
from models.classifier import Classifier

results_dir = "<path to results folder>"
hparams_path = os.path.join(results_dir, "hparams.pickle")
ckpt_path = os.path.join(results_dir, "ckpts", "<checkpoint.pth>")
checkpoint = torch.load(ckpt_path)
with open(hparams_path, "rb") as fp:
    hparams = pickle.load(fp)
model = Classifier(hparams.cfg)
print(model.load_state_dict(checkpoint['model_state_dict']))

# to access just the pretrained LEAF frontend
frontend = model.features

[1] If you use this repository, kindly cite the LEAF paper:

@article{zeghidour2021leaf,
  title={LEAF: A Learnable Frontend for Audio Classification},
  author={Zeghidour, Neil and Teboul, Olivier and de Chaumont Quitry, F{\'e}lix and Tagliasacchi, Marco},
  journal={ICLR},
  year={2021}
}

Please also consider citing this implementation using the following bibtex or from the citation widget on the sidebar.

@software{Yadav_leaf-pytorch_2021,
author = {Yadav, Sarthak},
month = {12},
title = {{leaf-pytorch}},
version = {0.0.1},
year = {2021}
}