After thinking a bit more, it would be nice to add a few lines to allow loading from other devices. Someone like @Mihonarium might want to load a pre-trained checkpoint and further train with different music data. As mentioned in the paper section 4.2, I've seen small performance gains by using 10% of test-dummy-set as training data. I call this use case as "seen database" to distinguish with general cases of fully "unseen" test set.

...............................................................................................................................................................................................................................................
What's even more strange, the issue with a lot of warnings appears only with run.py train and doesn't appear for generate.

The notebook: https://gist.github.com/Mihonarium/e3fd355cb560b82373fd2186139f1bc2 (the last cells show that generate and training from scratch work).

Originally posted by @Mihonarium in #10 (comment)

I'm using NVIDIA RTX A5000 .
I'm getting loss values as nan during training with default values.I'm getting this ..please clarify and solve the issue
W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation. Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal : Value 'sm_86' is not defined for option 'gpu-name'

Hello, I would like to know if the 500 test data you mentioned in your paper were randomly selected?Did you choose it only once? Or take the average of multiple times?
Is your database set up to ensure that it contains 500 test data?Because in general, in the audio query task, there should be a difference between the recall rate and the precision rate. I understand that the accuracy rate in your paper is the precision rate.
I'm using a program implemented by someone else in pytorch, have you checked the program's implementation and results?
When I tested, I found that the accuracy of 1-second and 2-second query tests fluctuated greatly with different data. Did you find this out?
And I think that there is also the possibility of muted segments for 1-second and 2-second queries, which will have a greater impact on the results. Therefore, your 1-second and 2-second accuracy rates are as stated in your paper. Is it right? Or is it more volatile?

Hello,
I executed the following command:

python run.py train CHECKPOINT --max_epoch=100 -c default

The code is throwing the following error on line 207:
if cfg['TRAIN']['SAVE_IMG'] and sim_mtx is not None:
UnboundLocalError: local variable 'sim_mtx' referenced before assignment

I am facing this issue and not sure how to fix it. Any insights on what might be causing this error?

Hi,
I'm interested in finding near-duplicate audio files. My dataset is about 3000 thousands short audio files, between 0.5 seconds to 5 seconds. Unlike Shazam, both the "target" audio (i.e. the songs in Shazam's case) and the user input are short, and both might contain noise.

Can this library help?
If so, are there any recommendations for tuning parameters?

N.B - if a file is matched to multiple other files, it's fine - I have a less efficient algorithm that can verify which match is correct. In other words, I can handle some amount of false positives, but I don't want false negatives.

Whether to check the similarity before audio storage?
Is there a simple demo for query and other operations ?

If I query the start and end positions of an audio, through faiss.search, it is possible that top n is a non-continuous value, how can I align it?

I followed the actions in the readme documentation to configure the environment（Create a virtual environment via .yml), and downloaded the Dataset-mini v1.1 to ../ . But the loss calculated when running run.py training is nan. When debugging, I found that when the data passed through the front_conv layer of the FingerPrinter model, the values of the calculated tensor were all 0 or nan. What‘s wrong and why is this happening?

Upload a few examples with Top1 score

• Image
• Audio
• Demo page

Hi, first of all great work. I am trying to use this model to generate some fingerprints for a single movie audio (let's say 2 hours long). I trained the model, following your instructions, and for the -s argument in the python run.py generate command I used the directory containing the movie audio. Then the fingerprints are generated, all good. I then proceed to read the produced memmap, I convert it into a numpy array and it's of size (16353,128) with duration=1 and hop=0.5 as parameters. All good until now. However, when I inspect the produced fingerprints, they are repeated every 125 fingerprints, meaning that it produces 16353/125=131 batches which are the same. Why is that? Don't you crop the audio file into segments and then pass it through the model, or should I do that beforehand? Thanks!

My model gradient explosion

Hi

Could you give a date on which the code and data set will be available.

Regards

First of all, thank you very much for the work, having access to the repository gave me a jump start to the field.

After reading the paper, downloading the dataset from IEEE dataport and running some experiments I noticed an issue.

• The paper reports that train, val, test_query, and test_dummy sets are disjoint with corresponding sizes 10,000, 500, 500, and 100,000. When you get the union of these disjoint sets, the total size is 110,000. When you consider the NAFP dataset to be a subset of the FMA dataset, this can not be true as FMA dataset (fma_full) contains 106,574 tracks. You can check this information at the FMA Github Repository.
• In fact, the IEEE dataport downloaded neural-audio-fp-dataset/music/test-dummy-db-100k-full/fma_full directory contains 93,458 wav tracks. This can be verified with running find -type f -name "*.wav" | wc -l from the directory mentioned previously.
• Therefore, running the checksum does not match the provided value at this repo.

Could you clarify this, please?

I think this can be the reason why the reported metrics on the paper and the metrics that we obtain by training an Adam N=120 model or evaluating the provided 640_lamb model are inconsistent. See: Issue related.

I did some experiments to find these 6,542 tracks. I will report them on the corresponding issue.

Hi, it might be related to this, but I'm trying to generate fingerprints from custom source using the pretranied model you shared here: #10 (comment) and I was wondering if you could tell me what's the expectated time for generating a fingerprint from a single query? Since it took 1629seconds to generate fingerprints corresponding to 2 queries (1-min length) [even though in the source directory there are 3 wav files, I'm studying why this as well ]
From the CLI Output: 2/2 [==============================] - 1629s 47ms/step

I'm using a 40-cpu server with a RTX3090.

Also, can you help me understanding the shape of the resulting db? I understand that the shape is n_items x d, and n_items is #num audios x batchsize. I don't see what this batchsize mean and therefore, the resulting db shape.

Thanks in advance!

Originally posted by @guillemcortes in #8 (comment)

Hello, I am using a pretrained model that you commented in one of your issues, and I want to use it for my own dataset, I modified the dataset class:

def get_custom_db_ds(self, source_root_dir):
    """ Construc DB (or query) from custom source files. """

    query_path = sorted(
        glob.glob(source_root_dir + 'query/*.wav'))
    db_path = sorted(
        glob.glob(source_root_dir + 'db/*.wav'))
    
    _ts_n_anchor = self.ts_batch_sz # Only anchors...

    query_ds = genUnbalSequence(
        db_path,
        self.ts_batch_sz,
        _ts_n_anchor,
        self.dur,
        self.hop,
        self.fs,
        shuffle=False,
        random_offset_anchor=False,
        drop_the_last_non_full_batch=False) # No augmentations, No drop-samples.
    db_ds = genUnbalSequence(
        query_path,
        self.ts_batch_sz,
        _ts_n_anchor,
        self.dur,
        self.hop,
        self.fs,
        shuffle=False,
        random_offset_anchor=False,
        drop_the_last_non_full_batch=False)
    return query_ds, db_ds

and the get_data_source function in generate.py:

def get_data_source(cfg, source_root_dir, skip_dummy):
    dataset = Dataset(cfg)
    ds = dict()
    if source_root_dir:
        ds['custom_source'] = dataset.get_custom_db_ds(source_root_dir)
    else:
        if skip_dummy:
            tf.print("Excluding \033[33m'dummy_db'\033[0m from source.")
            pass
        else:
            ds['dummy_db'] = dataset.get_test_dummy_db_ds()

        if dataset.datasel_test_query_db in ['unseen_icassp', 'unseen_syn']:
            ds['query'], ds['db'] = dataset.get_test_query_db_ds()
        elif dataset.datasel_test_query_db == 'custom_dataset':
            ds['query'], ds['db'] = dataset.get_custom_db_ds(cfg['DIR']['SOURCE_ROOT_DIR'])
        else:
            raise ValueError(dataset.datasel_test_query_db)

    tf.print(f'\x1b[1;32mData source: {ds.keys()}\x1b[0m',
             f'{dataset.datasel_test_query_db}')
    return ds

I am using a sample of 3 db audios files and 2 query audio files.

1. Generating the fingerprint gave a warning that the size does not match also note that I am using skip dummy since I am using my own dataset. I want to ask if this is fine.
2. also regarding search and evaluation, a part of the code depend on the dummy_db like training the faiss index, how should I adapt it for my case.
3. The test_seq_len argument is regarding how many segment from the query we should consider to match against the db?
4. can this model help me get from which timestamp in the top1 detected db audio does a start from.
5. If a query contains parts from two database audios is it possible to get a probability score so that I can condition on a threshold to detect if it comes from more than one audio db or not.

Thank in advance, and thanks for you greate work

While typing this command
$python run.py evaluate CHECKPOINT_NAME CHECKPOINT_INDEX

FileNotFoundError: [Errno 2] No such file or directory: './logs/emb/CHECKPOINT_NAME/CHECKPOINT_INDEX/query_shape.npy'

Im getting this error, please help me for search and evaluation process.
Thank you!

how much the loss when the model convergence

hello, Can I refer to your code for my advisement retrival ?

• config file
  • default tau: 0.1 --> 0.05
  • added new preset for BSZ=640
• run.py
  • correct evaluate command's default to 'icassp'
• eval/eval_faiss.py
  • test_ids='all' works
• model/train.py:
  • validatation size 250 --> 500

dinesh@fourier:~/neural-audio-fp$ docker build -t neural-audio-fp .
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&shmsize=0&t=neural-audio-fp&target=&ulimits=null&version=1": dial unix /var/run/docker.sock: connect: permission denied

hi, I have about 900 short audios of users speaking, after training the main model for 20 epochs(converging very well) I try to finetune it on my own data but after 130 epochs the loss does not converge and stays in the range of 0.99-0.95 . Are there any configurations I am missing that need to be altered in the case of nonmusic data

or checksum of other files separately so that I can locate the different files...

Lets say dog is barking in two seconds by more than 4 times,So that each time sound is similar to remaning barks in the same file.We have defined postive pairs are ORIGINAL and It's AUGMENTED REPLICAS.But here bark sounds from one dog is almost similar then how can they be differentiate with augmented or else will they also become postive pairs(REAL SAMPLES)?In pairwise similarity simply real circle (not dashed) presented more than ones in a row.What is the case at this moment?

Dataset update

• Early downloaders (before July 8, 2021, 2:31 PM GMT) need this update via Dataport.
• The test set included in the initial package was labeld as SNR [0,10] dB, but it was actually [10,10] dB (easier test). Mistake during directory cleanup. It's fixed now.
• Now v1.1 with SNR= [0,10]dB, 0dB, -3dB queries.
• I found 24 duplicate (between the dummy and test sets) songs in the previous data set for publication results. I also had to replace a few songs for the training set. Datasets v1-v1.1 is very clean, as I double-checked.
• One problem (?) is that after the data set correction, the performance improved by almost 5 to 6 percent for 1 second query.

In progress:

• Fast download with command-line interface (kaggle public dataset?)
• md5/sha1 checksum
• Upload full dataset after correcting dataset duplicates

Hey team! I'm looking into the model definition here.

Should

 front_strides=[[(1,2), (2,1)], [(1,2), (2,1)],
                [(1,2), (2,1)], [(1,2), (2,1)],
                [(1,1), (2,1)], [(1,2), (2,1)],
                [(1,1), (2,1)], [(1,2), (2,1)]],

be

 front_strides=[[(1,2), (2,1)], [(1,2), (2,1)],
                [(1,2), (2,1)], [(1,2), (2,1)],
                [(1,2), (2,1)], [(1,2), (2,1)],
                [(1,2), (2,1)], [(1,2), (2,1)]],

Or is the former intended and, if so, why?

Hi all, sorry for my lateness!! I've been replying to emails, but I saw the issue board for the first time today. My bad...

• For someone in a hurry, I am releasing a working code (training + mini evaluation) in advance now.
• I can complete everything (dataset upload, easy setup for FAISS-GPU, fully repoducing all experiments, etc.) by the end of this week.

Just wait a few more days. I'm working on this project full time now... :)

1. I cloned this repo.
2. Downloaded the dataset mini from the google drive.
3. Created the docker container by pulling the image provided.
4. Ran python run.py train test123 --max_epoch=100 -c default inside the docker.

I am seeing all NaNs in the tensorboard logs for training loss. I am running this on a machine with 3 NVIDIA GeForce 3090 GPUs. What might be going wrong?

Is it possible include code for the Google's now playing model here? Even if it's not functional.

While it's relatively easy to train the model on the Dataset-mini (even Colab allows that), it's not as easy to reproduce the paper's results with the Dataset-full. It would be great if you could publish a model trained on the full dataset.

(By the way, congratulations on the paper, and thanks for publishing the work, it's really cool!)

I'm tring to reproduce your paper results.

In my resource, I can use only TR_BATCH_SZ=120.
So your paper said top-1 exact match rate 55.9% and near match rate 62.3% at query length 1s in Table 3.
But I get 67.95% of exact match rate and 73.0% of near match rate. (see below)
Is it an applicable result?

$ time -p python run.py evaluate test 100
cli: Configuration from ./config/default.yaml
Load 29,500 items from ./logs/emb/test/100/query.mm.
Load 29,500 items from ./logs/emb/test/100/db.mm.
Load 54,336,000 items from ./logs/emb/test/100/dummy_db.mm.
Creating index: ivfpq
Copy index to GPU.
Training index using 18.40 % of data...
Elapsed time: 31.04 seconds.
54336000 items from dummy DB
29500 items from reference DB
Added total 54365500 items to DB. 104.26 sec.
Created fake_recon_index, total 54365500 items. 0.11 sec.
test_id: icassp,  n_test: 2000
========= Top1 hit rate (%) of segment-level search =========
               ---------------- Query length ----------------
   segments      1        3       5       9       11      19     
   seconds      (1s)     (2s)    (3s)    (5s)    (6s)   (10s)    

  Top1 exact   67.95    88.90   94.30   97.35   98.10   99.15
  Top1 near    73.00    90.30   94.70   97.40   98.15   99.20
  Top3 exact   76.65    92.05   95.65   98.25   98.85   99.55
 Top10 exact   80.00    92.65   96.05   98.45   99.00   99.60
=============================================================
average search + evaluation time 18.77 ms/query
Saved test_ids and raw score to ./logs/emb/test/100/.
real 364.41
user 544.26
sys 115.40

I use dataset-full v1.1 from ieee-dataport.org.

• https://ieee-dataport.org/open-access/neural-audio-fingerprint-dataset#files

I change configuration TEST_DUMMY_DB=100k_full_icassp.
Then train & generate, evaluate

I want to generate just the fingerprints for custom wav file as input. I just want to generate fingerprint for one single wav file. Could someone point me to a script which can do so.
e.g. If I have a wav file of 10s and my seg length is 1s and hop 0.5s, then I want a fingerprint output of size (19 x fp_dim).

Hello, I found this paper quite interesting and encountered some issues during the testing process with my custom data. I apologize if my questions seem naive, as I lack some knowledge in this area.

In my case, I don't need an algorithm to identify a specific audio from various songs using audio fragments. I have a single audio file (2-3 hours) and need to find where in this file a certain audio fragment (3-5 seconds) begins (I understand from the issue check that I need to customize the process for obtaining the start timestamp).

1. Is this code suitable for such a scenario?

2. I trained using the provided dataset mini. Then I used the command python run.py generate --source CUSTOM_SOURCE_ROOT_DIR --output FP_OUTPUT_DIR --skip_dummy to generate fingerprints for my custom data, which is an audio file. Afterward, I wanted to evaluate a short audio fragment (3-5 second wav file) but wasn't sure how to proceed. Also, is this a meaningful process?

3. Should my custom audio data also be included in the training?

Thank you.

Unable to open the file "../demo_template.ipynb"

We generate segment-wise embeddings zt∈Z that can represent a unit segment of audio from the acoustic features S at time step t.
In this line do each zt is of dimension d or dimension 1.

Is this repository helpful for the below situation?
There could be situations where we need to detect the presence of new
sound sources for whose prerecorded acoustic signatures are available.
In order to handle such novel situations, the proposed system will be
equipped with audio search capability, which performs template
matching to detect the sound sources which match the given acoustic
signature (pre-recorded sound from that source).

I tried to reproduce your paper, and my code is https://github.com/stdio2016/pfann .
In my code, I generate queries by:

1. Randomly slice a x second segment from test music, x is query length
2. Add one noise file to this segment
3. Add 2 IRs to this segment, one is for room reverb, and the other is for microphone IR
4. Save this segment as query file

It seems that your code does these:

1. Split test music into 1 second segments
2. For each segment:
3. Randomly time shift the segment within +/-0.2s
4. Add one noise segment to the segment. The added noises of each segment seem to maintain time order.
5. Add one IR file to the segment.
6. Concatenate these segments and save as query file

My question:

1. Why do you add different IR to different 1s segments of the same query file? I do not think that the reverb environment would change every 1s.
2. I use random slicing to simulate query start time, while you randomly shift each 1s segment independently. Isn't uniform time shifting enough?
3. In your paper, you said "microphone and room impulse response (IR) are sequentially applied by convolution operation." However, I can only find one convolution operation per segment. How do you apply 2 IRs (microphone and room IRs) in one convolution? Do you merge these two datasets, or preprocess so that every new IR is a combination of one microphone and one room IR?