Utility function under utils.py that transforms .wav to .png spectrogram.

Make sure that output spectrogram has enough length to be visible, else time expand the spectrogram.

• Can it be deployed in cloud? Or just personal use?
• Can it classify types of bat calls?
• Licensing?

Report results here.

https://github.com/umr-ds/transformer4bats

• Transformers are pretty data hungry?
• What data did they use to train?
• What are its classification capabilities?

under utils.py, create a data_download function with following requirements.

Input: URL path/directory path where raw data resides
Output: the downloaded directory containing raw data, will be in a form of directory.

We would like to be able to snippet spectrograms for each label identified by RavenPro in Python. The advantage will be twofold:

1. We will be able to identify False Positives much faster by observing individual .png files
2. We will start to develop useful dataprep code for our final model.

What are the details?

From Dr. Lee:

Forgot if I mentioned this but just thought of it: I think the package koogu can take in RavenPro detection table as inputs for training, and it is a complete framework for the detection from the look of it

time expand the original .wav file so that short calls can be captured

Run snippeter code with all current folders

Wu-Jung has already taken the time to upload to google drive a big enough sample of WAV files here. So we are only missing the labeling of bat calls using RavenPro.

I will chat with Wu-Jung and Aditya asking for this data, this will ideally solve our initial database problem and get us up and running.

Ensure have good understanding with the following workflow.

1. Open .wav and .txt files using RavenPro.
2. Know how to identify specific time box in .txt file.
3. Know how to annotate and make amendments to existing annotation.

http://visual.cs.ucl.ac.uk/pubs/batDetective/index.html

Questions to answer:
Model:

• Detect calls other than Search?
• Y-values of bounding box seem nonsensical (0 to 288?) What does it mean?
• Detection cuts early. Are there classification thresholds that can be adjusted?

Data:

• Are non-search calls labeled at all?
• Are non-search calls labeled as feeding and social?
• Quality of training data?

Misc:

• They used a "citizen scientists" to label their data. Can we leverage their system?

Sharing all insights and next steps found during the bat call detection and classification exploration:

A. Bat call detection

The idea here is to be able to detect any type of call in any frequency range and from any type of bat species.
Raven Pro: We explored RavenPro's capabilities, trying out the band limited detector, we noticed that it take very long and the project should be all held within one same pipeline so RavenPro would be a problem

Scikit-maad

We found a useful library that has most of the required capabilities for the project such as sound processing, segmentation and feature extraction. We believe this library will be very useful throughout the project. We found the following most useful for now:

1. trim: snipper that trims a sound file into smaller components
2. find_rois(): band limited detector, it works at the same level as RavenPro but inside Python which is a big plus.
3. spectrogram generation and plotting of bounding boxes.
4. shape and centroid feature extraction: It extracts from a set of bounding boxes the shape and frequency characteristics into a set of features that can later use for training any model

batdetect

This model based on the following paper: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005995 uses a pre-trained CNN using labeled calls from multiple different bat species around the globe. The authors have published open-source code here: https://github.com/macaodha/batdetect that uses Python 2.7 and for visualizations they reference another library that also uses Python 2.7 called AudioTagger. For our project we have found the following:

1. The code works even though it requires python 2.7, it does a good job at detecting a good amount of calls. In this case, we use the 20221012_030000.wav file and it detects a total of 23 calls (compared to the 42 detections found by RavenPro's Band-Limited Detector). Manually observing, it misses some fainter calls but captures all of the clear calls, there are no false positives detected. We can observe the results for this specific audio as follows:
   `
   LabelStartTime_Seconds | LabelEndTime_Seconds | DetectorConfidence

-- | -- | --

48.152 | 48.153 | 0.977

48.336 | 48.337 | 0.973

48.534 | 48.535 | 1

48.633 | 48.634 | 0.995

48.732 | 48.733 | 1

48.835 | 48.836 | 1

48.939 | 48.94 | 1

49.024 | 49.025 | 1

49.184 | 49.185 | 0.955

49.278 | 49.279 | 0.99

49.372 | 49.373 | 1

49.575 | 49.576 | 0.999

49.674 | 49.675 | 0.998

49.763 | 49.764 | 1

49.764 | 49.765 | 0.998

49.863 | 49.864 | 0.997

49.966 | 49.967 | 0.953

50.141 | 50.142 | 0.965

50.348 | 50.349 | 1

50.452 | 50.453 | 0.982

50.65 | 50.651 | 0.995

50.852 | 50.853 | 0.993

51.573 | 51.574 | 0.978

`
It starts to recognize from here:

But it misses multiple fainter such as these:

2. The GUI and plotter AudioTagger is old and we were not able to make it run.
3. The output format is a .csv that doesn't have the y-axis/frequency to be able to generate a bounding box and the precision on the x-axis is in the 10e-3 seconds, this could chop off part of the horizontal bounding box.

batdetect2

We have found that the same authors of batdetect have recently released a new version of batdetect called batdetect2. The open-source code can be found here: https://github.com/macaodha/batdetect2 and it is code explianed and used in the following paper: https://www.biorxiv.org/content/10.1101/2022.12.14.520490v1. This version includes multiple enhancements such as:

1. They use a transformer that considers the time aspect of bat calls, claiming to have much better accuracy.
2. It uses Python 3.10
3. They increase their dataset for training to include more species and countries
4. Their prediction now includes a potential species (class prediction)
5. They develop their own easy and intuitive GUI (Python 3.10) to annotate new audio as well as visualize results.
6. Their visualizations and output include detailed bounding boxes.
7. They have developed a Colab version of their model to run right away

For our project, we were able to easily clone this repository and run the pre-trained model. The results are also good, for the same sample audio file 20221012_030000.wav it identifies 59 calls, all of the ones that batdetect missed, all of them or almost all look like bat calls (need Wu Jung or Aditya to confirm) but clearly the ones that look less like bat calls have lower probabilities so with a higher threshold we would be able to remove these FP. There are no FN in this case, we can observe excerpts of this detection below:
Clearest part detection:

We obserrve that these detections might not be bat calls but they are assigned lower probabilities as we can see with the brightness of the box:

This model is robust, easy to run and fast when run with excerpts of less than 10 seconds.

B. Bat call classification

Even though batdetect2 is the best we have found to detect bat calls, it is robust only for social and search calls, but it hasn't proven to be able to detect feed buzzes.

As of now we have an idea in development to figure out if we are able to detect feed buzzes independently using template matching detection in scikit.maad library. If this detection works, we will be able to separately recognize feedbuzzes from social and search calls. Allowing us to automatically have a segmentation between these two types of calls. The preliminary results for this model in one audio file with 3 feedbing buzzes (20210910_030000_time1220_HFbuzz_LF.wav) are shown below:

The template matching was able to identify the 3 feed-buzzes using the first one as the template to recognize the following two.

C. Pipeline

We will initially implement these models into a Pipeline that approximately looks like this (subject to change):

1. User inputs 30 minute audio file
2. Audio file will be cut into 30 smaller 1 minute audio files using scikit.maad.trim()
3. Each of the 30 files will be processed for social or search call detection using batdetect2 labeled 'social or search call' and feedbuzz detection using template detection (or alternative feedbuzz detection model) labeled 'feed buzz'.
4. All detection will be re-contextualized to the 30 minute span of the audio file when merged toghether in a single .txt detection and classification file
5. This .txt file is output for the user.

D. Call with Juan Ulloa (creator of scikit-maad)

We were able to contact the creator of Scikit-maad who kindly dedicated an hour to respond to any of our questions, this summarizes what we found:

Q: Batdetec2 is not robust against buzz feeds, what do you recommend us to do?
A: He recommends to use template matching that is a function that is still in the development code of scikit-maad but it is stable so he told us to use it by accessing directly to that branch.

Q: What are centroid and shape features, how and for what can we use them?
A: Shape features are completely based on the bounding box, they describe using numbers what we observe within the bounding box. Centroid features allow us to understand where in the spectrogram (y-axis) this bounding box is. So if we mix both of these we will have a lot of information to describe a bounding box. This data can be used to feed into classifiers, ML models, etc. There are 3 types, 'Low', 'Med' and 'Hi', this is the resolution of the shape features, as you increase, it will describe better your shape (more angles "you can think of them as more convolutional filters") but it will be much more heavy (larger file). This link helps to understand this more: https://scikit-maad.github.io/generated/maad.features.opt_shape_presets.html#maad.features.opt_shape_presets

Q: How do we plot bounding boxes with maad?
A: Follow this tutorial page: https://scikit-maad.github.io/_auto_examples/1_basic/plot_find_rois_simple.html#sphx-glr-auto-examples-1-basic-plot-find-rois-simple-py

Q: What model would you recommend if we go for supervised learning with anbnotated datasets
A: He recommends the following paper: https://www.sciencedirect.com/science/article/pii/S1574954120300637?via%3Dihub

Q: Is there an example to be able to do template matching?
A: Yes, we have an example in the production branch: https://github.com/scikit-maad/scikit-maad/tree/production/example_gallery/1_basic

It seems currently batdetect2 is installed as a submodule (see here). How about in the requirements.txt specify pip install of a particular commit?

1. Git clone https://github.com/umr-ds/transformer4bats.
2. Try to tune parameter and see if classification works on our test data.

• Parallel processing capability (@ccharp)
• Create runtime comparison after parallel processing (@ccharp)
• Output TSV in format consumable by RavenPro (@ccharp)
• Fix "There is an error" bug occurring in audo segmenter @ccharp
• Investigate extraneous output from CLI application and remove (@ccharp )
• Try putting all code inside of src directory @ccharp
• Clear out all TODOs in the current repo (@ccharp )
• Move template_dict.pickle to be local with the code that uses it (similar to BatDetect2's model)
• Create conclusions around FP and FN, what do these detections have in common? ( @Kirsteenng and @emcediel )
• Add results, graphs and conclusions to poster (due Wednesday) ( whole team )
• Verify that Feed Buzz model works End-to-End (@ccharp and @emcediel )
• Verify that FP removal is at the right level and works (@ccharp and @emcediel )
• Create PR to add FeedBuzz and FP removal ( @emcediel )
• Make random subsample of FP and modify FP results with a better estimate of true FP (annotator error) ( @Kirsteenng and @emcediel )
• Check if Feed Buzz FN are truly FN (send Wu - Jung pictures) ( @Kirsteenng and @emcediel )
• Write documentation for updating feed buzz detection templates and scikit-maad library (@Kirsteenng )
• In-line documentations for all functions for template matching pipeline involve
• Write documentation on analysis and results (@emcediel )
• Write documentation on how to batch process multiple files using bash. (@ccharp )

Please let me know any other TODOs to include in this list!

https://osg-htc.org/