olgaliak / active-learning-detect Goto Github PK

After getting the workflow running, I have been impressed with the ease of training, reviewing, and retraining a model. I will add ideas here for improved flow.

on the tagging computer currently, we need to delete the csvs in the tag folder and the images and json in the tagging_location before running download_vott_json.py and beginning a new round of tagging. It would be nice for the files to be automatically deleted or maybe moved when running download_vott_json.py or at least an error that those files still exist and should be deleted prior to running the script.

Hello,

I am currently doing preliminary research into a project similar to this and had some questions regarding the cloud aspect.

I saw that your project used Azure in particular as your cloud service of choice, and I was wondering if there were particular benefits to using Azure over AWS/GCP in this specific case. Also, as this project was done 3 years, have the services/options changed significantly since then?

Thank You and Best Regards,

Matthew

I could see it being really useful to instead of making a new csv with performance each time you train a model to actually append the new run to the same csv.

I would love a table that had a summary of the training run. Maybe a csv with

1. date/time of run,
2. number of training images,
3. number of each class,
4. performance on the cross val or test data (recall and precision and maybe to numbers of FP, TP, FN, TN)
5. mAP for each run

I am trying the workflow with user_folders=True for the first time and it looks like after I initialized project it only is pulling images from one of the folders when I run download_vott_json.py. Ping me when you are ready to address this and I can provide info that you need to recreate.

Sometimes the newer version of the model trained with more data actually seems worse than an older version (maybe more diversity in the training data within each class?). It would be nice to be able to quickly change what model predictions you are using to download new photos

Prediction is rather slow, greatly slowing down the workflow when there are many images. for instance, I have 6,446 images in my container and it takes >45 minutes to predict (that is an estimate).
It would be great if we could speed this up.

Ideas:

• only predict on unlabeled images/test set (#17)
• only predict on the above plus non-NULL labeled images i.e. images that have something in them
• predict on a random (10xdownload_number) unlabeled images and sample the download_number from them?

Abram and I were discussing a few edge cases that we are running into, specifically when we can identify a blurry or cut-off image down to a general group but not down to a species (for eg, can tell it's a mammal but can't tell which, or when we have a bad angle on a shearwater/petrel and cannot determine which), and we were wondering about the possibility of having hierarchical classes. In our biological use-case, this makes a lot of sense - for example, if we label something as a hawaiian petrel, we can assume it's an animal and part of the Procellariiformes order, part of the Pterodroma genus, etc etc. Then, if we couldn't ID a bird down to species, we could simply label it with the most specific label we have for it, whether it's Pterodroma, Procellariiformes, or just Aves.

I'm not sure how much re-invention would be required to make that work. In a perfect world, the model would be able to use all of the levels of the tag hierarchy, meaning that all of the specific tags would also be incorporated in each more general class, instead of each level being considered its own unique class.

Just curious if this is possible!

Sometimes there are images that have an example of a class in them that is not a good image to train on (the object is not typical, or partially obscured, or blurry). It would be nice to have a mechanism to easily exclude images with a specific tag or text string (bad, exclude, *_blurry, *_bad).

In VoTT the only option is to tag something, or no, so in terms of creating a labeled dataset, it is important to label everything in the dataset, so it would be nice to have the exclusion mechanism in the active learning side.

This is definitely a "nice thing to potentially have" instead of an issue but something that might make more "pythonic" sense is to have config be a class defined like this:

class Config():
    def __init__(self, config_file):
        self.config = {}
        with open(config_file) as file_:
            for line in file_:
                line = line.strip()
                if line and line[0] is not "#":
                    var,value = line.split('=', 1)
                    self.config[var.strip()] = value.strip()

    def __getitem__(self, key):
        # Potential error-checking code if key not in self.config
        return self.config[key]

Then you can just do config = Config(config_file) and call config["attr"] to get the value of attr in config. This sounds like a pretty useless change but Mike and I had spent a bit of time thinking about how to do a config file in python so I feel like it would be good to have. Let me know if you have any suggestions!

Sometimes a model preforms very poorly at first. In this case sometimes it is helpful to set the confidence threshold at some low number so that you get some predictions and can visualize what is going on. This number seems a little hard to pick in an informed way. It would be nice if predictions were made and kept above some very very low number and then the user could choose the threshold above which (or and lower and upper bound) to target for tagging. This way when you run download_vott_json.py it would have an extra argument (or in the config.ini) that would be the confidence threshold above which to sample.

When there are no photos left that are untagged the function runs with no message. It would be nice if there was a warning that there are no unlabeled files left.

If you have a file_type defined in the config.ini such "*.jpg" and have images that are *.JPG then during prediction creation you will have zero files and the TF Detector code will fail the script. Root cause seems to be the way code is handled here for instance. Why wait all the way up until Create Predictions for this to surface? Do we need to be case sensitive here?

Just logging this as a bug. I will probably fix

1. I think I ran out of unlabeled photos that I uploaded to blob storage but would like to continue to tag and train. how do I add new photos to the workflow?

2. How to add classes after a project has been initialized? I want to add new tags that were not included in the beginning when I initialized the project, what is the process to make sure the tags are included correctly??

3. Changing tag class once it has bee labeled Some of the tags that were uploaded may need to be changed due to tagger inconsistency, can I change them manually in the tagged csv or do I need to change them somewhere else too?

It would be nice to have a method to pull some (random or matching a search string) or all of the images that have been labeled so that you can fix any inconsistencies or split/group classes.

We have multiple labels in our dataset and it appears as if there is a bug that is assigning all the tags to a single class. We get predictions that accurately put bounding boxes around all of the classes we have trained on but they are labeled as the 'petrel' class We only get predictions for a two classes (petrel and rat) if you look in the tagged_preds.csv but there are only 3 rows with this classes predicted compared to 3,047 petrel predictions. Most (~50%??) of the petrel predictions are other classes.

There is a class imbalance in the training data:
petrel:

Let me know what info is needed to troubleshoot/fix this issue?

Abram and I are having an issue getting the VoTT workflow set up on a second windows machine. Downloading the images and tagging works just fine, the json is properly updated after tagging, but tagging.csv is never updated with the script, and tagged.csv is not created. We made sure all of the modules we needed are up to date, and our parameters are set properly in our config_cats.ini file, and we aren't sure what else to try.

It would be helpful to understand which files are used for which purpose during the active learning workflow. There are many different files with similar names (tagged, untagged, totag, tagging all with various suffixes and stored/copied/duplicated among blob storage, the training machine, and the tagging machine. It would be nice to know which is used for what purpose where they are created, etc.

Consider using sys.argv[0] instead of Path.cwd() so that regardless of where the file is run from it will find the config correctly. Also more generally find a better way to import config.

When there are a lot of images in the blob storage training is fast but prediction is slow. It would be a nice option to only predict on unlabeled images to improve speed and be able to iterate faster and an option to only do metrics on the test set / not predict on the rest of the labeled images.

Another thought that I have been pondering with the active-learning pipeline is how to avoid biasing your detector to the types of images that you labeled first? For instance, say I have 10 cameras and each camera has taken a different number of images from 1000 to 100,000 over a 1-year deployment. If you label 100 randomly selected images to start with, the majority will be from the camera that took the most images, and maybe the background in that camera is distinct. if you train a model with those initial 100 images, it may be highly biased toward detecting things in images from that camera (because of some characteristic of those images). Images from other cameras might not even have detections and might not get "served" to the person tagging?
Essentially I see it as the same idea as the class imbalance, but instead, it is an imbalance in the raw data. How/does this normally get addressed in active learning?

In the README.md it says you only need python 3.5. Update readme! #28

It would be nice to be able to easily specify even class balance (all classes same number), or native (all classes in relative proportion to what is found in the dataset). There might be some other pre-canned options that would be nice that I have not though of yet.

Sometimes when tagging new labels that were not thought of when the project was initialized are needed. Essentially, it would be nice to be able to add classes to the config.ini and have the pascal_label_map update automatically with the new labels.

@yashpande provided this code
The code to make a new label map file:

with open(config_file["label_map_path"], "w") as map_file:
       for index, name in enumerate(config_file["classes"].split(","), 1):
           map_file.write("item {{\n  id: {}\n  name: '{}'\n}}".format(index, name))

The easiest way is probably just to manually fill in the variables config_file["label_map_path"] and config_file["classes"] then run it as an independent script.
You might want to add an issue about just having that code run each time training kicks off.

The iteration loop here grabs all files and attempt to extract height and width without checking to see if the file is actually an image.

Yash and I worked through a bug where I was getting errors when there were photos that were reviewed but did not have a label. These images received a bounding box of 0,0,0,0 and a NULL label. In the totag.csv files, there are a bunch of rows that have NULL class. and the confidence is 0.

This makes me think that when we use download_vott_json we will get these files if we use pick_max = False which is not all that useful when generating new tags for training.

It would be really great to be able to change the confidence threshold on the fly to be able to display bounding boxes in VoTT that have only high or low confidence.

It would be nice to have a way to assure that the correct dependencies were installed to run the active learning routine

Eg pip install -r requirements.txt

@olgaliak I am not sure this is a result from this PR but I pulled the latest and I am working on master. I updated my config and I am trying to download images for tagging and asked for 5 images and it started downloading 1000s! and each photo is listed multiple times. Seems fishy... here is a screenshot of output.

and the config is attached.
config_bbalous.zip

Let me know if I am misspecifying something or if you need more info to look into this

Originally posted by @abfleishman in #41 (comment)

With the kittiwake model, I have now run many iterations and it is performing very well for kittiwakes. I have a second class in the model (egg) which is much rarer ( i think I have only found ~5 examples) it would be really great if I could specifically download photos that have predictions for a certain class (egg) so that I could find more of that class (i have not seen a single prediction for egg in the random photos that I have tagged).