Giter VIP home page Giter VIP logo

eyecandies's Introduction

The Eyecandies Dataset for Unsupervised Multimodal Anomaly Detection and Localization

This repository contains sample code to download and use the Eyecandies dataset in your project. Please refer to the project page for more information about the publication. If you use this dataset in your research, please cite the following paper:

@inproceedings{bonfiglioli2022eyecandies,
    title={The Eyecandies Dataset for Unsupervised Multimodal Anomaly Detection and Localization},
    author={Bonfiglioli, Luca and Toschi, Marco and Silvestri, Davide and Fioraio, Nicola and De Gregorio, Daniele},
    booktitle={Proceedings of the 16th Asian Conference on Computer Vision (ACCV2022},
    note={ACCV},
    year={2022},
}

Warning

This package is just an example on how to use the Eyecandies dataset and it is not meant to reproduce the results in the paper.

Get Started

Installing the package brings in everything you need to download and use the Eyecandies dataset.

If you don't want to try the sample auto-encoder, you can simply install the package with pip, either directly from github:

$ pip install git+https://github.com/eyecan-ai/eyecandies.git

or by cloning the repository and installing it locally:

$ git clone https://github.com/eyecan-ai/eyecandies.git
$ pip install -e eyecandies

Instead, to run the NN code you need pytorch and torchvision. First, create a virtual environment and install pytorch following the instructions on the official website. For instance, to get pytorch 1.12.1 with pip and venv on Linux:

$ python3 -m venv <venv_path>
$ source <venv_path>/bin/activate
$ pip install torch==1.12.1 torchvision --extra-index-url https://download.pytorch.org/whl/cu116

Now you can get Eyecandies with the torch flavor either directly from github:

$ pip install git+https://github.com/eyecan-ai/eyecandies.git[torch]

or by cloning the repository and installing it locally:

$ git clone https://github.com/eyecan-ai/eyecandies.git
$ pip install -e eyecandies[torch]

This package is built on top of Pipelime. You can find more information about Pipelime and its features in the documentation. For example, you can list the available pipelime commands in this package with:

$ eyec list

Then, to see the help for a specific command, e.g. ec-get:

$ eyec help ec-get

Download The Dataset

First, you should download the dataset. You can either get them one by one:

$ eyec ec-get +o <output_root_folder> +c <category_name>

or download all the categories at once:

$ eyec ec-get +o <output_root_folder>

Already downloaded categories will be skipped, of course. Here a list of the available categories:

  • Candy Cane
  • Chocolate Cookie
  • Chocolate Praline
  • Confetto
  • Gummy Bear
  • Hazelnut Truffle
  • Licorice Sandwich
  • Lollipop
  • Marshmallow
  • Peppermint Candy

Train A Model

We provide a naive auto-encoder implementation to train a model on the Eyecandies dataset within the Pipelime framework.

Warning

This naive auto-encoder is not the one deployed in the paper and it is not meant to be used in practice!

Just create your own tps_context.yaml by copying dags/tps_ctx_template.yaml and filling the missing variables, namely:

  • data.root: the root folder of the Eyecandies dataset
  • data.name: the name of the category
  • result_folder: the output folder

Then, run the full pipeline with:

$ eyec run --config dags/train_predict_stats.yaml --context dags/context.yaml

To understand what's going on, the pipeline can be drawn (you need Graphviz installed, see Pipelime documentation for more info):

$ eyec draw --config dags/train_predict_stats.yaml --context dags/context.yaml
flowchart TB
    classDef operation fill:#03A9F4,stroke:#fafafa,color:#fff;
    classDef data fill:#009688,stroke:#fafafa,color:#fff;
    path/to/CATEGORY/test_public[("path/to/&lt;CATEGORY&gt;/test_public")]:::data-->|test_dataset|predict_public:::operation
    path/to/CATEGORY/test_public[("path/to/&lt;CATEGORY&gt;/test_public")]:::data-->|targets|stats:::operation
    predict_public:::operation-->|predictions|path/to/results/autoenc/CATEGORY/DATE_test_public[("path/to/results/autoenc/&lt;CATEGORY&gt;/&lt;DATE&gt;_test_public")]:::data
    path/to/results/autoenc/CATEGORY/DATE.ckpt[("path/to/results/autoenc/&lt;CATEGORY&gt;/&lt;DATE&gt;.ckpt")]:::data-->|ckpt|predict_public:::operation
    path/to/results/autoenc/CATEGORY/DATE.ckpt[("path/to/results/autoenc/&lt;CATEGORY&gt;/&lt;DATE&gt;.ckpt")]:::data-->|ckpt|predict_private:::operation
    path/to/results/autoenc/CATEGORY/DATE_test_public[("path/to/results/autoenc/&lt;CATEGORY&gt;/&lt;DATE&gt;_test_public")]:::data-->|predictions|stats:::operation
    path/to/CATEGORY/test_private[("path/to/&lt;CATEGORY&gt;/test_private")]:::data-->|test_dataset|predict_private:::operation
    predict_private:::operation-->|predictions|path/to/results/autoenc/CATEGORY/DATE_test_private[("path/to/results/autoenc/&lt;CATEGORY&gt;/&lt;DATE&gt;_test_private")]:::data
    path/to/CATEGORY/train[("path/to/&lt;CATEGORY&gt;/train")]:::data-->|train_dataset|train:::operation
    train:::operation-->|last_ckpt|path/to/results/autoenc/CATEGORY/DATE.ckpt[("path/to/results/autoenc/&lt;CATEGORY&gt;/&lt;DATE&gt;.ckpt")]:::data
    stats:::operation-->|output_folder|path/to/results/autoenc/CATEGORY[("path/to/results/autoenc/&lt;CATEGORY&gt")]:::data
Loading

As you can see, the pipeline is composed of three main steps:

  1. autoenc-train: the autoencoder is trained and a checkpoint is produced
  2. autoenc-predict: the trained model predicts anomaly heatmaps on the public and private test set
  3. ec-metrics: ROC and AUROC are computed on the predictions using the public ground truth

To get the results on any other method, just replace the first two nodes, then run ec-metrics on the new predictions. Note that results are given for the public test set only: to get the results on the private test set as well, please follow the instructions on the Eyecandies website.

Metric Depths And Pointclouds

If you want to work with depth maps, you may want to convert them in meters. To do so, we provide a stage that can be run as:

$ eyec -m eyecandies.stages map +s depth2mt +i path/to/eyecandies/dataset +o path/to/output/dataset

The output dataset has, for each sample, a depth.npy item, which is a float32 depth map in meters. Note that all other files are hard-linked, so no extra space is used.

If you use the depth2mt stage in a python script, you should use its class name:

from pipelime.sequences import SamplesSequence
from eyecandies.stages import DepthToMetersStage

seq = SamplesSequence.from_underfolder("path/to/eyecandies/dataset")
seq = seq.map(DepthToMetersStage())

To project depth points to a pointcloud, we provide the stage depth2pc too. To run both stages in a row, you can use the compose builtin stage:

$ eyec map +i path/to/eyecandies/dataset +o path/to/output/dataset +stage.compose.stages[0] depth2mt +stage.compose.stages[1] depth2pc +g 4,10

where +g 4,10 means that 4 processes will be spawned with a prefetch of 10 samples each. The default behavior is to save a PLY file with points, normals and colors.

Train And Testing With Anomalib

Anomalib is a popular framework to benchmark anomaly detection algorithms. To train and test on Eyecandies, you should first rearrange your data. We provide a ready-to-use Pipelime dag to extract and split the files, you just have to create your own to_anomalib_context.yaml by copying dags/to_anomalib_ctx_template.yaml and filling the missing variables, namely:

  • categories: the list of categories to convert
  • eyecandies_root: the path to the Eyecandies dataset root folder
  • output_root: the path to the output dataset root folder
  • image_key: the key of the image item to use for anomalib, e.g., image_5

Then run:

$ pipelime run --config dags/to_anomalib.yaml --context dags/to_anomalib_context.yaml

Pipelime can also create for you the dataset section of the configuration file required by Anomalib. Just run:

$ pipelime audit --config dags/anomalib_conf_template.yaml --context dags/to_anomalib_context.yaml -o dags/anomalib_conf.yaml

The output file dags/anomalib_conf.yaml contains a dataset section for each category you can copy-and-paste to your Anomalib configuration file.

To further customize the output, you can also set some other variables in the context:

anomalib:
  image_size: 256
  train_batch_size: 64
  test_batch_size: 64
  num_workers: 4
  train_transform_config: null

or on the command line:

$ [...] @anomalib.image_size 256 @anomalib.train_batch_size 64 @anomalib.test_batch_size 64 @anomalib.num_workers 4 @anomalib.train_transform_config null

eyecandies's People

Contributors

astrodave85 avatar lucabonfiglioli avatar m4nh avatar nfioraio-ec avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

eyecandies's Issues

Dataset download

Hello, I am very interested in your research, but I can't download your dataset. Do you have any other ways to share your datasets?
1671202721626

Question about AUROC Calculation in Eyecandies Dataset

Hi,
I read your paper on the Eyecandies dataset and have a question about the AUROC calculation. In your experiments, did you calculate AUROC for each image individually under different lighting conditions, or were these metrics calculated based on the combined set of 6 images for each object?

Thanks for your help!
Snipaste_2024-02-01_18-42-16

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.