toshas / torch-fidelity Goto Github PK

High-fidelity performance metrics for generative models in PyTorch

License: Other

Python 98.07% Shell 0.90% Dockerfile 1.03%

pytorch metrics gan generative-model evaluation inception-score frechet-inception-distance kernel-inception-distance perceptual-path-length precision

torch-fidelity's People

Contributors

Stargazers

Watchers

torch-fidelity's Issues

Unable to use ‘mps’ device for m1 macs

Image format specification for torch.utils.data.Dataset

I read everywhere trying to figure out how my datasets should output an image (i.e. which transforms to use). There is nowhere in the docs. As far I found in the code, it seems to be a torch uint8 tensor of the shape (C, H, W):

register_dataset(
    'cifar10-train',
    lambda root, download: Cifar10_RGB(root, train=True, transform=TransformPILtoRGBTensor(), download=download)
)

class TransformPILtoRGBTensor:
    def __call__(self, img):
        vassert(type(img) is Image.Image, 'Input is not a PIL.Image')
        width, height = img.size
        img = torch.ByteTensor(torch.ByteStorage.from_buffer(img.tobytes())).view(height, width, 3)
        img = img.permute(2, 0, 1)
        return img

Is that it? The metrics expects torch uint8 tensors? What about sizes? What size of images the dataset should output? The same size as the generator?

Thanks

Does it support to calculate inception score for coco dataset?

Help on: "RuntimeError: stack expects each tensor to be equal size"

Hi,

Thanks for your great work!
When trying to run a very simple example (code pasted below):

# -*- coding: utf-8 -*-
"""
Created on Fri Feb  4 13:18:44 2022

@author: User
"""

import torch_fidelity

path1 = r'/home/rsmalbraak/data/TRAIN_3classes_nofolder'
path2 = r'/home/rsmalbraak/data/Generated_3dlayer_decay_nofolder'

metrics_dict = torch_fidelity.calculate_metrics(
    input1=path1, 
    input2=path2, 
    cuda=True, 
    isc=True, 
    fid=True, 
    kid=False, 
    verbose=False,
    ppl_sample_similarity_resize=64,
)
print(metrics_dict)

I get the following error message:

Traceback (most recent call last):
  File "/home/rsmalbraak/FID_IS.py", line 40, in <module>
    metrics_dict = torch_fidelity.calculate_metrics(
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch_fidelity/metrics.py", line 239, in calculate_metrics
    featuresdict_1 = extract_featuresdict_from_input_id_cached(1, feat_extractor, **kwargs)
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch_fidelity/utils.py", line 372, in extract_featuresdict_from_input_id_cached
    featuresdict = fn_recompute()
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch_fidelity/utils.py", line 360, in fn_recompute
    return extract_featuresdict_from_input_id(input_id, feat_extractor, **kwargs)
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch_fidelity/utils.py", line 345, in extract_featuresdict_from_input_id
    featuresdict = get_featuresdict_from_dataset(input, feat_extractor, batch_size, cuda, save_cpu_ram, verbose)
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch_fidelity/utils.py", line 111, in get_featuresdict_from_dataset
    for bid, batch in enumerate(dataloader):
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 521, in __next__
    data = self._next_data()
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1203, in _next_data
    return self._process_data(data)
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data
    data.reraise()
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch/_utils.py", line 434, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 1.
Original Traceback (most recent call last):
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
    return self.collate_fn(data)
  File "/home/rsmalbraak/.local/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 56, in default_collate
    return torch.stack(batch, 0, out=out)
RuntimeError: stack expects each tensor to be equal size, but got [3, 1024, 1024] at entry 0 and [3, 256, 256] at entry 38

As I have used "ppl_sample_similarity_resize", I do not understand why this error occurs.
Do you know how I can fix this?

Looking forward to your reply and thank you very much in advance!

Check features length before calling `choice` in KID

Traceback (most recent call last):
  File "/cluster/home/alugmayr/.local/bin/fidelity", line 8, in <module>
    sys.exit(main())
  File "/cluster/home/alugmayr/.local/lib/python3.8/site-packages/torch_fidelity/fidelity.py", line 98, in main
    metrics = calculate_metrics(args.input1, input_2=args.input2, **vars(args))
  File "/cluster/home/alugmayr/.local/lib/python3.8/site-packages/torch_fidelity/metrics.py", line 145, in calculate_metrics
    metric_kid = kid_featuresdict_to_metric(featuresdict_1, featuresdict_2, feature_layer_kid, **kwargs)
  File "/cluster/home/alugmayr/.local/lib/python3.8/site-packages/torch_fidelity/metric_kid.py", line 113, in kid_featuresdict_to_metric
    metric = kid_features_to_metric(features_1, features_2, **kwargs)
  File "/cluster/home/alugmayr/.local/lib/python3.8/site-packages/torch_fidelity/metric_kid.py", line 91, in kid_features_to_metric
    f1 = features_1[rng.choice(len(features_1), kid_subset_size, replace=False)]
  File "mtrand.pyx", line 954, in numpy.random.mtrand.RandomState.choice
ValueError: Cannot take a larger sample than population when 'replace=False'

ValueError: KID subset size 1000 cannot be smaller than the number of samples (input_1: 876, input_2: 876). Consider using "kid_subset_size" kwarg or "--kid-subset-size" command line key to proceed.

Hi I am so happy to get your sharing valuable source code to calculate the metrics I do appreciate it and I am currently facing this bug when I train with my custom generated images. May I know how I should solve it? Please help me where should I need to change. Thank you so much.

Score calculation for paired dataset

Hi,

I also calculate the FID , KID and LPIPS scores between pair images (predictions and GTs), and I am comparing two networks, but the results show the model that has a lower LPIPS value but higher FID and KID scores at the same time.

So I wanna know is this what you say hard to interpret? What may be the reason for this phenomenon? Could it be a lack of diversity since the higher similarity?

I will very appreciate if you could help me.

Add support for computing sFID metric inside feature_extractor_inceptionv3.py

As stated in the paper https://arxiv.org/pdf/2103.03841v1.pdf the computation of sFID metric requires the extraction of the first 7 channels from mixed6/conv feature map. Currently the repo does not provide it. What about adding this feature? It could be then exploited inside torchmetrics package.

FID fails when `feature_layer_fid!=2048`

Script to reproduce

import torch
from torch.utils.data import Dataset
from torch_fidelity import calculate_metrics

img1 = torch.randint(0, 255, (5, 3, 299, 299), dtype=torch.uint8)
img2 = torch.randint(0, 255, (5, 3, 299, 299), dtype=torch.uint8)

class _ImgDataset(Dataset):
    def __init__(self, imgs):
        self.imgs = imgs
        
    def __getitem__(self, idx):
        return self.imgs[idx]

    def __len__(self):
        return self.imgs.shape[0]

def main(feature_size):
    torch_fid = calculate_metrics(
        _ImgDataset(img1),
        _ImgDataset(img2),
        fid=True, 
        feature_layer_fid=feature_size
    )
    print(torch_fid)

if __name__ == "__main__":
    main("64") # fails
    main("192") # fails
    main("768") # fails
    main("2048") # succeeds

will fail with the following assert:

it seems that the shape of the calculated features when feature_layer_fid!=2048 are not 2D but instead:
[N, feature_size, 1, 1]
it should be simple to fix by introducing a feature.squeeze() somewhere appropriately.

Running the code

Hi,
Thank you for your work
While running fidelity --gpu 0 --isc --input1 ~/images/
I got following error:

ValueError: Input descriptor "input" field can be either an instance of Dataset, GenerativeModelBase class, or a string, such as a path to a name of a registered dataset (cifar10-train, cifar10-val, stl10-train, stl10-test, stl10-unlabeled), a directory with file samples, or a path to an ONNX or PTH (JIT) module

Could you help me to solve this issue, please?
Thank you

Suggestions for Colab notebook use?

I do not have CUDA installed since I am running an Apple imac. However, I use Google Colab for most machine learning experiments. I started a new notebook with pip install torch-fidelity, which worked fine and received:

.....
Installing collected packages: torch-fidelity
Successfully installed torch-fidelity-0.3.0

However, I don't know where the bin "fidelity" is installed in Colab space. I looked in user/local/bin and it was not there. Same issue with /usr/bin. In the notebook, "whereis fidelity" or "type -a fidelity" are not yielding anything. I also tried accessing fidelity directly from python code in the readme.md

Any suggestions to move forward on this?

Add a new tf feature extractor

Hello,

I have tf inception model finetuned specific for my dataset. I would like to use this model to compute IS and FID using torch fidelity. How do I convert this inception model to torch and preserve behavior as you did?

Can you share how you did it?

Thanks,

Douglas

Error when loading COCO-val2017

Hi,

I downloaded the COCO-val2017 dataset from this link and unzipped the data to get 5000 jpg images. I then use the path to the extracted images to compare it with other images I generated but get an error:

import torch_fidelity
metrics_dict = torch_fidelity.calculate_metrics(
        input1="path_to_my_images", # fill this with your generated images path
        input2="path_to_coco",
        cuda=True, 
        isc=True, 
        fid=True, 
        kid=False,
        cache_root='./', # fill this with your own path
        cache=True,
        verbose=True,
        samples_find_deep=True,
        samples_find_ext="jpg,jpeg,png"
    )

print(metrics_dict)

Here is the output I get:

Creating feature extractor "inception-v3-compat" with features ['logits_unbiased', '2048']
Extracting features from input1
Looking for samples recursively in "outputs/baseline" with extensions jpg,jpeg,png
Found 10000 samples
Processing samples                                                                                                                                                                                                                            
Extracting features from input2
Looking for samples recursively in "/home/ubuntu/val2017" with extensions jpg,jpeg,png
Found 5000 samples, some are lossy-compressed - this may affect metrics
Traceback (most recent call last):                                                                                                                                                                                                            
  File "scripts/calculate_fid.py", line 43, in <module>
    main()
  File "scripts/calculate_fid.py", line 26, in main
    metrics_dict = torch_fidelity.calculate_metrics(
  File "/home/ubuntu/q-diffusion/src/torch-fidelity/torch_fidelity/metrics.py", line 322, in calculate_metrics
    return calculate_metrics_one_feature_extractor(**kwargs)
  File "/home/ubuntu/q-diffusion/src/torch-fidelity/torch_fidelity/metrics.py", line 71, in calculate_metrics_one_feature_extractor
    featuresdict_2 = extract_featuresdict_from_input_id_cached(2, feat_extractor, **kwargs)
  File "/home/ubuntu/q-diffusion/src/torch-fidelity/torch_fidelity/utils.py", line 407, in extract_featuresdict_from_input_id_cached
    featuresdict = fn_recompute()
  File "/home/ubuntu/q-diffusion/src/torch-fidelity/torch_fidelity/utils.py", line 395, in fn_recompute
    return extract_featuresdict_from_input_id(input_id, feat_extractor, **kwargs)
  File "/home/ubuntu/q-diffusion/src/torch-fidelity/torch_fidelity/utils.py", line 380, in extract_featuresdict_from_input_id
    featuresdict = get_featuresdict_from_dataset(input, feat_extractor, batch_size, cuda, save_cpu_ram, verbose)
  File "/home/ubuntu/q-diffusion/src/torch-fidelity/torch_fidelity/utils.py", line 127, in get_featuresdict_from_dataset
    for bid, batch in enumerate(dataloader):
  File "/opt/conda/envs/qdiff/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 630, in __next__
    data = self._next_data()
  File "/opt/conda/envs/qdiff/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1345, in _next_data
    return self._process_data(data)
  File "/opt/conda/envs/qdiff/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1371, in _process_data
    data.reraise()
  File "/opt/conda/envs/qdiff/lib/python3.8/site-packages/torch/_utils.py", line 694, in reraise
    raise exception
RuntimeError: Caught RuntimeError in DataLoader worker process 0.
Original Traceback (most recent call last):
  File "/opt/conda/envs/qdiff/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 308, in _worker_loop
    data = fetcher.fetch(index)
  File "/opt/conda/envs/qdiff/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 54, in fetch
    return self.collate_fn(data)
  File "/opt/conda/envs/qdiff/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 265, in default_collate
    return collate(batch, collate_fn_map=default_collate_fn_map)
  File "/opt/conda/envs/qdiff/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 119, in collate
    return collate_fn_map[elem_type](batch, collate_fn_map=collate_fn_map)
  File "/opt/conda/envs/qdiff/lib/python3.8/site-packages/torch/utils/data/_utils/collate.py", line 161, in collate_tensor_fn
    out = elem.new(storage).resize_(len(batch), *list(elem.size()))
RuntimeError: Trying to resize storage that is not resizable

I installed the latest version following this link. I am using torch version 2.1.0+cu118.

Can you help with this?

Thanks

can torch-fidelity be installed into conda env?

CelebA Dataset

Thank you for sharing the implementation of the FID and IS scores.
I wanted to use the code to find the metrics for the samples generated by a GAN model trained on the CelebA dataset as well as the dataset with transformations (custom version of CelebA dataset), thereby I thought of using input1 and input2. However when I use the command

fidelity --gpu 0 --fid --input1 /images/sample_dir/ --input2 /images/CelebA_DATASET

I am getting the following error (which suggests to use given datasets only)

ValueError: Input descriptor "input" field can be either an instance of Dataset, GenerativeModelBase class, or a string, such as a path to a name of a registered dataset (cifar10-train, cifar10-val, stl10-train, stl10-test, stl10-unlabeled), a directory with file samples, or a path to an ONNX or PTH (JIT) module

I request you to guide me in this regard.

Kind regards

Prabhav

Feature extractor torch.compile

when there will be next release?

Hello, we have implemented a metric based on the latest version of this package but PyPPI has the last 0.3.0 and PyPI does not allow any dependency outside their registry so when we refer master no user could use it... Would it be possible to make a release any time soon? Thank you

cc: @SkafteNicki

Question about PRC merge to master

Hello.

Thanks for the awesome repo. When is the PRC branch going to be merged to main? Is it going to happen anytime soon?

About the value of FID

Hello, I would like to ask why I get a negative value when calculating FID with pictures in the same folder. Is this normal?

Read the docs documentation

AssertionError in FID calculation.

Hi,

when trying to calculate the FID score for images between two directories as inputs, the following error is given:
AssertionError: Imaginary component 2.0889527537971523e+111

I tried fixes from other implementations of FID, but increasing the dataset size to be over 2048 and different scipy versions are not a fix for me.

Is there any clue what else might be causing this?

A better way to compute the FID

Hello, I think the following implementation of the Fréchet distance is faster than the current one and would allow to drop the scipy dependency.

def frechet_distance(mu_x: Tensor, sigma_x: Tensor, mu_y: Tensor, sigma_y: Tensor) -> Tensor:
    a = (mu_x - mu_y).square().sum(dim=-1)
    b = sigma_x.trace() + sigma_y.trace()
    c = torch.linalg.eigvals(sigma_x @ sigma_y).sqrt().real.sum(dim=-1)

    return a + b - 2 * c

The implementation is based on two facts:

The trace of $A$ equals the sum of its eigenvalues.
The eigenvalues of $\sqrt{A}$ are the square-roots of the eigenvalues of $A$.

0.3.0 miss requirements.txt

$ wget https://files.pythonhosted.org/packages/source/t/torch-fidelity/torch_fidelity-0.3.0.tar.gz
...
$ tar vtaf torch_fidelity-0.3.0.tar.gz|grep requirements.txt
$ tar vtaf torch_fidelity-0.3.0.tar.gz
drwxr-xr-x anton/staff       0 2021-06-15 22:53 torch_fidelity-0.3.0/
-rw-r--r-- anton/staff     554 2021-06-12 00:54 torch_fidelity-0.3.0/LICENSE.md
-rw-r--r-- anton/staff    1828 2021-06-15 22:53 torch_fidelity-0.3.0/PKG-INFO
-rw------- anton/staff    6501 2021-06-15 22:46 torch_fidelity-0.3.0/README.md
-rw-r--r-- anton/staff      38 2021-06-15 22:53 torch_fidelity-0.3.0/setup.cfg
-rw-r--r-- anton/staff    2440 2021-06-12 00:54 torch_fidelity-0.3.0/setup.py
drwxr-xr-x anton/staff       0 2021-06-15 22:53 torch_fidelity-0.3.0/torch_fidelity/
-rw-r--r-- anton/staff    1094 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/__init__.py
-rw-r--r-- anton/staff    2007 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/datasets.py
-rw-r--r-- anton/staff    1460 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/defaults.py
-rw-r--r-- anton/staff    2243 2021-06-14 18:32 torch_fidelity-0.3.0/torch_fidelity/feature_extractor_base.py
-rw-r--r-- anton/staff   16856 2021-06-14 20:17 torch_fidelity-0.3.0/torch_fidelity/feature_extractor_inceptionv3.py
-rw-r--r-- anton/staff   12071 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/fidelity.py
-rw-r--r-- anton/staff     892 2021-06-14 18:17 torch_fidelity-0.3.0/torch_fidelity/generative_model_base.py
-rw-r--r-- anton/staff    2663 2021-06-14 18:33 torch_fidelity-0.3.0/torch_fidelity/generative_model_modulewrapper.py
-rw-r--r-- anton/staff    4005 2021-06-14 20:33 torch_fidelity-0.3.0/torch_fidelity/generative_model_onnx.py
-rw------- anton/staff     523 2020-12-29 19:23 torch_fidelity-0.3.0/torch_fidelity/helpers.py
-rw-r--r-- anton/staff    6370 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/interpolate_compat_tensorflow.py
-rw-r--r-- anton/staff    5038 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/metric_fid.py
-rw-r--r-- anton/staff    2206 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/metric_isc.py
-rw-r--r-- anton/staff    5004 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/metric_kid.py
-rw-r--r-- anton/staff    6033 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/metric_ppl.py
-rw------- anton/staff   13752 2021-06-14 20:39 torch_fidelity-0.3.0/torch_fidelity/metrics.py
-rw-r--r-- anton/staff    1403 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/noise.py
-rw------- anton/staff    6797 2021-06-14 20:18 torch_fidelity-0.3.0/torch_fidelity/registry.py
-rw-r--r-- anton/staff     798 2021-06-14 20:17 torch_fidelity-0.3.0/torch_fidelity/sample_similarity_base.py
-rw-r--r-- anton/staff    7415 2021-06-14 20:17 torch_fidelity-0.3.0/torch_fidelity/sample_similarity_lpips.py
-rw-r--r-- anton/staff   15367 2021-06-12 00:54 torch_fidelity-0.3.0/torch_fidelity/utils.py
-rw-r--r-- anton/staff      22 2021-06-15 22:23 torch_fidelity-0.3.0/torch_fidelity/version.py
drwxr-xr-x anton/staff       0 2021-06-15 22:53 torch_fidelity-0.3.0/torch_fidelity.egg-info/
-rw-r--r-- anton/staff    1828 2021-06-15 22:53 torch_fidelity-0.3.0/torch_fidelity.egg-info/PKG-INFO
-rw-r--r-- anton/staff     985 2021-06-15 22:53 torch_fidelity-0.3.0/torch_fidelity.egg-info/SOURCES.txt
-rw-r--r-- anton/staff       1 2021-06-15 22:53 torch_fidelity-0.3.0/torch_fidelity.egg-info/dependency_links.txt
-rw-r--r-- anton/staff      59 2021-06-15 22:53 torch_fidelity-0.3.0/torch_fidelity.egg-info/entry_points.txt
-rw-r--r-- anton/staff      42 2021-06-15 22:53 torch_fidelity-0.3.0/torch_fidelity.egg-info/requires.txt
-rw-r--r-- anton/staff      15 2021-06-15 22:53 torch_fidelity-0.3.0/torch_fidelity.egg-info/top_level.txt

No requirements.txt. However, https://github.com/toshas/torch-fidelity/blob/master/setup.py#L10

with open('requirements.txt') as f:

It will make this program broken when it is installing.

Gated CRF Loss for Weakly Supervised Semantic Image Segmentation

hi,
The Gated CRF loss is a interesting work,I except your work code,i am working in weak-segmentation,and see your paper,i hope communicate with you about this,my email:[email protected], Excepting your reply

Passing in two TensorDataset instances results in wrongly failed assertion

Hi! Thanks for the package. While it looks exactly like what we need, we couldn't get it to work.

In our training loop, we call torch_fidelity.calculate_metrics as follows with two TensorDatasets made on the fly. We have verified their contents to be be torch.uint8 images, however we get the assertion error that the input is not torch.uint8. Investigating a little more, the cause seems to be that the input is a list instead.

# in training loop
tensors1 = ... # image from network, already on GPU
tensors2 = ... # image from network, already on GPU
imgs1 = tensor_to_image(tensors1)  # changes BCWH to BWHC and converts to uint8
imgs2 = tensor_to_image(tensors2)  
dataset1 = TensorDataset(imgs1)
dataset2 = TensorDataset(imgs2)
frechet  = torch_fidelity.calculate_metrics(dataset1, dataset2, cuda=False, fid=True, save_cpu_ram=True)

Which results in:

File "/opt/conda/lib/python3.7/site-packages/torch_fidelity/feature_extractor_inceptionv3.py", line 81, in forward
    vassert(torch.is_tensor(x) and x.dtype == torch.uint8, 'Expecting image as torch.Tensor with dtype=torch.uint8')
File "/opt/conda/lib/python3.7/site-packages/torch_fidelity/helpers.py", line 9, in vassert
    raise ValueError(message)
ValueError: Expecting image as torch.Tensor with dtype=torch.uint8

Investigating, I found that it isn't x.dtype == torch.uint8 which fails, but torch.is_tensor(x). When dataset1 and dataset2 are created their inputs are all tensors, but somewhere within calculate_metrics these incorrectly get converted to lists.

What's going wrong here and how can we fix this?

Outputs of inception network seems to be batch size dependent

As mentionned in this issue Lightning-AI/torchmetrics#1620, the inception network seems to suffer from different results depending from the batch_size

Something i tried that seems to solve this issue is to run the network in float64

This batch independence seems to have considerable differences in FIDs as mentionned in Lightning-AI/torchmetrics#1620 the batch bias in FID seems to be higher from small batch-sizes. If we compute FID between 2 uniformly sampled distributions with a 1000 points each, if we compute it with a batch size of 1000, we get FID 1.9 but if we compute it with batch-size=2, then the FID is 10. Since we sample from the same distribution, FID should be as close too zero as possible.

Possible Fix

inception = inception.to(torch.float64)
imgs = imgs.to(torch.float64)
inception(imgs)

Split tests into full and smoke, use smoke with circleci

Geometry Score

https://github.com/KhrulkovV/geometry-score

Improved precision-recall

Implement https://arxiv.org/pdf/1904.06991.pdf

Precision / recall reproducibility

Was the reproducibility of these new metrics checked with the official repo's results? I tested with my mode-collapsed generative model that produces nearly the same images as below:

and got these results:

inception_score_mean: 1.12824
inception_score_std: 0.0006825597
kernel_inception_distance_mean: 0.239309
kernel_inception_distance_std: 0.002847237
precision: 4e-05
recall: 0.99084
f_score: 7.999677e-05

We can see that precision is nearly zero and recall is close to 1. Recall is supposed to measure the diversity of generated samples; it should be close to zero in this case. Also, it seems that the car lies on the true data manifold, meaning that precision should be close to one. Results seem to be flipped.

I used 50000 generated samples. This is the command I used:

fidelity --prc --isc --kid --input1 ${dir}/${iteration}-50k/samples --input2 cifar10-train --gpu 0 | tee ${dir}/${iteration}-50k/fidelity.txt

Some question about 'calculate_metrics'

Hi,

I have some problems when using 'calculate_metrics' to calculate FID and KID, and I hope to get your help.
Q: I only have 200 groups of images (each containing gt.png and out.png), how do I call calculate_metrics?
How to change the input in calculate_metrics(input1, input2,kid=True)?

Thank you.

toshas / torch-fidelity Goto Github PK

torch-fidelity's People

Contributors

Stargazers

Watchers

Forkers

torch-fidelity's Issues

Recommend Projects

Recommend Topics

Recommend Org