Could you provide your MAML implementation or the repo you used to get the reported MAML results ? Thanks

Hi,
Submitting the paper on arxiv takes a while to get the arxiv link. And our current state on arxiv is on hold. I'm afraid we won't be able to get the link by the deadline.
So if we don't get the arxiv link by the deadline, can we just upload the PDF of our paper on CMT3?

I'm running this command that is provided in README to train a proto net model:

    python ./train.py --dataset miniImageNet --model ResNet10  --method protonet --n_shot 5 --train_aug

I'm getting these scores in test phase:

which are less than scores reported in paper:

What setting should I use to get the scores close to the ones that reported in paper?

With such large domain differences (miniImageNet-->Chest-X), does domain generation make sense? Will there be such a big domain difference in the real world? In this case there seems to be no valuable information that can be transfered from the source domain to the target domain.

Could you please clarify what version of dataset ChestX did you test in your original paper? On that original CVPR 2017 paper, the author used ChestX-ray8, which contained 108,948 images. However, currently the data provided by the kaggle link is Chest-Xray 14, which is a modified version. Should we use ChestX-ray8 to reproduce your results?

For track 2, unlabelled data is available. Is it still allowed to use mini-ImageNet for training or can training only be performed on that unlabelled set?

Hi @yunhuiguo,

sorry, I have one more question about the arXiv paper.

Since we can't finish our paper before your workshop deadline, so we will put it on arXiv as the rule required.
I would like to know is it available to submit the arXiv paper to other conferences? (if the conference accepts an arXiv submission).

We will extend the paper with extra experiments, but most of the content will be the same as the arXiv paper.

Thank you for your help.

Hi,

Thanks for sharing the code. I have two questions:

1. Can finetune.py also finetune features learned by meta learning methods (e.g. ProtoNet)?

2. I notice there is a command option --adaptation to adapt meta-learned features on the support set. And when I run:

       python test_with_saved_features.py --method protonet --dataset EuroSAT --n_shot 50 --train_aug --adaptation
   

I get 88.82% +- 0.46% which is much higher than what is reported in Table 1 ( 80.48% ± 0.57%). Further adaptation also brings improvement to ProtoNet on other datasets. Have you tried this option?

Thanks.

Hello! Can we use unlabeled target data to train our model?

Current way requires installing entire ImageNet of 150GB... Previous methods had a much more direct way by having a google drive link?

[1] https://github.com/WenbinLee/DN4
[2] https://github.com/floodsung/LearningToCompare_FSL

Hi,

Was just wondering if the leaderboard/ invitations will be sent out today, as it is 27 May? Thanks.

I think this is an interesting and inspiring work, but in your released code, I don't find the code for "Fine-tuning last-k", "Transductive fine-tuning" and "Transfer from Multiple Pretrained Models" as shown in your paper. I hope you can release these parts as soon. I also want to know the start date of the CVPR2020-VL3-Challenge. Looking forward to your reply!

Hi,

I was wondering why the code seems to have the RAM blowing up (>25 Gb) in the data loading process. Do we need to force the code to utilise disk space as RAM if our RAM size limit is 25 Gb?

Thanks.

Hallo, I see in your paper, the results of approach based Transfer learning is obviously better than the results of methods based Meta learning. I wonder if the challenge will evaluate the results of the two kind of methods separately.
And could we use a different backbone networks, such as ResNet18?

Is the model pre-trained on the entire miniImageNet (100 classes) or on the meta-training set (64 base classes)?
Besides, why the parameter --num_classes is 200 in io_utils.py?
Thank you very much!

Hi @yunhuiguo,

Is it allowed to have any operation between query images if we make sure there is no assumption about their class? (distance calculation etc.)
Or should we keep the query images being independent too?

If we can have those operations, is it allowed to check the predicted class of query images from our model? (e.g. when predicting the second query image, can we check the predicted class of the first images? we won't assume the number of the query images predicted to each class should be same.)

Thank you!

Hi,

I was wondering if we are allowed to use ensemble boosting by training multiple neural networks and reporting the results of the ensemble? Thanks.

Hello, About this challege, 15 query samples of each category can be used for unsupervised training in each episode??

Are there any model checkpoints for this very interesting benchmark?

Hi,

Was just wondering what time the submission is due on May 20 (and which timezone)? Thanks.

I have some question about this challenge:

1. In the setting that we use miniImageNet as the source domain and use CropDiseases, EuroSAT, ISIC2018 and ChestX as the target domains, if we use the base classes (64) of the miniImageNet as the training classes, which classes do we use as the validation classes and do we use all classes in the target domain as test classes?

2. As we can see, ImageNet data (155G) needs to be downloaded to construct the miniImageNet dataset. So can we use a simpler way, just like in CrossDomainFewShot.

3. Can you provide data processing scripts for the four datasets: CropDiseases, EuroSAT, ISIC2018 and ChestX? I only saw the ones of miniImageNet and CUB datasets?

Hi,

I was just wondering if there was an implicit use of unlabelled data in the process of transductive fine-tuning in the baseline paper. This is because transductive fine-tuning uses the batch norm statistics to fine-tune, and thus we are not predicting each query image in the batch independently of other query images in the batch.

This is an issue, because based on the above explanation, transduction fine-tuning should only be allowed for Track 2 and not for Track 1 results? An alternative is for the transductive fine-tuning to only use the statistics of the support images - then this should be permitted in Track 1. May I kindly ask which version this paper has implemented?

Hello, I saw that you use Relation Net as one of the comparison models for your method.

In the paper you use ResNet10 for the backbone of Relation Net encoder, however there's no information about relation module.

Can I get the configuration of the relation module that you used for your experiment?

Hi,

Is it alright to make slight modifications of the evaluation code to make it easier to interface with the models we have? We will make sure that the evaluation protocol is unchanged (which you can verify by checking the code).

Hi, could you please tell us how many GPUs we need to train a few-shot model (Res-12+any classifier) in the setting of 5-way 50-shot. Also, is it necessary？ I mean, 50 samples may not be considered as "few-shot"

Hi, thanks for organizing the challenge!!

We have several questions about the rules:

1. Are we allowed to use ImageNet validation or test set?
2. We should only use ResNet-10 as the backbone?
3. Do we need to respect the preprocessing in the provided code? especially the resolution of images? (many academic papers working on lower resolution)

Thanks and looking forward to your reply.

we used some of the same technology in the two technical files PDFs submitted by track1 and track2. Can the two PDFs use the same or similar descriptions(or Figure) in the public part? and Can I submit track1 and track2 documents in the same submission ?

Hi, could you provide the links to the saved models: MatchingNet, MatchingNet+FWT, MAML, ProtoNet, ProtoNet+FWT, RelationNet, RelationNet+FWT, MetaOpt for the cross-domain experiments in Table 1.

On the challenge website it states: "For each evaluation, the same 600 randomly sampled few-shot episodes should be used for consistency, ... . Participants must employ the supplied code in order to conduct their evaluations." (https://www.learning-with-limited-labels.com/challenge#h.p_h74e3HJnMMqn)

As far as I can see, the the supplied code in finetune.py, does not guarantee, that always the same 600 episodes will be sampled. The only control over randomness is line 164 np.random.seed(10), but this does not seem to be enough.

Could you please clarify, if one should use this exact code or if one should instead take additional measures to guarantee, that always the same 600 episodes will be sampled?

Thank you for your help!

The current way requires to downloading the entire NIH Chest X-ray Dataset of 42GB...This will not be easy in China. o(╥﹏╥)o
We can infer from the "./datasets/Chest?_few_shot.py" file that not all the images in this dataset need to be used, so can you provide a simplified version of the dataset and annotations with a google drive link?

I would like to check implicit transduction learning is allowed to used in Track 1 or not?
We followed the instructions in your paper and other issues.
For the current setting, we use implicit transduction learning in our result for Track 1.

#2
#8

But, I just noticed that in issue 22, you mention that the query samples are not predicted independently if applying implicit transduction learning.

So it's forbidden in Track 1?
We may not have enough time to change the result now...

1. Can we use own processing of datasets, i.e., normalizing the size of all picture in a dataset and packaging it into a large tensor for storage.

2. Can Transductive Inference [1] be used in track 1? Or is implicit transduction learning (i.e., MAML uses batchnorm to share information in test inference) OK.

[1] Transductive Episodic-Wise Adaptive Metric for Few-Shot Learning, ICCV2019

Hi, thanks your code. I wonder before you train the model with miniImageNet, did you pre-trained it with ImageNet first? Thank you.

Hello, we would like to check if we can submit for both Track 1 and Track 2 without using unlabeled data or we are only supposed to submit for Track 1 ?

Hi, I want to reproduce the result in Table 1 in your paper with Protonet provided in your code. But I am not able to reproduce the result. The 5-way 5-shot classification accuracy I can get is 69.23 +- 0.75, which is also 10% lower than reported. Any suggestion?

It seems that the direct link for the ISIC 2018 dataset is not working. Could you update the working link ?

Hi @yunhuiguo .

Thank you for hosting this competition, it's really interesting and challenging!
After carefully reading the rules on your website and in your paper, I still have some questions, please correct me if I misunderstood.

1. Is it allowed to use the paper submitted to your workshop as the paper required in this competition?
2. Is it allowed to modify those hyper-parameters for training? (e.g. epoch, n_query)
   Intuitively, training a module by the normal way will have a better performance than by the meta-train way in current setting. (baseline will see more images)

In one epoch:
baseline will see: 16 (batch size) * 2400 (dataloader length) = 38400 images
protonet will see: (5+16)(n_support + n_query) * 5 (n_way) * 100 (dataloader length) = 10500 images

3. During testing (in finetune.py), it seems only concat the trained backbone with a learnable classifier.
   Other mechanisms in the module will not affect, right? (e.g. calculate the prototype in PN, relation module in RN).
   Or we can modify the code, let other mechanisms also work during testing?

Looking forward to your reply, thank you so much.

Hello, e have obtained the arxiv link of our paper, but CMT3 cannot be modified. How can we upload our arxiv link?

Hello, I've noticed that there are only code of Baseline and ProtoNet. Could you please update the code of other models used in the experiment?

Hi, I have a question regards to the competition rule.

For the transductive setting, do we allow to use intra-class relationships for data? For example, if we have a similarity matrix (shape 100 * 100 in 5way-5shot-15query setting, 5*(5+15) ) and we want to initialize it, can we assume the entry for queries from the same class to have similarity 1? That is, we don't know the current query belongs to which class, but we do know it belongs to the same class with some other query.

Thanks!

The paper says that it failed to operate on the entire benchmark and I was curious what that meant and why.

Thanks in advance for the response and thanks for sharing this awesome benchmark for meta-learning and AGI.

Thank you for sharing the code for the CDFSL benchmark.

I'm currently working on a research project based on the code you've provided, but I noticed that there is a lot of duplicate code in the data and datasets folders. This makes it harder to modify the code, e.g., to customize the transformations across all the datasets.

Would you consider accepting a PR if I work on some refactorizations to remove these duplicates?

Thanks.

Do we have to use PyTorch and python version as below for the competition?

Python 3.5.5

Pytorch 0.4.1

h5py 2.9.0

Thanks

could we change the backbone network?

Hi, I visualized some images in the aircraft carrier category and compared them with the original mini-ImageNet dataset provided by matching network in this website. The image id I checked is n02687172_5357.JPEG (from dataset of this competition), which could not be found on the txt file. Then I checked the category name, there are only 4 categories matched from the competition dataset to the list of txt file provided by the matching network. Since the training set is chosen 64 categories from 100 categories of ImageNet, the number of the matched category is unbelievable. I wonder do you use the subset of ImageNet2012 or modified something else?