Have you tried to convert it with https://github.com/clcarwin/convert_torch_to_pytorch?

I have created a patch dataset, I used the hardnet model to train it from scratch. Strangely, the fpr95 is very high, it can drop from the initial value of 0.82 to 0.68. At the same time, the AP, auc metric is also very high, they can reach up to 0.84. So I do not interpret these metrics value. Additionly, I used the sklearn package to calculate the AP and auc. The code is following:

from sklearn.metrics import average_precision_score,roc_auc_score
def ErrorRateAt95Recall(labels, scores):
    distances = 1.0 / (scores + 1e-8)
    recall_point = 0.95
    labels = labels[np.argsort(distances)]
    # Sliding threshold: get first index where recall >= recall_point.
    # This is the index where the number of elements with label==1 below the threshold reaches a fraction of
    # 'recall_point' of the total number of elements with label==1.
    # (np.argmax returns the first occurrence of a '1' in a bool array).
    threshold_index = np.argmax(np.cumsum(labels) >= recall_point * np.sum(labels))

    FP = np.sum(labels[:threshold_index] == 0) # Below threshold (i.e., labelled positive), but should be negative
    TN = np.sum(labels[threshold_index:] == 0) # Above threshold (i.e., labelled negative), and should be negative
    # FP +TN denotes the number of Negative Samples.
    return float(FP) / float(FP + TN)
def Average_Precision_Score(lables,scores):
    precision=average_precision_score(lables,scores)
    return precision
def Roc_Auc_Score(labels,scores):
    return roc_auc_score(labels,scores)

Can you help me to interpret the problem? @ducha-aiki @DagnyT

pytorch_sift

I can not find the file pytorch_sift @DagnyT @ducha-aiki @spongezhang

Hello,

I want to know if while training Hardnet+ you used --fliprot only or --augmentation as well? I am assuming it is with fliprot only. If yes, then when is the --augmentation used?

Thanks,

Hi,

when the number of keypoints is a multiple of the batch size (128) code crashes with:

Traceback (most recent call last):
  File "./extract_hardnet_desc_from_hpatches_file.py", line 135, in <module>
    data_a = torch.from_numpy(data_a)
RuntimeError: the given numpy array has zero-sized dimensions. Zero-sized dimensions are not supported in PyTorch

I found it is due to the fact that an empty batch to be processed is given. Currently I roughly solved this issue by replacing:

    data_a = torch.from_numpy(data_a)

with

    if data_a.size == 0:
		continue
    data_a = torch.from_numpy(data_a)

Hello,

I have tested your pretrained model of liberty with augmentation with python implementation of HPatches. I was able to reproduce results but when I tested with matlab implementation of HPatches to generate graphs I wasn't able to reproduce retrieval results (patch verification and matching results were correct.) can you please guide what I possibly be doing incorrect. Split is "full", I have checked. Does it has something to do with distractors?

Hardnetlibertywithaug (mAP with python implementation) = 69.66
Hardnetlibertywithaug (mAP with matlab implementation) = 60.64

Thanks in advance

Hi
When I use the demo code HardNet.py with the parameter --fliprot=False --experiment-name=liberty_train/ --gpu-id=4 , there always occur an interruption because of the memory usage. I found that the memory usage will increase drastically along with the training epochs, then it will be interrupted by the OS.
Even I reset the --n-triplets=500000, the memory usage will still cause an interruption.

Hi,
I wonder if we feed the network the whole image and use convolutional sliding window technique ( which instead of outputting 1*128 -> n * 128) where n is number of patches in the image(given that the patch size 32 * 32) and retrain the network, will it enhance the model performance in image description or it will be harder to train ?
Thanks in advance

while trained on different datasets, including ### Lib, Not and Yos, the Test FPR95 accuracy is ### almostly the same. I just run the### run_me.sh using the default parameter.

Hardnet trained on Lib:

�[91mTest set: Accuracy(FPR95): 0.95152000
�[91mTest set: Accuracy(FPR95): 0.95264000
Found cached data data/sets/liberty.pt
Generating 5000000 triplets
Found cached data data/sets/notredame.pt
Found cached data data/sets/yosemite.pt

Hardnet trained on Not:

�[91mTest set: Accuracy(FPR95): 0.95128000
�[91mTest set: Accuracy(FPR95): 0.95264000
Found cached data data/sets/notredame.pt
Generating 5000000 triplets
Found cached data data/sets/liberty.pt
Found cached data data/sets/yosemite.pt

Hardnet trained on Yos:

�[91mTest set: Accuracy(FPR95): 0.95128000
�[91mTest set: Accuracy(FPR95): 0.95152000
Found cached data data/sets/yosemite.pt
Generating 5000000 triplets
Found cached data data/sets/liberty.pt
Found cached data data/sets/notredame.pt

## Meanwhile, Hardnet++ met the same problems. May you explain that??? Thank you

labels = torch.cat([labels,datasets[i][1]])+torch.max(labels)+1

Seemingly, assume that length of datasets[0][1] and dataset[1][1] is n1, and n2 respectively and (n1<n2). So, datasets[0][1][:n1]==datasets[1][1][:n1], but they are not the positive pairs.
Is it right? the original line is a bug ? @ducha-aiki @DagnyT @spongezhang

When I try to run the train code , I get a error : killed by OOM killer
Where in the script may cause this problem？

Looking forward to your apply.

Hi, I made two Euclidean distance histograms, one for positive pairs, one for negative pairs, using the HPatches and the pretrained HardNet++ to extract features, and the extracted features are stored in folder "repeatharnet+" as csv files

I tried "repeatharnet+" features with image verification task, which is very good.
But when I plot the distance histograms, I found out that the margin is insufficient. I believe during training the distance margin was set to 1. the clearly the distance of positive pairs are too high

positive pair distance histogram

negative pair distance histogram

code
`import os
from numpy import genfromtxt
import numpy as np
import matplotlib.pyplot as plt
import pdb

feature_dir = '/HPatches/hpatches-benchmark/data/descriptors/repeathardnet+/'
sequence_list = os.listdir(feature_dir)

negative_dist = np.array([])
for i in range(len(sequence_list)):
for j in range(i+1, len(sequence_list)):
if negative_dist.shape[0] > 18000000:
break
feature_sequence_dir_i = os.path.join(feature_dir, sequence_list[i])
feature_sequence_dir_j = os.path.join(feature_dir, sequence_list[j])
featureList_i = []
featureList_j = []
for img in ['e1', 'e2', 'e3', 'e4', 'e5', 'ref', 'h1', 'h2', 'h3', 'h4', 'h5', 't1', 't2', 't3', 't4', 't5']:
csv_file_i = os.path.join(feature_sequence_dir_i, img) + ".csv"
featureList_i.append(genfromtxt(csv_file_i, delimiter=','))
csv_file_j = os.path.join(feature_sequence_dir_j, img) + ".csv"
featureList_j.append(genfromtxt(csv_file_j, delimiter=','))
for fi in featureList_i:
for fj in featureList_j:
dim = min(fi.shape[0], fj.shape[0])
dij = np.linalg.norm(fi[:dim, :]-fj[:dim,:], axis=1)
negative_dist = np.hstack([negative_dist, dij])
print("negative_dist", negative_dist.shape)

plt.hist(negative_dist, bins='auto') # arguments are passed to np.histogram
plt.title("negative_dist pair dist")
plt.savefig("{} negative_dist pair dist.jpg".format(negative_dist.shape[0]))

positive_dist = np.array([])
for sequence in sequence_list:
feature_sequence_dir = os.path.join(feature_dir, sequence)

featureList = []
for img in ['e1', 'e2', 'e3', 'e4', 'e5', 'ref', 'h1', 'h2', 'h3', 'h4', 'h5', 't1', 't2', 't3', 't4', 't5']:
    csv_file = os.path.join(feature_sequence_dir, img) + ".csv"
    featureList.append(genfromtxt(csv_file, delimiter=','))
# pdb.set_trace()
for i in range(len(featureList)):
    for j in range(i+1, len(featureList)):
        fi = featureList[i]
        fj = featureList[j]
        dij = np.linalg.norm(fi-fj, axis=1)
        positive_dist = np.hstack([positive_dist, dij])
        print("positive_dist", positive_dist.shape)

plt.hist(positive_dist, bins='auto') # arguments are passed to np.histogram
plt.title("positive pair dist")
plt.savefig("{} positive pair dist.jpg".format(positive_dist.shape[0])) `

Hi, When I run the "run_me.sh", it shows "FileNotFoundError: [Errno 2] No such file or directory: 'data/sets/liberty.zip' " error. How to solve it. Thanks

There are so many different type of HardNet. Can you explain the with more details ? Thanks !

I wonder is this loss the same with Batch Hard Loss proposed in this paper In Defense of the Triplet Loss for Person Re-Identification. The idea looks quite similar.

Hey @DagnyT
This issue is regarding the last few changes in the repo where you added the extract_hardnet_desc_from_hpatches_file.py .
The script is not working as it is suppose to be and I have few doubts regarding it

• The paper suggests two step normalization one global and one locally for a patch itself
  

  hardnet/examples/extract_hardnet_desc_from_hpatches_file.py

  Lines 105 to 106 in 1f033f3

  	patches -= 0.443728476019
  	patches /= 0.20197947209

  
  How do you justify this ? should'nt it be
  patches = (patches - patches.mean())/patches.std()

*Next issue is an important one because of which the script fails to run

The way the code expands the mean tensor(1) or std tensor (2) is incorrect, because of which the error

(utkarsh) utkarsh@Viper:~/project/codeBase/hardnet/examples$ python extract_hardnet_desc_from_hpatches_file.py imgs/ref.png out.txt
(72995, 65)
Traceback (most recent call last):
  File "extract_hardnet_desc_from_hpatches_file.py", line 125, in <module>
    out_a = model(data_a)
  File "/home/utkarsh/anaconda3/envs/utkarsh/lib/python2.7/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "extract_hardnet_desc_from_hpatches_file.py", line 72, in forward
    (input - mp.unsqueeze(-1).unsqueeze(-1).expand_as(input)) / sp.unsqueeze(-1).unsqueeze(1).expand_as(input))
  File "/home/utkarsh/anaconda3/envs/utkarsh/lib/python2.7/site-packages/torch/autograd/variable.py", line 725, in expand_as
    return Expand.apply(self, (tensor.size(),))
  File "/home/utkarsh/anaconda3/envs/utkarsh/lib/python2.7/site-packages/torch/autograd/_functions/tensor.py", line 111, in forward
    result = i.expand(*new_size)
RuntimeError: The expanded size of the tensor (1) must match the existing size (128) at non-singleton dimension 1. at /opt/conda/conda-bld/pytorch_1502006348621/work/torch/lib/THC/generic/THCTensor.c:323

I had resolved this by adding an extra .unsqueeze(-1) in the above mentioned lines.

What are your thoughts over this ?

when I run python extract_hardnet_desc_from_hpatches_file.py imgs/ref.png out.txt
error occured :

THCudaCheck FAIL file=/pytorch/aten/src/THC/THCTensorRandom.cu line=25 error=30 : unknown error
Traceback (most recent call last):
File "extract_hardnet_desc_from_hpatches_file.py", line 90, in
checkpoint = torch.load(model_weights)
File "/usr/local/lib/python2.7/dist-packages/torch/serialization.py", line 303, in load
return _load(f, map_location, pickle_module)
File "/usr/local/lib/python2.7/dist-packages/torch/serialization.py", line 469, in _load
result = unpickler.load()
File "/usr/local/lib/python2.7/dist-packages/torch/serialization.py", line 437, in persistent_load
data_type(size), location)
File "/usr/local/lib/python2.7/dist-packages/torch/serialization.py", line 88, in default_restore_location
result = fn(storage, location)
File "/usr/local/lib/python2.7/dist-packages/torch/serialization.py", line 70, in _cuda_deserialize
return obj.cuda(device)
File "/usr/local/lib/python2.7/dist-packages/torch/_utils.py", line 68, in _cuda
with torch.cuda.device(device):
File "/usr/local/lib/python2.7/dist-packages/torch/cuda/init.py", line 228, in enter
_lazy_init()
File "/usr/local/lib/python2.7/dist-packages/torch/cuda/init.py", line 161, in _lazy_init
torch._C._cuda_init()
RuntimeError: cuda runtime error (30) : unknown error at /pytorch/aten/src/THC/THCTensorRandom.cu:25

ubuntu16.04 + cuda8.0+cudnn5.1+pyTorch0.4.0
Thanks for help!

I want to use HPatches as training dataset , and i try to use HPatchesDatasetCreator to generate it through the following command. But i do not know what "path_to_splits_json" mean:

python HPatchesDatasetCreator.py path_to_hpatches path_to_splits_json output_dir

What's splits_json should be?

From your paper, it could be find that u evaluate paris dataset with model trained on Oxford5k. But i don't know what protocol you set on Oxford5k and Paris.
Have you cropped Oxford5k' query images? And Junk images are splited to no-match images?

0%| | 0/5000000 [00:00<?, ?it/s]
Traceback (most recent call last):

Found cached data ../datasets/notredame.pt

Generating 5000000 triplets
File "./code/HardNet.py", line 599, in
train_loader, test_loaders = create_loaders(load_random_triplets = triplet_flag)
File "./code/HardNet.py", line 365, in create_loaders
transform=transform),
File "./code/HardNet.py", line 200, in init
self.triplets = self.generate_triplets(self.labels, self.n_triplets)
File "./code/HardNet.py", line 232, in generate_triplets
if len(indices[c1]) == 2: # hack to speed up process
KeyError: 43567

Can anyone help me? Thank you very much.

Hello,

I have a small query regarding the training epochs used for different versions. Have you kept 10 epochs for all variants of HardNet? Even for training on full Brown data set?
And what did you use to generate graphs for HPatches evaluation? Did you write the script in Matlab for that?

Thanks in advance!

In HardNet.py, line 554, it should be this:
DATASET_DIR=args.w1bsroot.replace('wxbs-descriptors-benchmark/code', 'wxbs-descriptors-benchmark/data/W1BS')).
Because of the RUNPATH in run_me.sh , /code appears twice in args.w1bsroot .
Pardon me if i was wrong.

Thanks for your sharing, when I run hardnet.py by following command ,it raises this error,can you tell me how to solve this problem?

python ./code/HardNet.py --w1bsroot "$DATASETS/wxbs-descriptors-benchmark/code/" --fliprot=False --n-triplets=500 --experiment-name=liberty_train/ $@ | tee -a "$DATALOGS/log_HardNet_Lib.log"

I am very interested in your work but do not how to repeat the experiments of Oxford5k and Paris6k.
Would you like to share me the code and give me a hand?

When I use all 6 mbrown datasets as the training set, the mean, std, lens is unset in the /path/to/pythonlib/site-pachages/torchvision/datasets/phototour.py.
How should I set them?

I think all the np.random.randint(0, len(sth) - 1) should be changed to np.random.randint(0, len(sth)) instead since the larger integer is excluded according to NumPy manual.

Hi, I try to use hardnet to replace SIFT as the local feature extractor for landmark recognition.
But I got a worse results than SIFT.
Here is my pipeline:
(1) Use SIFT key point detector implemented by openCV to get the location and diamater of key points.
(2) sample a patch according to the diameter for each key point an resize it to 32x32.
(For example, for a key point with with diameter 15, I sample 15x15 patch and then resize it to 32x32)
(3) use hardnet to extract 128-dimensional feature

For step 2, I found that the diameter of most key points are pretty small (<8 pixels), I suspect this is the reason why hardnet feature perform worse.

Do I need to always sample 32x32 patch for each key point instead of using the diameter of the key point? Thanks for answering :).

When I run train_hardnet_on_HPatches_per_split.sh, key error occurred. It seems that something wrong in data generation?

How many dicractors do you set in your retrieval task in the Hpatches dataset?
5K ro 1W?

Just for advice, when you running run_me.sh, cause Pytorch get updated, the grammar about tensor has changed also. just replace loss.data[0] with loss.data.item(), the error would be perfectly solved.

The test accuracy is always change. Did you get the average precision of the test data ?

I am wondering what is the version of pytorch

Hi，have you try to test or train HardNet on the indoor datasets?do pretrained models fit the indoor environment matching task？

When I run HardNetClassicalHardNegMiningSiftInit, it performance worse heavily than HardNetMultipleDatasets. In theory, the performance of HardNetClassicalHardNegMiningSiftInit should not be worse than HardNetMultipleDatasets.
I note there are some difference in convoloution layer, bias=False for HardNetMultipleDatasets, but True for HardNetClassicalHardNegMiningSiftInit.
Another difference is that the learning rate, they have 100 times difference.
Why HardNetClassicalHardNegMiningSiftInit have a worse performance than HardNetMultipleDatasets? @DagnyT

Hello,

Can you please elaborate the differences in hardnet.py , hardnetclassicalhardnegmining.py and hardnetclassicalhardnegminingsiftinit.py?

Also, when I run run_me.sh, the default experiment is train on liberty and test on yosemite and notredame?

I use your code to test Notredame and Yosemite using Yosemite dataset. Here is the result.

Train Epoch: 9 [4997120/5000000 (100%)] Loss: 0.645969: : 4883it [16:58, 4.80it/s]
notredame Test Epoch: 9 [92160/100000 (92%)]: : 98it [00:40, 2.43it/s]
Test set: Accuracy(FPR95): 0.00488000

yosemite Test Epoch: 9 [92160/100000 (92%)]: : 98it [00:42, 2.30it/s]
Test set: Accuracy(FPR95): 0.01864000

The test result is better than that in your paper. So when you wrote your paper, did you average several tests ?

When I use the test provide example:

cd examples/caffe
python extract_hardnetCaffe_desc_from_hpatches_file.py ../imgs/ref.png hardnet_caffe.txt

1123 patches to describe in ../imgs/ref.png
[libprotobuf ERROR google/protobuf/text_format.cc:245] Error parsing text-format caffe.NetParameter: 27:3: Invalid value for boolean field "bias_term". Value: "False".
WARNING: Logging before InitGoogleLogging() is written to STDERR
F0424 00:21:30.764016 12746 upgrade_proto.cpp:88] Check failed: ReadProtoFromTextFile(param_file, param) Failed to parse NetParameter file: HardNet.prototxt
*** Check failure stack trace: **

Something is wrong with HardNet.prototxt or HardNet++.caffemodel . I guess the error comes from the caffemodel. Hope you can modify it !

Could you fix the link?

Could you provide some information on the label formatting for the provided training sets?

What are the two columns of info.txt? Why do some numbers in the first column repeat? What are the columns of interest.txt and m50_XXXX.txt?

For context, I am trying to create my own training set for a different type of data and need to train from scratch.

Can you tell me if different data sets have a significant impact on descriptor performance？
For example, would it be better for me to train with the vehicle data set for the matching of vehicle features？And when I matched the features of the architectural image, I used the architectural image for training?

Hello,I trained on liberty and tested on notredame successfully. But the following bug appears when i try to create test_loaders with yosemite dataset :

RuntimeError: $ Torch: invalid memory size -- maybe an overflow? at ..\aten\src\TH\THGeneral.cpp:188