I'm trying to use triplet loss train resnet on arbitrary datasets but I can't. When I train mnist using your repo, I see fraction_positive decreasing over time but then, for resnet it goes up instead. batch all and batch hard losses start way above margin and decrease over time to margin, making mean distance to be about zero.

First I thought my data is not sufficient for triplet loss but when I saw the very same pattern for mnist, I think there might be a problem somewhere.

Hi Omoindrot,
great explanation and implementation of triplet loss.

I’m building a model that uses the online triplet loss, but in my case, the network requires 2 domains: the draws domain and the images domain. The purpose of this network is to learn a good embedding for SBIR (Sketch-based image retrieval), corresponding to filling the gap between draws and images (the 2 branches shares partially the weights).

I tried to change the function for the 2 domains as follows:

1. I have builded a (Keras) generator that outputs 2 embeddings vectors (draws and icons) with the same class indexes:

e.g. :

X_out = [ 
  [ draw_emb_class2, draw_emb_class1, draw_emb_class3, … ],
  [ icon_emb_class2, icon_emb_class1, icon_emb_class3, … ],
]
y_out = [ 
  [ 2, 1, 3, … ]
]

2. Then I have builded a matrix with the pairwise distances with the draw embeddings as anchors and the images as pos/neg (not the best code...). I obtained a matrix of distances as it follows:

# d_ is the distance
pairwise_dist = [
 d_draw1-draw1, d_draw1-image2, d_draw1-image3, 
 d_draw2-image1, d_draw2-draw2, d_draw2-image3, 
 d_draw3-image1, d_draw3-image2, d_draw3-draw3, 
]

Here is the code (changes are in the "multi domain version" sections):

def batch_hard_triplet_loss_multi_domain(y_true, y_pred, margin=1, squared=False):

    # I think keras need this
    labels = tf.squeeze(y_true, axis=-1)
  

    # *************************multi domain version*****************************
    _batch_size = y_pred.shape[1] // 2

    # embedding = (batch, feat)
    embedding_d = y_pred[:, :_batch_size]
    embedding_i = y_pred[:, _batch_size:]
    
    pairwise_dist_same = multidomain_pairwise_dist(embedding_i, embedding_i)
    pairwise_dist_diff = multidomain_pairwise_dist(embedding_i, embedding_d)

    pairwise_dist = tf.linalg.set_diag(
        pairwise_dist_diff,
        tf.linalg.diag_part(                                                    # all zeros
            pairwise_dist_same,
        )
    )
    # **************************************************************************


    # For each anchor, get the hardest positive
    # First, we need to get a mask for every valid positive (they should have same label)
    

    # *************************multi domain version*****************************
    mask_anchor_positive = _get_anchor_positive_triplet_mask_multi_domain(labels)
    # **************************************************************************


    mask_anchor_positive = tf.compat.v1.to_float(mask_anchor_positive)

    # We put to 0 any element where (a, p) is not valid (valid if a != p and label(a) == label(p))
    anchor_positive_dist = tf.math.multiply(mask_anchor_positive, pairwise_dist)

    # shape (batch_size, 1)
    hardest_positive_dist = tf.math.reduce_max(anchor_positive_dist, axis=1, keepdims=True)
    tf.summary.scalar("hardest_positive_dist", tf.math.reduce_mean(hardest_positive_dist))

    # For each anchor, get the hardest negative
    # First, we need to get a mask for every valid negative (they should have different labels)
    mask_anchor_negative = _get_anchor_negative_triplet_mask(labels)
    mask_anchor_negative = tf.compat.v1.to_float(mask_anchor_negative)

    # We add the maximum value in each row to the invalid negatives (label(a) == label(n))
    max_anchor_negative_dist = tf.math.reduce_max(pairwise_dist, axis=1, keepdims=True)
    anchor_negative_dist = pairwise_dist + max_anchor_negative_dist * (1.0 - mask_anchor_negative)

    # shape (batch_size,)
    hardest_negative_dist = tf.math.reduce_min(anchor_negative_dist, axis=1, keepdims=True)
    tf.summary.scalar("hardest_negative_dist", tf.math.reduce_mean(hardest_negative_dist))

    # Combine biggest d(a, p) and smallest d(a, n) into final triplet loss
    triplet_loss = tf.math.maximum(hardest_positive_dist - hardest_negative_dist + margin, 0.0)

    # Get final mean triplet loss
    triplet_loss = tf.math.reduce_mean(triplet_loss)

    return triplet_loss

And these are the called functions:

def _get_anchor_positive_triplet_mask_multi_domain(labels):

    # *************************multi domain version*****************************
    mask = tf.math.equal(tf.expand_dims(labels, 0), tf.expand_dims(labels, 1))
    # **************************************************************************

    return mask



def multidomain_pairwise_dist(A, B):
    """
    Computes pairwise distances between each elements of A and each elements of B.
    Args:
      A,    [m,d] matrix
      B,    [n,d] matrix
    Returns:
      D,    [m,n] matrix of pairwise distances
    """
    #with tf.variable_scope('pairwise_dist'):
    # squared norms of each row in A and B
    na = tf.reduce_sum(tf.square(A), 1)
    nb = tf.reduce_sum(tf.square(B), 1)

    # na as a row and nb as a co"lumn vectors
    na = tf.reshape(na, [-1, 1])
    nb = tf.reshape(nb, [1, -1])

    # return pairwise euclidean difference matrix
    D = tf.sqrt(tf.maximum(na - 2 * tf.matmul(A, B, False, True) + nb, 0.0))
    return D

Now, do you think this approach could make sense and could work with online triplet loss?

Thank you.

Hi,

I have made some experiments about batch all with 70 million triplet samples from our own vehicle tracking system.

Then, I found a loss is converging to a margin, while distances are collapsing or exploding. In the worst case, all the distances get close to 0, in which case such a loss converges to a margin, and no progress could be made further.

Such a model can not be reused even for verification because a positive distance is not guaranteed to be less than a margin, so is negative.

I think this is because the loss function is not enough.

triplet_loss = anchor_positive_dist - anchor_negative_dist + margin

This does not penalize when both distances of positive and negative collapse or explode together because it is a subtraction.

So, I am experimenting this version:

if balanced:
        # add two more loss elements to prevent distances from collapsing or exploding
        loss_positive = anchor_positive_dist * margin
        loss_negative = margin / anchor_negative_dist
        triplet_loss += loss_positive + loss_negative

https://github.com/ggsato/tensorflow-triplet-loss/blob/triplet_for_trackings/model/triplet_loss.py#L198

Still under experiments, but so far, a training with the balanced one described above works better. A positive mean distance keeps stay around 0.03, while a negative mean distance is forced to be apart beyond 4.0 or more.

Did anyone face with this issue? And what's your workaround?

Could you give me some clues or papers？ thx

hello i utilize your useful program for face recognition,my implemented network is inception resnet v2 and

1. batch_size=16,
   2.lr=0.001
   3.embeding_size=128
   4.train_size=708
   5.eval_size=396
   6.margin=0.5
   7."triplet_strategy": "batch_all"
   loss = 0.5458313, step = 1
   loss = 0.0, step = 101 (3229.453 sec)
   without pretrained weights and only with your implementation two layers CNN ,loss for evaluation goes to about 0.35(batch_size=64,lr=1e-4)with data augmentation .i use embeddings vector from PREDICTION mode as input of my SVM and KNN classifier but accuracies of both are too low about 10%. what do think about this low accuracy for recognition?which part do you think have caused this low accuracy (triplet part or classification part)?

Why the code at line 34 in triplet_loss.py is "if not Squared"?
Should it be "if Squared", for the following codes are to calculate the squared euclidean distance matrix?

Thank you for you tutorial and implementation of triplet loss. I have one questions about how to use the triplet loss for classification. If we need to do the classification, we could use the "embeddings" and then take the "embeddings" as feature then train model like SVM,GBDT to classify? This way the triplet-loss works like a feature extractor. Do you have any other suggestions for classification? Thanks.

I have a training set with N different identities [I_1, I_2 ... I_N] and for each identity I have several images extracted from a video. I trained the triplet loss model on this set. Now I have another set of images related to a new identity I_N+1 and I want all the future images related to I_N+1 to be recognized. What I thought is:

• Compute the mean embedding of the available images related to I_N+1
• When a new image is provided (i.e. at serving time), compute the embedding and compare it with the mean embedding. If the difference is below a given threshold, the identity is confirmed.

Is this approach correct? Is it better to compute several embeddings related to I_N+1 and use them separately?

Hello @omoindrot , thanks for your great work. I build a Resnet to do speaker recognition job, but when I use triplet loss to train my model the loss is 0.5 and didn't decrease. Am I use triplet loss in wrong way?

    def _inference(self, inp):
        for i in range(self.n_blocks):
            if i > 0:
                inp_channel = inp.get_shape().as_list()[-1]
                inp = self._conv2d(name='conv5%d'%i, shape=[5, 5, inp_channel, self.out_channel[i]],
                                   strides=[1, 2, 2, 1], x=inp, padding='SAME')
                inp = self._residual_block(inp, self.out_channel[i], "residual_block_%d" % i,
                                           is_first_layer=False)

            else:
                inp_channel = inp.get_shape().as_list()[-1]
                inp = self._conv2d(name='conv5%d' % i, shape=[5, 5, inp_channel, self.out_channel[i]],
                                   strides=[1, 2, 2, 1], x=inp, padding='SAME')
                inp = self._residual_block(inp, self.out_channel[i], "residual_block_%d" % i,
                                           is_first_layer=True)
            print(inp.get_shape().as_list())

        inp = tf.reduce_mean(inp, axis=1)

        inp = tf.reshape(inp, [-1, inp.get_shape().as_list()[1]*inp.get_shape().as_list()[2]])
        
        weight_affine = self._new_variable("affine_weight", [inp.get_shape().as_list()[-1], 512],
                                           weight_type="FC")
        bias_affine = self._new_variable("affine_bias", [512], "FC")
        inp = tf.nn.relu(tf.matmul(inp, weight_affine) + bias_affine)
        print(inp.get_shape().as_list())
        output = tf.nn.l2_normalize(inp)
        return output

    def _triplet_loss(self, inp, targets):
        if targets.get_shape().as_list()[-1] != 1:
            targets = tf.argmax(targets, axis=1)
        loss = triplet_loss.batch_hard_triplet_loss(targets, inp, 0.5)
        return loss

Hi, first of all thanks for your great work!
I'm using your code to topic clustering texts, and apparently it's working fine on the first tests with few classes. I have the problem that I want to train a multi-label classification (say, a text can have label "politics" and "opinion"). I know that the code should be changed for this purpose, but I want to know if you know if you know whether this model of triplet-loss will work with multi-label.

On the other hand, after trained a model, i want to use it to classify new or test images/texts. Say I have an image of a "7" and I want to predict it's embedding and calculate the distances to know what label is the "nearest" from it. How could it be done?

Thanks again for all.

Can you release the code for market1501 dataset, please?

Is base_model defined by myself?

I think below line is some misleading.
https://github.com/omoindrot/tensorflow-triplet-loss/blob/master/model/triplet_loss.py#L29
Instead of distances = tf.expand_dims(square_norm, 0) - 2.0 * dot_product + tf.expand_dims(square_norm, 1) , I prefer
distances = tf.expand_dims(square_norm, 1) - 2.0 * dot_product + tf.expand_dims(square_norm, 0) , which is more consisten with ||a - b||^2 = ||a||^2 - 2 <a, b> + ||b||^2.

The current implementation uses tf.data for the input pipeline and only create random batches.

The triplet loss implementation accepts the following inputs:

def batch_all_triplet_loss(labels, embeddings, margin, squared=False):

where the labels could be anything.

Why a high number of classes is an issue

In the case of MNIST, if the batch size is big enough you are assured to have a high amount of useful triplets. For instance if you have a batch of 100 random images, you will on average have 10 images of each digit and you will be able to easily construct triplets.

However when there are a lot of classes, this approach breaks down. For instance if you have 10,000 classes and a batch size of 100, the probability that all images in the batch have distinct labels is around 60%. If this happens, then no triplet can be built and the loss will be useless.

Proposed solution

I'll work on building a data pipeline that automatically creates balanced batches of data. This should use the following arguments in the params.json file:

• num_classes_per_batch: number of different classes to include in the batch
• num_examples_per_class: number of examples of each class to include

The batch size will be the product of these two numbers.

Additionally, the previous signature of batch_all_triplet_loss(labels, embeddings, margin) can be replaced with balanced_batch_all_triplet_loss(num_classes, num_examples, embeddings, margin) since we don't need the labels in this case.

We don't need the labels because all the batches will always contain the same order of examples. The labels will always look like:

[1, 1, 1, 1, 5, 5, 5, 5, 3, 3, 3, 3]

(in this example, we have num_classes=3 and num_examples=4 for a total batch size of 12)

The reason to do this is that we don't need to compute the masks dynamically with _get_triplet_mask(labels) since the mask will always be the same, so we can hard-code it statically in the graph. This will lead to some performance improvements.

I would like to train the model on my own dataset. Where can I find instructions on how to build the dataset?

Hi @omoindrot I am utilizing your foundation code on a custom dataset and I'm getting multiple clusters for same class when used tsne to visualize. My embeddings are 128 dimensions. Am I doing something wrong or there might be a single cluster for each class and when dimension is reduced it is moving into 3 different cluster??

Hi,

I am currently new to all this and still learning. Is your implementation very similar to FaceNet approach? As I am currently working on a facial recognition project and I am trying FaceNet approach. I have read David Sandberg's on FaceNet and noticed he used a model architecture of inception ResNet v1 to train the dataset.

May I know what is the model architecture that you are using and if I were to use your approach, must I retrain a huge dataset of photos and use this pretrained model, to train my own much smaller dataset? Any suggestions of what I need to do? Thank you.

During training, the projector shows some points in the PCA space. Every point has the same color. What do they represent? What can I infer about the training process from this graph?

i read the post and understand that the network will be for feature extraction (is this right ?)
i want to make model on my own data set

when i read the blog post i understand that the steps to generate my network is
1- prepare my data set using tensor flow
for example i will use below code to prepare data set

import cv2

# Use a custom OpenCV function to read the image, instead of the standard
# TensorFlow `tf.read_file()` operation.
def _read_py_function(filename, label):
  image_decoded = cv2.imread(filename.decode(), cv2.IMREAD_GRAYSCALE)
  return image_decoded, label

# Use standard TensorFlow operations to resize the image to a fixed shape.
def _resize_function(image_decoded, label):
  image_decoded.set_shape([None, None, None])
  image_resized = tf.image.resize_images(image_decoded, [28, 28])
  return image_resized, label

filenames = ["/var/data/image1.jpg", "/var/data/image2.jpg", ...]
labels = [0, 37, 29, 1, ...]

dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))
dataset = dataset.map(
    lambda filename, label: tuple(tf.py_func(
        _read_py_function, [filename, label], [tf.uint8, label.dtype])))
dataset = dataset.map(_resize_function) 

2- then i will pass this data set to train.py as arg
but what i can't understand is params file (i don't know what is the values inside it )
and also will the code generate model for me ?and after that i can send image to that model and the model will recognize it

i am bit confused can you help me to understand and make model on my own data set
thanks in advance
i appreciate your help

I managed to train the network. I see that there is a script to evaluate it on a test set. How can I make predictions in deploy phase? Should I keep using the tf Estimator with single images?

Hi. I am new to this repo. I was able to get the MNIST demo running quickly. Nice work!

I am interested in training my own embeddings for various vehicle makes/models. For example, there is the "Cars" dataset [ http://ai.stanford.edu/~jkrause/cars/car_dataset.html ] where annotations are available for the model/make of a vehicle.

My question is -- is the CNN architecture here general enough to handle vehicles? Do I need something more sophisticated like the VGG 1024 CNN in Faster RCNN? Thanks very much.

Hi, thanks for sharing your project,

In my own project, I trained the densenet network first, and then used the output as input for the new network, training a very simple DNN network.

def build_model(is_training, images, params):
"""Compute outputs of the model (embeddings for triplet loss).

Args:
    is_training: (bool) whether we are training or not
    images: (dict) contains the inputs of the graph (features)
            this can be `tf.placeholder` or outputs of `tf.data`
    params: (Params) hyperparameters

Returns:
    output: (tf.Tensor) output of the model
"""
out = images    

with tf.variable_scope('fc_1'):
    out = tf.layers.dense(out, 1024，activation=tf.nn.relu)
       
with tf.variable_scope('fc_2'):
    out = tf.layers.dense(out, params.embedding_size)


return out

But there were the following bugs:
Traceback (most recent call last):
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64,64,64] vs. [0,0,0]
[[Node: Mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ToFloat_1, add_2)]]
[[Node: truediv_1/_77 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_168_truediv_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 50, in
estimator.train(lambda: train_input_fn('data/val.tfrecords', params))
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 859, in _train_model_default
saving_listeners)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 1059, in _train_with_estimator_spec
_, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 567, in run
run_metadata=run_metadata)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1043, in run
run_metadata=run_metadata)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1134, in run
raise six.reraise(*original_exc_info)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/six.py", line 693, in reraise
raise value
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1119, in run
return self._sess.run(*args, **kwargs)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 1191, in run
run_metadata=run_metadata)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/monitored_session.py", line 971, in run
return self._sess.run(*args, **kwargs)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64,64,64] vs. [0,0,0]
[[Node: Mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ToFloat_1, add_2)]]
[[Node: truediv_1/_77 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_168_truediv_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'Mul', defined at:
File "train.py", line 50, in
estimator.train(lambda: train_input_fn('data/val.tfrecords', params))
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 363, in train
loss = self._train_model(input_fn, hooks, saving_listeners)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 843, in _train_model
return self._train_model_default(input_fn, hooks, saving_listeners)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 856, in _train_model_default
features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/estimator/estimator.py", line 831, in _call_model_fn
model_fn_results = self._model_fn(features=features, **kwargs)
File "/home/popzq/zsl/tensorflow-triplet-loss-master-zsl/model/model_fn.py", line 95, in model_fn
squared=params.squared)
File "/home/popzq/zsl/tensorflow-triplet-loss-master-zsl/model/triplet_loss.py", line 160, in batch_all_triplet_loss
triplet_loss = tf.multiply(mask, triplet_loss)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_ops.py", line 337, in multiply
return gen_math_ops.mul(x, y, name)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 4759, in mul
"Mul", x=x, y=y, name=name)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
op_def=op_def)
File "/home/popzq/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1718, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Incompatible shapes: [64,64,64] vs. [0,0,0]
[[Node: Mul = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ToFloat_1, add_2)]]
[[Node: truediv_1/_77 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_168_truediv_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

I would appreciate it if you could help me！

batch_all_triplet_loss is giving this issue however hard loss is working fine.

AttributeError Traceback (most recent call last)
in
4 #return batch_hard_triplet_loss(y_true, y_pred,margin=0.1)
5
----> 6 model.compile(optimizer='sgd', loss=func, metrics=["accuracy"])
7
8 model.fit_generator(

c:\users\akshay\anaconda3\envs\pythongpu\lib\site-packages\keras\engine\training.py in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, weighted_metrics, target_tensors, **kwargs)
340 with K.name_scope(self.output_names[i] + '_loss'):
341 output_loss = weighted_loss(y_true, y_pred,
--> 342 sample_weight, mask)
343 if len(self.outputs) > 1:
344 self.metrics_tensors.append(output_loss)

c:\users\akshay\anaconda3\envs\pythongpu\lib\site-packages\keras\engine\training_utils.py in weighted(y_true, y_pred, weights, mask)
415 if weights is not None:
416 # reduce score_array to same ndim as weight array
--> 417 ndim = K.ndim(score_array)
418 weight_ndim = K.ndim(weights)
419 score_array = K.mean(score_array,

c:\users\akshay\anaconda3\envs\pythongpu\lib\site-packages\keras\backend\tensorflow_backend.py in ndim(x)
617 ```
618 """
--> 619 dims = x.get_shape()._dims
620 if dims is not None:
621 return len(dims)

AttributeError: 'tuple' object has no attribute 'get_shape'

how can i load my own dataset instead of Mnist dataset?( dataset = mnist_dataset.train(data_dir)

I am trying to inference using the trained network, however I am not sure where to put the image in, and what is the network node of the embedding. Since the estimator with dataset is not friendly for a raw image, could you provide an interface more easy for inference?

Hi，omoindrot
Thanks for your code,I meet some question, loss value is approximate of margin,i found that the distance is close to 0，I don't know how it was caused.
Does the output of the network need to be L2 normalized,?what is the role of L2 normalization?How to set the value of margin，if i don't use L2 normalized？
thanks

I was following your article in link here Triplet Loss and Online Triplet Mining in TensorFlow. I wanted to save the model weights that are defined in embedImages(Images) method. Later I wanted to use these weights to calculate the embeddings of given input image.
I tried running this by running new session as given below ;
im = cv2.imread(img_path)
img = np.reshape(im, (1,374,388,3))
with tf.Session() as session:
result = session.run(embedded_images, feed_dict={Images: img})
print(result)

But didn't get any useful results?
Can you share idea how to do it.
@omoindrot

I successfully create a dataset with 224x224x3 images and I'm feeding it to the estimator instead of the MNIST dataset. However, when I run the train script, I got the error

ValueError: An initializer for variable model/block_1/conv2d/kernel of <dtype: 'string'> is required

Could it be related to the structure of the dataset?

Hey!

Thank you for a well written blog, and a well commented piece of code to go with it. The problem I am working on requires that I compute the similarity between two images. As you suggested in another issue, I created my own pipeline to use images. For the purpose of testing, I used the MNIST dataset again. I am having some trouble understanding what to normalise so that pairwise distances lie between 0 and 1. In other words, two images of the same class must have a similarity of 0, and an image ( from possibly untrained class ) from a dissimlar class have a similarity score of 1. Any suggestions?

Hi, @omoindrot

Thanks for sharing your code! And I am right now crafting a small project using triplet loss.

I've cloned the code, run it, and all fine. But when applying to my code, in a mini-batch context, code went crack at this line. The shapes of tensors were examined and compared to this mnist example. I found dimension of label is the most possible reason.

Hope you could shed me some light. I am really stucked.... Thanks in advance.

Following is the error message I got:

ValueError: Dimensions must be equal, but are 82398 and 100 for 'Mul' (op: 'Mul') with input shapes: [100,100,100,82398], [100,100,100].

Next the shapes of tensors:
In my code (batch_size is 100, number of classes is 82398):

NFO:tensorflow:pairwise_dist: (100, 100)
INFO:tensorflow:labels: (100, 82398)
INFO:tensorflow:embeddings: (100, 512)
INFO:tensorflow:anchor_p_dist: (100, 100, 1)
INFO:tensorflow:anchor_n_dist: (100, 1, 100)
INFO:tensorflow:indices_equal: (?, ?)
INFO:tensorflow:indices_not_equal: (?, ?)
INFO:tensorflow:i_not_equal_j: (?, ?, 1)
INFO:tensorflow:i_not_equal_k: (?, 1, ?)
INFO:tensorflow:j_not_equal_k: (1, ?, ?)
INFO:tensorflow:label_equal: (100, 100, 82398)
INFO:tensorflow:i_equal_j: (100, 100, 1, 82398)
INFO:tensorflow:i_equal_k: (100, 1, 100, 82398)
INFO:tensorflow:distinct_indices: (?, ?, ?)
INFO:tensorflow:valid_labels: (100, 100, 100, 82398)
INFO:tensorflow:mask: (100, 100, 100, 82398)
INFO:tensorflow:triplet_loss: (100, 100, 100)

In your code:

INFO:tensorflow:pairwise_dist: (?, ?)
INFO:tensorflow:labels: (?,)
INFO:tensorflow:embeddings: (?, 64)
INFO:tensorflow:anchor_p_dist: (?, ?, 1)
INFO:tensorflow:anchor_n_dist: (?, 1, ?)
INFO:tensorflow:indices_equal: (?, ?)
INFO:tensorflow:indices_not_equal: (?, ?)
INFO:tensorflow:i_not_equal_j: (?, ?, 1)
INFO:tensorflow:i_not_equal_k: (?, 1, ?)
INFO:tensorflow:j_not_equal_k: (1, ?, ?)
INFO:tensorflow:label_equal: (?, ?)
INFO:tensorflow:i_equal_j: (?, ?, 1)
INFO:tensorflow:i_equal_k: (?, 1, ?)
INFO:tensorflow:distinct_indices: (?, ?, ?)
INFO:tensorflow:valid_labels: (?, ?, ?)
INFO:tensorflow:mask: (?, ?, ?)
INFO:tensorflow:triplet_loss: (?, ?, ?)

TypeError: Expected binary or unicode string, got <PrefetchDataset shapes: ((?, 784), (?,)), types: (tf.float32, tf.int32)>

Hi, thanks for sharing your project,

I would like to know how to produce visual embeddings like shown in project page.
I guessed that after execute python visualize_embeddings.py --model_dir experiments/base_model, I could see it in Tensorboard at projector's tab, but I get hang on computing PCA.

Dear developers,

I observed some TensorFlow API usage in your code that might lead to serious compatibility problems(like a crash).

For tf API tf.reduce_max and tf.reduce_min, their parameter keepdims appears only since tf v1.5.0-rc0. If use this keyword to pass the value, the program will crash in older versions of TensorFlow. It's better to use keep_dims for earlier versions.

I have made a patch for this problem for you in #41
Look forward to your response! @ymcasky @dlfelps @omoindrot
Thanks!

How can I use tf.data.dataset. Zip ((images, labels)) as you do when I use tfrecorder to read my data sets? What format do these two parameters need to be in?

Hi Omoindrot.

Thanks for the great explanation of triplet loss on your blog. It is one of the few resources I found online which talked about the implementation. I have been trying to simulate a dataset for embedding tests. I am attaching the full code for your reference. Maybe you could have a look if you have some time and let me know what I have been doing wrong:

I am sampling data points from 2 different distributions and wanting to see if I can generate lower dimensional embeddings for both.

`# -*- coding: utf-8 -*-
"""
Created on Tue Sep 24 18:44:04 2019

@author: aangris
"""

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from sklearn import datasets
from sklearn.decomposition import PCA
from omnidrot_loss_utils import batch_all_triplet_loss

# import some data to play with
iris = datasets.load_iris()

def load_data():
    X = iris.data #[:, :2]  # we only take the first two features.
    y = iris.target.reshape(-1,1)
    XY_labeled = np.hstack((X,y))
    np.random.shuffle(XY_labeled)
    return XY_labeled[:,:-1] , XY_labeled[:,-1]

def make_shared_branch(input_tensor):
    layer_1_units = 5
    layer_2_units = 3
    with tf.variable_scope('layer1_scope',reuse = tf.AUTO_REUSE):
        weights = tf.get_variable('weights1',
                                  shape = [input_tensor.shape[1].value,layer_1_units],
                                  initializer = tf.truncated_normal_initializer(1/np.sqrt(layer_1_units)))
        biases = tf.get_variable('b1',
                                 initializer=tf.ones_initializer(),
                                 shape = [layer_1_units])
        layer1 = tf.nn.relu(tf.matmul(input_tensor,weights)+biases)
        
        drop_out1 = tf.nn.dropout(layer1,rate = 0.5)
        
    with tf.variable_scope("layer2_scope",reuse = tf.AUTO_REUSE):
        weights = tf.get_variable(
            'weights2', 
            shape=[layer_1_units, layer_2_units],
            initializer=tf.truncated_normal_initializer(
                stddev=1.0/np.sqrt(float(layer_2_units))))
        biases = tf.get_variable(
            'biases2', 
            initializer=tf.ones_initializer(),
            shape=[layer_2_units])
        layer2 = tf.nn.relu(tf.add(tf.matmul(drop_out1, weights),biases))
    #this is the only thing that has been changed.
    #sigmoid gives some problems with embeddings sticking to each other
    #this (with relu) gives pretty good results
#    with tf.variable_scope("lambda_layer"):
#        lambda_layer  = tf.nn.l2_normalize(layer2,axis=1)
#        
        return layer2

num_imgs = 200
dims = 10  

anchor = tf.placeholder(tf.float32,(None,dims),"anchor")
labels = tf.placeholder(tf.int32,shape = (None),name = "labels")

learning_rate = tf.placeholder(tf.float32,shape=[])

anchor_branch = make_shared_branch(anchor)

#X,y = load_data()
np.random.seed(1)
X1 = np.random.multivariate_normal(mean = np.array([40]*dims),cov = np.eye(10),size = num_imgs)
X2 = np.random.multivariate_normal(mean = np.array([50]*dims),cov = np.eye(10),size = num_imgs)
X = np.vstack((X1,X2))
y = np.repeat([1,2],200).reshape(-1,1)
XY = np.hstack((X,y))
np.random.shuffle(XY)
X = XY[:,:-1]
y = XY[:,-1]
#test
#trips = generate_triplets(X,y)
loss_mean_collector = []
val_loss_mean_collector = []
train_cutoff = len(X)*0.8
test_cutoff = len(X)*0.2
EPOCHS = 500
lr = 1e-3
#A very high learning rate collapses the embeddings. Try smaller learning rates if you want to prevent collapse
#Seems like starting from 0.1 for triplet loss (NON BPR) is a bad strategy.
#Margin fixed at 0.5 

loss = batch_all_triplet_loss(labels,anchor_branch,margin=0.05,squared=False)
train_step = tf.train.AdamOptimizer(learning_rate = learning_rate).minimize(loss[0])
init = tf.global_variables_initializer()
sess = tf.Session()
sess.run(init)
batch_size = 30
from tqdm import tqdm
loss_collector = []
frac_pos =[]
loss_mean_collector = []
with sess.as_default():
    for i in range(EPOCHS):
        for b in tqdm(range(int(train_cutoff//batch_size))):
            X_train= X[b:batch_size+b]
            y_train = y[b:batch_size+b]
            '''   
            shape1, text1, im_labels, shape_text_margin = args[0],args[1],args[2],args[3]
            im_im_labels = args[4]
            shape_margin = args[5]
            '''
            custom_dict = {anchor : X_train,
                           labels : y_train,
                           learning_rate : lr}
            _ = sess.run(train_step,feed_dict = custom_dict)
            loss_value,fraction_pos = sess.run(loss,feed_dict = custom_dict)            
            frac_pos.append(fraction_pos)
            loss_collector.append(loss_value)
        if i%500==0:
            lr = lr*0.5
            print("Epoch #",i)
        print(np.mean(loss_collector))
        print(np.mean(frac_pos))
        loss_mean_collector.append(np.mean(loss_collector))
    frac_pos = []

plt.plot(loss_mean_collector)
def rand_jitter(arr):
    stdev = .01*(max(arr)-min(arr))
    return arr + np.random.randn(len(arr)) * stdev
with sess.as_default():
    Xt,yt = X,y
    newX = sess.run(anchor_branch,feed_dict={anchor : Xt})
    newX2 = newX - np.mean(newX,axis=0)
    fig = plt.figure()
    ax = plt.axes(projection="3d")
#    ax.scatter3D(newX[:,0][y==0],
#                 newX[:,1][y==0],
#                 newX[:,2][y==0],
#                 c = "r",marker = "o")
    #with mean shift
    ax.scatter3D(rand_jitter(newX[:,0][np.where(yt==1)[0]] - np.mean(newX[:,0][np.where(yt==1)[0]])),
                 rand_jitter(newX[:,1][np.where(yt==1)[0]] - np.mean(newX[:,0][np.where(yt==1)[0]])),
                 rand_jitter(newX[:,2][np.where(yt==1)[0]] - np.mean(newX[:,0][np.where(yt==1)[0]])),
                 c = "g",marker = "o")
    ax.scatter3D(rand_jitter(newX[:,0][np.where(yt==2)[0]] - np.mean(newX[:,0][np.where(yt==2)[0]])),
                 rand_jitter(newX[:,1][np.where(yt==2)[0]] - np.mean(newX[:,0][np.where(yt==2)[0]])),
                 rand_jitter(newX[:,2][np.where(yt==2)[0]] - np.mean(newX[:,0][np.where(yt==2)[0]])),
                 c = "b",marker = "^")
    #without mean shift
    fig = plt.figure()
    ax = plt.axes(projection="3d")
    ax.scatter3D(rand_jitter(newX[:,0][np.where(yt==1)[0]]),
                 rand_jitter(newX[:,1][np.where(yt==1)[0]]),
                 rand_jitter(newX[:,2][np.where(yt==1)[0]]),
                 c = "g",marker = "o")
    ax.scatter3D(rand_jitter(newX[:,0][np.where(yt==2)[0]]),# - np.mean(newX[:,0][np.where(yt==2)[0]])),
                 rand_jitter(newX[:,1][np.where(yt==2)[0]]),# - np.mean(newX[:,0][np.where(yt==2)[0]])),
                 rand_jitter(newX[:,2][np.where(yt==2)[0]]),# - np.mean(newX[:,0][np.where(yt==2)[0]])),
                 c = "b",marker = "^")

The hope is that newX parameter is able to allow me to visualize the newly generated embeddings and i can see the differences visually. Of course I could be doing something wrong. If you could have a look at my code and let me know where my mistake is, that would be awesome. Thanks for your time.

Please let me know if I need to share more details.

AA

Thanks for your nice work for tensorflow triplet loss, I can clearly understand the theory and implement of it. tensorflow-triplet-loss uses Batch all strategy in default to train on MNIST, I find the value of loss vibrate near the margin 0.5.
I meet a problem when I use triplet loss to train another dataset which including 3000 clusters according to content similarity, each cluster has several videos, each video feature size is 512 vector (0-1). The euclidean metrics between two origin data features are between 1 and 10.
I use 3 fully connected layer to learn the embedding, the input batch includes couple in the same cluster, the negative is not. however the triplet loss become to value of margin after several steps, and the embeddings among the anchor, positive, negative are the same. I try to not use hard mining, change lr etc, but can not solve it. Could you help me, Many thanks.

Hello. I am unable to make work hard triplet mining and balanced batches. I think that we had a discussion about it here, but so far I think embedding are always collapsing into one point. I tried many combinations of "margins", "num_classes_per_batch" , "num_images_per_class". But nothing seems to work. Could you please take a look at the code if there is some obvious problem? Noting that with batch_all strategy, it works well.
Thanks,
Tom

def train_input_fn(data_dir, params):
    data_root = pathlib.Path(data_dir)
    all_image_paths = list(data_root.glob('**/*.jpg'))
    all_directories = {'/'.join(str(i).split("/")[:-1]) for i in all_image_paths}
    print("-----")
    print("num of labels: ")
    print(len(all_directories))
    print("-----")
    labels_index = list(i.split("/")[-1] for i in  all_directories)

    # Create the list of datasets creating filenames
    datasets = [tf.data.Dataset.list_files("{}/*.jpg".format(image_dir), shuffle=False) for image_dir in all_directories]

    num_labels = len(all_directories)
    print(datasets)
    num_classes_per_batch = params.num_classes_per_batch
    num_images_per_class = params.num_images_per_class

    def get_label_index(s):
        return labels_index.index(s.numpy().decode("utf-8").split("/")[-2])

    def preprocess_image(image):   
      image = tf.cast(image, tf.float32)
      image = tf.math.divide(image, 255.0)     
      return image

    def load_and_preprocess_image(path):
        image = tf.read_file(path)
        return tf.py_function(preprocess_image, [image], tf.float32), tf.py_function(get_label_index, [path], tf.int64)

    def generator():
        while True:
            # Sample the labels that will compose the batch
            labels = np.random.choice(range(num_labels),
                                      num_classes_per_batch,
                                      replace=False)
            for label in labels:
                for _ in range(num_images_per_class):
                    yield label

    choice_dataset = tf.data.Dataset.from_generator(generator, tf.int64)
    dataset = tf.data.experimental.choose_from_datasets(datasets, choice_dataset)


    dataset = dataset.map(load_and_preprocess_image, num_parallel_calls=tf.data.experimental.AUTOTUNE)

    batch_size = num_classes_per_batch * num_images_per_class
    print("----------------------")
    print(batch_size)
    print("----------------------")
    dataset = dataset.batch(batch_size)
    dataset = dataset.repeat(params.num_epochs)
    dataset = dataset.prefetch(1)

    print(dataset)
    return dataset

Hi, in the first place, thanks for sharing this project!

We noticed that embeddings to mining functions are not L2 normalized in pairwise distances functions, nor in layers before embeddings.

So, is my understanding that L2 normalization is not done as described in the FaceNet paper, correct?

If correct, can you explain why? In my understanding, without L2 normalization, it would be quite difficult to guess a proper margin.

In test_triplet_mask

distinct = (i != j and i != j and j != k) should be
distinct = (i != j and i != k and j != k)

Thanks for your contribution!

When using batch hard, is it correct to say that the hardest negative distance should increase and the hardest positive distance should decrease during training? Is it a good method to monitor the training process?

I tried to export the model using the official guide (using savedmodel with estimators). I tried both approaches:

features = {'x': tf.placeholder(tf.float32, [224, 224, 3], name="x")}
input_fn = tf.estimator.export.build_raw_serving_input_receiver_fn(features, 1)
exported_model_path = estimator.export_savedmodel(os.path.join(args.model_dir, 'exported'), input_fn)

feature_spec = {'x': tf.FixedLenFeature([224, 224, 3], tf.float32)}
input_fn = tf.estimator.export.build_parsing_serving_input_receiver_fn(feature_spec, 1)
exported_model_path = estimator.export_savedmodel(args.model_dir, input_fn)

However I get the following error:

TypeError: Failed to convert object of type <type 'dict'> to Tensor. Contents: {'x': <tf.Tensor 'ParseExample/ParseExample:0' shape=(1, 224, 224, 3) dtype=float32>}. Consider casting elements to a supported type.

I trained a network with the triplet loss over a small dataset with 6 different identities and about 1000 total images. I noticed that the loss goes to 0 after 100 steps. However, the performances are not good: the embeddings seem just noise. I used a small learning rate (1e-4) and a simple network with 6 convolutions (conv+BN+relu) and 3 pooling. This happens both with batch hard and batch all. What could be the problem?

hi,omoindrot
I have encountered some problems，after training for a while,pairwise_dist drops to 0，loss is near the margin and can't go down,visualize the training set and discover that they are all together,I don't know what caused it.
Learning_rate is 0.0001,The network structure is vgg16 and the output dimension is 128,Data augmentation is used because of the small amount of data（random crop and horizontal flip)，This will not happen if I don't use data augmentation.I hope you can reply，thanks!

Hi ! A dead link is on the repo's readme :
When I click on the link "params.json", i arrive on this page: https://github.com/omoindrot/tensorflow-triplet-loss/blob/master/experiments/base_model/params.json

i propose to switch the link with the following (or a new one):
https://github.com/omoindrot/tensorflow-triplet-loss/blob/master/experiments/batch_all/params.json

I hope my english is enough good.
Best regards.

Hi
i use this for face recognition and i need calculate False rejection rate and false accept rate ,can you help me?

Epoch 1/60
97/97 [==============================] - 24s - loss: 1.0072 - mAP: 0.1649 - val_loss: 0.9624 - val_mAP: 0.1296
Epoch 2/60
97/97 [==============================] - 22s - loss: 1.0060 - mAP: 0.1959 - val_loss: 0.9647 - val_mAP: 0.0784
Epoch 3/60
97/97 [==============================] - 21s - loss: 1.0051 - mAP: 0.2268 - val_loss: 0.9851 - val_mAP: 0.1536
Epoch 4/60
97/97 [==============================] - 21s - loss: 1.0051 - mAP: 0.1650 - val_loss: 0.9519 - val_mAP: 0.1808
Epoch 5/60
97/97 [==============================] - 21s - loss: 1.0034 - mAP: 0.2474 - val_loss: 0.9696 - val_mAP: 0.3072
Epoch 6/60
97/97 [==============================] - 21s - loss: 1.0025 - mAP: 0.2577 - val_loss: 0.9895 - val_mAP: 0.3584
Epoch 7/60
97/97 [==============================] - 21s - loss: 1.0044 - mAP: 0.2990 - val_loss: 0.9717 - val_mAP: 0.5392
Epoch 8/60
97/97 [==============================] - 21s - loss: 1.0007 - mAP: 0.2784 - val_loss: 0.9902 - val_mAP: 0.4096

Hi, I found something wrong for the loss value.
The loss value is almost not changing at the time model training.
And the loss value is changed when I change margin value, loss value is approximate of margin

Excuse me, Sir. Thanks for you code of triplet loss, but i don't understand the 'num_channels=32' means in 'params.json'. In general, the channels of a picture are 1 or 3, 'num_channels=32' perplexes me. Could you help me , thanks much.

Hello, this is more question than a problem. I would like to utilize mobilenet v2 with pretrained weights to use features learned by this great model and also to have small footprint since I would like to use it on mobile phone. Is it possible?

I tried to use mobilenet v2 with your code, but I am unable to make it work for now.