adapt dataloader to also run on primitives dataset.

• Filter for max length of descriptor
• save processed data
• save vocabulary
  reload processed data and vocabulary --> move to dataloader

Currently exception gets caught an image dropped to keep drawing and program alive.
Line of problems:

ax.voxels(data_crop, facecolors=colors_reshape, edgecolors=edges_reshape)

Error:

#91378ff' is neither a valid single color nor a color sequence consisting of single character color specifiers such as 'rgb'. Note also that the latter is deprecated.

Would be easier to also get the model ID for making it possible to as example show the shape corresponding to the text2shape verification

• split data
• generate batches (shuffled)
• use preprocessed data
• desc2vec and vice versa

generate for triplet loss:

• Anchor
• Positive as vector
• Negative as vector

combine:

• txt encoder
• shape encoder

handles:

• triplets to tensor
• loss function
• pdate routine

Implement whole loss function:

• Cross-Modal Associations
• Instance-Level Associations
• Multimodal Metric Learning
• Full Multimodal Loss

Set up something to make visualization of voxel possible.
Useful at the end of project.

Verify if training was successful.
In this case just check for the text encoder (NNs?).

So far all data are stored within RetriavelLoader class as tensors.
Memory consumption for whole data set (shapes and captions) is about 9GB.

TripletLoader during training (list of class triplets) is about 4 GB.
Easiest solution:

• adapt NNs for using triplets itself instead of list of tensors
• provides retrievals to use them during training as metric

Implement architecture for shape encoder

query text and find nearest shape neighbors

use config file to set up hyperparameter, directories ...

Solution so far is not pretty good.
Focusing on cross modal retrievals?

Implement architecture for text encoder.

Metric learning approach

check which nearest neighbors are found

reduce dimension of embedding and plot results

How could we select "smart" batches.

So far only s2t triplets are generated. Adapt whole pipeline to make also t2s triplets possible.

• adapt data loader
• generate triplets with description as anchor and pos/neg matching shapes
• adapt loss function to figure out what kind of triplet
• how to randomly generate different batch

maybe use normalized discounted cumulative gain