As we are using TF dataset objects to stream data, we need to reconsider how to implement normalisation of input to the VAE

When executing train.slurm, the training fails on the following line of tile.py:

    # add the tiles overlapping with the labels to the test set
    mask = tiles.intersects(labels.unary_union)       #<- Fails on this line
    test_set = test_set.append(tiles[mask])

My hypothesis is that in the label data, some polylines are added, which is causing the failure of labels.unary_union . My temporal solution is to grow the labels with a very small buffer:

    # add the tiles overlapping with the labels to the test set
    labels['geometry'] = labels.buffer(0.0000001) # Add very small buffer to get rid of Polylines
    mask = tiles.intersects(labels.unary_union)
    test_set = test_set.append(tiles[mask])

In this way, all labels are converted into polygon/multipolygon, and there is no failure.

We need to think of a solution to this. Do we want to adapt to both polygon and polyline in the labels? Or there should be a check to enforce all labels to be polygon?

@sonjageorgievska @rogerkuou @maaikeizb @fnattino please check out and comment/accept the schematic diagram in the merging branch

Right now, training on GPU only takes place on a single card (2 are available on Cartesius node). Implement multi-card training as in: https://keras.io/guides/distributed_training/

We should rename the preprocessing repository, and/or separate the network from the ingestion and (pre-)processing