Example of Embedding Learning about unsupervisedscalablerepresentationlearningtimeseries HOT 9 CLOSED

we do already provide all the software and pre-trained models. maybe you can try it on your dataset and let us know how it performs?

from unsupervisedscalablerepresentationlearningtimeseries.

@White-Link Many thanks for your great support so far!

I now move on to try your method on time series with unequal lengths, padding them with NaNs. I might come back with some questions though :-)

from unsupervisedscalablerepresentationlearningtimeseries.

@martinjaggi Thanks for your reply.

Since there are multiple, potentially related series, I assume I need to adjust the number of input channels of the time series, as stated in Section 5.1.2 of the paper

To complement our evaluation on the UCR archive which exclusively contains univariate series, we
evaluate our method on multivariate time series. This can be done by simply changing the number of
input filters of the first convolutional layer of the proposed encoder.

Since I am using 133 different series, I assume I need to set in_channels = 133? But I get the following error

hyperparameters = {
        "compared_length": 441,          # Maximum length of randomly chosen time series
        "nb_random_samples": 10,         # Number of randomly chosen intervals to select the final negative sample in the loss
        "penalty": None,                 # Penalty term for the SVM classifier. If None and if the number of samples is high enough, performs a hyperparameter search to find a suitable constant.
        "negative_penalty": 1,           # Multiplicative coefficient for the negative sample loss
        "batch_size": 10,                # Batch size used during the training of the encoder 
        "nb_steps": 200,                 # Number of optimization steps to perform for the training of the encoder
        "lr": 0.001,                     # learning rate of the Adam optimizer used to train the encoder    
        "early_stopping": None,          # Enables, if not None, early stopping heuristic for the training of the representations, based on the final score. Representations are still learned unsupervisedly in this case. If the number of samples per class is no more than 10, disables this heuristic. If not None, accepts an integer representing the patience of the early stopping strategy.
        "channels": 40,                  # Number of channels manipulated in the causal CNN
        "depth": 10,                     # Depth of the causal CNN
        "reduced_size": 80,              # Fixed length to which the output time series of the causal CNN is reduced
        "out_channels": 160,             # Number of features in the final output
        "kernel_size": 3,                # Kernel size of the applied non-residual convolutions 
        "in_channels": np.shape(target_df)[0],                # Number of input channels of the time series
        "cuda": cuda,
        "gpu": gpu
}

encoder.fit_encoder(target_df, save_memory=True, verbose=True)

RuntimeError: Given groups=1, weight of size [40, 133, 3], expected input[10, 1, 429] to have 133 channels, but got 1 channels instead

where type(target_df) = numpy.ndarray. I am not sure how to proceed.

Appreciate your help!

from unsupervisedscalablerepresentationlearningtimeseries.

Hi! Setting in_channels to 133 means that your input time series should be multidimensional, with a vector of size 133 for each time step. Therefore, the input shape should be (nb_time_series, 133, max_length), which does not seem to be your case.

Note that, in the case of time series of unequal lengths, we require to pad the series with NaNs up to max_length.

from unsupervisedscalablerepresentationlearningtimeseries.

@White-Link Many thanks for the clarification! I now have the right input shape, after realizing that in_channels = 1. For clarification: I am using a dataset of 133 time series, each with 441 months of observations, which looks as folllows:

Each column is a separate time series and each row indicating time. So dataset.shape = (133, 1, 441).

@White-Link I guess this is what you call a univariate time series data set in your paper, composed of several time series, right?

I ran the encoder with the following settings:

CausalCNNEncoderClassifier(batch_size=10, channels=40, compared_length=441,
                           cuda=True, depth=10, kernel_size=3, nb_steps=200,
                           in_channels=1,out_channels=160, penalty=None, reduced_size=80)

I want to extract, for each of the 133 different time series, the embeddings that I can then use to find meaningful clusters of similar time series. Hence, the output shape should be embedding_matrix.shape = (133, 160), for 133 time series, each with an embedding size of 160, as specified by out_channels=160. It is not necessary to have the embeddings on a yearly, or bi-annual granularity, as demonstrated in your HouseholdPowerConsumption Example.

@White-Link: I am not quite sure how to use encoder.encode_window(dataset, window = ???). What would be a proper choice for window?

Also, I am using these settings

hyperparameters = {
        "compared_length": 441,           # Maximum length of randomly chosen time series
        "nb_random_samples": 10,                           # Number of randomly chosen intervals to select the final negative sample in the loss
        "penalty": None,                                   # Penalty term for the SVM classifier. If None and if the number of samples is high enough, performs a hyperparameter search to find a suitable constant.
        "negative_penalty": 1,                             # Multiplicative coefficient for the negative sample loss
        "batch_size": 10,                                  # Batch size used during the training of the encoder 
        "nb_steps": 200,                                   # Number of optimization steps to perform for the training of the encoder
        "lr": 0.001,                                       # learning rate of the Adam optimizer used to train the encoder    
        "early_stopping": None,                            # Enables, if not None, early stopping heuristic for the training of the representations, based on the final score. Representations are still learned unsupervisedly in this case. If the number of samples per class is no more than 10, disables this heuristic. If not None, accepts an integer representing the patience of the early stopping strategy.
        "channels": 40,                                    # Number of channels manipulated in the causal CNN
        "depth": 10,                                       # Depth of the causal CNN
        "reduced_size": 80,                                # Fixed length to which the output time series of the causal CNN is reduced
        "out_channels": 160,                               # Number of features in the final output
        "kernel_size": 3,                                  # Kernel size of the applied non-residual convolutions 
        "in_channels": 1,               # Number of input channels of the time series
        "cuda": cuda,
        "gpu": gpu
}

However, training stops after 15 iterations, even though nb_steps: 200 and early_stopping=None

@White-Link: Any suggestions why that happens?

Really appreciate your support!

from unsupervisedscalablerepresentationlearningtimeseries.

No worries! I would say that you are indeed correctly using the code, with 133 univariate time series. About your other questions:

• Regarding the encoding window, it depends on what you would like to compare. If you wish to compare time series over their whole length, you should instead use the encode method to compute their associated representations. If you wish to compare shorter intervals, you could indeed use encode_window to encode any subseries of size window in each of your input time series. For example, if you are interested in decade-long series, you can set window=120.
• Training should perform exactly 200 optimization steps. Did you mean that it went through 15 epochs, which would correspond to around 200 iterations with a batch size equal to 10? In this case, the code behaves correctly.

from unsupervisedscalablerepresentationlearningtimeseries.

Many thanks for the detailed explanation! I am now using encode to extract the embeddings! I didn't know that nb_steps / (n_series / batch_size) = n_epochs. In that case, 15 epochs is just fine. I was assuming that nb_steps referrs to the number of epochs.

• Do you have any experience on how to select compared_length? Currently I am setting it to the full length of each series (all have the same length). Would you recommend to select a value smaller than this?
• Also, I was wondering how the size of the embeddings out_channels influences the results?

So far, the results look like this (showing a subset of series only)

from unsupervisedscalablerepresentationlearningtimeseries.

• compared_length is meant to restrict the length of the input time series of the encoder. Therefore, if you are interested in comparing your time series over their full lengths, I would suggest keeping as is.
• We unfortunately did not study the effect of out_channels since we had to choose a single one for all time series of each archive. We only chose high enough to make sure that it could be suitable for all considered time series. It is possible that increasing could increase the performance of the method.

We are always interested in possible applications of our method, so I hope that it will help you in your clustering task! I may also add that increasing the number of optimization steps may help to learn better representations as well.

from unsupervisedscalablerepresentationlearningtimeseries.

Closing this issue for now, please let us know if you have any other question!

from unsupervisedscalablerepresentationlearningtimeseries.