• x_train: (8437, 114)
• x_test: (1428, 114)
• y_train: (8437, 3)
• y_test: (1428, 3)

The dataset consists of continious data in the all columns x1-x112. Therefore repeating values seem to be dubious.

Every single row and column contain NaN value.

wo_nan0 = x_train_df.dropna(axis=0)
wo_nan0.shape == (0, 114)

wo_nan1 = x_train_df.dropna(axis=1)
wo_nan1.shape == (8437, 2) # only 'key' and 'date' columns

yrv = y_train_df["y"].value_counts()
yrv[yrv > 1].index

y_train dubious values: [ 10000.0, 2444.444444, 2666.666667, 3111.111111, 2222.222222, 1777.777778, 3777.777778, 2888.888889, 4000.0, 1555.555556, 3555.555556, 2000.0, 1333.333333, 3333.333333, 5000.0, 444.4444444, ]

The above picture of y_train distribution demonstrates an anomality appearing with values 0 and 10000, meanwhile these values are borders of the interval containing all the y_train values. This makes me think that values outside of the interval were somehow aggregated to its borders.

the correlation matrix showed presence of 16 blocks of narrowly correlated columns.

Initially I used several Imputer strategies (SimpleImputer with strategies mean, median, most_frequent) and KNNImputer (n_neighbors=3,5,7). The mean strategy followed by unconverging the PCA algorithm. Nevertheless all others convergated therefore I choose the SimpleImputer with the strategy median.

The PCA algorithm suggested I should choose n_components=41.

imputer.fit(xs)
x_train_filled = imputer.transform(xs)

pca.fit(x_train_filled)
cumsum = np.cumsum(pca.explained_variance_ratio_)

d = np.argmax(cumsum > 0.95) + 1
d

>>> d=41

• A Straightforward Guide to Cleaning and Preparing Data in Python

• Distribution Analysis 1

• Distribution Analysis 2

• Kaggle: Time series with Linear Regression