broadinstitute / featureaggregation_single_cell Goto Github PK

@EchteRobert asked

last week I believe you mentioned that we should perhaps pick a different dataset to start than the Stain5(?) dataset. Do you remember which one(s) you had in mind instead?

@niranjchandrasekaran had said:

Here are some options
Stain2 and Stain3 - Lots of different experimental conditions; 1 plate per condition (no replicates). These two could be good datasets to burn through while Robert is coming up with his methods.
Stain4, Plate1, Reagent1 and Stain5 - Lots of different conditions; 3 or 4 replicate plates per condition. I guess Stain5 is the best dataset, so it could perhaps be used as a holdout set.

Based on this, I vote for starting with Stain2, @EchteRobert

This first line of experiments will be designed to beat the PR baseline of profiles created with the current aggregation method (which takes the mean). All information about the compound plates used can be found here: https://github.com/jump-cellpainting/2021_Chandrasekaran_submitted.

Creating plate clusters for training and validation splits

In order to split all the Stain2, Stain3, Stain4, and Stain5 (condition C) plates into clusters that are most similar to each other, I created a hierarchical cluster map based on the PC1 loadings of the mean aggregate profiles of these plates. First including all the outliers and then iteratively removing these until I find x clusters which are similar enough so that they can be used for training and validation.

Main takeaways

• The final 7 clusters were chosen based on the largest clusters that can be seen in the final clustermap iteration.
• Cluster number 5, 6, and 7 are the highest quality clusters, i.e. have the highest correlation of the PC1 loadings of the plates.
• We already now that the model beats the baseline on cluster 1 for most plates, except on BR00113818 and BR00112199. However, note that this is actually one of the most diverse diverse clusters of the 7 I have created here.

It is now clear that this feature aggregation model will only serve a certain feature set (meaning a certain dataset line), and is not developed to be able to aggregate any feature set (it is only invariant to the number of cells per well). I will start with creating a model that is able to beat the 'mean aggregation' baselines of the Stain2 batches, and then move forward to Stain3, Stain4, and finally use Stain5 as a final testset.

Because of that it would be ideal if all features across Stain datasets were the same. This is (somewhat) the case across Stain2, Stain3, and Stain4. However, Stain5 has a slightly different cellprofiler pipeline resulting in a different and larger feature set. During preprocessing I found that the pipeline from raw single-cell features to data that can directly be fed to the model, is quite a slow process. This is especially the case when all features are used (in this case 4295 for Stain 2-4 and 5794 for Stain 5). The model inference and training also becomes increasingly slower as the number of features increases. From the initial experiments on CPJUMP1 we saw that not all features are needed to create a better profile than the baseline (#1). This is why I have chosen to select only all common features across Stain 2-5. This has the advantage of speed, both in preprocessing and inference, and compatibility, as no separate model will have to be trained to use Stain5 as the test set.

Assuming that the features across Stain2, Stain3, Stain4, and Stain5 are consistent within each experiment, there are 1324 features which are measured in all of them. The features are well distributed in terms of category: Cells: 441 features, Cytoplasm: 433 features, and Nuclei: 450 features. 1124 of them are decently uncorrelated (<abs(0.5) Pearsson correlation) [one plate tested]. From hereon these are the features that will be used to train the model.

This issue is used to test more general aspects of model development not directly related to, but likely still influenced by, the dataset or model hyperparameters that are used at that time.

Here I trained a model on all data available from batch 1 in the LINCS dataset, which can be found like this: aws s3 ls s3://cellpainting-gallery/cpg0004-lincs/broad/workspace/backend/2016_04_01_a549_48hr_batch1/

The model uses 1745 features, because of an issue with 10 plates (broadinstitute/lincs-cell-painting#88 (comment)). In total, I trained the model on 136 plates, 5965 wells, including 1228 unique compounds using the 10 uM dose point. During preprocessing I removed 1587 wells due to missing MoA or compound name (pert_iname) annotation. I used the following hyperparameters:

I assess the model on the 10 uM dose point using replicate and MoA prediction and similarly on the 3.33 uM dose, which is considered the test set.

Results

• The model significantly improves upon the average baseline for replicate and MoA prediction for both the 10 and 3.33 uM dose points.
• It improves the mAP by 60% and 30% for the 10 uM dose point (training) and 3.33 uM dose point (test) data, respectively.
• I could have trained the model a bit longer, e.g. 150 epochs, as the validation mAP and loss did not fully converge yet.

Loss curves

In this issue I will post all the results on Stain5 with various models and with various evaluation metrics. I first trained a model on 5 plates from Stain2, Stain3, and Stain4, for a total of 15 training plates. I then use this model to run inference directly on Stain5. After that I fine-tune the model (transfer learning) using 3 plates from Stain5 and 10, 20, 40, and 80% of the available compounds for training/fine-tuning. I also tested this with 1 plate and the same fractions, but multiple plates are required for generalizing to the feature patterns of Stain5.
I fine-tune the model by training for 100 epochs and then taking the best validation mAP model. There are perhaps better ways to do this, but this is a first proof-of-concept experiment.

Main takeaways

• As expected, the model does not directly translate to Stain5. This is not due plate/batch effects as those are much smaller. Instead, it is due to different experimental conditions which cause a shift in the features distributions, so that the model is no longer able to correctly aggregated the single cell data. See #7 (comment) for the hierarchical cluster map showing that Stain2, Stain3, and Stain4 are in a different cluster than Stain5 altogether.
• Secondly, fine-tuning the model does increase the performance on plates that are similar to the training data (i.e. using confocal plates will increase performance on confocal plates and using widefield will increase performance on widefield, not both at the same time). However, in order to generalize to unseen compounds I need to use more than 3 plates. This is the same issue I was having before with training the models: using 1 training plate does not generalize to unseen plates and using 3 training plates does not generalize to unseen compounds. So I probably need to use 5 or more plates to generalize to this type of data.

Results

In this issue, I will outline experiments that may be useful in the future to further investigate the inner-workings of the model, but for which I currently do not have the time (or have low priority).

Two cluster training data (T: S3+S4)

Some final tweaks to training the model will be made in this issue. All of these tweaks will be made with Stain2, Stain3, and Stain4 in mind at the same time, in stead of 1 at a time. The first model is trained on 3 plates from Stain3 and Stain4 at the same time and evaluated on Stain2, Stain3, and Stain4.

Main takeaways

• It's possible to generalize to clusters outside of the trained clusters by using training data from at least two clusters at the same time.
• This actually also improves overall performance on validation mAP for plates within the training cluster. It is slightly worse than the best model trained on the Stain2 cluster specifically.

To test the generalization of the model trained on Stain2, I will now evaluate it on Stain3. Based on the results, further advancements will be made by training on plates of Stain3 (and possibly then evaluating on Stain2 in turn).

Stain 3 consists of 17 plates which were divided into 4 "batches" which are defined based on the analysis pipeline used.

• Standard
• Multiplane
• HighExp
• Bin1

To analyze the relations between the different plates in Stain3, I calculated the correlation between the PC1 loadings of the mean aggregated profiles of every plate. The BR00115130 plate stands out. This agrees with the findings in https://github.com/jump-cellpainting/pilot-analysis/issues/20 where this plate achieved the lowest PR, likely due to a lot wells having evaporated or something else being messed up. Another plate that stand out is BR00115132. This plate contains a 4-fold dilution of all dyes. BR00115130 will be left out of analysis altogether. BR00115132 will not be left out, although it will not be used as a training plate.

To test the generalization of the model trained on Stain3 (and tested on Stain2), I will now evaluate it on Stain4. Based on the results, further advancements will be made by training on plates of Stain4 (and then evaluating on Stain2 and Stain3 in turn).

Stain 4 consists of 30 plates which were divided into 5 batches, each with different staining conditions.

• Baseline staining conditions used in Stain3 (Stain2)
• Baseline staining conditions used in Stain3 with 2-fold dilution of all dyes (Stain2_2)
• Baseline staining conditions used in Stain3 with 2-fold dilution of Phalloidin (Stain2_Phalloidin_2)
• Baseline staining conditions used in Stain3 with 2-fold dilution of Phalloidin and ConA (Stain2_Phalloidin_ConA_2)
• Bray et al. staining conditions (Bray)

Apart from that, standard exposure vs. high exposure and Binning 1 vs. Binning 2 comparisons were also made.

To analyze the relations between the different plates in Stain4, I calculated the correlation between the PC1 loadings of the mean aggregated profiles of every plate. I only included the plates that were similar enough to form a large cluster.

Click here for clusters!

Click here for cells per well per plate!

Only the plates I have downloaded so far are available, but will still give a good indication of this dataset

LINCS contains 6 dose points: 0.04 µM, 0.12 µM, 0.37 µM, 1.11 µM, 3.33 µM, and 10 µM. For my experiments, I will use the highest dose (10 µM) as the training set and the validation set. The model is trained to create profiles that attract replicate compound profiles and repel non-replicate compound profiles. It is then validated by evaluating the ability of these profiles to predict MoAs (or find sister compounds). Finally, the model will be tested on the 3.3 µM dose point data as a hold-out set. This data should look significantly different from the training and validation data.

I will follow the same data exclusion protocol as Michael did in his research:

“The evaluation metrics all require the number of compounds in an MOA to be at least two (MOA class size of at least two). A compound that does not have other compounds annotated with the same MOA cannot have a precision value and does not affect Enrichment and Hit@k. Additionally, there is no use for compounds with unknown MOA labels, because I can not apply any metrics. Finally, I used only a single dose per compound to make experimental iterations practical. I picked the 10uM dose point because we found that this dose produces the strongest phenotypes across compounds without reducing the ability to group MOAs [42]. These three requirements lead to a subset of the LINCS data in which I 1) only keep the highest dose of each perturbation, 2) delete all compounds that have unknown MOA labels, and 3) drop all compounds with MOA sizes smaller than two. All further processing and experiment are based on this subset of data. This subset process is documented in my repository.”

Which should result in similar data numbers:

“The resulting (sub-selected) dataset comprises 8,818 wells spanning over 136 plates, five different batches, and 18.2 million cells. Each of the 1,144 perturbations (compounds) have five technical replicates - each on a separate plate but in the same position on the plate (same well position). Four of these technical replicate wells were produced in the same batch. This is highly relevant for training and technical artifacts, because I expect cells in the same well and same batch to have similar technical artifacts, but the corresponding wells from a different batch may look very different from one another. Each of the 136 plates has 24 negative controls (DMSO) distributed across the plate (3264 DMSOs in total). The 1,144 compounds can be categorized into 235 MOAs, and every MOA has at least two compounds (MOA class size > 1). Some MOAs, such as phosphodiesterase inhibitor, have an MOA class size of 35, while most other MOAs are only found for two compounds.”

I have put some relevant quotes from Michael's work and the LINCS manuscript here: https://docs.google.com/document/d/1z2U5o91vzBwB-4xtryYn5d3kSWJi8ZSerE_MAzdLT_0/edit#heading=h.sbi6l2r6p5ec

The first results here are generated by the lastly presented model (BS64) in #1. Note that Stain2 confocal data contains 1433 features instead of the 1938 in CPJUMP1. So some features are randomly selected and repeated to feed the data to the model.

Main takeaways

As expected, the model does not directly translate to the Stain2 dataset. I expect the cause to be two-fold:

1. Stain2 has measured less and perhaps even different features, the model has likely not learned to interpret general feature distributions yet but has overfit to the features present in CPJUMP1.
2. Stain2 contains different compounds, which contain different feature distributions than the compounds found in CPJUMP1.

Next steps: find a way for the model to generalize to different features and compounds.

Model trained on CPJUMP1 compound data on Stain2_Batch2_Binned PR // BENCHMARK: 0.6 PS

Model trained on CPJUMP1 compound data on Stain2_Batch2_Confocal PR // BENCHMARK: 0.533