We demonstrated that the CNNs, including U-Net and Mask R-CNN, are instrumental to provide:

• efficient evaluation of pathological lung lesions.
• detailed characterization of the normal lung histology.
• precise detection and classification for BALF cells.

Overall, these advanced methods allow improved efficiency and quantification of lung cytology and histopathology.

The convolutional neural network architecture used in this project was inspired by U-Net and dual frame U-Net with added transfer learning from pre-trained models in keras (keras-applications).

After training on 14 image pairs, the neural network is able to reach >90% accuracy (dice coefficient) in identifying lung parenchymal region and >60% for severe inflammation in the lung in the validation set. The prediction results on a separate image, including segmentation mask and area stats, was shown below.

Multi-label overlay (blue: parenchyma, red: severe inflammation)

After training and validating (3:1) on 16 whole slide scans, the neural network is able to identify a variety of areas in a normal mouse lung section (equivalent to 10X, cropped from whole slide scan).

Variations of U-Nets were built to perform

• single-class segmentation
  • output: sigmoid
  • loss: dice & binary crossentropy
  • metrics: dice
• multi-class segmentation
  • output: softmax
  • loss: multiclass crossentropy
  • metrics: accuracy

Among them, dual-frame slightly outperform U-Net with single-frame. Although more time consuming, single-class segmentation combined with argmax achieved a better classification results than those done by one multi-class segmentation model, especially for the underrepresented categories.

The best results are listed below:

• single-class segmentation (dice)
  • background: 97%
  • conducting airway: 84%
  • connective tissue: 83%
  • large blood vessel: 78%
  • respiratory airway: 97%
  • small blood vessel: 63%
• multi-class segmentation (accuracy)
  • all six categories: 96%

These methods are helpful for identifing and quantifing various structures or tissue types in the lung and extensible to developmental abnormality or diseased areas.

Non-Parenchymal Region Highlighted Image

Six-Color Segmentation Map

Mask RCNN was developed by Kaiming He, 2017 to simultaneously perform instance segmentation, bounding-box object detection, and keypoint detection.

This project was based on the implementation of matterport with additional functionalities:

• support more convolutional backbone, including vgg and densenet.
• support large images by performing slicing and merging images & detections.
• simulate bronchoalveolar lavage from background & representative cell images for efficient training.
• batch-evaluate models for the mean average precision (mAP).

After training and validating (3:1) on 21 background image with 26 lymphocytes, 95 monocytes, and 22 polymorphonuclear leukocytes, the neural network is able to detect and categorize these cell types in a mouse lung bronchoalveolar lavage fluid (20X objective).

Within one day of training, the accuracy represented by mean average precision has reached 75% for all categories. The accuracy is highest for the monocyte category.

Data credits: Jeanine D'Armiento, Monica Goldklang, Kyle Stearns; Columbia University Medical Center