Background: 

Research suggests that treatment of multiple brain metastases (BMs) with stereotactic radiosurgery shows improvement when metastases are detected early, providing a case for BM detection capabilities on small lesions.

Purpose: 

To demonstrate automatic detection of BM on three MRI datasets using a deep learning-based approach. To improve the performance of the network is iteratively co-trained with datasets from different domains. A systematic approach is proposed to prevent catastrophic forgetting during co-training.

Study type: Retrospective.

Population: 

A total of 156 patients (105 ground truth and 51 pseudo labels) with 1502 BM (BrainMetShare); 121 patients with 722 BM (local); 400 patients with 447 primary gliomas (BrATS). Training/pseudo labels/validation data were distributed 84/51/21 (BrainMetShare). Training/validation data were split: 121/23 (local) and 375/25 (BrATS).

Field strength/sequence: 

A 5 T and 3 T/T1 spin-echo postcontrast (T1-gradient echo) (BrainMetShare), 3 T/T1 magnetization prepared rapid acquisition gradient echo postcontrast (T1-MPRAGE) (local), 0.5 T, 1 T, and 1.16 T/T1-weighted-fluid-attenuated inversion recovery (T1-FLAIR) (BrATS).

Assessment: 

The ground truth was manually segmented by two (BrainMetShare) and four (BrATS) radiologists and manually annotated by one (local) radiologist. Confidence and volume based domain adaptation (CAVEAT) method of co-training the three datasets on a 3D nonlocal convolutional neural network (CNN) architecture was implemented to detect BM.

Statistical tests: 

The performance was evaluated using sensitivity and false positive rates per patient (FP/patient) and free receiver operating characteristic (FROC) analysis at seven predefined (1/8, 1/4, 1/2, 1, 2, 4, and 8) FPs per scan.

Results: 

The sensitivity and FP/patient from a held-out set registered 0.811 at 2.952 FP/patient (BrainMetShare), 0.74 at 3.130 (local), and 0.723 at 2.240 (BrATS) using the CAVEAT approach with lesions as small as 1 mm being detected.

Data conclusion: 

Improved sensitivities at lower FP can be achieved by co-training datasets via the CAVEAT paradigm to address the problem of data sparsity.

1. Baid, U., Ghodasara, S., Mohan, S., Bilello, M., Calabrese, E., Colen, R., ... & Bakas, S. (2021). The RSNA-ASNR-MICCAI BraTS 2021 benchmark on brain tumor segmentation and radiogenomic classification (arXiv:2107.02314). arXiv. https://doi.org/10.48550/arxiv.2107.02314
2. Bakas, S., Akbari, H., Sotiras, A., Bilello, M., Rozycki, M., Kirby, J. S., ... & Davatzikos, C. (2017). Advancing the cancer genome atlas glioma MRI collections with expert segmentation labels and radiomic features. Scientific Data, 4(1), Article 170117. https://doi.org/10.1038/sdata.2017.117
3. Bengio, Y., Louradour, J., Collobert, R., & Weston, J. (2009). Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning (pp. 41–48). Association for Computing Machinery. https://doi.org/10.1145/1553374.1553380
4. Cao, Y., Xu, J., Lin, S., Wei, F., & Hu, H. (2019). GCNet: Non-local networks meet squeeze-excitation networks and beyond. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) (pp. 1971–1980). IEEE. https://doi.org/10.1109/ICCVW.2019.00246
5. Clèrigues, A., Valverde, S., Bernal, J., Freixenet, J., Oliver, A., & Lladó, X. (2019). Acute ischemic stroke lesion core segmentation in CT perfusion images using fully convolutional neural networks. Computers in Biology and Medicine, 115, Article 103487. https://doi.org/10.1016/j.compbiomed.2019.103487
6. De Fauw, J., Ledsam, J. R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., ... & Suleyman, M. (2018). Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature Medicine, 24(9), 1342–1350. https://doi.org/10.1038/s41591-018-0107-6
7. Dikici, E., Ryu, J. L., Demirer, M., Bigelow, M., White, R. D., Slone, H., ... & Prevedello, L. M. (2019). Automated brain metastases detection framework for T1-weighted contrast-enhanced 3D MRI. IEEE Journal of Biomedical and Health Informatics, 24(10), 2883–2893. https://doi.org/10.1109/JBHI.2019.2957097
8. Eisenhauer, E. A., Therasse, P., Bogaerts, J., Schwartz, L. H., Sargent, D., Ford, R., ... & Verweij, J. (2009). New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1). European Journal of Cancer, 45(2), 228–247. https://doi.org/10.1016/j.ejca.2008.10.026
9. Ellenbogen, R. G., Abdulrauf, S. I., & Sekhar, L. N. (2012). Principles of neurological surgery (3rd ed.). Elsevier Health Sciences.
10. Grøvik, E., Yi, D., Iv, M., Tong, E., Rubin, D., & Zaharchuk, G. (2020). Deep learning enables automatic detection and segmentation of brain metastases on multisequence MRI. Journal of Magnetic Resonance Imaging, 51(1), 175–182. https://doi.org/10.1002/jmri.26761
11. Grøvik, E., Yi, D., Iv, M., Tong, E., Rubin, D. L., & Zaharchuk, G. (2021). Handling missing MRI sequences in deep learning segmentation of brain metastases: A multicenter study. npj Digital Medicine, 4, Article 41. https://doi.org/10.1038/s41746-021-00411-8
12. Hamwood, J., Alonso-Caneiro, D., Read, S. A., Vincent, S. J., & Collins, M. J. (2018). Effect of patch size and network architecture on a convolutional neural network approach for automatic segmentation of OCT retinal layers. Biomedical Optics Express, 9(7), 3049–3066. https://doi.org/10.1364/BOE.9.003049
13. Han, C., Murao, K., Noguchi, T., Suzuki, Y., Nakayama, R., Satoh, S., & Racoceanu, D. (2019). Learning more with less: Conditional PGGAN-based data augmentation for brain metastases detection using highly-rough annotation on MR images. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management (pp. 119–127). Association for Computing Machinery. https://doi.org/10.1145/3357384.3357890
14. He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 770–778). IEEE. https://doi.org/10.1109/CVPR.2016.90
15. Hu, J., Shen, L., & Sun, G. (2018). Squeeze-and-excitation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 7132–7141). IEEE. https://doi.org/10.1109/CVPR.2018.00745
16. Isensee, F., Jaeger, P. F., Kohl, S. A. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: A self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 18(2), 203–211. https://doi.org/10.1038/s41592-020-01008-z
17. Jünger, S. T., Hoyer, U. C. I., Schaufler, D., Keller, J., Kocher, M., Loges, W., ... & Pennig, L. (2021). Fully automated MR detection and segmentation of brain metastases in non-small cell lung cancer using deep learning. Journal of Magnetic Resonance Imaging, 54(5), 1608–1622. https://doi.org/10.1002/jmri.27740
18. Liao, F., Liang, M., Li, Z., Hu, X., & Song, S. (2019). Evaluate the malignancy of pulmonary nodules using the 3D deep leaky noisy-or network. IEEE Transactions on Neural Networks and Learning Systems, 30(11), 3484–3495. https://doi.org/10.1109/TNNLS.2019.2892409
19. Ma, L., Fogh, S., Gupta, N., Hwang, A., Sneed, P., & Larson, D. (2012). A technique for achieving submillimeter accuracy of volume-staged stereotactic radiosurgery. Journal of Radiosurgery and SBRT, 2(1), 11–18.
20. Menze, B. H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., ... & Van Leemput, K. (2015). The multimodal brain tumor image segmentation benchmark (BRATS). IEEE Transactions on Medical Imaging, 34(10), 1993–2024. https://doi.org/10.1109/TMI.2014.2377694
21. Mittal, S., Galesso, S., & Brox, T. (2021). Essentials for class incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (pp. 3535–3544). IEEE. https://doi.org/10.1109/CVPRW53098.2021.00392
22. Ren, S., He, K., Girshick, R., & Sun, J. (2015). Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 91–99. https://doi.org/10.48550/arXiv.1506.01497
23. Setio, A. A. A., Traverso, A., de Bel, T., Berens, M. S. N., Bogaard, C., Cerello, P., ... & Jacobs, C. (2017). Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: The LUNA16 challenge. Medical Image Analysis, 42, 1–13. https://doi.org/10.1016/j.media.2017.06.015
24. Strohm, L., Hehakaya, C., Ranschaert, E. R., Boon, W. P. C., & Moors, E. H. M. (2020). Implementation of artificial intelligence (AI) applications in radiology: Hindering and facilitating factors. European Radiology, 30(10), 5525–5532. https://doi.org/10.1007/s00330-020-06946-9
25. Tao, A., Sapra, K., & Catanzaro, B. (2020). Hierarchical multi-scale attention for semantic segmentation (arXiv:2005.10821). arXiv. https://doi.org/10.48550/arXiv.2005.10821
26. Thakur, S., Doshi, J., Pati, S., Rathore, S., Sako, C., Erus, G., ... & Davatzikos, C. (2020). Brain extraction on MRI scans in presence of diffuse glioma: Multi-institutional performance evaluation of deep learning methods and robust modality-agnostic training. NeuroImage, 220, Article 117081. https://doi.org/10.1016/j.neuroimage.2020.117081
27. Tong, E., McCullagh, K. L., & Iv, M. (2020). Advanced imaging of brain metastases: From augmenting visualization and improving diagnosis to evaluating treatment response. Frontiers in Neurology, 11, Article 270. https://doi.org/10.3389/fneur.2020.00270
28. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008.
29. Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 7794–7803). IEEE. https://doi.org/10.1109/CVPR.2018.00813
30. Wolf, A., Kvint, S., Chachoua, A., Heinemann, M., Saraf, S., Shiau, C. Y., ... & Kondziolka, D. (2017). Toward the complete control of brain metastases using surveillance screening and stereotactic radiosurgery. Journal of Neurosurgery, 128(1), 23–31. https://doi.org/10.3171/2016.10.JNS16147
31. Xue, J., Wang, B., Ming, Y., Liu, X., Jiang, Z., Wang, C., ... & Wang, J. (2020). Deep learning-based detection and segmentation-assisted management of brain metastases. Neuro-Oncology, 22(4), 505–514. https://doi.org/10.1093/neuonc/noz234
32. Xie, Q., Luong, M. T., Hovy, E., & Le, Q. V. (2020). Self-training with noisy student improves ImageNet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 10687–10698). IEEE. https://doi.org/10.1109/CVPR42600.2020.01070
33. Zhang, M., Young, G. S., Chen, H., Rigolo, L., Barbieri, M., McDuff, D., ... & Bi, W. L. (2020). Deep-learning detection of cancer metastases to the brain on MRI. Journal of Magnetic Resonance Imaging, 52(4), 1227–1236. https://doi.org/10.1002/jmri.27129
34. Zhou, Z., Sanders, J. W., Johnson, J. M., Gidley, P. W., Blanchard, P., Sio, T. T., ... & Aristophanous, M. (2020). Computer-aided detection of brain metastases in T1-weighted MRI for stereotactic radiosurgery using deep learning single-shot detectors. Radiology, 295(2), 407–415. https://doi.org/10.1148/radiol.2020191479
35. Zhou, Z., Sodha, V., Pang, J., Gotway, M. B., & Liang, J. (2021). Models Genesis. Medical Image Analysis, 67, Article 101840. https://doi.org/10.1016/j.media.2020.101840
36. Zhu, Y., Zhang, Z., Wu, C., Zhang, Z., He, T., Zhang, Z., ... & Lin, J. (2023). Improving semantic segmentation via self-training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4), 5216–5228. https://doi.org/10.1109/TPAMI.2021.3138337