Abstract:
Deep learning-based approaches have demonstrated great results; in handling
the complexities of multimodal data and learning informative repre-sentations
from heterogeneous modalities, these multimodal fusion techniques have
attracted considerable attention for their role in the integration of infor-mation
from different data modalities. In computer aided diagnosis (CAD) sys-tems, the
mixture of different information extracted from heterogeneous modal-ities, like
medical images, clinical data, genetic data, or textual reports, can pro-vide a
more comprehensive and reliable assessment of diseases or conditions. This
review article examines advances in deep multimodal fusion using hetero
geneous neural networks for medical computer-aided-diagnosis (CAD) systems.
We present an overview of the main methodologies and architectures used to
combine information from numerous modalities. This review defines the vari
ous challenges of the fusion approaches, including early fusion, late fusion, and
hybrid fusion. Additionally, we discuss the advantages and limitations of differ
ent neural network architectures used for multimodal data fusion. Finally, we
mention future research directions and open challenges in this area, paving the
way for further advances in deep multimodal fusion using heterogeneous neural
networks.
