Yudi Candra, Roslidar Roslidar, Mulkan Azhary, Fitri Arnia, Kahlil Muchtar
Tuberculosis (TB) remains a major global health challenge that requires diagnostic solutions capable of integrating structural and functional information from the respiratory system. This study proposes a novel lesion-guided multimodal framework that fuses chest X-ray (CXR) and lung sound data for TB detection, addressing the lack of synchronized datasets by leveraging a data-driven pairing mechanism. A YOLOv8-seg model was employed to localize pulmonary lesions, which are then mapped to anatomically corresponding auscultation sites using a rib-guided quadrant extraction algorithm. This process generates a physiologically coherent paired dataset containing 450 CXR images and 2700 lung sound segments. A dual-branch architecture is developed, consisting of a ResNet50 encoder for visual features and a convolutional recurrent neural network (CRNN) encoder for mel-frequency cepstral coefficient (MFCC)–based acoustic representations. Their probabilistic outputs are integrated using an optimized weighted decision-level fusion strategy, with weights derived from unimodal validation performance. Simulation results show that the proposed multimodal framework outperforms both unimodal baselines, achieving 92% accuracy and an AUC of 0.98, compared with 0.95 (CXR-only) and 0.81 (audio-only). Qualitative Grad-CAM and saliency analyses further confirm that the model focuses on interpretable radiographic lesions and crackle-like audio patterns. These findings demonstrate that the proposed lesion-guided multimodal framework offers a physiologically consistent and data-efficient approach for TB screening, highlighting its potential for future clinically deployable diagnostic systems. Copyright © 2026 Yudi Candra et al. Applied Computational Intelligence and Soft Computing published by John Wiley & Sons Ltd.
Department of Electrical and Computer Engineering, Universitas Syiah Kuala, Banda Aceh, 23111, Indonesia; Faculty of Medicine, Universitas Syiah Kuala, Banda Aceh, 23111, Indonesia
Research at a Glance
Register to unlock
Topics & SDG Alignment
Register to unlock
Collaboration
Register to unlock
Author Profile (Selected)
Register to unlock
References Overview
Register to unlock
Journal & Source
Register to unlock
Metadata & Integrity
Register to unlock