VS-BIM: a cognitive map-driven framework enhancing MLLM for automatic safety inspection in construction

The rise of Multimodal Large Language Models (MLLMs) offers new potential for automated construction safety inspection. However, current discriminative vision-language alignment approaches struggle with spatial understanding and complex reasoning, limiting proactive risk detection. To address this, we propose VS-BIM, a generative zero-shot framework driven by cognitive maps. We reconstruct 3D scenes from panoramic video and Building Information Models (BIM), and align visual and semantic information into a queryable 3D cognitive map that serves as the spatial working memory for MLLMs. VS-BIM includes three modules: (1) panoramic video and BIM-based 3D reconstruction; (2) pixel-level semantic embedding using segmentation and pretrained vision-language models; and (3) cognitive map generation via multi-view consistency constraints. We evaluate VS-BIM using public datasets and custom construction environments. In addition, we introduce the CER-QA benchmark to test its performance in configuration recognition, spatial reasoning, spatiotemporal inference, and risk assessment. Results demonstrate VS-BIM excels in 3D object detection, achieves near-human spatial reasoning, and surpasses human averages in spatial estimation. Overall, the cognitive-map-based zero-shot paradigm endows MLLMs with stronger spatial reasoning while greatly reducing the need for large-scale annotation and retraining. By adapting seamlessly to complex, dynamic sites, it shifts safety inspection from passive checks to proactive risk prediction, enabling more efficient construction-site management.

Text 122268.pdf - Published Version Restricted to Repository staff only Download (4MB) Official URL or Download Paper: https://linkinghub.elsevier.com/retrieve/pii/S1474...

Actions (login required)

View Item