Multimodal classifier for disaster response

Abstract

Data obtained from social media has a massive effect on making correct decisions in time-critical situations and natural disasters. Social media content generally consists of messages, images, and videos. In situations of disasters, using multimedia files such as images can significantly help in understanding the damage caused by disasters compared to using text only. In other words, the exact situation and the effect of disaster are better understood using visual data. So far, researchers widely use text datasets for building efficient disaster management systems, and a limited number of studies have focused on using other content, such as images and videos. This is due to the lack of available multimodal datasets. We addressed this limitation in this work by introducing a new Turkish multimodal dataset. This dataset was created by collecting disaster-related Turkish texts and their related images from Twitter. Then, by three evaluators and the majority voting, each sample was annotated as a disaster or not a disaster. Next, multimodal classification studies were carried out with the late fusion technique. The BERT embedding approach and a pre-trained LSTM model are used to classify the text, and a pre-trained CNN model is used for the visual content (images). Overall, concatenating both inputs in a multimodal learning architecture using late fusion achieved an accuracy of 91.87% compared to early fusion, which achieved 86.72%. © 2024, The Author(s), under exclusive license to Springer Nature Switzerland AG.