Implementación de un prototipo de consulta de vehículos en videos urbanos Integrando Modelos Multimodales y RAG

Abstract

Abstract: A prototype was developed and implemented that allows natural language queries of vehicular events within urban videos captured from drones, integrating computer vision detection and augmented generation through retrieval. The system detects vehicles with YOLO, records class and location, attaches frames as evidence, and converts events into embeddings that are stored in a vector database to retrieve and rank relevant information. Consequently, with the context retrieved, a multimodal language model generates explanatory and substantiated responses, reducing hallucinations through groundedness and the use of visual evidence. For this purpose, an XP methodology, a large dataset of labeled aerial images, and five test videos in different resolutions and lighting conditions were used. The evaluation showed that YOLOv8 achieved the best balance between accuracy and coverage compared to YOLOv5 and YOLOv11, validating the feasibility of the approach for near-real-time urban analysis.