Implementación de un sistema de preguntasrespuestas usando representaciones vectoriales densas y modelos de lenguaje pre-entrenados. Caso de aplicación en un dominio cerrado

Abstract

Abstract: In the current scenario, Question-answering (Q&A) systems have established themselves as essential tools in various aspects of contemporary society. This project has been oriented towards the design and development of a QA system specialized in the contamination domain. The solution stands out for its unique ability to be adapted to different user-determined domains. Using an iterative approach, the project has been developed over three iterations, successfully addressing, and overcoming key challenges and achieving remarkable advances in terms of operational efficiency and quality of responses. The implementation of specialized technologies has been a crucial component for the success of the project: 1) The Haystack framework was used for effective text segmentation, i.e. for partitioning Wikipedia content without compromising its semantic coherence. 2) Milvus, as a vector database, has proven its efficiency in managing large volumes of vector data, contributing significantly to the optimization of response times. 3) In the area of pre-trained model loading, the use of Hugging Face has been essential; this platform has facilitated the agile integration of language models, enriching the system's ability to understand and generate natural language responses. The successful deployment of the system has culminated in the delivery of accurate and relevant responses in the specific domain defined by the user. Overall, this project not only highlights the importance of domain customization in QA systems, but also underscores the need for strategic integration of specialized tools. The ability to adapt to specific domains and the careful selection of technologies have been key elements in achieving a system that is agile, efficient, and capable of providing relevant answers in the desired context.