This repository contains the reference code for the paper Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering.
Please cite with the following BibTeX:
@article{cocchi2024augmenting,
title={{Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering}},
author={Cocchi, Federico and Moratelli, Nicholas and Cornia, Marcella and Baraldi, Lorenzo and Cucchiara, Rita},
journal={arXiv},
year={2024}
}
Multimodal LLMs (MLLMs) are the natural extension of
large language models to handle multimodal inputs, combining text and image data.
They have recently garnered attention due to their capability to address complex tasks involving both modalities.
However, their effectiveness is limited to the knowledge acquired during training, which restricts their practical utility.
In this work, we introduce a novel method to enhance the adaptability of MLLMs by integrating external knowledge sources.
Our proposed model, Reflective LLaVA (ReflectiVA), utilizes reflective tokens to dynamically determine the need for external knowledge
and predict the relevance of information retrieved from an external database, ultimately enables the MLLM to manage external knowledge
while preserving fluency and performance on tasks where external knowledge is not needed.
To create the conda environment named reflectiva use the following instructions. With this environment you have all the packages to run the code inside this repo.
conda create -n reflectiva python==3.8.16
conda activate reflectiva
pip install -r requirements.txt
You can access the official model weights for the ReflectiVA model on 🤗 Hugging Face.
Coming soon ...
Coming soon ...