InstructBLIP

InstructBLIP is a vision-language instruction-tuning framework from Salesforce Research, released in May 2023 and presented at NeurIPS 2023. It builds on the frozen BLIP-2 architecture and adds an instruction-aware Q-Former: the instruction text is fed into the Q-Former alongside its learned queries, so the visual features extracted from an image are conditioned on the task at hand.

The team converted 26 public datasets into instruction format, training on 13 held-in sets and evaluating zero-shot on 13 held-out sets. The resulting models set state-of-the-art zero-shot results across all held-out datasets, substantially outperforming BLIP-2 and the much larger Flamingo, and also lead when finetuned on downstream tasks such as ScienceQA.

All InstructBLIP models were open-sourced through the Salesforce LAVIS library, with Flan-T5 and Vicuna LLM backbones. It remains a landmark in instruction tuning for multimodal models.