CogVLM

CogVLM, from Zhipu AI and Tsinghua University (paper November 2023, CogVLM-17B released October 2023), is an open vision-language foundation model that departs from shallow alignment methods. Instead of mapping image features into the language model's input space, it adds a trainable visual expert module inside the attention and FFN layers, bridging the frozen pretrained language model and the image encoder for deep vision-language fusion without hurting NLP performance.

CogVLM-17B took state-of-the-art on 10 classic cross-modal benchmarks including NoCaps, RefCOCO variants, GQA, and ScienceQA, matching or surpassing the much larger PaLI-X 55B. The family later grew into CogVLM2 and the GUI-agent CogAgent. Code and checkpoints are open; verify the current model-card license before commercial use, as terms vary across releases.