MM1

MM1 is a family of multimodal large language models published by Apple researchers in March 2024. The models combine image encoders with language models to handle text-rich visual reasoning, captioning, and visual question answering, scaling from 3B dense models up to 30B mixture-of-experts variants.

The paper's ablations showed that the choice of image encoder, visual resolution, and pre-training data mixture matter more than raw parameter count for multimodal performance. MM1-Chat variants demonstrated strong few-shot and chain-of-thought visual reasoning.

Apple published the research findings but did not release weights or code. The work signaled Apple's serious investment in multimodal foundation models ahead of its on-device AI push.