Falcon Mamba 7B

Falcon Mamba 7B is the Technology Innovation Institute's base large language model built on the pure Mamba state-space architecture, trained on 5.8 trillion tokens of curated data. Its technical report showed it surpassing transformer-based open-weight models like Mistral 7B, Llama 3.1 8B, and Falcon2 11B, while matching Gemma 7B — demonstrating that even a pure (non-hybrid) Mamba design can equal or exceed transformer and hybrid designs at this scale.

Because the architecture processes tokens sequentially in linear time, Falcon Mamba is significantly faster at inference and needs far less memory for long sequence generation than its transformer peers. Released in August 2024 under the permissive TII Falcon-Mamba License 2.0 (Apache-2.0-based), it remains the reference implementation for state-space language modeling in the open ecosystem.