Orca

Orca, published by Microsoft Research in June 2023, is a 13-billion-parameter model built on LLaMA/Vicuna foundations that learns to imitate the reasoning process, not just the style, of large foundation models. It trains on rich signals from GPT-4 including explanation traces and step-by-step thought processes, with teacher assistance from ChatGPT and large-scale diverse imitation data sampled from the Flan 2022 collection.

The paper reports Orca surpassing Vicuna-13B by over 100% on Big-Bench Hard and 42% on AGIEval, reaching parity with ChatGPT on BBH and competitive scores on SAT, LSAT, GRE, and GMAT exams, while trailing GPT-4. Important caveats: Microsoft never released public weights (a weight diff was discussed but held up on legal review), so this remains a research artifact rather than a usable model, and its training lineage inherits LLaMA's non-commercial restrictions.