Microsoft OmniParser
Turns any UI screenshot into structured, clickable elements so vision models can drive a computer.
About
OmniParser is Microsoft Research's screen-parsing toolkit for pure vision-based GUI agents. Feed it a screenshot and it returns a structured list of interactable elements — buttons, icons, text fields — with bounding boxes and functional captions, using a fine-tuned YOLOv8 detector plus a Florence-2 captioning model. That structured output is what lets a vision model like GPT-4V ground its actions to the right pixel coordinates instead of guessing.
V2 cut latency ~60% (about 0.6s per frame on an A100), scored 39.6 average accuracy on ScreenSpot Pro, and added OmniTool — a ready-made harness that pairs OmniParser with your vision model of choice to control a Windows 11 VM. With 25,000+ stars it is the reference implementation the computer-use crowd builds on. The rough edges: running it locally wants a GPU, the repo's license metadata is ambiguous (GitHub reports CC-BY-4.0 while the README badge says MIT — check before commercial use), and parsing accuracy still degrades on dense or unusual UIs.
What it can do
- Screenshot-to-structured-UI parsing: interactable regions, bounding boxes, icon captions
- Fine-tuned YOLOv8 + Florence-2 models purpose-built for UI grounding
- OmniTool harness for controlling a Windows 11 VM with any supported vision model
- ~0.6s/frame latency on A100; 39.6 avg accuracy on ScreenSpot Pro
- Works across PC and phone screenshots and varied applications
- Hugging Face demo and model hub for try-before-you-install
Setup
- Clone the repo:
git clone https://github.com/microsoft/OmniParserand create a Python 3.12 conda env. - Install requirements:
pip install -r requirements.txt. - Download the V2 model weights from Hugging Face (
microsoft/OmniParser-v2.0). - Run the Gradio demo or the API server to parse screenshots into structured elements.
- (Optional) Pair with OmniTool and your vision model (GPT-4o, DeepSeek-R1, Qwen2.5-VL, or Anthropic Computer Use) to drive a Windows 11 VM.
Prerequisites
Python 3.12; a CUDA-capable GPU recommended for local inference (CPU works but slow); Hugging Face access for model weights; a vision-model API key for the agent loop.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (metrics tier, metrics-v1). Evidence: 25,528 GitHub stars (GitHub API), pushed Jul 20 2026, arXiv paper (2408.00203) and Microsoft Research blog corroborate the V2 claims; Replicate's model page confirms the latency and ScreenSpot Pro figures. Strengths: the reference screen-parsing stack for computer-use agents, real benchmarks, huge adoption. Docked for: (1) license metadata conflict — GitHub reports CC-BY-4.0 while the README badge says MIT, verify before commercial use; (2) local inference realistically needs a GPU; (3) accuracy drops on dense or unusual interfaces. License: ambiguous (CC-BY-4.0 per GitHub metadata, MIT per README badge).
★ 25,528 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 8.0/10
- Maintenance · recency, releases, issue load
- 8.0/10
- Openness · license + docs
- 9.0/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · Adoption 8: starScore from 25,528 stars on microsoft/OmniParser
- · Maintenance 8: pushed 2026-07-20 (recency 8); research cadence, not a product cadence
- · Openness 9: license mixed (0.7*8) + project page and docs verified (0.3*10)
- · Manual 8: arXiv paper 2408.00203, Microsoft Research blog, and Replicate page corroborate V2 latency/accuracy claims; docked for the license ambiguity, GPU requirement for local runs, accuracy drops on dense UIs
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.