microsoft.com
OmniParser v2: Turning Any LLM into a Computer Use Agent
Microsoft Research introduces a model for converting screenshots into structured interface elements.
https://www.microsoft.com/en-us/research/articles/omniparser-v2-turning-any-llm-into-a-computer-use-agent/
About This Resource
Microsoft Research introduces a model for converting screenshots into structured interface elements. The article explains how this information can help agents locate and interact with controls on a screen.
Saved Notes
Original notes saved with this link. They may describe an earlier version.
GitHub - microsoft/OmniParser: A simple screen parsing tool towards pure vision based GUI agent
OmniParser V2: Turning Any LLM into a Computer Use Agent - Microsoft Research
OmniParser is a comprehensive method for parsing user interface screenshots into structured and easy-to-understand elements, which significantly enhances the ability of GPT-4V to generate actions that can be accurately grounded in the corresponding regions of the interface.
v2 Announcement
Graphic User interface (GUI) automation requires agents with the ability to understand and interact with user screens. However, using general purpose LLM models to serve as GUI agents faces several challenges:
- Reliably identifying interactable icons within the user interface
- Understanding the semantics of various elements in a screenshot and accurately associating the intended action with the corresponding region on the screen
OmniParser closes this gap by 'tokenizing' UI screenshots from pixel spaces into structured elements in the screenshot that are interpretable by LLMs. This enables the LLMs to do retrieval based next action prediction given a set of parsed interactable elements.
OmniParser V2 takes this capability to the next level. Compared to its predecessor (opens in new tab), it achieves higher accuracy in detecting smaller interactable elements and faster inference, making it a useful tool for GUI automation.
In particular, OmniParser V2 is trained with a larger set of interactive element detection data and icon functional caption data. By decreasing the image size of the icon caption model, OmniParser V2 reduces the latency by 60% compared to the previous version.
Notably, Omniparser+GPT-4o achieves state-of-the-art average accuracy of 39.6 on a recently released grounding benchmark ScreenSpot Pro (opens in new tab), which features high resolution screen and tiny target icons. This is a substantially improvement on GPT-4o's original score of 0.8.