๐ง Come fanno gli #AI Agent come #Operator a eseguire azioni sui browser e su qualunque interfaccia grafica?
๐๏ธ Questo รจ un esempio di utilizzo di #OmniParser V2 in esecuzione in locale. Il sistema elabora ciรฒ che "vede" nello schermo, e lo converte in dati strutturati che mappano e classificano ogni elemento.
โ๏ธ Questi dati diventano contesto per un #LLM, che puรฒ eseguire operazioni sugli elementi.
๐ง #Microsoft ha rilasciato #OmniParser V2: un sistema open source in grado di compiere azioni nell'interfaccia utente.
โจ Non solo sul browser, ma un sistema che usa un #LLM in un Computer Use Agent.
๐ Il progetto: https://github.com/microsoft/OmniParser
___
โ๏ธ ๐ฆ๐ฒ ๐๐๐ผ๐ถ ๐ฟ๐ถ๐บ๐ฎ๐ป๐ฒ๐ฟ๐ฒ ๐ฎ๐ด๐ด๐ถ๐ผ๐ฟ๐ป๐ฎ๐๐ผ/๐ฎ ๐๐ ๐พ๐๐ฒ๐๐๐ฒ ๐๐ฒ๐บ๐ฎ๐๐ถ๐ฐ๐ต๐ฒ, ๐ถ๐๐ฐ๐ฟ๐ถ๐๐ถ๐๐ถ ๐ฎ๐น๐น๐ฎ ๐บ๐ถ๐ฎ ๐ป๐ฒ๐๐๐น๐ฒ๐๐๐ฒ๐ฟ: https://bit.ly/newsletter-alessiopomaro
#AI #GenAI #GenerativeAI #IntelligenzaArtificiale #LLM #AIAgent
Yadong Lu, Senior Researcher; Thomas Dhome-Casanova (opens in new tab), Software Engineer; Jianwei Yang, Principal Researcher; Ahmed Awadallah, Partner Research Manager Graphic User interface (GUI) automation requires agents with the ability to understand and interact with user screens. However, using general purpose LLM models to serve as GUI agents faces several challenges: 1) reliably identifying [โฆ]
#้ๆบๅไบซ ๅพฎ่ป้ๆบไบไธๆฌพๅฏไปฅ่งฃๆๅ่ญๅฅ่ขๅนไธๅฏไบคไบๅ็คบ็ๅทฅๅ ท๏ผOmniParser๏ผๅฎ่ฝๆบ็ขบ็่ญๅฅๅบ็จๆถ็้ขไธญ็ๅฏไบคไบๅ็คบ๏ผๅจ่งฃๆๆน้ขๅชๆผGPT-4V
็น้ป๏ผ
1ใ้้่ญๅฅ่ฝๅ๏ผ่ฝๆพๅบ็้ขไธๆๆๅฏไปฅ้ปๆ็ๅฐๆน๏ผๅ
ทๅ่ช็พฉ็่งฃ่ฝๅ๏ผ่ฝ็่งฃๆ้ๆๅ็คบ็ๅ
ท้ซๅ่ฝ
2ใๅฏไปฅไฝ็บๆไปถ๏ผ่Phi-3.5-Vใ Llama-3.2-Vไปฅๅๅ ถไปๆจกๅ็ตๅไฝฟ็จ
3ใๆฏๆ็ตๆงๅ่ผธๅบ๏ผ้คไบ่ญๅฅ่ขๅนไธ็ๅ ็ด ๏ผ้่ฝๅฐ้ไบๅ ็ด ่ฝๆๆ็ตๆงๅ็ๆธๆ
ๅฐๆกๅฐๅ๏ผ github.com/microsoft/OmniParser
็ถฒ็ซ๏ผ microsoft.github.io/OmniParser
๐ #Microsoft introduces #OmniParser, a new screen parsing module for #GUI interactions:
โข Converts UI screenshots into structured elements for improved #AI agent navigation
โข Works with #GPT4V to generate precise actions for interface regions
โข Achieves top performance on #WindowsAgentArena benchmark
๐ ๏ธ Key Components:
โข Specialized datasets for icon detection and description
โข Fine-tuned detection model for identifying actionable regions
โข Captioning model for extracting functional semantics
๐ Performance Highlights:
โข Outperforms standard #GPT4V on #ScreenSpot benchmarks
โข Compatible with #Phi35V and #Llama32V models
โข Functions across PC and mobile platforms without HTML dependencies
๐ Learn more: https://www.microsoft.com/en-us/research/articles/omniparser-for-pure-vision-based-gui-agent/
By Yadong Lu, Senior Researcher; Jianwei Yang, Principal Researcher; Yelong Shen, Principal Research Manager; Ahmed Awadallah, Partner Research Manager Recent advancements in large vision-language models (VLMs), such as GPT-4V and GPT-4o, have demonstrated considerable promise in driving intelligent agent systems that operate within user interfaces (UI). However, the full potential of these multimodal models remains [โฆ]
Alessio Pomaro
Carlos Mendible 
Erik Jonker
็ๆฐธๅธฅ๐ฅ
michabbb