[Feat-Req] Add support for Omniparser V2

[Emasoft]

Microsoft presented Omniparser V2 and it is showing the best performances in GUI parsing. You can read about the details here.
It is not a multimodal but a visual parsing model specialized in detecting the GUI elements and can be combined with any LLM, including Claude.

It would be great to add support for Omniparser V2 in Claude-Debugs-For-You, so that it would be able to debug GUIs and web apps visual output!

https://github.com/microsoft/OmniParser/

[jasonjmcghee]

Thanks for your interest in the project and filing an issue @Emasoft!

My first question would be - does it make sense to directly integrate this, or is this something that should be a tool, separately defined? Like, a web search might be a great feature to be able to attempt to understand a specific cryptic error message, but not positive it should be something built here - as opposed to a separate "web search" tool.

The intent of this extension / tool is to allow an llm to step through code and inspect stack frames. Performing visual observation on a GUI seems related in certain contexts, definitely, but would it make sense for it to be a separate MCP server which is responsible for / specializes in observing / interacting with a GUI?

[Emasoft]

Of course it makes sense! Stepping through the code while the GUI is rendered is essential to debug programs with GUI or even TUI! If I step in and read that a value is set to color red, but the GUI is not updated or updated with the wrong color, then I found an error. But if I don't have the ability to see the results on the value set to red, how can I know if it works or not? How can the LLM catch a bug? The LLM must have the ability to see the changes to the UI after each step to debug a GUI. So it is essential for "Claude-Debug-For-You" to have a secondary llm that return the description of the GUI after each debug step, making a diff from the previous state and see what changed and how. I would say that it is hopeless to try to debug any program with a UI (being it web, graphic, TUI, game, svg rendering, etc.) without a secondary model like Omniparser to report about the UI changes! Just the other day I discovered how impossible is to develop with AI models the programs made with Textual, since the LLM have no way to understand the output TUI. But now with your debugger integrated with Omniparser, it would become possible!

[jasonjmcghee]

As far as I can tell from reading about OmniParser, there's the model and then the interactive tool- the tool is a windows VM.

If you just want to be able to interpret a screenshot (what the model does) you should just expose an appropriate MCP server. Then Claude (or any LLM) can "understand" screenshots.

These MCP servers are built to be modular on purpose!

This actually specifically references exposing OmniParser to an LLM via MCP: https://www.mcpservers.ai/servers/evalstate/Hugging%20Face%20Spaces%20(mcp-hfspace)

You might need to directly tell it to use it, or you could wrap it as a standalone MCP server with good descriptions/prompts.

And if you want to be able to interact with it, that's kind of a different ball game- but again these could be exposed as commands (playwright for web, omnitool or Claude computer for OS / native interaction).

To me, these are distinct servers (single responsibility).

One for interacting with the debugger (setting breakpoints, stepping, evaluation, etc) in VSCode

One for understanding / interacting with a UI.

Both are useful independently.

If you want both, you enable both. You'd have a seamless UI debugging experience.

[Emasoft]

They are not independent al all. You didn't realize it yet, but you already know that.
You said it youself:

One for interacting with the debugger (setting breakpoints, stepping, evaluation, etc) in VSCode

One for understanding / interacting with a UI.

The "evaluation" step of the debugger and the "understanding" step of the UI parser, are one and the same.

Both are the act of learning what is changed with the step.
Both must be evaluated at the same time by the main LLM to decide if there is an error or not.
The value of a variable or the color of a button are both changes that happen when the debugger makes a step.
To be able to debug like a human programmer, the LLM must be able to evaluate ALL the changes that happen after each step.
This is not something it can delegate. The main LLM must get both at the same time to decide the next action.
As your extension makes Claude informed of the changes in the variables, it should make it also informed about the changes in the GUI.
Of course Claude is able to read screenshots too, but not as well as Omniparser V2! More importantly, Claude is already focusing on the VSCode window with the code, so it cannot evaluate the GUI in the app window (or in the browser, or in the terminal).
Omniparser can provide the informations about the changes in the GUI, so that those are 'read' by the LLM along with the variables from the stack when the LLM make the tool call to receive the update after it requested the debugger to make a new step with another tool call.

The full cycle of the LLM is then:
1 - Claude examines the code
2 - Claude Sets a Breakpoint and starts the debugger
3 - Claude Makes a Step
4 - Claude READS the changes (both in the variables AND in the GUI)
5 - Claude uses this information to determine if there is an error/anomaly
6 - Claude stops the debugger
7 - Claude commits a change to the code to attempt to fix the issue
8 - Repeat

The 4th step is the most important, since evaluating the state of the variables and the state of the GUI at the same time is essential to discover mismatches between the two. That is the only way to find a bug. This is what debugging is all about.
So why are you saying that the two processes must happen separately? Those two processes must be the result of a single tool call from the LLM, returning both the variables state AND the GUI state at the same time. So this function call must be written by the same person who wrote the rest of the extension. It must be integrated in the extension to work correctly. You cannot use the Microsoft tool or any other program. You must handle the Omniparser model directly in your code to be sure that when it reads the screen and produce the textual description, the debugger is active and freezed in a breakpoint step, so that you can be sure that the state of the screen is exactly the one matching the current step state, not one from before or after.
If you are worried about handling two models at the same time, I can understand. It is more complicated than just letting Claude do it all. But there are 3 reasons to do it that way: 1) to get much better visual parsing results from the GUI we need Omniparser; 2) making the same LLM parse the GUI and parse the debug step state at the same time is going to mess with the LLM context, confusing the AI and producing more errors. 3) The LLM must receive the data about the changes of both the variables and the GUI and evaluate them at the same time, otherwise any mismatch between them cannot be found by the LLM.
You want to handle such delicate loop directly, and not delegating it to other tools.

But maybe I'm wrong. I just can't see why you want to separate those two things instead of creating a well integrated debug solution.

[nightscape]

I don't understand why 4. can't be two three steps?
4.a Claude reads debugger information
4.b Claude inspects UI
4.c Claude continues running the application

That should yield the exact same data as running them sequentially from within one MCP, wouldn't it?
If your concern is timing between debugging and UI:
The 4.a step would block until a breakpoint is hit, then return the data from the debugging session.
Then there would be sufficient time for the OmniVision MCP to take a screenshot.
Only after that would Claude let the application continue.

[nightscape]

To be safe, it would maybe have to be more steps:

1. Blocking-wait for breakpoint
2. Take screenshot (pre-step state)
3. Wait brief period for any pending UI updates
4. Take screenshot again (stabilized pre-step)
5. Make a debugger step
6. Take screenshot (immediate post-step)
7. Wait brief period for any pending UI updates
8. Take screenshot (stabilized post-step)
9. Resume application

[jasonjmcghee]

So why are you saying that the two processes must happen separately? Those two processes must be the result of a single tool call from the LLM, returning both the variables state AND the GUI state at the same time. So this function call must be written by the same person who wrote the rest of the extension

I'm not saying they have to, I'm saying they should be separate. For the same reason you build a class or a library for a single responsibility. Loose coupling reduces complexity.

You can call a function from one library and then call one from another without them being tightly coupled.

Any single action can be performed by any tool- the llm is facilitating everything.

• LLM makes plan
• Set breakpoint (debugger tool)
• Take screenshot and interpret (UI tool)
• LLM understands where to click
• Interact with UI to trigger breakpoint (UI tool)
• Explicitly wait for breakpoint (debugger tool - new functionality)
• Hit breakpoint (program is paused)
• LLM now knows where it's suspended
• Take screenshot and interpret (UI tool)
• Evaluate expressions (debugger tool)
• Delete breakpoint and/or Resume execution (debugger tool)
• Decide what to do next etc

Maybe there's some additional waiting in there, where "wait" is something the UI tool can do as well. (This ended up looking a lot like @nightscape 's but I was writing it out and labeling which tool was being used)

I would argue the only thing that would need to be modified in this MCP server would be "wait for breakpoint hit" functionality- which shouldn't be too much of a lift.

---

To clarify, I think an OmniParser + UI interaction MCP Server is a great idea, but I do think it's a distinct MCP server. It's not about not wanting to deal with another model, I just see it as independent and nicely self contained this way, easier to maintain independently, and more flexible.

[Emasoft]

@jasonjmcghee : Ok, I understand your POV now. At least I helped you to realize the need for the "wait for breakpoint hit" functionality. 😄