Skip to content

Computer Use: adding playwright as a tool for tool_use #272

Open
@sumitmehta1025

Description

@sumitmehta1025

Hello Team!
Please see attached code and help me solve the issue.

Goal: Leverage computer use to navigate a website, close all pop-ups and scroll to the end to take screenshot of all sections and save these screenshots in a local folder.
Requirement: use API only as task needs to be automated.
Issue:

  1. not able to capture entire stream even after using .stream
  2. not able to take screenshots even after providing playwright as tool to the computer

Please help, here are the details!

Code:
playwright_tool_computer_use.txt

Output after running the code:

I'll take screenshots of www.example.com with 30% overlap after closing any popups or cookie notices. Let me start by taking a screenshot to see the desktop environment.

Full response: I'll take screenshots of www.example.com with 30% overlap after closing any popups or cookie notices. Let me start by taking a screenshot to see the desktop environment.
Completed 0 tool calls

Final message content: [BetaThinkingBlock(signature='ErUBCkYIAxgCIkBW4YaznWGny3JOTqSg6AhVIxbZ9nf5iY32LmXGIITiZQ5uSuBc6z3Cu4coJy/Z/kY3xo35MKwo1Qo2YGLID8apEgz/LumPN6wau4ne6pUaDDBaQd6FNGd1d8REzSIwRuBjLMschj7+idryMGaeJcx/vwFYlzgesHquN5Vbm1o39RK6bNSgxBLhoyIe901KKh0APsc49DDQ2GOZUPjoIb0muxgh5VQMsuVpzrGqdxgC', thinking="I need to use the `website_screenshot` function to capture screenshots of www.example.com with 30% overlap. Before that, I should check if there are any popups or cookie notices that need to be closed, which would require me to first navigate to the website using a browser.\n\nLet me break this down into steps:\n1. Take a screenshot to see the desktop\n2. Open a web browser (likely Firefox)\n3. Navigate to www.example.com\n4. Check for and close any popups or cookie notices\n5. Use the website_screenshot function to capture the entire page with 30% overlap\n\nLet's start by taking a screenshot to see the desktop.", type='thinking'), BetaTextBlock(citations=None, text="I'll take screenshots of www.example.com with 30% overlap after closing any popups or cookie notices. Let me start by taking a screenshot to see the desktop environment.", type='text'), BetaToolUseBlock(id='toolu_01R9oGirVu6QjhBBcFyPPg9v', input={'action': 'screenshot'}, name='computer', type='tool_use')]

Thanks!

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions