[Feature]: Option to disable `inspect.stack()` calls for performance optimization

[jl-martins]

🚀 Feature Request

Summary

playwright-python currently makes frequent calls to inspect.stack, which significantly impacts performance in certain use cases, such as web scraping with scrapy-playwright. This feature request proposes adding an option to disable stack inspections to improve execution speed when debugging information is not required.

Use case

Although Playwright is primarily designed for end-to-end testing, some projects leverage it for web scraping, where performance is a key concern. In my case, I am using scrapy-playwright, and profiling results from cProfile indicate that inspect.stack calls contribute to roughly 25% of the total execution time, as illustrated in the following icicle and call graph:

As far as I can tell, these calls are primarily used for debugging and are not critical for normal execution.

Proposed solution

Introduce an environment variable (e.g.: PW_INSPECT_STACK) that allows users to disable stack inspections when debugging is not required. The default value would be 1 to preserve the current behavior.

Motivation

• Performance Boost: Reducing unnecessary function calls can significantly speed up Playwright for scraping-heavy applications.
• Flexibility: Users who require stack traces for debugging can leave the option enabled.
• Backward Compatibility: The default behavior remains unchanged, ensuring no disruption for existing users.

[luisferreira93]

+1

[mxschmitt]

Do you have in comparison how much time the actual page.goto/click/fill etc's took? Lets say 1s in average with network resources around different origins and sometimes even iframes etc. This should be way larger and demonstrate that its not worth investing into that end (inspector.stack).

[neoncube2]

I think I might be affected by this, too.

In my case, I load a page, check to make sure that a certain amount of buttons are visible, then execute a few thousand page.evaluate() calls in a tight loop.

Profiling results

Use case

I have a game written in JavaScript, and I'm using Pytorch to train an AI to play the game. I use Playwright to load the page and then play the game via JS calls: await evaluate('doGameAction', action_index)

As an aside: Is evaluate() truly async? It seems like it uses a lot of CPU, and I'm guessing that internally it may involve a loop with a bunch of wait() calls to see if the browser has finished executing the JS code.

Thanks! :)

[neoncube2]

I think I might have a fix for this that improves performance and doesn't require a special flag! :)

I'm currently running some tests, and if they're successful, I'll probably open a PR within the next few days, God willing.

[neoncube2]

I've opened two PRs to fix this:

• _connect.py: Don't request context when inspecting stack #2835 is a small change and provides a 2.5x speedup
• Even faster stack traces for _connection.py #2836 is a slightly more involved fix but provides almost a ~6x speedup

[SichangHe]

No joke. MASSIVE speed up here to be had. For my crawler, with playwright==1.52.0, ~200 browser tabs, my main Python process CPU stays ~100% and the asyncio event loop hangs constantly. With an aggressive monkey patch inspired by @neoncube2's PRs, CPU is around 30% and the event loop almost doesn't hang.

I attach the monkey patch I am using in case it can help anyone.

"""Patch Playwright and reexport its async API."""

import inspect
import sys
from pathlib import Path
from types import FrameType


def _playwright_frames_generator(start_frame: FrameType, context=0):
    """Yield `FrameInfo`s for the call stack, starting from `start_frame`,
    ending after the second non-Playwright frame, optimizing for speed for
    Playwright.
    Unlike `inspect.stack`, do not call the expensive `getsourcefile` and
    instead simply provide the file name in the frame.
    Aggressively inline the implementation in `inspect.py`."""
    frame = start_frame
    n_non_playwright_frame = 0
    while frame is not None:
        if inspect.istraceback(frame):
            lineno = frame.tb_lineno
            frame = frame.tb_frame
        else:
            lineno = frame.f_lineno
        filename = frame.f_code.co_filename
        function_name = frame.f_code.co_name
        if context > 0:
            start = lineno - 1 - context // 2
            try:
                lines, _lnum = inspect.findsource(frame)
            except OSError:
                lines = index = None
            else:
                start = max(0, min(start, len(lines) - context))
                lines = lines[start : start + context]
                index = lineno - 1 - start
        else:
            lines = index = None
        yield inspect.FrameInfo(frame, filename, lineno, function_name, lines, index)
        if not filename.startswith(playwright_module_path_str):
            n_non_playwright_frame += 1
            if n_non_playwright_frame > 1:
                frame = None
        else:
            frame = frame.f_back


def _optimized_inspect_stack(context=None):
    """Replacement for `inspect.stack`.
    If called from Playwright, return a lazy and fast generator.
    Otherwise, follow the original `inspect.stack` implementation."""
    caller_frame = sys._getframe(1)
    caller_filename = caller_frame.f_code.co_filename
    if caller_filename.startswith(playwright_module_path_str):
        return _playwright_frames_generator(caller_frame, context or 0)
    else:
        return inspect.getouterframes(caller_frame, context or 1)


def patch_inspect_stack_for_playwright():
    """Monkey patch `inspect.stack` to speed it up when called from Playwright."""
    inspect.stack = _optimized_inspect_stack


patch_inspect_stack_for_playwright()

import playwright  # noqa:E402
from playwright.async_api import *  # type:ignore[reportWildcardImportFromLibrary]#noqa:E402,F403
from playwright.async_api import _context_manager  # noqa:E402

PlaywrightContextManager = _context_manager.PlaywrightContextManager
playwright_module_path_str = str(Path(playwright.__file__).parent)

What I mean by the event loop hanging.

Assuming all the blocking code have been moved to other threads, but there are more tasks than the event loop can handle, it will still have tasks piled up into a backlog, meaning it may not wake up an IO-bound/time-sleep task even if the response is here or the time is up.

Specifically, I use asyncio.sleep not waking up a while after the specified time has passed to detect when the event loop hanged.

Flamegraphs of my crawler before and after the monkey patch shows the time inspect.stack took went down from ~80% to negligible.

Original	Patched

This shows inspect.getsourcefile being the culprit.

[jl-martins]

No joke. MASSIVE speed up here to be had. For my crawler, with playwright==1.52.0, ~200 browser tabs, my main Python process CPU stays ~100% and the asyncio event loop hangs constantly. With an aggressive monkey patch inspired by @neoncube2's PRs, CPU is around 30% and the event loop almost doesn't hang.

I attach the monkey patch I am using in case it can help anyone.

"""Patch Playwright and reexport its async API."""

import inspect
import sys
from pathlib import Path
from types import FrameType


def _playwright_frames_generator(start_frame: FrameType, context=0):
    """Yield `FrameInfo`s for the call stack, starting from `start_frame`,
    ending after the second non-Playwright frame, optimizing for speed for
    Playwright.
    Unlike `inspect.stack`, do not call the expensive `getsourcefile` and
    instead simply provide the file name in the frame.
    Aggressively inline the implementation in `inspect.py`."""
    frame = start_frame
    n_non_playwright_frame = 0
    while frame is not None:
        if inspect.istraceback(frame):
            lineno = frame.tb_lineno
            frame = frame.tb_frame
        else:
            lineno = frame.f_lineno
        filename = frame.f_code.co_filename
        function_name = frame.f_code.co_name
        if context > 0:
            start = lineno - 1 - context // 2
            try:
                lines, _lnum = inspect.findsource(frame)
            except OSError:
                lines = index = None
            else:
                start = max(0, min(start, len(lines) - context))
                lines = lines[start : start + context]
                index = lineno - 1 - start
        else:
            lines = index = None
        yield inspect.FrameInfo(frame, filename, lineno, function_name, lines, index)
        if not filename.startswith(playwright_module_path_str):
            n_non_playwright_frame += 1
            if n_non_playwright_frame > 1:
                frame = None
        else:
            frame = frame.f_back


def _optimized_inspect_stack(context=None):
    """Replacement for `inspect.stack`.
    If called from Playwright, return a lazy and fast generator.
    Otherwise, follow the original `inspect.stack` implementation."""
    caller_frame = sys._getframe(1)
    caller_filename = caller_frame.f_code.co_filename
    if caller_filename.startswith(playwright_module_path_str):
        return _playwright_frames_generator(caller_frame, context or 0)
    else:
        return inspect.getouterframes(caller_frame, context or 1)


def patch_inspect_stack_for_playwright():
    """Monkey patch `inspect.stack` to speed it up when called from Playwright."""
    inspect.stack = _optimized_inspect_stack


patch_inspect_stack_for_playwright()

import playwright  # noqa:E402
from playwright.async_api import *  # type:ignore[reportWildcardImportFromLibrary]#noqa:E402,F403
from playwright.async_api import _context_manager  # noqa:E402

PlaywrightContextManager = _context_manager.PlaywrightContextManager
playwright_module_path_str = str(Path(playwright.__file__).parent)

What I mean by the event loop hanging.

Assuming all the blocking code have been moved to other threads, but there are more tasks than the event loop can handle, it will still have tasks piled up into a backlog, meaning it may not wake up an IO-bound/time-sleep task even if the response is here or the time is up.

Specifically, I use asyncio.sleep not waking up a while after the specified time has passed to detect when the event loop hanged.

Flamegraphs of my crawler before and after the monkey patch shows the time inspect.stack took went down from ~80% to negligible.

Original	Patched

This shows inspect.getsourcefile being the culprit.

I did a similar monkey patch a while back and got similar improvements. Massive speed up indeed.

Thanks for sharing, @SichangHe!

[neoncube2]

Thanks, @SichangHe! :D Let me see if I can integrate this into my PR! :)