Cannot attach Python debugger to debugpy within a future inside Docker container

[sebastko]

Type: Bug

High-level setup:

• VSCode connected via SSH extension to a Linux VM (w/ Python 3.8.5, although probably that doesn't matter?)
• Python code executing inside a Docker container (also Linux image, w/ Python 3.9.10)
• Trying to set a debugpy breakpoint inside a future and attach a debugger to the process inside the container

Consider the following script futures-example.py, with a debugpy breakpoint inside a future:

import concurrent.futures

def my_function(arg1, arg2):
    import debugpy;
    debugpy.listen(5678)
    print('waiting')
    debugpy.wait_for_client()
    debugpy.breakpoint()
    return arg1 * arg2

with concurrent.futures.ThreadPoolExecutor() as executor:
    future = executor.submit(my_function, 2, 3)
    result = future.result()
    print(result)

Now let's run it inside a Docker container:

docker run -v ~/src:/src --network host -t <image ID> python /src/futures-example.py

After seeing in the terminal:

waiting

...I try attaching a debugger with the following launch.json configuration:

{
    "version": "0.2.0",
    "configurations": [
        {
            "name": "Python: Attach",
            "type": "python",
            "request": "attach",
            "justMyCode": false,
            "connect": {
              "host": "0.0.0.0",
              "port": 5678
            },
            "pathMappings": [
              {
                "localRoot": "/home/azureuser/src",
                "remoteRoot": "/src"
              }
            ]
          }
    ]
}

You can see the program execution continued, but the breakpoint is not hit in VSCode. Also, the process is left hanging at the end:

waiting
6

You can also see that debugger seem to be partially attached, even though the execution doesn't stop at the breakpoint:

Notes:

• If you run it on the host Linux VM (outside Docker container), and change "host" to "localhost", it works fine.
• If you put the breakpoint somewhere in the main process (not in the future), and still run it inside the Docker container, it also works fine.
• Interestingly, if you put breakpoints inside Ray actors, it also works fine -- the problem is just with Python futures...

Extension version: 2023.20.0
VS Code version: Code 1.84.2 (1a5daa3a0231a0fbba4f14db7ec463cf99d7768e, 2023-11-09T10:51:52.184Z)
OS version: Windows_NT x64 10.0.22631
Modes:
Remote OS version: Linux x64 5.15.0-1040-azure
Remote OS version: Linux x64 5.15.0-1040-azure

System Info

Item	Value
CPUs	Intel(R) Xeon(R) Platinum 8370C CPU @ 2.80GHz (16 x 2793)
GPU Status	2d_canvas: unavailable_software canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: disabled_software multiple_raster_threads: enabled_on opengl: disabled_off rasterization: disabled_software raw_draw: disabled_off_ok video_decode: disabled_software video_encode: disabled_software vulkan: disabled_off webgl: unavailable_software webgl2: unavailable_software webgpu: unavailable_software
Load (avg)	undefined
Memory (System)	63.95GB (47.31GB free)
Process Argv	--crash-reporter-id 3d536ca0-305a-4040-ad86-5d746b939f7a
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: host name removed by the author
OS	Linux x64 5.15.0-1040-azure
CPUs	AMD EPYC 7V12 64-Core Processor (96 x 2445)
Memory (System)	1771.74GB (1753.62GB free)
VM	0%

A/B Experiments

vsliv368:30146709
vsreu685:30147344
python383cf:30185419
vspor879:30202332
vspor708:30202333
vspor363:30204092
vslsvsres303:30308271
vserr242cf:30382550
pythontb:30283811
vsjup518:30340749
pythonptprofiler:30281270
vshan820:30294714
vstes263:30335439
vscorecescf:30445987
vscod805:30301674
binariesv615:30325510
bridge0708:30335490
bridge0723:30353136
vsaa593:30376534
pythonvs932:30410667
py29gd2263:30880072
vsclangdc:30486549
c4g48928:30535728
dsvsc012cf:30540253
pynewext54:30695312
azure-dev_surveyone:30548225
3biah626:30602489
89544117:30613380
showlangstatbar:30737416
pythonfmttext:30731395
fixshowwlkth:30771522
showindicator:30805244
pythongtdpath:30769146
i26e3531:30792625
pythonnosmt12:30797651
pythonidxpt:30866567
pythonnoceb:30805159
dsvsc013:30795093
dsvsc014:30804076
dsvsc015:30845448
pythontestfixtcf:30871695
pythonregdiag2:30871582
pyreplss1:30886140
pythonmypyd1:30879173
pythoncet0:30885854
pythontbext0:30879054
dsvsc016:30886110
dsvsc017:30886112
dsvsc018cf:30886115
aa_t_chat:30882232

[sebastko]

Update/clarification: the "hanging at the end" part is perhaps simply caused by the debugpy listener inside the future... so if you write the code like this:

import concurrent.futures
import debugpy;
debugpy.listen(5678)

def my_function(arg1, arg2):
    print('waiting')
    debugpy.wait_for_client()
    debugpy.breakpoint()
    return arg1 * arg2

with concurrent.futures.ThreadPoolExecutor() as executor:
    future = executor.submit(my_function, 2, 3)
    result = future.result()
    print(result)

...then the process exits nicely at the end, but all other symptoms (i.e., breakpoint not breaking) are still true.

[int19h]

Given that you're using a thread pool executor, this would create a bunch of threads in the same process, with each trying to set up a debug server. I would expect all threads except for the first one to fail to do so and abort with exception, but it might be hitting a bug where we don't detect it correctly and instead just reopen the socket on the same port several times, which would likely get it into a very weird state with multiple debug adapters contending for the same process.

Does it work for you if debugpy.listen() etc is outside of the function?

[sebastko]

Given that you're using a thread pool executor, this would create a bunch of threads in the same process, with each trying to set up a debug server.

I believe only one future is being executed in the example, so only a single debugpy server should listen, as far as I understand.

Does it work for you if debugpy.listen() etc is outside of the function?

@int19h: please see my follow-up post in the issue - the breakpoint still doesn't break:

Update/clarification: the "hanging at the end" part is perhaps simply caused by the debugpy listener inside the future... so if you write the code like this:

import concurrent.futures
import debugpy;
debugpy.listen(5678)

def my_function(arg1, arg2):
    print('waiting')
    debugpy.wait_for_client()
    debugpy.breakpoint()
    return arg1 * arg2

with concurrent.futures.ThreadPoolExecutor() as executor:
    future = executor.submit(my_function, 2, 3)
    result = future.result()
    print(result)

...then the process exits nicely at the end, but all other symptoms (i.e., breakpoint not breaking) are still true.