JSONDecodeError on Windows when running Python produces warnings

[jleclanche]

Symptom

Running any poetry command would produce the following error:

Expecting value: line 1 column 1 (char 0)

Diagnosis

Under certain circumstances, running Python produces warnings which cannot simply be silenced by -W ignore. This confuses parsing methods.

By looking with -vvv, I debugged the issue.

In my case, the output looked like this, with the warning included:

'\'\\\\wsl.localhost\\Ubuntu-20.04\\home\\adys\\src\'\r\nCMD.EXE was started with the above path as the current directory.\r\nUNC paths are not supported.  Defaulting to Windows directory.\r\n{"implementation_name": "cpython", "implementation_version": "3.10.0", "os_name": "nt", "platform_machine": "AMD64", "platform_release": "10", "platform_system": "Windows", "platform_version": "10.0.22621", "python_full_version": "3.10.0", "platform_python_implementation": "CPython", "python_version": "3.10", "sys_platform": "win32", "version_info": [3, 10, 0, "final", 0], "interpreter_name": "cp", "interpreter_version": "3_10"}\r\n'

In poetry/utils/env.py, this causes run_python_script() to include those warnings in the output (the return value). The output of run_python_script() is parsed very strictly by sys_path, get_version_info, get_supported_tags, get_marker_env, etc.

In none of the cases above should the warning be included in the returned output value.

Resolution

run_python_script calls run(), which is a generic way of getting a command and running it using subprocess. However, further down the stack, the stderr gets merged with the stdout.

To solve the issue, I added a stderr argument to _run(), defaulting to its old behaviour. But run_python_script now specifies it, setting it to subprocess.DEVNULL. This silences other warnings which were not caught by -W ignore (I suppose a similar issue happened before, with python-specific warnings, causing this to be included).

Will include a PR.

[neersighted]

I feel like the lede might be buried here -- are you attempting to install into a Windows Python from Linux via WSL? Or are you running a Windows Poetry from WSL?

Either way, there are definitely dragons here, and these are only likely to be a few of them. However, it is worth noting that the output that cannot be silenced by -W ignore is from the operating system and not Python -- this is simply an extremely sharp edge of WSL.

[jleclanche]

@neersighted I was inside a directory that indeed looked like \\\\wsl.localhost\\Ubuntu-20.04\\home\\adys\\src\, which something in the stack doesn't like, making the execution of Poetry produce that warning, completely outside of Python's control.

I suspect similar edge cases can happen outside of WSL. For example, on Linux, I believe libc itself can throw warnings under extreme circumstances, which you would not be able to silence either. The PR likely fixes that.

My proposed fix is safe because none of the run_python_script calls expect anything in stderr, so anything that does pop up there is garbage.

[neersighted]

The 'something' there would be cmd.exe which is used to host any PAE executable started by WSL interop, as there has to be a bound console for input/output.

I don't believe there are any cases on a Linux system where you should get output on stderr from non-application code and still have the program run as expected (and it wouldn't be from libc, but instead from the dynamic linker).

Anyway, my concerns are on the PR.