Black inserts trailing comma after **kwargs in non-3.6 file

[dgingrich]

Operating system: macOS 10.13.6
Python version: 3.6.3
Black version: 18.6b4
Does also happen on master: yes

I'm using black --line-length=101 --safe -v to format a file that contains a snippet like below:

# No f strings or trailing commas after *, *args, or **kwarg parameters elsewhere in the file
class MyClass(BaseClass):
    def __init__(
        self,
        x,
        *args,
       # Below line is changed by black: trailing comma added
        **kwargs
    ):
        try:
            super().__init__(
                x=x
                *args,
                # ...But only if the next line also has a trialing comma
                **kwargs,
            )
        except Exception as e:
            raise

Black is always adding a trailing comma after the **kwargs in the def __init__ (below the first comment). This happens even on master and even after clearing black's cache.

A workaround is removing the trailing comma from the **kwargs in the super().__init__ call (below the second comment), even though this comma is legal in Py3.5:

$ python3.5
>>> class Base:
...     def __init__(self, *a, **ka):
...             self.a = a
...             self.ka = ka
...
>>> class Child(Base):
...     def __init__(self, *a, **ka):
...             super().__init__(
...                     *a,
...                     **ka,
...             )
...
>>> c = Child(1, 2, a=3, b=4)
>>> c
<__main__.Child object at 0x10a44e2b0>
>>> c.a
(1, 2)
>>> c.ka
{'b': 4, 'a': 3}

[zsol]

Thanks for reporting! This is clearly broken behavior on black's part. TBH I was a little surprised trailing commas are accepted after star args in python <3.6 and this complicates things a bit. Right now the simplistic verison detection used by black can only differentiate between 3.6 and non-3.6 code by (incorrectly) looking at trailing commas in declarations or function calls. Just simply looking in declarations is not enough, since black itself generates code with trailing commas in function calls, so I'll need to poke this a bit more to really fix the problem.

[asottile]

Having implemented (essentially) the same logic, trailing commas are allowed after unpackings in function calls in python 3.5+ and in function definitions in python 3.6+. The fix here is probably to add a --py35 option and detection, or to drop auto-detection based on calls altogether

[ambv]

Yeah, this is over-reaching on Black's part. We should probably stop discovering 3.6+ based on this then, which is unfortunate since in most cases by now it is a 3.6+ file which just doesn't happen to use other Python 3.6 features.

[zsol]

To demonstrate my point earlier, consider this code:

def f(
    a,
    **kwargs,
):
    return SomeLongClassName(
        the_first_parameter_is_really_long_to_really_make_sure_there_is_a_break,
        **kwargs,
     )

Today black collapses the function declaration to a single line and keeps the return statement broken over mutliple lines, and this results in a stable format (i.e. running black over the formatted output doesn't reformat it again).

If we stop discovering 3.6+ based on trailing commas in function calls, running black over the formatted output will reformat it again because black doesn't put the trailing comma in function calls pre-3.6.

Now we could solve this by introducing 3.5 detection as well. This might be worth doing in the short term, but I think it highlights a serious shortcoming in black around python version detection: if we're trying to guess the version based on observed features in the input code, we can't rely on the output code having the same features. Looks like we've been lucky so far (or I'm just discounting @ambv's brilliance - in which case I apologize ;) ) because the current rules for version detection seem stable wrt black's reformatting.

I don't have any better suggestion for how to design this better apart from having the user explicitly tell us and defaulting to the highest version black supports.

edit: Maybe we could detect various features instead of python versions. The downside: an explosion of configuration options because people will (rightly so) want to turn them on even if the input doesn't use them.

[ambv]

Our problem is that we're sniffing trailing commas after kwargs in function calls to detect 3.6+ whereas 3.5 also supports this. In this case other 3.6+ formattings kick in and code breaks.

In your example 3.6+ will be discovered due to the trailing comma in the function signature so the function call is irrelevant.

If you're trying to say that if this trailing comma would be the only 3.6-mode feature available in the file then you're right, after removing sniffing based on this, it would be removed. This is a temporary problem that applies to Python 3.5 only. That's too bad but I don't want to add a "py35" mode for this alone. Today Python 3.5 in two releases behind. It will reach end-of-life in less 2 years (September 2020). At that point Python 3.8.0 will be already a year old.

As for whether we'd like 3.7+ or 3.8+ specific formattings: there are none at the moment. However, we can use new features like from __future__ import annotations or the walrus operator to discover 3.6+ mode just fine.

[jedie]

Because of this bug, black seems to be unusable, isn't it? Is there a work-a-round or must we wait for a new release?

EDIT: Hm, ok... After black has reformat the code and add the trailing comma after **kwargs then i can remove it and black will not add it again... Maybe until black will reformat the __init__() line ?!?

[ambv]

@jedie, consider how you word your feedback, you come across as unnecessarily heated and dramatic.

This problem affects people who used a Python 3.5-specific feature in their file which was erroneously considered a Python 3.6-specific feature by Black's 3.6+ auto-detection. This does affect users and we will fix it but it does affect a very small minority of people. The tool is perfectly usable for everybody else.

[penguinolog]

Also py36 = false for pyproject.toml not changes behavior: trailing comma added and compatibility with python 3.5 is broken

[asvetlov]

Hi.
Any progress on it?
I've tried to apply black to aiohttp.
Everything works fine except Python 3.5 is broken (SyntaxError) by trailing commas.