Allow variadic generics

[NYKevin]

C++11 recently introduced the notion of variadic templates, which I believe Python could benefit from in a simplified form.

The idea is that you can have a generic class with a variable number of type variables, like typing.Tuple[] has. Here is a real-world variadic-generic class which is not Tuple; it is variadic in TagVar. As you can see, TagVar only appears in tuple contexts. Those tuples are sometimes heterogenous (Caution: annotations are a homegrown 3.4-compatible mishmash of nonsense), so repeating TagVar as shown is actually incorrect (but the closest approximation I could find).

Here's one possible syntax:

class MultiField(AbstractField[GetSetVar], Generic[(*TagVar,)]):
    def __init__(self, nbt_names: ty.Sequence[str], *, default:
                 GetSetVar=None) -> None:
        ...

    @abc.abstractmethod
    def to_python(self, *tags: (*TagVar,)) -> GetSetVar:
        ...

    @abc.abstractmethod
    def from_python(self, value: GetSetVar) -> ty.Tuple[(*TagVar,)]:
        ...

This is syntactically valid in Python 3.5 (if a bit ugly with the parentheses and trailing comma, which cannot be omitted without language changes), but doesn't currently work because type variables are not sequences and cannot be unpacked. It could be implemented by adding something like this to the TypeVar class:

def __iter__(self):
    yield StarredTypeVar(self)

StarredTypeVar would be a wrapper class that prefixes the repr with a star and delegates all other functionality to the wrapped TypeVar.

Of course, syntax isn't everything; I'd be fine with any syntax that lets me do this. The other immediately obvious syntax is to follow the TypeVar with an ellipsis, which conveniently does not require changes to typing.py. However, that might require disambiguation in some contexts (particularly since Tuple is likely to be involved with these classes).

[gvanrossum]

Hmm... why not make this a special property of type variables.Tthen you wouldn't need the funny syntax, you could just use a type variable that has a keyword saying it is variadic.

But maybe the bigger question is how exactly we should type check this.

[NYKevin]

Hmm... why not make this a special property of type variables.Tthen you wouldn't need the funny syntax, you could just use a type variable that has a keyword saying it is variadic.

This isn't obviously objectionable, but for the following reasons I'm not sure it's actually a good idea. First, I would like to point out that I failed to notice a third possible syntax, which doesn't require the trailing comma:

class MultiField(AbstractField[GetSetVar], Generic[[*TagVar]]):
    def __init__(self, nbt_names: ty.Sequence[str], *, default:
                 GetSetVar=None) -> None:
        ...

    @abc.abstractmethod
    def to_python(self, *tags: [*TagVar]) -> GetSetVar:
        ...

    @abc.abstractmethod
    def from_python(self, value: GetSetVar) -> ty.Tuple[[*TagVar]]:
        ...

I'll return to this syntax in a moment.

But maybe the bigger question is how exactly we should type check this.

We should enforce that the variable only appears as the sole argument to Tuple (with a trailing ellipsis?), as a variable of a generic class, possibly (?) in the instantiation of a generic class other than Tuple, or as the type of a variadic parameter (*args, **kwargs). The generic class case is complicated, however, because there might be multiple variadic type variables in play under multiple inheritance. Even if there aren't, it could be difficult to parse.

We can overcome both problems by requiring specialized syntax when the generic class is subclassed or instantiated:

class UUIDField(MultiField[uuid.UUID, [tags.LongTag, tags.LongTag]]):
    ...

Note the extra pair of brackets. They indicate where the variadic typevar begins and ends, which removes any ambiguity when there are multiple variadic typevars, and simplifies parsing when there are non-variadic typevars in the same class (as in this case).

For reasons of uniformity, I would recommend we use the bracket syntax I showed above rather than making variadic-ness a property of the type variable. That makes instantiation and declaration look more like one another, and seems more intuitive to me.

[JukkaL]

Variadic generics would likely be useful at least occasionally. It would make sense to generalize them also to argument types in Callable. I remember that there are a few places in the std lib stubs where these would have let us have a more precise type for a library function. The canonical example might be something like this (in my counter-proposal being variadic is a property of a type variable, as it seems more readable to me):

Ts = TypeVar('Ts', variadic=True)
RT = TypeVar('RT')

def call_async(function: Callable[Ts, RT], args: Tuple[Ts]) -> Future[RT]:
    ...

In my proposal a variadic type variable Ts would be valid in at least these contexts:

• As Tuple[Ts] (the only argument to Tuple)
• As Callable[Ts, X] (the first argument to Callable)
• As type of *args, such as *args: Ts

It's less obvious how to generalize this to user-defined generic types. Maybe like this:

class C(Generic[Ts]):
    def __init__(self, *x: Ts) -> None: ...

c = C(1, '')  # type: C[int, str]   # ok!

The limitation would be that in order to have both variadic and non-variadic arguments, you'd have to create a dummy wrapper type for the variadic args:

class ArgTypes(Generic[Ts]): pass   # dummy wrapper type

class C(Generic[T, ArgTypes[Ts]]):
    def __init__(self, t: T, *args: Ts) -> None: ...

The apparent awkwardness of the above example probably wouldn't matter much as I expect this use case to be very rare.

There are additional implementation details that a type checker should probably get right to make this useful, but I'm not even trying to enumerate them here. The implementation would likely be tricky, but probably not excessively so.

I'm still unconvinced that this is a very important feature -- look at how long it took C++ to pick up this feature. I'd propose looking for more use cases to justify variadic generics and once we have a sufficient number of use cases collected here, we'd send this to python-ideas@ for discussion.

[JukkaL]

Examples where these could be useful:

• zip
• contextlib.contextmanager

[sixolet]

@JukkaL There's one extension to your proposal that would solve a problem I've been trying to figure out: I have some Query objects, which vary in the types of results they produce. I also have an execute function, which takes in some Querys and would like to return a tuple, with one result per Query it was passed.

R = TypeVar('R')
class Query(Generic[R]):
   ...

Rs = TypeVar('Rs', variadic=True)
Qs = Rs.map(Query) 

def execute(*args: Qs) -> Tuple[Rs]:
   ...

I am not at all wed to this syntax for mapping over variadic types variables, but I want to be able to do it, and I think it should be a thing we should consider when considering variadic type variables.

[gvanrossum]

It's not very clear from that example that execute() returns a tuple. I think Jukka's proposal would have you write Tuple[Rs] for the return type. I also wish the map() functionality was expressible as part of the signature of execute() rather than in the definition of Qs.

Anyway, this would be shorthand for an infinite sequence of definitions for execute(), like this, right?

R = TypeVar('R')
R1 = TypeVar('R1')
R2 = TypeVar('R2')
R3 = TypeVar('R3')
# etc. until R999
@overload
def execute(q1: Query[R]) -> R: ...
@overload
def execute(q1: Query[R1], q2: Query[R2]) -> Tuple[R1, R2]: ...
@overload
def execute(q1: Query[R1], q2: Query[R2], q3: Query[R3]) -> Tuple[R1, R2, R3]: ...
# etc. until a total of 999 variants

NOTE: The special case for the first overload is not part of the special semantics for variadic type variables; instead there could be two overload variants, one for a single query, one for 2 or more.

[sixolet]

Yes, execute returns a Tuple[Rs] and I simply was posting too quickly to check my work, apologies.

[sixolet]

And yes, this would be exactly that infinite series of overloads.

[sixolet]

Making the type mapping in the argument list would be nice, but requires some care as to exactly where you are parameterizing a type over each element of your variadic type variable, and where you are parameterizing a type using all elements of your variadic type variable.

For example, I'd love to be able to write Query[Rs] and mean "a variadic type variable that is a query type for each result type in Rs" but then I want to also be able to write something like Tuple[Rs] meaning "a tuple of every return type in Rs".

One possibility is, borrowing some stars and parens from @NYKevin to have, I think, somewhat of a different meaning:

def execute(*args: Query[Rs]) -> Tuple[(*Rs,)]:
    ...

[gvanrossum]

So maybe the notation ought to reflect that, and we should be able to write

Rn = TypeVar('Rn', variadic=True)
def execute(*q1: Query[Rn.one]) -> Tuple[Rn.all]: ...

Where .one and .all try to give hints on how to expand these.

[sixolet]

Ooh, I like not having it be some kind of obtuse operator but rather english words. Consider each as a possibility to mean what you're using one for above?