Handle "impossible" intersections

[CarliJoy]

This issue is about handling an "impossible" intersection by static type checkers. We consider two cases:

It must be transformed to Never

Arguments in favor:

• @mikeshardmind argues that is unsafe.
• @gandhis1 argues not doing so creates divergent behavior between typecheckers
• @mikeshardmind argues not doing so creates divergent behavior between typecheckers in a different way in a related case

It is up to the type checker to reduce it or not

Arguments in favor:

• @erictraut argues that the conditions are complex and expensive.
• @erictraut argues in the context of reduction and normalization.
• @CarliJoy argues that there are problems caused by inconsistent typing definitions in the standard library.

(Please edit this comment. I haven't gone through the whole issue to add links to everyone's arguments.)

[mikeshardmind]

I don't really think this needs its own issue. We can detect some cases where an intersection is impossible to satisfy, and that should result in Never, I'm not sure why this would be something requiring any amount of bikeshedding.

[CarliJoy]

@mikeshardmind I already wrote an algorithm for protocols in the current specification under https://github.com/CarliJoy/intersection_examples/blob/main/specification.rst#protocol-reduction

Could you adopt this algorithm to accompany this?

There is already a section with evaluation to never, we could add things there, too.

But in this case I would like to define also, that type checkers should deliver a good error msg. Otherwise finding these issues will be pain.

[mikeshardmind]

I'm not in a position to do anything formal at the moment, but that said I already posted a partial list of "trivial" reductions in the discord server, I'll post them here as well.

Note, removal of Any was included prior to some people finding that controversial, don't read into it as a mandate that this approach is the resolution

Trivial Removal of Any
Any & P -> P
Any & P & Q -> P & Q

Subtype relationship example
(copied directly from draft, already included)
(Note: examples for Any above also follow this rule of more specific types taking precedence)

Intersection[str, Employee, Manager] is Intersection[str, Manager]

concrete types which fully satisfy protocols

SupportsIndex & int -> int

Reductions involving None

None & ConcreteType -> Never
None & ProtocolNoneDoesntSatisfy -> Never
None & ProtocolNoneSatisfies -> None (ie. bool)

Interaction with other Final types

(Note: None follows all of these rules, no special casing of None)

FT is a final type
NBFT is another type that isn't in FT's bases
BFT is a type that is in FT's bases
SFTP is a protocol (heh) that FT satisfies
NSFTP is a protocol that FT does not satisfy

FT & NBFT -> Never
FT & BFT -> FT
FT & NSFTP -> Never
FT & SFTP -> FT

---

In addition to this post, there's also if you have two types with incompatible interfaces -> Never as it would be impossible to satisfy both requirements requested by use of the intersection. I don't think artificially creating overloads works, for reasons above, complexity, and so on. It also behaves more like a Union at that point, so if you want a Union, just use a Union.

[CarliJoy]

I already included them in the specification. Did you have look at it?

[CarliJoy]

In addition to this post, there's also if you have two types with incompatible interfaces -> Never as it would be impossible to satisfy both requirements requested by use of the intersection. I don't think artificially creating overloads works, for reasons above, complexity, and so on. It also behaves more like a Union at that point, so if you want a Union, just use a Union.

That's where we disagree.
Creating overloads is actually one use case for intersections. I.e. think of Callable[[int], str] & Callable[[float], [str].
People already said that would expect an overload there.

Of course the special cases about staticmethods and classmethods are non trivial and need some evaluation.

[mikeshardmind]

I would expect this to be an error. If I wanted that, the correct tool is a Union, not an intersection.

Callable[[int], str] | Callable[[float], str] has the behavior you are putting into the intersection here by creating overloads artifically.

[mikeshardmind]

The specification is also stricter on allowable type combinations than what is required by python. It is possible to have multiple inheritance from multiple builtin types without it being in error. This is why my post was specific to mention type compatibility as well as if a class was marked as being final

[CarliJoy]

The specification is also stricter on allowable type combinations than what is required by python. It is possible to have multiple inheritance from multiple builtin types without it being in error. This is why my post was specific to mention type compatibility as well as if a class was marked as being final

That is not correct, see https://github.com/CarliJoy/intersection_examples/blob/main/test_basetypes.py I didn't find any combination that is valid.

[mikeshardmind]

I should have been so much more specific. There is no rule in the interpreter or at runtime that multiple inheritance from builtin classes is impossible. The specification implies this off it being true for the current types there, but one can imagine that it may not remain true. It should be sufficient, more succinct, and more permanently correct to say that A & B is not valid for concrete types if class(A, B) is not valid.

[mikeshardmind]

Anyhow, if you are resolving this by adding overloads, I'm no longer interested in intersections. I consider this to be antithetical to why I want an intersection in python. Everything else there's been disagreement about I could see multiple paths out of, but this would result in me just not using the feature ever. It would be more correct to just hand out protocols that specify exactly my intent at that point.

If I specify A & B where A is mine, and B is a library's, I expect that A & B are out-of-the-box not incompatible interfaces. Hiding the incompatibility with artificially inserted overloads that then have to be checked for an outcome when used breaks expectations and some of the reasons for Intersections. I want to be able to use this as if it is A and as if it is B. If I wanted to check which interface I had, I wouldn't need an intersection.

[CarliJoy]

Well the correct definition would be A & B is never if class (A, B) or class(B, A) raises TypeError.

There are two Problems with this definition:

1. Performance: You need to check all permutations of all arguments
2. It requires running the code what a type checker never should do (strongest arguemt)
3. How should type checker not implemented in python know this? And there are even multiple implementations of python next to CPython. So maybe in one of this implementation class (int, float) maybe even works. So typing will now depend on the interpreter?

[mikeshardmind]

The type error here is predictable and arises from a base class layout conflict. This can be statically determined and does not require running the code.

All permutations of arguments should be checked, it shouldn't be possible to create an impossible requirement.

Realistically, though, this goes back to what was brought up in discord by @DiscordLiz that even maintainers of type checkers thought intersections on anything other than protocols would be too complex.

[DiscordLiz]

pragmatically speaking, it's worth just disallowing multiple inheritance of builtin types. The manner in which they are implemented would need to change for this to no longer be true, and other implementations could allow it.

you do need to verify that a combination of types is possible, which would be checking that there exists at least 1 valid permutation of bases.

Really should just have intersections be for fully compatible protocols only. It's simple, and and solves a real problem. All of these edge cases come in from things nobody should ever need to care about in practice, but for it to be defined robustly and allow more than protocols, do.

[DiscordLiz]

I read through the current specification, and I need to state more than just simple agreement that the overloads are a mistake. This creates a situation where intersections do something that could never be valid on real classes for protocols, and only for protocols. Protocols are already only a definition structurally, I would expect any incompatibility to be a mistake on the part of whoever requested that something works with both.

You can't hack this together for real classes, because it would become an LSP violation. This breaks symmetry between structural and nominal typing.