Skip to content

Assignment of Any does not get rid of Optional (add AnyUnion type?) #3526

Closed
@ddfisher

Description

@ddfisher
Collaborator

Consider this code (run with --strict-optional):

from typing import Optional
def f():
  # untyped function; often from silent import
  return 0  # always returns an int

def g(x: Optional[int]) -> int
  if x is None:
    x = f()
  return x  # E: Incompatible return value type (got "Optional[int]", expected "int")

Mypy doesn't infer that x is not None in g. I think this a bug -- mypy
should be permissive here.

If we wanted to fix this, we'd run into a problem: what type should x have
after x = f()? We don't want it to be Any, because we don't want that
behavior in the general case. It shouldn't be int -- f() could be intended
to unconditionally return None instead. To fix this, I think we need to
introduce a new special type, which I will call AnyUnion. AnyUnion[A, B, C] would represent a type which is allowed to be used as any of A, B, or
C (similarly to how Any is allowed to be used as any type). In this case,
we could have x's type bound to AnyUnion[int, None], which would fix this
error.

Other helpful uses of AnyUnion:

  • pow will no longer need to return Any in builtins. Instead, it could
    return the much less broad AnyUnion[int, float].
  • AnyUnion[X, None] would provide a per-variable opt-out of strict None
    checking. This could be helpful for classes with delayed initialization where
    a lot of effort would be needed to refactor them into a state that would work
    properly with strict None checking.

Cons:

  • Additional complexity of the type system affects implementation complexity and
    makes things slightly harder for users to understand.
  • The motivating problem also exists with subclasses (with the way the binder
    currently works), but AnyUnion doesn't help much with that.

Activity

ilevkivskyi

ilevkivskyi commented on Jun 13, 2017

@ilevkivskyi
Member

pow will no longer need to return Any in builtins. Instead, it could
return the much less broad AnyUnion[int, float].

But doesn't #3501 (or similar) provides a better solution?

AnyUnion[X, None] would provide a per-variable opt-out of strict None
checking. This could be helpful for classes with delayed initialization where
a lot of effort would be needed to refactor them into a state that would work
properly with strict None checking.

Do I understand correctly that this has been solved by PEP 526?

In general, I don't think that we really need this feature. What worries me is that in addition to making type system more complex it also can break transitivity of subtyping (or maybe better call this compatibility). IIUC your proposal, then int <: AnyUnion[int, str] <: str. Currently, we have only one special type that does this -- Any. Having many (user defined) types that break transitivity might lead to unexpected consequences.

ilevkivskyi

ilevkivskyi commented on Jun 13, 2017

@ilevkivskyi
Member

Do I understand correctly that this has been solved by PEP 526?

Probably it was only partially fixed by PEP 526. But for remaining cases one can just do something like:

UNINITIALIZED: Any = None

class C:
    def __init__(self) -> None:
        self.complicated: int = UNINITIALIZED
        # existing code below
        while self.complicated is None:
            ...
gvanrossum

gvanrossum commented on Jun 13, 2017

@gvanrossum
Member

I'm not sure this is needed all that often, but if we do, we could just spell it Any -- Any[A, B, C] is acceptable as a A, B or C, while plain Any is unchanged. Alternatively, maybe this type is really the same as Intersection (python/typing#213)?

Regardless I'm not sure that in your original motivating example we should do anything different. If you want to signal at the call site that f() returns an int, you can write

def g(x: Optional[int]) -> int
  if x is None:
    y = f()  # type: int
    x = y
  return x  # OK
JukkaL

JukkaL commented on Jun 13, 2017

@JukkaL
Collaborator

This is probably the same feature as 'unsafe unions' that have been discussed before (though I can't find the discussion -- maybe it was in person or over email). Unsafe unions are different from Intersection. For example, unlike intersections, compatibility goes both ways: int is compatible with AnyUnion[int, str] and vice versa.

My take is that this feature would complicate mypy significantly. Normal union types are already a very complicated feature, and they still aren't even fully implemented. It would be a likely cause of some user confusion, since we'd have two pretty similar but different concepts -- union types and these any/unsafe unions. If we decide to add intersection types at some point, there would be even more potential for confusion, since they are also similar in some ways to any unions. All in all, I don't think that there is enough evidence of the usefulness of the feature to support implementing it.

changed the title [-]Add `AnyUnion` type[/-] [+]Assignment of Any does not get rid of Optional (add `AnyUnion` type?)[/+] on Jun 23, 2017
JukkaL

JukkaL commented on Jun 23, 2017

@JukkaL
Collaborator

Solving the original issue is a priority because this may come up often when using ignored imports. I'm not convinced that adding AnyUnion is the right course of action, though.

ilevkivskyi

ilevkivskyi commented on Jun 23, 2017

@ilevkivskyi
Member

Concerning the original example:

from typing import Optional
def f():
  # untyped function; often from silent import
  return 0  # always returns an int

def g(x: Optional[int]) -> int
  if x is None:
    x = f()
  return x

what is wrong with x having type Union[Any, int] after the if block? Thus seems logical, initially it was Union[None, int] but then the None is replaced by Any in the if block by x = f(). Fortunately, such unions are not simplified anymore:

x: Union[int, Any]
reveal_type(x) # Revealed type is 'Union[builtins.int, Any]'
x.whatever # Error: Item "int" of "Union[int, Any]" has no attribute "whatever"

Also, Union[int, Any] is a (non-proper) subtype of int so that there should be no error in the original example.

ddfisher

ddfisher commented on Jun 23, 2017

@ddfisher
CollaboratorAuthor

@ilevkivskyi: That doesn't work, as variables don't change type in general. In particular, we want to maintain this behavior:

x = 0  # type: int
x = any()
reveal_type(x)  # int
ilevkivskyi

ilevkivskyi commented on Jun 23, 2017

@ilevkivskyi
Member

@ddfisher

as variables don't change type in general

But there is the type binder that can already do this:

x: Optional[int]

if x is None:
    x = int()

reveal_type(x)  # Revealed type is 'builtins.int'

Why can't it be special-cased to re-bind to Any inside unions? (I agree that binding to Any in general is clearly bad.)

ddfisher

ddfisher commented on Jun 23, 2017

@ddfisher
CollaboratorAuthor

Unless there's a really good reason, Unions should work as similarly as possible to non-Unions. I don't think this is a strong enough reason.

The binder only binds to subtypes of the declared type. If we change that, the declared type starts to become pretty meaningless.

gvanrossum

gvanrossum commented on Jun 23, 2017

@gvanrossum
Member

34 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @ddfisher@msullivan@JukkaL@gvanrossum@ilevkivskyi

      Issue actions

        Assignment of Any does not get rid of Optional (add `AnyUnion` type?) · Issue #3526 · python/mypy