Skip to content

incremental: Initial "always dirty" queries #44234

@alexcrichton

Description

@alexcrichton
Member

In working on #44142 I've come across the need/desire a few times to have "always dirty" nodes that are always recomputed at the base of the dependency graph. One example of this is handling today's extern_mod_stmt_cnum method.

Today there's a method CrateStore::extern_mod_stmt_cnum, but as part of #41417 we want to move this to a query. That's relatively simple but what's actually happening here I think is a bit more subtle in terms of dependencies. This query will take the ID of an extern crate statement and return the CrateNum that it loaded. This happens very early in the compiler when we're loading crates and it's basically "however CrateStore is implemented picks these numbers".

Right now, on my currently unmerged branch, the implementation looks like this:

        extern_mod_stmt_cnum: |tcx, id| {
            let id = tcx.hir.definitions().find_node_for_hir_id(id);
            tcx.sess.cstore.extern_mod_stmt_cnum_untracked(id)
        },

So in other words this is just defining a query that doesn't actually have any dependencies! The extern_mod_stmt_cnum_untracked method just reaches into the internals of CStore and plucks a CrateNum seemingly out of thin air, returning it.

Ideally I think what we'll want here is a way of saying that queries like this need to be computed 100% of the time to determine if they're red or green. Typically they'll instantly turn green again which'll allow us to have lots of cache hits, but I'm worried about assumign they're always green because they have no inputs.

I believe this is a similar-ish issue to many of the maps in #44137 as well. For all the maps calculated in resolve then then hidden behind a query in TyCtxt those nodes don't actually have any dependencies, they're just reading internal tables. We should always rerun the query though to ensure that it is properly tracked!

cc @nikomatsakis, @michaelwoerister

Activity

added
C-enhancementCategory: An issue proposing an enhancement or a PR with one.
T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.
on Sep 1, 2017
michaelwoerister

michaelwoerister commented on Sep 4, 2017

@michaelwoerister
Member

Yes, I agree. I think what @nikomatsakis calls "constant data" in #44137 is really "input data" as far as the query system is concerned. Inputs have no dependencies, they always need to be read and checked for changes.

michaelwoerister

michaelwoerister commented on Sep 4, 2017

@michaelwoerister
Member

Right now a DepNode can be marked as [anon]. One way to handle this would be to add a [input] modifier that informs the query system that it must treat the dependency differently.

michaelwoerister

michaelwoerister commented on Sep 4, 2017

@michaelwoerister
Member
nikomatsakis

nikomatsakis commented on Sep 5, 2017

@nikomatsakis
Contributor

Yes so I'm in favor of this. I think roughly speaking the idea of an "input" query would just be that it's a query where we can never re-use the results. Seems like a simple thing for us to add and it will no doubt remain useful until such time as we push the query system all the way throughout the compiler (and maybe even then...).

(For example, it could eventually subsume the existing handling of HIR nodes -- i.e., we could just have a hir(def-id) query that returns the data, and let the natural hashing take care of it.)

@michaelwoerister this does suggest that perhaps moving towards a lazy model of deciding when (e.g.) an input HIR node has changed might be better? i.e., closer to this scheme?

michaelwoerister

michaelwoerister commented on Sep 5, 2017

@michaelwoerister
Member

this does suggest that perhaps moving towards a lazy model of deciding when (e.g.) an input HIR node has changed might be better? i.e., closer to this scheme?

I think so, yes.

alexcrichton

alexcrichton commented on Sep 13, 2017

@alexcrichton
MemberAuthor

Niko just had an excellent idea. Let's add an always red query called "Input", and then any other query that wants to always run will simply depend on this.

eddyb

eddyb commented on Sep 13, 2017

@eddyb
Member

Just to be clear: having the compiler arguments and files read during parsing/macro expansion would serve the same purpose, if we can sneak all of that into queries? If so I'm fine with "Input".

michaelwoerister

michaelwoerister commented on Sep 14, 2017

@michaelwoerister
Member

Yet, they are not always red :) An input can be green too, it's just that we cannot rely on their (non-existing) dependencies for determining if they are. It would rather have to be a "always re-compute" query.

michaelwoerister

michaelwoerister commented on Dec 6, 2017

@michaelwoerister
Member

❤️ This has been implement in #45353 by @wesleywiser

added a commit that references this issue on Jan 7, 2019
c89b07a
added a commit that references this issue on Oct 20, 2024
added a commit that references this issue on Oct 20, 2024
a860657
added a commit that references this issue on Oct 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    A-incr-compArea: Incremental compilationC-enhancementCategory: An issue proposing an enhancement or a PR with one.T-compilerRelevant to the compiler team, which will review and decide on the PR/issue.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @alexcrichton@eddyb@nikomatsakis@shepmaster@michaelwoerister

        Issue actions

          incremental: Initial "always dirty" queries · Issue #44234 · rust-lang/rust