[Bug] dbt ls cache key ignores requested --output-keys, so upgrades/config changes can silently serve stale cached node data

[tatiana]

Description

Cosmos's dbt ls result cache (DbtGraph.should_use_dbt_ls_cache/save_dbt_ls_cache in cosmos/dbt/graph.py, keyed via cache._calculate_dbt_ls_cache_current_version in cosmos/cache.py) decides whether to reuse a previously-cached dbt ls --output json result by hashing only:

• the dbt project folder's contents (_create_folder_version_hash), and
• dbt_ls_cache_key_args: select/exclude/vars/selector/--no-partial-parse (dbt_ls_args), plus env vars, plus a configurable allowlist of Airflow Variables (RenderConfig.airflow_vars_to_purge_dbt_ls_cache).

Nothing in that hash reflects: (a) which --output-keys Cosmos actually requested from dbt ls, (b) the Cosmos package version, or (c) RenderConfig.source_rendering_behavior, which independently controls whether --output-keys is passed at all (run_dbt_ls's specify_output_keys logic in cosmos/dbt/graph.py).

This means two distinct scenarios silently return stale node data instead of re-running dbt ls:

1. Config-driven (already possible today): flipping RenderConfig.source_rendering_behavior between NONE and any other value changes whether --output-keys is passed to dbt ls at all (with vs without Cosmos's explicit key list), but that option is not part of dbt_ls_cache_key_args, so an existing cache entry keeps being served across the flip.
2. Upgrade-driven: if a future Cosmos release changes the --output-keys list itself (for example, [Feature] Unify Asset/Dataset URI construction across Cosmos execution modes #2959 proposes adding database/schema/alias/relation_name) or changes how parse_dbt_ls_output interprets the JSON, a user who upgrades Cosmos but doesn't also touch their dbt project files, select/exclude/vars, or the specific Airflow Variables in airflow_vars_to_purge_dbt_ls_cache keeps being served their pre-upgrade cached output - silently missing whatever the new Cosmos version expected to be there.

Use case/motivation

Surfaced while scoping #2959 (extending the requested --output-keys to carry relation identity): shipping that change alone would leave existing users' caches silently stale post-upgrade, with no error and no changed behavior until an unrelated cache-bust event (a project file edit, a different --select, etc.) happens to occur. The underlying gap is general - any future change to what Cosmos asks dbt ls for, or how it parses the response, has the same exposure, and the source_rendering_behavior case shows this is not purely hypothetical.

Proposed approach (sketch, open for discussion)

Fold something that changes whenever Cosmos's own expectations of the dbt ls output change into dbt_ls_cache_key_args (or a new input to _calculate_dbt_ls_cache_current_version), so the cache version changes independent of the user's project files or select/exclude/vars. Two options:

• Include the actual resolved --output-keys list (whatever run_dbt_ls decided to pass, accounting for specify_output_keys/source_rendering_behavior) directly in the cache-key inputs, so any future change to that list busts the cache automatically without a contributor needing to remember to update anything.
• Or maintain an explicit, manually-bumped "dbt-ls cache schema version" constant in cosmos/cache.py, incremented whenever run_dbt_ls's command construction or parse_dbt_ls_output's parsing changes, and fold it into the hash. Simpler to reason about, but relies on contributors remembering to bump it.

The first option is more robust against future omissions but needs confirming that including the output-keys list is cheap enough to compute on every parse.

Related issues

• [Feature] Unify Asset/Dataset URI construction across Cosmos execution modes #2959 - the specific change that would be silently unsafe without this fix

Are you willing to submit a PR?

• Yes, I am willing to submit a PR!