Skip to content

Manifest load mode: graph operator "+" only resolves exact node names, not folder/package/fqn roots #2962

Description

@pankajkoti

Environment

Field Value
Astronomer Cosmos version 1.15.1
dbt Core version 1.12.0
dbt adapter versions Not applicable (rendering from a committed manifest.json, no adapter invoked)
LoadMode DBT_MANIFEST
ExecutionMode LOCAL (rendering only, not relevant to the bug)
InvocationMode Not applicable
Airflow version 3.3.0
Python 3.11
Operating System macOS (Darwin)

Summary

In LoadMode.DBT_MANIFEST, a selector that combines the graph operator (+) with a bare folder name or package name (for example my_folder+ or my_package+) resolves to an empty set. Cosmos resolves the graph-operator root by exact node name only, so folder and package tokens match nothing and the select or exclude silently has no effect. The same selectors work under dbt ls, which delegates resolution to dbt.

For exclude this is a correctness issue. The intended nodes are not removed and remain in the rendered DAG.

Background: what #2357 addressed

PR #2357 ("Fix inclusion of package models, and selecting/excluding") added manifest-mode support for:

  • Including installed-package models in the graph with correct file paths.
  • package:<name> select and exclude.
  • Bare identifiers resolving the dbt way (by package name, node name, or folder name).
  • Select and exclude by folder name.

All of the above work. The gap below is the graph-operator (+) case layered on top of the folder and package tokens.

What still needs addressing

Graph-operator root resolution should match dbt. The token attached to + should resolve through the same methods as a bare identifier (node name, folder name, package name, fqn, path), not by exact node name alone.

How to reproduce

Project: fhir-dbt-analytics (public), loaded from its target/manifest.json with LoadMode.DBT_MANIFEST.

from cosmos.config import ProjectConfig, RenderConfig, ExecutionConfig
from cosmos.constants import LoadMode
from cosmos.dbt.graph import DbtGraph

def n_models(selector):
    g = DbtGraph(
        project=ProjectConfig(dbt_project_path=DBT_DIR, manifest_path=DBT_DIR / "target" / "manifest.json"),
        render_config=RenderConfig(load_method=LoadMode.DBT_MANIFEST, select=[selector]),
        execution_config=ExecutionConfig(dbt_project_path=DBT_DIR),
    )
    g.load(method=LoadMode.DBT_MANIFEST)
    return sum(1 for x in g.filtered_nodes.values() if x.resource_type.value == "model")

Results:

kind selector models selected status
node name metric_definition 1 PASS
node + descendants metric_definition+ 7 PASS
folder name post_processing 6 PASS
folder + descendants post_processing+ 0 FAIL
path path:models/post_processing 6 PASS
path + descendants path:models/post_processing+ 11 PASS
bare package fhir_dbt_utils 25 PASS
bare package + descendants fhir_dbt_utils+ 0 FAIL
package selector package:fhir_dbt_utils 25 PASS
package selector + descendants package:fhir_dbt_utils+ 182 PASS

So <node>+, path:...+, and package:...+ all work. Only <folder>+ and <package>+ (a bare non-node token plus the operator) return nothing.

Root cause

In cosmos/dbt/selector.py, GraphSelector resolves its root through a node_by_name[node.name] lookup (exact node name). If the token is not a node name it logs Selector <x> not found and returns without selecting anything. Explicit method prefixes inside a graph selector (path:, package:) are handled, which is why path:...+ and package:...+ work, but bare folder, package, and fqn tokens are not. The non-graph bare-identifier path already resolves folder and package names, so the two paths are inconsistent.

Impact

Any <folder>+ or <package>+ selector in manifest mode silently matches nothing, which diverges from dbt ls. On exclude the targeted nodes are not excluded.

Workaround

Add an explicit method prefix so the graph selector can resolve the root. Use package:<name>+ or path:<folder>+ instead of the bare <name>+.

Suggested fix

When the GraphSelector root token is not an exact node name, fall back to the same resolution the non-graph bare-identifier path already uses (folder name, package name, fqn), so <folder>+ and <package>+ behave as they do under dbt. Adding manifest-mode selector tests for these two shapes would lock in the parity.

Are you willing to submit a PR?

Yes, I am willing to submit a PR.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions