Skip to content

Log rendered profile contents when profile caching is enabled #2739

Description

@tatiana

Description

Cosmos converts an Airflow connection into a dbt profiles.yml via the configured profile mapping. The contents of the rendered profile are only logged at INFO when profile caching is disabled — when caching is enabled (the default, since enable_cache_profile = True), only the cache path is logged, not the contents.

This is a meaningful diagnostic gap: a substantial class of "Cosmos generates the wrong profile" or "Cosmos drops a connection field" issues can only be confirmed by inspecting the rendered file, which requires either:

  1. Toggling AIRFLOW__COSMOS__ENABLE_CACHE_PROFILE=False and re-running, or
  2. Shelling into the worker and cat-ing the file under /tmp/cosmos/profile/<hash>/profiles.yml.

Both are workable but slow, and on managed/serverless deployments (1) requires a redeploy and (2) is often impossible.

Where this happens

RenderConfig.ensure_profile (around lines 373-411 at the time of writing):

  • Non-cached branch logs the full contents via logger.info("... with the following contents:\n%s", profile_contents).
  • Cached branch logs only "Profile not found in cache storing and using profile: %s." — the contents are silently elided.

Proposed fix

Log the rendered profile contents on the cached path too, with secret fields redacted (e.g., replace values for fields listed in the mapping's secret_fields with ***). The log line should appear once per cache miss, at the same level as the non-cached branch (INFO).

Why this matters

Without this, every Cosmos profile-mapping troubleshooting cycle requires a back-and-forth with the user to either flip a config flag or pull the file off the worker. Adding the redacted contents to the standard log surfaces them immediately and saves a round-trip in support / debugging contexts.

Acceptance criteria

  • When profile caching is enabled and a cache miss occurs, the rendered profiles.yml contents are logged at INFO.
  • Values for fields in the mapping's secret_fields are redacted before logging.
  • The existing non-cached branch behaviour is unchanged.
  • Cache hits may also log the path (already happens) — but logging contents on a hit is optional.

Metadata

Metadata

Assignees

Labels

bugSomething isn't workinggood first issueGood for newcomersstaleIssue has not had recent activity or appears to be solved. Stale issues will be automatically closed

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions