Description
Cosmos converts an Airflow connection into a dbt profiles.yml via the configured profile mapping. The contents of the rendered profile are only logged at INFO when profile caching is disabled — when caching is enabled (the default, since enable_cache_profile = True), only the cache path is logged, not the contents.
This is a meaningful diagnostic gap: a substantial class of "Cosmos generates the wrong profile" or "Cosmos drops a connection field" issues can only be confirmed by inspecting the rendered file, which requires either:
- Toggling
AIRFLOW__COSMOS__ENABLE_CACHE_PROFILE=False and re-running, or
- Shelling into the worker and
cat-ing the file under /tmp/cosmos/profile/<hash>/profiles.yml.
Both are workable but slow, and on managed/serverless deployments (1) requires a redeploy and (2) is often impossible.
Where this happens
RenderConfig.ensure_profile (around lines 373-411 at the time of writing):
- Non-cached branch logs the full contents via
logger.info("... with the following contents:\n%s", profile_contents).
- Cached branch logs only
"Profile not found in cache storing and using profile: %s." — the contents are silently elided.
Proposed fix
Log the rendered profile contents on the cached path too, with secret fields redacted (e.g., replace values for fields listed in the mapping's secret_fields with ***). The log line should appear once per cache miss, at the same level as the non-cached branch (INFO).
Why this matters
Without this, every Cosmos profile-mapping troubleshooting cycle requires a back-and-forth with the user to either flip a config flag or pull the file off the worker. Adding the redacted contents to the standard log surfaces them immediately and saves a round-trip in support / debugging contexts.
Acceptance criteria
- When profile caching is enabled and a cache miss occurs, the rendered
profiles.yml contents are logged at INFO.
- Values for fields in the mapping's
secret_fields are redacted before logging.
- The existing non-cached branch behaviour is unchanged.
- Cache hits may also log the path (already happens) — but logging contents on a hit is optional.
Description
Cosmos converts an Airflow connection into a dbt
profiles.ymlvia the configured profile mapping. The contents of the rendered profile are only logged at INFO when profile caching is disabled — when caching is enabled (the default, sinceenable_cache_profile = True), only the cache path is logged, not the contents.This is a meaningful diagnostic gap: a substantial class of "Cosmos generates the wrong profile" or "Cosmos drops a connection field" issues can only be confirmed by inspecting the rendered file, which requires either:
AIRFLOW__COSMOS__ENABLE_CACHE_PROFILE=Falseand re-running, orcat-ing the file under/tmp/cosmos/profile/<hash>/profiles.yml.Both are workable but slow, and on managed/serverless deployments (1) requires a redeploy and (2) is often impossible.
Where this happens
RenderConfig.ensure_profile(around lines 373-411 at the time of writing):logger.info("... with the following contents:\n%s", profile_contents)."Profile not found in cache storing and using profile: %s."— the contents are silently elided.Proposed fix
Log the rendered profile contents on the cached path too, with secret fields redacted (e.g., replace values for fields listed in the mapping's
secret_fieldswith***). The log line should appear once per cache miss, at the same level as the non-cached branch (INFO).Why this matters
Without this, every Cosmos profile-mapping troubleshooting cycle requires a back-and-forth with the user to either flip a config flag or pull the file off the worker. Adding the redacted contents to the standard log surfaces them immediately and saves a round-trip in support / debugging contexts.
Acceptance criteria
profiles.ymlcontents are logged at INFO.secret_fieldsare redacted before logging.