Skip to content

Improve profile mapping error when a required field is missing from the Airflow connection #2968

Description

@pankajastro

When a profile mapping's required secret is absent from the Airflow connection, Cosmos fails with:

cosmos.exceptions.CosmosValueError: Could not find a value for secret field password.

The message names neither the profile mapping, the connection, nor where Cosmos looked (cosmos/profiles/base.py:238), so users read it as a stale manifest or a cached profile rather than a connection/mapping mismatch.

A common trigger: a Snowflake connection is switched from user/password to key-pair auth, while ProfileConfig still uses SnowflakeUserPasswordProfileMapping — which requires password (cosmos/profiles/snowflake/user_pass.py:24-34) and resolves it from profile_args["password"], then conn.password (base.py:288-317). Both are empty after the switch, and the resulting traceback points at env_vars rather than at the connection.

Suggested message

SnowflakeUserPasswordProfileMapping could not resolve the required dbt profile field 'password'.
Cosmos looked in ProfileConfig(profile_args={'password': ...}) and in Airflow connection
'snowflake_prod' (connection field 'password') — both are empty. Check that the connection exists
and has 'password' set. If it uses a different authentication method, switch ProfileConfig to the
matching mapping — other mappings for conn_type 'snowflake':
SnowflakeEncryptedPrivateKeyFilePemProfileMapping, SnowflakeEncryptedPrivateKeyPemProfileMapping,
SnowflakePrivateKeyPemProfileMapping, SnowflakePrivateKeyFilePemProfileMapping.

The useful additions are: the mapping class name, the conn_id, the connection field(s) consulted via airflow_param_mapping, and the other mappings registered for the same airflow_connection_type.

Also worth covering

get_automatic_profile_mapping raises Could not find a profile mapping for connection {conn_id} (cosmos/profiles/__init__.py:80), which is vague in the same way. It should include the connection's conn_type and a hint that the connection is likely missing the fields required by its authentication method, or that the conn_type is unsupported.

No behaviour change — error messages only.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:loggingRelated to logging, like log levels, log formats, error logging, etcarea:profileRelated to ProfileConfig, like Athena, BigQuery, Clickhouse, Spark, Trino, etcenhancementNew feature or requestgood first issueGood for newcomers

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions