S3DagBundle causes continuous DAG version increments despite no code changes

[marcosmartinezfco]

Apache Airflow version

Other Airflow 3 version (please specify below)

If "Other Airflow 3 version" selected, which one?

3.1.4

What happened?

We are observing continuous DAG version increments for DAGs sourced from an S3 DAG bundle (S3DagBundle).

Even though no changes have been made to the DAG files in S3 for several weeks, Airflow keeps generating new DAG versions—for example, the UI currently shows:

Dag Version(s): v2516

This issue happens without any DAG code updates, suggesting that the scheduler or DAG processor is detecting artificial "changes" during bundle sync or serialization.

Effects we’re seeing:

• DAG version numbers increase constantly (v1000 → v2000+).
• Metadata DB (serialized_dag, dag_version) grows unnecessarily.
• UI makes it appear as if DAGs are constantly changing.
• Scheduler and DAG processor logs do not indicate code changes.

What you think should happen instead?

Airflow should only create new DAG versions when:

• The DAG file content changes, or
• Structural changes to the DAG occur.

Since the S3 objects have unchanged ETags and last-modified timestamps, new versions should not be created.

How to reproduce

No idea, this only happens with one of my Dags which doesn't have any particular special code or anthing

Operating System

AlmaLinux 9.5 (Teal Serval)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.18.0
apache-airflow-providers-celery==3.14.0
apache-airflow-providers-cncf-kubernetes==10.11.0
apache-airflow-providers-common-compat==1.10.0
apache-airflow-providers-common-io==1.7.0
apache-airflow-providers-common-messaging==2.0.1
apache-airflow-providers-common-sql==1.30.0
apache-airflow-providers-docker==4.5.0
apache-airflow-providers-elasticsearch==6.4.0
apache-airflow-providers-fab==3.0.3
apache-airflow-providers-ftp==3.14.0
apache-airflow-providers-git==0.1.0
apache-airflow-providers-google==19.1.0
apache-airflow-providers-grpc==3.9.0
apache-airflow-providers-hashicorp==4.4.0
apache-airflow-providers-http==5.6.0
apache-airflow-providers-microsoft-azure==12.9.0
apache-airflow-providers-mysql==6.4.0
apache-airflow-providers-odbc==4.11.0
apache-airflow-providers-openlineage==2.9.0
apache-airflow-providers-postgres==6.5.0
apache-airflow-providers-redis==4.4.0
apache-airflow-providers-sendgrid==4.2.0
apache-airflow-providers-sftp==5.5.0
apache-airflow-providers-slack==9.6.0
apache-airflow-providers-smtp==2.4.0
apache-airflow-providers-snowflake==6.7.0
apache-airflow-providers-ssh==3.14.0
apache-airflow-providers-standard==1.10.0

Deployment

Docker-Compose

Deployment details

No response

Anything else?

Orchestration

• Docker Compose deployment.

• Scheduler + webserver + DAG processor running on a r7i.8xlarge instance

• Celery workers running on additional EC2 instances.

Executor

• CeleryExecutor

Metadata Database

• PostgreSQL Aurora cluster

• Highly available, no failovers detected.

• No write issues or DB-level instability.

Storage

• DAGs pulled from S3 DAG Bundle (S3DagBundle)

• S3 object ETags and LastModified timestamps confirmed stable.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[marcosmartinezfco]

The refresh rate is 30s so seems like every refresh triggers a new version change

[tirkarthi]

Can you please share the dag so that it is reproducible? Is it also reproducible using LocalDagBundle?

[marcosmartinezfco]

Can you please share the dag so that it is reproducible? Is it also reproducible using LocalDagBundle?

Sorry can't share the dag, but it's basically connecting to bastion host that has access to internet using the ssh operator and then running a script that lives in a shared network filesystem mounted in all our machines.

I haven't managed to reproduce it with the LocalDagBundle. Happy to help with the debugging if there's anything else I can do

[tirkarthi]

You can check the md5 hash to see if it's changing. You can also take a diff of the two serialised dag entries in data column for a given dag_id to see the difference that might help. Please note if the dag constructor has some dynamic elements like changing timestamp in dag.description it could also lead to this.

airflow/airflow-core/src/airflow/models/serialized_dag.py

Line 354 in d34942e

return md5(data_json).hexdigest()

[wjddn279]

Please check whether any arguments in the DAG or tasks include values that change on each execution, such as datetime.now().

[marcosmartinezfco]

Please check whether any arguments in the DAG or tasks include values that change on each execution, such as datetime.now().

yeah we use that kind of stuff a lot.

Is there any sort of best practices around this?

[potiuk]

Please check whether any arguments in the DAG or tasks include values that change on each execution, such as datetime.now().

yeah we use that kind of stuff a lot.

Is there any sort of best practices around this?

We are working on having a warning issued when this happens #59430 - you can take a lok at this PR in the meantime and avoid those things that are checked there.