Airflow 3.1.1 task terminated after 24h: "Server indicated the task shouldn't be running anymore"

[Leocy17]

Environment

• Airflow version: 3.1.1
• Executor: CeleryExecutor (previously LocalExecutor)
• Deployment: Docker Compose (single host)
• Python version: 3.12
• OS: Ubuntu 22.04

---

💡 Problem Description

After upgrading from LocalExecutor to CeleryExecutor on Airflow 3.1.1,
some of our long-running SSH tasks (over 24h) fail with the following error:

[potiuk]

Likely this https://airflow.apache.org/docs/apache-airflow-providers-celery/stable/configurations-ref.html#visibility-timeout - but I am guessing.

[Leocy17]

Thank you for your answer! I tried your suggestion and also set the following parameters:

AIRFLOW__CELERY__TASK_ACKS_LATE="True"
AIRFLOW__CELERY_BROKER_TRANSPORT_OPTIONS__VISIBILITY_TIMEOUT="604800"

The task now runs perfectly. Thank you so much for your help; this solution really worked for me.

[buswedg]

Getting the same issue on 3.1.2, running CeleryExecutor via Docker on a Debian VM. I just tried increasing the visibility timeout per above and will report back if this doesn't correct the issue. Though I see the default timeout Celery puts on this is 1 hour (if set to None via Airflow). So have my doubts this will fix the issue.

Curious, could setting AIRFLOW__CELERY__TASK_ACKS_LATE to True also address this @potiuk ?

[potiuk]

Curious, could setting AIRFLOW__CELERY__TASK_ACKS_LATE to True also address this @potiuk ?

No idea.