Is on_execute_callback can be used to initialize operators parameters at execution ?

[darkag]

Hello,

I wonder if I might have a problem with this type of use of on_execute_callback, as far as I've tested I suppose that's ok but I may miss something important :

def on_exec_callback_bash(context):
        from airflow.hooks.base import BaseHook
        vc = BaseHook.get_connection("my_connection")
        context['ti'].task.env = { "varenv": vc.host }
    
bash = BashOperator(task_id="also_run_this", bash_command='echo $varenv', on_execute_callback = on_exec_callback_bash)

[enchant3dmango]

Your code works, but it’s a hack. Prefer setting env directly in the operator or using templating. Keep callbacks for side effects, not for parameter wiring. In your case, I recommend pulling the connection inside the bash_command or setting it at the DAG parse time.

1. Pull the connection inside the bash_command

from airflow.hooks.base import BaseHook

vc = BaseHook.get_connection("my_connection")

bash = BashOperator(
    task_id="also_run_this",
    bash_command=f'echo {vc.host}'
)

2. Set env at the DAG parse time

from airflow.hooks.base import BaseHook

vc = BaseHook.get_connection("my_connection")

bash = BashOperator(
    task_id="also_run_this",
    bash_command="echo $varenv",
    env={"varenv": vc.host},
)

[potiuk]

Actually the "hack" is better. The problem with your solution @enchant3dmango is that it retrieves connection during Dag parsing, which is a bad idea - and against good practices.

The on_exec_callback is actually guaranteed to be executed in the same interpreter, with the same context as the operaror. so this is actually quite a good solution if it works. Maybe even a good idea to contribute it to the docs as recommended @darkag