Need Help with SSL_CERTIFICATE_VERIFY_FAILED issue getting 'Server Returned error' after adding CA certificated directly to certifi/cacert.pem

[prayaagmahajan]

Apache Airflow version
3.1.3

What happened?
When API server has setup SSL certificate with local Certificate Authority, airflow-worker is not enable to finish task and service log contains the following error:
2025-12-01_13:02:47.60287 [2025-12-01 13:02:47,601: ERROR/MainProcess] Task execute_workload[4fc778e9-71ee-44f6-a27e-40f813091e1d] raised unexpected: ServerResponseError('Server returned error') 2025-12-01_13:02:47.60292 Traceback (most recent call last): 2025-12-01_13:02:47.60293 File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 453, in trace_task 2025-12-01_13:02:47.60294 R = retval = fun(*args, **kwargs) 2025-12-01_13:02:47.60295 ^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60295 File "/usr/local/lib/python3.11/site-packages/celery/app/trace.py", line 736, in __protected_call__ 2025-12-01_13:02:47.60296 return self.run(*args, **kwargs) 2025-12-01_13:02:47.60297 ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60298 File "/usr/local/lib/python3.11/site-packages/airflow/providers/celery/executors/celery_executor_utils.py", line 182, in execute_workload 2025-12-01_13:02:47.60299 supervise( 2025-12-01_13:02:47.60307 File "/usr/local/lib/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py", line 1940, in supervise 2025-12-01_13:02:47.60309 process = ActivitySubprocess.start( 2025-12-01_13:02:47.60310 ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60311 File "/usr/local/lib/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py", line 954, in start 2025-12-01_13:02:47.60311 proc._on_child_started(ti=what, dag_rel_path=dag_rel_path, bundle_info=bundle_info) 2025-12-01_13:02:47.60314 File "/usr/local/lib/python3.11/site-packages/airflow/sdk/execution_time/supervisor.py", line 965, in _on_child_started 2025-12-01_13:02:47.60315 ti_context = self.client.task_instances.start(ti.id, self.pid, start_date) 2025-12-01_13:02:47.60316 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60321 File "/usr/local/lib/python3.11/site-packages/airflow/sdk/api/client.py", line 215, in start 2025-12-01_13:02:47.60323 resp = self.client.patch(f"task-instances/{id}/run", content=body.model_dump_json()) 2025-12-01_13:02:47.60324 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60325 File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1218, in patch 2025-12-01_13:02:47.60326 return self.request( 2025-12-01_13:02:47.60326 ^^^^^^^^^^^^^ 2025-12-01_13:02:47.60327 File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 338, in wrapped_f 2025-12-01_13:02:47.60328 return copy(f, *args, **kw) 2025-12-01_13:02:47.60329 ^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60329 File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 477, in __call__ 2025-12-01_13:02:47.60330 do = self.iter(retry_state=retry_state) 2025-12-01_13:02:47.60333 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60334 File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 378, in iter 2025-12-01_13:02:47.60335 result = action(retry_state) 2025-12-01_13:02:47.60336 ^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60337 File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 400, in <lambda> 2025-12-01_13:02:47.60340 self._add_action_func(lambda rs: rs.outcome.result()) 2025-12-01_13:02:47.60341 ^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60342 File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 449, in result 2025-12-01_13:02:47.60343 return self.__get_result() 2025-12-01_13:02:47.60343 ^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60344 File "/usr/lib64/python3.11/concurrent/futures/_base.py", line 401, in __get_result 2025-12-01_13:02:47.60345 raise self._exception 2025-12-01_13:02:47.60346 File "/usr/local/lib/python3.11/site-packages/tenacity/__init__.py", line 480, in __call__ 2025-12-01_13:02:47.60347 result = fn(*args, **kwargs) 2025-12-01_13:02:47.60348 ^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60349 File "/usr/local/lib/python3.11/site-packages/airflow/sdk/api/client.py", line 885, in request 2025-12-01_13:02:47.60350 return super().request(*args, **kwargs) 2025-12-01_13:02:47.60351 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60353 File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 825, in request 2025-12-01_13:02:47.60354 return self.send(request, auth=auth, follow_redirects=follow_redirects) 2025-12-01_13:02:47.60355 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60356 File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 914, in send 2025-12-01_13:02:47.60357 response = self._send_handling_auth( 2025-12-01_13:02:47.60358 ^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60358 File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 942, in _send_handling_auth 2025-12-01_13:02:47.60359 response = self._send_handling_redirects( 2025-12-01_13:02:47.60361 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60362 File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 999, in _send_handling_redirects 2025-12-01_13:02:47.60362 raise exc 2025-12-01_13:02:47.60363 File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 982, in _send_handling_redirects 2025-12-01_13:02:47.60365 hook(response) 2025-12-01_13:02:47.60369 File "/usr/local/lib/python3.11/site-packages/airflow/sdk/api/client.py", line 186, in raise_on_4xx_5xx_with_note 2025-12-01_13:02:47.60370 return get_json_error(response) or response.raise_for_status() 2025-12-01_13:02:47.60374 ^^^^^^^^^^^^^^^^^^^^^^^^ 2025-12-01_13:02:47.60375 File "/usr/local/lib/python3.11/site-packages/airflow/sdk/api/client.py", line 176, in get_json_error 2025-12-01_13:02:47.60376 raise err 2025-12-01_13:02:47.60376 **airflow.sdk.api.client.ServerResponseError: Server returned error**

Workaround:

• adding CA certificated directly to certifi/cacert.pem file - airflow-worker works correctly but each certifi update needs to cacert.pem redefinition
• Added to enable SSL: [AIRFLOW__API__SSL_CERT,AIRFLOW__API__SSL_KEY]

Previous Error

• before adding CA certificate directly to certifi/cacert.pem file I was getting error "CERTIFICATE_VERIFY_FAILED". After adding this change directly now I am receiving "Server Returned Error"