Skip to content

[BUG] Container doesn't start because of Celery workers unavailable in newest version #372

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Dr-Electron opened this issue Mar 13, 2025 · 8 comments · Fixed by #373
Closed

[BUG] Container doesn't start because of Celery workers unavailable in newest version #372

Dr-Electron opened this issue Mar 13, 2025 · 8 comments · Fixed by #373
Labels
bug Something isn't working fixed in dev Fixed in the development branch

Comments

@Dr-Electron
Copy link

Describe the bug

After upgrading to 0.20.4 I get the Celery workers unavailable error followed by a timeout

To Reproduce

Upgrade to 0.20.4 (from 0.20.3) I guess. Not sure if my setup is somewhat special

Error logs

Logs 

Operations to perform:
  Apply all migrations: account, app, auth, contenttypes, db, django_celery_beat, django_celery_results, events, lists, mfa, sessions, socialaccount, users
Running migrations:
  No migrations to apply.
2025-03-13 18:37:16,528 INFO Set uid to user 0 succeeded
2025-03-13 18:37:16,560 INFO supervisord started with pid 1
2025-03-13 18:37:17,563 INFO spawned: 'nginx' with pid 14
2025-03-13 18:37:17,566 INFO spawned: 'gunicorn' with pid 15
2025-03-13 18:37:17,568 INFO spawned: 'celery' with pid 16
2025-03-13 18:37:17,570 INFO spawned: 'celery-beat' with pid 17
2025-03-13 18:37:18,573 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-03-13 18:37:18,574 INFO success: gunicorn entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-03-13 18:37:18,574 INFO success: celery entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2025-03-13 18:37:18,574 INFO success: celery-beat entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
celery beat v5.4.0 (opalescent) is starting.
[2025-03-13 18:37:19 +0000] [15] [INFO] Starting gunicorn 23.0.0
[2025-03-13 18:37:19 +0000] [15] [INFO] Listening at: http://127.0.0.1:8001 (15)
[2025-03-13 18:37:19 +0000] [15] [INFO] Using worker: sync
[2025-03-13 18:37:19 +0000] [23] [INFO] Booting worker with pid: 23
__    -    ... __   -        _
LocalTime -> 2025-03-13 18:37:21
Configuration ->
    . broker -> redis://redis:6379//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> django_celery_beat.schedulers.DatabaseScheduler
    . logfile -> [stderr]@%INFO
    . maxinterval -> 5.00 seconds (5s)
[2025-03-13 18:37:21 +0000] [17] [INFO] beat: Starting...
 
 -------------- celery@1e8973e1cda3 v5.4.0 (opalescent)
--- ***** ----- 
-- ******* ---- Linux-4.4.302+-x86_64-with 2025-03-13 18:37:21
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         yamtrack:0x7f17758e4b30
- ** ---------- .> transport:   redis://redis:6379//
- ** ---------- .> results:     
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                
[tasks]
  . Import from AniList
  . Import from Kitsu
  . Import from MyAnimeList
  . Import from SIMKL
  . Import from Trakt
  . Import from Yamtrack
  . Reload calendar
  . health_check.contrib.celery.tasks.add
[2025-03-13 18:37:21 +0000] [17] [INFO] DatabaseScheduler: Schedule changed.
[2025-03-13 18:37:21 +0000] [16] [INFO] Connected to redis://redis:6379//
[2025-03-13 18:37:21 +0000] [16] [INFO] celery@1e8973e1cda3 ready.
[2025-03-13 18:37:41 +0000] [16] [INFO] Task health_check.contrib.celery.tasks.add[6a0f8817-9441-419c-ae22-3fa8c8d1785a] received
[2025-03-13 18:37:41 +0000] [27] [INFO] Task health_check.contrib.celery.tasks.add[6a0f8817-9441-419c-ae22-3fa8c8d1785a] succeeded in 0.052795181050896645s: 8
[2025-03-13 18:37:42 +0000] [23] [ERROR] unavailable: Celery workers unavailable
[2025-03-13 18:37:42 +0000] [23] [ERROR] Internal Server Error: /health/
[2025-03-13 18:38:13 +0000] [23] [ERROR] unavailable: Celery workers unavailable
[2025-03-13 18:38:16 +0000] [23] [ERROR] unavailable: TimeoutError: The task took too long to return a result
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/health_check/contrib/celery/backends.py", line 20, in check_status
    result.get(timeout=result_timeout)
  File "/usr/local/lib/python3.12/site-packages/celery/result.py", line 251, in get
    return self.backend.wait_for_pending(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/celery/backends/base.py", line 755, in wait_for_pending
    meta = self.wait_for(
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/celery/backends/base.py", line 791, in wait_for
    raise TimeoutError('The operation timed out.')
celery.exceptions.TimeoutError: The operation timed out.
[2025-03-13 18:38:16 +0000] [23] [ERROR] Internal Server Error: /health/

Yamtrack version: 0.20.4
Database: SQLite (default) or PostgreSQL
@Dr-Electron Dr-Electron added the bug Something isn't working label Mar 13, 2025
@FuzzyGrim FuzzyGrim linked a pull request Mar 13, 2025 that will close this issue
refactor: simplify healthcheck #373
Merged
@FuzzyGrim
Copy link
Owner

I created a pull request with some changes, you can test that build by changing the image tag in your docker-compose.yml to:

services:
  yamtrack:
    container_name: Yamtrack
    image: ghcr.io/fuzzygrim/yamtrack:pr-373
...

If everything seems to work fine, I will merge the pull request and create a new release with the changes.

@FuzzyGrim FuzzyGrim added the fixed in dev Fixed in the development branch label Mar 15, 2025
@Dr-Electron
Copy link
Author

Sorry totally forgot to answer. But it is still not working for me 🤔

@FuzzyGrim
Copy link
Owner

Have you upgraded to 0.20.5? If you have and it's still not working, could you show the logs with DEBUG=True?

services:
  yamtrack:
    container_name: yamtrack
    image: ghcr.io/fuzzygrim/yamtrack
    restart: unless-stopped
    environment:
      - DEBUG=True
...

@Dr-Electron
Copy link
Author

Dr-Electron commented Apr 14, 2025 •
edited
Loading

Sorry for the late reply. Tried updating to 0.20.5 again. Let me know if I can help some more.

    . maxinterval -> 5.00 seconds (5s)
[2025-04-14 22:09:30 +0000] [17] [INFO] beat: Starting...
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def reload_calendar(user=0, items_to_process=1):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def chain(*args, **kwargs):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def accumulate(self, *args, **kwargs):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def chord(self, header, body, partial_args=0, interval=1, countdown=2, max_retries=3, eager=4, **kwargs):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def unlock_chord(self, group_id, callback, interval=0, max_retries=1, result=2, Result=3, GroupResult=4, result_from_tuple=5, **kwargs):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def import_simkl(username, user_id, mode):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def import_yamtrack(file, user_id, mode):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def import_anilist(username, user_id, mode):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def xmap(task, it):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def import_mal(username, user_id, mode):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def xstarmap(task, it):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def import_trakt(username, user_id, mode):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def chunks(task, it, n):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def import_kitsu(username, user_id, mode):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def group(self, tasks, result, group_id, partial_args, add_to_parent=0):
�������
    return 1
[2025-04-14 22:09:30 +0000] [16] [DEBUG] 
def backend_cleanup():
�������
    return 1
[2025-04-14 22:09:30 +0000] [17] [INFO] DatabaseScheduler: Schedule changed.
 
 -------------- celery@5da47faf1705 v5.4.0 (opalescent)
--- ***** ----- 
-- ******* ---- Linux-4.4.302+-x86_64-with 2025-04-14 22:09:30
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         yamtrack:0x7f21df6405f0
- ** ---------- .> transport:   redis://redis:6379//
- ** ---------- .> results:     
- *** --- * --- .> concurrency: 1 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery
                
[tasks]
  . Import from AniList
  . Import from Kitsu
  . Import from MyAnimeList
  . Import from SIMKL
  . Import from Trakt
  . Import from Yamtrack
  . Reload calendar
[2025-04-14 22:09:30 +0000] [16] [INFO] Connected to redis://redis:6379//
[2025-04-14 22:09:30 +0000] [16] [INFO] celery@5da47faf1705 ready.
[2025-04-14 22:09:50 +0000] [23] [DEBUG] Got redis://redis:6379 as the redis_url. Connecting to redis...
[2025-04-14 22:09:50 +0000] [23] [DEBUG] Attempting to connect to redis...
[2025-04-14 22:09:50 +0000] [23] [DEBUG] Connection established. Redis is healthy.
[2025-04-14 22:09:51 +0000] [23] [ERROR] unavailable: Celery workers unavailable
Internal Server Error: /health/
[2025-04-14 22:09:51 +0000] [23] [ERROR] Internal Server Error: /health/

@FuzzyGrim
Copy link
Owner

I don't know why your celery worker isn't working. Could you try to disable the health check by adding to your docker-compose.yml

    healthcheck:
      disable: true

And then check the celery status with:

docker exec yamtrack celery -A config status

where yamtrack is the name of your Yamtrack container.

@Dr-Electron
Copy link
Author

I either get

Error: No nodes replied within time constraint

or

->  celery@1aeb716fe838: OK

1 node online.

@FuzzyGrim
Copy link
Owner

Seeing that sometimes it runs correctly, it seems that it could be because the worker is slow, could you try increasing the timeout?

docker exec yamtrack celery -A config inspect ping --timeout 3

@FuzzyGrim
Copy link
Owner

Added HEALTHCHECK_CELERY_PING_TIMEOUT variable to increase the healthcheck timeout in v0.22.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working fixed in dev Fixed in the development branch
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

refactor: simplify healthcheck FuzzyGrim/Yamtrack
2 participants
@FuzzyGrim @Dr-Electron