Skip to content

[Jobs] Show pool health in the wait spinner for pool-targeted launches#9520

Open
lloyd-brown wants to merge 2 commits intomasterfrom
lloyd/show-pool-health-in-wait-spinner
Open

[Jobs] Show pool health in the wait spinner for pool-targeted launches#9520
lloyd-brown wants to merge 2 commits intomasterfrom
lloyd/show-pool-health-in-wait-spinner

Conversation

@lloyd-brown
Copy link
Copy Markdown
Collaborator

Summary

When sky jobs launch --pool <pool> can't dispatch immediately (e.g. no idle replicas), the job sits in PENDING showing only the generic "Waiting for task to start" spinner. The user has no signal whether the pool is fundamentally broken (zero READY workers) or just busy with someone else's work — they have to run sky jobs pool status in another terminal to find out.

This PR adds a one-line pool-health summary to that spinner, refreshed each tick, so the situation is visible inline:

⠋ Waiting for task to start (status: PENDING). It may take a few minutes.
  Pool 'foo': 1/3 ready (2 provisioning)
  View controller logs: sky jobs logs --controller 42
  • A small helper _get_pool_health_summary in sky/jobs/utils.py queries serve_state.get_replica_infos and produces strings like Pool 'foo': 5/11 ready (3 provisioning, 2 starting, 1 failed).
  • All FAILED_* replica statuses (FAILED, FAILED_CLEANUP, FAILED_INITIAL_DELAY, FAILED_PROBING, FAILED_PROVISION, UNKNOWN) collapse to a single failed count via ReplicaStatus.failed_statuses() so the line stays readable for large/chaotic pools.
  • A new {pool_str} field is added to _JOB_WAITING_STATUS_MESSAGE. When pool is unset (non-pool launches), it stays empty, so non-pool behavior is unchanged.
  • Any error querying replicas is swallowed; the spinner falls back to the original message.

Test plan

Renderings across pool scales (verified via unittest.mock.patch.object(serve_state, 'get_replica_infos', ...)):

Scenario Rendered summary
Empty / nonexistent pool Pool 'demo-pool': 0 workers
Healthy small pool, all ready Pool 'demo-pool': 3/3 ready
Small pool warming up Pool 'demo-pool': 1/3 ready (2 provisioning)
Small pool, broken (mixed FAILED_*) Pool 'demo-pool': 0/3 ready (3 failed)
Mid pool (mixed) Pool 'demo-pool': 5/11 ready (3 provisioning, 2 starting, 1 failed)
Large pool (50) Pool 'demo-pool': 30/50 ready (10 provisioning, 5 starting, 3 not_ready, 2 failed)
Chaotic 200-pool Pool 'demo-pool': 80/200 ready (50 provisioning, 30 starting, 10 not_ready, 10 pending, 5 shutting_down, 15 failed)

End-to-end manual test:

  • sky jobs launch --pool <healthy-pool> -- echo hi — confirm spinner shows Pool '<name>': N/N ready while waiting, transitions to streaming logs after dispatch.
  • sky jobs launch --pool <pool-with-no-ready-workers> -- echo hi — confirm spinner shows 0/N ready (...) instead of leaving the user in the dark.
  • sky jobs launch -- echo hi (no --pool) — confirm spinner is byte-identical to pre-change output (no pool line).
  • format.sh clean (10.00/10 pylint, mypy, yapf, isort, black all pass).

🤖 Generated with Claude Code

Pool jobs that can't dispatch (e.g. no idle replicas) sit in PENDING
showing only the generic "Waiting for task to start" spinner — the user
has no signal whether the pool is healthy or just busy. Add a one-line
pool-health summary to the spinner that updates each tick, so the user
can immediately tell "0/3 ready (3 provisioning)" from "3/3 ready" (a
legitimate queue) without needing to run sky jobs pool status separately.

All FAILED_* replica statuses (FAILED, FAILED_CLEANUP, FAILED_INITIAL_DELAY,
FAILED_PROBING, FAILED_PROVISION, UNKNOWN) collapse into a single "failed"
count to keep the line readable for large/chaotic pools. Any error
querying replicas falls back silently to the original spinner message.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds a pool health summary to the job waiting spinner in sky/jobs/utils.py, providing users with visibility into worker statuses (e.g., ready, failed, provisioning) for pending jobs. Feedback suggests refactoring the _get_pool_health_summary function to use a single-pass iteration and collections.Counter for better efficiency and deterministic output.

Comment thread sky/jobs/utils.py Outdated
…mary

Address Gemini review on PR #9520:
- Fold the ready count and other-status counts into a single pass.
- Use collections.Counter (more idiomatic).
- Sort the non-ready breakdown by count descending (alphabetical
  tiebreaker) so the spinner output is deterministic and the dominant
  state appears first — the alphabetical sort suggested in review would
  put 'failed' first in many cases, misleading the reader on where the
  bulk of replicas actually are.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@lloyd-brown lloyd-brown requested a review from cblmemo May 4, 2026 20:57
@cblmemo
Copy link
Copy Markdown
Collaborator

cblmemo commented May 5, 2026

According to the scenario you described, should we show how many pool replica is occupied by other jobs as well?

@lloyd-brown
Copy link
Copy Markdown
Collaborator Author

According to the scenario you described, should we show how many pool replica is occupied by other jobs as well?

Ah yes I like that, will work on adding that!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants