[core][autoscaler][v1] add heartbeat timeout logic to determine node activity status

[nadongjun]

Why are these changes needed?

This PR is related to #52198.
When a worker is reported as active by the node provider, the autoscaler marks it as active in the summary—even if the worker’s raylet has already exited. This results in stale nodes being incorrectly shown as active in the cluster status.

This issue still occurs even after the previous PR was applied, which intended to fix the heartbeat-based activity check. The reason is that the logic used to determine node activeness in non_terminated_nodes still relies solely on the presence of the following internal mappings:

• ray_nodes_last_used_time_by_ip
• static_resources_by_ip
• raylet_id_by_ip
• dynamic_resources_by_ip
• last_heartbeat_time_by_ip

If these entries exist, the autoscaler assumes the node is active—even if the underlying raylet process has already exited.

Thus, when a raylet exits after idle timeout but the NodeProvider continues to return the node, the autoscaler still marks it as active. This behavior is incorrect and leads to inaccurate autoscaler summaries.

Proposed

The is_active() method should not rely solely on the existence of the above variables. Instead, it should check whether the time since the last heartbeat exceeds AUTOSCALER_HEARTBEAT_TIMEOUT_S. This way, nodes with outdated or missing heartbeats can be correctly identified as inactive.

Current Behavior (Observed)

1. Marked as Active Despite Raylet Exit (ip: 172.28.0.16)

(Raylet has exited, but node is still reported as active by the autoscaler)

2. Worker Only Removed After Idle Timeout Expires

(Terminated worker persists until autoscaler clears stale state)

In the current system, the following sequence occurs:

1. The node provider reports a worker node as active(172.28.0.16).
2. The worker’s raylet exits.
3. The autoscaler does not detect the raylet exit and continues to treat the node as active.
4. After the idle timeout(5 minute) passes, the autoscaler finally removes the node’s entries from its internal state.
5. Until the cleanup, the node remains incorrectly marked as active.

Related issue number

#52198

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(