[Feature] [PD] TransferEngine fault auto-recovery: allows Prefill node failures without requiring a restart of Decode.

[whybeyoung]

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

Currently, if a Prefill node fails, the Decode side may hang during data transfer if cached information about the failed node is still in use. A mechanism is needed to notify Decode to re-establish the connection with a healthy Prefill node.

CC @ByronHsu @ShangmingCai

Related resources

No response

[ShangmingCai]

I am on it. I can make a PR today or tomorrow to clean the outdated connection with the failed prefill node, and mark the requests as kv fail state. But I think we still need a mechanism to notify the mini_lb to let it stop pair requests to the failed prefill node.

[whybeyoung]

Solid of you @ShangmingCai