[lighthouse] fast failure on missing heartbeat instead of timeout

[rualark]

My understanding is that there are always some collective operations between replication groups to allreduce the gradients (if any form of ddp or hsdp is used). If one node fails in a replication group, all other groups will timeout because they will not finish allreduce. As lighthouse already knows that the node failed, should allreduce be aborted to avoid waiting for the timeout?

[WarrenZhu050413]

Hi Rualark,

From how I am understanding the question, if one node fails, the lighthouse should know based on the node heartbeat information that it failed, so it would be nice for us to have a mechanism for the lighthouse send a message to the individual managers to abort the allreduce?

I am indeed working on a solution to this effect. You would need a listening thread to listen from the messages, and abort the NCCL communication group when listening to an error message.

[rualark]

Sounds great! Do you want to use this issue for your PR, or do you want to close it as a duplicate of another existing issue?