Detect that Redis Server has Gone Down and Exit Gracefully

[melrom]

From Jack Smith:

BTW, is there a way for a BigJob (BJ) script to detect that the Redis server has shut down while the script is running (i.e., having survived the initial startup while Redis is up)? I submitted a BJ script yesterday, and this morning I found the BJ script still running, but it had not made any progress. And then I noticed that no batch pilot job (PJ) was running or waiting. I then peeked at the PJ’s stderr file and saw the “Please start Redis server!” message. This apparently happened about 23 hrs after the script was launched.

The .sge prolog/epilog file times confirm that was the time the SGE batch PJ finally starting running and almost immediately terminated. So apparently the Redis server had shut down sometime in the interim. However the BJ script seemed to be oblivious to both the Redis server being down and the batch PJ terminating.

I'm not entirely sure if this is possible for BigJob, but maybe if the PJ.get_state reports a Fail or Cancel or can't access the batch PJ anymore, we could do some kind of error handling or at least a warning, "Cannot reach redis server"

[drelu]

This is not as trivial as it seems: http://www.cs.mcgill.ca/~carl/impossible.pdf

For example, terminating everything, i.e. setting the state to fail, because of a transient network fault (which one cannot be sure of whether this is a network fault or machine fault) is not a good option. Since 80% of the faults are of transient nature, they are ignored. This way BJ is able to survive e.g. a reboot of the Redis server machine or a flaky network without a problem. You can't optimize for all possible error modes, you need to chose the most frequent ones.

[JackS9]

Why not let the user set the criteria in a Config file? Like a maximum wait time?

On the flip-side, the user should also be able to specify (override) the criteria used for when (if) the batch job (Pilot Job) should ever terminate for lack of any new requests or resources (cores) going idle.

[oleweidner]

Any progress on this one?