Replace previous checkpoints while resuming training

[suprosanna]

CheckpointSaver saves model checkpoints for key_metric_n_saved number of best validation score and n_savednumber of the last model. While resuming training, the filename buffer starts from empty. Thus it creates a whole new set of checkpoint files after each resume. Replacing the previous checkpoints would be a cool feature to have for conveniently resuming training. Thanks.

[Nic-Ma]

Hi @suprosanna ,

Thanks for your interest here.
Actually, if you want to use a fixed filename and automatically replace it, you can set the key_metric_filename:
https://github.com/Project-MONAI/MONAI/blob/master/monai/handlers/checkpoint_saver.py#L56

Thanks.

[suprosanna]

That doesn't solve the problem since the handler does not allow to save more than one model based on the validation metric. Also, it does not help with the latest checkpoint file. I hope the following message clarifies the issue further.

File "~/venv/lib/python3.6/site-packages/monai-0.4.0+77.gf7565bf-py3.6.egg/monai/handlers/checkpoint_saver.py", line 150, in init
raise ValueError("if using fixed filename to save the best metric model, we should only save 1 model.")
ValueError: if using fixed filename to save the best metric model, we should only save 1 model.

[Nic-Ma]

Hi @suprosanna ,

Sorry maybe I misunderstood your request, may I know what the Replacing the previous checkpoints do you expect? Delete all the previous checkpoints before resuming?
@vfdev-5 do you have any better suggestion from ignite side?

Thanks in advance.

[vfdev-5]

@suprosanna in ignite Checkpoint has a method state_dict/load_state_dict that could be saved to keep and be able to restore the "filename buffer". We also provide a convenient argument include_self such that every time Checkpoint is used it includes its state_dict to the saved file. Then, when using stored checkpoint file to resume the training, we can pass required checkpoint handler to restore their states. Let us know if it helps. Thanks

[wyli]

I think this is a nice feature request. we should at least have include_self in https://github.com/Project-MONAI/MONAI/blob/master/monai/handlers/checkpoint_saver.py#L29 @Nic-Ma

[Nic-Ma]

Sounds good, I will try to enhance it later.
Thanks.