SkyPilot - How to troubleshoot evaluation score discrepancies in ML training with pre-emptions using Skypilot and PyTorch-Lightning?

skypilot-users

How to troubleshoot evaluation score discrepancies in ML training with pre-emptions using Skypilot and PyTorch-Lightning?

I am testing Skypilot to run my ML Training using OnSpot instances. When we have a pre-emption training restarts automatically, but the ending model we get has very different evaluation score much worse than the training without pre-emptions. Any suggestions on what could be the issue?

Bi

Bilal Yousaf

Asked on Feb 16, 2024

Launch the training in an on-demand instance and manually start the run commands to simulate preemption.
Check if all random seeds, including data seeds, are saved and restored properly.
Share how you trigger saving and restoring inside PyTorch-Lightning.

Feb 17, 2024Edited by