SkyPilot - Controller error after ~21 hours: 'Controller's latest status is INIT; jobs will not be shown until it becomes UP'. What could be the issue?

SkyPilot

skypilot-org.slack.com

skypilot-users

Controller error after ~21 hours: 'Controller's latest status is INIT; jobs will not be shown until it becomes UP'. What could be the issue?

I'm running into this error after having the controller be up for ~21 hours. I can ssh into the machine and I also see all sorts of errors in the logs from gcp, though nothing obvious. Does anyone know what this could be? Also, is there any way to restart the controller when it's in a bad state?

Alex Kouzemtchenko

Asked on Oct 27, 2023

To get the spot controller out of abnormal state, you can try sky start -f sky-spot-controller-<hash>.

Oct 27, 2023Edited by