Can the startup time for autoscaling be improved by using a starting volume with pre-downloaded weights?
I'm wondering if the startup time for autoscaling can be improved by using a starting volume that already has the weights for the model downloaded. Currently, the startup time is slow because the weights need to be downloaded. Is there a way to speed up this process?
Caleb Welton
Asked on Feb 02, 2024
Yes, there are a couple of options to improve the startup time for autoscaling by pre-downloading the weights:
-
Using a Docker image: You can put the weights into a Docker image and specify
resources.image_idfor each replica. This way, the weights will already be available in the Docker image, reducing the download time. -
Using a machine image: Another option is to create a machine image that contains the weights and specify
resources.image_idto use that image for starting each replica. This can be faster than using a Docker image since the weights are already included in the machine image.
Both options have their pros and cons. Using a Docker image allows for more flexibility and can be used across different clouds, but there is still a download time from the Docker registry. On the other hand, using a machine image is faster since the weights are already included, but it is limited to a single cloud/region.