SkyPilot - Is there a solution for managing datasets on k8s clusters in Skypilot?

skypilot-users

Is there a solution for managing datasets on k8s clusters in Skypilot?

Andrew Aikawa is looking for a solution to manage datasets on k8s clusters in Skypilot for a client who wants to move onto k8s. Romil Bhardwaj has posted a workaround on the GitHub issue #2497. Will this workaround be sufficient for managing datasets on k8s clusters in Skypilot?

An

Andrew Aikawa (asaiacai)

Asked on Feb 22, 2024

Romil Bhardwaj has posted a workaround on the GitHub issue #2497 to manage datasets on k8s clusters in Skypilot.
The workaround involves specifying volumes through the task yaml.
Skypilot aims to support specifying volumes through the task yaml in the future, but it requires careful design of the abstractions.
Andrew Aikawa found the workaround provided by Romil Bhardwaj to be perfect and suitable for his client's use case.

Feb 22, 2024Edited by