How can I ensure a VM on AWS authenticates with GCP using the same service account as my local machine?
I'm using Skypilot to launch a job on an AWS VM that needs to access a GCP bucket. Locally, I'm authenticated with a GCP service account and everything works fine. However, when the job runs on the AWS VM, it fails to authenticate with GCP, even though I've configured a file mount in Skypilot that points to the GCP bucket. The error message suggests that the OAuth 2.0 Service Account credentials are invalid and it cannot find the adc.json file. I can manually copy the GCP credential files to the VM and authenticate successfully, but I'm looking for a way to automate this within the Skypilot configuration. How can I achieve this?
Lucas Young
Asked on Feb 20, 2024
Zongheng Yang from Skypilot suggested that the issue might be due to the AWS node not being able to "assume" the role of the GCP service account from the local machine. He recommended uploading the ~/.config/gcloud/legacy_credentials/<currently active account> directory to the AWS node, rather than the entire ~/.config/gcloud/legacy_credentials/ directory. This approach should bypass the original error related to missing adc.json file. He provided a potential fix in a GitHub commit and advised to check it out and install it using pip install -e . to see if it resolves the issue.