The issue might be caused by a mismatch in the ray version. Skypilot relies on ray==2.4.1 to schedule jobs on the remote cluster. If you have ray 2.6.3 installed, it can cause issues with the skypilot runtime. To fix this, you can create a new conda environment in the remote machine and install ray==2.4.1 in that environment. Here's an example of how to do it:

setup: |
  conda activate my-env
  if [ $? -eq 0 ]; then
    echo 'conda env exists'
  else
    # Setup the environment
    conda create -n my-env python=3.10 -y
    conda activate my-env
  fi
  pip install -r requirements/requirements.txt

run: |
  conda activate my-env
  echo "hello"

Additionally, it is recommended to install torch<2.2 instead of the latest torch==2.2 as the latest version can cause a segmentation fault on the VM image.