I'm trying to run inference with Lorax in the GPU Provider runpod. I've successfully launched the cluster, but it seems HTTP requests are not available. I'm looking for a workaround or an example to access the ports on RunPod for inference. Here's what I've tried so far:
# Launching the cluster
# Attempting to make HTTP requestsAny suggestions on how to proceed?
Carlos Leyson
Asked on Apr 04, 2024
RunPod currently doesn't support opening ports. However, you can use SSH local port-forwarding as a workaround for development purposes. Here's an example of how to set it up:
ssh -L 8080:localhost:8080 my-cluster -NKeep the SSH command running and then you can access the forwarded port on your local machine. If you encounter issues with connections being reset, make sure to check the configuration and ensure that the services inside the cluster are running correctly and listening on the expected ports.