Automatically restart port-forwarding in Kubernetes or OpenShift

When you set up a port-forward to a Kubernetes or OpenShift pod, the connection will disconnect automatically with the message “lost connection to pod” when the pod completes or you reach the port-forward time limit. It could be a few minutes or a few hours.

The automatic disconnection is a useful feature for security and to save resources, but the closed connection will interrupt your flow of thought and force you to restart port forwarding. For example, while connecting to a database and running queries, you will find the database connection suddenly closed and a query fails to run.

For example:

oc port-forward --address 12.0.0.1 pod/ste4site-db-0 5432:5432 --namespace=ste4site-test

Forwarding from 127.0.0.1:5432 -> 5432
E0726 19:03:15.297057    8264 portforward.go:233] lost connection to pod

I started using scripts with a polite loop to restart the connection when it breaks. I call it a “polite loop” because it attempts to avoid hammering the server if anything goes wrong, and it stops trying to restart the connection when the session token expires.

There are two versions of the same script below: one written in PowerShell for Windows clients, for when I’m working in Windows, and one in Bash shell for when I’m working in WSL or Linux.

They are very simple and I have separate scripts for each service and environment I use. I just call the appropriate script and don’t need to think about it again until the next day. You could add parameters if you prefer. Update the pod name, local and remote ports, and namespace as needed.

PowerShell:

# ------------------------------------------------------------------------
# PowerScript to forward a local port to a pod until login session expires
# ------------------------------------------------------------------------

$ListenInterface = "127.0.0.1"
$PodName =         "svc/my-service-name-test"
$LocalPort =       "5432"
$RemotePort =      "5432"
$NameSpace =       "abc123-test"
$RetrySeconds =    3

# Exit with an error if not logged in
oc whoami | out-null
if ( !$? ) { exit 2 } 

do
{
    Write-Output ""
    Get-Date -UFormat "%Y-%m-%d %T %Z: ${NameSpace} ${PodName}"
    oc port-forward --address ${ListenInterface} ${PodName} ${LocalPort}:${RemotePort} --namespace=${NameSpace}

    # Try not to bombard server with requests
    Start-Sleep $RetrySeconds

    # Check to see if I'm still logged in
    oc whoami | out-null
} while ( $? ) # Exit with an error if authentication token has expired

Bash shell:

#!/usr/bin/env bash
# ------------------------------------------------------------------------
# Bash script to forward a local port to a pod until login session expires
# ------------------------------------------------------------------------

LISTEN_INTERFACE=127.0.0.1
POD_NAME=svc/my-service-name-test
LOCAL_PORT=5432
REMOTE_PORT=5432
NAMESPACE=abc123-test
RETRY_SECONDS=3

while [ true ]
do
    # Exit with an error if not logged in
    oc whoami > /dev/null 2>&1
    if [ "${?}" -ne 0 ]; then
      echo Not logged in to OpenShift.
      exit
    fi
    date "+%Y-%m-%d %T %Z: ${NAMESPACE} ${POD_NAME}"
    oc port-forward --address ${LISTEN_INTERFACE} ${POD_NAME} ${LOCAL_PORT}:${REMOTE_PORT} --namespace=${NAMESPACE}

    # Try not to bombard the server with requests
    sleep ${RETRY_SECONDS}
done

I find this to be a robust and useful solution to a difficult problem.