How to interactively SSH into a build job on any CI system

If you ever debug some tricky CI failure and say to your self: "If I can SSH into this server or container that running the build, I will be able to figure out immediately". Then this post is for you.

Traditionally, if we own the build server we can SSH into the server and inspect its state, or SSH into a server then docker exec a container to interactively run something or set a breakpoint in the code. But most of time the build servers are usually remain in a private network which we may not even had direct access as a developer, or when we use CI like github action, travis ci, circle ci, etc.

Today I want to share a very simple trick to SSH into any build server or container. That's simply using SSH Remote Port Forwarding or an external tools such as tmate.io

Method 1. Using tmate

On your CI job, simply install https://tmate.io/ and drop a command like this somewhere in middle of your script

tmate

Upon doing this, tmate will print a server address like this ssh [email protected] and anyone can SSH to it. From there we can manually execute build script and debug it

Beside tmate, there is alternative such as https://upterm.dev/ which works similarly.

Method 2. Using SSH

Above method is very simple but require us to trust tmate server. We can use a more secure method where we own the server with ssh remote port forwarding.

To do so, in our build script we include 2 things:

Step 1: Spin up a SSH server process

If the build run on a VM directly then maybe there is a SSH server already. If not here is what we can do. We will add the script below into any build script.

apt-get update && apt-get install -y openssh-server
mkdir /var/run/sshd
echo 'root:mypassword' | chpasswd # this will be the password we used to ssh to server
sed -i 's/PermitRootLogin prohibit-password/PermitRootLogin yes/' /etc/ssh/sshd_config
sed 's@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g' -i /etc/pam.d/sshd
/usr/sbin/sshd

We install openssh-server, change root password to mypassword , enable root login and sshd in daemon mode .

Now we had a SSH server running, but obviously we cannot SSH into it. Let's use remote port forwarding.

Step 2: Create a tunnel so we can SSH in

For this to work, we need a jumpbox. From inside your build script, we can do

ssh -i path-to-private-key-of-my-user -R ssh -R 2345:localhost:22 myuser@jumpox-ip

We should bundle/generate the private key somehow in the build script since we control the jumpbox.

Now SSH will listen on port 2323 on jumpbox and forward it back to our SSH server run on step 1.

Now, from inside jumpbox, we can SSH back the ci build server:

# run this on jumpbox
ssh -p 2345 root@localhost

Using the password we setup on step 1. Now, you are SSH into the server or container that run the build.

I hope this will be a useful tip to help debug build on any CI.