SSH Summer School: jump hosts and file transferring
This blog post is the final part of a 3 part series about SSH (Secure Shell) connections with OpenSSH. In this last part of the series, I’ll talk about jump/bastion hosts, using ‘screen’ and transferring files. The aim of this blog post series is to teach you more about SSH. In these blog posts, I assume you are using a UNIX-like operating system with OpenSSH.
Let’s start off by talking about jump/bastion hosts and why they’re so handy. It’s becoming more and more common that the remote server you want to access isn’t directly reachable. Instead, you need to pass through a jump or bastion host. And that’s a good thing, because:
- application servers, most of the times, should not be exposed directly to the internet.
- if you are using IPv4, there is no need for a public IPv4 address for each of your servers.
- only a single host’s firewall needs to be configured to allow your office IP, which saves a lot of work when this IP changes.
Passing through a jump host can be annoying because you need to SSH through multiple servers. Luckily, there are multiple ways to do this. I’ve listed 4 methods below!
How to pass through a jump host
1. 'Manually'
One method of passing through a jump host is to SSH into the jump host and then from the jump host SSH into your server.
user@local-computer $ ssh jumphostuser@my-jump-host
jumphostuser@my-jump-host $ ssh remoteserveruser@my-remote-server
[remoteserveruser@my-remote-server ~]$
You’ll soon come to the conclusion that connecting to your remote server from the jump host requires an SSH key on the jump host. A solution could be to create an SSH key on the jump host, but this isn’t an option for jump hosts that are replaced on a regular basis. This probably isn’t a suitable method for most of us, so let’s go on to the next method. Oh, and by the way: never copy your local SSH key to another host such as a jump host!
2. Agent forwarding
I have already explained the ssh-agent concept in the first blog post of this series. With agent forwarding, you can expose your local ssh-agent on a remote server, such as a jump host. You can either use the command line option -A or the config file option ForwardAgent.
user@local-computer $ ssh-add
Enter passphrase for /Users/janbeerden/.ssh/id_rsa:
Identity added: /Users/user/.ssh/id_rsa (/Users/user/.ssh/id_rsa)
user@local-computer $ ssh -A jumphostuser@my-jump-host
jumphostuser@my-jump-host $ ssh remoteserveruser@my-remote-server
[remoteserveruser@my-remote-server ~]$By doing this, you no longer need an intermediary key. Your account on the jump host and the remote server should only contain the public part of your local SSH key. Easy enough, right? 
However, because you are exposing your local ssh-agent to the jump host, the jump host can use your SSH key without needing the passphrase. If you are the only one using (and maintaining) this jump host, you’re probably fine. If that isn’t the case (probably most of the times), someone could use your key while it’s being forwarded to the jump host.
When starting an SSH connection with agent forwarding enabled, the SSH server will create a socket to expose your ssh-agent. Only your user on the jump host will have permissions on this socket, but these restrictions do not apply to the root user or any user that has enough permissions via sudo.
3. ProxyCommand
With the ProxyCommand option, you can specify a command to be used to connect to the remote server. This option can be used to ‘chain’ SSH connections. By using “ssh -W” as ProxyCommand, we request that standard input and output on the client is forwarded to Hostname over the secure channel to my-jump-host.
Host my-jump-host
Hostname 192.168.0.10
User jumphostuser
Host my-remote-server
Hostname 192.168.5.10
User remoteserveruser
ProxyCommand ssh my-jump-host -W %h:%pAs you can see, this requires some configuration, but that only needs to be done once.
user@local-computer $ ssh my-remote-server
[remoteserveruser@my-remote-server ~]$Once this configuration is in place, you can ssh ‘directly’ in to your remote server. Passing through the bastion host is taken care of by the ProxyCommand.
4. ProxyJump (I recommend this one!)
Since OpenSSH 7.3, there is an easier to configure alternative to ProxyCommand called ProxyJump.
Host my-jump-host
Hostname 192.168.0.10
User jumphostuser
Host my-remote-server
Hostname 192.168.5.10
User remoteserveruser
ProxyJump my-jump-hostAs you can see, this option’s configuration is a lot simpler.
user@local-computer $ ssh my-remote-server
[remoteserveruser@my-remote-server ~]$Connecting to your remote server is no different from the ProxyCommand method. If you want to connect to a remote server without having to add it to your config file via a jump host (that you have added to your config file), ProxyJump is also very simple.
user@local-computer $ ssh remoteserveruser@192.168.0.20 -J my-jump-hostWhen you share an SSH config file with your team, you can also include the config shown above. They will certainly appreciate not having to create or add the configuration themselves!
I recommend this method instead of the others because:
- method 1 is too much effort each time you want to SSH to your remote server,
- method 2 is a security risk and still requires multiple steps each time you want to SSH to your remote server,
- method 3’s syntax is more complex and thus less easy to remember than the ProxyJump method.
With methods 3 and 4, you will also have the benefit of being able to transfer files directly to your remote server using scp or sftp without having to copy them to the jump host first. But how exactly do you start transferring files then? We’ll dive into the details of file transfer, but first let me tell you about using ‘screen’.
Using 'screen'
Since each jump host introduces an extra point of failure, now is the perfect time to start using ‘screen’. Have you ever had your SSH session disconnect while running a lengthy and/or delicate process on a remote server? When your SSH session gets disconnected, all processes started in the session that are running on the foreground will be terminated. This will require you to start all over or, depending on the process, leave you with a mess to clean up manually. Luckily, you can prevent this from becoming a problem by using ‘screen’. From its man page:
Screen is a full-screen window manager that multiplexes a physical terminal between several processes (typically interactive shells).
Some benefits of using screen:
- your shell remains active when your SSH session gets disconnected,
- you can run lengthy processes on a remote server without the need of maintaining an active SSH connection,
- you can use multiple windows from a single SSH session.
First of all, screen needs to be installed on your remote server. If screen isn’t installed yet, you can use your systems package manager to install it.
Starting a new screen
Starting a screen is very simple:
[remoteserveruser@my-remote-server ~]$ screenScreen will create a new shell, similarly as SSH’ing into a server would.
Disconnecting from a screen
The difference from a ‘standard’ shell is that you can disconnect from it, without killing all the sub processes. First you need to enter the escape command, which is CTRL+a, followed by CTRL+d
[detached from 4389.pts-0.my-remote-server]
[remoteserveruser@my-remote-server ~]$Screen will tell you it detached you from a screen session. In this case, the session is called 4389.pts-0.my-remote-server. If you want to exit the shell instead of disconnecting for the screen session, you can do so as if it is any other shell. Enter:
- exit
- CTRL + d
- ...
However, be careful! Screen does not protect you from accidentally exiting your shell.
Listing active screen sessions
With the -ls option, you can list all active screen sessions.
[remoteserveruser@my-remote-server ~]$ screen -ls
There are screens on:
4405.pts-0.my-remote-server (Detached)
4389.pts-0.my-remote-server (Detached)
2 Sockets in /var/run/screen/S-remoteserveruser.Each line consists of the process id and the session name, separated by a period.
Starting a new screen with a custom session name
As you can see from the output above, screen sessions can be hard to identify. Therefore, it might be wise to name your screen session when starting it:
[remoteserveruser@my-remote-server ~]$ screen -S server-upgrade
[remoteserveruser@my-remote-server ~]$ screen -ls
There are screens on:
4475.server-upgrade (Attached)
1 Socket in /var/run/screen/S-remoteserveruser.Attaching to a detached screen session
When you want to re-attach to a screen session, you can do so with the -r option. This option takes either the process id or the session name as parameter.
[remoteserveruser@my-remote-server ~]$ screen -r server-upgradeMultiple windows in a single screen session
Each of the following subsections assume you are attached to a screen session.
Creating a new window inside your screen session
By issuing CTRL+a followed by CTRL+c, you create a new window.
List of open windows
By issuing CTRL+a followed by CTRL+w, you get a list of open windows.
[remoteserveruser@my-remote-server ~]$
0$ bash 1-$ bash 2*$ bashAs you can see from the output above, I have 3 open windows, identified by a number followed by its title.
Switching between windows
There are 4 (5) ways to switch between windows:
- switch to a specific window
- CTRL+a followed by the number of the window
- CTRL+a followed by “, this method is probably the easiest since it presents you with an interactive window selector. By using the arrow keys, you can select the desired screen and press the return key to select it.
- switch to the next window: CTRL+a followed by n
- switch to the previous window: CTRL+a followed by p
- switch to the last window: CTRL+a followed by o
Other screen key bindings
There are several keys bindings that can be used inside a screen session, after issuing the control command (CTRL+a). You can get an overview of the available key bindings by pressing the ? key. Hit CTRL+a first, then hit the ? key.
Screen key bindings, page 1 of 2.
Command key: ^A Literal ^A: a
break ^B b license , removebuf =
clear C lockscreen ^X x reset Z
colon : log H screen ^C c
copy ^[ [ login L select '
detach ^D d meta a silence _
digraph ^V monitor M split S
displays * next ^@ ^N sp n suspend ^Z z
dumptermcap . number N time ^T t
fit F only Q title A
flow ^F f other ^A vbell ^G
focus ^I pow_break B version v
hardcopy h pow_detach D width W
help ? prev ^H ^P p ^? windows ^W w
history { } quit \ wrap ^R r
info i readbuf < writebuf >
kill K k redisplay ^L l xoff ^S s
lastmsg ^M m remove X xon ^Q q
[Press Space for next page; Return to end.]One of my favorites is copy. As the name gives away, it can be used to copy text from one window to another. For a complete overview and explanation of the available command line options and key bindings, please have a look at the man page.
Transferring files
As promised above, we’ll touch upon transferring files. There are a number of methods to transfer files to, from and between remote hosts. In this section, I’ll go over file transfer via SCP, SFTP and SSH ‘streaming’. All three of these methods also work when using a jump host when you use the ProxyCommand or ProxyJump directive as described above.
SCP
scp or secure copy allows you to securely copy files from, to or between remote hosts.
From
user@local-computer $ scp my-remote-server:~/file.txt ~/Downloads/local/To
user@local-computer $ scp ~/Downloads/local/file.txt my-remote-server:~/Between
user@local-computer $ scp my-remote-server:~/file.txt my-other-remote-server:~/For more information about using scp, have a look at it’s man page.
user@local-computer $ man scpsftp or secure file transfer protocol is an interactive file transfer program similar to ftp which allows you to securely copy files to or from a remote server. By entering the “?” command you will get a list of possible commands.
user@local-computer $ sftp my-remote-server
sftp> ?
Available commands:
bye Quit sftp
cd path Change remote directory to 'path'
chgrp grp path Change group of file 'path' to 'grp'
chmod mode path Change permissions of file 'path' to 'mode'
chown own path Change owner of file 'path' to 'own'
df [-hi] [path] Display statistics for current directory or
filesystem containing 'path'
exit Quit sftp
get [-afPpRr] remote [local] Download file
reget [-fPpRr] remote [local] Resume download file
reput [-fPpRr] [local] remote Resume upload file
help Display this help text
lcd path Change local directory to 'path'
lls [ls-options [path]] Display local directory listing
lmkdir path Create local directory
ln [-s] oldpath newpath Link remote file (-s for symlink)
lpwd Print local working directory
ls [-1afhlnrSt] [path] Display remote directory listing
lumask umask Set local umask to 'umask'
mkdir path Create remote directory
progress Toggle display of progress meter
put [-afPpRr] local [remote] Upload file
pwd Display remote working directory
quit Quit sftp
rename oldpath newpath Rename remote file
rm path Delete remote file
rmdir path Remove remote directory
symlink oldpath newpath Symlink remote file
version Show SFTP version
!command Execute 'command' in local shell
! Escape to local shell
? Synonym for helpFor more information about using sftp, have a look at it’s man page.
user@local-computer $ man sftp‘Stream’ through SSH
Let’s say you need an archive of the log files on your remote server, but you don’t have enough space on that server to create the archive. Instead of creating the archive on the remote server and then copying the archive, you could also ‘stream’ it to your local machine. Just like scp, this method can be used to copy files from, to or between remote servers.
From
user@local-computer $ ssh remoteserveruser@my-remote-server "tar -zcf - /var/log/" > ~/log-archive.tgzTo
user@local-computer $ tar -zcf - /var/log/ | ssh remoteserveruser@my-remote-server "cat - > ~/log-archive.tgz"Between
user@local-computer $ ssh remoteserveruser@my-remote-server "tar -zcf - /var/log/" | ssh remoteserveruser@my-remote-server "cat - > ~/log-archive.tgz"And there we have it! As always, if you have any questions or remarks, let us know.
