SSH Tips for Remote Development
SSH is undoubtedly one of the most powerful tools for enabling remote development. Over a history spanning decades, SSH has proven itself to be reliable, security-focused, and flexible. Since SSH is one of Mutagen’s core transports, I felt that it would be useful to share a handful of simple but powerful features that I use on a daily basis to drastically improve my SSH experience and remote development workflows.
None of these features will be new or surprising for SSH aficionados, but new or
infrequent SSH users will hopefully find them valuable. I’ve intentionally kept
this list short and restricted to features that are simple and broadly useful.
I’ve also restricted it to OpenSSH features since OpenSSH is by far the most
common SSH implementation, but most of these features have analogs in other SSH
clients. For a full list of SSH’s superpowers, I’d recommend you grab a cup of
tea and drop into the ssh man page—it’s
well worth the time investment.
Aliases and configuration
Perhaps the most powerful SSH feature that users tend to avoid is its
configuration system. The SSH client configuration file (stored in
~/.ssh/config or specified via the -F flag) allows users to define hosts,
aliases for those hosts, and host-specific configuration. This allows for less
typing and significantly reduces the risk of command line typos.
A minimal configuration block in ~/.ssh/config might look like the following:
Host example-server
HostName host.example.org
User me
This configuration would cause the command ssh example-server to behave like
ssh [email protected]. This may not seem like a huge gain, but it already
avoids the need to specify the username when invoking SSH (how many times have
you had to Ctrl-C SSH after seeing
<local-username>@host.example.org?). It also creates an alias for the SSH
target (example-server) and adds it to SSH’s autocomplete functionality,
meaning that if you type ssh ex and press Tab, you’ll see example-server
in SSH’s list of suggestions (assuming you have autocomplete support enabled for
your shell and SSH).
A more complex configuration block might look like the following:
Host example-server
HostName host.example.org
Port 24
User me
IdentityFile ~/.ssh/id_rsa_example_org
TCPKeepAlive yes
ServerAliveInterval 60
In this example, a custom port has been specified, a non-default SSH private key has been provided, and other connection parameters have been set. Having to type each of these configuration parameters as separate flags when invoking SSH would be both tedious and error-prone.
The best part of all of this is that programs that use SSH as a transport (usually) automatically inherit this functionality, meaning that you can insert the host alias into the location where you’d typically insert a more explicit specification, for example:
scp <local-file> example-server:~/<remote-file>
or
mutagen sync create <local-path> example-server:~/<remote-path>
These examples only scratch the surface of what is possible. For more
information, including information on wildcard host matching, check out the
ssh_config man page.
Factoring out SSH configuration from the command line to a configuration file has a huge impact on most SSH workflows, making it easier to manage large sets of servers and deal with complex server setups. The time invested in setting up this configuration is usually recouped within hours.
Proxying connections
One of the coolest features of SSH is its ProxyCommand option. This option
allows SSH to connect to a remote server via execution of an intermediate
command, which is particularly useful in cases where an intermediate bastion
server is used to access another internal SSH server.
The SSH configuration for a bastion setup might look something like:
Host bastion-server
HostName bastion.example.org
User bastionuser
IdentityFile ~/.ssh/id_rsa_bastion
Host internal-server
HostName 10.0.1.25
User internaluser
IdentityFile ~/.ssh/id_rsa_internal
ProxyCommand ssh bastion-server -W %h:%p
In this setup, a bastion server (bastion.example.org) is being used to access
a private internal server (with the IP address 10.0.1.25). The ProxyCommand
option is used to specify a command that will open a TCP connection to the
target SSH server and connect it to the standard input and output of the proxy
command process. SSH will then use this proxied stream to communicate with the
target SSH server. In this case, we’re using ssh itself as the proxy command,
specifying the handy -W flag to open a connection to the internal SSH server
via the bastion server, but you can also use tools like nc as the proxy
command.
The ProxyCommand option provides support for templated placeholders like %h
and %p that are used to pass information about the target SSH server to the
proxy command. In this case, the hostname passed to the proxy command (filling
in the %h placeholder) will be 10.0.1.25 and the port value (filling in
%p) will default to 22. Generally speaking, you’ll want the HostName value
specified for the target SSH server to be relative to the intermediate server,
so it’s quite possible that you might have an internal IP or hostname specified
for HostName, as is the case here.
What do we get for all this trouble? Well, now we can do:
ssh internal-server
and be directly connected to the internal server. No more intermediate SSH sessions (at least none that are visible to the user). Moreover, because other tools inherit SSH aliases and their behavior, we can do things like:
scp <local-file> internal-server:~/<remote-file>
or
mutagen forward create tcp:localhost:8080 internal-server:tcp:localhost:8080
The flexibility of ProxyCommand (and its relative ProxyJump) cannot be
overstated. While it’s typically used for connecting via bastion servers, it can
be used for other types of proxies as well (or to chain multiple proxies
together). The only requirement is that the proxy command somehow create a TCP
connection to the target SSH server and connect it to the process’ standard
input/output. In theory you could write a proxy command that sends SSH data via
a web socket, connects via a SOCKS proxy, or even bounces data off the moon
using lasers (though the latency probably wouldn’t be ideal).
Loading private keys
When using public key authentication, you ideally want to password-protect your
private keys, though many people don’t. Those that do set a password when
creating a private key are rewarded by needing to endlessly type in the password
for that private key every time they connect to a server where the key is being
used for authentication (which is doubly fun when using ProxyCommand).
Fortunately there’s a way to have your cake and secure it too. The ssh-agent
command is a daemon that lives on the local system and stores your unlocked
private keys in-memory, allowing SSH to grab them when necessary and only
requiring you to type in the password once. You don’t actually start ssh-agent
directly, but instead use the ssh-add command to unlock a private key and
automatically start the ssh-agent daemon if it’s not already running. This
basically looks like:
# Unlock a private key and add it to ssh-agent.
ssh-add ~/.ssh/id_rsa
Enter passphrase:
All you have to do is specify the private key that you want to unlock. Once
added to ssh-agent, the private key will be available to SSH (and any other
programs using SSH) without you needing to enter the password for the key.
There’s a lot more that one can do with ssh-agent than what’s outlined here,
and there are also some security implementations worth reading about (especially
on multi-user systems), but for single-user systems with secure SSH keys,
ssh-agent can be a game changer. For more information, check out the
ssh-agent man page.
Port and socket forwarding
Besides creating interactive terminal sessions and forwarding input and output
to and from remote commands, SSH’s third major utility is forwarding network
traffic. This forwarding is controlled by the ssh command’s -L and -R
flags and the exact format for these flags is detailed extensively in the
ssh man page, so I won’t regurgitate that
here. Instead, I just want to touch on a few related topics. Of course I’ll also
point out that you can perform the exact same forwarding with Mutagen, and it
behaves better in some ways (for example, cleaning up sockets), but sometimes
SSH is the right tool for the job (especially if it’s the only one available).
The first thing to mention is the -N flag, which essentially tells the ssh
command that it’s only being used for forwarding and that it shouldn’t start an
interactive session or execute any command. Using this flag, a typical TCP
forwarding setup might look something like this:
# Start an SSH session that performs port forwarding and does nothing else.
ssh -N -L 'localhost:8080:localhost:8080' [email protected]
Nothing will be printed when you run this command (unless SSH needs a password or some other input), but the TCP forwarding will be established and continue to function until the command is terminated.
Second, while we’re on the topic of TCP forwarding, it’s really important to
mention that the default bind if no hostname is specified is all
interfaces, not localhost. This means that you don’t want to do something
like:
# WARNING: This binds to port 5432 on *ALL* local interfaces!
ssh -N -L '5432:localhost:5432' [email protected]
Doing so while sitting on a public network and without any local firewall would mean that your secure server was now exposed to anyone on that network who might be scanning for open ports. This same caveat also applies to Mutagen’s network forwarding.
Third, just in case it wasn’t clear, you can forward multiple ports with the
same ssh command invocation (using repeated -L and/or -R specifications)
and you can mix the -L and -R flags. Combining multiple forwarding
specifications can be a simple and secure mechanism for tying together different
application components (e.g. a local web application and a remote database).
Fourth, you can mix TCP and Unix domain socket endpoints when forwarding (for example, forwarding connections from a remote Unix domain socket to a local TCP server). This doesn’t have a lot of practical applications, but it is cool.
Fifth, you may want to remove stale Unix domain sockets when starting
forwarding. SSH doesn’t clean up the Unix domain sockets that it creates, so
starting a new forwarding command can require tedious removal of the previously
created socket. The next best thing is using the StreamLocalBindUnlink option,
which will remove any conflicting Unix domain sockets when starting forwarding,
for example:
# Tell SSH to remove any conflicting Unix domain sockets when forwarding.
ssh -N -o 'StreamLocalBindUnlink=yes' -L '/home/me/local.sock:/var/remote.sock' example.org
Note that StreamLocalBindUnlink doesn’t determine whether or not the
conflicting socket is in use—it simply removes any conflicting socket.
Sixth, you may want to set permissions on the socket. This is particularly
important when forwarding sockets to a remote daemon that needs to remain
secure. This can be done using the StreamLocalBindMask option, which overrides
the default umask when creating
sockets. The default mask is 0177, which results in sockets which are only
accessible by the current user (which is the safest default). Here’s an example
of what explicitly specifying such a mask would look like:
# Tell SSH to use a umask of 0177 for setting Unix domain socket permissions.
ssh -N -o 'StreamLocalBindMask=0177' -L '/home/me/local.sock:/var/remote.sock' example.org
It’s worth noting that not all platforms use socket permissions to restrict access (though most modern systems do), so you should consult the relevant documentation for your platform.
Finally, it’s worth mentioning one important application of this forwarding: accessing a remote Docker® daemon. For example, if you had a Docker daemon installation accessible via SSH, this forwarding might look like:
# Forward a local socket to a remote Docker daemon.
ssh -N -o 'StreamLocalBindUnlink=yes' -o 'StreamLocalBindMask=0177' -L '/Users/me/docker.sock:/var/run/docker.sock' [email protected]
You can then tell your Docker client (and any software that runs on top of the
client, e.g. Docker Compose or Mutagen) to connect to the remote daemon via this
forwarded socket using the DOCKER_HOST environment variable, e.g.
# Tell the Docker client to use a non-standard socket to connect to the daemon.
export DOCKER_HOST=unix:///Users/me/docker.sock
This is a fantastic way to set up and access a Docker daemon without having to
virtualize it on your laptop. All you need in this case is the Docker client,
which is significantly easier to install. You can also use Mutagen to set up
this forwarding, in which case the Mutagen daemon will keep this forwarding
alive in the background and allow you to just add the DOCKER_HOST setting to
your shell initialization file (e.g. .profile or .bashrc) and access it all
the time.
Multiplexing
All SSH connections are inherently multiplexed, meaning that the single TCP
connection made from the ssh command to the SSH server handles multiple data
streams. For example, one data stream might be used to create an interactive
terminal session and another data stream might be used to forward a network
connection (depending on what the ssh command has been told to do). SSH allows
us to exploit this multiplexing even further by using a single multiplexed TCP
connection to serve multiple ssh command invocations. Instead of opening a new
TCP connection every time ssh is invoked, SSH can cache and store the TCP
connection created for an ssh command and re-use it for subsequent ssh
invocations. Of course, this also applies to commands that use the ssh command
internally, such as scp, rsync, or git. This can drastically improve the
performance of commands that regularly interface with the same SSH host (such as
git push and git pull commands), and makes establishing new interactive SSH
sessions lightning fast.
The way this multiplexing is established is via the ControlMaster directive.
A typical configuration might look like:
Host git.example.org
ControlMaster auto
ControlPath ~/.ssh/connections/%r_%h_%p
ControlPersist 1h
The way that ControlMaster connections work is by creating a Unix domain
socket (set via the ControlPath parameter) that will be used by ssh commands
as the default mechanism for connecting to the remote host. By specifying auto
for ControlMaster, we’re telling SSH that it should first try to connect via
this Unix domain socket, but fall back to creating a standard TCP connection if
the socket doesn’t exist. If an ssh command does create a TCP connection, it
will then make that connection available via a Unix domain socket at the target
path. Thus, you need to specfy a path where those connections should be created
and make sure the parent directory (~/.ssh/connections in this example) is
secure (usually with 0700 permissions). The ControlPath option also needs to
uniquely correspond to the target connection (so that, for example,
ssh [email protected] and ssh [email protected] don’t use
the same Unix domain socket). Fortunately, the ControlPath option also allows
for placeholders to be used in the path, and in the example above we use the
%r, %h, and %p placeholders (username, hostname, and port, respectively)
to uniquely identify connection parameters.
It sounds complicated, but basically the three lines starting with Control in
the example above are all you need to add. You can even use them as-is, though
you’ll need to create and secure ~/.ssh/connections.
While ControlMaster is a game-changer for many SSH performance woes, it can
actually hurt certain applications, because all traffic to the remote SSH server
is going over the same TCP connection. This can lead to contention and (in my
experience) head-of-line blocking in certain use cases. For things like git
operations and interactive sessions, it works exceptionally well. For tools like
Mutagen and rsync, I’ve found it’s actually best to avoid using ControlMaster.
Fortunately, SSH’s flexible configuration makes it easy to enable
ControlMaster only where it’s helpful.
Conclusion
So that’s my list. There’s still much more that SSH can do, and by combining SSH’s behavioral primitives via other mechanisms, the setups that you can create are effectively limitless. In any case, I hope that these features are helpful in sparking workflow ideas and making your complex remote development experience just a little bit easier.