Tunnels

Tunnels are a new peer-to-peer transport supporting synchronization and forwarding sessions across arbitrary infrastructure. They are particularly well-suited to container infrastructure, but they can also be used to work with virtual machines, remote servers, and even IoT devices.

NOTE: Tunnels and the mutagen.io infrastructure are still in beta. Please report any issues you experience via the issue tracker.

Design

Tunnels are designed to function as a transport between two locations: the Mutagen daemon and the remote environment where the tunnel is being hosted. When a tunnel is created, a credential file is created that can be used to host the tunnel. This file can be copied to the target remote environment manually (for example, via scp) or via a secret distribution mechanism (for example, kubectl create secret). Once a tunnel is created, multiple synchronization and forwarding sessions can be created across the tunnel.

Tunnel initiation is brokered by the mutagen.io API, though tunnels communicate data directly between locations with end-to-end encryption. This brokering allows tunnels to be instantly notified when remote infrastructure becomes available or goes offline, allowing for faster reconnection to ephemeral infrastructure like containers.

Requirements

Mutagen tunnels require Mutagen v0.11.0 or later.

Tunnels also require use of of the mutagen.io API for initiation, so you’ll need to sign in to mutagen.io using either your GitHub or Google account to get an API key.

Usage

Tunnels are managed using the mutagen tunnel commands, namely create, list, monitor, pause, resume, and terminate. These commands are almost identical to those for managing synchronization and forwarding sessions, for which example usage can be found in the Getting started guide. As with synchronization and forwarding sessions, tunnels have an identifier, an optional name, and optional labels. Once tunnels are created, their identifiers and/or names can be used to establish synchronization and/or forwarding sessions on top of them.

Logging in

Before creating your first tunnel, you’ll need to sign in to mutagen.io and grab your API key from the dashboard. You’ll then need to configure Mutagen to use this API key by using the mutagen login command, for example:

# Log in to mutagen.io. You'll be prompted for an API token.
mutagen login

This procedure only needs to be completed once.

Creating a tunnel

After logging in, tunnels can be created using the mutagen tunnel create command, for example:

# Create a tunnel with the name "mytunnel" and store the hosting credentials in
# a file called "tunnel.tunn".
mutagen tunnel create --name=mytunnel > tunnel.tunn

In the example above, a tunnel is created with the name mytunnel and the hosting credentials are piped into a file called tunnel.tunn. Tunnel hosting credentials can alternatively be piped directly to a secret storage mechanism. For example, a Docker® secret can be created using:

# Create a tunnel and store the hosting credentials in a Docker secret.
mutagen tunnel create | docker secret create <secret-name> -

A Kubernetes® secret can be created using one of the following invocations:

# Creating a Kubernetes secret on POSIX systems.
mutagen tunnel create | kubectl create secret generic <secret-name> --from-file=/dev/stdin
# Creating a Kubernetes secret on Windows systems via an intermediate file.
mutagen tunnel create > tunnel.tunn
kubectl create secret generic <secret-name> --from-file=tunnel.tunn
del tunnel.runn

Tunnels are designed to be long-lived and reusable. An example application for tunnels would be creating synchronization and forwarding sessions into a container host or cluster that’s regularly used for development. As long as the hosting credential file is stored and kept secure in the cluster, the tunnel can be used indefinitely.

Hosting a tunnel

Once the hosting credential file is available in the target location, a tunnel can be hosted using the mutagen tunnel host command:

# Host a Mutagen tunnel.
mutagen tunnel host <credential-file-path>

For a Docker secret setup, this might look something like:

# Host a Mutagen tunnel using a Docker secret.
mutagen tunnel host /run/secrets/<secret-name>

Tunnels should only be hosted in one location at a time. Invoking mutagen tunnel host with the same hosting credential file in multiple locations will cause the Mutagen daemon to connect to whichever host reaches the mutagen.io API first.

The hosting credential file path can also be specified to mutagen tunnel host using the MUTAGEN_TUNNEL_HOST_CREDENTIALS environment variable, which can work better for container images with mutagen tunnel host as their entry point where it might be desirable to avoid freezing the credentials path into the image.

Hosting in containers

In containerized setups, the best option is usually to have a sidecar container that acts as a tunnel host. Shared persistent volumes (accessible to both the sidecar and other containers) can be used as file synchronization targets. Similarly, if the sidecar container can access other containers through container-based or service-based hostnames, it provides an ideal location through which to route network forwarding sessions.

NOTE: It's important to use persistent shared volumes for synchronizing files with ephemeral infrastructure. Mutagen tracks synchronization session history, and thus a non-persistent storage location will appear to Mutagen to be deleting files. If using a bidirectional synchronization mode, this could result in file deletion. Mutagen's safety checks can avoid this deletion in certain cases, but it's best to either use a persistent volume for file storage or recreate/reset sessions before recreating containers.

To handle permissions, the sidecar container can host the tunnel as root, allowing synchronization sessions to set arbitrary user and group ownership for files. Alternatively, the sidecar container can run as the user in the application container, in which case files will automatically inherit that user’s ownership settings. In either case, it’s best to build the sidecar container image using the same base image as your other containers so that user and group ID mappings are consistent. To this end, a provisioning script is provided in the Mutagen repository that can be used create a sidecar container image. The script requires that the curl command be available, but beyond that is relatively trivial to use, for example:

# Use a minimal base image.
FROM alpine:latest

# Install supplementary tools.
RUN ["apk", "add", "curl"]

# Download and run the setup script.
RUN curl -fsSL https://raw.githubusercontent.com/mutagen-io/mutagen/master/scripts/tunnel/sidecar.sh | sh

# Set tunnel hosting to be the container entrypoint.
ENTRYPOINT ["mutagen", "tunnel", "host"]

This script will install the latest release of Mutagen into the container, including a Mutagen agent for that version. Because Mutagen is only compatible with agents of the same minor version, you may wish to include additional agent versions into the container if you team is using multiple versions of Mutagen. This can be accomplished by setting the MUTAGEN_TUNNEL_AGENT_VERSIONS environment variable to a list of colon-separated agent versions to install, for example:

ENV MUTAGEN_TUNNEL_AGENT_VERSIONS 0.12.1:0.11.3

This must be done before invoking the provisioning script. There should also only be one version specified per minor release series (i.e. 0.12.1 and 0.11.3 together makes sense, 0.12.1 and 0.12.0 do not).

Synchronization

Tunnel-based filesystem endpoints can be specified to the mutagen sync create command using URLs of the form:

tunnel://<tunnel-name-or-identifier>/<path>

These URLs support Unicode names and paths and neither require nor support URL escape encoding (i.e. just type “ö”, not “%C3%B6”, etc.).

The <tunnel-name-or-identifier> component can specify either a tunnel name or identifier. The specification must be unambiguous or an error will occur when sessions attempt to connect via the tunnel.

The <path> component must be non-empty (i.e. at least a / character) and can take one of four forms: an absolute path, a home-directory-relative path, a home-directory-relative path for an alternate user, or a Windows absolute path:

# Example absolute path (/var/www).
tunnel://<tunnel-name-or-identifier>/var/www

# Example home-directory-relative path (~/project).
tunnel://<tunnel-name-or-identifier>/~/project

# Example alternate user home-directory-relative path (~otheruser/project).
tunnel://<tunnel-name-or-identifier>/~otheruser/project

# Example Windows absolute path (C:\path). Note that, in POSIX shells, the
# backslash character will require quoting or escaping.
tunnel://<tunnel-name-or-identifier>/C:\path

A tunnel must be unpaused in order to create synchronization sessions and to allow synchronization to run.

Fowarding

Tunnel network endpoints can be specified to the mutagen forward create command using URLs of the form:

tunnel://<tunnel-name-or-identifier>:<network-endpoint>

The <tunnel-name-or-identifier> component of this URL are the same those in the synchronization URL format described above, while the <network-endpoint> component is described in the forwarding documentation. In the case of relative Unix domain socket paths, path resolution will be performed relative to the working directory in which mutagen tunnel host is invoked.

Examples of tunnel forwarding URLs include the following:

# Binds to all network interfaces on port 8080.
tunnel://<tunnel-name-or-identifier>:tcp::8080
# Binds to or targets the IPv4 loopback interface on port 8080.
tunnel://<tunnel-name-or-identifier>:tcp4:localhost:8080
# Binds to or targets the IPv6 loopback interface on port 8080.
tunnel://<tunnel-name-or-identifier>:tcp6:localhost:8080
# Binds to or targets to a specific IP address on port 6060.
tunnel://<tunnel-name-or-identifier>:tcp:10.0.1.25:6060
# Binds to or targets a Unix domain socket relative to the user's home
# directory.
tunnel://<tunnel-name-or-identifier>:unix:~/path/to/socket.sock
# Binds to or targets a Unix domain socket via an absolute path.
tunnel://<tunnel-name-or-identifier>:unix:/path/to/socket.sock
# Binds to or targets a Unix domain socket via a relative path. Path resolution
# in this case is relative to the directory in which mutagen tunnel host is
# invoked.
tunnel://<tunnel-name-or-identifier>:unix:path/to/socket.sock

Examples

An example data science environment using tunnels can be found in the Mutagen repository.

Limitations

Each tunnel is limited to 65,533 active synchronization and forwarding sessions, though this limit shouldn’t be a problem in practice. In the unlikely event that this limit is reached, additional tunnels can be created.

Tunnel URLs don’t currently support user specification. Remote tunnel-based Mutagen agent processes run as the user invoking the mutagen tunnel host command.

Networking

Tunnels use UDP NAT traversal to establish peer-to-peer connections. While this mechanism works in the majority of cases, tunnels may experience difficulty connecting in network environments with strict firewalls or NAT setups.

One way to work around these issues is to restrict the range of UDP ports that a tunnel will use and then enforce that the range is exposed to the internet. This can be accomplished by setting the MUTAGEN_TUNNEL_UDP_PORT_MINIMUM and MUTAGEN_TUNNEL_UDP_PORT_MAXIMUM environment variables for the mutagen tunnel host command to a specific port range and then adjusting NAT and firewalls to accommodate this range. A range of 10-50 ports is recommended.

An example of this can be found in the data science example project, where the UDP port range for the tunnel host is restricted to a set of ports that are then published to the container host. If you are unable to work around NAT or firewall issues, please open a GitHub issue or reach out to [email protected] so that we can debug the issue and document a fix.