Tunnels are a new peer-to-peer transport supporting synchronization and forwarding sessions across arbitrary infrastructure. They are particularly well-suited to container infrastructure, but they can also be used to work with virtual machines, remote servers, and even IoT devices.
Tunnels are designed to function as a transport between two locations: the
Mutagen daemon and the remote environment where the tunnel is being hosted.
When a tunnel is created, a credential file is created that can be used to host
the tunnel. This file can be copied to the target remote environment manually
(for example, via
scp) or via a secret distribution mechanism (for example,
kubectl create secret). Once a tunnel is created, multiple synchronization and
forwarding sessions can be created across the tunnel.
Tunnel initiation is brokered by the mutagen.io API, though tunnels communicate data directly between locations with end-to-end encryption. This brokering allows tunnels to be instantly notified when remote infrastructure becomes available or goes offline, allowing for faster reconnection to ephemeral infrastructure like containers.
Mutagen tunnels require Mutagen v0.11.0-beta1 or later (which can currently be
installed via Homebrew
Tunnels are managed using the
mutagen tunnel commands, namely
terminate. These commands are almost
identical to those for managing synchronization and forwarding sessions, for
which example usage can be found in the
Getting started guide. As with
synchronization and forwarding sessions, tunnels have an identifier, an optional
name, and optional labels. Once tunnels are created, their identifiers and/or
names can be used to establish synchronization and/or forwarding sessions on top
Before creating your first tunnel, you'll need to
sign in to mutagen.io and grab your API key from the
dashboard. You'll then need to configure Mutagen to use
this API key by using the
mutagen login command, for example:
# Log in to mutagen.io. You'll be prompted for an API token. mutagen login
This procedure only needs to be completed once.
Creating a tunnel
After logging in, tunnels can be created using the
mutagen tunnel create
command, for example:
# Create a tunnel with the name "mytunnel" and store the hosting credentials in # a file called "tunnel.tunn". mutagen tunnel create --name=mytunnel > tunnel.tunn
In the example above, a tunnel is created with the name
mytunnel and the
hosting credentials are piped into a file called
tunnel.tunn. Tunnel hosting
credentials can alternatively be piped directly to a secret storage mechanism.
For example, a Docker® secret can be created using:
# Create a tunnel and store the hosting credentials in a Docker secret. mutagen tunnel create | docker secret create <secret-name> -
A Kubernetes® secret can be created using one of the following invocations:
# Creating a Kubernetes secret on POSIX systems. mutagen tunnel create | kubectl create secret generic <secret-name> --from-file=/dev/stdin
REM Creating a Kubernetes secret on Windows systems via an intermediate file. mutagen tunnel create > tunnel.tunn kubectl create secret generic <secret-name> --from-file=tunnel.tunn del tunnel.runn
Tunnels are designed to be long-lived and reusable. An example application for tunnels would be creating synchronization and forwarding sessions into a container host or cluster that's regularly used for development. As long as the hosting credential file is stored and kept secure in the cluster, the tunnel can be used indefinitely.
Hosting a tunnel
Once the hosting credential file is available in the target location, a tunnel
can be hosted using the
mutagen tunnel host command:
# Host a Mutagen tunnel. mutagen tunnel host <credential-file-path>
For a Docker secret setup, this might look something like:
# Host a Mutagen tunnel using a Docker secret. mutagen tunnel host /run/secrets/<secret-name>
Tunnels should only be hosted in one location at a time. Invoking
mutagen tunnel host with the same hosting credential file in multiple
locations will cause the Mutagen daemon to connect to whichever host reaches the
Mutagen API first.
The hosting credential file path can also be specified to
mutagen tunnel host
MUTAGEN_TUNNEL_HOST_CREDENTIALS environment variable, which can work
better for container images with
mutagen tunnel host as their entry point
where it might be desirable to avoid freezing the credentials path into the
Hosting in containers
When using containerized setups, the current recommended practice is to have a sidecar container that acts as the tunnel host and to use shared persistent volumes as synchronization endpoints. Similarly, if this sidecar container can access other containers through container-based or service-based hostnames, it provides the best location through which to route forwarding sessions.
To handle permissions, the sidecar container can host the tunnel as
allowing synchronization sessions to set arbitrary user and group ownership for
files. Alternatively, the sidecar container can run as the user in the
application container, in which case files will automatically inherit that
user's ownership settings. In either case, it's best to build the sidecar
container image using the same base image as your other containers so that user
and group ID mappings are consistent. To this end, a
is provided in the Mutagen repository that can be used create a sidecar
container image. The script requires that the
curl command be available, but
beyond that is relatively trivial to use, for example:
# Use a minimal base image. FROM alpine:latest # Install supplementary tools. RUN ["apk", "add", "curl"] # Download and run the setup script. RUN curl -fsSL https://raw.githubusercontent.com/mutagen-io/mutagen/master/scripts/tunnel/sidecar.sh | sh # Set tunnel hosting to be the container entrypoint. ENTRYPOINT ["mutagen", "tunnel", "host"]
This script will install the latest release of Mutagen into the container,
including a Mutagen agent for that version. Because Mutagen is only compatible
with agents of the same minor version, you may wish to include additional agent
versions into the container if you team is using multiple versions of Mutagen.
This can be accomplished by setting the
environment variable to a list of colon-separated agent versions to install,
ENV MUTAGEN_TUNNEL_AGENT_VERSIONS 0.12.1:0.11.3
This must be done before invoking the provisioning script. There should also only be one version specified per minor release series (i.e. 0.12.1 and 0.11.3 together makes sense, 0.12.1 and 0.12.0 do not).
Tunnel-based filesystem endpoints can be specified to the
mutagen sync create
command using URLs of the form:
These URLs support Unicode names and paths and neither require nor support URL escape encoding (i.e. just type “ö”, not “%C3%B6”, etc.).
<tunnel-name-or-identifier> component can specify either a tunnel name or
identifier. The specification must be unambiguous or an error will occur when
sessions attempt to connect via the tunnel.
<path> component must be non-empty (i.e. at least a
/ character) and can
take one of four forms: an absolute path, a home-directory-relative path, a
home-directory-relative path for an alternate user, or a Windows absolute path:
# Example absolute path (/var/www). tunnel://<tunnel-name-or-identifier>/var/www # Example home-directory-relative path (~/project). tunnel://<tunnel-name-or-identifier>/~/project # Example alternate user home-directory-relative path (~otheruser/project). tunnel://<tunnel-name-or-identifier>/~otheruser/project # Example Windows absolute path (C:\path). Note that, in POSIX shells, the # backslash character will require quoting or escaping. tunnel://<tunnel-name-or-identifier>/C:\path
A tunnel must be unpaused in order to create synchronization sessions and to allow synchronization to run.
Tunnel network endpoints can be specified to the
mutagen forward create
command using URLs of the form:
<tunnel-name-or-identifier> component of this URL are the same those in
the synchronization URL format described above, while the
component is described in the
forwarding documentation. In the
case of relative Unix domain socket paths, path resolution will be performed
relative to the working directory in which
mutagen tunnel host is invoked.
Examples of tunnel forwarding URLs include the following:
# Binds to all network interfaces on port 8080. tunnel://<tunnel-name-or-identifier>:tcp::8080 # Binds to or targets the IPv4 loopback interface on port 8080. tunnel://<tunnel-name-or-identifier>:tcp4:localhost:8080 # Binds to or targets the IPv6 loopback interface on port 8080. tunnel://<tunnel-name-or-identifier>:tcp6:localhost:8080 # Binds to or targets to a specific IP address on port 6060. tunnel://<tunnel-name-or-identifier>:tcp:10.0.1.25:6060 # Binds to or targets a Unix domain socket relative to the user's home # directory. tunnel://<tunnel-name-or-identifier>:unix:~/path/to/socket.sock # Binds to or targets a Unix domain socket via an absolute path. tunnel://<tunnel-name-or-identifier>:unix:/path/to/socket.sock # Binds to or targets a Unix domain socket via a relative path. Path resolution # in this case is relative to the directory in which mutagen tunnel host is # invoked. tunnel://<tunnel-name-or-identifier>:unix:path/to/socket.sock
Each tunnel is limited to 65,533 active synchronization and forwarding sessions, though this limit shouldn't be a problem in practice. In the unlikely event that this limit is reached, additional tunnels can be created.
Tunnel URLs don't currently support user specification. Remote tunnel-based
Mutagen agent processes run as the user invoking the
mutagen tunnel host
Tunnels use UDP NAT traversal to establish peer-to-peer connections. While this mechanism works in the majority of cases, tunnels may experience difficulty connecting in network environments with strict firewalls or NAT setups.
One way to work around these issues is to restrict the range of UDP ports that a
tunnel will use and then enforce that the range is exposed to the internet.
This can be accomplished by setting the
MUTAGEN_TUNNEL_UDP_PORT_MAXIMUM environment variables for the
mutagen tunnel host command to a range falling inside [49152, 65535] and then
adjusting NAT and firewalls to accommodate this range. A range of 10-50 ports is
An example of this can be found in the data science example project, where the UDP port range for the tunnel host is restricted to a set of ports that are then published to the container host. If you are unable to work around NAT or firewall issues, please open a GitHub issue or reach out to [email protected] so that we can debug the issue and document a fix.