Mutagen Compose

Mutagen Compose is a Mutagen sibling project that provides Mutagen integration with Docker® Compose, allowing you to automatically create Mutagen synchronization and forwarding sessions alongside your Compose-based services, volumes, and networks.

Using Mutagen’s synchronization and forwarding sessions, you can replace bind mounts and exposed ports, allowing you to run your project on a remote and/or cloud-based Docker engine, while still editing code and accessing your application locally.

Mutagen Compose can also be used with Docker Desktop to provide a high-performance alternative to the virtual filesystems used in Docker Desktop bind mounts.

Getting started

The following sections provide detailed information about using Mutagen’s support for Docker Compose, but the best way to experience it is to look at an example project and try it for yourself. You can find an example web application that demonstrates using this integration in the Mutagen repository. Have a look at the corresponding Compose YAML file to get an idea of what’s involved with integrating Mutagen, and try starting the project using mutagen-compose up.

Here are a few other examples to get you started:

• A simple data science environment that uses Mutagen Compose integration
• A port of the Example Voting App that has been modified to support Mutagen Compose integration

Usage

Mutagen Compose uses x-mutagen extension fields in Compose YAML files to define synchronization and forwarding sessions that target Docker volumes and networks, respectively. The format of these extension fields is described below.

Once you’ve added these fields, you can use the mutagen-compose command to start and operate your Compose project. The mutagen-compose command is a complete Compose implementation that understands x-mutagen configuration information and creates, updates, and terminates Mutagen synchronization and forwarding sessions alongside your Compose-based services, volumes, and networks.

Extension fields

To define synchronization and forwarding sessions, you just need to include an x-mutagen field at the root of your Compose YAML with Mutagen configuration and session definitions. Here’s an example of what such a YAML file might look like:

# Define networks
networks:
  frontend:
  backend:

 # Define volumes
volumes:
  code:

# Define services
services:
  database:
    ...
    networks:
      - backend
  web:
    ...
    volumes:
      - code:/code
    networks:
      - frontend
      - backend
    ...

# Define synchronization and forwarding sessions
x-mutagen:
  sync:
    defaults:
      ignore:
        vcs: true
    code:
      alpha: "."
      beta: "volume://code"
  forward:
    database:
      source: "tcp:localhost:5432"
      destination: "network://backend:tcp:database:5432"
    web:
      source: "tcp::8080"
      destination: "network://frontend:tcp:web:8080"

In this example, we define a Mutagen session to synchronize code from the Compose project directory to a shared volume that we mount into our application container. We also define two Mutagen forwarding sessions: one that exposes the database to our local system (so that you can access it via a local client for development purposes) and one that exposes a web appplication on all interfaces on our local system (so that you can test it locally and from other devices on your local network). In this example, we use a separate volume to store code and we create networks manually. While volumes are necessary for synchronization targets, you don’t need to explicitly create networks as in this example—you can reference the default network by name. We’ve also avoided any major synchronization configuration here, whereas in real-world usage you’ll likely want to set up ignores and conflict resolution behavior. See this example for a more complete setup.

The format of the x-mutagen field is an extension of the global configuration format described in the introduction. In addition to a defaults key for synchronization and forwarding, it allows you to define keys that refer to named sessions. Sessions defined in Compose projects can also have endpoint-specific configuration options, which are specified using the configurationAlpha and configurationBeta keys for synchronization sessions and configurationSource and configurationDestination keys for forwarding sessions. The most general x-mutagen configuration looks something like:

x-mutagen:
  sync:
    defaults:
      ...default synchronization configuration...
    example-sync-session:
      alpha: ...
      beta: ...
      ...configuration...
      configurationAlpha:
        ...alpha-specific configuration...
      configurationBeta:
        ...beta-specific configuration...
    ...

  forward:
    defaults:
      ...default forwarding configuration...
    example-forwarding-session:
      source: ...
      destination: ...
      ...configuration...
      configurationSource:
        ...source-specific configuration...
      configurationDestination:
        ...destination-specific configuration...
    ...

As with normal Compose projects, you can specify these configuration elements across multiple YAML files. When merging YAML files, Mutagen will merge x-mutagen fields at the level below the sync and forward keys (i.e. sessions with different names will be combined into a set, while sessions with the same name (and session type) found in different files will just use the last definition). This also applies to defaults keys.

Endpoint URLs

Mutagen’s Compose integration currently supports only a limited set of endpoint URL formats, though these should suffice for typical Compose usage. For synchronization sessions, one of the endpoint URLs must be a local URL and the other must be a volume URL. For forwarding, the source endpoint URL must be a local URL and the destination endpoint URL must be a network URL. Support for Mutagen’s other transports is under development.

Volume URLs

Volume URLs are of the form:

volume://<volume-name>[<path>]

The <volume-name> component must refer to the name of a volume defined in the Compose YAML.

The <path> component is optional and should be an absolute path rooted at the volume root. If the <path> component is omitted, then the volume root is used as the synchronization target.

Network URLs

Network URLs are of the form:

network://<network-name>:<network-endpoint>

The <network-name> component must refer to the name of a network defined in the Compose YAML. The <network-endpoint> component is described in the forwarding documentation. Only TCP-based network endpoints are currently supported (i.e those prefixed with tcp, tcp4, or tcp6) and the host component of the endpoint should refer to a service defined in the Compose YAML and attached to the specified network.

Implementation

Mutagen Compose is implemented as a small layer of lifecycle hooks on top of Docker Compose V2. This allows Mutagen Compose to focus only on managing Mutagen sessions, while leaving the heavy lifting of creating volumes, networks, and containers to the underlying Docker Compose implementation.

When a Compose project is started, Mutagen Compose injects an additional sidecar service (called mutagen) through which synchronization and forwarding can be routed. The mutagen service service uses the Mutagen sidecar image, which is essentially a no-op entrypoint suitable for hosting Mutagen agents. Creation of this service is fully managed by the mutagen-compose command, so there’s no need to define it as part of your Compose YAML (though it will show up alongside other containers in your project).

The mutagen service and all synchronization and forwarding sessions are started before any of the project’s services are created. Additionally, all synchronization sessions are flushed (forced to complete an initial synchronization cycle) before any services are started, so you can be sure that any code or other files being synchronized will be available on the target volumes before any of your containers start. The synchronization and forwarding sessions operate using Mutagen’s Docker container transport.

The Mutagen sidecar image also causes a few minor behavioral changes in Mutagen agents that are designed to make working with Compose projects easier. Specifically:

• If a synchronization session targets an empty volume root and a non-default owner, group, or directory mode setting has been specified, then Mutagen will set the permissions of the volume root to match. This is a heuristic to work around the fact that Docker volumes don’t have an easy way to set permissions upon creation.
• If no file staging mode is explicitly set in the container, then Mutagen will will stage files into a temporary directory in their target volumes, avoiding the need for file copy operations when applying changes. If this behavior is undesirable (e.g. due to conflicts with filesystem watchers), then you can choose to stage in the sidecar container filesystem itself by selecting the mutagen staging mode.

Known limitations

There are a few minor limitations with Mutagen Compose that shouldn’t restrict most use cases. These will be improved/fixed in future releases.

• The Mutagen Compose version you install must match the version of Mutagen that you have installed.
• POSIX users and groups targeted by ownership settings in the x-mutagen section must be specified by numeric ID rather than name. This is because ownership setting happens in the Mutagen sidecar container, where (a) most names likely aren’t set to resolve to a numeric ID and (b) may resolve to different numeric IDs than in other images given that the sidecar image uses an Alpine base (e.g. www-data resolves to user/group ID 82 in Alpine-based images, but 33 in Debian-based images).
• Docker Compose V2 (and thus Mutagen Compose) will only use BuildKit (which you should use) if explicitly opted-into on the Docker engine (via JSON configuration) or on the client (by setting DOCKER_BUILDKIT=1 in the environment). Docker Desktop already opts-in to using BuildKit, so this caveat doesn’t affect it, but remote Docker engines typically do not. In this case, Docker Compose V2 will fall back to using the classic builder, but that currently leaves behind a number of excess containers and images. As such, it’s highly recommended that you set DOCKER_BUILDKIT=1 when using Mutagen Compose if you’re targeting a non-Docker Desktop engine. This applies to Docker Compose V2 as much as Mutagen Compose.
• Octal file modes (e.g. 0755) must be specified with a leading 0 and should optimally be specified with a 0o prefix (e.g. 0o755) in YAML for complete future-proofing. This restriction will likely migrate to the rest of Mutagen’s configuration soon, as the YAML spec no longer supports 0-only prefixes for octal modes (and Mutagen Compose doesn’t have the ability to assume an octal value in the absence of any prefix because it doesn’t control YAML parsing when loading Compose files).
• Only local and Compose-related (i.e. volume:// and network://) URLs are currently supported by Mutagen Compose. This is a temporary limitation that will be removed as development continues.
• Reverse network forwarding is not currently supported by Mutagen Compose. This is a temporary limitation that should be removed as development continues. You can still use Mutagen to manually set up reverse forwarding from Docker containers in your project if necessary.
• Only Linux containers are currently supported by Mutagen’s Compose integration. This is a temporary limitation that will be removed as development continues. Support for Windows containers is expected (and mostly implemented).
• Because the Mutagen service is created and started before all other services, containers cannot rely on volumes being populated with container contents. See mutagen-io/mutagen#302 for additional discussion.