Compose Integration

Mutagen provides integration with Docker® Compose, allowing you to automatically create Mutagen synchronization and forwarding sessions alongside your Compose-based services, volumes, and networks. Mutagen’s synchronization and forwarding sessions can be used replace bind mounts and exposed ports, respectively, allowing you to run your project on a cloud-based Docker host while still editing code and accessing your application locally.

NOTE: Support for Docker Compose is new in Mutagen v0.12 (which is still in beta) so you'll need to install it from the Mutagen beta channel and it may have a few rough edges. Please help make Mutagen better by reporting issues and feedback.
NOTE: Mutagen's support for Docker Compose is separate from the Mutagen integration in Docker for Mac. Docker for Mac's Mutagen integration is focused on file caching inside of the Docker for Mac VM, whereas Mutagen's Docker Compose support is focused on enabling cloud-based development of Compose projects. However, when combined with Docker Desktop for Mac or Windows, Mutagen's Compose support can be used to provide the same caching benefits of Docker for Mac's Mutagen integration, with more granular control over synchronization behaviors.

Getting started

The following sections provide detailed information about using Mutagen’s support for Docker Compose, but the best way to experience it is to look at an example project and try it for yourself. You can find an example web application that demonstrates using this integration in the Mutagen repository. Have a look at the corresponding Compose YAML file to get an idea of what’s involved with integrating Mutagen, and try starting the project using mutagen compose up.

Here are a few other examples to get you started:

Usage

Mutagen’s integration with Docker Compose uses x-mutagen extension fields in Compose YAML files to define synchronization and forwarding sessions that target Docker volumes and networks, respectively. The format of these extension fields is described below. Note that using top-level extension fields requires a Compose file with version 3.7 or later.

Once you’ve added these fields, you can use the mutagen compose command to start and operate your Compose project. The mutagen compose command is a wrapper around the docker-compose command that reads x-mutagen configuration information and creates, updates, and terminates Mutagen synchronization and forwarding sessions alongside your Compose-based services, volumes, and networks.

Extension fields

To define synchronization and forwarding sessions, you just need to include an x-mutagen field at the root of your Compose YAML with Mutagen configuration and session definitions. Here’s an example of what such a YAML file might look like:

# Set the Compose format version
version: "3.7"

# Define networks
networks:
  frontend:
  backend:

 # Define volumes
volumes:
  code:

# Define services
services:
  database:
    ...
    networks:
      - backend
  web:
    ...
    volumes:
      - code:/code
    networks:
      - frontend
      - backend
    ...

# Define synchronization and forwarding sessions
x-mutagen:
  sync:
    defaults:
      ignore:
        vcs: true
    code:
      alpha: "."
      beta: "volume://code"
  forward:
    database:
      source: "tcp:localhost:5432"
      destination: "network://backend:tcp:database:5432"
    web:
      source: "tcp::8080"
      destination: "network://frontend:tcp:web:8080"

In this example, we define a Mutagen session to synchronize code from the Docker Compose project directory to a shared volume that we mount into our application container. We also define two Mutagen forwarding sessions: one that exposes the database to our local system (so that you can access it via a local client for development purposes) and one that exposes a web appplication on all interfaces on our local system (so that you can test it locally and from other devices on your local network). In this example, we use a separate volume to store code and we create networks manually. While volumes are necessary for synchronization targets, you don’t need to explicitly create networks as in this example—you can reference the default network by name. We’ve also avoided any major synchronization configuration here, whereas in real-world usage you’ll likely want to set up ignores and conflict resolution behavior. See this example for a more complete setup.

The format of the x-mutagen field is an extension of the global configuration format described in the introduction. In addition to a defaults key for synchronization and forwarding, it allows you to define keys that refer to named sessions. Sessions defined in Compose projects can also have endpoint-specific configuration options, which are specified using the configurationAlpha and configurationBeta keys for synchronization sessions and configurationSource and configurationDestination keys for forwarding sessions. The most general x-mutagen configuration looks something like:

x-mutagen:
  sync:
    defaults:
      ...default synchronization configuration...
    example-sync-session:
      alpha: ...
      beta: ...
      ...configuration...
      configurationAlpha:
        ...alpha-specific configuration...
      configurationBeta:
        ...beta-specific configuration...
    ...

  forward:
    defaults:
      ...default forwarding configuration...
    example-forwarding-session:
      source: ...
      destination: ...
      ...configuration...
      configurationSource:
        ...source-specific configuration...
      configurationDestination:
        ...destination-specific configuration...
    ...

As with normal Compose projects, you can specify these configuration elements across multiple YAML files. When merging YAML files, Mutagen will merge x-mutagen fields at the level below the sync and forward keys (i.e. sessions with different names will be combined into a set, while sessions with the same name (and session type) found in different files will just use the last definition). This also applies to defaults keys.

Endpoint URLs

Mutagen’s Compose integration currently supports only a limited set of endpoint URL formats, though these should suffice for typical Compose usage. For synchronization sessions, one of the endpoint URLs must be a local URL and the other must be a volume URL. For forwarding, the source endpoint URL must be a local URL and the destination endpoint URL must be a network URL. Support for Mutagen’s other transports is under development.

Volume URLs

Volume URLs are of the form:

volume://<volume-name>[<path>]

The <volume-name> component must refer to the name of a volume defined in the Compose YAML.

The <path> component is optional and should be an absolute path rooted at the volume root. If the <path> component is omitted, then the volume root is used as the synchronization target.

Network URLs

Network URLs are of the form:

network://<network-name>:<network-endpoint>

The <network-name> component must refer to the name of a network defined in the Compose YAML. The <network-endpoint> component is described in the forwarding documentation. Only TCP-based network endpoints are currently supported (i.e those prefixed with tcp, tcp4, or tcp6) and the host component of the endpoint should refer to a service defined in the Compose YAML and attached to the specified network.

Wrapper command

In order to use the extension fields added to your Compose YAML, you’ll need to use Mutagen’s Docker Compose wrapper command, mutagen compose. This command behaves identically to the docker-compose command and provides awareness of the synchronization and forwarding sessions defined in x-mutagen extension fields.

The mutagen compose wrapper is designed to faithfully emulate the docker-compose command in every respect (including configuration loading, override files and YAML merging, .env file support, commands, flags, and so on). If you find any observable difference in behavior, please open an issue. In fact, mutagen compose uses docker-compose under the hood for most of the heavy lifting, but it adds hooks to the up, down, ps, stop, and start commands to manage Mutagen synchronization and forwarding sessions. These hooks behave as follows:

  • When running up (with or without any service specifications), Mutagen synchronization and forwarding sessions are checked against their definitions in the x-mutagen extension fields and recreated or terminated if their configuration has been modified or deleted, respectively. Any newly created synchronization sessions are flushed (i.e. they have a synchronization cycle forced) before any containers are created, allowing containers to be sure that code already exists on any relevant volumes before they start.
  • When running down without any service specifications, Mutagen sessions are terminated before terminating services or other infrastructure.
  • When running ps without any flags or service specifications, Mutagen session statuses for the project are printed after container statuses. You can use docker-compose ps directly to avoid this behavior.
  • When using stop without any service specifications, Mutagen sessions are paused for the project.
  • When using start (with or without any service specifications), Mutagen sessions are resumed for the project before any services are started.

In short: you can use your normal Docker Compose workflow and expect Mutagen to manage sessions accordingly.

Implementation

Mutagen’s Compose support works by emulating Docker Compose’s configuration loading and determining the volume and network dependencies of synchronization and forwarding sessions. It then defines an additional sidecar service (called mutagen) attached to those volumes and networks through which synchronization and forwarding can be routed. All of the heavy lifting of creating volumes, networks, and containers is still provided by Docker Compose. The mutagen service service uses the Mutagen sidecar image (source code available here), which is essentially a no-op entrypoint suitable for hosting Mutagen agents. Creation of this service is fully automated and managed by the mutagen compose up and mutagen compose down commands, so there’s no need to define it as part of your Compose YAML (though it will show up alongside other containers in your project).

NOTE: The Mutagen sidecar service runs as root, so all files synchronized onto volumes will have their default ownership set to root. You can use Mutagen's permission settings to control the target UID/GID for files created by Mutagen. You can see an example of this in the data science demo.

The mutagen service and all synchronization and forwarding sessions are started before any of the project’s services are created. Additionally, all synchronization sessions are flushed (forced to complete an initial synchronization cycle) before any services are started, so you can be sure that any code or other files being synchronized will be available on the target volumes before any of your containers start. The synchronization and forwarding sessions operate using Mutagen’s Docker container transport.

Docker Compose and Docker CLI search paths

Mutagen uses the first docker-compose executable found in the user’s path. This search strategy can be overridden by setting the MUTAGEN_DOCKER_COMPOSE_PATH environment variable to a path containing the desired docker-compose executable. This environment variable should be set for mutagen compose commands, not the Mutagen daemon.

Because Mutagen’s Compose support also uses the Docker CLI, the docker command search behavior described in the Docker container transport documentation also applies and can be overridden using the MUTAGEN_DOCKER_PATH environment variable. Unlike the MUTAGEN_DOCKER_COMPOSE_PATH variable, the MUTAGEN_DOCKER_PATH variable should be set for the Mutagen daemon and not the mutagen compose command.

Known limitations

There are a few known limitations with Mutagen’s Compose support that shouldn’t restrict most use cases. These will be improved/fixed in future releases.

  • Implicit Docker contexts set using docker context use aren’t supported at the moment because Mutagen sessions don’t query the current context setting from the Docker context store, meaning that any context selected in this way won’t be persisted. Docker contexts specified using the -c/--context flags are supported and will persist correctly, and host configuration can also still be managed using DOCKER_HOST and other environment variables. This is a temporary limitation that will be removed as development continues.
  • Flags that only exist in a short form (e.g. the -T flag of exec) are missing from shell completion in mutagen compose. This is due to a limitation in Cobra’s flag parsing library tracked by spf13/pflag#139. Using these flags is still fully supported—only shell completion is affected.
  • The --skip-hostname-check flag is not supported because it’s not supported by the docker command that Mutagen uses for its Docker container transport.
  • Only local and Compose-related URLs (i.e. volume:// and network:// URLs) are currently supported by Mutagen’s Compose integration. This is a temporary limitation that will be removed as development continues.
  • Reverse network forwarding is not currently supported by Mutagen’s Compose integration. This is a temporary limitation that should be removed as development continues. You can still manually set up reverse forwarding with Docker containers in your project if necessary.
  • Only Linux/x86-64 containers are currently supported by Mutagen’s Compose integration. This is a temporary limitation that will be removed as development continues. Support for Windows containers and additional architectures is expected.
  • The run command is not fully supported because it creates ephemeral containers that Mutagen can’t easily track. The run command may work fine in certain cases and support for it should improve as development continues.

Windows and Chocolatey

If you’re using Chocolatey to install Docker Compose on Windows, please be aware that Chocolatey generates shim executables that can interfere with Mutagen’s wrapping. Fortunately, you can point Mutagen to the unshimmed executables to avoid this breakage by setting the MUTAGEN_DOCKER_COMPOSE_PATH environment variable. This setting should look like:

MUTAGEN_DOCKER_COMPOSE_PATH=C:\ProgramData\chocolatey\lib\docker-compose\tools

You may need to adjust this path depending on your Chocolatey installation. You can also use the %ChocolateyInstall% environment variable as part of this definition, for example:

MUTAGEN_DOCKER_COMPOSE_PATH=%ChocolateyInstall%\lib\docker-compose\tools