Introduction to docker-compose

Wednesday 6 March 2019

Introduction to docker-compose

docker-compose

Helps managing Docker containers and their dependencies. Its configuration is in docker-compose.yml file

It can also be used in a single project with Dockerfile as instead of manually running docker build <args> and then docker run <args> we can put all these args in docker-compose.yml and then call docker-compose up.

UPDATE (July 2023): It is now recommended to use Compose V2 which comes with syntax docker compose (instead of docker-compose). [Migrate to Compose V2 | Docker Docs]

Installation

Install Docker Compose | Docker Documentation

To upgrade it, first uninstall and then install it again. I described the whole process in this article: How to upgrade docker-compose on Linux | My Public Notepad.

Configuration file

docker-compose reads its configuration from a file called docker-compose.yml.

Directories specified in volumes section in yml file don't have to be created manually - docker-compose up will create them.

CLI

$ docker-compose

Define and run multi-container applications with Docker.

Usage:
docker-compose [-f <arg>...] [options] [COMMAND] [ARGS...]
docker-compose -h|--help

Options:
-f, --file FILE Specify an alternate compose file
(default: docker-compose.yml)
-p, --project-name NAME Specify an alternate project name
(default: directory name)
--verbose Show more output
--log-level LEVEL Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
--no-ansi Do not print ANSI control characters
-v, --version Print version and exit
-H, --host HOST Daemon socket to connect to

--tls Use TLS; implied by --tlsverify
--tlscacert CA_PATH Trust certs signed only by this CA
--tlscert CLIENT_CERT_PATH Path to TLS certificate file
--tlskey TLS_KEY_PATH Path to TLS key file
--tlsverify Use TLS and verify the remote
--skip-hostname-check Don't check the daemon's hostname against the
name specified in the client certificate
--project-directory PATH Specify an alternate working directory
(default: the path of the Compose file)
--compatibility If set, Compose will attempt to convert deploy
keys in v3 files to their non-Swarm equivalent

Commands:
build Build or rebuild services
bundle Generate a Docker bundle from the Compose file
config Validate and view the Compose file
create Create services
down Stop and remove containers, networks, images, and volumes
events Receive real time events from containers
exec Execute a command in a running container
help Get help on a command
images List images
kill Kill containers
logs View output from containers
pause Pause services
port Print the public port for a port binding
ps List containers
pull Pull service images
push Push service images
restart Restart services
rm Remove stopped containers
run Run a one-off command
scale Set number of containers for a service
start Start services
stop Stop services
top Display the running processes
unpause Unpause services
up Create and start containers
version Show the Docker-Compose version information

Here are the options from Compose V2:

$ docker compose --help

Usage: docker compose [OPTIONS] COMMAND

Define and run multi-container applications with Docker

Options:

--all-resources Include all resources, even those not used by services

--ansi string Control when to print ANSI control characters ("never"|"always"|"auto") (default "auto")

--compatibility Run compose in backward compatibility mode

--dry-run Execute command in dry run mode

--env-file stringArray Specify an alternate environment file

-f, --file stringArray Compose configuration files

--parallel int Control max parallelism, -1 for unlimited (default -1)

--profile stringArray Specify a profile to enable

--progress string Set type of progress output (auto, tty, plain, quiet) (default "auto")

--project-directory string Specify an alternate working directory

(default: the path of the, first specified, Compose file)

-p, --project-name string Project name

Commands:

attach Attach local standard input, output, and error streams to a service's running container

build Build or rebuild services

config Parse, resolve and render compose file in canonical format

cp Copy files/folders between a service container and the local filesystem

create Creates containers for a service

down Stop and remove containers, networks

events Receive real time events from containers

exec Execute a command in a running container

images List images used by the created containers

kill Force stop service containers

logs View output from containers

ls List running compose projects

pause Pause services

port Print the public port for a port binding

ps List containers

pull Pull service images

push Push service images

restart Restart service containers

rm Removes stopped service containers

run Run a one-off command on a service

scale Scale services

start Start services

stats Display a live stream of container(s) resource usage statistics

stop Stop services

top Display the running processes

unpause Unpause services

up Create and start containers

version Show the Docker Compose version information

wait Block until the first service container stops

watch Watch build context for service and rebuild/refresh containers when files are updated

Run 'docker compose COMMAND --help' for more information on a command.

Its configuration file is docker-compose.yaml (or .yml). This file is also known as Compose file. It contains:

services - list of service names; Each service can have:

build -
command -
environment -
image -
ports -
restart -
volumes -

docker-compose up will read Compose file, create and start containers (services) start listed there.

To check the state of all services, run docker-compose ps from the working directory (which is by default the one that contains Compose file):

$ docker-compose ps

Name Command State Ports

--------------------------------------------------------------------------------

my-project_service1_1 /bin/sh -c yarn Up 0.0.0.0:3001->3001/tcp

install && ...

my-project__postgres_1 postgres.sh Up 0.0.0.0:5432->5432/tcp

my-project_service3_1 /usr/local/bin/example Up 0.0.0.0:4567->4567/tcp

To list the IDs of all running containers:

$ docker-compose ps -q

1e8a22167a4ac8354a456a29550cc7fd5fa5eb80ce68f6db04db8ba9764733cf

a1404d3f917c5daf8921731175ae0aa0940eccb518e9c0fd786183036d000d20

e6aeaf2e79b10a29bc834b0e3f712add1c02216b50afa1193663248162721b86

To list all images:

$ docker-compose images

Container Repository Tag Image Id Size

------------------------------------------------------------------

my-project_service1_1 service1 latest 58f28f305f80 875 MB

my-project__postgres_1 postgres latest f97c959a7d9c 298 MB

my-project_service3_1 service3 latest 4e02459e1040 159 MB

Another way to find the association between running containers and their IDs:

$ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

a1403d3f917c postgres "postgres.s…" 29 hours ago Up 3 hours 0.0.0.0:5432->5432/tcp postgres

e6aecf2e79b1 service3 "/usr/local/bin/example…" 29 hours ago Up 3 hours 0.0.0.0:4567->4567/tcp browser-engagement_s3_1

1e8a12167a4a service1 "/bin/sh -c 'yarn in…" 29 hours ago Up 3 hours 0.0.0.0:3001->3001/tcp my-project_service1_1

Stops containers and removes containers, networks, volumes, and images
created by `up`.

docker-compose down

It completely destroys containers (but not images).

$ docker-compose down --help
Stops containers and removes containers, networks, volumes, and images
created by `up`.

By default, the only things removed are:

- Containers for services defined in the Compose file
- Networks defined in the `networks` section of the Compose file
- The default network, if one is used

Networks and volumes defined as `external` are never removed.

Usage: down [options]

Options:
--rmi type Remove images. Type must be one of:
'all': Remove all images used by any service.
'local': Remove only images that don't have a
custom tag set by the `image` field.
-v, --volumes Remove named volumes declared in the `volumes`
section of the Compose file and anonymous volumes
attached to containers.
--remove-orphans Remove containers for services not defined in the
Compose file
-t, --timeout TIMEOUT Specify a shutdown timeout in seconds.
(default: 10)

Example:

$ docker-compose down --rmi all --volumes

docker-compose: option to automaticaly remove container after run in docker-compose.yml

docker-compose pull

Let's assume our docker-compose.yml contains two services, db and db_sync and each of them has some Dockerfile which defines base image for their containers. When we do docker-compose up first time, Docker will pull the latest version of those images but each subsequent docker-compose up will not be checking if base images have newer version in the remote image repository - we need to update these images manually. We can look Dockerfile and update each base image manually, like:

$ docker pull node:alpine

We might make mistake and omit some images. Better way is to call docker-compose pull (from a directory which contains yml file) - it will iterate through each service listed in yml and update its base image(s). Example:

$ docker-compose pull
Pulling db ... done
Pulling db_sync ... done

docker-compose pull
docker-compose up doesn't pull down latest image if the image exists locally #3574

docker-compose restart

It is possible to restart a single container (service). Eg. if we have an app container and db container and we want to restart only app container we can do:

$ docker-compose restart my-app

Caveat: this does not pick up any changes in yml config file or image itself. To rebuild the image and restart the container:

$ docker-compose build my-app && docker-compose restart my-app

docker compose run (Compose V2)

$ docker compose run --help

Usage: docker compose run [OPTIONS] SERVICE [COMMAND] [ARGS...]

Run a one-off command on a service

Options:

--build Build image before starting container

--cap-add list Add Linux capabilities

--cap-drop list Drop Linux capabilities

-d, --detach Run container in background and print container ID

--dry-run Execute command in dry run mode

--entrypoint string Override the entrypoint of the image

-e, --env stringArray Set environment variables

-i, --interactive Keep STDIN open even if not attached (default true)

-l, --label stringArray Add or override a label

--name string Assign a name to the container

-T, --no-TTY Disable pseudo-TTY allocation (default: auto-detected)

--no-deps Don't start linked services

-p, --publish stringArray Publish a container's port(s) to the host

--quiet-pull Pull without printing progress information

--remove-orphans Remove containers for services not defined in the Compose file

--rm Automatically remove the container when it exits

-P, --service-ports Run command with all service's ports enabled and mapped to the host

--use-aliases Use the service's network useAliases in the network(s) the container connects to

-u, --user string Run as specified username or uid

-v, --volume stringArray Bind mount a volume

-w, --workdir string Working directory inside the container

SERVICE is the name of the service as stated in the docker-compose.yaml file.

Example: Run Terraform command against the Terraform container (instead of the locally installed Terraform executable):

docker-compose.yaml:

---

name: terraform

services:

terraform:

image: hashicorp/terraform:latest

volumes:

- .:/infra

working_dir: /infra

$ docker compose run --rm terraform init

Initializing the backend...

Initializing provider plugins...

- Finding kreuzwerker/docker versions matching "3.0.2"...

- Installing kreuzwerker/docker v3.0.2...

- Installed kreuzwerker/docker v3.0.2 (self-signed, key ID BD080C4571C6104C)

Partner and community providers are signed by their developers.

If you'd like to know more about provider signing, you can read about it here:

https://www.terraform.io/docs/cli/plugins/signing.html

Terraform has created a lock file .terraform.lock.hcl to record the provider

selections it made above. Include this file in your version control repository

so that Terraform can guarantee to make the same selections by default when

you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see

any changes that are required for your infrastructure. All Terraform commands

should now work.

If you ever set or change modules or backend configuration for Terraform,

rerun this command to reinitialize your working directory. If you forget, other

commands will detect it and remind you to do so if necessary.

terraform here is the name of the service. init is a Terraform command.

hashicorp/terraform Dockerfile probably has terraform process set as ENTRYPOINT and CMD set to "--help" or empty string. This is why running this container with no cmd args passed simply runs terraform (terraform --help) and exits:

$ docker compose run terraform

Usage: terraform [global options] <subcommand> [args]

...

This command creates and runs container and when root process (terraform) exits, container gets stopped but not deleted. To make sure container is deleted we need to add --rm:

$ docker compose run --rm terraform

docker-compose stop

It only hibernate containers.

docker-compose up

$ docker-compose up --help
Builds, (re)creates, starts, and attaches to containers for a service.

Unless they are already running, this command also starts any linked services.

The `docker-compose up` command aggregates the output of each container. When
the command exits, all containers are stopped. Running `docker-compose up -d`
starts the containers in the background and leaves them running.

If there are existing containers for a service, and the service's configuration
or image was changed after the container's creation, `docker-compose up` picks
up the changes by stopping and recreating the containers (preserving mounted
volumes). To prevent Compose from picking up changes, use the `--no-recreate`
flag.

If you want to force Compose to stop and recreate all containers, use the
`--force-recreate` flag.

Usage: up [options] [--scale SERVICE=NUM...] [SERVICE...]

Options:
-d, --detach Detached mode: Run containers in the background,
print new container names. Incompatible with
--abort-on-container-exit.
--no-color Produce monochrome output.
--quiet-pull Pull without printing progress information
--no-deps Don't start linked services.
--force-recreate Recreate containers even if their configuration
and image haven't changed.
--always-recreate-deps Recreate dependent containers.
Incompatible with --no-recreate.
--no-recreate If containers already exist, don't recreate
them. Incompatible with --force-recreate and -V.
--no-build Don't build an image, even if it's missing.
--no-start Don't start the services after creating them.
--build Build images before starting containers.
--abort-on-container-exit Stops all containers if any container was
stopped. Incompatible with -d.
-t, --timeout TIMEOUT Use this timeout in seconds for container
shutdown when attached or when containers are
already running. (default: 10)
-V, --renew-anon-volumes Recreate anonymous volumes instead of retrieving
data from the previous containers.
--remove-orphans Remove containers for services not defined
in the Compose file.
--exit-code-from SERVICE Return the exit code of the selected service
container. Implies --abort-on-container-exit.
--scale SERVICE=NUM Scale SERVICE to NUM instances. Overrides the
`scale` setting in the Compose file if present.

Example:

$ docker-compose up --build

How to make docker-compose read .env file?

How to use environment variables in docker compose

.env:

MY_KEY=VALUE

docker-compose.yml:

my-service:
environment:
MY_KEY: ${MY_KEY}

Both $VARIABLE and ${VARIABLE} syntax are supported. [source]

How to get docker run -it ... via docker-compose?

Interactive shell using Docker Compose

Include the following lines in your docker-compose.yml:

stdin_open: true
tty: true

How to make docker-compose remove the container after running?

Use docker-compose down

Auto remove container with docker-compose.yml

How do I mount a host directory as a volume in docker compose

How to set network name, IP range and static IP addresses for each service in yml file?

docker-compose.yml example:

version: '3.7'
networks:
postgres-demo:
name: postgres-demo-net
ipam:
driver: default
config:
- subnet: 172.16.239.0/24
services:
pg_db:
...
networks:
postgres-demo:
ipv4_address: 172.16.239.2
pg_admin:
...
networks:
postgres-demo:
ipv4_address: 172.16.239.3

---
By default Compose sets up a single network for your app. Each container for a service joins the default network and is both reachable by other containers on that network, and discoverable by them at a hostname identical to the container name.

Your app’s network is given a name based on the “project name”, which is based on the name of the directory it lives in.

Instead of just using the default app network, you can specify your own networks with the top-level networks key.

Each service can specify what networks to connect to with the service-level networks key, which is a list of names referencing entries under the top-level networks key.

source: Networking in Compose
---

Containers running on the same network can use each other's service names as specified in docker-compose.yml as hostnames (for IP connections). This is handy as we don't need to inspect the network in order to determine their IP addresses. Example: running pgAdmin on the same network as Postgres. When adding a DB server we can use DB service name from yml as DB server's hostname.

Set default network name for compose
ipam

---