How to upgrade docker-compose on Linux

Let's check the version of currently installed docker-compose:

 

$ docker-compose --version
docker-compose version 1.27.4, build 40524192
 

Let's verify the path to executable:

$ which docker-compose
/usr/local/bin/docker-compose

To upgrade docker-compose, we first need to remove the old version:

$ sudo rm /usr/local/bin/docker-compose
[sudo] password for user: 

 

We can then download the binary of the latest stable version (1.29.2 at time of writing)

 

$ sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   664  100   664    0     0   2114      0 --:--:-- --:--:-- --:--:--  2121
100 12.1M  100 12.1M    0     0  2544k      0  0:00:04  0:00:04 --:--:-- 2723k
 

Add permissions to execute the newly downloaded binary: 

$ sudo chmod +x /usr/local/bin/docker-compose
 

Let's verify the new version:

 

$ docker-compose --version
docker-compose version 1.29.2, build 5becea4c

References

Install Docker Compose | Docker Documentation

Running pgAdmin in Docker container

pgAdmin is a browser-based DB client. It is possible to run it from a Docker container  - an image is available at DockerHub: dpage/pgadmin4.

I assume we are also running PostgresDB Docker container.

To run pgAdmin Docker container on the same network as PostgresDB container execute:

$  docker run \
-p 5051:5051 \
-d \
-e "PGADMIN_DEFAULT_EMAIL=xxxxxxx@example.com" \
-e "PGADMIN_DEFAULT_PASSWORD=xxxxxxx" \
-e "PGADMIN_LISTEN_PORT=5051" \
--rm \
--name pgadmin \
--network my_network_default \
dpage/pgadmin4

my_network_default is the name of the Docker network on which Postgres DB container is running. This allows using DB service name (as specified in docker-compose.yml) as the DB hostname when adding DB server in pgAdmin4. This is possible if DB container is run via docker-compose.

Once this container is up we can go to http://localhost:5051 in local browser and log in with credentials specified via PGADMIN_DEFAULT_EMAIL and PGADMIN_DEFAULT_PASSWORD. To add a new DB server we need to know either its hostname or its IP address.

If both containers are running on the same Docker network (which is our case here as we use --network) then for DB container hostname we can simply use the name of the DB service from docker-compose.yml e.g. db.

If we want to go via DB container IP address route we can find it out we can inspect Postgres container network:

$ docker inspect my_network_default

If you're using VSCode for development, use Docker plugin feature NETWORKS, right-click network of interest and select inspect.

Resolving Issues

In case of any issues, remove -d from this command line in order to run this container undetached in which case all useful output will appear in terminal.

How to update pgAdmin Docker image? 

To update pgAdmin Docker image and then run it, use:

$ docker pull dpage/pgadmin4 && docker run (...) dpage/pgadmin4

How to Persist Data between pgAdmin sessions?

If we restart pgAdmin container, all serves we added before will be lost and we'll need to add them again. To prevent that we can use local host directory as a mounted volume for persistence:

$ docker run \
-p 5051:5051 \
-d \
-e "PGADMIN_DEFAULT_EMAIL=xxxxxxx@example.com" \
-e "PGADMIN_DEFAULT_PASSWORD=xxxxxxx" \
-e "PGADMIN_LISTEN_PORT=5051" \
--rm \
--name pgadmin \
--network my_network_default \
-v "$(pwd)/pgadmin_data/servers.json":/pgadmin4/servers.json \
-v "$(pwd)/pgadmin_data/pgadmin":/var/lib/pgadmin \
dpage/pgadmin4

It is not necessary to manually create local directory ./pgadmin_data/, it will be created by Docker but it will set both the owner and the group to root:

$ ls -la 

drwxr-xr-x  4 root  root  4096 Oct  7 16:39 pgadmin_data

From pgAdmin4 docs:

/var/lib/pgadmin - This is the working directory in which pgAdmin stores session data, user files, configuration files, and it’s configuration database. Mapping this directory onto the host machine gives you an easy way to maintain configuration between invocations of the container.

/pgadmin4/servers.json - If this file is mapped, server definitions found in it will be loaded at launch time. This allows connection information to be pre-loaded into the instance of pgAdmin in the container. Note that server definitions are only loaded on first launch, i.e. when the configuration database is created, and not on subsequent launches using the same configuration database.

pgAdmin runs as the pgadmin user (UID: 5050) in the pgadmin group (GID: 5050) in the container. You must ensure that all files are readable, and where necessary (e.g. the working/session directory) for this user on the host machine. 

As pgAdmin can't access directory owned by root, we need to change the ownership manually:

$ sudo chown -R 5050:5050 <host_directory>

or, in our case:

$ sudo chown -R 5050:5050 ./pgadmin_data/

After this, reload http://localhost:5051 and the login page should load with no issues.

I came across this pgadmin user issue when running pgAdmin from docker-compose. 

docker-compose.yml:

version: '3.7'

services:

  pg_db:

    image: postgres

    container_name: my-postgres

    volumes:

      - ./database_data:/var/lib/postgresql/data

    ports:

      - 5432:5432

    environment:

      POSTGRES_DB: $PGDATABASE

      POSTGRES_USER: $PGUSER

      POSTGRES_PASSWORD: $PGPASSWORD

    stdin_open: true

    tty: true

  pg_admin:

    image: dpage/pgadmin4

    container_name: my-pgadmin

    depends_on:

      - pg_db

    restart: always

    environment:

      PGADMIN_DEFAULT_EMAIL: test@example.com

      PGADMIN_DEFAULT_PASSWORD: postgres

      PGADMIN_LISTEN_PORT: 5052

    volumes:

      - ./pgadmin:/var/lib/pgadmin

    ports:

    - 5052:5052

docker-compose up output:

$ docker-compose up

...

my-pgadmin | WARNING: Failed to set ACL on the directory containing the configuration database:

my-pgadmin |            [Errno 1] Operation not permitted: '/var/lib/pgadmin'

my-pgadmin | HINT   : You may need to manually set the permissions on

my-pgadmin |          /var/lib/pgadmin to allow pgadmin to write to it.

my-pgadmin | ERROR  : Failed to create the directory /var/lib/pgadmin/sessions:

my-pgadmin |            [Errno 13] Permission denied: '/var/lib/pgadmin/sessions'

my-pgadmin | HINT   : Create the directory /var/lib/pgadmin/sessions, ensure it is writeable by

my-pgadmin |          'pgadmin', and try again, or, create a config_local.py file

my-pgadmin |          and override the SESSION_DB_PATH setting per

my-pgadmin |          https://www.pgadmin.org/docs/pgadmin4/4.26/config_py.html

my-pgadmin | sudo: setrlimit(RLIMIT_CORE): Operation not permitted

my-pgadmin | [2021-04-19 15:26:59 +0000] [1] [INFO] Starting gunicorn 19.9.0

my-pgadmin | [2021-04-19 15:26:59 +0000] [1] [INFO] Listening at: http://[::]:5052 (1)

my-pgadmin | [2021-04-19 15:26:59 +0000] [1] [INFO] Using worker: threads

my-pgadmin | /usr/local/lib/python3.8/os.py:1023: RuntimeWarning: line buffering (buffering=1) isn't supported in binary mode, the default buffer size will be used

my-pgadmin |   return io.open(fd, *args, **kwargs)

my-pgadmin | [2021-04-19 15:26:59 +0000] [89] [INFO] Booting worker with pid: 89

...

linux - Permission denied: '/var/lib/pgadmin/sessions' in Docker - Stack Overflow

$ sudo chown -R 5050:5050 ./pgadmin/

$ ls -la  ./pgadmin/

total 8

drwxr-xr-x 2  5050  5050 4096 Apr 19 16:26 .

drwxrwxr-x 7 bojan bojan 4096 Apr 19 16:26 ..

How to export and view exported table?

If we export some table into a csv file, we can see that the following command is executed:

This command is run when using pgAdmin web application to export DB table my_table to csv file:

"/usr/local/pgsql-11/psql" --command " "\\copy public.my_table (column1_name, column2_name...) TO '<STORAGE_DIR>/my_table.csv' CSV QUOTE '\"' ESCAPE '''';""

STORAGE_DIR path is defined in /pgadmin4/config.py.

We can view csv file it if we attach to pgadmin container's terminal:

$ docker exec -it pgadmin /bin/sh
/pgadmin4 # ls -la

...

/var/lib/pgadmin/storage/test_example.com # ls

my_table_export.csv

/var/lib/pgadmin/storage/test_example.com # cat my_table.csv 

How to check pgAdmin4 log file?

Path to pgAdmin log file on Linux is: ~/.pgadmin/pgadmin4.log [FAQ]

Introduction to docker-compose

docker-compose

Helps managing Docker containers and their dependencies. Its configuration is in docker-compose.yml file

It can also be used in a single project with Dockerfile as instead of manually running docker build <args> and then docker run <args> we can put all these args in docker-compose.yml and then call docker-compose up.

UPDATE (July 2023): It is now recommended to use Compose V2 which comes with syntax docker compose (instead of docker-compose). [Migrate to Compose V2 | Docker Docs]

Installation

Install Docker Compose | Docker Documentation

To upgrade it, first uninstall and then install it again. I described the whole process in this article: How to upgrade docker-compose on Linux | My Public Notepad.

Configuration file

docker-compose reads its configuration from a file called docker-compose.yml.

Directories specified in volumes section in yml file don't have to be created manually - docker-compose up will create them.

CLI

$ docker-compose

Define and run multi-container applications with Docker.

Usage:
  docker-compose [-f <arg>...] [options] [COMMAND] [ARGS...]
  docker-compose -h|--help

Options:
  -f, --file FILE             Specify an alternate compose file
                              (default: docker-compose.yml)
  -p, --project-name NAME     Specify an alternate project name
                              (default: directory name)
  --verbose                   Show more output
  --log-level LEVEL           Set log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)
  --no-ansi                   Do not print ANSI control characters
  -v, --version               Print version and exit
  -H, --host HOST             Daemon socket to connect to

  --tls                       Use TLS; implied by --tlsverify
  --tlscacert CA_PATH         Trust certs signed only by this CA
  --tlscert CLIENT_CERT_PATH  Path to TLS certificate file
  --tlskey TLS_KEY_PATH       Path to TLS key file
  --tlsverify                 Use TLS and verify the remote
  --skip-hostname-check       Don't check the daemon's hostname against the
                              name specified in the client certificate
  --project-directory PATH    Specify an alternate working directory
                              (default: the path of the Compose file)
  --compatibility             If set, Compose will attempt to convert deploy
                              keys in v3 files to their non-Swarm equivalent

Commands:
  build              Build or rebuild services
  bundle             Generate a Docker bundle from the Compose file
  config             Validate and view the Compose file
  create             Create services
  down               Stop and remove containers, networks, images, and volumes
  events             Receive real time events from containers
  exec               Execute a command in a running container
  help               Get help on a command
  images             List images
  kill               Kill containers
  logs               View output from containers
  pause              Pause services
  port               Print the public port for a port binding
  ps                 List containers
  pull               Pull service images
  push               Push service images
  restart            Restart services
  rm                 Remove stopped containers
  run                Run a one-off command
  scale              Set number of containers for a service
  start              Start services
  stop               Stop services
  top                Display the running processes
  unpause            Unpause services
  up                 Create and start containers
  version            Show the Docker-Compose version information

Here are the options from Compose V2:

$ docker compose --help

Usage:  docker compose [OPTIONS] COMMAND

Define and run multi-container applications with Docker

Options:

      --all-resources              Include all resources, even those not used by services

      --ansi string                Control when to print ANSI control characters ("never"|"always"|"auto") (default "auto")

      --compatibility              Run compose in backward compatibility mode

      --dry-run                    Execute command in dry run mode

      --env-file stringArray       Specify an alternate environment file

  -f, --file stringArray           Compose configuration files

      --parallel int               Control max parallelism, -1 for unlimited (default -1)

      --profile stringArray        Specify a profile to enable

      --progress string            Set type of progress output (auto, tty, plain, quiet) (default "auto")

      --project-directory string   Specify an alternate working directory

                                   (default: the path of the, first specified, Compose file)

  -p, --project-name string        Project name

Commands:

  attach      Attach local standard input, output, and error streams to a service's running container

  build       Build or rebuild services

  config      Parse, resolve and render compose file in canonical format

  cp          Copy files/folders between a service container and the local filesystem

  create      Creates containers for a service

  down        Stop and remove containers, networks

  events      Receive real time events from containers

  exec        Execute a command in a running container

  images      List images used by the created containers

  kill        Force stop service containers

  logs        View output from containers

  ls          List running compose projects

  pause       Pause services

  port        Print the public port for a port binding

  ps          List containers

  pull        Pull service images

  push        Push service images

  restart     Restart service containers

  rm          Removes stopped service containers

  run         Run a one-off command on a service

  scale       Scale services 

  start       Start services

  stats       Display a live stream of container(s) resource usage statistics

  stop        Stop services

  top         Display the running processes

  unpause     Unpause services

  up          Create and start containers

  version     Show the Docker Compose version information

  wait        Block until the first service container stops

  watch       Watch build context for service and rebuild/refresh containers when files are updated

Run 'docker compose COMMAND --help' for more information on a command.

Its configuration file is docker-compose.yaml (or .yml). This file is also known as Compose file. It contains:

• services - list of service names; Each service can have:
• build - 
• command - 
• environment - 
• image - 
• ports - 
• restart - 
• volumes - 

docker-compose up will read Compose file, create and start containers (services) start listed there.

To check the state of all services, run docker-compose ps from the working directory (which is by default the one that contains Compose file):

$ docker-compose ps

         Name                   Command           State           Ports         

--------------------------------------------------------------------------------

my-project_service1_1    /bin/sh -c yarn          Up      0.0.0.0:3001->3001/tcp

                         install && ...                                         

my-project__postgres_1   postgres.sh              Up      0.0.0.0:5432->5432/tcp                                             

my-project_service3_1    /usr/local/bin/example   Up      0.0.0.0:4567->4567/tcp                              

To list the IDs of all running containers:

$ docker-compose ps -q

1e8a22167a4ac8354a456a29550cc7fd5fa5eb80ce68f6db04db8ba9764733cf

a1404d3f917c5daf8921731175ae0aa0940eccb518e9c0fd786183036d000d20

e6aeaf2e79b10a29bc834b0e3f712add1c02216b50afa1193663248162721b86

To list all images:

$ docker-compose images

       Container       Repository   Tag       Image Id      Size 

------------------------------------------------------------------

my-project_service1_1   service1   latest   58f28f305f80   875 MB

my-project__postgres_1  postgres   latest   f97c959a7d9c   298 MB

my-project_service3_1   service3   latest   4e02459e1040   159 MB

Another way to find the association between running containers and their IDs:

$ docker ps

CONTAINER ID    IMAGE   COMMAND                     CREATED       STATUS     PORTS                    NAMES

a1403d3f917c  postgres  "postgres.s…"               29 hours ago  Up 3 hours 0.0.0.0:5432->5432/tcp   postgres

e6aecf2e79b1  service3  "/usr/local/bin/example…"   29 hours ago  Up 3 hours 0.0.0.0:4567->4567/tcp   browser-engagement_s3_1

1e8a12167a4a  service1  "/bin/sh -c 'yarn in…"      29 hours ago  Up 3 hours 0.0.0.0:3001->3001/tcp   my-project_service1_1

Stops containers and removes containers, networks, volumes, and images
created by `up`.

docker-compose down

It completely destroys containers (but not images).

$ docker-compose down --help
Stops containers and removes containers, networks, volumes, and images
created by `up`.

By default, the only things removed are:

- Containers for services defined in the Compose file
- Networks defined in the `networks` section of the Compose file
- The default network, if one is used

Networks and volumes defined as `external` are never removed.

Usage: down [options]

Options:
    --rmi type              Remove images. Type must be one of:
                              'all': Remove all images used by any service.
                              'local': Remove only images that don't have a
                              custom tag set by the `image` field.
    -v, --volumes           Remove named volumes declared in the `volumes`
                            section of the Compose file and anonymous volumes
                            attached to containers.
    --remove-orphans        Remove containers for services not defined in the
                            Compose file
    -t, --timeout TIMEOUT   Specify a shutdown timeout in seconds.
                            (default: 10)


Example:

$ docker-compose down --rmi all --volumes

docker-compose: option to automaticaly remove container after run in docker-compose.yml

docker-compose pull

Let's assume our docker-compose.yml contains two services, db and db_sync and each of them has some Dockerfile which defines base image for their containers. When we do docker-compose up first time, Docker will pull the latest version of those images but each subsequent docker-compose up will not be checking if base images have newer version in the remote image repository - we need to update these images manually. We can look Dockerfile and update each base image manually, like:

$ docker pull node:alpine

We might make mistake and omit some images. Better way is to call docker-compose pull (from a directory which contains yml file) - it will iterate through each service listed in yml and update its base image(s). Example:

$ docker-compose pull
Pulling db     ... done
Pulling db_sync ... done

docker-compose pull
docker-compose up doesn't pull down latest image if the image exists locally #3574

docker-compose restart

It is possible to restart a single container (service). Eg. if we have an app container and db container and we want to restart only app container we can do:

$ docker-compose restart my-app

Caveat: this does not pick up any changes in yml config file or image itself. To rebuild the image and restart the container:

$ docker-compose build my-app && docker-compose restart my-app