I had a case where I run a Terraform Docker container but it would fail to reach a provider package repository.
docker-compose.yaml:
---
name: terraform
services:
terraform:
image: hashicorp/terraform:latest
volumes:
- .:/infra
working_dir: /infra
I wanted to execute (terraform) init command (terraform argument here refers to the name of the service, not the terraform executable itself):
$ docker compose run --rm terraform init
[+] Creating 1/1
✔ Network import_demo_default Created 0.1s
[+] Running 5/5
✔ terraform Pulled 11.2s
✔ ec99f8b99825 Pull complete 4.2s
✔ 47bfda048af5 Pull complete 8.4s
✔ 755b9030e6bd Pull complete 8.4s
✔ db586b81a2dc Pull complete 9.4s
Initializing the backend...
Initializing provider plugins...
- Finding kreuzwerker/docker versions matching "3.0.2"...
╷
│ Error: Failed to query available provider packages
│
│ Could not retrieve the list of available versions for provider kreuzwerker/docker: could not connect to registry.terraform.io: failed to request discovery document: Get
│ "https://registry.terraform.io/.well-known/terraform.json": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
╵
I changed entrypoint in order to execute debugging tools. With either
entrypoint: ["wget", "https://registry.terraform.io/.well-known/terraform.json"]
or
entrypoint: ["ping", "registry.terraform.io"]
$ docker compose run --rm terraform
returned:
bad address 'registry.terraform.io'
...and
entrypoint: ["nslookup", "registry.terraform.io"]
returned:
;; connection timed out; no servers could be reached
To check the DNS servers used I set entrypoint to print the resolv.conf file:
entrypoint: ["cat", "/etc/resolv.conf"]
This returned:
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.
nameserver 127.0.0.11
search bigcorp.com
options edns0 trust-ad ndots:0
# Based on host file: '/etc/resolv.conf' (internal resolver)
# ExtServers: [10.10.1.255 10.11.5.183]
# Overrides: [nameservers search]
# Option ndots from: internal
By default, Docker provides a DNS server (daemon's embedded DNS resolver) at 127.0. 0.11, so all DNS requests from containers come to it. Daemon then forwards these requests to uplink DNS servers as defined via --dns arguments, /etc/docker/daemon.json or host's /etc/resolv.conf.
Containers use the same DNS servers as the host by default, but you can override this with --dns.By default, containers inherit the DNS settings as defined in the /etc/resolv.conf configuration file. Containers that attach to the default bridge network receive a copy of this file. Containers that attach to a custom network use Docker's embedded DNS server. The embedded DNS server forwards external DNS lookups to the DNS servers configured on the host.
Using --dns is the same as adding dns attribute to /etc/docker/daemon.json. Same applies for --dns-search. DNS settings in /etc/docker/daemon.json will override those set in the local /etc/resolv.conf file.
My local /etc/resolv.conf file:
$ cat /etc/resolv.conf
# This is /run/systemd/resolve/stub-resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
#
# This file might be symlinked as /etc/resolv.conf. If you're looking at
# /etc/resolv.conf and seeing this text, you have followed the symlink.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "resolvectl status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs should typically not access this file directly, but only
# through the symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a
# different way, replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0 trust-ad
search Home
In my case, uplink DNS server is my local router:
$ resolvectl status
Global
Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub
Link 2 (enp0s31f6)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 3 (wlp2s0)
Current Scopes: DNS
Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: xxxx:a25c:xxxx:0:xxxx:7aff:fe4d:3700
DNS Servers: 192.168.0.1 xxxx:a25c:xxxx:0:xxxx:7aff:fe4d:3700
DNS Domain: Home
Link 4 (br-a7ba833104f5)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 5 (br-d39e3c16b90f)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 6 (docker0)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 7 (br-1d4f7fd2e5cc)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 8 (br-3c8c9487a095)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 9 (br-7bfedc7c4369)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 26 (veth846e490)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 27 (br-c06da6a5a65a)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 30 (br-c1e0d2aed078)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Link 71 (enxa44cc8e41d0f)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
I discovered that I had DNS settings set in /etc/docker/daemon.json:
$ cat /etc/docker/daemon.json
{
"dns": ["10.10.1.255", "10.11.5.183"],
"dns-search": ["bigcorp.com"]
}
As there is no need to use these custom (corporate) DNS servers, I can remove these settings (basically empty) /etc/docker/daemon.json.
To reload the new (empty) config, I had to flush changes and restart Docker:
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker
Let's check how container's /etc/resolv.conf changed:
$ docker compose run --rm terraform
# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.
nameserver 127.0.0.11
search Home
options edns0 trust-ad ndots:0
# Based on host file: '/etc/resolv.conf' (internal resolver)
# ExtServers: [host(127.0.0.53)]
# Overrides: []
# Option ndots from: internal
Switching entrypoint to nslookup:
entrypoint: ["nslookup", "registry.terraform.io"]
...gives now the expected result:
$ docker compose run --rm terraform
Server: 127.0.0.11
Address: 127.0.0.11:53
Non-authoritative answer:
registry.terraform.io canonical name = d3rdzqodp6w8cx.cloudfront.net
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 2600:9000:225d:ee00:16:1aa3:1440:93a1
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 2600:9000:225d:7c00:16:1aa3:1440:93a1
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 2600:9000:225d:4a00:16:1aa3:1440:93a1
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 2600:9000:225d:4800:16:1aa3:1440:93a1
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 2600:9000:225d:8200:16:1aa3:1440:93a1
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 2600:9000:225d:7200:16:1aa3:1440:93a1
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 2600:9000:225d:2000:16:1aa3:1440:93a1
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 2600:9000:225d:e000:16:1aa3:1440:93a1
Non-authoritative answer:
registry.terraform.io canonical name = d3rdzqodp6w8cx.cloudfront.net
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 143.204.68.98
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 143.204.68.95
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 143.204.68.128
Name: d3rdzqodp6w8cx.cloudfront.net
Address: 143.204.68.94
Finally, after removing entrypoint alltogether:
$ docker compose run --rm terraform init
Initializing the backend...
Initializing provider plugins...
- Finding kreuzwerker/docker versions matching "3.0.2"...
- Installing kreuzwerker/docker v3.0.2...
- Installed kreuzwerker/docker v3.0.2 (self-signed, key ID BD080C4571C6104C)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
In case there were issues with any of the further uplink DNS resolvers (DNS setting on my local router, DNS issues with my Internet provider etc...) I would try using Google DNS servers directly:
$ cat /etc/docker/daemon.json
{
"dns": ["8.8.4.4", "8.8.8.8"],
}
But for now I can keep that config file empty.