Tuesday, 26 November 2019

How to make HTTPS request through HTTP proxy with Axios

I have a Node.js app which runs in node:alpine Docker image on TeamCity. It uses axios HTTP client to make HTTP(S) requests. To make it run it behind the proxy, I used:

docker run \
...
-e https_proxy="http://proxy.proxyhost.com:8080" \
...

...but request would fail with this error:

url: https://endpoint.target.com/whatever...
Error: Error: write EPROTO 139652420734280:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:

It is worth mentioning that for these cases:

  • No proxy specified at all
  • -e HTTPS_PROXY="http://proxy.proxyhost.com:8080" \
  • -e HTTP_PROXY="http://proxy.proxyhost.com:8080" \
  • -e http_proxy="http://proxy.proxyhost.com:8080" \


...the output of docker run was:

url: https://target.endpoint.com/whatever...
Error: Error: connect ETIMEDOUT 11.22.33.44:443

As Docker documentation [Configure Docker to use a proxy server | Docker Documentation] states, passing HTTPS_PROXY environment variable to docker run would make container use specified proxy.

Axios documentation [axios - npm] states:
  // 'proxy' defines the hostname and port of the proxy server.
  // You can also define your proxy using the conventional `http_proxy` and
  // `https_proxy` environment variables. 

I was passing both HTTPS_PROXY and https_proxy environment variable to docker run in hope that Axios would pick one up but this didn't happen. HTTPS requests would always time out.

Then I found that there is a bug in Axios...and some solutions:

Request to HTTPS with HTTP proxy fails · Issue #925 · axios/axios
Axios proxy is not working. · Issue #2072 · axios/axios
Node.js Axios behind corporate proxies - Jan Molak

Three npm packages were used in proposed workarounds:

https-proxy-agent - npm has 8,439,302 weekly downloads
tunnel - npm has 523,094 weekly downloads
axios-https-proxy-fix - npm has 6,132 weekly downloads

For its popularity I decided to go for solution with https-proxy-agent and fix that worked from me was this.

I tried to recreate the same environment in order to try to reproduce this error and try the fix. I created a temp/test build step in my TeamCity build config with these properties:

This is the temp step I used:

Proxy Test >> Build Step (1 of 1): Test Proxy from Node Docker container
Runner type: command line
Run: custom script
Run step within Docker container: docker.mydockerhub.com/node:alpine
Docker image platform: Linux
Additional docker run arguments: NONE!!!
Custom script:

echo Path:
echo $PATH

# Comment this if do not need to use custom APK package registry
APK_PACKAGE_REGISTRY=apks.myapkpackageregistry.com/alpine
sed -i "s:http\:\/\/dl-cdn.alpinelinux.org:https\:\/\/${APK_PACKAGE_REGISTRY}:g" /etc/apk/repositories
cat /etc/apk/repositories
apk --no-cache add ca-certificates
update-ca-certificates
apk add curl

echo First request...
curl -v -x http://proxy.proxyhost.com:8080 -L ' endpoint.target.com/whatever/...'

cd ~
mkdir proxytest
cd proxytest
npm init -y
echo 'registry=https://myresourceserver.com/npm
# reset auth related options if specified in ~/.npmrc file
_auth=""
always-auth=false' > .npmrc
cat .npmrc
npm install axios
npm install https-proxy-agent

echo "axios = require('axios');
const HttpsProxyAgent = require('https-proxy-agent');

const agent = new HttpsProxyAgent({host: 'proxy.proxyhost.com', port: '8080'});

//use axios as you normally would, but specify httpsAgent in the config
axios = axios.create({
    httpsAgent: agent
});

axios.get(' endpoint.target.com/whatever/...')
.then((response) => {
  console.log('response: ' + JSON.stringify(response.data));
}, (error) => {
  console.log('error: ' + error);
  process.exit(1);
});" > main.js
cat main.js
node main.js

Log:

Step 1/12: Test Proxy from Node Docker container (Command Line) (29s)
[08:14:30]Running step within Docker container docker.mydockerhub.com/node:alpine
[08:14:30]Starting: /bin/sh -c "docker pull docker.mydockerhub.com/node:alpine && ... && docker run --rm ... --entrypoint /bin/sh "docker.mydockerhub.com/node:alpine" /home/docker-agent/temp/agentTmp/docker-shell-script-1558438850936.sh"
[08:14:30]in directory: /home/docker-agent/work/8ede9871249b4
[08:14:30]alpine: Pulling from node
[08:14:31]89d9c30c1d48: Pulling fs layer
[08:14:31]5320ee7fe9ff: Pulling fs layer
[08:14:31]0a42696890fc: Pulling fs layer
[08:14:31]12c581b27455: Pulling fs layer
[08:14:31]12c581b27455: Waiting
[08:14:31]0a42696890fc: Verifying Checksum
[08:14:31]0a42696890fc: Download complete
[08:14:31]12c581b27455: Download complete
[08:14:31]89d9c30c1d48: Verifying Checksum
[08:14:31]89d9c30c1d48: Download complete
[08:14:31]89d9c30c1d48: Pull complete
[08:14:32]5320ee7fe9ff: Verifying Checksum
[08:14:32]5320ee7fe9ff: Download complete
[08:14:33]5320ee7fe9ff: Pull complete
[08:14:33]0a42696890fc: Pull complete
[08:14:33]12c581b27455: Pull complete
[08:14:33]Digest: sha256:bdf054f006078036f72dedfd53f3b11176c1c00d5451d8fc2af206636eb54d70
[08:14:33]Status: Downloaded newer image for docker.mydockerhub.com/node:alpine
[08:14:33]docker.mydockerhub.com/node:alpine
[08:14:34]Path:
[08:14:34]/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
...
[08:14:35](1/3) Installing nghttp2-libs (1.39.2-r0)
[08:14:35](2/3) Installing libcurl (7.66.0-r0)
[08:14:35](3/3) Installing curl (7.66.0-r0)
[08:14:35]Executing busybox-1.30.1-r2.trigger
[08:14:35]OK: 8 MiB in 20 packages
[08:14:35]First request...
[08:14:35]  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[08:14:35]                                 Dload  Upload   Total   Spent    Left  Speed
[08:14:35]
[08:14:35]  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.17.129.70:8080...
[08:14:35]* TCP_NODELAY set
[08:14:35]* Connected to proxy.proxyhost.com (11.22.33.44) port 8080 (#0)
[08:14:35]* allocate connect buffer!
[08:14:35]* Establish HTTP proxy tunnel to endpoint.target.com/whatever:443
[08:14:35]> CONNECT endpoint.target.com:443 HTTP/1.1
[08:14:35]> Host: endpoint.target.com:443
[08:14:35]> User-Agent: curl/7.66.0
[08:14:35]> Proxy-Connection: Keep-Alive
[08:14:35]> 
[08:14:35]< HTTP/1.1 200 Connection established
[08:14:35]< 
[08:14:35]* Proxy replied 200 to CONNECT request
[08:14:35]* CONNECT phase completed!
[08:14:35]* ALPN, offering h2
[08:14:35]* ALPN, offering http/1.1
[08:14:35]* successfully set certificate verify locations:
[08:14:35]*   CAfile: /etc/ssl/certs/ca-certificates.crt
[08:14:35]  CApath: none
[08:14:35]} [5 bytes data]
[08:14:35]* TLSv1.3 (OUT), TLS handshake, Client hello (1):
[08:14:35]} [512 bytes data]
[08:14:35]* CONNECT phase completed!
[08:14:35]* CONNECT phase completed!
[08:14:36]{ [5 bytes data]
[08:14:36]* TLSv1.3 (IN), TLS handshake, Server hello (2):
[08:14:36]{ [108 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Certificate (11):
[08:14:36]{ [2994 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
[08:14:36]{ [333 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Server finished (14):
[08:14:36]{ [4 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
[08:14:36]} [70 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
[08:14:36]} [1 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS handshake, Finished (20):
[08:14:36]} [16 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Finished (20):
[08:14:36]{ [16 bytes data]
[08:14:36]* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
[08:14:36]* ALPN, server accepted to use http/1.1
[08:14:36]* Server certificate:
[08:14:36]*  subject: C=CZ; L=London; O=MyCompany; OU=Devs; CN=*.target.com
[08:14:36]*  start date: Nov  1 00:00:00 2018 GMT
[08:14:36]*  expire date: Nov 14 12:00:00 2020 GMT
[08:14:36]*  subjectAltName: host "endpoint.target.com" matched cert's "*.target.com"
[08:14:36]*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
[08:14:36]*  SSL certificate verify ok.
[08:14:36]} [5 bytes data]
[08:14:36]> GET /whatever... HTTP/1.1
[08:14:36]> Host: endpoint.target.com
[08:14:36]> User-Agent: curl/7.66.0
[08:14:36]> Accept: */*
[08:14:36]> 
[08:14:36]{ [5 bytes data]
[08:14:36]* Mark bundle as not supporting multiuse
[08:14:36]< HTTP/1.1 301 Moved Permanently
[08:14:36]< Server: nginx/1.16.1
[08:14:36]< Date: Tue, 26 Nov 2019 07:14:36 GMT
[08:14:36]< Content-Type: text/html
[08:14:36]< Content-Length: 169
[08:14:36]< Location: https://endpoint.target.com/whatever...
[08:14:36]< Connection: keep-alive
[08:14:36]< 
[08:14:36]* Ignoring the response-body
[08:14:36]{ [169 bytes data]
[08:14:36]
[08:14:36]100   169  100   169    0     0    290      0 --:--:-- --:--:-- --:--:--   290
[08:14:36]* Connection #0 to host proxy.proxyhost.com left intact
[08:14:36]* Issue another request to this URL: 'https://endpoint.target.com/whatever...'
[08:14:36]* Found bundle for host endpoint.target.com: 0x561df56fb80 [serially]
[08:14:36]* Can not multiplex, even if we wanted to!
[08:14:36]* Re-using existing connection! (#0) with proxy proxy.proxyhost.com
[08:14:36]* Connected to proxy proxy.proxyhost.com (10.17.129.70) port 8080 (#0)
[08:14:36]} [5 bytes data]
[08:14:36]> GET /api/?post_type=block_filterlist&json=1&count=1000 HTTP/1.1
[08:14:36]> Host:  endpoint.target.com
[08:14:36]> User-Agent: curl/7.66.0
[08:14:36]> Accept: */*
[08:14:36]> 
[08:14:37]{ [5 bytes data]
[08:14:37]
[08:14:37]  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Mark bundle as not supporting multiuse
[08:14:37]< HTTP/1.1 200 OK
[08:14:37]< Server: nginx/1.16.1
[08:14:37]< Date: Tue, 26 Nov 2019 07:14:36 GMT
[08:14:37]< Content-Type: application/json; charset=UTF-8
[08:14:37]< Transfer-Encoding: chunked
[08:14:37]< Connection: keep-alive
[08:14:37]< 
[08:14:37]{ [13618 bytes data]
[08:14:37]
[08:14:37]100 13597    0 13597    0     0   9961      0 --:--:--  0:00:01 --:--:-- 3319k
[08:14:37]* Connection #0 to host proxy proxy.proxyhost.com left intact
[08:14:37]{"status":"ok",...}Wrote to /root/proxytest/package.json:
[08:14:37]
[08:14:37]{
[08:14:37]  "name": "proxytest",
[08:14:37]  "version": "1.0.0",
[08:14:37]  "description": "",
[08:14:37]  "main": "index.js",
[08:14:37]  "scripts": {
[08:14:37]    "test": "echo \"Error: no test specified\" && exit 1"
[08:14:37]  },
[08:14:37]  "keywords": [],
[08:14:37]  "author": "",
[08:14:37]  "license": "ISC"
[08:14:37]}
[08:14:37]
[08:14:37]
[08:14:37]registry=https://myresourceserver.com/npm
[08:14:37]# reset auth related options if specified in ~/.npmrc file
[08:14:37]_auth=""
[08:14:37]always-auth=false
[08:14:38]npm notice created a lockfile as package-lock.json. You should commit this file.
[08:14:38]npm WARN proxytest@1.0.0 No description
[08:14:38]npm WARN proxytest@1.0.0 No repository field.
[08:14:38]
[08:14:39]+ axios@0.19.0
[08:14:39]added 5 packages from 8 contributors and audited 5 packages in 1.441s
[08:14:39]found 0 vulnerabilities
[08:14:39]
[08:14:40]npm WARN proxytest@1.0.0 No description
[08:14:40]npm WARN proxytest@1.0.0 No repository field.
[08:14:40]
[08:14:41]+ https-proxy-agent@3.0.1
[08:14:41]added 4 packages from 3 contributors and audited 11 packages in 1.632s
[08:14:41]found 0 vulnerabilities
[08:14:41]
[08:14:41]axios = require('axios');
[08:14:41]const HttpsProxyAgent = require('https-proxy-agent');
[08:14:41]
[08:14:41]const agent = new HttpsProxyAgent({host: 'proxy proxy.proxyhost.com', port: '8080'});
[08:14:41]
[08:14:41]//use axios as you normally would, but specify httpsAgent in the config
[08:14:41]axios = axios.create({
[08:14:41]    httpsAgent: agent
[08:14:41]});
[08:14:41]
[08:14:41]axios.get('https:// endpoint.target.com/whatever...')
[08:14:41].then((response) => {
[08:14:41]  console.log('response: ' + JSON.stringify(response.data));
[08:14:41]}, (error) => {
[08:14:41]  console.log('error: ' + error);
[08:14:41]  process.exit(1);
[08:14:41]});
[08:14:43]response: {"status":"ok", ...}
[08:14:43]Process exited with code 0



1 comment:

hajikhatri said...

Really impressed! Everything is very open and very clear clarification of issues. It contains truly facts. Your website is very valuable. Thanks for sharing. free vpn