Tuesday, 26 November 2019

How to make HTTPS request through HTTP proxy with Axios

I have a Node.js app which runs in node:alpine Docker image on TeamCity. It uses axios HTTP client to make HTTP(S) requests. To make it run it behind the proxy, I used:

docker run \
...
-e https_proxy="http://proxy.proxyhost.com:8080" \
...

...but request would fail with this error:

url: https://endpoint.target.com/whatever...
Error: Error: write EPROTO 139652420734280:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:

It is worth mentioning that for these cases:

  • No proxy specified at all
  • -e HTTPS_PROXY="http://proxy.proxyhost.com:8080" \
  • -e HTTP_PROXY="http://proxy.proxyhost.com:8080" \
  • -e http_proxy="http://proxy.proxyhost.com:8080" \


...the output of docker run was:

url: https://target.endpoint.com/whatever...
Error: Error: connect ETIMEDOUT 11.22.33.44:443

As Docker documentation [Configure Docker to use a proxy server | Docker Documentation] states, passing HTTPS_PROXY environment variable to docker run would make container use specified proxy.

Axios documentation [axios - npm] states:
  // 'proxy' defines the hostname and port of the proxy server.
  // You can also define your proxy using the conventional `http_proxy` and
  // `https_proxy` environment variables. 

I was passing both HTTPS_PROXY and https_proxy environment variable to docker run in hope that Axios would pick one up but this didn't happen. HTTPS requests would always time out.

Then I found that there is a bug in Axios...and some solutions:

Request to HTTPS with HTTP proxy fails · Issue #925 · axios/axios
Axios proxy is not working. · Issue #2072 · axios/axios
Node.js Axios behind corporate proxies - Jan Molak

Three npm packages were used in proposed workarounds:

https-proxy-agent - npm has 8,439,302 weekly downloads
tunnel - npm has 523,094 weekly downloads
axios-https-proxy-fix - npm has 6,132 weekly downloads

For its popularity I decided to go for solution with https-proxy-agent and fix that worked from me was this.

I tried to recreate the same environment in order to try to reproduce this error and try the fix. I created a temp/test build step in my TeamCity build config with these properties:

This is the temp step I used:

Proxy Test >> Build Step (1 of 1): Test Proxy from Node Docker container
Runner type: command line
Run: custom script
Run step within Docker container: docker.mydockerhub.com/node:alpine
Docker image platform: Linux
Additional docker run arguments: NONE!!!
Custom script:

echo Path:
echo $PATH

# Comment this if do not need to use custom APK package registry
APK_PACKAGE_REGISTRY=apks.myapkpackageregistry.com/alpine
sed -i "s:http\:\/\/dl-cdn.alpinelinux.org:https\:\/\/${APK_PACKAGE_REGISTRY}:g" /etc/apk/repositories
cat /etc/apk/repositories
apk --no-cache add ca-certificates
update-ca-certificates
apk add curl

echo First request...
curl -v -x http://proxy.proxyhost.com:8080 -L ' endpoint.target.com/whatever/...'

cd ~
mkdir proxytest
cd proxytest
npm init -y
echo 'registry=https://myresourceserver.com/npm
# reset auth related options if specified in ~/.npmrc file
_auth=""
always-auth=false' > .npmrc
cat .npmrc
npm install axios
npm install https-proxy-agent

echo "axios = require('axios');
const HttpsProxyAgent = require('https-proxy-agent');

const agent = new HttpsProxyAgent({host: 'proxy.proxyhost.com', port: '8080'});

//use axios as you normally would, but specify httpsAgent in the config
axios = axios.create({
    httpsAgent: agent
});

axios.get(' endpoint.target.com/whatever/...')
.then((response) => {
  console.log('response: ' + JSON.stringify(response.data));
}, (error) => {
  console.log('error: ' + error);
  process.exit(1);
});" > main.js
cat main.js
node main.js

Log:

Step 1/12: Test Proxy from Node Docker container (Command Line) (29s)
[08:14:30]Running step within Docker container docker.mydockerhub.com/node:alpine
[08:14:30]Starting: /bin/sh -c "docker pull docker.mydockerhub.com/node:alpine && ... && docker run --rm ... --entrypoint /bin/sh "docker.mydockerhub.com/node:alpine" /home/docker-agent/temp/agentTmp/docker-shell-script-1558438850936.sh"
[08:14:30]in directory: /home/docker-agent/work/8ede9871249b4
[08:14:30]alpine: Pulling from node
[08:14:31]89d9c30c1d48: Pulling fs layer
[08:14:31]5320ee7fe9ff: Pulling fs layer
[08:14:31]0a42696890fc: Pulling fs layer
[08:14:31]12c581b27455: Pulling fs layer
[08:14:31]12c581b27455: Waiting
[08:14:31]0a42696890fc: Verifying Checksum
[08:14:31]0a42696890fc: Download complete
[08:14:31]12c581b27455: Download complete
[08:14:31]89d9c30c1d48: Verifying Checksum
[08:14:31]89d9c30c1d48: Download complete
[08:14:31]89d9c30c1d48: Pull complete
[08:14:32]5320ee7fe9ff: Verifying Checksum
[08:14:32]5320ee7fe9ff: Download complete
[08:14:33]5320ee7fe9ff: Pull complete
[08:14:33]0a42696890fc: Pull complete
[08:14:33]12c581b27455: Pull complete
[08:14:33]Digest: sha256:bdf054f006078036f72dedfd53f3b11176c1c00d5451d8fc2af206636eb54d70
[08:14:33]Status: Downloaded newer image for docker.mydockerhub.com/node:alpine
[08:14:33]docker.mydockerhub.com/node:alpine
[08:14:34]Path:
[08:14:34]/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
...
[08:14:35](1/3) Installing nghttp2-libs (1.39.2-r0)
[08:14:35](2/3) Installing libcurl (7.66.0-r0)
[08:14:35](3/3) Installing curl (7.66.0-r0)
[08:14:35]Executing busybox-1.30.1-r2.trigger
[08:14:35]OK: 8 MiB in 20 packages
[08:14:35]First request...
[08:14:35]  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[08:14:35]                                 Dload  Upload   Total   Spent    Left  Speed
[08:14:35]
[08:14:35]  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.17.129.70:8080...
[08:14:35]* TCP_NODELAY set
[08:14:35]* Connected to proxy.proxyhost.com (11.22.33.44) port 8080 (#0)
[08:14:35]* allocate connect buffer!
[08:14:35]* Establish HTTP proxy tunnel to endpoint.target.com/whatever:443
[08:14:35]> CONNECT endpoint.target.com:443 HTTP/1.1
[08:14:35]> Host: endpoint.target.com:443
[08:14:35]> User-Agent: curl/7.66.0
[08:14:35]> Proxy-Connection: Keep-Alive
[08:14:35]> 
[08:14:35]< HTTP/1.1 200 Connection established
[08:14:35]< 
[08:14:35]* Proxy replied 200 to CONNECT request
[08:14:35]* CONNECT phase completed!
[08:14:35]* ALPN, offering h2
[08:14:35]* ALPN, offering http/1.1
[08:14:35]* successfully set certificate verify locations:
[08:14:35]*   CAfile: /etc/ssl/certs/ca-certificates.crt
[08:14:35]  CApath: none
[08:14:35]} [5 bytes data]
[08:14:35]* TLSv1.3 (OUT), TLS handshake, Client hello (1):
[08:14:35]} [512 bytes data]
[08:14:35]* CONNECT phase completed!
[08:14:35]* CONNECT phase completed!
[08:14:36]{ [5 bytes data]
[08:14:36]* TLSv1.3 (IN), TLS handshake, Server hello (2):
[08:14:36]{ [108 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Certificate (11):
[08:14:36]{ [2994 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
[08:14:36]{ [333 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Server finished (14):
[08:14:36]{ [4 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
[08:14:36]} [70 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
[08:14:36]} [1 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS handshake, Finished (20):
[08:14:36]} [16 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Finished (20):
[08:14:36]{ [16 bytes data]
[08:14:36]* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
[08:14:36]* ALPN, server accepted to use http/1.1
[08:14:36]* Server certificate:
[08:14:36]*  subject: C=CZ; L=London; O=MyCompany; OU=Devs; CN=*.target.com
[08:14:36]*  start date: Nov  1 00:00:00 2018 GMT
[08:14:36]*  expire date: Nov 14 12:00:00 2020 GMT
[08:14:36]*  subjectAltName: host "endpoint.target.com" matched cert's "*.target.com"
[08:14:36]*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
[08:14:36]*  SSL certificate verify ok.
[08:14:36]} [5 bytes data]
[08:14:36]> GET /whatever... HTTP/1.1
[08:14:36]> Host: endpoint.target.com
[08:14:36]> User-Agent: curl/7.66.0
[08:14:36]> Accept: */*
[08:14:36]> 
[08:14:36]{ [5 bytes data]
[08:14:36]* Mark bundle as not supporting multiuse
[08:14:36]< HTTP/1.1 301 Moved Permanently
[08:14:36]< Server: nginx/1.16.1
[08:14:36]< Date: Tue, 26 Nov 2019 07:14:36 GMT
[08:14:36]< Content-Type: text/html
[08:14:36]< Content-Length: 169
[08:14:36]< Location: https://endpoint.target.com/whatever...
[08:14:36]< Connection: keep-alive
[08:14:36]< 
[08:14:36]* Ignoring the response-body
[08:14:36]{ [169 bytes data]
[08:14:36]
[08:14:36]100   169  100   169    0     0    290      0 --:--:-- --:--:-- --:--:--   290
[08:14:36]* Connection #0 to host proxy.proxyhost.com left intact
[08:14:36]* Issue another request to this URL: 'https://endpoint.target.com/whatever...'
[08:14:36]* Found bundle for host endpoint.target.com: 0x561df56fb80 [serially]
[08:14:36]* Can not multiplex, even if we wanted to!
[08:14:36]* Re-using existing connection! (#0) with proxy proxy.proxyhost.com
[08:14:36]* Connected to proxy proxy.proxyhost.com (10.17.129.70) port 8080 (#0)
[08:14:36]} [5 bytes data]
[08:14:36]> GET /api/?post_type=block_filterlist&json=1&count=1000 HTTP/1.1
[08:14:36]> Host:  endpoint.target.com
[08:14:36]> User-Agent: curl/7.66.0
[08:14:36]> Accept: */*
[08:14:36]> 
[08:14:37]{ [5 bytes data]
[08:14:37]
[08:14:37]  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Mark bundle as not supporting multiuse
[08:14:37]< HTTP/1.1 200 OK
[08:14:37]< Server: nginx/1.16.1
[08:14:37]< Date: Tue, 26 Nov 2019 07:14:36 GMT
[08:14:37]< Content-Type: application/json; charset=UTF-8
[08:14:37]< Transfer-Encoding: chunked
[08:14:37]< Connection: keep-alive
[08:14:37]< 
[08:14:37]{ [13618 bytes data]
[08:14:37]
[08:14:37]100 13597    0 13597    0     0   9961      0 --:--:--  0:00:01 --:--:-- 3319k
[08:14:37]* Connection #0 to host proxy proxy.proxyhost.com left intact
[08:14:37]{"status":"ok",...}Wrote to /root/proxytest/package.json:
[08:14:37]
[08:14:37]{
[08:14:37]  "name": "proxytest",
[08:14:37]  "version": "1.0.0",
[08:14:37]  "description": "",
[08:14:37]  "main": "index.js",
[08:14:37]  "scripts": {
[08:14:37]    "test": "echo \"Error: no test specified\" && exit 1"
[08:14:37]  },
[08:14:37]  "keywords": [],
[08:14:37]  "author": "",
[08:14:37]  "license": "ISC"
[08:14:37]}
[08:14:37]
[08:14:37]
[08:14:37]registry=https://myresourceserver.com/npm
[08:14:37]# reset auth related options if specified in ~/.npmrc file
[08:14:37]_auth=""
[08:14:37]always-auth=false
[08:14:38]npm notice created a lockfile as package-lock.json. You should commit this file.
[08:14:38]npm WARN proxytest@1.0.0 No description
[08:14:38]npm WARN proxytest@1.0.0 No repository field.
[08:14:38]
[08:14:39]+ axios@0.19.0
[08:14:39]added 5 packages from 8 contributors and audited 5 packages in 1.441s
[08:14:39]found 0 vulnerabilities
[08:14:39]
[08:14:40]npm WARN proxytest@1.0.0 No description
[08:14:40]npm WARN proxytest@1.0.0 No repository field.
[08:14:40]
[08:14:41]+ https-proxy-agent@3.0.1
[08:14:41]added 4 packages from 3 contributors and audited 11 packages in 1.632s
[08:14:41]found 0 vulnerabilities
[08:14:41]
[08:14:41]axios = require('axios');
[08:14:41]const HttpsProxyAgent = require('https-proxy-agent');
[08:14:41]
[08:14:41]const agent = new HttpsProxyAgent({host: 'proxy proxy.proxyhost.com', port: '8080'});
[08:14:41]
[08:14:41]//use axios as you normally would, but specify httpsAgent in the config
[08:14:41]axios = axios.create({
[08:14:41]    httpsAgent: agent
[08:14:41]});
[08:14:41]
[08:14:41]axios.get('https:// endpoint.target.com/whatever...')
[08:14:41].then((response) => {
[08:14:41]  console.log('response: ' + JSON.stringify(response.data));
[08:14:41]}, (error) => {
[08:14:41]  console.log('error: ' + error);
[08:14:41]  process.exit(1);
[08:14:41]});
[08:14:43]response: {"status":"ok", ...}
[08:14:43]Process exited with code 0



3 comments:

hajikhatri said...

Really impressed! Everything is very open and very clear clarification of issues. It contains truly facts. Your website is very valuable. Thanks for sharing. free vpn

Unknown said...

Thanks for sharing the details really helpful. The post help me to sort thing easily. Great work sharing with the free for free

micheal pan said...

BE SMART AND BECOME RICH IN LESS THAN 3DAYS....It all depends on how fast 
you can be to get the new PROGRAMMED blank ATM card that is capable of
hacking into any ATM machine,anywhere in the world. I got to know about 
this BLANK ATM CARD when I was searching for job online about a month 
ago..It has really changed my life for good and now I can say I'm rich and 
I can never be poor again. The least money I get in a day with it is about 
$50,000.(fifty thousand USD) Every now and then I keeping pumping money 
into my account. Though is illegal,there is no risk of being caught 
,because it has been programmed in such a way that it is not traceable,it 
also has a technique that makes it impossible for the CCTVs to detect 
you..For details on how to get yours today, email the hackers on : (
atmmachinehackers1@gmail.com ). Tell your 
loved once too, and start to live large. That's the simple testimony of how 
my life changed for good...Love you all ...the email address again is ;
atmmachinehackers1@gmail.com