Tuesday 26 November 2019

How to make HTTPS request through HTTP proxy with Axios

I have a Node.js app which runs in node:alpine Docker image on TeamCity. It uses axios HTTP client to make HTTP(S) requests. To make it run it behind the proxy, I used:

docker run \
...
-e https_proxy="http://proxy.proxyhost.com:8080" \
...

...but request would fail with this error:

url: https://endpoint.target.com/whatever...
Error: Error: write EPROTO 139652420734280:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:

It is worth mentioning that for these cases:

  • No proxy specified at all
  • -e HTTPS_PROXY="http://proxy.proxyhost.com:8080" \
  • -e HTTP_PROXY="http://proxy.proxyhost.com:8080" \
  • -e http_proxy="http://proxy.proxyhost.com:8080" \


...the output of docker run was:

url: https://target.endpoint.com/whatever...
Error: Error: connect ETIMEDOUT 11.22.33.44:443

As Docker documentation [Configure Docker to use a proxy server | Docker Documentation] states, passing HTTPS_PROXY environment variable to docker run would make container use specified proxy.

Axios documentation [axios - npm] states:
  // 'proxy' defines the hostname and port of the proxy server.
  // You can also define your proxy using the conventional `http_proxy` and
  // `https_proxy` environment variables. 

I was passing both HTTPS_PROXY and https_proxy environment variable to docker run in hope that Axios would pick one up but this didn't happen. HTTPS requests would always time out.

Then I found that there is a bug in Axios...and some solutions:

Request to HTTPS with HTTP proxy fails · Issue #925 · axios/axios
Axios proxy is not working. · Issue #2072 · axios/axios
Node.js Axios behind corporate proxies - Jan Molak

Three npm packages were used in proposed workarounds:

https-proxy-agent - npm has 8,439,302 weekly downloads
tunnel - npm has 523,094 weekly downloads
axios-https-proxy-fix - npm has 6,132 weekly downloads

For its popularity I decided to go for solution with https-proxy-agent and fix that worked from me was this.

I tried to recreate the same environment in order to try to reproduce this error and try the fix. I created a temp/test build step in my TeamCity build config with these properties:

This is the temp step I used:

Proxy Test >> Build Step (1 of 1): Test Proxy from Node Docker container
Runner type: command line
Run: custom script
Run step within Docker container: docker.mydockerhub.com/node:alpine
Docker image platform: Linux
Additional docker run arguments: NONE!!!
Custom script:

echo Path:
echo $PATH

# Comment this if do not need to use custom APK package registry
APK_PACKAGE_REGISTRY=apks.myapkpackageregistry.com/alpine
sed -i "s:http\:\/\/dl-cdn.alpinelinux.org:https\:\/\/${APK_PACKAGE_REGISTRY}:g" /etc/apk/repositories
cat /etc/apk/repositories
apk --no-cache add ca-certificates
update-ca-certificates
apk add curl

echo First request...
curl -v -x http://proxy.proxyhost.com:8080 -L ' endpoint.target.com/whatever/...'

cd ~
mkdir proxytest
cd proxytest
npm init -y
echo 'registry=https://myresourceserver.com/npm
# reset auth related options if specified in ~/.npmrc file
_auth=""
always-auth=false' > .npmrc
cat .npmrc
npm install axios
npm install https-proxy-agent

echo "axios = require('axios');
const HttpsProxyAgent = require('https-proxy-agent');

const agent = new HttpsProxyAgent({host: 'proxy.proxyhost.com', port: '8080'});

//use axios as you normally would, but specify httpsAgent in the config
axios = axios.create({
    httpsAgent: agent
});

axios.get(' endpoint.target.com/whatever/...')
.then((response) => {
  console.log('response: ' + JSON.stringify(response.data));
}, (error) => {
  console.log('error: ' + error);
  process.exit(1);
});" > main.js
cat main.js
node main.js

Log:

Step 1/12: Test Proxy from Node Docker container (Command Line) (29s)
[08:14:30]Running step within Docker container docker.mydockerhub.com/node:alpine
[08:14:30]Starting: /bin/sh -c "docker pull docker.mydockerhub.com/node:alpine && ... && docker run --rm ... --entrypoint /bin/sh "docker.mydockerhub.com/node:alpine" /home/docker-agent/temp/agentTmp/docker-shell-script-1558438850936.sh"
[08:14:30]in directory: /home/docker-agent/work/8ede9871249b4
[08:14:30]alpine: Pulling from node
[08:14:31]89d9c30c1d48: Pulling fs layer
[08:14:31]5320ee7fe9ff: Pulling fs layer
[08:14:31]0a42696890fc: Pulling fs layer
[08:14:31]12c581b27455: Pulling fs layer
[08:14:31]12c581b27455: Waiting
[08:14:31]0a42696890fc: Verifying Checksum
[08:14:31]0a42696890fc: Download complete
[08:14:31]12c581b27455: Download complete
[08:14:31]89d9c30c1d48: Verifying Checksum
[08:14:31]89d9c30c1d48: Download complete
[08:14:31]89d9c30c1d48: Pull complete
[08:14:32]5320ee7fe9ff: Verifying Checksum
[08:14:32]5320ee7fe9ff: Download complete
[08:14:33]5320ee7fe9ff: Pull complete
[08:14:33]0a42696890fc: Pull complete
[08:14:33]12c581b27455: Pull complete
[08:14:33]Digest: sha256:bdf054f006078036f72dedfd53f3b11176c1c00d5451d8fc2af206636eb54d70
[08:14:33]Status: Downloaded newer image for docker.mydockerhub.com/node:alpine
[08:14:33]docker.mydockerhub.com/node:alpine
[08:14:34]Path:
[08:14:34]/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
...
[08:14:35](1/3) Installing nghttp2-libs (1.39.2-r0)
[08:14:35](2/3) Installing libcurl (7.66.0-r0)
[08:14:35](3/3) Installing curl (7.66.0-r0)
[08:14:35]Executing busybox-1.30.1-r2.trigger
[08:14:35]OK: 8 MiB in 20 packages
[08:14:35]First request...
[08:14:35]  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[08:14:35]                                 Dload  Upload   Total   Spent    Left  Speed
[08:14:35]
[08:14:35]  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.17.129.70:8080...
[08:14:35]* TCP_NODELAY set
[08:14:35]* Connected to proxy.proxyhost.com (11.22.33.44) port 8080 (#0)
[08:14:35]* allocate connect buffer!
[08:14:35]* Establish HTTP proxy tunnel to endpoint.target.com/whatever:443
[08:14:35]> CONNECT endpoint.target.com:443 HTTP/1.1
[08:14:35]> Host: endpoint.target.com:443
[08:14:35]> User-Agent: curl/7.66.0
[08:14:35]> Proxy-Connection: Keep-Alive
[08:14:35]> 
[08:14:35]< HTTP/1.1 200 Connection established
[08:14:35]< 
[08:14:35]* Proxy replied 200 to CONNECT request
[08:14:35]* CONNECT phase completed!
[08:14:35]* ALPN, offering h2
[08:14:35]* ALPN, offering http/1.1
[08:14:35]* successfully set certificate verify locations:
[08:14:35]*   CAfile: /etc/ssl/certs/ca-certificates.crt
[08:14:35]  CApath: none
[08:14:35]} [5 bytes data]
[08:14:35]* TLSv1.3 (OUT), TLS handshake, Client hello (1):
[08:14:35]} [512 bytes data]
[08:14:35]* CONNECT phase completed!
[08:14:35]* CONNECT phase completed!
[08:14:36]{ [5 bytes data]
[08:14:36]* TLSv1.3 (IN), TLS handshake, Server hello (2):
[08:14:36]{ [108 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Certificate (11):
[08:14:36]{ [2994 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
[08:14:36]{ [333 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Server finished (14):
[08:14:36]{ [4 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
[08:14:36]} [70 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
[08:14:36]} [1 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS handshake, Finished (20):
[08:14:36]} [16 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Finished (20):
[08:14:36]{ [16 bytes data]
[08:14:36]* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
[08:14:36]* ALPN, server accepted to use http/1.1
[08:14:36]* Server certificate:
[08:14:36]*  subject: C=CZ; L=London; O=MyCompany; OU=Devs; CN=*.target.com
[08:14:36]*  start date: Nov  1 00:00:00 2018 GMT
[08:14:36]*  expire date: Nov 14 12:00:00 2020 GMT
[08:14:36]*  subjectAltName: host "endpoint.target.com" matched cert's "*.target.com"
[08:14:36]*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
[08:14:36]*  SSL certificate verify ok.
[08:14:36]} [5 bytes data]
[08:14:36]> GET /whatever... HTTP/1.1
[08:14:36]> Host: endpoint.target.com
[08:14:36]> User-Agent: curl/7.66.0
[08:14:36]> Accept: */*
[08:14:36]> 
[08:14:36]{ [5 bytes data]
[08:14:36]* Mark bundle as not supporting multiuse
[08:14:36]< HTTP/1.1 301 Moved Permanently
[08:14:36]< Server: nginx/1.16.1
[08:14:36]< Date: Tue, 26 Nov 2019 07:14:36 GMT
[08:14:36]< Content-Type: text/html
[08:14:36]< Content-Length: 169
[08:14:36]< Location: https://endpoint.target.com/whatever...
[08:14:36]< Connection: keep-alive
[08:14:36]< 
[08:14:36]* Ignoring the response-body
[08:14:36]{ [169 bytes data]
[08:14:36]
[08:14:36]100   169  100   169    0     0    290      0 --:--:-- --:--:-- --:--:--   290
[08:14:36]* Connection #0 to host proxy.proxyhost.com left intact
[08:14:36]* Issue another request to this URL: 'https://endpoint.target.com/whatever...'
[08:14:36]* Found bundle for host endpoint.target.com: 0x561df56fb80 [serially]
[08:14:36]* Can not multiplex, even if we wanted to!
[08:14:36]* Re-using existing connection! (#0) with proxy proxy.proxyhost.com
[08:14:36]* Connected to proxy proxy.proxyhost.com (10.17.129.70) port 8080 (#0)
[08:14:36]} [5 bytes data]
[08:14:36]> GET /api/?post_type=block_filterlist&json=1&count=1000 HTTP/1.1
[08:14:36]> Host:  endpoint.target.com
[08:14:36]> User-Agent: curl/7.66.0
[08:14:36]> Accept: */*
[08:14:36]> 
[08:14:37]{ [5 bytes data]
[08:14:37]
[08:14:37]  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Mark bundle as not supporting multiuse
[08:14:37]< HTTP/1.1 200 OK
[08:14:37]< Server: nginx/1.16.1
[08:14:37]< Date: Tue, 26 Nov 2019 07:14:36 GMT
[08:14:37]< Content-Type: application/json; charset=UTF-8
[08:14:37]< Transfer-Encoding: chunked
[08:14:37]< Connection: keep-alive
[08:14:37]< 
[08:14:37]{ [13618 bytes data]
[08:14:37]
[08:14:37]100 13597    0 13597    0     0   9961      0 --:--:--  0:00:01 --:--:-- 3319k
[08:14:37]* Connection #0 to host proxy proxy.proxyhost.com left intact
[08:14:37]{"status":"ok",...}Wrote to /root/proxytest/package.json:
[08:14:37]
[08:14:37]{
[08:14:37]  "name": "proxytest",
[08:14:37]  "version": "1.0.0",
[08:14:37]  "description": "",
[08:14:37]  "main": "index.js",
[08:14:37]  "scripts": {
[08:14:37]    "test": "echo \"Error: no test specified\" && exit 1"
[08:14:37]  },
[08:14:37]  "keywords": [],
[08:14:37]  "author": "",
[08:14:37]  "license": "ISC"
[08:14:37]}
[08:14:37]
[08:14:37]
[08:14:37]registry=https://myresourceserver.com/npm
[08:14:37]# reset auth related options if specified in ~/.npmrc file
[08:14:37]_auth=""
[08:14:37]always-auth=false
[08:14:38]npm notice created a lockfile as package-lock.json. You should commit this file.
[08:14:38]npm WARN proxytest@1.0.0 No description
[08:14:38]npm WARN proxytest@1.0.0 No repository field.
[08:14:38]
[08:14:39]+ axios@0.19.0
[08:14:39]added 5 packages from 8 contributors and audited 5 packages in 1.441s
[08:14:39]found 0 vulnerabilities
[08:14:39]
[08:14:40]npm WARN proxytest@1.0.0 No description
[08:14:40]npm WARN proxytest@1.0.0 No repository field.
[08:14:40]
[08:14:41]+ https-proxy-agent@3.0.1
[08:14:41]added 4 packages from 3 contributors and audited 11 packages in 1.632s
[08:14:41]found 0 vulnerabilities
[08:14:41]
[08:14:41]axios = require('axios');
[08:14:41]const HttpsProxyAgent = require('https-proxy-agent');
[08:14:41]
[08:14:41]const agent = new HttpsProxyAgent({host: 'proxy proxy.proxyhost.com', port: '8080'});
[08:14:41]
[08:14:41]//use axios as you normally would, but specify httpsAgent in the config
[08:14:41]axios = axios.create({
[08:14:41]    httpsAgent: agent
[08:14:41]});
[08:14:41]
[08:14:41]axios.get('https:// endpoint.target.com/whatever...')
[08:14:41].then((response) => {
[08:14:41]  console.log('response: ' + JSON.stringify(response.data));
[08:14:41]}, (error) => {
[08:14:41]  console.log('error: ' + error);
[08:14:41]  process.exit(1);
[08:14:41]});
[08:14:43]response: {"status":"ok", ...}
[08:14:43]Process exited with code 0



Tuesday 19 November 2019

How to access Amazon Web Services via AWS Go SDK

The centre point of the AWS client application is AWS Session. To create it, we need to do the following:

(1) import github.com/aws/aws-sdk-go/aws/session package
(2) call NewSession() on it

sess, err := session.NewSession()

By default, session uses environment variables for its configuration. Some of them are:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION

For the full list see AWS CLI - Environment Variables.

So, if we call our AWS client app (let's name it aws-client) from command line, we can first set these environment variables:

export AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID_VALUE
export AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY_VALUE
export AWS_REGION=AWS_REGION_VALUE
aws-client

If aws-client is entrypoint for Docker container, we can pass these environment variables via command line:

docker run \
-e AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID_VALUE \
-e AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY_VALUE \
-e AWS_REGION=AWS_REGION_VALUE \
--rm \
aws-client-image

We can create session configuration explicitly and pass options which are programmatically set. For this, we need to:

(1) import github.com/aws/aws-sdk-go/aws package
(2) create Config structure and pass it to session constructor

sess, err := session.NewSession(&aws.Config{})

In the example above, environment variables will still be used. If we want to pass particular options, we can set matching properties of the Config structure:

// S3Region is the name of the geo region e.g. "eu-west-2"
const S3Region = "us-east-1"
...
sess, err := session.NewSession(
   &aws.Config{
      Region: aws.String(S3Region)
   }
)

For complete list of regions see Regions and Availability Zones - Amazon Relational Database Service.



TO BE CONTINUED...

Monday 18 November 2019

How to install OpenCV on Ubuntu 18.04

I needed to create (a virtual) environment on my Ubuntu dev box for project which uses Python 3 and OpenCV. To install OpenCV in it I followed, with some small modifications, instructions listed in Adrian Rosebrock's article Ubuntu 18.04: How to install OpenCV. Here I want to share my experience with this process.

When installing virtualenv make sure you use pip3. If you by mistake use pip:

$ sudo pip install virtualenv virtualenvwrapper
The directory '/home/bojan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/bojan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting virtualenv
Downloading https://files.pythonhosted.org/packages/c5/97/00dd42a0fc41e9016b23f07ec7f657f636cb672fad9cf72b80f8f65c6a46/virtualenv-16.7.7-py2.py3-none-any.whl (3.4MB)
100% || 3.4MB 442kB/s
Collecting virtualenvwrapper
Downloading https://files.pythonhosted.org/packages/c1/6b/2f05d73b2d2f2410b48b90d3783a0034c26afa534a4a95ad5f1178d61191/virtualenvwrapper-4.8.4.tar.gz (334kB)
100% || 337kB 2.9MB/s
Collecting stevedore (from virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/b1/e1/f5ddbd83f60b03f522f173c03e406c1bff8343f0232a292ac96aa633b47a/stevedore-1.31.0-py2.py3-none-any.whl (43kB)
100% || 51kB 11.4MB/s
Collecting virtualenv-clone (from virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/ba/f8/50c2b7dbc99e05fce5e5b9d9a31f37c988c99acd4e8dedd720b7b8d4011d/virtualenv_clone-0.5.3-py2.py3-none-any.whl
Collecting pbr!=2.1.0,>=2.0.0 (from stevedore->virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/46/a4/d5c83831a3452713e4b4f126149bc4fbda170f7cb16a86a00ce57ce0e9ad/pbr-5.4.3-py2.py3-none-any.whl (110kB)
100% || 112kB 6.7MB/s
Requirement already satisfied: six>=1.10.0 in /home/bojan/.local/lib/python2.7/site-packages (from stevedore->virtualenvwrapper)
Installing collected packages: virtualenv, pbr, stevedore, virtualenv-clone, virtualenvwrapper
Running setup.py install for virtualenvwrapper ... done
Successfully installed pbr-5.4.3 stevedore-1.31.0 virtualenv-16.7.7 virtualenv-clone-0.5.3 virtualenvwrapper-4.8.4

...then sourcing bashrc will fail:

$ source ~/.bashrc
/usr/bin/python3: Error while finding module specification for 'virtualenvwrapper.hook_loader' (ModuleNotFoundError: No module named 'virtualenvwrapper')
virtualenvwrapper.sh: There was a problem running the initialization hooks.

If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3 and that PATH is
set properly.

We need to use pip3 so virtual environment is installed with Python3:

$ sudo pip3 install virtualenv virtualenvwrapper
The directory '/home/bojan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/bojan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting virtualenv
Downloading https://files.pythonhosted.org/packages/c5/97/00dd42a0fc41e9016b23f07ec7f657f636cb672fad9cf72b80f8f65c6a46/virtualenv-16.7.7-py2.py3-none-any.whl (3.4MB)
100% || 3.4MB 516kB/s
Collecting virtualenvwrapper
Downloading https://files.pythonhosted.org/packages/c1/6b/2f05d73b2d2f2410b48b90d3783a0034c26afa534a4a95ad5f1178d61191/virtualenvwrapper-4.8.4.tar.gz (334kB)
100% || 337kB 3.2MB/s
Collecting stevedore (from virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/b1/e1/f5ddbd83f60b03f522f173c03e406c1bff8343f0232a292ac96aa633b47a/stevedore-1.31.0-py2.py3-none-any.whl (43kB)
100% || 51kB 9.7MB/s
Collecting virtualenv-clone (from virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/ba/f8/50c2b7dbc99e05fce5e5b9d9a31f37c988c99acd4e8dedd720b7b8d4011d/virtualenv_clone-0.5.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.10.0 in /home/bojan/.local/lib/python3.6/site-packages (from stevedore->virtualenvwrapper)
Collecting pbr!=2.1.0,>=2.0.0 (from stevedore->virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/46/a4/d5c83831a3452713e4b4f126149bc4fbda170f7cb16a86a00ce57ce0e9ad/pbr-5.4.3-py2.py3-none-any.whl (110kB)
100% || 112kB 6.2MB/s
Installing collected packages: virtualenv, pbr, stevedore, virtualenv-clone, virtualenvwrapper
Running setup.py install for virtualenvwrapper ... done
Successfully installed pbr-5.4.3 stevedore-1.31.0 virtualenv-16.7.7 virtualenv-clone-0.5.3 virtualenvwrapper-4.8.4


$ sudo rm -rf ~/get-pip.py ~/.cache/pip


$ source ~/.bashrc
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/premkproject
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/postmkproject
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/initialize
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/premkvirtualenv
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/postmkvirtualenv
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/prermvirtualenv
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/postrmvirtualenv
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/predeactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/postdeactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/preactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/postactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/get_env_details

Now we can create virtual environment for our project:

$ mkvirtualenv py3.6_cv2 -p python3.6
Running virtualenv with interpreter /usr/bin/python3.6
Already using interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /home/bojan/.virtualenvs/py3.6_cv2/bin/python3.6
Also creating executable in /home/bojan/.virtualenvs/py3.6_cv2/bin/python
Installing setuptools, pip, wheel...
done.
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/py3.6_cv2/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/py3.6_cv2/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/py3.6_cv2/bin/preactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/py3.6_cv2/bin/postactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/py3.6_cv2/bin/get_env_details


(py3.6_cv2) bojan@...$ workon py3.6_cv2

Let's check pip version in this virtual environment:

(py3.6_cv2) bojan@...$ which pip
/home/bojan/.virtualenvs/py3.6_cv2/bin/pip

(py3.6_cv2) bojan@...$ pip --version
pip 19.3.1 from /home/bojan/.virtualenvs/py3.6_cv2/lib/python3.6/site-packages/pip (python 3.6)

Installing necessary packages (only these were required for the OpenCV functionality in my project; some other packages might be necessary for different OpenCV features):

$ pip install numpy
$ sudo apt install clang cmake libgtk2.0-dev

Downloading and unpacking OpenCV source code (version 4.1.1 was the last one at the time of writing):

$ wget https://github.com/opencv/opencv/archive/4.1.1.zip
$ unzip 4.1.1.zip 
$ cd 4.1.1/
$ mkdir build
$ cd build/

If we don't explicitly specify Python version that cmake should use, cmake's output will contain reference to Python 2:

$ cmake -D ...
...
Python (for build): /usr/bin/python2.7

But if we have add options set in bold:

$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D WITH_CUDA=OFF \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_ENABLE_NONFREE=OFF \
-D BUILD_EXAMPLES=OFF \
-D CMAKE_C_COMPILER=/usr/bin/clang \
-D CMAKE_CXX_COMPILER=/usr/bin/clang++ \
-D BUILD_NEW_PYTHON_SUPPORT=ON \
-D BUILD_opencv_python3=ON \
-D HAVE_opencv_python3=ON \
-D PYTHON_DEFAULT_EXECUTABLE=/usr/bin/python3.6 \
.. \

...cmake will output:

...
Python (for build): /usr/bin/python3.6
...

Let's now build and install OpenCV:

~/Downloads/opencv-4.1.1/build$ make install
...
file cannot create directory: /usr/local/share/licenses/opencv4. Maybe
need administrative privileges.

We need to use sudo:

~/Downloads/opencv-4.1.1/build$ sudo make install
$ sudo ldconfig

Let's check OpenCV version:

(py3.6_cv2) bojan@...$ opencv_version 
4.1.1

cv2 package still can't be loaded to Python code:

$ python
Python 3.6.8 (default, Oct 7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'cv2'
>>>
KeyboardInterrupt
>>> exit()


Lets' see where is Python binding for OpenCV:

$ sudo ls /usr/local/lib/python3.6/site-packages/cv2/python-3.6/
cv2.cpython-36m-x86_64-linux-gnu.so



We need to create soft link:

(py3.6_cv2) bojan@...:~/Downloads/opencv-4.1.1/build$ cd ~/.virtualenvs/py3.6_cv2/lib/python3.6/site-packages/

(py3.6_cv2) bojan@...:~/.virtualenvs/py3.6_cv2/lib/python3.6/site-packages$ ln -s /usr/local/lib/python3.6/site-packages/cv2/python-3.6/cv2.cpython-36m-x86_64-linux-gnu.so cv2.so


Importing cv2 package works now:

(py3.6_cv2) bojan@...:~/.virtualenvs/py3.6_cv2/lib/python3.6/site-packages$ python
Python 3.6.8 (default, Oct 7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.1.1'
>>> quit()