Tuesday, 26 November 2019

How to make HTTPS request through HTTP proxy with Axios

I have a Node.js app which runs in node:alpine Docker image on TeamCity. It uses axios HTTP client to make HTTP(S) requests. To make it run it behind the proxy, I used:

docker run \
...
-e https_proxy="http://proxy.proxyhost.com:8080" \
...

...but request would fail with this error:

url: https://endpoint.target.com/whatever...
Error: Error: write EPROTO 139652420734280:error:1408F10B:SSL routines:ssl3_get_record:wrong version number:../deps/openssl/openssl/ssl/record/ssl3_record.c:332:

It is worth mentioning that for these cases:

  • No proxy specified at all
  • -e HTTPS_PROXY="http://proxy.proxyhost.com:8080" \
  • -e HTTP_PROXY="http://proxy.proxyhost.com:8080" \
  • -e http_proxy="http://proxy.proxyhost.com:8080" \


...the output of docker run was:

url: https://target.endpoint.com/whatever...
Error: Error: connect ETIMEDOUT 11.22.33.44:443

As Docker documentation [Configure Docker to use a proxy server | Docker Documentation] states, passing HTTPS_PROXY environment variable to docker run would make container use specified proxy.

Axios documentation [axios - npm] states:
  // 'proxy' defines the hostname and port of the proxy server.
  // You can also define your proxy using the conventional `http_proxy` and
  // `https_proxy` environment variables. 

I was passing both HTTPS_PROXY and https_proxy environment variable to docker run in hope that Axios would pick one up but this didn't happen. HTTPS requests would always time out.

Then I found that there is a bug in Axios...and some solutions:

Request to HTTPS with HTTP proxy fails · Issue #925 · axios/axios
Axios proxy is not working. · Issue #2072 · axios/axios
Node.js Axios behind corporate proxies - Jan Molak

Three npm packages were used in proposed workarounds:

https-proxy-agent - npm has 8,439,302 weekly downloads
tunnel - npm has 523,094 weekly downloads
axios-https-proxy-fix - npm has 6,132 weekly downloads

For its popularity I decided to go for solution with https-proxy-agent and fix that worked from me was this.

I tried to recreate the same environment in order to try to reproduce this error and try the fix. I created a temp/test build step in my TeamCity build config with these properties:

This is the temp step I used:

Proxy Test >> Build Step (1 of 1): Test Proxy from Node Docker container
Runner type: command line
Run: custom script
Run step within Docker container: docker.mydockerhub.com/node:alpine
Docker image platform: Linux
Additional docker run arguments: NONE!!!
Custom script:

echo Path:
echo $PATH

# Comment this if do not need to use custom APK package registry
APK_PACKAGE_REGISTRY=apks.myapkpackageregistry.com/alpine
sed -i "s:http\:\/\/dl-cdn.alpinelinux.org:https\:\/\/${APK_PACKAGE_REGISTRY}:g" /etc/apk/repositories
cat /etc/apk/repositories
apk --no-cache add ca-certificates
update-ca-certificates
apk add curl

echo First request...
curl -v -x http://proxy.proxyhost.com:8080 -L ' endpoint.target.com/whatever/...'

cd ~
mkdir proxytest
cd proxytest
npm init -y
echo 'registry=https://myresourceserver.com/npm
# reset auth related options if specified in ~/.npmrc file
_auth=""
always-auth=false' > .npmrc
cat .npmrc
npm install axios
npm install https-proxy-agent

echo "axios = require('axios');
const HttpsProxyAgent = require('https-proxy-agent');

const agent = new HttpsProxyAgent({host: 'proxy.proxyhost.com', port: '8080'});

//use axios as you normally would, but specify httpsAgent in the config
axios = axios.create({
    httpsAgent: agent
});

axios.get(' endpoint.target.com/whatever/...')
.then((response) => {
  console.log('response: ' + JSON.stringify(response.data));
}, (error) => {
  console.log('error: ' + error);
  process.exit(1);
});" > main.js
cat main.js
node main.js

Log:

Step 1/12: Test Proxy from Node Docker container (Command Line) (29s)
[08:14:30]Running step within Docker container docker.mydockerhub.com/node:alpine
[08:14:30]Starting: /bin/sh -c "docker pull docker.mydockerhub.com/node:alpine && ... && docker run --rm ... --entrypoint /bin/sh "docker.mydockerhub.com/node:alpine" /home/docker-agent/temp/agentTmp/docker-shell-script-1558438850936.sh"
[08:14:30]in directory: /home/docker-agent/work/8ede9871249b4
[08:14:30]alpine: Pulling from node
[08:14:31]89d9c30c1d48: Pulling fs layer
[08:14:31]5320ee7fe9ff: Pulling fs layer
[08:14:31]0a42696890fc: Pulling fs layer
[08:14:31]12c581b27455: Pulling fs layer
[08:14:31]12c581b27455: Waiting
[08:14:31]0a42696890fc: Verifying Checksum
[08:14:31]0a42696890fc: Download complete
[08:14:31]12c581b27455: Download complete
[08:14:31]89d9c30c1d48: Verifying Checksum
[08:14:31]89d9c30c1d48: Download complete
[08:14:31]89d9c30c1d48: Pull complete
[08:14:32]5320ee7fe9ff: Verifying Checksum
[08:14:32]5320ee7fe9ff: Download complete
[08:14:33]5320ee7fe9ff: Pull complete
[08:14:33]0a42696890fc: Pull complete
[08:14:33]12c581b27455: Pull complete
[08:14:33]Digest: sha256:bdf054f006078036f72dedfd53f3b11176c1c00d5451d8fc2af206636eb54d70
[08:14:33]Status: Downloaded newer image for docker.mydockerhub.com/node:alpine
[08:14:33]docker.mydockerhub.com/node:alpine
[08:14:34]Path:
[08:14:34]/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
...
[08:14:35](1/3) Installing nghttp2-libs (1.39.2-r0)
[08:14:35](2/3) Installing libcurl (7.66.0-r0)
[08:14:35](3/3) Installing curl (7.66.0-r0)
[08:14:35]Executing busybox-1.30.1-r2.trigger
[08:14:35]OK: 8 MiB in 20 packages
[08:14:35]First request...
[08:14:35]  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
[08:14:35]                                 Dload  Upload   Total   Spent    Left  Speed
[08:14:35]
[08:14:35]  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.17.129.70:8080...
[08:14:35]* TCP_NODELAY set
[08:14:35]* Connected to proxy.proxyhost.com (11.22.33.44) port 8080 (#0)
[08:14:35]* allocate connect buffer!
[08:14:35]* Establish HTTP proxy tunnel to endpoint.target.com/whatever:443
[08:14:35]> CONNECT endpoint.target.com:443 HTTP/1.1
[08:14:35]> Host: endpoint.target.com:443
[08:14:35]> User-Agent: curl/7.66.0
[08:14:35]> Proxy-Connection: Keep-Alive
[08:14:35]> 
[08:14:35]< HTTP/1.1 200 Connection established
[08:14:35]< 
[08:14:35]* Proxy replied 200 to CONNECT request
[08:14:35]* CONNECT phase completed!
[08:14:35]* ALPN, offering h2
[08:14:35]* ALPN, offering http/1.1
[08:14:35]* successfully set certificate verify locations:
[08:14:35]*   CAfile: /etc/ssl/certs/ca-certificates.crt
[08:14:35]  CApath: none
[08:14:35]} [5 bytes data]
[08:14:35]* TLSv1.3 (OUT), TLS handshake, Client hello (1):
[08:14:35]} [512 bytes data]
[08:14:35]* CONNECT phase completed!
[08:14:35]* CONNECT phase completed!
[08:14:36]{ [5 bytes data]
[08:14:36]* TLSv1.3 (IN), TLS handshake, Server hello (2):
[08:14:36]{ [108 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Certificate (11):
[08:14:36]{ [2994 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
[08:14:36]{ [333 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Server finished (14):
[08:14:36]{ [4 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
[08:14:36]} [70 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
[08:14:36]} [1 bytes data]
[08:14:36]* TLSv1.2 (OUT), TLS handshake, Finished (20):
[08:14:36]} [16 bytes data]
[08:14:36]* TLSv1.2 (IN), TLS handshake, Finished (20):
[08:14:36]{ [16 bytes data]
[08:14:36]* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
[08:14:36]* ALPN, server accepted to use http/1.1
[08:14:36]* Server certificate:
[08:14:36]*  subject: C=CZ; L=London; O=MyCompany; OU=Devs; CN=*.target.com
[08:14:36]*  start date: Nov  1 00:00:00 2018 GMT
[08:14:36]*  expire date: Nov 14 12:00:00 2020 GMT
[08:14:36]*  subjectAltName: host "endpoint.target.com" matched cert's "*.target.com"
[08:14:36]*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
[08:14:36]*  SSL certificate verify ok.
[08:14:36]} [5 bytes data]
[08:14:36]> GET /whatever... HTTP/1.1
[08:14:36]> Host: endpoint.target.com
[08:14:36]> User-Agent: curl/7.66.0
[08:14:36]> Accept: */*
[08:14:36]> 
[08:14:36]{ [5 bytes data]
[08:14:36]* Mark bundle as not supporting multiuse
[08:14:36]< HTTP/1.1 301 Moved Permanently
[08:14:36]< Server: nginx/1.16.1
[08:14:36]< Date: Tue, 26 Nov 2019 07:14:36 GMT
[08:14:36]< Content-Type: text/html
[08:14:36]< Content-Length: 169
[08:14:36]< Location: https://endpoint.target.com/whatever...
[08:14:36]< Connection: keep-alive
[08:14:36]< 
[08:14:36]* Ignoring the response-body
[08:14:36]{ [169 bytes data]
[08:14:36]
[08:14:36]100   169  100   169    0     0    290      0 --:--:-- --:--:-- --:--:--   290
[08:14:36]* Connection #0 to host proxy.proxyhost.com left intact
[08:14:36]* Issue another request to this URL: 'https://endpoint.target.com/whatever...'
[08:14:36]* Found bundle for host endpoint.target.com: 0x561df56fb80 [serially]
[08:14:36]* Can not multiplex, even if we wanted to!
[08:14:36]* Re-using existing connection! (#0) with proxy proxy.proxyhost.com
[08:14:36]* Connected to proxy proxy.proxyhost.com (10.17.129.70) port 8080 (#0)
[08:14:36]} [5 bytes data]
[08:14:36]> GET /api/?post_type=block_filterlist&json=1&count=1000 HTTP/1.1
[08:14:36]> Host:  endpoint.target.com
[08:14:36]> User-Agent: curl/7.66.0
[08:14:36]> Accept: */*
[08:14:36]> 
[08:14:37]{ [5 bytes data]
[08:14:37]
[08:14:37]  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0* Mark bundle as not supporting multiuse
[08:14:37]< HTTP/1.1 200 OK
[08:14:37]< Server: nginx/1.16.1
[08:14:37]< Date: Tue, 26 Nov 2019 07:14:36 GMT
[08:14:37]< Content-Type: application/json; charset=UTF-8
[08:14:37]< Transfer-Encoding: chunked
[08:14:37]< Connection: keep-alive
[08:14:37]< 
[08:14:37]{ [13618 bytes data]
[08:14:37]
[08:14:37]100 13597    0 13597    0     0   9961      0 --:--:--  0:00:01 --:--:-- 3319k
[08:14:37]* Connection #0 to host proxy proxy.proxyhost.com left intact
[08:14:37]{"status":"ok",...}Wrote to /root/proxytest/package.json:
[08:14:37]
[08:14:37]{
[08:14:37]  "name": "proxytest",
[08:14:37]  "version": "1.0.0",
[08:14:37]  "description": "",
[08:14:37]  "main": "index.js",
[08:14:37]  "scripts": {
[08:14:37]    "test": "echo \"Error: no test specified\" && exit 1"
[08:14:37]  },
[08:14:37]  "keywords": [],
[08:14:37]  "author": "",
[08:14:37]  "license": "ISC"
[08:14:37]}
[08:14:37]
[08:14:37]
[08:14:37]registry=https://myresourceserver.com/npm
[08:14:37]# reset auth related options if specified in ~/.npmrc file
[08:14:37]_auth=""
[08:14:37]always-auth=false
[08:14:38]npm notice created a lockfile as package-lock.json. You should commit this file.
[08:14:38]npm WARN proxytest@1.0.0 No description
[08:14:38]npm WARN proxytest@1.0.0 No repository field.
[08:14:38]
[08:14:39]+ axios@0.19.0
[08:14:39]added 5 packages from 8 contributors and audited 5 packages in 1.441s
[08:14:39]found 0 vulnerabilities
[08:14:39]
[08:14:40]npm WARN proxytest@1.0.0 No description
[08:14:40]npm WARN proxytest@1.0.0 No repository field.
[08:14:40]
[08:14:41]+ https-proxy-agent@3.0.1
[08:14:41]added 4 packages from 3 contributors and audited 11 packages in 1.632s
[08:14:41]found 0 vulnerabilities
[08:14:41]
[08:14:41]axios = require('axios');
[08:14:41]const HttpsProxyAgent = require('https-proxy-agent');
[08:14:41]
[08:14:41]const agent = new HttpsProxyAgent({host: 'proxy proxy.proxyhost.com', port: '8080'});
[08:14:41]
[08:14:41]//use axios as you normally would, but specify httpsAgent in the config
[08:14:41]axios = axios.create({
[08:14:41]    httpsAgent: agent
[08:14:41]});
[08:14:41]
[08:14:41]axios.get('https:// endpoint.target.com/whatever...')
[08:14:41].then((response) => {
[08:14:41]  console.log('response: ' + JSON.stringify(response.data));
[08:14:41]}, (error) => {
[08:14:41]  console.log('error: ' + error);
[08:14:41]  process.exit(1);
[08:14:41]});
[08:14:43]response: {"status":"ok", ...}
[08:14:43]Process exited with code 0



Tuesday, 19 November 2019

How to access Amazon Web Services via AWS Go SDK

The centre point of the AWS client application is AWS Session. To create it, we need to do the following:

(1) import github.com/aws/aws-sdk-go/aws/session package
(2) call NewSession() on it

sess, err := session.NewSession()

By default, session uses environment variables for its configuration. Some of them are:

AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_REGION

For the full list see AWS CLI - Environment Variables.

So, if we call our AWS client app (let's name it aws-client) from command line, we can first set these environment variables:

export AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID_VALUE
export AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY_VALUE
export AWS_REGION=AWS_REGION_VALUE
aws-client

If aws-client is entrypoint for Docker container, we can pass these environment variables via command line:

docker run \
-e AWS_ACCESS_KEY_ID=AWS_ACCESS_KEY_ID_VALUE \
-e AWS_SECRET_ACCESS_KEY=AWS_SECRET_ACCESS_KEY_VALUE \
-e AWS_REGION=AWS_REGION_VALUE \
--rm \
aws-client-image

We can create session configuration explicitly and pass options which are programmatically set. For this, we need to:

(1) import github.com/aws/aws-sdk-go/aws package
(2) create Config structure and pass it to session constructor

sess, err := session.NewSession(&aws.Config{})

In the example above, environment variables will still be used. If we want to pass particular options, we can set matching properties of the Config structure:

// S3Region is the name of the geo region e.g. "eu-west-2"
const S3Region = "us-east-1"
...
sess, err := session.NewSession(
   &aws.Config{
      Region: aws.String(S3Region)
   }
)

For complete list of regions see Regions and Availability Zones - Amazon Relational Database Service.



TO BE CONTINUED...

Monday, 18 November 2019

How to install OpenCV on Ubuntu 18.04

I needed to create (a virtual) environment on my Ubuntu dev box for project which uses Python 3 and OpenCV. To install OpenCV in it I followed, with some small modifications, instructions listed in Adrian Rosebrock's article Ubuntu 18.04: How to install OpenCV. Here I want to share my experience with this process.

When installing virtualenv make sure you use pip3. If you by mistake use pip:

$ sudo pip install virtualenv virtualenvwrapper
The directory '/home/bojan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/bojan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting virtualenv
Downloading https://files.pythonhosted.org/packages/c5/97/00dd42a0fc41e9016b23f07ec7f657f636cb672fad9cf72b80f8f65c6a46/virtualenv-16.7.7-py2.py3-none-any.whl (3.4MB)
100% || 3.4MB 442kB/s
Collecting virtualenvwrapper
Downloading https://files.pythonhosted.org/packages/c1/6b/2f05d73b2d2f2410b48b90d3783a0034c26afa534a4a95ad5f1178d61191/virtualenvwrapper-4.8.4.tar.gz (334kB)
100% || 337kB 2.9MB/s
Collecting stevedore (from virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/b1/e1/f5ddbd83f60b03f522f173c03e406c1bff8343f0232a292ac96aa633b47a/stevedore-1.31.0-py2.py3-none-any.whl (43kB)
100% || 51kB 11.4MB/s
Collecting virtualenv-clone (from virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/ba/f8/50c2b7dbc99e05fce5e5b9d9a31f37c988c99acd4e8dedd720b7b8d4011d/virtualenv_clone-0.5.3-py2.py3-none-any.whl
Collecting pbr!=2.1.0,>=2.0.0 (from stevedore->virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/46/a4/d5c83831a3452713e4b4f126149bc4fbda170f7cb16a86a00ce57ce0e9ad/pbr-5.4.3-py2.py3-none-any.whl (110kB)
100% || 112kB 6.7MB/s
Requirement already satisfied: six>=1.10.0 in /home/bojan/.local/lib/python2.7/site-packages (from stevedore->virtualenvwrapper)
Installing collected packages: virtualenv, pbr, stevedore, virtualenv-clone, virtualenvwrapper
Running setup.py install for virtualenvwrapper ... done
Successfully installed pbr-5.4.3 stevedore-1.31.0 virtualenv-16.7.7 virtualenv-clone-0.5.3 virtualenvwrapper-4.8.4

...then sourcing bashrc will fail:

$ source ~/.bashrc
/usr/bin/python3: Error while finding module specification for 'virtualenvwrapper.hook_loader' (ModuleNotFoundError: No module named 'virtualenvwrapper')
virtualenvwrapper.sh: There was a problem running the initialization hooks.

If Python could not import the module virtualenvwrapper.hook_loader,
check that virtualenvwrapper has been installed for
VIRTUALENVWRAPPER_PYTHON=/usr/bin/python3 and that PATH is
set properly.

We need to use pip3 so virtual environment is installed with Python3:

$ sudo pip3 install virtualenv virtualenvwrapper
The directory '/home/bojan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/bojan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting virtualenv
Downloading https://files.pythonhosted.org/packages/c5/97/00dd42a0fc41e9016b23f07ec7f657f636cb672fad9cf72b80f8f65c6a46/virtualenv-16.7.7-py2.py3-none-any.whl (3.4MB)
100% || 3.4MB 516kB/s
Collecting virtualenvwrapper
Downloading https://files.pythonhosted.org/packages/c1/6b/2f05d73b2d2f2410b48b90d3783a0034c26afa534a4a95ad5f1178d61191/virtualenvwrapper-4.8.4.tar.gz (334kB)
100% || 337kB 3.2MB/s
Collecting stevedore (from virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/b1/e1/f5ddbd83f60b03f522f173c03e406c1bff8343f0232a292ac96aa633b47a/stevedore-1.31.0-py2.py3-none-any.whl (43kB)
100% || 51kB 9.7MB/s
Collecting virtualenv-clone (from virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/ba/f8/50c2b7dbc99e05fce5e5b9d9a31f37c988c99acd4e8dedd720b7b8d4011d/virtualenv_clone-0.5.3-py2.py3-none-any.whl
Requirement already satisfied: six>=1.10.0 in /home/bojan/.local/lib/python3.6/site-packages (from stevedore->virtualenvwrapper)
Collecting pbr!=2.1.0,>=2.0.0 (from stevedore->virtualenvwrapper)
Downloading https://files.pythonhosted.org/packages/46/a4/d5c83831a3452713e4b4f126149bc4fbda170f7cb16a86a00ce57ce0e9ad/pbr-5.4.3-py2.py3-none-any.whl (110kB)
100% || 112kB 6.2MB/s
Installing collected packages: virtualenv, pbr, stevedore, virtualenv-clone, virtualenvwrapper
Running setup.py install for virtualenvwrapper ... done
Successfully installed pbr-5.4.3 stevedore-1.31.0 virtualenv-16.7.7 virtualenv-clone-0.5.3 virtualenvwrapper-4.8.4


$ sudo rm -rf ~/get-pip.py ~/.cache/pip


$ source ~/.bashrc
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/premkproject
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/postmkproject
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/initialize
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/premkvirtualenv
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/postmkvirtualenv
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/prermvirtualenv
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/postrmvirtualenv
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/predeactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/postdeactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/preactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/postactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/get_env_details

Now we can create virtual environment for our project:

$ mkvirtualenv py3.6_cv2 -p python3.6
Running virtualenv with interpreter /usr/bin/python3.6
Already using interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /home/bojan/.virtualenvs/py3.6_cv2/bin/python3.6
Also creating executable in /home/bojan/.virtualenvs/py3.6_cv2/bin/python
Installing setuptools, pip, wheel...
done.
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/py3.6_cv2/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/py3.6_cv2/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/py3.6_cv2/bin/preactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/py3.6_cv2/bin/postactivate
virtualenvwrapper.user_scripts creating /home/bojan/.virtualenvs/py3.6_cv2/bin/get_env_details


(py3.6_cv2) bojan@...$ workon py3.6_cv2

Let's check pip version in this virtual environment:

(py3.6_cv2) bojan@...$ which pip
/home/bojan/.virtualenvs/py3.6_cv2/bin/pip

(py3.6_cv2) bojan@...$ pip --version
pip 19.3.1 from /home/bojan/.virtualenvs/py3.6_cv2/lib/python3.6/site-packages/pip (python 3.6)

Installing necessary packages (only these were required for the OpenCV functionality in my project; some other packages might be necessary for different OpenCV features):

$ pip install numpy
$ sudo apt install clang cmake libgtk2.0-dev

Downloading and unpacking OpenCV source code (version 4.1.1 was the last one at the time of writing):

$ wget https://github.com/opencv/opencv/archive/4.1.1.zip
$ unzip 4.1.1.zip 
$ cd 4.1.1/
$ mkdir build
$ cd build/

If we don't explicitly specify Python version that cmake should use, cmake's output will contain reference to Python 2:

$ cmake -D ...
...
Python (for build): /usr/bin/python2.7

But if we have add options set in bold:

$ cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D WITH_CUDA=OFF \
-D INSTALL_C_EXAMPLES=OFF \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D OPENCV_ENABLE_NONFREE=OFF \
-D BUILD_EXAMPLES=OFF \
-D CMAKE_C_COMPILER=/usr/bin/clang \
-D CMAKE_CXX_COMPILER=/usr/bin/clang++ \
-D BUILD_NEW_PYTHON_SUPPORT=ON \
-D BUILD_opencv_python3=ON \
-D HAVE_opencv_python3=ON \
-D PYTHON_DEFAULT_EXECUTABLE=/usr/bin/python3.6 \
.. \

...cmake will output:

...
Python (for build): /usr/bin/python3.6
...

Let's now build and install OpenCV:

~/Downloads/opencv-4.1.1/build$ make install
...
file cannot create directory: /usr/local/share/licenses/opencv4. Maybe
need administrative privileges.

We need to use sudo:

~/Downloads/opencv-4.1.1/build$ sudo make install
$ sudo ldconfig

Let's check OpenCV version:

(py3.6_cv2) bojan@...$ opencv_version 
4.1.1

cv2 package still can't be loaded to Python code:

$ python
Python 3.6.8 (default, Oct 7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
Traceback (most recent call last):
File "", line 1, in
ModuleNotFoundError: No module named 'cv2'
>>>
KeyboardInterrupt
>>> exit()


Lets' see where is Python binding for OpenCV:

$ sudo ls /usr/local/lib/python3.6/site-packages/cv2/python-3.6/
cv2.cpython-36m-x86_64-linux-gnu.so



We need to create soft link:

(py3.6_cv2) bojan@...:~/Downloads/opencv-4.1.1/build$ cd ~/.virtualenvs/py3.6_cv2/lib/python3.6/site-packages/

(py3.6_cv2) bojan@...:~/.virtualenvs/py3.6_cv2/lib/python3.6/site-packages$ ln -s /usr/local/lib/python3.6/site-packages/cv2/python-3.6/cv2.cpython-36m-x86_64-linux-gnu.so cv2.so


Importing cv2 package works now:

(py3.6_cv2) bojan@...:~/.virtualenvs/py3.6_cv2/lib/python3.6/site-packages$ python
Python 3.6.8 (default, Oct 7 2019, 12:59:55)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cv2
>>> cv2.__version__
'4.1.1'
>>> quit()

Sunday, 20 October 2019

How to install Docker on Ubuntu 18.04

To install Docker on my Ubuntu 18.04 machines I followed the instructions from the Docker's website: https://docs.docker.com/install/linux/docker-ce/ubuntu/

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic

$ sudo apt-get update

sudo apt-get install \
>     apt-transport-https \
>     ca-certificates \
>     curl \
>     gnupg-agent \
>     software-properties-common


$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
OK

$ apt-key fingerprint 0EBFCD88
pub   rsa4096 2017-02-22 [SCEA]
      9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid           [ unknown] Docker Release (CE deb) <docker@docker.com>
sub   rsa4096 2017-02-22 [S]

$ sudo add-apt-repository \
>    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
>    $(lsb_release -cs) \
>    stable"


$ sudo apt-get update

$ sudo apt-get install docker-ce docker-ce-cli containerd.io

sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
1b930d010525: Pull complete
Digest: sha256:2557e3c07ed1e38f26e389462d03ed943586f744621577a99efb77324b0fe535
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

$ docker -v
Docker version 19.03.4, build 9013bf583a

How to run Docker as non-root user?


Your regular user has to be added to the docker group (whose members are allowed to communicate wtih Docker daemon which runs as root):

$ sudo usermod -aG docker $USER

$ newgrp docker

$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/



References:

https://docs.docker.com/v17.09/engine/installation/linux/linux-postinstall/

Friday, 11 October 2019

How to install Meld on Ubuntu

Meld is popular cross-platform visual diff and merge tool. It can compare files and directories.

Clone Meld repository (use HTTPS URI, not the SSH-based one unless you've set up SSH key for that repo):

$ git clone https://gitlab.gnome.org/GNOME/meld.git

Check the status of master and release branches. When I built and run the app from master, I got this error:

$ meld
Meld requires GtkSourceView 4 4.0.0 or higher.




Then I found the following comment here:
> If Meld is complaining about GtkSourceView 4, you're using master, not 3.20.1.

This means I should have checked out the release branch for the desired version:

$ cd meld/
$ git checkout meld-3-20

When I was running the installer:

$ python3 setup.py install --prefix=/usr

...I was getting errors related to missing packages:

unable to execute 'intltool-update': No such file or directory
error: command 'intltool-update' failed with exit status 1

unable to execute 'xmllint': No such file or directory
error: command 'xmllint' failed with exit status 1

unable to execute 'glib-compile-resources': No such file or directory
error: command 'glib-compile-resources' failed with exit status 1

I installed the required packages (the package needed for xmllint is libxml2-utils):

$ sudo apt update
$ sudo apt upgrade
$ sudo apt install intltool 
$ sudo apt install xmllint
E: Unable to locate package xmllint
$ sudo apt install libxml2-utils
$ glib-compile-resources --help
Command 'glib-compile-resources' not found, but can be installed with:
sudo apt install libglib2.0-dev-bin
$ sudo apt install libglib2.0-dev-bin

...and run again installer:

$ python3 setup.py install --prefix=/usr

...which now complained about permissions:

creating /usr/lib/python3/dist-packages/meld
error: could not create '/usr/lib/python3/dist-packages/meld': Permission denied

Meld was finally installed with sudo user:

$ sudo python3 setup.py install --prefix=/usr
...
running install_egg_info
Writing /usr/lib/python3/dist-packages/meld-3.21.0.egg-info

When I run it first time I got this error:

$ meld
Meld requires pygobject 3.30.0 or higher.




To install the latest version of PyGObject:

$ pip3 install PyGObject 

Installing PyGObject complained couple of times about missing packages:

No package 'glib-2.0' found
Try installing it with: 'sudo apt install libglib2.0-dev'

No package 'gobject-introspection-1.0' found
Try installing it with: 'sudo apt install libgirepository1.0-dev'

No package 'cairo' found
Try installing it with: 'sudo apt install libcairo2-dev'

Installing PyGObject was successful once I installed all missing packages:

$ sudo apt install libglib2.0-dev
$ sudo apt install libgirepository1.0-dev
$ sudo apt install libcairo2-dev
$ pip3 install pycairo

I rebuilt the app and run it again:

$ sudo python3 setup.py install --prefix=/usr
$ meld

This time it opened with no issues:


References:

Monday, 7 October 2019

What do you need to do before upgrading Postgres Docker image

If you just use the new Postgres version, it is very likely you'll get a message like this:

FATAL:  database files are incompatible with server
DETAIL:  The data directory was initialized by PostgreSQL version X, which is not compatible with this version Y

In my case, I had Postgres and pgAdmin running as services in docker-compose:

$ docker-compose pull && docker-compose up
Pulling pg_db    ... done
Pulling pg_admin ... done
Recreating postgres-demo-pg ... done
Recreating postgres-demo-pgadmin ... done
Attaching to postgres-demo-pg, postgres-demo-pgadmin
postgres-demo-pg | 2019-10-07 09:46:59.270 UTC [1] FATAL:  database files are incompatible with server
postgres-demo-pg | 2019-10-07 09:46:59.270 UTC [1] DETAIL:  The data directory was initialized by PostgreSQL version 11, which is not compatible with this version 12.0 (Debian 12.0-1.pgdg100+1).
postgres-demo-pg exited with code 1

To prevent this situation do the following:

Revert postgres version to the previous one (that matches data version, 11 in my case):

docker-compose.yml:

services:
   pg_db:
      image: posgres:11

Run postgres docker container so db loads data as usual:

$ docker-compose up

Use docker exec to dump data into file:

$ docker exec postgres-demo-pg pg_dumpall -U postgres > dump.sql

Stop db container:

$ docker-compose down

Delete old mount dir:

$ sudo rm -rf database_data/

Recreate it:

$ mkdir database_data

Copy dump file to local dir that will be mounted as Postgres data dir:

$ sudo cp dump.sql database_data/

Set postgres version to the latest one:

docker-compose.yml:

services:
   pg_db:
      image: postgres


Run docker container based on the image with the latest Postgres version:

$ docker-compose pull && docker-compose up

Load dump file into db:

$ docker exec -it postgres-demo-pg bash
root@2873483c35ac:/# cd /var/lib/postgresql/data/
root@2873483c35ac:/var/lib/postgresql/data# psql -U postgres < dump.sql
SET
...
SET
root@2873483c35ac:/var/lib/postgresql/data# exit
exit

Verify db version - that it is indeed the latest:

$ docker exec -it postgres-demo-pg bashroot@2873483c35ac:/# postgres --version
postgres (PostgreSQL) 12.0 (Debian 12.0-1.pgdg100+1)
root@2873483c35ac:/# exit
exit