Monday, 22 February 2021

How to check file GPG signature on Linux

Retrieving and installation of Linux software can be done via package management systems (like apt) or manually downloading and running the executable. 

Public Key Cryptography is used on Linux to verify the identity of the file provider. A public and private key are created, file sender signs the file with private key and publishes public key. Receiver uses the public key to verify the signature. 

GNU Privacy Guard (GnuPG or GPG) is set of open-source tools for key exchange, signing, verification, encrypting and decrypting files.




I want to show here how to use gpg to verify a file downloaded from Internet on the example of Tor browser installer. It is not necessary to do this when using package managers to install some software as they use this automatically (see SecureApt - Debian Wiki).

From Tor's website we need to download:

1) an installation archive (https://www.torproject.org/dist/torbrowser/10.0.10/tor-browser-linux64-10.0.10_en-US.tar.xz) 

2) its PGP signature (https://dist.torproject.org/torbrowser/10.0.10/tor-browser-linux64-10.0.10_en-US.tar.xz.asc) which looks like this:

-----BEGIN PGP SIGNATURE-----

iQIcBAABCgAGBQJgG5QbAAoJEOt3RJHZ/wbinYQP/RVTZOY9YLe7y2azJs2K3IcF
Wr20e2zMGVGpvV2WZ3uYij7nMQGujPU4V8cJ/5vSb3ONKSwr/YXfqSGOC6gL8EAF
yvJsS5JjVdGYpBxg9ye1Cb3AFtKgQLBwFLvDWtWuLH1lk7VDLY5dDPbRpX6x3HKE
LbulR1IsUdQLdrbUfkoUpYBf1GTb+4+F4a//tLEnCwWojO72jL3UNzRmKPCOZ+KD
IxbGxHCtqAHMU8dBYZ5yQugW7mLnzmTAyFtfem9KuJXMFlpKICadOK6yZDYH2U3Y
SPGONS3vRZEhlfuDmyh0U18L7A3H25ELdbUTYhQLfZj0/OCy0cyqESzn65cpSPS8
/Fg2BdnmyzkWqo45KF5l4KCoT3+Jd2BHC6uZ8na/LpW6WncQ43+8obBbthgHODLA
rHGTTA7ccR+2aWFHjlZaGHErj7mjdRiNuOzlpWVpfMwcvMoCxN71DUxrVIdQchY4
j7K4fvt5ZpR3Ln0pFd9QgbwES6bqi2U0MZ872ChQ/LPoYhGxqrsEpJTJoZBnJ7Ik
IL4MiHLcN+9RKwamBhC9qjOp9V1IHVexxPvRxWc1Ix3Vwp5W7hFSZoM+pBruMjsF
9gc8pEL3RETBomr8zSGRPKDAvHv/V+r31pHTEr9pgEn3rN7wU7VCABVrK7pHJltl
03nm5LAu7HdLxFYysxur
=6o0j
-----END PGP SIGNATURE-----

Now we want to verify the authenticity of the downloaded file. The tool that we're going to use is gpg. gpg is included in Linux distributions (I am using Ubuntu).

$ gpg --help
gpg (GnuPG) 2.2.19
libgcrypt 1.8.5
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Home: /home/bojan/.gnupg
Supported algorithms:
Pubkey: RSA, ELG, DSA, ECDH, ECDSA, EDDSA
Cypher: IDEA, 3DES, CAST5, BLOWFISH, AES, AES192, AES256, TWOFISH,
        CAMELLIA128, CAMELLIA192, CAMELLIA256
Hash: SHA1, RIPEMD160, SHA256, SHA384, SHA512, SHA224
Compression: Uncompressed, ZIP, ZLIB, BZIP2

Syntax: gpg [options] [files]
Sign, check, encrypt or decrypt
Default operation depends on the input data

Commands:
 
 -s, --sign                  make a signature
     --clear-sign            make a clear text signature
 -b, --detach-sign           make a detached signature
 -e, --encrypt               encrypt data
 -c, --symmetric             encryption only with symmetric cypher
 -d, --decrypt               decrypt data (default)
     --verify                verify a signature
 -k, --list-keys             list keys
     --list-signatures       list keys and signatures
     --check-signatures      list and check key signatures
     --fingerprint           list keys and fingerprints
 -K, --list-secret-keys      list secret keys
     --generate-key          generate a new key pair
     --quick-generate-key    quickly generate a new key pair
     --quick-add-uid         quickly add a new user-id
     --quick-revoke-uid      quickly revoke a user-id
     --quick-set-expire      quickly set a new expiration date
     --full-generate-key     full featured key pair generation
     --generate-revocation   generate a revocation certificate
     --delete-keys           remove keys from the public keyring
     --delete-secret-keys    remove keys from the secret keyring
     --quick-sign-key        quickly sign a key
     --quick-lsign-key       quickly sign a key locally
     --sign-key              sign a key
     --lsign-key             sign a key locally
     --edit-key              sign or edit a key
     --change-passphrase     change a passphrase
     --export                export keys
     --send-keys             export keys to a keyserver
     --receive-keys          import keys from a keyserver
     --search-keys           search for keys on a keyserver
     --refresh-keys          update all keys from a keyserver
     --import                import/merge keys
     --card-status           print the card status
     --edit-card             change data on a card
     --change-pin            change a card's PIN
     --update-trustdb        update the trust database
     --print-md              print message digests
     --server                run in server mode
     --tofu-policy VALUE     set the TOFU policy for a key

Options:
 
 -a, --armor                 create ASCII armoured output
 -r, --recipient USER-ID     encrypt for USER-ID
 -u, --local-user USER-ID    use USER-ID to sign or decrypt
 -z N                        set compress level to N (0 disables)
     --textmode              use canonical text mode
 -o, --output FILE           write output to FILE
 -v, --verbose               verbose
 -n, --dry-run               do not make any changes
 -i, --interactive           prompt before overwriting
     --openpgp               use strict OpenPGP behaviour

(See the man page for a complete listing of all commands and options)

Examples:

 -se -r Bob [file]          sign and encrypt for user Bob
 --clear-sign [file]        make a clear text signature
 --detach-sign [file]       make a detached signature
 --list-keys [names]        show keys
 --fingerprint [names]      show fingerprints

Please report bugs to <https://bugs.gnupg.org>.


From Tor's website we can find that it is signed by Tor Browser Developers and that their signing key is 0xEF6E286DDA85EA2A4BA7DE684E2C6E8793298290. 

gpg can import the public key of the author from Web Key Directory it deducts from author's public email ID:

$ gpg --auto-key-locate nodefault,wkd --locate-keys torbrowser@torproject.org
gpg: key 4E2C6E8793298290: public key "Tor Browser Developers (signing key) <torbrowser@torproject.org>" imported
gpg: Total number processed: 1
gpg:               imported: 1
pub   rsa4096 2014-12-15 [C] [expires: 2025-07-21]
      EF6E286DDA85EA2A4BA7DE684E2C6E8793298290
uid           [ unknown] Tor Browser Developers (signing key) <torbrowser@torproject.org>
sub   rsa4096 2018-05-26 [S] [expires: 2021-06-12]

--auto-key-locate defines how will gpg locate keys:

--auto-key-locate mechanisms
--no-auto-key-locate
              GnuPG  can  automatically  locate  and  retrieve  keys  as  needed using this option.  This happens when encrypting to an email address (in the "user@example.com" form), and there are no "user@example.com" keys on the local keyring.  This option takes any number of the mechanisms listed below, in the order they are to be tried.  Instead of listing the mechanisms as comma delimited  arguments,  the  option  may also be given several times to add more mechanism.  The option --no-auto-key-locate or the mechanism "clear" resets the list.  The default is "local,wkd".

              cert   Locate a key using DNS CERT, as specified in RFC-4398.
              pka    Locate a key using DNS PKA.
              dane   Locate a key using DANE, as specified in draft-ietf-dane-openpgpkey-05.txt.
              wkd    Locate a key using the Web Key Directory protocol.
              ldap   Using DNS Service Discovery, check the domain in question for any LDAP keyservers to use.  If this fails, attempt to locate the key using the  PGP  Universal  method  of  checking ‘ldap://keys.(thedomain)’.
              keyserver Locate a key using a keyserver.
              keyserver-URL In addition, a keyserver URL as used in the dirmngr configuration may be used here to query that particular keyserver.
              local  Locate  the  key  using  the  local keyrings.  This mechanism allows the user to select the order a local key lookup is done.  Thus using ‘--auto-key-locate local’ is identical to --no-auto-key-locate.
              nodefault
                     This flag disables the standard local key lookup, done before any of the mechanisms defined by the --auto-key-locate are tried.  The position of this mechanism in  the  list  does not matter.  It is not required if local is also used.
              clear  Clear all defined mechanisms.  This is useful to override mechanisms given in a config file.  Note that a nodefault in mechanisms will also be cleared unless it is given after the clear.


Web Key Directories: 
  • provide an easy way to discover public keys through HTTPS
  • way to retrieve PGP keys without using a keyserver
  • keep email addresses private (as you need to know an address already to ask for a public key)
  • are authoritative source for a public key for its domain
Web Key Directory protocol:
  • The sender's mail client checks a "well known" URL on the domain of the recipient.
  • If a public key is available for that mail address, will be downloaded via HTTPS.
  • The downloaded pubkey can now be used without further user interaction.
How does gpg find the URL of the key? It uses the host name from the provided email address and constructs it. See here: Attacks on GnuPG's Web Key Directory (WKD) | SektionEins GmbH and here: draft-koch-openpgp-webkey-service-08 - OpenPGP Web Key Directory.

torbrowser@torproject.org =>
    username (local-part): torbrowser
    domain: torproject.org

$ echo -n "torbrowser" | sha1sum 
5426242bb720dfdd6dc01bfcddfe3f5b92d9f065 

I used Cryptii online tool which uses z-base-32 Encoder to encrypt the string (sha1 sum):

kounek7zrdx745qydx6p59t9mqjpuhdf

We now have zbase32 username.

Every domain can host host the WKD from a special openpgpkey subdomain: openpgpkey.

It should have a folder structure similar to this:

.well-known
.well-known/openpgpkey
.well-known/openpgpkey/example.com
.well-known/openpgpkey/example.com/policy
.well-known/openpgpkey/example.com/hu
.well-known/openpgpkey/example.com/hu/<zbase32_username>

In our case:

https://openpgpkey.torproject.org/.well-known/openpgpkey/torproject.org/hu/kounek7zrdx745qydx6p59t9mqjpuhdf

We can see that Tor Browser Developers (signing key) is at that URL (I'm printing here only first 4 lines of the output): 

$ curl https://openpgpkey.torproject.org/.well-known/openpgpkey/hu/kounek7zrdx745qydx6p59t9mqjpuhdf?l=torbrowser \
--output - \
| head -n4
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
T����2ϐ��,�c݉0�     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
� Jʸ�X�jV_���>�P�AD���2B�>[���Ԟrv�K
Ge�ӧGWo��31�S#�2�H�+Յ5�Y�8{�Պbq'M�����D\l}�~9�S!/�v(�w�(;���bX/���@Tor Browser Developers (signing key) <torbrowser@torproject.org>�Te%�G�'��`;��X�;D�5q_���¤�f=��N��)�>�'�n}��K��TGhi��7;����[�
>
 
  5025    0     0  14075      0 --:--:-- --:--:-- --:--:-- 14036
(23) Failed writing body

Let's save this key in tor.keyring file:

$ gpg --output ./tor.keyring --export 0xEF6E286DDA85EA2A4BA7DE684E2C6E8793298290


Verification of the signature is done via gpgv tool:

$ gpgv --help
gpgv (GnuPG) 2.2.19
libgcrypt 1.8.5
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Syntax: gpgv [options] [files]
Check signatures against known trusted keys


Options:
 
 -v, --verbose                verbose
 -q, --quiet                  be somewhat more quiet
     --keyring FILE           take the keys from the keyring FILE
 -o, --output FILE            write output to FILE
     --ignore-time-conflict   make timestamp conflicts only a warning
     --status-fd FD           write status info to this FD
     --weak-digest ALGO       reject signatures made with ALGO

Please report bugs to <https://bugs.gnupg.org>.


$ gpgv \
--keyring ./tor.keyring \
~/Downloads/tor-browser-linux64-10.0.10_en-US.tar.xz.asc \ ~/Downloads/tor-browser-linux64-10.0.10_en-US.tar.xz
gpgv: Signature made Thu 04 Feb 2021 06:28:43 GMT
gpgv:                using RSA key EB774491D9FF06E2
gpgv: Good signature from "Tor Browser Developers (signing key) <torbrowser@torproject.org>"

Wednesday, 10 February 2021

How to detect if computer supports CUDA?

To run applications that use CUDA we need to make sure that our computer has GPUs that support CUDA.

From Google Chrome Browser


Go to 

chrome://gpu/ 

and in the result page look for:

GPU CUDA compute capability major version

It is also possible to check here the presence of GPUs on the system:

Driver Information
...
GPU0 VENDOR= 0x10de, DEVICE=0x13b4
GPU1 VENDOR= 0x8086, DEVICE=0x591b *ACTIVE*
...

0x10DE is NVIDIA's vendor ID.
0x8086 is Intel's vendor ID.

Check Graphic Cards


GPUs on NVIDA graphic cards support CUDA co check if you have any of them on your board:


$ lspci
...
01:00.0 VGA compatible controller: NVIDIA Corporation GK107 [GeForce GT 640 OEM] (rev a1)
...


Run nvidia-settings


nvidia-settings can be installed with:

$ sudo apt install nvidia-settings

$ nvidia-settings




Run nvidia-smi


nvidia-smi can be installed with:

sudo apt install nvidia-340               # version 340.108-0ubuntu5.20.04.1, or
sudo apt install nvidia-utils-390         # version 390.141-0ubuntu0.20.04.1
sudo apt install nvidia-utils-450         # version 450.102.04-0ubuntu0.20.04.1
sudo apt install nvidia-utils-450-server  # version 450.102.04-0ubuntu0.20.04.1
sudo apt install nvidia-utils-460         # version 460.32.03-0ubuntu0.20.04.1
sudo apt install nvidia-utils-435         # version 435.21-0ubuntu7
sudo apt install nvidia-utils-440         # version 440.82+really.440.64-0ubuntu6
sudo apt install nvidia-utils-418-server  # version 418.181.07-0ubuntu0.20.04.1

$ sudo apt install nvidia-utils-460
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libgfortran4 libhdf5-100 libnetcdf13
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  libnvidia-compute-460
Suggested packages:
  nvidia-driver-460
The following NEW packages will be installed
  libnvidia-compute-460 nvidia-utils-460
0 to upgrade, 2 to newly install, 0 to remove and 1 not to upgrade.
Need to get 22.2 MB of archives.
After this operation, 103 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 libnvidia-compute-460 amd64 460.32.03-0ubuntu0.20.04.1 [21.8 MB]
Get:2 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 nvidia-utils-460 amd64 460.32.03-0ubuntu0.20.04.1 [362 kB]     
Fetched 22.2 MB in 2min 5s (177 kB/s)                                                                                                   
Selecting previously unselected package libnvidia-compute-460:amd64.
(Reading database ... 304171 files and directories currently installed.)
Preparing to unpack .../libnvidia-compute-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking libnvidia-compute-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package nvidia-utils-460.
Preparing to unpack .../nvidia-utils-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking nvidia-utils-460 (460.32.03-0ubuntu0.20.04.1) ...
Setting up libnvidia-compute-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Setting up nvidia-utils-460 (460.32.03-0ubuntu0.20.04.1) ...
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
/sbin/ldconfig.real: /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5 is not a symbolic link

$ nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

https://www.nvidia.com/en-us/drivers/unix/

https://www.nvidia.com/Download/driverResults.aspx/170134/en-us contains download section with a link to file NVIDIA-Linux-x86_64-460.39.run (so the latest version is 460) but it also contains this note:
Note that many Linux distributions provide their own packages of the NVIDIA Linux Graphics Driver in the distribution's native package management format. This may interact better with the rest of your distribution's framework, and you may want to use this rather than NVIDIA's official package.

So let's check what apt contains:

$ apt search nvidia-driver
Sorting... Done
Full Text Search... Done
nvidia-384/focal-updates,focal-security 390.141-0ubuntu0.20.04.1 amd64
  Transitional package for nvidia-driver-390

nvidia-384-dev/focal-updates,focal-security 390.141-0ubuntu0.20.04.1 amd64
  Transitional package for nvidia-driver-390

nvidia-driver-390/focal-updates,focal-security 390.141-0ubuntu0.20.04.1 amd64
  NVIDIA driver metapackage

nvidia-driver-418/focal 430.50-0ubuntu3 amd64
  Transitional package for nvidia-driver-430

nvidia-driver-418-server/focal-updates,focal-security 418.181.07-0ubuntu0.20.04.1 amd64
  NVIDIA Server Driver metapackage

nvidia-driver-430/focal-updates,focal-security 440.100-0ubuntu0.20.04.1 amd64
  Transitional package for nvidia-driver-440

nvidia-driver-435/focal-updates 455.45.01-0ubuntu0.20.04.1 amd64
  Transitional package for nvidia-driver-455

nvidia-driver-440/focal-updates,focal-security 450.102.04-0ubuntu0.20.04.1 amd64
  Transitional package for nvidia-driver-450

nvidia-driver-440-server/focal-updates,focal-security 450.102.04-0ubuntu0.20.04.1 amd64
  Transitional package for nvidia-driver-450-server

nvidia-driver-450/focal-updates,focal-security 450.102.04-0ubuntu0.20.04.1 amd64
  NVIDIA driver metapackage

nvidia-driver-450-server/focal-updates,focal-security 450.102.04-0ubuntu0.20.04.1 amd64
  NVIDIA Server Driver metapackage

nvidia-driver-455/focal-updates,focal-security 460.32.03-0ubuntu0.20.04.1 amd64
  Transitional package for nvidia-driver-460

nvidia-driver-460/focal-updates,focal-security 460.32.03-0ubuntu0.20.04.1 amd64
  NVIDIA driver metapackage

nvidia-headless-390/focal-updates,focal-security 390.141-0ubuntu0.20.04.1 amd64
  NVIDIA headless metapackage

nvidia-headless-418-server/focal-updates,focal-security 418.181.07-0ubuntu0.20.04.1 amd64
  NVIDIA headless metapackage

nvidia-headless-450/focal-updates,focal-security 450.102.04-0ubuntu0.20.04.1 amd64
  NVIDIA headless metapackage

nvidia-headless-450-server/focal-updates,focal-security 450.102.04-0ubuntu0.20.04.1 amd64
  NVIDIA headless metapackage

nvidia-headless-460/focal-updates,focal-security 460.32.03-0ubuntu0.20.04.1 amd64
  NVIDIA headless metapackage

nvidia-headless-no-dkms-390/focal-updates,focal-security 390.141-0ubuntu0.20.04.1 amd64
  NVIDIA headless metapackage - no DKMS

nvidia-headless-no-dkms-418-server/focal-updates,focal-security 418.181.07-0ubuntu0.20.04.1 amd64
  NVIDIA headless metapackage - no DKMS

nvidia-headless-no-dkms-450/focal-updates,focal-security 450.102.04-0ubuntu0.20.04.1 amd64
  NVIDIA headless metapackage - no DKMS

nvidia-headless-no-dkms-450-server/focal-updates,focal-security 450.102.04-0ubuntu0.20.04.1 amd64
  NVIDIA headless metapackage - no DKMS

nvidia-headless-no-dkms-460/focal-updates,focal-security 460.32.03-0ubuntu0.20.04.1 amd64
  NVIDIA headless metapackage - no DKMS

xserver-xorg-video-nvidia-390/focal-updates,focal-security 390.141-0ubuntu0.20.04.1 amd64
  NVIDIA binary Xorg driver

xserver-xorg-video-nvidia-418-server/focal-updates,focal-security 418.181.07-0ubuntu0.20.04.1 amd64
  NVIDIA binary Xorg driver

xserver-xorg-video-nvidia-450/focal-updates,focal-security 450.102.04-0ubuntu0.20.04.1 amd64
  NVIDIA binary Xorg driver

xserver-xorg-video-nvidia-450-server/focal-updates,focal-security 450.102.04-0ubuntu0.20.04.1 amd64
  NVIDIA binary Xorg driver

xserver-xorg-video-nvidia-460/focal-updates,focal-security 460.32.03-0ubuntu0.20.04.1 amd64
  NVIDIA binary Xorg driver

Let's install the one with the highest number (that one should be the latest one):

$ sudo apt install nvidia-driver-460
[sudo] password for bojan: 
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libgfortran4 libhdf5-100 libnetcdf13
Use 'sudo apt autoremove' to remove them.
The following additional packages will be installed:
  dkms libnvidia-cfg1-460 libnvidia-common-460 libnvidia-compute-460:i386 libnvidia-decode-460 libnvidia-decode-460:i386
  libnvidia-encode-460 libnvidia-encode-460:i386 libnvidia-extra-460 libnvidia-fbc1-460 libnvidia-fbc1-460:i386 libnvidia-gl-460
  libnvidia-gl-460:i386 libnvidia-ifr1-460 libnvidia-ifr1-460:i386 nvidia-compute-utils-460 nvidia-dkms-460 nvidia-kernel-common-460
  nvidia-kernel-source-460 nvidia-prime nvidia-settings screen-resolution-extra xserver-xorg-video-nvidia-460
Suggested packages:
  menu
The following NEW packages will be installed
  dkms libnvidia-cfg1-460 libnvidia-common-460 libnvidia-compute-460:i386 libnvidia-decode-460 libnvidia-decode-460:i386
  libnvidia-encode-460 libnvidia-encode-460:i386 libnvidia-extra-460 libnvidia-fbc1-460 libnvidia-fbc1-460:i386 libnvidia-gl-460
  libnvidia-gl-460:i386 libnvidia-ifr1-460 libnvidia-ifr1-460:i386 nvidia-compute-utils-460 nvidia-dkms-460 nvidia-driver-460
  nvidia-kernel-common-460 nvidia-kernel-source-460 nvidia-prime nvidia-settings screen-resolution-extra xserver-xorg-video-nvidia-460
0 to upgrade, 24 to newly install, 0 to remove and 1 not to upgrade.
Need to get 154 MB of archives.
After this operation, 531 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Get:1 http://gb.archive.ubuntu.com/ubuntu focal/main amd64 dkms all 2.8.1-5ubuntu1 [66.6 kB]
Get:2 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 libnvidia-cfg1-460 amd64 460.32.03-0ubuntu0.20.04.1 [77.2 kB]
Get:3 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 libnvidia-common-460 all 460.32.03-0ubuntu0.20.04.1 [9,168 B]
Get:4 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted i386 libnvidia-compute-460 i386 460.32.03-0ubuntu0.20.04.1 [23.2 MB]
Get:5 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted i386 libnvidia-decode-460 i386 460.32.03-0ubuntu0.20.04.1 [1,289 kB] 
Get:6 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 libnvidia-decode-460 amd64 460.32.03-0ubuntu0.20.04.1 [1,300 kB]
Get:7 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 libnvidia-encode-460 amd64 460.32.03-0ubuntu0.20.04.1 [39.6 kB]
Get:8 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted i386 libnvidia-encode-460 i386 460.32.03-0ubuntu0.20.04.1 [42.2 kB]  
Get:9 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 libnvidia-extra-460 amd64 460.32.03-0ubuntu0.20.04.1 [39.7 kB] 
Get:10 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted i386 libnvidia-fbc1-460 i386 460.32.03-0ubuntu0.20.04.1 [48.5 kB]   
Get:11 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 libnvidia-fbc1-460 amd64 460.32.03-0ubuntu0.20.04.1 [45.4 kB] 
Get:12 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted i386 libnvidia-gl-460 i386 460.32.03-0ubuntu0.20.04.1 [19.1 MB]     
Get:13 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 libnvidia-gl-460 amd64 460.32.03-0ubuntu0.20.04.1 [80.6 MB]   
Get:14 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted i386 libnvidia-ifr1-460 i386 460.32.03-0ubuntu0.20.04.1 [69.1 kB]   
Get:15 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 libnvidia-ifr1-460 amd64 460.32.03-0ubuntu0.20.04.1 [67.6 kB] 
Get:16 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 nvidia-compute-utils-460 amd64 460.32.03-0ubuntu0.20.04.1 [105 kB]
Get:17 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 nvidia-kernel-source-460 amd64 460.32.03-0ubuntu0.20.04.1 [24.5 MB]
Get:18 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 nvidia-kernel-common-460 amd64 460.32.03-0ubuntu0.20.04.1 [19.1 kB]
Get:19 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 nvidia-dkms-460 amd64 460.32.03-0ubuntu0.20.04.1 [28.3 kB]    
Get:20 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 xserver-xorg-video-nvidia-460 amd64 460.32.03-0ubuntu0.20.04.1 [1,557 kB]
Get:21 http://gb.archive.ubuntu.com/ubuntu focal-updates/restricted amd64 nvidia-driver-460 amd64 460.32.03-0ubuntu0.20.04.1 [431 kB]   
Get:22 http://gb.archive.ubuntu.com/ubuntu focal-updates/main amd64 nvidia-prime all 0.8.15.3~0.20.04.1 [9,248 B]                       
Get:23 http://gb.archive.ubuntu.com/ubuntu focal/main amd64 screen-resolution-extra all 0.18build1 [4,456 B]                            
Get:24 http://gb.archive.ubuntu.com/ubuntu focal-updates/main amd64 nvidia-settings amd64 440.82-0ubuntu0.20.04.1 [963 kB]              
Fetched 154 MB in 52s (2,967 kB/s)                                                                                                      
Selecting previously unselected package dkms.
(Reading database ... 304197 files and directories currently installed.)
Preparing to unpack .../00-dkms_2.8.1-5ubuntu1_all.deb ...
Unpacking dkms (2.8.1-5ubuntu1) ...
Selecting previously unselected package libnvidia-cfg1-460:amd64.
Preparing to unpack .../01-libnvidia-cfg1-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking libnvidia-cfg1-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-common-460.
Preparing to unpack .../02-libnvidia-common-460_460.32.03-0ubuntu0.20.04.1_all.deb ...
Unpacking libnvidia-common-460 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-compute-460:i386.
Preparing to unpack .../03-libnvidia-compute-460_460.32.03-0ubuntu0.20.04.1_i386.deb ...
Unpacking libnvidia-compute-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-decode-460:i386.
Preparing to unpack .../04-libnvidia-decode-460_460.32.03-0ubuntu0.20.04.1_i386.deb ...
Unpacking libnvidia-decode-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-decode-460:amd64.
Preparing to unpack .../05-libnvidia-decode-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking libnvidia-decode-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-encode-460:amd64.
Preparing to unpack .../06-libnvidia-encode-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking libnvidia-encode-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-encode-460:i386.
Preparing to unpack .../07-libnvidia-encode-460_460.32.03-0ubuntu0.20.04.1_i386.deb ...
Unpacking libnvidia-encode-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-extra-460:amd64.
Preparing to unpack .../08-libnvidia-extra-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking libnvidia-extra-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-fbc1-460:i386.
Preparing to unpack .../09-libnvidia-fbc1-460_460.32.03-0ubuntu0.20.04.1_i386.deb ...
Unpacking libnvidia-fbc1-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-fbc1-460:amd64.
Preparing to unpack .../10-libnvidia-fbc1-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking libnvidia-fbc1-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-gl-460:amd64.
Preparing to unpack .../11-libnvidia-gl-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking libnvidia-gl-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-gl-460:i386.
Preparing to unpack .../12-libnvidia-gl-460_460.32.03-0ubuntu0.20.04.1_i386.deb ...
Unpacking libnvidia-gl-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-ifr1-460:i386.
Preparing to unpack .../13-libnvidia-ifr1-460_460.32.03-0ubuntu0.20.04.1_i386.deb ...
Unpacking libnvidia-ifr1-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package libnvidia-ifr1-460:amd64.
Preparing to unpack .../14-libnvidia-ifr1-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking libnvidia-ifr1-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package nvidia-compute-utils-460.
Preparing to unpack .../15-nvidia-compute-utils-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking nvidia-compute-utils-460 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package nvidia-kernel-source-460.
Preparing to unpack .../16-nvidia-kernel-source-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking nvidia-kernel-source-460 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package nvidia-kernel-common-460.
Preparing to unpack .../17-nvidia-kernel-common-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking nvidia-kernel-common-460 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package nvidia-dkms-460.
Preparing to unpack .../18-nvidia-dkms-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking nvidia-dkms-460 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package xserver-xorg-video-nvidia-460.
Preparing to unpack .../19-xserver-xorg-video-nvidia-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking xserver-xorg-video-nvidia-460 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package nvidia-driver-460.
Preparing to unpack .../20-nvidia-driver-460_460.32.03-0ubuntu0.20.04.1_amd64.deb ...
Unpacking nvidia-driver-460 (460.32.03-0ubuntu0.20.04.1) ...
Selecting previously unselected package nvidia-prime.
Preparing to unpack .../21-nvidia-prime_0.8.15.3~0.20.04.1_all.deb ...
Unpacking nvidia-prime (0.8.15.3~0.20.04.1) ...
Selecting previously unselected package screen-resolution-extra.
Preparing to unpack .../22-screen-resolution-extra_0.18build1_all.deb ...
Unpacking screen-resolution-extra (0.18build1) ...
Selecting previously unselected package nvidia-settings.
Preparing to unpack .../23-nvidia-settings_440.82-0ubuntu0.20.04.1_amd64.deb ...
Unpacking nvidia-settings (440.82-0ubuntu0.20.04.1) ...
Setting up libnvidia-common-460 (460.32.03-0ubuntu0.20.04.1) ...
Setting up libnvidia-fbc1-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Setting up libnvidia-fbc1-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Setting up nvidia-compute-utils-460 (460.32.03-0ubuntu0.20.04.1) ...
Warning: The home dir /nonexistent you specified can't be accessed: No such file or directory
Adding system user `nvidia-persistenced' (UID 128) ...
Adding new group `nvidia-persistenced' (GID 137) ...
Adding new user `nvidia-persistenced' (UID 128) with group `nvidia-persistenced' ...
Not creating home directory `/nonexistent'.
Setting up nvidia-prime (0.8.15.3~0.20.04.1) ...
Setting up libnvidia-compute-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Setting up nvidia-kernel-common-460 (460.32.03-0ubuntu0.20.04.1) ...
update-initramfs: deferring update (trigger activated)
Setting up dkms (2.8.1-5ubuntu1) ...
Setting up libnvidia-cfg1-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Setting up xserver-xorg-video-nvidia-460 (460.32.03-0ubuntu0.20.04.1) ...
Setting up nvidia-kernel-source-460 (460.32.03-0ubuntu0.20.04.1) ...
Setting up screen-resolution-extra (0.18build1) ...
Setting up libnvidia-gl-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Setting up libnvidia-gl-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Setting up nvidia-settings (440.82-0ubuntu0.20.04.1) ...
Setting up libnvidia-extra-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Setting up libnvidia-decode-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Setting up libnvidia-decode-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Setting up libnvidia-ifr1-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Setting up libnvidia-ifr1-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Setting up nvidia-dkms-460 (460.32.03-0ubuntu0.20.04.1) ...
update-initramfs: deferring update (trigger activated)
INFO:Enable nvidia
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/lenovo_thinkpad
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/put_your_quirks_here
DEBUG:Parsing /usr/share/ubuntu-drivers-common/quirks/dell_latitude
Loading new nvidia-460.32.03 DKMS files...
Building for 5.4.0-65-generic
Building for architecture x86_64
Building initial module for 5.4.0-65-generic
Done.

nvidia.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-65-generic/updates/dkms/

nvidia-modeset.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-65-generic/updates/dkms/

nvidia-drm.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-65-generic/updates/dkms/

nvidia-uvm.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/5.4.0-65-generic/updates/dkms/

depmod.......

DKMS: install completed.
Setting up libnvidia-encode-460:amd64 (460.32.03-0ubuntu0.20.04.1) ...
Setting up libnvidia-encode-460:i386 (460.32.03-0ubuntu0.20.04.1) ...
Setting up nvidia-driver-460 (460.32.03-0ubuntu0.20.04.1) ...
Processing triggers for desktop-file-utils (0.24-1ubuntu3) ...
Processing triggers for mime-support (3.64ubuntu1) ...
Processing triggers for gnome-menus (3.36.0-1ubuntu1) ...
Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
/sbin/ldconfig.real: /usr/local/cuda-8.0/targets/x86_64-linux/lib/libcudnn.so.5 is not a symbolic link

Processing triggers for man-db (2.9.1-1) ...
Processing triggers for bamfdaemon (0.5.3+18.04.20180207.2-0ubuntu2) ...
Rebuilding /usr/share/applications/bamf-2.index...
Processing triggers for initramfs-tools (0.136ubuntu6.3) ...
update-initramfs: Generating /boot/initrd.img-5.4.0-65-generic
cryptsetup: ERROR: Couldn't resolve device 
    UUID=998a1126-c156-4290-ba9e-6265633e7e3a
W: initramfs-tools configuration sets RESUME=UUID=998a1126-c156-4290-ba9e-6265633e7e3a
W: but no matching swap device is available.

$ sudo reboot

After reboot:

$ nvidia-smi
Thu Feb 11 00:59:07 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GT 640      Off  | 00000000:01:00.0 N/A |                  N/A |
| 40%   28C    P8    N/A /  N/A |    148MiB /  1992MiB |     N/A      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+


$ nvidia-smi -L
GPU 0: GeForce GT 640 (UUID: GPU-f2583df9-404d-2564-d332-e7878a94d087)

Friday, 15 January 2021

Python Package Management with pip

 pip -  Package Manager



$ pip

Usage:   
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible dependencies.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host as trusted, even though it does not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file containing the private key and the certificate in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index.

To check its version and for which Python version it has been installed:

pip -V
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)

To list packages installed via pip:

pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
change-case (0.5.2)
cryptography (2.1.4)
dnspython (1.15.0)
enum34 (1.1.6)
gyp (0.1)
idna (2.6)
ipaddress (1.0.17)
iso8601 (0.1.12)
JSONSchema2DB (1.0.1)
...


Installing packages


Let's see pip install help page first:

$ pip install --help

Usage:   
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

Description:
  Install packages from:
  
  - PyPI (and other indexes) using requirement specifiers.
  - VCS project urls.
  - Local project directories.
  - Local or remote source archives.
  
  pip also supports installing from "requirements files", which provide
  an easy way to specify a whole environment to be installed.

Install Options:
  -r, --requirement <file>    Install from the given requirements file. This
                              option can be used multiple times.
  -c, --constraint <file>     Constrain versions using the given constraints
                              file. This option can be used multiple times.
  --no-deps                   Don't install package dependencies.
  --pre                       Include pre-release and development versions. By
                              default, pip only finds stable versions.
  -e, --editable <path/url>   Install a project in editable mode (i.e.
                              setuptools "develop mode") from a local project
                              path or a VCS url.
  -t, --target <dir>          Install packages into <dir>. By default this
                              will not replace existing files/folders in
                              <dir>. Use --upgrade to replace existing
                              packages in <dir> with new versions.
  --platform <platform>       Only use wheels compatible with <platform>.
                              Defaults to the platform of the running system.
  --python-version <python_version>
                              The Python interpreter version to use for wheel
                              and "Requires-Python" compatibility checks.
                              Defaults to a version derived from the running
                              interpreter. The version can be specified using
                              up to three dot-separated integers (e.g. "3" for
                              3.0.0, "3.7" for 3.7.0, or "3.7.3"). A major-
                              minor version can also be given as a string
                              without dots (e.g. "37" for 3.7.0).
  --implementation <implementation>
                              Only use wheels compatible with Python
                              implementation <implementation>, e.g. 'pp',
                              'jy', 'cp',  or 'ip'. If not specified, then the
                              current interpreter implementation is used.  Use
                              'py' to force implementation-agnostic wheels.
  --abi <abi>                 Only use wheels compatible with Python abi
                              <abi>, e.g. 'pypy_41'.  If not specified, then
                              the current interpreter abi tag is used.
                              Generally you will need to specify
                              --implementation, --platform, and --python-
                              version when using this option.
  --user                      Install to the Python user install directory for
                              your platform. Typically ~/.local/, or
                              %APPDATA%\Python on Windows. (See the Python
                              documentation for site.USER_BASE for full
                              details.)  On Debian systems, this is the
                              default when running outside of a virtual
                              environment and not as root.
  --root <dir>                Install everything relative to this alternate
                              root directory.
  --prefix <dir>              Installation prefix where lib, bin and other
                              top-level folders are placed
  --system                    Install using the system scheme (overrides
                              --user on Debian systems)
  -b, --build <dir>           Directory to unpack packages into and build in.
                              Note that an initial build still takes place in
                              a temporary directory. The location of temporary
                              directories can be controlled by setting the
                              TMPDIR environment variable (TEMP on Windows)
                              appropriately. When passed, build directories
                              are not cleaned in case of failures.
  --src <dir>                 Directory to check out editable projects into.
                              The default in a virtualenv is "<venv
                              path>/src". The default for global installs is
                              "<current dir>/src".
  -U, --upgrade               Upgrade all specified packages to the newest
                              available version. The handling of dependencies
                              depends on the upgrade-strategy used.
  --upgrade-strategy <upgrade_strategy>
                              Determines how dependency upgrading should be
                              handled [default: only-if-needed]. "eager" -
                              dependencies are upgraded regardless of whether
                              the currently installed version satisfies the
                              requirements of the upgraded package(s). "only-
                              if-needed" -  are upgraded only when they do not
                              satisfy the requirements of the upgraded
                              package(s).
  --force-reinstall           Reinstall all packages even if they are already
                              up-to-date.
  -I, --ignore-installed      Ignore the installed packages, overwriting them.
                              This can break your system if the existing
                              package is of a different version or was
                              installed with a different package manager!
  --ignore-requires-python    Ignore the Requires-Python information.
  --no-build-isolation        Disable isolation when building a modern source
                              distribution. Build dependencies specified by
                              PEP 518 must be already installed if this option
                              is used.
  --use-pep517                Use PEP 517 for building source distributions
                              (use --no-use-pep517 to force legacy behaviour).
  --install-option <options>  Extra arguments to be supplied to the setup.py
                              install command (use like --install-option="--
                              install-scripts=/usr/local/bin"). Use multiple
                              --install-option options to pass multiple
                              options to setup.py install. If you are using an
                              option with a directory path, be sure to use
                              absolute path.
  --global-option <options>   Extra global options to be supplied to the
                              setup.py call before the install command.
  --compile                   Compile Python source files to bytecode
  --no-compile                Do not compile Python source files to bytecode
  --no-warn-script-location   Do not warn when installing scripts outside PATH
  --no-warn-conflicts         Do not warn about broken dependencies
  --no-binary <format_control>
                              Do not use binary packages. Can be supplied
                              multiple times, and each time adds to the
                              existing value. Accepts either :all: to disable
                              all binary packages, :none: to empty the set, or
                              one or more package names with commas between
                              them (no colons). Note that some packages are
                              tricky to compile and may fail to install when
                              this option is used on them.
  --only-binary <format_control>
                              Do not use source packages. Can be supplied
                              multiple times, and each time adds to the
                              existing value. Accepts either :all: to disable
                              all source packages, :none: to empty the set, or
                              one or more package names with commas between
                              them. Packages without binary distributions will
                              fail to install when this option is used on
                              them.
  --prefer-binary             Prefer older binary packages over newer source
                              packages.
  --no-clean                  Don't clean up build directories.
  --require-hashes            Require a hash to check each requirement
                              against, for repeatable installs. This option is
                              implied when any package in a requirements file
                              has a --hash option.
  --progress-bar <progress_bar>
                              Specify type of progress to be displayed
                              [off|on|ascii|pretty|emoji] (default: on)

Package Index Options:
  -i, --index-url <url>       Base URL of the Python Package Index (default
                              https://pypi.org/simple). This should point to a
                              repository compliant with PEP 503 (the simple
                              repository API) or a local directory laid out in
                              the same format.
  --extra-index-url <url>     Extra URLs of package indexes to use in addition
                              to --index-url. Should follow the same rules as
                              --index-url.
  --no-index                  Ignore package index (only looking at --find-
                              links URLs instead).
  -f, --find-links <url>      If a url or path to an html file, then parse for
                              links to archives. If a local path or file://
                              url that's a directory, then look for archives
                              in the directory listing.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring
                              environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be
                              used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be
                              used up to 3 times (corresponding to WARNING,
                              ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form
                              [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should
                              attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists:
                              (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host or host:port pair as trusted,
                              even though it does not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file
                              containing the private key and the certificate
                              in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine
                              whether a new version of pip is available for
                              download. Implied with --no-index.
  --no-color                  Suppress colored output
  --no-python-version-warning
                              Silence deprecation warnings for upcoming
                              unsupported Pythons.



To install some package via pip:

pip install jsonschema2db


To install some specific version of the package:

$ pip3 install psycopg2==2.7.6


To list properties of the installed package (including its location on disk!) use pip show:

pip show jsonschema2db
Name: JSONSchema2DB
Version: 1.0.1
Summary: Generate database tables from JSON schema
Home-page: https://better.engineering/convoys
Author: Erik Bernhardsson
Author-email: erikbern@better.com
License: MIT
Location: /home/bojan/.local/lib/python2.7/site-packages
Requires: iso8601, change-case, psycopg2

pip - How do I install a Python package with a .whl file? - Stack Overflow

pip install --user  /path/to/package.whl


Globally installed packages are located in:

/home/user/.local/lib/python3.6/site-packages/


Uninstalling Packages


To uninstall a package, use:

pip uninstall JSONSchema2DB
Uninstalling JSONSchema2DB-1.0.1:
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/DESCRIPTION.rst
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/INSTALLER
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/METADATA
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/RECORD
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/WHEEL
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/metadata.json
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/top_level.txt
  /home/bojan/.local/lib/python2.7/site-packages/jsonschema2db.py
Proceed (y/n)? y
  Successfully uninstalled JSONSchema2DB-1.0.1

Example how to uninstall globally installed package:

pip uninstall psycopg2
Uninstalling psycopg2-2.7.2:
  Would remove:
    /home/bojan/.local/lib/python3.6/site-packages/psycopg2-2.7.2.dist-info/*
    /home/bojan/.local/lib/python3.6/site-packages/psycopg2/*
Proceed (y/n)? y
  Successfully uninstalled psycopg2-2.7.2

Package Management in Virtual Environments




Example:

(my_env)$ pip freeze
asn1crypto==0.24.0
change-case==0.5.2
cryptography==2.1.4
dnspython==1.15.0
enum34==1.1.6
...

To save this info in requirements.txt (this file can have an arbitrary name):

(my_env)$ pip freeze > requirements.txt

Once we have requirements file, we can recreate this environment anywhere else by installing all the packages listed there:

(my_env2) $ pip3 install -r requirements.txt --verbose


Example where virtual env is named venv_basic and requirements.txt contains a single package, python-dotenv:

$ pip3 install -r requirements_basic.txt --verbose
Using pip 20.3.3 from /home/bojan/dev/github/python-demo/venv_basic/lib/python3.8/site-packages/pip (python 3.8)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-9xisk4la
Created temporary directory: /tmp/pip-req-tracker-r1kgbsxz
Initialized build tracking at /tmp/pip-req-tracker-r1kgbsxz
Created build tracker: /tmp/pip-req-tracker-r1kgbsxz
Entered build tracker: /tmp/pip-req-tracker-r1kgbsxz
Created temporary directory: /tmp/pip-install-vqtt3gqw
1 location(s) to search for versions of python-dotenv:
* https://pypi.org/simple/python-dotenv/
Fetching project page and analyzing links: https://pypi.org/simple/python-dotenv/
Getting page https://pypi.org/simple/python-dotenv/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/python-dotenv/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/python-dotenv/ HTTP/1.1" 200 5997
Updating cache with response from "https://pypi.org/simple/python-dotenv/"
Caching due to etag
  Found link https://files.pythonhosted.org/packages/ad/3b/16a207ced20a1be0cc2ec5a53ab68ce1976ad643fe7f135d3addd763abf0/python-dotenv-0.1.0.tar.gz#sha256=26156022355d4457bc9062bcead7fde4824623d4463488b623bdd5ce912462fa (from https://pypi.org/simple/python-dotenv/), version: 0.1.0
  Found link https://files.pythonhosted.org/packages/e6/95/e9239c25d2c06c4a262360375d4e0c61b243d1c763d647137aca4d7df9fe/python-dotenv-0.1.2.tar.gz#sha256=aaefda0e975b702374f7c86edcf948590bd4f47737b33027de1bc72afbed745b (from https://pypi.org/simple/python-dotenv/), version: 0.1.2
  ...
  ...
  ... 
https://files.pythonhosted.org/packages/53/04/1a8126516c8febfeb2015844edee977c9b783bdff9b3bcd89b1cc2e1f372/python-dotenv-0.15.0.tar.gz#sha256=587825ed60b1711daea4832cf37524dfd404325b7db5e25ebe88c495c9f807a0 (from https://pypi.org/simple/python-dotenv/), version: 0.15.0
  Found link https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl#sha256=0c8d1b80d1a1e91717ea7d526178e3882732420b03f08afea0406db6402e220e (from https://pypi.org/simple/python-dotenv/), version: 0.15.0
Given no hashes to check 64 links for project 'python-dotenv': discarding no candidates
Collecting python-dotenv
  Created temporary directory: /tmp/pip-unpack-gp764zed
  Looking up "https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl HTTP/1.1" 200 18253
  Downloading python_dotenv-0.15.0-py2.py3-none-any.whl (18 kB)
  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl"
  Caching due to etag
  Added python-dotenv from https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl#sha256=0c8d1b80d1a1e91717ea7d526178e3882732420b03f08afea0406db6402e220e (from -r requirements_basic.txt (line 1)) to build tracker '/tmp/pip-req-tracker-r1kgbsxz'
  Removed python-dotenv from https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl#sha256=0c8d1b80d1a1e91717ea7d526178e3882732420b03f08afea0406db6402e220e (from -r requirements_basic.txt (line 1)) from build tracker '/tmp/pip-req-tracker-r1kgbsxz'
Created temporary directory: /tmp/pip-unpack-bhhhx138
Installing collected packages: python-dotenv

  changing mode of /home/bojan/dev/github/python-demo/venv_basic/bin/dotenv to 775
Successfully installed python-dotenv-0.15.0
Removed build tracker: '/tmp/pip-req-tracker-r1kgbsxz'


This example also shown that pip installs the most recent version of the package if package version is not specified in requirements.txt.

To check that packages from requirements file are indeed installed we can use:

(my_env2) $ pip3 list
Package       Version
------------- -------
pip           20.3.3
python-dotenv 0.15.0
setuptools    51.1.2
wheel         0.36.2



If for some reason installation from requirements.txt fails, you can always try to manually install packages, one by one:

(my_env2)$ pip3 install python-dotenv


We can only list names of packages in this file but sometimes it is necessary to specify their versions as otherwise we might get pip error like this:

ERROR: jsonschema2db 1.0.1 has requirement psycopg2==2.7.2, but you'll have psycopg2 2.8.2 which is incompatible.


Package Management in Dockerfile


Here is an example how to use previously described approach when we need to install packages in Docker image:

Dockerfile before requirements.txt:

RUN python3 -m pip install psycopg2
RUN python3 -m pip install jsonschema2db

requirements.txt:

JSONSchema2DB==1.0.1
psycopg2==2.7.2

Dockerfile after requirements.txt:

RUN python3 -m pip install -r requirements.txt


pip3


Pip3 is the Python3 version of pip. If you just use pip, then only the python2.7 version will be installed. You have to use pip3 for it to be installed on Python3. [source]

If pip3 is not installed:

$ python3 -m pip install JSONSchema2DB
/usr/bin/python3: No module named pip

$ pip3
Command 'pip3' not found, but can be installed with:
sudo apt install python3-pip

Let's install it:

$ sudo apt install python3-pip
...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up python3.6-dev (3.6.7-1~18.04) ...
Setting up python3-lib2to3 (3.6.7-1~18.04) ...
Setting up python3-distutils (3.6.7-1~18.04) ...
Setting up libpython3-dev:amd64 (3.6.7-1~18.04) ...
Setting up python3-pip (9.0.1-2.3~ubuntu1) ...
Setting up python3-setuptools (39.0.1-2) ...
Setting up dh-python (3.20180325ubuntu2) ...
Setting up python3-dev (3.6.7-1~18.04) ...

Now we have:

$ pip -V
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
$ pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

We can now use it to install some package for python3.


$ python3 -m pip install JSONSchema2DB
Collecting JSONSchema2DB
  Using cached 
...
Installing collected packages: change-case, iso8601, psycopg2, JSONSchema2DB
Successfully installed JSONSchema2DB-1.0.1 change-case-0.5.2 iso8601-0.1.12 psycopg2-2.7.2

Let's verify it is installed for Python3:

$ pip3 show JSONSchema2DB
Name: JSONSchema2DB
Version: 1.0.1
Summary: Generate database tables from JSON schema
Home-page: https://better.engineering/convoys
Author: Erik Bernhardsson
Author-email: erikbern@better.com
License: MIT
Location: /home/bojan/.local/lib/python3.6/site-packages
Requires: change-case, iso8601, psycopg2