Friday 15 January 2021

Python Package Management with pip

 pip -  Package Manager



$ pip

Usage:   
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible dependencies.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host as trusted, even though it does not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file containing the private key and the certificate in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index.

To check its version and for which Python version it has been installed:

pip -V
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)

To list packages installed via pip:

pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
change-case (0.5.2)
cryptography (2.1.4)
dnspython (1.15.0)
enum34 (1.1.6)
gyp (0.1)
idna (2.6)
ipaddress (1.0.17)
iso8601 (0.1.12)
JSONSchema2DB (1.0.1)
...


Installing packages


Let's see pip install help page first:

$ pip install --help

Usage:   
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

Description:
  Install packages from:
  
  - PyPI (and other indexes) using requirement specifiers.
  - VCS project urls.
  - Local project directories.
  - Local or remote source archives.
  
  pip also supports installing from "requirements files", which provide
  an easy way to specify a whole environment to be installed.

Install Options:
  -r, --requirement <file>    Install from the given requirements file. This
                              option can be used multiple times.
  -c, --constraint <file>     Constrain versions using the given constraints
                              file. This option can be used multiple times.
  --no-deps                   Don't install package dependencies.
  --pre                       Include pre-release and development versions. By
                              default, pip only finds stable versions.
  -e, --editable <path/url>   Install a project in editable mode (i.e.
                              setuptools "develop mode") from a local project
                              path or a VCS url.
  -t, --target <dir>          Install packages into <dir>. By default this
                              will not replace existing files/folders in
                              <dir>. Use --upgrade to replace existing
                              packages in <dir> with new versions.
  --platform <platform>       Only use wheels compatible with <platform>.
                              Defaults to the platform of the running system.
  --python-version <python_version>
                              The Python interpreter version to use for wheel
                              and "Requires-Python" compatibility checks.
                              Defaults to a version derived from the running
                              interpreter. The version can be specified using
                              up to three dot-separated integers (e.g. "3" for
                              3.0.0, "3.7" for 3.7.0, or "3.7.3"). A major-
                              minor version can also be given as a string
                              without dots (e.g. "37" for 3.7.0).
  --implementation <implementation>
                              Only use wheels compatible with Python
                              implementation <implementation>, e.g. 'pp',
                              'jy', 'cp',  or 'ip'. If not specified, then the
                              current interpreter implementation is used.  Use
                              'py' to force implementation-agnostic wheels.
  --abi <abi>                 Only use wheels compatible with Python abi
                              <abi>, e.g. 'pypy_41'.  If not specified, then
                              the current interpreter abi tag is used.
                              Generally you will need to specify
                              --implementation, --platform, and --python-
                              version when using this option.
  --user                      Install to the Python user install directory for
                              your platform. Typically ~/.local/, or
                              %APPDATA%\Python on Windows. (See the Python
                              documentation for site.USER_BASE for full
                              details.)  On Debian systems, this is the
                              default when running outside of a virtual
                              environment and not as root.
  --root <dir>                Install everything relative to this alternate
                              root directory.
  --prefix <dir>              Installation prefix where lib, bin and other
                              top-level folders are placed
  --system                    Install using the system scheme (overrides
                              --user on Debian systems)
  -b, --build <dir>           Directory to unpack packages into and build in.
                              Note that an initial build still takes place in
                              a temporary directory. The location of temporary
                              directories can be controlled by setting the
                              TMPDIR environment variable (TEMP on Windows)
                              appropriately. When passed, build directories
                              are not cleaned in case of failures.
  --src <dir>                 Directory to check out editable projects into.
                              The default in a virtualenv is "<venv
                              path>/src". The default for global installs is
                              "<current dir>/src".
  -U, --upgrade               Upgrade all specified packages to the newest
                              available version. The handling of dependencies
                              depends on the upgrade-strategy used.
  --upgrade-strategy <upgrade_strategy>
                              Determines how dependency upgrading should be
                              handled [default: only-if-needed]. "eager" -
                              dependencies are upgraded regardless of whether
                              the currently installed version satisfies the
                              requirements of the upgraded package(s). "only-
                              if-needed" -  are upgraded only when they do not
                              satisfy the requirements of the upgraded
                              package(s).
  --force-reinstall           Reinstall all packages even if they are already
                              up-to-date.
  -I, --ignore-installed      Ignore the installed packages, overwriting them.
                              This can break your system if the existing
                              package is of a different version or was
                              installed with a different package manager!
  --ignore-requires-python    Ignore the Requires-Python information.
  --no-build-isolation        Disable isolation when building a modern source
                              distribution. Build dependencies specified by
                              PEP 518 must be already installed if this option
                              is used.
  --use-pep517                Use PEP 517 for building source distributions
                              (use --no-use-pep517 to force legacy behaviour).
  --install-option <options>  Extra arguments to be supplied to the setup.py
                              install command (use like --install-option="--
                              install-scripts=/usr/local/bin"). Use multiple
                              --install-option options to pass multiple
                              options to setup.py install. If you are using an
                              option with a directory path, be sure to use
                              absolute path.
  --global-option <options>   Extra global options to be supplied to the
                              setup.py call before the install command.
  --compile                   Compile Python source files to bytecode
  --no-compile                Do not compile Python source files to bytecode
  --no-warn-script-location   Do not warn when installing scripts outside PATH
  --no-warn-conflicts         Do not warn about broken dependencies
  --no-binary <format_control>
                              Do not use binary packages. Can be supplied
                              multiple times, and each time adds to the
                              existing value. Accepts either :all: to disable
                              all binary packages, :none: to empty the set, or
                              one or more package names with commas between
                              them (no colons). Note that some packages are
                              tricky to compile and may fail to install when
                              this option is used on them.
  --only-binary <format_control>
                              Do not use source packages. Can be supplied
                              multiple times, and each time adds to the
                              existing value. Accepts either :all: to disable
                              all source packages, :none: to empty the set, or
                              one or more package names with commas between
                              them. Packages without binary distributions will
                              fail to install when this option is used on
                              them.
  --prefer-binary             Prefer older binary packages over newer source
                              packages.
  --no-clean                  Don't clean up build directories.
  --require-hashes            Require a hash to check each requirement
                              against, for repeatable installs. This option is
                              implied when any package in a requirements file
                              has a --hash option.
  --progress-bar <progress_bar>
                              Specify type of progress to be displayed
                              [off|on|ascii|pretty|emoji] (default: on)

Package Index Options:
  -i, --index-url <url>       Base URL of the Python Package Index (default
                              https://pypi.org/simple). This should point to a
                              repository compliant with PEP 503 (the simple
                              repository API) or a local directory laid out in
                              the same format.
  --extra-index-url <url>     Extra URLs of package indexes to use in addition
                              to --index-url. Should follow the same rules as
                              --index-url.
  --no-index                  Ignore package index (only looking at --find-
                              links URLs instead).
  -f, --find-links <url>      If a url or path to an html file, then parse for
                              links to archives. If a local path or file://
                              url that's a directory, then look for archives
                              in the directory listing.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring
                              environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be
                              used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be
                              used up to 3 times (corresponding to WARNING,
                              ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form
                              [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should
                              attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists:
                              (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host or host:port pair as trusted,
                              even though it does not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file
                              containing the private key and the certificate
                              in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine
                              whether a new version of pip is available for
                              download. Implied with --no-index.
  --no-color                  Suppress colored output
  --no-python-version-warning
                              Silence deprecation warnings for upcoming
                              unsupported Pythons.



To install some package via pip:

pip install jsonschema2db


To install some specific version of the package:

$ pip3 install psycopg2==2.7.6


To list properties of the installed package (including its location on disk!) use pip show:

pip show jsonschema2db
Name: JSONSchema2DB
Version: 1.0.1
Summary: Generate database tables from JSON schema
Home-page: https://better.engineering/convoys
Author: Erik Bernhardsson
Author-email: erikbern@better.com
License: MIT
Location: /home/bojan/.local/lib/python2.7/site-packages
Requires: iso8601, change-case, psycopg2

pip - How do I install a Python package with a .whl file? - Stack Overflow

pip install --user  /path/to/package.whl


Globally installed packages are located in:

/home/user/.local/lib/python3.6/site-packages/


Uninstalling Packages


To uninstall a package, use:

pip uninstall JSONSchema2DB
Uninstalling JSONSchema2DB-1.0.1:
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/DESCRIPTION.rst
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/INSTALLER
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/METADATA
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/RECORD
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/WHEEL
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/metadata.json
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/top_level.txt
  /home/bojan/.local/lib/python2.7/site-packages/jsonschema2db.py
Proceed (y/n)? y
  Successfully uninstalled JSONSchema2DB-1.0.1

Example how to uninstall globally installed package:

pip uninstall psycopg2
Uninstalling psycopg2-2.7.2:
  Would remove:
    /home/bojan/.local/lib/python3.6/site-packages/psycopg2-2.7.2.dist-info/*
    /home/bojan/.local/lib/python3.6/site-packages/psycopg2/*
Proceed (y/n)? y
  Successfully uninstalled psycopg2-2.7.2

Package Management in Virtual Environments




Example:

(my_env)$ pip freeze
asn1crypto==0.24.0
change-case==0.5.2
cryptography==2.1.4
dnspython==1.15.0
enum34==1.1.6
...

To save this info in requirements.txt (this file can have an arbitrary name):

(my_env)$ pip freeze > requirements.txt

Once we have requirements file, we can recreate this environment anywhere else by installing all the packages listed there:

(my_env2) $ pip3 install -r requirements.txt --verbose


Example where virtual env is named venv_basic and requirements.txt contains a single package, python-dotenv:

$ pip3 install -r requirements_basic.txt --verbose
Using pip 20.3.3 from /home/bojan/dev/github/python-demo/venv_basic/lib/python3.8/site-packages/pip (python 3.8)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-9xisk4la
Created temporary directory: /tmp/pip-req-tracker-r1kgbsxz
Initialized build tracking at /tmp/pip-req-tracker-r1kgbsxz
Created build tracker: /tmp/pip-req-tracker-r1kgbsxz
Entered build tracker: /tmp/pip-req-tracker-r1kgbsxz
Created temporary directory: /tmp/pip-install-vqtt3gqw
1 location(s) to search for versions of python-dotenv:
* https://pypi.org/simple/python-dotenv/
Fetching project page and analyzing links: https://pypi.org/simple/python-dotenv/
Getting page https://pypi.org/simple/python-dotenv/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/python-dotenv/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/python-dotenv/ HTTP/1.1" 200 5997
Updating cache with response from "https://pypi.org/simple/python-dotenv/"
Caching due to etag
  Found link https://files.pythonhosted.org/packages/ad/3b/16a207ced20a1be0cc2ec5a53ab68ce1976ad643fe7f135d3addd763abf0/python-dotenv-0.1.0.tar.gz#sha256=26156022355d4457bc9062bcead7fde4824623d4463488b623bdd5ce912462fa (from https://pypi.org/simple/python-dotenv/), version: 0.1.0
  Found link https://files.pythonhosted.org/packages/e6/95/e9239c25d2c06c4a262360375d4e0c61b243d1c763d647137aca4d7df9fe/python-dotenv-0.1.2.tar.gz#sha256=aaefda0e975b702374f7c86edcf948590bd4f47737b33027de1bc72afbed745b (from https://pypi.org/simple/python-dotenv/), version: 0.1.2
  ...
  ...
  ... 
https://files.pythonhosted.org/packages/53/04/1a8126516c8febfeb2015844edee977c9b783bdff9b3bcd89b1cc2e1f372/python-dotenv-0.15.0.tar.gz#sha256=587825ed60b1711daea4832cf37524dfd404325b7db5e25ebe88c495c9f807a0 (from https://pypi.org/simple/python-dotenv/), version: 0.15.0
  Found link https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl#sha256=0c8d1b80d1a1e91717ea7d526178e3882732420b03f08afea0406db6402e220e (from https://pypi.org/simple/python-dotenv/), version: 0.15.0
Given no hashes to check 64 links for project 'python-dotenv': discarding no candidates
Collecting python-dotenv
  Created temporary directory: /tmp/pip-unpack-gp764zed
  Looking up "https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl HTTP/1.1" 200 18253
  Downloading python_dotenv-0.15.0-py2.py3-none-any.whl (18 kB)
  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl"
  Caching due to etag
  Added python-dotenv from https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl#sha256=0c8d1b80d1a1e91717ea7d526178e3882732420b03f08afea0406db6402e220e (from -r requirements_basic.txt (line 1)) to build tracker '/tmp/pip-req-tracker-r1kgbsxz'
  Removed python-dotenv from https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl#sha256=0c8d1b80d1a1e91717ea7d526178e3882732420b03f08afea0406db6402e220e (from -r requirements_basic.txt (line 1)) from build tracker '/tmp/pip-req-tracker-r1kgbsxz'
Created temporary directory: /tmp/pip-unpack-bhhhx138
Installing collected packages: python-dotenv

  changing mode of /home/bojan/dev/github/python-demo/venv_basic/bin/dotenv to 775
Successfully installed python-dotenv-0.15.0
Removed build tracker: '/tmp/pip-req-tracker-r1kgbsxz'


This example also shown that pip installs the most recent version of the package if package version is not specified in requirements.txt.

To check that packages from requirements file are indeed installed we can use:

(my_env2) $ pip3 list
Package       Version
------------- -------
pip           20.3.3
python-dotenv 0.15.0
setuptools    51.1.2
wheel         0.36.2



If for some reason installation from requirements.txt fails, you can always try to manually install packages, one by one:

(my_env2)$ pip3 install python-dotenv


We can only list names of packages in this file but sometimes it is necessary to specify their versions as otherwise we might get pip error like this:

ERROR: jsonschema2db 1.0.1 has requirement psycopg2==2.7.2, but you'll have psycopg2 2.8.2 which is incompatible.


Package Management in Dockerfile


Here is an example how to use previously described approach when we need to install packages in Docker image:

Dockerfile before requirements.txt:

RUN python3 -m pip install psycopg2
RUN python3 -m pip install jsonschema2db

requirements.txt:

JSONSchema2DB==1.0.1
psycopg2==2.7.2

Dockerfile after requirements.txt:

RUN python3 -m pip install -r requirements.txt


pip3


Pip3 is the Python3 version of pip. If you just use pip, then only the python2.7 version will be installed. You have to use pip3 for it to be installed on Python3. [source]

If pip3 is not installed:

$ python3 -m pip install JSONSchema2DB
/usr/bin/python3: No module named pip

$ pip3
Command 'pip3' not found, but can be installed with:
sudo apt install python3-pip

Let's install it:

$ sudo apt install python3-pip
...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up python3.6-dev (3.6.7-1~18.04) ...
Setting up python3-lib2to3 (3.6.7-1~18.04) ...
Setting up python3-distutils (3.6.7-1~18.04) ...
Setting up libpython3-dev:amd64 (3.6.7-1~18.04) ...
Setting up python3-pip (9.0.1-2.3~ubuntu1) ...
Setting up python3-setuptools (39.0.1-2) ...
Setting up dh-python (3.20180325ubuntu2) ...
Setting up python3-dev (3.6.7-1~18.04) ...

Now we have:

$ pip -V
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
$ pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

We can now use it to install some package for python3.


$ python3 -m pip install JSONSchema2DB
Collecting JSONSchema2DB
  Using cached 
...
Installing collected packages: change-case, iso8601, psycopg2, JSONSchema2DB
Successfully installed JSONSchema2DB-1.0.1 change-case-0.5.2 iso8601-0.1.12 psycopg2-2.7.2

Let's verify it is installed for Python3:

$ pip3 show JSONSchema2DB
Name: JSONSchema2DB
Version: 1.0.1
Summary: Generate database tables from JSON schema
Home-page: https://better.engineering/convoys
Author: Erik Bernhardsson
Author-email: erikbern@better.com
License: MIT
Location: /home/bojan/.local/lib/python3.6/site-packages
Requires: change-case, iso8601, psycopg2

Using Python Virtual Environments

Different Python projects depend on different packages. To keep them separated and avoid polluting the global space with package installations we can use the concept of virtual environments. There are several ways to work with virtual environments in Python:
  • virtualenv
  • virtualenvwrapper
  • Conda (part of Anaconda)

virtualenv



To use virtualenv we first need to install all necessary components:

$ sudo apt update
$ sudo apt install python3-dev python3-pip
$ sudo pip3 install -U virtualenv  # system-wide install

Verification:

$ python3 --version
$ pip3 --version
$ virtualenv --version

or

$ virtualenv
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--read-only-app-data] [--app-data APP_DATA] [--reset-app-data] [--upgrade-embed-wheels] [--discovery {builtin}] [-p py] [--try-first-with py_exe]
                  [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_sep_list] [--clear] [--no-vcs-ignore] [--system-site-packages] [--symlinks | --copies] [--no-download | --download]
                  [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel] [--no-periodic-update] [--symlink-app-data] [--prompt prompt] [-h]
                  dest
virtualenv: error: the following arguments are required: dest
SystemExit: 2

or

$ virtualenv --help
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--read-only-app-data] [--app-data APP_DATA] [--reset-app-data] [--upgrade-embed-wheels] [--discovery {builtin}] [-p py] [--try-first-with py_exe]
                  [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_sep_list] [--clear] [--no-vcs-ignore] [--system-site-packages] [--symlinks | --copies] [--no-download | --download]
                  [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel] [--no-periodic-update] [--symlink-app-data] [--prompt prompt] [-h]
                  dest

optional arguments:
  --version                     display the version of the virtualenv package and its location, then exit
  --with-traceback              on failure also display the stacktrace internals of virtualenv (default: False)
  --read-only-app-data          use app data folder in read-only mode (write operations will fail with error) (default: False)
  --app-data APP_DATA           a data folder used as cache by the virtualenv (default: /home/bojan/.local/share/virtualenv)
  --reset-app-data              start with empty app data folder (default: False)
  --upgrade-embed-wheels        trigger a manual update of the embedded wheels (default: False)
  -h, --help                    show this help message and exit

verbosity:
  verbosity = verbose - quiet, default INFO, mapping => CRITICAL=0, ERROR=1, WARNING=2, INFO=3, DEBUG=4, NOTSET=5

  -v, --verbose                 increase verbosity (default: 2)
  -q, --quiet                   decrease verbosity (default: 0)

discovery:
  discover and provide a target interpreter

  --discovery {builtin}         interpreter discovery method (default: builtin)
  -p py, --python py            interpreter based on what to create environment (path/identifier) - by default use the interpreter where the tool is installed - first found wins (default: [])
  --try-first-with py_exe       try first these interpreters before starting the discovery (default: [])

creator:
  options for creator builtin

  --creator {builtin,cpython3-posix,venv}
                                create environment via (builtin = cpython3-posix) (default: builtin)
  dest                          directory to create virtualenv at
  --clear                       remove the destination directory if exist before starting (will overwrite files otherwise) (default: False)
  --no-vcs-ignore               don't create VCS ignore directive in the destination directory (default: False)
  --system-site-packages        give the virtual environment access to the system site-packages dir (default: False)
  --symlinks                    try to use symlinks rather than copies, when symlinks are not the default for the platform (default: True)
  --copies, --always-copy       try to use copies rather than symlinks, even when symlinks are the default for the platform (default: False)

seeder:
  options for seeder app-data

  --seeder {app-data,pip}       seed packages install method (default: app-data)
  --no-seed, --without-pip      do not install seed packages (default: False)
  --no-download, --never-download
                                pass to disable download of the latest pip/setuptools/wheel from PyPI (default: True)
  --download                    pass to enable download of the latest pip/setuptools/wheel from PyPI (default: False)
  --extra-search-dir d [d ...]  a path containing wheels to extend the internal wheel list (can be set 1+ times) (default: [])
  --pip version                 version of pip to install as seed: embed, bundle or exact version (default: bundle)
  --setuptools version          version of setuptools to install as seed: embed, bundle or exact version (default: bundle)
  --wheel version               version of wheel to install as seed: embed, bundle or exact version (default: bundle)
  --no-pip                      do not install pip (default: False)
  --no-setuptools               do not install setuptools (default: False)
  --no-wheel                    do not install wheel (default: False)
  --no-periodic-update          disable the periodic (once every 14 days) update of the embedded wheels (default: False)
  --symlink-app-data            symlink the python packages from the app-data folder (requires seed pip>=19.3) (default: False)

activators:
  options for activation scripts

  --activators comma_sep_list   activators to generate - default is all supported (default: bash,cshell,fish,powershell,python,xonsh)
  --prompt prompt               provides an alternative prompt prefix for this environment (default: None)

config file /home/bojan/.config/virtualenv/virtualenv.ini missing (change via env var VIRTUALENV_CONFIG_FILE)


Create virtual environment (we chose to use python3 as Python interpreter and venv as the directory where to store virtual environment):

virtualenv -p python3 ./venv_dir_name

or

virtualenv --system-site-packages -p python3 ./venv_dir_name


If you build with virtualenv --system-site-packages ENV, your virtual environment will inherit packages from /usr/lib/python2.7/site-packages (or wherever your global site-packages directory is).

This can be used if you have control over the global site-packages directory, and you want to depend on the packages there. If you want isolation from the global system, do not use this flag. BK: This flag should be avoided as it defeats the entire point of virtualenvs (which is why --no-site-packages was made the default).

Example:

$ virtualenv -p python3 ./venv
created virtual environment CPython3.8.5.final.0-64 in 253ms
  creator CPython3Posix(dest=/home/bojan/dev/github/python-demo/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/bojan/.local/share/virtualenv)
    added seed packages: pip==20.3.3, setuptools==51.1.2, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator


I once experienced this error when tried to create a virtual environment:

$ virtualenv -p python3 ./venv
Traceback (most recent call last):
  File "/usr/local/bin/virtualenv", line 5, in <module>
    from virtualenv.__main__ import run_with_catch
  File "/usr/local/lib/python3.8/dist-packages/virtualenv/__init__.py", line 3, in <module>
    from .run import cli_run, session_via_cli
  File "/usr/local/lib/python3.8/dist-packages/virtualenv/run/__init__.py", line 6, in <module>
    from ..app_data import make_app_data
  File "/usr/local/lib/python3.8/dist-packages/virtualenv/app_data/__init__.py", line 9, in <module>
    from appdirs import user_data_dir
ModuleNotFoundError: No module named 'appdirs'

I then installed the missing module via pip but this showed another error:

$ sudo pip3 install appdirs
Collecting appdirs
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
ERROR: virtualenv 20.3.1 requires filelock<4,>=3.0.0, which is not installed.
Installing collected packages: appdirs
Successfully installed appdirs-1.4.4

I resolved this by reinstalling the virtualenv:

$ sudo pip3 install virtualenv
Requirement already satisfied: virtualenv in /usr/local/lib/python3.8/dist-packages (20.3.1)
Requirement already satisfied: appdirs<2,>=1.4.3 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (1.4.4)
Requirement already satisfied: six<2,>=1.9.0 in /usr/lib/python3/dist-packages (from virtualenv) (1.14.0)
Requirement already satisfied: distlib<1,>=0.3.1 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (0.3.1)
Collecting filelock<4,>=3.0.0
  Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Installing collected packages: filelock
Successfully installed filelock-3.0.12

Once virtual environment is created, we'll have venv/bin/activate script created. Let's explore it:

$ cat venv/bin/activate
...

deactivate() {
...
}
...
VIRTUAL_ENV="/home/bojan/dev/github/tensorflow-demo/venv"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
...
pydoc () {
    python -m pydoc "$@"
}
...


We can see that we'll now have a new environment variable set - VIRTUAL_ENV and also a deactivate commmand.

To activate virtual environment:

$ source ./venv/bin/activate

We can also use a dot command:

$ . ./venv/bin/activate

When virtual environment is active, your shell prompt is prefixed with its name in form: (venv).

(venv) $ which pip
/home/bojan/dev/github/python-demo/venv/bin/pip

(venv) $ which pip3
/home/bojan/dev/github/python-demo/venv/bin/pip3


If we've saved requirements.txt, we can install all dependencies now with on of the following commands (see Python Package Management with pip | My Public Notepad):

$ pip install -r requirements.txt
pip3 install -r requirements.txt

It is possible to install a single new package in the virtual environment:

$ pip install package_name


Example:

(venv) $ pip3 install psycopg2
Collecting psycopg2
  Using cached psycopg2-2.8.6-cp38-cp38-linux_x86_64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.8.6

To verify installation path:

(venv) $ pip3 show psycopg2
Name: psycopg2
Version: 2.8.6
Summary: psycopg2 - Python-PostgreSQL Database Adapter
Home-page: https://psycopg.org/
Author: Federico Di Gregorio
Author-email: fog@initd.org
License: LGPL with exceptions
Location: /home/user/dev/my-app/venv/lib/python3.8/site-packages
Requires: 
Required-by: 

To check the version of the package installed in virtual environment:

(venv) $ pip3 list | grep psycopg
psycopg2          2.8.6


To exit virtual environment:

(venv) $ deactivate

How to rename virtual environment?

directory - How to rename a virtualenv in Python? - Stack Overflow

From inside active virtual environment:

$ pip freeze > requirements.txt
$ deactivate

From python - Pip freeze vs. pip list - Stack Overflow
pip list shows ALL packages. 
pip freeze shows packages YOU installed via pip (or pipenv if using that tool) command in a requirements format.
Also, be aware of `$ pip freeze > requirements.txt` considered harmful

Then delete old environment directory. E.g.

$ rm -r ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.14.0/

...and create a new one with correct name:

$ python3 -m virtualenv -p python3 ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/nvidia/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/python3
Also creating executable in /home/nvidia/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

Activate it and install all packages from the saved list:

$ source ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/activate
$ pip install -r requirements.txt
$ pip3 install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v43 tensorflow-gpu==1.15.0+nv19.12


virtualenvwrapper


TBD...


Conda



References