Friday 15 January 2021

Using Python Virtual Environments

Different Python projects depend on different packages. To keep them separated and avoid polluting the global space with package installations we can use the concept of virtual environments. There are several ways to work with virtual environments in Python:
  • virtualenv
  • virtualenvwrapper
  • Conda (part of Anaconda)

virtualenv



To use virtualenv we first need to install all necessary components:

$ sudo apt update
$ sudo apt install python3-dev python3-pip
$ sudo pip3 install -U virtualenv  # system-wide install

Verification:

$ python3 --version
$ pip3 --version
$ virtualenv --version

or

$ virtualenv
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--read-only-app-data] [--app-data APP_DATA] [--reset-app-data] [--upgrade-embed-wheels] [--discovery {builtin}] [-p py] [--try-first-with py_exe]
                  [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_sep_list] [--clear] [--no-vcs-ignore] [--system-site-packages] [--symlinks | --copies] [--no-download | --download]
                  [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel] [--no-periodic-update] [--symlink-app-data] [--prompt prompt] [-h]
                  dest
virtualenv: error: the following arguments are required: dest
SystemExit: 2

or

$ virtualenv --help
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--read-only-app-data] [--app-data APP_DATA] [--reset-app-data] [--upgrade-embed-wheels] [--discovery {builtin}] [-p py] [--try-first-with py_exe]
                  [--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_sep_list] [--clear] [--no-vcs-ignore] [--system-site-packages] [--symlinks | --copies] [--no-download | --download]
                  [--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel] [--no-periodic-update] [--symlink-app-data] [--prompt prompt] [-h]
                  dest

optional arguments:
  --version                     display the version of the virtualenv package and its location, then exit
  --with-traceback              on failure also display the stacktrace internals of virtualenv (default: False)
  --read-only-app-data          use app data folder in read-only mode (write operations will fail with error) (default: False)
  --app-data APP_DATA           a data folder used as cache by the virtualenv (default: /home/bojan/.local/share/virtualenv)
  --reset-app-data              start with empty app data folder (default: False)
  --upgrade-embed-wheels        trigger a manual update of the embedded wheels (default: False)
  -h, --help                    show this help message and exit

verbosity:
  verbosity = verbose - quiet, default INFO, mapping => CRITICAL=0, ERROR=1, WARNING=2, INFO=3, DEBUG=4, NOTSET=5

  -v, --verbose                 increase verbosity (default: 2)
  -q, --quiet                   decrease verbosity (default: 0)

discovery:
  discover and provide a target interpreter

  --discovery {builtin}         interpreter discovery method (default: builtin)
  -p py, --python py            interpreter based on what to create environment (path/identifier) - by default use the interpreter where the tool is installed - first found wins (default: [])
  --try-first-with py_exe       try first these interpreters before starting the discovery (default: [])

creator:
  options for creator builtin

  --creator {builtin,cpython3-posix,venv}
                                create environment via (builtin = cpython3-posix) (default: builtin)
  dest                          directory to create virtualenv at
  --clear                       remove the destination directory if exist before starting (will overwrite files otherwise) (default: False)
  --no-vcs-ignore               don't create VCS ignore directive in the destination directory (default: False)
  --system-site-packages        give the virtual environment access to the system site-packages dir (default: False)
  --symlinks                    try to use symlinks rather than copies, when symlinks are not the default for the platform (default: True)
  --copies, --always-copy       try to use copies rather than symlinks, even when symlinks are the default for the platform (default: False)

seeder:
  options for seeder app-data

  --seeder {app-data,pip}       seed packages install method (default: app-data)
  --no-seed, --without-pip      do not install seed packages (default: False)
  --no-download, --never-download
                                pass to disable download of the latest pip/setuptools/wheel from PyPI (default: True)
  --download                    pass to enable download of the latest pip/setuptools/wheel from PyPI (default: False)
  --extra-search-dir d [d ...]  a path containing wheels to extend the internal wheel list (can be set 1+ times) (default: [])
  --pip version                 version of pip to install as seed: embed, bundle or exact version (default: bundle)
  --setuptools version          version of setuptools to install as seed: embed, bundle or exact version (default: bundle)
  --wheel version               version of wheel to install as seed: embed, bundle or exact version (default: bundle)
  --no-pip                      do not install pip (default: False)
  --no-setuptools               do not install setuptools (default: False)
  --no-wheel                    do not install wheel (default: False)
  --no-periodic-update          disable the periodic (once every 14 days) update of the embedded wheels (default: False)
  --symlink-app-data            symlink the python packages from the app-data folder (requires seed pip>=19.3) (default: False)

activators:
  options for activation scripts

  --activators comma_sep_list   activators to generate - default is all supported (default: bash,cshell,fish,powershell,python,xonsh)
  --prompt prompt               provides an alternative prompt prefix for this environment (default: None)

config file /home/bojan/.config/virtualenv/virtualenv.ini missing (change via env var VIRTUALENV_CONFIG_FILE)


Create virtual environment (we chose to use python3 as Python interpreter and venv as the directory where to store virtual environment):

virtualenv -p python3 ./venv_dir_name

or

virtualenv --system-site-packages -p python3 ./venv_dir_name


If you build with virtualenv --system-site-packages ENV, your virtual environment will inherit packages from /usr/lib/python2.7/site-packages (or wherever your global site-packages directory is).

This can be used if you have control over the global site-packages directory, and you want to depend on the packages there. If you want isolation from the global system, do not use this flag. BK: This flag should be avoided as it defeats the entire point of virtualenvs (which is why --no-site-packages was made the default).

Example:

$ virtualenv -p python3 ./venv
created virtual environment CPython3.8.5.final.0-64 in 253ms
  creator CPython3Posix(dest=/home/bojan/dev/github/python-demo/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/bojan/.local/share/virtualenv)
    added seed packages: pip==20.3.3, setuptools==51.1.2, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator


I once experienced this error when tried to create a virtual environment:

$ virtualenv -p python3 ./venv
Traceback (most recent call last):
  File "/usr/local/bin/virtualenv", line 5, in <module>
    from virtualenv.__main__ import run_with_catch
  File "/usr/local/lib/python3.8/dist-packages/virtualenv/__init__.py", line 3, in <module>
    from .run import cli_run, session_via_cli
  File "/usr/local/lib/python3.8/dist-packages/virtualenv/run/__init__.py", line 6, in <module>
    from ..app_data import make_app_data
  File "/usr/local/lib/python3.8/dist-packages/virtualenv/app_data/__init__.py", line 9, in <module>
    from appdirs import user_data_dir
ModuleNotFoundError: No module named 'appdirs'

I then installed the missing module via pip but this showed another error:

$ sudo pip3 install appdirs
Collecting appdirs
  Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
ERROR: virtualenv 20.3.1 requires filelock<4,>=3.0.0, which is not installed.
Installing collected packages: appdirs
Successfully installed appdirs-1.4.4

I resolved this by reinstalling the virtualenv:

$ sudo pip3 install virtualenv
Requirement already satisfied: virtualenv in /usr/local/lib/python3.8/dist-packages (20.3.1)
Requirement already satisfied: appdirs<2,>=1.4.3 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (1.4.4)
Requirement already satisfied: six<2,>=1.9.0 in /usr/lib/python3/dist-packages (from virtualenv) (1.14.0)
Requirement already satisfied: distlib<1,>=0.3.1 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (0.3.1)
Collecting filelock<4,>=3.0.0
  Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Installing collected packages: filelock
Successfully installed filelock-3.0.12

Once virtual environment is created, we'll have venv/bin/activate script created. Let's explore it:

$ cat venv/bin/activate
...

deactivate() {
...
}
...
VIRTUAL_ENV="/home/bojan/dev/github/tensorflow-demo/venv"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
...
pydoc () {
    python -m pydoc "$@"
}
...


We can see that we'll now have a new environment variable set - VIRTUAL_ENV and also a deactivate commmand.

To activate virtual environment:

$ source ./venv/bin/activate

We can also use a dot command:

$ . ./venv/bin/activate

When virtual environment is active, your shell prompt is prefixed with its name in form: (venv).

(venv) $ which pip
/home/bojan/dev/github/python-demo/venv/bin/pip

(venv) $ which pip3
/home/bojan/dev/github/python-demo/venv/bin/pip3


If we've saved requirements.txt, we can install all dependencies now with on of the following commands (see Python Package Management with pip | My Public Notepad):

$ pip install -r requirements.txt
pip3 install -r requirements.txt

It is possible to install a single new package in the virtual environment:

$ pip install package_name


Example:

(venv) $ pip3 install psycopg2
Collecting psycopg2
  Using cached psycopg2-2.8.6-cp38-cp38-linux_x86_64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.8.6

To verify installation path:

(venv) $ pip3 show psycopg2
Name: psycopg2
Version: 2.8.6
Summary: psycopg2 - Python-PostgreSQL Database Adapter
Home-page: https://psycopg.org/
Author: Federico Di Gregorio
Author-email: fog@initd.org
License: LGPL with exceptions
Location: /home/user/dev/my-app/venv/lib/python3.8/site-packages
Requires: 
Required-by: 

To check the version of the package installed in virtual environment:

(venv) $ pip3 list | grep psycopg
psycopg2          2.8.6


To exit virtual environment:

(venv) $ deactivate

How to rename virtual environment?

directory - How to rename a virtualenv in Python? - Stack Overflow

From inside active virtual environment:

$ pip freeze > requirements.txt
$ deactivate

From python - Pip freeze vs. pip list - Stack Overflow
pip list shows ALL packages. 
pip freeze shows packages YOU installed via pip (or pipenv if using that tool) command in a requirements format.
Also, be aware of `$ pip freeze > requirements.txt` considered harmful

Then delete old environment directory. E.g.

$ rm -r ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.14.0/

...and create a new one with correct name:

$ python3 -m virtualenv -p python3 ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/nvidia/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/python3
Also creating executable in /home/nvidia/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

Activate it and install all packages from the saved list:

$ source ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/activate
$ pip install -r requirements.txt
$ pip3 install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v43 tensorflow-gpu==1.15.0+nv19.12


virtualenvwrapper


TBD...


Conda



References


1 comment:

John said...

Thank you very much. You saved the day!