Different Python projects depend on different packages. To keep them separated and avoid polluting the global space with package installations we can use the concept of virtual environments. There are several ways to work with virtual environments in Python:
- virtualenv
- virtualenvwrapper
- Conda (part of Anaconda)
virtualenv
To use virtualenv we first need to install all necessary components:
$ sudo apt update
$ sudo apt install python3-dev python3-pip
$ sudo pip3 install -U virtualenv # system-wide install
Verification:
$ python3 --version
$ pip3 --version
$ virtualenv --version
or
$ virtualenv
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--read-only-app-data] [--app-data APP_DATA] [--reset-app-data] [--upgrade-embed-wheels] [--discovery {builtin}] [-p py] [--try-first-with py_exe]
[--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_sep_list] [--clear] [--no-vcs-ignore] [--system-site-packages] [--symlinks | --copies] [--no-download | --download]
[--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel] [--no-periodic-update] [--symlink-app-data] [--prompt prompt] [-h]
dest
virtualenv: error: the following arguments are required: dest
SystemExit: 2
or
$ virtualenv --help
usage: virtualenv [--version] [--with-traceback] [-v | -q] [--read-only-app-data] [--app-data APP_DATA] [--reset-app-data] [--upgrade-embed-wheels] [--discovery {builtin}] [-p py] [--try-first-with py_exe]
[--creator {builtin,cpython3-posix,venv}] [--seeder {app-data,pip}] [--no-seed] [--activators comma_sep_list] [--clear] [--no-vcs-ignore] [--system-site-packages] [--symlinks | --copies] [--no-download | --download]
[--extra-search-dir d [d ...]] [--pip version] [--setuptools version] [--wheel version] [--no-pip] [--no-setuptools] [--no-wheel] [--no-periodic-update] [--symlink-app-data] [--prompt prompt] [-h]
dest
optional arguments:
--version display the version of the virtualenv package and its location, then exit
--with-traceback on failure also display the stacktrace internals of virtualenv (default: False)
--read-only-app-data use app data folder in read-only mode (write operations will fail with error) (default: False)
--app-data APP_DATA a data folder used as cache by the virtualenv (default: /home/bojan/.local/share/virtualenv)
--reset-app-data start with empty app data folder (default: False)
--upgrade-embed-wheels trigger a manual update of the embedded wheels (default: False)
-h, --help show this help message and exit
verbosity:
verbosity = verbose - quiet, default INFO, mapping => CRITICAL=0, ERROR=1, WARNING=2, INFO=3, DEBUG=4, NOTSET=5
-v, --verbose increase verbosity (default: 2)
-q, --quiet decrease verbosity (default: 0)
discovery:
discover and provide a target interpreter
--discovery {builtin} interpreter discovery method (default: builtin)
-p py, --python py interpreter based on what to create environment (path/identifier) - by default use the interpreter where the tool is installed - first found wins (default: [])
--try-first-with py_exe try first these interpreters before starting the discovery (default: [])
creator:
options for creator builtin
--creator {builtin,cpython3-posix,venv}
create environment via (builtin = cpython3-posix) (default: builtin)
dest directory to create virtualenv at
--clear remove the destination directory if exist before starting (will overwrite files otherwise) (default: False)
--no-vcs-ignore don't create VCS ignore directive in the destination directory (default: False)
--system-site-packages give the virtual environment access to the system site-packages dir (default: False)
--symlinks try to use symlinks rather than copies, when symlinks are not the default for the platform (default: True)
--copies, --always-copy try to use copies rather than symlinks, even when symlinks are the default for the platform (default: False)
seeder:
options for seeder app-data
--seeder {app-data,pip} seed packages install method (default: app-data)
--no-seed, --without-pip do not install seed packages (default: False)
--no-download, --never-download
pass to disable download of the latest pip/setuptools/wheel from PyPI (default: True)
--download pass to enable download of the latest pip/setuptools/wheel from PyPI (default: False)
--extra-search-dir d [d ...] a path containing wheels to extend the internal wheel list (can be set 1+ times) (default: [])
--pip version version of pip to install as seed: embed, bundle or exact version (default: bundle)
--setuptools version version of setuptools to install as seed: embed, bundle or exact version (default: bundle)
--wheel version version of wheel to install as seed: embed, bundle or exact version (default: bundle)
--no-pip do not install pip (default: False)
--no-setuptools do not install setuptools (default: False)
--no-wheel do not install wheel (default: False)
--no-periodic-update disable the periodic (once every 14 days) update of the embedded wheels (default: False)
--symlink-app-data symlink the python packages from the app-data folder (requires seed pip>=19.3) (default: False)
activators:
options for activation scripts
--activators comma_sep_list activators to generate - default is all supported (default: bash,cshell,fish,powershell,python,xonsh)
--prompt prompt provides an alternative prompt prefix for this environment (default: None)
config file /home/bojan/.config/virtualenv/virtualenv.ini missing (change via env var VIRTUALENV_CONFIG_FILE)
Create virtual environment (we chose to use python3 as Python interpreter and venv as the directory where to store virtual environment):
$ virtualenv -p python3 ./venv_dir_name
or$ virtualenv -p python3 ./venv_dir_name
$ virtualenv --system-site-packages -p python3 ./venv_dir_name
From virtualenv · PyPI:
If you build with virtualenv --system-site-packages ENV, your virtual environment will inherit packages from /usr/lib/python2.7/site-packages (or wherever your global site-packages directory is).This can be used if you have control over the global site-packages directory, and you want to depend on the packages there. If you want isolation from the global system, do not use this flag. BK: This flag should be avoided as it defeats the entire point of virtualenvs (which is why --no-site-packages was made the default).
Example:
$ virtualenv -p python3 ./venv
created virtual environment CPython3.8.5.final.0-64 in 253ms
creator CPython3Posix(dest=/home/bojan/dev/github/python-demo/venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/bojan/.local/share/virtualenv)
added seed packages: pip==20.3.3, setuptools==51.1.2, wheel==0.36.2
activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
I once experienced this error when tried to create a virtual environment:
$ virtualenv -p python3 ./venv
Traceback (most recent call last):
File "/usr/local/bin/virtualenv", line 5, in <module>
from virtualenv.__main__ import run_with_catch
File "/usr/local/lib/python3.8/dist-packages/virtualenv/__init__.py", line 3, in <module>
from .run import cli_run, session_via_cli
File "/usr/local/lib/python3.8/dist-packages/virtualenv/run/__init__.py", line 6, in <module>
from ..app_data import make_app_data
File "/usr/local/lib/python3.8/dist-packages/virtualenv/app_data/__init__.py", line 9, in <module>
from appdirs import user_data_dir
ModuleNotFoundError: No module named 'appdirs'
I then installed the missing module via pip but this showed another error:
$ sudo pip3 install appdirs
Collecting appdirs
Downloading appdirs-1.4.4-py2.py3-none-any.whl (9.6 kB)
ERROR: virtualenv 20.3.1 requires filelock<4,>=3.0.0, which is not installed.
Installing collected packages: appdirs
Successfully installed appdirs-1.4.4
I resolved this by reinstalling the virtualenv:
$ sudo pip3 install virtualenv
Requirement already satisfied: virtualenv in /usr/local/lib/python3.8/dist-packages (20.3.1)
Requirement already satisfied: appdirs<2,>=1.4.3 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (1.4.4)
Requirement already satisfied: six<2,>=1.9.0 in /usr/lib/python3/dist-packages (from virtualenv) (1.14.0)
Requirement already satisfied: distlib<1,>=0.3.1 in /usr/local/lib/python3.8/dist-packages (from virtualenv) (0.3.1)
Collecting filelock<4,>=3.0.0
Downloading filelock-3.0.12-py3-none-any.whl (7.6 kB)
Installing collected packages: filelock
Successfully installed filelock-3.0.12
Once virtual environment is created, we'll have venv/bin/activate script created. Let's explore it:
$ cat venv/bin/activate
...
deactivate() {
...
}
...
VIRTUAL_ENV="/home/bojan/dev/github/tensorflow-demo/venv"export VIRTUAL_ENV
_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
...
pydoc () {
python -m pydoc "$@"
}
...
We can see that we'll now have a new environment variable set - VIRTUAL_ENV and also a deactivate commmand.
To activate virtual environment:
$ source ./venv/bin/activate
We can also use a dot command:
$ . ./venv/bin/activate
When virtual environment is active, your shell prompt is prefixed with its name in form: (venv).
(venv) $ which pip
/home/bojan/dev/github/python-demo/venv/bin/pip
(venv) $ which pip3
/home/bojan/dev/github/python-demo/venv/bin/pip3
If we've saved requirements.txt, we can install all dependencies now with on of the following commands (see Python Package Management with pip | My Public Notepad):
$ pip install -r requirements.txt
$ pip3 install -r requirements.txt
$ pip3 install -r requirements.txt --verbose
Installing packages using pip and virtual environments — Python Packaging User Guide
Installing packages using pip and virtual environments — Python Packaging User Guide
It is possible to install a single new package in the virtual environment:
$ pip install package_name
Example:
(venv) $ pip3 install psycopg2
Collecting psycopg2
Using cached psycopg2-2.8.6-cp38-cp38-linux_x86_64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.8.6
To verify installation path:
(venv) $ pip3 show psycopg2
Name: psycopg2
Version: 2.8.6
Summary: psycopg2 - Python-PostgreSQL Database Adapter
Home-page: https://psycopg.org/
Author: Federico Di Gregorio
Author-email: fog@initd.org
License: LGPL with exceptions
Location: /home/user/dev/my-app/venv/lib/python3.8/site-packages
Requires:
Required-by:
To check the version of the package installed in virtual environment:
(venv) $ pip3 list | grep psycopg
psycopg2 2.8.6
To exit virtual environment:
(venv) $ deactivate
How to rename virtual environment?
directory - How to rename a virtualenv in Python? - Stack Overflow
From inside active virtual environment:
$ pip freeze > requirements.txt
$ deactivate
From python - Pip freeze vs. pip list - Stack Overflow:
pip list shows ALL packages.
pip freeze shows packages YOU installed via pip (or pipenv if using that tool) command in a requirements format.
Also, be aware of `$ pip freeze > requirements.txt` considered harmful.
Then delete old environment directory. E.g.
$ rm -r ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.14.0/
...and create a new one with correct name:
$ python3 -m virtualenv -p python3 ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/nvidia/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/python3
Also creating executable in /home/nvidia/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.
Activate it and install all packages from the saved list:
$ source ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/activate
$ pip install -r requirements.txt
$ pip3 install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v43 tensorflow-gpu==1.15.0+nv19.12
virtualenvwrapper
TBD...
Conda
References
Setting up your python development environment (with pyenv, virtualenv, and virtualenvwrapper)
The Right Way to Use Virtual Environments
Pipenv & Virtual Environments
Create a virtual environment
Virtual Environments and Packages
Python Virtual Environments: A Primer
The Right Way to Use Virtual Environments
Pipenv & Virtual Environments
Create a virtual environment
Virtual Environments and Packages
Python Virtual Environments: A Primer
1 comment:
Thank you very much. You saved the day!
Post a Comment