Saturday 3 October 2020

Managing Python Environments with Conda

At the end of my article How to install Anaconda on Ubuntu I verified that Anaconda was installed successfully by executing conda list. You've probably noticed (base) that appears in front of command prompt:

(base) $

This happens after we activate conda by executing ~/.bashrc:

$ cat ~/.bashrc
...
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/bojan/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/bojan/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/home/bojan/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/bojan/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<
...




As conda is also an environment manager it is capable of setting a specified or a default environment. 

Each environment has associated:
  • name
  • collection of packages 
  • resources which are stored in a dedicated directory

When conda is activated, it sets a default environment unless some other is specified. Its name appears within parentheses before the prompt in terminal. In our case that was a default environment named base.

To list all available environments with their names and associated directories we can use:

$ conda env list
# conda environments:
#
base                  *  /home/bojan/anaconda3

Asterisk (*) appears next to the currently active environment.

Another way to get the same information is:

$ conda info --envs
# conda environments:
#
base                  *  /home/bojan/anaconda3

Here is the content of the base environment directory:

$ ls -1a /home/bojan/anaconda3
.
..
bin
compiler_compat
condabin
conda-meta
doc
envs
etc
include
lib
libexec
LICENSE.txt
man
mkspecs
phrasebooks
pkgs
plugins
qml
resources
sbin
share
shell
ssl
translations
var
x86_64-conda_cos6-linux-gnu

To deactivate the current environment:

(base) $ conda deactivate
$

To activate default environment again:

$ conda activate
(base) $ 


After installing Conda, base environment is set to be auto activated by default:

(base) $ conda config --show | grep auto_activate_base
auto_activate_base: True


(base) $ conda config --set auto_activate_base False

Let's verify it:

(base) $ conda config --show | grep auto_activate_base
auto_activate_base: False

Next time we open a new terminal, we should not see (base) in front of the prompt.

Note that when we have a base environment activated, Terminal shell uses Python from Anaconda installaton:

(base) $ which python
/home/bojan/anaconda3/bin/python

(base) $ which python3
/home/bojan/anaconda3/bin/python3

When no environment is activated, Terminal shell uses Python from a system installation:

$ which python
/usr/bin/python

$ which python3
/usr/bin/python3

Same applies to pip and pip3:

(base) $ which pip
/home/bojan/anaconda3/bin/pip
(base) $ which pip3
/home/bojan/anaconda3/bin/pip3

$ which pip
/home/bojan/.local/bin/pip
$ which pip3
/home/bojan/.local/bin/pip3


Each environment can be specified with its YAML (.yml) file. This file contains all necessary information for conda to recreate the same environment on some other computer:
  • name of the environment
  • channels
  • dependencies
Environment yaml file generic example:

name: my_env
channels:
  - channel1
dependencies:
  - python=3.6
  - some_package=1.2 # it is possible to add comments
  - some_package2
  - jupyter=1.0.0
  - jupyterlab=0.34.9
  - keras=2.2.2=0
  - matplotlib=2.2.3
  - ...


We can create this yaml file manually or can use conda command:

$ conda env export > my_environment.yml

To re-create the environment from a given yaml file:

$ conda env create -f my_environment.yml 
Collecting package metadata (repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.3
  latest version: 4.8.5

Please update conda by running

    $ conda update -n base -c defaults conda

Downloading and Extracting Packages
gmp-6.1.2            | 751 KB    | ##################################### | 100% 
prometheus_client-0. | 44 KB     | ##################################### | 100% 
libstdcxx-ng-9.3.0   | 4.0 MB    | ##################################### | 100% 
defusedxml-0.6.0     | 22 KB     | ##################################### | 100% 
ipython-7.16.1       | 1.1 MB    | ##################################### | 100% 
kiwisolver-1.2.0     | 87 KB     | ##################################### | 100% 
libgcc-ng-9.3.0      | 7.8 MB    | ##################################### | 100% 
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: \ b'Enabling notebook extension jupyter-js-widgets/extension...\n      - Validating: \x1b[32mOK\x1b[0m\n'
done
#
# To activate this environment, use
#
#     $ conda activate my_environment
#
# To deactivate an active environment, use
#
#     $ conda deactivate


We can verify that this environment has now been added to conda:

(base) $ conda env list
# conda environments:
#
base                  *  /home/bojan/anaconda3
my_environment           /home/bojan/anaconda3/envs/my_environment
test-opencv              /home/bojan/anaconda3/envs/test-opencv


To activate this environment:

(base)$ conda activate my_environment
(my_environment)$


To check which packages are installed in the current environment:

(my_environment) $ conda list
# packages in environment at /home/bojan/anaconda3/envs/my_environment:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
absl-py                   0.10.0           py36h9f0ad1d_0    conda-forge
argon2-cffi               20.1.0           py36h8c4c3a4_1    conda-forge
astor                     0.8.1              pyh9f0ad1d_0    conda-forge
async_generator           1.10                       py_0    conda-forge
atk                       2.25.90           hf2eb9ee_1001    conda-forge
attrs                     20.2.0             pyh9f0ad1d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
...

To install additional package to the current environment:

(my_environment) $ conda install pillow
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/bojan/anaconda3/envs/my_environment

  added / updated specs:
    - pillow


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2020.10.14 |                0         121 KB
    certifi-2020.6.20          |           py36_0         156 KB
    olefile-0.46               |           py36_0          48 KB
    openssl-1.0.2u             |       h7b6447c_0         2.2 MB
    pillow-7.2.0               |   py36hb39fc2d_0         619 KB
    ------------------------------------------------------------
                                           Total:         3.1 MB

The following NEW packages will be INSTALLED:

  lcms2              pkgs/main/linux-64::lcms2-2.11-h396b838_0
  olefile            pkgs/main/linux-64::olefile-0.46-py36_0
  pillow             pkgs/main/linux-64::pillow-7.2.0-py36hb39fc2d_0

The following packages will be UPDATED:

  ca-certificates    conda-forge::ca-certificates-2020.6.2~ --> pkgs/main::ca-certificates-2020.10.14-0

The following packages will be SUPERSEDED by a higher-priority channel:

  certifi            conda-forge::certifi-2020.6.20-py36h9~ --> pkgs/main::certifi-2020.6.20-py36_0
  openssl            conda-forge::openssl-1.0.2u-h516909a_0 --> pkgs/main::openssl-1.0.2u-h7b6447c_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
openssl-1.0.2u       | 2.2 MB    | ################################################################ | 100% 
certifi-2020.6.20    | 156 KB    | ################################################################ | 100% 
pillow-7.2.0         | 619 KB    | ################################################################ | 100% 
ca-certificates-2020 | 121 KB    | ################################################################ | 100% 
olefile-0.46         | 48 KB     | ################################################################ | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

We can verify that a new package is indeed installed in new environment:

(my_environment) $ conda list
# packages in environment at /home/bojan/anaconda3/envs/my_environment:
#
# Name                    Version                   Build  Channel
...
pillow                    7.2.0            py36hb39fc2d_0 
...

To install a new package from a specific channel in the current environment:

(my_environment) $ conda install -c channel_name package_name


To install a specific version of a new package from a specific channel in the current environment:

(my_environment) $ conda install -c channel_name package_name=version


To uninstall some package from the current environment:

(my_environment) $ conda remove pillow

To uninstall multiple packages from the current environment:

(my_environment) $ conda remove package1 package2 ...

To update an environment with (new) dependencies in (new) yml file:

(my_environment$ conda deactivate
$ conda env update --file my_environment.yml
$ conda activate my_environment


To update conda (I've deactivated the current environment):

$ conda update -n base -c defaults conda
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/bojan/anaconda3

  added / updated specs:
    - conda


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-4.8.5                |           py38_0         2.9 MB
    ------------------------------------------------------------
                                           Total:         2.9 MB

The following packages will be UPDATED:

  conda                                        4.8.3-py38_0 --> 4.8.5-py38_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
conda-4.8.5          | 2.9 MB    | ################################################################################################################################################################# | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done


To remove given environment:

$ conda env remove --name test-opencv

Remove all packages in environment /home/bojan/anaconda3/envs/test-opencv:


We can verify that its directory does not exist:

$ ls /home/bojan/anaconda3/envs/test-opencv
ls: cannot access '/home/bojan/anaconda3/envs/test-opencv': No such file or directory

References:


No comments: