Saturday, 31 October 2020

How to update conda

If conda is outdated, a warning will be displayed each time we try to use it:

Example #1:

(my-env) $ conda remove opencv
Collecting package metadata (repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.5
  latest version: 4.9.0

Please update conda by running

    $ conda update -n base -c defaults conda
...


Example #2:

(my-env) $ conda install -c conda-forge opencv
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.5
  latest version: 4.9.0

Please update conda by running

    $ conda update -n base -c defaults conda

...

Conda itself is updated just like any other conda package, with conda update command:

$ conda update --help
usage: conda update [-h] [-n ENVIRONMENT | -p PATH] [-c CHANNEL] [--use-local]
                    [--override-channels] [--repodata-fn REPODATA_FNS]
                    [--strict-channel-priority] [--no-channel-priority]
                    [--no-deps | --only-deps] [--no-pin] [--copy] [-C] [-k]
                    [--offline] [-d] [--json] [-q] [-v] [-y] [--download-only]
                    [--show-channel-urls] [--file FILE] [--force-reinstall]
                    [--freeze-installed | --update-deps | -S | --update-all | --update-specs]
                    [--clobber]
                    [package_spec [package_spec ...]]

Updates conda packages to the latest compatible version.

    This command accepts a list of package names and updates them to the latest
    versions that are compatible with all other packages in the environment.

    Conda attempts to install the newest versions of the requested packages. To
    accomplish this, it may update some packages that are already installed, or
    install additional packages. To prevent existing packages from updating,
    use the --no-update-deps option. This may force conda to install older
    versions of the requested packages, and it does not prevent additional
    dependency packages from being installed.

Options:

positional arguments:
  package_spec          Packages to install or update in the conda
                        environment.

optional arguments:
  -h, --help            Show this help message and exit.
  --file FILE           Read package versions from the given file. Repeated
                        file specifications can be passed (e.g. --file=file1
                        --file=file2).

Target Environment Specification:
  -n ENVIRONMENT, --name ENVIRONMENT
                        Name of environment.
  -p PATH, --prefix PATH
                        Full path to environment location (i.e. prefix).

Channel Customization:
  -c CHANNEL, --channel CHANNEL
                        Additional channel to search for packages. These are
                        URLs searched in the order they are given (including
                        local directories using the 'file://' syntax or simply
                        a path like '/home/conda/mychan' or '../mychan').
                        Then, the defaults or channels from .condarc are
                        searched (unless --override-channels is given). You
                        can use 'defaults' to get the default packages for
                        conda. You can also use any name and the .condarc
                        channel_alias value will be prepended. The default
                        channel_alias is http://conda.anaconda.org/.
  --use-local           Use locally built packages. Identical to '-c local'.
  --override-channels   Do not search default or .condarc channels. Requires
                        --channel.
  --repodata-fn REPODATA_FNS
                        Specify name of repodata on remote server. Conda will
                        try whatever you specify, but will ultimately fall
                        back to repodata.json if your specs are not
                        satisfiable with what you specify here. This is used
                        to employ repodata that is reduced in time scope. You
                        may pass this flag more than once. Leftmost entries
                        are tried first, and the fallback to repodata.json is
                        added for you automatically.

Solver Mode Modifiers:
  --strict-channel-priority
                        Packages in lower priority channels are not considered
                        if a package with the same name appears in a higher
                        priority channel.
  --no-channel-priority
                        Package version takes precedence over channel
                        priority. Overrides the value given by `conda config
                        --show channel_priority`.
  --no-deps             Do not install, update, remove, or change
                        dependencies. This WILL lead to broken environments
                        and inconsistent behavior. Use at your own risk.
  --only-deps           Only install dependencies.
  --no-pin              Ignore pinned file.
  --force-reinstall     Ensure that any user-requested package for the current
                        operation is uninstalled and reinstalled, even if that
                        package already exists in the environment.
  --freeze-installed, --no-update-deps
                        Do not update or change already-installed
                        dependencies.
  --update-deps         Update dependencies.
  -S, --satisfied-skip-solve
                        Exit early and do not run the solver if the requested
                        specs are satisfied. Also skips aggressive updates as
                        configured by 'aggressive_update_packages'. Similar to
                        the default behavior of 'pip install'.
  --update-all, --all   Update all installed packages in the environment.
  --update-specs        Update based on provided specifications.

Package Linking and Install-time Options:
  --copy                Install all packages using copies instead of hard- or
                        soft-linking.
  --clobber             Allow clobbering of overlapping file paths within
                        packages, and suppress related warnings.

Networking Options:
  -C, --use-index-cache
                        Use cache of channel index files, even if it has
                        expired.
  -k, --insecure        Allow conda to perform "insecure" SSL connections and
                        transfers. Equivalent to setting 'ssl_verify' to
                        'false'.
  --offline             Offline mode. Don't connect to the Internet.

Output, Prompt, and Flow Control Options:
  -d, --dry-run         Only display what would have been done.
  --json                Report all output as json. Suitable for using conda
                        programmatically.
  -q, --quiet           Do not display progress bar.
  -v, --verbose         Can be used multiple times. Once for INFO, twice for
                        DEBUG, three times for TRACE.
  -y, --yes             Do not ask for confirmation.
  --download-only       Solve an environment and ensure package caches are
                        populated, but exit prior to unlinking and linking
                        packages into the prefix.
  --show-channel-urls   Show channel urls. Overrides the value given by `conda
                        config --show show_channel_urls`.

Examples:

    conda update -n myenv scipy

 

This is the command for updating the conda:


(my-env) $ conda update -n base -c defaults conda
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/bojan/anaconda3

  added / updated specs:
    - conda


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-4.9.0                |           py38_0         2.9 MB
    conda-package-handling-1.7.2|   py38h03888b9_0         886 KB
    ------------------------------------------------------------
                                           Total:         3.7 MB

The following packages will be UPDATED:

  conda                                        4.8.5-py38_0 --> 4.9.0-py38_0
  conda-package-han~                   1.6.1-py38h7b6447c_0 --> 1.7.2-py38h03888b9_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
conda-4.9.0          | 2.9 MB    | ################################################################################################ | 100% 
conda-package-handli | 886 KB    | ################################################################################################ | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

Sunday, 4 October 2020

Introduction to JupyterLab

JupyterLab is a new interface of Jupyter Notebook server. It is a web-based interactive development environment for Jupyter notebooks, code, and data.


Installation via Anaconda


env.yml file should contain:

name: my-env # arbitrary name
...
dependencies:
  - jupyter=1.0.0
  - jupyterlab=0.34.9
...
  - python=3.6.6 # for running python kernels
...


Let's first check that python we'll be using is the right one:

(my-env) $ which python
/home/bojan/anaconda3/envs/my-env/bin/python

(my-env) $ python --version
Python 3.6.6

To launch jupyter lab:

(my-env) $ jupyter-lab
[I 07:37:01.479 LabApp] JupyterLab extension loaded from /home/bojan/anaconda3/envs/my-env/lib/python3.6/site-packages/jupyterlab
[I 07:37:01.479 LabApp] JupyterLab application directory is /home/bojan/anaconda3/envs/my-env/share/jupyter/lab
[I 07:37:01.483 LabApp] Serving notebooks from local directory: /home/bojan/dev/github/jupyterlab-demo
[I 07:37:01.484 LabApp] The Jupyter Notebook is running at:
[I 07:37:01.484 LabApp] http://localhost:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
[I 07:37:01.484 LabApp]  or http://127.0.0.1:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
[I 07:37:01.484 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 07:37:01.491 LabApp] 
    
    To access the notebook, open this file in a browser:
        file:///home/bojan/.local/share/jupyter/runtime/nbserver-20890-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
     or http://127.0.0.1:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
Opening in existing browser session.
[1004/073701.855894:ERROR:nacl_helper_linux.cc(308)] NaCl helper process running without a sandbox!
Most likely you need to configure your SUID sandbox correctly
[I 07:37:24.221 LabApp] Kernel started: 8144639c-36e9-41ec-8e0c-06b9c593a426
[I 07:37:24.780 LabApp] Build is up to date
[I 07:37:46.613 LabApp] 302 GET /?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff (127.0.0.1) 0.66ms
[I 07:37:53.685 LabApp] Starting buffering for 8144639c-36e9-41ec-8e0c-06b9c593a426:3d1e75cf-7a1f-4999-a91a-013da0761314
[I 07:37:55.524 LabApp] Build is up to date
[I 07:38:59.153 LabApp] Saving file at /Untitled.ipynb
...
[I 07:46:51.165 LabApp] Starting buffering for 8144639c-36e9-41ec-8e0c-06b9c593a426:84ef87af-47d6-473a-b8a6-cbe3f2c0da45
[I 07:46:53.082 LabApp] Build is up to date
[I 07:46:55.971 LabApp] Starting buffering for 8144639c-36e9-41ec-8e0c-06b9c593a426:58fc2ee2-42b3-408d-baa9-b9d3fc5ba29c
^C[I 07:47:02.242 LabApp] interrupted
Serving notebooks from local directory: /home/bojan/dev/github/jupyterlab-demo
1 active kernel
The Jupyter Notebook is running at:
http://localhost:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
 or http://127.0.0.1:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
Shutdown this notebook server (y/[n])? y
[C 07:47:03.991 LabApp] Shutdown confirmed
[I 07:47:03.992 LabApp] Shutting down 1 kernel
[I 07:47:04.294 LabApp] Kernel shutdown: 8144639c-36e9-41ec-8e0c-06b9c593a426

We can see that jupyterlab starts a local web server which listens on port 8888. This is a classical Jupyter Notebook server (that's why we needed to include jupyter in the environment packages). We can then access it from a browser with http://localhost:8888 or http://127.0.0.1:8888. 

jupyterlab is an extension of this server:

(my-env) $ jupyter serverextension list
config dir: /home/bojan/anaconda3/envs/my-env/etc/jupyter
    jupyterlab  enabled 
    - Validating...
      jupyterlab 0.34.9 OK

jupyter lab is accessible at /lab path so when we run it, it will automatically open http://localhost:8888/lab in a browser.

We can also run jupyter notebook:

(my-env) $ jupyter notebook
[I 08:05:37.951 NotebookApp] JupyterLab extension loaded from /home/bojan/anaconda3/envs/my-env/lib/python3.6/site-packages/jupyterlab
[I 08:05:37.951 NotebookApp] JupyterLab application directory is /home/bojan/anaconda3/envs/my-env/share/jupyter/lab
[I 08:05:37.954 NotebookApp] Serving notebooks from local directory: /home/bojan/dev/github/jupyterlab-demo
[I 08:05:37.954 NotebookApp] The Jupyter Notebook is running at:
[I 08:05:37.954 NotebookApp] http://localhost:8888/?token=90194de8634a4a8e3c0ee9d4ec760382b4aa545d7b9260f6
[I 08:05:37.954 NotebookApp]  or http://127.0.0.1:8888/?token=90194de8634a4a8e3c0ee9d4ec760382b4aa545d7b9260f6
[I 08:05:37.954 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 08:05:37.960 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///home/bojan/.local/share/jupyter/runtime/nbserver-24138-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=90194de8634a4a8e3c0ee9d4ec760382b4aa545d7b9260f6
     or http://127.0.0.1:8888/?token=90194de8634a4a8e3c0ee9d4ec760382b4aa545d7b9260f6
Opening in existing browser session.
[1004/080538.264120:ERROR:nacl_helper_linux.cc(308)] NaCl helper process running without a sandbox!
Most likely you need to configure your SUID sandbox correctly
[I 08:05:49.878 NotebookApp] Kernel started: 3aa4ef5c-e260-40af-8e54-a4f509223a64
[I 08:05:50.492 NotebookApp] Build is up to date
[I 08:06:31.324 NotebookApp] Saving file at /Untitled.ipynb




...and manually go to /lab:





References:

Saturday, 3 October 2020

Managing Python Environments with Conda

At the end of my article How to install Anaconda on Ubuntu I verified that Anaconda was installed successfully by executing conda list. You've probably noticed (base) that appears in front of command prompt:

(base) $

This happens after we activate conda by executing ~/.bashrc:

$ cat ~/.bashrc
...
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/bojan/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/bojan/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/home/bojan/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/bojan/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<
...




As conda is also an environment manager it is capable of setting a specified or a default environment. 

Each environment has associated:
  • name
  • collection of packages 
  • resources which are stored in a dedicated directory

When conda is activated, it sets a default environment unless some other is specified. Its name appears within parentheses before the prompt in terminal. In our case that was a default environment named base.

To list all available environments with their names and associated directories we can use:

$ conda env list
# conda environments:
#
base                  *  /home/bojan/anaconda3

Asterisk (*) appears next to the currently active environment.

Another way to get the same information is:

$ conda info --envs
# conda environments:
#
base                  *  /home/bojan/anaconda3

Here is the content of the base environment directory:

$ ls -1a /home/bojan/anaconda3
.
..
bin
compiler_compat
condabin
conda-meta
doc
envs
etc
include
lib
libexec
LICENSE.txt
man
mkspecs
phrasebooks
pkgs
plugins
qml
resources
sbin
share
shell
ssl
translations
var
x86_64-conda_cos6-linux-gnu

To deactivate the current environment:

(base) $ conda deactivate
$

To activate default environment again:

$ conda activate
(base) $ 


After installing Conda, base environment is set to be auto activated by default:

(base) $ conda config --show | grep auto_activate_base
auto_activate_base: True


(base) $ conda config --set auto_activate_base False

Let's verify it:

(base) $ conda config --show | grep auto_activate_base
auto_activate_base: False

Next time we open a new terminal, we should not see (base) in front of the prompt.

Note that when we have a base environment activated, Terminal shell uses Python from Anaconda installaton:

(base) $ which python
/home/bojan/anaconda3/bin/python

(base) $ which python3
/home/bojan/anaconda3/bin/python3

When no environment is activated, Terminal shell uses Python from a system installation:

$ which python
/usr/bin/python

$ which python3
/usr/bin/python3

Same applies to pip and pip3:

(base) $ which pip
/home/bojan/anaconda3/bin/pip
(base) $ which pip3
/home/bojan/anaconda3/bin/pip3

$ which pip
/home/bojan/.local/bin/pip
$ which pip3
/home/bojan/.local/bin/pip3


Each environment can be specified with its YAML (.yml) file. This file contains all necessary information for conda to recreate the same environment on some other computer:
  • name
  • channels
  • dependencies
Environment yaml file generic example:

name: my_env
channels:
  - channel1
dependencies:
  - python=3.6
  - some_package=1.2 # it is possible to add comments
  - some_package2
  - jupyter=1.0.0
  - jupyterlab=0.34.9
  - keras=2.2.2=0
  - matplotlib=2.2.3
  - ...


We can create this yaml file manually or can use conda command:

$ conda env export > my_environment.yml

To re-create the environment from a given yaml file:

$ conda env create -f my_environment.yml 
Collecting package metadata (repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.3
  latest version: 4.8.5

Please update conda by running

    $ conda update -n base -c defaults conda

Downloading and Extracting Packages
gmp-6.1.2            | 751 KB    | ##################################### | 100% 
prometheus_client-0. | 44 KB     | ##################################### | 100% 
libstdcxx-ng-9.3.0   | 4.0 MB    | ##################################### | 100% 
defusedxml-0.6.0     | 22 KB     | ##################################### | 100% 
ipython-7.16.1       | 1.1 MB    | ##################################### | 100% 
kiwisolver-1.2.0     | 87 KB     | ##################################### | 100% 
libgcc-ng-9.3.0      | 7.8 MB    | ##################################### | 100% 
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: \ b'Enabling notebook extension jupyter-js-widgets/extension...\n      - Validating: \x1b[32mOK\x1b[0m\n'
done
#
# To activate this environment, use
#
#     $ conda activate my_environment
#
# To deactivate an active environment, use
#
#     $ conda deactivate


We can verify that this environment has now been added to conda:

(base) $ conda env list
# conda environments:
#
base                  *  /home/bojan/anaconda3
my_environment           /home/bojan/anaconda3/envs/my_environment
test-opencv              /home/bojan/anaconda3/envs/test-opencv


To activate this environment:

(base)$ conda activate my_environment
(my_environment)$


To check which packages are installed in the current environment:

(my_environment) $ conda list
# packages in environment at /home/bojan/anaconda3/envs/my_environment:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
absl-py                   0.10.0           py36h9f0ad1d_0    conda-forge
argon2-cffi               20.1.0           py36h8c4c3a4_1    conda-forge
astor                     0.8.1              pyh9f0ad1d_0    conda-forge
async_generator           1.10                       py_0    conda-forge
atk                       2.25.90           hf2eb9ee_1001    conda-forge
attrs                     20.2.0             pyh9f0ad1d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
...

To install additional package to the current environment:

(my_environment) $ conda install pillow
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/bojan/anaconda3/envs/my_environment

  added / updated specs:
    - pillow


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2020.10.14 |                0         121 KB
    certifi-2020.6.20          |           py36_0         156 KB
    olefile-0.46               |           py36_0          48 KB
    openssl-1.0.2u             |       h7b6447c_0         2.2 MB
    pillow-7.2.0               |   py36hb39fc2d_0         619 KB
    ------------------------------------------------------------
                                           Total:         3.1 MB

The following NEW packages will be INSTALLED:

  lcms2              pkgs/main/linux-64::lcms2-2.11-h396b838_0
  olefile            pkgs/main/linux-64::olefile-0.46-py36_0
  pillow             pkgs/main/linux-64::pillow-7.2.0-py36hb39fc2d_0

The following packages will be UPDATED:

  ca-certificates    conda-forge::ca-certificates-2020.6.2~ --> pkgs/main::ca-certificates-2020.10.14-0

The following packages will be SUPERSEDED by a higher-priority channel:

  certifi            conda-forge::certifi-2020.6.20-py36h9~ --> pkgs/main::certifi-2020.6.20-py36_0
  openssl            conda-forge::openssl-1.0.2u-h516909a_0 --> pkgs/main::openssl-1.0.2u-h7b6447c_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
openssl-1.0.2u       | 2.2 MB    | ################################################################ | 100% 
certifi-2020.6.20    | 156 KB    | ################################################################ | 100% 
pillow-7.2.0         | 619 KB    | ################################################################ | 100% 
ca-certificates-2020 | 121 KB    | ################################################################ | 100% 
olefile-0.46         | 48 KB     | ################################################################ | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

We can verify that a new package is indeed installed in new environment:

(my_environment) $ conda list
# packages in environment at /home/bojan/anaconda3/envs/my_environment:
#
# Name                    Version                   Build  Channel
...
pillow                    7.2.0            py36hb39fc2d_0 
...

To install a new package from a specific channel in the current environment:

(my_environment) $ conda install -c channel_name package_name


To install a specific version of a new package from a specific channel in the current environment:

(my_environment) $ conda install -c channel_name package_name=version


To uninstall some package from the current environment:

(my_environment) $ conda remove pillow

To uninstall multiple packages from the current environment:

(my_environment) $ conda remove package1 package2 ...

To update conda (I've deactivated the current environment):

$ conda update -n base -c defaults conda
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/bojan/anaconda3

  added / updated specs:
    - conda


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-4.8.5                |           py38_0         2.9 MB
    ------------------------------------------------------------
                                           Total:         2.9 MB

The following packages will be UPDATED:

  conda                                        4.8.3-py38_0 --> 4.8.5-py38_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
conda-4.8.5          | 2.9 MB    | ################################################################################################################################################################# | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done


To remove given environment:

$ conda env remove --name test-opencv

Remove all packages in environment /home/bojan/anaconda3/envs/test-opencv:


We can verify that its directory does not exist:

$ ls /home/bojan/anaconda3/envs/test-opencv
ls: cannot access '/home/bojan/anaconda3/envs/test-opencv': No such file or directory

References: