Sunday, 4 October 2020

Introduction to JupyterLab

JupyterLab is a new interface of Jupyter Notebook server. It is a web-based interactive development environment for Jupyter notebooks, code, and data.


Installation via Anaconda


env.yml file should contain:

name: my-env # arbitrary name
...
dependencies:
  - jupyter=1.0.0
  - jupyterlab=0.34.9
...
  - python=3.6.6 # for running python kernels
...


Let's first check that python we'll be using is the right one:

(my-env) $ which python
/home/bojan/anaconda3/envs/my-env/bin/python

(my-env) $ python --version
Python 3.6.6

To launch jupyter lab:

(my-env) $ jupyter-lab
[I 07:37:01.479 LabApp] JupyterLab extension loaded from /home/bojan/anaconda3/envs/my-env/lib/python3.6/site-packages/jupyterlab
[I 07:37:01.479 LabApp] JupyterLab application directory is /home/bojan/anaconda3/envs/my-env/share/jupyter/lab
[I 07:37:01.483 LabApp] Serving notebooks from local directory: /home/bojan/dev/github/jupyterlab-demo
[I 07:37:01.484 LabApp] The Jupyter Notebook is running at:
[I 07:37:01.484 LabApp] http://localhost:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
[I 07:37:01.484 LabApp]  or http://127.0.0.1:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
[I 07:37:01.484 LabApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 07:37:01.491 LabApp] 
    
    To access the notebook, open this file in a browser:
        file:///home/bojan/.local/share/jupyter/runtime/nbserver-20890-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
     or http://127.0.0.1:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
Opening in existing browser session.
[1004/073701.855894:ERROR:nacl_helper_linux.cc(308)] NaCl helper process running without a sandbox!
Most likely you need to configure your SUID sandbox correctly
[I 07:37:24.221 LabApp] Kernel started: 8144639c-36e9-41ec-8e0c-06b9c593a426
[I 07:37:24.780 LabApp] Build is up to date
[I 07:37:46.613 LabApp] 302 GET /?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff (127.0.0.1) 0.66ms
[I 07:37:53.685 LabApp] Starting buffering for 8144639c-36e9-41ec-8e0c-06b9c593a426:3d1e75cf-7a1f-4999-a91a-013da0761314
[I 07:37:55.524 LabApp] Build is up to date
[I 07:38:59.153 LabApp] Saving file at /Untitled.ipynb
...
[I 07:46:51.165 LabApp] Starting buffering for 8144639c-36e9-41ec-8e0c-06b9c593a426:84ef87af-47d6-473a-b8a6-cbe3f2c0da45
[I 07:46:53.082 LabApp] Build is up to date
[I 07:46:55.971 LabApp] Starting buffering for 8144639c-36e9-41ec-8e0c-06b9c593a426:58fc2ee2-42b3-408d-baa9-b9d3fc5ba29c
^C[I 07:47:02.242 LabApp] interrupted
Serving notebooks from local directory: /home/bojan/dev/github/jupyterlab-demo
1 active kernel
The Jupyter Notebook is running at:
http://localhost:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
 or http://127.0.0.1:8888/?token=22151a5520697ab97bfe48f44bcdd6248a0c1bd7bc5100ff
Shutdown this notebook server (y/[n])? y
[C 07:47:03.991 LabApp] Shutdown confirmed
[I 07:47:03.992 LabApp] Shutting down 1 kernel
[I 07:47:04.294 LabApp] Kernel shutdown: 8144639c-36e9-41ec-8e0c-06b9c593a426

We can see that jupyterlab starts a local web server which listens on port 8888. This is a classical Jupyter Notebook server (that's why we needed to include jupyter in the environment packages). We can then access it from a browser with http://localhost:8888 or http://127.0.0.1:8888. 

jupyterlab is an extension of this server:

(my-env) $ jupyter serverextension list
config dir: /home/bojan/anaconda3/envs/my-env/etc/jupyter
    jupyterlab  enabled 
    - Validating...
      jupyterlab 0.34.9 OK

jupyter lab is accessible at /lab path so when we run it, it will automatically open http://localhost:8888/lab in a browser.

We can also run jupyter notebook:

(my-env) $ jupyter notebook
[I 08:05:37.951 NotebookApp] JupyterLab extension loaded from /home/bojan/anaconda3/envs/my-env/lib/python3.6/site-packages/jupyterlab
[I 08:05:37.951 NotebookApp] JupyterLab application directory is /home/bojan/anaconda3/envs/my-env/share/jupyter/lab
[I 08:05:37.954 NotebookApp] Serving notebooks from local directory: /home/bojan/dev/github/jupyterlab-demo
[I 08:05:37.954 NotebookApp] The Jupyter Notebook is running at:
[I 08:05:37.954 NotebookApp] http://localhost:8888/?token=90194de8634a4a8e3c0ee9d4ec760382b4aa545d7b9260f6
[I 08:05:37.954 NotebookApp]  or http://127.0.0.1:8888/?token=90194de8634a4a8e3c0ee9d4ec760382b4aa545d7b9260f6
[I 08:05:37.954 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 08:05:37.960 NotebookApp] 
    
    To access the notebook, open this file in a browser:
        file:///home/bojan/.local/share/jupyter/runtime/nbserver-24138-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/?token=90194de8634a4a8e3c0ee9d4ec760382b4aa545d7b9260f6
     or http://127.0.0.1:8888/?token=90194de8634a4a8e3c0ee9d4ec760382b4aa545d7b9260f6
Opening in existing browser session.
[1004/080538.264120:ERROR:nacl_helper_linux.cc(308)] NaCl helper process running without a sandbox!
Most likely you need to configure your SUID sandbox correctly
[I 08:05:49.878 NotebookApp] Kernel started: 3aa4ef5c-e260-40af-8e54-a4f509223a64
[I 08:05:50.492 NotebookApp] Build is up to date
[I 08:06:31.324 NotebookApp] Saving file at /Untitled.ipynb




...and manually go to /lab:





References:

Saturday, 3 October 2020

Managing Python Environments with Conda

At the end of my article How to install Anaconda on Ubuntu I verified that Anaconda was installed successfully by executing conda list. You've probably noticed (base) that appears in front of command prompt:

(base) $

This happens after we activate conda by executing ~/.bashrc:

$ cat ~/.bashrc
...
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/bojan/anaconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
    eval "$__conda_setup"
else
    if [ -f "/home/bojan/anaconda3/etc/profile.d/conda.sh" ]; then
        . "/home/bojan/anaconda3/etc/profile.d/conda.sh"
    else
        export PATH="/home/bojan/anaconda3/bin:$PATH"
    fi
fi
unset __conda_setup
# <<< conda initialize <<<
...




As conda is also an environment manager it is capable of setting a specified or a default environment. 

Each environment has associated:
  • name
  • collection of packages 
  • resources which are stored in a dedicated directory

When conda is activated, it sets a default environment unless some other is specified. Its name appears within parentheses before the prompt in terminal. In our case that was a default environment named base.

To list all available environments with their names and associated directories we can use:

$ conda env list
# conda environments:
#
base                  *  /home/bojan/anaconda3

Asterisk (*) appears next to the currently active environment.

Another way to get the same information is:

$ conda info --envs
# conda environments:
#
base                  *  /home/bojan/anaconda3

Here is the content of the base environment directory:

$ ls -1a /home/bojan/anaconda3
.
..
bin
compiler_compat
condabin
conda-meta
doc
envs
etc
include
lib
libexec
LICENSE.txt
man
mkspecs
phrasebooks
pkgs
plugins
qml
resources
sbin
share
shell
ssl
translations
var
x86_64-conda_cos6-linux-gnu

To deactivate the current environment:

(base) $ conda deactivate
$

To activate default environment again:

$ conda activate
(base) $ 

Each environment can be specified with its YAML (.yml) file. This file contains all necessary information for conda to recreate the same environment on some other computer:
  • name
  • channels
  • dependencies
Environment yaml file generic example:

name: my_env
channels:
  - channel1
dependencies:
  - python=3.6
  - some_package=1.2 # it is possible to add comments
  - some_package2
  - jupyter=1.0.0
  - jupyterlab=0.34.9
  - keras=2.2.2=0
  - matplotlib=2.2.3
  - ...


We can create this yaml file manually or can use conda command:

$ conda env export > my_environment.yml

To re-create the environment from a given yaml file:

$ conda env create -f my_environment.yml 
Collecting package metadata (repodata.json): done
Solving environment: done


==> WARNING: A newer version of conda exists. <==
  current version: 4.8.3
  latest version: 4.8.5

Please update conda by running

    $ conda update -n base -c defaults conda

Downloading and Extracting Packages
gmp-6.1.2            | 751 KB    | ##################################### | 100% 
prometheus_client-0. | 44 KB     | ##################################### | 100% 
libstdcxx-ng-9.3.0   | 4.0 MB    | ##################################### | 100% 
defusedxml-0.6.0     | 22 KB     | ##################################### | 100% 
ipython-7.16.1       | 1.1 MB    | ##################################### | 100% 
kiwisolver-1.2.0     | 87 KB     | ##################################### | 100% 
libgcc-ng-9.3.0      | 7.8 MB    | ##################################### | 100% 
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: \ b'Enabling notebook extension jupyter-js-widgets/extension...\n      - Validating: \x1b[32mOK\x1b[0m\n'
done
#
# To activate this environment, use
#
#     $ conda activate my_environment
#
# To deactivate an active environment, use
#
#     $ conda deactivate


We can verify that this environment has now been added to conda:

(base) $ conda env list
# conda environments:
#
base                  *  /home/bojan/anaconda3
my_environment           /home/bojan/anaconda3/envs/my_environment
test-opencv              /home/bojan/anaconda3/envs/test-opencv


To activate this environment:

(base)$ conda activate my_environment
(my_environment)$


To check which packages are installed in the current environment:

(my_environment) $ conda list
# packages in environment at /home/bojan/anaconda3/envs/my_environment:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
absl-py                   0.10.0           py36h9f0ad1d_0    conda-forge
argon2-cffi               20.1.0           py36h8c4c3a4_1    conda-forge
astor                     0.8.1              pyh9f0ad1d_0    conda-forge
async_generator           1.10                       py_0    conda-forge
atk                       2.25.90           hf2eb9ee_1001    conda-forge
attrs                     20.2.0             pyh9f0ad1d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
...

To install additional package to the current environment:

(my_environment) $ conda install pillow
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/bojan/anaconda3/envs/my_environment

  added / updated specs:
    - pillow


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ca-certificates-2020.10.14 |                0         121 KB
    certifi-2020.6.20          |           py36_0         156 KB
    olefile-0.46               |           py36_0          48 KB
    openssl-1.0.2u             |       h7b6447c_0         2.2 MB
    pillow-7.2.0               |   py36hb39fc2d_0         619 KB
    ------------------------------------------------------------
                                           Total:         3.1 MB

The following NEW packages will be INSTALLED:

  lcms2              pkgs/main/linux-64::lcms2-2.11-h396b838_0
  olefile            pkgs/main/linux-64::olefile-0.46-py36_0
  pillow             pkgs/main/linux-64::pillow-7.2.0-py36hb39fc2d_0

The following packages will be UPDATED:

  ca-certificates    conda-forge::ca-certificates-2020.6.2~ --> pkgs/main::ca-certificates-2020.10.14-0

The following packages will be SUPERSEDED by a higher-priority channel:

  certifi            conda-forge::certifi-2020.6.20-py36h9~ --> pkgs/main::certifi-2020.6.20-py36_0
  openssl            conda-forge::openssl-1.0.2u-h516909a_0 --> pkgs/main::openssl-1.0.2u-h7b6447c_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
openssl-1.0.2u       | 2.2 MB    | ################################################################ | 100% 
certifi-2020.6.20    | 156 KB    | ################################################################ | 100% 
pillow-7.2.0         | 619 KB    | ################################################################ | 100% 
ca-certificates-2020 | 121 KB    | ################################################################ | 100% 
olefile-0.46         | 48 KB     | ################################################################ | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

We can verify that a new package is indeed installed in new environment:

(my_environment) $ conda list
# packages in environment at /home/bojan/anaconda3/envs/my_environment:
#
# Name                    Version                   Build  Channel
...
pillow                    7.2.0            py36hb39fc2d_0 
...

To update conda (I've deactivated the current environment):

$ conda update -n base -c defaults conda
Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /home/bojan/anaconda3

  added / updated specs:
    - conda


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    conda-4.8.5                |           py38_0         2.9 MB
    ------------------------------------------------------------
                                           Total:         2.9 MB

The following packages will be UPDATED:

  conda                                        4.8.3-py38_0 --> 4.8.5-py38_0


Proceed ([y]/n)? y


Downloading and Extracting Packages
conda-4.8.5          | 2.9 MB    | ################################################################################################################################################################# | 100% 
Preparing transaction: done
Verifying transaction: done
Executing transaction: done


To remove given environment:

$ conda env remove --name test-opencv

Remove all packages in environment /home/bojan/anaconda3/envs/test-opencv:


We can verify that its directory does not exist:

$ ls /home/bojan/anaconda3/envs/test-opencv
ls: cannot access '/home/bojan/anaconda3/envs/test-opencv': No such file or directory

References: