Friday, 10 May 2019

Introduction to Python

Python versions

To find out which version the executable python refers to, execute this:

$ python --version
Python 2.7.15rc1


Installing packages


$ pip install jsonschema2db

pip - How do I install a Python package with a .whl file? - Stack Overflow

$ pip install --user  /path/to/package.whl


Importing packages


import datetime
from jsonschema2db import JSONSchemaToPostgres

Getting Package Info


$ python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import JSONSchema2DB
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'JSONSchema2DB'
>>> import jsonschema2db
>>> help(jsonschema2db)
>>> exit()

help(package_name) opens a documentation mode which lists all classes, functions etc.:

Help on module jsonschema2db:

NAME
    jsonschema2db

CLASSES
    builtins.object
        JSONSchemaToDatabase
            JSONSchemaToPostgres
            JSONSchemaToRedshift
    
    class JSONSchemaToDatabase(builtins.object)
     |  JSONSchemaToDatabase is the mother class for everything
     |  
     |  :param schema: The JSON schema, as a native Python dict
     |  :param database_flavor: Either "postgres" or "redshift"
...

FILE
    /home/bojan/.local/lib/python3.6/site-packages/jsonschema2db.py


Another example:

(venv) xxx:~/path/to/tensorflow-demo$ python
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help
Type help() for interactive help, or help(object) for help about object.
>>> help("packages")
No Python documentation found for 'packages'.
Use help() to get the interactive help utility.
Use help(str) for help on the str class.

>>> help("modules")
...
_operator           copyreg             nis                 tensorflow
_osx_support        crypt               nntplib             tensorflow_core
_pickle             cryptography        notebook            tensorflow_estimator

...

>>> help("tensorflow")

>>> help("tensorflow_core")



pydoc is is Python module which generates documentation about modules, classes, functions and methods.

First run:

$ pydoc modules

or

$ python 
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help('modules')

...to get a list of all modules.

Then run:

$ pydoc module

Example:

$ pydoc tensorflow_core
$ pydoc tensorflow_core.keras
$ pydoc tensorflow_core.keras.datasets.mnist

pydoc path:

$ which pydoc
/usr/bin/pydoc

pydoc can be used from virtual environment because. Virtual env's activate script exposes pydoc function (see section about virtual environments in this article).

How to find package version programmatically?


Almost every normal package in python assigns the variable .__version__ or VERSION to the current version. So if you want to find the version of some package you can do the following

import a
a.__version__ # or a.VERSION

[source]

Functions


Example:

def add(a, b):
    return a+b


Python 3 (3.5, from 2011) introduces (optional) function annotations:

def add(a: number, b: number) -> number:
    return a+b

After importing some module, we can use dir(module) to list all its variables and functions. To find out types of arguments and return value of some function we can use inspect.signature or pydoc.render_doc. Example:

print("inspect.signature(mnist.load_data): {}".format(inspect.signature(mnist.load_data)))

print("pydoc.render_doc(mnist.load_data): {}".format(pydoc.render_doc(mnist.load_data)))

introspection - How do I look inside a Python object? - Stack Overflow
PEP 257 -- Docstring Conventions | Python.org



Classes



class MyClass:

In Python 3 each class implicitly inherit from object.

Inheritance

class Derived(Base):

---

Data Types

List

Python List


Dictionary


Python Dictionary Methods
python dictionaries and JSON (crash course)

Environment




Dependencies & Package Management


pip -  Package Manager



$ pip

Usage:   
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible dependencies.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host as trusted, even though it does not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file containing the private key and the certificate in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index.

To check its version and for which Python version it has been installed:

$ pip -V
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)

To list packages installed via pip:

$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
change-case (0.5.2)
cryptography (2.1.4)
dnspython (1.15.0)
enum34 (1.1.6)
gyp (0.1)
idna (2.6)
ipaddress (1.0.17)
iso8601 (0.1.12)
JSONSchema2DB (1.0.1)
...

To install some package via pip:

$ pip install jsonschema2db

To list properties of the installed package (including its location on disk!) use pip show:

$ pip show jsonschema2db
Name: JSONSchema2DB
Version: 1.0.1
Summary: Generate database tables from JSON schema
Home-page: https://better.engineering/convoys
Author: Erik Bernhardsson
Author-email: erikbern@better.com
License: MIT
Location: /home/bojan/.local/lib/python2.7/site-packages
Requires: iso8601, change-case, psycopg2

To uninstall a package, use:

$ pip uninstall JSONSchema2DB
Uninstalling JSONSchema2DB-1.0.1:
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/DESCRIPTION.rst
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/INSTALLER
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/METADATA
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/RECORD
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/WHEEL
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/metadata.json
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/top_level.txt
  /home/bojan/.local/lib/python2.7/site-packages/jsonschema2db.py
Proceed (y/n)? y
  Successfully uninstalled JSONSchema2DB-1.0.1



Example:

$ pip freeze
asn1crypto==0.24.0
change-case==0.5.2
cryptography==2.1.4
dnspython==1.15.0
enum34==1.1.6
...

To save this info in requirements.txt:

$ pip freeze > requirements.txt

Dockerfile before requirements.txt:

RUN python3 -m pip install psycopg2
RUN python3 -m pip install jsonschema2db

requirements.txt:

JSONSchema2DB==1.0.1
psycopg2==2.7.2

Dockerfile after requirements.txt:

RUN python3 -m pip install -r requirements.txt

We can only list names of packages in this file but sometimes it is necessary to specify their versions as otherwise we might get pip error like this:

ERROR: jsonschema2db 1.0.1 has requirement psycopg2==2.7.2, but you'll have psycopg2 2.8.2 which is incompatible.


pip3


Pip3 is the Python3 version of pip. If you just use pip, then only the python2.7 version will be installed. You have to use pip3 for it to be installed on Python3. [source]

If pip3 is not installed:

$ python3 -m pip install JSONSchema2DB
/usr/bin/python3: No module named pip

$ pip3
Command 'pip3' not found, but can be installed with:
sudo apt install python3-pip

Let's install it:

$ sudo apt install python3-pip
...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up python3.6-dev (3.6.7-1~18.04) ...
Setting up python3-lib2to3 (3.6.7-1~18.04) ...
Setting up python3-distutils (3.6.7-1~18.04) ...
Setting up libpython3-dev:amd64 (3.6.7-1~18.04) ...
Setting up python3-pip (9.0.1-2.3~ubuntu1) ...
Setting up python3-setuptools (39.0.1-2) ...
Setting up dh-python (3.20180325ubuntu2) ...
Setting up python3-dev (3.6.7-1~18.04) ...

Now we have:

$ pip -V
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
$ pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

We can now use it to install some package for python3.


$ python3 -m pip install JSONSchema2DB
Collecting JSONSchema2DB
  Using cached 
...
Installing collected packages: change-case, iso8601, psycopg2, JSONSchema2DB
Successfully installed JSONSchema2DB-1.0.1 change-case-0.5.2 iso8601-0.1.12 psycopg2-2.7.2

Let's verify it is installed for Python3:

$ pip3 show JSONSchema2DB
Name: JSONSchema2DB
Version: 1.0.1
Summary: Generate database tables from JSON schema
Home-page: https://better.engineering/convoys
Author: Erik Bernhardsson
Author-email: erikbern@better.com
License: MIT
Location: /home/bojan/.local/lib/python3.6/site-packages
Requires: change-case, iso8601, psycopg2


An Introduction To Venv


Install all necessary components:

$ sudo apt update
$ sudo apt install python3-dev python3-pip
$ sudo pip3 install -U virtualenv  # system-wide install

Verification:

$ python3 --version
$ pip3 --version
$ virtualenv --version

Create virtual environment (we chose to use python3 as Python interpreter and venv as the directory where to store virt. env.):

$ virtualenv --system-site-packages -p python3 ./venv


Once virtual environment is created, we'll have venv/bin/activate script created. To explore it:

$ cat venv/bin/activate
...

deactivate() {
...
}
...
VIRTUAL_ENV="/home/bojan/dev/github/tensorflow-demo/venv"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH
...
pydoc () {
    python -m pydoc "$@"
}
...


We can see that we'll now have a new environment variable set - VIRTUAL_ENV and also a deactivate commmand.

To activate virtual environment:

$ source ./venv/bin/activate

When virtualenv is active, your shell prompt is prefixed with its name in form: (venv).

If we've saved requirements.txt, we can install all dependencies now with:

$ pip install -r requirements.txt

To exit virtual environment:

(venv) $ deactivate

How to rename virtual environment?

directory - How to rename a virtualenv in Python? - Stack Overflow

From inside active virtual environment:

$ pip freeze > requirements.txt
$ deactivate

Then delete old environment directory. E.g.

$ rm -r ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.14.0/

...and create a new one with correct name:

$ python3 -m virtualenv -p python3 ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/nvidia/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/python3
Also creating executable in /home/nvidia/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/python
Installing setuptools, pkg_resources, pip, wheel...done.

Activate it and install all packages from the saved list:

$ source ~/python-envs/tensorrt-5.1.5.1-tensorflow-1.15.0/bin/activate
$ pip install -r requirements.txt
$ pip3 install --extra-index-url https://developer.download.nvidia.com/compute/redist/jp/v43 tensorflow-gpu==1.15.0+nv19.12


Installing Python in Docker


https://stackoverflow.com/questions/22938679/error-trying-to-install-postgres-for-python-psycopg2


How to compare version strings?


How do I compare version numbers in Python?

from packaging import version
>>> version.parse("2.3.1") < version.parse("10.1.2")

If module packaging is not installed, you might get the following error:

No module named 'packaging'


Solution is to install it via pip (after optionally updating pip itself):

$ pip3 install --upgrade pip
$ pip3 install packaging

References and Further Reading


Learn Python Programming - The Definitive Guide



How to install multiple python packages at once using pip


No comments: