Friday, 10 May 2019

Introduction to Python

Python versions

To find out which version the executable python refers to, execute this:

$ python --version
Python 2.7.15rc1

Installing packages

$ pip install jsonschema2db

pip - How do I install a Python package with a .whl file? - Stack Overflow

$ pip install --user  /path/to/package.whl

Globally installed packages are located in:


Uninstalling Packages

Example how to uninstall globally installed package:

$ pip uninstall psycopg2
Uninstalling psycopg2-2.7.2:
  Would remove:
Proceed (y/n)? y
  Successfully uninstalled psycopg2-2.7.2

Importing packages

import datetime
from jsonschema2db import JSONSchemaToPostgres

Getting Package Info

$ python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import JSONSchema2DB
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'JSONSchema2DB'
>>> import jsonschema2db
>>> help(jsonschema2db)
>>> exit()

help(package_name) opens a documentation mode which lists all classes, functions etc.:

Help on module jsonschema2db:


    class JSONSchemaToDatabase(builtins.object)
     |  JSONSchemaToDatabase is the mother class for everything
     |  :param schema: The JSON schema, as a native Python dict
     |  :param database_flavor: Either "postgres" or "redshift"


If we want to see docs for some particular class within the package we can start package examination from the top level and go deeper:

$ python3
Python 3.6.9 (default, Apr 18 2020, 01:56:04) 
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import scrapy
>>> help(scrapy)
>>> help(scrapy.http)
>>> help(scrapy.http.response)
>>> help(scrapy.http.response.Response)

Another example:

(venv) xxx:~/path/to/tensorflow-demo$ python
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help
Type help() for interactive help, or help(object) for help about object.
>>> help("packages")
No Python documentation found for 'packages'.
Use help() to get the interactive help utility.
Use help(str) for help on the str class.

>>> help("modules")
_operator           copyreg             nis                 tensorflow
_osx_support        crypt               nntplib             tensorflow_core
_pickle             cryptography        notebook            tensorflow_estimator


>>> help("tensorflow")

>>> help("tensorflow_core")

pydoc is is Python module which generates documentation about modules, classes, functions and methods.

First run:

$ pydoc modules


$ python 
Python 3.6.8 (default, Oct  7 2019, 12:59:55) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> help('modules') get a list of all modules.

Then run:

$ pydoc module


$ pydoc tensorflow_core
$ pydoc tensorflow_core.keras
$ pydoc tensorflow_core.keras.datasets.mnist

pydoc path:

$ which pydoc

pydoc can be used from virtual environment because. Virtual env's activate script exposes pydoc function (see section about virtual environments in this article).

How to find package version programmatically?

Almost every normal package in python assigns the variable .__version__ or VERSION to the current version. So if you want to find the version of some package you can do the following

import a
a.__version__ # or a.VERSION




def add(a, b):
    return a+b

Python 3 (3.5, from 2011) introduces (optional) function annotations:

def add(a: number, b: number) -> number:
    return a+b

After importing some module, we can use dir(module) to list all its variables and functions. To find out types of arguments and return value of some function we can use inspect.signature or pydoc.render_doc. Example:

print("inspect.signature(mnist.load_data): {}".format(inspect.signature(mnist.load_data)))

print("pydoc.render_doc(mnist.load_data): {}".format(pydoc.render_doc(mnist.load_data)))

introspection - How do I look inside a Python object? - Stack Overflow
PEP 257 -- Docstring Conventions |


class MyClass:

In Python 3 each class implicitly inherit from object.


class Derived(Base):

Member Functions

class C:
    def foo(self, a):

    def bar(self, b):

Python call function within class - Stack Overflow

Data Types


Python List


Python Dictionary Methods
python dictionaries and JSON (crash course)


Dependencies & Package Management

pip -  Package Manager

$ pip

  pip <command> [options]

  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible dependencies.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host as trusted, even though it does not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file containing the private key and the certificate in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
                              Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index.

To check its version and for which Python version it has been installed:

$ pip -V
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)

To list packages installed via pip:

$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
change-case (0.5.2)
cryptography (2.1.4)
dnspython (1.15.0)
enum34 (1.1.6)
gyp (0.1)
idna (2.6)
ipaddress (1.0.17)
iso8601 (0.1.12)
JSONSchema2DB (1.0.1)

To install some package via pip:

$ pip install jsonschema2db

To list properties of the installed package (including its location on disk!) use pip show:

$ pip show jsonschema2db
Name: JSONSchema2DB
Version: 1.0.1
Summary: Generate database tables from JSON schema
Author: Erik Bernhardsson
License: MIT
Location: /home/bojan/.local/lib/python2.7/site-packages
Requires: iso8601, change-case, psycopg2

To uninstall a package, use:

$ pip uninstall JSONSchema2DB
Uninstalling JSONSchema2DB-1.0.1:
Proceed (y/n)? y
  Successfully uninstalled JSONSchema2DB-1.0.1


$ pip freeze

To save this info in requirements.txt:

$ pip freeze > requirements.txt

Dockerfile before requirements.txt:

RUN python3 -m pip install psycopg2
RUN python3 -m pip install jsonschema2db



Dockerfile after requirements.txt:

RUN python3 -m pip install -r requirements.txt

We can only list names of packages in this file but sometimes it is necessary to specify their versions as otherwise we might get pip error like this:

ERROR: jsonschema2db 1.0.1 has requirement psycopg2==2.7.2, but you'll have psycopg2 2.8.2 which is incompatible.


Pip3 is the Python3 version of pip. If you just use pip, then only the python2.7 version will be installed. You have to use pip3 for it to be installed on Python3. [source]

If pip3 is not installed:

$ python3 -m pip install JSONSchema2DB
/usr/bin/python3: No module named pip

$ pip3
Command 'pip3' not found, but can be installed with:
sudo apt install python3-pip

Let's install it:

$ sudo apt install python3-pip
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up python3.6-dev (3.6.7-1~18.04) ...
Setting up python3-lib2to3 (3.6.7-1~18.04) ...
Setting up python3-distutils (3.6.7-1~18.04) ...
Setting up libpython3-dev:amd64 (3.6.7-1~18.04) ...
Setting up python3-pip (9.0.1-2.3~ubuntu1) ...
Setting up python3-setuptools (39.0.1-2) ...
Setting up dh-python (3.20180325ubuntu2) ...
Setting up python3-dev (3.6.7-1~18.04) ...

Now we have:

$ pip -V
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
$ pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

We can now use it to install some package for python3.

$ python3 -m pip install JSONSchema2DB
Collecting JSONSchema2DB
  Using cached 
Installing collected packages: change-case, iso8601, psycopg2, JSONSchema2DB
Successfully installed JSONSchema2DB-1.0.1 change-case-0.5.2 iso8601-0.1.12 psycopg2-2.7.2

Let's verify it is installed for Python3:

$ pip3 show JSONSchema2DB
Name: JSONSchema2DB
Version: 1.0.1
Summary: Generate database tables from JSON schema
Author: Erik Bernhardsson
License: MIT
Location: /home/bojan/.local/lib/python3.6/site-packages
Requires: change-case, iso8601, psycopg2

Special Variables


When the Python interpeter reads a source file, it first defines a few special variables e.g. __name__ variable. If you are running your module (the source file) as the main program, e.g.

$ python

...the interpreter will assign the hard-coded string "__main__" to the __name__ variable.

If there's a statement like this in the main program, or in some other module the main program imports:

# Suppose this is in some other main program.
import foo

The interpreter will search for your file (along with searching for a few other variants), and prior to executing that module, it will assign the name "foo" from the import statement to the __name__ variable, i.e.

# It's as if the interpreter inserts this at the top
# of your module when it's imported from another module.
__name__ = "foo"

An Introduction To Venv

Install all necessary components:

$ sudo apt update
$ sudo apt install python3-dev python3-pip
$ sudo pip3 install -U virtualenv  # system-wide install


$ python3 --version
$ pip3 --version
$ virtualenv --version

Create virtual environment (we chose to use python3 as Python interpreter and venv as the directory where to store virt. env.):

$ virtualenv --system-site-packages -p python3 ./venv

Once virtual environment is created, we'll have venv/bin/activate script created. To explore it:

$ cat venv/bin/activate

deactivate() {

export PATH
pydoc () {
    python -m pydoc "$@"

We can see that we'll now have a new environment variable set - VIRTUAL_ENV and also a deactivate commmand.

To activate virtual environment:

$ source ./venv/bin/activate

When virtualenv is active, your shell prompt is prefixed with its name in form: (venv).

If we've saved requirements.txt, we can install all dependencies now with:

$ pip install -r requirements.txt

To exit virtual environment:

(venv) $ deactivate

How to rename virtual environment?

directory - How to rename a virtualenv in Python? - Stack Overflow

From inside active virtual environment:

$ pip freeze > requirements.txt
$ deactivate

From python - Pip freeze vs. pip list - Stack Overflow
pip list shows ALL packages. 
pip freeze shows packages YOU installed via pip (or pipenv if using that tool) command in a requirements format.
Also, be aware of `$ pip freeze > requirements.txt` considered harmful

Then delete old environment directory. E.g.

$ rm -r ~/python-envs/tensorrt-

...and create a new one with correct name:

$ python3 -m virtualenv -p python3 ~/python-envs/tensorrt-
Already using interpreter /usr/bin/python3
Using base prefix '/usr'
New python executable in /home/nvidia/python-envs/tensorrt-
Also creating executable in /home/nvidia/python-envs/tensorrt-
Installing setuptools, pkg_resources, pip, wheel...done.

Activate it and install all packages from the saved list:

$ source ~/python-envs/tensorrt-
$ pip install -r requirements.txt
$ pip3 install --extra-index-url tensorflow-gpu==1.15.0+nv19.12

Installing Python in Docker

How to compare version strings?

How do I compare version numbers in Python?

from packaging import version
>>> version.parse("2.3.1") < version.parse("10.1.2")

If module packaging is not installed, you might get the following error:

No module named 'packaging'

Solution is to install it via pip (after optionally updating pip itself):

$ pip3 install --upgrade pip
$ pip3 install packaging

References and Further Reading

Learn Python Programming - The Definitive Guide

How to install multiple python packages at once using pip

No comments: