Friday, 15 January 2021

Python Package Management with pip

 pip -  Package Manager



$ pip

Usage:   
  pip <command> [options]

Commands:
  install                     Install packages.
  download                    Download packages.
  uninstall                   Uninstall packages.
  freeze                      Output installed packages in requirements format.
  list                        List installed packages.
  show                        Show information about installed packages.
  check                       Verify installed packages have compatible dependencies.
  search                      Search PyPI for packages.
  wheel                       Build wheels from your requirements.
  hash                        Compute hashes of package archives.
  completion                  A helper command used for command completion.
  help                        Show help for commands.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be used up to 3 times (corresponding to WARNING, ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists: (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host as trusted, even though it does not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file containing the private key and the certificate in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine whether a new version of pip is available for download. Implied with --no-index.

To check its version and for which Python version it has been installed:

pip -V
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)

To list packages installed via pip:

pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
asn1crypto (0.24.0)
change-case (0.5.2)
cryptography (2.1.4)
dnspython (1.15.0)
enum34 (1.1.6)
gyp (0.1)
idna (2.6)
ipaddress (1.0.17)
iso8601 (0.1.12)
JSONSchema2DB (1.0.1)
...


Installing packages


Let's see pip install help page first:

$ pip install --help

Usage:   
  pip install [options] <requirement specifier> [package-index-options] ...
  pip install [options] -r <requirements file> [package-index-options] ...
  pip install [options] [-e] <vcs project url> ...
  pip install [options] [-e] <local project path> ...
  pip install [options] <archive url/path> ...

Description:
  Install packages from:
  
  - PyPI (and other indexes) using requirement specifiers.
  - VCS project urls.
  - Local project directories.
  - Local or remote source archives.
  
  pip also supports installing from "requirements files", which provide
  an easy way to specify a whole environment to be installed.

Install Options:
  -r, --requirement <file>    Install from the given requirements file. This
                              option can be used multiple times.
  -c, --constraint <file>     Constrain versions using the given constraints
                              file. This option can be used multiple times.
  --no-deps                   Don't install package dependencies.
  --pre                       Include pre-release and development versions. By
                              default, pip only finds stable versions.
  -e, --editable <path/url>   Install a project in editable mode (i.e.
                              setuptools "develop mode") from a local project
                              path or a VCS url.
  -t, --target <dir>          Install packages into <dir>. By default this
                              will not replace existing files/folders in
                              <dir>. Use --upgrade to replace existing
                              packages in <dir> with new versions.
  --platform <platform>       Only use wheels compatible with <platform>.
                              Defaults to the platform of the running system.
  --python-version <python_version>
                              The Python interpreter version to use for wheel
                              and "Requires-Python" compatibility checks.
                              Defaults to a version derived from the running
                              interpreter. The version can be specified using
                              up to three dot-separated integers (e.g. "3" for
                              3.0.0, "3.7" for 3.7.0, or "3.7.3"). A major-
                              minor version can also be given as a string
                              without dots (e.g. "37" for 3.7.0).
  --implementation <implementation>
                              Only use wheels compatible with Python
                              implementation <implementation>, e.g. 'pp',
                              'jy', 'cp',  or 'ip'. If not specified, then the
                              current interpreter implementation is used.  Use
                              'py' to force implementation-agnostic wheels.
  --abi <abi>                 Only use wheels compatible with Python abi
                              <abi>, e.g. 'pypy_41'.  If not specified, then
                              the current interpreter abi tag is used.
                              Generally you will need to specify
                              --implementation, --platform, and --python-
                              version when using this option.
  --user                      Install to the Python user install directory for
                              your platform. Typically ~/.local/, or
                              %APPDATA%\Python on Windows. (See the Python
                              documentation for site.USER_BASE for full
                              details.)  On Debian systems, this is the
                              default when running outside of a virtual
                              environment and not as root.
  --root <dir>                Install everything relative to this alternate
                              root directory.
  --prefix <dir>              Installation prefix where lib, bin and other
                              top-level folders are placed
  --system                    Install using the system scheme (overrides
                              --user on Debian systems)
  -b, --build <dir>           Directory to unpack packages into and build in.
                              Note that an initial build still takes place in
                              a temporary directory. The location of temporary
                              directories can be controlled by setting the
                              TMPDIR environment variable (TEMP on Windows)
                              appropriately. When passed, build directories
                              are not cleaned in case of failures.
  --src <dir>                 Directory to check out editable projects into.
                              The default in a virtualenv is "<venv
                              path>/src". The default for global installs is
                              "<current dir>/src".
  -U, --upgrade               Upgrade all specified packages to the newest
                              available version. The handling of dependencies
                              depends on the upgrade-strategy used.
  --upgrade-strategy <upgrade_strategy>
                              Determines how dependency upgrading should be
                              handled [default: only-if-needed]. "eager" -
                              dependencies are upgraded regardless of whether
                              the currently installed version satisfies the
                              requirements of the upgraded package(s). "only-
                              if-needed" -  are upgraded only when they do not
                              satisfy the requirements of the upgraded
                              package(s).
  --force-reinstall           Reinstall all packages even if they are already
                              up-to-date.
  -I, --ignore-installed      Ignore the installed packages, overwriting them.
                              This can break your system if the existing
                              package is of a different version or was
                              installed with a different package manager!
  --ignore-requires-python    Ignore the Requires-Python information.
  --no-build-isolation        Disable isolation when building a modern source
                              distribution. Build dependencies specified by
                              PEP 518 must be already installed if this option
                              is used.
  --use-pep517                Use PEP 517 for building source distributions
                              (use --no-use-pep517 to force legacy behaviour).
  --install-option <options>  Extra arguments to be supplied to the setup.py
                              install command (use like --install-option="--
                              install-scripts=/usr/local/bin"). Use multiple
                              --install-option options to pass multiple
                              options to setup.py install. If you are using an
                              option with a directory path, be sure to use
                              absolute path.
  --global-option <options>   Extra global options to be supplied to the
                              setup.py call before the install command.
  --compile                   Compile Python source files to bytecode
  --no-compile                Do not compile Python source files to bytecode
  --no-warn-script-location   Do not warn when installing scripts outside PATH
  --no-warn-conflicts         Do not warn about broken dependencies
  --no-binary <format_control>
                              Do not use binary packages. Can be supplied
                              multiple times, and each time adds to the
                              existing value. Accepts either :all: to disable
                              all binary packages, :none: to empty the set, or
                              one or more package names with commas between
                              them (no colons). Note that some packages are
                              tricky to compile and may fail to install when
                              this option is used on them.
  --only-binary <format_control>
                              Do not use source packages. Can be supplied
                              multiple times, and each time adds to the
                              existing value. Accepts either :all: to disable
                              all source packages, :none: to empty the set, or
                              one or more package names with commas between
                              them. Packages without binary distributions will
                              fail to install when this option is used on
                              them.
  --prefer-binary             Prefer older binary packages over newer source
                              packages.
  --no-clean                  Don't clean up build directories.
  --require-hashes            Require a hash to check each requirement
                              against, for repeatable installs. This option is
                              implied when any package in a requirements file
                              has a --hash option.
  --progress-bar <progress_bar>
                              Specify type of progress to be displayed
                              [off|on|ascii|pretty|emoji] (default: on)

Package Index Options:
  -i, --index-url <url>       Base URL of the Python Package Index (default
                              https://pypi.org/simple). This should point to a
                              repository compliant with PEP 503 (the simple
                              repository API) or a local directory laid out in
                              the same format.
  --extra-index-url <url>     Extra URLs of package indexes to use in addition
                              to --index-url. Should follow the same rules as
                              --index-url.
  --no-index                  Ignore package index (only looking at --find-
                              links URLs instead).
  -f, --find-links <url>      If a url or path to an html file, then parse for
                              links to archives. If a local path or file://
                              url that's a directory, then look for archives
                              in the directory listing.

General Options:
  -h, --help                  Show help.
  --isolated                  Run pip in an isolated mode, ignoring
                              environment variables and user configuration.
  -v, --verbose               Give more output. Option is additive, and can be
                              used up to 3 times.
  -V, --version               Show version and exit.
  -q, --quiet                 Give less output. Option is additive, and can be
                              used up to 3 times (corresponding to WARNING,
                              ERROR, and CRITICAL logging levels).
  --log <path>                Path to a verbose appending log.
  --proxy <proxy>             Specify a proxy in the form
                              [user:passwd@]proxy.server:port.
  --retries <retries>         Maximum number of retries each connection should
                              attempt (default 5 times).
  --timeout <sec>             Set the socket timeout (default 15 seconds).
  --exists-action <action>    Default action when a path already exists:
                              (s)witch, (i)gnore, (w)ipe, (b)ackup, (a)bort.
  --trusted-host <hostname>   Mark this host or host:port pair as trusted,
                              even though it does not have valid or any HTTPS.
  --cert <path>               Path to alternate CA bundle.
  --client-cert <path>        Path to SSL client certificate, a single file
                              containing the private key and the certificate
                              in PEM format.
  --cache-dir <dir>           Store the cache data in <dir>.
  --no-cache-dir              Disable the cache.
  --disable-pip-version-check
                              Don't periodically check PyPI to determine
                              whether a new version of pip is available for
                              download. Implied with --no-index.
  --no-color                  Suppress colored output
  --no-python-version-warning
                              Silence deprecation warnings for upcoming
                              unsupported Pythons.



To install some package via pip:

pip install jsonschema2db


To install some specific version of the package:

$ pip3 install psycopg2==2.7.6


To list properties of the installed package (including its location on disk!) use pip show:

pip show jsonschema2db
Name: JSONSchema2DB
Version: 1.0.1
Summary: Generate database tables from JSON schema
Home-page: https://better.engineering/convoys
Author: Erik Bernhardsson
Author-email: erikbern@better.com
License: MIT
Location: /home/bojan/.local/lib/python2.7/site-packages
Requires: iso8601, change-case, psycopg2

pip - How do I install a Python package with a .whl file? - Stack Overflow

pip install --user  /path/to/package.whl


Globally installed packages are located in:

/home/user/.local/lib/python3.6/site-packages/


Uninstalling Packages


To uninstall a package, use:

pip uninstall JSONSchema2DB
Uninstalling JSONSchema2DB-1.0.1:
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/DESCRIPTION.rst
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/INSTALLER
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/METADATA
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/RECORD
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/WHEEL
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/metadata.json
  /home/bojan/.local/lib/python2.7/site-packages/JSONSchema2DB-1.0.1.dist-info/top_level.txt
  /home/bojan/.local/lib/python2.7/site-packages/jsonschema2db.py
Proceed (y/n)? y
  Successfully uninstalled JSONSchema2DB-1.0.1

Example how to uninstall globally installed package:

pip uninstall psycopg2
Uninstalling psycopg2-2.7.2:
  Would remove:
    /home/bojan/.local/lib/python3.6/site-packages/psycopg2-2.7.2.dist-info/*
    /home/bojan/.local/lib/python3.6/site-packages/psycopg2/*
Proceed (y/n)? y
  Successfully uninstalled psycopg2-2.7.2

Package Management in Virtual Environments




Example:

(my_env)$ pip freeze
asn1crypto==0.24.0
change-case==0.5.2
cryptography==2.1.4
dnspython==1.15.0
enum34==1.1.6
...

To save this info in requirements.txt (this file can have an arbitrary name):

(my_env)$ pip freeze > requirements.txt

Once we have requirements file, we can recreate this environment anywhere else by installing all the packages listed there:

(my_env2) $ pip3 install -r requirements.txt --verbose


Example where virtual env is named venv_basic and requirements.txt contains a single package, python-dotenv:

$ pip3 install -r requirements_basic.txt --verbose
Using pip 20.3.3 from /home/bojan/dev/github/python-demo/venv_basic/lib/python3.8/site-packages/pip (python 3.8)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-9xisk4la
Created temporary directory: /tmp/pip-req-tracker-r1kgbsxz
Initialized build tracking at /tmp/pip-req-tracker-r1kgbsxz
Created build tracker: /tmp/pip-req-tracker-r1kgbsxz
Entered build tracker: /tmp/pip-req-tracker-r1kgbsxz
Created temporary directory: /tmp/pip-install-vqtt3gqw
1 location(s) to search for versions of python-dotenv:
* https://pypi.org/simple/python-dotenv/
Fetching project page and analyzing links: https://pypi.org/simple/python-dotenv/
Getting page https://pypi.org/simple/python-dotenv/
Found index url https://pypi.org/simple
Looking up "https://pypi.org/simple/python-dotenv/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/python-dotenv/ HTTP/1.1" 200 5997
Updating cache with response from "https://pypi.org/simple/python-dotenv/"
Caching due to etag
  Found link https://files.pythonhosted.org/packages/ad/3b/16a207ced20a1be0cc2ec5a53ab68ce1976ad643fe7f135d3addd763abf0/python-dotenv-0.1.0.tar.gz#sha256=26156022355d4457bc9062bcead7fde4824623d4463488b623bdd5ce912462fa (from https://pypi.org/simple/python-dotenv/), version: 0.1.0
  Found link https://files.pythonhosted.org/packages/e6/95/e9239c25d2c06c4a262360375d4e0c61b243d1c763d647137aca4d7df9fe/python-dotenv-0.1.2.tar.gz#sha256=aaefda0e975b702374f7c86edcf948590bd4f47737b33027de1bc72afbed745b (from https://pypi.org/simple/python-dotenv/), version: 0.1.2
  ...
  ...
  ... 
https://files.pythonhosted.org/packages/53/04/1a8126516c8febfeb2015844edee977c9b783bdff9b3bcd89b1cc2e1f372/python-dotenv-0.15.0.tar.gz#sha256=587825ed60b1711daea4832cf37524dfd404325b7db5e25ebe88c495c9f807a0 (from https://pypi.org/simple/python-dotenv/), version: 0.15.0
  Found link https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl#sha256=0c8d1b80d1a1e91717ea7d526178e3882732420b03f08afea0406db6402e220e (from https://pypi.org/simple/python-dotenv/), version: 0.15.0
Given no hashes to check 64 links for project 'python-dotenv': discarding no candidates
Collecting python-dotenv
  Created temporary directory: /tmp/pip-unpack-gp764zed
  Looking up "https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl" in the cache
  No cache entry available
  Starting new HTTPS connection (1): files.pythonhosted.org:443
  https://files.pythonhosted.org:443 "GET /packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl HTTP/1.1" 200 18253
  Downloading python_dotenv-0.15.0-py2.py3-none-any.whl (18 kB)
  Ignoring unknown cache-control directive: immutable
  Updating cache with response from "https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl"
  Caching due to etag
  Added python-dotenv from https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl#sha256=0c8d1b80d1a1e91717ea7d526178e3882732420b03f08afea0406db6402e220e (from -r requirements_basic.txt (line 1)) to build tracker '/tmp/pip-req-tracker-r1kgbsxz'
  Removed python-dotenv from https://files.pythonhosted.org/packages/32/2e/e4585559237787966aad0f8fd0fc31df1c4c9eb0e62de458c5b6cde954eb/python_dotenv-0.15.0-py2.py3-none-any.whl#sha256=0c8d1b80d1a1e91717ea7d526178e3882732420b03f08afea0406db6402e220e (from -r requirements_basic.txt (line 1)) from build tracker '/tmp/pip-req-tracker-r1kgbsxz'
Created temporary directory: /tmp/pip-unpack-bhhhx138
Installing collected packages: python-dotenv

  changing mode of /home/bojan/dev/github/python-demo/venv_basic/bin/dotenv to 775
Successfully installed python-dotenv-0.15.0
Removed build tracker: '/tmp/pip-req-tracker-r1kgbsxz'


This example also shown that pip installs the most recent version of the package if package version is not specified in requirements.txt.

To check that packages from requirements file are indeed installed we can use:

(my_env2) $ pip3 list
Package       Version
------------- -------
pip           20.3.3
python-dotenv 0.15.0
setuptools    51.1.2
wheel         0.36.2



If for some reason installation from requirements.txt fails, you can always try to manually install packages, one by one:

(my_env2)$ pip3 install python-dotenv


We can only list names of packages in this file but sometimes it is necessary to specify their versions as otherwise we might get pip error like this:

ERROR: jsonschema2db 1.0.1 has requirement psycopg2==2.7.2, but you'll have psycopg2 2.8.2 which is incompatible.


Package Management in Dockerfile


Here is an example how to use previously described approach when we need to install packages in Docker image:

Dockerfile before requirements.txt:

RUN python3 -m pip install psycopg2
RUN python3 -m pip install jsonschema2db

requirements.txt:

JSONSchema2DB==1.0.1
psycopg2==2.7.2

Dockerfile after requirements.txt:

RUN python3 -m pip install -r requirements.txt


pip3


Pip3 is the Python3 version of pip. If you just use pip, then only the python2.7 version will be installed. You have to use pip3 for it to be installed on Python3. [source]

If pip3 is not installed:

$ python3 -m pip install JSONSchema2DB
/usr/bin/python3: No module named pip

$ pip3
Command 'pip3' not found, but can be installed with:
sudo apt install python3-pip

Let's install it:

$ sudo apt install python3-pip
...
Processing triggers for man-db (2.8.3-2ubuntu0.1) ...
Setting up python3.6-dev (3.6.7-1~18.04) ...
Setting up python3-lib2to3 (3.6.7-1~18.04) ...
Setting up python3-distutils (3.6.7-1~18.04) ...
Setting up libpython3-dev:amd64 (3.6.7-1~18.04) ...
Setting up python3-pip (9.0.1-2.3~ubuntu1) ...
Setting up python3-setuptools (39.0.1-2) ...
Setting up dh-python (3.20180325ubuntu2) ...
Setting up python3-dev (3.6.7-1~18.04) ...

Now we have:

$ pip -V
pip 9.0.1 from /usr/lib/python2.7/dist-packages (python 2.7)
$ pip3 -V
pip 9.0.1 from /usr/lib/python3/dist-packages (python 3.6)

We can now use it to install some package for python3.


$ python3 -m pip install JSONSchema2DB
Collecting JSONSchema2DB
  Using cached 
...
Installing collected packages: change-case, iso8601, psycopg2, JSONSchema2DB
Successfully installed JSONSchema2DB-1.0.1 change-case-0.5.2 iso8601-0.1.12 psycopg2-2.7.2

Let's verify it is installed for Python3:

$ pip3 show JSONSchema2DB
Name: JSONSchema2DB
Version: 1.0.1
Summary: Generate database tables from JSON schema
Home-page: https://better.engineering/convoys
Author: Erik Bernhardsson
Author-email: erikbern@better.com
License: MIT
Location: /home/bojan/.local/lib/python3.6/site-packages
Requires: change-case, iso8601, psycopg2

1 comment:

micheal pan said...

BE SMART AND BECOME RICH IN LESS THAN 3DAYS....It all depends on how fast 
you can be to get the new PROGRAMMED blank ATM card that is capable of
hacking into any ATM machine,anywhere in the world. I got to know about 
this BLANK ATM CARD when I was searching for job online about a month 
ago..It has really changed my life for good and now I can say I'm rich and 
I can never be poor again. The least money I get in a day with it is about 
$50,000.(fifty thousand USD) Every now and then I keeping pumping money 
into my account. Though is illegal,there is no risk of being caught 
,because it has been programmed in such a way that it is not traceable,it 
also has a technique that makes it impossible for the CCTVs to detect 
you..For details on how to get yours today, email the hackers on : (
atmmachinehackers1@gmail.com ). Tell your 
loved once too, and start to live large. That's the simple testimony of how 
my life changed for good...Love you all ...the email address again is ;
atmmachinehackers1@gmail.com