Friday, 10 July 2020

Using ESLint

ESLint is one of most popular linters for JavaScript and TypeScript (via typescript-eslint).




To use, we need to install it first. It can be installed locally (per project) or globally. To install it globally:

$ npm install -g eslint

We can verify that package is installed:

$ npm -g list  | grep eslint
├─┬ eslint@7.4.0
│ ├─┬ eslint-scope@5.1.0
│ ├─┬ eslint-utils@2.1.0
│ │ └── eslint-visitor-keys@1.3.0 deduped
│ ├── eslint-visitor-keys@1.3.0
│ │ └── eslint-visitor-keys@1.3.0 deduped

...and also check its command line args:

$ eslint --help
eslint [options] file.js [file.js] [dir]

Basic configuration:
  --no-eslintrc                   Disable use of configuration from .eslintrc.*
  -c, --config path::String       Use this configuration, overriding .eslintrc.* config options if present
  --env [String]                  Specify environments
  --ext [String]                  Specify JavaScript file extensions
  --global [String]               Define global variables
  --parser String                 Specify the parser to be used
  --parser-options Object         Specify parser options
  --resolve-plugins-relative-to path::String  A folder where plugins should be resolved from, CWD by default

Specifying rules and plugins:
  --rulesdir [path::String]       Use additional rules from this directory
  --plugin [String]               Specify plugins
  --rule Object                   Specify rules

Fixing problems:
  --fix                           Automatically fix problems
  --fix-dry-run                   Automatically fix problems without saving the changes to the file system
  --fix-type Array                Specify the types of fixes to apply (problem, suggestion, layout)

Ignoring files:
  --ignore-path path::String      Specify path of ignore file
  --no-ignore                     Disable use of ignore files and patterns
  --ignore-pattern [String]       Pattern of files to ignore (in addition to those in .eslintignore)

Using stdin:
  --stdin                         Lint code provided on <STDIN> - default: false
  --stdin-filename String         Specify filename to process STDIN as

Handling warnings:
  --quiet                         Report errors only - default: false
  --max-warnings Int              Number of warnings to trigger nonzero exit code - default: -1

Output:
  -o, --output-file path::String  Specify file to write report to
  -f, --format String             Use a specific output format - default: stylish
  --color, --no-color             Force enabling/disabling of color

Inline configuration comments:
  --no-inline-config              Prevent comments from changing config or rules
  --report-unused-disable-directives  Adds reported errors for unused eslint-disable directives

Caching:
  --cache                         Only check changed files - default: false
  --cache-file path::String       Path to the cache file. Deprecated: use --cache-location - default: .eslintcache
  --cache-location path::String   Path to the cache file or directory

Miscellaneous:
  --init                          Run config initialization wizard - default: false
  --env-info                      Output execution environment information - default: false
  --no-error-on-unmatched-pattern  Prevent errors when pattern is unmatched
  --debug                         Output debugging information
  -h, --help                      Show help
  -v, --version                   Output the version number
  --print-config path::String     Print the configuration for the given file




To initialise and configure ESLint launch configuration wizard:

$ eslint --init


ESLint configuration will be saved in file .eslintrc.json (if you opt json file to be used). Example content:

.eslintrc.json:

{
    "env": {
        "browser": true,
        "es2020": true
    },
    "extends": "eslint:recommended",
    "parserOptions": {
        "ecmaVersion": 11
    },
    "rules": {
        "indent": [
            "error",
            4
        ],
        "quotes": [
            "error",
            "single"
        ],
        "semi": [
            "error",
            "always"
        ]
    }
}

If you use VS Code, you can install ESLint plugin which will pick up this configuration and automatically lint your code and show warnings and error son the go.

ESLint should be added to Node project as a dev dependency.

package.json (created by $npm init):

  "devDependencies": {
    "eslint": "^7.4.0"
  }


It is also possible to run this linter from the terminal. For example, to lint all files in the current project recursively with respect to .eslintrc.json:

$eslint .

...
  2:32009  error  Strings must use singlequote                                               quotes
  3:23     error  Strings must use singlequote                                               quotes
  3:42     error  Missing semicolon                                                          semi
  3:43     error  Strings must use singlequote                                               quotes
  3:70     error  Strings must use singlequote                                               quotes
  3:166    error  Strings must use singlequote                                               quotes
  3:207    error  Strings must use singlequote                                               quotes
  3:280    error  Strings must use singlequote                                               quotes
  3:338    error  Missing semicolon                                                          semi
  3:460    error  Missing semicolon                                                          semi
  3:555    error  Missing semicolon                                                          semi
...
  4:22578  error  Missing semicolon                                                          semi
  4:22601  error  Missing semicolon                                                          semi

✖ 11436 problems (11436 errors, 0 warnings)


To ignore rules in .eslintrc.json and run the linter only for JavaScript files and only to check one particular rule:

/my-project$ eslint . --ext .js --no-eslintrc --rule 'indent: ["error", 4, { "SwitchCase": 1 }]'

/my-project/path/to/file.js
   28:1  error  Expected indentation of 12 spaces but found 16  indent
   46:1  error  Expected indentation of 12 spaces but found 16  indent
  186:1  error  Expected indentation of 4 spaces but found 8    indent
  187:1  error  Expected indentation of 4 spaces but found 8    indent
  188:1  error  Expected indentation of 4 spaces but found 8    indent
  189:1  error  Expected indentation of 4 spaces but found 8    indent

Rules used here was indent.

References






Image processing on Linux

mogrify is a part of ImageMagick package which enables processing a set of input images (instead of a single one).

Resizing JPG images


If image is e.g. 4608px wide and we want to make it 2 times smaller (maintaining the aspect ratio) we can do:

# assume all .jpg images are in current directory
$ mkdir output
$ mogrify -path ./output -resize 2304  *.jpg

or 

$ mogrify -path ./output -resize 50%  *.jpg


Converting JPG images to PNG


$ mkdir output_png
$ mogrify -path ./output_png -format png  *.jpg


mogrify - full list of arguments and options


$ mogrify
Version: ImageMagick 6.9.7-4 Q16 x86_64 20170114 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC Modules OpenMP 
Delegates (built-in): bzlib djvu fftw fontconfig freetype jbig jng jpeg lcms lqr ltdl lzma openexr pangocairo png tiff wmf x xml zlib
Usage: mogrify-im6.q16 [options ...] file [ [options ...] file ...]

Image Settings:
  -adjoin              join images into a single multi-image file
  -affine matrix       affine transform matrix
  -alpha option        activate, deactivate, reset, or set the alpha channel
  -antialias           remove pixel-aliasing
  -authenticate password
                       decipher image with this password
  -attenuate value     lessen (or intensify) when adding noise to an image
  -background color    background color
  -bias value          add bias when convolving an image
  -black-point-compensation
                       use black point compensation
  -blue-primary point  chromaticity blue primary point
  -bordercolor color   border color
  -caption string      assign a caption to an image
  -cdl filename        color correct with a color decision list
  -channel type        apply option to select image channels
  -colors value        preferred number of colors in the image
  -colorspace type     alternate image colorspace
  -comment string      annotate image with comment
  -compose operator    set image composite operator
  -compress type       type of pixel compression when writing the image
  -decipher filename   convert cipher pixels to plain pixels
  -define format:option
                       define one or more image format options
  -delay value         display the next image after pausing
  -density geometry    horizontal and vertical density of the image
  -depth value         image depth
  -direction type      render text right-to-left or left-to-right
  -display server      get image or font from this X server
  -dispose method      layer disposal method
  -dither method       apply error diffusion to image
  -encipher filename   convert plain pixels to cipher pixels
  -encoding type       text encoding type
  -endian type         endianness (MSB or LSB) of the image
  -family name         render text with this font family
  -features distance   analyze image features (e.g. contrast, correlation)
  -fill color          color to use when filling a graphic primitive
  -filter type         use this filter when resizing an image
  -flatten             flatten a sequence of images
  -font name           render text with this font
  -format "string"     output formatted image characteristics
  -function name       apply a function to the image
  -fuzz distance       colors within this distance are considered equal
  -gravity type        horizontal and vertical text placement
  -green-primary point chromaticity green primary point
  -intensity method    method to generate intensity value from pixel
  -intent type         type of rendering intent when managing the image color
  -interlace type      type of image interlacing scheme
  -interline-spacing value
                       set the space between two text lines
  -interpolate method  pixel color interpolation method
  -interword-spacing value
                       set the space between two words
  -kerning value       set the space between two letters
  -label string        assign a label to an image
  -limit type value    pixel cache resource limit
  -loop iterations     add Netscape loop extension to your GIF animation
  -mask filename       associate a mask with the image
  -matte               store matte channel if the image has one
  -mattecolor color    frame color
  -monitor             monitor progress
  -morphology method kernel
                       apply a morphology method to the image
  -orient type         image orientation
  -page geometry       size and location of an image canvas (setting)
  -path path           write images to this path on disk
  -ping                efficiently determine image attributes
  -pointsize value     font point size
  -precision value     maximum number of significant digits to print
  -preview type        image preview type
  -quality value       JPEG/MIFF/PNG compression level
  -quiet               suppress all warning messages
  -red-primary point   chromaticity red primary point
  -regard-warnings     pay attention to warning messages
  -remap filename      transform image colors to match this set of colors
  -respect-parentheses settings remain in effect until parenthesis boundary
  -sampling-factor geometry
                       horizontal and vertical sampling factor
  -scene value         image scene number
  -seed value          seed a new sequence of pseudo-random numbers
  -size geometry       width and height of image
  -stretch type        render text with this font stretch
  -stroke color        graphic primitive stroke color
  -strokewidth value   graphic primitive stroke width
  -style type          render text with this font style
  -synchronize         synchronize image to storage device
  -taint               declare the image as modified
  -texture filename    name of texture to tile onto the image background
  -tile-offset geometry
                       tile offset
  -treedepth value     color tree depth
  -transparent-color color
                       transparent color
  -undercolor color    annotation bounding box color
  -units type          the units of image resolution
  -verbose             print detailed information about the image
  -view                FlashPix viewing transforms
  -virtual-pixel method
                       virtual pixel access method
  -weight type         render text with this font weight
  -white-point point   chromaticity white point

Image Operators:
  -adaptive-blur geometry
                       adaptively blur pixels; decrease effect near edges
  -adaptive-resize geometry
                       adaptively resize image using 'mesh' interpolation
  -adaptive-sharpen geometry
                       adaptively sharpen pixels; increase effect near edges
  -alpha option        on, activate, off, deactivate, set, opaque, copy
                       transparent, extract, background, or shape
  -annotate geometry text
                       annotate the image with text
  -auto-gamma          automagically adjust gamma level of image
  -auto-level          automagically adjust color levels of image
  -auto-orient         automagically orient (rotate) image
  -bench iterations    measure performance
  -black-threshold value
                       force all pixels below the threshold into black
  -blue-shift          simulate a scene at nighttime in the moonlight
  -blur geometry       reduce image noise and reduce detail levels
  -border geometry     surround image with a border of color
  -bordercolor color   border color
  -brightness-contrast geometry
                       improve brightness / contrast of the image
  -canny geometry      detect edges in the image
  -cdl filename        color correct with a color decision list
  -charcoal radius     simulate a charcoal drawing
  -chop geometry       remove pixels from the image interior
  -clamp               keep pixel values in range (0-QuantumRange)
  -clip                clip along the first path from the 8BIM profile
  -clip-mask filename  associate a clip mask with the image
  -clip-path id        clip along a named path from the 8BIM profile
  -colorize value      colorize the image with the fill color
  -color-matrix matrix apply color correction to the image
  -connected-components connectivity
                       connected-components uniquely labeled
  -contrast            enhance or reduce the image contrast
  -contrast-stretch geometry
                       improve contrast by `stretching' the intensity range
  -convolve coefficients
                       apply a convolution kernel to the image
  -cycle amount        cycle the image colormap
  -decipher filename   convert cipher pixels to plain pixels
  -deskew threshold    straighten an image
  -despeckle           reduce the speckles within an image
  -distort method args
                       distort images according to given method ad args
  -draw string         annotate the image with a graphic primitive
  -edge radius         apply a filter to detect edges in the image
  -encipher filename   convert plain pixels to cipher pixels
  -emboss radius       emboss an image
  -enhance             apply a digital filter to enhance a noisy image
  -equalize            perform histogram equalization to an image
  -evaluate operator value
                       evaluate an arithmetic, relational, or logical expression
  -extent geometry     set the image size
  -extract geometry    extract area from image
  -hough-lines geometry
                       identify lines in the image
  -features distance   analyze image features (e.g. contrast, correlation)
  -fft                 implements the discrete Fourier transform (DFT)
  -flip                flip image vertically
  -floodfill geometry color
                       floodfill the image with color
  -flop                flop image horizontally
  -frame geometry      surround image with an ornamental border
  -function name parameters
                       apply function over image values
  -gamma value         level of gamma correction
  -gaussian-blur geometry
                       reduce image noise and reduce detail levels
  -geometry geometry   preferred size or location of the image
  -grayscale method    convert image to grayscale
  -help                print program options
  -identify            identify the format and characteristics of the image
  -ift                 implements the inverse discrete Fourier transform (DFT)
  -implode amount      implode image pixels about the center
  -kuwahara geometry   edge preserving noise reduction filter
  -lat geometry        local adaptive thresholding
  -layers method       optimize, merge,  or compare image layers
  -level value         adjust the level of image contrast
  -level-colors color,color
                       level image with the given colors
  -linear-stretch geometry
                       improve contrast by `stretching with saturation'
  -liquid-rescale geometry
                       rescale image with seam-carving
  -local-contrast geometry
                       enhance local contrast
  -magnify             double the size of the image with pixel art scaling
  -mean-shift geometry delineate arbitrarily shaped clusters in the image
  -median geometry     apply a median filter to the image
  -mode geometry       make each pixel the 'predominant color' of the
                       neighborhood
  -modulate value      vary the brightness, saturation, and hue
  -monochrome          transform image to black and white
  -morphology method kernel
                       apply a morphology method to the image
  -motion-blur geometry
                       simulate motion blur
  -negate              replace every pixel with its complementary color 
  -noise geometry      add or reduce noise in an image
  -normalize           transform image to span the full range of colors
  -opaque color        change this color to the fill color
  -ordered-dither NxN
                       add a noise pattern to the image with specific
                       amplitudes
  -paint radius        simulate an oil painting
  -perceptible epsilon
                       pixel value less than |epsilon| become epsilon or
                       -epsilon
  -polaroid angle      simulate a Polaroid picture
  -posterize levels    reduce the image to a limited number of color levels
  -profile filename    add, delete, or apply an image profile
  -quantize colorspace reduce colors in this colorspace
  -radial-blur angle   radial blur the image
  -raise value         lighten/darken image edges to create a 3-D effect
  -random-threshold low,high
                       random threshold the image
  -region geometry     apply options to a portion of the image
  -render              render vector graphics
  -repage geometry     size and location of an image canvas
  -resample geometry   change the resolution of an image
  -resize geometry     resize the image
  -roll geometry       roll an image vertically or horizontally
  -rotate degrees      apply Paeth rotation to the image
  -sample geometry     scale image with pixel sampling
  -scale geometry      scale the image
  -segment values      segment an image
  -selective-blur geometry
                       selectively blur pixels within a contrast threshold
  -sepia-tone threshold
                       simulate a sepia-toned photo
  -set property value  set an image property
  -shade degrees       shade the image using a distant light source
  -shadow geometry     simulate an image shadow
  -sharpen geometry    sharpen the image
  -shave geometry      shave pixels from the image edges
  -shear geometry      slide one edge of the image along the X or Y axis
  -sigmoidal-contrast geometry
                       increase the contrast without saturating highlights or
                       shadows
  -sketch geometry     simulate a pencil sketch
  -solarize threshold  negate all pixels above the threshold level
  -sparse-color method args
                       fill in a image based on a few color points
  -splice geometry     splice the background color into the image
  -spread radius       displace image pixels by a random amount
  -statistic type radius
                       replace each pixel with corresponding statistic from the neighborhood
  -strip               strip image of all profiles and comments
  -swirl degrees       swirl image pixels about the center
  -threshold value     threshold the image
  -thumbnail geometry  create a thumbnail of the image
  -tile filename       tile image when filling a graphic primitive
  -tint value          tint the image with the fill color
  -transform           affine transform image
  -transparent color   make this color transparent within the image
  -transpose           flip image vertically and rotate 90 degrees
  -transverse          flop image horizontally and rotate 270 degrees
  -trim                trim image edges
  -type type           image type
  -unique-colors       discard all but one of any pixel color
  -unsharp geometry    sharpen the image
  -vignette geometry   soften the edges of the image in vignette style
  -wave geometry       alter an image along a sine wave
  -wavelet-denoise threshold
                       removes noise from the image using a wavelet transform
  -white-threshold value
                       force all pixels above the threshold into white

Image Sequence Operators:
  -affinity filename   transform image colors to match this set of colors
  -append              append an image sequence
  -clut                apply a color lookup table to the image
  -coalesce            merge a sequence of images
  -combine             combine a sequence of images
  -compare             mathematically and visually annotate the difference between an image and its reconstruction
  -complex operator    perform complex mathematics on an image sequence
  -composite           composite image
  -copy geometry offset
                       copy pixels from one area of an image to another
  -crop geometry       cut out a rectangular region of the image
  -deconstruct         break down an image sequence into constituent parts
  -evaluate-sequence operator
                       evaluate an arithmetic, relational, or logical expression
  -flatten             flatten a sequence of images
  -fx expression       apply mathematical expression to an image channel(s)
  -hald-clut           apply a Hald color lookup table to the image
  -layers method       optimize, merge, or compare image layers
  -morph value         morph an image sequence
  -mosaic              create a mosaic from an image sequence
  -poly terms          build a polynomial from the image sequence and the corresponding
                       terms (coefficients and degree pairs).
  -print string        interpret string and print to console
  -process arguments   process the image with a custom image filter
  -separate            separate an image channel into a grayscale image
  -smush geometry      smush an image sequence together
  -write filename      write images to this file

Image Stack Operators:
  -delete indexes      delete the image from the image sequence
  -duplicate count,indexes
                       duplicate an image one or more times
  -insert index        insert last image into the image sequence
  -reverse             reverse image sequence
  -swap indexes        swap two images in the image sequence

Miscellaneous Options:
  -debug events        display copious debugging information
  -distribute-cache port
                       distributed pixel cache spanning one or more servers
  -help                print program options
  -list type           print a list of supported option arguments
  -log format          format of debugging information
  -version             print version information

By default, the image format of `file' is determined by its magic
number.  To specify a particular image format, precede the filename
with an image format name and a colon (i.e. ps:image) or specify the
image type as the filename suffix (i.e. image.ps).  Specify 'file' as
'-' for standard input or output.

Wednesday, 24 June 2020

Go Language: Errors


errors (standard package)

errors - The Go Programming Language

Go by Example: Errors

If function returns type error and we want to return error object with custom message:

return errors.New("Argument is nil: file")

To create an error with formatted message use fmt.Errorf:

err := fmt.Errorf("user %q (id %d) not found", name, id)


Example when type assertion is used to convert from error to some specific error type:
What is err.(*os.PathError) in Go?

Example:

fi, err := file.Stat()
if err != nil {
return 0, fmt.Errorf("Failed to access file %s", (err.(*os.PathError)).Path)
}


github.com/pkg/errors package


Go default package "errors" does not support capturing stack trace info at the moment when error is created. When the error is created somewhere deep on the stack, it can be returned from the chain of function calls to the main error handler where the only information which can be extracted is error text message, without its context. When such message appears in the log, it is difficult to find out where it came from, from which function. 

Example of the Go app log snippet:

...
Error: Bad status: 404 Not Found
Terminating with error: 1
Process exited with code 1


All we know from error message "Bad status: 404 Not Found" that some HTTP response returned 404 but we don't know which one and in which part of the code.

To capture stack trace at the moment when error gets created and later be able to print it out, we can use github.com/pkg/errors package.

errors package · pkg.go.dev

This article explains best practices and how you need to change our code in order to use this package efficiently:




Here are the changes in the code base that need to be done when switching from errors to github.com/pkg/errors package:

import (
"errors"
...
)

will be replaced with 

import (
        ...
"github.com/pkg/errors"
)

...so go.mod will contain something like:

github.com/pkg/errors v0.8.1


Error creation errors.New("...") will remain the same but 

fmt.Errorf("%s value is empty string", key)

will be replaced with 

errors.Errorf("%s value is empty string", key)


Instead of passing the original error and message:

if err := bg.foo(br, df); err != nil {
return err
}

...we can add a context info by adding a contextual error message (and still keeping the original one):

if err := bg.foo(br, df); err != nil {
return errors.Wrapf(err, "Fetching resource with ID = %s failed for entity with ID = %s", br.ResID, br.ID)
}


Instead of just printing the error message:

func LogAndExit(err error, code int) {
if err != nil {
log.Printf("Error: %s\nTerminating with error: %d", err.Error(), code)
os.Exit(code)
}
}

we can now print message AND stack trace:
 
func LogAndExit(err error, code int) {
if err != nil {
log.Printf("Error: %+v\nTerminating with error: %d", err, code)
os.Exit(code)
}
}

Monday, 15 June 2020

Go Linters in VS Code

If you are using VS Code for Go development, you have probably installed Go for Visual Studio Code extension. It allows configuring a linter of choice by going to settings (CTRL+,), typing go.lintTool and selecting a desired linter:



golint


golint is a default linter. To run it across all files in the project, execute this from the project's root directory:

$ golint ./...

For each issue detected, golint's output contains source code file name, line and column numbers and the linter message.

When I run golint on one of my projects that I used for benchmark, it found 9 issues.



Staticcheck


Staticcheck is an advanced linter. VS Code Go extension can install it automatically if we select staticcheck as go.lintTool value and opt to install it in the popup notification that will apper upon selection:


Installation log can be verified in the output window:


To run it across all files in the project, execute this from the project's root directory:

$ staticcheck ./...

Similar to golinit, for each issue detected the output contains source code file name, line and column numbers and the linter message.

When I run Staticcheck on my benchmark project, it found 57 issues.

Static check's repo is at https://github.com/dominikh/go-tools and shows active development. At the time of writing has 72 watchers, 3.2k stars and 199 forks.

revive


revive is another Go linting tool which is listed among other linters in the default linters list in go.lintTools:



Installation output log:




To run it across all files in the project, execute this from the project's root directory:

$ revive ./...

Similar to golinit, for each issue detected the output contains source code file name, line and column numbers and the linter message.

When I run revive on my benchmark project, it found 11 issues.

Revive repo is at https://github.com/mgechev/revive and shows active development. At the time of writing has 29 watchers, 2.3k stars and 107 forks.


golangci-lint


golangci-lint is a Go linters aggregator which currently includes 48 linters. Some are enabled by default (e.g. staticcheck) and some are not. There is no need to install all linters, this tool does it all, just select golangci-lint as go.lintTool value:




Installation log output:



To run it across all files in the project, execute this from the project's root directory:

$ golangci-lint run

The output is more verbose, in addition to source code file name, line and column numbers, for each issue detected golangci-lint also prints that code line with a caret that shows exact character in line for which the linter message is issued. All messages are coloured.

When I run golangci-lint on my benchmark project, it found the same number of issues as Staticcheck. When compared the output, I found that issues reported were mostly the same as those from Staticcheck but were reported as the catch of some other enabled linters (e.g. gosimple, govet, deadcheck, unused etc...).

golangci-lint repo is at https://github.com/golangci/golangci-lint and shows active development. At the time of writing has 77 watchers, 5.5k stars and 495 forks.


Conclusion


Given the more complete and verbose output and larger community around the project, I would opt for using golangci-lint as the preferred Go linting tool.


Wednesday, 20 May 2020

Generator Functions in various Programming Languages

A generator is a special type of function which can pause its execution, yield result back to the caller and resume later, at caller’s convenience. And this happens as long as the generator has something to return, some value to yield back. Caller is usually iterating over yielded values in a loop, it takes a new value as soon as generator yields it. The main benefit of generators is that not all elements in a  sequence have to be kept in memory at the same time but only a single one. 



Generators in Python


Sunday, 26 April 2020

How to install Scrapy on Ubuntu

$ which python3
/usr/bin/python3

$ python3 --version
Python 3.6.9

$ virtualenv --python=python3.6 venv
Running virtualenv with interpreter /usr/bin/python3.6
Already using interpreter /usr/bin/python3.6
Using base prefix '/usr'
New python executable in /home/bojan/dev/github/scrapy-demo/venv/bin/python3.6
Also creating executable in /home/bojan/dev/github/scrapy-demo/venv/bin/python
Installing setuptools, pip, wheel...
done.

$ source venv/bin/activate

(venv) $ pip install scrapy
Collecting scrapy
  Downloading Scrapy-2.1.0-py2.py3-none-any.whl (239 kB)
Collecting service-identity>=16.0.0
  Downloading service_identity-18.1.0-py2.py3-none-any.whl (11 kB)
Collecting cryptography>=2.0
  Downloading cryptography-2.9.2-cp35-abi3-manylinux2010_x86_64.whl (2.7 MB)
Collecting cssselect>=0.9.1
  Downloading cssselect-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting w3lib>=1.17.0
  Downloading w3lib-1.21.0-py2.py3-none-any.whl (20 kB)
Collecting pyOpenSSL>=16.2.0
  Downloading pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
Collecting zope.interface>=4.1.3
  Downloading zope.interface-5.1.0-cp36-cp36m-manylinux2010_x86_64.whl (234 kB)
Collecting protego>=0.1.15
  Downloading Protego-0.1.16.tar.gz (3.2 MB)
Collecting parsel>=1.5.0
  Downloading parsel-1.5.2-py2.py3-none-any.whl (12 kB)
Collecting queuelib>=1.4.2
  Downloading queuelib-1.5.0-py2.py3-none-any.whl (13 kB)
Collecting lxml>=3.5.0
  Downloading lxml-4.5.0-cp36-cp36m-manylinux1_x86_64.whl (5.8 MB)
Collecting PyDispatcher>=2.0.5
  Downloading PyDispatcher-2.0.5.tar.gz (34 kB)
Collecting Twisted>=17.9.0
  Downloading Twisted-20.3.0-cp36-cp36m-manylinux1_x86_64.whl (3.1 MB)
Collecting pyasn1-modules
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
Collecting pyasn1
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
Collecting attrs>=16.0.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting six>=1.4.1
  Using cached six-1.14.0-py2.py3-none-any.whl (10 kB)
Collecting cffi!=1.11.3,>=1.8
  Downloading cffi-1.14.0-cp36-cp36m-manylinux1_x86_64.whl (399 kB)
Requirement already satisfied: setuptools in ./venv/lib/python3.6/site-packages (from zope.interface>=4.1.3->scrapy) (46.1.3)
Collecting incremental>=16.10.1
  Downloading incremental-17.5.0-py2.py3-none-any.whl (16 kB)
Collecting Automat>=0.3.0
  Downloading Automat-20.2.0-py2.py3-none-any.whl (31 kB)
Collecting hyperlink>=17.1.1
  Downloading hyperlink-19.0.0-py2.py3-none-any.whl (38 kB)
Collecting PyHamcrest!=1.10.0,>=1.9.0
  Downloading PyHamcrest-2.0.2-py3-none-any.whl (52 kB)
Collecting constantly>=15.1
  Downloading constantly-15.1.0-py2.py3-none-any.whl (7.9 kB)
Collecting pycparser
  Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting idna>=2.5
  Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Building wheels for collected packages: protego, PyDispatcher
  Building wheel for protego (setup.py) ... done
  Created wheel for protego: filename=Protego-0.1.16-py3-none-any.whl size=7765 sha256=7696d4fa63732f3509349e932b23800b7dacc6034fc150eaa3f4448fa401c6aa
  Stored in directory: /home/bojan/.cache/pip/wheels/b2/74/25/517a0ec6186297704db56664268e72686f5cfa8ab398582f33
  Building wheel for PyDispatcher (setup.py) ... done
  Created wheel for PyDispatcher: filename=PyDispatcher-2.0.5-py3-none-any.whl size=11515 sha256=17b011eb905d7eccda1eaafb51833fbe7a1b8406fb4a7764e7216ef19fafd698
  Stored in directory: /home/bojan/.cache/pip/wheels/28/db/61/691c759da06ba9b86da079bdd17cb3e01828d49d5c152cb3af
Successfully built protego PyDispatcher
Installing collected packages: six, pycparser, cffi, cryptography, pyasn1, pyasn1-modules, attrs, service-identity, cssselect, w3lib, pyOpenSSL, zope.interface, protego, lxml, parsel, queuelib, PyDispatcher, incremental, Automat, idna, hyperlink, PyHamcrest, constantly, Twisted, scrapy
Successfully installed Automat-20.2.0 PyDispatcher-2.0.5 PyHamcrest-2.0.2 Twisted-20.3.0 attrs-19.3.0 cffi-1.14.0 constantly-15.1.0 cryptography-2.9.2 cssselect-1.1.0 hyperlink-19.0.0 idna-2.9 incremental-17.5.0 lxml-4.5.0 parsel-1.5.2 protego-0.1.16 pyOpenSSL-19.1.0 pyasn1-0.4.8 pyasn1-modules-0.2.8 pycparser-2.20 queuelib-1.5.0 scrapy-2.1.0 service-identity-18.1.0 six-1.14.0 w3lib-1.21.0 zope.interface-5.1.0

(venv) $ pip list --local
Package          Version
---------------- -------
attrs            19.3.0 
Automat          20.2.0 
cffi             1.14.0 
constantly       15.1.0 
cryptography     2.9.2  
cssselect        1.1.0  
hyperlink        19.0.0 
idna             2.9    
incremental      17.5.0 
lxml             4.5.0  
parsel           1.5.2  
pip              20.0.2 
Protego          0.1.16 
pyasn1           0.4.8  
pyasn1-modules   0.2.8  
pycparser        2.20   
PyDispatcher     2.0.5  
PyHamcrest       2.0.2  
pyOpenSSL        19.1.0 
queuelib         1.5.0  
Scrapy           2.1.0  
service-identity 18.1.0 
setuptools       46.1.3 
six              1.14.0 
Twisted          20.3.0 
w3lib            1.21.0 
wheel            0.34.2 
zope.interface   5.1.0  

(venv) $ pip freeze 
attrs==19.3.0
Automat==20.2.0
cffi==1.14.0
constantly==15.1.0
cryptography==2.9.2
cssselect==1.1.0
hyperlink==19.0.0
idna==2.9
incremental==17.5.0
lxml==4.5.0
parsel==1.5.2
Protego==0.1.16
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
PyDispatcher==2.0.5
PyHamcrest==2.0.2
pyOpenSSL==19.1.0
queuelib==1.5.0
Scrapy==2.1.0
service-identity==18.1.0
six==1.14.0
Twisted==20.3.0
w3lib==1.21.0
zope.interface==5.1.0


Verification:

(venv) $ scrapy
Scrapy 2.1.0 - no active project

Usage:
  scrapy <command> [options] [args]

Available commands:
  bench         Run quick benchmark test
  fetch         Fetch a URL using the Scrapy downloader
  genspider     Generate new spider using pre-defined templates
  runspider     Run a self-contained spider (without creating a project)
  settings      Get settings values
  shell         Interactive scraping console
  startproject  Create new project
  version       Print Scrapy version
  view          Open URL in browser, as seen by Scrapy

  [ more ]      More commands available when run from project directory

Use "scrapy <command> -h" to see more info about a command

(venv) $ scrapy bench
2020-04-25 23:57:01 [scrapy.utils.log] INFO: Scrapy 2.1.0 started (bot: scrapybot)
2020-04-25 23:57:01 [scrapy.utils.log] INFO: Versions: lxml 4.5.0.0, libxml2 2.9.10, cssselect 1.1.0, parsel 1.5.2, w3lib 1.21.0, Twisted 20.3.0, Python 3.6.9 (default, Apr 18 2020, 01:56:04) - [GCC 8.4.0], pyOpenSSL 19.1.0 (OpenSSL 1.1.1g  21 Apr 2020), cryptography 2.9.2, Platform Linux-4.15.0-96-generic-x86_64-with-Ubuntu-18.04-bionic
2020-04-25 23:57:01 [scrapy.crawler] INFO: Overridden settings:
{'CLOSESPIDER_TIMEOUT': 10, 'LOGSTATS_INTERVAL': 1, 'LOG_LEVEL': 'INFO'}
2020-04-25 23:57:01 [scrapy.extensions.telnet] INFO: Telnet Password: 6642b95b9fd73c04
2020-04-25 23:57:01 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.closespider.CloseSpider',
 'scrapy.extensions.logstats.LogStats']
2020-04-25 23:57:01 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy.downloadermiddlewares.stats.DownloaderStats']
2020-04-25 23:57:01 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2020-04-25 23:57:01 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2020-04-25 23:57:01 [scrapy.core.engine] INFO: Spider opened
2020-04-25 23:57:01 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:01 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-04-25 23:57:03 [scrapy.extensions.logstats] INFO: Crawled 93 pages (at 5580 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:03 [scrapy.extensions.logstats] INFO: Crawled 157 pages (at 3840 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:04 [scrapy.extensions.logstats] INFO: Crawled 230 pages (at 4380 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:06 [scrapy.extensions.logstats] INFO: Crawled 301 pages (at 4260 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:07 [scrapy.extensions.logstats] INFO: Crawled 365 pages (at 3840 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:08 [scrapy.extensions.logstats] INFO: Crawled 422 pages (at 3420 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:09 [scrapy.extensions.logstats] INFO: Crawled 485 pages (at 3780 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:10 [scrapy.extensions.logstats] INFO: Crawled 541 pages (at 3360 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:11 [scrapy.extensions.logstats] INFO: Crawled 597 pages (at 3360 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:12 [scrapy.core.engine] INFO: Closing spider (closespider_timeout)
2020-04-25 23:57:12 [scrapy.extensions.logstats] INFO: Crawled 646 pages (at 2940 pages/min), scraped 0 items (at 0 items/min)
2020-04-25 23:57:12 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 301439,
 'downloader/request_count': 661,
 'downloader/request_method_count/GET': 661,
 'downloader/response_bytes': 2092067,
 'downloader/response_count': 661,
 'downloader/response_status_count/200': 661,
 'elapsed_time_seconds': 10.596581,
 'finish_reason': 'closespider_timeout',
 'finish_time': datetime.datetime(2020, 4, 25, 22, 57, 12, 551514),
 'log_count/INFO': 20,
 'memusage/max': 55128064,
 'memusage/startup': 55128064,
 'request_depth_max': 22,
 'response_received_count': 661,
 'scheduler/dequeued': 661,
 'scheduler/dequeued/memory': 661,
 'scheduler/enqueued': 13220,
 'scheduler/enqueued/memory': 13220,
 'start_time': datetime.datetime(2020, 4, 25, 22, 57, 1, 954933)}

2020-04-25 23:57:12 [scrapy.core.engine] INFO: Spider closed (closespider_timeout)