Thursday, 18 April 2019

Http Client in Node.js

Making HTTP(S) calls from the application is very common requirement. Node.JS world offers lots of options about how to implement it, or better to say - which package to use.

Native solution


Node.JS comes with native http and https packages.


Request package


It does not support Promises so if we need to wrap it into a Promise ourselves...something like:

/**
 * Function which fetches http response asynchronously.
 * It is implemented as a simple Promise wrapper around request API.
 * @param url Request URL.
 */
function fetchData(url: string): Promise<string> {
    return new Promise((resolve, reject) => {
        if (!url) {
            reject(ErrorFactory.createInvalidArgumentError("url"));
        }

        request(url, (error: any, response: request.Response, body: any) => {
            if (error) {
                reject(error);
            } else {
                if (response) {
                    console.log("statusCode: ", response.statusCode);
                    if (body) {
                        resolve(body);
                    } else {
                        reject(ErrorFactory.createFalsyObjectError("body"));
                    }
                } else {
                    reject(ErrorFactory.createFalsyObjectError("response"));
                }
            }
        });
    });
}


Request package with native promises



node-fetch package


window.fetch API for Node.js 

Axios package


Promise based HTTP client for the browser and node.js

Some References



Conclusion


I've been looking solutions which support Promises. node-fetch seems to be traditional approach and Axios seems to be very popular among developers now.

Posted by at No comments:
Labels: , ,

Friday, 12 April 2019

Introduction to JavaScript

Here are some notes & links I used on my quest of learning JavaScript. You won't find anything spectacular here :)

JS in general


http://blog.thefirehoseproject.com/posts/exactly-makes-javascript-weird-programming-language/
https://www.crockford.com/javascript/javascript.html

Coding convention

http://www.crockford.com/javascript/code.html

When to use and when not semicolons?

https://stackoverflow.com/questions/444080/do-you-recommend-using-semicolons-after-every-statement-in-javascript
https://hackernoon.com/an-open-letter-to-javascript-leaders-regarding-no-semicolons-82cec422d67d
https://stackoverflow.com/questions/8528557/why-doesnt-a-javascript-return-statement-work-when-the-return-value-is-on-a-new

What are the rules for JavaScript's automatic semicolon insertion (ASI)?

https://stackoverflow.com/questions/2846283/what-are-the-rules-for-javascripts-automatic-semicolon-insertion-asi

Memory Management


Memory management model

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Memory_Management

Causes of memory leaks

https://auth0.com/blog/four-types-of-leaks-in-your-javascript-code-and-how-to-get-rid-of-them/
https://blog.sessionstack.com/how-javascript-works-memory-management-how-to-handle-4-common-memory-leaks-3f28b94cfbec

How to free up memory?

https://stackoverflow.com/questions/8467350/how-to-free-up-the-memory-in-javascript

JS in Front-End


What is DOM?

https://www.quora.com/What-is-meant-by-Document-Object-Model-and-how-does-it-works

Web page lifecycle

https://javascript.info/onload-ondomcontentloaded

How to check which events on the page/window have handlers attached?

https://github.com/deanoemcke/thegreatsuspender/issues/272
When does DOMContentLoaded fire? What does its handler usually do?

https://www.quora.com/What-does-it-mean-exactly-that-the-DOM-Document-Object-Model-is-ready
https://api.jquery.com/ready/
https://eager.io/blog/how-to-decide-when-your-code-should-run/

What is the vanilla JS equivalent of '$(document).ready()'?

https://stackoverflow.com/questions/2304941/what-is-the-non-jquery-equivalent-of-document-ready

When does window.onload fire? What does its handler usually do?

https://stackoverflow.com/questions/799981/document-ready-equivalent-without-jquery
https://api.jquery.com/ready/

How to prevent web page from closing?

https://www.maki-chan.de/preventclose.htm

window.onbeforeunload = function() { return "Would you really like to close your browser?"; }

Threading system


Is JS single or multi-threaded language and why?
How does event system work if it’s single-threaded?
Reentrancy in JavaScript


Statements 


for...in


  • for iterating over object properties

for (const prop in obj) {
  if (obj.hasOwnProperty(prop)) {
    console.log(`obj.${prop} = ${obj[prop]}`);
  } 
}

for..in (MDN)


for...of

  • for iterating over arrays



Variables

Assigning and passing to functions by value and by reference

https://hackernoon.com/grasp-by-value-and-by-reference-in-javascript-7ed75efa1293

Is JavaScript a pass-by-reference or pass-by-value language?

It's always pass by value, but for objects the value of the variable is a reference. Because of this, when you pass an object and change its members, those changes persist outside of the function.

Objects

Built-in objects

undefined


https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/undefined

Error

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Error


How to re-throw the error?


https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/throw
https://blog.joefallon.net/2018/09/typescript-try-catch-finally-and-custom-errors/.
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/try...catch


https://developer.mozilla.org/en-US/docs/Web/JavaScript/Enumerability_and_ownership_of_properties
http://zduck.com/2013/non-enumerable-properties-in-javascript/
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/defineProperty

Data types


JavaScript is a weakly typed language, not un-typed language. Primitive types are passed into functions by value, while non-primitives are passed by reference. The typeof operator in JavaScript explicitly informs you of what the type of a variable is (hint is in the name of the operator). There are 5 primitive types (string, number, boolean, undefined, symbol) and two non-primitives (object, function) that are available in user land. [source]
Which 2 types of data types are defined by the latest ECMAScript?

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures

List all primitives.


https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures

null vs unsigned


https://codeburst.io/javascript-null-vs-undefined-20f955215a2

List all 6 falsy values in JS


  • false
  • 0 (zero)
  • “” (empty string)
  • null
  • undefined
  • NaN (Not A Number)


https://codeburst.io/javascript-null-vs-undefined-20f955215a2
JavaScript Showdown: == vs ===

Object


https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object

What is an “object” in computer science?
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Data_structures

Object​.assign()

What are the pros and cons of using object literal notation vs class notation? : javascript

Object literals vs constructors in JavaScript - Internal Pointers

Why Object Literals in JavaScript Are Cool

Object initializer - JavaScript | MDN


Object.hasOwnProperty()


Object.hasOwnProperty determines if the whole property is defined in the object itself or in the prototype chain.
Object.keys returns an array of string where its items are the own properties from a given object 
When you're using for (var key in obj) it will loop through the given object + its parent objects' properties on the prototype chain until it reaches the end of the chain. As you want to check only specific object's properties, you need to use hasOwnProperty.
This is not needed in for (var i = 0; i < length; i++) or data.forEach()


Inheritance and the prototype chain - JavaScript | MDN

Object.getPrototypeOf() - JavaScript | MDN

Why is JavaScript prototype property undefined on new objects? - Stack Overflow

How do I enumerate the properties of a JavaScript object? - Stack Overflow

object - Javascript what is property in hasOwnProperty? - Stack Overflow

 

Object.keys()

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/keys


Computed property name syntax (ES6)

It's a shorthand for the someObject[someKey] assignment (from ES3/5):

var a = "b"
var c = {[a]: "d"}

is same as:

var a = "b"
var c = {}
c[a] = "d"

Square Brackets Javascript Object Key


Object.values()

What is the difference between Object.values() and for..in?
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Object/values


Remove a property in an object immutably

const {prop1, prop2, ...outputObject} = originalObject;

String


In JavaScript everything is an object (or may at least be treated as an object), except primitives (booleans, null, numbers, strings and the value undefined (and symbol in ES6)).

In JavaScript strings can be literals or objects.

Why does instanceof return false for some literals?

Primitives are a different kind of type than objects created from within Javascript.
Literal is a primitive => Literal is NOT an object!


template string


https://stackoverflow.com/questions/19105009/how-to-insert-variables-in-javascript-strings

How to make template string substitution in runtime?

https://stackoverflow.com/questions/30003353/can-es6-template-literals-be-substituted-at-runtime-or-reused
split()
https://medium.com/@jeanpan/javascript-splice-slice-split-745b1c1c05d2
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/split

Array


How to find if array contains some element which matches some criteria?
https://www.w3schools.com/jsref/jsref_find.asp

jQuery - What happens if some operation is executed on the empty array?
e.g. what happens if find returns empty array in:
$footer_ip.find(".user--ip, .virtual--ip").addClass("animated fadeInUp");
https://stackoverflow.com/questions/5705067/what-happens-when-a-jquery-selector-wasnt-found

How to create an array?
https://stackoverflow.com/questions/1629539/use-of-square-brackets-around-javascript-variables

How to redefine Array.toString function?
https://stackoverflow.com/questions/1629539/use-of-square-brackets-around-javascript-variables

Simplify your JavaScript – Use .map(), .reduce(), and .filter()

.map()

  • takes 2 arguments:
    • a callback
    • an optional context (will be considered as this in the callback)
  • callback runs for each value in the array and returns each new value in the resulting array
  • resulting array will always be the same length as the original array
  • Whenever you see .forEach in your code, think about using .map


.reduce()

  • runs a callback for each element of an array
  • passes the result of this callback (the accumulator) from one array element to the other
  • accumulator can be pretty much anything (integer, string, object, etc.) and must be instantiated or passed when calling .reduce()
  • .reduce() is an easy way to generate a single value or object from an array

.filter()

  • if the callback function returns true, the current element will be in the resulting array. If it returns false, it won’t be.


How to remove an element from an array?
"delete" should not be used on arrays

How do I remove a property from a JavaScript object?
How to remove a property from a JavaScript object
3 Ways to clone objects in Javascript
Copying Objects in JavaScript
How to deep clone a JavaScript object
What is the most efficient way to deep clone an object in JavaScript?
How do I correctly clone a JavaScript object?
Clone a js object except for one key
How to handle immutability in JavaScript

Its Lodash features that JS doesn't have (or sometimes needs polyfills for), that you should definitely use when needed. Examples of that are object deep clone and object difference - two of the most important Lodash functionalities for me, personally. [ Lodash and usefulness of utility libraries ]



Hoisting


https://www.sitepoint.com/back-to-basics-javascript-hoisting/

Expressions and operators


Loose equality operator (==)

It tests for strict equality between two values. Both the type and the value you’re comparing have to be exactly the same.

https://codeburst.io/javascript-null-vs-undefined-20f955215a2

Double Equals(==) vs. Triple Equals(===)

Triple equals tests for loose equality and performs type coercion.

https://codeburst.io/javascript-double-equals-vs-triple-equals-61d4ce5a121a
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Comparison_Operators
When is it OK to use == in JavaScript?

Destructuring assignment


https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Destructuring_assignment

[a, b, ...rest] = [10, 20, 30, 40, 50];


Spread syntax

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Spread_syntax
https://zendev.com/2018/05/09/understanding-spread-operator-in-javascript.html
https://stackoverflow.com/questions/11704267/in-javascript-how-to-conditionally-add-a-member-to-an-object


Short-circuiting

https://codeburst.io/javascript-what-is-short-circuit-evaluation-ff22b2f5608c


Null Coalescing Operator

let a = b || c;

Regardless of the type of the first operand, if casting it to a Boolean results in false (like for null, undefined, 0, ""...but not for "false", "undefined", "null", "0", "empty", "deleted"... as they are all true since they are non-empty strings.), the assignment will use the second operand.

Is there a “null coalescing” operator in JavaScript?


The || operator in JavaScript doesn't necessarily return true or false. It's exact behavior is this:
If the first operand is truthy, it evaluates to the first operand. Otherwise, it evaluates to the second.

Why does (false || null) return null, while (null || false) returns false?



Functions

Built-in (global) functions


parseInt - what is the purpose of radix argument?
https://www.w3schools.com/jsref/jsref_parseint.asp
https://davidwalsh.name/parseint-radix
https://stackoverflow.com/questions/6611824/why-do-we-need-to-use-radix
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/parseInt
https://eloquentjavascript.net/03_functions.html

How to verify that parsing some string as integer was successful?
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/parseInt

Function.prototype.bind()


https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Function/bind

Arrow functions

Arrow functions which return a single value don't need braces or parenthesis at all:

The exception is the case when it returns an empty object:

const f = () => {}; 
// f returns an empty object


This is same as:

const f = () => ({})

or

const f = () => { return {}; }

As shown above, if we want to use curly braces, we need to use return keyword.

http://2ality.com/2012/04/arrow-functions.html
Arrow function without curly braces

Self-Executing Anonymous Functions

http://markdalgleish.com/2011/03/self-executing-anonymous-functions/

Anonymous self-invoking function (often called "module's side effect") will run when the module is evaluated, which happens when module is imported at least once in some other module (by importing any of its exported variables, functions or classes). [Importing a self invoking function]
[Code Example]

I learned this hard way. Had a (Jest) unit test file which was supposed to test some function exported from some other file/module which also contained anonymous self-invoking function. Unit test file was importing named function but each time I'd run tests, this anonymous function would be executed.

Named Arguments



Cool Javascript 9: Named arguments — Functions that get and return Objects
This helped me to understand the code which omits some properties while cloning an object [source]:

const obj = {a: 1, b: 2, c: 3, d: 4};
const clone = (({b, c, ...others}) => ({...others}))(obj); // remove b and c

This solution puts on display the beauty of JavaScript. Function declared and called in same line. Object passed as an argument is deconstructed to match the object set as argument in function definition. Spread operator. No need for return statement...I am impressed!

How to get a subset of a javascript object's properties

Pure Functions


Pure functions are functions that accept an input and returns a value without modifying any data outside its scope(Side Effects). Its output or return value must depend on the input/arguments and pure functions must return a value.

Immutability comes when we want to preserve our state. To keep our state from changing we have to create a new instance of our state objects. Immutability makes our app state predictable, ups the performance rate of our apps and to easily track changes in state. [ Understanding Javascript Mutation and Pure Functions ]

JavaScript’s object arguments are references, which means that if a function were to mutate a property on an object or array parameter, that would mutate state that is accessible outside the function. Pure functions must not mutate external state. [ Master the JavaScript Interview: What is a Pure Function? ]

Reassigning parameters deoptimizes in many engines, specifically v8 - it's a horrible idea to do it. Variables are free, and creating new ones rather than reusing old ones makes code much clearer.
(In addition, nothing in JS is passed by reference, everything is passed by value, where objects are a form of "reference value" - this is a good read on the subject) [Thoughts on "Never mutate parameters"]

To mutate, or not to mutate, in JavaScript
Don’t change objects in functions
Do not change objects after construction.

Never mutate parameters. 

Never reassign parameters.


Closures


https://developer.mozilla.org/en-US/docs/Web/JavaScript/Closures

Events


Is addEventListener (using listeners) preferred over attaching a handler?

https://www.reddit.com/r/learnjavascript/comments/6erpv2/whats_the_difference_between_using_an_event/
https://teamtreehouse.com/community/addeventlistener-vs-things-like-onclick-onchange-etc-pure-javascript

What is the correct attaching a handler (if go down this route)?
How to avoid wrong double handler execution?
https://stackoverflow.com/questions/7794301/window-onunload-is-not-working-properly-in-chrome-browser-can-any-one-help-me
Describe event phases
(capturing, bubbling...)
https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Examples#Example_5:_Event_Propagation

jQuery - Why returning false from (click) event handler?
https://stackoverflow.com/questions/11184276/return-false-from-jquery-click-event

Event.stopPropagation()
https://developer.mozilla.org/en-US/docs/Web/API/Event/stopPropagation

Promises


http://www.ecma-international.org/ecma-262/6.0/#sec-promise-objects
https://italonascimento.github.io/applying-a-timeout-to-your-promises/
https://html.spec.whatwg.org/multipage/timers-and-user-prompts.html#list-of-active-timers
https://stackoverflow.com/questions/31324110/why-does-the-promise-constructor-require-a-function-that-calls-resolve-when-co
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/then
https://javascript.info/promise-chaining
JavaScript: async/await with forEach()

Should a Promise.reject message be wrapped in Error?
https://stackoverflow.com/questions/26020578/should-a-promise-reject-message-be-wrapped-in-error
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise/reject#Using_the_static_Promise.reject_method
https://stackoverflow.com/questions/26711243/promise-resolve-vs-new-promiseresolve

resolve & reject

`resolve` and `reject` functions inside a Promise's executor

async-await

https://javascript.info/async-await

http://2ality.com/2016/10/async-function-tips.html

(Especially pay attention to point #3 - Returned Promises are not wrapped - which explains when and why await can be omitted)

JavaScript loops - how to handle async/await
JavaScript: async/await with forEach()


RegEx


Comes very handy when need to test string arguments. Instead of lengthy code which would use string functions, testing a string to match some format is a one-liner with RegEx:

test: ({ app_name, app_version }) =>
   /myapp/i.test(app_name) && /^12\.3/.test(app_version)

RegExp.prototype.test()


What does the forward slash mean within a JavaScript regular expression?
The slashes indicate the start and end of the regular expression.


Regular expressions have four optional flags that allow for global and case insensitive searching. 
  • To indicate a global search, use the g flag. 
  • To indicate a case-insensitive search, use the i flag. 
  • To indicate a multi-line search, use the m flag. 
  • To perform a "sticky" search, that matches starting at the current position in the target string, use the y flag. 

These flags can be used separately or together in any order, and are included as part of the regular expression.

To include a flag with the regular expression, use this syntax:

var re = /pattern/flags;



Web APIs


WindowOrWorkerGlobalScope

setTimeout()

https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout

How to set text in a <span> element?

https://stackoverflow.com/questions/1358810/how-do-i-change-the-text-of-a-span-element-in-javascript#

querySelector vs querySelectorAll

https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelector
https://developer.mozilla.org/en-US/docs/Web/API/Document/querySelectorAll
https://stackoverflow.com/questions/14377590/queryselector-and-queryselectorall-vs-getelementsbyclassname-and-getelementbyid

When to use (set) innerHTML and when innerText?
tbd...

Misc How Tos


How to declare a constant data member in a class?
https://stackoverflow.com/questions/32647215/declaring-static-constants-in-es6-classes

Best Practices

Airbnb JavaScript Style Guide

JavaScript Rules

https://www.w3schools.com/js/js_best_practices.asp

Check that there is no unnecessary console.log which goes into production.
Check that there are no unused variables.
Check return values of functions (is it null, undefined, empty array, error code etc...)

Make sure of the time order of events and functions...e.g.

document.addEventListener("DOMContentLoaded", function() {..xxx...});

If xxx is never executed but you can see that document has event handler for DOMContentLoaded event that means that the document.addEventListener("DOMContentLoaded" code was executed too late, after DOMContentLoaded event has already been fired!

var btn = document.querySelectorAll("a.btn")[0];

Use querySelector.

I prefer:

var btn = document.querySelector("a.btn");
btn.addEventListener('click', function(e) {
   const url = "secure://extensions/?id=dmfdacibleoapmpfdgonigdfinmekhgp";
   chrome.tabs.create({ url: url });
});

to:

var btn = document.querySelector("a.btn");
if (btn !== null) {
   btn.addEventListener('click', function(e) {
      const url = "secure://extensions/?id=dmfdacibleoapmpfdgonigdfinmekhgp";
      chrome.tabs.create({ url: url });
   });
} else {
   console.log(“WARNING: a.btn element not found.”);
}

as console.log message would probably be unnoticed in the log while exception thrown in case btn is null would be marked in red in console log.

use _xxx for private member variables

use no special naming for global variables because try to avoid globals outside of application constants

Consts write as MY_CONST.

This function has a lot of side effects on the DOM and global multimap. Not necessarily a problem, although the function name doesn't communicate this. A nicer way may be restructure the app to have a function that returns a state object and then hands this off to a function which generates/mutates the DOM. It's a more "functional" way of doing it anyway! Personal preference.

when you say "state object", do you mean data (e.g. optimalGateway and multimap) in this case?

I mean an object representing all data within the application. Granted, this is quite a Redux/React way of thinking and is harder to implement with a bunch of jquery, so it may not be appropriate. Still, if I'm calling a function called loadDataFromBackgroundPage at the top level with no return value, it is very unclear that is going to have side effects on the DOM. I would expect it to have a return value of that data, and no side effects. Then that data could be handed off to another function to handle the rendering.

$() equivalent to $(document).ready()

$(handler) is now preferred to $(document).ready(handler) (https://api.jquery.com/ready/)

I have two similar handlers for $(document).click in this code so I'm going to merge them.

Function name capitalisation consistency.

selectedGatewayId = vpnExtensionEngine.getGateways().find(g => g.city.name + ", " + g.country.name === selectedLocation).id;

If find returns undefined, we'll get an error here. Might be worth adding a check for this or catching the error.

All parseInt calls should specify the base. e.g. parseInt(unixTimestampString, 10)

(https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/parseInt confirms this)

Another DOM ready handler. Can we condense these in to one?

var url = "https://extension.avastbrowser.com/vpn/about/";

Move to const?

replace var with let wherever applicable.

Instead of looping infinitely and throwing, can we adjust this condition?

That's certainly better approach as exception is used here to control the flow of execution (and not to signal exceptional behaviour).

use setTimeout vs setInterval and clearing it immediately in the callback
Callback has to be executed only once, after timeout so it makes more sense to use setTimeout

const serverApiVersionMajor = lightVpnControllerApiVersion.split(".")[0];
const serverApiVersionMinor = lightVpnControllerApiVersion.split(".")[1];

Array destructuring supported in Chrome 49+, so could rewrite these two lines as:

const [serverApiVersionMajor, serverApiVersionMinor] = lightVpnControllerApiVersion.split(".")

Arrow functions are used elsewhere, so could use one here as well for compactness. Personal preference. Also, you wouldn't need the .bind(this) as the arrow function doesn't create a closure.

Chrome 42+ supports the fetch API. More compact and returns a promise, so don't need to wrap it like this. https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/fetch

General tips:
https://news.ycombinator.com/item?id=17466391

Stack Overflow:
https://stackoverflow.com/questions/tagged/javascript


Packages


https://socket.io/
http://tesseract.projectnaptha.com/

Cool Tips and Tricks


How to print the name of the JS file currently loaded in html?

console.log('Loading script: ' + document.currentScript.src);

https://stackoverflow.com/questions/22095529/hiding-everything-until-the-page-has-finished-loading?noredirect=1&lq=1
https://en.wikipedia.org/wiki/Flash_of_unstyled_content
https://en.wikipedia.org/wiki/Screen_reader
https://stackoverflow.com/questions/2690865/visibility-attribute-question

How to add a line break to the string in messages.json?

https://phraseapp.com/docs/guides/formats/chrome-json/
https://github.com/angular-translate/angular-translate/issues/595
https://developer.mozilla.org/en-US/docs/Mozilla/Localization/Localization_content_best_practices
Where should I put <script> tags in HTML markup?
Remove Render-Blocking JavaScript
JavaScript Where To

String literals:

console.log(‘Hello, world!’)

or

console.log("Hello, world!")

What is lexical scope?
What is "this"?
What is Lexical Scope Anyway?
How do I write a named arrow function in ES2015?

Why is the content of some js files wrapped inside a function which is wrapped inside anonymous self-executed function?

(function () {...})();

What is the purpose of wrapping whole Javascript files in anonymous functions like “(function(){ … })()”?
Self-Executing Anonymous Functions
Immediately Invoked Function Expression (IIFE)
What does “use strict” do in JavaScript, and what is the reasoning behind it?
Should I 'use strict' for every single javascript function I write?
Not recommended to use “use strict” in ES6?
Strict Mode
What is this Javascript “require”?
What is require?
Requiring modules in Node.js: Everything you need to know
Commonly accepted best practices around code organization in JavaScript
ECMA-262, 9th edition, June 2018


What is the difference between String.slice and String.substring?

Create a JavaScript array containing 1…N
Tasks, microtasks, queues and schedules
https://codeburst.io/javascript-quickie-dot-notation-vs-bracket-notation-333641c0f781
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Operator_Precedence
Posted by at No comments:
Labels: ,

Wednesday, 10 April 2019

Introduction to RegEx

RegEx is used often when working with strings, paths, configurations etc...so here is a little breakdown of commonly used RegEx expressions. I will be adding examples I come across in my daily work and down below I'll be adding explanations on how to interpret them.

RegEx examples:


'/.*?\\.(test|spec)\\.js$'


RegEx Special Characters (Metacharacters)


\d = matches any single digit in most regex grammar styles and is equivalent to [0-9]
\t = match tab

[ ] = Square brackets are characters that help define a character set, which is a set of characters we want to match.
  • [aeiou] matches all vowels. 

Some characters have special meanings inside square brackets:
  • caret (^) before any other character inside square brackets means we’re negating the characters that follow the caret. We’re telling the regex engine not to match those characters. 
    • [^aeiou] matches all characters apart from vowels.
  • hyphen (-) between two characters inside square brackets means range. [a-z] means "match all characters between a and z, inclusive"
If we want to match square brackets, we need to escape them with backslash. \[|\] means "match [ or ]".

RegEx Expressions & Interpretation:


.
Dot matches any single character except the newline character, by default.
If s flag ("dotAll") is true, it also matches newline characters. If we want to match a dot character itself, we need to escape it: \.


*
This quantifier (asterisk) matches the preceding expression 0 or more (unlimited) times, as many times as possible, giving back as needed (greedy)

Example: Find any text between two digits OR a single digit:

"\\d(.*\\d)*"

 In string LeadingText-1-TrailingText found pattern 1
In string LeadingText-12-TrailingText found pattern 12
In string LeadingText-1.2-TrailingText found pattern 1.2
In string LeadingText-11.2-TrailingText found pattern 11.2
In string LeadingText-1.22-TrailingText found pattern 1.22
In string LeadingText-11.22-TrailingText found pattern 11.22
In string LeadingText-1234-TrailingText found pattern 1234 

.*
Matches any character greedily - as many characters as possible.

Example:
1.*1 in 101000001 will match 101000001

?
Matches the preceding expression 0 or 1 time.
If used immediately after any of the quantifiers *, +, ?, or {}, makes the quantifier non-greedy (matching the fewest possible characters), as opposed to the default, which is greedy (matching as many characters as possible).

.*?
Matches any character in non-greedy mode - as little as enough to match the pattern.

Example:
1.*1 in 101000001 will match 101

What is the difference between .*? and .* regular expressions?
(this answer also contains nice explanation of backtracking and how non-greedy expression can return multiple matches within a string)


+
This quantifier matches the preceding expression 1 or more (unlimited) times, as many times as possible, giving back as needed (greedy)

\
A backslash that precedes a non-special character indicates that the next character is special and is not to be interpreted literally.
A backslash that precedes a special character indicates that the next character is not special and should be interpreted literally (this is called escaping).
Example: \. matches the character . literally (case sensitive)

\\
The first backslash escapes the one after it, so the expression searches for a single literal backslash.

^
Caret.
  • ^ means "not the following" when inside and at the start of [], so [^...].
  • When it's inside [] but not at the start, it means the actual ^ character.
  • When it's escaped (\^), it also means the actual ^ character.
  • In all other cases it means start of the string or line (which one is language / setting dependent). If a caret (^) is at the beginning of the entire regular expression, it matches the beginning of a line.
Examples:
  • [^abc] -> not a, b or c
  • [ab^cd] -> a, b, ^ (character), c or d
  • \^ -> a ^ character
  • Anywhere else -> start of string / line.
  • ^[ \t]* = match all spaces and tabs at the beginning of line

^[b-d]t$ means:
  • Start of line
  • b/c/d character
  • t character
  • End of line

Carets in Regular Expressions


$

If a dollar sign ($) is at the end of the entire regular expression, it matches the end of a line.

If an entire regular expression is enclosed by a caret and dollar sign (^lorem ipusm$), it matches an entire line.

Capturing Groups


Part of a pattern can be enclosed in parentheses (...). This is called a capturing group.
Multiple characters in that group are treated as a single unit that we want to match.
It allows to get a part of the match as a separate item in the result array.
If we put a quantifier after the parentheses, it applies to the parentheses as a whole.


String: abababa 
Goal: find all matches of sequence ab. 
Result: There are 3 matches. 
Regex: (ab)

String: ab123cd345ef785
Goal: find all sequences of numbers
Result: 123, 345, 785
Regex: (\d+)

String: abc345-1.23.456.7890+whatever.ext
Goal: extract only numbers which form a valid version number (greedy - M.m.r.b or  M.m.r or M.m )
Result: 1.23.456.7890

Regex (\d+) returns 5 groups: 345, 1, 23, 456, 7890
Regex (\d+)\. returns all groups of numbers that are followed by dot. There are 3 such groups: 1, 23 and 456.

Let's look at some examples:

1.23
11.23
123.45
1.23.456
1.23.456.7890

We can see that all version numbers:
  • start with a sequence of 1 or more digits which are followed by dot: \d+\.
  • end with a sequence of 1 or more digits: \d+
So far we have: \d+\.\d+
Regex \d+\.\d+ returns 2 groups: 1.23 and 456.7890

Between these two sequences can be 0 or more (max 2 but let's ignore this) sequences of 1 or more digits that are followed by dot: \d+\.
This sequence is optional so let's put it in brackets that sequence to form a group and append * to it:
Regex  \d+\.(\d+\.)*\d+ does match 1.23.456.7890 but as (...) is capturing group it captures it and result is a single group: 456.

Here we just want regex to match this group but not to capture it (not to return it in results). We want this group to be a non-capturing group and there is a special syntax for it: (?: ... ).

Regex \d+\.(?:\d+\.)*\d+ fully matches 1.23.456.7890


Capturing group can be used to match characters between delimiters but excluding delimiters. 

E.g. extract file name from: https://raw.example.com/aws/path/to/karpenter.sh_nodeclaims.yaml

/[^/]*$ captures leading /, any character apart from / (0 or more times), before the end of the string:
/karpenter.sh_nodeclaims.yaml

If we use capturing group, we can capture anything between the last / and the end of the string:
/([^/]*)$ matches /karpenter.sh_nodeclaims.yaml and a capturing group karpenter.sh_nodeclaims.yaml. Capturing group match is the desired file name.



Online Regex tools





Posted by at No comments:
Labels:

Tuesday, 9 April 2019

Introduction to Node.js

How to check Node location and version?


(!) They can be different for normal and sudo/root user: 

$ which node
/home/bojan/.nvm/versions/node/v14.1.0/bin/node

 $ /home/bojan/.nvm/versions/node/v14.1.0/bin/node -v
v14.1.0

$ node -v
v14.1.0

 $ sudo which node
/usr/bin/node

 $ /usr/bin/node -v
v8.10.0

 $ sudo node -v
v8.10.0


In this case running a node application with sudo can make application fail with unexpected errors. Always make sure your code is compatible with the version of Node you want to run it against.

How to check all Node versions installed locally?


$ nvm list
       v11.13.0
       v11.14.0
->      v14.1.0
         system
default -> 11.13.0 (-> v11.13.0)
node -> stable (-> v14.1.0) (default)
stable -> 14.1 (-> v14.1.0) (default)
iojs -> N/A (default)
unstable -> N/A (default)
lts/* -> lts/erbium (-> N/A)
lts/argon -> v4.9.1 (-> N/A)
lts/boron -> v6.17.1 (-> N/A)
lts/carbon -> v8.17.0 (-> N/A)
lts/dubnium -> v10.20.1 (-> N/A)
lts/erbium -> v12.16.3 (-> N/A)

How to check Node versions available to install from remote repository?


$ nvm ls-remote
        v0.1.14
        v0.1.15
        ...
        ...
       v13.12.0
       v13.13.0
       v13.14.0
        v14.0.0
->      v14.1.0

How to install some particular version of Node?


This can be done even if the same user has already installed some other Node version.
It can be useful to have different Node versions installed when we need to debug our app for compatibility issues with some particular Node version.

$ nvm install 13.14.0
Downloading and installing node v13.14.0...
Downloading https://nodejs.org/dist/v13.14.0/node-v13.14.0-linux-x64.tar.xz...
####################################################################################################################################################################### 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v13.14.0 (npm v6.14.4)

How to use in terminal some particular version of Node?


Open a new terminal (shell) and execute:

$ nvm use node 13.14.0
Now using node v13.14.0 (npm v6.14.4)

 $ node -v 
v13.14.0


This is cool, isn't it? :-)

NVM - Install and Manage Multiple Node.js Versions in Linux



First program: Hello, world!


Let's write our first Node.js program which outputs a string onto the terminal:

helloworld.js:

console.log(‘Hello, world!’)

Output:

>node helloworld.js
Hello, world!

Let's now see how Node handles erroneous code:

helloworld.js:

console.log(Hello, world!)

Output:

C:\wherever\HelloWorld\helloworld.js:1

 (function (exports, require, module, __filename, __dirname) { console.log(Hello, world!)
                                                                                                                        ^^^^^
SyntaxError: missing ) after argument list
   at createScript (vm.js:56:10)
   at Object.runInThisContext (vm.js:97:10)
   at Module._compile (module.js:542:28)
   at Object.Module._extensions..js (module.js:579:10)
   at Module.load (module.js:487:32)
   at tryModuleLoad (module.js:446:12)
   at Function.Module._load (module.js:438:3)
   at Module.runMain (module.js:604:10)
   at run (bootstrap_node.js:383:7)
   at startup (bootstrap_node.js:149:9)

Introducing dependencies


There are many Node packages around and the chances are that your Node application will use (depend on) some of them. To manage these dependencies we'll use Node Package Manager (npm). All dependencies will be listed in a file called package.json which is located at project's root directory. This file will also contain some other information about the application and so it will be a kind of an application manifest. To create it (which is usually one of the first things to do when setting up a Node project), we call npm init. For example:

>npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (rollup-js-demo)
version: (1.0.0)
description: Application which demoes Rollup.JS basic features.
entry point: (index.js) main.js
test command: echo \"Error: no test specified\" && exit 1
git repository: (https://github.com/BojanKomazec/rollup-js-demo.git)
keywords: rollup.js, demo
author: Bojan Komazec
license: (ISC)
About to write to C:\wherever\rollup-js-demo\package.json:

{
  "name": "rollup-js-demo",
  "version": "1.0.0",
  "description": "Application which demoes Rollup.JS basic features.",
  "main": "main.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/BojanKomazec/rollup-js-demo.git"
  },
  "keywords": [
    "rollup.js",
    "demo"
  ],
  "author": "Bojan Komazec",
  "license": "ISC",
  "bugs": {
    "url": "https://github.com/BojanKomazec/rollup-js-demo/issues"
  },
  "homepage": "https://github.com/BojanKomazec/rollup-js-demo#readme"
}

Is this OK? (yes)

Running test command indeed echoes message specified:

>npm test

> rollup-js-demo@1.0.0 test C:\wherever\rollup-js-demo
> echo "Error: no test specified" && exit 1

"Error: no test specified"
npm ERR! Test failed.  See above for more details.

What is the test command while creating package.json?
How to properly use “keywords” property in package.json?

If we want to execute npm init without prompting us to customize configutation, we can execute:

> npm init --yes

or shorter:

> npm init -y

Here is the full list of npm init options:

$ npm init --help

 npm init [--force|-f|--yes|-y|--scope]
npm init <@scope> (same as `npx <@scope>/create`)
npm init [<@scope>/]<name> (same as `npx [<@scope>/]create-<name>`)

 aliases: create, innit

If we leave description field empty, npm install will issue a warning like:

npm WARN my-app@1.0.0 No description

Installing dependencies


Dependencies (Node packages) are installed with npm install command:

$ npm install --help

 npm install (with no args, in package dir)
npm install [<@scope>/]<pkg>
npm install [<@scope>/]<pkg>@<tag>
npm install [<@scope>/]<pkg>@<version>
npm install [<@scope>/]<pkg>@<version range>
npm install <folder>
npm install <tarball file>
npm install <tarball url>
npm install <git:// url>
npm install <github username>/<github project>

 aliases: i, isntall, add
common options: [--save-prod|--save-dev|--save-optional] [--save-exact] [--no-save]

To see detailed help use:

$ npm help install


 npm install
- reads list of dependencies from packages.json, creates a local node_modules directory (if not created) and installs there all dependencies (and dependencies of dependencies).
- this command is usually used upon cloning the app's repository

npm install package_name
- Installs package_name locally
- new package appears within dependencies property in packages.json
- If not created, it creates a local node_modules directory and installs there specified package (and its dependencies)
- As node_modules directory contains binaries, it shall be added to local .gitignore file
- this command is usually used during development, when we want to add a new dependency

npm install -g 
- installs dependency globally.

"I have a personal philosophy of not installing npm modules as global unless absolutely needed." [link]

npm install --save-dev
- installs dependency locally and as a development dependency (within "devDependencies" property in packages.json; otherwise it would be in "dependencies"). If --production option is used then installing all development dependencies is omitted.

In the next example we want to install unit testing framework which should only be a development dependency of our application:

$ npm install --save-dev jest

dependencies or devDependencies?

If you install some package as dev dependency and then want to move it to become runtime dependency (to move a module from devDependencies to dependencies), use this:

$ npm install <module_name> --save-prod

Move a module from devDependencies to dependencies in npm package.json

To move a module from dependencies to devDependencies:

$ npm install <module_name> --save-dev

or shorter:

$ npm i <module_name> -D



Updating dependencies


To update both development and production dependencies, use npm update. Example:

$ npm update
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

 + @types/jest@24.0.15
+ tslib@1.10.0
+ axios@0.18.1
+ tslint@5.18.0
+ jest@24.8.0
+ @types/node@11.13.17
+ typescript@3.5.3
added 19 packages from 44 contributors, removed 32 packages, updated 100 packages and audited 873808 packages in 20.39s
found 0 vulnerabilities

 We can use git diff to verify that versions for both dependency types have indeed been updated in package.json:
 $ git diff package.json
diff --git a/package.json b/package.json
index 801557e..b516f83 100644
--- a/package.json
+++ b/package.json
@@ -17,17 +17,17 @@
   "author": "",
   "license": "ISC",
   "devDependencies": {
-    "@types/jest": "^24.0.11",
-    "@types/node": "^11.13.2",
-    "jest": "^24.7.1",
+    "@types/jest": "^24.0.15",
+    "@types/node": "^11.13.17",
+    "jest": "^24.8.0",
     "ts-jest": "^24.0.2",
-    "tslib": "^1.9.3",
-    "tslint": "^5.15.0",
-    "typescript": "^3.4.2"
+    "tslib": "^1.10.0",
+    "tslint": "^5.18.0",
+    "typescript": "^3.5.3"
   },
   "dependencies": {
     "@types/flat": "0.0.28",
-    "axios": "^0.18.0",
+    "axios": "^0.18.1",
     "dotenv": "^7.0.0",
     "flat": "^4.1.0",
     "pg": "^7.11.0"

---


How to run another process from Node app?


Use child_process module.

Example:

async function createTables(): Promise<void> {
    return new Promise((resolve, reject) => {
        const { spawn } = require("child_process");
        const schema2db = spawn("python", ["deps/schema2db/schema2db.py"]);

         schema2db.stdout.on("data", (data: any) => {
            console.log(`stdout: ${data}`);
        });

         schema2db.stderr.on("data", (data: any) => {
            console.log(`stderr: ${data}`);
        });

         schema2db.on("close", (code: any) => {
            console.log(`child process exited with code ${code}`);
            if (code === 0) {
                resolve(code);
            } else {
                reject(code);
            }
        });
    });
}

It is assumed that example project has a dependency - a Python app named schema2db, nested under deps/schema2db/.


How to interact with PostgreSQL DB?


Use pg module.

Examples:

async function testDbConnection() {
        const { Client } = require("pg");
        const client = new Client();
        await client.connect();
        const res = await client.query("SELECT $1::text as message", ["Hello world!"]);
        console.log(res.rows[0].message); // Hello world!
        await client.end();
}

 async function createTable(createTableQuery: string) {
        const { Client } = require("pg");
        const client = new Client();
        await client.connect();
        await client.query(createTableQuery);
        const { rows } = await client.query('SELECT * FROM users')
        console.log(rows)
        await client.end();
}

 async function populateTable(insertTableQuery: string, posts: any[]) {
    const { Client } = require("pg");
    const client = new Client();
    await client.connect();
    for (const post of posts) {
        await client.query(insertTableQuery, Object.values(post));
    }
    await client.end();
}

Misc


 Should I add node_modules under version control?
No
http://blog.gvm-it.eu/post/20404719601/getting-started-with-nodejs-on-windows
In Node.js, how do I “include” functions from my other files?
https://nodejs.org/api/esm.html

How to edit .npmrc on OSX?

$ cat ~/.npmrc
registry=http://some.domain.com:5678
$ vi ~/.npmrc <-- add ; at the beginning of the line to comment it
$ cat ~/.npmrc
;registry=http://some.domain.com:5678

How to update node on OSX?

$ sudo npm install -g n
Password:
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
/usr/local/lib
└── n@2.1.12 
$ sudo n latest

     install : node-v10.11.0
       mkdir : /usr/local/n/versions/node/10.11.0
       fetch : https://nodejs.org/dist/v10.11.0/node-v10.11.0-darwin-x64.tar.gz
######################################################################################################################################################### 100.0%
   installed : v10.11.0


API

path.join [Node.js path module].
node how to create a directory if doesn't exist?

child_process module - to spawn another process


Asynchronous programming


To call async function in main module wrap the call in anonymous self-invoking async function:

(async () => {
  await fooAsync();
})();


How Node.Js Single Thread mechanism Work ? Understanding Event Loop in NodeJs
The Node.js Event Loop, Timers, and process.nextTick()

Node.js Async Function Best Practices
How to keep a node.js script alive while promises are being resolved?
Node exits without error and doesn't await promise (Event callback)
Promise prevent node process from exiting
How does a node.js process know when to stop?
Promise keep node process from exit
How can I use async/await at the top level?


Node.js Project Structure

Entry point (main file) is usually named index.js.
Node Hero - Node.js Project Structure Tutorial

Modules


require


Requiring modules in Node.js: Everything you need to know

The module.exports object in every module is what the require function returns when we require that module. 

You Can Use require() To Load JSON (JavaScript Object Notation) Files In Node.js

var config = require("./config")

If we omit the file extension and Node.js will look for a .js file first and then, if not found, look for a .json file.

Normally, when referencing a file with require() a relative path is used. This path must reflect the position of the current file within your site's directory structure. 
[Absolute paths & require()]

Node require absolute path
How to make node.js require absolute? (instead of relative)
Conditional require in express?

require vs import


From difference between require and import ? 

The main difference between require and import is that import is always run at the very beginning of the file (if you look at Babel's output, they get hoisted to the top), and can't be run conditionally. This means that you can guarantee all the imports have completed before any of your code runs. Whereas require can be used inline, conditionally, etc.

 imports get sorted to the top of the file, and requires stay where they were put. So the run-order only changes with import.

From: Webpack: Import vs Require, and why
In order to take advantage of tree shaking in your application, you need to keep a few things in mind.
First, make sure you use ES2015 import and export statements in your code wherever possible.

From: An Update on ES6 Modules in Node.js
It’s important to keep in mind that all import and export statements are resolved to their targets before any code is actually evaluated. It is also important to note that the ES6 specification allows this resolution step to occur asynchronously. In Node.js terms, this means loading the contents of the script, resolution of the module imports and exports, and evaluation of the module code would occur over multiple turns of the event loop.

Application Configuration

Configuration Files 

How to store Node.js deployment settings/configuration files?
Node.js Best Practices — Smarter Ways to Manage Config Files and Variables

Environment Variables

How to set Environment variables from within package.json [Node.js]
Working with Environment Variables in Node.js
Node, the difference between development and production

Logging


Before all, read this: XI. Logs - Treat logs as event streams

Docker can be such production environment as it's capable of capturing stdin/stderr and forwarding it to e.g. AWS or Google Cloud logging. Docker logging drivers; HOW TO REDIRECT DOCKER LOGS TO A SINGLE FILE

Node.js Logging Tutorial

Common levels of logging in Node.js:

  • error
  • warn
  • info
  • verbose
  • debug

Log levels “error” and “warn” will go to stderr when called from the console.
Log levels “info” or “log” will both go to stdout from the console functions.
Log levels “debug” and “verbose” are where you’ll send detailed information.

Debug level can contain things like stack traces, input and output parameters, or special messages for developers.
Verbose should include every event that happened, but not necessarily as much detail as debug. Debug is the highest level in this hierarchy.

console.log and console.info will log to stdout.
Console.warn and console.error will log to stderr.
By default, these will print the output to your console.

To redirect stdout and stderr to a file do this redirect when you launch your application:

node app.js > app.log 2>&1

This redirects stdout to a file named app.log and redirects stderr to stdout.



How to Get Node.js Logging Right
4 Node.js Logging libraries which make sophisticated logging simpler

debug (npm package)
ulog (npm package)
chalk
log4js (npm package)
Node-Loggly
Bunyan

Winston


Morgan

What JS Logging library / tool do you use?  (forum discussion)

console.log()
console.error()

node app.js > app.log



Deployment



Best practice for nodejs deployment - Directly moving node_modules to server or run npm install command


Running Node.JS in Docker

How To Build a Node.js Application with Docker
Docker and Node.js Best Practices
Selecting A Node.js Image for Docker

If you specify “FROM node” without a version number tag, you will always be given the latest version of Node.js. While this may sound like a great thing, at first, you will quickly run into headaches when you rebuild your image and find your application doesn’t run because a new release of Node.js no longer supports an API you’re calling or makes other changes.

 One option is to use the latest LTS (long term support) version of Node available from the Docker Hub. Go to https://nodejs.org/en/ and take version described as LTS (or "Recommended for most users"). Obviously, your app has to support it.

I recommend to put the app in /home/node/app instead of root (e.g. /app) to avoid permission issues. [source]

you should place your app in a folder that is owned by the executing user or writable, e.g. /home/node/webapp. [source]

Dockerfile:

FROM node

# Create app directory
WORKDIR /usr/src/app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./

RUN npm install
# If you are building your code for production
# RUN npm ci --only=production

# Bundle app source
COPY . .

# EXPOSE 8080
CMD [ "node", "app.js" ]

.dockerignore:

node_modules
npm-debug.log



Building Docker image which resolves npm packages from custom domain

By default npm uses https://registry.npmjs.org registry to find and fetch packages (npm install). To set some custom npm package registry we can add to the project's root .npmrc file with following content:

registry=https://my.example.com/what/ever/npm

npm install will try to fetch packages from there. If npm install is in Dockerfile then this means that this custom domain has be resolvable by Docker daemon during docker build.

I had a case where npm install (with custom registry set in .npmrc) run during docker build would fail with the following error:

Step 7/20 : RUN npm install --loglevel ${NPM_LOG_LEVEL}
 ---> Running in f3bde2ab9d02
npm info it worked if it ends with ok
npm verb cli [
npm verb cli   '/usr/local/bin/node',
npm verb cli   '/usr/local/bin/npm',
npm verb cli   'install',
npm verb cli   '--loglevel',
npm verb cli   'verbose'
npm verb cli ]
npm info using npm@6.12.0
npm info using node@v13.0.1
npm verb npm-session 244bb6c393f2097a
npm info lifecycle my-app@1.0.0~preinstall: my-app@1.0.0
npm timing stage:loadCurrentTree Completed in 9ms
npm timing stage:loadIdealTree:cloneCurrentTree Completed in 1ms
npm timing stage:loadIdealTree:loadShrinkwrap Completed in 355ms
npm timing stage:loadIdealTree:loadAllDepsIntoIdealTree Completed in 22ms
npm timing stage:loadIdealTree Completed in 555ms
npm timing stage:generateActionsToTake Completed in 211ms
npm verb correctMkdir /root/.npm/_locks correctMkdir not in flight; initializing
npm verb lock using /root/.npm/_locks/staging-62530c0ac0331caf.lock for /usr/local/my-app/node_modules/.staging
npm timing action:extract Completed in 1218ms
npm verb unlock done using /root/.npm/_locks/staging-62530c0ac0331caf.lock for /usr/local/my-app/node_modules/.staging
npm timing stage:rollbackFailedOptional Completed in 1ms
npm timing stage:runTopLevelLifecycles Completed in 3414ms
npm verb stack RangeError: Maximum call stack size exceeded
npm verb stack     at RegExp.test (<anonymous>)
npm verb stack     at isDepOptional (/usr/local/lib/node_modules/npm/lib/install/deps.js:423:45)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:432:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack     at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb cwd /usr/local/my-app
npm verb Linux 4.15.0-65-generic
npm verb argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "--loglevel" "verbose"
npm verb node v13.0.1
npm verb npm  v6.12.0
npm ERR! Maximum call stack size exceeded
npm verb exit [ 1, true ]
npm timing npm Completed in 4583ms

 npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-10-31T16_26_16_246Z-debug.log
ERROR: The command 'npm install --loglevel ${NPM_LOG_LEVEL}' returned a non-zero code: 1

.npmrc:

registry=https://my.example.com/what/ever/npm/

 # reset auth related options if specified in ~/.npmrc file
_auth=""
always-auth=false

Dockerfile:

# installing dependencies
FROM my.example.com/node:alpine as node_cache
WORKDIR /usr/local/my-app
COPY package.json ./
# COPY package-lock.json ./
COPY .npmrc ./
# NPM_LOG_LEVEL = [silent|warn|info|silly|verbose]
ARG NPM_LOG_LEVEL=verbose
RUN npm install --loglevel ${NPM_LOG_LEVEL}

I tried to do the following to no avail:

  • npm rebuild
  • omit copying package-lock.json (and so forcing its re-creation)
  • setting authentication in .npmrc:
    • always-auth=true 
    • _auth was set to the output of curl -u<USERNAME>:<PASSWORD> https://my.example.com/what/ever/npm/auth) 
Problem would not be present if I'd comment this line in Dockerfile:

COPY .npmrc ./  

I switched log level to silly and now could see more log output and that the issue is in that npm can't resolve custom npm registry domain:

npm sill fetchPackageMetaData error for pg@^7.11.0 request to https://my.example.com/what/ever/npm/pg failed, reason: getaddrinfo ENOTFOUND my.example.com
npm timing stage:rollbackFailedOptional Completed in 1ms
npm timing stage:runTopLevelLifecycles Completed in 334ms
npm sill saveTree my-app@1.0.0
npm verb type system
npm verb stack FetchError: request to https://my.example.com/what/ever/npm/@types%2fflat failed, reason: getaddrinfo ENOTFOUND my.example.com
npm verb stack     at ClientRequest.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-fetch-npm/src/index.js:68:14)
npm verb stack     at ClientRequest.emit (events.js:210:5)
npm verb stack     at TLSSocket.socketErrorListener (_http_client.js:407:9)
npm verb stack     at TLSSocket.emit (events.js:210:5)
npm verb stack     at emitErrorNT (internal/streams/destroy.js:84:8)
npm verb stack     at processTicksAndRejections (internal/process/task_queues.js:80:21)
npm verb cwd /usr/local/my-app
npm verb Linux 4.15.0-65-generic
npm verb argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "--loglevel" "silly"
npm verb node v13.0.1
npm verb npm  v6.12.0
npm ERR! code ENOTFOUND
npm ERR! errno ENOTFOUND

npm install worked fine when I run npm install on my dev machine directly so I knew this was definitely a Docker DNS resolution issue.

Temp solution was to hardcode the resolved IP address for the problematic domain:

RUN echo x.y.z.q my.example.com >> /etc/hosts && npm install --loglevel ${NPM_LOG_LEVEL}

From here:
  • in Dockerfile, edit /etc/resolv.conf to point to your custom DNS
  • at end of build operations that require such custom DNS, restore the normal container runtime resolv.conf; this is done automatically as /etc/resolv.conf is intentionally not persisted to the image (intermediate container)

This is unreliable solution as it would break each time domain's IP address changes. Plus we'd need to do this for any other unresolved domain.

Proper solutions are described here: Fix Docker's networking DNS config and here. I opted for simply using --network=host in docker build:

$ docker build --network=host --pull --build-arg NPM_LOG_LEVEL=silly -t my-app .


Network calls fail during image build on corporate network


By the way these were some useful outputs:

$ cat /etc/resolv.conf
nameserver 127.0.0.53
options edns0
search my.example.com

 $ docker run busybox nslookup google.com
Server:         8.8.8.8
Address:        8.8.8.8:53

 Non-authoritative answer:
Name:   google.com
Address: 172.217.169.14

 *** Can't find google.com: No answer
$ docker run --network=host busybox nslookup google.com 
Server:         127.0.0.53
Address:        127.0.0.53:53

 Non-authoritative answer:
Name:   google.com
Address: 172.217.169.14

 *** Can't find google.com: No answer


At some other occasion I experienced similar error when building a Docker image for running Node.js application. npm install was failing with:

npm ERR! Maximum call stack size exceeded

And this was happening only if I'd set in .npmrc a custom npm registry which sits on corporate network.

Using --network=host (even when on corporate network) didn't help.
The docker build uses the DNS settings of the docker engine running on the host so I tried setting DNS servers for Docker daemon and that helped.


When on VPN, I checked which DNS servers are set:

$ cat /etc/resolv.conf
#@VPNC_GENERATED@ -- this file is generated by vpnc
# and will be overwritten by vpnc
# as long as the above mark is intact
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.

 options edns0
nameserver A.B.C.D
nameserver E.F.G.H
search Home company.com

 /etc/docker/daemon.json did not exist so I had to create it:

$ sudo touch /etc/docker/daemon.json
$ sudo gedit /etc/docker/daemon.json

and edit it as here:

{
  "dns": ["A.B.C.D", "E.F.G.H"],
  "dns-search": ["company.com"]
}

I then restarted Docker service:

$ systemctl restart docker

After this npm install worked fine.





References & Further Reading


node how to create a directory if doesn't exist?
Check synchronously if file/directory exists in Node.js
Writing files in Node.js
How to append to a file in Node?
Writing JSON object to a JSON file with fs.writeFileSync
How to get file size in Node.js
Difference between a module and a package in Node?
Create an empty file in Node.js?
What is the purpose of Node.js module.exports and how do you use it?
In Node.js, how do I “include” functions from my other files?
How to exit in Node.js
Node.js Development Tips



Posted by at No comments:
Labels: , , , ,
Subscribe to: Posts (Atom)