Tuesday, 9 April 2019

Introduction to Node.js

First program: Hello, world!


Let's write our first Node.js program which outputs a string onto the terminal:

helloworld.js:

console.log(‘Hello, world!’)

Output:

>node helloworld.js
Hello, world!

Let's now see how Node handles erroneous code:

helloworld.js:

console.log(Hello, world!)

Output:

C:\wherever\HelloWorld\helloworld.js:1

(function (exports, require, module, __filename, __dirname) { console.log(Hello, world!)
                                                                                                                        ^^^^^
SyntaxError: missing ) after argument list
   at createScript (vm.js:56:10)
   at Object.runInThisContext (vm.js:97:10)
   at Module._compile (module.js:542:28)
   at Object.Module._extensions..js (module.js:579:10)
   at Module.load (module.js:487:32)
   at tryModuleLoad (module.js:446:12)
   at Function.Module._load (module.js:438:3)
   at Module.runMain (module.js:604:10)
   at run (bootstrap_node.js:383:7)
   at startup (bootstrap_node.js:149:9)

Introducing dependencies


There are many Node packages around and the chances are that your Node application will use (depend on) some of them. To manage these dependencies we'll use Node Package Manager (npm). All dependencies will be listed in a file called package.json which is located at project's root directory. This file will also contain some other information about the application and so it will be a kind of an application manifest. To create it (which is usually one of the first things to do when setting up a Node project), we call npm init. For example:

>npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.

See `npm help json` for definitive documentation on these fields
and exactly what they do.

Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.

Press ^C at any time to quit.
package name: (rollup-js-demo)
version: (1.0.0)
description: Application which demoes Rollup.JS basic features.
entry point: (index.js) main.js
test command: echo \"Error: no test specified\" && exit 1
git repository: (https://github.com/BojanKomazec/rollup-js-demo.git)
keywords: rollup.js, demo
author: Bojan Komazec
license: (ISC)
About to write to C:\wherever\rollup-js-demo\package.json:

{
  "name": "rollup-js-demo",
  "version": "1.0.0",
  "description": "Application which demoes Rollup.JS basic features.",
  "main": "main.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "repository": {
    "type": "git",
    "url": "git+https://github.com/BojanKomazec/rollup-js-demo.git"
  },
  "keywords": [
    "rollup.js",
    "demo"
  ],
  "author": "Bojan Komazec",
  "license": "ISC",
  "bugs": {
    "url": "https://github.com/BojanKomazec/rollup-js-demo/issues"
  },
  "homepage": "https://github.com/BojanKomazec/rollup-js-demo#readme"
}

Is this OK? (yes)

Running test command indeed echoes message specified:

>npm test

> rollup-js-demo@1.0.0 test C:\wherever\rollup-js-demo
> echo "Error: no test specified" && exit 1

"Error: no test specified"
npm ERR! Test failed.  See above for more details.

What is the test command while creating package.json?
How to properly use “keywords” property in package.json?

If we want to execute npm init without prompting us to customize configutation, we can execute:

> npm init --yes

or shorter:

> npm init -y

Here is the full list of npm init options:

$ npm init --help

npm init [--force|-f|--yes|-y|--scope]
npm init <@scope> (same as `npx <@scope>/create`)
npm init [<@scope>/]<name> (same as `npx [<@scope>/]create-<name>`)

aliases: create, innit

Installing dependencies


Dependencies (Node packages) are installed with npm install command:

$ npm install --help

npm install (with no args, in package dir)
npm install [<@scope>/]<pkg>
npm install [<@scope>/]<pkg>@<tag>
npm install [<@scope>/]<pkg>@<version>
npm install [<@scope>/]<pkg>@<version range>
npm install <folder>
npm install <tarball file>
npm install <tarball url>
npm install <git:// url>
npm install <github username>/<github project>

aliases: i, isntall, add
common options: [--save-prod|--save-dev|--save-optional] [--save-exact] [--no-save]

To see detailed help use:

$ npm help install


npm install
- reads list of dependencies from packages.json, creates a local node_modules directory (if not created) and installs there all dependencies (and dependencies of dependencies).
- this command is usually used upon cloning the app's repository

npm install package_name
- Installs package_name locally
- new package appears within dependencies property in packages.json
- If not created, it creates a local node_modules directory and installs there specified package (and its dependencies)
- As node_modules directory contains binaries, it shall be added to local .gitignore file
- this command is usually used during development, when we want to add a new dependency

npm install -g 
- installs dependency globally.

"I have a personal philosophy of not installing npm modules as global unless absolutely needed." [link]

npm install --save-dev
- installs dependency locally and as a development dependency (within "devDependencies" property in packages.json; otherwise it would be in "dependencies"). If --production option is used then installing all development dependencies is omitted.

In the next example we want to install unit testing framework which should only be a development dependency of our application:

$ npm install --save-dev jest

dependencies or devDependencies?

If you install some package as dev dependency and then want to move it to become runtime dependency (to move a module from devDependencies to dependencies), use this:

$ npm install <module_name> --save-prod

Move a module from devDependencies to dependencies in npm package.json

To move a module from dependencies to devDependencies:

$ npm install <module_name> --save-dev

or shorter:

$ npm i <module_name> -D



Updating dependencies


To update both development and production dependencies, use npm update. Example:

$ npm update
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})

+ @types/jest@24.0.15
+ tslib@1.10.0
+ axios@0.18.1
+ tslint@5.18.0
+ jest@24.8.0
+ @types/node@11.13.17
+ typescript@3.5.3
added 19 packages from 44 contributors, removed 32 packages, updated 100 packages and audited 873808 packages in 20.39s
found 0 vulnerabilities

We can use git diff to verify that versions for both dependency types have indeed been updated in package.json:
$ git diff package.json
diff --git a/package.json b/package.json
index 801557e..b516f83 100644
--- a/package.json
+++ b/package.json
@@ -17,17 +17,17 @@
   "author": "",
   "license": "ISC",
   "devDependencies": {
-    "@types/jest": "^24.0.11",
-    "@types/node": "^11.13.2",
-    "jest": "^24.7.1",
+    "@types/jest": "^24.0.15",
+    "@types/node": "^11.13.17",
+    "jest": "^24.8.0",
     "ts-jest": "^24.0.2",
-    "tslib": "^1.9.3",
-    "tslint": "^5.15.0",
-    "typescript": "^3.4.2"
+    "tslib": "^1.10.0",
+    "tslint": "^5.18.0",
+    "typescript": "^3.5.3"
   },
   "dependencies": {
     "@types/flat": "0.0.28",
-    "axios": "^0.18.0",
+    "axios": "^0.18.1",
     "dotenv": "^7.0.0",
     "flat": "^4.1.0",
     "pg": "^7.11.0"

---


How to run another process from Node app?


Use child_process module.

Example:

async function createTables(): Promise<void> {
    return new Promise((resolve, reject) => {
        const { spawn } = require("child_process");
        const schema2db = spawn("python", ["deps/schema2db/schema2db.py"]);

        schema2db.stdout.on("data", (data: any) => {
            console.log(`stdout: ${data}`);
        });

        schema2db.stderr.on("data", (data: any) => {
            console.log(`stderr: ${data}`);
        });

        schema2db.on("close", (code: any) => {
            console.log(`child process exited with code ${code}`);
            if (code === 0) {
                resolve(code);
            } else {
                reject(code);
            }
        });
    });
}

It is assumed that example project has a dependency - a Python app named schema2db, nested under deps/schema2db/.


How to interact with PostgreSQL DB?


Use pg module.

Examples:

async function testDbConnection() {
        const { Client } = require("pg");
        const client = new Client();
        await client.connect();
        const res = await client.query("SELECT $1::text as message", ["Hello world!"]);
        console.log(res.rows[0].message); // Hello world!
        await client.end();
}

async function createTable(createTableQuery: string) {
        const { Client } = require("pg");
        const client = new Client();
        await client.connect();
        await client.query(createTableQuery);
        const { rows } = await client.query('SELECT * FROM users')
        console.log(rows)
        await client.end();
}

async function populateTable(insertTableQuery: string, posts: any[]) {
    const { Client } = require("pg");
    const client = new Client();
    await client.connect();
    for (const post of posts) {
        await client.query(insertTableQuery, Object.values(post));
    }
    await client.end();
}

Misc


Should I add node_modules under version control?
No
http://blog.gvm-it.eu/post/20404719601/getting-started-with-nodejs-on-windows
In Node.js, how do I “include” functions from my other files?
https://nodejs.org/api/esm.html

How to edit .npmrc on OSX?

$ cat ~/.npmrc
registry=http://some.domain.com:5678
$ vi ~/.npmrc <-- add ; at the beginning of the line to comment it
$ cat ~/.npmrc
;registry=http://some.domain.com:5678

How to update node on OSX?

$ sudo npm install -g n
Password:
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
/usr/local/lib
└── n@2.1.12 
$ sudo n latest

     install : node-v10.11.0
       mkdir : /usr/local/n/versions/node/10.11.0
       fetch : https://nodejs.org/dist/v10.11.0/node-v10.11.0-darwin-x64.tar.gz
######################################################################################################################################################### 100.0%
   installed : v10.11.0


API

path.join [Node.js path module].
node how to create a directory if doesn't exist?

child_process module - to spawn another process


Asynchronous programming


To call async function in main module wrap the call in anonymous self-invoking async function:

(async () => {
  await fooAsync();
})();


How Node.Js Single Thread mechanism Work ? Understanding Event Loop in NodeJs
The Node.js Event Loop, Timers, and process.nextTick()

Node.js Async Function Best Practices
How to keep a node.js script alive while promises are being resolved?
Node exits without error and doesn't await promise (Event callback)
Promise prevent node process from exiting
How does a node.js process know when to stop?
Promise keep node process from exit
How can I use async/await at the top level?


Node.js Project Structure

Entry point (main file) is usually named index.js.
Node Hero - Node.js Project Structure Tutorial

Modules


require


Requiring modules in Node.js: Everything you need to know

The module.exports object in every module is what the require function returns when we require that module. 

You Can Use require() To Load JSON (JavaScript Object Notation) Files In Node.js

var config = require("./config")

If we omit the file extension and Node.js will look for a .js file first and then, if not found, look for a .json file.

Normally, when referencing a file with require() a relative path is used. This path must reflect the position of the current file within your site's directory structure. 
[Absolute paths & require()]

Node require absolute path
How to make node.js require absolute? (instead of relative)
Conditional require in express?

require vs import


From difference between require and import ? 

The main difference between require and import is that import is always run at the very beginning of the file (if you look at Babel's output, they get hoisted to the top), and can't be run conditionally. This means that you can guarantee all the imports have completed before any of your code runs. Whereas require can be used inline, conditionally, etc.

imports get sorted to the top of the file, and requires stay where they were put. So the run-order only changes with import.

From: Webpack: Import vs Require, and why
In order to take advantage of tree shaking in your application, you need to keep a few things in mind.
First, make sure you use ES2015 import and export statements in your code wherever possible.

From: An Update on ES6 Modules in Node.js
It’s important to keep in mind that all import and export statements are resolved to their targets before any code is actually evaluated. It is also important to note that the ES6 specification allows this resolution step to occur asynchronously. In Node.js terms, this means loading the contents of the script, resolution of the module imports and exports, and evaluation of the module code would occur over multiple turns of the event loop.

Application Configuration

Configuration Files 

How to store Node.js deployment settings/configuration files?
Node.js Best Practices — Smarter Ways to Manage Config Files and Variables

Environment Variables

How to set Environment variables from within package.json [Node.js]
Working with Environment Variables in Node.js
Node, the difference between development and production

Logging


Before all, read this: XI. Logs - Treat logs as event streams

Docker can be such production environment as it's capable of capturing stdin/stderr and forwarding it to e.g. AWS or Google Cloud logging. Docker logging driversHOW TO REDIRECT DOCKER LOGS TO A SINGLE FILE

Node.js Logging Tutorial

Common levels of logging in Node.js:

  • error
  • warn
  • info
  • verbose
  • debug

Log levels “error” and “warn” will go to stderr when called from the console.
Log levels “info” or “log” will both go to stdout from the console functions.
Log levels “debug” and “verbose” are where you’ll send detailed information.

Debug level can contain things like stack traces, input and output parameters, or special messages for developers.
Verbose should include every event that happened, but not necessarily as much detail as debug. Debug is the highest level in this hierarchy.

console.log and console.info will log to stdout.
Console.warn and console.error will log to stderr.
By default, these will print the output to your console.

To redirect stdout and stderr to a file do this redirect when you launch your application:

node app.js > app.log 2>&1

This redirects stdout to a file named app.log and redirects stderr to stdout.



How to Get Node.js Logging Right
4 Node.js Logging libraries which make sophisticated logging simpler

debug (npm package)
ulog (npm package)
chalk
log4js (npm package)
Node-Loggly
Bunyan

Winston


Morgan

What JS Logging library / tool do you use?  (forum discussion)

console.log()
console.error()

node app.js > app.log



Deployment



Best practice for nodejs deployment - Directly moving node_modules to server or run npm install command


Running Node.JS in Docker

How To Build a Node.js Application with Docker
Docker and Node.js Best Practices
Selecting A Node.js Image for Docker

If you specify “FROM node” without a version number tag, you will always be given the latest version of Node.js. While this may sound like a great thing, at first, you will quickly run into headaches when you rebuild your image and find your application doesn’t run because a new release of Node.js no longer supports an API you’re calling or makes other changes.

One option is to use the latest LTS (long term support) version of Node available from the Docker Hub. Go to https://nodejs.org/en/ and take version described as LTS (or "Recommended for most users"). Obviously, your app has to support it.

I recommend to put the app in /home/node/app instead of root (e.g. /app) to avoid permission issues. [source]

you should place your app in a folder that is owned by the executing user or writable, e.g. /home/node/webapp. [source]

Dockerfile:

FROM node

# Create app directory
WORKDIR /usr/src/app

# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./

RUN npm install
# If you are building your code for production
# RUN npm ci --only=production

# Bundle app source
COPY . .

# EXPOSE 8080
CMD [ "node", "app.js" ]

.dockerignore:

node_modules
npm-debug.log




References & Further Reading


node how to create a directory if doesn't exist?
Check synchronously if file/directory exists in Node.js
Writing files in Node.js
How to append to a file in Node?
Writing JSON object to a JSON file with fs.writeFileSync
How to get file size in Node.js
Difference between a module and a package in Node?
Create an empty file in Node.js?
What is the purpose of Node.js module.exports and how do you use it?
In Node.js, how do I “include” functions from my other files?
How to exit in Node.js
Node.js Development Tips



No comments: