How to check Node location and version?
(!) They can be different for normal and sudo/root user:
/home/bojan/.nvm/versions/node/v14.1.0/bin/node
$ /home/bojan/.nvm/versions/node/v14.1.0/bin/node -v
v14.1.0
v14.1.0
$ sudo which node
/usr/bin/node
$ /usr/bin/node -v
v8.10.0
$ sudo node -v
v8.10.0
In this case running a node application with sudo can make application fail with unexpected errors. Always make sure your code is compatible with the version of Node you want to run it against.
How to check all Node versions installed locally?
$ nvm list
v11.13.0
v11.14.0
-> v14.1.0
system
default -> 11.13.0 (-> v11.13.0)
node -> stable (-> v14.1.0) (default)
stable -> 14.1 (-> v14.1.0) (default)
iojs -> N/A (default)
unstable -> N/A (default)
lts/* -> lts/erbium (-> N/A)
lts/argon -> v4.9.1 (-> N/A)
lts/boron -> v6.17.1 (-> N/A)
lts/carbon -> v8.17.0 (-> N/A)
lts/dubnium -> v10.20.1 (-> N/A)
lts/erbium -> v12.16.3 (-> N/A)
How to check Node versions available to install from remote repository?
$ nvm ls-remote
v0.1.14
v0.1.15
...
...
v13.12.0
v13.13.0
v13.14.0
v14.0.0
-> v14.1.0
How to install some particular version of Node?
This can be done even if the same user has already installed some other Node version.
It can be useful to have different Node versions installed when we need to debug our app for compatibility issues with some particular Node version.
$ nvm install 13.14.0
Downloading and installing node v13.14.0...
Downloading https://nodejs.org/dist/v13.14.0/node-v13.14.0-linux-x64.tar.xz...
####################################################################################################################################################################### 100.0%
Computing checksum with sha256sum
Checksums matched!
Now using node v13.14.0 (npm v6.14.4)
How to use in terminal some particular version of Node?
Open a new terminal (shell) and execute:
$ nvm use node 13.14.0
Now using node v13.14.0 (npm v6.14.4)
$ node -v
v13.14.0
This is cool, isn't it? :-)
NVM - Install and Manage Multiple Node.js Versions in Linux
First program: Hello, world!
Let's write our first Node.js program which outputs a string onto the terminal:
helloworld.js:
console.log(‘Hello, world!’)
Output:
>node helloworld.js
Hello, world!
Hello, world!
Let's now see how Node handles erroneous code:
helloworld.js:
console.log(Hello, world!)
Output:
C:\wherever\HelloWorld\helloworld.js:1
(function (exports, require, module, __filename, __dirname) { console.log(Hello, world!)
^^^^^
SyntaxError: missing ) after argument list
at createScript (vm.js:56:10)
at Object.runInThisContext (vm.js:97:10)
at Module._compile (module.js:542:28)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.runMain (module.js:604:10)
at run (bootstrap_node.js:383:7)
at startup (bootstrap_node.js:149:9)
(function (exports, require, module, __filename, __dirname) { console.log(Hello, world!)
^^^^^
SyntaxError: missing ) after argument list
at createScript (vm.js:56:10)
at Object.runInThisContext (vm.js:97:10)
at Module._compile (module.js:542:28)
at Object.Module._extensions..js (module.js:579:10)
at Module.load (module.js:487:32)
at tryModuleLoad (module.js:446:12)
at Function.Module._load (module.js:438:3)
at Module.runMain (module.js:604:10)
at run (bootstrap_node.js:383:7)
at startup (bootstrap_node.js:149:9)
Introducing dependencies
There are many Node packages around and the chances are that your Node application will use (depend on) some of them. To manage these dependencies we'll use Node Package Manager (npm). All dependencies will be listed in a file called package.json which is located at project's root directory. This file will also contain some other information about the application and so it will be a kind of an application manifest. To create it (which is usually one of the first things to do when setting up a Node project), we call npm init. For example:
>npm init
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.
See `npm help json` for definitive documentation on these fields
and exactly what they do.
Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.
Press ^C at any time to quit.
package name: (rollup-js-demo)
version: (1.0.0)
description: Application which demoes Rollup.JS basic features.
entry point: (index.js) main.js
test command: echo \"Error: no test specified\" && exit 1
git repository: (https://github.com/BojanKomazec/rollup-js-demo.git)
keywords: rollup.js, demo
author: Bojan Komazec
license: (ISC)
About to write to C:\wherever\rollup-js-demo\package.json:
{
"name": "rollup-js-demo",
"version": "1.0.0",
"description": "Application which demoes Rollup.JS basic features.",
"main": "main.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
"type": "git",
"url": "git+https://github.com/BojanKomazec/rollup-js-demo.git"
},
"keywords": [
"rollup.js",
"demo"
],
"author": "Bojan Komazec",
"license": "ISC",
"bugs": {
"url": "https://github.com/BojanKomazec/rollup-js-demo/issues"
},
"homepage": "https://github.com/BojanKomazec/rollup-js-demo#readme"
}
Is this OK? (yes)
This utility will walk you through creating a package.json file.
It only covers the most common items, and tries to guess sensible defaults.
See `npm help json` for definitive documentation on these fields
and exactly what they do.
Use `npm install <pkg>` afterwards to install a package and
save it as a dependency in the package.json file.
Press ^C at any time to quit.
package name: (rollup-js-demo)
version: (1.0.0)
description: Application which demoes Rollup.JS basic features.
entry point: (index.js) main.js
test command: echo \"Error: no test specified\" && exit 1
git repository: (https://github.com/BojanKomazec/rollup-js-demo.git)
keywords: rollup.js, demo
author: Bojan Komazec
license: (ISC)
About to write to C:\wherever\rollup-js-demo\package.json:
{
"name": "rollup-js-demo",
"version": "1.0.0",
"description": "Application which demoes Rollup.JS basic features.",
"main": "main.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"repository": {
"type": "git",
"url": "git+https://github.com/BojanKomazec/rollup-js-demo.git"
},
"keywords": [
"rollup.js",
"demo"
],
"author": "Bojan Komazec",
"license": "ISC",
"bugs": {
"url": "https://github.com/BojanKomazec/rollup-js-demo/issues"
},
"homepage": "https://github.com/BojanKomazec/rollup-js-demo#readme"
}
Is this OK? (yes)
Running test command indeed echoes message specified:
>npm test
> rollup-js-demo@1.0.0 test C:\wherever\rollup-js-demo
> echo "Error: no test specified" && exit 1
"Error: no test specified"
npm ERR! Test failed. See above for more details.
> rollup-js-demo@1.0.0 test C:\wherever\rollup-js-demo
> echo "Error: no test specified" && exit 1
"Error: no test specified"
npm ERR! Test failed. See above for more details.
How to properly use “keywords” property in package.json?
If we want to execute npm init without prompting us to customize configutation, we can execute:
> npm init --yes
or shorter:
> npm init -y
Here is the full list of npm init options:
$ npm init --help
npm init [--force|-f|--yes|-y|--scope]
npm init <@scope> (same as `npx <@scope>/create`)
npm init [<@scope>/]<name> (same as `npx [<@scope>/]create-<name>`)
aliases: create, innit
If we leave description field empty, npm install will issue a warning like:
npm WARN my-app@1.0.0 No description
Installing dependencies
Dependencies (Node packages) are installed with npm install command:
$ npm install --help
npm install (with no args, in package dir)
npm install [<@scope>/]<pkg>
npm install [<@scope>/]<pkg>@<tag>
npm install [<@scope>/]<pkg>@<version>
npm install [<@scope>/]<pkg>@<version range>
npm install <folder>
npm install <tarball file>
npm install <tarball url>
npm install <git:// url>
npm install <github username>/<github project>
aliases: i, isntall, add
common options: [--save-prod|--save-dev|--save-optional] [--save-exact] [--no-save]
To see detailed help use:
$ npm help install
npm install
- reads list of dependencies from packages.json, creates a local node_modules directory (if not created) and installs there all dependencies (and dependencies of dependencies).
- this command is usually used upon cloning the app's repository
npm install package_name
- Installs package_name locally
- new package appears within dependencies property in packages.json
- If not created, it creates a local node_modules directory and installs there specified package (and its dependencies)
- As node_modules directory contains binaries, it shall be added to local .gitignore file
- this command is usually used during development, when we want to add a new dependency
npm install -g
- installs dependency globally.
"I have a personal philosophy of not installing npm modules as global unless absolutely needed." [link]
npm install --save-dev
- installs dependency locally and as a development dependency (within "devDependencies" property in packages.json; otherwise it would be in "dependencies"). If --production option is used then installing all development dependencies is omitted.
In the next example we want to install unit testing framework which should only be a development dependency of our application:
$ npm install --save-dev jest
dependencies or devDependencies?
If you install some package as dev dependency and then want to move it to become runtime dependency (to move a module from devDependencies to dependencies), use this:
$ npm install <module_name> --save-prod
Move a module from devDependencies to dependencies in npm package.json
To move a module from dependencies to devDependencies:
$ npm install <module_name> --save-dev
or shorter:
$ npm i <module_name> -D
Updating dependencies
To update both development and production dependencies, use npm update. Example:
$ npm update
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@1.2.9 (node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.2.9: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
+ @types/jest@24.0.15
+ tslib@1.10.0
+ axios@0.18.1
+ tslint@5.18.0
+ jest@24.8.0
+ @types/node@11.13.17
+ typescript@3.5.3
added 19 packages from 44 contributors, removed 32 packages, updated 100 packages and audited 873808 packages in 20.39s
found 0 vulnerabilities
We can use git diff to verify that versions for both dependency types have indeed been updated in package.json:
$ git diff package.json
diff --git a/package.json b/package.json
index 801557e..b516f83 100644
--- a/package.json
+++ b/package.json
@@ -17,17 +17,17 @@
"author": "",
"license": "ISC",
"devDependencies": {
- "@types/jest": "^24.0.11",
- "@types/node": "^11.13.2",
- "jest": "^24.7.1",
+ "@types/jest": "^24.0.15",
+ "@types/node": "^11.13.17",
+ "jest": "^24.8.0",
"ts-jest": "^24.0.2",
- "tslib": "^1.9.3",
- "tslint": "^5.15.0",
- "typescript": "^3.4.2"
+ "tslib": "^1.10.0",
+ "tslint": "^5.18.0",
+ "typescript": "^3.5.3"
},
"dependencies": {
"@types/flat": "0.0.28",
- "axios": "^0.18.0",
+ "axios": "^0.18.1",
"dotenv": "^7.0.0",
"flat": "^4.1.0",
"pg": "^7.11.0"
---
How to run another process from Node app?
Use child_process module.
Example:
async function createTables(): Promise<void> {
return new Promise((resolve, reject) => {
const { spawn } = require("child_process");
const schema2db = spawn("python", ["deps/schema2db/schema2db.py"]);
schema2db.stdout.on("data", (data: any) => {
console.log(`stdout: ${data}`);
});
schema2db.stderr.on("data", (data: any) => {
console.log(`stderr: ${data}`);
});
schema2db.on("close", (code: any) => {
console.log(`child process exited with code ${code}`);
if (code === 0) {
resolve(code);
} else {
reject(code);
}
});
});
}
It is assumed that example project has a dependency - a Python app named schema2db, nested under deps/schema2db/.
How to interact with PostgreSQL DB?
Use pg module.
Examples:
async function testDbConnection() {
const { Client } = require("pg");
const client = new Client();
await client.connect();
const res = await client.query("SELECT $1::text as message", ["Hello world!"]);
console.log(res.rows[0].message); // Hello world!
await client.end();
}
async function createTable(createTableQuery: string) {
const { Client } = require("pg");
const client = new Client();
await client.connect();
await client.query(createTableQuery);
const { rows } = await client.query('SELECT * FROM users')
console.log(rows)
await client.end();
}
async function populateTable(insertTableQuery: string, posts: any[]) {
const { Client } = require("pg");
const client = new Client();
await client.connect();
for (const post of posts) {
await client.query(insertTableQuery, Object.values(post));
}
await client.end();
}
Misc
Should I add node_modules under version control?
No
Should I add package-lock.json under version control?
Yes
https://stackoverflow.com/questions/44206782/do-i-commit-the-package-lock-json-file-created-by-npm-5
Your first Node.js package
How to decide when to use Node.js?
In Node.js, how do I “include” functions from my other files?
https://nodejs.org/api/esm.html
How to edit .npmrc on OSX?
$ cat ~/.npmrc
registry=http://some.domain.com:5678
$ vi ~/.npmrc <-- add ; at the beginning of the line to comment it
$ cat ~/.npmrc
;registry=http://some.domain.com:5678
How to update node on OSX?
$ sudo npm install -g n
Password:
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
/usr/local/lib
└── n@2.1.12
registry=http://some.domain.com:5678
$ vi ~/.npmrc <-- add ; at the beginning of the line to comment it
$ cat ~/.npmrc
;registry=http://some.domain.com:5678
How to update node on OSX?
$ sudo npm install -g n
Password:
/usr/local/bin/n -> /usr/local/lib/node_modules/n/bin/n
/usr/local/lib
└── n@2.1.12
$ sudo n latest
install : node-v10.11.0
mkdir : /usr/local/n/versions/node/10.11.0
fetch : https://nodejs.org/dist/v10.11.0/node-v10.11.0-darwin-x64.tar.gz
######################################################################################################################################################### 100.0%
installed : v10.11.0
API
path.join [Node.js path module].node how to create a directory if doesn't exist?
child_process module - to spawn another process
Asynchronous programming
To call async function in main module wrap the call in anonymous self-invoking async function:
(async () => {
await fooAsync();
})();
How Node.Js Single Thread mechanism Work ? Understanding Event Loop in NodeJs
The Node.js Event Loop, Timers, and process.nextTick()
How to keep a node.js script alive while promises are being resolved?
Node exits without error and doesn't await promise (Event callback)
Promise prevent node process from exiting
How does a node.js process know when to stop?
Promise keep node process from exit
How can I use async/await at the top level?
Node.js Project Structure
Entry point (main file) is usually named index.js.Node Hero - Node.js Project Structure Tutorial
Modules
require
Requiring modules in Node.js: Everything you need to know
The module.exports object in every module is what the require function returns when we require that module.
You Can Use require() To Load JSON (JavaScript Object Notation) Files In Node.jsThe module.exports object in every module is what the require function returns when we require that module.
var config = require("./config")
If we omit the file extension and Node.js will look for a .js file first and then, if not found, look for a .json file.
Normally, when referencing a file with require() a relative path is used. This path must reflect the position of the current file within your site's directory structure.
[Absolute paths & require()]
Node require absolute path
How to make node.js require absolute? (instead of relative)
Conditional require in express?
require vs import
From difference between require and import ?
The main difference between require and import is that import is always run at the very beginning of the file (if you look at Babel's output, they get hoisted to the top), and can't be run conditionally. This means that you can guarantee all the imports have completed before any of your code runs. Whereas require can be used inline, conditionally, etc.
imports get sorted to the top of the file, and requires stay where they were put. So the run-order only changes with import.
From: Webpack: Import vs Require, and why
In order to take advantage of tree shaking in your application, you need to keep a few things in mind.
First, make sure you use ES2015 import and export statements in your code wherever possible.
From: An Update on ES6 Modules in Node.js
It’s important to keep in mind that all import and export statements are resolved to their targets before any code is actually evaluated. It is also important to note that the ES6 specification allows this resolution step to occur asynchronously. In Node.js terms, this means loading the contents of the script, resolution of the module imports and exports, and evaluation of the module code would occur over multiple turns of the event loop.
Application Configuration
Configuration Files
How to store Node.js deployment settings/configuration files?Node.js Best Practices — Smarter Ways to Manage Config Files and Variables
Environment Variables
How to set Environment variables from within package.json [Node.js]Working with Environment Variables in Node.js
Node, the difference between development and production
Node.js Everywhere with Environment Variables!
What is NODE_ENV in Express?
What You Should Know About NODE_ENV
dotenv
What is NODE_ENV in Express?
What You Should Know About NODE_ENV
dotenv
Logging
Before all, read this: XI. Logs - Treat logs as event streams
Docker can be such production environment as it's capable of capturing stdin/stderr and forwarding it to e.g. AWS or Google Cloud logging. Docker logging drivers; HOW TO REDIRECT DOCKER LOGS TO A SINGLE FILE
Node.js Logging Tutorial
Common levels of logging in Node.js:
- error
- warn
- info
- verbose
- debug
Log levels “error” and “warn” will go to stderr when called from the console.
Log levels “info” or “log” will both go to stdout from the console functions.
Log levels “debug” and “verbose” are where you’ll send detailed information.
Debug level can contain things like stack traces, input and output parameters, or special messages for developers.
Verbose should include every event that happened, but not necessarily as much detail as debug. Debug is the highest level in this hierarchy.
console.log and console.info will log to stdout.
Console.warn and console.error will log to stderr.
By default, these will print the output to your console.
To redirect stdout and stderr to a file do this redirect when you launch your application:
node app.js > app.log 2>&1
This redirects stdout to a file named app.log and redirects stderr to stdout.
How to Get Node.js Logging Right
4 Node.js Logging libraries which make sophisticated logging simpler
debug (npm package)
ulog (npm package)
chalk
log4js (npm package)
Node-Loggly
Bunyan
Winston
- examples: 1, 2,
- Ultimate Winston Logging Tutorial: Best Practices, Resources, and Tips
- Using Winston 3 for Logging in Node.js
Morgan
What JS Logging library / tool do you use? (forum discussion)
console.log()
console.error()
node app.js > app.log
Deployment
Best practice for nodejs deployment - Directly moving node_modules to server or run npm install command
Running Node.JS in Docker
How To Build a Node.js Application with Docker
Docker and Node.js Best Practices
Selecting A Node.js Image for Docker
If you specify “FROM node” without a version number tag, you will always be given the latest version of Node.js. While this may sound like a great thing, at first, you will quickly run into headaches when you rebuild your image and find your application doesn’t run because a new release of Node.js no longer supports an API you’re calling or makes other changes.
One option is to use the latest LTS (long term support) version of Node available from the Docker Hub. Go to https://nodejs.org/en/ and take version described as LTS (or "Recommended for most users"). Obviously, your app has to support it.
I recommend to put the app in /home/node/app instead of root (e.g. /app) to avoid permission issues. [source]
you should place your app in a folder that is owned by the executing user or writable, e.g. /home/node/webapp. [source]
Docker and Node.js Best Practices
Selecting A Node.js Image for Docker
If you specify “FROM node” without a version number tag, you will always be given the latest version of Node.js. While this may sound like a great thing, at first, you will quickly run into headaches when you rebuild your image and find your application doesn’t run because a new release of Node.js no longer supports an API you’re calling or makes other changes.
One option is to use the latest LTS (long term support) version of Node available from the Docker Hub. Go to https://nodejs.org/en/ and take version described as LTS (or "Recommended for most users"). Obviously, your app has to support it.
I recommend to put the app in /home/node/app instead of root (e.g. /app) to avoid permission issues. [source]
you should place your app in a folder that is owned by the executing user or writable, e.g. /home/node/webapp. [source]
Dockerfile:
FROM node
# Create app directory
WORKDIR /usr/src/app
# Install app dependencies
# A wildcard is used to ensure both package.json AND package-lock.json are copied
# where available (npm@5+)
COPY package*.json ./
RUN npm install
# If you are building your code for production
# RUN npm ci --only=production
# Bundle app source
COPY . .
# EXPOSE 8080
CMD [ "node", "app.js" ]
.dockerignore:
node_modules
npm-debug.log
Building Docker image which resolves npm packages from custom domain
By default npm uses https://registry.npmjs.org registry to find and fetch packages (npm install). To set some custom npm package registry we can add to the project's root .npmrc file with following content:registry=https://my.example.com/what/ever/npm
npm install will try to fetch packages from there. If npm install is in Dockerfile then this means that this custom domain has be resolvable by Docker daemon during docker build.
I had a case where npm install (with custom registry set in .npmrc) run during docker build would fail with the following error:
Step 7/20 : RUN npm install --loglevel ${NPM_LOG_LEVEL}
---> Running in f3bde2ab9d02
npm info it worked if it ends with ok
npm verb cli [
npm verb cli '/usr/local/bin/node',
npm verb cli '/usr/local/bin/npm',
npm verb cli 'install',
npm verb cli '--loglevel',
npm verb cli 'verbose'
npm verb cli ]
npm info using npm@6.12.0
npm info using node@v13.0.1
npm verb npm-session 244bb6c393f2097a
npm info lifecycle my-app@1.0.0~preinstall: my-app@1.0.0
npm timing stage:loadCurrentTree Completed in 9ms
npm timing stage:loadIdealTree:cloneCurrentTree Completed in 1ms
npm timing stage:loadIdealTree:loadShrinkwrap Completed in 355ms
npm timing stage:loadIdealTree:loadAllDepsIntoIdealTree Completed in 22ms
npm timing stage:loadIdealTree Completed in 555ms
npm timing stage:generateActionsToTake Completed in 211ms
npm verb correctMkdir /root/.npm/_locks correctMkdir not in flight; initializing
npm verb lock using /root/.npm/_locks/staging-62530c0ac0331caf.lock for /usr/local/my-app/node_modules/.staging
npm timing action:extract Completed in 1218ms
npm verb unlock done using /root/.npm/_locks/staging-62530c0ac0331caf.lock for /usr/local/my-app/node_modules/.staging
npm timing stage:rollbackFailedOptional Completed in 1ms
npm timing stage:runTopLevelLifecycles Completed in 3414ms
npm verb stack RangeError: Maximum call stack size exceeded
npm verb stack at RegExp.test (<anonymous>)
npm verb stack at isDepOptional (/usr/local/lib/node_modules/npm/lib/install/deps.js:423:45)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:432:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb stack at failedDependency (/usr/local/lib/node_modules/npm/lib/install/deps.js:448:9)
npm verb cwd /usr/local/my-app
npm verb Linux 4.15.0-65-generic
npm verb argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "--loglevel" "verbose"
npm verb node v13.0.1
npm verb npm v6.12.0
npm ERR! Maximum call stack size exceeded
npm verb exit [ 1, true ]
npm timing npm Completed in 4583ms
npm ERR! A complete log of this run can be found in:
npm ERR! /root/.npm/_logs/2019-10-31T16_26_16_246Z-debug.log
ERROR: The command 'npm install --loglevel ${NPM_LOG_LEVEL}' returned a non-zero code: 1
.npmrc:
registry=https://my.example.com/what/ever/npm/
# reset auth related options if specified in ~/.npmrc file
_auth=""
always-auth=false
Dockerfile:
# installing dependencies
FROM my.example.com/node:alpine as node_cache
WORKDIR /usr/local/my-app
COPY package.json ./
# COPY package-lock.json ./
COPY .npmrc ./
# NPM_LOG_LEVEL = [silent|warn|info|silly|verbose]
ARG NPM_LOG_LEVEL=verbose
RUN npm install --loglevel ${NPM_LOG_LEVEL}
I tried to do the following to no avail:
- npm rebuild
- omit copying package-lock.json (and so forcing its re-creation)
- setting authentication in .npmrc:
- always-auth=true
- _auth was set to the output of curl -u<USERNAME>:<PASSWORD> https://my.example.com/what/ever/npm/auth)
COPY .npmrc ./
I switched log level to silly and now could see more log output and that the issue is in that npm can't resolve custom npm registry domain:
npm sill fetchPackageMetaData error for pg@^7.11.0 request to https://my.example.com/what/ever/npm/pg failed, reason: getaddrinfo ENOTFOUND my.example.com
npm timing stage:rollbackFailedOptional Completed in 1ms
npm timing stage:runTopLevelLifecycles Completed in 334ms
npm sill saveTree my-app@1.0.0
npm verb type system
npm verb stack FetchError: request to https://my.example.com/what/ever/npm/@types%2fflat failed, reason: getaddrinfo ENOTFOUND my.example.com
npm verb stack at ClientRequest.<anonymous> (/usr/local/lib/node_modules/npm/node_modules/node-fetch-npm/src/index.js:68:14)
npm verb stack at ClientRequest.emit (events.js:210:5)
npm verb stack at TLSSocket.socketErrorListener (_http_client.js:407:9)
npm verb stack at TLSSocket.emit (events.js:210:5)
npm verb stack at emitErrorNT (internal/streams/destroy.js:84:8)
npm verb stack at processTicksAndRejections (internal/process/task_queues.js:80:21)
npm verb cwd /usr/local/my-app
npm verb Linux 4.15.0-65-generic
npm verb argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "--loglevel" "silly"
npm verb node v13.0.1
npm verb npm v6.12.0
npm ERR! code ENOTFOUND
npm ERR! errno ENOTFOUND
npm install worked fine when I run npm install on my dev machine directly so I knew this was definitely a Docker DNS resolution issue.
Temp solution was to hardcode the resolved IP address for the problematic domain:
RUN echo x.y.z.q my.example.com >> /etc/hosts && npm install --loglevel ${NPM_LOG_LEVEL}
From here:
- in Dockerfile, edit /etc/resolv.conf to point to your custom DNS
- at end of build operations that require such custom DNS, restore the normal container runtime resolv.conf; this is done automatically as /etc/resolv.conf is intentionally not persisted to the image (intermediate container)
This is unreliable solution as it would break each time domain's IP address changes. Plus we'd need to do this for any other unresolved domain.
Proper solutions are described here: Fix Docker's networking DNS config and here. I opted for simply using --network=host in docker build:
$ docker build --network=host --pull --build-arg NPM_LOG_LEVEL=silly -t my-app .
Network calls fail during image build on corporate network
By the way these were some useful outputs:
$ cat /etc/resolv.conf
nameserver 127.0.0.53
options edns0
search my.example.com
$ docker run busybox nslookup google.com
Server: 8.8.8.8
Address: 8.8.8.8:53
Non-authoritative answer:
Name: google.com
Address: 172.217.169.14
*** Can't find google.com: No answer
$ docker run --network=host busybox nslookup google.com
Server: 127.0.0.53
Address: 127.0.0.53:53
Non-authoritative answer:
Name: google.com
Address: 172.217.169.14
*** Can't find google.com: No answer
At some other occasion I experienced similar error when building a Docker image for running Node.js application. npm install was failing with:
npm ERR! Maximum call stack size exceeded
And this was happening only if I'd set in .npmrc a custom npm registry which sits on corporate network.
Using --network=host (even when on corporate network) didn't help.
The docker build uses the DNS settings of the docker engine running on the host so I tried setting DNS servers for Docker daemon and that helped.
When on VPN, I checked which DNS servers are set:
$ cat /etc/resolv.conf
#@VPNC_GENERATED@ -- this file is generated by vpnc
# and will be overwritten by vpnc
# as long as the above mark is intact
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
options edns0
nameserver A.B.C.D
nameserver E.F.G.H
search Home company.com
/etc/docker/daemon.json did not exist so I had to create it:
$ sudo touch /etc/docker/daemon.json
$ sudo gedit /etc/docker/daemon.json
and edit it as here:
{
"dns": ["A.B.C.D", "E.F.G.H"],
"dns-search": ["company.com"]
}
I then restarted Docker service:
$ systemctl restart docker
After this npm install worked fine.
References & Further Reading
Check synchronously if file/directory exists in Node.js
Writing files in Node.js
How to append to a file in Node?
Writing JSON object to a JSON file with fs.writeFileSync
How to get file size in Node.js
Difference between a module and a package in Node?
Create an empty file in Node.js?
What is the purpose of Node.js module.exports and how do you use it?
In Node.js, how do I “include” functions from my other files?
How to exit in Node.js
Node.js Development Tips
No comments:
Post a Comment