Conventions
# - requires given linux commands to be executed with root privileges either directly as a root user or by use of sudo command
$ - requires given linux commands to be executed as a regular non-privileged user
Useful Command Line Utility Tools
To print last N commands form Terminal:
$ history N
Using the history command - Media Temple
The output is command history in form:
2000 sudo apt install ipython ipython-notebook
2001 sudo apt list --installed | grep python3-pip
2002 sudo apt list --installed | grep python3-dev
...
To copy only commands (without command number) keep CTRL+ALT pressed and select desired text columns and rows with left mouse click button.
Operating System
How to find what's the version of the installed OS:
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic
User management
To request security privileges of superuser (root):
$ sudo
/usr/local/etc/sudoers
To log in and run the current shell as root use:
user@computer:~$ sudo -i
root@computer:~# whoami
root
root@computer:~# exit
logout
user@computer:~$
To find info about user use id:
$ id --help
Usage: id [OPTION]... [USER]
Print user and group information for the specified USER,
or (when USER omitted) for the current user.
-a ignore, for compatibility with other versions
-Z, --context print only the security context of the process
-g, --group print only the effective group ID
-G, --groups print all group IDs
-n, --name print a name instead of a number, for -ugG
-r, --real print the real ID instead of the effective ID, with -ugG
-u, --user print only the effective user ID
-z, --zero delimit entries with NUL characters, not whitespace;
not permitted in default format
--help display this help and exit
--version output version information and exitftime
Without any OPTION, print some useful set of identified information.
GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Full documentation at: <http://www.gnu.org/software/coreutils/id>
or available locally via: info '(coreutils) id invocation'
$ id
uid=1000(test_user) gid=1000(test_user) groups=1000(test_user),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),116(lpadmin),126(sambashare),999(docker)
$ id -u
1000
$ id -g
1000
test_user
$ id -gn
test_user
$ id -G
1000 3 23 26 29 45 115 125 999
$ id -Gn
test_user adm cdrom sudo dip plugdev lpadmin sambashare docker
To get the current user:
$ whoami
test_user
To list all groups:
$ groups
test_user adm cdrom sudo dip plugdev lpadmin sambashare docker
To find out to which groups belong given user:
$ groups test_user
test_user : test_user adm cdrom sudo dip plugdev lpadmin sambashare docker
.profile file
There is one global profile file (executed when anyone logs in):
/etc/profile
There are three user-specific bash profile files (executed when current/specific user logs in):
~/.profile
~/.bash_profile
~/.bashrc
If ~/.profile doesn't exist, just create it.
This is the comment at the beginning of ~/.profile:
# ~/.profile: executed by the command interpreter for login shells.
# This file is not read by bash(1), if ~/.bash_profile or ~/.bash_login exists.
# see /usr/share/doc/bash/examples/startup-files for examples.
# the files are located in the bash-doc package.
# the default umask is set in /etc/profile; for setting the umask
# for ssh logins, install and configure the libpam-umask package.
#umask 022
To add en environment variable during the session of a particular user (and also make them available in terminal) append the desired var name and its value to ~/.profile file. Example:
export GOROOT=/usr/local/go
export GOPATH=$HOME/go
export PATH=$GOPATH/bin:$GOROOT/bin:$PATH
We'd need to restart the terminal in order to get these changes visible but to make terminal fetch them in the current session, we can update the current shell session with:
source ~/.profile
To add a new or modify existing environment variable permanently (for non-root user) we need to change ~/.bashrc:
Example of ~/.bashrc snippet:
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
...
export GOPATH=$HOME/dev/go
export PATH=$PATH:$GOPATH/bin
To do the same for root user, open /etc/environment:
$ sudo gedit /etc/environment
...and add desired path:
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games"
GOPATH="/home/test_user/dev/go"
How to logout current user from the Terminal?
$ gnome-session-quit
or
$ gnome-session-quit --no-prompt
to suppress showing Logout confirmation dialog.
File & Directory Ownership
To see permissions (r= read, w=write, x=execute) and ownership (user:group) over some file or directory use:$ ls -la
To change ownership (e.g. from root to current user) of directory and all its content use chown (change owner):
$ sudo chown -R $USER directory
or
$ sudo chown -R username:group directory
-R stands for --recursive.
Change folder permissions and ownership
To change user:group of a file:
$ sudo chown user:group file_name
Permissions
Chmod permissions (flags) explained
os.MkDir and os.MkDirAll permission value?
To allow only file user to read and write (but not to execute):
$ chmod 600 filename
chmod 600 file – owner can read and write
chmod 700 file – owner can read, write and execute
chmod 666 file – all can read and write
chmod 777 file – all can read, write and execute
It is very common to add executable permissions for some bash script (apart from adding shebang #!/bin/bash at the beginning of the script so it is not necessary to call bash explictily):
$ chmod +x script.sh
How do I run a shell script without using “sh” or “bash” commands?
Example:
Before chmod +x, the output of ls -la is:
-rw-rw-r-- 1 test_user test_user 300 Oct 30 15:54 download.sh
After:
-rwxrwxr-x 1 test_user test_user 300 Oct 30 15:54 download.sh
Working with directories
To list the content of the directory:
ls
To list directories in the current directory, names only, and each name in its own line:
$ ls -1d *
-1 list one file per line
-d list directories themselves, not their contents
* list all items in the current directory (they might be filtered by other flags, like -d)
To list all files (including hidden) in the current directory:
ls -a
$ ls target_directory
Example:
$ ls /usr/local/go/
To list directories and files in form of a tree install tree package:
$ sudo apt install tree
...and use it as e.g.:
$ tree -I *node*
-I = ignores directories that match given pattern
To diff two directories use:
$ diff -r dir1 dir2
This shows which files are only in dir1 and those only in dir2 and also the changes of the files present in both directories if any. If any file does not end with a newline character, this will be reported as well:
$ diff -r dir1 dir2
diff -r dir1/test.txt dir2/test.txt
1c1
< First line in dir1/test.txt
\ No newline at end of file
---
> First line in dir2/test.txt
\ No newline at end of file
Binary files /dir1/test.bin and /dir2/test.bin differ
Only in /dir1/subdir1/: subsubdir1
Only in /dir2/subdir1/: file.zip
---What does 1c1 in diff tool mean?
1c1 indicates that line 1 in the first file was c hanged somehow to produce line 1 in the second file.
They probably differ in whitespace (perhaps trailing spaces, or Unix versus Windows line endings?).
man diff
Difference between two directories in Linux [closed]
How to make diff check a symlink link itself?
diff reports two files differ, although they are the same!
man cmp
diff reports two files differ, although they are the same!
man cmp
To find the location of some directory starting from the root directory (/) use:
$ find / -type d -name dir_name
Time and Date
To convert Unix time to human readable, use:
$ date -d @1583944041
Wed 11 Mar 16:27:21 GMT 2020
Working with files
Creating a file
To create a file use touch:
$ touch filename
It is possible to use redirection operators > and >> to achieve this:
- > will overwrite existing file or crate a new file
- >> will append text to existing file or created a new file
> file.txt
What does “>” do vs “>>”?
Writing multi-line text into file
shell - How to append multiple lines to a file - Unix & Linux Stack Exchange
Ending file with new line character
[No newline at end of file]
It is a good style to always put the newline as a last character if it is allowed by the file format.
Unix historically had a convention of all human-readable text files ending in a newline. Reasons:
Practically, because many Unix tools require or expect it for proper display.
Philosophically, because each line in a text file terminates with an "end-of-line" character--the last line shouldn't be any exception.
To write into file a set of lines which end with a new line character:
$ echo $'first line\nsecond line\nthirdline\n' > foo.txt
$'...' construct expands embedded ANSI escape sequences
How to put a newline special character into a file using the echo command and redirection operator?
Getting the information about a file
To get the number of lines (well, newline characters) in the file:
$ wc -l myfile.txt
23 myfile.txt
(This is why it's important to follow the convention and end each line with newline character.)
To get the number of words on some webpage:
$ curl "https://example.com/" 2>/dev/null | grep -i "word1|word2" | wc -l
To see the last couple of lines in the file use command tail:
$ tail myfile
To find various hash sums of a file:
$ md5sum file_name
$ sha1sum file_name
$ sha256sum file_name
If file is Windows executable, it is possible to examine it with:
$ exiftool somefile.exe
To install exiftool:
$ sudo apt install libimage-exiftool-perl
linux - viewing dll and exe file properties/attributes via the command line - Unix & Linux Stack Exchange
if test -f "$symlink_file"; then
echo "$symlink_file" exists and is regular file.
else
echo "$symlink_file" does not exist or is not a regular file.
fi
if test -L "$regular_file"; then
echo "$regular_file" exists and is symlink file.
else
echo "$regular_file" does not exist or is not a symlink file.
fi
How to Check if a File or Directory Exists in Bash
cp - copy
cp [OPTIONS] SOURCE DEST
SOURCE - file or directory
DEST - file or directory
An error is reported if directory is specified as source and file as destination.
$ cp -r test test.txt
cp: cannot overwrite non-directory 'test.txt' with directory 'test'
To copy a directory recursively use -r flag:
$ scp -r nvidia@nvidia-nano:/usr/src/tensorrt/samples/python/uff_ssd ~/dev/
nvidia@nvidia-nano's password:
inference.py 100% 12KB 3.3MB/s 00:00
engine.py 100% 5750 4.5MB/s 00:00
__init__.py 100% 0 0.0KB/s 00:00
boxes.py 100% 6725 5.2MB/s 00:00
coco.py 100% 4755 4.2MB/s 00:00
...
shell - How do I copy a folder from remote to local using scp? - Stack Overflow
To copy file which has spaces in path and/or name use double backslash before space and wrap entire file path in double quotation marks:
~/Downloads$ scp nvidia@nvidia-nano:"/home/nvidia/Pictures/Object\\ Detection\\ -\\ SSD.png" ~/Pictures
nvidia@192.168.0.10's password:
Object Detection - SSD.png 100% 444KB 1.2MB/s 00:00
$ mv *.{jpg,gif,png} ~/Pictures
To rename all .new files in the current directory to *.old:
$ rename -f -v 's/.new/.old/' *
-f = force; allows overwriting existing *.old files
-v = verbose
$ cat filename
To edit some file, you can use vi editor. Example:
$ vi ~/.profile
gedit can also be used as graphic editor:
sudo gedit ~/.profile
To enter some special character (e.g. bulletpoint) press CTRL+SHIFT+U and underscored "u" should appear (u). Then use numeric keyboard to type in the Unicode code of the character (e.g. 2022) and press Enter. [source]
To see the content of the file as binary and hexadecimal:
xxd -b file
xxd file
To search file from the root directory use /:
$ find / -name "file_name.ext"
How do I find all files containing specific text on Linux?
man grep
$ grep -rnw '/path/to/somewhere/' -e 'pattern'
-r or -R = recursive,
-n = line number
-w = match the whole word.
-l (lower-case L) = just give the file name of matching files
Example:
$ find ./go/src/pkg -type f -name "*.go" | xargs egrep '^type.*(er|or) interface {'
xargs manual - xargs builds and executes command lines from standard input
egrep manual - egrep prints lines matching a pattern
How to ignore line endings when comparing files?
$ diff --strip-trailing-cr file1 file2
How to detect file ends in newline?
Running a command prefixed by the time command will tell us how long our code took to execute.
$ time myapp
real 0m13.761s
user 0m0.262s
sys 0m0.039s
If an executable is present but some of its dependencies are missing bash (or sh) might display an error messages stating that main executable is not found (which might be a bit misleading).
Example:
/ # ls
bin myapp data-vol dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var
/ # myapp
/bin/sh: myapp: not found
linux - Find out symbolic link target via command line - Server Fault
What are they?
What is their purpose?
How do they work?
How to create them?
How to: Linux / UNIX create soft link with ln command
Use ln command:
NAME
ln - make links between files
SYNOPSIS
ln [OPTION]... [-T] TARGET LINK_NAME (1st form)
ln [OPTION]... TARGET (2nd form)
ln [OPTION]... TARGET... DIRECTORY (3rd form)
ln [OPTION]... -t DIRECTORY TARGET... (4th form)
DESCRIPTION
In the 1st form, create a link to TARGET with the name LINK_NAME. In the 2nd form, create a link to TARGET in the current directory. In the 3rd and 4th forms, create links to each TARGET in DIRECTORY. Create hard links by default, symbolic links with --symbolic. By default, each destination (name of new link) should not already exist. When creating hard links, each TARGET must exist.
Symbolic links can hold arbitrary text; if later resolved, a relative link is interpreted in relation to its parent directory.
Mandatory arguments to long options are mandatory for short options too.
--backup[=CONTROL]
make a backup of each existing destination file
-b like --backup but does not accept an argument
-d, -F, --directory
allow the superuser to attempt to hard link directories (note: will probably fail due to system restrictions, even for the superuser)
-f, --force
remove existing destination files
-i, --interactive
prompt whether to remove destinations
-L, --logical
dereference TARGETs that are symbolic links
-n, --no-dereference
treat LINK_NAME as a normal file if it is a symbolic link to a directory
-P, --physical
make hard links directly to symbolic links
-r, --relative
create symbolic links relative to link location
-s, --symbolic
make symbolic links instead of hard links
-S, --suffix=SUFFIX
override the usual backup suffix
-t, --target-directory=DIRECTORY
specify the DIRECTORY in which to create the links
-T, --no-target-directory
treat LINK_NAME as a normal file always
-v, --verbose
print name of each linked file
--help display this help and exit
--version
output version information and exit
The backup suffix is '~', unless set with --suffix or SIMPLE_BACKUP_SUFFIX. The version control method may be selected via the --backup option or through the VERSION_CONTROL environment variable.
Here are the values:
none, off
never make backups (even if --backup is given)
numbered, t
make numbered backups
existing, nil
numbered if numbered backups exist, simple otherwise
simple, never
always make simple backups
Using -s ignores -L and -P. Otherwise, the last option specified controls behavior when a TARGET is a symbolic link, defaulting to -P.
$ sudo ln -s /usr/local/go/bin/go /usr/local/bin/go
Creating a symlink from one folder to another with different names?
Types of symlinks:
$ pwd
/home/beau
$ ln -s foo/bar.txt bar.txt
$ readlink -f /home/beau/bar.txt
/home/beau/foo/bar.txt
Or:
$ cd foo
$ ln -s foo/bar.txt ../bar.txt
[man find]: If no paths are given, the current directory is used.
[How to list all symbolic links in a directory]
How do I tell if a folder is actually a symlink and how do I fix it if it's broken?
Here are some ways that can be used to verify symlink:
$ stat ./data-vol/content/app/74.0.1365.76
File: ./data-vol/content/app/74.0.1365.76 -> data-vol/content/app/win/x86/74.0.1365.76
Size: 45 Blocks: 0 IO Block: 4096 symbolic link
Device: fd01h/64769d Inode: 26479224 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2019-07-12 17:17:09.278071996 +0100
Modify: 2019-07-12 17:17:08.666073171 +0100
Change: 2019-07-12 17:17:08.666073171 +0100
Birth: -
$ stat -L ./data-vol/content/app/74.0.1365.76
stat: cannot stat './data-vol/content/app/74.0.1365.76': No such file or directory
$ file -L ./data-vol/content/app/74.0.1365.76
./data-vol/content/app/74.0.1365.76: cannot open `./data-vol/content/app/74.0.1365.76' (No such file or directory)
$ ls ./data-vol/content/app/74.0.1365.76
./data-vol/content/app/74.0.1365.76
$ ll ./data-vol/content/app/74.0.1365.76
lrwxrwxrwx 1 root root 45 Jul 12 17:17 ./data-vol/content/app/74.0.1365.76 -> data-vol/content/app/win/x86/74.0.1365.76
How to see full symlink path
$ readlink -f symlinkName
How to create hardlink of one file in different directories in linux
$ env
To display the value of some particular env var use echo $ENV_VAR_NAME. Example:
$ echo $GOPATH
/home/test_user/dev/go
To set environment variables for the single command:
Example:
$ env GOOS=linux GOARCH=amd64 go build cmd/main.go
From the executable's point of view, the same would have been achieved without using env:
$ GOOS=linux GOARCH=amd64 go build cmd/main.go
To set environment variables for the current terminal session:
$ export GOPATH=/mnt/c/dev/go
export is a bash builtin. export key=value is extended syntax and should not be used in portable scripts (i.e. #! /bin/sh)
What's the difference between set, export and env and when should I use each?
What is the difference between set, env, declare and export when setting a variable in a Linux shell?
If some bash script calls executable which requires some env variables, we also need to use export.Example:
demo.sh:
#!/bin/bash
...
echo
echo Env variables:
go env
export CGO_ENABLED=0
export GOOS=linux
export GOARCH=amd64
echo
echo Env variables:
go env
go build -o './bin/myapp' -v './cmd/main.go'
...gives the output:
Env variables:
[16:33:25][Step 4/7] GOARCH="amd64"
[16:33:25][Step 4/7] GOOS="linux"
[16:33:25][Step 4/7] CGO_ENABLED="1"
...
[16:33:25][Step 4/7]
[16:33:25][Step 4/7] Env variables:
[16:33:25][Step 4/7] GOARCH="amd64"
[16:33:25][Step 4/7] GOOS="linux"
[16:33:25][Step 4/7] CGO_ENABLED="0"
...
How do I add environment variables?
How to set an environment variable only for the duration of the script?
apt (Advanced Packaging Tool) - It is not a command itself but a package which contains set of tools which manage installation and removal of other packages.
apt-get
apt-get update - downloads the package lists from the repositories and "updates" them to get information on the newest versions of packages and their dependencies. It will do this for all repositories and PPAs.
http://askubuntu.com/questions/222348/what-does-sudo-apt-get-update-do
apt-cche
add-apt-repository - adds a repository to the list of repositories
To apply latest security updates on Ubuntu:
sudo apt-get update
sudo apt-get -y upgrade
Difference Between apt and apt-get Explained
Should I use apt or apt-get?
You might be thinking if you should use apt or apt-get. And as a regular Linux user, my answer is to go with apt.
apt is the command that is being recommended by the Linux distributions. It provides the necessary option to manage the packages. Most important of all, it is easier to use with its fewer but easy to remember options.
I see no reason to stick with apt-get unless you are going to do specific operations that utilize more features of apt-get.
-----------------------------------------------------------------------------------------------------------------------
To download a file into some specific directory:
cd /dest_dir
wget https://example.com/archive_file.tar.gz
Checking whether file exists
if test -f "$symlink_file"; then
echo "$symlink_file" exists and is regular file.
else
echo "$symlink_file" does not exist or is not a regular file.
fi
if test -L "$regular_file"; then
echo "$regular_file" exists and is symlink file.
else
echo "$regular_file" does not exist or is not a symlink file.
fi
How to Check if a File or Directory Exists in Bash
Copying files
cp - copy
cp [OPTIONS] SOURCE DEST
SOURCE - file or directory
DEST - file or directory
An error is reported if directory is specified as source and file as destination.
$ cp -r test test.txt
cp: cannot overwrite non-directory 'test.txt' with directory 'test'
Copying files and directories from remote machine
To copy a directory recursively use -r flag:
$ scp -r nvidia@nvidia-nano:/usr/src/tensorrt/samples/python/uff_ssd ~/dev/
nvidia@nvidia-nano's password:
inference.py 100% 12KB 3.3MB/s 00:00
engine.py 100% 5750 4.5MB/s 00:00
__init__.py 100% 0 0.0KB/s 00:00
boxes.py 100% 6725 5.2MB/s 00:00
coco.py 100% 4755 4.2MB/s 00:00
...
shell - How do I copy a folder from remote to local using scp? - Stack Overflow
To copy file which has spaces in path and/or name use double backslash before space and wrap entire file path in double quotation marks:
~/Downloads$ scp nvidia@nvidia-nano:"/home/nvidia/Pictures/Object\\ Detection\\ -\\ SSD.png" ~/Pictures
nvidia@192.168.0.10's password:
Object Detection - SSD.png 100% 444KB 1.2MB/s 00:00
Moving files
$ mv *.{jpg,gif,png} ~/Pictures
Renaming files
To rename all .new files in the current directory to *.old:
$ rename -f -v 's/.new/.old/' *
-f = force; allows overwriting existing *.old files
-v = verbose
File viewing and editing
To simply view the content of some file, use cat:$ cat filename
To edit some file, you can use vi editor. Example:
$ vi ~/.profile
sudo gedit ~/.profile
To see the content of the file as binary and hexadecimal:
xxd -b file
xxd file
Searching for Files
To search file from the root directory use /:
$ find / -name "file_name.ext"
Searching for text across files
How do I find all files containing specific text on Linux?
man grep
$ grep -rnw '/path/to/somewhere/' -e 'pattern'
-r or -R = recursive,
-n = line number
-w = match the whole word.
-l (lower-case L) = just give the file name of matching files
--include=\*.{c,h} = search through those files which have .c or .h extensions
--exclude=*.o = exclude searching all the files ending with .o extension
--exclude-dir={dir1,dir2,*.dst} = exclude a particular directory(ies)
-e PATTERN = string pattern to be searched
-i = ignore the case
Example:
$ grep -r /var/lib/go/src/ -e "CodeDecode"
/var/lib/go/src/encoding/json/bench_test.go:func BenchmarkCodeDecoder(b *testing.B) {
$ find ./go/src/pkg -type f -name "*.go" | xargs egrep '^type.*(er|or) interface {'
xargs manual - xargs builds and executes command lines from standard input
egrep manual - egrep prints lines matching a pattern
Comparing Files
How to ignore line endings when comparing files?
$ diff --strip-trailing-cr file1 file2
How to detect file ends in newline?
Working with executable files
Running a command prefixed by the time command will tell us how long our code took to execute.
$ time myapp
real 0m13.761s
user 0m0.262s
sys 0m0.039s
If an executable is present but some of its dependencies are missing bash (or sh) might display an error messages stating that main executable is not found (which might be a bit misleading).
Example:
/ # ls
bin myapp data-vol dev etc home lib media mnt opt proc root run sbin srv sys tmp usr var
/ # myapp
/bin/sh: myapp: not found
Symbolic links
What are they?
- files that contain a reference to another file or directory on the same system
- like shortcuts on Windows OS
What is their purpose?
- to avoid copying the same binary (usually a library) at multiple locations but simply creating a symlink where file is required to be
- various clients might require the same file but with name in different format so instead of having multiple copies of the same file but with different names we'd have multiple symlink, each with the name that satisfies requirements of each service
How do they work?
- opening/running the symlink would open/run the target file
- editing the content of the symlink edits the content of the target file
- if target file is deleted symlink becomes a dangling symlink
- if symlink is deleted, target file remains unaffected
- it is possible to create symlink that refers to another symlink [How can I create a symlink which points to another symlink?]
How to create them?
How to: Linux / UNIX create soft link with ln command
Use ln command:
NAME
ln - make links between files
SYNOPSIS
ln [OPTION]... [-T] TARGET LINK_NAME (1st form)
ln [OPTION]... TARGET (2nd form)
ln [OPTION]... TARGET... DIRECTORY (3rd form)
ln [OPTION]... -t DIRECTORY TARGET... (4th form)
DESCRIPTION
In the 1st form, create a link to TARGET with the name LINK_NAME. In the 2nd form, create a link to TARGET in the current directory. In the 3rd and 4th forms, create links to each TARGET in DIRECTORY. Create hard links by default, symbolic links with --symbolic. By default, each destination (name of new link) should not already exist. When creating hard links, each TARGET must exist.
Symbolic links can hold arbitrary text; if later resolved, a relative link is interpreted in relation to its parent directory.
Mandatory arguments to long options are mandatory for short options too.
--backup[=CONTROL]
make a backup of each existing destination file
-b like --backup but does not accept an argument
-d, -F, --directory
allow the superuser to attempt to hard link directories (note: will probably fail due to system restrictions, even for the superuser)
-f, --force
remove existing destination files
-i, --interactive
prompt whether to remove destinations
-L, --logical
dereference TARGETs that are symbolic links
-n, --no-dereference
treat LINK_NAME as a normal file if it is a symbolic link to a directory
-P, --physical
make hard links directly to symbolic links
-r, --relative
create symbolic links relative to link location
-s, --symbolic
make symbolic links instead of hard links
-S, --suffix=SUFFIX
override the usual backup suffix
-t, --target-directory=DIRECTORY
specify the DIRECTORY in which to create the links
-T, --no-target-directory
treat LINK_NAME as a normal file always
-v, --verbose
print name of each linked file
--help display this help and exit
--version
output version information and exit
The backup suffix is '~', unless set with --suffix or SIMPLE_BACKUP_SUFFIX. The version control method may be selected via the --backup option or through the VERSION_CONTROL environment variable.
Here are the values:
none, off
never make backups (even if --backup is given)
numbered, t
make numbered backups
existing, nil
numbered if numbered backups exist, simple otherwise
simple, never
always make simple backups
Using -s ignores -L and -P. Otherwise, the last option specified controls behavior when a TARGET is a symbolic link, defaulting to -P.
Example:
$ sudo ln -s /usr/local/go/bin/go /usr/local/bin/go
Creating a symlink from one folder to another with different names?
Types of symlinks:
- absolute
- relative; If you create a symbolic link to a relative path, it will store it as a relative symbolic link [Make a symbolic link to a relative pathname]
$ pwd
/home/beau
$ ln -s foo/bar.txt bar.txt
$ readlink -f /home/beau/bar.txt
/home/beau/foo/bar.txt
Or:
$ cd foo
$ ln -s foo/bar.txt ../bar.txt
How to list all symbolic links in the current directory?
$ find -type l[man find]: If no paths are given, the current directory is used.
[How to list all symbolic links in a directory]
# save symlinks and their targets (relative to ./path/to/dir/) in csv file which will also be pushed to S3
cd ./path/to/dir/
find . -type l -ls | awk '{print $11,",",$13}' > symlinks.csv
How do I tell if a folder is actually a symlink and how do I fix it if it's broken?
Here are some ways that can be used to verify symlink:
$ stat ./data-vol/content/app/74.0.1365.76
File: ./data-vol/content/app/74.0.1365.76 -> data-vol/content/app/win/x86/74.0.1365.76
Size: 45 Blocks: 0 IO Block: 4096 symbolic link
Device: fd01h/64769d Inode: 26479224 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2019-07-12 17:17:09.278071996 +0100
Modify: 2019-07-12 17:17:08.666073171 +0100
Change: 2019-07-12 17:17:08.666073171 +0100
Birth: -
$ stat -L ./data-vol/content/app/74.0.1365.76
stat: cannot stat './data-vol/content/app/74.0.1365.76': No such file or directory
$ file -L ./data-vol/content/app/74.0.1365.76
./data-vol/content/app/74.0.1365.76: cannot open `./data-vol/content/app/74.0.1365.76' (No such file or directory)
$ ls ./data-vol/content/app/74.0.1365.76
./data-vol/content/app/74.0.1365.76
$ ll ./data-vol/content/app/74.0.1365.76
lrwxrwxrwx 1 root root 45 Jul 12 17:17 ./data-vol/content/app/74.0.1365.76 -> data-vol/content/app/win/x86/74.0.1365.76
How to see full symlink path
$ readlink -f symlinkName
Hard links
How to create hardlink of one file in different directories in linux
Working with Environment Variables
To list all environment variables and their values use:$ env
To display the value of some particular env var use echo $ENV_VAR_NAME. Example:
$ echo $GOPATH
/home/test_user/dev/go
To set environment variables for the single command:
Example:
$ env GOOS=linux GOARCH=amd64 go build cmd/main.go
From the executable's point of view, the same would have been achieved without using env:
$ GOOS=linux GOARCH=amd64 go build cmd/main.go
$ export GOPATH=/mnt/c/dev/go
export is a bash builtin. export key=value is extended syntax and should not be used in portable scripts (i.e. #! /bin/sh)
What's the difference between set, export and env and when should I use each?
What is the difference between set, env, declare and export when setting a variable in a Linux shell?
demo.sh:
#!/bin/bash
...
echo
echo Env variables:
go env
export CGO_ENABLED=0
export GOOS=linux
export GOARCH=amd64
echo
echo Env variables:
go env
go build -o './bin/myapp' -v './cmd/main.go'
...gives the output:
Env variables:
[16:33:25][Step 4/7] GOARCH="amd64"
[16:33:25][Step 4/7] GOOS="linux"
[16:33:25][Step 4/7] CGO_ENABLED="1"
...
[16:33:25][Step 4/7]
[16:33:25][Step 4/7] Env variables:
[16:33:25][Step 4/7] GOARCH="amd64"
[16:33:25][Step 4/7] GOOS="linux"
[16:33:25][Step 4/7] CGO_ENABLED="0"
...
How do I add environment variables?
How to set an environment variable only for the duration of the script?
Package management
apt (Advanced Packaging Tool) - It is not a command itself but a package which contains set of tools which manage installation and removal of other packages.
apt-get
apt-get update - downloads the package lists from the repositories and "updates" them to get information on the newest versions of packages and their dependencies. It will do this for all repositories and PPAs.
http://askubuntu.com/questions/222348/what-does-sudo-apt-get-update-do
apt-cche
add-apt-repository - adds a repository to the list of repositories
To apply latest security updates on Ubuntu:
sudo apt-get update
sudo apt-get -y upgrade
Difference Between apt and apt-get Explained
Should I use apt or apt-get?
You might be thinking if you should use apt or apt-get. And as a regular Linux user, my answer is to go with apt.
apt is the command that is being recommended by the Linux distributions. It provides the necessary option to manage the packages. Most important of all, it is easier to use with its fewer but easy to remember options.
I see no reason to stick with apt-get unless you are going to do specific operations that utilize more features of apt-get.
-----------------------------------------------------------------------------------------------------------------------
To download a file into some specific directory:
cd /dest_dir
wget https://example.com/archive_file.tar.gz
To install a package/software in Ubuntu, it is usually enough to copy it to /usr/local directory.
To move dir1 to some other location e.g. /usr/local use:
mv new_app /usr/local
Run man hier to see extensive list of directories and description of the filesystem hierarchy.
/srv
/usr
/usr/src
/usr/local
/usr/local/src
/opt
$ sudo apt-get update
$ sudo apt-get install vlc
It's best to run sudo apt-get update first as this updates local information about what packages are available from where in what versions. This can prevent a variety of installation errors (including some "unmet dependencies" errors), and also ensures you get the latest version provided by your enabled software sources.
There is also an apt version of this command:
$ sudo apt update
...
Reading package lists... Done
Building dependency tree
Reading state information... Done
23 packages can be upgraded. Run 'apt list --upgradable' to see them.
Example: Installing PowerShell from Microsoft Package Repository
# Download the Microsoft repository GPG keys
wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
# Register the Microsoft repository GPG keys
sudo dpkg -i packages-microsoft-prod.deb
# Update the list of products
sudo apt-get update
# Enable the "universe" repositories
sudo add-apt-repository universe
# Install PowerShell
sudo apt-get install -y powershell
# Start PowerShell
pwsh
If you install the wrong version of packages-microsoft-prod.deb you can uninstall it with:
$ sudo dpkg -r packages-microsoft-prod
(Reading database ... 254902 files and directories currently installed.)
Removing packages-microsoft-prod (1.0-3) ...
$ sudo dpkg -i /path/to/deb/file
$ sudo apt-get install -f
The latter is necessary in order to fix broken packages (install eventual missing/unmet dependencies).
How to install a deb file, by dpkg -i or by apt?
Snap updates automatically and carries all its library dependencies, so is a better choice for users who want ease of deployment and to stay on top of the latest developments.
Snapcraft - Snaps are universal Linux packages
Installing snap on Ubuntu | Snapcraft documentation
How to Install and Use Snap on Ubuntu 18.04 - codeburst
To unpack the .tar.gz in the current directory use:
$ tar -zxvf archive_file.tar.gz
-x = extract
-f (--file) = use archive file; this flag has to be the last in the list of flags and to be followed by the archive file name
-v (--verbose) = verbose output
-z (--gzip, --gunzip, --ungzip) = filter the archive through gzip
To unpack only the specific directory from the archive use:
$ tar -zxvf archive_file.tar.gz dir_name
To unpack archive to the specific directory:
$ tar -zxvf archive_file.tar -C path/to/dest_dir
-C (--directory) stands for "Change to directory"
Example:
$ tar -xzf go1.12.1.linux-amd64.tar -C /usr/local
$ sudo tar -xzvf Postman-linux-x64-7.5.0.tar.gz -C /opt
Why compressed directories cannot be extracted in /opt?
To unpack multiple rar files first install unrar:
$ sudo apt-get install rar unrar
then go to the directory where all x.party.rar files are and execute:
$ unrar x -e file.part1.rar
ZIP format unzipping into the specified directory:
$ unzip bookmarks.zip -d bookmarks-dev-413
$ unzip bazel-1.2.1-dist.zip -d bazel bazel-1.2.1-dist
Unzip files in particular directory or folder under Linux or UNIX - nixCraft
What is the difference between curl and wget?
curl vs Wget
How to download files in Linux from command line with dynamic url
to download a file when it is pointed by a dynamic url, all you need to to is to use single quotes for the url; -O = specify the output filename
$ wget 'http://some.site.com/download?id=234&status=download' -O output_filename
To verify if you're running a 64-bit system use GNU Core Utility: uname.
To check virtual memory statistics (the latest reading) use:
$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 138816 9157820 1739872 11514740 0 1 1559 366 102 64 26 7 66 0 0
or, to see readings every 1 second:
$ vmstat 1
To disable and then reenable swap:
$ sudo swapoff -a
$ sudo swapon -a
performance - How to empty swap if there is free RAM? - Ask Ubuntu
$ xrandr --listmonitors
Monitors: 2
0: +*HDMI-1 2560/677x1440/381+0+0 HDMI-1
1: +eDP-1 1920/344x1080/193+112+1440 eDP-1
To see details for the current display:
$ xrandr --current
Screen 0: minimum 320 x 200, current 1920 x 1080, maximum 8192 x 8192
eDP-1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 344mm x 193mm
1920x1080 60.03*+ 60.01 59.97 59.96 59.93 48.02
1680x1050 59.95 59.88
1600x1024 60.17
1400x1050 59.98
1600x900 59.99 59.94 59.95 59.82
...
360x202 59.51 59.13
320x180 59.84 59.32
HDMI-1 disconnected (normal left inverted right x axis y axis)
DP-1 disconnected (normal left inverted right x axis y axis)
HDMI-2 disconnected (normal left inverted right x axis y axis)
DP-2 disconnected (normal left inverted right x axis y axis)
HDMI-3 disconnected (normal left inverted right x axis y axis)
To list all video devices (e.g. webcams):
$ ls /dev/video*
To check the number of processors:
$ grep -c ^processor /proc/cpuinfo | sed 's/^0$/1/'
4
To see details for all processors:
$ cat /proc/cpuinfo
To list CPU architecture information, use lscpu. E.g.
$ lscpu | grep "Byte Order"
Byte Order: Little Endian
How to generate SSH key pair on Ubuntu
To move dir1 to some other location e.g. /usr/local use:
mv new_app /usr/local
A word on Linux directories
[source]Run man hier to see extensive list of directories and description of the filesystem hierarchy.
/srv
- owner is root
- contains site-specific data which is served by this system
- place for your workspace - your software development tree.
- code repository (possibly in /srv/sourcerepo or something), and then developers would check out their own working copies into their home directories.
- place for my source code (though I usually use /srv/vcs/sourcerepo)
/usr
- the location where Distribution-based items are placed
/usr/src
- owner is root
- Source files for different parts of the system, included with some packages for reference purposes. Don't work here with your own projects, as files below /usr should be read-only except when installing software (optional).
- meant for source code for the binaries that come with your system's installation. For example, it could contain the source code for your kernel, tools such as ls, passwd, cp, mv etc, which are all installed BY YOUR DISTRIBUTION. If you upgrade your OS from source, all the source code would go here, when you rebuild your system. You DON'T want to put any software that you install BY YOURSELF in here, because they may get overwritten when you upgrade your system. In general, files that are in /, /usr/bin, /usr/sbin, /bin, /sbin etc. have their source code in /usr/src.
- is a system directory. You should not change the owner from root for security reasons
- contains the linux headers and source code of the kernel. Since the system compiles the kernel from there, it IS a security breach to change the ownership to anything other than root
- if you wanted to recompile an Ubuntu package from source, their package manager would place the source for package in /usr/src/{package dir}
/usr/local
- owner is root
- location where you'd place your own localized changesod the Distribution (/usr/local will be empty after a base install)
- contains the following subdirectories:
- bin
- etc
- games
- include
- lib
- man
- sbin
- share
- src
- this directory tree is meant to be used for software that you install by yourself, without using the distribution CD(s). For example, /usr/local/bin, /usr/local/sbin are for the binaries that are installed by you separately, /usr/local/etc is for config files etc. Thus /usr/local/src is for source files that you yourself downloaded. Example: go (binary distribution) gets installed in /usr/local/go.
- If you upgrade your system, files under the /usr tree get overwritten, such as /usr/bin, /usr/sbin, /usr/src etc. However, anything under /usr/local will not be touched. This is why all the software you installed separately should go to /usr/local tree.
- a place to install files built by the administrator, typically by using the make command (e.g., ./configure; make; make install). The idea is to avoid clashes with files that are part of the operating system, which would either be overwritten or overwrite the local ones otherwise (e.g., /usr/bin/foo is part of the OS while /usr/local/bin/foo is a local alternative).
- for self, inhouse, compiled and maintained software.
- reserved for software installed locally by the sysadmin
- place where you want to install software along with source files (for other programs to use or for people to look at)
- It's not meant, however, to be a workspace. Since it is local, you can do whatever you want, of course, but this isn't designed to be the place to put your software development tree.
/usr/local/src
- owner is root
- Source code for locally installed software
- If you downloaded a program not managed by your distribution and wanted to compile/install it, FHS dictates that you do that in /usr/local/src.
- a good place for downloading third party source code (eg for patching and rebuilding packages), not my own source code
/opt
- This directory is reserved for all the software and add-on packages that are not part of the default installation. All third party applications should be installed in this directory. (Linux Filesystem Hierarchy: /opt)
- a directory for installing unbundled packages (i.e. packages not part of the Operating System distribution, but provided by an independent source), each one in its own subdirectory. They are already built whole packages provided by an independent third party software distributor. Unlike /usr/local stuff, these packages follow the directory conventions (or at least they should). For example, someapp would be installed in /opt/someapp, with one of its command being /opt/someapp/bin/foo, its configuration file would be in /etc/opt/someapp/foo.conf, and its log files in /var/opt/someapp/logs/foo.access. (What is the difference between /opt and /usr/local?)
- for non-self, external, prepackaged binary/application bundle installation
- directory where you can just toss things and see if they work makes a whole lot of sense. I know I'm not going to go through the effort of packaging things myself to try them out. If the app doesn't work out, you can simply rm the /opt/mytestapp directory and that application is history.
- used for third-party software, which in the context of Ubuntu, means precompiled software that is not distributed via Debian packages
- A program that is installed in /opt is supposed to be self-contained.
- The main reason for using /opt is to provide a common standard path where external software can be installed without interfering with the rest of the installed system. /opt does not appear in standard compiler or linker paths (gcc -print-search-dirs or /etc/ld.so.conf etc.), so headers and libraries installed there are somewhat isolated from the main system and shouldn't interfere with already-installed programs. (Why should I move everything into /opt?)
Installing Software
How to install software available in Package Repository?
Installing from Ubuntu Package Repository
Example: Installing VLC player$ sudo apt-get update
$ sudo apt-get install vlc
It's best to run sudo apt-get update first as this updates local information about what packages are available from where in what versions. This can prevent a variety of installation errors (including some "unmet dependencies" errors), and also ensures you get the latest version provided by your enabled software sources.
There is also an apt version of this command:
$ sudo apt update
...
Reading package lists... Done
Building dependency tree
Reading state information... Done
23 packages can be upgraded. Run 'apt list --upgradable' to see them.
...
To list all upgradable packages:
$ sudo apt list --upgradable
To upgrade all packages:
$ sudo apt upgrade
To see all installed packages:
$ sudo apt list --installed
To check if some package has already been installed:
$ sudo apt list --installed | grep package_name
Installing from non-default (3rd Party) Package Repository
Example: Installing PowerShell from Microsoft Package Repository
# Download the Microsoft repository GPG keys
wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb
# Register the Microsoft repository GPG keys
sudo dpkg -i packages-microsoft-prod.deb
# Update the list of products
sudo apt-get update
# Enable the "universe" repositories
sudo add-apt-repository universe
# Install PowerShell
sudo apt-get install -y powershell
# Start PowerShell
pwsh
If you install the wrong version of packages-microsoft-prod.deb you can uninstall it with:
$ sudo dpkg -r packages-microsoft-prod
(Reading database ... 254902 files and directories currently installed.)
Removing packages-microsoft-prod (1.0-3) ...
How to install software distributed via Debian package (.deb) files?
Some applications are not available in Debian Package Repository but can be downloaded as .deb files.$ sudo dpkg -i /path/to/deb/file
$ sudo apt-get install -f
The latter is necessary in order to fix broken packages (install eventual missing/unmet dependencies).
How to install a deb file, by dpkg -i or by apt?
Another example: Etcher
Debian and Ubuntu based Package Repository (GNU/Linux x86/x64)
Add Etcher debian repository:
echo "deb https://deb.etcher.io stable etcher" | sudo tee /etc/apt/sources.list.d/balena-etcher.list
Trust Bintray.com's GPG key:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 379CE192D401AB61
Update and install:
sudo apt-get update
sudo apt-get install balena-etcher-electron
Uninstall
sudo apt-get remove balena-etcher-electron
sudo rm /etc/apt/sources.list.d/balena-etcher.list
sudo apt-get update
Debian and Ubuntu based Package Repository (GNU/Linux x86/x64)
Add Etcher debian repository:
echo "deb https://deb.etcher.io stable etcher" | sudo tee /etc/apt/sources.list.d/balena-etcher.list
Trust Bintray.com's GPG key:
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 379CE192D401AB61
Update and install:
sudo apt-get update
sudo apt-get install balena-etcher-electron
Uninstall
sudo apt-get remove balena-etcher-electron
sudo rm /etc/apt/sources.list.d/balena-etcher.list
sudo apt-get update
How to install applications distributed via snaps?
Snaps are containerised software packages. They're designed to install the programs within them on all major Linux systems without modification. Snaps do this by developers bundling a program's latest libraries in the containerized app.Snap updates automatically and carries all its library dependencies, so is a better choice for users who want ease of deployment and to stay on top of the latest developments.
Snapcraft - Snaps are universal Linux packages
Installing snap on Ubuntu | Snapcraft documentation
How to Install and Use Snap on Ubuntu 18.04 - codeburst
Working with Archive files
To unpack the .tar.gz in the current directory use:
$ tar -zxvf archive_file.tar.gz
-x = extract
-f (--file) = use archive file; this flag has to be the last in the list of flags and to be followed by the archive file name
-v (--verbose) = verbose output
-z (--gzip, --gunzip, --ungzip) = filter the archive through gzip
To unpack only the specific directory from the archive use:
$ tar -zxvf archive_file.tar.gz dir_name
To unpack archive to the specific directory:
$ tar -zxvf archive_file.tar -C path/to/dest_dir
-C (--directory) stands for "Change to directory"
Example:
$ tar -xzf go1.12.1.linux-amd64.tar -C /usr/local
$ sudo tar -xzvf Postman-linux-x64-7.5.0.tar.gz -C /opt
Why compressed directories cannot be extracted in /opt?
$ sudo apt-get install rar unrar
then go to the directory where all x.party.rar files are and execute:
$ unrar x -e file.part1.rar
ZIP format unzipping into the specified directory:
$ unzip bookmarks.zip -d bookmarks-dev-413
$ unzip bazel-1.2.1-dist.zip -d bazel bazel-1.2.1-dist
Unzip files in particular directory or folder under Linux or UNIX - nixCraft
Networking
What is the difference between curl and wget?
curl vs Wget
How to download files in Linux from command line with dynamic url
to download a file when it is pointed by a dynamic url, all you need to to is to use single quotes for the url; -O = specify the output filename
$ wget 'http://some.site.com/download?id=234&status=download' -O output_filename
What to do if ping does not help?
Options are:
$ ping example.com
$ traceroute example.com
$ curl -v example.com
$ wget example.com
How to find what DNS servers are used on the local machine?
$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0
search whatever.example.com
To find out which DNS servers are used by each network adapter:
$ systemd-resolve --status
Global
DNSSEC NTA: 10.in-addr.arpa
16.172.in-addr.arpa
168.192.in-addr.arpa
17.172.in-addr.arpa
18.172.in-addr.arpa
19.172.in-addr.arpa
20.172.in-addr.arpa
21.172.in-addr.arpa
22.172.in-addr.arpa
23.172.in-addr.arpa
24.172.in-addr.arpa
25.172.in-addr.arpa
26.172.in-addr.arpa
27.172.in-addr.arpa
28.172.in-addr.arpa
29.172.in-addr.arpa
30.172.in-addr.arpa
31.172.in-addr.arpa
corp
d.f.ip6.arpa
home
internal
intranet
lan
local
private
test
Link 137 (enxa44cc8e41d0f)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 136 (vethed1f504)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 38 (br-53b4f1b3fbda)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 5 (br-3c8c9487a095)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 4 (docker0)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 3 (wlp2s0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: x.y.z.q
x.y.z.v
DNS Domain: ~.
whatever.example.com
Link 2 (enp0s31f6)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
lines 44-81/81 (END)
Another way:
Options are:
- wget
- curl
- traceroute
- installation: $ sudo apt install traceroute
- lft
- "layer 4 traceroute"
- $ lft -S 192.168.0.20
$ ping example.com
$ traceroute example.com
$ curl -v example.com
$ wget example.com
How to find what DNS servers are used on the local machine?
$ cat /etc/resolv.conf
# This file is managed by man:systemd-resolved(8). Do not edit.
#
# This is a dynamic resolv.conf file for connecting local clients to the
# internal DNS stub resolver of systemd-resolved. This file lists all
# configured search domains.
#
# Run "systemd-resolve --status" to see details about the uplink DNS servers
# currently in use.
#
# Third party programs must not access this file directly, but only through the
# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
# replace this symlink by a static file or a different symlink.
#
# See man:systemd-resolved.service(8) for details about the supported modes of
# operation for /etc/resolv.conf.
nameserver 127.0.0.53
options edns0
search whatever.example.com
To find out which DNS servers are used by each network adapter:
$ systemd-resolve --status
Global
DNSSEC NTA: 10.in-addr.arpa
16.172.in-addr.arpa
168.192.in-addr.arpa
17.172.in-addr.arpa
18.172.in-addr.arpa
19.172.in-addr.arpa
20.172.in-addr.arpa
21.172.in-addr.arpa
22.172.in-addr.arpa
23.172.in-addr.arpa
24.172.in-addr.arpa
25.172.in-addr.arpa
26.172.in-addr.arpa
27.172.in-addr.arpa
28.172.in-addr.arpa
29.172.in-addr.arpa
30.172.in-addr.arpa
31.172.in-addr.arpa
corp
d.f.ip6.arpa
home
internal
intranet
lan
local
private
test
Link 137 (enxa44cc8e41d0f)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 136 (vethed1f504)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 38 (br-53b4f1b3fbda)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 5 (br-3c8c9487a095)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 4 (docker0)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
Link 3 (wlp2s0)
Current Scopes: DNS
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
DNS Servers: x.y.z.q
x.y.z.v
DNS Domain: ~.
whatever.example.com
Link 2 (enp0s31f6)
Current Scopes: none
LLMNR setting: yes
MulticastDNS setting: no
DNSSEC setting: no
DNSSEC supported: no
lines 44-81/81 (END)
Another way:
$ nmcli dev show | grep 'IP4.DNS'
IP4.DNS[1]: x.y.z.q
IP4.DNS[2]: x.y.z.v
DNS on Ubuntu 18.04
How to block some domain?
Add entry e.g.
0.0.0.0 domain.to.be.blocked
to /etc/hosts and then flush DNS cache:
How To Flush Linux / UNIX DNS Cache - nixCraft
How to direct network traffic use Proxy server?
Use environment variables
export http_proxy=10.21.32.70:8080
export https_proxy=10.21.32.70:8080
Domain names can be used instead of IP addresses.
How To Use Proxy Server To Access Internet at Shell Prompt With http_proxy Variable - nixCraft
How to test proxy?
curl -v -x 11.22.33.44:8080 -L 'https://www.example.com/examples/1'
-x, --proxy [protocol://]host[:port]
Use the specified proxy.
The proxy string can be specified with a protocol:// prefix. No
protocol specified or http:// will be treated as HTTP proxy. Use
socks4://, socks4a://, socks5:// or socks5h:// to request a spe‐
cific SOCKS version to be used. (The protocol support was added
in curl 7.21.7)
HTTPS proxy support via https:// protocol prefix was added in
7.52.0 for OpenSSL, GnuTLS and NSS.
To let cURL follow redirects (3xx statuses) add -L:
-L, --location
(HTTP) If the server reports that the requested page has moved
to a different location (indicated with a Location: header and a
3XX response code), this option will make curl redo the request
on the new place.
To see HTTP status code, use verbose flag:
-v, --verbose
How to block some domain?
Add entry e.g.
0.0.0.0 domain.to.be.blocked
to /etc/hosts and then flush DNS cache:
How To Flush Linux / UNIX DNS Cache - nixCraft
How to direct network traffic use Proxy server?
Use environment variables
export http_proxy=10.21.32.70:8080
export https_proxy=10.21.32.70:8080
Domain names can be used instead of IP addresses.
How To Use Proxy Server To Access Internet at Shell Prompt With http_proxy Variable - nixCraft
How to test proxy?
curl -v -x 11.22.33.44:8080 -L 'https://www.example.com/examples/1'
-x, --proxy [protocol://]host[:port]
Use the specified proxy.
The proxy string can be specified with a protocol:// prefix. No
protocol specified or http:// will be treated as HTTP proxy. Use
socks4://, socks4a://, socks5:// or socks5h:// to request a spe‐
cific SOCKS version to be used. (The protocol support was added
in curl 7.21.7)
HTTPS proxy support via https:// protocol prefix was added in
7.52.0 for OpenSSL, GnuTLS and NSS.
To let cURL follow redirects (3xx statuses) add -L:
-L, --location
(HTTP) If the server reports that the requested page has moved
to a different location (indicated with a Location: header and a
3XX response code), this option will make curl redo the request
on the new place.
To see HTTP status code, use verbose flag:
-v, --verbose
Hardware management
To verify if you're running a 64-bit system use GNU Core Utility: uname.
Swap Memory Management
To check virtual memory statistics (the latest reading) use:
$ vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 138816 9157820 1739872 11514740 0 1 1559 366 102 64 26 7 66 0 0
$ vmstat 1
To disable and then reenable swap:
$ sudo swapoff -a
$ sudo swapon -a
performance - How to empty swap if there is free RAM? - Ask Ubuntu
Displays/Monitors Management
$ xrandr --listmonitors
Monitors: 2
0: +*HDMI-1 2560/677x1440/381+0+0 HDMI-1
1: +eDP-1 1920/344x1080/193+112+1440 eDP-1
To see details for the current display:
$ xrandr --current
Screen 0: minimum 320 x 200, current 1920 x 1080, maximum 8192 x 8192
eDP-1 connected primary 1920x1080+0+0 (normal left inverted right x axis y axis) 344mm x 193mm
1920x1080 60.03*+ 60.01 59.97 59.96 59.93 48.02
1680x1050 59.95 59.88
1600x1024 60.17
1400x1050 59.98
1600x900 59.99 59.94 59.95 59.82
...
360x202 59.51 59.13
320x180 59.84 59.32
HDMI-1 disconnected (normal left inverted right x axis y axis)
DP-1 disconnected (normal left inverted right x axis y axis)
HDMI-2 disconnected (normal left inverted right x axis y axis)
DP-2 disconnected (normal left inverted right x axis y axis)
HDMI-3 disconnected (normal left inverted right x axis y axis)
$ ls /dev/video*
To check the number of processors:
$ grep -c ^processor /proc/cpuinfo | sed 's/^0$/1/'
4
To see details for all processors:
$ cat /proc/cpuinfo
To list CPU architecture information, use lscpu. E.g.
$ lscpu | grep "Byte Order"
Byte Order: Little Endian
SSH
How to generate SSH key pair on Ubuntu
How to test SSH key password on Ubuntu
---
To get base64 encoding of a string:
$ echo -n my_string | base64
or
$ printf my_string | base64
---
TBC...
How to get the GPU info?
lshw = list hardware
-C = class
$ sudo lshw -C display
*-display UNCLAIMED
description: 3D controller
product: GM107GLM [Quadro M620 Mobile]
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:01:00.0
version: a2
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress bus_master cap_list
configuration: latency=0
resources: memory:ee000000-eeffffff memory:d0000000-dfffffff memory:e0000000-e1ffffff ioport:e000(size=128) memory:ef000000-ef07ffff
*-display
description: VGA compatible controller
product: Intel Corporation
vendor: Intel Corporation
physical id: 2
bus info: pci@0000:00:02.0
version: 04
width: 64 bits
clock: 33MHz
capabilities: pciexpress msi pm vga_controller bus_master cap_list rom
configuration: driver=i915 latency=0
resources: irq:130 memory:ed000000-edffffff memory:c0000000-cfffffff ioport:f000(size=64) memory:c0000-dffff
NVidia and Intel in same Laptop: which card is used?
$ dig TXT +short o-o.myaddr.l.google.com @ns1.google.com | awk -F'"' '{ print $2}'
Bash script should start with shebang e.g.
#!/bin/bash
---
Escaping and preserving special characters in bash
Bash wildcards: ? (question mark) and * (asterisk) | LinuxG.net
What is the meaning of a question mark in bash variable parameter expansion as in ${var?}? - Stack Overflow
shell - Expansion of variable inside single quotes in a command in Bash - Stack Overflow
linux command line find escape question mark - Server Fault
---
What does 'cd $_' mean?
$_ expands to the last argument to the previous simple command* or to previous command if it had no arguments. Typical use:
mkdir dirA && cd $_
---
To see all Bash commands, execute:
$ help
---
$ help set
set: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
Set or unset values of shell options and positional parameters.
Change the value of shell attributes and positional parameters, or
display the names and values of shell variables.
Options:
-a Mark variables which are modified or created for export.
-b Notify of job termination immediately.
-e Exit immediately if a command exits with a non-zero status.
-f Disable file name generation (globbing).
-h Remember the location of commands as they are looked up.
-k All assignment arguments are placed in the environment for a
command, not just those that precede the command name.
-m Job control is enabled.
-n Read commands but do not execute them.
-o option-name
Set the variable corresponding to option-name:
allexport same as -a
braceexpand same as -B
emacs use an emacs-style line editing interface
errexit same as -e
errtrace same as -E
functrace same as -T
hashall same as -h
histexpand same as -H
history enable command history
ignoreeof the shell will not exit upon reading EOF
interactive-comments
allow comments to appear in interactive commands
keyword same as -k
monitor same as -m
noclobber same as -C
noexec same as -n
noglob same as -f
nolog currently accepted but ignored
notify same as -b
nounset same as -u
onecmd same as -t
physical same as -P
pipefail the return value of a pipeline is the status of
the last command to exit with a non-zero status,
or zero if no command exited with a non-zero status
posix change the behaviour of bash where the default
operation differs from the Posix standard to
match the standard
privileged same as -p
verbose same as -v
vi use a vi-style line editing interface
xtrace same as -x
-p Turned on whenever the real and effective user ids do not match.
Disables processing of the $ENV file and importing of shell
functions. Turning this option off causes the effective uid and
gid to be set to the real uid and gid.
-t Exit after reading and executing one command.
-u Treat unset variables as an error when substituting.
-v Print shell input lines as they are read.
-x Print commands and their arguments as they are executed.
-B the shell will perform brace expansion
-C If set, disallow existing regular files to be overwritten
by redirection of output.
-E If set, the ERR trap is inherited by shell functions.
-H Enable ! style history substitution. This flag is on
by default when the shell is interactive.
-P If set, do not resolve symbolic links when executing commands
such as cd which change the current directory.
-T If set, the DEBUG and RETURN traps are inherited by shell functions.
-- Assign any remaining arguments to the positional parameters.
If there are no remaining arguments, the positional parameters
are unset.
- Assign any remaining arguments to the positional parameters.
The -x and -v options are turned off.
Using + rather than - causes these flags to be turned off. The
flags can also be used upon invocation of the shell. The current
set of flags may be found in $-. The remaining n ARGs are positional
parameters and are assigned, in order, to $1, $2, .. $n. If no
ARGs are given, all shell variables are printed.
Exit Status:
Returns success unless an invalid option is given.
---
What is the difference between [] and [[]]?
Bash Brackets Quick Reference
Test if a command outputs an empty string
if [[ $(ls -A) ]]; then
echo "there are files"
else
echo "no files found"
fi
---
Create a directory if this does not exist:
[ -d path/to/mydir ] || mkdir -p path/to/mydir
(-p option instructs mkdir to create all intermediate parent directories to mydir)
---
Semicolon in conditional structures
The semicolon is needed only when the end of line is missing:
if [ "a" == "a" ] ; then echo "true" ; fi
Without semicolons, you get Syntax error.
---
How to capture exit code of the application most recently executed in Terminal?
$ echo $?
It can also be used in a bash script, e.g.:
ginkgo -r
if [[ $? != 0 ]]; then
echo "Unit tests failed. Terminating build process..."
exit 1
fi
---
$ dependencies=(build-essential cmake pkg-config libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libavresample-dev python3-dev libtbb2 libtbb-dev libtiff-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev libgtk-3-dev libcanberra-gtk3-module libatlas-base-dev gfortran wget unzip)
We can print the array:
$ echo ${dependencies[@]}
build-essential cmake pkg-config libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libavresample-dev python3-dev libtbb2 libtbb-dev libtiff-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev libgtk-3-dev libcanberra-gtk3-module libatlas-base-dev gfortran wget unzip
We can print elements with specific indexes in the array:
$ echo ${dependencies[0]}
$ echo ${dependencies[1]}
---
---
To get base64 encoding of a string:
$ echo -n my_string | base64
or
$ printf my_string | base64
---
TBC...
Getting Information about Hardware
How to get the GPU info?
lshw = list hardware
-C = class
$ sudo lshw -C display
*-display UNCLAIMED
description: 3D controller
product: GM107GLM [Quadro M620 Mobile]
vendor: NVIDIA Corporation
physical id: 0
bus info: pci@0000:01:00.0
version: a2
width: 64 bits
clock: 33MHz
capabilities: pm msi pciexpress bus_master cap_list
configuration: latency=0
resources: memory:ee000000-eeffffff memory:d0000000-dfffffff memory:e0000000-e1ffffff ioport:e000(size=128) memory:ef000000-ef07ffff
*-display
description: VGA compatible controller
product: Intel Corporation
vendor: Intel Corporation
physical id: 2
bus info: pci@0000:00:02.0
version: 04
width: 64 bits
clock: 33MHz
capabilities: pciexpress msi pm vga_controller bus_master cap_list rom
configuration: driver=i915 latency=0
resources: irq:130 memory:ed000000-edffffff memory:c0000000-cfffffff ioport:f000(size=64) memory:c0000-dffff
NVidia and Intel in same Laptop: which card is used?
$ lspci -k | grep -EA2 'VGA|3D'
00:02.0 VGA compatible controller: Intel Corporation Device 591b (rev 04)
Subsystem: Dell Device 07a9
Kernel driver in use: i915
--
01:00.0 3D controller: NVIDIA Corporation GM107GLM [Quadro M620 Mobile] (rev a2)
Subsystem: Dell GM107GLM [Quadro M620 Mobile]
Kernel modules: nvidiafb, nouveau
Networking
How to get public IP address of the computer?
$ dig TXT +short o-o.myaddr.l.google.com @ns1.google.com | awk -F'"' '{ print $2}'
GNU Bash Shell Commands
Robert Muth: Better Bash Scripting in 15 MinutesBash script should start with shebang e.g.
#!/bin/bash
---
Escaping and preserving special characters in bash
Bash wildcards: ? (question mark) and * (asterisk) | LinuxG.net
What is the meaning of a question mark in bash variable parameter expansion as in ${var?}? - Stack Overflow
shell - Expansion of variable inside single quotes in a command in Bash - Stack Overflow
linux command line find escape question mark - Server Fault
---
$_
What does 'cd $_' mean?
mkdir dirA && cd $_
---
To see all Bash commands, execute:
$ help
---
set
$ help set
set: set [-abefhkmnptuvxBCHP] [-o option-name] [--] [arg ...]
Set or unset values of shell options and positional parameters.
Change the value of shell attributes and positional parameters, or
display the names and values of shell variables.
Options:
-a Mark variables which are modified or created for export.
-b Notify of job termination immediately.
-e Exit immediately if a command exits with a non-zero status.
-f Disable file name generation (globbing).
-h Remember the location of commands as they are looked up.
-k All assignment arguments are placed in the environment for a
command, not just those that precede the command name.
-m Job control is enabled.
-n Read commands but do not execute them.
-o option-name
Set the variable corresponding to option-name:
allexport same as -a
braceexpand same as -B
emacs use an emacs-style line editing interface
errexit same as -e
errtrace same as -E
functrace same as -T
hashall same as -h
histexpand same as -H
history enable command history
ignoreeof the shell will not exit upon reading EOF
interactive-comments
allow comments to appear in interactive commands
keyword same as -k
monitor same as -m
noclobber same as -C
noexec same as -n
noglob same as -f
nolog currently accepted but ignored
notify same as -b
nounset same as -u
onecmd same as -t
physical same as -P
pipefail the return value of a pipeline is the status of
the last command to exit with a non-zero status,
or zero if no command exited with a non-zero status
posix change the behaviour of bash where the default
operation differs from the Posix standard to
match the standard
privileged same as -p
verbose same as -v
vi use a vi-style line editing interface
xtrace same as -x
-p Turned on whenever the real and effective user ids do not match.
Disables processing of the $ENV file and importing of shell
functions. Turning this option off causes the effective uid and
gid to be set to the real uid and gid.
-t Exit after reading and executing one command.
-u Treat unset variables as an error when substituting.
-v Print shell input lines as they are read.
-x Print commands and their arguments as they are executed.
-B the shell will perform brace expansion
-C If set, disallow existing regular files to be overwritten
by redirection of output.
-E If set, the ERR trap is inherited by shell functions.
-H Enable ! style history substitution. This flag is on
by default when the shell is interactive.
-P If set, do not resolve symbolic links when executing commands
such as cd which change the current directory.
-T If set, the DEBUG and RETURN traps are inherited by shell functions.
-- Assign any remaining arguments to the positional parameters.
If there are no remaining arguments, the positional parameters
are unset.
- Assign any remaining arguments to the positional parameters.
The -x and -v options are turned off.
Using + rather than - causes these flags to be turned off. The
flags can also be used upon invocation of the shell. The current
set of flags may be found in $-. The remaining n ARGs are positional
parameters and are assigned, in order, to $1, $2, .. $n. If no
ARGs are given, all shell variables are printed.
Exit Status:
Returns success unless an invalid option is given.
set -x
- enables a mode of the shell where all executed commands are printed to the terminal.
- typically used for debugging
What is the difference between [] and [[]]?
Bash Brackets Quick Reference
Test if a command outputs an empty string
if [[ $(ls -A) ]]; then
echo "there are files"
else
echo "no files found"
fi
---
Create a directory if this does not exist:
[ -d path/to/mydir ] || mkdir -p path/to/mydir
(-p option instructs mkdir to create all intermediate parent directories to mydir)
---
Semicolon in conditional structures
The semicolon is needed only when the end of line is missing:
if [ "a" == "a" ] ; then echo "true" ; fi
Without semicolons, you get Syntax error.
---
How to capture exit code of the application most recently executed in Terminal?
$ echo $?
It can also be used in a bash script, e.g.:
ginkgo -r
if [[ $? != 0 ]]; then
echo "Unit tests failed. Terminating build process..."
exit 1
fi
---
Variables
$ dependencies=(build-essential cmake pkg-config libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libavresample-dev python3-dev libtbb2 libtbb-dev libtiff-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev libgtk-3-dev libcanberra-gtk3-module libatlas-base-dev gfortran wget unzip)
We can print the array:
$ echo ${dependencies[@]}
build-essential cmake pkg-config libavcodec-dev libavformat-dev libswscale-dev libv4l-dev libxvidcore-dev libavresample-dev python3-dev libtbb2 libtbb-dev libtiff-dev libjpeg-dev libpng-dev libtiff-dev libdc1394-22-dev libgtk-3-dev libcanberra-gtk3-module libatlas-base-dev gfortran wget unzip
We can print elements with specific indexes in the array:
$ echo ${dependencies[0]}
$ echo ${dependencies[1]}
---
No comments:
Post a comment