Working with files in Linux

 

Creating a file

To create a file use touch:

$ touch filename

Writing into a file

It is possible to use redirection operators > and >> to achieve this:

• > will overwrite existing file or crate a new file
• >> will append text to existing file or create a new file

> file.txt

What does “>” do vs “>>”?

Next two commands have the same output - if file.txt does not exist, it will be created and string test (without quotes will be written into it):

$ echo "test" > file.txt

$ echo test > file.txt

$ cat test.txt 

test

tee command

If we want command's output to be written in the terminal but also into a file, we can pipe the  command into the tee command which takes stdin and writes it to two outputs, one being a file and another stdout. tee will crate the file if it does not exist:

$ echo test | tee test3.txt

test

$ cat test3.txt 

test

How does tee differ from redirecting/appending? - Unix & Linux Stack Exchange

shell - What is the purpose of 'tee'? - Super User

[SOLVED] usage of tee and redirecting output to /dev/null

tee command can also help when we need to write into a file that only superuser has write permissions. Our current bash terminal might be running as non-privileged user and so e.g. this command will fail:

echo line >> /etc/apt/sources.list

Using

sudo echo line >> /etc/apt/sources.list

...will not help as only echo will be executed as sudo.

The solution is:

echo line | sudo tee /etc/apt/sources.list > /dev/null

> /dev/null makes stdout being dumped to /dev/null.

Writing multi-line text into file

shell - How to append multiple lines to a file - Unix & Linux Stack Exchange

Ending file with new line character

[No newline at end of file]

It is a good style to always put the newline as a last character if it is allowed by the file format.

Unix historically had a convention of all human-readable text files ending in a newline. Reasons:
Practically, because many Unix tools require or expect it for proper display.
Philosophically, because each line in a text file terminates with an "end-of-line" character--the last line shouldn't be any exception.

To write into file a set of lines which end with a new line character:

$ echo $'first line\nsecond line\nthirdline\n' > foo.txt

$'...' construct expands embedded ANSI escape sequences

How to put a newline special character into a file using the echo command and redirection operator?

Difference between printf and echo:

$ printf "hello \n"

hello 

$ printf "hello " // note that new line is not appended automatically

hello $ echo "hello \n"

hello \n

$ echo "hello"

hello

$  // new line is appended automatically

Getting the information about a file

To get the number of lines (well, newline characters) in the file:

$ wc -l myfile.txt
23 myfile.txt

(This is why it's important to follow the convention and end each line with newline character.)

To get the number of words on some webpage:

$ curl "https://example.com/" 2>/dev/null | grep -i "word1|word2" | wc -l


To see the last couple of lines in the file use command tail:

$ tail myfile

To find various hash sums of a file:

$ md5sum file_name
$ sha1sum file_name
$ sha256sum file_name


If file is Windows executable, it is possible to examine it with:

$ exiftool somefile.exe 

To install exiftool:

$ sudo apt install libimage-exiftool-perl

linux - viewing dll and exe file properties/attributes via the command line - Unix & Linux Stack Exchange

Checking whether file exists

if test -f "$symlink_file"; then
   echo "$symlink_file" exists and is regular file.
else
   echo "$symlink_file" does not exist or is not a regular file.
fi

if test -L "$regular_file"; then
   echo "$regular_file" exists and is symlink file.
else
   echo "$regular_file" does not exist or is not a symlink file.
fi

How to Check if a File or Directory Exists in Bash

Copying files

cp - copy

cp [OPTIONS] SOURCE DEST 
SOURCE - file or directory
DEST - file or directory

An error is reported if directory is specified as source and file as destination.

$ cp -r test test.txt
cp: cannot overwrite non-directory 'test.txt' with directory 'test'

Copying files and directories to/from remote machine

Secure File Copy (scp) Tool | My Public Notepad

 

Moving files

$ mv *.{jpg,gif,png} ~/Pictures

Renaming files

To rename all .new files in the current directory to *.old:

$ rename -f -v 's/.new/.old/' *

-f = force; allows overwriting existing *.old files
-v = verbose

File viewing and editing

To simply view the content of some file, use cat:

$ cat filename

To edit some file, you can use vi editor. Example:

$ vi ~/.profile 

gedit can also be used as graphic editor:

sudo gedit ~/.profile

To enter some special character (e.g. bulletpoint) press CTRL+SHIFT+U and underscored "u" should appear (u). Then use numeric keyboard to type in the Unicode code of the character (e.g. 2022) and press Enter. [source]

To see the content of the file as binary and hexadecimal:

xxd -b file
xxd file

Searching for Files

To search file from the root directory use /:

$ find / -name "file_name.ext"

To find any file or directory which contains some string in their name, recursively, starting from the current directory:

$ find . -name "*STRING*" 

Searching for text across files

How do I find all files containing specific text on Linux?

man grep

$ grep -rnw '/path/to/somewhere/' -e 'pattern'

-r or -R = recursive,
-n = line number
-w = match the whole word.
-l (lower-case L) = just give the file name of matching files

--include=\*.{c,h} =  search through those files which have .c or .h extensions

--exclude=*.o = exclude searching all the files ending with .o extension

--exclude-dir={dir1,dir2,*.dst} = exclude a particular directory(ies)

-e PATTERN = string pattern to be searched

-i = ignore the case

Example:

$ grep -r /var/lib/go/src/ -e "CodeDecode"

/var/lib/go/src/encoding/json/bench_test.go:func BenchmarkCodeDecoder(b *testing.B) {

Example:

$ find ./go/src/pkg -type f -name "*.go" | xargs egrep '^type.*(er|or) interface {'

xargs manual - xargs builds and executes command lines from standard input
egrep manual - egrep prints lines matching a pattern

Comparing Files

How to ignore line endings when comparing files?

$ diff --strip-trailing-cr file1 file2


How to detect file ends in newline?

Working with executable files

Running a command prefixed by the time command will tell us how long our code took to execute.

$ time myapp
real 0m13.761s
user 0m0.262s
sys 0m0.039s

If an executable is present but some of its dependencies are missing bash (or sh) might display an error messages stating that main executable is not found (which might be a bit misleading).

Example:

/ # ls
bin       myapp      data-vol  dev       etc       home      lib       media     mnt       opt       proc      root      run       sbin      srv       sys       tmp       usr       var
/ #  myapp
/bin/sh:  myapp: not found