bash-workshop

references

preface

goals of this workshops
1. introduction to linux filesystem
  - files
  - directories
  - permissions
  - symbolic links
  - hard links
2. introduction to command line
  - standard commands with examples
  - data streams (with redirection)
3. workshop tasks are in workshop file and the answers are in answers file

linux filesystem

file is a collection of data items stored on disk
- is always associated with devices like hard disk, floppy disk, USB pen drive and more
- inode
  - is a file data structure that stores information about any file except its name and data
  - stores the file’s metadata, including all the storage blocks on which the file’s data can be found
    - if you save a file that exceeds a standard block, your computer will find the next available segment on which to store the rest of your file
      - over time, that can get super confusing
directory is a group of files
- two types
  - root directory
    - only one root directory
    - denoted by /
    - root of your entire file system tree
  - sub directory
    - directory under root
- inside every directory, you will find out two default sub-directories
  - . – current directory
  - .. – pointer to previous directory
    - directory immediately above the current one I am in now
- some typical directories
  - / - root directory
  - /bin - executable programs which are needed in single user mode
    - contains common commands like cat, cp, cd, ls, etc.
  - /dev - files, which refer to physical devices such as hard disk, keyboard, monitor, mouse and modem etc
  - /etc- all your system configuration files in it
  - /home - for a particular user of the system and consists of individual files
  - /lib - hold shared libraries that are necessary to boot the system and to run the commands
  - /opt - contains packages not part of the Operating System distribution, but provided by an independent source
  - /root - home directory for the root user
  - /tmp - temporary files which may be deleted with no notice
permissions
- four groups X(XXX)(XXX)(XXX)
  - example: drwxr-xr-x
- group 1: type
  - -: regular file
  - d: directory
  - l: symbolic link
  - s: socket
- group 2: owner permissions
  - r: read only file permission
  - w: write only file permission
  - x: execute only file permission
  - –: no permission
- group 3: group permissions
- group 4: other permissions
symbolic link vs hard link
- symbolic link
  - shortcut that reference to a file instead of its inode value
  - like a shortcut in Windows
  - can be seen as a static link to the last known location of the original file
- hard links
  - direct reference to a file via its inode
  - only files and not directories

terminal

terminal: user@computerName:~$
- ~ shortcut for current user home directory
commands are just text you type in the terminal
commands are interpreted by the shell
- shell - program that interprets the commands and passes them on to the operating system
the terminal is the window to the shell
commands structure
- commandName options inputs
- echo $PATH
  - in what directories the shell will search for a specific command
  - example: which echo
    - echo is /usr/bin/echo
    - you could type /usr/bin/echo 'Hello' or echo 'Hello'
    - type -a echo
      - echo is a shell builtin
  - that's how the shell knows where to find your commands
  - commandNames need to be on the shell's search path
- there can be input for options
  - cal -A 1 12 2017
    - cal (-A 1) 12 2017: 1 is input for A option
    - cal -A=1 -B=1 12 2017
open: ctrl alt t
close: ctrl d
terminal: user@computerName:~$
- ~ shortcut for current user home directory
search in history
- ctrl r
suggestions: double tap TAB

data streams

types
- standard input
- standard output
- standard error
redirections
- data streams can be redirected from their default locations to wherever you wish
- piping: redirecting the standard output of one commands to the standard input of another
- example
  - standard input: 0
    - wc -l < test.txt
  - standard output: 1
    - discard stdout: > /dev/null // or 1> /dev/null > is just a shortcut
      - /dev/null is known as the null device
      - whatever you write to /dev/null will be discarded
  - standard error: 2
    - cat -k bla 2>> error.txt // k not existing option
    - no error in terminal
  - &>> result.txt
    - redirect and append both stdout and stderr to file result.txt
ways to get data into and out of a command
- standard input (2)
  - cmd arguments and standard
- standard output (2)
  - error and standard
- standard output is connected by default to terminal
- standard input is connected by default to the keybord

commands

awk
- operations
  1. scans a file line by line
  2. splits each input line into fields
  3. compares line/fields to pattern
  4. performs action(s) on matched lines
- used for
  - transform (processing, manipulating, formatting) data files
  - searching
  - produce reports
- awk '{print $1}' file
- awk '{print $1 "," $2}' file
- awk -F '.' '{print $1 "," $2}' file // separator
- awk 'NR>=122 && NR<=129 { print }' // for row numbers
- awk '$1>5 { print }' // filtering by column
cat
- name derived from its function to concatenate files
- view content
  - cat file
- create new file with content
  - cat > FileWithContent then enter then insert content ctr c
- copy content of 1 file to another
  - cat file > file2
  - cat file1 file2 > file3
- append data
  - cat file1 >> file2
cd
- change directory
- cd dirName
- cd
  - return to home
cp source target
- used for copying files and directories to another location
- cp a.txt b.txt
- cp -r source target
  - for copying dirs
cut
- used to extract sections from each line of input — usually from a file
- cut -d "." -f1 // delimited by . and only first column
  - default delimiter is "TAB"
  - split only by single character
- cut -d "." -f1-3 // delimited by . and range
- cut c5- file // since 5 to the end
file filename
- shows type of the file
- if you change the name of the file to xxx.jpg it still prints png
  - in linux file extensions doesn't matter
find haystack/ -type f -name "needle.txt" -exec command {} ;
- used to find files and directories and perform subsequent operations on them
- for each result, command {} is executed
  - all occurences of {} are replaced by the filename
grep textToSearch file
- used for searching plain-text data sets for lines that match a regular expression
- grep -v textToSearch fileToSearchIn // line that does not have textToSearch
- grep -f - fileToSearchIn
  - -f tells grep to obtain its search pattern from a file and - tells it that this file is actually stdin
less file
- displays the contents of a file or a command output, one page at a time
locate *.log
- search for files
  - doesn't read the file system for the searched file or directory name
  - refers to a database (prepared by the command updatedb)
- very fast but requires updating
  - updatedb
    - sudo updatedb (to run as an root)
  - sudo - super user do
  - done automatically daily
ls
- list files or directories
- ls -F (shows differences between dirs and files)
  - directories: Downloads/
  - files: file.txt - without "/"
- ls -a
  - plus hidden files / directories
head/tail file
- display the beginning of a text file or piped data
history
- list history of commands
- then !1 - chose first and executes
- !! - most recent cmd
- !v latest cmd that starts with v
- !. latest cmd that start with .
mkdir folder
- create folder
- mkdir -p /a/b/c
  - create whole path even if dir doesn't exist
  - mkdir a/b/c - error
mv source target // for files and directories
- moves files or directories from one place to another
- mv foo.txt bar.txt
  - rename foo.txt to bar.txt
- mv file1.txt file.2.txt file3.txt folder
pwd
- writes the full pathname of the current working directory
- $oldpwd - prev directory
- ~+ for pwd
- ~- for oldpwd
rm
- remove file
rm -r
- remove directory
- rmdir - removes if empty
sed 's/search-pattern/replacement-string/flags' file
- display and edit data
- edit
  - sed 's/a/b/' 'a'
    - result: 'b'
  - doesn't modify the original file
  - replace only first occurrence
  - g flag
    - replace all
- display
  - sed -n '3p' file // 3rd row
  - sed -n '2,4p' file // range of lines
- sed vs awk
  - sed is used to process and modify text, awk is used for analysis and reporting
sort words.txt
- sort -r words.txt // sort words.txt | tac
- sort -n numbers.txt // sort by number not lexicographical
- sort -u number.txt // unique
- sort -k 2 employee.txt
- sort -t : -k 2n employee.txt
  - set delimiter to : // default is blank spaces
tar
- create, extract, or list files from a tar file
- tar -cvf filename.tar file[1..3].txt
  - c – create an archive file
  - v – show the progress of the archive file
  - f – filename of the archive file
  - t – viewing the content of the archive file
- tar -xvf ourarchive.tar
  - x - extract
- compression
  - gzip
    - gzip ourarchive.tar // compressed in place
    - gunzip ourarchive.tar // uncompress in place
    - tar -cvzf archive.tar.gz ... // z - gzip
    - tar -xvzf archive.tar.gz // extract gzip
tee
- save a data snapshot without breaking pipeline
  - redirection of stdout breaks pipeline
- find "4DOS" wikipedia.txt | tee 4DOS.txt | sort > 4DOSsorted.txt
  - searches the file wikipedia.txt for any lines containing the string "4DOS"
  - makes a copy of the matching lines in 4DOS.txt
  - sorts the lines
  - writes them to the output file 4DOSsorted.txt
touch filename
- create empty file
wildcards
- *
  - matches zero or more occurrences
  - ls Documents/ Downloads/ Pictures/
    - same as: ls Do* Pi* // assuming that there are only
- ?
  - matches single occurrence
  - ls file?.txt
- [123456]
  - matches 1 place
  - ls file[12345].txt
xargs
- converts input from standard input into arguments to a command
- cat file | xargs echo // vs echo file

bash

conventions
- https://google.github.io/styleguide/shellguide.html
bash = most commonly used linux shell today
what is script
- a shell script is a file containing commands for the shell
- different shells can interpret same text different ways
how should bash script begin
```
`#!/bin/bash`

set -o errexit
set -o pipefail
set -o nounset
```
- #!/bin/bash
  - first line; means: use bash to interpret this file
  - path to the bash shell
    - to obtain that path type which bash
  - = sharp, ! = bang, #! = shebang
- set -o errexit
  - exit immediately if a command exits with a non-zero status
- set -o pipefail
  - return value is the status of the last command to exit with a non-zero status, or zero if no command exited with a non-zero status
- set -o nounset
  - treat unset variables as an error when substituting
- other flags: help set
declaring/calling structures
- variable: str="Hello World!"
  - echo ${STR}
- function
```
hello() {
    local first_arg=$1 // access to first argument
}
```
  then calling: hello "arg"

control structures

if [[ condition ]] // ex. $NUM -gt 5
then
  command // ex. echo "Hello word!"
fi

for

for i in {0..3}
do
  echo "Number: $i"
done

while

while [ $i -le 2 ]
do
  echo Number: $i
  ((i++))
done

arguments
- $1 // without default
- ${1:-foo} // with default
options
- if an invalid option is provided, the option variable is assigned the value ?
  - this behaviour is only true when you prepend the list of valid options with : to disable the default error handling of invalid options
  - it is recommended to always disable the default error handling in your scripts
- shift $((OPTIND -1))
  - it is common practice to call the shift command at the end of your processing loop to remove options that have already been handled from $@
- options that themselves have arguments are signified with a :
  - if no argument is provided getopts will set opt to :
- example
```
while getopts ":d:h" opt; do
  case ${opt} in
    d )
      dir_count=$OPTARG
      ;;
    h )
      echo "Usage:"
      echo "-h                      help"
      echo "-d                      how many directories to create"
      exit 0
      ;;
    \? )
      echo "use -h option for help"
      exit 1
      ;;
    : )
      echo "Invalid option: $OPTARG requires an argument" 1>&2
      exit 1
      ;;
  esac
done
shift $((OPTIND -1))
```
intercepting output of commands
- $(command)
- called command substitution
  - bash performs the expansion by executing command in a subshell environment and replacing the command substitution with the standard output of the command
- subshell starts out as an almost identical copy of the original shell process
  - under the hood, the shell calls the fork system call, which creates a new process whose code and memory are copies
  - in particular, they have the same variables
  - subshell is thus different from executing a script

debugging

add lines before section you want to debug

set -x // print commands and their arguments as they are executed
set -v // print shell input lines as they are read

traps
- allows you to catch signals and execute code when they occur
  - signals are asynchronous notifications that are sent to your script when certain events occur
- bash provides a psuedo-signal called "EXIT", which is executed when your script exits
  - this can be used to make sure that your script executes some cleanup on exit

mtumilowicz / linux-terminal-bash-workshop Goto Github PK

linux-terminal-bash-workshop's Introduction

bash-workshop

preface

linux filesystem

terminal

data streams

commands

bash

= sharp, ! = bang, #! = shebang

linux-terminal-bash-workshop's People

Contributors

Watchers

Forkers

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent