Tuesday, August 21, 2007

Introduction To Bash Shell Scripting

Review

  • Getting Help

    • man Get off information about commands
    • man -k Get off information via keyword
    • info Read info documents
    • help Show help on shell built-in commands
    • command --help Help on command .i.e. cp --help Will give help on cp
    • bash -c "help" Short help on bash
    • bash -c "help set" Short help on bash options
  • Special Characters

    • Space Argument separator
    • \ Quote a single character A \ followed by a carriage return extend the current line
    • ' Quote rext with spaces in it i.e. 'Hello world'
    • $' Quote allows string expansion, backslash-escaped characters
    • " Quote rext, expands variables and command substitution
    • ` Command substitution i.e. echo "The date is `date`"
    • $ Denote a shell variable
    • # Comments the rest of the line
    • ; Commands separator
    • (commands) Run multiple commands in a subshell
    • Control-C Interrupt a command
    • Control-D Sends End of File from terminal
    • Control-U Erases the entire command line
    • Control-\ Is a stronger terminate than Control-c
    • Control-Z Suspend process
  • Redirection, Pipes and Filters

    • - Standard In for some commands
    • 0 Standard In
    • 1 Standard Out
    • 2 Standard Error
    • >2&1 Redirect standard error to standard out
    • &> Redirect standard error & standard out
    • <>
    • > Redirect output
    • >> Concatenate output to file
    • <<>
    • | Pipe
    • tee Command write to file and standard out
    • tee -a FILE #Allow appending to FILE
    • xargs Build command from standardin
  • Wildcards

    • * Wildcard for any character(s)
    • ? Wildcard for single character
    • [set] Wildcard for character in set
    • [^set ] Wildcard for not the character in the set
    • [!set ] Wildcard for not the character in the set
    • {ab,dc} Wildcard for alternate between commas
    • All wildcard work with existing files
    • Only {} alternate work to create files
  • Process Control

    • (command1; command) Run command1, then command2 in subshell
    • command1&&command Run command1, then command2 if command1 successes
    • command1||command Run command1, then command2 if command1 fails
  • Regular Expressions

    • Definition: Text pattern of text character and meta characters
    • Some Meta characters
      • Escape character \
      • Single Character Meta characters
        • . Matches any one character
        • [...] Matches any one character in a set
        • [^...] Matches any one character not in the set
      • Quantifiers
          >
        • * Matches the previous character zero or more times
        • \{n\} Matches the previous character n times
        • \{n,m\} Matches the previous character at least n & at most m
        • \{n,\} Matches the previous character n or more times
      • Anchors
        • ^ Matching at the start the line
        • $ Matching at the end of line
      • Grouping \( \)
  • Commands that use regular expressions
    • awk Pattern scanning and text processing language
    • ed Line-oriented text editor
    • egrep extended grep
    • emacs Emacs full screen text editor
    • ex Line-oriented text editor
    • expr Command evaluates an expression
    • fgrep Grep from patterns in a file
    • gawk GNU pattern scanning and processing language
    • grep Searches file for pattern (also see fgrep & egrep)
        grep [OPTIONS] PATTERN [INPUT-FILE...]
      • -E same as egrep
      • -c Count
      • -e pattern (for multiple pattern on line)
      • -f same as fgrep
      • -i Ignore case
      • -l Only list files containing pattern
      • -q Quit (No output, only Return Code)
      • -v Invert sense mode
    • perl Perl scripting
    • python Python scripting
    • sed Applies a set of user-specified editing command to a file
        sed [OPTIONS] 'sed_command' [INPUT_FILE...]
      • -n Suppress automatic printing
      • -e expression - sed_command
      • substitute other_text for some_text sed 's/some_text/other_text/g' FILE > NEWFILE
      • multiple changes sed -e 's@abc@def@g' -e 's@xyz@mno@g' FILE
      • print out line with faq in them sed -n '/faq/p' FILE
      • change Page ### to (Page ###) at end of line sed 's/Page [0-9]+$/(&)/' file # & replace the match
      • delete blank lines sed '/^[ \t]*$/d
    • tcl Tool command language
    • vi Full screen text editor

Shell Variables

  • Built-in Shell Variables

    • CDPATH Path of shortcuts for cd (like PATH)
    • COLUMNS Numbers of columns on the display
    • EDITOR Path for editor
    • HISTSIZE Number of commands in command history (default 500)
    • IFS Input Field Separator
    • LINES Numbers of lines on the display
    • OFS Output Field Separator
    • SECONDS Seconds that this shell is running
    • SHELLOPT Colon separate list of shell options
  • Environment Variables

    • export var Will make a variable an environment variable
    • HOME User's home directory
    • LOGNAME User's name
    • MAIL Name of user's mailbox
    • PATH List of directories to be search by the shell
      to find programs whose names are type as commands
    • PS1 String that is used by the shell prompt
    • PWD Name of current directory
    • SHELL Name of current shell
    • TERM The kind of terminal being used
    • Environment variable are global to shell and subshells
  • User Variables

    • Can be either upper or lower case
    • var=value Define a variable
    • var="" Define a variable as null
    • local var Define a variable local to its scope
  • Positional Variables

    • $0 Name of function or script being called
    • $1 ... $9 Replace by arguments to shell or function
    • ${n} Replace by n-th arguments to shell or function required if number of argument is over 9
  • Special Variables

    • $? Exit status or return code of last command
    • $# Number of arguments
    • $@ Argument 1 thru n with Input Field Separator
    • $* "$1" $2" ... $n
    • $! Process id of last background process
    • $$ Process id of shell running this script
    • $- The current shell flags

Some Command Useful With Scripts

  • basename Strip directory and option suffix
  • declare Built-in command declares variable
  • dirname Strip non-directory part
  • echo Built-in command display message to standard out
  • echo -n Built-in command display message to standard out without newline
  • echo -e Builtin Command display message to standout with escape sequences
  • enable Built-in command to enable/disable (-n) built-in commands
  • env Built-in command displays environment variables
  • eval Built-in command evaluate arguments before executing results
  • exec Built-in command runs command as the shell process
  • exit Built-in command exits shell
  • expr Evaluates an expression and output its value
  • false Built-in command always return false condition
  • local Built-in command make variable local
  • read Built-in command reads data into variable from standard in
  • seq Print a sequence of numbers
    • seq [OPTION]... LAST
    • seq [OPTION]... FIRST LAST
    • seq [OPTION]... FIRST INCREMENT LAST
    • -w equalize width by padding with leading zeroes
  • sleep Command causes execution to stop for a specified number of seconds
  • test Built-in command tests for various conditions, such as existence of a file, useful for controlling conditional script execution
  • time Built-in command times commands
  • times Print accumulated user and system times.
  • true Built-in command always return true condition
  • type Built-in command show what word is
  • wait wait for the completion of background processing; is used to ensure that critical processing is complete before proceeding in a script

Making Scripts Colorful

  • tput sgr0 #Reset text attributes to normal without clearing screen
  • Escape sequence to change colors #\e[${forground};${background}m example white on black sequence is \e[37;40m #Forground colors Backgrounfd Color black=30; bgblace=40 red=31; bgred=41 green=32; bggreen=42 yellow=33; bgyellow=43 blue=34; bgblue=44 magenta=35; bgmagenta=45 cyan=36; bgcyan=46 white=37; bgwhite=47
  • Turn text attribute off \e[0m
  • Change text attribute bold \e[1m
  • Change text attribute underline \e[4m
  • Change text attributer everse \e[7m

Shell Functions and Scripts

  • Functions

    • Function must be sourced just like .bashrc
    • type function will list the function
    • function functionname { shell commands }
    • functionname () { shell commands }
    • Example - a helpful function function givehelp { exec $1 --help | more; }
  • Scripts

    • #!/bin/bash Script name run script is run if execute bit is set
    • #!/bin/bash -x As above, but script lines are displayed
    • bash -x script As above
    • /usr/bin/env bash Use the environment bash for the script
  • Script Debugging options

      set -o OPTION Command line Action
    • set -o noexec sh -n Don't run command, just check for syntax
    • set -o verbose sh -v Echo commands before running them
    • set -o xtrace sh -x Echo commands after running them
    • set +o OPTION Turns OPTION off

Operation With Variables

  • Variable Usage

    • $var Value of variable var
    • "$var" Is null if variable is undefined avoids some shell syntax errors if variable is undefined
    • ${var} Value of variable var avoids confusion when concatenating with text
    • ${#var} Gives the length of the string contained in var
    • ${var:FIRST:N} Extract string from var starting at FIRST position and continuing for N-1 characters. Note FIRST starts at 0.
  • Passing an variable to a program or script

    • echo $var | command
  • Arrays

    • declare -a name Declare an array
    • name[index]=value Just assigning a value defines it
    • Index starts at zero
    • No maximum limit
    • Need not be contiguous
  • Setting Variable From Execution of Command

    • var=`command` var is set to output of command
    • var=$(command) same as above
  • Arithmetic Operation

    • $(( expression )) Almost the same as 'expr expression'
    • $[ expression ] Same as above
    • ( expression ) Groups expression within $(( ... ))
    • expr expression Note must quote *
  • Arithmetic operator

    • + add
    • - substract
    • * multiply
    • / divide
    • % remainder
    • # number conversion (Only in bash not expr)
  • Test Operation

    • [ -option arg ] Same as 'test -option arg
    • [[ -option arg ]] Same as 'test -option arg
    • [ arg1 -option arg2 ] Same as 'test arg1 -option arg2'
    • [[ arg1 -option arg2 ]] Same as 'test arg1 -option arg2'
    • See test command for test complete list of options
      • string1 = string2 String 1 equals string 2
      • string1 !=string2 String 1 not equals string 2
      • -n string String is not zero length
      • -z string String is zero length
      • -d FILE File is a directory
      • -e FILE File exists
      • -f FILE File exists and is a regular file
      • -r FILE File exists and is readable
      • -s FILE File exists and has length greater than zero
      • -w FILE File exists and is writable
      • -x FILE File exists and is executable
      • num1 -eq num2 Number 1 equals number 2
      • num1 -ne num2 Number 1 not equals number 2
      • num1 -lt num2 Number 1 less than number 2
      • num1 -le num2 Number 1 less than or equals number 2
      • num1 -gt num2 Number 1 greater than number 2
      • num1 -ge num2 Number 1 greater than or equals number 2
  • String Operation

    • ${varname:-word} Return var if exists and is not null, else word
    • ${varname:+word} Return word if var exists and is not null, else null
    • ${varname:?mess} Return var if exists and is not null, else display mess and return from script with error
  • Pattern Operations

    • ${variable#pattern} If the pattern matches the beginning of the variable value, delete the shortest part that matches and return the rest
    • ${variable##pattern} If the pattern matches the beginning the of variable value, delete the longest part that matches and return the rest
    • ${variable%pattern} If the pattern matches the end of the variable value, delete the shortest part that matches and return the rest
    • ${variable%%pattern} If the pattern matches the end of the variable value, delete the longest part that matches and return the rest

    The handy 'expr'

    • expr string : regrep Anchored pattern match
    • expr match string pattern Same as above
    • expr substr string pos {len] Substring beginning at pos of length len
    • expr length string Length of string

    Flow Control

    • if - general information

        if command               Test return code of command if command1 && command2  Test return code of command1 and command2 if command1 || command2  Test return code of command1 or command2  command can be condition i.e. [ $# -eq 0 ]
    • if/then/else or if/then/elif..

        if condition   then statements...   [else statements...] fi  if condition    then statements...   [elif condition     then statements...]     [else] statements...] fi 
    • return from Function

        return [ numeric expression or variable ]  This is the return code 
    • for

        for name [ in list ] do   statements  done  for  variable = start to end do   statements done 
    • while/until

        while condition do   statements done  until condition do   statements done 
  • break/continue

    • break [n] Break out of loop & select
    • continue [n] Continue next iteration of loop
  • case

      case expression in   pattern1[|pattern11] } statements ;;   pattern2[|pattern21] } statements ;;   ...  esac 

    User Interfaces

  • select

      select name [in list] do   statements that can use $name  done 
    • Generate a menu of each item in the list formatted with a number for each choice
    • Prompt the user for the number
    • Store the selected choice in name. The number is stored in REPLY
    • execute the statements in the do
    • Repeat the process again

    Command Line Options

    • shift Shift 1st argument form the argument list
    • getopts Used to process command line options
    • OPTIND Variable contains number of options
    • OPTARG Default variable for option
    • Example while getopts ":ab:c" opt; do case $opt in a) process_option_a;; b) process_option_b $OPTARG is the option's argument;; c) process_option_c;; esac done shift $(($OPTIND - 1))

    I/O in Scripts

    • printf Built-in command for formatted output
    • read Built-in command to read one line into variable(s) read #Everything entered goes to REPLY read var #Everything is read into var read a b #Read 1st word in to a and rest into b read -t 300 var #Read with 300 second timeout
    • Example how to use read from $file while read line; do do_something_with_line done <$file

    Process Handling

    • Signals

        Name  Number  Control Character 
      • EXIT 0 Used to trap exiting script
      • HUP 1 Logout
      • INT 2 Control-C
      • QUIT 3 Control-\
      • KILL 9 can not be ignored or trapped
      • TERM 15 Default kill
      • TSTP 24 Control-Z
    • Traps

      • trap "" signal-list Ignore signal
      • trap "cmds" signal-list Execute commands if signal is caught
      • trap signal-list Reset signal to original condition
      • trap : signal-list (undocumented) ignore signal, pass to child Signal are normally not passed to subprocesses
      • Examples trap 'rm tmpfile; exit' 0 1 2 #remove tmpfile on exit, logout, interrupt trap "echo 'You hit Control-C'" INT while true ; do sleep 60 done
      • Example parent child process #!/bin/bash #parent echo parent running trap 'echo parent exiting; exit' 0 trap :2 child sleep 1000 #!/bin/bash #child echo child started. pid is $$ trap 'echo child got signal 2; exit' INT sleep 1000

    Examples Useful Scripts or Function

    • # Function top5 Example how to set defaults # Usage top5 {n} #list n processes function top5 { ps -ef | head -${1:-5} }
    • # Function hereis Example of HERE IS FILE and handling arguments # Usage hereis word1 word2 ... function hereis { for name in "$@" do cat <
    • # Function pick Return selected items by user # Usage: .e.g var=`pick *` function pick { for name in $@ ; do #for each item in argument list echo -n "$name (y/n/q)?" >/dev/tty #ask user to select read ans #read answer from standard in case $ans in #Check choices y*) echo $name;; #selected q*) break;; #skip rest of arguments *) continue;; #skip item esac done }
    • # Function acal Display a nicer calendar # but will accept Alphabetic month function acal { m="" case $# in 0) cal; return;; #no arguments 1) m=$1; y=`date +%Y`;; #1 argument 2) m=$1; y=$2;; #2 arguments esac case $m in Jan*|jan* ) m=1;; Feb*|feb* ) m=2;; Mar*|mar* ) m=3;; Apr*|apr* ) m=4;; May|may } m=5;; Jun*|jun* ) m=6;; Jul*|jul* ) m=7;; Aug*|aug* ) m=8;; Sep*|sep* } m=9;; Oct*|oct* ) m=10;; Nov*|nov* ) m=11;; Dec*|dec* ) m=12;; [1-9]|1[0-2] ) ;; #numeric month *) ) y=$m; m="";; esac cal $m $y }
    • ## Function selectex - Example select # function selectex () { choices="/bin /usr /home" select selection in $choices; do if [ $selection ]; then ls $selection break else echo 'Invalid selection' fi done }
    • # Function fwhich Which command in $PATH is executed # function fwhich { if [[ $# -eq 0 ]] ; then cat <<>&2; return 2 Usage fwhich command #Example of parsing the $PATH Return 0 - found Return 1 - not found Return 2 - No arguments EndOfHelp fi for path in `echo $PATH | sed 's/^:/.:/ s/::/.:/g s/:$/:./ s/:/ /g'` do if [[ -x $path/$1 ]] ; # does executable file exists here? then echo $path/$1 # found it return 0 fi done return 1 # not found }
    • # Name: overwrite Copy standard input to output after EOF function overwrite { if [[ $# -lt 2 ]] ; then echo "Usage: overwrite file command [args]" 1>&2; return 2 fi file=$1; shift new=/tmp/overwrite1$$; old=/tmp/overwrite2$$ trap 'rm -f $new $old; return 1' 1 2 15 # clean up files "$@" > $new if [[ $? -eq 0 ]] ; # collect output then # command completed successfully cp $file $old # save original file trap '' 1 2 15 # we are committed; ignore signals cp $new $file # copy new file into file rm -f $new $old # remove temp files else echo "overwrite: $1 failed, $file unchanged" 1>$2 return 1 fi }
    • # Name: zgrep # Purpose: caseless grep of gzip files # Usage: zgrep text files.gz # function zgrep { if [ $# -eq 0 ] ; then echo "Usage: zgrep grep_text files.gz" return 2 fi text=$1 shift while [ $# -gt 0 ] do echo $1 gzip -cd $1 | grep -i $text shift done }
    • # Name: hgrep # Purpose: highlighting grep # Usage: hgrep pattern files # function hgrep { if [ $# -lt 2 ] ; then echo "Usage: hgrep pattern files" return 2 fi pattern=$1;shift sep=$'\001' #note use of $' ' to create control characters bold=$'\e[1m'; off=$'\e[0m' underline=$'\e[4m'; reverse=$'\e[7m' #other choices of highlighting sed -n "s${sep}${pattern}${sep}${reverse}&${off}${sep}gp" $* }
  • No comments: