Bash hints to know or how to make complex issues simple


Expantion is at heart of bash. It changes prefixed '$' or special characterer like '*' expression by their computed values and occurs before evaluating command. One may use this feature for write shorter script. Pathname expansion is an incredibly powerful tool to avoid having to specify exact pathnames in our arguments, or to go looking through our file system for the files we need. Brace expantion use the same idea but do not depend on filesystem.
# basic list
echo {1,2,3,4,5}
# result : 1 2 3 4 5
# range list
echo {1..5}
# result : 1 2 3 4 5
# usefull for loop 
for i in {1..5}  ; do echo $i ; done
# Preamble and postscript ( ie    preamble{case1,case2} or {case1,case2}postscript )
echo file{.txt,.log}
# result : file.txt file.log
# brace expension and file expantion
ls *{.txt,.log} # equivalent to "ls *.txt  *.log"

String operations

#trailing substitution
echo ${I%bar}
# result :foo
#leading substitution
echo ${I#foo}
# result :bar
#full substitution
STR="the wall is black"
echo ${STR/black/blue}
# result : the wall is blue
F="hello HELLO"
echo ${F^^}
# result : HELLO HELLO
echo ${F,,}
# result :hello hello
echo ${F:5:3}
# result : fgh
echo ${#F}
# result : 8
echo ${F%.log}.txt
# result : my_file.txt

Processing many files at once

for F in `find . -name "*.java"` ; do
  echo $F
  # rename .java to .txt
  mv $F ${}.txt

Processing larger dataset or files

Let's dive into the details about those two loops : for and while. Let's face it 'for' is fine but while both versions will work in general, the second variant is better for the following reasons:
for i in `find . -type d`
     # do some processing on the found directory
Should be replace by
find . -type d | while read i
    # do some processing on the found directory
As more cores are available and when parallel processing is required, one may use gnu parallel 'for loop'. Methode using gnu parallel should be prefered as it has been developped on purpose to handle this issue.
# standard sequential for loop
for F in $(find . -name *.mp3) ; do mpg -w $F ${F%.mp3}.wav ; done
# using gnu parallel : parallel creation of bz2 archive of all *.bin file
find . -name '*.bin' 2>/dev/null | parallel tar  -cjf {1}.tar.bz2 {1}
#  parallel but without gnu parallel : parallel execution in max N-process concurrent
for i in {1..100}; do
        # .. do your stuff here
        echo "starting task $i.."
        sleep $(( (RANDOM % 3) + 1))
    ) &
    # allow only to execute $N jobs in parallel
    if [[ $(jobs -r -p | wc -l) -gt $N ]]; then
        # wait only for first job
        wait -n
# wait for pending jobs
echo "all done"

Special trick

# list of arguments passed to script as string
echo $*
# list of arguments passed to script as delimited list
echo $@
#number of arguments passed to current script
echo $# 
# name of shell script (relative path from the current directory)
echo $0 
# argument 1
echo $1
#pid of the current shell
echo $$
bash -c "exit 24"
#last exit status
echo $?
#PID of the most recent background ( empty if not set)
echo "PID :"$!
cat /proc/cpuinfo >/dev/null &
echo "PID :"$!
#the (input) field separator 
echo -n "$IFS" | od -abc
SET="1 2 3 4 5"
for I in $SET ; do  echo "I="$I ;done
# result : 
echo -n "$IFS" | od -abc
for I in $SET ; do   echo "I="$I ;done
# result : 
#I=1 2 3 4 5
touch 'test test'
for F in `find . -name "* *"` ; do
   OLDF=`echo $F | sed -e 's# #\ #g'`
   echo $OLDF   
   NEWF=`echo $F | sed -e 's/ /_/g'`
   echo $NEWF
   mv $OLDF $NEWF