Unix and Linux network configuration. Multiple network interfaces. Bridged NICs. High-availability network configurations.


Reviews of latest Unix and Linux software. Helpful tips for application support admins. Automating application support.


Disk partitioning, filesystems, directories, and files. Volume management, logical volumes, HA filesystems. Backups and disaster recovery.


Distributed server monitoring. Server performance and capacity planning. Monitoring applications, network status and user activity.

Commands & Shells

Cool Unix shell commands and options. Command-line tools and application. Things every Unix sysadmin needs to know.

Home » Commands & Shells

Useful Bash One-Liners

Submitted by on May 2, 2016 – 8:43 am

Here’s a  loosely-organized pile ‘o shell quickies I kept googling now and then and finally decided to write ’em down.

[accordion] [spoiler title=”Shady shit” style=”fancy”]

Set custom mtime/atime on a file.

NOTE: Keep in mind, this does not affect ctime. To change ctime you may need to change system time first, or use some other trick. Here’s a cool one: “Faking the Uptime in Linux“.
touch -t 201303120234 /tmp/oldfile
touch -d '-1 year' /tmp/oldfile

Change a file’s mtime/atime based on its current timestamp
touch -r /tmp/oldfile -d '-1 year' /tmp/oldfile

Hide command from w (will still show up in ps output, though)
perl -e '
perl -e '${0} = "fake_command"; system("real_command");'
= "fake_command"; system("real_command");'

Clear your Bash history
cat /dev/null > ~/.bash_history && history -c && exit

Share contents of current directory via HTTP:
cd /tmp ; python -m SimpleHTTPServer <port>

Create a tunnel from localhost:2001 to remotehost:80
ssh -f -N -L2001:localhost:80 remotehost

Create a tunnel from localhost:2001 to remotehost:80 via bridgehost
ssh -f -N -L2001:remotehost:80 bridgehost

Tunnel your SSH connection via intermediary:
ssh -t reachable_host ssh unreachable_host

Output your microphone to other computer’s speaker:
dd if=/dev/dsp | ssh username@host dd of=/dev/dsp

Securely delete files and directories:
yum -y install coreutils srm wipe

shred -zvu -n  5 /tmp/dir1/secret_file
wipe -rfi /tmp/dir1/*
srm -vz /tmp/dir1/*
for i in `seq 1 5`; do cat /dev/random > /tmp/dir1/secret_file; done && /bin/rm -f /tmp/1gb_file

Using Bash brace expansion to generate commands with no spaces
Connection to 443 port [tcp/https] succeeded!

[/spoiler] [spoiler title=”Awk/sed/tr stuff” style=”fancy”]

Replace newline with comma

sed ':a;N;$!ba;s/\n/ /g'

Remove leading spaces and tabs
sed 's/^[ \t]*//'

Remove single spaces only (leave multiple spaces):
sed 's/\(.\) //g'

Move first line to the end of list
sed '1,1{H;1h;d;};$G'

Show allocated disk space:
df -klP -t xfs -t ext2 -t ext3 -t ext4 -t reiserfs | grep -oE ' [0-9]{1,}( +[0-9]{1,})+' | awk '{sum_used += $1} END {printf "%.0f GB\n", sum_used/1024/1024}'

Show used disk space:
df -klP -t xfs -t ext2 -t ext3 -t ext4 -t reiserfs | grep -oE ' [0-9]{1,}( +[0-9]{1,})+' | awk '{sum_used += $2} END {printf "%.0f GB\n", sum_used/1024/1024}'

Summarizing line data with awk:
# Sample data

# Add up the values in the second column
awk -F"," '{s+=$2}END{print s}' temp

# Add up the values in the second column only for ID2
awk -F, '$1=="ID2"{s+=$2;}END{print s}' temp
v="ID2"; awk -F, -v v="${v}" '$1==v{s+=$2;}END{print s}' temp

# List unique values in the first column
awk -F, '{a[$1];}END{for (i in a)print i;}' temp

# Add up values in the second column for each ID
awk -F, '{a[$1]+=$2;}END{for(i in a)print i", "a[i];}' temp

# Add up values in the second column for each ID and print total
awk -F, '{a[$1]+=$2;x+=$2}END{for(i in a)print i", "a[i];print "Total,"x}' temp

# Print the maximum second-column value for each group
awk -F, '{if (a[$1] < $2)a[$1]=$2;}END{for(i in a){print i,a[i];}}' OFS=, temp

# Print the number of occurrences for each ID
awk -F, '{a[$1]++;}END{for (i in a)print i, a[i];}' temp

# Print the first entry for each ID
awk -F, '!a[$1]++' temp

# Concatenate values for each ID
awk -F, '{if(a[$1])a[$1]=a[$1]":"$2; else a[$1]=$2;}END{for (i in a)print i, a[i];}' OFS=, temp

Extract URLs:
sed -n 's/.*href="\([^"]*\).*//p'

Preserve symlinks when using sed -i
cd /etc/httpd/conf.d && sed -i --follow-symlinks 's/192.168.1/192.168.2/g' *.conf

Append each string with a consecutive number:
awk -vRS=string '{$0=n$0;ORS=RT}++n'

Flush awk buffers when piping from STDIN for continuous output:
| awk {'print $1; fflush();'}

Print fields set in a Shell variable:
fields="1 3 4"
command | awk -v fields="${fields}" 'BEGIN{ n = split(fields,f) }
{ for (i=1; i<=n; ++i) printf "%s%s", $f[i], (i<n?OFS:ORS) }'

Print lines if third field is unique (PPID in this example):
ps -ef | grep [s]plunk | awk '!seen[$3]++'

Similar to above, but print second field (PID) if third field (PPID) is unique:
ps -ef | grep [s]plunk | awk '!seen[$3]++ {print $2}'

Show the primary IP of a local machine:
ifconfig | sed -rn 's/;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*//p'

Verify that local machine’s IP matches DNS:
if [ "$(ifconfig | sed -rn 's/;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*//p')" == "$(dig +short $(host -TtA $(hostname -s) | grep "has address" | awk '{print $1}'))" ]; then echo 0 ; else echo 1 ; fi

Show primary NIC:
route | grep -m1 ^default | awk '{print $NF}'

Show prefix (netmask in CIDR notation):
ip addr show "$(route | grep -m1 ^default | awk '{print $NF}')" | grep -w inet | grep -v | awk '{ print $2}' | cut -d "/" -f 2

Show broadcast address:
ip addr show "$(route | grep -m1 ^default | awk '{print $NF}')" | grep -w inet |grep -v|awk '{ print $4}'

Show local machine’s network in CIDR notation:
eval $(ipcalc -np $(ifconfig $(route | grep -m1 ^default | awk '{print $NF}') | sed -n "s/inet addr:\([^ ]*\).*Mask:\([^ ]*\).*/ /p")) ; echo $NETWORK/$PREFIX

Calculate sum from stdout and do math
| awk '{ SUM += $1} END { print ( SUM/1024 )"MB" }'

Calculate allocated and used local filesystem storage
df -klP -t ext2 -t ext3 -t ext4 -t reiserfs | grep -oE ' [0-9]{1,}( +[0-9]{1,})+' | awk '{sum_alloc +=$1; sum_used += $2} END {printf "%.2f / %.2f (GB)\n", sum_alloc/1024/1024, sum_used/1024/1024}'

Find gaps is numerical sequences
awk '$1!=p+1{print p+1"-"$1-1}{p=$1}'

Grepping with awk
echo "514/tcp   open  shell" | awk '{match($1,"^[0-9]+/[a-z]+") && match($2,"open")}END{print $1,$2,$3}'

Grepping with awk on a specific column
ls -l | awk '$3 == "root"'

Grepping with sed and also printing the headers (first line)
sed '1p;/pattern/!d'

Extract lines between unique tags using sed. Sample input file:
cat /tmp/testfile.txt

# Header 1
Line 11
Line 12
# Header 2
Line 21
Line 22
Line 23
# Header 3
Line 31
Line 32
Line 33

sed -n '/# Header 2/{:a;n;/# Header 3/b;p;ba}' /tmp/testfile.txt

Line 21
Line 22
Line 23

Extract lines contained within the second set of <header></header> tags using sed. Sample input file:
cat /tmp/testfile2.txt

Line 11
Line 12
Line 21
Line 22
Line 23
Line 31
Line 32
Line 33

sed -n '\|<header>|{:n;\|</header>|!{N;bn};y|\n| |;p}' /tmp/testfile2.txt | sed -n '2{p;q}'

<header> Line 21 Line 22 Line 23 </header>

Delete lines between two tags not including the tags:
sed "/<tag open>/,/<\/tag close>/{//!d}"

Delete line between two tags including the tags:
sed "/<tag open>/,/<\/tag close>/d"

Delete all lines after a tag not including the tag:
sed "/<\tag close>/,$d"

Delete lines 12 through 23:
sed "12,23d"

Remove dupes, spaces, extra semicolons from BASh PATH
PATH=$(xargs -d: -n1 <<<${PATH} | sed 's/ //g' | sort -u | xargs | sed 's/\b*//g;s/ /:/g')

Remove duplicate words in a line:
awk '{ while(++i<=NF) printf (!a[$i]++) ? $i FS : ""; i=split("",a); print "" }'

Remove duplicate lines in a file without sorting:
awk '!a[$0]++'

Print number of characters for each line in a file:
awk '{ print length($0)"\t"$0; }' file.txt

Insert a Unicode character into specific column position in a file:
sed -r -e 's/^.{15}/&\xe2\x86\x92\x0/' file.txt

Replace multiple newlines with a single newline
sed '/^$/N;/^\n$/D' file.txt

Preserve original search string and add to it
(Example: replace every [0-9]. with [0-9]..)
ls | sed -e 's/\([0-9]\.\)/\./g'

(Example 2: enclose every four-digit number followed by a dot in parentheses, i.e. 2014. becomes (2014).)
| sed -e 's/\([0-9]\{4\}\)\./\(\)\./g')"

Merge every two adjacent lines (sed wins):
awk 'NR%2{printf $0" ";next;}1'
# or
sed 'N;s/\n/ /'

Get hard drive model and size:
for i in `fdisk -l 2>/dev/null | egrep -o "/dev/sd[a-z]" | sort -u` ; do hdparm -I ${i} 2>/dev/null; done | egrep "Model|size.*1000" | awk -F: '{print $NF}' | awk 'NR%2{printf $0" ";next;}1'

Identify server’s primary IP address:
/sbin/ifconfig | sed -rn 's/;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*//p'

Print all fields but first:
awk '{$1=""; print $0}'

Print all fields but last:
awk '{$NF=""; print $0}'

Print all fields but last and preserve field delimiters:
awk -F'/' -v OFS='/' '{$NF=""; print $0}')"

Print all fields but the first two:
awk '{$1=$2=""; print $0}'

Print fields from 9th to last:
awk '{ s = ""; for (i = 9; i <= NF; i++) s = s $i " "; print s }'

Comment-out a line in a file containing a regex match:
sed -re '/REGEX/ s/^#*/#/' -i /tmp/file

Uncomment a file containing a regex match:
sed -re '/REGEX/ s/^#*//' -i /tmp/file

Convert upper- to lower-case with tr and sed:
tr '[:upper:]' '[:lower:]'
sed -e 's/\(.*\)/\L/'

Convert to “Title Case”:
sed 's/.*/\L&/; s/[a-z]*/\u&/g'

Insert “E” into string at position #3:
sed -r -e 's/^.{3}/&E/'

Print text between the first occurrence of tag “foo” and the last occurrence of tag “bar”:
sed -n '/foo/{:a;N;/^\n/s/^\n//;/bar/{p;s/.*//;};ba};'

Prepend a shell variable to a string using awk:
| awk -v var="${shell_var}" '{print var$0}'

Roundup number to the nearest multiple of 10:
awk '{print sprintf("%.0f",$0/10)*10}'

[/spoiler] [spoiler title=”Sequences and Combinations” style=”fancy”]
# Brace expansion

echo {a,c}{a,c}
aa ac ca cc

echo {a..c}{a..c}
aa ab ac ba bb bc ca cb cc

echo {'word1 ','word2 '}{'word1, ','word2, '}
word1 word1,  word1 word2,  word2 word1,  word2 word2,

# For loop with brace expansion

charset={a,b}; group=2; rep=; for ((i=0; i<${group}; i++)); do rep="${charset}${rep}"; done; eval echo ${rep}
aa ab ba bb

charset={1..3}; group=2; rep=; for ((i=0; i<${group}; i++)); do rep="${charset}${rep}"; done; eval echo ${rep}
11 12 13 21 22 23 31 32 33

charset=({a..c} {A,Z} {0..2})
  (($1 == 0)) && { echo "$2"; return; }
  for i in "${charset[@]}"
    permute "$(( - 1 ))" "${i}"
permute 3|tail -5

# Crunch word list generator

v="3.6" && wget -O /tmp/crunch-${v}.tgz${v}.tgz && \
tar xvfz crunch-${v}.tgz && cd /tmp/crunch-${v} && make && make install

crunch 3 3 ab 2>/dev/null

crunch 0 0 -p abc 2>/dev/null

# With Unicode characters
echo | crunch 0 0 -p яйца 2>/dev/null

[/spoiler] [spoiler title=”xargs & parallel” style=”fancy”]

SSH to prdweb001 through prdweb007 and lookup the OS version and the number of CPUs. The number of parallel processes for xargs is set to the number of CPU cores. The ts command comes from the moreutils package.

seq 1 7 | xargs -P$(grep -c proc /proc/cpuinfo) -I% bash -c "ssh -qtT -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=3 -o BatchMode=yes prdweb00% 'head -1 /etc/issue;grep -c proc /proc/cpuinfo' | ts prdweb00%:" | sort

prdweb001: 8
prdweb001: CentOS release 5.10 (Final)
prdweb002: 2
prdweb002: CentOS release 5.10 (Final)

Query name servers through for SOA record, grab the 10-digit serial and verify that all name servers have the same DNS zone serial number. This can be useful for identifying name servers that are not updating in a timely fashion.
seq 1 8 | xargs -P8 -I% dig +nsid -t SOA | grep -oP "[0-9]{10}" | sort -u | wc -l

Similar to above, but the names of the name servers are non-sequential. The number of xargs threads is set to the number of array element.
declare -a nsarray=('sojwiu01' 'kwjsiu01' 'sljhuw01' 'hwikdj01' 'lskwid01' 'ldhuwy01' 'sjducn01' 'vfjqod01')
printf '%s\n' ${nsarray[@]} | xargs -P$(printf '%s\n' ${#nsarray[@]}) -I% dig +nsid -t SOA | grep -oP "[0-9]{10}" | sort -u | wc -l

[/spoiler] [spoiler title=”wget & curl” style=”fancy”]

Download tar.gz and uncompress with a single command:

wget -q -O - | tar xz

Download tar.bz2 and uncompress with a single command:
wget -q -O - | tar xj

Download in background, limit bandwidth to 200KBps, do not ascend to parent URL, download only newer files, do not create new directories, download only htm*,php and, pdf, set 5-second timeout per link.
wget -b --limit-rate=200k -np -N -m -nd --accept=htm,html,php,pdf --wait=5 "${url}"

Download recursively, span multiple hosts, convert links to local, limit recursion level to 4, fake “mozilla” user agent, ignore “robots” directives.
wget -r -H --convert-links --level=4 --user-agent=Mozilla "${url}" -e robots=off

Generate a list of broken links:
wget --spider -o broken_links.log --wait 2 -r -p "${url}" -e robots=off

Download new PDFs from a list of URLs:
wget -r --level=1 -H --timeout=2 -nd -N -np --accept=pdf -e robots=off -i urls.txt

Save and use authentication cookie:
wget -O ~/.domain_cookie_tmp "${username}&amp;password=${password}"
grep "^cookie" ~/.domain_cookie_tmp | awk -F'=' '{print $2}' > ~/.domain_cookie
wget -c --no-cookies --header="Cookie: enc=`cat ~/.domain_cookie`" -i "${url_file}" -nc

Use wget with anonymous proxy:
export http_proxy=proxy_server:port
wget -Y -O /tmp/yahoo.htm ""

Use wget with authorized proxy:
export http_proxy=proxy_server:port
wget -Y --proxy-user=${username} --proxy-passwd=${password} \
-O /tmp/yahoo.htm ""

Make a local mirror of a Web site:
wget -U Mozilla -m -k -D ${domain} --follow-ftp \
--limit-rate=50k --wait=5 --random-wait -np "${url}" -e robots=off

Download images from a Web site:
wget -r -l 0 -U Mozilla -t 1 -nd -D ${domain} \
-A jpg,jpeg,gif,png "${url}" -e robots=off

Download and run a remote script:
bash <(curl -s0

Same as above, but start a python Web server instance on the remote server first:
# On the remote_server:
d=/var/adm/bin && f=${d}/ && echo -e "#\!/bin/bash\necho 'This is a test'" > ${f} && chmod 755 ${f}
cd ${d} && python -m SimpleHTTPServer 81

# On the local_server
bash <(curl -s0
This is a test

[/spoiler] [spoiler title=”Shell arrays” style=”fancy”]

Declare and populate array manually:

declare -a a=('first element' 'second element' 'fifth element')

Store output of shell commands in an array
IFS=$'\n' ; a=($(command1 | command2)) ; unset IFS

Output contents of an array one element per line
printf '%s\n' ${a[@]}

Output contents of an array one element per line when elements contains spaces
for ((i = 0; i < ${#a[@]}; i++)) ; do echo "${a[$i]}" ; done

Read contents of a file into an array
old_IFS=$IFS ; IFS=$'\n' ; a=($(grep ERROR /var/log/messages))
printf '%s\n' \"${a[@]}\"

[/spoiler] [spoiler title=”Sorting and somesuch” style=”fancy”]

Sort by IP address

sort -n -t . -k 1,1 -k 2,2 -k 3,3 -k 4,4

Sort while ignoring the header line
# one-line header; data file
(head -n 1; sort) < file.txt

# one-line header; read from pipe
command | (read -r; printf "%s\n" "$REPLY"; sort)

# three-line header; read from pipe
command | (for i in $(seq 3); do read -r; printf "%s\n" "$REPLY"; done; sort)

Find missing lines between two files
diff --new-line-format="" --unchanged-line-format="" <(sort file1) <(sort file2)

See if file1 is different from file2:
diff -q file1 file2
Files file1 and file2 differ

See difference between two files side-by-side:
diff --side-by-side file1 file2
654                                                           <
987                                                           <
123                                                           <
321                                                             321
                                                              > 987
789                                                             789
                                                              > 456
                                                              > 123

Similar to above but sort the files first and compare only the unique lines:
diff --side-by-side <(sort -u file1) <(sort -u file2)
123                                                             123
321                                                             321
654                                                           | 456
789                                                             789
987                                                             987

Find lines in two files containing matching fields:
awk 'FNR==NR{a[$1];next}($1 in a){print}' file2 file1

Print every line from file1 that is also in file2 (requires moreutils package):
combine file1 and file2

Print lines from file1 that are not in file2:
combine file1 not file2

Same as above but reading from STDIN:
cat file1 | combine - not file2

Same as above but using grep:
grep -Fxv -f file2 file1

Print line that are unique to file1 or file2:
combine file1 xor file2

List unique words in a file and count their frequency:
tr -c a-zA-Z '\n' < /file.txt  | sed '/^$/d' | sort | uniq -i -c | sort -rn

Print text from the last occurrence of a tag line to the end of file
tac ${yourfile} | grep "${yourtag}" -m 1 -B 9999 | tac

[/spoiler] [spoiler title=”Comparing apples and oysters” style=”fancy”]

Check if variable is an integer:

[[ ${var} =~ ^-?[0-9]+$ ]]

Check if variable is an integer or a decimal:
[[ "${var}" =~ ^-?[0-9]+(\.[0-9]+)?$ ]]

Check if a file contains non-ASCII characters:
if LC_ALL=C grep -q '[^[:print:][:space:]]' ${f}; then echo "non-ASCII"; fi

Setting a default value for a shell variable:
# if  is null, set var="/tmp"

If variable not set, show error message and exit:
var=${1:?"Missing argument"}

If variable not set, show error message, run a command, and exit
var=${1:?"Missing argument" $(date)}

Find the length of a variable:

Remove patterns from the variable’s value:
var="/dir1/dir2/file.txt" && echo "${var#/dir1/dir2/}" && echo "${var##*/}"

var="/dir1/dir2/file.txt" && echo "${var%/*}" && echo "${var%%/file*}"

Find/replace strings in the value of a variable:
var="kitty cat" && echo "${var/kitty/taco}"
taco cat

Extract a substring from the value of a variable:
var="taco cat" && echo "${var:0:4}"

Check if the value of a variable contain a string that matches a REGEX (Bash v3):
var="taco cat" && [[ ${var} =~ ^[acot]{4} ]] && echo yes || echo no

Rename multiple files using patterns:
[root@ncc1701:/tmp/poi] # ls
file_01_2013_archive.tgz  file_03_2013_archive.tgz  file_05_2013_archive.tgz  file_07_2013_archive.tgz  file_09_2013_archive.tgz
file_02_2013_archive.tgz  file_04_2013_archive.tgz  file_06_2013_archive.tgz  file_08_2013_archive.tgz  file_10_2013_archive.tgz

[root@ncc1701:/tmp/poi] # mmv "*_2013_*" '#1_2017_#2'

[root@ncc1701:/tmp/poi] # ls
file_01_2017_archive.tgz  file_03_2017_archive.tgz  file_05_2017_archive.tgz  file_07_2017_archive.tgz  file_09_2017_archive.tgz
file_02_2017_archive.tgz  file_04_2017_archive.tgz  file_06_2017_archive.tgz  file_08_2017_archive.tgz  file_10_2017_archive.tgz

Compare two tarballs:
diff <(tar -tvf file1.tgz | sort) <(tar -tvf file2.tgz | sort)

See if there’s anything in a tarball that’s missing from the server:
tar -dz --file=file1.tgz -C /base_path

[/spoiler] [spoiler title=”Looking for things” style=”fancy”]

Identify first and last occurrence of an error message in /var/log/logname*

zgrep -h "error message" `find /var/log/ -type f -name "logname*" | sort -V | sed '1,1{H;1h;d;};$G'` | sed -n '1p;$p'

List and search tarball contents:
tar -ztvf file1.tgz '*pattern*'

Show OS release:
grep -m1 -h [0-9] /etc/{*elease,issue} 2>/dev/null | head -1

Find ASCII files
find . -type f -exec grep -Iq . {} \; -and -print

Find ASCII files and extract IP addresses
find . -type f -exec grep -Iq . {} \; -exec grep -oE "(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)" {} /dev/null \;

Find broken symlinks
find . -type l -xtype l

Find files containing specified keywords:
grep -m1 -lrnw "/path/name/" -E -e "keyword1|keyword2" 2>/dev/null | sed -r 's/(^Binary file | matches$)//g' | sort -u

# Example:

IFS=''; grep -m1 -lrnw "/etc/sysconfig/" -E -e "`hostname -s`" 2>/dev/null | sed -r 's/(^Binary file | matches$)//g' | sort -u | while read line ; do file "${line}" ; done | column -s: -t

/etc/sysconfig/network                             ASCII text
/etc/sysconfig/networking/profiles/default/hosts   ASCII English text
/etc/sysconfig/rhn/systemid                        XML  document text
/etc/sysconfig/rhn/systemid.old                    XML  document text
/etc/sysconfig/rhn/                   XML  document text

Use find command with regex
 find "$(pwd)" -type f -regextype posix-extended -regex '^.*\.(mkv|avi|mp4|mov|qt|wmv|mng|webm|flv|vob|ogg|ogv|rm|mpg|mpeg|ts4)$'

Limit find to current mount and exclude NFS
find . -mount ! -fstype nfs

Find files modified on a specific date:
find /etc/ -newermt 2016-03-04 ! -newermt 2016-03-05 -ls

Find files modified on a specific date using older version of find:
touch /tmp/mark.start -d "2017-01-26 23:59"
touch /tmp/mark.end -d "2017-01-27 23:59"
find /etc -newer /tmp/mark.start ! -newer /tmp/mark.end

Fuzzy matching with fzf:
git clone --depth 1 ~/.fzf
find /etc -type f | fzf

Find ten largest open files:
lsof / | awk '{ if($7 > 1048576) print $7/1048576 "MB" " " $9 " " $1 }' | sort -n -u | tail

Find ten largest files without crossing mountpoints:
find / -xdev -type f | xargs du | sort -r -n -k 1 | head -n 10 | awk '{ split( "KB MB GB" , v ); s=1; while( $1>1024 ){ $1/=1024; s++ } print int($1) v[s]"\t"substr($0, index($0,$2))}'

Show the size of all subfolders in the current directory:
du -h --max-depth=1

Find ten largest files in a directory:
du -sh /var/log/* | sort -hr | head -10

Find ten largest files owned by oracle user, modified in the past five days, don’t cross mountpoints, don’t search NFS:
for i in \
$(find /opt -mount ! -fstype nfs -type f -user oracle -mtime -5 -printf '%s %p\n' 2>/dev/null | sort -nr | head -10 | \
awk '{ s = ""; for (i = 2; i <= NF; i++) s = s $i " "; print s }')
do ls -alh "${i}"

Find the latest file in a filesystem
find . -type f -printf '%T@ %p\n' | sort -n | tail -1 | cut -f2- -d" "

Continuous grep on I/O redirection
netstat -T -tupac | grep --line-buffered "1699/java"

Another method for continuous grep (and other commands) using unbuffer (yum -y install expect)
unbuffer netstat -T -tupac | grep "1699/java"

Grep with a non-capturing group

Extract a four-digit number in parentheses from string `2013 Monkeys in 1999 (2014).txt`

echo "2013 Monkeys in 1999 (2014).txt" | grep -oP "(?<=\()[0-9]{4}(?=\))"

Count occurrences of multiple patterns with a single grep
grep -EIho "pattern1|pattern2|pattern3" | sort | uniq -c

Grep with a pattern file:
grep -v -f pattern_file

Count word frequency in the English translation of Tolstoy’s “War and Peace”, excluding 5000 most common English words:
wget -q -O ${t}
wget -q -O ${p}
head -${i} ${p} | awk '{print " "$1"$"}' > ${p}_${i}
tr -c a-zA-Z '\n' < ${t}  | egrep -v "^(ll|ve)$" | sed '/^$/d' | \
sort | uniq -ic | sort -rn | grep -iv -f ${p}_${i} | more

A follow-up to the above: get a Wikipedia definition for the top 10 most common words in “War and Peace”:
if [ ! -d ${d} ] ; then mkdir -p ${d} ; fi
wget -q -O ${d}/wped.php
sed -i "s/'limit'=>3,/'limit'=>1,/g" ${d}/wped.php
chmod 755 ${d}/wped.php
ln -s ${d}/wped.php /usr/bin/wped
for w in `tr -c a-zA-Z '\n' < ${t}  | egrep -v "^(ll|ve)$" | sed '/^$/d' | sort | \
uniq -ic | sort -rn | grep -iv -f ${p}_${i} | awk '{print $NF}' | head -10` ; do \
wped ${w} ; done | grep -v "Search results"

Find 20 most frequently-used shell commands:
tr "\|\;" "\n" < ~/.bash_history | sed -e "s/^ //g" | cut -d " " -f 1 | sort | uniq -c | sort -rn | head -20

Checking if file is older than so many seconds:
if [ `expr $(date +%s) - $(stat -c %Y ${testfile})` -gt ${threshold_seconds} ]

Running diff on files from remote servers:
diff <(ssh -qtT server01 "sudo su - root -c 'fdisk -l 2>/dev/null | grep ^Disk'") <(ssh -qtT server02 "sudo su - root -c 'fdisk -l 2>/dev/null | grep ^Disk'")

[/spoiler] [spoiler title=”Formatting output” style=”fancy”]

Prepend comma-separated stdout with a header and arrange in columns

| (echo "COLUMN1_HEADER COLUMN2_HEADER COLUMN3_HEADER" && cat) | column -s',' -t

Print a horizontal line:
rule () {
	printf -v _hr "%*s" $(tput cols) && echo ${_hr// /${1--}}

rule -

Print a horizontal line with a message:
rulem ()  {
	if [ $# -eq 0 ]; then
		echo "Usage: rulem MESSAGE [RULE_CHARACTER]"
		return 1
	printf -v _hr "%*s" $(tput cols) && echo -en ${_hr// /${2--}} && echo -e "\r3[2C$1"

Right-align text:
alias right="printf '%*s' $(tput cols)"

[/spoiler] [spoiler title=”Process Control” style=”fancy”]

Background and disown a foreground process:

CTRL-Z; disown -h %1; bg 1; logout

Background and disown any process, including another user’s:
kill -TSTP $PID && kill -CONT $PID

[/spoiler] [spoiler title=”System checks” style=”fancy”]

Find dead system services

for i in $(chkconfig --list | grep "`runlevel | awk '{print $NF}'`:on" | awk '{print $1}' | sort) ; do /sbin/service ${i} status 2>&1 | egrep "not [a-z]{1,}ing|[kpsd][ea]d\b"; done

Find active system services that shouldn’t be running (i.e. were started manually)
for i in $(chkconfig --list | grep "`runlevel | awk '{print $NF}'`:off" | awk '{print $1}' | sort) ; do /sbin/service ${i} status 2>&1 | egrep "is running"; done

[/spoiler] [spoiler title=”Network stuff and SSH” style=”fancy”]

Nmap subnet scan

nmap -sn -oG - | awk '$4=="Status:" && $5=="Up" {print $2}'

Basic nmap port scan
nmap -oG -T4 -F | grep "\bopen\b"

Check host port access using only Bash:
s="$(cat 2>/dev/null < /dev/null > /dev/tcp/${target_ip}/${target_port} & WPID=$!; sleep 3 && kill $! >/dev/null 2>&1 & KPID=$!; wait $WPID && echo 1)" ; s="${s:-0}"; echo "${s}" | sed 's/0/2/;s/1/0/;s/2/1/'

mtr – traceroute and ping combined:

Mount a remote folder through SSH:
sshfs name@server:/path/to/folder /path/to/mount/point

To unmount the previous:
fusermount -u /path/to/mount/point

Download a website recursively with wget:
wget --random-wait -r -p -e robots=off -U Mozilla

Start an SMTP server:
python -m smtpd -n -c DebuggingServer localhost:1025

Shutdown a Windows machine
net rpc shutdown -I IP_ADDRESS -U username%password

Run a local script/complex command on remote machines via SSH with passwordless sudo
for i in 1 2 3; do ssh -qTt host0$i "echo $(base64 -w0 /tmp/ | base64 -d 2>/dev/null| sudo bash"; done

[/spoiler] [spoiler title=”MySQL tricks” style=”fancy”]

Non-locking queries


Select distinct with count
SELECT column1, COUNT(*) AS column1_count FROM some_table GROUP BY column1 ORDER BY column1_count DESC

Evade “myisamchk: error: myisam_sort_buffer_size is too small”
myisamchk --sort_buffer_size=2G -r -f table_name.MYI

Select via shell script when column names have spaces
SELECT \`Column One\`,\`Column Two\`,\`Column Three\`

Select and replace spaces in values with underscores
SELECT REPLACE(column1, ' ', '_'),REPLACE(column2, ' ', '_')

Correct MySQL  grant syntax
mysql -u${user} -p${passwd}
CREATE DATABASE ${tbl_name} ;
RANT ALL PRIVILEGES ON ${tbl_name}.* to ${user}@'%' IDENTIFIED BY 'password' WITH GRANT OPTION ;


[/spoiler] [spoiler title=”Sync and Backup” style=”fancy”]

Backup a remote folder to the local machine with TAR/SSH:

ssh username@hostname tar czf - /folder/ > /target/hostname_folder.tgz

Backup a local folder to a remote server with TAR/SSH:
tar zcvf - /folder | ssh username@hostname "cat > /target/folder.tgz"

Restore remote backup to the local machine with TAR/SSH:
cd / && ssh username@hostname "cat /target/folder.tgz" | tar zxvf -

[/spoiler] [spoiler title=”Hotkeys” style=”fancy”]
ALT-.		# Previous command's final parameter
CTRL-r		# History reverse incremental serch
TAB		# Complete unambiguous command
TAB TAB		# List all possible completions
ALT-*		# Insert all possible completions
CTRL-ALT-e	# Inline alias, history, and shell expansion
CTRL-x CTRL-e	# Load current command into default text editor
CTRL-a		# Move cursor to the beginning of the line
CTRL-e		# Move cursor to the end of the line
CTRL-w		# Delete previous word
CTRL-k		# Delete to the end of the line
CTRL-u		# Delete to the beginning of the line
CTRL-y		# Paste last deleted command
ALT-y		# Cycle through last deleted commands and paste

[/spoiler] [spoiler title=”Productivity shortcuts” style=”fancy”]

Run the last command as root:

sudo !!

Save changes to a read-only file in vi:
:w !sudo tee %

Change to the previous working directory:
cd -

Run the previous command replacing first “foo” with “bar”:

Run the previous command replacing all “foo” with “bar”:

Quickly backup or copy a file:
cp -p file.txt{,_`date +'%Y-%m-%d_%H%M%S'`}

Find config files in /etc and make compressed backup tarball in /etc/backup
t=/etc/backup ; d=`date +'%Y-%m-%d_%H%M%S'` ; mkdir -p ${t}/${d}
find /etc -not -path "${t}/${d}*" -type f \( -name "*\.conf*" -o -name "*\.cgf" -o -name "*\.cnf" \) -execdir /bin/cp -pf {} /etc/backup/${d}/{} \;
cd ${t} ; tar cfz ${d}.tgz ${d} ; /bin/rm -rf ${d}

Find the last command that begins with “foo”, but don’t run it:

Capture video of a Linux desktop:
ffmpeg -f x11grab -s wxga -r 25 -i :0.0 -sameq /tmp/out.mpg

Empty a file or create a new empty file:
> file.txt

Tweet from the shell:
curl -u user:pass -d status='Tweeting from the shell'

Quickly access ASCII table:
man 7 ascii

Manual timer:
time read

Execute a command in a sub-shell
(cd /tmp && ls)

List 10 most often used commands:
history | awk '{a[$2]++}END{for(i in a){print a[i] " " i}}' | sort -rn | head

[/spoiler] [spoiler title=”Package Management” style=”fancy”]

List all files contained in all packages that have “httpd” in their name:

rpm -ql $(rpm -qa | grep httpd)

Execute a command at midnight:
echo cmd | at midnight

Compare a remote file with a local file:
ssh user@host cat /path/to/remotefile | diff /path/to/localfile -

Display currently mounted file systems nicely:
mount | column -t

[/spoiler] [spoiler title=”Pipes and Redirects” style=”fancy”]

Get individual exit code of each piped command in a chain

ls /var/www/icons | grep gif | ls -als >/dev/null ; echo ${PIPESTATUS[*]}
0 0 0

Redirect SDTOUT to multiple commands:
echo "something anything" | tee >(sed 's/some/any/g') >(sed 's/thing/one/g') >(sed 's/any/some/g')
something anything
anything anything
someone anyone
something something

Similar to above, but using pee (requires moreutils package):
echo "something anything" | pee "sed 's/some/any/g'" "sed 's/thing/one/g'" "sed 's/any/some/g'"
anything anything
someone anyone
something something

[/spoiler] [spoiler title=”Clever Loops” style=”fancy”]

Get individual exit code of each piped command in a chain

ls /var/www/icons | grep gif | ls -als >/dev/null ; echo ${PIPESTATUS[*]}
0 0 0

Dynamic variables with variable names
i=1 ; eval "$(echo var${i})"=value ; eval echo $(echo $`eval echo "var${i}"`)

[/spoiler] [spoiler title=”System performance” style=”fancy”]

Create and mount a temporary RAM partition:

mount -t tmpfs -o size=1024m tmpfs /mnt

Top for files:
watch -d -n 1 'df; ls -FlAt /path'

Display the top ten running processes sorted by memory usage:
ps aux | sort -nk +4 | tail

# or

ps aux | awk '{if ($5 != 0 ) print $2,$5,$6,$11}' | sort -k2rn | head -10 | column -t

Free memory pagecache, dentries, and inodes:
free -m && sync && echo 3 > /proc/sys/vm/drop_caches && free -m

Find processes constantly in wait state:
for i in `seq 1 1 10`; do ps -eo state,pid,cmd | grep "^D"; echo "----"; sleep 5; done

[/spoiler] [spoiler title=”Fun stuff” style=”fancy”]

Watch Star-Wars via telnet:


Steam Locomotive
yum -y install sl >/dev/null 2>&1; sl

Read your fortune
yum -y install fortune >/dev/null 2>&1; fortune

Read things in reverse
yum install util-linux-ng >/dev/null 2>&1; date | rev

Prime factorization
yum -y install coreutils >/dev/null 2>&1; echo 5 10 10234 | factor

Talking cow
yum -y install cowsay >/dev/null 2>&1; cowsay `date`

Echo string over and over until CTRL-C
yes I am awesome

Matrix-like effect for the terminal screen
if [ `yum install cmatrix >/dev/null 2>&1; echo $?` -ne 0 ]
   cd /tmp
   wget -q
   tar xzf cmatrix-1.2a.tar.gz
   cd cmatrix-1.2a
   make install

[/spoiler] [/accordion]


Print Friendly, PDF & Email

No Comment »

1 Pingbacks »

Leave a Reply

%d bloggers like this: