Shell Utilities

Table of Contents

1 Tmp

1.1 find

find ~/data/fast/pick-master/ -name '*.[ch]'

1.2 sort

  • sort -k 4 -n

1.3 tee

  for name in data/github-bench/*; do 
      echo "===== $name"\
          | tee -a log.txt; { time helium --create-cache $name; } 2>&1\
          | tee -a log.txt; done

1.4 xz

a general-purpose data compression tool

1.5 cpio

copy files between archives and directories

2 Q&A

2.1 redirect output of time

{ time sleep 1 ; } 2> time.txt
{ time sleep 1 ; } 2>&1 | tee -a time.txt

3 Mathematics

3.1 random number generation

shuf -i 1-100 -n 1
  • bc calculator

4 Text Editing

4.1 Patch System

Create a patch (notice the order: old then new): 

diff -u hello.c hello_new.c > hello.patch
diff -Naur /usr/src/openvpn-2.3.2 /usr/src/openvpn-2.3.4 > openvpn.patch

To apply a patch 

patch -p3 < /path/to/openvpn.patch
patch -p1 <patch -d /path/to/old/file

the number after p indicates how many the leading slashes are skipped when find the old file

To reverse (un-apply) a patch: 

patch -p1 -R <patch

This works as if you swapped the old and new file when creating the patch.

4.2 grep

# -i: 忽略大小写
# -n: 显示行号
# -v: 输出不匹配的行
# -H: 同时输出此行所在的文件名
grep <pattern> <file>

4.3 tr: translate characters

tr <string1> <string2>

the characters in string1 are translated into the characters in string2 where the first character in string1 is translated into the first character in string2 and so on. If string1 is longer than string2, the last character found in string2 is duplicated until string1 is exhausted.

characters in the string can be:

any characters will represent itself if not:

  • \\octal: A backslash followed by 1, 2 or 3 octal digits
  • \n, \t
  • a-z: inclusive, ascending
  • [:class:]: space, upper, lower, alnum
    • if [:upper:] and [:lower:] appears in the same relative position, they will correlate.

4.4 uniq: report or filter out repeated lines in a file

Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first.

  • uniq -c: Precede each output line with the count of the number of times the line occurred in the input, followed by a single space. You can then comtine this with sort -n
  • -u: Only output lines that are not repeated in the input.
  • -i: Case insensitive comparison of lines.

4.5 Other

  • cat: -n output with line number
  • tail
    • -<n> 显示后n行
    • +<n> 显示第n行到结尾
    • -F 跟踪显示不断增长的文件结尾
  • column -s , -t xxx.csv

5 System Management

5.1 man

man [num] <item>

  • 2: system call, kernel function
  • 3: library call

5.2 sort

  • -n: compare according to string numerical value
  • -r: reverse

5.3 xargs

相当于~~，并把输出放 结尾 

find /etc -name '*.conf' | xargs ls -l
# the same as:
ls -l ~find ...~

5.4 Find

find . -type f -name *.flac -exec mv {} ../out/ \;

Copy file based on find, and take care of quotes and spaces: 

find CloudMusic -type f -name "*mp3" -exec cp "{}" all_music \;

5.5 Filesystem related

from relative path to absolute path 

realpath -f /relative/path # this will follow symbolic link

du: estimate file space usage 

du -sm * | sort -nr

5.6 unzip

  • -o overwrite existing file without prompt
  • -d extract to specified directory

5.7 tar

tar zcvf file.tar.gz –exclude=a.out –exclude=*.gmout folder

  • exclude must come before folder
  • exclude use shell pattern
  • exclude can be specified multiple times

5.8 Other

  • time <command>: # the total user and system time consumed by the shell and its children
  • column: formats its input into multiple columns. mount | column -t
  • dd: dd if=xxx.iso of=/dev/sdb bs=4m; sync
  • convert: convert xxx.jpg -resize 800 xxx.out.jpg # 800x<height>
  • nl: nl <filename> 添加行号。输出到stdout
  • ln: ln -s <target> <linkname> 记忆：新的东西总要最后才发布。
  • ls: order: -r reverse; -s file size; X extension; -t time
  • tree
    • -d 只显示目录
    • -f 显示路径
    • -F 条目后有 [\*/=@|]
    • -r 倒序
    • -t 修改时间排序
    • -L(\d) 显示n层