UNIX pipe and stream toolbox

(in progress)

I am asked on an almost daily basis "how do I do 'X'" where 'X' is "find some text in a set of files" or "replace some phrase in a set of files" that it occurs to me: people have no idea as to what tools are at their disposal on a UNIX system, or how to use them.


First, read the friendly manual(s). The man command goes a long way if you remember at least the name of the command.

$ man grep

Sometimes the manpage doesn't quite give you all the details you would like. Many of your tools include voluminous tomes of information, chapter by chapter, describing in more detail how to use the tool:

$ info sed


While there are many more useful tools at your disposal (at, cron, wget, screen, etc.) this one is all about the basic tools, which include (apart from the features of your shell itself, which are many):

cat - concatenate files and print on the standard output

$ # send the contents of myfile.txt to the console
$ cat myfile.txt

$ # send the contents of file1.txt and file2.txt
$ # one after the other, to the console
$ cat file1.txt file2.txt

$ # send the contents of all files matching "*.txt" ...
$ cat *.txt

sed - stream editor for filtering and transforming text

$ # output myfile.txt, with 4 leading spaces
$ sed "s/^/    /" myfile.txt

$ # output myfile.txt, changing "Foo" to "Bar"
$ sed "s/Foo/Bar/" myfile.txt

grep - print lines matching a pattern

$ # print only the lines with "Foo" in them
$ grep "Foo" myfile.txt













Before I talk in more detail any of the specific tools, an incredibly quick primer on the underlying philosophy behind all these tools in your toolbox: streams and pipes.

The input and output for all these tools is (in theory) simple streams of text. That's it. No bizarre XML mangling (although the tools can un-mangle and re-mangle them for you), just lines of text going in and lines of text going out. Think of the tools as boxes, into which you hand pieces of paper with lines of text, and out of which you are handed pieces of paper with lines of text.

While most of the tools can operate directly on files:

$ grep "foo" mybigfile.txt

The real power of your toolbox is in combining these boxes in a chain (more accurately a directed graph), one to the next. Nearly all the tools will happily read from "standard input" and place their results on "standard output", which is handy not only when you are sitting at the shell and waiting for their result, but especially so when you pipe or redirect the output of one command to the input of another command:

$ cat mybigfile.txt | grep "foo"

But vertical bar pipes are not the only game in town. The above could have been done:

$ grep "foo" < mybigfile.txt


Combining the "top shelf" tools using pipes and redirection is where some of the more interesting tools come in:

tee - read from standard input and write to standard output and files

changed October 10, 2007