Concepts Of Standard I/O, Pipe And Filters In Linux/Unix

Standard streams

In Linux programming, standard streams are preconnected input-output communication channels between the computer program and environment in which the program gets executed. There are three types of I/O communication channels i.e standard input (stdin), standard output(stdout), and standard error(stderr). This input or output can be redirected to or from a text file using a redirection operator. stderr of a program is generally redirected into a log file which is further used for error analysis.

Input redirection –

The less than sign i.e < is used as an input redirection operator. Input redirection is a feature in which input to a program is given from a separate file which is generally a text file.
For example –

wc < filename

In the above image above hello is a text file that contains 51 lines, 241 words, and 1738 characters.

Output redirection –

The greater than sign i.e > is used as an output redirection operator. Output redirection is a feature in which the output of a program is redirected to a separate file.

For example –

ls >file

You can see the content of the file above. The given content is redirected output of ls command.

Please note that if the file on which output is redirected contains any data, it will be overwritten. So don’t redirect it to a file that already contains some data.

Append the output to an existing file-

If a file already contains some data and you want to add a program’s output in this existing data. You can append data by using >> operator.

For example-

echo This is appended text>>file
cat file

You can see the appended text in the file.

Pipe And Filters

The pipe allows you to connect two or more than two commands in Linux/Unix so that the output of the first command becomes the input for the second, similarly, the output of the second command becomes the input for the third and so on. You can use it by inserting a horizontal bar | (don’t confuse it with a capital i or l) between the two commands.
For example-
ls -l | grep "Aug"

grep command will filter the lines from the output of ls -l command containing “Aug” string.

A filter is used to modify the stream that means, unlike a pipe which simply passes the standard output of one command as the input of others. It performs some operation on the input and does something useful and then return it to standard output. There are various filters that are available in Linux/Unix. Some useful ones are – grep, awk, sed, wc, spell, etc.

grep command-

grep searches for a regular expression globally. The regular expression can be plain text or some special character. Syntax of command is given below-

grep option pattern file

For example-

Other options that are used with grep are –

-vprints all lines that do not match the pattern
-nprints the matched line with line number
-lprints only the names of files with matching lines
-cprints only count of matching lines
-imatches either upper or lowercase

sort command-

Used to sort the content of a file in alphabetical order
sort options file

For example-

You can see that the content of file is sorted in the reverse order.

options used with sort commands are-

-bIgnore blanks at the start of the line
-rReverse the sorting order
-oSpecify the output file
-nUse the numerical value to sort
-MSort as per calendar month specified
-uSuppress lines that repeat an earlier key.
-cCheck whether the input is sorted
-dconsider only blanks and alphanumeric characters

awk command-

Basically awk is a scripting language that stands for “Aho Weinberger, and Kernighan(developers)”. It scans each input either it is from stdin or from a file for lines that matches any set of patterns specified are displayed as output.

awk program input_file(s)

For example-

ls -l | awk '{print $1}'

The command awk '{print $1}' will take the output of the command ls -l as input and return the first word of each line which is displayed in the terminal similarly $2 will display the second word of each line. You can read the detailed user guide of awk The GNU Awk User’s Guide.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.