Bash Tips and Tricks

Bash, or the Bourne Again Shell, is the default shell in most Linux distributions. The popularity of the bash shell amongst Linux and UNIX users is no accident. It has many features to enhance user-friendliness and productivity. Unfortunately, you can’t take advantage of those features unless you know they exist.

When I first started using Linux, the only bash feature I took advantage of was going back through the command history using the up arrow. I soon learned additional features by watching others and asking questions. In this article, I’d like to share some bash tricks I’ve learned over the years.

This article isn’t meant to cover all of the features of the bash shell; that would require a book, and plenty of books are available that cover this topic. Instead, this article is a summary of the bash tricks I use most often and would be lost without.

Brace Expansion

One of my favorite bash tricks is brace expansion. Brace expansion takes a list of strings separated by commas and expands those strings into separate arguments for you. The list is enclosed by braces, the symbols { and }, and there should be no spaces around the commas. For example:

$ echo {one,two,red,blue}
one two red blue

Using brace expansion as illustrated in this simple example doesn’t offer too much to the user. In fact, the above example requires typing two more characters than simply typing:

echo one two red blue

which produces the same result. However, brace expansion becomes quite useful when the brace-enclosed list occurs immediately before, after or inside another string:

$ echo {one,two,red,blue}baloon
onebaloon twobaloon redbaloon bluebaloon

$ echo fish{one,two,red,blue}
fishone fishtwo fishred fishblue

$ echo fi{one,two,red,blue}sh
fionesh fitwosh firedsh fibluesh

Notice that there are no spaces inside the brackets or between the brackets and the adjoining strings. If you include spaces, it breaks things:

$ echo {one, two, red, blue }fi
{one, two, red, blue }fi

$ echo "{one,two,red,blue} fi"
{one,two,red,blue} fi

However, you can use spaces if they’re enclosed in quotes outside the braces or within an item in the comma-separated list:

$ echo {"one ","two ","red ","blue "}fish
one fish two fish red fish blue fish

$ echo {one,two,red,blue}" fish"
one fish two fish red fish blue fish

You also can nest braces, but you must use some caution here too:

$ echo {{1,2,3},1,2,3}
1 2 3 1 2 3

$ echo {{1,2,3}1,2,3}
11 21 31 2 3

Now, after all these examples, you might be thinking to yourself, “Those are great parlor tricks, but why should I care about brace expansion?”

Brace expansion becomes useful when you need to make a backup of a file. This is why it’s my favorite shell trick. I use it almost every day when I need to make a backup of a config file before changing it. For example, if I’m making a change to my Apache configuration, I can do the following and save some typing:

$ cp /etc/apache2/apache2.conf{,.bak}

Notice that there is no character between the opening brace and the first comma. It’s perfectly acceptable to do this and is useful when adding characters to an existing filename or when one argument is a substring of the other. Then, if I need to see what changes I made later in the day, I use the diff command and reverse the order of the strings inside the braces:

$ diff /etc/apache2/apache2.conf{.bak,}
1050a1051
> # I added this comment earlier

Redirecting Standard Error

Have you ever looked for a file using the find command, only to learn the file you were looking for is lost in a sea of permission denied error messages that quickly fill your terminal window?

If you are the administrator of the system, you can become root and execute find again as root. Because root can read any file, you don’t get that error anymore. Unfortunately, not everyone has root access on the system being used. Besides, it’s bad practice to be root unless it’s absolutely necessary. So what can you do?

One thing you can do is redirect your output to a file. Basic output redirection should be nothing new to anyone who has spent a reasonable amount of time using any UNIX or Linux shell, so I won’t go into detail regarding the basics of output redirection. To save the useful output from the find command, you can redirect the output to a file:

$ find / -name foo > output.txt

You still see the error messages on the screen but not the path of the file you’re looking for. Instead, that is placed in the file output.txt. When the find command completes, you can cat the file output.txt to get the location(s) of the file(s) you want.

That’s an acceptable solution, but there’s a better way. Instead of redirecting the standard output to a file, you can redirect the error messages to a file. This can be done by placing a 2 directly in front of the redirection angle bracket. If you are not interested in the error messages, you simply can send them to /dev/null:

$ find / -name foo 2> /dev/null

This shows you the location of file foo, if it exists, without those pesky permission denied error messages. I almost always invoke the find command in this way.

The number 2 represents the standard error output stream. Standard error is where most commands send their error messages. Normal (non-error) output is sent to standard output, which can be represented by the number 1. Because most redirected output is the standard output, output redirection works only on the standard output stream by default. This makes the following two commands equivalent:

$ find / -name foo > output.txt
$ find / -name foo 1> output.txt

Sometimes you might want to save both the error messages and the standard output to file. This often is done with cron jobs, when you want to save all the output to a log file. This also can be done by directing both output streams to the same file:

$ find / -name foo > output.txt 2> output.txt

This works, but again, there’s a better way to do it. You can tie the standard error stream to the standard output stream using an ampersand. Once you do this, the error messages goes to wherever you redirect the standard output:

$ find / -name foo > output.txt 2>&1

One caveat about doing this is that the tying operation goes at the end of the command generating the output. This is important if piping the output to another command. This line works as expected:

find -name test.sh 2>&1 | tee /tmp/output2.txt

but this line doesn’t:

find -name test.sh | tee /tmp/output2.txt 2>&1

and neither does this one:

find -name test.sh 2>&1 > /tmp/output.txt

I started this discussion on output redirection using the find command as an example, and all the examples used the find command. This discussion isn’t limited to the output of find, however. Many other commands can generate enough error messages to obscure the one or two lines of output you need.

Output redirection isn’t limited to bash, either. All UNIX/Linux shells support output redirection using the same syntax.

Using Loops from the Command Line

One last tip I’d like to offer is using loops from the command line. The command line is not the place to write complicated scripts that include multiple loops or branching. For small loops, though, it can be a great time saver. Unfortunately, I don’t see many people taking advantage of this. Instead, I frequently see people use the up arrow key to go back in the command history and modify the previous command for each iteration.

If you are not familiar with creating for loops or other types of loops, many good books on shell scripting discuss this topic. A discussion on for loops in general is an article in itself.

You can write loops interactively in two ways. The first way, and the method I prefer, is to separate each line with a semicolon. A simple loop to make a backup copy of all the files in a directory would look like this:

$ for file in * ; do cp $file $file.bak; done

Another way to write loops is to press Enter after each line instead of inserting a semicolon. bash recognizes that you are creating a loop from the use of the for keyword, and it prompts you for the next line with a secondary prompt. It knows you are done when you enter the keyword done, signifying that your loop is complete:

$ for file in *
> do cp $file $file.bak
> done

And Now for Something Completely Different

When I originally conceived this article, I was going to name it “Stupid bash Tricks”, and show off some unusual, esoteric bash commands I’ve learned. The tone of the article has changed since then, but there is one stupid bash trick I’d like to share.

About 2 years ago, a Linux system I was responsible for ran out of memory. Even simple commands, such as ls, failed with an insufficient memory error. The obvious solution to this problem was simply to reboot. One of the other system administrators wanted to look at a file that may have held clues to the problem, but he couldn’t remember the exact name of the file. We could switch to different directories, because the cd command is part of bash, but we couldn’t get a list of the files, because even ls would fail. To get around this problem, the other system administrator created a simple loop to show us the files in the directory:

$ for file in *; do echo $file; done

This worked when ls wouldn’t, because echo is a part of the bash shell, so it already is loaded into memory. It’s an interesting solution to an unusual problem. Now, can anyone suggest a way to display the contents of a file using only bash built-ins?

Conclusion

Bash has many great features to make life easier for its users. I hope this summary of bash tricks I like to use has shown you some new ways to take advantage of the power bash has to offer.


Tags : Bash, Bash find, Bash loops, CLI, Command line, Redirecting errors, shell, Tips, Tricks
Categories : HOW TO