In the shell, what does " 2>&1 " mean? – Dev

The best answers to the question “In the shell, what does " 2>&1 " mean?” in the category Dev.

QUESTION:

In a Unix shell, if I want to combine stderr and stdout into the stdout stream for further manipulation, I can append the following on the end of my command:

2>&1

So, if I want to use head on the output from g++, I can do something like this:

g++ lots_of_errors 2>&1 | head

so I can see only the first few errors.

I always have trouble remembering this, and I constantly have to go look it up, and it is mainly because I don’t fully understand the syntax of this particular trick.

Can someone break this up and explain character by character what 2>&1 means?

ANSWER:

echo test > afile.txt

redirects stdout to afile.txt. This is the same as doing

echo test 1> afile.txt

To redirect stderr, you do:

echo test 2> afile.txt

>& is the syntax to redirect a stream to another file descriptor – 0 is stdin, 1 is stdout, and 2 is stderr.

You can redirect stdout to stderr by doing:

echo test 1>&2 # or echo test >&2

Or vice versa:

echo test 2>&1

So, in short… 2> redirects stderr to an (unspecified) file, appending &1 redirects stderr to stdout.

ANSWER:

File descriptor 1 is the standard output (stdout).
File descriptor 2 is the standard error (stderr).

Here is one way to remember this construct (although it is not entirely accurate): at first, 2>1 may look like a good way to redirect stderr to stdout. However, it will actually be interpreted as “redirect stderr to a file named 1“. & indicates that what follows and precedes is a file descriptor and not a filename. So the construct becomes: 2>&1.

Consider >& as redirect merger operator.

ANSWER:

I found this brilliant post on redirection: All about redirections

Redirect both standard output and standard error to a file

$ command &>file

This one-liner uses the &> operator to redirect both output streams – stdout and stderr – from command to file. This is Bash’s shortcut for quickly redirecting both streams to the same destination.

Here is how the file descriptor table looks like after Bash has redirected both streams:

Enter image description here

As you can see, both stdout and stderr now point to file. So anything written to stdout and stderr gets written to file.

There are several ways to redirect both streams to the same destination. You can redirect each stream one after another:

$ command >file 2>&1

This is a much more common way to redirect both streams to a file. First stdout is redirected to file, and then stderr is duplicated to be the same as stdout. So both streams end up pointing to file.

When Bash sees several redirections it processes them from left to right. Let’s go through the steps and see how that happens. Before running any commands, Bash’s file descriptor table looks like this:

Enter image description here

Now Bash processes the first redirection >file. We’ve seen this before and it makes stdout point to file:

Enter image description here

Next Bash sees the second redirection 2>&1. We haven’t seen this redirection before. This one duplicates file descriptor 2 to be a copy of file descriptor 1 and we get:

Enter image description here

Both streams have been redirected to file.

However be careful here! Writing

command >file 2>&1

is not the same as writing:

$ command 2>&1 >file

The order of redirects matters in Bash! This command redirects only the standard output to the file. The stderr will still print to the terminal. To understand why that happens, let’s go through the steps again. So before running the command, the file descriptor table looks like this:

Enter image description here

Now Bash processes redirections left to right. It first sees 2>&1 so it duplicates stderr to stdout. The file descriptor table becomes:

Enter image description here

Now Bash sees the second redirect, >file, and it redirects stdout to file:

Enter image description here

Do you see what happens here? Stdout now points to file, but the stderr still points to the terminal! Everything that gets written to stderr still gets printed out to the screen! So be very, very careful with the order of redirects!

Also note that in Bash, writing

$ command &>file

is exactly the same as:

$ command >&file

ANSWER:

Some tricks about redirection

Some syntax particularity about this may have important behaviours. There is some little samples about redirections, STDERR, STDOUT, and arguments ordering.

1 – Overwriting or appending?

Symbol > means redirection.

  • > means send to as a whole completed file, overwriting target if exist (see noclobber bash feature at #3 later).
  • >> means send in addition to would append to target if exist.

In any case, the file would be created if they not exist.

2 – The shell command line is order dependent!!

For testing this, we need a simple command which will send something on both outputs:

$ ls -ld /tmp /tnt
ls: cannot access /tnt: No such file or directory
drwxrwxrwt 118 root root 196608 Jan  7 11:49 /tmp

$ ls -ld /tmp /tnt >/dev/null
ls: cannot access /tnt: No such file or directory

$ ls -ld /tmp /tnt 2>/dev/null
drwxrwxrwt 118 root root 196608 Jan  7 11:49 /tmp

(Expecting you don’t have a directory named /tnt, of course ;). Well, we have it!!

So, let’s see:

$ ls -ld /tmp /tnt >/dev/null
ls: cannot access /tnt: No such file or directory

$ ls -ld /tmp /tnt >/dev/null 2>&1

$ ls -ld /tmp /tnt 2>&1 >/dev/null
ls: cannot access /tnt: No such file or directory

The last command line dumps STDERR to the console, and it seem not to be the expected behaviour… But…

If you want to make some post filtering about standard output, error output or both:

$ ls -ld /tmp /tnt | sed 's/^.*$/<-- & --->/'
ls: cannot access /tnt: No such file or directory
<-- drwxrwxrwt 118 root root 196608 Jan  7 12:02 /tmp --->

$ ls -ld /tmp /tnt 2>&1 | sed 's/^.*$/<-- & --->/'
<-- ls: cannot access /tnt: No such file or directory --->
<-- drwxrwxrwt 118 root root 196608 Jan  7 12:02 /tmp --->

$ ls -ld /tmp /tnt >/dev/null | sed 's/^.*$/<-- & --->/'
ls: cannot access /tnt: No such file or directory

$ ls -ld /tmp /tnt >/dev/null 2>&1 | sed 's/^.*$/<-- & --->/'

$ ls -ld /tmp /tnt 2>&1 >/dev/null | sed 's/^.*$/<-- & --->/'
<-- ls: cannot access /tnt: No such file or directory --->

Notice that the last command line in this paragraph is exactly same as in previous paragraph, where I wrote seem not to be the expected behaviour (so, this could even be an expected behaviour).

Well, there is a little tricks about redirections, for doing different operation on both outputs:

$ ( ls -ld /tmp /tnt | sed 's/^/O: /' >&9 ) 9>&2  2>&1  | sed 's/^/E: /'
O: drwxrwxrwt 118 root root 196608 Jan  7 12:13 /tmp
E: ls: cannot access /tnt: No such file or directory

Note: &9 descriptor would occur spontaneously because of ) 9>&2.

Addendum: nota! With the new version of bash (>4.0) there is a new feature and more sexy syntax for doing this kind of things:

$ ls -ld /tmp /tnt 2> >(sed 's/^/E: /') > >(sed 's/^/O: /')
O: drwxrwxrwt 17 root root 28672 Nov  5 23:00 /tmp
E: ls: cannot access /tnt: No such file or directory

And finally for such a cascading output formatting:

$ ((ls -ld /tmp /tnt |sed 's/^/O: /' >&9 ) 2>&1 |sed 's/^/E: /') 9>&1| cat -n
     1  O: drwxrwxrwt 118 root root 196608 Jan  7 12:29 /tmp
     2  E: ls: cannot access /tnt: No such file or directory

Addendum: nota! Same new syntax, in both ways:

$ cat -n <(ls -ld /tmp /tnt 2> >(sed 's/^/E: /') > >(sed 's/^/O: /'))
     1  O: drwxrwxrwt 17 root root 28672 Nov  5 23:00 /tmp
     2  E: ls: cannot access /tnt: No such file or directory

Where STDOUT go through a specific filter, STDERR to another and finally both outputs merged go through a third command filter.

3 – A word about noclobber option and >| syntax

That’s about overwriting:

While set -o noclobber instruct bash to not overwrite any existing file, the >| syntax let you pass through this limitation:

$ testfile=$(mktemp /tmp/testNoClobberDate-XXXXXX)

$ date > $testfile ; cat $testfile
Mon Jan  7 13:18:15 CET 2013

$ date > $testfile ; cat $testfile
Mon Jan  7 13:18:19 CET 2013

$ date > $testfile ; cat $testfile
Mon Jan  7 13:18:21 CET 2013

The file is overwritten each time, well now:

$ set -o noclobber

$ date > $testfile ; cat $testfile
bash: /tmp/testNoClobberDate-WW1xi9: cannot overwrite existing file
Mon Jan  7 13:18:21 CET 2013

$ date > $testfile ; cat $testfile
bash: /tmp/testNoClobberDate-WW1xi9: cannot overwrite existing file
Mon Jan  7 13:18:21 CET 2013

Pass through with >|:

$ date >| $testfile ; cat $testfile
Mon Jan  7 13:18:58 CET 2013

$ date >| $testfile ; cat $testfile
Mon Jan  7 13:19:01 CET 2013

Unsetting this option and/or inquiring if already set.

$ set -o | grep noclobber
noclobber           on

$ set +o noclobber

$ set -o | grep noclobber
noclobber           off

$ date > $testfile ; cat $testfile
Mon Jan  7 13:24:27 CET 2013

$ rm $testfile

4 – Last trick and more…

For redirecting both output from a given command, we see that a right syntax could be:

$ ls -ld /tmp /tnt >/dev/null 2>&1

for this special case, there is a shortcut syntax: &> … or >&

$ ls -ld /tmp /tnt &>/dev/null

$ ls -ld /tmp /tnt >&/dev/null

Nota: if 2>&1 exist, 1>&2 is a correct syntax too:

$ ls -ld /tmp /tnt 2>/dev/null 1>&2

4b- Now, I will let you think about:

$ ls -ld /tmp /tnt 2>&1 1>&2  | sed -e s/^/++/
++/bin/ls: cannot access /tnt: No such file or directory
++drwxrwxrwt 193 root root 196608 Feb  9 11:08 /tmp/

$ ls -ld /tmp /tnt 1>&2 2>&1  | sed -e s/^/++/
/bin/ls: cannot access /tnt: No such file or directory
drwxrwxrwt 193 root root 196608 Feb  9 11:08 /tmp/

4c- If you’re interested in more information

You could read the fine manual by hitting:

man -Len -Pless\ +/^REDIRECTION bash

in a bash console 😉