mongodump -db somedb --collection somecollection --out - | gzip > dump.gz
What is the dash mean in above picture? I seems something to do with tunnel. But after looking up bash reference, i still can't get what't the meaning.
It means output goes to the stdout, so the pipe would pass it to the gzip. And it's not about bash(1), it's up to the program mongodump. In unix and linux, the dash - related to input or output, usually stands for stdin or stdout.
oxnz is right on target. To expand upon that a little bit, some commands want a file as an argument typically at the end. In the command you describe, mongodump is hoping to output a file.
That's great, but your intent is to just compress it into a file dump_xyz. So instead of mongodump take a file as the final argument, the - indicates standard output (depending on the context, - can mean standard input). The output of mongodump directly goes to the standard output and pipes right into the waiting gzip. Gzip takes mongodump's output, compresses it on the fly and stores the compressed information into dump_xyz file.
Here's some light reading about - special character: http://tldp.org/LDP/abs/html/special-chars.html#DASHREF2
Related
Examples:
Create an ISO image and burn it directly to a CD.
mkisofs -V Photos -r /home/vivek/photos | cdrecord -v dev=/dev/dvdrw -
Change to the previous directory.
cd -
Listen on port 12345 and untar data sent to it.
nc -l -p 12345 | tar xvzf -
What is the purpose of the dash and how do I use it?
If you mean the naked - at the end of the tar command, that's common on many commands that want to use a file.
It allows you to specify standard input or output rather than an actual file name.
That's the case for your first and third example. For example, the cdrecord command is taking standard input (the ISO image stream produced by mkisofs) and writing it directly to /dev/dvdrw.
With the cd command, every time you change directory, it stores the directory you came from. If you do cd with the special - "directory name", it uses that remembered directory instead of a real one. You can easily switch between two directories quite quickly by using that.
Other commands may treat - as a different special value.
It's not magic. Some commands interpret - as the user wanting to read from stdin or write to stdout; there is nothing special about it to the shell.
- means exactly what each command wants it to mean. There are several common conventions, and you've seen examples of most of them in other answers, but none of them are 100% universal.
There is nothing magic about the - character as far as the shell is concerned (except that the shell itself, and some of its built-in commands like cd and echo, use it in conventional ways). Some characters, like \, ', and ", are "magical", having special meanings wherever they appear. These are "shell metacharacters". - is not like that.
To see how a given command uses -, read the documentation for that command.
It means to use the program's standard input stream.
In the case of cd, it means something different: change to the prior working directory.
The magic is in the convention. For millennia, people have used '-' to distinguish options from arguments, and have used '-' in a filename to mean either stdin or stdout, as appropriate. Do not underestimate the power of convention!
I know you can pipe output of a one command to another - for example
ls -la | less
to see output of ls -la inside of less instead of terminal stdio.
But if you use a command with a parameter that saves the output to a file
command --save-to-file file.txt
Then how to pipe that to another command?
this way will not work:
command --save-to-file | less
because command will complain that you used --save-to-file without any argument (filename)
If I remember well there was something like a buffer or a temp file in ram you could put instead of file.txt so you could do something like:
command --save-to-file ram-buffer.txt && cat ram-buffer.txt
Without even creating a file on disk, it that right?
Why I need that?
Some of commands have only a basic output to the stdio and more useful type of output cannot be printed by them but only saved to file. The thing is I am not interested in saving the more useful type of output to any file at all but just to print it in terminal or pipe to chain of another commands that do the filtering etc and then eventually print the processed output.
I would not like to be responsible to crating a tmp file then delete it etc. Perfectly I would like to just use a kind of magic file (or redirection) in place of file.txt that I could pipe to another command.
It is important to me to not write any content of the output to a disk if this is possible. Just print it in terminal or pipe to other command(s).
At this moment I'm trying to capture output of PHPUnit
phpunit --log-junit log.xml
which is not a shell command but a PHP script that uses:
#!/usr/bin/env php
But I remember I used to have an example with linux command that I wanted get the output but the form of it was only available with a parameter --save-to-file outputfile.txt
Perhaps because piping/redirecting an output designed to be saved to a file is not binary safe and therefore such output can be corrupted when piped/redirected - can it be?
Some programs have special handling for -. For example, you can tell tar to write to stdout so it can be used in a pipeline. This would create a tarball locally and untar it remotely without the tarball ever being written to disk:
tar -cf - *.txt | ssh user#host tar -C /dir/ -xf -
You can use /dev/stdout with nearly all programs, as long as they don't need a seekable file.
command --save-to-file /dev/stdout
As #Benjamin W. pointed out in the comments, you can save it to /dev/stdout, which is the output and then pipe the output to whatever you want(e.g. less)
command --save-to-file /dev/stdout | less
Take care because there may be additional output to stdout. In this case you could throw that away, save it to stderr and redirect it to stdout:
command --save-to-file /dev/stderr >/dev/null 2>/dev/stdout | less
If both, stderr and stdout are used you may be able to write your own driver for this of manipulate /proc/pid/mem or something like this.
Examples:
Create an ISO image and burn it directly to a CD.
mkisofs -V Photos -r /home/vivek/photos | cdrecord -v dev=/dev/dvdrw -
Change to the previous directory.
cd -
Listen on port 12345 and untar data sent to it.
nc -l -p 12345 | tar xvzf -
What is the purpose of the dash and how do I use it?
If you mean the naked - at the end of the tar command, that's common on many commands that want to use a file.
It allows you to specify standard input or output rather than an actual file name.
That's the case for your first and third example. For example, the cdrecord command is taking standard input (the ISO image stream produced by mkisofs) and writing it directly to /dev/dvdrw.
With the cd command, every time you change directory, it stores the directory you came from. If you do cd with the special - "directory name", it uses that remembered directory instead of a real one. You can easily switch between two directories quite quickly by using that.
Other commands may treat - as a different special value.
It's not magic. Some commands interpret - as the user wanting to read from stdin or write to stdout; there is nothing special about it to the shell.
- means exactly what each command wants it to mean. There are several common conventions, and you've seen examples of most of them in other answers, but none of them are 100% universal.
There is nothing magic about the - character as far as the shell is concerned (except that the shell itself, and some of its built-in commands like cd and echo, use it in conventional ways). Some characters, like \, ', and ", are "magical", having special meanings wherever they appear. These are "shell metacharacters". - is not like that.
To see how a given command uses -, read the documentation for that command.
It means to use the program's standard input stream.
In the case of cd, it means something different: change to the prior working directory.
The magic is in the convention. For millennia, people have used '-' to distinguish options from arguments, and have used '-' in a filename to mean either stdin or stdout, as appropriate. Do not underestimate the power of convention!
Examples:
Create an ISO image and burn it directly to a CD.
mkisofs -V Photos -r /home/vivek/photos | cdrecord -v dev=/dev/dvdrw -
Change to the previous directory.
cd -
Listen on port 12345 and untar data sent to it.
nc -l -p 12345 | tar xvzf -
What is the purpose of the dash and how do I use it?
If you mean the naked - at the end of the tar command, that's common on many commands that want to use a file.
It allows you to specify standard input or output rather than an actual file name.
That's the case for your first and third example. For example, the cdrecord command is taking standard input (the ISO image stream produced by mkisofs) and writing it directly to /dev/dvdrw.
With the cd command, every time you change directory, it stores the directory you came from. If you do cd with the special - "directory name", it uses that remembered directory instead of a real one. You can easily switch between two directories quite quickly by using that.
Other commands may treat - as a different special value.
It's not magic. Some commands interpret - as the user wanting to read from stdin or write to stdout; there is nothing special about it to the shell.
- means exactly what each command wants it to mean. There are several common conventions, and you've seen examples of most of them in other answers, but none of them are 100% universal.
There is nothing magic about the - character as far as the shell is concerned (except that the shell itself, and some of its built-in commands like cd and echo, use it in conventional ways). Some characters, like \, ', and ", are "magical", having special meanings wherever they appear. These are "shell metacharacters". - is not like that.
To see how a given command uses -, read the documentation for that command.
It means to use the program's standard input stream.
In the case of cd, it means something different: change to the prior working directory.
The magic is in the convention. For millennia, people have used '-' to distinguish options from arguments, and have used '-' in a filename to mean either stdin or stdout, as appropriate. Do not underestimate the power of convention!
I'm combining sox and lame to generate a new music file, but in order to do everything on one line using pipes, it seems necessary to 'mark' the output and input boundaries with a - character. I've inherited this code, so let me show.
sox $DIRNAME/$BASENAME -e signed-integer -r 8000 -c 2 -t wav - trim $POSITIONS | lame -v -V4 --resample 8 - $DIRNAME/${NOEXT}.mp3
The - between wav and trim is the output file, and the - between --resample 8 and $DIRNAME/${NOEXT}.mp3 is the input file.
I'm trying to find further information on this, like whether any character can be used, or if - is special in this way. What is this called, and what makes it work?
Many Unix command-line utilities use "-" as a shorthand to mean "don't use a real file here, use stdin (or stdout) instead". Sox is one of these programs:
This is from the sox manpage
SoX can be used in simple pipeline operations by using the special
filename '-' which, if used in place of an input filename, will cause
SoX will read audio data from 'standard input' (stdin), and which, if
used in place of the output filename, will cause SoX will send audio
data to 'standard output' (stdout). Note that when using this option,
the file-type (see -t below) must also be given.
By convention, unix and friends use - to represent stdin and stdout.
It's not 100% universal, but it's a pretty widely used.
In your example, it's the same thing as
/dev/stdin
Try to replace your - with it, you will see.
"-" is often use as a convention instead of a file name to say "use the standard input (instead or reading for the file) or standard output (instead of writing to the file)". This is not a feature of the command shell (i.e. bash), so in that sense, it is not a special character. It is a feature of some commands (like in your case "sox" and "lame") and is very useful to chain these commands through pipes.