Examples:
Create an ISO image and burn it directly to a CD.
mkisofs -V Photos -r /home/vivek/photos | cdrecord -v dev=/dev/dvdrw -
Change to the previous directory.
cd -
Listen on port 12345 and untar data sent to it.
nc -l -p 12345 | tar xvzf -
What is the purpose of the dash and how do I use it?
If you mean the naked - at the end of the tar command, that's common on many commands that want to use a file.
It allows you to specify standard input or output rather than an actual file name.
That's the case for your first and third example. For example, the cdrecord command is taking standard input (the ISO image stream produced by mkisofs) and writing it directly to /dev/dvdrw.
With the cd command, every time you change directory, it stores the directory you came from. If you do cd with the special - "directory name", it uses that remembered directory instead of a real one. You can easily switch between two directories quite quickly by using that.
Other commands may treat - as a different special value.
It's not magic. Some commands interpret - as the user wanting to read from stdin or write to stdout; there is nothing special about it to the shell.
- means exactly what each command wants it to mean. There are several common conventions, and you've seen examples of most of them in other answers, but none of them are 100% universal.
There is nothing magic about the - character as far as the shell is concerned (except that the shell itself, and some of its built-in commands like cd and echo, use it in conventional ways). Some characters, like \, ', and ", are "magical", having special meanings wherever they appear. These are "shell metacharacters". - is not like that.
To see how a given command uses -, read the documentation for that command.
It means to use the program's standard input stream.
In the case of cd, it means something different: change to the prior working directory.
The magic is in the convention. For millennia, people have used '-' to distinguish options from arguments, and have used '-' in a filename to mean either stdin or stdout, as appropriate. Do not underestimate the power of convention!
Related
Examples:
Create an ISO image and burn it directly to a CD.
mkisofs -V Photos -r /home/vivek/photos | cdrecord -v dev=/dev/dvdrw -
Change to the previous directory.
cd -
Listen on port 12345 and untar data sent to it.
nc -l -p 12345 | tar xvzf -
What is the purpose of the dash and how do I use it?
If you mean the naked - at the end of the tar command, that's common on many commands that want to use a file.
It allows you to specify standard input or output rather than an actual file name.
That's the case for your first and third example. For example, the cdrecord command is taking standard input (the ISO image stream produced by mkisofs) and writing it directly to /dev/dvdrw.
With the cd command, every time you change directory, it stores the directory you came from. If you do cd with the special - "directory name", it uses that remembered directory instead of a real one. You can easily switch between two directories quite quickly by using that.
Other commands may treat - as a different special value.
It's not magic. Some commands interpret - as the user wanting to read from stdin or write to stdout; there is nothing special about it to the shell.
- means exactly what each command wants it to mean. There are several common conventions, and you've seen examples of most of them in other answers, but none of them are 100% universal.
There is nothing magic about the - character as far as the shell is concerned (except that the shell itself, and some of its built-in commands like cd and echo, use it in conventional ways). Some characters, like \, ', and ", are "magical", having special meanings wherever they appear. These are "shell metacharacters". - is not like that.
To see how a given command uses -, read the documentation for that command.
It means to use the program's standard input stream.
In the case of cd, it means something different: change to the prior working directory.
The magic is in the convention. For millennia, people have used '-' to distinguish options from arguments, and have used '-' in a filename to mean either stdin or stdout, as appropriate. Do not underestimate the power of convention!
This question already has answers here:
How to obtain the first letter in a Bash variable?
(7 answers)
Closed 3 years ago.
I am trying to my a custom terminal command. I just learned I am supposed to do it using the Unix script? I don't really know much of what that is and am still trying to figure it out. What I do know is that $1 is an arg is it possible to make it a variable and then get the first letter like you could in python?
EX:
str = 'happy'
str[0] = 'h'
You're asking a few different things here.
I am trying to my a custom terminal command.
That could mean a few different things, but the most obvious meaning is that you want to add an executable to your path so that when you type it at the terminal, it runs just like any other executable on your system. This requires just a few things:
the executable permission must be set.
the file must specify how it can be executed. For interpreted programs such as bash scripts or python scripts, you can do so by beginning the file with a "shebang line" that specifies the interpreter for the file.
the file must be in one of the locations specified by your $PATH.
I just learned I am supposed to do it using the Unix script?
there's no such thing as a "unix script", but what you seem to be referring to is a "shell script". Though these are commonly associated with unix, they're no more inherently a unix script than any other language. A shell, such as bash, sh, or any other, is just an interpreted language that is designed so that it is convenient to be used interactively by a human as well as being programmatically executed as part of a saved file.
I don't really know much of what that is and am still trying to figure it out.
Let's get into some specifics.
First I edit a file called 'hello-world' to contain:
#!/bin/bash
echo "Hello, world!"
Note that this filename has no "extension". Though heuristics based on file extension are sometimes used (espeically in windows) to determine a file type, unix typically sees a file "extension" as part of the arbitrary file name. The thing that makes this a potentially executable bash script is the specification of that interpreter on the shebang line.
We can run our script right now from bash, just as we could if we wrote a python script.
$ bash hello-world
hello, world!
To make the bash implicit, we mark the file as executable. This enables the linux operating system to consult the beginning "magic bytes" of the file to determine how to run it. Thes beginning bytes might signify an ELF file (a compiled executable, written in eg C, C++, or go). Or, it might be #! which just so happens means , "read the rest of this first line to determine the command to run, and pass the rest of this file into that command to be interpreted.
$ chmod +x hello-world
ls -l will show us the "permissions" on the file (more accurately called the "file mode", hence chmod rather than chperm) . The x stands for executable, so we have enabled the use of the leading bytes to determine method of execution. Remember, the first two bytes of this file, and the rest of that first line, then specify that this file should be "run through bash" so to speak.
$ ls -l hello-world
-rwxr-xr-x 1 danfarrell staff 33 Dec 27 20:02 hello-world
Now we can run the file from the current directory:
$ ./hello-world
hello, world!
At this point, the only difference between this command and any other on the system, is that you have to specify its location. That's because my current directory is not in the system path. In short, the path (accessible in a unix shell via the $PATH variable) specifies an ordered list of locations that should be searched for a specified command whose location is not otherwise specified.
For example, there's a very common program called whoami. I can run it directly from my terminal without specifying a location of the executable:
$ whoami
danfarrell
This is because there's a location in my $PATH in which the shell was able to find that command. Let's take a closer look. First, here's my path:
$ echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/go/bin
And there's also a convenient program called whereis which can help show which path elements supply a named executable:
$ whereis whoami
/usr/bin/whoami
Sure enough, whoami is in one of the elements of the $PATH. (Actually I shared a simplified $PATH. Yours might be somewhat longer).
Finally, then, we can get to the last thing. If I put hello-world in one of the $PATH elements, I will be able to invoke it without a path. There are two ways to do this: we can move the executable to a location specified in the path, or we can add a new location to the path. For simplicity's sake I'll choose the first of these.
$ sudo cp hello-world /usr/local/bin/
Password:
I needed to use sudo to write to /usr/local/bin because it's not accessible as my user directly - that's quite standard.
Finally, I've achieved the goal of being able to run my very important program from any location, without specifying the executable's location.
$ hello-world
hello, world!
$ which hello-world
/usr/local/bin/hello-world
It works! I've created what might be described as a "custom terminal command".
What I do know is that $1 is an arg is it possible to make it a variable and then get the first letter like you could in python?
Well, one option would be to simply write the custom terminal command in python. If python is available,
$ which python
/usr/bin/python
You can specify it in a shebang just like a shell can be:
#!/usr/bin/env python
print("hello, world!"[0])
$ hello-world
h
it works!
Okay, confession time. I actually used #!/usr/bin/env python, not /usr/bin/python. env helps find the correct python to use in the user's environment, rather than hard coding one particular python. If you've been using python during the very long running python 2 to python 3 migration, you can no doubt understand why I"m reticent to hard code a python executable in my program.
It's certainly possible to get the first letter of a string in a bash script. But it's also very possible to write a custom command in a program other than shell. Python is an excellent choice for string manipulation, if you know it. I often use python for shell one-liners that need to interact with json, a format that doesn't lend itself well to standard unix tool stream editing.
Anyway, at the expense of incurring SO community's ire by reanswering an "already answered" question, I'll include a version in shell (Credit goes to David C Rankin)
#!/bin/bash
echo "${1:0:1}"
$ hello-world hiworld
h
I am trying to rename using "mv", because "rename" command doesn't work in my Mac terminal.
I have a bunch of files named
DTM001_ACGGT-TTAGGC.fq
DTM156_GGTTG-ACAGTG.fq
...etc
I wish to rename them to
DTM001.fq
DTM156.fq
I suppose the easier way is to remove the last 13 characters before the file extension?
I tried these links:
mac os x terminal batch rename
Rename file by removing last n characters
Removing last n characters from Unix Filename before the extension
but none have worked for me, perhaps because I do not fully understand how to manipulate the answers for my specific case or some answers use "rename" command which I cannot access.
The macOS Terminal is simply an interface to an interactive program called a shell. The default shell's name is bash.
What you are looking for is known as a shell script, or a bash script, to rename files.
The questions you referenced have the answer. To reiterate:
cd directory_with_the_files
for file in *.fq; do
mv -vn "${file}" "${file%_*}.fq"
done
You can type this all in at the command line, or place it into a file and execute it with:
bash file_containing_the_commands
This will go through all .fq files in the current directory, renaming them to what you want. The -v option to mv simply means to print the rename as it happens (useful to know that it's doing something), and the -n flag means don't accidentally overwrite any files (in case you type something in wrong or come across duplicate numbers).
All the magic is happening in the ${file%_*}.fq, which says
"remove everything after the first _ and add the .fq back". This is known as a "shell parameter expansion," which you can read more about in the Bash Reference Manual. It's somewhat obtusely worded, but here is the relevant bit to this particular use case:
${parameter%word}
The word is expanded to produce a
pattern just as in filename expansion. If the pattern matches a
trailing portion of the expanded value of parameter, then the result
of the expansion is the value of parameter with the shortest matching
pattern (the '%' case) deleted.
The simplest way is to use rename - see instructions at the end for installation on a Mac.
So, in answer to your question, you can see what would happen if you replace (the command is actually s for "substitute") everything from the first underscore to the end of the filename with .fq:
rename --dry-run 's/_.*/.fq/' *fq
'DTM001_ACGGT-TTAGGC.fq' would be renamed to 'DTM001.fq'
'DTM156_GGTTG-ACAGTG.fq' would be renamed to 'DTM156.fq'
If that looks good, remove the --dry-run and run it again for real.
You can use rename on your Mac, if you install it. By default, Apple doesn't ship a package manager with macOS. So, many folk use homebrew from the homebrew website.
If you have that, you can simply install rename with:
brew install rename
Then, you'll have a package manager and you can benefit for all sorts of lovely software including new, up-to-date versions of all the out-of-date, ancient versions of your favourite tools that Apple ships:
PHP
Perl
ImageMagick
GNU sed
GNU awk
GNU find
GNU Parallel
zeromq
htop
socat
sox
ffmpeg
youtube-dl
zenity
redis
feh
mosquitto
doxygen
pandoc etc.
Examples:
Create an ISO image and burn it directly to a CD.
mkisofs -V Photos -r /home/vivek/photos | cdrecord -v dev=/dev/dvdrw -
Change to the previous directory.
cd -
Listen on port 12345 and untar data sent to it.
nc -l -p 12345 | tar xvzf -
What is the purpose of the dash and how do I use it?
If you mean the naked - at the end of the tar command, that's common on many commands that want to use a file.
It allows you to specify standard input or output rather than an actual file name.
That's the case for your first and third example. For example, the cdrecord command is taking standard input (the ISO image stream produced by mkisofs) and writing it directly to /dev/dvdrw.
With the cd command, every time you change directory, it stores the directory you came from. If you do cd with the special - "directory name", it uses that remembered directory instead of a real one. You can easily switch between two directories quite quickly by using that.
Other commands may treat - as a different special value.
It's not magic. Some commands interpret - as the user wanting to read from stdin or write to stdout; there is nothing special about it to the shell.
- means exactly what each command wants it to mean. There are several common conventions, and you've seen examples of most of them in other answers, but none of them are 100% universal.
There is nothing magic about the - character as far as the shell is concerned (except that the shell itself, and some of its built-in commands like cd and echo, use it in conventional ways). Some characters, like \, ', and ", are "magical", having special meanings wherever they appear. These are "shell metacharacters". - is not like that.
To see how a given command uses -, read the documentation for that command.
It means to use the program's standard input stream.
In the case of cd, it means something different: change to the prior working directory.
The magic is in the convention. For millennia, people have used '-' to distinguish options from arguments, and have used '-' in a filename to mean either stdin or stdout, as appropriate. Do not underestimate the power of convention!
I'm combining sox and lame to generate a new music file, but in order to do everything on one line using pipes, it seems necessary to 'mark' the output and input boundaries with a - character. I've inherited this code, so let me show.
sox $DIRNAME/$BASENAME -e signed-integer -r 8000 -c 2 -t wav - trim $POSITIONS | lame -v -V4 --resample 8 - $DIRNAME/${NOEXT}.mp3
The - between wav and trim is the output file, and the - between --resample 8 and $DIRNAME/${NOEXT}.mp3 is the input file.
I'm trying to find further information on this, like whether any character can be used, or if - is special in this way. What is this called, and what makes it work?
Many Unix command-line utilities use "-" as a shorthand to mean "don't use a real file here, use stdin (or stdout) instead". Sox is one of these programs:
This is from the sox manpage
SoX can be used in simple pipeline operations by using the special
filename '-' which, if used in place of an input filename, will cause
SoX will read audio data from 'standard input' (stdin), and which, if
used in place of the output filename, will cause SoX will send audio
data to 'standard output' (stdout). Note that when using this option,
the file-type (see -t below) must also be given.
By convention, unix and friends use - to represent stdin and stdout.
It's not 100% universal, but it's a pretty widely used.
In your example, it's the same thing as
/dev/stdin
Try to replace your - with it, you will see.
"-" is often use as a convention instead of a file name to say "use the standard input (instead or reading for the file) or standard output (instead of writing to the file)". This is not a feature of the command shell (i.e. bash), so in that sense, it is not a special character. It is a feature of some commands (like in your case "sox" and "lame") and is very useful to chain these commands through pipes.