I've been trying to write a function for my bash profile for quite some time now.
The problem I'm trying to overcome is I'm usually provided with file paths that include spaces and it's a pain having to go through and escape all the spaces before I try to open it up in terminal.
e.g.
File -> /Volumes/Company/Illustrators/Website Front Page Design.ai
What I'm trying to end up with is '/Volumes/Company/Illustrators/Website\ Front\ Page\ Design.ai' being opened from my terminal.
So far I've managed to escape the spaces out, but I then get the error "The file ..... does not exist."
My code so far is
function opn { open "${1// /\\ }";}
Any help would be very much appreciated.
The important thing to understand is the difference between syntax and literal data.
When done correctly, escaping is syntax: It's read and discarded by the shell. That is, when you run
open "File With Spaces"
or
open File\ With\ Spaces
or even
open File" "With\ Spaces
...the quoting and escaping is parsed and removed by the shell, and the actual operating system call that gets executed is this:
execv("/usr/bin/open", "open", "File With Spaces")
Note that there aren't any backslashes (or literal quotes) in that syscall's arguments! If you put literal backslashes in your data, then you cause this to be run:
/* this is C syntax, so "\\" is a single-character backslash literal */
execv("/usr/bin/open", "open", "File\\ With\\ Spaces")
...and unless there's a file with backslashes in its name, that just doesn't work, giving the "file does not exist" error you report.
So -- just call open with your name in quotes:
open "$1"
...there's no need for an opn wrappper.
Spaces are problematic in filenames because they're part of bash's default IFS (Internal Field Separator), which is used to separate tokens in a command line. That means that by default, when you use command an argument with spaces, the command will receive 4 arguments rather than 1 containing spaces.
I'm guessing you called your opn function in the same way, thus resulting in only the first part of your path as $1.
Hopefully, the fix is easy : enclose your path in quotes so that bash does not interpret the spaces. By using this, the need for your opn function disappears : open "/Volumes/Company/Illustrators/Website Front Page Design.ai" should work just fine.
Related
#!/bin/bash
documents=("/datalog/AB errors.txt");
/usr/bin/bash /home/user/download.sh "${documents[*]}"
the download.sh SCPs into a server and downloads documents but in the case above there is a space in the file name which leads it to seek /datalog/AB instead of AB errors.txt file.
I thought surrounding in double quotes fixes the space in filename issue. I also tried AB\ errors.txt but that caused the entire bash script to not run.
The form
documents=("/datalog/AB errors.txt")
is bad form (turning string into an array !!!). The correct form for assignment is
documents="/datalog/AB errors.txt"
Always assume full-path variables can have spaces or other characters, and always double-quote such variable instantiations, i.e.
ls "${variable}" .
As Gordon Davisson said, you must code in this fashion at every level of scripting/programming to ensure the embedded special characters don't lead to unexpected results.
i'm very new to stackoverflow and to bash/python scripting.
I'm in need to resize some Data Terrain Model files (300+) in .tif format to be able to convert 'em into .hgt and i'm able to do it all using gdal tool but only per single file at once.
Guess you alredy spotted where scripting comes in: need to automatize the process for the 300+ files!
So i started looking a bit about how bash works and came out with this:
#!/bin/bash
for filename in "'/home/fenix/1\ Vari\ HDD/MTB/DTM\ Alos/'"*.tif; do
PATH=/usr/bin/ gdalwarp -of Gtiff -ts 3601 3601 $filename.tif "'/home/fenix/1\ Vari\ HDD/MTB/DTM Alos/temp/'"$filename.tif
done
I always used the backslash to move into "spaced" name directories or files but seems not working with scripting.... googleing i found using quotes or double quotes would fix it but still no success
As you have seen in the code above i used double quote, quote and backslash alone and any combination of the 3 but i'm always getting
ERROR 4: '/home/fenix/1: No such file or directory
Why?!?!
Thanks in advance and sorry for my english!
EDIT:
Following tripleee golden suggestions i edited the script like:
#!/bin/bash
PATH=/usr/bin/
for filename in "/home/fenix/1 Vari HDD/MTB/DTM Alos/"*.tif; do
gdalwarp -of Gtiff -ts 3601 3601 "$filename" "/home/fenix/1 Vari HDD/MTB/DTM Alos/temp/${filename##*/}"
done
And worked like a charm!
Your excessive quoting is getting in the way.
#!/bin/bash
for filename in "/home/fenix/1 Vari HDD/MTB/DTM Alos/"*.tif; do
PATH=/usr/bin/ gdalwarp -of Gtiff -ts 3601 3601 "$filename" "${filename##*/}"
done
The string /home/fenix/stuff with spaces can be expressed as either of
/home/fenix/stuff\ with\ spaces
"/home/fenix/stuff with spaces"
'/home/fenix/stuff with spaces'
A backslash or quote within quotes produces a literal backslash or quote, as part of the quoted string. A backslashed backslash or quote similarly produces a literal backslash or quote.
Single quotes are stronger; everything between them is literal. Double quotes allow for variable and backtick expansion, and backslash quoting.
So "'/home/fenix/1\ Vari\ HDD/MTB/DTM\ Alos/'" refers to ./'/home/fenix/1\ Vari\ HDD/MTB/DTM\ Alos/ which probably isn't a valid path, unless the current directory contains a directory whose name is literally a single quote, etc (where I put in the leading ./ just to make this more explicit).
Perhaps a complication is that the quotes inhibit wildcard expansion; so the wildcard *.tif needs to be unquoted. (Well, strictly speaking, only the wildcard needs to be unquoted; *'.tif' or *".tif" or *\.\t\i\f would work, too.)
Notice also that the value of $filename is the full path to each expanded value of the wildcard, with no directory prefix or extension suffix trimmed off or any other magic like that. I have speculatively shown how to pass the last argument as the filename with the directory path trimmed off (the parameter substitution ${variable##pattern} retrieves the value of variable with any prefix matching pattern trimmed off). So the output file should land in the current directory, with inp^t from the wildcard match (hopefully then in a different directory, so you don't end up overwriting your input files).
Finally, observe how we take care to always use double quotes around variables which contain file names. (Failing to do this is a common error even in some fairly popular tutorials. The script will appear to work until you try to handle file names with irregular spacing or literal asterisks, etc.)
The wacky PATH assignment looks weird, too. Does gdalwarp execute external commands, and do you really then want it to only find external commands in /usr/bin? Or perhaps you mean to run /usr/bin/gdalwarp (though setting the correct PATH at the beginning of the script would arguably be better than hardcoding a specific absolute pathname).
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'
This is my very simple code, which isn't working, for some reason I can't figure out.
#!/usr/bin/perl
use File::Copy;
$old = "car_lexusisf_gray_30inclination_000azimuth.png";
$new = "C:\Users\Lenovo\Documents\mycomp\simulation\cars\zzzorganizedbyviews\00inclination_000azimuth\lexuscopy.png";
copy ($old, $new) or die "File cannot be copied.";
I get the error that the file can't be copied.
I know there's nothing wrong with the copy command because if I set the value of $new to something simple without a path, it works. But what is wrong in the representation of the path as I've written it above? If I copy and past it into the address bar of windows explorer, it reaches that folder fine.
Tip: print out the paths before you perform the copy. You'll see this:
C:SERSenovodocumentsmycompsimulationrszzzorganizedbyviewsinclination_000azimuthexuscopy.png
Not what we wanted. The backslash is an escape character in Perl, which needs to be escaped itself. If the backslash sequence does not form a valid escape, then it's silently ignored. With escaped backslashes, your string would look like:
"C:\\Users\\Lenovo\\Documents\\mycomp\\simulation\\cars\\zzzorganizedbyviews\\00inclination_000azimuth\\lexuscopy.png";
or just use forward slashes instead – in most cases, Unix-style paths work fine on Windows too.
Here is a list of escapes you accidentally used:
\U uppercases the rest
\L lowercases the rest
\ca is a control character (ASCII 1, the start of heading)
\00 is an octal character, here the NUL byte
\l lowercases the next character.
If no interpolation is intended, use single quotes instead of double quotes.
I want to run a command from a bash script which has single quotes and some other commands inside the single quotes and a variable.
e.g. repo forall -c '....$variable'
In this format, $ is escaped and the variable is not expanded.
I tried the following variations but they were rejected:
repo forall -c '...."$variable" '
repo forall -c " '....$variable' "
" repo forall -c '....$variable' "
repo forall -c "'" ....$variable "'"
If I substitute the value in place of the variable the command is executed just fine.
Please tell me where am I going wrong.
Inside single quotes everything is preserved literally, without exception.
That means you have to close the quotes, insert something, and then re-enter again.
'before'"$variable"'after'
'before'"'"'after'
'before'\''after'
Word concatenation is simply done by juxtaposition. As you can verify, each of the above lines is a single word to the shell. Quotes (single or double quotes, depending on the situation) don't isolate words. They are only used to disable interpretation of various special characters, like whitespace, $, ;... For a good tutorial on quoting see Mark Reed's answer. Also relevant: Which characters need to be escaped in bash?
Do not concatenate strings interpreted by a shell
You should absolutely avoid building shell commands by concatenating variables. This is a bad idea similar to concatenation of SQL fragments (SQL injection!).
Usually it is possible to have placeholders in the command, and to supply the command together with variables so that the callee can receive them from the invocation arguments list.
For example, the following is very unsafe. DON'T DO THIS
script="echo \"Argument 1 is: $myvar\""
/bin/sh -c "$script"
If the contents of $myvar is untrusted, here is an exploit:
myvar='foo"; echo "you were hacked'
Instead of the above invocation, use positional arguments. The following invocation is better -- it's not exploitable:
script='echo "arg 1 is: $1"'
/bin/sh -c "$script" -- "$myvar"
Note the use of single ticks in the assignment to script, which means that it's taken literally, without variable expansion or any other form of interpretation.
The repo command can't care what kind of quotes it gets. If you need parameter expansion, use double quotes. If that means you wind up having to backslash a lot of stuff, use single quotes for most of it, and then break out of them and go into doubles for the part where you need the expansion to happen.
repo forall -c 'literal stuff goes here; '"stuff with $parameters here"' more literal stuff'
Explanation follows, if you're interested.
When you run a command from the shell, what that command receives as arguments is an array of null-terminated strings. Those strings may contain absolutely any non-null character.
But when the shell is building that array of strings from a command line, it interprets some characters specially; this is designed to make commands easier (indeed, possible) to type. For instance, spaces normally indicate the boundary between strings in the array; for that reason, the individual arguments are sometimes called "words". But an argument may nonetheless have spaces in it; you just need some way to tell the shell that's what you want.
You can use a backslash in front of any character (including space, or another backslash) to tell the shell to treat that character literally. But while you can do something like this:
reply=\”That\'ll\ be\ \$4.96,\ please,\"\ said\ the\ cashier
...it can get tiresome. So the shell offers an alternative: quotation marks. These come in two main varieties.
Double-quotation marks are called "grouping quotes". They prevent wildcards and aliases from being expanded, but mostly they're for including spaces in a word. Other things like parameter and command expansion (the sorts of thing signaled by a $) still happen. And of course if you want a literal double-quote inside double-quotes, you have to backslash it:
reply="\"That'll be \$4.96, please,\" said the cashier"
Single-quotation marks are more draconian. Everything between them is taken completely literally, including backslashes. There is absolutely no way to get a literal single quote inside single quotes.
Fortunately, quotation marks in the shell are not word delimiters; by themselves, they don't terminate a word. You can go in and out of quotes, including between different types of quotes, within the same word to get the desired result:
reply='"That'\''ll be $4.96, please," said the cashier'
So that's easier - a lot fewer backslashes, although the close-single-quote, backslashed-literal-single-quote, open-single-quote sequence takes some getting used to.
Modern shells have added another quoting style not specified by the POSIX standard, in which the leading single quotation mark is prefixed with a dollar sign. Strings so quoted follow similar conventions to string literals in the ANSI standard version of the C programming language, and are therefore sometimes called "ANSI strings" and the $'...' pair "ANSI quotes". Within such strings, the above advice about backslashes being taken literally no longer applies. Instead, they become special again - not only can you include a literal single quotation mark or backslash by prepending a backslash to it, but the shell also expands the ANSI C character escapes (like \n for a newline, \t for tab, and \xHH for the character with hexadecimal code HH). Otherwise, however, they behave as single-quoted strings: no parameter or command substitution takes place:
reply=$'"That\'ll be $4.96, please," said the cashier'
The important thing to note is that the single string that gets stored in the reply variable is exactly the same in all of these examples. Similarly, after the shell is done parsing a command line, there is no way for the command being run to tell exactly how each argument string was actually typed – or even if it was typed, rather than being created programmatically somehow.
Below is what worked for me -
QUOTE="'"
hive -e "alter table TBL_NAME set location $QUOTE$TBL_HDFS_DIR_PATH$QUOTE"
EDIT: (As per the comments in question:)
I've been looking into this since then. I was lucky enough that I had repo laying around. Still it's not clear to me whether you need to enclose your commands between single quotes by force. I looked into the repo syntax and I don't think you need to. You could used double quotes around your command, and then use whatever single and double quotes you need inside provided you escape double ones.
just use printf
instead of
repo forall -c '....$variable'
use printf to replace the variable token with the expanded variable.
For example:
template='.... %s'
repo forall -c $(printf "${template}" "${variable}")
Variables can contain single quotes.
myvar=\'....$variable\'
repo forall -c $myvar
I was wondering why I could never get my awk statement to print from an ssh session so I found this forum. Nothing here helped me directly but if anyone is having an issue similar to below, then give me an up vote. It seems any sort of single or double quotes were just not helping, but then I didn't try everything.
check_var="df -h / | awk 'FNR==2{print $3}'"
getckvar=$(ssh user#host "$check_var")
echo $getckvar
What do you get? A load of nothing.
Fix: escape \$3 in your print function.
Does this work for you?
eval repo forall -c '....$variable'