Im running the below script to run some checks on file paths within a function
Some of the paths contains single quotes and the temporary files contain "$"
If I enclose the variable in single quotes (name variable below) then the string in truncated if there is a ' in the file path. If I use a double quote around the filepath then it truncates if there is a "$" in the path.
Is there any way out of this circular conundrum?
File=/root/fed/~$reader.txt
Is echoing as if there is a $ in the file path
/root/fed/eager.txt
if there is a ' in the file path and I enclose around single quote (to stop the above from happening) then
File='/root/fed/reader's'
(this wont echo)
Code is :
find / -type f -print0 | xargs -0 -I name bash -c "fnchash 'name'"
fnchash () {
echo "$1"
}
Single-quoted strings may not contain single quotes. Not even by escaping them. Unquoted or double-quoted and unescaped $ introduces a variable reference that expands to the value of the referenced shell variable, or to nothing if no such variable is defined.
One solution involves double quotes and escape characters. For example:
File=/root/fed/~\$reader.txt
File=/root/fed/reader\'s
File="/root/fed/\$reader's.txt"
Note, too, that quotes are not necessarily string delimiters -- they are substring delimiters. Thus, these work, too:
File=/root/fed/~'$'reader.txt
File=/root/fed/reader"'"s
If you need to perform automatic quoting of data read at runtime, then you should also be aware of bash's built-in printf command (which is more featureful than what you may have as a standalone command). Note in particular that the %q field descriptor performs all needed quoting on its arguments to make them re-usable as shell input.
printf '%q' $File
The way out of this conundrum is to stop trying to treat data as code. This snippet passes filenames as data (arguments) instead of trying to inject them into a code string, thereby preventing these issues entirely:
$ cat myscript
fnchash () {
for arg
do
printf "Now processing: %s\n" "$arg"
done
}
export -f fnchash
find . -type f -exec bash -c 'fnchash "$#"' _ {} +
$ ls -1
File with space and '.txt
myscript
some file.txt
$ bash myscript
Now processing: ./some file.txt
Now processing: ./myscript
Now processing: ./File with space and '.txt
This uses the fact that you can call bash -c 'command' argv0 argv1 argv2.. to run 'command' with the positional parameters $0, $1, $2.. set to arguments passed to bash and process them as if they were arguments to a script (man bash under -c).
-exec bash -c 'command' _ {} + is then used to run command with $0 set to a dummy value _ and the rest of the parameters set to filenames that find finds (man find under -exec).
The bash command can then process these arguments just like any script would.
The same technique can also be used with xargs, here to parallelize the process 5 ways in chunks of 20 files:
find . -type f -print0 | xargs -0 -P 5 -n 20 bash -c 'fnchash "$#"' _
You can solve this by using single quotes and replacing all single quotes with escaped ones,
Or
Using double quotes and replacing all $ with escaped ones.
Related
I have being trying to write a bash script that can search recursively in a directory and replace multiple strings e.g. #{DEMO_STRING_1} etc with an environment variable e.g. $sample1.
Full script:
#!/bin/sh
find /my/path/here -type f -name '*.js' -exec sed -i \
-e 's/#{DEMO_STRING_1}/'"$sample1"'/g' \
-e 's/#{DEMO_STRING_2}/'"$sample2"'/g' \
-e 's/#{DEMO_STRING_3}/'"$sample3"'/g' \
-e 's/#{DEMO_STRING_4}/'"$sample4"'/g' \
-e 's/#{DEMO_STRING_5}/'"$sample5"'/g' \
-e 's/#{DEMO_STRING_6}/'"$sample6"'/g' \
-e 's/#{DEMO_STRING_7}/'"$sample7"'/g' \
-e 's/#{DEMO_STRING_8}/'"$sample8"'/g' \
{} +
I can not figure out how to replace strings with hashtag with curly brackets.
I tried this example: sed find and replace with curly braces or Environment variable substitution in sed but I can not figure out how to combine them.
What I am missing? I searched also for characters that need to be escaped e.g. What characters do I need to escape when using sed in a sh script? but again not the characters that I need.
The specific format is throwing the following error:
sed: bad option in substitution expression
Where am I going so wrong?
Update: Sample of environment variables:
https://www.example.com
/sample string/
12345-abcd-54321-efgh
base64 string
All the cases above are environment variables that I would like to replace. All environment variables are within double quotes.
It is important to understand that the environment variable references are expanded by the shell, as it prepares to execute the command, not by the command itself (sed in this case). The command sees only the results of the expansions.
In your case, that means that if any of the environment variables' values contain characters that are meaningful to sed in context, such as unescaped (to sed) slashes (/), then sed will attribute special significance to them instead of interpreting them as ordinary characters. For example, given a sed command such as
sed -e "s/X/${var}/" <<EOF
Replacement: X
EOF
, if the value of $var is Y then the output will be
Replacement: Y
, but if the value of $var is /path/to/Y then sed will fail with the same error you report. This happens because the sed command actually run is the same as if you had typed
sed -e s/X//path/to/Y
, which contains an invalid s instruction. Probably the best alternative would be to escape the replacement-string characters that otherwise would be significant to sed. You can do that by interposing a shell function:
escape_replacement() {
# replace all \ characters in the first argument with double backslashes.
# Note that in order to do that here, we need to escape them from the shell
local temp=${1//\\/\\\\}
# Replace all & characters with \&
temp=${temp//&/\\&}
# Replace all / characters with \/, and write the result to standard out.
# Use printf instead of echo to avoid edge cases in which the value to print
# is interpreted to be or start with an option.
printf -- "%s" "${temp//\//\\/}"
}
Then the script would use it like this:
find /my/path/here -type f -name '*.js' -exec sed -i \
-e 's/#{DEMO_STRING_1}/'"$(escape_replacement "$sample1")"'/g' \
...
Note that you probably also want to use a shebang line that explicitly specifies a shell that supports substitution references (${parameter/pattern/replacement}), because these are not required by POSIX, and you might run into a system where /bin/sh is a shell that does not support them. If you're willing to rely on Bash then that should be reflected in your shebang line. Alternatively, you could prepare a version of the escape_replacement function that does not rely on substitution references.
If you use perl - you don't need to escape anything.
With your shell variable exported you can access it via $ENV{name} inside perl.
examples:
samples=(
https://www.example.com
'/sample string/'
12345-abcd-54321-efgh
'base64 string'
$'multi\nline'
)
for sample in "${samples[#]}"
do
echo '---'
export sample
echo 'A B #{DEMO_STRING_1} C' |
perl -pe 's/#{DEMO_STRING_1}/$ENV{sample}/g'
done
echo '---'
Output:
---
A B https://www.example.com C
---
A B /sample string/ C
---
A B 12345-abcd-54321-efgh C
---
A B base64 string C
---
A B multi
line C
---
To add the -i option you can: perl -pi -e 's///'
Is it possible to use basename and readlink in a one line? Something like:
ln -s /usr/local/src symlink
echo `basename <(readlink -f "./symlink")`
except that script above prints 63 instead of src.
Use the command substitution instead of the process substitution:
echo "$(basename "$(readlink -f "./symlink")")"
or, if that's your complete line, echo is redundant:
basename "$(readlink -f "./symlink")"
Multiple $(..) command substitutions can be nested without any escaping or quoting needed (unlike with the old-style backquote version). Also note that if the substitution appears within double quotes, word splitting and filename expansion are not performed on the results.
To clarify the difference: when you say <(cmd), the cmd is executed and the result is made available in a file, handle to which is returned, something like /dev/fd/63. Since the basename acts on a filename given, not its contents, it returns 63.
Unlike process substitution, the $(cmd) will execute the cmd and return the result of command (its standard output). You can then store it in a variable, like res=$(cmd), or reuse it in-place, like cmd "$(cmd)".
I'm trying to make a little script as a wrapper around this command:
$ egrep target /usr/lusers/me/test/*test.txt
/usr/lusers/me/test/1test.txt:target
That directory has files called 1test.txt and 2test.txt, one of which contains some text I want to find.
Here is my whole script, called mygrep.sh:
set -v
set -x
egrep "$1" '/usr/lusers/me/test/*test.txt'
Here's the output:
$ ./mygrep.sh target
set -x
egrep "$1" '/usr/lusers/me/test/*test.txt'
++ egrep targ '/usr/lusers/me/test/*test.txt'
egrep: /usr/lusers/me/test/*test.txt: No such file or directory
Note the 's around the file path in the set -x output, and that the command fails.
Now compare this variation of the script:
set -v
set -x
egrep "$1" '/usr/lusers/me/test/1test.txt'
Note that the only difference is the asterisk vs the literal file name.
Output:
$ ./mygrep.sh target
set -x
egrep "$1" '/usr/lusers/me/test/1test.txt'
++ egrep target /usr/lusers/me/test/1test.txt
target
No single quotes after expansion, and the command works.
So why are those single quotes added when there's an asterisk, and why is the command failing in that case?
The output resulting from set -x is for debugging purposes. No quotes are added to the argument; they are just for display purposes.
The correct command is egrep "$1" /usr/lusers/me/test/*.test.txt, because the shell must expand the pattern (if possible) before passing the results to egrep. You don't have an actual file named *.test.txt.
The globbing character must be outside (single or double) quotes, as quotes disable globbing.
Use this instead :
egrep "$1" '/usr/lusers/me/test/'*'test.txt'
Or this :
egrep "$1" "/usr/lusers/me/test/"*"test.txt"
Or, since there is nothing inside this specific pattern that would cause word splitting to occur (but that would not be a generally safe way to do it if the path is not known safe in advance) :
egrep "$1" /usr/lusers/me/test/*test.txt
I have just recently got back into learning bash. Currently working on a project of mine and when using sed I've run into an issue, I've tried looking around the web for help but haven't had any joy. I suspect as I may not be using the correct terminology so I can't find what I'm looking for. ANYHOW.
So in my script I'm trying to assign the output of date to a variable. Here's the line from my script.
origdate=$(date)
When I call it the output looks like this:
Wed Oct 5 19:40:45 BST 2016
Part of my script then generates a file and writes information to it, part of which I am trying to use sed to find lines and replace parts of it. This is the first I've been playing around with sed, I've used it successfully so far for my needs. However I'm getting stuck when I try this:
sed -i '/origdate=empty/c\'$origdate'' $sd/pingcheck-email-$job.txt
When I run the script and it gets to this line, this is the error I'm getting:
sed: can't read Oct: No such file or directory
sed: can't read 5: No such file or directory
sed: can't read 19:52:56: No such file or directory
sed: can't read BST: No such file or directory
sed: can't read 2016: No such file or directory
I suspect it's something to do with the spaces in the date (variable), my question is: how can I work around this? Can I get sed to 'ignore' the spaces? or should I just use cut to cut the field for the date, and set that to a variable and the same thing again to set the time to another variable?
Even if someone could kindly point me in the right direction that'd be great!
Thanks in advance!
double quote the variable
sed -i '/origdate=empty/c\'"$origdate"'' $sd/pingcheck-email-$job.txt
or alternatively, the whole script
sed -i "/origdate=empty/c\$origdate" $sd/pingcheck-email-$job.txt
The problem is not with sed but rather with how bash word splits on your date given your command.
Bash
In bash, word splitting is performed on the command line so that text is broken up into a list of arguments. To illustrate, I'm going to run a simple script that outputs the first argument only.
bash -c 'echo $1' ignored_0 foo bar
Think of bash -c 'echo $1' ignored_0 as the command (sed in your case) and foo bar as the arguments. In this case, foo bar is split into two arguments, foo and bar.
To pass foo bar in as the first parameter, you need to have the text in either single or double quotes. See the GNU manual on quoting.
bash -c 'echo $1' ignored_0 'foo bar'
bash -c 'echo $1' ignored_0 "foo bar"
Parameter expansion does not occur when the variable is inside a single quote.
var="foo bar"
bash -c 'echo $1' ignored_0 '$var'
bash -c 'echo $1' ignored_0 "$var"
NOTE: In the command `bash -c 'echo $1', I do not want $1 to expand before being passed as an argument to bash because that's part of the code I want to execute.
Parameter expansion occurs when variables are outside of quotes, but word splitting will apply after the parameter is expanded. From the bash man page in the Word Splitting section:
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
From the GNU bash manual on Word Splitting:
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
var="foo bar"
bash -c 'echo $1' ignored_0 $var
The last step in Shell Expansions in Quote Removal where unquoted quote characters are removed before being passed to commands. The following command shows that ''"" has no effect on the arguments passed.
bash -c 'echo $1' ignored_0 foo''""
Application
In your example, the trailing '' after $origdate is extraneous. The important part is that $origdate is not quoted so word splitting applies to the expanded variable.
When -e is not passed to the sed command, sed expects the expression to be in one argument, or word from bash. When you run your command, your expression is /origdate=empty/c\Wed and the rest of the date is considered to be files for the expression to be applied to.
The simple fix is to put double quotes around the string for which you want to prevent word splitting. I've modified the command so that anyone can run this example without having the files on their system.
In this example, the \ must be escaped so that it is not considered an escape character for $.
echo "origdate=empty" | sed "/origdate=empty/c\\$origdate"
You can also change the type of quotes you are using without affecting word splitting like so.
echo "origdate=empty" | sed '/origdate=empty/c\'"$origdate"
You need escape by double slash
\ / \%
I created a bash script to thumbnail all images in a tree. It is the following:
#!/bin/bash
find -path "thumbnails/" -prune -or -iname "*.jpg" -exec \
bash -c 'convert "$0" -resize 256x256 thumbnails/`sha512sum "$0" | awk "{ print \\$1 }"`.jpg' {} \;
# ^^
In the awk command, there is a double \\. (I've marked it with ^^ on the commented line, but you'll probably need to scroll →) Why do I need two backslashes here? I need one to prevent the shell from attempting to expand $1, but otherwise, we are working within just a single set of single-quotes, which shouldn't be messing with the number of slashes. Yet, with just one backslash, awk { print } gets executed, which isn't correct.
Why the \\?
Why do I need two backslashes here? I need one to prevent the shell from attempting to expand $1, but otherwise, we are working within just a single set of single-quotes, which shouldn't be messing with the number of slashes.
There are actually two shells here which do all the usual variable/path substitions/expansions, one is
bash -c
the other is the backtick operator:
`command`
You need another backlash to prevent expanding of $1.
You want to run
convert "$0" -resize 256x256 thumbnails/`sha512sum "$0" | awk "{ print \$1 }"`.jpg {}
through a bash -c for each file. So when you add the command as an argument of bash, you wrap it with single quote ''. In which case you need to escape the backslash with one more backslash. That is
bash -c 'convert "$0" -resize 256x256 thumbnails/`sha512sum "$0" | awk "{ print \\$1 }"`.jpg ' {}
A double backslash ensures that a single backslash will be part of what is printed, in other words if $1 holds the value "xyz" the script will print \xyz