How can I pass my variables inside an sh -c command?
~$ echo $from
/tmp/from
~$ echo $to
/tmp/to
~$ find /tmp/from -type f | xargs -r -d '\n' -n1 sh -c 'f="$0" ; a=`echo "$f" | sed "s|^$from/|$to/|"` ; echo "${f}\t${a}"'
/tmp/from/lulu /tmp/from/lulu
/tmp/from/toto /tmp/from/toto
It looks like my $from and $to are not getting inside ... I tried double quotes and escaping inside quotes, but without much success so far.
Thanks!
P.S.: input files list is normally stored in a file, find is only here for the sake of example.
Related
I'm trying to pipe a series of manipulations into an xargs call that I can use to swap the first value with the second using the sed command (sed is optional if there's a better way).
Basically I'm grabbing method signature in camel case and appending a prefix while trying to retain camel case.
So it should take...
originalMethodSignature
and replace it with...
givenOriginalMethodSignature
Because I'm using a series of pipes to find and modify the text, I was hoping to use multiple params with xargs, but it seems that most of the questions involving that use sh -c which would be fine but in order for the sed command to be interactive on a Mac terminal I need to use single quotes inside the shell calls' single quotes.
Something like this, where the double quotes preserve the functionality of the single quotes in the sed command...
echo "somePrecondition SomePrecondition" | xargs -L1 sh -c 'find ~/Documents/BDD/Definitions/ -type f -name "Given$1.swift" -exec sed -i "''" "'"s/ $0/ given$1/g"'" {} +'
assuming there's a file called "~/Documents/BDD/Definitions/GivenSomePrecondition.swift" with below code...
protocol GivenSomePrecondition { }
extension GivenSomePrecondition {
func somePrecondition() {
print("empty")
}
}
The first awk is going through a list of swift protocols that start with the Given keyword (e.g. GivenSomePrecondition), then they strip it down to "somePrecondition SomePrecondition" before hitting the final pipe. My intent is that the final xargs call can replace $0 with given$1 interactively (overwriting the file).
The original command in context...
awk '{ if ($1 ~ /^Given/) print $0;}' ~/Documents/Sell/SellUITests/BDDLite/Definitions/HasStepDefinitions.swift \
| tr -d "\t" \
| tr -d " " \
| tr -d "," \
| sort -u \
| xargs -I string sh -c 'str=$(echo string); echo ${str#"Given"}' \
| awk '{ print tolower(substr($1,1,1)) substr($1, 2)" "$1 }' \
| xargs -L1 sh -c '
find ~/Documents/Sell/SellUITests/BDDLite/Definitions/ \
-type f \
-name "Given$1.swift" \
-exec sed -i '' "'"s/ $0/ given$1/g"'" {} +'
You don't need xargs or sh -c, and taking them out reduces the amount of work involved.
echo "somePrecondition SomePrecondition" |
while read -r source replace; do
find ~/Documents/BDD/Definitions/ -type f -name "Given${replace}.swift" -print0 |
while IFS= read -r -d '' filename; do
sed -i '' -e "s/ ${source}/ given${replace}/g" "$filename"
done
done
However, to answer your questions as opposed to sidestepping it, you can write functions that use any kind of quotes you want, and export them into your subshell, either with export -f yourFunction in a parent process or by putting "$(declare -f yourFunction)" inside the string passed after bash -c (assuming that bash is the same shell used in the parent process defining those functions).
#!/usr/bin/env bash
replaceOne() {
local source replace
source=$1; shift || return
replace=$1; shift || return
sed -i '' -e "s/ $1/ given$2/g" "$#"
}
# substitute replaceOne into a new copy of bash, no matter what kind of quotes it has
bash -c "$(declare -f replaceOne)"'; replaceOne "$#"'
I have a problem with a code similar to the following:
function echotee() { echo $1 | tee -a ${FILE}; }
export -f echotee
find . -delete -exec sh -c 'echotee "Deleting: {}"' \;
The function echotee usually works as expected. However, during the -exec it does not. Indeed, it just prints on the terminal, omitting tee.
Hoping the question is not too trivial, thanks in advance.
Why don't you just use this:
find . -delete -exec sh -c 'echo "Deleting: $1" | tee -a "$2"' _ {} "${FILE}" \;
No need to define and call a function.
You mentioned in a comment that you want to use echotee as a central point to print and log information. Have you considered a setup like this instead:
#!/usr/bin/env bash
# Send all script output to console and logfile
LOGFILE="..."
exec > >(tee -ia "${LOGFILE}") 2>&1
find . -delete -printf "Deleting: %f\n"
or this:
#!/usr/bin/env bash
# Set up fd 3 to send output to console and logfile on demand
LOGFILE="..."
exec 3> >(tee -ia "${LOGFILE}")
find . -delete -printf "Deleting: %f\n" 1>&3 2>&1
Use name() instead of function name().
You did not set nor export FILE variable.
sh does not support exporting functions. It's a feature of bash, you have to call bash.
sh -c ' .... "{}"' will break on filenames containing " character. Put it as positional argument and use $1.
$1 and $FILE expansions are not quoted and are subject to word splitting and filename expansion.
echo $1 will break on filenames like -e. Prefer printf.
Check your scripts with shellcheck - it will catch many such mistakes.
I think you meant to:
FILE=/tmp/log.txt
echotee() { printf "%s\n" "$1" | tee -a "$FILE"; }
export -f echotee
export FILE
find . -exec bash -c 'echotee "Deleting: $1"' -- {} \;
But the version from Shawn with -printf "Deleting: %p\n" | tee "$FILE" looks just nicer.
I think spawning tee and pipe will be slower then, I think doing like so could be a bit faster:
echotee() { printf "%s\n" "$1" >> "$FILE"; printf "%s\n" "$1"; }
or like:
exec 10>>"$FILE"
echotee() { printf "%s\n" "$1" >&10; printf "%s\n" "$1"; }
You could remove the pipe either way, just:
echotee() { tee -a "$FILE" <<<"$1"; }
I need to escape single quotes in a variable.
ssh_command 'file=$(hostname)_server-setup_$(date +%Y-%m-%d).tar.gz && cd /var && tar -zcvf $file ini | wc -l | xargs printf \'Num files: %d, File: $file\''
I can surround the variable with double quotes but then the internal variables will be evaluated when the variable is declared, and that's not what I want.
ssh_command "file=$(hostname)_server-setup_$(date +%Y-%m-%d).tar.gz && cd /var && tar -zcvf $file ini | wc -l | xargs printf 'Num files: %d, File: $file'"
update
Have now come up with this, but then $file is just printed as $file
ssh_command (){
ssh root#host $1
}
ssh_command 'file=$(hostname)_server-setup_$(date +%Y-%m-%d).tar.gz && cd /var && tar -zcvf $file ini | wc -l | xargs printf '"'"'Num files: %d, File: $file'"'"
output
Num files: 61, File: $file
When sending over a complicated command over SSH (using quotes, dollar signs, semi-colons), then I prefer to base64 encode/decode it. Here I've made base64_ssh.bash:
#!/bin/bash
script64=$(cat script.txt | base64 -w 0)
ssh 127.0.0.1 "echo $script64 | base64 -d | bash"
In the example above, simply put the command you would like to run on the remote server in script.txt, and then run the bash script.
This does require one extra file, but not having to escape quotes or other special characters makes this a better solution in my opinion.
This will also work with creating functions too.
The way it works, is it converts the command into a base64 encoded string which has a simpler character set (Wikipedia base64), and these characters will never need to be escaped. Then once it's on the other side, it is decoded and then piped through to the bash interpreter.
To make this work with your example above, put the following into script.txt:
file=$(hostname)_server-setup_$(date +%Y-%m-%d).tar.gz && cd /var && tar -zcvf $file ini | wc -l | xargs printf "Num files: %d, File: $file"
For this not too big command I would not overcomplicate it and stick to the original double quotes, and escape just the $ which should not be expanded locally.
ssh_command "file=\$(hostname)_server-setup_$(date +%Y-%m-%d).tar.gz && cd /var && tar -zcvf \$file ini | wc -l | xargs printf 'Num files: %d, File: \$file'"
I have a function that generates a random file name
#generate random file names
get_rand_filename() {
if [ "$ASCIIONLY" == "1" ]; then
for ((i=0; i<$((MINFILENAMELEN+RANDOM%MAXFILENAMELEN)); i++)) {
printf \\$(printf '%03o' ${AARR[RANDOM%aarrcount]});
}
else
# no need to escape double quotes for filename
cat /dev/urandom | tr -dc '[ -~]' | tr -d '[$></~:`\\]' | head -c$((MINFILENAMELEN+RANDOM%MAXFILENAMELEN)) #| sed 's/\(["]\)/\\\1/g'
fi
printf "%s" $FILEEXT
}
export -f get_rand_filename
When I call it from within another function
cf(){
fD=$1
echo "the target dir recieved is " $fD
CFILE="$(get_rand_filename)"
echo "the file name is "$CFILE
}
export -f cf
when I call
echo "$targetdir" | xargs -0 sh -c 'cf $1' sh
I only get the FILEXT (no random file name)
when I call
cf "$targetdir"
I get a valid result
I need to be able to handle spaces in the $targetdir and file name string.
echo "$targetdir" | xargs -0 sh -c 'cf $1' sh
You should invoke bash rather than sh. Function exporting is a bash feature.
$ foo() { echo bar; }
$ export -f foo
$ sh -c 'foo'
sh: 1: foo: not found
$ bash -c 'foo'
bar
Also, get rid of the -0 option since the input isn't NUL-separated. Use -d'\n' instead. And quote "$1" for robustness.
echo "$targetdir" | xargs -d'\n' bash -c 'cf "$1"' bash
Actually, you could use -0 if you change the input format.
printf '%s\0' "$targetdir" | xargs -0 bash -c 'cf "$1"' bash
For what it's worth, mktemp creates random temporary files, and does it safely. It makes sure the file doesn't already exist and then creates it to prevent anybody else from snatching up the name in the split second between the name being generated and it being returned to the caller.
I'm trying to rename commands in a bash script. If I run for example:
echo /home/scientist/mySalesData/campaignData_1482386214.24417.csv | sed 's/\(.*\)\(_.*\)/mv \"&" \"\1.csv\"/' | bash
It works fine and gives me campaignData.csv in the directory /home/scientist/mySalesData/ .
However, if I put this in a bash script as follows:
for f in /home/scientist/SalesData/*; do
if [ -f "$f" ];
cp "$f" /home/scientist/SalesForce/SalesData/Backups/
echo $f$ | sed 's/\(.*\)\(_.*\)/mv \"&" \"\1.csv\"/' | bash |
fi
done
I get:
mv: cannot stat '/home/scientist/SalesData/campaignData_1482386214.24417.csv$': No such file or directory
Any help would be much appreciated!
cd "$srcdir"
for f in *; do
if [ -f "$f" ]; then
cp "./$f" "$dstdir/${f%_*}.csv"
fi
done
The % is the strip shortest suffix pattern operator.
You have a trailing $ here:
echo $f$
remove that (and quote the expansion):
echo "$f"
You could use here string too:
sed ... <<<"$f"