I'm trying to pipe a series of manipulations into an xargs call that I can use to swap the first value with the second using the sed command (sed is optional if there's a better way).
Basically I'm grabbing method signature in camel case and appending a prefix while trying to retain camel case.
So it should take...
originalMethodSignature
and replace it with...
givenOriginalMethodSignature
Because I'm using a series of pipes to find and modify the text, I was hoping to use multiple params with xargs, but it seems that most of the questions involving that use sh -c which would be fine but in order for the sed command to be interactive on a Mac terminal I need to use single quotes inside the shell calls' single quotes.
Something like this, where the double quotes preserve the functionality of the single quotes in the sed command...
echo "somePrecondition SomePrecondition" | xargs -L1 sh -c 'find ~/Documents/BDD/Definitions/ -type f -name "Given$1.swift" -exec sed -i "''" "'"s/ $0/ given$1/g"'" {} +'
assuming there's a file called "~/Documents/BDD/Definitions/GivenSomePrecondition.swift" with below code...
protocol GivenSomePrecondition { }
extension GivenSomePrecondition {
func somePrecondition() {
print("empty")
}
}
The first awk is going through a list of swift protocols that start with the Given keyword (e.g. GivenSomePrecondition), then they strip it down to "somePrecondition SomePrecondition" before hitting the final pipe. My intent is that the final xargs call can replace $0 with given$1 interactively (overwriting the file).
The original command in context...
awk '{ if ($1 ~ /^Given/) print $0;}' ~/Documents/Sell/SellUITests/BDDLite/Definitions/HasStepDefinitions.swift \
| tr -d "\t" \
| tr -d " " \
| tr -d "," \
| sort -u \
| xargs -I string sh -c 'str=$(echo string); echo ${str#"Given"}' \
| awk '{ print tolower(substr($1,1,1)) substr($1, 2)" "$1 }' \
| xargs -L1 sh -c '
find ~/Documents/Sell/SellUITests/BDDLite/Definitions/ \
-type f \
-name "Given$1.swift" \
-exec sed -i '' "'"s/ $0/ given$1/g"'" {} +'
You don't need xargs or sh -c, and taking them out reduces the amount of work involved.
echo "somePrecondition SomePrecondition" |
while read -r source replace; do
find ~/Documents/BDD/Definitions/ -type f -name "Given${replace}.swift" -print0 |
while IFS= read -r -d '' filename; do
sed -i '' -e "s/ ${source}/ given${replace}/g" "$filename"
done
done
However, to answer your questions as opposed to sidestepping it, you can write functions that use any kind of quotes you want, and export them into your subshell, either with export -f yourFunction in a parent process or by putting "$(declare -f yourFunction)" inside the string passed after bash -c (assuming that bash is the same shell used in the parent process defining those functions).
#!/usr/bin/env bash
replaceOne() {
local source replace
source=$1; shift || return
replace=$1; shift || return
sed -i '' -e "s/ $1/ given$2/g" "$#"
}
# substitute replaceOne into a new copy of bash, no matter what kind of quotes it has
bash -c "$(declare -f replaceOne)"'; replaceOne "$#"'
Related
I can use md5 -r foo.txt > md5.txt to create a text file with the md5 of the file followed by a space and then the local path to that file .. but how would I go about getting those two items separated by a TAB character?
For reference and context, the full command I'm using is
find . -type f -exec \
bash -c '
md=$(md5 -r "$0")
siz=$(wc -c <"$0")
echo -e "${md}\t${siz}"
' {} \; \
> listing.txt
Note that the filepath item of md5 output might also contain spaces, like ./path to file/filename, and these should not be converted to tabs.
sed is another option:
find directory/ -type f -exec md5 -r '{}' '+' | sed 's/ /\t/' > listing.txt
This will replace the first space on each line with a tab.
(Note that the file you're redirecting output to should not be in the directory tree being searched by find)
Try the builtin printf and P.E. parameter expansion, to split the md variable.
find . -type f -exec sh -c '
md=$(md5 -r "$0") siz=$(wc -c <"$0")
printf "%s\t%s\t%s\n" "${md%% *}" "${md#*"${md%% *}"}" "${siz}"
' {} \; > listing.txt
Output
d41d8cd98f00b204e9800998ecf8427e ./bar.txt 0
d41d8cd98f00b204e9800998ecf8427e ./foo.txt 0
d41d8cd98f00b204e9800998ecf8427e ./more.txt 0
d41d8cd98f00b204e9800998ecf8427e ./baz.txt 0
314a1673b94e05ed5d9757b6ee33e3b1 ./qux.txt 0
See the online manual for bash ParameExpansion
The local man pages if available. PAGER='less +/^[[:space:]]*parameter\ expansion' man bash
Looks like you are simply left with spaces between the hash and file name that you don't want. A quick pass through awk can clean that up for you. By default input awk delimiter is any amount of white space. Simply running though awk and printing the fields with a new OFS (output field separator) is all you need. In fact, it makes the pass through echo pointless.
time find . -type f -exec bash -c 'md=$(md5 -r "$0"); siz=$(wc -c <"$0"); awk -vOFS="\t" "{print \$1,\$2,\$3}" <<< "${md} ${siz}" ' > listing.txt {} \;
Personally, I would have ran the output of that find command into a while loop. This is basically the same as above, but a little easier to follow.
time find . -type f | \
while read -r file; do
md=$(md5 -r "$file")
siz=$(wc -c < "$file")
awk -vOFS="\t" '{print $1,$2,$3}' <<< "${md} ${siz}"
done > listing.txt
I have a function that generates a random file name
#generate random file names
get_rand_filename() {
if [ "$ASCIIONLY" == "1" ]; then
for ((i=0; i<$((MINFILENAMELEN+RANDOM%MAXFILENAMELEN)); i++)) {
printf \\$(printf '%03o' ${AARR[RANDOM%aarrcount]});
}
else
# no need to escape double quotes for filename
cat /dev/urandom | tr -dc '[ -~]' | tr -d '[$></~:`\\]' | head -c$((MINFILENAMELEN+RANDOM%MAXFILENAMELEN)) #| sed 's/\(["]\)/\\\1/g'
fi
printf "%s" $FILEEXT
}
export -f get_rand_filename
When I call it from within another function
cf(){
fD=$1
echo "the target dir recieved is " $fD
CFILE="$(get_rand_filename)"
echo "the file name is "$CFILE
}
export -f cf
when I call
echo "$targetdir" | xargs -0 sh -c 'cf $1' sh
I only get the FILEXT (no random file name)
when I call
cf "$targetdir"
I get a valid result
I need to be able to handle spaces in the $targetdir and file name string.
echo "$targetdir" | xargs -0 sh -c 'cf $1' sh
You should invoke bash rather than sh. Function exporting is a bash feature.
$ foo() { echo bar; }
$ export -f foo
$ sh -c 'foo'
sh: 1: foo: not found
$ bash -c 'foo'
bar
Also, get rid of the -0 option since the input isn't NUL-separated. Use -d'\n' instead. And quote "$1" for robustness.
echo "$targetdir" | xargs -d'\n' bash -c 'cf "$1"' bash
Actually, you could use -0 if you change the input format.
printf '%s\0' "$targetdir" | xargs -0 bash -c 'cf "$1"' bash
For what it's worth, mktemp creates random temporary files, and does it safely. It makes sure the file doesn't already exist and then creates it to prevent anybody else from snatching up the name in the split second between the name being generated and it being returned to the caller.
I'm getting mad with a script performance.
Basically I have to replace 600 strings in more than 35000 files.
I have got something like this:
patterns=(
oldText1 newText1
oldText2 newText2
oldText3 newText3
)
pattern_count=${#patterns[*]}
files=(`find \. -name '*.js'`);
files_count=${#files[*]}
for ((i=0; i < $pattern_count ; i=i+2)); do
search=${patterns[i]};
replace=${patterns[i+1]};
echo -en "\e[0K\r Status "$proggress"%. Iteration: "$i" of " $pattern_count;
for ((j=0; j < $files_count; j++)); do
command sed -i s#$search#$replace#g ${files[j]};
proggress=$(($i*100/$files_count));
echo -en "\e[0K\r Inside the second loop: " $proggress"%. File: "$j" of "$files_count;
done
proggress=$(($i*100/$pattern_count));
echo -en "\e[0K\r Status "$proggress"%. Iteration: "$i" of " $pattern_count;
done
But this takes tons of minutes. There is another solution? Probably using sed just one time and not in a double loop?
Thanks a lot.
Create a proper sed script:
s/pattern1/replacement1/g
s/pattern2/replacement2/g
...
Run this script with sed -f script.sed file (or in whatever way is required).
You may create that sed script using your array:
printf 's/%s/%s/g\n' "${patterns[#]}" >script.sed
Applying it to the files:
find . -type f -name '*.js' -exec sed -i -f script.sed {} ';'
I don't quite know how GNU sed (which I assume you're using) is handling multiple files when you use -i, but you may also want to try
find . -type f -name '*.js' -exec sed -i -f script.sed {} +
which may potentially be much more efficient (executing as few sed commands as possible). As always, test on data that you can afford to throw away after testing.
For more information about using -exec with find, see https://unix.stackexchange.com/questions/389705
You don't need to run sed multiple times over one file. You can separate sed commands with ';'
You can execute multiple seds in parallel
For example:
patterns=(
oldText1 newText1
oldText2 newText2
oldText3 newText3
)
// construct sed argument such as 's/old/new/g;s/old2/new2/g;...'
sedarg=$(
for ((i = 0; i < ${#patterns[#]}; i += 2)); do
echo -n "s/${patterns[i]}/${patterns[i+1]}/g;"
done
)
// find all files named '*.js' and pass them to args with zero as separator
// xargs will parse them:
// -0 use zero as separator
// --verbose will print the line before execution (ie. sed -i .... file)
// -n1 pass one argument/one line to one sed
// -P8 run 8 seds simulteneusly (experiment with that value, depends on how fast your cpu and harddrive is)
find . -type f -name '*.js' -print0 | xargs -0 --verbose -n1 -P8 sed -i "$sedarg"
If you need the progress bar so much, I guess you can count the lines xargs --verbose returns or better use parallel --bar, see this post.
How can I pass my variables inside an sh -c command?
~$ echo $from
/tmp/from
~$ echo $to
/tmp/to
~$ find /tmp/from -type f | xargs -r -d '\n' -n1 sh -c 'f="$0" ; a=`echo "$f" | sed "s|^$from/|$to/|"` ; echo "${f}\t${a}"'
/tmp/from/lulu /tmp/from/lulu
/tmp/from/toto /tmp/from/toto
It looks like my $from and $to are not getting inside ... I tried double quotes and escaping inside quotes, but without much success so far.
Thanks!
P.S.: input files list is normally stored in a file, find is only here for the sake of example.
I'm wasting so much time right now trying to figure out something so simple....
pseudo code (mixture of several syntax's, sorry):
cmd1 = "find /my/starting/path -type f | grep -v -f /my/exclude/files"
cmd2 = " nl -ba -s' ' "
cmd3 = " | xargs mv -t /move/here/dir "
echo run_command_and_return_output($cmd1$cmd2)
$cmd1$cmd3 # just this now...
# i don't actually want a function... but the name explains what i want to do
function run_command_and_return_output(){ /* magic */ }
this works....
FIND=$(find $LOG_DIR -type f | grep -v -f $EXCLUDE | nl -ba -s' ')
printf "%s\n" "$FIND"
this does not...
NL="nl -ba -s' '"
FIND=$(find $LOG_DIR -type f -mtime +$ARCH_AGE | grep -v -f $EXCLUDE | $NL)
printf "%s\n" "$FIND"
and neither does this...
NL='nl -ba -s'\'' '\'' '
this definitely does work, though:
find /my/starting/path -type f | grep -v -f /my/exclude/files | nl -ba -s' '
or
FIND=$(find $LOG_DIR -type f -mtime +$ARCH_AGE | grep -v -f $EXCLUDE | nl -ba -s' ' )
Short form: Expanding $foo unquoted runs the content through string-splitting and glob expansion, but not syntactical parsing. This means that characters which would do quoting and escaping in a different context aren't honored as syntax, but are only treated as data.
If you want to run a string through syntactical parsing, use eval -- but mind the caveats, which are large and security-impacting.
Much better is to use the right tools for the job -- building individual simple commands (not pipelines!) in shell arrays, and using functions as the composable unit for constructing complex commands. BashFAQ #50 describes these tools -- and goes into in-depth discussion on which of them is appropriate when.
To get a bit more concrete:
nl=( nl -ba -s' ' )
find_output=$(find "$log_dir" -type f -mtime "+$arch_age" | grep -v -f "$exclude" | "${nl[#]}")
printf "%s\n" "$find_output"
...would be correct, since it tracks the simple command nl as an array.