A bash function, prepend_line, takes two parameters: a string and a fully-qualified path to a file. It's used for logging, inserting the current date/time and the string at the top of the log file.
Standalone use works fine: prepend_line "test string" "$log_file"
How can I get the output from a command, e.g. mv -fv "$fileOne" "$fileTwo" to be used as the first parameter for prepend_line?
I've tried various combinations of piping to xargs, but I don't understand how it works and I'm not convinced it's the best way in any case.
If you really have to:
export -f prepend_line
mv -fv "$fileOne" "$fileTwo" |
xargs -0 bash -c 'prepend_line "$1" "$log_file"' --
The -0 parses the line as beeing zero delimetered. As there should be no zeros in mv -v output, as filenames can't have a zero byte, you will get only a single element. This element/line will be passed as the first argument to the bash subshell.
Tested with:
prepend_line() {
printf "%s\n" "$#" | xxd -p
}
fileOne=$'1\x01\x02\x031234566\n\t\e'
fileTwo=$'2\x01\x02\x031234566\n\t\e \n\n\n'
export -f prepend_line
printf "%s\n" "$fileOne -> $fileTwo" |
xargs -0 bash -c 'prepend_line "$1" "$log_file"' --
The script will output (output from the xxd -p inside prepend_line):
31010203313233343536360a091b202d3e2032010203313233343536360a
091b200a0a0a0a0a0a
Same hex output with some extra newlines and comments:
# first filename $'1\x01\x02\x031234566\n\t\e'
31010203313233343536360a091b
# the string: space + '->' + space
202d3e20
# second filename $'2\x01\x02\x031234566\n\t\e \n\n\n'
32010203313233343536360a091b200a0a0a0a0a0a
If you really have to parse some strange input's you can convert your string to hex with xxd -p. Then, later, convert it back to machine representation with xxd -r -p and streaming right into the output:
prepend_line() {
# some work
# append the output of the "$1" command to the log_file
<<<"$1" xxd -p -r >> "$2"
# some other work
}
prepend_line "$(mv -fv "$fileOne" "$fileTwo" | xxd -p)" "$log_file"
But I doubt you will ever need to handle such cases. Who names filenames using $'\x01' and suffixes with empty newlines 'great_script.sh'$'\n\n'?
Anyway, objectively I would rather see the interface as using a stream:
mv -fv "$fileOne" "$fileTwo" | prepend_line "$log_file"
It needs set -o pipefail to propagate errors correctly. Inside prepend_line I would just redirect the output to the log file or some temporary file, sparing the need of parsing and corner cases.
Related
I'm trying to kick off multiple processes to work through some test suites. In my bash script I have the following
printf "%s\0" "${SUITE_ARRAY[#]}" | xargs -P 2 -0 bash -c 'run_test_suite "$#" ${EXTRA_ARG}'
Below is the defined script, cut down to it's basics.
SUITE_ARRAY will be a list of suites that may have 1 or more, {Suite 1, Suite 2, ..., Suite n}
EXTRA_ARG will be like a specific name to store values in another script
#!/bin/bash
run_test_suite(){
suite=$1
someArg=$2
someSaveDir=someArg"/"suite
# some preprocess work happens here, but isn't relevant to running
runSomeScript.sh suite someSaveDir
}
export -f run_test_suite
SUITES=$1
EXTRA_ARG=$2
IFS=','
SUITECOUNT=0
for csuite in ${SUITES}; do
SUITE_ARRAY[$SUITECOUNT]=$csuite
SUITECOUNT=$(($SUITECOUNT+1))
done
unset IFS
printf "%s\0" "${SUITE_ARRAY[#]}" | xargs -P 2 -0 bash -c 'run_test_suite "$#" ${EXTRA_ARG}'
The issue I'm having is how to get the ${EXTRA_ARG} passed into xargs. From how I've come to understand it, xargs will take whatever is piped into it, so the way I have it doesn't seem correct.
Any suggestions on how to correctly pass the values? Thanks in advance
If you want EXTRA_ARG to be available to the subshell, you need to export it. You can do that either explicitly, with the export keyword, or by putting the var=value assignment in the same simple command as xargs itself:
#!/bin/bash
run_test_suite(){
suite=$1
someArg=$2
someSaveDir=someArg"/"suite
# some preprocess work happens here, but isn't relevant to running
runSomeScript.sh suite someSaveDir
}
export -f run_test_suite
# assuming that the "array" in $1 is comma-separated:
IFS=, read -r -a suite_array <<<"$1"
# see the EXTRA_ARG="$2" just before xargs on the same line; this exports the variable
printf "%s\0" "${suite_array[#]}" | \
EXTRA_ARG="$2" xargs -P 2 -0 bash -c 'run_test_suite "$#" "${EXTRA_ARG}"' _
The _ prevents the first argument passed from xargs to bash from becoming $0, and thus not included in "$#".
Note also that I changed "${suite_array[#]}" to be assigned by splitting $1 on commas. This or something like it (you could use IFS=$'\n' to split on newlines instead, for example) is necessary, as $1 cannot contain a literal array; every shell command-line argument is only a single string.
This is something of a guess:
#!/bin/bash
run_test_suite(){
suite="$1"
someArg="$2"
someSaveDir="${someArg}/${suite}"
# some preprocess work happens here, but isn't relevant to running
runSomeScript.sh "${suite}" "${someSaveDir}"
}
export -f run_test_suite
SUITE_ARRAY="$1"
EXTRA_ARG="$2"
printf "%s\0" "${SUITE_ARRAY[#]}" |
xargs -n 1 -I '{}' -P 2 -0 bash -c 'run_test_suite {} '"${EXTRA_ARG}"
Using GNU Parallel it looks like this:
#!/bin/bash
run_test_suite(){
suite="$1"
someArg="$2"
someSaveDir="$someArg"/"$suite"
# some preprocess work happens here, but isn't relevant to running
echo runSomeScript.sh "$suite" "$someSaveDir"
}
export -f run_test_suite
EXTRA_ARG="$2"
parallel -d, -q run_test_suite {} "$EXTRA_ARG" ::: "$1"
Called as:
mytester 'suite 1,suite 2,suite "three"' 'extra "quoted" args here'
If you have the suites in an array:
parallel -q run_test_suite {} "$EXTRA_ARG" ::: "${SUITE_ARRAY[#]}"
Added bonus: Any output from the jobs will not be mixed, so you will not have to deal with http://mywiki.wooledge.org/BashPitfalls#Using_output_from_xargs_-P
What I want to achieve is the following :
I want the subtitles for my TV Show downloaded automatically.
The script "getSubtitle.sh" is ran as soon as the show is downloaded, but it can happen that no subtitle are released yet.
So what I am doing to counter this :
Creating a file each time "getSubtitle.sh" is ran. It contain the location of the script with its arguments, for example :
/Users/theo/logSubtitle/getSubtitle.sh "The Walking Dead - 5x10 - Them.mp4" "The.Walking.Dead.S05E10.480p.HDTV.H264.mp4" "/Volumes/Window HD/Série/The Walking Dead"
If a subtitle has been found, this file will contain only this line, if no subtitle has been found, this file will have 2 lines (the first one being "no subtitle downloaded", and the second one being the path to the script as explained above)
Now, once I get this, I'm planning to run a cron everyday that will do the following :
Remove all file that have only 1 line (Subtitle found), and execute the script again for the remaining file. Here is the full script :
cd ~/logSubtitle/waiting/
for f in *
do nbligne=$(wc -l $f | cut -c 8)
if [ "$nbligne" = "1" ]
then
rm $f
else
command=$(sed -n "2 p" $f)
sh $command 3>&1 1>&2 2>&3 | grep down > $f ; echo $command >> $f
fi
done
This is unfortunately not working, I have the feeling that the script is not called.
When I replace $command by the line in the text file, it is working.
I am sure that $command match the line because of the "echo $command >> $f" at the end of my script.
So I really don't get what I am missing here, any ideas ?
Thanks.
I'm not sure what you're trying to achieve with the cut -c 8 part in wc -l $f | cut -c 8. cut -c 8 will select the 8th character of the output of wc -l.
A suggestion: to check whether your file contains 1 or two lines (and since you'll need the content of the second line, if any, anyway), use mapfile. This will slurp the file in an array, one line per field. You can use the option -n 2 to read at most 2 lines. This will be much more efficient, safe and nice than your solution:
mapfile -t -n 2 ary < file
Then:
if ((${#ary[#]}==1)); then
printf 'File contains one line only: %s\n' "${ary[0]}"
elif ((${#ary[#]==2)); then
printf 'File contains (at least) two lines:\n'
printf ' %s\n' "${ary[#]}"
else
printf >&2 'Error, no lines found in file\n'
fi
Another suggestion: use more quotes!
With this, a better way to write your script:
#!/bin/bash
dir=$HOME/logSubtitle/waiting/
shopt -s nullglob
for f in "$dir"/*; do
mapfile -t -n 2 ary < "$f"
if ((${#ary[#]}==1)); then
rm -- "$f" || printf >&2 "Error, can't remove file %s\n" "$f"
elif ((${#ary[#]}==2)); then
{ sh -c "${ary[1]}" 3>&1 1>&2 2>&3 | grep down; echo "${ary[1]}"; } > "$f"
else
printf >&2 'Error, file %s contains no lines\n' "$f"
fi
done
After the done keyword you can even add the redirection 2>> logfile to a log file if you wish. Make sure the cron job is run with your user: check crontab -l and, if needed, edit it with crontab -e.
Use eval instead of sh. The reason it works with eval and not sh is due to the number of passes to evaluate variables. sh will treat the sed command as its command to execute while eval will evaluate the sed command first and then execute the result.
Briefly explained.
I want to make UNIX script, which will automatically move my working directory files to newly created directories.
Example: In you dir you got files:
001-file.html,
001-file.rb,
002-file.html,
002-file.rb
And 2 files will be moved to ./NewDir/001-file and another 2 to ./NewDir/002-file
My problem is that after I get correct string from Unix commands I cannot assign it to variable.
Here is my code:
clear
echo "Starting script"
echo "Dir = "$(pwd)
read -p "Please enter count(max '999') of different file groups:" max_i
read -p "Enter new dir name:" outer_dir_name
for ((i=0; i<=$max_i;i++)) do
a1=$(($i/100))
a2=$((($i-$a1*100)/10))
a3=$(($i-($a2*10)-($a1*100)))
inner_dir_name=$((ls *[$a1][$a2][$a3]* 2>/dev/null | head -n 1 | cut -f1 -d"."))
echo $inner_dir_name
echo "--------------"
done
One pair of round parentheses is enough for command substitution.
inner_dir_name=$(ls *[$a1][$a2][$a3]* 2>/dev/null | head -n 1 | cut -f1 -d".")
It looks like you're going about the operation the hard way. I would probably do something like this, assuming that there are no spaces in the file names:
ls | sed 's/\..*$//' | sort -u |
while read prefix
do
mkdir -p $outer_dir_name/$prefix
mv $prefix.* $outer_dir_name/$prefix
done
The ls could be made more precise with:
ls [0-9][0-9][0-9]-file.*
If I was worried about blanks and other odd-ball characters in the file names, I'd have to use something more careful:
for file in [0-9][0-9][0-9]-file.*
do
prefix=${file%%.*}
[ -d "$outer_dir_name/$prefix" ] || mkdir -p "$outer_dir_name/$prefix"
mv "$file" "$outer_dir_name/$prefix"
done
This executes more mv commands, in general.
Im currently using 3 different commands to achieve my goal of downloading a zip, extracting it, converting the txt file to utf8 and then converting the csv to json!
First I have:
wget https://www.example.com/example.zip -O temp.zip; unzip -o temp.zip; rm temp.zip
Which is good, but the problem to start with is how do I rename the file that is extracted so its the same every time for the next processes as it can be a different name within the zip every day? Next I run this script depending on the filename that converts the ISO to utf8:
sh dir_iconv.sh example1.txt ISO8859-1 UTF-8
Which is this script:
#!/bin/bash
ICONVBIN='/usr/bin/iconv' # path to iconv binary
if [ $# -lt 3 ]
then
echo "$0 dir from_charset to_charset"
exit
fi
for f in $1/*
do
if test -f $f
then
echo -e "\nConverting $f"
/bin/mv $f $f.old
$ICONVBIN -f $2 -t $3 $f.old > $f
rm -f $f.old
else
echo -e "\nSkipping $f - not a regular file";
fi
done
And then finally I run a ruby script csv2json - https://github.com/darwin/csv2json - that is called as follows (pipe delimited) to give me a json output:
csv2json -s '|' example1.txt > example1.json
Is there a simple way to roll this into one command or script that can be called?
Pipe all your commands one after another and, if necessary, throw them in a shell script file.
wget -qO- https://www.example.com/example.zip | unzip | iconv -f ISO8859-1 -t UTF-8 | csv2json > example.json
This is a follow-up to this question's answer.
How can I modify the code so that the annoying CRLF of a DOS created file can be stripped away before being passed to xargs?
Example file 'arglist.dos'.
# cat > arglist.unix
src/file1 dst/file1
src/file2 dst/file2
src/file3 dst/file3
^c
# sed 's/$/\r/' arglist.unix > arglist.dos
The unix variant of the file works with this:
$ xargs -n2 < arglist.unix echo cp
cp src/file1 dst/file1
cp src/file2 dst/file2
cp src/file3 dst/file3
For my own education, how can I change it to accept either the 'arglist.unix' or 'arglist.dos' files on the same command line?
cat arglist.dos | tr -d "\r" | xargs -n2 echo cp
gives you the same result as
cat arglist.unix | tr -d "\r" | xargs -n2 echo cp
so it works on both files.
tr -d "\r" removes all the CR characters
Use d2u to remove the CR before passing the file to xargs.