I have the following two scripts:
#script1.sh:
#!/bin/bash
this_chunk=(1 2 3 4)
printf "%s\n" "${this_chunk[#]}" | ./script2.sh
#script2.sh:
#!/bin/bash
while read -r arr
do
echo "--$arr"
done
When I execute script1.sh, the output is as expected:
--1
--2
--3
--4
which shows that I was able to pipe the elements of the array this_chunk as arguments to script2.sh. However, if I change the line calling script2.sh to
printf "%s\n" "${this_chunk[#]}" | xargs ./script2.sh
there is no output. My question is, how to pass the array this_chunk using xargs, rather than simple piping? The reason is that I will have to deal with large arrays and thus long argument lists which will be a problem with piping.
Edit:
Based on the answers and comments, this is the correct way to do it:
#script1.sh
#!/bin/bash
this_chunk=(1 2 3 4)
printf "%s\0" "${this_chunk[#]}" | xargs -0 ./script2.sh
#script2.sh
#!/bin/bash
for i in "${#}"; do
echo $i
done
how to pass the array this_chunk using xargs
Note that xargs by default interprets ' " and \ sequences. To disable the interpretation, either preprocess the data, or better use GNU xargs with -d '\n' option. -d option is not part of POSIX xargs.
printf "%s\n" "${this_chunk[#]}" | xargs -d '\n' ./script2.sh
That said, with GNU xargs prefer zero terminated streams, to preserve newlines:
printf "%s\0" "${this_chunk[#]}" | xargs -0 ./script2.sh
Your script ./script2.sh ignores command line arguments, and your xargs spawns the process with standard input closed. Because the input is closed, read -r arr fails, so your scripts does not print anything, as expected. (Note that in POSIX xargs, when the spawned process tries to read from stdin, the result is unspecified.)
Related
I have a little command line utility rjp2tif that extracts radiometric data from a jpeg file into a tiff file. I was hoping to be able to pipe the filepath to ImageJ on the command line and have ImageJ open the tiff file. To this end, rjp2tif writes the filepath of the tiff file to standard output. I tried the following in bash:
$ rjp2tif /path/to/rjpeg | open -a imagej
and
$ rjp2tif /path/to/rjpeg | open -a imagej -f
The first opens ImageJ but doesn't open the file.
The second opens ImageJ with a text window with the filepath in it.
This is on macOS Monterey, but I don't think that matters.
Anyone tried to do this and been successful? TIA.
Assuming the rjp2tif command returns a file-path in standard output, and you want to pass this output as a regular CLI argument to another command, you may be interested in the xargs command. But note that in the general case, you may hit some issue if the file-path contains spaces or so:
Read space, tab, newline and end-of-file delimited arguments from standard input and execute the specified utility with them as arguments.
The arguments are typically a long list of filenames (generated by ls or find, for example) that get passed to xargs via a pipe.
So in this case, assuming each file-path takes only one line (which is obviously the case if there's only one line overall), you can use the following NUL-based tip relying on the tr command.
Here is the command you'd obtain:
rjp2tif /path/to/rjpeg | tr '\n' '\0' | xargs -0 open -a imagej
Note: I have a GNU/Linux OS, so can you please confirm it does work under macOS?
FTR, below is a comprehensive shell code allowing one to test two different modes of xargs: generating one command per line-argument (-n1), or a single command with all line-arguments in one go:
$ printf 'one \ntwo\nthree and four' | tr '\n' '\0' | xargs -0 -n1 \
bash -c 'printf "Run "; for a; do printf "\"$a\" "; done; echo' bash
Run "one "
Run "two"
Run "three and four"
$ printf 'one \ntwo\nthree and four' | tr '\n' '\0' | xargs -0 \
bash -c 'printf "Run "; for a; do printf "\"$a\" "; done; echo' bash
Run "one " "two" "three and four"
######################################
# or alternatively (with no for loop):
######################################
$ printf 'one \ntwo\nthree and four' | tr '\n' '\0' | xargs -0 -n1 \
bash -c 'printf "Run "; printf "\"%s\" " "$#"; echo' bash
Run "one "
Run "two"
Run "three and four"
$ printf 'one \ntwo\nthree and four' | tr '\n' '\0' | xargs -0 \
bash -c 'printf "Run "; printf "\"%s\" " "$#"; echo' bash
Run "one " "two" "three and four"
I am trying to use basename utility in xargs piped from printf as below:
printf "%s" "$ACTUAL_FILES" | xargs -d ' ' -i printf "%s\n" "$(basename {})"
Here $ACTUAL_FILES is an array of absolute file paths, each delimited with a space.
With the above snippet I am trying to print filename without path in each line. But the output I am getting is same as in $ACTUAL_FILES with each element in new line.
I know that we can achieve this with bash sub shell and echo with xargs, but I was informed to use printf with xargs.
How can I use basename or any other utility to get the filename.
You need to strip the path after processing xargs (I write your var in lowercase):
printf "%s" "${actual_files}" | xargs -d ' ' -i printf "%s\n" "{}" | sed 's#.*/##'
Processing can be easier when you start with replacing spaces by newlines.
tr ' ' '\n' <<< "${actual_files}"| sed 's#.*/##'
You can avoid tr with
grep -Eo "[^/]*( |$)" <<< "${actual_files}"
xargs sends your arguments directly to the command. There is no shell intervention anymore when xargs makes the arguments and sends them to the command. Using xargs you cannot make use of a subshell call ($(..)). You can either preparse them before sending them to xargs, or you can let xargs make a shell instead in which you can make use of all the shell features.
printf "%s" "$ACTUAL_FILES" | xargs -d ' ' -i bash -c 'printf "%s\n" "$(basename {})"'
If it is possible for you to use, GNU parallel comes with much more features in building a command.
printf "%s" "$ACTUAL_FILES" | parallel -q printf "%s\n" "{/}"
In here {/} automatically is the basename of the input arguments and -q preserves the quoting used.
This question already has an answer here:
POSIX shell equivalent to <()
(1 answer)
Closed 3 years ago.
I want to create a script to read a .txt file. This is my code:
while IFS= read -r lines
do
echo "$lines"
done < <(tail -n +2 filename.txt)
I tried a lot of things like:
<<(tail -n +2 in.txt)
< < (tail -n +2 in.txt)
< (tail -n +2 in.txt)
<(tail -n +2 in.txt)
(tail -n +2 in.txt)
I expected to print me from the second line but instead I get an error:
Syntax error: redirection unexpected
If you just want to ignore the first line, there's no good reason to use tail at all!
{
read -r first_line
while IFS= read -r line; do
printf '%s\n' "$line"
done
} <filename.txt
Using read to consume the first line leaves the original file pointer intact, so following code can read directly from the file, instead of reading from a FIFO attached to the output of the tail program; it's thus much lower-overhead.
If you did want to use tail, for the specific case raised, you don't need to use a process substitution (<(...)), but can simply pipe into your while loop. Note that this has a serious side effect, insofar as any variables you set in the loop will no longer be available after it exits; this is documented (in a cross-shell manner) in BashFAQ #24.
tail -n +2 filename.txt | while IFS= read -r line
do
printf '%s\n' "$line"
done
As it says in this answer
POSIX shell equivalent to <()
you could use named pipes to simulate process substitution in
POSIX. Your script would look like that:
#!/usr/bin/env sh
mkfifo foo.fifo
tail -n +2 filename.txt >foo.fifo &
while IFS= read -r lines
do
echo "$lines"
done < foo.fifo
rm foo.fifo
for i in $( find . -name 'x.txt' ); do; if [ grep 'vvvv' ];
then; grep 'vvvv' -A 2 $i | grep -v vvvv | grep -v '-' >> y.csv; else
grep 0 $i >> y.csv; fi; done
What might be wrong with this?
Thanks!
A ; is not permitted after do.
This is automatically detected by http://shellcheck.net/
That said, what you probably want is something more like:
while IFS= read -r -d '' i; do
if grep -q -e vvvv -- "$i"; then
grep -e 'vvvv' -A 2 -- "$i" | egrep -v -e '(vvvv|-)'
else
grep 0 -- "$i"
fi
done < <(find . -name 'x.txt' -print0) >y.csv
Note:
Using find -print0 and IFS= read -r -d '' ensures that all possible filenames (including filenames containing spaces, newlines, etc) can be handled correctly. See BashFAQ #1 for more background on this idiom.
if grep ... should be used if you want if to check the output of grep. Making it if [ grep ... ] means you're passing grep as an argument to the test command, not running it as a command itself.
We open y.csv only once for the entire loop, rather than re-opening the file over and over, only to write a single line (or short number of lines) and close it.
The argument -- should be used to separate options from positional arguments if you don't control those positional arguments.
When - is passed to grep as a string to search for, it should be preceded by -e. That said, in the present case, we can combine both grep -v invocations and avoid the need altogether.
Expansions should always be quoted. That is, "$i", not $i. Otherwise, the values are split on whitespace, and each piece generated is individually evaluated as a glob, preventing correct handling of filenames modified by either of these operations.
Why isn't this bash array populating? I believe I've done them like this in the past. Echoing ${#XECOMMAND[#]} shows no data..
DIR=$1
TEMPFILE=/tmp/dir.tmp
ls -l $DIR | tail -n +2 | sed 's/\s\+/ /g' | cut -d" " -f5,9 > $TEMPFILE
i=0
cat $TEMPFILE | while read line ;do
if [[ $(echo $line | cut -d" " -f1) == 0 ]]; then
XECOMMAND[$i]="$(echo "$line" | cut -d" " -f2)"
(( i++ ))
fi
done
When you run the while loop like
somecommand | while read ...
then the while loop is executed in sub-shell, i.e. a different process than the main script. Thus, all variable assignments that happen in the loop, will not be reflected in the main process. The workaround is to use input redirection and/or command substitution, so that the loop executes in the current process. For example if you want to read from a file you do
while read ....
do
# do stuff
done < "$filename"
or if you wan't the output of a process you can do
while read ....
do
# do stuff
done < <(some command)
Finally, in bash 4.2 and above, you can set shopt -s lastpipe, which causes the last command in the pipeline to be executed in the current process.
I think you're trying to construct an array consisting of the names of all zero-length files and directories in $DIR. If so, you can do it like this:
mapfile -t ZERO_LENGTH < <(find "$DIR" -maxdepth 1 -size 0)
(Add -type f to the find command if you're only interested in regular files.)
This sort of solution is almost always better than trying to parse ls output.
The use of process substitution (< <(...)) rather than piping (... |) is important, because it means that the shell variable will be set in the current shell, not in an ephimeral subshell.