I would like to get the stdout from a process into another process not using stdin, as that one is used for another purpose.
In short I want to accomplish something like that:
echo "a" >&4
cat | grep -f /dev/fd/4
I got it running using an file as source for file descriptor 4, but that is not what I want:
# Variant 1
cat file | grep -f /dev/fd/4 4<pattern
# Variant 2
exec 4<pattern
cat | grep -f /dev/fd/4
exec 4<&-
My best try is that, but I got the following error message:
# Variant 3
cat | (
echo "a" >&4
grep -f /dev/fd/4
) <&4
Error message:
test.sh: line 5: 4: Bad file descriptor
What is the best way to accomplish that?
You don't need to use multiple streams to do this:
$ printf foo > pattern
$ printf '%s\n' foo bar | grep -f pattern
foo
If instead of a static file you want to use the output of a command as the input to -f you can use a process substitution:
$ printf '%s\n' foo bar | grep -f <(echo foo)
foo
For POSIX shells that lack process substitution, (e.g. dash, ash, yash, etc.).
If the command allows string input, (grep allows it), and the input string containing search targets isn't especially large, (i.e. the string doesn't exceed the length limit for the command line), there's always command substitution:
$ printf '%s\n' foo bar baz | grep $(echo foo)
foo
Or if the input file is multi-line, separating quoted search items with '\n' works the same as grep OR \|:
$ printf '%s\n' foo bar baz | grep "$(printf "%s\n" foo bar)"
foo
bar
Related
I have a file, foo.txt, with 2 lines:
foo bar
hello world
I am trying to output the second line using:
cut -d$'\n' -f2 foo.txt
But it is printing both lines instead.
Is cut able to accept a newline as a delimiter?
Background
cut operates on all lines of the input, so I don't think that a newline can
work as a delimiter.
From cut(1p) of POSIX.1-2017:
NAME
cut - cut out selected fields of each line of a file
SYNOPSIS
cut -b list [-n] [file...]
cut -c list [file...]
cut -f list [-d delim] [-s] [file...]
DESCRIPTION
The cut utility shall cut out bytes (-b option), characters (-c option), or
character-delimited fields ( -f option) from each line in one or more
files, concatenate them, and write them to standard output.
Example:
$ printf 'foo bar1\nfoo bar2\n'
foo bar1
foo bar2
$ printf 'foo bar1\nfoo bar2\n' | cut -f 2 -d ' '
bar1
bar2
Solution
To print only the second line of input, sed can be used:
$ printf 'foo bar1\nfoo bar2\n'
foo bar1
foo bar2
$ printf 'foo bar1\nfoo bar2\n' | sed '2p;d'
foo bar2
It's not very clear what you want, but this is splitting on empty lines and collapsing the others using awk:
awk 'BEGIN {RS=""} {gsub("\n",""); print}'
this might not be a very elegant solution, but i often just use head and tail, since they typically outperform sed or awk in speed
#!/bin/sh
input='test0\ntest1\ntest2\ntest3\ntest4\ntest5'
echo -e "$input" | head --lines 3
# test0
# test1
# test2
echo -e "$input" | tail --lines 3
# test3
# test4
# test5
get_index() {
echo -e "$1" |
head --lines "$(( $2 + 1 ))" |
tail --lines "1"
}
get_index "$input" "0"
# test0
get_index "$input" "4"
# test4
I want to transform a string given in this form:
xyx some commands
into this form:
alias xyz="some commands"
I tried different combinations in the terminal. It seems (i'm not sure) that it worked once, but never when i run this from the script. I've read somewhere that this is a variable problem.
Alias for readability:
alias first="sed 's/\s.*//'"
alias rest="sed 's/\S*\s*//'"
cat f_in | tee -a >(one=$(first)) >(two=$(rest)) | tee >(awk '{print "alias "$1"=\""$2"\""}' > f_out )
I used awk in this way to parse "cat f_in" into "print". It doesn't work. Then, i used "awk -v" but it still doesn't work too. How to redirect variable $one and $two into awk:
{one=$(first) === first | read -r one }?
Is this what you're trying to do:
$ echo 'xyx some commands' |
awk '{var=$1; sub(/^[^[:space:]]+[[:space:]]+/,""); printf "alias %s=\"%s\"\n", var, $0}'
alias xyx="some commands"
$ echo 'xyx some commands' |
sed 's/\([^[:space:]]*\)[[:space:]]*\(.*\)/alias \1="\2"/'
alias xyx="some commands"
Problem
(1) Given a string, I replace spaces with $'\n' using sed:
echo "one two" | sed 's/ /$'"'"'\\n'"'"'/g'
This outputs:
# one$'\n'two
(2) Note that echoing this output of (1):
echo one$'\n'two
results in:
# one
# two
(3) I echo the output of (1) in another way, by piping the output of (1) into xargs echo:
echo "one two" | sed 's/ /$'"'"'\\n'"'"'/g' | xargs echo
But I don't get the same output as (2):
# one$\ntwo
Question
What does xargs do when formatting the input of a string containing $'\n'?
Why is echoing a string with $'\n' not the same as using xargs echo on the same string?
When you write
echo one$'\n'two
at the command line, bash replaces the "$'\n'" with a newline. But when you pass it to xargs no such replacement can happen.
But piping it to xargs will still not do what you want, since by default xargs uses the newline as an argument separator:
$ echo "one two" | tr ' ' '\n' | xargs echo
one two
You must tell xargs to use a different separator, even if it is a bogus one:
$ echo "one two" | tr ' ' '\n' | xargs -0 echo
one
two
Unsure if answering your question, but a trick I've used in the past for similar cases is to use printf instead, which loops over passed arguments in a loop (if not enough % to consume them), e.g.:
$ printf "%s\n" one two
one
two
Use shell own white-space separator if above are in a single string
$ args="one two"
$ printf "%s\n" $args
one
two
Just for completeness, feed to xargs -n1 with some foo scriptlet
$ printf "%s\n" one two |xargs -n1 sh -c 'echo [$(date -R)] foo=$1' --
[Sun, 03 Jun 2018 21:34:17 -0300] foo=one
[Sun, 03 Jun 2018 21:34:17 -0300] foo=two
Is there a difference between command1 | command2 and command2 <(command1)?
For example, git diff | more vs more <(git diff)
My understanding is that both take the stdout of command2 and pipe it to the stdin of command1.
The main difference is that <(...), called "process substitution", is translated by the shell into a filename that is passed as a regular argument to the command; it doesn't send anything to the command's standard input. This means that it can't be used directly with commands such as tr which don't take a filename argument:
$ tr a-z A-Z <(echo hello)
usage: tr [-Ccsu] string1 string2
tr [-Ccu] -d string1
tr [-Ccu] -s string1
tr [-Ccu] -ds string1 string2
However, you can always put another < in front of the <(...) to turn it into an input redirection instead:
$ tr a-z A-Z < <(echo hello)
HELLO
And because it generates a filename, you can use process substitution with commands that take more than one file argument:
$ diff -u <(echo $'foo\nbar\nbaz') <(echo $'foo\nbaz\nzoo')
--- /dev/fd/63 2016-07-15 14:48:52.000000000 -0400
+++ /dev/fd/62 2016-07-15 14:48:52.000000000 -0400
## -1,3 +1,3 ##
foo
-bar
baz
+zoo
The other significant difference is that a pipe creates subshells which can't have side effects in the parent environment:
$ echo hello | read x
$ echo $x
# nothing - x is not set
But with process substitution, only the process inside the parentheses is in a subshell; the surrounding command can still have side effects:
$ read x < <(echo hello)
$ echo $x
hello
Worth mentioning that you can also write into a process with >(...), although there are fewer cases where that's useful:
$ echo hello > >(cat)
hello
a | b takes the stdout output from executable a and feeds it to executable b as b's stdin.
a > b takes the stdout from executable a and redirects/writes it out to file b.
a < b takes the contents of file b and redirects/inputs it to executable a as its stdin
In other words, | pipes output between programs, while < and > pipes files into/out of programs.
Your version with () runs an extra process, while accomplishing essentially the same thing.
I have a command line tool which receives two arguments:
TOOL arg1 -o arg2
I would like to invoke it with the same argument provided it for arg1 and arg2, and to make that easy for me, i thought i would do:
each <arg1_value> | TOOL $1 -o $1
but that doesn't work, $1 is not replaced, but is added once to the end of the commandline.
An explicit example, performing:
cp fileA fileA
returns an error fileA and fileA are identical (not copied)
While performing:
echo fileA | cp $1 $1
returns the following error:
usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file target_file
cp [-R [-H | -L | -P]] [-fi | -n] [-apvX] source_file ... target_directory
any ideas?
If you want to use xargs, the [-I] option may help:
-I replace-str
Replace occurrences of replace-str in the initial-arguments with names read from standard input. Also, unquoted blanks do not terminate input items; instead the separa‐
tor is the newline character. Implies -x and -L 1.
Here is a simple example:
mkdir test && cd test && touch tmp
ls | xargs -I '{}' cp '{}' '{}'
Returns an Error cp: tmp and tmp are the same file
The xargs utility will duplicate its input stream to replace all placeholders in its argument if you use the -I flag:
$ echo hello | xargs -I XXX echo XXX XXX XXX
hello hello hello
The placeholder XXX (may be any string) is replaced with the entire line of input from the input stream to xargs, so if we give it two lines:
$ printf "hello\nworld\n" | xargs -I XXX echo XXX XXX XXX
hello hello hello
world world world
You may use this with your tool:
$ generate_args | xargs -I XXX TOOL XXX -o XXX
Where generate_args is a script, command or shell function that generates arguments for your tool.
The reason
each <arg1_value> | TOOL $1 -o $1
did not work, apart from each not being a command that I recognise, is that $1 expands to the first positional parameter of the current shell or function.
The following would have worked:
set - "arg1_value"
TOOL "$1" -o "$1"
because that sets the value of $1 before calling you tool.
You can re-run a shell to perform variable expansion, with sh -c. The -c takes an argument which is command to run in a shell, performing expansion. Next arguments of sh will be interpreted as $0, $1, and so on, to use in the -c. For example:
sh -c 'echo $1, i repeat: $1' foo bar baz will print execute echo $1, i repeat: $1 with $1 set to bar ($0 is set to foo and $2 to baz), finally printing bar, i repeat: bar
The $1,$2...$N are only visible to bash script to interpret arguments to those scripts and won't work the way you want them to. Piping redirects stdout to stdin and is not what you are looking for either.
If you just want a one-liner, use something like
ARG1=hello && tool $ARG1 $ARG1
Using GNU parallel to use STDIN four times, to print a multiplication table:
seq 5 | parallel 'echo {} \* {} = $(( {} * {} ))'
Output:
1 * 1 = 1
2 * 2 = 4
3 * 3 = 9
4 * 4 = 16
5 * 5 = 25
One could encapsulate the tool using awk:
$ echo arg1 arg2 | awk '{ system("echo TOOL " $1 " -o " $2) }'
TOOL arg1 -o arg2
Remove the echo within the system() call and TOOL should be executed in accordance with requirements:
echo arg1 arg2 | awk '{ system("TOOL " $1 " -o " $2) }'
Double up the data from a pipe, and feed it to a command two at a time, using sed and xargs:
seq 5 | sed p | xargs -L 2 echo
Output:
1 1
2 2
3 3
4 4
5 5