How can I pipe the stdout of multiple commands to a single command?
Example 1: combine and sort the output of all three echo commands:
echo zzz; echo aaa; echo kkk
desired output:
aaa
kkk
zzz
Example 2: rewrite the following so that all the commands are in a single command-line using pipes, without redirects to a temp file:
setopt > /tmp/foo; unsetopt >> /tmp/foo; set >> /tmp/foo; sort /tmp/foo
Use parentheses ()'s to combine the commands into a single process, which will concatenate the stdout of each of them.
Example 1 (note that $ is the shell prompt):
$ (echo zzz; echo aaa; echo kkk) | sort
aaa
kkk
zzz
Example 2:
(setopt; unsetopt; set) | sort
You can use {} for this and eliminate the need for a sub-shell as in (list), like so:
{ echo zzz; echo aaa; echo kkk; } | sort
We do need a whitespace character after { and before }. We also need the last ; when the sequence is written on a single line.
We could also write it on multiple lines without the need for any ;:
Example 1:
{
echo zzz
echo aaa
echo kkk
} | sort
Example 2:
{
setopt
unsetopt
set
} | sort
In Windows it would be as follow: (echo zzz & echo aaa & echo kkk) | sort
Or if it is inside a batch file it can be mono line (like sample) as well as multiline:
(
echo zzz
echo aaa
echo kkk
) | sort
Note: The original post does not mention it is only for Linux, so I added the solution for Windows command line... it is very useful when working with VHD/VHDX with diskpart inside scripts (echo diskpart_command) instead of the echo on the same, but let there the echo, there is also another way without echos and with > redirector, but it is very prone to errors and much more complex to write (why use a complicated prone to errors way if exists a simple way that allways work well)... also remember %d% gives you the actual path (very useful for not hardcoding the path of VHD/VHDX files).
Related
I have a bash script which generates a motd. The problem is depending on some terminal settings which I am not sure about the color will extend to the end of the line. Othertimes it doesn't:
e.g.
v.s.
IIRC one is just the normal gnome-terminal and the other is my tmux term. So my question is how can I get this to extend to 80 character (or really to the terminal width). Of course I can pad to 80 chars but that really doesn't solve the problem.
Here is a snip of my code which generates the motd:
TC_RESET="^[[0m"
TC_SKY="^[[0;37;44m"
TC_GRD="^[[0;30;42m"
TC_TEXT="^[[38;5;203m"
echo -n "${TC_SKY}
... lots of printing..."
echo -e "\n Welcome to Mokon's Linux! \n"
echo -n "${TC_GRD}"
nodeinfo # Just prints the info seen below...
echo ${TC_RESET}
How can I programmatically from bash change the terminal settings or something change the color to the end of the line?
Maybe use the Escape sequence to clear-to-EOL
For some reason (on my MacOS terminal!) I only needed specify this sequence and then it worked for all the lines but for completeness I list it for all
TC_RESET=$'\x1B[0m'
TC_SKY=$'\x1B[0;37;44m'
TC_GRD=$'\x1B[0;30;42m'
TC_TEXT=$'\x1B[38;5;203m'
CLREOL=$'\x1B[K'
echo -n "${TC_SKY}${CLREOL}"
echo -e "\n ABC${CLREOL}\n"
echo -e "\n DEFG${CLREOL}\n"
echo -n "${TC_GRD}"
echo -e "\n ABC${CLREOL}\n"
echo -e "\n DEFG${CLREOL}\n"
echo ${TC_RESET}
Padding filter
Unfortunely, you have to pad each line with exact number of spaces for changing the color of the whole line's background.
As you're speaking about bash, my solution will use bashisms (Won't work under other shell, or older version of bash).
syntax printf -v VAR FORM ARGS assign to varianble VAR then result of sprintf FORM ARGS. That's bashism, under other kind of shell, you have to replace this line by TC_SPC=$(printf "%${COLUMNS}s" '')
You may try this:
... lots of printing..."
echo -e "\n Welcome to Mokon's Linux! \n"
echo -n "${TC_GRD}"
printf -v TC_SPC "%${COLUMNS}s" ''
nodeinfo |
sed "s/$/$TC_SPC/;s/^\\(.\\{${COLUMNS}\\}\\) */\\1/" # Just prints the info seen below...
echo ${TC_RESET}
Maybe you have to ensure that $COLUMNS is correctly setted:
COLUMNS=$(tput cols)
As you could see, only the result of command filtered by sed is fully colored.
you may
use same filter many times:
cmd1 | sed '...'
cmd2 | sed '...'
or group your commands to use only one filter:
( cmd1 ; cmd 2 ) | sed '...'
But there is an issue in case you try to filter ouptut that contain formatting escapes:
(
echo $'\e[33;44;1mYellow text on blue background';
seq 1 6;
echo $'\e[0m'
) | sed "
s/$/$TC_SPC/;
s/^\\(.\\{${COLUMNS}\\}\\) */\\1/"
Il the lines you have to pad to contain escapes, you have to isolate thems:
(
echo $'\e[33;44;1mYellow text on blue background';
seq 1 6;
echo $'\e[0m'
) | sed "
s/\$/$TC_SPC/;
s/^\\(\\(\\o33\\[[0-9;]*[a-zA-Z]\\)*\\)\\([^\o033]\\{${COLUMNS}\\}\\) */\\1\\3/
"
And finally to be able to fill terminate very long lines:
(
echo $'\e[33;44;1mYellow text on blue background';
seq 1 6;
echo "This is a very very long long looooooooooong line that contain\
more characters than the line could hold...";
echo $'\e[0m';
) | sed "
s/\$/$TC_SPC/;
s/^\\(\\(\\o33\\[[0-9;]*[a-zA-Z]\\)*\\)\\(\\([^\o033]\\{${COLUMNS}\\}\\)*\\) */\\1\\3/"
Nota: This only work if formating escapes are located at begin of line.
Try with this:
echo -e '\E[33;44m'"yellow text on blue background"; tput sgr0
How to I concatenate stdin to a string, like this?
echo "input" | COMMAND "string"
and get
inputstring
A bit hacky, but this might be the shortest way to do what you asked in the question (use a pipe to accept stdout from echo "input" as stdin to another process / command:
echo "input" | awk '{print $1"string"}'
Output:
inputstring
What task are you exactly trying to accomplish? More context can get you more direction on a better solution.
Update - responding to comment:
#NoamRoss
The more idiomatic way of doing what you want is then:
echo 'http://dx.doi.org/'"$(pbpaste)"
The $(...) syntax is called command substitution. In short, it executes the commands enclosed in a new subshell, and substitutes the its stdout output to where the $(...) was invoked in the parent shell. So you would get, in effect:
echo 'http://dx.doi.org/'"rsif.2012.0125"
use cat - to read from stdin, and put it in $() to throw away the trailing newline
echo input | COMMAND "$(cat -)string"
However why don't you drop the pipe and grab the output of the left side in a command substitution:
COMMAND "$(echo input)string"
I'm often using pipes, so this tends to be an easy way to prefix and suffix stdin:
echo -n "my standard in" | cat <(echo -n "prefix... ") - <(echo " ...suffix")
prefix... my standard in ...suffix
There are some ways of accomplish this, i personally think the best is:
echo input | while read line; do echo $line string; done
Another can be by substituting "$" (end of line character) with "string" in a sed command:
echo input | sed "s/$/ string/g"
Why i prefer the former? Because it concatenates a string to stdin instantly, for example with the following command:
(echo input_one ;sleep 5; echo input_two ) | while read line; do echo $line string; done
you get immediatly the first output:
input_one string
and then after 5 seconds you get the other echo:
input_two string
On the other hand using "sed" first it performs all the content of the parenthesis and then it gives it to "sed", so the command
(echo input_one ;sleep 5; echo input_two ) | sed "s/$/ string/g"
will output both the lines
input_one string
input_two string
after 5 seconds.
This can be very useful in cases you are performing calls to functions which takes a long time to complete and want to be continuously updated about the output of the function.
You can do it with sed:
seq 5 | sed '$a\6'
seq 5 | sed '$ s/.*/\0 6/'
In your example:
echo input | sed 's/.*/\0string/'
I know this is a few years late, but you can accomplish this with the xargs -J option:
echo "input" | xargs -J "%" echo "%" "string"
And since it is xargs, you can do this on multiple lines of a file at once. If the file 'names' has three lines, like:
Adam
Bob
Charlie
You could do:
cat names | xargs -n 1 -J "%" echo "I like" "%" "because he is nice"
Also works:
seq -w 0 100 | xargs -I {} echo "string "{}
Will generate strings like:
string 000
string 001
string 002
string 003
string 004
...
The command you posted would take the string "input" use it as COMMAND's stdin stream, which would not produce the results you are looking for unless COMMAND first printed out the contents of its stdin and then printed out its command line arguments.
It seems like what you want to do is more close to command substitution.
http://www.gnu.org/software/bash/manual/html_node/Command-Substitution.html#Command-Substitution
With command substitution you can have a commandline like this:
echo input `COMMAND "string"`
This will first evaluate COMMAND with "string" as input, and then expand the results of that commands execution onto a line, replacing what's between the ‘`’ characters.
cat will be my choice: ls | cat - <(echo new line)
With perl
echo "input" | perl -ne 'print "prefix $_"'
Output:
prefix input
A solution using sd (basically a modern sed; much easier to use IMO):
# replace '$' (end of string marker) with 'Ipsum'
# the `e` flag disables multi-line matching (treats all lines as one)
$ echo "Lorem" | sd --flags e '$' 'Ipsum'
Lorem
Ipsum#no new line here
You might observe that Ipsum appears on a new line, and the output is missing a \n. The reason is echo's output ends in a \n, and you didn't tell sd to add a new \n. sd is technically correct because it's doing exactly what you are asking it to do and nothing else.
However this may not be what you want, so instead you can do this:
# replace '\n$' (new line, immediately followed by end of string) by 'Ipsum\n'
# don't forget to re-add the `\n` that you removed (if you want it)
$ echo "Lorem" | sd --flags e '\n$' 'Ipsum\n'
LoremIpsum
If you have a multi-line string, but you want to append to the end of each individual line:
$ ls
foo bar baz
$ ls | sd '\n' '/file\n'
bar/file
baz/file
foo/file
I want to prepend my sql script with "set" statement before running it.
So I echo the "set" instruction, then pipe it to cat. Command cat takes two parameters : STDIN marked as "-" and my sql file, cat joins both of them to one output. Next I pass the result to mysql command to run it as a script.
echo "set #ZERO_PRODUCTS_DISPLAY='$ZERO_PRODUCTS_DISPLAY';" | cat - sql/test_parameter.sql | mysql
p.s. mysql login and password stored in .my.cnf file
I am trying to write a short bash hack that requires piping keystrokes of the F-Keys
basically what I am trying to do is:
(echo "1"; "for x in 1..9; do echo "123<F1>34<F3>"; done; echo "<F1>")|./program
where is the F-key with that #
is this possible? if so can some one point me to the docs, or something
Depending on your terminal, a function-key is just a sequence of characters. You can see what they are with cat:
$ cat
^[OP
^[OQ
^[OR
This is me hitting F1, F2, F3 in sequence. So to echo them into your program, you can just echo those control codes (note the first one there is ctrl-ESC), and you should be all set.
To expand on zigdon's answer, you can use tput to produce the right sequences for your terminal:
f1=$(tput kf1)
f2=$(tput kf2)
# ...
(echo 1; for x in {1..9}; do echo "123${f1}34$f3"; done; echo "$f1") | ./program
The "kf1" and "kf2" names are the terminfo capabilities for the F1 and F2 keys, respectively.
For example look at the following line of bash-code
eval `echo "ls *.jpg"`
It lists all jpgs in the current directory. Now I want it to just print the line to the prompt so I can edit it before executing. (Like key-up does for example)
How do I do that?
The reason for this question comes from a much more usefull alias:
alias ac="history 2 | sed -n '1 s/[ 0-9]*//p' >> ~/.commands; sort -fu ~/.commands > ~/.commandsTmp; mv ~/.commandsTmp ~/.commands"
alias sc='oldIFS=$IFS; IFS=$'\n'; text=(); while read line ; do text=( ${text[#]-} "${line}") ; done < ~/.commands; PS3="Choose command by number: " ; eval `select selection in ${text[#]}; do echo "$selection"; break; done`; IFS=$oldIFS'
alias rc='awk '"'"'{print NR,$0}'"'"' ~/.commands; read -p "Remove number: " number; sed "${number} d" ~/.commands > ~/.commandsTmp; mv ~/.commandsTmp ~/.commands'
Where ac adds or remembers the last typed command, sc shows the available commands and executes them and rc deletes or forgets a command. (You need to touch ~/.commands before it works)
It would be even more usefull if I could edit the output of sc before executing it.
history -s whatever you want
will append "whatever you want" to your bash history. Then a simple up arrow (or !! followed by enter if you have shopt histreedit enabled --- I think that's the option I'm thinking of, not 100% sure), will give you "whatever you want" on the command line, ready to be edited.
Some comments on your aliases:
Simplified quoting:
alias rc='awk "{print NR,\$0}" ~/.commands ...'
No need for tail and you can combine calls to sed:
alias ac="history 2 | sed -n '1 s/[ 0-9]*//p'..."
Simplified eval and no need for $IFS:
alias sc='text=(); while read line ; do text+=("${line}") ; done < ~/.commands; PS3="Choose command by number: " ; select selection in "${text[#]}"; do eval "$selection"; break; done'
#OP, you should really put those commands into subroutines, and when you want to use them, source it. (taken from dennis's answers)
rc(){
awk "{print NR,\$0}" ~/.commands ...
}
ac(){
history 2 | sed -n '1 s/[ 0-9]*//p'...
}
sc(){
text=()
while read line
do
text+=("${line}")
done < ~/.commands
PS3="Choose command by number: "
select selection in "${text[#]}"
do
eval "$selection"
break
done
}
then save it as "library.sh" or something and when you want to use it
$ source /path/to/library.sh
Or
$ . /path/to/library.sh
Maybe you could use preexec.bash?
http://www.twistedmatrix.com/users/glyph/preexec.bash.txt
(On a related note, you can edit the current command line by using ctrl-x-e as well!)
cheers,
tavod
I've got a script 'myscript' that outputs the following:
abc
def
ghi
in another script, I call:
declare RESULT=$(./myscript)
and $RESULT gets the value
abc def ghi
Is there a way to store the result either with the newlines, or with '\n' character so I can output it with 'echo -e'?
Actually, RESULT contains what you want — to demonstrate:
echo "$RESULT"
What you show is what you get from:
echo $RESULT
As noted in the comments, the difference is that (1) the double-quoted version of the variable (echo "$RESULT") preserves internal spacing of the value exactly as it is represented in the variable — newlines, tabs, multiple blanks and all — whereas (2) the unquoted version (echo $RESULT) replaces each sequence of one or more blanks, tabs and newlines with a single space. Thus (1) preserves the shape of the input variable, whereas (2) creates a potentially very long single line of output with 'words' separated by single spaces (where a 'word' is a sequence of non-whitespace characters; there needn't be any alphanumerics in any of the words).
Another pitfall with this is that command substitution — $() — strips trailing newlines. Probably not always important, but if you really want to preserve exactly what was output, you'll have to use another line and some quoting:
RESULTX="$(./myscript; echo x)"
RESULT="${RESULTX%x}"
This is especially important if you want to handle all possible filenames (to avoid undefined behavior like operating on the wrong file).
In case that you're interested in specific lines, use a result-array:
declare RESULT=($(./myscript)) # (..) = array
echo "First line: ${RESULT[0]}"
echo "Second line: ${RESULT[1]}"
echo "N-th line: ${RESULT[N]}"
In addition to the answer given by #l0b0 I just had the situation where I needed to both keep any trailing newlines output by the script and check the script's return code.
And the problem with l0b0's answer is that the 'echo x' was resetting $? back to zero... so I managed to come up with this very cunning solution:
RESULTX="$(./myscript; echo x$?)"
RETURNCODE=${RESULTX##*x}
RESULT="${RESULTX%x*}"
Parsing multiple output
Introduction
So your myscript output 3 lines, could look like:
myscript() { echo $'abc\ndef\nghi'; }
or
myscript() { local i; for i in abc def ghi ;do echo $i; done ;}
Ok this is a function, not a script (no need of path ./), but output is same
myscript
abc
def
ghi
Considering result code
To check for result code, test function will become:
myscript() { local i;for i in abc def ghi ;do echo $i;done;return $((RANDOM%128));}
1. Storing multiple output in one single variable, showing newlines
Your operation is correct:
RESULT=$(myscript)
About result code, you could add:
RCODE=$?
even in same line:
RESULT=$(myscript) RCODE=$?
Then
echo $RESULT $RCODE
abc def ghi 66
echo "$RESULT"
abc
def
ghi
echo ${RESULT#Q}
$'abc\ndef\nghi'
printf '%q\n' "$RESULT"
$'abc\ndef\nghi'
but for showing variable definition, use declare -p:
declare -p RESULT RCODE
declare -- RESULT="abc
def
ghi"
declare -- RCODE="66"
2. Parsing multiple output in array, using mapfile
Storing answer into myvar variable:
mapfile -t myvar < <(myscript)
echo ${myvar[2]}
ghi
Showing $myvar:
declare -p myvar
declare -a myvar=([0]="abc" [1]="def" [2]="ghi")
Considering result code
In case you have to check for result code, you could:
RESULT=$(myscript) RCODE=$?
mapfile -t myvar <<<"$RESULT"
declare -p myvar RCODE
declare -a myvar=([0]="abc" [1]="def" [2]="ghi")
declare -- RCODE="40"
3. Parsing multiple output by consecutives read in command group
{ read firstline; read secondline; read thirdline;} < <(myscript)
echo $secondline
def
Showing variables:
declare -p firstline secondline thirdline
declare -- firstline="abc"
declare -- secondline="def"
declare -- thirdline="ghi"
I often use:
{ read foo;read foo total use free foo ;} < <(df -k /)
Then
declare -p use free total
declare -- use="843476"
declare -- free="582128"
declare -- total="1515376"
Considering result code
Same prepended step:
RESULT=$(myscript) RCODE=$?
{ read firstline; read secondline; read thirdline;} <<<"$RESULT"
declare -p firstline secondline thirdline RCODE
declare -- firstline="abc"
declare -- secondline="def"
declare -- thirdline="ghi"
declare -- RCODE="50"
After trying most of the solutions here, the easiest thing I found was the obvious - using a temp file. I'm not sure what you want to do with your multiple line output, but you can then deal with it line by line using read. About the only thing you can't really do is easily stick it all in the same variable, but for most practical purposes this is way easier to deal with.
./myscript.sh > /tmp/foo
while read line ; do
echo 'whatever you want to do with $line'
done < /tmp/foo
Quick hack to make it do the requested action:
result=""
./myscript.sh > /tmp/foo
while read line ; do
result="$result$line\n"
done < /tmp/foo
echo -e $result
Note this adds an extra line. If you work on it you can code around it, I'm just too lazy.
EDIT: While this case works perfectly well, people reading this should be aware that you can easily squash your stdin inside the while loop, thus giving you a script that will run one line, clear stdin, and exit. Like ssh will do that I think? I just saw it recently, other code examples here: https://unix.stackexchange.com/questions/24260/reading-lines-from-a-file-with-bash-for-vs-while
One more time! This time with a different filehandle (stdin, stdout, stderr are 0-2, so we can use &3 or higher in bash).
result=""
./test>/tmp/foo
while read line <&3; do
result="$result$line\n"
done 3</tmp/foo
echo -e $result
you can also use mktemp, but this is just a quick code example. Usage for mktemp looks like:
filenamevar=`mktemp /tmp/tempXXXXXX`
./test > $filenamevar
Then use $filenamevar like you would the actual name of a file. Probably doesn't need to be explained here but someone complained in the comments.
How about this, it will read each line to a variable and that can be used subsequently !
say myscript output is redirected to a file called myscript_output
awk '{while ( (getline var < "myscript_output") >0){print var;} close ("myscript_output");}'