In a Bash loop, I have a mix of commands that append directly to a file, and commands that output to stdout, and then the whole loop is appended to the same file, like this:
for i in {1..3}; do
echo foo >> mylog
echo bar
done >> mylog
The result is that the lines in the mylog file are interleaved, as if the output from the echo bar is appended to mylog in real time:
foo
bar
foo
bar
foo
bar
However, if I pipe the output of the loop through any command (I use the trivial cat command here for simplicity, but it could be any other command where I take each input line and output a processed line), and then pipe it to the file, like this:
for i in {1..3}; do
echo foo >> mylog
echo bar
done | cat >> mylog
the output is not interleaved, with all the bars printed after all the foos, as if it waited until all the output was done from the cat before appending the lines to the file:
foo
foo
foo
bar
bar
bar
My questions are:
Why does this happen?
How can I modify my second code (assuming I still want to have each line processed through a custom command) so that the output is interleaved?
Here is what happens:
loop repeat 3 times:
echo foo >> mylog: Open file mylog, append foo, close file mylog.
echo bar: Append bar to the already opened stdout file handle.
whole loop as a single block:
done |: redirect the whole stdout for the loop to the pipe which has a small buffer for until the receiving process is ready to listen and read stdin data from the pipe.
cat >> mylog: The sub-process running the cat command, open file mylog, append what has been read from stdin, close file mylog.
Technically it is:
Repeat 3 times:
Inefficiently append foo to mylog.
Append bar to buffer.
Content of mylog at this stage:
foo
foo
foo
Then:
Read buffer content and append it all to mylog
Content of mylog at this stage:
foo
foo
foo
bar
bar
bar
If more content was generated in the loop, it would probably block or interleave content once the cat sub-process starts appending concurrently to the file.
Related
The use case is, in my case, CSS file concatenation, before it gets minimized. To concat two CSS files:
cat 1.css 2.css > out.css
To add some text at one single position, I can do
cat 1.css <<SOMESTUFF 2.css > out.css
This will end in the middle.
SOMESTUFF
To add STDOUT from one other program:
sed 's/foo/bar/g' 3.css | cat 1.css - 2.css > out.css
So far so good. But I regularly come in situations, where I need to mix several strings, files and even program output together, like copyright headers, files preprocessed by sed(1) and so on. I'd like to concatenate them together in as little steps and temporary files as possible, while having the freedom of choosing the order.
In short, I'm looking for a way to do this in as little steps as possible in Bash:
command [string|file|output]+ > concatenated
# note the plus ;-) --------^
(Basically, having a cat to handle multiple STDINs would be sufficient, I guess, like
<(echo "FOO") <(sed ...) <(echo "BAR") cat 1.css -echo1- -sed- 2.css -echo2-
But I fail to see, how I can access those.)
This works:
cat 1.css <(echo "FOO") <(sed ...) 2.css <(echo "BAR")
You can do:
echo "$(command 1)" "$(command 2)" ... "$(command n)" > outputFile
You can add all the commands in a subshell, which is redirected to a file:
(
cat 1.css
echo "FOO"
sed ...
echo BAR
cat 2.css
) > output
You can also append to a file with >>. For example:
cat 1.css > output
echo "FOO" >> output
sed ... >> output
echo "BAR" >> output
cat 2.css >> output
(This potentially opens and closes the file repeatedly)
Have this input:
bar foo
foo ABC/DEF
BAR ABC
ABC foo DEF
foo bar
on the above I need do 4 (sequential) actions:
select only lines containing "foo" (lowercase)
on the selected lines, remove everything but UPPERCASE letters
delete empty lines (if some is created by the previous action)
and on the remaining from the above - enclose every char with [x]
I'm able to solve the above, but need two sed invocations piped together. Script:
#!/bin/bash
data() {
cat <<EOF
bar foo
foo ABC/DEF
BAR ABC
ABC foo DEF
foo bar
EOF
}
echo "Result OK"
data | sed -n '/foo/s/[^A-Z]//gp' | sed '/^\s*$/d;s/./[&]/g'
# in the above it is solved using 2 sed invocations
# trying to solve it using only one invocation,
# but the following doesn't do what i need.. :( :(
echo "Variant 2 - trying to use only ONE invocation of sed"
data | sed -n '/foo/s/[^A-Z]//g;/^\s*$/d;s/./[&]/gp'
output from the above:
Result OK
[A][B][C][D][E][F]
[A][B][C][D][E][F]
Variant 2 - trying to use only ONE invocation of sed
[A][B][C][D][E][F]
[B][A][R][ ][A][B][C]
[A][B][C][D][E][F]
The variant 2 should be also only
[A][B][C][D][E][F]
[A][B][C][D][E][F]
It is possible to solve the above using only by one sed invocation?
sed -n '/foo/{s/[^A-Z]//g;/^$/d;s/./[&]/g;p;}' inputfile
Output:
[A][B][C][D][E][F]
[A][B][C][D][E][F]
Alternative sed approach:
sed '/foo/!d;s/[^A-Z]//g;/./!d;s/./[&]/g' file
The output:
[A][B][C][D][E][F]
[A][B][C][D][E][F]
/foo/!d - deletes all lines that don't contain foo
/./!d - deletes all empty lines
How can I define a function bar in a Bash script such that echo foo | bar in that script will read an input from the script's stdin and not the pipe? In other words, if bar is:
function bar(){
read ZOO
}
I want it to wait for my input rather than setting ZOO to "foo"
The idea of a pipe is to connect stdout of the process on the left side of the pipe with stdin of the process on the right side of the pipe. So foo is in this case piped into stdin of the function bar().
If you want to read explicitly from the current terminal, then pass the special device /dev/tty to stdin of read:
function bar() {
read ZOO < /dev/tty
}
The use case is, in my case, CSS file concatenation, before it gets minimized. To concat two CSS files:
cat 1.css 2.css > out.css
To add some text at one single position, I can do
cat 1.css <<SOMESTUFF 2.css > out.css
This will end in the middle.
SOMESTUFF
To add STDOUT from one other program:
sed 's/foo/bar/g' 3.css | cat 1.css - 2.css > out.css
So far so good. But I regularly come in situations, where I need to mix several strings, files and even program output together, like copyright headers, files preprocessed by sed(1) and so on. I'd like to concatenate them together in as little steps and temporary files as possible, while having the freedom of choosing the order.
In short, I'm looking for a way to do this in as little steps as possible in Bash:
command [string|file|output]+ > concatenated
# note the plus ;-) --------^
(Basically, having a cat to handle multiple STDINs would be sufficient, I guess, like
<(echo "FOO") <(sed ...) <(echo "BAR") cat 1.css -echo1- -sed- 2.css -echo2-
But I fail to see, how I can access those.)
This works:
cat 1.css <(echo "FOO") <(sed ...) 2.css <(echo "BAR")
You can do:
echo "$(command 1)" "$(command 2)" ... "$(command n)" > outputFile
You can add all the commands in a subshell, which is redirected to a file:
(
cat 1.css
echo "FOO"
sed ...
echo BAR
cat 2.css
) > output
You can also append to a file with >>. For example:
cat 1.css > output
echo "FOO" >> output
sed ... >> output
echo "BAR" >> output
cat 2.css >> output
(This potentially opens and closes the file repeatedly)
Say I have file A, in middle of which have a tag string "#INSERT_HERE#". I want to put the whole content of file B to that position of file A. I tried using pipe to concatenate those contents, but I wonder if there is more advanced one-line script to handle it.
$ cat file
one
two
#INSERT_HERE#
three
four
$ cat file_to_insert
foo bar
bar foo
$ awk '/#INSERT_HERE#/{while((getline line<"file_to_insert")>0){ print line };next }1 ' file
one
two
foo bar
bar foo
three
four
cat file | while read line; do if [ "$line" = "#INSERT_HERE#" ]; then cat file_to_insert; else echo $line; fi; done
Use sed's r command:
$ cat foo
one
two
#INSERT_HERE#
three
four
$ cat bar
foo bar
bar foo
$ sed '/#INSERT_HERE#/{ r bar
> d
> }' foo
one
two
foo bar
bar foo
three
four