When I want to redirect output to a file, I usually do this:
$ echo 'a' > b
$ cat b
a
However, I've seen people use tee instead of redirecting directly to a file. I'm wondering what the difference is. What I mean in this pattern:
$ echo 'a' | tee c
a
$ cat c
a
It doesn't seem to be doing anything differently than a simple redirect. I know they are conceptually not the same thing, but I'm wondering why people would use one over the other.
In simple word
echo 'a' > b , it will write "a" to file b.
#echo 'a' > b
#cat b
#a
echo 'a' | tee b , it will write "a" to file b and will display output(echo output) in the terminal.
#echo 'a' | tee b
#a
#cat b
#a
Using tee let's you split the output. You can either view it (by directing stdout to the tty you are looking at) or pass it on to further processing. It is handy for keeping track of intermediate stages of a pipeline.
Related
I'm currently working on a larger script, but I can't get this single function to work properly.
for f in app1/*; do
sort -u $f "temp.txt" > "temp.txt"
done
Directory app1 has a few text files in it. What I am trying to do is take each file one by one and merge it with temp.txt to build an updated sorted temp.txt file without duplicates.
Example:
temp.txt starts as an empty file.
app1/1.txt
a
b
c
d
app1/2.txt
d
e
f
End result at the end of the loop
temp.txt
a
b
c
d
e
f
The problem I'm running into is that the temp.txt file only has the data from the last file passed through the loop.
If all the files combined are not large, you can sort them at once:
sort -u *.txt > all
If the files are large and sorting must be done at one file level, you can do
sort -u $f all -o all
You have 2 problems.
You are using the outputfile as input (as stated by others) and you overwrite the outputfile in each loop. See the next incorrect fix
for f in app1/*; do
sort -u $f "temp.txt" > "temp1.txt"
done
This code will reset the outputfile for each f. Remember: When you redirect to a file in a loop, always append (>> "temp1.txt").
The problem seems to be fixed with the ugly loop:
for f in app1/*; do
cp temp.txt fix1.txt
sort -u $f "fix1.txt" > "temp.txt"
done
The way you should do it is writing to output outside the loop. Since you start with an empty temp.txt you have
for f in app1/*; do
sort -u $f
done > "fix2.txt"
sort -u "fix2.txt" > "temp.txt"
Or is #Andrey right and can you use
for f in app1/*; do
sort -u $f
done | sort -u > "temp.txt"
or
sort -u app1/* > "temp.txt"
You may want to append - using double angle-bracket:
sort -u $f "temp.txt" >> "temp.txt"
This may be another way to do it:
reut#reut-work-room:~/srt$ cat 1.txt
a
b
c
d
reut#reut-work-room:~/srt$ cat 2.txt
d
e
f
reut#reut-work-room:~/srt$ sort -u *.txt > out.txt
reut#reut-work-room:~/srt$ cat out.txt
a
b
c
d
e
f
The shell process redirections before launching the command. So
sort foo bar > bar
will first truncate "bar" to zero bytes. Then the sort command has the "normal" foo file and a now empty bar file to work with.
ref: http://www.gnu.org/software/bash/manual/bashref.html#Redirections
I have two programs that produce data on stdout, and I'd like to paste their output together. I can successfully do this like so:
paste <(./prog1) <(./prog2)
But I find that this method will print all lines from both inputs,
and what I really want is to stop paste after either input program is finished.
So if ./prog1 produces the output:
a
b
c
But ./prog2 produces:
Hello
World
I would expect the output:
a Hello
b World
Also note that one of the input programs may actually produce infinite output, and I want to be able to handle that case as well. For example, if my inputs are yes and ./prog2, I should get:
y Hello
y World
Use join instead, with a variation on the Schwartzian transform:
numbered () {
nl -s- -ba -nrz
}
join -j 1 <(prog1 | numbered) <(prog2 | numbered) | sed 's/^[^-]*-//'
Piping to nl numbers each line, and join -1 1 will join corresponding lines with the same number. The extra lines in the longer file will have no join partner and be omitted. Once the join is complete, pipe through sed to remove the line numbers.
Here's one solution:
while IFS= read -r -u7 a && IFS= read -r -u8 b; do echo "$a $b"; done 7<$file1 8<$file2
This has the slightly annoying effect of ignoring the last line of an input file if it is not terminated with a newline (but such a file is not a valid text file).
You can wrap this in a function, of course:
paste_short() {
(
while IFS= read -r -u7 a && IFS= read -r -u8 b; do
echo "$a $b"
done
) 7<"$1" 8<"$2"
}
Consider using awk:
awk 'FNR==NR{a[++i]=$0;next} FNR>i{exit}
{print a[FNR], $0}' <(printf "hello\nworld\n") <(printf "a\nb\nc\n")
hello a
world b
Keep the longer output producing program as your 2nd input.
I have written a script to automate a routine, but can't understand the difference between the 2 blocks below. The first works and the second doesn't.
This works:
echo "$(pull_data)" > data.csv
cat data.csv | while read a b c d; do
This doesn't work:
cat "$(pull_data)" | while read a b c d; do
Why is that?
cat concatenates and outputs files - I think you want echo in your second statement:
echo "$(pull_data)" | while read a b c d; do
cat is used to work with files. You don't have a file in this case. If you don't need to store your data in data.csv, you should be able to pipe it directly to the loop:
echo "$(pull_data)" | while read a b c d; do
I am trying to write a program that will print out every line from a file with another line of that file added at the end, basically creating pairs from a portion of each line. If the line is the same, it will do nothing. Also, it must avoid repeating the same pairs. A B is the same as B A
In short
FileInput:
otherstuff A
otherstuff B
otherstuff C
otherstuff D
Output:
A B
A C
A D
B C
B D
C D
I was trying to do this with a BASH script, but was having trouble because I could not get my nested while loops to work. It would read the first line, compare it to each other line, and then stop (Basically only outputting the first 3 lines in the example output above, the outer while loop only ran once).
I also suspect I might be able to do this using MATLAB, so suggestions using that are also welcome.
Here is the bash script that I have thus far. As I said, it is no printing out correctly for me, as the outer loop only runs once.
#READS IN file from terminal
FILE1=$1
#START count at 0
count0=
exec 3<&0
exec 0< $FILE1
while read LINEa; do
while read LINEb; do
eventIDa=$(echo $LINEa | cut -c20-23)
eventIDb=$(echo $LINEb | cut -c20-23)
echo $eventIDa $eventIDb
done
done
Using bash:
#!/bin/bash
[ -f "$1" ] || { echo >&2 "File not found"; exit 1; }
mapfile -t lines < <(cut -c20-23 <"$1" | sort | uniq)
for i in ${!lines[#]}; do
elem1=${lines[$i]}
unset lines[$i]
for elem2 in "${lines[#]}"; do
echo "$elem1" "$elem2"
done
done
This will read a file given as a parameter on the command line, sort and filter out duplicates, and output all combinations. You can modify the parameter to cut to adjust to your particular input file.
Due to the particular way you seem to indent to use cut, your input example above won't work. Instead, use something with the correct line length, such as:
123456789012345678 A
123456789012345678 B
123456789012345678 C
123456789012345678 D
Assuming the otherstuff is not relevant (otherwise you can of course add it later) this should do the trick in Matlab:
combnk({'A' 'B' 'C' 'D'},2)
How can I pipe the stdout of multiple commands to a single command?
Example 1: combine and sort the output of all three echo commands:
echo zzz; echo aaa; echo kkk
desired output:
aaa
kkk
zzz
Example 2: rewrite the following so that all the commands are in a single command-line using pipes, without redirects to a temp file:
setopt > /tmp/foo; unsetopt >> /tmp/foo; set >> /tmp/foo; sort /tmp/foo
Use parentheses ()'s to combine the commands into a single process, which will concatenate the stdout of each of them.
Example 1 (note that $ is the shell prompt):
$ (echo zzz; echo aaa; echo kkk) | sort
aaa
kkk
zzz
Example 2:
(setopt; unsetopt; set) | sort
You can use {} for this and eliminate the need for a sub-shell as in (list), like so:
{ echo zzz; echo aaa; echo kkk; } | sort
We do need a whitespace character after { and before }. We also need the last ; when the sequence is written on a single line.
We could also write it on multiple lines without the need for any ;:
Example 1:
{
echo zzz
echo aaa
echo kkk
} | sort
Example 2:
{
setopt
unsetopt
set
} | sort
In Windows it would be as follow: (echo zzz & echo aaa & echo kkk) | sort
Or if it is inside a batch file it can be mono line (like sample) as well as multiline:
(
echo zzz
echo aaa
echo kkk
) | sort
Note: The original post does not mention it is only for Linux, so I added the solution for Windows command line... it is very useful when working with VHD/VHDX with diskpart inside scripts (echo diskpart_command) instead of the echo on the same, but let there the echo, there is also another way without echos and with > redirector, but it is very prone to errors and much more complex to write (why use a complicated prone to errors way if exists a simple way that allways work well)... also remember %d% gives you the actual path (very useful for not hardcoding the path of VHD/VHDX files).