I would like to redirect output to a file how described in this StackOverflow question but with an exception: let's suppose I do not want to store all the lines which start with the character r.
Specifically, I execute:
./command | tee /tmp/output.txt
and I get all the printed lines in the file. How to modify the instruction to allow exception?
This is a case for an output process substitution:
$ printf "%s\n" {o..u} | tee >(grep -v '^r' > outputfile)
o
p
q
r
s
t
u
$ cat outputfile
o
p
q
s
t
u
Related
I am trying to read a csv file using shell script,using the following command.
cat file.csv | while read -r a b c d e f; do echo "$a:$b:$c:$d:$e:$f"; done
When i run this command the first column in the file is not being read properly.
For Ex: If 1 st column contents are
number1,
number2,
number3,
number4,
(so on)
It outputs:
::::er1,
::::er2,
::::er3,
::::er4,
some characters are replaced by ':'
this happens only for the first column contents. Where am i going wrong?
The problem is due to most likely a couple of issues:-
You are reading the file without the IFS=,
Your csv file might likely have carriage returns(\r) which could mangle how read command processes the input stream.
To remove the carriage returns(\r) use tr -d '\r' < oldFile.csv > newFile.csv and in the new file do the parsing as mentioned below.
Without setting the Internal Field Separator (IFS=","), while reading from the input stream read doesn't know where to delimit your words. Add the same in the command as below.
cat file.csv | while IFS="," read -r a b c d e f; do echo "$a:$b:$c:$d:$e:$f"; done
You can see it working as below. I have the contents of the file.csv as follows.
$ cat file.csv
abc,def,ghi,ijk,lmn,opz
1,2,3,4,5,6
$ cat file.csv | while IFS="," read -r a b c d e f; do echo "$a:$b:$c:$d:$e:$f"; done
abc:def:ghi:ijk:lmn:opz
1:2:3:4:5:6
More over using cat and looping it over it is not recommended and bash enthusiasts often call it as UUOC - Useless Use Of Cat
You can avoid this by doing
#!/bin/bash
while IFS="," read -r a b c d e f;
do
echo "$a:$b:$c:$d:$e:$f"
done < file.csv
I'm currently working on a larger script, but I can't get this single function to work properly.
for f in app1/*; do
sort -u $f "temp.txt" > "temp.txt"
done
Directory app1 has a few text files in it. What I am trying to do is take each file one by one and merge it with temp.txt to build an updated sorted temp.txt file without duplicates.
Example:
temp.txt starts as an empty file.
app1/1.txt
a
b
c
d
app1/2.txt
d
e
f
End result at the end of the loop
temp.txt
a
b
c
d
e
f
The problem I'm running into is that the temp.txt file only has the data from the last file passed through the loop.
If all the files combined are not large, you can sort them at once:
sort -u *.txt > all
If the files are large and sorting must be done at one file level, you can do
sort -u $f all -o all
You have 2 problems.
You are using the outputfile as input (as stated by others) and you overwrite the outputfile in each loop. See the next incorrect fix
for f in app1/*; do
sort -u $f "temp.txt" > "temp1.txt"
done
This code will reset the outputfile for each f. Remember: When you redirect to a file in a loop, always append (>> "temp1.txt").
The problem seems to be fixed with the ugly loop:
for f in app1/*; do
cp temp.txt fix1.txt
sort -u $f "fix1.txt" > "temp.txt"
done
The way you should do it is writing to output outside the loop. Since you start with an empty temp.txt you have
for f in app1/*; do
sort -u $f
done > "fix2.txt"
sort -u "fix2.txt" > "temp.txt"
Or is #Andrey right and can you use
for f in app1/*; do
sort -u $f
done | sort -u > "temp.txt"
or
sort -u app1/* > "temp.txt"
You may want to append - using double angle-bracket:
sort -u $f "temp.txt" >> "temp.txt"
This may be another way to do it:
reut#reut-work-room:~/srt$ cat 1.txt
a
b
c
d
reut#reut-work-room:~/srt$ cat 2.txt
d
e
f
reut#reut-work-room:~/srt$ sort -u *.txt > out.txt
reut#reut-work-room:~/srt$ cat out.txt
a
b
c
d
e
f
The shell process redirections before launching the command. So
sort foo bar > bar
will first truncate "bar" to zero bytes. Then the sort command has the "normal" foo file and a now empty bar file to work with.
ref: http://www.gnu.org/software/bash/manual/bashref.html#Redirections
I have two programs that produce data on stdout, and I'd like to paste their output together. I can successfully do this like so:
paste <(./prog1) <(./prog2)
But I find that this method will print all lines from both inputs,
and what I really want is to stop paste after either input program is finished.
So if ./prog1 produces the output:
a
b
c
But ./prog2 produces:
Hello
World
I would expect the output:
a Hello
b World
Also note that one of the input programs may actually produce infinite output, and I want to be able to handle that case as well. For example, if my inputs are yes and ./prog2, I should get:
y Hello
y World
Use join instead, with a variation on the Schwartzian transform:
numbered () {
nl -s- -ba -nrz
}
join -j 1 <(prog1 | numbered) <(prog2 | numbered) | sed 's/^[^-]*-//'
Piping to nl numbers each line, and join -1 1 will join corresponding lines with the same number. The extra lines in the longer file will have no join partner and be omitted. Once the join is complete, pipe through sed to remove the line numbers.
Here's one solution:
while IFS= read -r -u7 a && IFS= read -r -u8 b; do echo "$a $b"; done 7<$file1 8<$file2
This has the slightly annoying effect of ignoring the last line of an input file if it is not terminated with a newline (but such a file is not a valid text file).
You can wrap this in a function, of course:
paste_short() {
(
while IFS= read -r -u7 a && IFS= read -r -u8 b; do
echo "$a $b"
done
) 7<"$1" 8<"$2"
}
Consider using awk:
awk 'FNR==NR{a[++i]=$0;next} FNR>i{exit}
{print a[FNR], $0}' <(printf "hello\nworld\n") <(printf "a\nb\nc\n")
hello a
world b
Keep the longer output producing program as your 2nd input.
When I want to redirect output to a file, I usually do this:
$ echo 'a' > b
$ cat b
a
However, I've seen people use tee instead of redirecting directly to a file. I'm wondering what the difference is. What I mean in this pattern:
$ echo 'a' | tee c
a
$ cat c
a
It doesn't seem to be doing anything differently than a simple redirect. I know they are conceptually not the same thing, but I'm wondering why people would use one over the other.
In simple word
echo 'a' > b , it will write "a" to file b.
#echo 'a' > b
#cat b
#a
echo 'a' | tee b , it will write "a" to file b and will display output(echo output) in the terminal.
#echo 'a' | tee b
#a
#cat b
#a
Using tee let's you split the output. You can either view it (by directing stdout to the tty you are looking at) or pass it on to further processing. It is handy for keeping track of intermediate stages of a pipeline.
I have a command which outputs in this format:
A
B
C
D
E
F
G
I
J
etc
I want the output to be in this format
A B C D E F G I J
I tried using ./script | tr "\n" " " but all it does is remove n from the output
How do I get all the output in one line. (Line wrapped)
Edit: I accidentally put in grep while asking the question. I removed
it. My original question still stands.
The grep is superfluous.
This should work:
./script | tr '\n' ' '
It did for me with a command al that lists its arguments one per line:
$ al A B C D E F G H I J
A
B
C
D
E
F
G
H
I
J
$ al A B C D E F G H I J | tr '\n' ' '
A B C D E F G H I J $
As Jonathan Leffler points out, you don't want the grep. The command you're using:
./script | grep tr "\n" " "
doesn't even invoke the tr command; it should search for the pattern "tr" in files named "\n" and " ". Since that's not the output you reported, I suspect you've mistyped the command you're using.
You can do this:
./script | tr '\n' ' '
but (a) it joins all its input into a single line, and (b) it doesn't append a newline to the end of the line. Typically that means your shell prompt will be printed at the end of the line of output.
If you want everything on one line, you can do this:
./script | tr '\n' ' ' ; echo ''
Or, if you want the output wrapped to a reasonable width:
./script | fmt
The fmt command has a number of options to control things like the maximum line length; read its documentation (man fmt or info fmt) for details.
No need to use other programs, why not use Bash to do the job? (-- added in edit)
line=$(./script.sh)
set -- $line
echo "$*"
The set sets command-line options, and one of the (by default) seperators is a "\n". EDIT: This will overwrite any existing command-line arguments, but good coding practice would suggest that you reassigned these to named variables early in the script.
When we use "$*" (note the quotes) it joins them alll together again using the first character of IFS as the glue. By default that is a space.
tr is an unnecessary child process.
By the way, there is a command called script, so be careful of using that name.
If I'm not mistaken, the echo command will automatically remove the newline chars when its argument is given unquoted:
tmp=$(./script.sh)
echo $tmp
results in
A B C D E F G H I J
whereas
tmp=$(./script.sh)
echo "$tmp"
results in
A
B
C
D
E
F
G
H
I
J
If needed, you can re-assign the output of the echo command to another variable:
tmp=$(./script.sh)
tmp2=$(echo $tmp)
The $tmp2 variable will then contain no newlines.