Now I have this code, which can show the result on my terminal
cat temp | sort -n | uniq -c | awk '{ print $2, $1 }'
but How can I redirect this into another file?
I tried this echo temp | sort -n | uniq -c | awk '{ print $2, $1 }' > temp2, but not working
Thanks
echo temp | sort -n | uniq -c | awk '{ print $2, $1 }' > temp2
You used echo:
cat temp | sort -n | uniq -c | awk '{ print $2, $1 }' > temp2
Also you don't need to use cat:
sort -n temp | uniq -c | awk '{ print $2, $1 }' > temp2
Any command that displays results to your terminal can be redirected to a file by adding to the end of the command a redirect: > out.txt
cat temp | sort -n | uniq -c | awk '{ print $2, $1 }' > temp2
Your second attempt (echo temp ...) simply sent the string "temp" to the sort command, which sent it to the uniq command, and so fort. echo temp is not a valid way to direct results of file "temp". echo prints the actual string "temp" to the terminal and has nothing to do with the file "temp"
[root#www ~]# echo THIS IS FILE CONTENTS > temp
[root#www ~]# cat temp
THIS IS FILE CONTENTS
[root#www ~]# echo temp
temp
[root#www ~]# cat temp > temp2
[root#www ~]# cat temp2
THIS IS FILE CONTENTS
[root#www ~]#
Related
I have a script like this that I would like to seek some suggestions on enhancing it.
cd /home/output/
cat R*op.txt > R.total.op.txt
awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' R.total.op.txt > temp && mv temp R.total.op.txt
cat S*op.txt > S.total.op.txt
awk '{if( (length($8)>9) || ($8 ~ /^AAA/) ) {print $0}}' S.total.op.txt > temp && mv temp S.total.op.txt
cat R.total.op.txt S.total.op.txt | awk '{print $4}' | sort -k1,1 | awk '!x[$1]++' > genes.txt
rm *total.op.txt
head genes.txt
cd /home/output/
for j in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
**for i in `cat genes.txt`; do cat $j'.op.txt' | grep -w $i >> $j'_'$i'_gene.txt'**;done
done
ls -m1 *gene.txt | wc -l
find . -size 0 -delete
ls -m1 *gene.txt | wc -l
rm genes.txt
cd /home/output/
for i in `ls *gene.txt`
do
paste <(awk '{print $4"\t"$8"\t"$9"\t"$13}' $i | awk '!x[$1]++' | awk '{print $1}') <(awk '{print $4"\t"$8"\t"$9}' $i | awk '{if( (length($2)>9) || ($2 ~ /^AAA/) ) {print $0}}' | sort -k2,2 | awk '{ sum += $3 } END { if (NR > 0) print sum / NR }') <(awk '{print $4"\t"$8"\t"$9}' $i| awk '{if( (length($2)>9) || ($2 ~ /^AAA/) ) {print $0}}' | sort -k2,2 | wc -l) <(awk '{print $4"\t"$8"\t"$9"\t"$13}' $i | awk '{if( (length($2)>9) || ($2 ~ /^AAA/) ) {print $0}}' | sort -k2,2 | grep -v ":::" | wc -l) > $i'_stats.txt'
done
rm *gene.txt
cd /home/output/
for j in R1_with-genename R2_with-genename S1_with-genename S2_with-genename
do
cat $j*stats.txt > $j'.final.txt'
done
rm *stats.txt
cd /home/output/
for i in `ls *final.txt`
do
sed "1iGene_Name\tMean1\tCalculated\tbases" $i > temp && mv temp $i
done
head *final.txt
The very first for loop (marked with asterisks) that has cat genes.txt is the grep loop that is taking 3 days to finish. Can someone please advice any enhancements to the command and if this entire script can be made into a single command? Thanks in advance.
Try replacing the nested loops with a single awk.
awk 'FNR = NR {words[$0] = "\\b" $0 "\\b"; next}
{ for (i in words) if ($0 ~ words[i]) {
fn = FILENAME "_" i "_gene.txt";
print >> fn;
close(fn);
}' genes.txt {{R,S}{1,2}_with-genename}.op.txt
I suggest creating a sed script:
# name script
SEDSCRIPT=split.sed
# Make sure it is empty
echo "" > ${SEDSCRIPT}
# Loop through all the words in genes.txt and
# create sed command that will write that line to a file
for word in `cat genes.txt`; do
echo "/${word}/w ${word}.txt" >> ${SEDSCRIPT}
done
basenames="R1_with-genename R2_with-genename S1_with-genename S2_with-genename"
# Loop over input files
for name in "${basenames}"; do
# Run sed script against file
sed -n -f ${SEDSCRIPT} ${name}.op.txt
# Move the temporary files created by sed to their permanent names
for word in `cat genes.txt`; do
mv ${word}.txt ${name}_${word}_gene.txt
done
done
How to use awk command, as I need to add or append a 000 to my below timestamp column. I try to use the below command,
head -n 10000001 ratings.csv | tail -n +2 | awk '{print $1 "000"}' >> ratings_1.csv
but data is not as expected.
$ cat ratings.csv |wc -l
20000264
$ head ratings.csv
userId,movieId,rating,timestamp
1,2,3.5,1112486027
1,29,3.5,1112484676
1,32,3.5,1112484819
1,47,3.5,1112484727
1,50,3.5,1112484580
1,112,3.5,1094785740
1,151,4.0,1094785734
1,223,4.0,1112485573
1,253,4.0,1112484940
My expected output should look like
1,2,3.5,1112486027000
awk '{ if (NR > 1) { $1 = $1 "000" } print }'
Maybe a faster version that wouldn't run the if on every line would be:
awk 'BEGIN { getline; print } { print $0 "000" }'
There is a large log file of 10GB, and formatted as following:
node123`1493000000`POST /api/info`app_id=123&token=123&sign=abc
node456`1493000000`POST /api/info`app_id=456&token=456&sign=abc
node456`1493000000`POST /api/info`token=456&app_id=456&sign=abc
node456`1493000000`POST /api/info`token=456&sign=abc&app_id=456
Now I want to get unique app_ids from the log file. For example, the expected result of the log file above should be:
123
456
I do that with shell script awk -F 'app_id=' '{print $2}' $filename | awk -F '&' '{print $1}' | sort | uniq, and is there a more efficient way?
If your log file's name is log_file.txt,you can use these commands:
grep -Po "(?<=&app_id=)[0-9]+" log_file.txt
awk -F "[&=]" '{print $4}' log_file.txt
Change the logfile name
awk '{print $17" "$18" "$19" "$20}' log.txt |sort -k1|uniq >> z #apache
# filename on line number(0-9) awk result
while read x;
do
echo $x
grep "$x" log.txt | wc -l
done < z
how can i make a loop for users of server using shell script
i wrote this code ..
#!/bin/bash
b=`awk -F: '{ print $1 }' /etc/passwd | sort`
for $b in /home/$b/ /home/$b/ /home/$b/
echo "$b"
done
i want to loop all users and show its
the users like in file
/etc/passwd
like :
root,admin,cpanel,adm,mysql,user1,user2,user3,user4,user5
i want the output :
/home/adm
/home/root
/home/admin
/home/mysql
/home/user1
/home/user2
and thanks
To print /home/ before each user name:
$ awk -F: ' { print "/home/"$1 }' /etc/passwd | sort
/home/avahi
/home/backup
/home/bin
/home/clamav
/home/colord
/home/daemon
[...snip...]
If you want actual home directories, you can find them in field 6:
$ awk -F: ' { print $6 }' /etc/passwd | sort
/dev/null
/home/john1024
/nonexistent
/root
You can also access the sixth field using cut:
cut -d: -f6 /etc/passwd | sort
Printing multiple directories
As per the revised question in the comments:
$ awk -F: ' { p="/home/"$1; printf "%s\n%s\n%s\n",p"/www",p"/ftp",p"/etc" }' /etc/passwd | sort
/home/avahi/etc
/home/avahi/ftp
/home/avahi/www
/home/backup/etc
/home/backup/ftp
/home/backup/www
[...snip...]
I'm using a few commands to cat a few files, like this:
cat somefile | grep example | awk -F '"' '{ print $2 }' | xargs cat
It nearly works, but my issue is that I'd like to add a newline after each file.
Can this be done in a one liner?
(surely I can create a new script or a function that does cat and then echo -n but I was wondering if this could be solved in another way)
cat somefile | grep example | awk -F '"' '{ print $2 }' | while read file; do cat $file; echo ""; done
Using GNU Parallel http://www.gnu.org/software/parallel/ it may be even faster (depending on your system):
cat somefile | grep example | awk -F '"' '{ print $2 }' | parallel "cat {}; echo"
awk -F '"' '/example/{ system("cat " $2 };printf "\n"}' somefile