I have the following file:
file1.txt
geek
for
geeks
I am using the tee command to perform two operations on the output. My question is about the the redirection character after the first tee. I want to get the first column of file1.txt and
write it to file2.txt. When I run the following command, I don't receive an error but it does not give me the first column:
wc -l file1.txt |tee awk '{print $1}' - > file2.txt | sed 's/4/6/g' > file3.txt
However, the following command works as expected. What does the > is doing here?
wc -l file1.txt |tee >(awk '{print $1}' - > file2.txt) | sed 's/4/6/g' > file3.txt
tee awk '{print $1}' - > file2.txt
does:
execute tee with 3 arguments awk and '{print $1}' and -.
tee will create a file named awk, another file named '{print $1}' and yet another file named -.
Then the output of tee will be redirected to file2.txt
tee will duplicate input to those 3 files and will output to file2.txt
Consequently | sed will receive no input, because the output of tee is redirected to the file and the subshell outputs nothing.
tee >(awk '{print $1}' - > file2.txt)
does:
>(...)
Run awk with two arguments '{print $1}' and -
------ '{print $1}' is interpreted as a script
------ - is interpreted as stdin (and could be omitted)
------ then the output of awk of redirected to file2.txt
Then bash creates a fifo or a /dev/fd/something file
Then the output of that file is connected to stdin of awk process
And the >(awk ...) is substituted for the filename of the file, most probably for /dev/fd/something
tee >(...)
executes tee with one argument, like tee /dev/fd/something
The /dev/fd/something is connected to awk process on the other side
So tee writes to /dev/fd/something and awk reads the data from stdin on the other side
the output of tee is redirected to | sed
What does the > is doing here?
The first occurrence is used to introduce a process substitution. The second occurrence is used to redirect output of awk command to a file named file2.txt. The third occurrence is used to redirect the output of sed command to file named file3.txt.
Here, Process substitution is used to capture output that would normally go to a file
The Bash syntax for writing to a process is >(command)
Related
For example:
ls -l -d */ | wc -l | awk '{print $1}' | tee /dev/tty | ls -l
This shell command print the result of wc and ls -l with single line, but tee is used.
Is it possible to using one Shell commend line to achieve multiple output without using “&&” “||” “>” “>>” “<” “;” “&”,tee and temp file?
When you want the output of date and ls -rtl | head -1 on one line, you can use
echo "$(date): $(ls -rtl | head -1)"
Yes, you can achieve writing to multiple files with awk which is not in the list of things you appear not to like:
echo hi | awk '{print > "a.txt"; print > "b.txt"}'
Then check a.txt and b.txt.
i want to svn blame lines of code which include "todo | fixme"
i have the general flow of the script but struggle to combine it into one
finding the lines with "todo"
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" .
blame the line of code
svn blame ${file} | cat -n |grep ${linenumber}
i could get $file and $linenumber from the first command with awk, but i dont know how to pipe the values i extract with awk into the second command.
i am missing the glue to combine these commands into one "script" (- :
You can build the command with awk and then pipe it to bash:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | grep \"%s\"\n", $1, $2}'
That prints one command per input line with the following format:
svn blame "${file}" | cat -n | grep "${linenumber}"
The varibales are replaces. When you execute the command as above they are only printed to the shell, that you can comfirm if everything is right. If yes add a last pipe to the in of the command that the ouput is redirected to bash. The complete command would look like this:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | grep \"%s\"\n", $1, $2}' | bash
A small notice: I think you want to print the line number extracterd in the first command, aren't you? But grep ${linenumber} just gives the line containing the string ${linenumber}. To print only the linenumber use that command: sed -n "2p" to print line number 2 for example. The complete command would then look like this:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | sed -n \"%sp\"\n", $1, $2}' | bash
I would like to know if somebody could help me with this error:
wc: Files/Unused-CMA_host.txt: No such file or directory
The file doesn't exist, then I want to redirect the output to /dev/null
I try with this sentence > /dev/null 2>&1 which is working in other case, but not here:
wc -l Files/Unused-CMA_host.txt | awk '{print $1}' > /dev/null 2>&1
somebody know why?
thanks.
Redirections apply to individual components of a pipeline, not a pipeline as a whole. In your example, you only redirect awk's standard output. To redirect the standard error and stardard output of the entire pipeline would require a command group such as
{ wc -l Files/Unused-CMA_host.txt | awk '{print $1}' ; } > /dev/null 2>&1
However, if the file doesn't exist, there won't be any standard output. Perhaps you want to redirect standard error to suppress the error message? Then you can simply use
wc -l Files/Unused-CMA_host.txt 2> /dev/null | awk '{print $1}'
In the case of a non-existent file, awk will simply read from an empty stream and do nothing.
I am attempting to filter a log file and am running into issues, what I have so far is the following, which does not work,
tail -f /var/log/squid/accesscustom.log | awk '/username/;/user-name/ {print $1; fflush("")}' | awk '!x[$0]++' > /var/log/squid/accesscustom-filtered.log
The goal is to take a file that contains
ipaddress1 username
ipaddress7
ipaddress2 user-name
ipaddress1 username
ipaddress5
ipaddress3 username
ipaddress4 user-name
and save to accesscustom-filtered.log
ipaddress1
ipaddress2
ipaddress3
ipaddress4
It works without the output to accesscustom-filtered.log but something in the > isn't working right and the file ends up empty.
Edit: Changed the original example to be correct
Use tee:
tail -f /var/log/squid/accesscustom.log | awk '/username/;/user-name/ {print $1}' | tee /var/log/squid/accesscustom-filtered.log
See also: Writing “tail -f” output to another file and Turn off buffering in pipe
Note: awk doesn't buffer like grep in the superuser example, so you shouldn't need to do anything special with your awk command. (more info)
What Linux commands would you use successively, for a bunch of files, to count the number of lines in a file and output to an output file with part of the corresponding input file as part of the output line. So for example we were looking at file LOG_Yellow and it had 28 lines, the the output file would have a line like this (Yellow and 28 are tab separated):
Yellow 28
wc -l [filenames] | grep -v " total$" | sed s/[prefix]//
The wc -l generates the output in almost the right format; grep -v removes the "total" line that wc generates for you; sed strips the junk you don't want from the filenames.
wc -l * | head --lines=-1 > output.txt
produces output like this:
linecount1 filename1
linecount2 filename2
I think you should be able to work from here to extend to your needs.
edit: since I haven't seen the rules for you name extraction, I still leave the full name. However, unlike other answers I'd prefer to use head rather then grep, which not only should be slightly faster, but also avoids the case of filtering out files named total*.
edit2 (having read the comments): the following does the whole lot:
wc -l * | head --lines=-1 | sed s/LOG_// | awk '{print $2 "\t" $1}' > output.txt
wc -l *| grep -v " total"
send
28 Yellow
You can reverse it if you want (awk, if you don't have space in file names)
wc -l *| egrep -v " total$" | sed s/[prefix]//
| awk '{print $2 " " $1}'
Short of writing the script for you:
'for' for looping through your files.
'echo -n' for printing the current file
'wc -l' for finding out the line count
And dont forget to redirect
('>' or '>>') your results to your
output file