Execute piped shell commands in Tcl - shell

I want to execute these piped shell commands in Tcl:
grep -v "#" inputfile | grep -v ">" | sort -r -nk7 | head
I try:
exec grep -v "#" inputfile | grep -v ">" | sort -r -nk7 | head
and get an error:
Error: grep: invalid option -- 'k'
When I try to pipe only 2 of the commands:
exec grep -v "#" inputfile | grep -v ">"
I get:
Error: can't specify ">" as last word in command
Update: I also tried {} and {bash -c '...'}:
exec {bash -c 'grep -v "#" inputfile | grep -v ">"'}
Error: couldn't execute "bash -c 'grep -v "#" inputfile | grep -v ">"'": no such file or directory
My question: how can I execute the initial piped commands in a tcl script?
Thanks

The problem is that exec does “special things” when it sees a > on its own (or at the start of a word) as that indicates a redirection. Unfortunately, there's no practical way to avoid this directly; this is an area where Tcl's syntax system doesn't help. You end up having to do something like this:
exec grep -v "#" inputfile | sh -c {exec grep -v ">"} | sort -r -nk7 | head
You can also move the entire pipeline to the Unix shell side:
exec sh -c {grep -v "#" inputfile | grep -v ">" | sort -r -nk7 | head}
Though to be frank this is something that you can do in pure Tcl, which will then make it portable to Windows too…

The > is causing problems here.
You need to escape it from tcl and the shell to make it work here.
exec grep -v "#" inputfile | grep -v {\\>} | sort -r -nk7 | head
or (and this is better since you have one less grep)
exec grep -Ev {#|>} inputfile | sort -r -nk7 | head
If you look in the directory you were running this from (assuming tclsh or similar) you'll probably see that you created an oddly named file (i.e. |) before.

In pure Tcl:
package require fileutil
set lines {}
::fileutil::foreachLine line inputfile {
if {![regexp #|> $line]} {
lappend lines $line
}
}
set lines [lsort -decreasing -integer -index 6 $lines]
set lines [lrange $lines 0 9]
puts [join $lines \n]\n
(-double might be more appropriate than -integer)
Edit: I mistranslated the (1-based) -k index for the command sort when writing the (0-based) -index option for lsort. It is now corrected.
Documentation: fileutil package, if, join, lappend, lrange, lsort, package, puts, regexp, set

Related

How to combine output from awk and pipe into htop?

$ ps -ef | grep python | awk -F' ' '{print $2}'
9825
4470
4619
$ htop -p 9825,4470,4619
For now, I have to make two separate commands in order to watch all python processes within htop. Is there a way that I can pipe all the results from awk and feed them into htop?
If you have pgrep (you probably do):
htop -p $(pgrep python | paste -sd,)
You could avoid grep and use only awk using something like:
ps -ef | awk '/[p]ython/{print $2}'
Then you could use:
htop -p $(ps -ef | awk -v ORS=, '/[p]ython/{print $2}')
Notice the [] around the p, this is a nice trick to avoid printing the second command itself:
ps -ef | awk '/[p]ython/{print $2}'
| |
cmd 1 cmd 2
it works because awk will translate the regex [p] to say something like "match characters from [p] in this case, p only, followed by ython:
[p]ython != python

Curl and xargs in piped commands

I want to process an old database where password are plain text (comma separated ; passwd is the 5th field in the csv file where the database has been exported) to crypt them for further use by dokuwiki. Here is my bash command (grep and sed are there to extract the crypted passwd from curl output) :
cat users.csv | awk 'FS="," { print $4 }' | xargs -l bash -c 'curl -s --data-binary "pass1=$0&pass2=$0" "https://sprhost.com/tools/SMD5.php" -o - ' | xargs | grep -o '<tt.*tt>' | sed -e 's/tt//g' | sed -e 's/<[^>]*>//g'
I get the following comment from xargs
xargs: unmatched single quote; by default quotes are special to xargs unless you use the -0 option
And only the first line of the file is processed, and nothing appends then.
Using the -0 option, and playing around with quotes, doesn't solve anything. Where am I wrong in the command line ? May be a more advanced language will be more adequate to do this.
Thank for help, LM
In general, if you have such a long pipe of commands, it is better to split them if things go wrong. Going through your pipe:
cat users.csv |
Nothing unexpected there.
awk 'FS="," { print $4 }' |
You probably wanted to do awk 'BEGIN {FS=","} { print $4 }'. Try the first two commands in the pipe and see if they produce the correct answer.
xargs -l bash -c 'curl -s --data-binary "pass1=$0&pass2=$0" "https://sprhost.com/tools/SMD5.php" -o - ' |
Nothing wrong there, although there might be better ways to do an MD5 hash.
xargs |
What is this xargs doing in the pipe? It should be removed.
grep -o '<tt.*tt>' |
Note that this will produce two lines:
<tt>$1$17ab075e$0VQMuM3cr5CtElvMxrPcE0</tt>
<tt><your_docuwiki_root>/conf/users.auth.php</tt>
which is probably not what you expected.
sed -e 's/tt//g' |
sed -e 's/<[^>]*>//g'
which will remove the html-tags, though
sed 's/<tt>//;s/<.tt>//'
will do the same.
So I'd say a wrong awk and an xargs too many.

Bash code error unexpected syntax error

I am not sure why i am getting the unexpected syntax '( err
#!/bin/bash
DirBogoDict=$1
BogoFilter=/home/nikhilkulkarni/Downloads/bogofilter-1.2.4/src/bogofilter
echo "spam.."
for i in 'cat full/index |fgrep spam |awk -F"/" '{if(NR>1000)print$2"/"$3}'|head -500'
do
cat $i |$BogoFilter -d $DirBogoDict -M -k 1024 -v
done
echo "ham.."
for i in 'cat full/index | fgrep ham | awk -F"/" '{if(NR>1000)print$2"/"$3}'|head -500'
do
cat $i |$BogoFilter -d $DirBogoDict -M -k 1024 -v
done
Error:
./score.bash: line 7: syntax error near unexpected token `('
./score.bash: line 7: `for i in 'cat full/index |fgrep spam |awk -F"/" '{if(NR>1000)print$2"/"$3}'|head -500''
Uh, because you have massive syntax errors.
The immediate problem is that you have an unpaired single quote before the cat which exposes the Awk script to the shell, which of course cannot parse it as shell script code.
Presumably you want to use backticks instead of single quotes, although you should actually not read input with for.
With a fair bit of refactoring, you might want something like
for type in spam ham; do
awk -F"/" -v type="$type" '$0 ~ type && NR>1000 && i++<500 {
print $2"/"$3 }' full/index |
xargs $BogoFilter -d $DirBogoDict -M -k 1024 -v
done
This refactors the useless cat | grep | awk | head into a single Awk script, and avoids the silly loop over each output line. I assume bogofilter can read file name arguments; if not, you will need to refactor the xargs slightly. If you can pipe all the files in one go, try
... xargs cat | $BogoFilter -d $DirBogoDict -M -k 1024 -v
or if you really need to pass in one at a time, maybe
... xargs sh -c 'for f; do $BogoFilter -d $DirBogoDict -M -k 1024 -v <"$f"; done' _
... in which case you will need to export the variables BogoFilter and DirBogoDict to expose them to the subshell (or just inline them -- why do you need them to be variables in the first place? Putting command names in variables is particularly weird; just update your PATH and then simply use the command's name).
In general, if you find yourself typing the same commands more than once, you should think about how to avoid that. This is called the DRY principle.
The syntax error is due to bad quoting. The expression whose output you want to loop over should be in command substitution syntax ($(...) or backticks), not single quotes.

how to pass values from stdout as parameter for the next command

i want to svn blame lines of code which include "todo | fixme"
i have the general flow of the script but struggle to combine it into one
finding the lines with "todo"
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" .
blame the line of code
svn blame ${file} | cat -n |grep ${linenumber}
i could get $file and $linenumber from the first command with awk, but i dont know how to pipe the values i extract with awk into the second command.
i am missing the glue to combine these commands into one "script" (- :
You can build the command with awk and then pipe it to bash:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | grep \"%s\"\n", $1, $2}'
That prints one command per input line with the following format:
svn blame "${file}" | cat -n | grep "${linenumber}"
The varibales are replaces. When you execute the command as above they are only printed to the shell, that you can comfirm if everything is right. If yes add a last pipe to the in of the command that the ouput is redirected to bash. The complete command would look like this:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | grep \"%s\"\n", $1, $2}' | bash
A small notice: I think you want to print the line number extracterd in the first command, aren't you? But grep ${linenumber} just gives the line containing the string ${linenumber}. To print only the linenumber use that command: sed -n "2p" to print line number 2 for example. The complete command would then look like this:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | sed -n \"%sp\"\n", $1, $2}' | bash

bash command quoted by single quote

I need to find process by string matching, and the kill it, need to do it in one line in another script file:
here's what I tried:
'kill $(ps -ef|grep xxx|grep -v grep | awk '{print $2 }' )'
"kill $(ps -ef|grep xxx|grep -v grep | awk '{print $2 }' )"
first one didn't work because of the nested single quote, second one didn't work because $2 is taken by the parent script to be argument 2 to parent script.
how do I do this?
The easiest way to accomplish that task is:
pkill xxx
(which you'll find in the debian/ubuntu world in package procps, if you don't have it installed.) You might need to use pkill -f xxx, depending on whether xxx is part of the process name or an argument, which is often the case with script execution.
However, to answer the more general question about shell-quoting, if you need to pass the string
kill $(ps aux | grep xxx | grep -v grep | awk '{print $2}')
as an argument, you need to use backslash escapes:
bash -c "kill \$(ps aux | grep xxx | grep -v grep | awk '{print \$2}')"
Or, you can paste together several quoted strings:
bash -c 'kill $(ps aux | grep xxx | grep -v grep | awk '"'"'{print $2}'"'"')'
Personally, I find the first one more readable but YMMV.
You can only backslash escape a few characters inside a double-quoted string: $, ", \, newline and backtick; and inside a single-quoted string backslash is just backslash. However, that's enough to let you type just about anything.

Resources