This question already has answers here:
Bash: execute content of variable including pipe [duplicate]
(2 answers)
Closed 1 year ago.
I want to use grep to search for matching strings in the file, but because the file is too big, I only search for the first 500 lines.
I wrote in the shell script:
#!/bin/bash
patterns=(
llc_prefetcher_operat
to_prefetch
llc_prefetcher_cache_fill
)
search_file_path="mix1-bimodal-no-bop-lru-4core.txt"
echo ${#patterns[*]}
cmd="head -500 ${search_file_path} | grep -a "
for(( i=0;i<${#patterns[#]};i++)) do
cmd=$cmd" -e "\"${patterns[i]}\"
done;
echo $cmd
$cmd >junk.log
The result of running the script is:
3
head -500 mix1-bimodal-no-bop-lru-4core.txt | grep -a -e "llc_prefetcher_operat" -e "to_prefetch" -e "llc_prefetcher_cache_fill"
head: invalid option -a
Try'head --help' for more information.
On the penultimate line, I printed out the string of the executed command. I ran it directly on the command line and it was successful.
That is the following sentence.
head -500 mix1-bimodal-no-bop-lru-4core.txt | grep -a -e "llc_prefetcher_operat" -e "to_prefetch" -e "llc_prefetcher_cache_fill"
Note that in the grep command, if I do not add the -a option, there will be a problem of matching the binary file.
Why does this problem occur? Thank you!
Instead of trying to build a string holding a complex command, you're better off using grep's -f option and bash process substitution to pass the list of patterns to search for:
head -500 "$search_file_path" | grep -Faf <(printf "%s\n" "${patterns[#]}") > junk.log
It's shorter, simpler and less error prone.
(I added -F to the grep options because none of your example patterns have any regular expression metacharacters; so fixed string searching will likely be faster)
The biggest problem with what you're doing is the | is treated as just another argument to head when $cmd is word split. It's not treated as a pipeline delimiter like it is when a literal one is present.
Related
x=rankings* && grep -A10 -P '^Ranked :$' $x | tail -n +2 > results$x
This is a command that I can't get to work no matter the approach and I haven't been able to find anything within 10+ searches of stack overflow.
Is there a way to feed a wildcard through as an argument to a single line of commands? I want to make a list of files based on existing files in the directory.
The closest I have gotten is
x=rankingsX.Y.Z && grep -A10 -P '^Ranked :$' $x | tail -n +2 > results$x
where X Y Z are some numbers, but this hard-coding individually is the opposite of my objective - a single line command(not script file) that searches and outputs specific text into files using the original names.
A redirection only has one destination at a time; thus, an attempt to redirect to an expression which, when string-split and glob-expanded, results in more than one filename causes a "bad redirection" error.
What you want is x to have one value at a time, for each value the glob matches to -- which is to say that this is a job for a for loop.
for x in rankings*; do grep -A10 -P '^Ranked :$' "$x" | tail -n +2 >"results$x"; done
...which could also be written over multiple lines (even at an interactive shell), as in:
for x in rankings*; do
grep -A10 -P '^Ranked :$' "$x" | tail -n +2 >"results$x"
done
This question already has answers here:
Why does shell ignore quoting characters in arguments passed to it through variables? [duplicate]
(3 answers)
Closed 6 years ago.
I'm creating a variable from an array which build up multiple -e clauses for a sed command.
The resulting variable is something like:
sedArgs="-e 's/search1/replace1/g' -e 's/search2/replace2/g' -e 's/search3/replace3/g'"
But when I try to call sed with this as the argument I get the error sed: -e expression #1, char 1: unknown command: ''
I've tried to call sed the following ways:
cat $myFile | sed $sedArgs
cat $myFile | sed ${sedArgs}
cat $myFile | sed `echo $sedArgs`
cat $myFile | sed "$sedArgs"
cat $myFile | sed `echo "$sedArgs"`
and all give the same error.
UPDATE - Duplicate question
As has been identified, this is a 'quotes expansion' issue - I thought it was something sed specific, but the duplicate question that has been identified put me on the right track.
I managed to resolve the issue by creating the sedArgs string as:
sedArgs="-e s/search1/replace1/g -e s/search2/replace2/g -e s/search3/replace3/g"
and calling it with:
cat $myFile | sed $sedArgs
which works perfectly.
Then I took the advice of tripleee and kicked the useless cat out!
sed $sedArgs $myFile
also works perfectly.
Use BASH arrays instead of simple string:
# sed arguments in an array
sedArgs=(-e 's/search1/replace1/g' -e 's/search2/replace2/g' -e 's/search3/replace3/g')
# then use it as
sed "${sedArgs[#]}" file
Here is no sane way to do that, but you can pass the script as a single string.
sedArgs='s/search1/replace1/g
s/search2/replace2/g
s/search3/replace3/g'
: then
sed "$sedArgs" "$myFile"
The single-quoted string spans multiple lines; this is scary when you first see it, but perfectly normal shell script. Notice also how the cat is useless as ever, and how the file name needs to be quoted, too.
This question already has answers here:
Using variable inside of sed -i (regex?) bash
(2 answers)
Why wont sed remove line from file?
(4 answers)
Closed 8 years ago.
I am trying to remove lines from a text file from a Bash Script using command sed.
Here is how this function works.
User enters record number
Program Searches for record number
Program deletes record
Here is my code:
r=$(grep -h "$record" student_records.txt|cut -d"," -f1) #find the record that needs to be deleted
echo $line -->> This proves that previous command works
sed -i '/^$r/d' student_records.txt -->> this does not work
Any ideas?
To remove a line containing $record from the file:
grep -v "$record" student_records.txt >delete.me && mv -f delete.me student_records.txt
In the above, $record is treated as a regular expression. This means, for example, that a period is a wildcard. If that is unwanted, add the -F option to grep to specify that $record is to be treated as a fixed string.
Comments
Consider these two line:
r=$(grep -h "$record" student_records.txt|cut -d"," -f1) #find the record that needs to be deleted
echo $line -->> This proves that previous command works
The first line defines a shell variable r. The second line prints the shell variable line, a variable which was unaffected by the previous command. Consequently, the second line is not a successful test on the first.
sed -i '/^$r/d' student_records.txt -->> this does not work
Obseve that the expression $r appears inside single-quotes. The shell does not alter any inside single quotes. Consequently, $r will remain a dollar sign followed by an r. Since a dollar sign matches the end of a line, this expression will match nothing. The following would work better:
sed -i "/^$r/d" student_records.txt
Unlike the grep command, however, the above sed command is potentially dangerous. It would be easy to construct a value of r that would cause sed to do surprising things. So, don't use this approach unless you trust the process by which you obtained r.
What if more than one line matches record?
If there is more than one line that matches record, then the following would generate an unterminated address regex error from sed:
r=$(grep -h "$record" student_records.txt|cut -d"," -f1)
sed -i "/^$r/d" student_records.txt
This error is an example of the surprising results that can happen when a shell variable is expanded into a sed command.
By contrast, this approach would remove all matching lines:
grep -v "$record" student_records.txt >delete.me && mv -f delete.me student_records.txt
We use some package called Autosys and there are some specific commands of this package. I have a list of variables which i like to pass in one of the Autosys commands as variables one by one.
For example one such variable is var1, using this var1 i would like to launch a command something like this
autosys_showJobHistory.sh var1
Now when I launch the below written command, it gives me the desired output.
echo "var1" | while read line; do autosys_showJobHistory.sh $line | grep 1[1..6]:[0..9][0..9] | grep 24.12.2012 | tail -1 ; done
But if i put the var1 in a file say Test.txt and launch the same command using cat, it gives me nothing. I have the impression that command autosys_showJobHistory.sh does not work in that case.
cat Test.txt | while read line; do autosys_showJobHistory.sh $line | grep 1[1..6]:[0..9][0..9] | grep 24.12.2012 | tail -1 ; done
What I am doing wrong in the second command ?
Wrote all of below, and then noticed your grep statement.
Recall that ksh doesn't support .. as an indicator for 'expand this range of values'. (I assume that's your intent). It's also made ambiguous by your lack of quoting arguments to grep. If you were using syntax that the shell would convert, then you wouldn't really know what reg-exp is being sent to grep. Always better to quote argments, unless you know for sure that you need the unquoted values. Try rewriting as
grep '1[1-6]:[0-9][0-9]' | grep '24.12.2012'
Also, are you deliberately using the 'match any char' operator '.' OR do you want to only match a period char? If you want to only match a period, then you need to escape it like \..
Finally, if any of your files you're processing have been created on a windows machine and then transfered to Unix/Linux, very likely that the line endings (Ctrl-MCtrl-J) (\r\n) are causing you problems. Cleanup your PC based files (or anything that was sent via ftp) with dos2unix file [file2 ...].
If the above doesn't help, You'll have to "divide and conquer" to debug your problem.
When I did the following tests, I got the expected output
$ echo "var1" | while read line ; do print "line=${line}" ; done
line=var1
$ vi Test.txt
$ cat Test.txt
var1
$ cat Test.txt | while read line ; do print "line=${line}" ; done
line=var1
Unrelated to your question, but certain to cause comment is your use of the cat commnad in this context, which will bring you the UUOC award. That can be rewritten as
while read line ; do print "line=${line}" ; done < Test.txt
But to solve your problem, now turn on the shell debugging/trace options, either by changing the top line of the script (the shebang line) like
#!/bin/ksh -vx
Or by using a matched pair to track the status on just these lines, i.e.
set -vx
while read line; do
print -u2 -- "#dbg: Line=${line}XX"
autosys_showJobHistory.sh $line \
| grep 1[1..6]:[0..9][0..9] \
| grep 24.12.2012 \
| tail -1
done < Test.txt
set +vx
I've added an extra debug step, the print -u2 -- .... (u2=stderror, -- closes option processing for print)
Now you can make sure no extra space or tab chars are creeping in, by looking at that output.
They shouldn't matter, as you have left your $line unquoted. As part of your testing, I'd recommend quoting it like "${line}".
Then I'd comment out the tail and the grep lines. You want to see what step is causing this to break, right? So does the autosys_script by itself still produce the intermediate output you're expecting? Then does autosys + 1 grep produce out as expected, +2 greps, + tail? You should be able to easily see where you're loosing your output.
IHTH
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
bash echo number of lines of file given in a bash variable
Was wondering how you output the number of lines in a text file to screen and then store it in a variable.
I have a file called stats.txt and when I run wc -l stats.txt it outputs 8 stats.txt
I tried doing x = wc -l stats.txt thinking it would store the number only and the rest is just for visual but it does not work :(
Thanks for the help
There are two POSIX standard syntax for doing this:
x=`cat stats.txt | wc -l`
or
x=$(cat stats.txt | wc -l)
They both run the program and replace the invocation in the script with the standard output of the command, in this case assigning it to the $x variable. However, be aware that both trim ending newlines (this is actually what you want here, but can be dangerous sometimes, when you expect a newline).
Also, the second case can be easily nested (example: $(cat $(ls | head -n 1) | wc -l)). You can also do it with the first case, but it is more complex:
`cat \`ls | head -n 1\` | wc -l`
There are also quotation issues. You can include these expressions inside double-quotes, but with the back-ticks, you must continue quoting inside the command, while using the parenthesis allows you to "start a new quoting" group:
"`echo \"My string\"`"
"$(echo "My string")"
Hope this helps =)
you may try:
x=`cat stats.txt | wc -l`
or (from the another.anon.coward's comment):
x=`wc -l < stats.txt`