x=rankings* && grep -A10 -P '^Ranked :$' $x | tail -n +2 > results$x
This is a command that I can't get to work no matter the approach and I haven't been able to find anything within 10+ searches of stack overflow.
Is there a way to feed a wildcard through as an argument to a single line of commands? I want to make a list of files based on existing files in the directory.
The closest I have gotten is
x=rankingsX.Y.Z && grep -A10 -P '^Ranked :$' $x | tail -n +2 > results$x
where X Y Z are some numbers, but this hard-coding individually is the opposite of my objective - a single line command(not script file) that searches and outputs specific text into files using the original names.
A redirection only has one destination at a time; thus, an attempt to redirect to an expression which, when string-split and glob-expanded, results in more than one filename causes a "bad redirection" error.
What you want is x to have one value at a time, for each value the glob matches to -- which is to say that this is a job for a for loop.
for x in rankings*; do grep -A10 -P '^Ranked :$' "$x" | tail -n +2 >"results$x"; done
...which could also be written over multiple lines (even at an interactive shell), as in:
for x in rankings*; do
grep -A10 -P '^Ranked :$' "$x" | tail -n +2 >"results$x"
done
Related
I am creating a script that parses some files and greps out the necessary information.
I have set up many different variables in arrays that search for different aspects in the files.
e.g. dates, locations, types.
However I wanted to make each of these variables optional which is where I run into an issue.
the syntax of the command would have been simple
grep ${dates} filename | grep ${locations} | grep ${types}
However, due to variables being optional, the above won't work if a variable is unset.
I was trying to find a way to get grep to find anything (i.e. like egrep .* filename)
that way I could set the variables to the proper regex and have the command still run.
unfortunately, when I set the variable to equal '' it freezes, when I set it to '.' it just greps everything from every file in the current directory and when I leave the variable blank it takes the filename as the variable and waits for a filename.
is there anyway that I can set a variable so that grep $variable file outputs the same as cat would?
Many thanks in advance!
To get grep to act like cat use an empty string as a search pattern, i.e. grep "". Therefore to make some of those variables optional, but not have piped greps fail, just quote the variables:
grep "${dates}" filename | grep "${locations}" | grep "${types}"
Demonstration. Search {250,255,260...280} for the digits 5, 2, and 7:
x=5 y=2 z=7 ; seq 250 5 280 | grep "$x" | grep "$y" | grep "$z"
275
Now unset two of the variables, and it still works:
unset x y ; seq 250 5 280 | grep "$x" | grep "$y" | grep "$z"
270
275
If there aren't any $dates in filename then there is nothing to feed the rest of the pipeline.
I think the best way to do it is to grep for each string separately.
If you get a match, then grep for the next string.
Can you post the source file and a target output. From your question it sounds like you just need to use grep -E in conjunction with the | pipe delimiter.
grep -E "${dates}|${locations}|${types}" fileName
The above line should automatically get you every occurrence of the any of the patterns. This is not even a regex yet.
I have a command that lists the full 8 level deep path of all folders we are backing up.
I also have a command that enumerates all 8 level deep folders on the system.
Both of these are stored as variables in a bash script.
I'm trying to get a loop together that takes file 1 and uses the first line entry as a variable in an if/then/else, and then moves onwards to through the end of the file.
I've tried so many things but its beyond my skillset to provide an example that won't confuse the reader of this post.
TempFile1=/ifs/data/scripts/ConfigMonitor/TempFile1.txt
TempFile2=/ifs/data/scripts/ConfigMonitor/TempFile2.txt
find /ifs/*/*/backup -maxdepth 4 -mindepth 4 -type d > $TempFile1
isi snapshot schedules list -v | grep Path: | awk '{print $2}' > $TempFile2
list line 1 on $TempFile1
Grep for line 1 within $TempFile2
if result yielded then
echo found
else
echo fullpath not being backed up
fi
Use Grep's -f Flag
grep(1) says:
-f FILE, --file=FILE
Obtain patterns from FILE, one per line. The empty file
contains zero patterns, and therefore matches nothing. (-f is
specified by POSIX.)
Therefore, the following should work:
grep -f patterns_to_match.txt file_to_examine.txt
Faster Reporting
Another way to think about this is that you can ask GNU grep to show you all the matches:
echo 'Lines that match a pattern in your pattern file.'
grep -f patterns_to_match.txt file_to_examine.txt
and then show you all the lines that don't match any of the patterns:
echo 'Lines that do not match any patterns in your pattern file.'
grep -f patterns_to_match.txt -v file_to_examine.txt
This is likely to be faster and more efficient than looping through the file one line at a time in Bash. You may or may not get similar results with a grep other than GNU grep; while the -f and -v flags are specified by POSIX, I only tested it against GNU grep 2.16, so your mileage may vary.
This should iterate through Tempfile1.txt and grep for the line in TempFile2.txt.
while read line; do
if grep $line /path/to/TempFile2.txt > /dev/null
then
echo "Found $line"
else
echo "Did not find $line"
fi
done < Tempfile1.txt
Tempfile1.txt:
a
b
c
Tempfile2.txt
b
d
z
Output:
Did not find a
Found b
Did not find c
I am trying to select the nth file in a folder of which the filename matches a certain pattern:
Ive tried using this with sed: e.g.,
sed -n 3p /path/to/files/pattern.txt
but it appears to return the 3rd line of the first matching file.
Ive also tried
sed -n 3p ls /path/to/files/*pattern*.txt
which doesnt work either.
Thanks!
Why sed, when bash is so much better at it?
Assuming some name n indicates the index you want:
Bash
files=(path/to/files/*pattern*.txt)
echo "${files[n]}"
Posix sh
i=0
for file in path/to/files/*pattern*.txt; do
if [ $i = $n ]; then
break
fi
i=$((i++))
done
echo "$file"
What's wrong with sed is that you would have to jump through many hoops to make it safe for the entire set of possible characters that can occur in a filename, and even if that doesn't matter to you you end up with a double-layer of subshells to get the answer.
file=$(printf '%s\n' path/to/files/*pattern*.txt | sed -n "$n"p)
Please, never parse ls.
ls -1 /path/to/files/*pattern*.txt | sed -n '3p'
or, if patterne is a regex pattern
ls -1 /path/to/files/ | egrep 'pattern' | sed -n '3p'
lot of other possibilities, it depend on performance or simplicity you look at
We use some package called Autosys and there are some specific commands of this package. I have a list of variables which i like to pass in one of the Autosys commands as variables one by one.
For example one such variable is var1, using this var1 i would like to launch a command something like this
autosys_showJobHistory.sh var1
Now when I launch the below written command, it gives me the desired output.
echo "var1" | while read line; do autosys_showJobHistory.sh $line | grep 1[1..6]:[0..9][0..9] | grep 24.12.2012 | tail -1 ; done
But if i put the var1 in a file say Test.txt and launch the same command using cat, it gives me nothing. I have the impression that command autosys_showJobHistory.sh does not work in that case.
cat Test.txt | while read line; do autosys_showJobHistory.sh $line | grep 1[1..6]:[0..9][0..9] | grep 24.12.2012 | tail -1 ; done
What I am doing wrong in the second command ?
Wrote all of below, and then noticed your grep statement.
Recall that ksh doesn't support .. as an indicator for 'expand this range of values'. (I assume that's your intent). It's also made ambiguous by your lack of quoting arguments to grep. If you were using syntax that the shell would convert, then you wouldn't really know what reg-exp is being sent to grep. Always better to quote argments, unless you know for sure that you need the unquoted values. Try rewriting as
grep '1[1-6]:[0-9][0-9]' | grep '24.12.2012'
Also, are you deliberately using the 'match any char' operator '.' OR do you want to only match a period char? If you want to only match a period, then you need to escape it like \..
Finally, if any of your files you're processing have been created on a windows machine and then transfered to Unix/Linux, very likely that the line endings (Ctrl-MCtrl-J) (\r\n) are causing you problems. Cleanup your PC based files (or anything that was sent via ftp) with dos2unix file [file2 ...].
If the above doesn't help, You'll have to "divide and conquer" to debug your problem.
When I did the following tests, I got the expected output
$ echo "var1" | while read line ; do print "line=${line}" ; done
line=var1
$ vi Test.txt
$ cat Test.txt
var1
$ cat Test.txt | while read line ; do print "line=${line}" ; done
line=var1
Unrelated to your question, but certain to cause comment is your use of the cat commnad in this context, which will bring you the UUOC award. That can be rewritten as
while read line ; do print "line=${line}" ; done < Test.txt
But to solve your problem, now turn on the shell debugging/trace options, either by changing the top line of the script (the shebang line) like
#!/bin/ksh -vx
Or by using a matched pair to track the status on just these lines, i.e.
set -vx
while read line; do
print -u2 -- "#dbg: Line=${line}XX"
autosys_showJobHistory.sh $line \
| grep 1[1..6]:[0..9][0..9] \
| grep 24.12.2012 \
| tail -1
done < Test.txt
set +vx
I've added an extra debug step, the print -u2 -- .... (u2=stderror, -- closes option processing for print)
Now you can make sure no extra space or tab chars are creeping in, by looking at that output.
They shouldn't matter, as you have left your $line unquoted. As part of your testing, I'd recommend quoting it like "${line}".
Then I'd comment out the tail and the grep lines. You want to see what step is causing this to break, right? So does the autosys_script by itself still produce the intermediate output you're expecting? Then does autosys + 1 grep produce out as expected, +2 greps, + tail? You should be able to easily see where you're loosing your output.
IHTH
I am writing some BASH shell script that will continuously check a file to see if the file already contains "Completed!" before proceeding. (Of course, assume the file is being updated and will eventually contain the phrase "Completed!")
I am not sure how to do this. Thank you for your help.
You can do something like:
while ! grep -q -e 'Completed!' file ; do
sleep 1 # Or some other number of seconds
done
# Here the file contains completed
Amongst the standard utilities, tail has an option to keep reading from a file: tail -f. So filter the output of tail -f.
<some_file tail -f -n +1 | grep 'Completed!' | head -n 1 >/dev/null
There may be a delay due to buffering. You can at least reduce the delay by using fewer tools in the pipeline. In fact, some implementations of tail never buffer when you do tail -f, so the following snippet will return as soon as Completed! is written to the file.
<some_file tail -f -n +1 | sed -e '/Completed!/ q'
This assumes that the file is being appended to by some other tool. If the file is overwritten by the data-producing program after you start tail, this solution won't work. You can search the file periodically. On some systems you can call a notification mechanism to know whenever the file changes, e.g. with inotifywait under Linux.
I've done this in Kornshell:
tail -f somefile | while read line
do
echo $line
[[ $line == *Completed!* ]] && break
done
Note no quotes around the *Completed!* string. This allows the double square brackets to do glob pattern matching instead of string matching.
This seems to work in BASH too. However, the line with the Completed must end in a NL. Otherwise, it'll take an extra line before it breaks the loop.
You can use grep too:
tail -f somefile | while read line
do
echo $line
grep -iq "Completed!" && break
done
The -q parameter means quiet. If your grep doesn't take the -q parameter, you might have to pipe it to /dev/null. The -i is ignore case. Whether you want to do that is up to you.
The advantage is that you aren't doing any processing unless there's a line to read. Using sleep may mean you miss the line, or that you're processing when no line has been added to the file.
Using grep in a pipe you may turn on line buffering mode by adding the --line-buffered option!