in my script i need to loop through lines in a file, once i find some specific line i need to save it to variable so later on i can use it outside the loop, i tried the following but it wont' work:
count=0
res=""
python my.py -p 12345 |
while IFS= read -r line
do
count=$((count+1))
if [ "$count" -eq 5 ]; then
res=`echo "$line" | xargs`
fi
done
echo "$res"
it output nothing, i also tried this,
res=""
... in the loop...
res=$res`echo "$line" | xargs`
still nothing. please help. thanks.
Update: Thanks for all the help. here is my final code:
res=python my.py -p 12345 | sed -n '5p' | xargs
for finding a specific line in a file, have you considered using grep?
grep "thing I'm looking for" /path/to/my.file
this will output the lines that match the thing you're looking for. Moreover this can be piped to xargs as in your question.
If you need to look at a particularly numbered line of a file, consider using the head and tail commands (which can also be piped to grep).
cat /path/to/my.file | head -n5 | tail -n1 | grep "thing I'm looking for"
These commands take the first lines specified (in this case, 5 and 1 respectively) and only prints those out. Hopefully this will help you accomplish your task.
Happy coding! Leave a comment if you have any questions.
Related
I want to apply a command to each line of piped stdin like so:
cat file.txt | grep ... | ./filter | wc -l
the problem is ./filter accepts only a single line of input and gives a single line of output. I've tried xargs but it spawns a subshell and I can't capture it's output to continue working with the result. Is there an easy way to do that?
If it accepts a single line, then you should put it in a loop if you want to process multiple lines,
cat file.txt |
grep ... |
while read line ; do
echo "$line" | ./filter
done |
wc -l
To call a command for each line you can read a line into a variable and use the variable as a standard input. (Also, let’s avoid UUOC.)
grep ... < file.txt |
while IFS= read -r line; do
./filter <<< "$line"
done |
wc -l
In this case it looks like things may get way easier if you instead write the whole filter in awk. Because it will give you wc -l for free (NR), plus line and record splitting and filtering better than what grep can do.
Not sure how to explain this but, what I am trying to achieve is this:
- tailing a file and grepping for a patter A
- then I want to pipe into another customGrepFunction where it matches pattern B, and if B matches echo something out. Need the customGrepFunction in order to do some other custom stuff.
The sticky part here is how to make the grepCustomFunction work here.In other words when only patternA matches echo the whole line and when both patterA & patternB match printout something custom:
when I only run:
tail -f file.log | grep patternA
I can see the pattenA rows are being printed/tailed however when I add the customGrepFunction nothing happens.
tail -f file.log | grep patternA | customGrepFunction
And the customGrepFunction should be available globally in my bin folder:
customGrepFunction(){
if grep patternB
then
echo "True"
fi
}
I have this setup however it doesn't do what I need it to do, it only echos True whenever I do Ctrl+C and exit the tailing.
What am I missing here?
Thanks
What's Going Wrong
The code: if grep patternB; then echo "true"; fi
...waits for grep patternB to exit, which will happen only when the input from tail -f file.log | grep patternA hits EOF. Since tail -f waits for new content forever, there will never be an EOF, so your if statement will never complete.
How To Fix It
Don't use grep on the inside of your function. Instead, process content line-by-line and use bash's native regex support:
customGrepFunction() {
while IFS= read -r line; do
if [[ $line =~ patternB ]]; then
echo "True"
fi
done
}
Next, make sure that grep isn't buffering content (if it were, then it would be written to your code only in big chunks, delaying until such a chunk is available). The means to do this varies by implementation, but with GNU grep, it would look like:
tail -f file.log | grep --line-buffered patternA | customGrepFunction
I am trying to select the nth file in a folder of which the filename matches a certain pattern:
Ive tried using this with sed: e.g.,
sed -n 3p /path/to/files/pattern.txt
but it appears to return the 3rd line of the first matching file.
Ive also tried
sed -n 3p ls /path/to/files/*pattern*.txt
which doesnt work either.
Thanks!
Why sed, when bash is so much better at it?
Assuming some name n indicates the index you want:
Bash
files=(path/to/files/*pattern*.txt)
echo "${files[n]}"
Posix sh
i=0
for file in path/to/files/*pattern*.txt; do
if [ $i = $n ]; then
break
fi
i=$((i++))
done
echo "$file"
What's wrong with sed is that you would have to jump through many hoops to make it safe for the entire set of possible characters that can occur in a filename, and even if that doesn't matter to you you end up with a double-layer of subshells to get the answer.
file=$(printf '%s\n' path/to/files/*pattern*.txt | sed -n "$n"p)
Please, never parse ls.
ls -1 /path/to/files/*pattern*.txt | sed -n '3p'
or, if patterne is a regex pattern
ls -1 /path/to/files/ | egrep 'pattern' | sed -n '3p'
lot of other possibilities, it depend on performance or simplicity you look at
I have been trying to get the head utility to display all but the last line of standard input. The actual code that I needed is something along the lines of cat myfile.txt | head -n $(($(wc -l)-1)). But that didn't work. I'm doing this on Darwin/OS X which doesn't have the nice semantics of head -n -1 that would have gotten me similar output.
None of these variations work either.
cat myfile.txt | head -n $(wc -l | sed -E -e 's/\s//g')
echo "hello" | head -n $(wc -l | sed -E -e 's/\s//g')
I tested out more variations and in particular found this to work:
cat <<EOF | echo $(($(wc -l)-1))
>Hola
>Raul
>Como Esta
>Bueno?
>EOF
3
Here's something simpler that also works.
echo "hello world" | echo $(($(wc -w)+10))
This one understandably gives me an illegal line count error. But it at least tells me that the head program is not consuming the standard input before passing stuff on to the subshell/command substitution, a remote possibility, but one that I wanted to rule out anyway.
echo "hello" | head -n $(cat && echo 1)
What explains the behavior of head and wc and their interaction through subshells here? Thanks for your help.
head -n -1 will give you all except the last line of its input.
head is the wrong tool. If you want to see all but the last line, use:
sed \$d
The reason that
# Sample of incorrect code:
echo "hello" | head -n $(wc -l | sed -E -e 's/\s//g')
fails is that wc consumes all of the input and there is nothing left for head to see. wc inherits its stdin from the subshell in which it is running, which is reading from the output of the echo. Once it consumes the input, it returns and then head tries to read the data...but it is all gone. If you want to read the input twice, the data will have to be saved somewhere.
Using sed:
sed '$d' filename
will delete the last line of the file.
$ seq 1 10 | sed '$d'
1
2
3
4
5
6
7
8
9
For Mac OS X specifically, I found an answer from a comment to this Q&A.
Assuming you are using Homebrew, run brew install coreutils then use the ghead command:
cat myfile.txt | ghead -n -1
Or, equivalently:
ghead -n -1 myfile.txt
Lastly, see brew info coreutils if you'd like to use the commands without the g prefix (e.g., head instead of ghead).
cat myfile.txt | echo $(($(wc -l)-1))
This works. It's overly complicated: you could just write echo $(($(wc -l)-1)) <myfile.txt or echo $(($(wc -l <myfile.txt)-1)). The problem is the way you're using it.
cat myfile.txt | head -n $(wc -l | sed -E -e 's/\s//g')
wc consumes all the input as it's counting the lines. So there is no data left to read in the pipe by the time head is started.
If your input comes from a file, you can redirect both wc and head from that file.
head -n $(($(wc -l <myfile.txt) - 1)) <myfile.txt
If your data may come from a pipe, you need to duplicate it. The usual tool to duplicate a stream is tee, but that isn't enough here, because the two outputs from tee are produced at the same rate, whereas here wc needs to fully consume its output before head can start. So instead, you'll need to use a single tool that can detect the last line, which is a more efficient approach anyway.
Conveniently, sed offers a way of matching the last line. Either printing all lines but the last, or suppressing the last output line, will work:
sed -n '$! p'
sed '$ d'
Here is a one-liner that can get you the desired output, and it can be used more generally for getting all lines from a file except the last n lines.
grep -n "" myfile.txt \ # output the line number for each line
| sort -nr \ # reverse the file by using those line numbers
| sed '1,4d' \ # delete first 4 lines (last 4 of the original file)
| sort -n \ # reverse the reversed file (correct the line order)
| sed 's/^[0-9]*://' # remove the added line numbers
Here is the above command in an actual single line and runnable (can't execute the above due to the added comments):
grep -n "" myfile.txt | sort -nr | sed '1,4d' | sort -n | sed 's/^[0-9]*://'
It's a little cumbersome, and this problem can be solved with more comprehensive commands like ghead, but when you can't or don't want to download such tools, it's nice to be able to do this with the more basic options. I've been in situations where it's simply not an option to get better tools.
awk 'NR>1{print p}{p=$0}'
For this job, an awk one-liner is a bit longer than a sed one.
This question already has answers here:
Closed 10 years ago.
Possible Duplicate:
bash echo number of lines of file given in a bash variable
Was wondering how you output the number of lines in a text file to screen and then store it in a variable.
I have a file called stats.txt and when I run wc -l stats.txt it outputs 8 stats.txt
I tried doing x = wc -l stats.txt thinking it would store the number only and the rest is just for visual but it does not work :(
Thanks for the help
There are two POSIX standard syntax for doing this:
x=`cat stats.txt | wc -l`
or
x=$(cat stats.txt | wc -l)
They both run the program and replace the invocation in the script with the standard output of the command, in this case assigning it to the $x variable. However, be aware that both trim ending newlines (this is actually what you want here, but can be dangerous sometimes, when you expect a newline).
Also, the second case can be easily nested (example: $(cat $(ls | head -n 1) | wc -l)). You can also do it with the first case, but it is more complex:
`cat \`ls | head -n 1\` | wc -l`
There are also quotation issues. You can include these expressions inside double-quotes, but with the back-ticks, you must continue quoting inside the command, while using the parenthesis allows you to "start a new quoting" group:
"`echo \"My string\"`"
"$(echo "My string")"
Hope this helps =)
you may try:
x=`cat stats.txt | wc -l`
or (from the another.anon.coward's comment):
x=`wc -l < stats.txt`