I have a file which contains varoius data (date,time,speed, distance from the front, distance from the back), the file looks like this, just with more rows:
2003.09.23.,05:05:21:64,134,177,101
2009.03.10.,17:46:17:81,57,102,57
2018.01.05.,00:30:37:04,354,145,156
2011.07.11.,23:21:53:43,310,125,47
2011.06.26.,07:42:10:30,383,180,171
I'm trying to write a simple Bash program, which tells the dates and times when the 'distance from the front' is less than the provided parameter ($1)
So far I wrote:
#!/bin/bash
if [ $# -eq 0 -o $# -gt 1 ]
then
echo "wrong number of parameters"
fi
i=0
fdistance=()
input='auto.txt'
while IFS= read -r line
do
year=${line::4}
month=${line:5:2}
day=${line:8:2}
hour=${line:12:2}
min=${line:15:2}
sec=${line:18:2}
hthsec=${line:21:2}
fdistance=$(cut -d, -f 4)
if [ "$fdistance[$i]" -lt "$1" ]
then
echo "$year[$i]:$month[$i]:$day[$i],$hour[$i]:$min[$i]:$sec[$i]:$hthsec[$i]"
fi
i=`expr $i + 1`
done < "$input"
but this gives the error "whole expression required" and doesn't work at all.
If you have the option of using awk, the entire process can be reduced to:
awk -F, -v dist=150 '$4<dist {split($1,d,"."); print d[1]":"d[2]":"d[3]","$2}' file
Where in the example above, any record with distance (field 4, $4) less than the dist variable value takes the date field (field 1, $1) and splits() the field into the array d on "." where the first 3 elements will be year, mo, day and then simply prints the output of those three elements separated by ":" (which eliminates the stray "." at the end of the field). The time (field 2, $2) is output unchanged.
Example Use/Output
With your sample data in file, you can do:
$ awk -F, -v dist=150 '$4<dist {split($1,d,"."); print d[1]":"d[2]":"d[3]","$2}' file
2009:03:10,17:46:17:81
2018:01:05,00:30:37:04
2011:07:11,23:21:53:43
Which provides the records in the requested format where the distance is less than 150. If you call awk from within your script you can pass the 150 in from the 1st argument to your script.
You can also accomplish this task by substituting a ':' for each '.' in the first field with gsub() and outputting a substring of the first field with substr() that drops the last character, e.g.
awk -F, -v dist=150 '$4<dist {gsub(/[.]/,":",$1); print substr($1,0,length($1)-1),$2}' file
(same output)
While parsing the data is a great exercise for leaning string handling in shell or bash, in practice awk will be Orders of Magnitude faster than a shell script. Processing a million line file -- the difference in runtime can be seconds with awk compared to minutes (or hours) with a shell script.
If this is an exercise to learn string handling in your shell, just put this in your hip pocket for later understanding that awk is the real Swiss Army-Knife for text processing. (well worth the effort to learn)
Would you try the following:
#/bin/bash
if (( $# != 1 )); then
echo "usage: $0 max_distance_from_the_front" >& 2 # output error message to the stderr
exit 1
fi
input="auto.txt"
while IFS=, read -r mydate mytime speed fdist bdist; do # split csv and assign variables
mydate=${mydate%.}; mydate=${mydate//./:} # reformat the date string
if (( fdist < $1 )); then # if the front disatce is less than $1
echo "$mydate,$mytime" # then print the date and time
fi
done < "$input"
Sample output with the same parameter as Keldorn:
$ ./test.sh 130
2009:03:10,17:46:17:81
2011:07:11,23:21:53:43
There are a few odd things in your script:
Why is fdistance an array. It is not necessary (and here done wrong) since the file is read line by line.
What is the cut of the line fdistance=$(cut -d, -f 4) supposed to cut, what's the input?
(Note: When invalid parameters, better end the script right away. Added in the example below.)
Here is a working version (apart from the parsing of the date, but that is not what your question was about so I skipped it):
#!/usr/bin/env bash
if [ $# -eq 0 -o $# -gt 1 ]
then
echo "wrong number of parameters"
exit 1
fi
input='auto.txt'
while IFS= read -r line
do
fdistance=$(echo "$line" | awk '{split($0,a,","); print a[4]}')
if [ "$fdistance" -lt "$1" ]
then
echo $line
fi
done < "$input"
Sample output:
$ ./test.sh 130
2009.03.10.,17:46:17:81,57,102,57
2011.07.11.,23:21:53:43,310,125,47
$
Related
So I get some URL's from my other team and I need to identiy a defined pattern in that URL and save the value after the pattern inside a variable.
Can this be achieved ?
**Input file: Just an example**
https://stackoverflow.com/questions/hakuna
https://stackoverflow.com/questions/simba
I wrote a simple for loop for this purpose
for i in `cat inputFile`
do
storeVal=awk -v $i -F"questions/" '{print$2}'
echo "The Name for the day is ${storeVal}"
length=`secondScript.sh ${storeVal}`
if [[ $length -gt 10 ]]
then
thirdScript.sh ${storeVal}
elif [[ $length -lt 10 ]]
then
fourthScript.sh ${storeVal}
else
echo "The length of for ${storeVal} is undefined"
done
Desired Output:
The Name for the day is hakuna
The length of for hakuna is greater than 10
Command1 hakuna executed
Command2 hakuna executed
The Name for the day is simba
Command1 simba executed
Command2 simba executed
And extra point to be noted.
The reason why I need to store the awk cut value in a variable is because I need to use that variable in multiple places with the loop.
Since it sounds like you want to run a command for every line in the input file, you can just use the built-in functionality of the shell:
while IFS=/ read -ra pieces; do
printf '%s\n' "${pieces[#]}" # prints each piece on a separate line
done < inputFile
If you always want the last part of the url (i.e. after the last /) on each line, then you can use "${pieces[-1]}":
while IFS=/ read -ra pieces; do
variable=${pieces[-1]} # do whatever you want with this variable
printf 'The Name for the day is %s\n' "$variable" # e.g. print it
done < inputFile
Since you want to extract all the string after the questions/ bit, you can simply use shell pattern substitution for that.
while IFS= read -r line; do
storeVal=${line##*questions/}
echo "The Name for the day is ${storeVal}"
done < 'inputFile'
Sample input file:
$ cat inputFile
https://stackoverflow.com/questions/hakuna
https://stackoverflow.com/questions/simba
https://stackoverflow.com/questions/simba/lion
Output of script:
The Name for the day is hakuna
The Name for the day is simba
The Name for the day is simba/lion
I'm a bit confused by the done < $1 notation.
I'm trying to write a program "sumnums" that reads in a file called "nums" that has a couple rows of numbers. Then it should print out the rows of the numbers followed by a sum of all the numbers.
Currently I have:
#!/bin/bash
sum=0;
while read myline
do
echo "Before for; Current line: \"$myline\""
done
for i in $myline; do
sum=$(expr $sum + $i)
done < $1
echo "Total sum is: $sum"
and it outputs the list of the numbers from nums correctly then says
./sumnums: line 10: $1: ambiguous redirect, then outputs Total sum is: 0.
So somehow it isn't adding. How do I rearrange these lines to fix the program and get rid of the "ambiguous redirect"?
Assuming your filename is in $1 (that is, that your script was called with ./yourscript nums):
#!/bin/bash
[[ $1 ]] || set -- nums ## use $1 if already set; otherwise, override with "nums"
sum=0
while read -r i; do ## read from stdin (which is redirected by < for this loop)
sum=$(( sum + i )) ## ...treat what we read as a number, and add it to our sum
done <"$1" ## with stdin reading from $1 for this loop
echo "Total sum is: $sum"
If $1 doesn't contain your filename, then use something that does contain your filename in its place, or just hardcode the actual filename itself.
Notes:
<"$1" is applied to a while read loop. This is essential, because read is (in this context) the command that actually consumes content from the file. It can make sense to redirect stdin to a for loop, but only if something inside that loop is reading from stdin.
$(( )) is modern POSIX sh arithmetic syntax. expr is legacy syntax; don't use it.
awk to the rescue!
awk '{for(i=1;i<=NF;i++) sum+=$i} END{print "Total sum is: " sum}' file
bash is not the right tool for this task.
Hi I am trying to print/echo line numbers that are multiple of 5. I am doing this in shell script. I am getting errors and unable to proceed. below is the script
#!/bin/bash
x=0
y=$wc -l $1
while [ $x -le $y ]
do
sed -n `$x`p $1
x=$(( $x + 5 ))
done
When executing above script i get below errors
#./echo5.sh sample.h
./echo5.sh: line 3: -l: command not found
./echo5.sh: line 4: [: 0: unary operator expected
Please help me with this issue.
For efficiency, you don't want to be invoking sed multiple times on your file just to select a particular line. You want to read through the file once, filtering out the lines you don't want.
#!/bin/bash
i=0
while IFS= read -r line; do
(( ++i % 5 == 0 )) && echo "$line"
done < "$1"
Demo:
$ i=0; while read line; do (( ++i % 5 == 0 )) && echo "$line"; done < <(seq 42)
5
10
15
20
25
30
35
40
A funny pure Bash possibility:
#!/bin/bash
mapfile ary < "$1"
printf "%.0s%.0s%.0s%.0s%s" "${ary[#]}"
This slurps the file into an array ary, which each line of the file in a field of the array. Then printf takes care of printing one every 5 lines: %.0s takes a field, but does nothing, and %s prints the field. Since mapfile is used without the -t option, the newlines are included in the array. Of course this really slurps the file into memory, so it might not be good for huge files. For large files you can use a callback with mapfile:
#!/bin/bash
callback() {
printf '%s' "$2"
ary=()
}
mapfile -c 5 -C callback ary < "$1"
We're removing all the elements of the array during the callback, so that the array doesn't grow too large, and the printing is done on the fly, as the file is read.
Another funny possibility, in the spirit of glenn jackmann's solution, yet without a counter (and still pure Bash):
#!/bin/bash
while read && read && read && read && IFS= read -r line; do
printf '%s\n' "$line"
done < "$1"
Use sed.
sed -n '0~5p' $1
This prints every fifth line in the file starting from 0
Also
y=$wc -l $1
wont work
y=$(wc -l < $1)
You need to create a subshell as bash will see the spaces as the end of the assignment, also if you just want the number its best to redirect the file into wc.
Dont know what you were trying to do with this ?
x=$(( $x + 5 ))
Guessing you were trying to use let, so id suggest looking up the syntax for that command. It would look more like
(( x = x + 5 ))
Hope this helps
There are cleaner ways to do it, but what you're looking for is this.
#!/bin/bash
x=5
y=`wc -l $1`
y=`echo $y | cut -f1 -d\ `
while [ "$y" -gt "$x" ]
do
sed -n "${x}p" "$1"
x=$(( $x + 5 ))
done
Initialize x to 5, since there is no "line zero" in your file $1.
Also, wc -l $1 will display the number of line counts, followed by the name of the file. Use cut to strip the file name out and keep just the first word.
In conditionals, a value of zero can be interpreted as "true" in Bash.
You should not have space between your $x and your p in your sed command. You can put them right next to each other using curly braces.
You can do this quite succinctly using awk:
awk 'NR % 5 == 0' "$1"
NR is the record number (line number in this case). Whenever it is a multiple of 5, the expression is true, so the line is printed.
You might also like the even shorter but slightly less readable:
awk '!(NR%5)' "$1"
which does the same thing.
I'm gonna tear my hair out: I have this script:
#!/bin/bash
if [[ $# -eq 2 ]]
then
total=0
IFS=' '
while read one two; do
total=$((total+two))
done < $2
echo "Total: $total"
fi
Its supposed to add up my gas receipts I have saved in a file in this format:
3/9/13 21.76
output:
./getgas: line 9: 21.76: syntax error: invalid arithmetic operator (error token is ".76")
I read online that its possible to do float math in bash, and I found an an example script that works and it has:
function float_eval()
{
local stat=0
local result=0.0
if [[ $# -gt 0 ]]; then
result=$(echo "scale=$float_scale; $*" | bc -q 2>/dev/null)
stat=$?
if [[ $stat -eq 0 && -z "$result" ]]; then stat=1; fi
fi
echo $result
return $stat
}
which looks awesome, and runs no problem
WTF is going on here. I can easily do this is C but this crap is making me mad
EDIT: I don't anything about awk. It looks promising but I don't even know how to run those one-liners you guys posted
awk '{ sum += $2 } END { printf("Total: %.2f\n", sum); }' $2
Add up column 2 (that's the $2 in the awk script) of the file named by shell script argument $2 (rife with opportunities for confusion) and print the result at the end.
I don't [know] anything about awk. It looks promising but I don't even know how to run those one-liners you guys posted.
In the context of your script:
#!/bin/bash
if [[ $# -eq 2 ]]
then
awk '{ sum += $2 } END { printf("Total: %.2f\n", sum); }' $2
else
echo "Usage: $0 arg1 receipts-file" >&2; exit 1
fi
Or just write it on the command line, substituting the receipts file name for the $2 after the awk command. Or leave that blank and redirect from the file. Or type the dates and values in. Or, …
Your script demands two arguments, but doesn't use the first one, which is a bit puzzling.
As noted in the comments, you could simplify that to:
#!/bin/bash
exec awk '{ sum += $2 } END { printf("Total: %.2f\n", sum) }' "$#"
Or even use the shebang to full power:
#!/usr/bin/awk -f
{ sum += $2 }
END { printf("Total: %.2f\n", sum) }
The kernel will execute awk for you, and that's the awk script written out as a two line program. Of course, if awk is in /bin/awk, then you have to fix the shebang line; the shell looks in many places for awk and will probably find it. So there are advantages to sticking with a shell script. Both these revisions simply sum what's on standard input if there are no files specified, or what is in all the files specified if there is one or more files specified on the command line.
In bash you can only operate on integers. The example script you posted uses bc which is an arbitrary-precision calculation, included with most UNIX-like OS-es. So the script prepares an expression and pipes it to bc (the initial scale=... expression configures the number of significant digits bc should display.
A simplified example would be:
echo -e 'scale=2\n1.234+5.67\nquit' | bc
You could also use awk:
awk 'BEGIN{print 1.234+5.67}'
My bash script doesn't work the way I want it to:
#!/bin/bash
total="0"
count="0"
#FILE="$1" This is the easier way
for FILE in $*
do
# Start processing all processable files
while read line
do
if [[ "$line" =~ ^Total ]];
then
tmp=$(echo $line | cut -d':' -f2)
count=$(expr $count + 1)
total=$(expr $total + $tmp)
fi
done < $FILE
done
echo "The Total Is: $total"
echo "$FILE"
Is there another way to modify this script so that it reads arguments into $1 instead of $FILE? I've tried using a while loop:
while [ $1 != "" ]
do ....
done
Also when I implement that the code repeats itself. Is there a way to fix that as well?
Another problem that I'm having is that when I have multiple files hi*.txt it gives me duplicates. Why? I have files like hi1.txt hi1.txt~ but the tilde file is of 0 bytes, so my script shouldn't be finding anything.
What i have is fine, but could be improved. I appreciate your awk suggestions but its currently beyond my level as a unix programmer.
Strager: The files that my text editor generates automatically contain nothing..it is of 0 bytes..But yeah i went ahead and deleted them just to be sure. But no my script is in fact reading everything twice. I suppose its looping again when it really shouldnt. I've tried to silence that action with the exit commands..But wasnt successful.
while [ "$1" != "" ]; do
# Code here
# Next argument
shift
done
This code is pretty sweet, but I'm specifying all the possible commands at one time. Example: hi[145].txt
If supplied would read all three files at once.
Suppose the user enters hi*.txt;
I then get all my hi files read twice and then added again.
How can I code it so that it reads my files (just once) upon specification of hi*.txt?
I really think that this is because of not having $1.
It looks like you are trying to add up the totals from the lines labelled 'Total:' in the files provided. It is always a good idea to state what you're trying to do - as well as how you're trying to do it (see How to Ask Questions the Smart Way).
If so, then you're doing in about as complicated a way as I can see. What was wrong with:
grep '^Total:' "$#" |
cut -d: -f2 |
awk '{sum += $1}
END { print sum }'
This doesn't print out "The total is" etc; and it is not clear why you echo $FILE at the end of your version.
You can use Perl or any other suitable program in place of awk; you could do the whole job in Perl or Python - indeed, the cut work could be done by awk:
grep "^Total:" "$#" |
awk -F: '{sum += $2}
END { print sum }'
Taken still further, the whole job could be done by awk:
awk -F: '$1 ~ /^Total/ { sum += $2 }
END { print sum }' "$#"
The code in Perl wouldn't be much harder and the result might be quicker:
perl -na -F: -e '$sum += $F[1] if m/^Total:/; END { print $sum; }' "$#"
When iterating over the file name arguments provided in a shell script, you should use '"$#"' in place of '$*' as the latter notation does not preserve spaces in file names.
Your comment about '$1' is confusing to me. You could be asking to read from the file whose name is in $1 on each iteration; that is done using:
while [ $# -gt 0 ]
do
...process $1...
shift
done
HTH!
If you define a function, it'll receive the argument as $1. Why is $1 more valuable to you than $FILE, though?
#!/bin/sh
process() {
echo "doing something with $1"
}
for i in "$#" # Note use of "$#" to not break on filenames with whitespace
do
process "$i"
done
while [ "$1" != "" ]; do
# Code here
# Next argument
shift
done
On your problem with tilde files ... those are temporary files created by your text editor. Delete them if you don't want them to be matched by your glob expression (wildcard). Otherwise, filter them in your script (not recommended).