Averaging Shell Script giving errors - shell

I have a shell script that should take the average of several data files and create a new file. Here is a copy of the script:
#! /bin/bash
cat *thist > tmp.dat
> aves.txt
nx=10
ss=5
for i in $(seq 1 $nx)
do
a=$i*2-1
export dummy=$(awk 'NR=='$a' {print $1}' tmp.dat)
awk '$1=='$dummy' {print $5}' tmp.dat > $dummy.dat
export ave=$(awk 'NR>='$ss' {sum+=$1 b++} END {print sum/b}' $dummy.dat)
echo $dummy $ave >> aves.txt
done
rm *.dat
After reading in 100 .thist files, this is what the output file looks like:
0 545.608
4e-07 290.349
8e-07 613.883
1.2e-06 295.655
1.6e-06 310.78
2e-06 305.01
2.4e-06 300.733
2.8e-06 308.319
3.2e-06 298.728
3.6e-06 311.961
I am getting an error on lines 1 and 3, as the numbers in the second column should be between 250 and 350. I can't figure out what I am doing wrong. I have checked all the individual data files and all of the second column numbers are between 250 and 350. I have also run this script reading in only 10 files, and it seems to work just fine. I'm sorry if this is a dumb question or if it's confusing, I'm pretty new to shell scripts. Thanks in advance for your help.

Sam, you do not post the actual errors, but it would appear your line-1 error is due to the space between #! and /bin/bash (remove it). Then to enable debugging output add set -x as line-2 (or run your script with bash -x scriptname, which will do the same thing.) Post the line and actual error that occurs.
Your line-3 error is likely due to no file matching the file glob *thist. If there are additional characters that follow thist in the filename, you will need *thist* (or *thist.txt if they all have .txt extensions).
You next line is more properly written as :> aves.txt (to truncate the file at 0).
Finally your arithmetic should be a=$((i * 2 - 1)) or not recommended, but you can use the old expr syntax a=$(expr $i \* 2 - 1) (note: you must escape the * with \*)

Related

how to check only lines that were added since last check in bash

I have a dilemma.
I need to check log files using bash script.
The script needs to run every 5-10 minutes (set it in crontab) and send email if there is a warning or error in logs.
But it has to check only lines that were added since last check and not go through the whole document again and again.
I don't know how to check only lines that were added since last check, or lines that were added in the last 10 minutes
Sleep won't work in my situation because the script shouldn't be running all the time it should be done once every 5-10 minutes
If your file is not too big, you could try storing the previous number of lines of your log file in an environment variable, and compare it with the current line number. Something like the following script should work :
#!/bin/bash
#here, detect_errors is a placeholder name for the function you already developped
#we use only the lines we haven't seen yet in our detect_errors function
detect_errors "$(head -n $(($(wc -l log.file) - OLD_LINE_COUNT)) log.file)"
#we update the new value for the $OLD_LINE_COUNT
export OLD_LINE_COUNT="$(wc -l log.file)"
DETAILED EXPLANATION
head -n X myfile : displays the first X lines of myfile
$(( ... )) : arithmetic calculations in bash
wc -l log.file : line count for our file
$OLD_LINE_COUNT : environement variable where we store the line count of the previous iteration (equals to 0 when we first launch the script)
If the file is unchanged, $(wc -l log.file) - OLD_LINE_COUNT will return 0, and head -n 0 returns empty.
If the log file is too big, wc -l will take a lot of time, so in this case my method wouldn't be recomended.
EDIT: I have not asked how your log file is incremented. If the additional lines are added AT THE END OF THE FILE, you should use tail -n instead of head -n

bash script to read my files and execute the command

I have a script which one can run as ./analyze file.txt file.root log.txt
file.txt is input file which contains executable root files with their paths, others are output. My problem is I have almost 30 input files and i do not want to write down the command each time to run the code. Bash script would be nice but I did not manage. It gives an error. see an example of the code below, I try to run:
#!/bin/bash
do
echo
./analyze runlist1.txt runlist1.root log1.txt
./analyze runlist2.txt runlist2.root log.txt
./analyze runlist3.txt runlist3.root log.txt
./analyze runlist4.txt runlist4.root res4.txt
done
I get the error "syntax error near unexpected token `do' ", Any help would be appreciated.
When all files are called Brunsplit.txt, the following will help
for file in Brunsplit*.txt; do
tmp=${file%.txt}
nr=${tmp#Brunsplit}
./analyze ${file} runlist${nr}.root res${nr}.txt
done
The tmp and nr vars are filled with special syntax, it is something like cutting off .txt at the end and Brunsplit from the start.
Like others indicated in comments, the do out of context is a syntax error (it is a valid keyword right after a for, while, and until control statement).
Apparently, there is no systematic mapping between input file names and the corresponding log file names, so you need a script which maintains this mapping. Something like this, then:
while read suffix logfile; do
./analyze "runlist$suffix.txt" "runlist$suffix.root" "$logfile"
done <<'____HERE'
1 log1.txt
2 log.txt
3 log.txt
4 res4.txt
____HERE
The here document (the stuff between << delimiter and the delimiter alone on a line) is just like a text file, except it is embedded as part of the script.
Check it out the primitive but the modified version.
!/bin/bash
for file in Brunsplit1.txt Brunsplit2.txt
do
echo $file
./analyze Brunsplit1.txt runlist1.root res1.txt
./analyze Brunsplit2.txt runlist2.root res2.txt
done
Thanks.

Use first 3 characters of a filename as a variable in shell script

this is my first post so hopefully I will make my question clear.
I am new to shell scripts and my task with this one is to add a new value to every line of a csv file. The value that needs added is based on the first 3 digits of the filename.
I bit of background. The csv files I am receiving are eventually being loaded into partitioned oracle tables. The start of the file name (e.g. BATTESTFILE.txt) contains the partitioned site so I need to write a script that takes the first 3 characters of the filename (in this example BAT) and add this to the end of each line of the file.
The closest I have got so far is when I stripped the code to the bare basics of what I need to do:
build_files()
{
OLDFILE=${filename[#]}.txt
NEWFILE=${filename[#]}.NEW.txt
ABSOLUTE='path/scripts/'
FULLOLD=$ABSOLUTE$OLDFILE
FULLNEW=$ABSOLUTE$NEWFILE
sed -e s/$/",${j}"/ "${FULLOLD}" > "${FULLNEW}"
}
set -A site 'BAT'
set -A filename 'BATTESTFILE'
for j in ${site[#]}; do
for i in ${filename[#]}; do
build_files ${j}
done
done
Here I have set up an array site as there will be 6 'sites' and this will make it easy to add additionals sits to the code as the files come through to me. The same is to be siad for the filename array.
This codes works, but it isn't as automated as I need. One of my most recent attempts has been below:
build_files()
{
OLDFILE=${filename[#]}.txt
NEWFILE=${filename[#]}.NEW.txt
ABSOLUTE='/app/dss/dsssis/sis/scripts/'
FULLOLD=$ABSOLUTE$OLDFILE
FULLNEW=$ABSOLUTE$NEWFILE
sed -e s/$/",${j}"/ "${FULLOLD}" > "${FULLNEW}"
}
set -A site 'BAT'
set -A filename 'BATTESTFILE'
for j in ${site[#]}; do
for i in ${filename[#]}; do
trust=echo "$filename" | cut -c1-3
echo "$trust"
if ["$trust" = 'BAT']; then
${j} = 'BAT'
fi
build_files ${j}
done
done
I found the code trust=echo "$filename" | cut -c1-3 through another question on StackOverflow as I was researching, but it doesn't seem to work for me. I added in the echo to test what trust was holding, but it was empty.
I am getting 2 errors back:
Line 17 - BATTESTFILE: not found
Line 19 - test: ] missing
Sorry for the long winded questions. Hopefully It contains helpful info and shows the steps I have taken. Any questions, comment away. Any help or guidance is very much appreciated. Thanks.
When you are new with shells, try avoiding arrays.
In an if statement use spaces before and after the [ and ] characters.
Get used to surrounding your shell variables with {} like ${trust}
I do not know how you fill your array, when the array is hardcoded, try te replace with
SITE=file1
SITE="${SITE} file2"
And you must tell unix you want to have the rightside eveluated with $(..) (better than backtics):
trust=$(echo "${filename}" | cut -c1-3)
Some guidelines and syntax help can be found at Google
Just use shell parameter expansion:
$ var=abcdefg
$ echo "${var:0:3}"
abc
Assuming you're using a reasonably capable shell like bash or ksh, for example
Just in case it is useful for anyone else now or in the future, I got my code to work as desired by using the below. Thanks Walter A below for his answer to my main problem of getting the first 3 characters from the filename and using them as a variable.
This gave me the desired output of taking the first 3 characters of the filename, and adding them to the end of each line in my csv file.
## Get the current Directory and file name, create a new file name
build_files()
{
OLDFILE=${i}.txt
NEWFILE=${i}.NEW.txt
ABSOLUTE='/app/dss/dsssis/sis/scripts/'
FULLOLD=$ABSOLUTE$OLDFILE
FULLNEW=$ABSOLUTE$NEWFILE
## Take the 3 characters from the filename and
## add them onto the end of each line in the csv file.
sed -e s/$/";${j}"/ "${FULLOLD}" > "${FULLNEW}"
}
## Loop to take the first 3 characters from the file names held in
## an array to be added into the new file above
set -A filename 'BATTESTFILE'
for i in ${filename[#]}; do
trust=$(echo "${i}" | cut -c1-3)
echo "${trust}"
j="${trust}"
echo "${i} ${j}"
build_files ${i} ${j}
done
Hope this is useful for someone else.

Get output filename in Bash Script

I would like to get just the filename (with extension) of the output file I pass to my bash script:
a=$1
b=$(basename -- "$a")
echo $b #for debug
if [ "$b" == "test" ]; then
echo $b
fi
If i type in:
./test.sh /home/oscarchase/test.sh > /home/oscarchase/test.txt
I would like to get:
test.txt
in my output file but I get:
test.sh
How can I procede to parse this first argument to get the right name ?
Try this:
#!/bin/bash
output=$(readlink /proc/$$/fd/1)
echo "output is performed to \"$output\""
but please remember that this solution is system-dependent (particularly for Linux). I'm not sure that /proc filesystem has the same structure in e.g. FreeBSD and certainly this script won't work in bash for Windows.
Ahha, FreeBSD obsoleted procfs a while ago and now has a different facility called procstat. You may get an idea on how to extract the information you need from the following screenshot. I guess some awk-ing is required :)
Finding out the name of the file that is opened on file descriptor 1 (standard output) is not something you can do directly in bash; it depends on what operating system you are using. You can use lsof and awk to do this; it doesn't rely on the proc file system, and although the exact call may vary, this command worked for both Linux and Mac OS X, so it is at least somewhat portable.
output=$( lsof -p $$ -a -d 1 -F n | awk '/^n/ {print substr($1, 2)}' )
Some explanation:
-p $$ selects open files for the current process
-d 1 selects only file descriptor 1
-a is use to require both -p and -d apply (the default is to show all files that match either condition
-F n modifies the output so that you get one line per field, prefixed with an identifier character. With this, you'll get two lines: one beginning with p and indicating the process ID, and one beginning with `n indicating the file name of the file.
The awk command simply selects the line starting with n and outputs the first field minus the initial n.

Weird characters when I print a number into a file (bash shell)

I have this line in my script.sh
printf "%d" "$endMS_line"
$endMS_line is a number. I get that number with
endMS_line=`cat file | awk '{if($1=='"$variable"') print NR}'`
And to print it I use
printf "%d" "$endMS_line"
or
echo $endMS_line
So everything works perfectly in the standard output. The problem is when I want to save that number into a file (because I want to use the result in another script, may be there is a clever way to do it than write a file and then read the number from the file, etc..)
But for now I am trying to do that. How? Well I write this in the standard output.
myscript.sh inputs > file.txt
But when I try to see the file (when I open the file) I see the result plus weird characteres
[H[2J867
The correct number in this example is 867. Anyone know how can I fix this?
Thank you!
At the begginning of the script I had the command:
clear
removing that and using:
echo "$endMS_line"
Then in the standard output:
myscript.sh input > file.txt
works perfectly.

Resources