So I am building a script to check for files with certain errors in a bunch of files, based on output from an SQL DB. The file with the error shall be sent to me via mail.
The problem is that when I try to send the mail, I get the message
"script.sh: 9: mutt: not found" Which does not occur, if I send the mail before the PATH variable is created.
The script looks as following:
JOB=$(sudo cat /tmp/sqltest.txt | awk '{ print $5 }')
DATE=$(sudo cat /tmp/sqltest.txt | awk '{ print $1 }')
CODE=$(sudo cat /tmp/sqltest.txt | awk '{ print $3 }')
PATH=$(grep ${CODE} /tmp/unzip/* | awk '{ print $1 }' | cut -d':' -f1 | head -n 1)
echo "File containing error message for job "${JOB}" at "${DATE}"" | mutt -a "/tmp/sqltest.txt" -s "Mail title" -- <mail#address>
In short, grep finds the file where the error code is, awk picks out the column with the path to the file, the column also comes with a timestamp which cut removes and head ensures that I only get one result, if the error is reported several places.
I can send the mail with mutt if I use it after variable CODE, instead of PATH, though I unfortunately need PATH instead of /tmp/sqltest.txt
Do you have any ideas on what might cause this?
What we got here is a classic case of trying to use an environment variable (and a pretty important one !) : just use another variable name to get rid of the error. As some suggested, it is good practice to try to avoid full-uppercase variables.
Environment Variables
There is a couple of environment variables inside Bash, PATH being one of it.
You can get the list of both environment and shell variables using the set command.
Source :
Environment Variable on Wikipedia
You can simply add the output obtained by command substitution to the PATH.
Change the line setting the PATH to
PATH=$(grep ${CODE} /tmp/unzip/* | awk '{ print $1 }' | cut -d':' -f1
| head -n 1):${PATH}
The change in PATH would be valid for the duration of the script.
Related
This question already has answers here:
How to pass parameters to a Bash script?
(4 answers)
Closed 1 year ago.
At the beginning I have a file.txt, which contains several informations that I will take using the grep command as you see in the script.
What I want is to give the script the file I want instead of file.txt but without changing the file name each time in the script for example if the file is named Me.txt I don’t want to go into the script and write Me.txt in each grep command especially if I have dozens of orders.
Is there a way to do this?
#!/bin/bash
grep teste file.txt > testline.txt
awk '{print $2}' testline.txt > test.txt
echo '#'
echo '#'
grep remote file.txt > remoteline.txt
awk '{print $3}' remoteline.txt > remote.txt
echo '#'
echo '#'
grep adresse file.txt > adresseline.txt
awk '{print $2}' adresseline.txt > adresse.txt
Using a parameter, as many contributors here suggested, is of course the obvious approach, and the one which is usually taken in such case, so I want to extend this idea:
If you do it naively as
filename=$1
you have to supply the name on every invocation. You can improve on this by providing a default value for the case the parameter is missing:
filename=${1:-file.txt}
But sometimes you are in a situation, where for some time (working on a specific task), you always need the same filename over and over, and the default value happens to be not the one you need. Another possibility to pass information to a program is via the environment. If you set the filename by
filename=${MOOFOO:-file.txt}
it means that - assuming your script is called myscript.sh - if you invoke your script by
MOOFOO=myfile.txt myscript.sh
it uses myfile.txt, while if you call it by
myscript.sh
it uses the default file.txt. You can also set MOOFOO in your shell, as
export MOOFOO=myfile.txt
and then, even a lone execution of
myscript.sh
with use myfile.txt instead of the default file.txt
The most flexible approach is to combine both, and this is what I often do in such a situation. If you do in your script a
filename=${1:-${MOOFOO:-file.txt}}
it takes the name from the 1st parameter, but if there is no parameter, takes it from the variable MOOFOO, and if this variable is also undefined, uses file.txt as the last fallback.
You should pass the filename as a command line parameter so that you can call your script like so:
script <filename>
Inside the script, you can access the command line parameters in the variables $1, $2,.... The variable $# contains the number of command line parameters passed to the script, and the variable $0 contains the path of the script itself.
As with all variables, you can choose to put the variable name in curly brackets which has advantages sometimes: ${1}, ${2}, ...
#!/bin/bash
if [ $# = 1 ]; then
filename=${1}
else
echo "USAGE: $(basename ${0}) <filename>"
exit 1
fi
grep teste "${filename}" > testline.txt
awk '{print $2}' testline.txt > test.txt
echo '#'
echo '#'
grep remote "${filename}" > remoteline.txt
awk '{print $3}' remoteline.txt > remote.txt
echo '#'
echo '#'
grep adresse "${filename}" > adresseline.txt
awk '{print $2}' adresseline.txt > adresse.txt
By the way, you don't need two different files to achieve what you want, you can just pipe the output of grep straight into awk, e.g.:
grep teste "${filename}" | awk '{print $2}' > test.txt
but then again, awk can do the regex match itself, reducing it all to just one command:
awk '/teste/ {print $2}' "${filename}" > test.txt
I am building a shell script.
The script gets git log such as:
"Updating 2f8b547d..eb94967a Fast-forward...."
but I want to get 2f8b547d..eb94967a snippet.
I am a new one for the shell. So, Thanks for you help.
Update:
For the more, I want use the snippet as a param. Because I will excute
git log 2f8b547d..eb94967a
You can pipe it to awk like so:
echo "Updating 2f8b547d..eb94967a Fast-forward...." | awk '{print $2}'
Your result will be 2f8b547d..eb94967a.
If it is a script, say, abc.sh that had such output, then you can run:
$> ./abc.sh | awk '{print $2}'
awk takes the output and splits the information by space. Updating is represented with $1. 2f8b547d..eb94967a is $2 and so on. In the above example, we ask awk to print out the 2nd item in the output.
As an alternative to awk (don't get me wrong, awk is super for this job as well), you can simply use cut with a space delimiter extract the second field, e.g.
cut -d' ' -f2 yourgit.log
You can also pipe output to cut or redirect the input file to it using < as well. It essentially does the same as the awk command, it just being a different choice of which utility to use.
Here another alternative:
echo "Updating 2f8b547d..eb94967a Fast-forward...." | read u hash rest
After this, the string you are looking for is stored on the variable hash:
echo $hash
I have a script that read log files and parse the data to insert them to mysql table..
My script looks like
while read x;do
var=$(echo ${x}|cut -d+ -f1)
var2=$(echo ${x}|cut -d_ -f3)
...
echo "$var,$var2,.." >> mysql.infile
done<logfile
The Problem is that log files are thousands of lines and taking hours....
I read that awk is better, I tried, but don't know the syntax to parse the variables...
EDIT:
inputs are structure firewall logs so they are pretty large files like
#timestamp $HOST reason="idle Timeout" source-address="x.x.x.x"
source-port="19219" destination-address="x.x.x.x"
destination-port="53" service-name="dns-udp" application="DNS"....
So I'm using a lot of grep for ~60 variables e.g
sourceaddress=$(echo ${x}|grep -P -o '.{0,0}
source-address=\".{0,50}'|cut -d\" -f2)
if you think perl will be better I'm open to suggestions and maybe a hint how to script it...
To answer your question, I assume the following rules of the game:
each line contains various variables
each variable can be found by a different delimiter.
This gives you the following awk script :
awk 'BEGIN{OFS=","}
{ FS="+"; $0=$0; var=$1;
FS="_"; $0=$0; var2=$3;
...
print var1,var2,... >> "mysql.infile"
}' logfile
It basically does the following :
set the output separator to ,
read line
set the field separator to +, re-parse the line ($0=$0) and determine the first variable
set the field separator to '_', re-parse the line ($0=$0) and determine the second variable
... continue for all variables
print the line to the output file.
The perl script below might help:
perl -ane '/^[^+]*/;printf "%s,",$&;/^([^_]*_){2}([^_]*){1ntf "%s\n",$+' logfile
Since, $& can result in performance penalty, you could also use the /p modifier like below :
perl -ane '/^[^+]*/p;printf "%s,",${^MATCH};/^([^_]*_){2}([^_]*){1}_.*/;printf "%s\n",$+' logfile
For more on perl regex matching refer to [ PerlDoc ]
if you're extracting the values in order, something like this will help
$ awk -F\" '{for(i=2;i<=NF;i+=2) print $i}' file
idle Timeout
x.x.x.x
19219
x.x.x.x
53
dns-udp
DNS
you can easily change the output format as well
$ awk -F\" -v OFS=, '{for(i=2;i<=NF;i+=2)
printf "%s", $i ((i>NF-2)?ORS:OFS)}' file
idle Timeout,x.x.x.x,19219,x.x.x.x,53,dns-udp,DNS
I have an input file that contains several path, including one referring to a initial solution. Corresponding line is the following:
initial_solution_file = ../../INIT/foo
What I would like to do is having an alias that would display this path so that I would type "init" and the shell would return " the initial solution is: ../../INIT/foo"
What I have tried is:
grep initial_solution_file input_file | awk '{print $3}' | echo "the initial solution is:" `xargs echo`
It provides the desired output, but I additionaly get something like:
[6] 48201 48202
What is this and how to prevent it from happening ?
Thanks in advance
echo "the initial solution is: $(awk '/initial_solution_file/{print $3}' input_file)"
the initial solution is: ../../INIT/foo
There is no need of pipes , you can do command substitution by using $(....) construct. Also, grep and awk can be done by awk alone.
I am trying to get the full path of a files within a directory. So far this is what I have in bash.
prefix="s3://${s3_bucket}/${s3_folder}/$(date --date="$i days ago" +"%Y/%m/%d")/"
#echo $prefix
aws s3 ls s3://${s3_bucket}/${s3_folder}/$(date --date="$i days ago" +"%Y/%m/%d")/ | sed -n 's/.*\([0-9][0-9]-h.*gz\)/\1/p' | awk '$0="${prefix}"$0' >> ${s3_files_1}
In my output, I am getting the following:
${prefix}file1.gz
${prefix}file2.gz
The output I am looking for is something like below.
s3://my_bucket/my_folder/file1.gz
s3://my_bucket/my_folder/file2.gz
My issue is with the way the awk command is interpreting the variable ${prefix}. Can anyone please help?
You can use -v to pass shell variable contents to awk:
prefix="s3://my_bucket/my_folder/"
echo "file1.gz" | awk -v myprefix="${prefix//\\/\\\\}" '{ print myprefix $0 }'
Sadly, awk -v is not data safe. This example uses parameter expansion to escape backslashes to avoid them being mangled.