how to parse a config file (*.conf) in shell script? - shell

I am new to shell script. I have a file app.conf as :
[MySql]
user = root
password = root123
domain = localhost
database = db_name
port = 3306
[Logs]
level = logging.DEBUG
[Server]
port = 8080
I want to parse this file in shell script and want to extract mysql credentials from the same. How can I achieve that?

I'd do this:
pw=$(awk '/^password/{print $3}' app.conf)
user=$(awk '/^user/{print $3}' app.conf)
echo $pw
root123
echo $user
root
The $() sets the variable pw to the output of the command inside. The command inside looks through your app.conf file for a line starting password and then prints the 3rd field in that line.
EDITED
If you are going to parse a bunch of values out of your config file, I would make a variable for the config file name:
CONFIG=app.conf
pw=$(awk '/^password/{print $3}' "${CONFIG}")
user=$(awk '/^user/{print $3}' "${CONFIG}")
Here's how to do the two different ports... by setting a flag to 1 when you come to the right section and exiting when you find the port.
mport=$(awk '/^\[MySQL\]/{f=1} f==1&&/^port/{print $3;exit}' "${CONFIG}")
sport=$(awk '/^\[Server\]/{f=1} f==1&&/^port/{print $3;exit}' "${CONFIG}")

You will want to search for "shell ini file parser". I would start with something like this:
ini_get () {
awk -v section="$2" -v variable="$3" '
$0 == "[" section "]" { in_section = 1; next }
in_section && $1 == variable {
$1=""
$2=""
sub(/^[[:space:]]+/, "")
print
exit
}
in_section && $1 == "" {
# we are at a blank line without finding the var in the section
print "not found" > "/dev/stderr"
exit 1
}
' "$1"
}
mysql_user=$( ini_get app.conf MySql user )

Using awk:
awk -F ' *= *' '$1=="user"||$1=="password"{print $2}' my.cnf
root
gogslab

I ran in a similar problem yesterday and thought the best solution might be, if you get an associative array like "key - value" after parsing the file.
I you like to see a running example have a look at https://github.com/philippkemmeter/set-resolution/blob/master/set-resolution.
Adapted to your problem, this might work:
function receive_assoc_declare_statement {
awk -F '=' 'BEGIN {ORS=" "}
{
gsub(/[ \t]+/, "", $1);
gsub(/[ \t]+/, "", $2);
print "[" $1 "]=" $2
}' app.conf
}
eval 'declare -A CONF=('`receive_assoc_declare_statement`')'
You then have access to for instance user via ${CONF[user]}.
The gsub is trimming keys and values, so that you can use tab etc. to format your config file.
It's lacking sections, but you could add this functionality using sed to create one config array per section:
sed -n '/\[MySql\]/, /\[/ {p}' test.removeme | sed '1 d; $ d'
So answering your question in total, this script might work:
MYSQL=`sed -n '/\[MySql\]/, /\[/ {p}' app.conf | sed '1 d; $ d' | awk -F '=' 'BEGIN {ORS=" "}
{
gsub(/[ \t]+/, "", $1);
gsub(/[ \t]+/, "", $2);
print "[" $1 "]=" $2
}' `
eval 'declare -A MYSQL=('$MYSQL')'
The other sections correspondingly.

Related

Extract specific substring in shell

I have a file which contains following line:
ro fstype=sd timeout=10 console=ttymxc1,115200 show=true
I'd like to extract and store fstype attribue "sd" in a variable.
I did the job using bash
IFS=" " read -a args <<< file
for arg in ${args[#]}; do
if [[ "$arg" =~ "fstype" ]]; then
id=$(cut -d "=" -f2 <<< "$arg")
echo $id
fi
done
and following awk command in another shell script:
awk -F " " '{print $2}' file | cut -d '=' -f2
Because 'fstype' argument position and file content can differ, how to do the same things and keep compatibility in shell script ?
Could you please try following.
awk 'match($0,/fstype=[^ ]*/){print substr($0,RSTART+7,RLENGTH-7)}' Input_file
OR more specifically to handle any string before = try following:
awk '
match($0,/fstype=[^ ]*/){
val=substr($0,RSTART,RLENGTH)
sub(/.*=/,"",val)
print val
val=""
}
' Input_file
With sed:
sed 's/.*fstype=\([^ ]*\).*/\1/' Input_file
awk code's explanation:
awk ' ##Starting awk program from here.
match($0,/fstype=[^ ]*/){ ##Using match function to match regex fstype= till first space comes in current line.
val=substr($0,RSTART,RLENGTH) ##Creating variable val which has sub-string of current line from RSTART to till RLENGTH.
sub(/.*=/,"",val) ##Substituting everything till = in value of val here.
print val ##Printing val here.
val="" ##Nullifying val here.
}
' Input_file ##mentioning Input_file name here.
Any time you have tag=value pairs in your data I find it best to start by creating an array (f[] below) that maps those tags (names) to their values:
$ awk -v tag='fstype' -F'[ =]' '{for (i=2;i<NF;i+=2) f[$i]=$(i+1); print f[tag]}' file
sd
$ awk -v tag='console' -F'[ =]' '{for (i=2;i<NF;i+=2) f[$i]=$(i+1); print f[tag]}' file
ttymxc1,115200
With the above approach you can do whatever you like with the data just by referencing it by it's name as the index in the array, e.g.:
$ awk -F'[ =]' '{
for (i=2;i<NF;i+=2) f[$i]=$(i+1)
if ( (f["show"] == "true") && (f["timeout"] < 20) ) {
print f["console"], f["fstype"]
}
}' file
ttymxc1,115200 sd
If your data has more than 1 row and there can be different fields on each row (doesn't appear to be true for your data) then add delete f as the first line of the script.
If the key and value can be matched by the regex fstype=[^ ]*, grep and -o option which extracts matched pattern can be used.
$ grep -o 'fstype=[^ ]*' file
fstype=sd
In addition, regex \K can be used with -P option (please make sure this option is only valid in GNU grep).
Patterns that are to the left of \K are not shown with -o.
Therefore, below expression can extract the value only.
$ grep -oP 'fstype=\K[^ ]*' file
sd

sed unterminated 's' command modify line of file

I'm trying to modify a groups.tsv file (I'm on repl.it so path to file is fine).
Each line in the file looks like this:
groupname \t amountofpeople \t lastadded
and I'm trying to count the occurences of both groupname($nomgrp) and a login($login), and change lastadded to login.
varcol2=$(grep "$nomgrp" groups | cut "-d " -f2- | awk -F"\t" '{print $2}' )
((varcol21=varcol2+1));
varcol3=$(awk -F"\t" '{print $3}' groups)
sed -i "s|${nomgrp}\t${varcol2}\t$varcol3|${nomgrp}\t${varcol21}\t${login}|" groups
However, I'm getting the error message:
sed : -e expression #1, char 27: unterminated 's' command
The groups file has lines such as " sudo 2 user1" (delimited with a tab): a user inputs "user" which is stored in $login, then "sudo" which is stored in $nomgrp.
What am I doing wrong?
Sorry if this has been answered/super easy to fix, I'm quite the newbie here...
If I understand what you are trying to do correctly and if you have GNU awk, you could do
gawk -i inplace -F '\t' -v group="$nomgrp" -v login="$login" -v OFS='\t' '$1 == group { $2 = $2 + 1; $3 = login; } { print }' groups.tsv
Example:
$ cat groups.tsv
wheel 1000 2019-12-10
staff 1234 2019-12-11
users 9001 2019-12-12
$ gawk -i inplace -F '\t' -v group=wheel -v login=2019-12-12 -v OFS='\t' '$1 == group { $2 = $2 + 1; $3 = login; } 1' groups.tsv
$ cat groups.tsv
wheel 1001 2019-12-12
staff 1234 2019-12-11
users 9001 2019-12-12
This works as follows:
-i inplace is a GNU awk extension that allows you to change a file in place,
-F '\t' sets the input field separator to a tab so that the input is interpreted as TSV and fields with spaces in them are not split apart,
-v variable=name sets an awk variable for use in awk's code,
specifically, -v OFS='\t' sets the output field separator variable to a tab, so that the output is again a TSV
So we set variables group, login to your shell variables and ensure that awk outputs a TSV. The code then works as follows:
$1 == group { # If the first field in a line is equal to the group variable
$2 = $2 + 1; # add 1 to the second field
$3 = login; # and overwrite the third with the login variable
}
{ # in all lines:
print # print
}
{ print } could also be abbreviated as 1, I'm sure people someone will point out, but I find this way easier to explain.
If you do not have GNU awk, you could achieve the same with a temporary file, e.g.
awk -F '\t' -v group="$nomgrp" -v login="$login" -v OFS='\t' '$1 == group { $2 = $2 + 1; $3 = login; } { print }' groups.tsv > groups.tsv.new
mv groups.tsv.new groups.tsv

nslookup for IP & replace the result FQDN with IP

Requirement
I have a txt file in which last column have URLs.
Some of the URL entries have IPs instead of FQDN.
So, for entries with IPs (e.g. url=https://174.37.243.85:443*), I need to do reverse nslookup for IP and replace the result (FQDN) with IP.
Text File Input
httpMethod=SSL-SNI destinationIPAddress=174.37.243.85 url=https://174.37.243.85:443*
httpMethod=SSL-SNI destinationIPAddress=183.3.226.92 url=https://pingtas.qq.com:443/*
httpMethod=SSL-SNI destinationIPAddress=184.173.136.86 url=https://v.whatsapp.net:443/*
Expected Output
httpMethod=SSL-SNI destinationIPAddress=174.37.243.85 url=https://55.f3.25ae.ip4.static.sl-reverse.com:443/*
httpMethod=SSL-SNI destinationIPAddress=183.3.226.92 url=https://pingtas.qq.com:443/*
httpMethod=SSL-SNI destinationIPAddress=184.173.136.86 url=https://v.whatsapp.net:443/*
Here's a quick and dirty attempt in pure Awk.
awk '$3 ~ /^url=https?:\/\/[0-9.]*([:\/?*].*)?$/ {
# Parse out the hostname part
split($3, n, /[\/:?\*]+/);
cmd = "dig +short -x " n[2]
cmd | getline reverse;
sub(/\.$/, "", reverse);
close(cmd)
# Figure out the tail after the hostname part
match($3, /^url=https:?\/\/[0-9.]*/); # update index
$3 = n[1] "://" reverse substr($3, RSTART+RLENGTH) } 1' file
If you don't have dig, you might need to resort to nslookup or host instead; but the only one of these which portably offers properly machine-readable output is dig so you might want to install it for that feature alone.
Solution 1st: Within single awk after discussion on comments adding this now:
awk '
{
if(match($0,/\/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/)){
val_match=substr($0,RSTART+1,RLENGTH-1);
system("nslookup " val_match " > temp")};
val=$0;
while(getline < "temp"){
if($0 ~ /name/){
num=split($0, array," ");
sub(/\./,"",array[num]);
sub(val_match,array[num],val);
print val}}
}
NF
' Input_file
Solution 2nd: It is my initial solution with awk and shell.
Following simple script may help you on same:
cat script.ksh
CHECK_IP () {
fdqn=$(echo "$1" | awk '{if(match($0,/\/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/)){system("nslookup " substr($0,RSTART+1,RLENGTH-1))}}')
actual_fdqn=$(echo "$fqdn" | awk '/name/{sub(/\./,""$NF);print $NF}')
echo "$actual_fdqn"
}
while read line
do
val=$(CHECK_IP "$line")
if [[ -n "$val" ]]
then
echo "$line" | awk -v var="$val" '{if(match($0,/\/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/)){ip_val=substr($0,RSTART+1,RLENGTH-1);sub(ip_val,var)}} 1'
else
echo "$line"
fi
done < "Input_file"

Search .netrc file in bash, and return username for given machine

I need a way (the most portable) in bash, to perform a search of the ~/.netrc file, for a particular machine api.mydomain.com and then on the next line, pull the username value.
The format is:
machine a.mydomain.com
username foo
passsword bar
machine api.mydomain.com
username boo
password far
machine b.mydomain.com
username doo
password car
So, it should matchin api.mydomain.com and return exactly boo from this example.
awk '/api.mydomain.com/{getline; print}' ~/.netrc
Get's me the line I want, but how do I find the username value?
$ awk '/api.mydomain.com/{getline; print $2}' ~/.netrc
boo
To capture it in a variable:
$ name=$(awk '/api.mydomain.com/{getline; print $2}' ~/.netrc)
$ echo "$name"
boo
By default, awk splits records (lines) into fields based on whitespace. Thus, on the line username boo, username is assigned to field 1, denoted $1, and boo is assigned to field 2, denoted $2.
If you like to avoid using the getline fuction use this:
awk '/api.mydomain.com/ {f=NR} f&&f+1==NR {print $2}' ~/.netrc
boo
As Ed write here: avoid using it.
http://awk.info/?tip/getline
This will find the line number of the pattern, and then
when line number is one more, print field #2
Can be shorten some to:
awk '/api.mydomain.com/ {f=NR} f&&f+1==NR&&$0=$2' ~/.netrc
or
awk 'f&&!--f&&$0=$2; /api.mydomain.com/ {f=1}' ~/.netrc
This may be the most robust way to do it.
If there are comments line or blank line after domain, other solution fails.
awk '/api.mydomain.com/ {f=1} f && /username/ {print $2;f=0}' ~/.netrc
boo
If domain is found, set flag f. If flag f is true and next line has username print field #2
This vanilla Bourne shell (which includes Bash, KSH, and more) function should parse any valid .netrc file: it handled everything I could think of. Invoke with netrc-fetch MACHINE [FIELD]. For your question, it would be netrc-fetch api.mydomain.com user.
netrc-fetch () {
cat $HOME/.netrc | awk -v host=$1 -v field=${2:-password} '
{
for (i=1; i <= NF; i += 2) {
j=i+1;
if ($i == "machine") {
found = ($j == host);
} else if (found && ($i == field)) {
print $j;
exit;
}
}
}
'
}
This sed is as portable as I can make it:
sed -n '
/machine[ ]\{1,\}api.mydomain.com/ {
# we have matched the machine
:a
# next line
n
# print username, if matched
s/^[ ]\{1,\}username[ ]\{1,\}//p
# goto b if matched
tb
# else goto a
ba
:b
q
}
' ~/.netrc
The whitespace in the brackets is a space and a tab character.
Looking at this with fresh eyes, this is what I would write now:
awk -v machine=api.mydomain.com '
$1 == "machine" {
if (m)
# we have already seen the requested domain but did not find a username
exit 1
if ($2 == machine) m=1
}
m && $1 == "username" {print $2; exit}
' ~/.netrc
or if you like unreadable oneliners
awk '$1=="machine"{if(m)exit 1;if($2==M)m=1} m&&$1=="username"{print $2;exit}' M=api.mydomain.com ~/.netrc
Here's what you could do:
awk '/api.mydomain.com/{getline;print $2}' ~/.netrc ##$2 will print username
I recently stumbled upon this issue and couldn't find an answer which covers the one-line format (i.e. machine x login y password z), and inline or intra-line comments.
What I ended up with is making the file into one line with xargs (so only single spaces remain), then using grep with lookbehind (for the keyword) and lookahead (for the next whitespace or end of line)
xargs < ~/.netrc | grep -oP '(?<=machine api\.domain\.com ).*?(?=( machine)|$)' | grep -oP '(?<=login ).*?(?=\s|$)'
This could be of course developed into a function of sorts with extending the dots in the remote host variable with backslashes, printing the password as well, etc.
get_netrc_entry() {
machine_entry=$(xargs < ~/.netrc | grep -oP "(?<=machine ${1//./\\.} ).*?(?=( machine)|$)")
grep -oP '(?<=login ).*?(?=\s|$)' <<<"${machine_entry}"
grep -oP '(?<=password ).*?(?=\s|$)' <<<"${machine_entry}"
}
Another solution to get user of password:
grep -A2 armdocker.rnd.ericsson.se ~/.config/artifactory.netrc | awk '/login/ {print $2}'
grep -A2 gives back the following 2 lines of a requested machine

concatenate two variables from 2 different awk commands in a single echo

Job = grep 'Job:' | awk '{ print $3 }'
Status = grep 'Job Status:' | awk '{ print $3 }'
Both the variables are printed correctly by using two echo statements.I want a result like Job name - status in a single line.I have tried below commands. But its printing only 2nd variable like - status
echo "$Job - $Status"
echo "${Job} - ${Status}"
echo -e "${Job} - ${Status}"
please help!
You can do it with a single awk command:
awk '/Job:/ { job = $3 } /Job Status:/ { status = $3 } END { print job " - " status }' file
If Job: comes before Job Status:
awk '/Job:/ { job = $3 } /Job Status:/ { print job " - " $3; exit }' file
Or vice versa:
awk '/Job Status:/ { status = $3 } /Job Status:/ { print $3 " - " status; exit }' file
I think that should work:
echo $(awk ' /Job:/ { print $3} ' file)" - "$(awk ' /Job Status:/ { print $3} ' file)
but konsolebox's version is probably better, as there is only one awk invocation.
I think you are trying to find out how to get the result of running some command and store it in a variable. Then you want to do that twice and print both variables on the same line.
So the basic syntax is:
result=$(some command)
e.g. if
date +'%Y'
tells you the year is 2014, but you want 2014 in a variable called year, you can do
year=$(date +'%Y')
then you can echo $year like this:
echo $year
2014
So, coming to your actual question, you want two variables, one for the output of each of two commands:
job=$(grep "Job:" someFile | awk '{print $3}')
status=$(grep "Job Status:" someFile | awk '{print $3}')
then you can do:
echo $job $status
and get both things on the same line.
The other answers are saying you can avoid invoking awk twice, which is true, but doesn't explain how to capture the result of running a command into a variable. In general, you don't need to use awk and grep, because this:
grep xyz | awk ...
is equivalent to
awk '/xyz/ {...}'
but uses one fewer processes (i.e. no grep) and therefore fewer resources.
And by the way, you must not put any spaces either side of = in bash either. It is
variable=something
not
variable = something

Resources