I am trying to go through a list of workstations, and add data to their corresponding individual files. I have already made the individual files, but now I need help using sed or something else to replace text in the individual XML files:
For example:
workstation_list.txt has the following lines:
workstation1
workstation1.domain.com
127.0.0.1
00:00:00:00:00:00
workstation2
workstation2.domain.com
127.0.0.2
11:11:11:11:11:11
I have two files: workstation1 and workstation2 with the following XML:
< HOST_NAME >workstation3< /HOST_NAME >
< HOST_FQDN >workstation3.domain.com< /HOST_FQDN >
< IP >127.0.0.0< /IP >
< MAC >33:33:33:33:33:33< /MAC >
I can do a "while read line do" without a problem, but I've never used more than one variable.
Thank you for your help!
The whole task can be accomplished with a single call to gawk:
awk -v RS='\n\n' '{
print "< HOST_NAME >" $1 "< /HOST_NAME >" > $1
print "< HOST_FQDN >" $2 "< /HOST_FQDN >" > $1
print "< IP >" $3 "< /IP >" > $1
print "< MAC >" $4 "< /MAC >" > $1
close($1)
}' file
Given your sample input it creates two files as follows:
workstation1:
< HOST_NAME >workstation1< /HOST_NAME >
< HOST_FQDN >workstation1.domain.com< /HOST_FQDN >
< IP >127.0.0.1< /IP >
< MAC >00:00:00:00:00:00< /MAC >
workstation2:
< HOST_NAME >workstation2< /HOST_NAME >
< HOST_FQDN >workstation2.domain.com< /HOST_FQDN >
< IP >127.0.0.2< /IP >
< MAC >11:11:11:11:11:11< /MAC >
Related
I'm trying to get the cassandra schema version into variable from output of nodetool command.
Here are some of the output of nodetool commands:
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
65e78f0e-e81e-30d8-a631-a65dff93bf82: [127.0.0.1]
When few nodes are not reachable here's the output.
Cluster Information:
Name: Production Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
UNREACHABLE: 1176b7ac-8993-395d-85fd-41b89ef49fbb: [10.202.205.203]
Can anyone suggest how to get schema version into variable irrespective of reachable or not?
Tried to use awk and grep commands but didn't work because of unreachable.
Another version of an awk script that will match only the UUID type REGEX can be written to use match() setting the internal RSTART and RLENGTH variables that can then be used with substr().
That would be:
awk '
/Schema versions:/ {
set=1
next
}
set {
match($0,/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/)
print substr($0, RSTART, RLENGTH)
exit
}' file
Example Use/Output
$ awk '
> /Schema versions:/ {
> set=1
> next
> }
> set {
> match($0,/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/)
> print substr($0, RSTART, RLENGTH)
> exit
> }' << 'eof'
> Cluster Information:
> Name: Test Cluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> 65e78f0e-e81e-30d8-a631-a65dff93bf82: [127.0.0.1]
>
> eof
65e78f0e-e81e-30d8-a631-a65dff93bf82
and
$ awk '
> /Schema versions:/ {
> set=1
> next
> }
> set {
> match($0,/[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/)
> print substr($0, RSTART, RLENGTH)
> exit
> }' << 'eof'
> Cluster Information:
> Name: Production Cluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> UNREACHABLE: 1176b7ac-8993-395d-85fd-41b89ef49fbb: [10.202.205.203]
> eof
1176b7ac-8993-395d-85fd-41b89ef49fbb
You can use the command in a command substitution in bash to capture the result in a variable.
Let me know if you have further questions.
Awk will do the job for that:
version=$(awk '/Schema versions:/ {
getline
gsub(/:/,"")
if ($1 == "UNREACHABLE") {
print $2
} else {
print $1
}
}' < <(nodetool_cmd)) # remplace "nodetool_cmd" by the correct command
$ echo "$version" #when reachable
65e78f0e-e81e-30d8-a631-a65dff93bf82
$ echo "$version" # when unreachable
1176b7ac-8993-395d-85fd-41b89ef49fbb
# or in single line:
version=$(awk '/version/ {getline;gsub(/:/,"");if ($1 == "UNREACHABLE") {print $2} else {print $1}}' < <(nodetool_cmd))
Requirement
I have a txt file in which last column have URLs.
Some of the URL entries have IPs instead of FQDN.
So, for entries with IPs (e.g. url=https://174.37.243.85:443*), I need to do reverse nslookup for IP and replace the result (FQDN) with IP.
Text File Input
httpMethod=SSL-SNI destinationIPAddress=174.37.243.85 url=https://174.37.243.85:443*
httpMethod=SSL-SNI destinationIPAddress=183.3.226.92 url=https://pingtas.qq.com:443/*
httpMethod=SSL-SNI destinationIPAddress=184.173.136.86 url=https://v.whatsapp.net:443/*
Expected Output
httpMethod=SSL-SNI destinationIPAddress=174.37.243.85 url=https://55.f3.25ae.ip4.static.sl-reverse.com:443/*
httpMethod=SSL-SNI destinationIPAddress=183.3.226.92 url=https://pingtas.qq.com:443/*
httpMethod=SSL-SNI destinationIPAddress=184.173.136.86 url=https://v.whatsapp.net:443/*
Here's a quick and dirty attempt in pure Awk.
awk '$3 ~ /^url=https?:\/\/[0-9.]*([:\/?*].*)?$/ {
# Parse out the hostname part
split($3, n, /[\/:?\*]+/);
cmd = "dig +short -x " n[2]
cmd | getline reverse;
sub(/\.$/, "", reverse);
close(cmd)
# Figure out the tail after the hostname part
match($3, /^url=https:?\/\/[0-9.]*/); # update index
$3 = n[1] "://" reverse substr($3, RSTART+RLENGTH) } 1' file
If you don't have dig, you might need to resort to nslookup or host instead; but the only one of these which portably offers properly machine-readable output is dig so you might want to install it for that feature alone.
Solution 1st: Within single awk after discussion on comments adding this now:
awk '
{
if(match($0,/\/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/)){
val_match=substr($0,RSTART+1,RLENGTH-1);
system("nslookup " val_match " > temp")};
val=$0;
while(getline < "temp"){
if($0 ~ /name/){
num=split($0, array," ");
sub(/\./,"",array[num]);
sub(val_match,array[num],val);
print val}}
}
NF
' Input_file
Solution 2nd: It is my initial solution with awk and shell.
Following simple script may help you on same:
cat script.ksh
CHECK_IP () {
fdqn=$(echo "$1" | awk '{if(match($0,/\/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/)){system("nslookup " substr($0,RSTART+1,RLENGTH-1))}}')
actual_fdqn=$(echo "$fqdn" | awk '/name/{sub(/\./,""$NF);print $NF}')
echo "$actual_fdqn"
}
while read line
do
val=$(CHECK_IP "$line")
if [[ -n "$val" ]]
then
echo "$line" | awk -v var="$val" '{if(match($0,/\/[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+/)){ip_val=substr($0,RSTART+1,RLENGTH-1);sub(ip_val,var)}} 1'
else
echo "$line"
fi
done < "Input_file"
I am new to shell script. I have a file app.conf as :
[MySql]
user = root
password = root123
domain = localhost
database = db_name
port = 3306
[Logs]
level = logging.DEBUG
[Server]
port = 8080
I want to parse this file in shell script and want to extract mysql credentials from the same. How can I achieve that?
I'd do this:
pw=$(awk '/^password/{print $3}' app.conf)
user=$(awk '/^user/{print $3}' app.conf)
echo $pw
root123
echo $user
root
The $() sets the variable pw to the output of the command inside. The command inside looks through your app.conf file for a line starting password and then prints the 3rd field in that line.
EDITED
If you are going to parse a bunch of values out of your config file, I would make a variable for the config file name:
CONFIG=app.conf
pw=$(awk '/^password/{print $3}' "${CONFIG}")
user=$(awk '/^user/{print $3}' "${CONFIG}")
Here's how to do the two different ports... by setting a flag to 1 when you come to the right section and exiting when you find the port.
mport=$(awk '/^\[MySQL\]/{f=1} f==1&&/^port/{print $3;exit}' "${CONFIG}")
sport=$(awk '/^\[Server\]/{f=1} f==1&&/^port/{print $3;exit}' "${CONFIG}")
You will want to search for "shell ini file parser". I would start with something like this:
ini_get () {
awk -v section="$2" -v variable="$3" '
$0 == "[" section "]" { in_section = 1; next }
in_section && $1 == variable {
$1=""
$2=""
sub(/^[[:space:]]+/, "")
print
exit
}
in_section && $1 == "" {
# we are at a blank line without finding the var in the section
print "not found" > "/dev/stderr"
exit 1
}
' "$1"
}
mysql_user=$( ini_get app.conf MySql user )
Using awk:
awk -F ' *= *' '$1=="user"||$1=="password"{print $2}' my.cnf
root
gogslab
I ran in a similar problem yesterday and thought the best solution might be, if you get an associative array like "key - value" after parsing the file.
I you like to see a running example have a look at https://github.com/philippkemmeter/set-resolution/blob/master/set-resolution.
Adapted to your problem, this might work:
function receive_assoc_declare_statement {
awk -F '=' 'BEGIN {ORS=" "}
{
gsub(/[ \t]+/, "", $1);
gsub(/[ \t]+/, "", $2);
print "[" $1 "]=" $2
}' app.conf
}
eval 'declare -A CONF=('`receive_assoc_declare_statement`')'
You then have access to for instance user via ${CONF[user]}.
The gsub is trimming keys and values, so that you can use tab etc. to format your config file.
It's lacking sections, but you could add this functionality using sed to create one config array per section:
sed -n '/\[MySql\]/, /\[/ {p}' test.removeme | sed '1 d; $ d'
So answering your question in total, this script might work:
MYSQL=`sed -n '/\[MySql\]/, /\[/ {p}' app.conf | sed '1 d; $ d' | awk -F '=' 'BEGIN {ORS=" "}
{
gsub(/[ \t]+/, "", $1);
gsub(/[ \t]+/, "", $2);
print "[" $1 "]=" $2
}' `
eval 'declare -A MYSQL=('$MYSQL')'
The other sections correspondingly.
I'm trying to make a program that sorts machines based on load, but i'm having a hard time parsing the ssh output. What i have so far is this:
gen_data()
{
declare -a machines=("machine1" "machine2" "machine3" "machine4" "machine5")
for i in ${machines[#]}; do
ssh $i "hostname && uptime"
done | awk ' BEGIN {cnt=0} \
{ printf("%s, ", $0)
cnt++
if(cnt % 3 == 0) {printf("\n") }
}' > ~/perf_data
}
#function check_data
# check for load averages (fields 6,7,8) which are greater than 7
check_data()
{
awk -F"," '{ if($6 < 9.0 && $7 < 9.0 && $8 < 9.0)
{print $0 }
}' ~/perf_data
}
most of this code is a modified version of a code that checked machine loads and emailed you if it was too high, but i can't quite get it to print out the machine names or make the perf_data file correctly.
What i'm trying to get is for a list of machines me#machine*.network.com, the program tests the load of the machine, and if it's low enough it prints the machine name:
me#machine1.network.com me#machine5.network.com me#machine10.network.com
that way i can pipe the output to another program that will use those machines.
Since i'm a n00b in awk i really need help with this.
Instead of this:
for i in ${machines[#]}; do
ssh $i "hostname && uptime"
done | awk ...
use this to make your life easier
for m in ${machines[#]}; do
ssh $i <<'COMMANDS'
echo "$(hostname):$(uptime)" | awk -F: '{gsub(/,/,"",$NF); print $1, $NF}'
COMMANDS
done > ~/perf_data
Then check_data can be
check_data() {
awk '$2 < 9 && $3 < 9 && $4 < 9 {print $1} ~/perf_data
}
Rather than modifying this script, you can write new one.
Here's a version replacing your script entirely, which fetches the load average in a Linux specific way:
for host in machine1 machine2 machine3
do
ssh "$host" '[ "$(awk "\$1 < 9" /proc/loadavg)" ] && hostname'
done > ~/perf_data
Alternately, you can do it through uptime:
for host in machine1 machine2 machine3
do
ssh "$host" '[ "$(uptime | awk -F"[ ,]+" "\$11 < 9")" ] && hostname'
done > ~/perf_data
Both these assume that you're interested in the current load, so it checks the 1 minute average rather than also caring about the 15 minute average.
I have two files: domainList and config.cnf. The domainList file simply has a list of domains like so:
facebook.com
yahoo.com
youtube.com
The config.cnf is a config file and has the same list with a little different format. I need to write a script that will update the config file when the list has been updated. I can execute a bash script whenever the first list is updated. Here is the format of the list in the config file...
*other config options/entries*
[my_list]
WWW.1 = facebook.com
WWW.2 = yahoo.com
WWW.3 = youtube.com
EOF
So, if yahoo is removed and ebay is added in domainList and I run my cool bash script I need the config file to be updated like so...
*other config options/entries*
[my_list]
WWW.1 = facebook.com
WWW.2 = youtube.com
WWW.3 = ebay.com
EOF
To complicate things (slightly) the domains can have subdomains and wild cards (ie. news.google.com or *.google.com). Any ideas on how to accomplish this would be appreciated! How do I do it without getting the numbers all out of whack? It'll probably just need to clear the list and regenerate it every time, huh?
Thanks!
EV
Here's a simple script to achieve that:
# delete all lines after [my_list]
sed -i '/my_list/q' config.cnf
# add the domain list to the bottom of the config
awk '{print "WWW." NR " = " $0}' domainList >> config.cnf
This script could be written as a one-liner with awk or sed, but the above is (hopefully) quite clear in its approach.
#!/usr/bin/env bash
FIN=domainList
FOUT=config.cnf
echo "config.cnf template header" > $FOUT
awk '{ print "WWW." FNR " = " $1 }' $FIN >> $FOUT
echo "config.cnf template footer" >> $FOUT
Here is a one-liner in awk
awk '
BEGIN{var=1}
NR==FNR{a[NR]=$1;next}
var && /WWW/{var=0; for (x=1;x<=length(a);x++) {print "WWW." x " = " a[x]};next}
!var && /WWW/ {next}
1' domainList config.cnf > config.cnf_new
Test:
$ cat domainList
facebook.com
youtube.com
ebay.com
$ cat config.cnf
*other config options/entries*
[my_list]
WWW.1 = facebook.com
WWW.2 = yahoo.com
WWW.3 = youtube.com
EOF
$ awk '
BEGIN{var=1}
NR==FNR{a[NR]=$1;next}
var && /WWW/{var=0; for (x=1;x<=length(a);x++) {print "WWW." x " = " a[x]};next}
!var && /WWW/ {next}
1' domainList config.cnf
*other config options/entries*
[my_list]
WWW.1 = facebook.com
WWW.2 = youtube.com
WWW.3 = ebay.com
EOF
$
bash with a touch of awk
while IFS= read -r line; do
echo "$line"
if [[ $line = '[my_list]' ]]; then
awk '{print "WWW." NR " = " $0}' domainList
echo "EOF" # is this literally in your config file?
break
fi
done < config.cnf > tmpfile && mv tmpfile config.cnf