I am trying to figure out how to do the following with sed:
I got a list of IPv4 addresses and I am trying to make them all uniform in the display. So for example: 1.2.4.32 would be 001.002.004.032. 10.125.62.1 would be 010.125.062.001.
I am trying to use sed to do it because that's what I am learning right now.
I got these two, which will take any one or two digit number and append zeros at the front.
sed 's/\<[0-9][0-9]\>/0&/g' file
sed 's/\<[0-9]\>/00&/g' file
But that runs into a more practical problem in that my input file will have single or double digits numbers in other non-IP address places. Example:
host-1 1.2.3.32
So I need a way for it to look for the full IP address, which I thought could be achieved by this
sed 's/\.\<[0-9]\>/00&/g'
but not only does that ignore the case of 1.something.something.something, but also it appends the 00 at the end of 3rd octet for some reason.
echo "10.10.88.5" | sed 's/\.\<[0-9]\>/00&/g'
10.10.8800.5
Sample file:
Jumpstart Server jumo 10.20.5.126
Jumpstart Server acob 10.20.5.168
NW1 H17 Node cluster 10.10.161.87
NW1 H17 Node-1 10.10.161.8
NW1 H17 Node-2 10.10.161.9
ts-nw1 10.10.8.6
The idiomatic way of changing only parts of a line is to copy it to the hold space, remove the parts we're not interested in from the pattern space, get the hold space back and then rearrange the pattern space to replace the part we've changed with our new version.
This should work (replace -r with -E for BSD sed):
sed -r 'h # Copy pattern space to hold space
# Remove everything except IP address from pattern space
s/.*\b(([0-9]{1,3}\.){3}[0-9]{1,3})\b.*/\1/
s/([0-9])+/00&/g # Prepend '00' to each group of digits
s/[0-9]*([0-9]{3})/\1/g # Only retain last three digits of each group
G # Append hold space to pattern space
# Replace old IP with new IP
s/(.*)\n(.*)\b([0-9]{1,3}\.){3}[0-9]{1,3}\b(.*)/\2\1\4/' infile
The last step is the most complicated one. Just before it, a line looks like this (newline as \n, end of line as $):
010.020.005.126\nJumpstart Server jumo 10.20.5.126$
i.e., our new and improved IP address, a newline, then the complete old line. We now capture the underlined groups:
010.020.005.126\nJumpstart Server jumo 10.20.5.126$
^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^ ^
(.*) \n (.*) \b...\b (.*)
\1 \2 \3 \4
and rearrange the line by using group 2, then groups 1 (our new IP) and 4. Notice that
There are four capture groups, but the third one is just there to help describe an IP address, we don't actually want to retain it, hence \2\1\4 in the substitution (there are no non-capturing groups in sed).
The last capturing group (after the IP address) is empty, but having it makes it possible to use this for lines that have the IP address anywhere.
This only replaces the first IP address on each line, in case there are several.
The overall output is
Jumpstart Server jumo 010.020.005.126
Jumpstart Server acob 010.020.005.168
NW1 H17 Node cluster 010.010.161.087
NW1 H17 Node-1 010.010.161.008
NW1 H17 Node-2 010.010.161.009
ts-nw1 010.010.008.006
The same as a solidly unreadable one-liner:
sed -r 'h;s/.*\b(([0-9]{1,3}\.){3}[0-9]{1,3})\b.*/\1/;s/([0-9])+/00&/g;s/[0-9]*([0-9]{3})/\1/g;G;s/(.*)\n(.*)\b([0-9]{1,3}\.){3}[0-9]{1,3}\b(.*)/\2\1\4/' infile
\b is a GNU extension. The script mostly works without it as well; using it makes sure that blah1.2.3.4blah is left alone.
$ cat 37222835.txt
Jumpstart Server jumo 10.20.5.126 10.29.23.24
Jumpstart Server acob 10.20.5.168 dig opt
Jumpstart Server reac 251.218.212.1 rel
NW1 H17 Node cluster 10.10.161.87
NW1 H17 Node-1 10.10.161.8
NW1 H17 Node-2 10.10.161.9
ts-nw1 10.10.8.6
Nw2 HW12 Node-3 192.168.0.1
cluster
Doing :
sed -n 's/\([1]\?[0-9][0-9]\?\|2[0-4][0-9]\|25[0-5]\)\{1\}\.'\
'\([1]\?[0-9][0-9]\?\|2[0-4][0-9]\|25[0-5]\)\{1\}\.'\
'\([1]\?[0-9][0-9]\?\|2[0-4][0-9]\|25[0-5]\)\{1\}\.'\
'\([1]\?[0-9][0-9]\?\|2[0-4][0-9]\|25[0-5] \)/00\1\.00\2\.00\3\.00\4/g;
s/0\+\([0-9]\{3\}\)/\1/g;p' 37222835.txt
gives :
Jumpstart Server jumo 010.020.005.126 010.029.023.024
Jumpstart Server acob 010.020.005.168 dig opt
Jumpstart Server reac 251.218.212.001 rel
NW1 H17 Node cluster 010.010.161.087
NW1 H17 Node-1 010.010.161.008
NW1 H17 Node-2 010.010.161.009
ts-nw1 010.010.008.006
Nw2 HW12 Node-3 192.168.000.001
cluster
Advantage over the approach mentioned by #benjamin-w
This can replace multiple ip addresses in the same line
Disadvantage(the approach mentioned by #benjamin-w remedy this)
Had there be a word say Node-000234 it would be changed to Node-234. In fact, you could work on the second substitution command to get the desired behaviour.
Related
Hi there.
I am very new to Shell Script so need your help...
I have config file with below info
config name AAAAA
root root
port number 00000
Hostname hahahahah
config name AAAAA
root less
port number 00001
Hostname nonononono
config name AAAAA
root less
port number 00002
Hostname nonononono
And inside my bash file, there's arraylist with below info
${array1[0]} # Has value of value11111
${array2[1]} # Has value of value22222
${array2[1]} # Has value of value33333
I want to change config file and save as below
config name value11111
root root
port number 00000
Hostname hahahahah
config name value22222
root less
port number 00001
Hostname nonononono
config name value33333
root less
port number 00002
Hostname nonononono
I tried awk and sed but no luck..... Could you please help this?
Check out some of these answers.
I second the advice by Ed and David (and in hindsight, this whole post could have been a comment instead of an answer), that awk/sed might not be the best tool for this job, and you'd want to take a step back and re-think the process. There's a whole bunch of things that could go wrong; the array values might not be populated correctly, there's no check that enough values for all substitutions exist, and in the end, you can't roll changes back.
Nevertheless, here's a starting point, just to illustrate some sed. It certainly is not the most performant one, and only works for GNU sed, but provides the output you required
#!/bin/bash
declare -a array
array=(value11111 value22222 value33333)
for a in "${array[#]}"; do
# Use sed inline, perform substitutions directly on the file
# Starting from the first line, search for the first match of `config name AAAAA`
# and then execute the substitution in curly brackets
sed -i "0,/config name AAAAA/{s/config name AAAAA/config name $a/}" yourinputconfigfile
done
# yourinputconfigfile
config name value11111
root root
port number 00000
Hostname hahahahah
config name value22222
root less
port number 00001
Hostname nonononono
config name value33333
root less
port number 00002
Hostname nonononono
I'm going to try to describe my problem and my end goal as best as I can, here it goes:
I have a script that fetches AWS ELB information (elb name + ports that's associated with a specific certifican arn).
So, in the end I have a text file (I call it elb_ports) and it looks something like this:
ccds-lb 636
cf-router 443 4443
dev-cf-router 443 4443
eng-jenkins-monit 443
gitlab-lb 443
gitlab-mattermost-elb 443
jenkins-np-elb 443
saml 443
uaa 443
I have another script that comes after that which I want it to go through that elb_ports file and replace the certificates with a new one, but according to Amazon's documentation: It says in order to replace the certificates, I need two things from that elb_ports file. The load balancer name and the load balancer port.
So basically their command looks like this
aws elb set-load-balancer-listener-ssl-certificate \
--load-balancer-name my-load-balancer \
--load-balancer-port 443 \
--ssl-certificate-id arn:aws:iam::123456789012:server-certificate/my-new-certificate
I want to be able to loop through the file and execute the command above to each elb and port, but my problem is with the elbs that has multiple ports associated with the cert like: cf-router 443 4443 for example.
So my idea was to split that into two lines, so like this:
cf-router 443
cf-router 4443
But I'm not sure how to add cf-router (for example) to the ports that come after the first one (there could be more than two ports using the same cert).
I hope I was able to explain my problem and end goal clearly, if this isn't a good method, I'm open to suggestions also.
EDIT: Perhaps something like this is beneficial, but not sure how to tailor it to my needs.. Like put each line in an array and the space as a delimiter and then loop through each line putting arr(1) (load balancer name) and then the load balancer port, but not sure how to count and go through >arr(2) in bash.
To split out your extra columns into separate lines:
while read -r lb_name lb_ports_str; do ## split line into lb name and port list string
read -r -a lb_ports <<<"$lb_ports_str" ## split out port list string into an array
for port in "${lb_ports[#]}"; do ## iterate through that array
printf '%s %s\n' "$lb_name" "$port" ## handle each port separately
done
done <elb_ports ## reading lines from elb_ports
Of course, that printf could be any other line referring to $lb_name and $port -- meaning you could potentially run your code that's installing new certificates here.
I've got a text file with a bunch of IPv4 addresses, and I'd like to know the hostname of each one in order to know if they are tor addresses. Is there a simple script that can help me to do that ?
You can loop using dig:
#!/bin/bash
while read line
do
dig -x "$line" +short
done
Then given IPs 1 per line, you can run something like ./reverse.sh < addrs.txt.
Caveats: DNS is not a 1-to-1 mapping, and reverse DNS is somewhat less reliable than forward DNS.
I am kicking machines with cobbler to install redhat 6 and also put in hostname.
I am managing my machines with /etc/hosts file right now (may switch to dns in the future) and would like to have puppet (or anything that will work) parse through /etc/hosts and find my ip address and hostname (based on the hostname supplied by cobbler at kickstart). The trick is that some machines have multiple IP's and hostnames in the /etc/hosts file, like such:
# Maintenance Network
192.168.80.192 testsrv01-maint
192.168.80.193 testsrv02-maint
192.168.80.194 testsrv03-maint
# Lights Out Network
192.168.120.192 testsrv01-ilo
192.168.120.193 testsrv02-ilo
192.168.120.194 testsrv03-ilo
# Primary Data Network
192.168.150.192 testsrv01-pri
192.168.150.193 testsrv02-pri
192.168.150.194 testsrv03-pri
# Secondary Data Network
192.168.200.192 testsrv01-sec
192.168.200.193 testsrv02-sec
192.168.200.194 testsrv03-sec
I need to capture each ip and hostname pair (in a line) and transpose into /etc/sysconfig/network-scripts/ifcfg-eth* (eth1, eth2, eth3, ...). Puppet will need to create as many ifcfg-eth* files as there are matches in /etc/hosts for the hostname.
I just need puppet to append the $IP and $hostname to the ifcfg-eth file, the rest of the content is common.
So how would I get 4x ifcfg-eth files for 'testsrv01', with puppet?
Puppet is a very ill fit for this task. This calls for a script, which Puppet does not support. Puppet allows you to declare a piece of machine state that can portably enforced on different platforms.
The task is simple enough for a Shell or Perl Script.
With Puppet, it would entail the following scripts
writing custom facts to retrieve each address / hostname pair
devising a defined type to render such a pair into an ethX file
You'd possibly even need to generate the respective manifest, so that the appropriate interface index is chosen for each address.
All things considered, you would not make use of Puppet's strengths and suffer some of its weaknesses.
Have a text file w/ around 3 million URL's of sites I want to block.
Trying to ping them one by one (yes, I know it is going to take some time).
Have a script (yes, I am a bit slow in BASH) which reads the lines one at a time from text file.
Obviously cannot print text file here. Text file was created >> w/ Python some time ago.
Problem is that ping returns "unknown host" w/ every entry. If I make a smaller file by hand using the same entries the script works. I thought it may be a white space or end of line issue so tried addressing that in script. What could the issue possibly be?
#!/bin/bash
while read line
do
li=$(echo $line|tr -d '\n')
li2=$(echo $li|tr -d ' ')
if [ ${#line} -lt 2 ]
then
continue
fi
ping -c 2 -- $li2>>/dev/null
if [ $? -gt 0 ]
then
echo 'bad'
else
echo 'good'
fi
done<'temp_file.txt'
Does the file contains URLs or hostnames ?
If it contains URLs you must extract the hostname from URLs before pinging:
hostname=$(echo "$li2"|cut -d/ -f3);
ping -c 2 -- "$hostname"
Ping is used to ping hosts. If you have URLs of websites, then it will not work. Check that you have hosts in your file , example www.google.com or an IP address and not actual full website urls. If you want to check actual URLs, use a tool like wget and another tool like grep/awk to grab for errors like 404 or others. Last but not least, people who are security conscious will sometimes block pinging from the outside, so take note.
C heck if the file contains windows-style \r\n line endings: head file | od -c
If so, to fix it: dos2unix filename filename
I wouldn't use ping for this. It can easily be blocked, and it's not the best way to check for either ip addresses or if a server presents web pages.
If you just want to find the corresponding IP, use host:
$ host www.google.com
www.google.com is an alias for www.l.google.com.
www.l.google.com has address 209.85.149.106
www.l.google.com has address 209.85.149.147
www.l.google.com has address 209.85.149.99
www.l.google.com has address 209.85.149.103
www.l.google.com has address 209.85.149.104
www.l.google.com has address 209.85.149.105
As you see, you get all the IPs registered to a host. (Note that this requires you to parse the hostname from your urls!)
If you want to see if a URL points at a web server, use wget:
wget --spider $url
The --spider flag makes wget not save the page, just check that it exists. You could look at the return code, or add the -S flag (which prints the HTTP headers returned)