I need to split an IPv4 address into octets, calculate the MD5 hash of each and print as a CGI output:
IP1=$(echo ${REMOTE_ADDR} | tr "." " " | awk '{print $1'} | md5sum | cut -c1-32)
printf $IP1
In this example, REMOTE_ADDR = 192.168.20.100
But the MD5 of 192 gives me a wrong MD5 IP1=6be7de648baa9067fa3087928d5ab0b4, while it should be 58a2fc6ed39fd083f55d4182bf88826d
If I do this:
cat /tmp/test.txt | md5sum | cut -c1-32
where test.txt contains 192,
I get the correct MD5 hash, i.e 58a2fc6ed39fd083f55d4182bf88826d
What am I doing wrong?
Your awk's print is adding a newline so you're computing the md5 of "192\n", not "192". Use
IP1=$(printf "%s" "${REMOTE_ADDR%.*.*.*}" | md5sum | cut -c1-32)
instead, which uses shell parameter expansion to remove all but the first octet of the IP address, and printf to write it without the newline.
As #Shawn said, the problem was on awk print.
Adding tr -d '\n' solved the problem.
Now it is working correctly; for other octects I had to change print $2..etc on awk
IP1=$(echo ${REMOTE_ADDR} | awk -F. '{print $1'} | tr -d '\n' | md5sum | cut -c1-32 )
Related
I am attempting to parse the nodejs package.json for the version number.. this line usually works but somehow I am now getting an error of invalid block
PACKAGE_VERSION=$(cat package.json | grep version | head -1 | awk -F: '{ print $2 }' | sed 's/[",]//g' | xargs)
specifically this area is highlighted in VSCode sed 's/......
How can I repair this line? is the " needing escaping or ?
The problem is YAML-related (it seems the combination of : and | is the culprit).
The easy fix is to use a YAML block:
script:
# …
- |-
PACKAGE_VERSION=$(cat package.json | grep version | head -1 | awk -F: '{ print $2 }' | sed 's/[",]//g' | xargs)
And while you’re at it you can use a folding block to make it more readable:
script:
# …
- >-
PACKAGE_VERSION=$(
cat package.json |
grep version |
head -1 |
awk -F: '{ print $2 }' |
sed 's/[",]//g' |
xargs
)
Chaining grep, head, awk and sed can usually be replaced by a single awk. And there’s a useless use of cat. Not to mention parsing JSON without a proper JSON parser.
I have a list of IP lookups and I wish to remove all but the last 3 parts, so:
98.254.237.114.broad.lyg.js.dynamic.163data.com.cn
would become
163data.com.cn
I have spent hours searching for clues, including parameter substitution, but the closest I got was:
$ string="98.254.237.114.broad.lyg.js.dynamic.163data.com.cn"
$ string1=${string%.*.*.*}
$ echo $string1
Which gives me the inverted answer of:
98.254.237.114.broad.lyg.js.dynamic
which is everything but the last 3 parts.
A script to do a list would be better than just the static example I have here.
Using CentOS 6, I don't mind if it by using sed, cut, awk, whatever.
Any help appreciated.
Thanks, now that I have working answers, may I ask as a follow up to then process the resulting list and if the last part (after last '.') is 3 characters - eg .com .net etc, then to just keep the last 2 parts.
If this is against protocol, please advise how to do a follow up question.
if parameter expansion inside another parameter expansion is supported, you can use this:
$ s='98.254.237.114.broad.lyg.js.dynamic.163data.com.cn'
$ # removing last three fields
$ echo "${s%.*.*.*}"
98.254.237.114.broad.lyg.js.dynamic
$ # pass output of ${s%.*.*.*} plus the extra . to be removed
$ echo "${s#${s%.*.*.*}.}"
163data.com.cn
can also reverse the line, get required fields and then reverse again.. this makes it easier to use change numbers
$ echo "$s" | rev | cut -d. -f1-3 | rev
163data.com.cn
$ echo "$s" | rev | cut -d. -f1-4 | rev
dynamic.163data.com.cn
$ # and easy to use with file input
$ cat ip.txt
98.254.237.114.broad.lyg.js.dynamic.163data.com.cn
foo.bar.123.baz.xyz
a.b.c.d.e.f
$ rev ip.txt | cut -d. -f1-3 | rev
163data.com.cn
123.baz.xyz
d.e.f
echo $string | awk -F. '{ if (NF == 2) { print $0 } else { print $(NF-2)"."$(NF-1)"."$NF } }'
NF signifies the total number of field separated by "." and so we want the last piece (NF), last but 1 (NF-1) and last but 2 (NF-2)
$ echo $string | awk -F'.' '{printf "%s.%s.%s\n",$(NF-2),$(NF-1),$NF}'
163data.com.cn
Brief explanation,
Set the field separator to .
Print only last 3 field using the awk parameter $(NF-2), $(NF-1),and $NF.
And there's also another option you may try,
$ echo $string | awk -v FPAT='[^.]+.[^.]+.[^.]+$' '{print $NF}'
163data.com.cn
It sounds like this is what you need:
awk -F'.' '{sub("([^.]+[.]){"NF-3"}","")}1'
e.g.
$ echo "$string" | awk -F'.' '{sub("([^.]+[.]){"NF-3"}","")}1'
163data.com.cn
but with just 1 sample input/output it's just a guess.
wrt your followup question, this might be what you're asking for:
$ echo "$string" | awk -F'.' '{n=(length($NF)==3?2:3); sub("([^.]+[.]){"NF-n"}","")}1'
163data.com.cn
$ echo 'www.google.com' | awk -F'.' '{n=(length($NF)==3?2:3); sub("([^.]+[.]){"NF-n"}","")}1'
google.com
Version which uses only bash:
echo $(expr "$string" : '.*\.\(.*\..*\..*\)')
To use it with a file you can iterate with xargs:
File:
head list.dat
98.254.237.114.broad.lyg.js.dynamic.163data.com.cn
98.254.34.56.broad.kkk.76onepi.co.cn
98.254.237.114.polst.a65dal.com.cn
iterating the whole file:
cat list.dat | xargs -I^ -L1 expr "^" : '.*\.\(.*\..*\..*\)'
Notice: it won't be very efficient in large scale, so you need to consider by your own whether it is good enough for you.
Regexp explanation:
.* \. \( .* \. .* \. .* \)
\___| | | | |
| \------------------------/> brakets shows which part we extract
| | |
| \-------/> the \. indicates the dots to separate specific number of words
|
|
-> the rest and the final dot which we are not interested in (out of brakets)
details:
http://tldp.org/LDP/abs/html/string-manipulation.html -> Substring Extraction
I have the file DATA, and within it there is:
Name | Karlstrom|
Description | New_Server|
Type | UNIX OS|
Formula | y=kx+j |
Severity | Critical|
I need to know how to display the data like this:
Name| Karlstrom|Description| New_Server|Type UNIX OS|Formula| y=kx+j|Severity| Critical|
USING KORN SHELL | KSH
The requirements do not explain all cases, but the following code will handle your example input:
sed -e 's/ *|/|/' DATA | tr -d "\n"; echo
# Output:
Name| Karlstrom|Description| New_Server|Type| UNIX OS|Formula| y=kx+j |Severity| Critical|
I added an echo after the command, so that the command prompt will be on the next line.
I don't think that shell matters here. Do you have cat, awk and sed utilities? You can do this, for example:
cat DATA | awk 'BEGIN {s=""} {s=s$0} END {print s}' | sed 's/ *//g'
I am writing a bash script to iterate through file lines with given value.
The command I am using to list the possible values is:
cat file.csv | cut -d';' -f2 | sort | uniq | head
When I use it in for loop like this it stops working:
for i in $( cat file.csv | cut -d';' -f2 | sort | uniq | head )
do
//do something else with these lines
done
How can I use piped commands in for loop?
You can use this awk command to get sum of 3rd column for each unique value of 2nd columns:
awk -F ';' '{sums[$2]+=$3} END{for (i in sums) print i ":", sums[i]}' file.csv
Input data:
asd;foo;0
asd;foo;2
asd;bar;1
asd;foo;4
Output:
foo: 6
bar: 1
I have a text file (list.txt) containing single and multi-word English phrases. My goal is to do a word count for each word and write the results to a CSV file.
I have figured out the command to write the amount of unique instances of each word, sorted from largest to smallest. That command is:
$ tr 'A-Z' 'a-z' < list.txt | tr -sc 'A-Za-z' '\n' | sort | uniq -c | sort -n -r | less > output.txt
The problem is the way the new file (output.txt) is formatted. There are 3 leading spaces, followed by the number of occurrences, followed by a space, followed by the word. Then on to a next line. Example:
9784 the
6368 and
4211 for
2929 to
What would I need to do in order to get the results in a more desired format, such as CSV? For example, I'd like it to be:
9784,the
6368,and
4211,for
2929,to
Even better would be:
the,9784
and,6368
for,4211
to,2929
Is there a way to do this with a Unix command, or do I need to do some post-processing within a text editor or Excel?
Use awk as follows:
> cat input
9784 the
6368 and
4211 for
2929 to
> cat input | awk '{ print $2 "," $1}'
the,9784
and,6368
for,4211
to,2929
You full pipeline will be:
$ tr 'A-Z' 'a-z' < list.txt | tr -sc 'A-Za-z' '\n' | sort | uniq -c | sort -n -r | awk '{ print $2 "," $1}' > output.txt
use sed to replace the spaces with comma
cat extra_set.txt | sort -i | uniq -c | sort -nr | sed 's/^ *//g' | sed 's/ /\, /'