I'm trying to build a csv of countries a list of ipv4 addresses come from.
I keep clobbering the IP in the output file;
#!/bin/bash
cat ipv4list.txt | while read ip ;do
echo -n "$ip", >> outputfile
whois -r "$ip">temp.txt
cat temp.txt | grep -i country >> outputfile
done
cat ipv4list.txt
1.1.1.1
2.2.2.2
What I'd like is outputfile to read;
1.1.1.1,country: AU
2.2.2.2,country: US
but I'm getting outputfile as follows;
,country: AU
,country: US
The echo statement need the refer to he ip variable:
echo -n "$ip", >> outputfile
Also, the clobbering indicate they input file may use window new lines (\r\n). Check the output file with editor, or hexdump
Going a bit outside the box, here. Please be kind, guys.
pop.ed:-
1d
wq
whois.sh:-
#!/bin/sh -x
init() {
cp ipv4list ipv4stack
}
next() {
[[ -s ipv4stack ]] && main
}
main() {
ip=$(echo "1p" | ed -s ipv4stack.txt)
wic=$(whois -r "${ip}")
echo "${ip},${wic}" >> outputfile
ed -s ipv4stack.txt < pop.ed
next
}
init
next
Ed apparently isn't installed by default in most distributions these days either, sadly; so you may need to install it if you want to use this.
Related
#!/bin/bash
{ cat sample.txt; echo; } | while read -r -a A_Name; do
if [ ! -z "${A_Name[0]}" ]; then
echo " ${A_Name[0]%.isx} "
fi
done
I am trying to display contents of a text file (which includes .isx files) using while loop but when i try to eliminate extension with %, it doesnt work.
Output
.isx is appearing for first two values:
./test.sh
abc.isx
def.isx
ghi
Input
sample.txt file:
abc.isx
def.isx
ghi.isx
Please, assist. Thank you.
why so complicated?
#!/bin/bash
cat sample.txt | while read line; do
echo "${line%.isx}"
done
or with sed
sed "s/\.isx//" sample.txt >output.txt
or with sed and inplace replacement
sed -i "s/\.isx//" sample.txt
I need to change the title on many pdf files. Pdftk works great and I tried to create a bash script (pdftitle) to make it a single pass:
#!/bin/bash
newtitle=$2
pdftk "$1" data_dump output "$1".data.txt;
sed 's/^InfoKey:\sTitle\nInfoValue:\s.*/InfoKey:\sTitle\nInfoValue:'"$newtitle/" "$1".data.txt > "$1".data.fixed.txt;
pdftk "$1" update_info *.data.fixed.txt output "$1".fixed;
mv "$1".fixed "$1";
rm -f ./*.txt
exit;
So on the cli I would enter
$> pdftitle mypdf.pdf "New Title"
The data.txt that pdftk creates has multiple lines, but only two relevant lines are the targets:
...
InfoBegin
InfoKey: Author
InfoValue: Not Me
InfoBegin
InfoKey: Title
InfoValue: Microsoft Word - Old Title.doc
InfoBegin
InfoKey: Creator
InfoValue: PScript5.dll Version 5.2
...
Of which the subsequent line needs to be replaced:
...
InfoKey: Title
InfoValue: Relevant New Title
...
No error messages are produced but the title remains intact. So it seems that sed is having problems here, but I cannot figure out where or how.
Any help will be greatly appreciated.
Here's a refactoring using Awk which assumes pdftk can write to and read from stdin/stdout using - as the pseudo-filename argument.
#!/bin/bash
filename=$1
shift
pdftk "$filename" data_dump output - |
awk -v title="$*" '/^InfoKey: Title/ { t=1 }
t && /^InfoValue:/ { $0 = "InfoValue: " title; t=0 }1' |
pdftk "$filename" update_info - output "$filename".fixed &&
mv "$filaname".fixed "$filename"
The pattern to set a flag variable when you see a pattern and then acting on a subsequent line if that variable is set is a simple and very common Awk idiom.
There is no need for trailing semicolons or an explicit exit at the end.
#tripleee provided the solution to make the bash script work perfectly:
#!/bin/bash
filename=$1
shift
pdftk "$filename" data_dump output |
awk -v title="$#" '/^InfoKey: Title/ { t=1 }
t && /^InfoValue:/ { $0 = "InfoValue: " title; t=0 }1' > data.txt
pdftk "$filename" update_info data.txt output "$filename".fixed &&
mv "$filename".fixed "$filename"
rm ./data.txt
#!/bin/bash
# This file will gather who is information
while IFS=, read url
do
whois $url > output.txt
echo "$url," >> Registrants.csv
grep "Registrant Email:" output.txt >> Registrants.csv
done < $1
How do I get the grep output to go into a new column instead of a new row? I Want column 1 to have the echo, column 2 to have the grep, then go down to a new row.
You can disable the trailing newline on echo with the -n flag.
#!/bin/bash
# This file will gather who is information
while IFS=, read url
do
whois $url > output.txt
echo -n "$url," >> Registrants.csv
grep "Registrant Email:" output.txt >> Registrants.csv
done < $1
Use printf, then you don't have to worry if the "echo" you are using accepts options.
printf "%s" "$url,"
printf is much more portable than "echo -n".
I need to remove subdomains from file:
.domain.com
.sub.domain.com -- this must be removed
.domain.com.uk
.sub2.domain.com.uk -- this must be removed
so i have used sed :
sed '/\.domain.com$/d' file
sed '/\.domain.com.uk$/d' file
and this part was simple, but when i try to do it in the loop problems appears:
while read line
do
sed '/\$line$/d' filename > filename
done < filename
I suppose it is "." and $ problem , have tried escaping it in many ways but i am out of ideas now.
A solution inspired by NeronLeVelu's idea:
#!/bin/bash
#set -x
domains=($(rev domains | sort))
for i in `seq 0 ${#domains[#]}` ;do
domain=${domains[$i]}
[ -z "$domain" ] && continue
for j in `seq $i ${#domains[#]}` ;do
[[ ${domains[$j]} =~ $domain.+ ]] && domains[$j]=
done
done
for i in `seq 0 ${#domains[#]}` ;do
[ -n "${domains[$i]}" ] && echo ${domains[$i]} | rev >> result.txt
done
For cat domains:
.domain.com
.sub.domain.com
.domain.co.uk
.sub2.domain.co.uk
sub.domain.co.uk
abc.yahoo.com
post.yahoo.com
yahoo.com
You get cat result.txt:
.domain.co.uk
.domain.com
yahoo.com
sed -n 's/.*/²&³/;H
$ {x;s/$/\
/
: again
s|\(\n\)²\([^³]*\)³\(.*\)\1²[^³]*\2³|\1\2\3|
t again
s/[²³]//g;s/.\(.*\)./\1/
p
}' YourFile
Load the file in working buffer then remove (iterative) any line that end with an earlier one, finally priont the result. Use of temporary edge delimiter easier to manage than \n in pattern
--posix -e for GNU sed (tested from AIX)
Your loop is a bit confusing because you're trying to use sed to delete patterns from a file but you take the patterns from the same file.
If you really want to remove subdomains from filename then I suppose you need more something like the following:
#!/bin/bash
set -x
cp domains domains.tmp
while read domain
do
sed -r -e "/[[:alnum:]]+${domain//./\\.}$/d" domains.tmp > domains.tmp2
cp domains.tmp2 domains.tmp
done < dom.txt
Where cat domains is:
.domain.com
.sub.domain.com
.domain.co.uk
.sub2.domain.co.uk
sub.domain.co.uk
abc.yahoo.com
post.yahoo.com
and cat dom.txt is:
.domain.com
.domain.co.uk
.yahoo.com
Running the script on these inputs results in:
$ cat domains.tmp
.domain.com
.domain.co.uk
Each iteration will remove subdomains of domain currently read from dom.txt, store it in a temporary file the contents of which is used in the next iteration for additional filtering.
It's good to try your scripts with set -x, you'll see some of the substitutions, etc.
While creating a simple script that grabs a blacklist of ip addresses and blocks them, I came across this issue:
## Function giving greif
function _droplist (){
while read line; do
$IPT -A droplist -i eth1 -s $line -j LOG --log-prefix "IP BlockList "
$IPT -A droplist -i eth1 -s $line -j DROP
done < $badlist ##IPT is /sbin/iptables
}
Through several iterations of this function I get the error:
Try `iptables -h' or 'iptables --help' for more information.
' not found.4.12: host/network `SO.ME.IPH.ERE
Running the same script with hard coded in IP's works fine, its either something to do with $line or m implementation of iptables.
cheers -- Baffled.
What does $badlist contain? A file name or a list of IPs?
if it's a filename it should work as you did it, but if it's a list of ip you have to change how you read them.
Assuming it's a new-line-delimited list of IPs like:
$ badlist="1.1.1.1\n2.2.2.2\n3.3.3.3"
$ echo -e "$badlist"
1.1.1.1
2.2.2.2
3.3.3.3
then you have to modify the loop as follows:
$ echo -e "$badlist"|while read line; do
# do stuff with $line
done
This was an early dive into bash scripting for me the code was also placed remotely on a friends box, the last rough iteration I own of it is on my pastebin:
#!/bin/bash
# ..
# ..
# ..
## Variables
stamp=$(date "+%d/%m/%Y %T")
seed="$RANDOM-$RANDOM-IPTABLES-$(date "+%d-%m-%Y")-TEMPORY" ## proverbial sandpit
log=/root/.IPTables.log; touch $log ## Always a logfile
dmp=/tmp/IPT_DUMP$seed.temp ## Intermediate
list=/tmp/IPT_LIST$seed.txt ## F**ing '\r\r\n' regex'rs
pos=0
## Link(s)
link=http://au.hive.sshhoneypot.com/downloads/iplist.php
## Log File
function _tolog (){
echo -e "$stamp - $#\r" >> $log
}
## Leadin'
_tolog " "
_tolog "-----Running rottweiler : A simple IP deny auto script "
sh -c "iptables --flush"; _tolog "--OK Tables have been flushed"; sleep 1
## Grab-blacklist(s) # Fortran array HO!
function _populate (){
wget $link -O $dmp | egrep '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' | xargs; _tolog "--OK blacklist stored from honeypot"
tr -d '\r' < $dmp > $list # See above rage comment
while read line; do
arrayIp[$pos]=$line
((pos++))
done < $list
_tolog "--OK IP array created!"
_tolog $(echo -e "---- Array size: ${#arrayIp[*]}")
for item in ${arrayIp[*]}
do
sh -c "/sbin/iptables -I INPUT -s $item -j DROP" # This drops the current blacklist
done; _tolog "--OK Commands passed to iptables DB" # Use: /sbin/iptables -L -v -n to get list back quickly ( no resolving crap )
/sbin/iptables-save > /root/iptables.backup; _tolog "--OK Table database saved to flatfile"
}
_populate
_tolog "-----Terminating script: Tables logged in ~/iptables.backup"
Had similar issues resulting from Windows line endings (\r\n). Converting to unix endings (\n) solved my problem.
Cheers, /phfff