While creating a simple script that grabs a blacklist of ip addresses and blocks them, I came across this issue:
## Function giving greif
function _droplist (){
while read line; do
$IPT -A droplist -i eth1 -s $line -j LOG --log-prefix "IP BlockList "
$IPT -A droplist -i eth1 -s $line -j DROP
done < $badlist ##IPT is /sbin/iptables
}
Through several iterations of this function I get the error:
Try `iptables -h' or 'iptables --help' for more information.
' not found.4.12: host/network `SO.ME.IPH.ERE
Running the same script with hard coded in IP's works fine, its either something to do with $line or m implementation of iptables.
cheers -- Baffled.
What does $badlist contain? A file name or a list of IPs?
if it's a filename it should work as you did it, but if it's a list of ip you have to change how you read them.
Assuming it's a new-line-delimited list of IPs like:
$ badlist="1.1.1.1\n2.2.2.2\n3.3.3.3"
$ echo -e "$badlist"
1.1.1.1
2.2.2.2
3.3.3.3
then you have to modify the loop as follows:
$ echo -e "$badlist"|while read line; do
# do stuff with $line
done
This was an early dive into bash scripting for me the code was also placed remotely on a friends box, the last rough iteration I own of it is on my pastebin:
#!/bin/bash
# ..
# ..
# ..
## Variables
stamp=$(date "+%d/%m/%Y %T")
seed="$RANDOM-$RANDOM-IPTABLES-$(date "+%d-%m-%Y")-TEMPORY" ## proverbial sandpit
log=/root/.IPTables.log; touch $log ## Always a logfile
dmp=/tmp/IPT_DUMP$seed.temp ## Intermediate
list=/tmp/IPT_LIST$seed.txt ## F**ing '\r\r\n' regex'rs
pos=0
## Link(s)
link=http://au.hive.sshhoneypot.com/downloads/iplist.php
## Log File
function _tolog (){
echo -e "$stamp - $#\r" >> $log
}
## Leadin'
_tolog " "
_tolog "-----Running rottweiler : A simple IP deny auto script "
sh -c "iptables --flush"; _tolog "--OK Tables have been flushed"; sleep 1
## Grab-blacklist(s) # Fortran array HO!
function _populate (){
wget $link -O $dmp | egrep '[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}' | xargs; _tolog "--OK blacklist stored from honeypot"
tr -d '\r' < $dmp > $list # See above rage comment
while read line; do
arrayIp[$pos]=$line
((pos++))
done < $list
_tolog "--OK IP array created!"
_tolog $(echo -e "---- Array size: ${#arrayIp[*]}")
for item in ${arrayIp[*]}
do
sh -c "/sbin/iptables -I INPUT -s $item -j DROP" # This drops the current blacklist
done; _tolog "--OK Commands passed to iptables DB" # Use: /sbin/iptables -L -v -n to get list back quickly ( no resolving crap )
/sbin/iptables-save > /root/iptables.backup; _tolog "--OK Table database saved to flatfile"
}
_populate
_tolog "-----Terminating script: Tables logged in ~/iptables.backup"
Had similar issues resulting from Windows line endings (\r\n). Converting to unix endings (\n) solved my problem.
Cheers, /phfff
Related
I am running into an issue where I am comparing two files (alert.txt and env.txt) and based on common value, I am pulling complete line data from env.txt based on matching entry. I am reading these values into while loop and then invoking a function as follows. the ssh call is not working and also the while loop inside start_admin not working
#!/bin/bash
start_admin()
{
ssh -n -f $user#$host "sh -c 'cd $domain; ./script.sh > /dev/null 2>&1'"
while !(netstat -na | grep -E $host:$port|grep -E LISTEN) 2>/dev/null
sleep 30
do
echo "waiting"
done
echo "started"
}
grep -Ff alert.txt env.txt | (while IFS=" " read -r r1 r2 r3 r4 r5
do
user=$r2
host=$r3
domain=$r4
port=$r5
done
start_admin $user $host $domain $port
)
and contents of alert.txt is:
env2
env3
and that of env.txt is :
env1 user1 host1 /app/domain1/ port1
env2 user2 host2 /app/domain2/ port2
env3 user3 host3 /app/domain3/ port3
I could solve this with multiple if else loops, but that is not a desired solution, please guide me in right direction as to what is missing ?
Use join instead of grep here to avoid false positives
Because your while read loop completes before you run start_admin, you only launch it once (done should be AFTER start_admin)
In start_admin, don't use $user, $host and so on, use $1, $2 (or use them but don't pass them as parameters when calling the function)
I'm not sure exactly what you try to achieve, but here is a revised version already.
#!/bin/bash
start_admin()
{
sanitized_domain=${domain//'"'/'\"'}
ssh -n -f "$user#$host" "sh -c 'cd \"$sanitized_domain\"; ./script.sh >/dev/null 2>&1'"
while ! netstat -na | grep -q " $host:$port .*LISTEN"; do
echo waiting
sleep 30
done
echo started
}
join alert.txt env.txt | while IFS=' ' read -r env user host domain port; do
start_admin
done
)
Here is a sample bash script:
#!/bin/bash
array[0]="google.com"
array[1]="yahoo.com"
array[2]="bing.com"
pasteCommand="/usr/bin/paste -d'|'"
for val in "${array[#]}"; do
pasteCommand="${pasteCommand} <(echo \$(/usr/bin/dig -t A +short $val)) "
done
output=`$pasteCommand`
echo "$output"
Somehow it shows an error:
/usr/bin/paste: invalid option -- 't'
Try '/usr/bin/paste --help' for more information.
How can I fix it so that it works fine?
//EDIT:
Expected output is to get result from the 3 dig executions in a string delimited with | character. Mainly I am using paste that way because it allows to run the 3 dig commands in parallel and I can separate output using a delimiter so then I can easily parse it and still know the dig output to which domain (e.g google.com for first result) is assigned.
First, you should read BashFAQ/050 to understand why your approach failed. In short, do not put complex commands inside variables.
A simple bash script to give intended output could be something like that:
#!/bin/bash
sites=(google.com yahoo.com bing.com)
iplist=
for site in "${sites[#]}"; do
# Capture command's output into ips variable
ips=$(/usr/bin/dig -t A +short "$site")
# Prepend a '|' character, replace each newline character in ips variable
# with a space character and append the resulting string to the iplist variable
iplist+=\|${ips//$'\n'/' '}
done
iplist=${iplist:1} # Remove the leading '|' character
echo "$iplist"
outputs
172.217.18.14|98.137.246.7 72.30.35.9 98.138.219.231 98.137.246.8 72.30.35.10 98.138.219.232|13.107.21.200 204.79.197.200
It's easier to ask a question when you specify input and desired output in your question, then specify your try and why doesn't it work.
What i want is https://i.postimg.cc/13dsXvg7/required.png
$ array=("google.com" "yahoo.com" "bing.com")
$ printf "%s\n" "${array[#]}" | xargs -n1 sh -c '/usr/bin/dig -t A +short "$1" | paste -sd" "' _ | paste -sd '|'
172.217.16.14|72.30.35.9 98.138.219.231 98.137.246.7 98.137.246.8 72.30.35.10 98.138.219.232|204.79.197.200 13.107.21.200
I might try a recursive function like the following instead.
array=(google.com yahoo.com bing.com)
paster () {
dn=$1
shift
if [ "$#" -eq 0 ]; then
dig -t A +short "$dn"
else
paster "$#" | paste -d "|" <(dig -t A +short "$dn") -
fi
}
output=$(paster "${array[#]}")
echo "$output"
Now finally clear with expected output:
domains_arr=("google.com" "yahoo.com" "bing.com")
out_arr=()
for domain in "${domains_arr[#]}"
do
mapfile -t ips < <(dig -tA +short "$domain")
IFS=' '
# Join the ips array into a string with space as delimiter
# and add it to the out_arr
out_arr+=("${ips[*]}")
done
IFS='|'
# Join the out_arr array into a string with | as delimiter
echo "${out_arr[*]}"
If the array is big (and not just 3 sites) you may benefit from parallelization:
array=("google.com" "yahoo.com" "bing.com")
parallel -k 'echo $(/usr/bin/dig -t A +short {})' ::: "${array[#]}" |
paste -sd '|'
I'm really new to bash. I have a list of domains in a .txt file (URLs.txt). I also want to have a .csv file which consists of 3 columns separated by , (myFile.csv). My code reads each line of URLs.txt (each domain), finds its IP address and then inserts them into myFile.csv (domain in the first column, its IP in the 2nd column.
Name, IP
ex1.com, 10.20.30.40
ex2.com, 20.30.40.30
ex3.com, 10.45.60.20
ex4.com, 10.20.30.40
Here is my code:
echo "Name,IP" > myFile.csv # let's overwrite, not appending
while IFS= read -r line; do
ipValue= # initialize the value
while IFS= read -r ip; do
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
ipValue+="${ip}-" # append the results with "-"
fi
done < <(dig +short "$line") # assuming the result has multi-line
ipValue=${ipValue%-} # remove trailing "-" if any
if [[ -n $ipValue ]]; then
# if the IP is not empty
echo "$line,$ipValue" >> myFile.csv
fi
done < URLs.txt
I want to add another column to myFile.csv for keeping open ports of each IP. So output would be like this:
Name, IP, Port
ex1.com, 10.20.30.40, 21/tcp
ex2.com, 20.30.40.30, 20/tcp
ex3.com, 10.45.60.20, 33/tcp
ex4.com, 10.20.30.40, 21/tcp
I want to use Nmap to do this. After I choose an IP address from the 2nd column of myFile.csv and find its open ports using Nmap, I want to write the Nmap result to the corresponding cell of the 3rd column.
Also, if there is another similar IP in the 2nd column I want to write the Nmap result for that line too. I mean I don't want to run Nmap again for the duplicated IP. For example, in my example, there are two "10.20.30.40" in the 2nd column. I want to use Nmap just once and for the 1st "10.20.30.40" (and write the result for the 2nd "10.20.30.40" as well, Nmap should not be run for the duplicated IP).
For this to happen, I changed the first line of my code to this:
echo "Name,IP,Port" > myFile.csv
and also here is the Nmap code to find the open ports:
nmap -v -Pn -p 1-100 $ipValue -oN out.txt
port=$(grep '^[0-9]' out.txt | tr '\n' '*' | sed 's/*$//')
but I don't know what to do next and how to apply these changes to my code.
I updated my code to something like this:
echo "Name,IP" > myFile.csv # let's overwrite, not appending
while IFS= read -r line; do
ipValue= # initialize the value
while IFS= read -r ip; do
if [[ $ip =~ ^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$ ]]; then
ipValue+="${ip}-" # append the results with "-"
fi
done < <(dig +short "$line") # assuming the result has multi-line
ipValue=${ipValue%-} # remove trailing "-" if any
if [[ -n $ipValue ]]; then
# if the IP is not empty
nmap -v -Pn -p 1-100 $ipValue -oN out.txt
port=$(grep '^[0-9]' out.txt | tr '\n' '*' | sed 's/*$//')
echo "$line,$ipValue,$port" >> myFile.csv
fi
done < URLs.txt
but this way, Nmap was used for finding the open ports of the duplicated IPs too, but I didn't want this. What should I do?
Here's a modified version of your script that roughly does what you want:
#!/usr/bin/env bash
# cache maps from IP addresses to open ports
declare -A cache
getports() {
local ip=$1
nmap -v -Pn -p 1-100 "$ip" -oG - \
| awk -F '\t' '
/Ports:/ {
n = split($2, a, /,? /)
printf "%s", a[2]
for (i = 3; i <= n; ++i)
printf ":%s", a[i]
}
'
}
{
echo 'Name,IP,Port'
while IFS= read -r url; do
# Read filtered dig output into array
readarray -t ips < <(dig +short "$url" | grep -E '^([0-9]+\.){3}[0-9]+$')
# Build array of open ports
unset ports
for ip in "${ips[#]}"; do
ports+=("${cache["$ip"]:=$(getports "$ip")}")
done
# Output
printf '%s,%s,%s\n' \
"$url" \
"$(IFS='-'; echo "${ips[*]}")" \
"$(IFS='-'; echo "${ports[*]}")"
done < URLs.txt
} > myFile.csv
The readarray line reads the filtered output from dig into an array of IP addresses; if that array has length zero, the rest of the loop is skipped.
Then, for each elements in the ips array, we get the ports. To avoid calling nmap if we've seen the IP address before, we use the ${parameter:=word} parameter expansion: if ${cache["$ip"]} is non-empty, use it, otherwise call the getports function and store the output in the cache associative array.
getports is called for IP addresses we haven't seen before; I've used -oG ("grepable output") to make parsing easier. The awk command filters for lines containing Ports:, which look something like
Host: 52.94.225.242 () Ports: 80/open/tcp//http/// Ignored State: closed (99)
with tab separated fields. We then split the second field on the regular expression /,? / (an optional comma followed by a blank) and store all but the first field of the resulting array, colon separated.
Finally, we print the line of CSV data; if ips or ports contain more than one element, we want to join the elements with -, which is achieved by setting IFS in the command substitution and then printing the arrays with [*].
The initial echo and the loop are grouped within curly braces so output redirection has to happen just once.
I'm trying to build a csv of countries a list of ipv4 addresses come from.
I keep clobbering the IP in the output file;
#!/bin/bash
cat ipv4list.txt | while read ip ;do
echo -n "$ip", >> outputfile
whois -r "$ip">temp.txt
cat temp.txt | grep -i country >> outputfile
done
cat ipv4list.txt
1.1.1.1
2.2.2.2
What I'd like is outputfile to read;
1.1.1.1,country: AU
2.2.2.2,country: US
but I'm getting outputfile as follows;
,country: AU
,country: US
The echo statement need the refer to he ip variable:
echo -n "$ip", >> outputfile
Also, the clobbering indicate they input file may use window new lines (\r\n). Check the output file with editor, or hexdump
Going a bit outside the box, here. Please be kind, guys.
pop.ed:-
1d
wq
whois.sh:-
#!/bin/sh -x
init() {
cp ipv4list ipv4stack
}
next() {
[[ -s ipv4stack ]] && main
}
main() {
ip=$(echo "1p" | ed -s ipv4stack.txt)
wic=$(whois -r "${ip}")
echo "${ip},${wic}" >> outputfile
ed -s ipv4stack.txt < pop.ed
next
}
init
next
Ed apparently isn't installed by default in most distributions these days either, sadly; so you may need to install it if you want to use this.
I'm trying to deploy a cluster with a script which uses a yaml file. Except for an entry called "RaftFS" each yaml entry represents a machine to deploy. I don't understand why the script does only one while cycle if the ssh command is executed (even if the command is a simple ls !) but if I delete it then everything is fine and it does a number of cycle equals to the number of machines defined in the yaml file!
cat RaftFS/servers.yaml | shyaml keys-0 |
while read -r -d $'\0' value; do
if [ ! $value == "RaftArgs" ]; then
address=$(cat RaftFS/servers.yaml | shyaml get-value $value.machineIP | xargs -0 -n 1 echo)
username=$(cat RaftFS/servers.yaml | shyaml get-value $value.username | xargs -0 -n 1 echo)
password=$(cat RaftFS/servers.yaml | shyaml get-value $value.password | xargs -0 -n 1 echo)
#uploading my fingerprint (in order to use pssh)
echo $address $username $password
echo "uploading my fingerprint on $username#$address $password"
sshpass -p $password ssh-copy-id -oStrictHostKeyChecking=no $username#$address
echo "creating RaftFS"
ssh $username#$address echo "MACHINE=$value vagrant up">>vagrantscript.sh
fi
echo $address $username $password
done
I think there is no issue with the ssh command but it is a delimiter's issue
I've played a little with read -r -d $'\0' and these are the results
echo "a\0b\0c" | while read -r -d $'\0' var; do echo $var; done
prints
a
b
and
echo "a\0b\0c\0" | while read -r -d $'\0' var; do echo $var; done
prints
a
b
c
I assume there is some difference in the end line when the $value == "RaftArgs"
The standard input to the while loop is also the standard input to every command within the while loop. ssh reads from standard input in order to pipe the data to the remote command. It's probably consuming the data intended for the read statement.
You can redirect the ssh command's input:
ssh $username#$address ... >>vagrantscript.sh < /dev/null
Or you can run ssh with the "-n" flag to prevent reading from stdin:
ssh -n $username#$address ... >>vagrantscript.sh