Passing multiple arguments from input file, to a command multiple times (Bash) - bash

I have found myself in several situations, where it would be handy to be able to feed a command arguments from an input file, on a per line basis. In the handful of times I've wanted to be able to do this, I've ended up finding a workaround or running the command multiple times manually.
I have a file input.txt which contains multiple lines, with an arbitrary number of arguments on each line. I am going to use my most recent need for this functionality as an example. I am trying to simply copy my iptables rules to ip6tables. I have run the command iptables -S > input.txt to generate the following file:
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT DROP
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m state --state ESTABLISHED -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -m state --state ESTABLISHED -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A OUTPUT -o lo -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 53 -m state --state NEW -j ACCEPT
-A OUTPUT -p udp -m udp --dport 53 -m state --state NEW -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 80 -m state --state NEW -j ACCEPT
-A OUTPUT -p tcp -m tcp --dport 443 -m state --state NEW -j ACCEPT
For each line, I would like to run something like sudo ip6tables $line, so that ip6tables runs with all the arguments from a given line, and N commands are run, where N is the number of lines in input.txt.
The answer to this question should be applicable to this scenario in general. I am not looking for a solution that only works for copying iptables rules to the IPv6 counterpart (through some functionality in iptables for example). I would also prefer a one liner that I can execute in the terminal if possible, instead of writing a bash script. Though, if a bash script turns out to be much less cumbersome than a one liner, then I would accept that as an answer as well.

This sounds like the perfect job for xargs. For your iptables example it works like this:
xargs -L 1 sudo ip6tables < input.txt
xargs reads command arguments from stdin and executes the provided command with the arguments added to the command line. Here the arguments are piped in to stdin from the input.txt file. -L 1 limits the arguments to one line per execution, i.e. one line is added to the command line, the resulting command gets executed, continue with the next line etc.

If it is just for iptables your system may already have an appropriate iptables-restore command you just have to call with:
sudo iptables-restore input.txt
Now if input.txt is to be used to pass arguments to other commands, there i s an easy solution with Bash:
arguments is an array that is read (filled) from each line of input.txt
Iterate over each line of the input.txt file.
Pass the arguments array to your command like this:
#!/usr/bin/env bash
arguments=()
while read -r -a arguments; do
your_command "${arguments[#]}"
done <input.txt
for a one-liner of the above:
while read -ra a;do your_command "${a[#]}";done<input.txt
Or for a POSIX compatible version (no array):
while read -r a;do set -- $a; your_command "$#";done <input.txt

Related

iptables rules is this correct? [duplicate]

This question already has answers here:
Are shell scripts sensitive to encoding and line endings?
(14 answers)
Closed 2 years ago.
I input this from a bash script
#!/bin/bash
#
# iptables example configuration script
# Drop ICMP echo-request messages sent to broadcast or multicast addresses
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
# Drop source routed packets
echo 0 > /proc/sys/net/ipv4/conf/all/accept_source_route
# Enable TCP SYN cookie protection from SYN floods
echo 1 > /proc/sys/net/ipv4/tcp_syncookies
# Don't accept ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/accept_redirects
# Don't send ICMP redirect messages
echo 0 > /proc/sys/net/ipv4/conf/all/send_redirects
# Enable source address spoofing protection
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
# Log packets with impossible source addresses
echo 1 > /proc/sys/net/ipv4/conf/all/log_martians
# Flush all chains
/sbin/iptables --flush
# Allow unlimited traffic on the loopback interface
/sbin/iptables -A INPUT -i lo -j ACCEPT
/sbin/iptables -A OUTPUT -o lo -j ACCEPT
# Set default policies
/sbin/iptables --policy INPUT DROP
/sbin/iptables --policy OUTPUT DROP
/sbin/iptables --policy FORWARD DROP
# Previously initiated and accepted exchanges bypass rule checking
# Allow unlimited outbound traffic
/sbin/iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A OUTPUT -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
/sbin/iptables -A INPUT -p tcp --dport 69 -m state --state NEW -m recent --update --seconds 60 --hitcount 4 -j DROP
/sbin/iptables -A INPUT -p tcp --dport 69 -m state --state NEW -m recent --set
/sbin/iptables -A INPUT -p tcp --dport 69 -m state --state NEW -j ACCEPT
# Allow certain ports to be accessible from the outside
/sbin/iptables -A INPUT -p tcp --dport 25565 -m state --state NEW -j ACCEPT #Minecraft
/sbin/iptables -A INPUT -p tcp --dport 1688 -m state --state NEW -j ACCEPT #Dynmap plugin
# Other rules for future use if needed. Uncomment to activate
/sbin/iptables -A INPUT -p tcp --dport 80 -m state --state NEW -j ACCEPT # http
/sbin/iptables -A INPUT -p tcp --dport 443 -m state --state NEW -j ACCEPT # https
# UDP packet rule. This is just a random udp packet rule as an example only
# /sbin/iptables -A INPUT -p udp --dport 5021 -m state --state NEW -j ACCEPT
# Allow pinging of your server
/sbin/iptables -A INPUT -p icmp --icmp-type 8 -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
# Drop all other traffic
/sbin/iptables -A INPUT -j DROP
# print the activated rules to the console when script is completed
/sbin/iptables -nL
and get output of this
firewall.sh: line 38: DROP: command not found
firewall.sh: line 39: tcp: command not found
firewall.sh: line 43: -p: command not found
firewall.sh: line 46: --dport: command not found
its weird im migrating servers and on the old one this script ran fine is something wrong with the script that im not seeing? What i am hosting on is a pi4 8gb with raspibian x64 is it possible that is giving me the issue with iptables currently? Or is it the code?
The error pointed by you is most likely caused by window-style line ending present in your file. you can try to use cat -A <filename> to debug and use the following command to convert your file with Linux style line endings.
dos2unix <file>

Expanding to a flag only if value is set

I have some logic like:
if [[ -n "$SSH_CLIENT" ]]
then
sflag="-s $(echo "$SSH_CLIENT" | awk '{ print $1}')"
else
sflag=''
fi
iptables -A MY_RULE "$sflag" -p tcp -m tcp --dport 9999 -m conntrack -j ACCEPT
In other words, I want to mimic only passing the -s flag to iptables if SSH_CLIENT is set. What actually happens is that the empty string is inadvertently passed.
I'm interested in whether it is possible, in the interest of not repeating two quite long iptables calls, to expand the flag name and value. E.g. the command above should expand to
iptables -A MY_RULE -s 10.10.10.10 -p tcp -m tcp ..., or
iptables -A MY_RULE -p tcp -m tcp ...
The problem is that in the second case, the expansion actually becomes:
iptables -A MY_RULE '' -p tcp -m tcp
and there is an extra empty string that is treated as a positional argument. How can I achieve this correctly?
All POSIX shells: Using ${var+ ...expansion...}
Using ${var+ ...words...} lets you have an arbitrary number of words only if a variable is set:
iptables -A MY_RULE \
${SSH_CLIENT+ -s "${SSH_CLIENT%% *}"} \
-p tcp -m tcp --dport 9999 -m conntrack -j ACCEPT
Here, if-and-only-if SSH_CLIENT is set, we add -s followed by everything in SSH_CLIENT up to the first space.
Bash (and other extended shells): Using Arrays
The more general approach is to use an array whenever you want to represent multiple strings as a single value:
ssh_client_args=( )
[[ $SSH_CLIENT ]] && ssh_client_args+=( -s "${SSH_CLIENT%% *}" )
iptables -A MY_RULE "${ssh_client_args[#]}" -p tcp -m tcp --dport 9999 -m conntrack -j ACCEPT
The syntax "${var%% *} is a parameter expansion which expands to var with the longest possible suffix starting with a space trimmed; thus, leaving the first word. This is much faster than running an external program like awk. See also BashFAQ #100 describing general best practices for native-bash string manipulation.
This is one of those occasions where you probably don't want to quote the variable expansion:
iptables -A MY_RULE $sflag -p tcp -m tcp ...
Alternatively, and more robustly, we can use ${var+...} expansion:
iptables -A MY_RULE ${SSH_CLIENT:+-s "${SSH_CLIENT%% *}"} -p tcp -m tcp ...
Read this as "if $SSH_CLIENT is set (and not null) then expand and substitute -s "${SSH_CLIENT%% *}", else nothing".
Note that we don't quote the $.+ expansion, but we do quote the individual arguments within it where needed (obviously -s doesn't need quotes, although you're free to add them if you want).
I removed the external awk command, as $.%% expansion is the simpler and more efficient way to truncate a string.

Extra single quote is appended in shell script

I am trying to write a simple script that loads the iptables rules from a file to the iptables:
#!/bin/bash
set -euxo pipefail
test_file="test.conf"
while read line; do
echo $line
echo "-----"
iptables $line
done < $test_file
And the test.conf has some rules inside:
-A INPUT -p tcp -m tcp --dport 123 -m comment --comment "test 1" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 112 -m comment --comment "test 2" -j ACCEPT
-A INPUT -p tcp -m tcp --dport 1231 -m comment --comment "test 3" -j ACCEPT
But when i run the script, it returns error as:
+ test_file=rules.conf
+ read line
+ echo -A INPUT -p tcp -m tcp --dport 123 -m comment --comment '"test' '1"'
-j ACCEPT
-A INPUT -p tcp -m tcp --dport 123 -m comment --comment "test 1" -j ACCEPT
+ echo -----
-----
+ iptables -A INPUT -p tcp -m tcp --dport 123 -m comment --comment '"test'
'1"' -j ACCEPT
Bad argument `1"'
Looks like whenever the file is run and if there is a space in the string, it will have a single quote to wrap around the space.
Is there a way to get rid of the behavior?
The last line is the main problem. The shell doesn't interpret quote marks during variable expansion. If you change that to
eval iptables $line
you should get (roughly) the right results.
Your read statement is also potentially problematic because you didn't set IFS to a newline first. This could cause multiple whitespace to be collapsed and other possibly undesirable behavior. So I would start with
IFS="
"
or similar.

IPTables Script to block Concurrent Connections

We are using Suse Linux Enterprise Server 12. We need to block concurrent IP Addresses which is hitting our web server for more thatn 50 times per second and block that ip address for 10 minutes. Also it should distinguish attacker and genuine traffic and block attacker's IP forever. We have currently blocked using iptables , below is the rule.
iptables -I INPUT -p tcp --dport 443 -i eth0 -m state --state NEW -m recent --set
iptables -I INPUT -p tcp --dport 443 -i eth0 -m state --state NEW -m recent --update --seconds 1 --hitcount 50 -j DROP
It will just block the IPAddress which exceeds 50 connections but wont blacklist the IPAddress. Please let us know if we have a script that will match all the scenarios which is metioned above. Please Help.
I tested this and it works really nice. If the behavior is detected, the IP is put into hold-down for 10 minutes and logged. You can verify it's operation by watching these files. /proc/net/xt_recent/NICE, /proc/net/xt_recent/NAUGHTY. You need to build a script to parse the log for bad IP's and commit them to a file that is loaded into iptables on startup if you want to blacklist permanently. That concept is already clear so no need for me to include it.
#flush and clear
iptables -F -t nat
iptables -F
iptables -X
#this is where naughty kids go
iptables -N GETCAUGHT
#you got added to the naughty list
iptables -A GETCAUGHT -m recent --name NAUGHTY --set #everyone here is bad
iptables -A GETCAUGHT -j LOG --log-prefix "iwasbad: " --log-level 4 #and it goes on your permanent record
#if you are on the NAUGHTY list you get a lump of coal
iptables -A INPUT -i eth0 -m recent --name NAUGHTY --rcheck --seconds 600 -j DROP #check everyone at the door
#though everyone starts out on the NICE list
iptables -A INPUT -i eth0 -p tcp --dport 443 -m conntrack --ctstate NEW -m recent --name NICE --set #you seem nice
#but if you GETCAUGHT doing this you are naughty
iptables -A INPUT -i eth0 -p tcp --dport 443 -m conntrack --ctstate NEW -m recent --name NICE --seconds 1 --hitcount 50 --update -j GETCAUGHT #that wasn't nice

Getting unterminated 's' command with sed (bash)

My final goal is to get a bunch of text above the COMMIT line in /etc/ufw/before.rules. I am trying to replace the COMMIT with something like this:
TEXT I WANT\nCOMMIT
I see this as an easy sed command: echo COMMIT | sed "s#COMMIT#${morerules}#g"
I have set the variable morerules to a string like this:
morerules="""
### Start HTTP ###
# Enter rule
-A ufw-before-input -p tcp --dport 80 -j ufw-http
-A ufw-before-input -p tcp --dport 443 -j ufw-http
# Limit connections per Class C
-A ufw-http -p tcp --syn -m connlimit --connlimit-above 50 --connlimit-mask 24 -j ufw-http-logdrop
# Limit connections per IP
-A ufw-http -m state --state NEW -m recent --name conn_per_ip --set
-A ufw-http -m state --state NEW -m recent --name conn_per_ip --update --seconds 10 --hitcount 20 -j ufw-http-logdrop
# Limit packets per IP
-A ufw-http -m recent --name pack_per_ip --set
-A ufw-http -m recent --name pack_per_ip --update --seconds 1 --hitcount 20 -j ufw-http-logdrop
# Finally accept
-A ufw-http -j ACCEPT
# Log
-A ufw-http-logdrop -m limit --limit 3/min --limit-burst 10 -j LOG --log-prefix \"[UFW HTTP DROP]\"
-A ufw-http-logdrop -j DROP
### End HTTP ##
"""
As you can see, I added a backslash to the quotations marks. Setting the variable is successful. When I run the command from above echo COMMIT | sed "s#COMMIT#${morerules}#g", this is the output:
sed: -e expression #1, char 9: unterminateds' command`
Is the error with my string or with my sed command? Also, I know other people have gotten this error, but none of their fixes seem to work.
To place a literal backslash before each newline in your replacement text, thus telling sed that it should be considered part of the same expression:
orig=$'\n'; replace=$'\\\n'
sed "s#COMMIT#${morerules//$orig/$replace}#g"
(Placing the values in variables makes it easier to implement this in a way that works across multiple versions of bash).

Resources