Using bash to generate random IPs Issue - bash

I have a list of URL and I'm trying construct a firewall log with that list.
Example:
$ cat urls.csv
a.com
I know how to construct the log with IP directly mentioned as variable.
$ cat processor.sh
#!/bin/bash
filename="$1"
while read -r line
do
URLS="$line"
IP='10.109.1.1'
today_date=`date +%d/%b/%Y`
conact_1=" - - ["
concat_2=":00:00:00 +0000] "
date_concat=$conact_1$today_date$concat_2
GET='"GET '
protocol=' HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0'
final_url=$IP$conact_1$today_date$concat_2$GET$URLS$protocol
echo $final_url
done < "$filename"
Result:
$ bash processor.sh urls.csv
10.109.1.1 - - [22/Jul/2018:00:00:00 +0000] "GET a.com HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0
Now If i have a log line of 50 or 100 or even more , How can I generate random IP for the total number of URLs and generate a log line ?
$cat urls.csv
a.com
b.com
c.com
d.com
....
Any suggestions on how to generate random IPs in bash?
Expected Result:
$ bash processor.sh urls.csv
1.1.1.1 - - [22/Jul/2018:00:00:00 +0000] "GET a.com HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0
1.1.1.2 - - [22/Jul/2018:00:00:00 +0000] "GET b.com HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0
1.1.1.3 - - [22/Jul/2018:00:00:00 +0000] "GET c.com HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0
1.1.1.4 - - [22/Jul/2018:00:00:00 +0000] "GET d.com HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0
............

For IPv4, the basics would be:
printf "%d.%d.%d.%d\n" "$((RANDOM % 256))" "$((RANDOM % 256))" "$((RANDOM % 256))" "$((RANDOM % 256))"
You'll probably want to filter out 192.168.0.0/16 and 10.0.0.0/8 and a lot of other addresses. For IPv6, you might try:
for ((i=0;i<8;i++)); do printf "%02x%02x:" $((RANDOM%256)) $((RANDOM%256)); done | sed 's/:$//'
($RANDOM only generates values up to 2^15-1, so $((RANDOM%65536)) is pointless)

Works as expected based on Inder's answer.
$cat processor.sh
#!/bin/bash
filename="$1"
while read -r line
do
URLS="$line"
#IP=$(printf "%d.%d.%d.%d\n" "$((RANDOM % 256 ))")
IP=$(printf "%d.%d.%d.%d\n" "$((RANDOM % 256 ))" "$((RANDOM % 256 ))" "$((RANDOM % 256 ))" "$((RANDOM % 256 ))")
#IP='10.109.1.1'
today_date=`date +%d/%b/%Y`
conact_1=" - - ["
concat_2=":00:00:00 +0000] "
date_concat=$conact_1$today_date$concat_2
GET='"GET '
protocol=' HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0'
final_url=$IP$conact_1$today_date$concat_2$GET$URLS$protocol
echo $final_url
done < "$filename"
$ bash processor.sh input.csv
103.137.94.111 - - [22/Jul/2018:00:00:00 +0000] "GET a.com HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0
74.123.232.234 - - [22/Jul/2018:00:00:00 +0000] "GET b.com HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0
102.13.8.154 - - [22/Jul/2018:00:00:00 +0000] "GET c.com HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0
210.106.160.230 - - [22/Jul/2018:00:00:00 +0000] "GET d.com HTTP/1.1" 304 0 304 0 0 0 655 456 645 368 0

Related

Strange requests from VPS IP address

In the Apache logs, I found out that strange requests are coming from the IP address of my VPS, which I did not make. Usually, requests from my VPS that I run through the CRON scheduler are as follows.
domain.ru:443 **.**.**.** - - [20/Sep/2021:19:55:01 +0300] "GET /test.php HTTP/1.0" 200 421 "-" "Wget/1.19.4 (linux-gnu)" 118650
Strange queries look like this
domain.ru:80 **.**.**.** - - [21/Sep/2021:09:06:52 +0300] "GET / HTTP/1.0" 400 0 "-" "-" 48
domain.ru:80 **.**.**.** - - [21/Sep/2021:08:10:59 +0300] "GET / HTTP/1.0" 400 0 "-" "-" 53
domain.ru:80 **.**.**.** - - [21/Sep/2021:07:27:17 +0300] "GET /boaform/admin/formLogin?username=adminisp&psd=adminisp HTTP/1.0" 400 0 "-" "-" 51
domain.ru:80 **.**.**.** - - [21/Sep/2021:06:25:03 +0300] "GET / HTTP/1.0" 400 0 "-" "-" 145
domain.ru:80 **.**.**.** - - [21/Sep/2021:04:11:17 +0300] "GET / HTTP/1.0" 400 0 "-" "-" 41
domain.ru:80 **.**.**.** - - [21/Sep/2021:02:52:44 +0300] "GET / HTTP/1.0" 400 0 "-" "-" 41
domain.ru:80 **.**.**.** - - [21/Sep/2021:02:36:17 +0300] "GET / HTTP/1.0" 400 0 "-" "-" 41
domain.ru:80 **.**.**.** - - [21/Sep/2021:01:51:52 +0300] "GET / HTTP/1.0" 400 0 "-" "Mozilla/5.0" 38
These requests are especially alarming.
domain.ru:80 **.**.**.** - - [21/Sep/2021:07:27:17 +0300] "GET /boaform/admin/formLogin?username=adminisp&psd=adminisp HTTP/1.0" 400 0 "-" "-" 51
domain.ru:80 **.**.**.** - - [21/Sep/2021:01:51:52 +0300] "GET / HTTP/1.0" 400 0 "-" "Mozilla/5.0" 38
domain.ru:80 **.**.**.** - - [20/Sep/2021:19:51:34 +0300] "GET / HTTP/1.0" 400 0 "-" "YahooBot" 54
As you can see, UserAgent YahooBot and Mozilla / 5.0 are transmitted, and even a strange request to the page /boaform/admin/formLogin?username=adminisp&psd=adminisp
Tell me what to do. Is it a virus?
These are bots probing for vulnerabilities. I suggest you modify LogFormat to include the client IP in case you want to block those. Here is what I use:
LogFormat "%h %l %u %t \"%r\" %>s %B \"%{Referer}i\" \"%{User-Agent}i\"" combined
CustomLog ${APACHE_LOG_DIR}/access.log combined
Here is the relevant documentation for LogFormat variables. If you want, the next step is to block abuse traffic. I use fail2ban for that. Consider enabling the unique_id module, that way you can log that id in both access and error log.

how to use cat | awk | xargs sed to replace a char

I wrote a little bash script to parse Apache Access log to count POST|GET request.
My script works fine but I have a little graphical issue when I want to remove "[" char from the date field return by awk command.
Here is my script:
clear
ls /var/log/httpd | egrep *access_log$ > temp.txt
while read line
do
linecount=$(cat /var/log/httpd/"$line" | wc -l)
#echo -e "$line"
#echo -e "$linecount"
if [ $linecount -gt 0 ]
then
echo -e "==========================================="
echo -e "$line"
echo -e "Date de debut du log :"
cat /var/log/httpd/"$line" | awk -v ligne=1 'NR == ligne, FS=":" {print $4}' | xargs -0 sed -i 's/\[//g'
echo -e "Date de fin du log :"
cat /var/log/httpd/"$line" | awk 'END {print $4}'
echo -e "Nombre de requêtes sur la période :"
egrep -i 'post|get' /var/log/httpd/"$line" | wc -l
fi
linecount=0
done < temp.txt
rm -f temp.txt
An example of standard output of this code looks like this :
===========================================
xxx.xxx.xxx-ssl_access_log
Date de debut du log :
sed: impossible de lire [01/Jan/2021:07:34:59
: Aucun fichier ou dossier de ce type
Date de fin du log :
[22/Jan/2021:07:44:44
Nombre de requêtes sur la période :
22
Why can't sed use the string piped by awk?
How can I correct it ?
Below an example of log imput file :
54.36.148.55 - - [29/Dec/2020:18:05:38 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.149.92 - - [29/Dec/2020:18:05:38 +0100] "GET / HTTP/1.1" 200 2394
54.36.148.185 - - [30/Dec/2020:17:51:06 +0100] "GET / HTTP/1.1" 200 2394
54.36.149.77 - - [31/Dec/2020:17:19:18 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.148.97 - - [31/Dec/2020:17:19:19 +0100] "GET / HTTP/1.1" 200 2394
54.36.149.61 - - [01/Jan/2021:14:45:59 +0100] "GET / HTTP/1.1" 200 2394
54.36.148.151 - - [02/Jan/2021:16:26:22 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.148.71 - - [02/Jan/2021:16:26:24 +0100] "GET / HTTP/1.1" 200 2394
54.36.148.108 - - [03/Jan/2021:15:21:28 +0100] "GET / HTTP/1.1" 200 2394
208.100.26.249 - - [03/Jan/2021:23:15:13 +0100] "GET / HTTP/1.1" 200 2394
54.36.149.95 - - [04/Jan/2021:15:28:31 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.148.202 - - [04/Jan/2021:15:28:32 +0100] "GET / HTTP/1.1" 200 2394
54.36.149.24 - - [05/Jan/2021:14:44:52 +0100] "GET / HTTP/1.1" 200 2394
54.36.148.184 - - [06/Jan/2021:15:00:55 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.149.54 - - [06/Jan/2021:15:00:55 +0100] "GET / HTTP/1.1" 200 2394
54.36.148.185 - - [07/Jan/2021:14:03:13 +0100] "GET / HTTP/1.1" 200 2394
51.158.103.247 - - [08/Jan/2021:12:31:33 +0100] "GET / HTTP/1.1" 200 2394
54.36.148.17 - - [08/Jan/2021:14:10:18 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.148.185 - - [08/Jan/2021:14:10:19 +0100] "GET / HTTP/1.1" 200 2394
54.36.148.101 - - [09/Jan/2021:14:17:39 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.148.94 - - [09/Jan/2021:14:17:40 +0100] "GET / HTTP/1.1" 200 2394
54.36.148.103 - - [10/Jan/2021:15:21:24 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.148.68 - - [10/Jan/2021:15:21:24 +0100] "GET / HTTP/1.1" 200 2394
54.36.148.208 - - [11/Jan/2021:18:15:40 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.149.78 - - [11/Jan/2021:18:15:41 +0100] "GET / HTTP/1.1" 200 2394
54.36.148.64 - - [12/Jan/2021:20:37:08 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.149.38 - - [12/Jan/2021:20:37:09 +0100] "GET / HTTP/1.1" 200 2394
54.36.149.66 - - [13/Jan/2021:20:40:09 +0100] "GET /robots.txt HTTP/1.1" 404 159
54.36.148.203 - - [13/Jan/2021:20:40:10 +0100] "GET / HTTP/1.1" 200 2394
51.158.127.119 - - [14/Jan/2021:11:41:05 +0100] "GET / HTTP/1.1" 200 2394
51.15.251.143 - - [14/Jan/2021:11:52:04 +0100] "GET / HTTP/1.1" 200 2394
54.36.149.76 - - [14/Jan/2021:20:05:36 +0100] "GET / HTTP/1.1" 200 2394
208.100.26.243 - - [18/Jan/2021:10:20:00 +0100] "GET / HTTP/1.1" 200 2394
208.100.26.248 - - [25/Jan/2021:04:10:37 +0100] "GET / HTTP/1.1" 200 2394
Using awk as a "complete" solution
awk 'FNR==1 {
gsub("[[]","",$4);
sdat=$4 # When the file record number (FNR) is 1, remove [ from the 4th space separated field with gsub and set sdat to this field
}
ENDFILE {
gsub("[[]","",$4);
fdat=$4; # When we reach the end of each file, remove [ gain from the 4th field and set fdat to this field
print "==========================================="
print FILENAME # Print the filename using awk's FILENAME variable
print "Date de debut du log :" # Print the data required
print sdat
print "Date de fin du log :"
print fdat
print "Nombre de requêtes sur la période :"
print FNR # Print the total number of records in the file (file number record)
} ' /var/log/httpd/*access_log
# user15097052 : you'll absolutely love the insane power afforded by AWK. It's great because of its simplicity - it doesn't come with every bell and whistle, but for the building blocks it does, they do it REALLY well.
These days I pretty much avoid touching wc, sed, cut, and the majority of the time, I prefer not having to deal with perl or python3. The URL encode/decode module on python3 slows me down compared to awk.

IIS Express site dosnt start on VS F5

Using Episerver in VS 2017
My site runs in IIS Express. When I start the project from VS in Chrome, I get the default IIS Screen (The blue screen with IIS info) - so IIS starts, but the site does not launch. If I reload the browser, the site launches.
Here is my site definition in the applicationhost.config:
<site name="MySite" id="3">
<application path="/" applicationPool="Clr4IntegratedAppPool">
<virtualDirectory path="/" physicalPath="C:\Projects\mysite-episerver\src\MySite" />
</application>
<bindings>
<binding protocol="http" bindingInformation="*:80:mysite.mydomain.com" />
</bindings>
</site>
Update
Looking at the log:
Software: Microsoft Internet Information Services 10.0
Version: 1.0
Date: 2018-05-25 18:09:24
Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2018-05-25 18:09:24 127.0.0.1 GET / - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 - 302 0 0 13991
2018-05-25 18:09:31 127.0.0.1 GET /personal/ - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 - 200 0 0 6389
2018-05-25 18:09:31 127.0.0.1 GET /dist/styles/main.css v=636627702363674333 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 37
2018-05-25 18:09:31 127.0.0.1 GET /dist/fonts/stylesheet.css - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 29
2018-05-25 18:09:31 127.0.0.1 GET /EPiServer.Forms/DataSubmit/GetFormInitScript formGuid=6bd570d3-3e91-447f-903e-566e9d9f9468&formLanguage=en 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 171
2018-05-25 18:09:31 127.0.0.1 GET /WebResource.axd d=TzRGlCddaaqtz0Im2nSJhBlH3P1Fze7Czb4HaEn0ZkkpUFjLB2tREFUTizpsCAfxQQOQDZCZqUiBZLAo7l7EhsGTwsuUFnRXRabRz6n3XcvVK2Bd0VQe-ZDaVHtSCoc3paD6qdAAJwaIPHOv9FJMVOt6_eqfG_NNb1zepx_9qUk1&t=636601600300000000 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 81
2018-05-25 18:09:31 127.0.0.1 GET /dist/scripts/sticky-header.js v=636627702363754120 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 130
2018-05-25 18:09:31 127.0.0.1 GET /dist/scripts/main.js v=636627702363724200 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 149
2018-05-25 18:09:31 127.0.0.1 GET /dist/scripts/forms.js v=636627702363704257 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 165
2018-05-25 18:09:31 127.0.0.1 GET /siteassets/settings/logo_mysiteusa.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 171
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/profile.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 185
2018-05-25 18:09:31 127.0.0.1 GET /siteassets/image-blocks/785x650.png - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 181
2018-05-25 18:09:31 127.0.0.1 GET /siteassets/heros/180800277.jpg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 294
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/BG-Pattern.png - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 70
2018-05-25 18:09:31 127.0.0.1 GET /siteassets/image-and-text-blocks/bbb.png - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 217
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/chevron.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 65
2018-05-25 18:09:31 127.0.0.1 GET /siteassets/image-and-text-blocks/bestmysite.png - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 255
2018-05-25 18:09:31 127.0.0.1 GET /siteassets/settings/logo_ehl.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 253
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/jumplink.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 89
2018-05-25 18:09:31 127.0.0.1 GET /siteassets/settings/footer-mountains.png - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 112
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/footer_linkedin.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 38
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/icon_quotation.png - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 37
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/footer_twitter.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 76
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/doublearrow.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 173
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/footer_facebook.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 68
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/mobilephone.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 43
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/email.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 115
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/footer_youtube.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 200
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/doublearrow-white.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 105
2018-05-25 18:09:31 127.0.0.1 GET /dist/images/worldwide.svg - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/dist/styles/main.css?v=636627702363674333 200 0 0 115
2018-05-25 18:09:31 127.0.0.1 GET /util/LiveMonitor/Tracking/Index contentId=34&languageId=en&r=0.6407196012207004&referrer= 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 131
2018-05-25 18:09:32 127.0.0.1 GET /util/LiveMonitor/Tracking/Index contentId=34&languageId=en&r=0.5590579783850214&referrer= 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/personal/ 200 0 0 565
2018-05-25 18:09:36 127.0.0.1 GET /login/ - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 - 200 0 0 223
2018-05-25 18:09:36 127.0.0.1 GET /util/LiveMonitor/Tracking/Index contentId=271&languageId=en&r=0.29442535680181825&referrer= 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/login/ 200 0 0 55
2018-05-25 18:09:36 127.0.0.1 GET /util/LiveMonitor/Tracking/Index contentId=271&languageId=en&r=0.21276262992917028&referrer= 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/login/ 200 0 0 578
2018-05-25 18:09:52 127.0.0.1 POST /login/post/ - 80 - 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/login/ 302 0 0 443
2018-05-25 18:09:52 127.0.0.1 GET /login/post/mysiteusa/cms/ - 80 admin 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/login/ 404 0 0 506
2018-05-25 18:09:52 127.0.0.1 GET /util/LiveMonitor/Tracking/Index contentId=9&languageId=en&r=0.2522424054704595&referrer=http%3A//mysite.mydomain.com/login/ 80 admin 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/login/post/%2Fmysiteusa%2Fcms%2F 200 0 0 37
2018-05-25 18:09:52 127.0.0.1 GET /util/LiveMonitor/Tracking/Index contentId=9&languageId=en&r=0.9305181288066644&referrer=http%3A//mysite.mydomain.com/login/ 80 admin 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/login/post/%2Fmysiteusa%2Fcms%2F 200 0 0 42
2018-05-25 18:09:52 127.0.0.1 GET /epiutil/images/quicknav/quicknav-arrow.png - 80 admin 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/66.0.3359.181+Safari/537.36 http://mysite.mydomain.com/epiutil/styles/quicknavigator.css 200 0 0 57
<WebProjectProperties>
<UseIIS>True</UseIIS>
<AutoAssignPort>True</AutoAssignPort>
<DevelopmentServerPort>0</DevelopmentServerPort>
<DevelopmentServerVPath>/</DevelopmentServerVPath>
<IISUrl>http://goalaskausa.mydelphic.com</IISUrl>
<NTLMAuthentication>False</NTLMAuthentication>
<UseCustomServer>False</UseCustomServer>
<CustomServerUrl>
</CustomServerUrl>
<SaveServerSettingsInUserFile>False</SaveServerSettingsInUserFile>
</WebProjectProperties>
UseIIS keeps getting reset to true. I think whats happening is that its hitting IIS, not IIS express. Then when I refresh it hits IIS express.

Summing downloaded data by IP addresses

I have this file:
2001:778:0:1::21 - - [16/Sep/2011:12:30:46 +0300] "GET / HTTP/1.1" 200 44
2001:778:0:1::21 - - [16/Sep/2011:12:30:46 +0300] "GET /favicon.ico HTTP/1.1" 200 1406
2001:778:0:1::21 - - [16/Sep/2011:12:32:15 +0300] "GET / HTTP/1.1" 200 66643
88.222.10.7 - - [17/Sep/2011:23:39:25 +0300] "GET / HTTP/1.1" 200 66643
88.222.10.7 - - [17/Sep/2011:23:39:25 +0300] "GET /favicon.ico HTTP/1.1" 200 1406
88.222.10.7 - - [18/Sep/2011:13:45:39 +0300] "GET / HTTP/1.1" 304 -
88.222.10.7 - - [19/Sep/2011:05:47:35 +0300] "GET / HTTP/1.1" 200 66643
88.222.10.7 - - [19/Sep/2011:05:47:36 +0300] "GET /favicon.ico HTTP/1.1" 200 1406
121.141.172.40 - - [19/Sep/2011:20:32:07 +0300] "CONNECT 64.12.202.43:443 HTTP/1.0" 405 235
And I have IP addresses data (last number in each line), for example 44, 1406, 66643, 6664, .....
I want to sum all data that belongs to same IP address. So my results should be:
2001:778:0:1::21 68093 (44+1406+66643)
88.222.10.7 136098 (66643+1406+66643+1406)
121.141.172.40 235 (235)
Is is possible to do that in shell?
This should give you the desired output:
# awk 'BEGIN{FS=" "}{arr[$1]+=$10}END{for(i in arr) print i,arr[i]}' file
88.222.10.7 136098
2001:778:0:1::21 68093
121.141.172.40 235

Rack requests files non stop

I create a simple rack application to serve static files on heroku. But I have a problem that when trying to run the application locally with rackup. It seems to requests files continuously, this results hanging tab in Chrome.
What is the problem I have and how to fix it ?
config.ru
use Rack::Static,
:urls => ["/images", "/scripts", "/styles", "/components"],
:root => "public"
run lambda { |env|
[
200,
{
'Content-Type' => 'text/html',
'Cache-Control' => 'public, max-age=86400'
},
File.open('public/index.html', File::RDONLY)
]
}
And the error :
...
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /scripts/scripts.js?_=1362996702033 HTTP/1.1" 200 8543 0.0007
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/jquery/jquery.min.js?_=1362996702034 HTTP/1.1" 200 92629 0.0021
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/jquery-ui/ui/minified/jquery-ui.custom.min.js?_=1362996702035 HTTP/1.1" 200 228147 0.0041
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/angular-strap/dist/angular-strap.min.js?_=1362996702040 HTTP/1.1" 200 11632 0.0010
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/bootstrap/docs/assets/js/bootstrap.min.js?_=1362996702041 HTTP/1.1" 200 28545 0.0010
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/angular-ui/build/angular-ui.min.js?_=1362996702042 HTTP/1.1" 200 17515 0.0009
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/angular-ui/build/angular-ui-ieshiv.min.js?_=1362996702043 HTTP/1.1" 200 1018 0.0006
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /scripts/scripts.js?_=1362996702044 HTTP/1.1" 200 8543 0.0007
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/jquery/jquery.min.js?_=1362996702045 HTTP/1.1" 200 92629 0.0021
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/jquery-ui/ui/minified/jquery-ui.custom.min.js?_=1362996702046 HTTP/1.1" 200 228147 0.0041
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/angular-strap/dist/angular-strap.min.js?_=1362996702051 HTTP/1.1" 200 11632 0.0007
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/bootstrap/docs/assets/js/bootstrap.min.js?_=1362996702052 HTTP/1.1" 200 28545 0.0060
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/angular-ui/build/angular-ui.min.js?_=1362996702053 HTTP/1.1" 200 17515 0.0008
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /components/angular-ui/build/angular-ui-ieshiv.min.js?_=1362996702054 HTTP/1.1" 200 1018 0.0006
127.0.0.1 - - [11/Mar/2013 11:11:43] "GET /scripts/scripts.js?_=1362996702055 HTTP/1.1" 200 8543 0.0007
...

Resources