I was looking my apache logs and I find out this.
Log Line: 192.168.1.2 - - [30/Nov/2016:15:46:52 +0100] "GET http://www.Mywebsite.... HTTP/1.1" 200 5539 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
This happen a few times with that IP, but i cannot understand how is bingbot executing from that IP.
Thanks in advance
Related
I have some strange lags issue with a sinatra thin app when the trafic get a little high (30-40 users).
It's a small game using long-polling so http IOs can be high compared to the number of users.
CPU load stay low and there's a lot of free memory.
Here are some typical logs lines when lags happen :
1 - [17/Jul/2015:16:50:17 -0400] "POST /play?next=word HTTP/1.1" 200 1 0.0018
2 - [17/Jul/2015:16:50:17 -0400] "GET /update?_=1437166100579 HTTP/1.1" 200 304 15.0046
3 - [17/Jul/2015:16:50:17 -0400] "GET /update?_=1437166102348 HTTP/1.1" 200 286 15.0045
4 - [17/Jul/2015:16:50:17 -0400] "POST /accept_replay? HTTP/1.1" 200 - 0.0021
5 - [17/Jul/2015:16:50:18 -0400] "GET /core HTTP/1.1" 200 3719 0.0015
6 - [17/Jul/2015:16:50:18 -0400] "GET /join HTTP/1.1" 302 - 0.0640
7 - [17/Jul/2015:16:50:18 -0400] "GET /core HTTP/1.1" 200 3719 0.0024
8 - [17/Jul/2015:16:50:19 -0400] "POST /play?next=word HTTP/1.1" 200 1 0.0034
9 - [17/Jul/2015:16:50:19 -0400] "GET /update?_=1437166215907 HTTP/1.1" 200 248 10.0018
10- [17/Jul/2015:16:50:19 -0400] "GET /update?_=1437166222579 HTTP/1.1" 200 252 11.0029
11- [17/Jul/2015:16:50:31 -0400] "GET /core HTTP/1.1" 200 3719 0.0034
12- [17/Jul/2015:16:50:31 -0400] "POST /sentiment/bad? HTTP/1.1" 200 - 0.0024
13- [17/Jul/2015:16:50:31 -0400] "GET / HTTP/1.1" 200 4449 0.0086
14- [17/Jul/2015:16:50:31 -0400] "POST /decline_replay HTTP/1.1" 302 - 0.0020
And 30 more exactly at [17/Jul/2015:16:50:31 -0400]
( get /update are longpolling requests so it can take up to 40seconds)
Everythings stops for 12seconds between 10 and 11. And all the requests received during this time seems to be processed simultaneously.
I start the app that way
thin start -p 80
Can it be a thin issue ?
Do i need a custom thin config file ?
Do i need nginx?
Any indication is welcome...
edit :
Errors I find in ObectSpace [SystemStackError, 1][NoMemoryError, 1][IOError, 1]
The kind of behavior smells a lot like request queuing, which means there aren't enough web processes free to handle incoming requests. So the requests sit waiting, and then then the backlog is cleared they suddenly all get processed super fast and all at once.
This guy wrote up a good post on how to use Thin, EventMachine, and Async Sinatra to handle long-polling requests.
I have an interesting issue occurring frequently in my Spring web app. Basically, a user logs into the site successfully, navigates around to protected pages for a while and then for some reason, the server starts to return 403 responses. The user calls into tech support and tech support asks them to refresh their browser and try again. Magically, everything works fine the second time around after the refresh. No one on my team is able to reproduce the issue in either our QA environment nor on Production. Does anyone have any ideas what may cause this?
UPDATE - these are logs from the apache access log file
These calls are what prompt the call to tech support (notice the 403s)
"GET /server-webapp/api/getCartContents? HTTP/1.1" 200 1273
"GET /js/config.js HTTP/1.1" 200 809
"GET /server-webapp/api/getCartContents? HTTP/1.1" 200 1273
"GET /server-webapp/api/getUserInfo? HTTP/1.1" 200 201
"GET /server-webapp/api/getPendingSalesOrder? HTTP/1.1" 200 183
"POST /server-webapp/api/getShoppingCartErrors HTTP/1.1" 200 40
"GET /server-webapp/generated/CountriesAndStates.json? HTTP/1.1" 200 3319
"GET /server-webapp/api/getAddresses? HTTP/1.1" 403 390
"POST /server-webapp/api/createPendingSalesOrder HTTP/1.1" 403 390
"GET /server-webapp/api/getAddresses?" 403 390
"POST /server-webapp/api/createPendingSalesOrder HTTP/1.1" 403 390
"GET /server-webapp/api/getAddresses? HTTP/1.1" 403 390
"POST /server-webapp/api/createPendingSalesOrder HTTP/1.1" 403 390
The user is asked to refresh their browser and those same calls are now returning 200s...
"GET /server-webapp/api/getCartContents? HTTP/1.1" 200 1273
"GET /server-webapp/api/getCartContents? HTTP/1.1" 200 1273
"GET /server-webapp/api/getUserInfo? HTTP/1.1" 200 261
"POST /server-webapp/api/getShoppingCartErrors HTTP/1.1" 200 40
"GET /server-webapp/api/getPendingSalesOrder? HTTP/1.1" 200 183
"GET /server-webapp/generated/CountriesAndStates.json? HTTP/1.1" 200 3319
"GET /server-webapp/api/getAddresses? HTTP/1.1" 200 50
"POST /server-webapp/api/createPendingSalesOrder HTTP/1.1" 200 184
"POST /server-webapp/api/updatePendingSalesOrderLines HTTP/1.1" 200 42
"GET /server-webapp/api/getPendingSalesOrder? HTTP/1.1" 200 206
The issue ended up not being with Tomcat or Apache and was, in fact, a nasty little bug in the authentication logic. In short, the user was being logged in but with no permissions but only if they took a very specific route to becoming logged in. Thanks to those that have taken a look and if there is way for me to delete the ticket then let me know since it turned out to be coding logic and not anything to do with Tomcat or Apache.
My VPS shutdown because the HDD is filling up and I realized that the microcache.log file is becoming 12GB after I delete it. The content of microcache.log file is:
23.88.110.68 - - [12/Jun/2014:16:09:45 -0400] "GET http://ib.adnxs.com/ttj?id=2168123&position=below HTTP/1.0" 502 166 "battercar.com/?p=436" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10_4_11; en) AppleWebKit/528.4+ (KHTML, like Gecko) Version/4.0dp1 Safari/526.11.2" nocache:
173.208.213.94 - - [12/Jun/2014:16:09:45 -0400] "GET ib.adnxs.com/tt?id=2962937 HTTP/1.0" 502 568 "http://www.existeducation.com/tag/tap/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/532.0 (KHTML, like Gecko) Chrome/4.0.202.0 Safari/532.0" nocache:
(continues thousans of lines...)
How can I repair my VPS? I got tired to delete this file everyday.
VPS : Centos 6 with nginx
Large LOG files should be deleted http://wikitechsolutions.com/2761/microcache-log-file-exceeds-my-vps-hdd
I am getting relentless requests on one of my domains which I believe is coming from the Pushdo virus (or similar) see log snippet below. Apparently it picks random domains to send traffic to in order to mask the requests to it's command node. I have tried Fail2Ban but the IPs keep changing constantly and was banning 50K+ and the banning was using more resources than the requests. I was hoping to take care of the HTTP requests (there are SMTP one too but that is another question!) by blocking the user agent.
I have tried using
iptables -A INPUT -p tcp --dport 80 -m string --algo bm --string "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)" -j DROP
But this does not work! What am I doing wrong? Also, any other suggestions for dealing with this - it has been going on for over a month now and I am pulling my hair out!
OS: CentOS 6.4
Log Snippet:
121.54.54.47 - - [20/Oct/2013:03:32:37 +0100] "POST / HTTP/1.1" 200 14772 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
125.60.156.224 - - [20/Oct/2013:03:32:37 +0100] "POST / HTTP/1.1" 200 14772 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
84.108.50.80 - - [20/Oct/2013:03:32:37 +0100] "POST / HTTP/1.1" 200 14772 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
110.143.55.42 - - [20/Oct/2013:03:32:37 +0100] "POST / HTTP/1.1" 200 14772 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
122.208.75.75 - - [20/Oct/2013:03:32:37 +0100] "POST / HTTP/1.1" 200 14772 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
1.2.248.56 - - [20/Oct/2013:03:32:38 +0100] "POST / HTTP/1.1" 200 14772 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
180.194.171.167 - - [20/Oct/2013:03:32:38 +0100] "POST / HTTP/1.1" 200 14772 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
190.200.59.125 - - [20/Oct/2013:03:32:39 +0100] "POST / HTTP/1.1" 200 14772 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
223.197.238.249 - - [20/Oct/2013:03:32:40 +0100] "POST / HTTP/1.1" 200 14772 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
200.121.4.163 - - [20/Oct/2013:03:32:39 +0100] "POST / HTTP/1.1" 200 14772 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"
You're using -A which appends the rule to your existing set, so it's probably not doing anything. Using -I would probably work, but you likely want to script this and get it put in the right order.
See:
Here are images are downloaded good:
https://polishwords.com.pl/dev/testAbort2.php
And here:
https://polishwords.com.pl/dev/testAbort.php
I get them in Firefox with HTTPS and randomly one of them is Aborted and does not display correctly.
In logs on server it looks like this:
[22/Mar/2013:23:29:11 +0100] "GET /images/mukonczeniestudiow.jpg HTTP/1.1" 200 6705 "-" "Mozilla/5.0 (Windows NT 6.1; rv:19.0) Gecko/20100101 Firefox/19.0"
And when the file is loaded ok:
[22/Mar/2013:23:30:41 +0100] "GET /images/mukonczeniestudiow.jpg HTTP/1.1" 200 6907 "https://polishwords.com.pl/dev/testAbort.php" "Mozilla/5.0 (Windows NT 6.1; rv:19.0) Gecko/20100101 Firefox/19.0"
What can be the cause of this problem?
In Chrome and in Opera it seems to work fine. I have latest Firefox.
It was something on server blocking several queries