Small question regarding how to interpret the IP of an access log of a Spring WebFlux application please.
I have a very simple Spring WebFlux Application, with access log enabled, I can see the log, happy.
This application was built for only one client. Meaning, I am sure, at any time, there is one and only one client application calling in for sure.
Moreover, I know as a fact the client application is also unique. It is one client application that rarely restarts, there is only one instance of it, deployed in a fix physical datacenter, inside one physical hardware server. TLDR, there is only one client.
Yet, in my access log, each time this unique client calls in, I see something like:
INFO [myservice,,] 10 --- [or-http-epoll-4] reactor.netty.http.server.AccessLog : aa.1aa.1aa.aa - - "POST /myapi HTTP/1.1" 200 563 382
INFO [myservice,,] 10 --- [or-http-epoll-2] reactor.netty.http.server.AccessLog : bb.2bb.2bb.bb - - "POST /myapi HTTP/1.1" 200 563 372
And this is confusing me, as in why would I see multiple IP in this log?
Thank you
Related
I dockerized my Application. If i run it with docker run, evertything works fine.
I tried to run it with ecs fargate and put an ALB infront of it.
If i try to access my Application via the ALB dns, i get an 504 Gateway Teme-out back.
While searching a solution, i found an post, which told me to set the Tomcat timeout higher than the ELB timeout, but it doesn't helped.
Dockerfile
FROM tomcat:8.0.20-jre8
RUN sed -i 's/connectionTimeout="20000"/connectionTimeout="70000"/' /usr/local/tomcat/conf/server.xml
CMD ["catalina.sh","run"]
COPY /target/Webshop.war /usr/local/tomcat/webapps/
ELB Log
http 2019-09-11T11:20:50.585293Z app/Doces-Backe-19RQJLVNHYG2P/8fb4f4079bb6ff9f 66.85.6.136:47767 - -1 -1 -1 503 - 18 348 "GET http://:8080/ HTTP/1.0" "-" - - arn:aws:elasticloadbalancing:eu-central-1:573575081005:targetgroup/ecs-Docest-de-webshop/8df4f0978484f8bd "Root=1-5d78d892-58886d3490906f0fa3914563" "-" "-" 0 2019-09-11T11:20:50.462000Z "forward" "-" "-"
http 2019-09-11T11:23:23.535869Z app/Doces-Backe-19RQJLVNHYG2P/8fb4f4079bb6ff9f 66.85.6.136:50950 10.10.11.140:8080 -1 -1 -1 504 - 18 303 "GET http://:8080/ HTTP/1.0" "-" - - arn:aws:elasticloadbalancing:eu-central-1:573575081005:targetgroup/ecs-Docest-de-webshop/8df4f0978484f8bd "Root=1-5d78d921-a236121716bd1bd209625fd8" "-" "-" 0 2019-09-11T11:23:13.415000Z "forward" "-" "-"
http 2019-09-11T11:23:56.286426Z app/Doces-Backe-19RQJLVNHYG2P/8fb4f4079bb6ff9f 66.85.6.136:51658 10.10.11.140:8080 -1 -1 -1 504 - 18 303 "GET http://:8080/ HTTP/1.0" "-" - - arn:aws:elasticloadbalancing:eu-central-1:573575081005:targetgroup/ecs-Docest-de-webshop/8df4f0978484f8bd "Root=1-5d78d942-22a1680464884762e02ec940" "-" "-" 0 2019-09-11T11:23:46.156000Z "forward" "-" "-"
http 2019-09-11T11:23:27.513803Z app/Doces-Backe-19RQJLVNHYG2P/8fb4f4079bb6ff9f 66.85.6.136:51034 10.10.11.140:8080 -1 -1 -1 504 - 18 303 "GET http://:8080/ HTTP/1.0" "-" - - arn:aws:elasticloadbalancing:eu-central-1:573575081005:targetgroup/ecs-Docest-de-webshop/8df4f0978484f8bd "Root=1-5d78d925-b6b5daf0d0f733140aea0f84" "-" "-" 0 2019-09-11T11:23:17.393000Z "forward" "-" "-"
I expected to see my application running at the elb.
Thanks for your help!
Solution:
The problem was that I set the correct port in the security group of the load balancer, but not in that of the ECS service.
So I opened the required port there and now it works.
Procedure:
Go to your cluster
Go to the service with the problem
Click on the Security Group under the item Network Access and open the required port
Thanks!
There can be multiple reasons behind gateway timeout. The only thing that I do not like about fargate is debug-log. #AWS team should enable log configuration for fargate service by default as its hard to debug these issues without logs.
Better to configure log driver and push logs to cloud watch and see the actual issue also double check your desired port in task definition and mapped port in service.
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "awslogs-spring",
"awslogs-region": "us-west-2",
"awslogs-stream-prefix": "awslogs-example"
}
or from AWS console
You need to assign permission or role of cloud watch logs to task definition or service to push the logs to Cloud watch.
Once logs are configured then goto cloudwatch loggroup and search the log group so you will insight to your application.
But still, to troubleshoot the actual issue first, you have to understand the error code and possible reason of Gateway Timeout.
HTTP 504: Gateway Timeout
Description: Indicates that the load balancer closed a connection because a request did not complete within the idle timeout period.
Cause 1: The application takes longer to respond than the configured idle timeout.
Solution 1: Monitor the HTTPCode_ELB_5XX and Latency metrics. If there
is an increase in these metrics, it could be due to the application
not responding within the idle timeout period. For details about the
requests that are timing out, enable access logs on the load balancer
and review the 504 response codes in the logs that are generated by
Elastic Load Balancing. If necessary, you can increase your capacity
or increase the configured idle timeout so that lengthy operations
(such as uploading a large file) can complete. For more information,
see Configure the Idle Connection Timeout for Your Classic Load
Balancer and How do I troubleshoot Elastic Load Balancing high
latency.
Cause 2: Registered instances closing the connection to Elastic Load Balancing.
Solution 2: Enable keep-alive settings on your EC2 instances and make
sure that the keep-alive timeout is greater than the idle timeout
settings of your load balancer.
I have a very simple Gatling scenario which hits a single HTTP endpoint with concurrent users.
When I run this for 30 seconds with 10 requests per second, everything is fine.
When I run this for 30 seconds at 60 requests per second on Windows, I get very strange errors that look to me like the underlying network connections are getting corrupted or are being misused. Perhaps there is a race condition or concurrency bug somewhere in Gatling or somewhere else in my system.
I don't get the same problems on a linux machine.
The web server is nginx and PHP. I don't suspect that is the cause of the problem, but I might be wrong.
How can I track down and fix this bug?
The scenario code
val scn = scenario("my scenario - one endpoint only")
.exec(http("fetch")
.get("http://my.website/page"))
.inject(
constantUsersPerSec(requestsPerSecond)
.during(30.seconds)
.randomized)
.protocols(httpProtocol)
setUp(scn)
Symptoms
The scenario reports about 8% failure rate, with errors that look like the server is replying with malformed HTTP responses, returning HTML code where the HTTP status line should be. These vary in the details, but here is a representative example:
2017-02-20 17:30:59,875 DEBUG org.asynchttpclient.netty.request.NettyRequestSender - invalid version format: <META
java.lang.IllegalArgumentException: invalid version format: <META
at io.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:130)
at io.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:84)
at io.netty.handler.codec.http.HttpResponseDecoder.createMessage(HttpResponseDecoder.java:118)
at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:219)
at io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:152)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:250)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:346)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:367)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:353)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
at java.lang.Thread.run(Thread.java:745)
Similarly, the server logs include invalid requests where the client has sent HTML where the HTTP request line should be:
10.56.4.130 - - [20/Feb/2017:17:30:59 +0000] "span class=\x22id4-cta-size-small id5-cta id4-cta-color-blue id4-cta-small-blue\x22><a hr" 400 166 "-" "-" "-" 0.179 -
10.56.4.130 - - [20/Feb/2017:17:31:00 +0000] "<!doctype html>" 400 166 "-" "-" "-" 0.070 -
Version info
I am using:
O/S: Windows 8.1 64 bit
Virus Scanner: I have Kapersky which intercepts network traffic. I tried turning it off, which made no difference. I don't know if it was "really" off.
VPN: My machine has a Windows Direct Connect VPN. The target site does not fall within that VPN.
Java: "1.8.0_121", Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
Scala: 2.11.8
Gatling: 2.2.3
Akka: 2.4.12
io.netty.netty-handler: 4.0.42.Final
(4.0.41.Final was requested by netty-reactive-streams v 1.0.8, I wonder if that's significant)
I am relatively new to the world of java. I am building a web application where I use Spring. So here's the case:
We have deployed the web app to our development server , and have it out in the open. This is how our network administrator explained it (don't really understand it but might be important for you):
55.555.55.5 is a NAT port to 77.77.77.77 ; 77.77.77.77 is an apache proxy server that makes a connection to 99.9.99.999.
We tell this server - 55.555.55.5 on http port to connect to 99.9.99.999 on port 8080 and give the client a connection on port 80 of 55.555.55.5; At the moment it is 1:1 connection...
And here is the problem:
On http://localhost:8080/myapp/ and on http://99.9.99.999:8080/myapp/ (our development server) everything works fine.
But on http://55.555.55.5:8080/myapp/ which is visible to the clients if I load the login page and right away login its ok, but if I wait like a couple of minutes on the page I get this:
HTTP Status 501 - Method email=something%40gmail.com&password=somethingGET is not implemented by this servlet for this URI
type Status report
description The server does not support the functionality needed to fulfill this request
Apache Tomcat/8.0.32
This is really strange and I don't know why it's happening?
I have noticed in my production Rails log that exactly every 5 minutes, I have a GET request to my root url from 127.0.0.1 which apparently is my localhost.
Started GET "/" for 127.0.0.1 at 2012-07-01 14:05:03 -0500
Processing by ApplicationController#landing as */*
Rendered shared/_header.html.erb (0.9ms)
Rendered shared/_footer.html.erb (0.5ms)
Rendered application/landing.html.erb (5.7ms)
Completed 200 OK in 8ms (Views: 7.9ms)
I have never seen this in any other Rails apps. I am using New Relic, MongoDB, Nginx, and Unicorn. Can anyone tell my why this is happening or what it means?
This is most likely a monitoring application, especially since it's only checking the root path for a successful connection (i.e. HTTP 200). Have you installed any tools such as monit? What hosting provider are you using? They may monitor without you knowing.
I'm running single instance CloudFoundry configuration with one web application. I turned on Hyperic monitoring with notification for case of web app unavailability.
Now I randomly receive alert emails (Subject "An alert has been triggered - Deployment myapp - context unavailable") that the application is not running, but it obviously is running fine.
In access log of Apache I see two requests every 15 seconds:
127.0.0.1 - - [17/Mar/2010:15:37:33 +0100] "GET /server-status?auto HTTP/1.1" 200 438 "-" "Jakarta Commons-HttpClient/3.1"
127.0.0.1 - - [17/Mar/2010:15:37:33 +0100] "GET /myapp HTTP/1.1" 200 - "-" "Jakarta Commons-HttpClient/3.1"
At the time when I get the alert emails, everything in log still seems to be fine - two requests.
Do you have idea what could be wrong? Did anybody have this kind of problem and solve it?
Thanks,
P
Ok, got info from CloudFoundry guys. The alerts are sent if either Apache or internal Tomcat request goes wrong or is timeouted. My problem apparently came from internal Tomcat requests that are not logged in access log.
They now simply changed the algorithm, so the alert is trigerred when unavailability (Apache / Tomcat) is reported at least two times in a row. Frequent alert emails problem is gone.