I'm getting an error of "gateway timeout. The gateway did not receive a timely response from the upstream server or application." What I'm doing is I'm getting data from sticky CRM by API and storing it into my database and it takes much time to execute. What I noticed is that I'm getting this error right after one minute. So I want to increase that time. I have install web server under ec2 instance and database under rds. I want to increase my gateway time from one minute to infinite or 15-20 minutes.
Help will be appreciated.
Thanks.
Related
I have AWS Elastic load balancer which has two healthy instances. If I make a POST request, it gets accepted. But consequent requests throw 504 gateway timeout error. After 5-10 minutes, it accepts 2-4 requests, and then start throwing 504 error. I try to reach is Spring Boot Application hosted on these two instances. There are no application level timeouts. Further time duration between failed and accepted requests vary, so I believe no fixed timeout configuration setting is causing an issue. How can I resolve this?
I have a REST service written in Spring Boot which have only GET and POST methods. The GET/POST method retrieve/post data from/to cassandra cluster.
I tried testing the REST service with 100 users running concurrently.
My jmeter configurations are:
When I run this test, for the first few request it gives errors and half request is successful.
Here is the result table
Then I checked the reason of error in the Result tree and found this response data:
{"timestamp":1474278650822,"status":500,"error":"Internal Server Error","exception":"com.datastax.driver.core.exceptions.NoHostAvailableException","message":"org.springframework.web.util.NestedServletException: Request processing failed; nested exception is com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (no host was tried)","path":"/post"}
I couldn't figure out why it failed to connect to cassandra cluster for the first few requests.
I am new to cassandra, REST service and jmeter. Can anyone point out what mistake I am making.
NOTE: I have tested my REST Service using Postman, and it functions well, both GET and POST request.
With this configuration you're hitting the service at once with 100 requests.
Each request will get a request thread, therefore I assume your service has a thread-pool with at least 100 connection threads.
Each request thread tries to obtain a connection to the DB from the DB Connection pool. If 50% of your requests fail, I'd guess, your DB Connection Pool size set to 50. In case your DB connection pool can handle 100 connections, check the connection settings of your Cassandra DB, maybe that one is limited to 50.
There is few more to add with the answer of #Gerald. The number of threadpool limitation in the DB might not be the only culprit, the other possibilities are the max thread connections set in the server level as well. If you have the access to the server log of the api your trying to test, you can get exactly at what point it breaks and fix it with help of developers
I have what appears to be a race condition related to losing responses coming from my heroku web service.
The heroku router delivers the request to the web service, the web service processes the request and returns a response, but in the interim the heroku router fails the request, either due to client (interrupt) or backend timeout.
The problem is that the web service request processing changed state on the backend and expected to send the state change to the client in the body of the response. The response never gets to the client, therefore the state change is lost forever.
The state change in my case happens to be the delivery and removal of a message from a RabbitMQ message queue. The web service request handler pops the request from the RabbitMQ queue, but it fails to reach the client and is never heard of again.
I could implement my own client-based message ACK system to mitigate this. However, I suspect that some of you might have a better solution regarding how to deal with ensuring that the responses get to the client. Is there any callback that I can use on my web service to determine if the response was lost? FWIW my web service is a JAX-RS service running embedded Jetty.
Thanks!
We're on four Amazon EC2 instances (one load balancer, one db, and two app) and are constantly getting random timeouts. We get at least one a day, sometimes more. Here are some examples:
Errno::ETIMEDOUT: Connection timed out - connect(2)
/usr/local/rvm/rubies/ruby-1.9.2-p180/lib/ruby/1.9.1/net/smtp.rb:546:in `initialize'
and
Timeout::Error: execution expired
[GEM_ROOT]/gems/activemodel-3.0.9/lib/active_model/attribute_methods.rb:354:in `match'
I'm not sure how to debug these as they are not related to application code or server load. CPU usage usually hovers below 10% with the biggest spike going up to 60%. The spikes are most likely due to running backups and do not correspond with the times of the timeout errors.
How can these types of errors be tracked down?
The first timeout looks like a legit connection timeout sending mail via SMTP. Are you hosting your own SMTP server or using a service?
Looks like sendgrid has been experiencing delays/timeouts the last couple of days:
We're currently seeing lots of volume in our queues and emails may be delayed for a short period. Stay tuned for updates. #status
Fix for SMTP Service Timeout/Fails
Setup a local mail relay that will hold mail and re-send if there are failures like this. We use a local Postfix relay in production for just this problem (so ActiveMailer uses sendmail to Postfix, which queues up mail and delivers via SMTP relay to Sendgrid).
I have inherited an application (internal to my company) that uses javascript running in Internet Explorer which makes Ajax calls to a Struts-based application running in WebLogic Server v10.
Certain server-side operations in the system are taking longer than 3 minutes. Users consistently noticed that the Ajax call returns 503 error at the 3 minute mark. My users can wait longer than 3 minutes, but 503 errors interrupt their work.
This application needs to be performance tuned, but we badly need a temporary workaround to extend how much time can occur before a 503 error is returned.
The current theory is that the 503 error is being raised by the IE XMLHttpRequest object. A team of supposed WebLogic experts poured over our code and WebLogic logs, and declared that there's no timeout occurring on the server side. But I have my doubts.
My question is, which piece of software is responsible for raising 503 error: the browser, the Ajax javascript, or the server? And can this timeout period be changed?
A 503 error is kind of a catch-all for a lot of different types of errors, usually on the server side. In your case it could be that the server is just rejecting the connection after a certain timeout, and responding back with a 503 to indicate that the server is overloaded or cannot process your request.
A lot of times with web services, a 503 will be returned when the server code throws an exception or error. If the server code doesn't properly handle the error, it will bubble up to the server, which will just respond back with a generic 503.
http://www.checkupdown.com/status/E503.html
Error code 5xx (alternate definition)
RFC 2616
503 is a server error. XMLHttpRequest will happily wait longer than 3 minutes. The first thing you should do is satisfy yourself of that by visiting the problem URL in telnet or netcat or similar and seeing the 503 with javascript out of the picture.
Then you can proceed to find the timeout on the server side.
Your web server has a request reply timeout which is being tripped by long-running service requests. It could be the WebLogic server or a proxy. It is certainly not the client.
Have you considered submitting an asynchronous HTTP request that will be responded to immediately, and then polling another location for the eventual results? Three minutes is about 170 seconds too long.
503 is most likely due to a timeout on the server. If you can tune your Apache server, read about the Timeout attribute that you can set in httpd.conf.
Look in the httpd/logs/error_log to see if timeouts are occurring.
Refer also to this answer: Mod cluster proxy timeout in apache error logs .