I am running a Meteor app on Heroku and the Heroku response time chart is constantly filled with high response time queries hitting 30 seconds limit
The router logs shows that all the 30 seconds queries are some kind of sockJS xhr call
14 Aug 2021 11:37:33.249334 <158>1 2021-08-14T03:37:32.837295+00:00 heroku router - - at=info method=POST path="/sockjs/950/pwiv10fk/xhr" host=www.abc.com request_id=edbe326a-cddd-4e5e-9347-0f5c1c49651b fwd="121.6.80.103,162.158.165.53" dyno=web.1 connect=0ms service=29493ms status=200 bytes=384 protocol=httpHigh Response Time
14 Aug 2021 11:37:33.358335 <158>1 2021-08-14T03:37:32.837093+00:00 heroku router - - at=info method=POST path="/sockjs/707/7haw05h4/xhr" host=www.abc.com request_id=20ca52a2-5aa4-43fa-bb61-39462eb0ab86 fwd="121.6.80.103,162.158.167.169" dyno=web.1 connect=0ms service=29484ms status=200 bytes=384 protocol=httpHigh Response Time
14 Aug 2021 11:38:03.405335 <158>1 2021-08-14T03:38:02.836998+00:00 heroku router - - at=info method=POST path="/sockjs/950/pwiv10fk/xhr" host=www.abc.com request_id=f3a51bc3-9319-4b80-91b5-ed76bf179493 fwd="121.6.80.103,162.158.166.136" dyno=web.1 connect=0ms service=29480ms status=200 bytes=384 protocol=httpHigh Response Time
The reality is that there is only a few users on the app, and no unusual activity happening on server. Actually the users did not experience slow down as well, so i suspect, the socket connections mechanisms on Meteor app is polluting the heroku response time charts, I also have to turn off slow response time alerts because they are too many.
These non-stops 30 seconds response times calls creates a problem which I do not really know the true speed of the web app.
What is happening and how to avoid this?
It seems that the issue was introduced by Cloudflare. They seem to have been interrupting the websocket connections, causing Meteor to fall back to polling, which is not efficient.
Related
in my application (deployed in Heroku), there is a request (GET) that is blocked in the infrastructure layer, the request does not get to execute the code of my application. It returns an error status=400 and connect=0ms and does not carry any associated Heroku error code and description. The request never reaches the application.
It only happens with this GET request, when it comes from the production server. if I make the request from postman it is received correctly with status=200
The other requests have not problem and are executed correctly from the production server.
This is an example:
2021-08-20T10: 27: 02.217551 + 00: 00 heroku [router]: at=info method=GET path="/api/get" host=myapp.herokuapp.com request_id=2920634e-87f2-4b2c-be60-b38497c53e58 dyno=web.1 connect=0ms service=1ms status=400 bytes=47 protocol=https
The problem was identified and corrected.
The problem was that one of the headers of the GET request was being sent to null and the request was rejected as Bad Request Exception before entering the APP.
regards
So I have an app deployed in Heroku, a Go backend with Angular 8 frontend using Hobby dinos. But today I noticed that one of my endpoints would not work if the result contains more than 40 elements. I tested this locally and this doesnt happen locally, so it must be a problem when running in Heroku. Any idea about what is going on here? Heroku throws the following error message:
sock=backend at=error code=H18 desc="Server Request Interrupted" method=POST path="/invoices/g/range?from=2020-08-01T00:00:00.000+00:00&to=2020-08-28T00:00:00.000+00:00" host=xxx-prod.herokuapp.com request_id=d113ba1c-f51a-4f57-8f02-31195da1b5f8 fwd="xx.xxx.xxx.xxx" dyno=web.1 connect=1ms service=60ms status=503 bytes= protocol=https
So I finally figured out what was going on here. It has to do with an unsolved Node issue, see: https://github.com/nodejs/node/issues/12339
When using the POST handler I was receiving a large response that triggered that Issue. What I did is I rebuilt my logic in order to retrieve the response using a GET. Now it works as expected.
I can launch my application still, and load a page that doesn't utilize any data from the pg database, but as soon as I hit a page that would look up data I get a time out error
2019-10-04T05:53:38.387579+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/sessions/login" host=get-organized.herokuapp.com request_id=735daa05-c11b-4da6-a7c5-90f8ae46759a fwd="73.158.160.78" dyno=web.1 connect=1ms service=30000ms status=503 bytes=0 protocol=http
201
These apps used to work fine. I have newer apps that didn't have the maintenance done and they still work fine. I have no idea how to further troubleshoot this, any help would be greatly appreciated.
I am looking on my drain log and I see this
327 <158>1 2018-04-17T22:03:27.578702+00:00 heroku router - - at=info method=GET path="/{url}" host={my_host} request_id=11bb9b05-dea3-42c2-b57a-9be6fb9b93d2 fwd="80.6.26.72,141.101.107.25" dyno=web.1 connect=0ms service=1ms status=200 bytes=6265 protocol=http
I am certain that this request doesn't come from a legit user, how is it possible to dig in more and get the remote server IP? I used https://stackoverflow.com/a/6837689/2513428 inside my script to check the ip's but I assume it returned the proxy of herocu servers.
Heroku makes the IP making the request available in the fwd log field: https://devcenter.heroku.com/articles/http-routing#heroku-router-log-format
You can also read it within your code by looking at the X-Forwarded-For HTTP header.
So in your case, the IP of the client making this request was 80.6.26.72.
Developing a web app with javascript/express/node. On my local machine and Heroku, after running for a short period of time (~5 mins), I have to restart the server or it never responds to requests. I have no idea what could be causing this. Has anyone dealt with something like this before?
2014-02-10T01:25:29.935130+00:00 app[web.1]: GET / 304 281ms
2014-02-10T01:25:29.936342+00:00 heroku[router]: at=info method=GET path=/ host=wwwcom request_id=8d2cd623-c379-4e6a-8b30-1021dd2f5d18 fwd="" dyno=web.1 connect=2ms service=284ms status=304 bytes=0
2014-02-10T01:25:30.096768+00:00 heroku[router]: at=info method=GET path=/css/style.css host=wwwcom request_id=953f1c0c-65ce-4ac7-bfc5-04d5bec2791a fwd="" dyno=web.1 connect=2ms service=7ms status=304 bytes=0
2014-02-10T01:24:10.997307+00:00 heroku[router]: at=info method=GET path=/img/money.jpg host=wwwcom request_id=b1eb7f92-98ae-466d-8e03-a7539215936a fwd="" dyno=web.1 connect=2ms service=2ms status=304 bytes=0
2014-02-10T01:25:30.171105+00:00 heroku[router]: at=info method=GET path=/img/money.jpg host=wwwcom request_id=fd9244d0-22eb-4e28-ac81-b820bcab753b fwd="" dyno=web.1 connect=16ms service=4ms status=304 bytes=0
2014-02-10T01:27:58.989371+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path=/ host=wwwcom request_id=ddcb47a6-b250-49ea-8ba8-9c9da4c41c91 fwd="" dyno=web.1 connect=1ms service=30001ms status=503 bytes=0
2014-02-10T01:27:58.997153+00:00 app[web.1]: GET / 200 30003ms
I think it would be good to start here, especially the Debugging request timeouts section could be of help:
Debugging request timeouts
One cause of request timeouts is an infinite loop in the code. Test locally (perhaps with a copy of the production database pulled down with pgbackups) and see if you can replicate the problem and fix the bug. Another possibility is that you are trying to do some sort of long-running task inside your web process, such as:
Sending an email
Accessing a remote API (posting to Twitter, querying Flickr, etc.)
Web scraping / crawling
Rendering an image or PDF
Heavy computation (computing a fibonacci sequence, etc.)
Heavy database usage (slow or numerous queries, N+1 queries)
If so, you should move this heavy lifting into a background job which can run asynchronously from your web request. See background workers for details.