I had a working deployment of Hasura on Heroku. It did run for about 2 years. However, recently, when I startup the application, there's a timeout error, resulting in the app just not starting anymore (see logs below). The credentials, urls etc are correct, since the application has been working before. There were no changes in settings.
I already had Heroku increase the timeout from 60 seconds to 180 seconds, but that doesn't help, but it gives me the reason for the failure: the connection to the postgres db fails.
However, the postgres database is running fine: I can connect to it and read the schemas and tables using psql (using the DATABASE_URL env variable from Heroku as URL). When I setup Hasura Cloud and connect to the postgres db on Heroku, it works fine (with the URL provided in the DATABASE_URL env variable in Heroku). So, the db runs, and the connection string is the right one.
This is my Dockerfile (updated Hasura to version 2.9.0):
$ cat Dockerfile
FROM hasura/graphql-engine:v2.9.0
# Enable the console
ENV HASURA_GRAPHQL_ENABLE_CONSOLE=true
# Enable debugging mode. It should be disabled in production.
ENV HASURA_GRAPHQL_DEV_MODE=true
# Heroku hobby tier PG has few limitations including 20 max connections
# https://devcenter.heroku.com/articles/heroku-postgres-plans#hobby-tier
ENV HASURA_GRAPHQL_PG_CONNECTIONS=15
# Change $DATABASE_URL to your heroku postgres URL if you're not using
# the primary postgres instance in your app
CMD graphql-engine \
--database-url $DATABASE_URL \
serve \
--server-port $PORT
## Comment the command above and use the command below to
## enable an access-key and an auth-hook
## Recommended that you set the access-key as a environment variable in heroku
#CMD graphql-engine \
# --database-url $DATABASE_URL \
# serve \
# --server-port $PORT \
# --access-key XXXXX \
# --auth-hook https://myapp.com/hasura-webhook
#
# Console can be enable/disabled by the env var HASURA_GRAPHQL_ENABLE_CONSOLE
Any ideas how I can find out the exact issue/how to debug it/how to solve it?
Thanks.
Output logs:
2022-08-08T07:26:47.538883+00:00 heroku[web.1]: Starting process with command `/bin/sh -c graphql-engine\ \ \ \ \ --database-url\ \postgres://edxvzxubxqcanh:68eaaa545e7742497f656a43c3d2946f039fc90806a07d691e02202a2541263c#ec2-108-128-201-166.eu-west-1.compute.amazonaws.com:5432/d25u4bqll5rg1a\ \ \ \ \ serve\ \ \ \ \ --server-port\ \52293`
2022-08-08T07:26:49.672834+00:00 app[web.1]: {"detail":{"info":{"admin_secret_set":true,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":true,"enabled_apis":["config","pgdump","graphql","metadata"],"enabled_log_types":["startup","webhook-log","websocket-log","http-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[{"audience":null,"claims_format":"json","claims_namespace":"https://hasura.io/jwt/claims","header":null,"issuer":null,"key":"<JWK REDACTED>","type":"<TYPE REDACTED>"}],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"info","port":52293,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":false,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":"public","use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"WSConnectionInitTimeout {unWSConnectionInitTimeout = Seconds {seconds = 3s}}","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Seconds {seconds = 5s}}"},"kind":"server_configuration"},"level":"info","timestamp":"2022-08-08T07:26:49.667+0000","type":"startup"}
2022-08-08T07:26:50.680661+00:00 app[web.1]: {"detail":{"info":{"database_url":"postgres://edxvzxubxqcanh:...#ec2-108-128-201-166.eu-west-1.compute.amazonaws.com:5432/d25u4bqll5rg1a","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2022-08-08T07:26:49.667+0000","type":"startup"}
2022-08-08T07:28:56.917452+00:00 app[web.1]: {"detail":{"message":"postgres connection failed, retrying(0)."},"level":"warn","timestamp":"2022-08-08T07:28:56.917+0000","type":"pg-client"}
2022-08-08T07:29:48.153244+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 180 seconds of launch
2022-08-08T07:29:48.246481+00:00 heroku[web.1]: Stopping process with SIGKILL
2022-08-08T07:29:48.392887+00:00 heroku[web.1]: Process exited with status 137
2022-08-08T07:29:48.786085+00:00 heroku[web.1]: State changed from starting to crashed
== EDIT 15/08/2022
After updating to hasura to version:
FROM hasura/graphql-engine:v2.11.0-beta.1
# Heroku only allows to install extensions in the heroku_ext schema
ENV HASURA_GRAPHQL_DEFAULT_EXTENSIONS_SCHEMA=heroku_ext
# https://github.com/hasura/graphql-engine/issues/8734
ENV HASURA_GRAPHQL_METADATA_DATABASE_EXTENSIONS_SCHEMA=heroku_ext
ENV HASURA_GRAPHQL_ENABLE_CONSOLE=true
ENV HASURA_GRAPHQL_DEV_MODE=true
ENV HASURA_GRAPHQL_PG_CONNECTIONS=15
CMD graphql-engine \
--database-url $DATABASE_URL \
serve \
--server-port $PORT
I still have the same issue, the connection to postgres doesn't work:
Starting process with command `/bin/sh -c graphql-engine\ \ \ \ \ --database-url\ \postgres://edxvzxubxqcanh:...#ec2-108-128-201-166.eu-west-1.compute.amazonaws.com:5432/d25u4bqll5rg1a\ \ \ \ \ serve\ \ \ \ \ --server-port\ \7315`
2022-08-15T14:09:24.454749+00:00 app[web.1]: {"detail":{"info":{"admin_secret_set":true,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":true,"enabled_apis":["config","pgdump","metadata","graphql"],"enabled_log_types":["startup","webhook-log","http-log","websocket-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[{"audience":null,"claims_format":"json","claims_namespace":"https://hasura.io/jwt/claims","header":null,"issuer":null,"key":"<JWK REDACTED>","type":"<TYPE REDACTED>"}],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"info","port":7315,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":false,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":"public","use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"WSConnectionInitTimeout {unWSConnectionInitTimeout = Seconds {seconds = 3s}}","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Seconds {seconds = 5s}}"},"kind":"server_configuration"},"level":"info","timestamp":"2022-08-15T14:09:24.404+0000","type":"startup"}
2022-08-15T14:09:25.468623+00:00 app[web.1]: {"detail":{"info":{"database_url":"postgres://edxvzxubxqcanh:...#ec2-108-128-201-166.eu-west-1.compute.amazonaws.com:5432/d25u4bqll5rg1a","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2022-08-15T14:09:24.404+0000","type":"startup"}
2022-08-15T14:11:29.990364+00:00 heroku[router]: at=error code=H20 desc="App boot timeout" method=OPTIONS path="/v1/graphql" host=systemic-nrr-store.herokuapp.com request_id=d4620807-2a79-468c-b658-be14fd31ec55 fwd="178.51.9.123" dyno= connect= service= status=503 bytes= protocol=https
2022-08-15T14:11:30.005688+00:00 heroku[router]: at=error code=H20 desc="App boot timeout" method=OPTIONS path="/v1/graphql" host=systemic-nrr-store.herokuapp.com request_id=2370dc41-749e-44f8-81c4-73052eec90fe fwd="178.51.9.123" dyno= connect= service= status=503 bytes= protocol=https
2022-08-15T14:11:30.109890+00:00 heroku[router]: at=error code=H20 desc="App boot timeout" method=OPTIONS path="/v1/graphql" host=systemic-nrr-store.herokuapp.com request_id=796a2090-79f4-4a3b-89a7-c00b55870af9 fwd="178.51.9.123" dyno= connect= service= status=503 bytes= protocol=https
2022-08-15T14:11:31.738877+00:00 app[web.1]: {"detail":{"message":"postgres connection failed, retrying(0)."},"level":"warn","timestamp":"2022-08-15T14:11:31.738+0000","type":"pg-client"}
My first time deploying streamlit on Heroku, and the scripts are on this GitHub. I'm actually deploying from local, so this is just to share my scripts to debug. The following is the error message.
2020-12-31T23:47:18.055589+00:00 heroku[router]:
at=error code=H20 desc="App boot timeout"
method=GET path="/" host=stormy-dawn-49490.herokuapp.com
request_id=b3df4074-d363-41e5-8ecd-d101070679f7
fwd="99.230.168.115"
dyno= connect= service= status=503 bytes= protocol=https
2020-12-31T23:48:01.327576+00:00 heroku[web.1]: Error R10 (Boot timeout) ->
Web process failed to bind to $PORT within 60 seconds of launch
I have four scripts, which is also linked to my GitHub repository the beginning of the question.
setup.sh
mkdir -p ~/.streamlit/
echo “\
[general]\n\
email = \”<dont#spam.us>\”\n\
“ > ~/.streamlit/credentials.toml
echo “\
[server]\n\
headless = true\n\
enableCORS=false\n\
port = $PORT\n\
“ > ~/.streamlit/config.toml
Procfile:
web: sh setup.sh && streamlit run app.py
requirements.txt:
fredapi==0.4.2
numpy==1.18.5
pandas==1.1.5
plotly==4.14.1
python-dateutil==2.8.1
python-slugify==4.0.1
Quandl==3.5.3
scikit-learn==0.23.2
streamlit==0.72.0
yfinance==0.1.55
gunicorn== 19.9.0
app.py: I'd suggest read here
I didn't put much in the app.py as I'm just trying how to deploy on Heroku
Thank you in advance!
Once I started using Channels with Phoenix, I started getting a lot of these errors on Heroku:
heroku[router]: at=error code=H15 desc="Idle connection" method=GET
path="/socket/websocket?guardian_token=eyJhbGciOiJIUz[...]&vsn=1.0.0" host=[...]
dyno=web.1 connect=0ms service=87803ms status=503 bytes=
A reliable way to reproduce this is on a laptop:
Open a Phoenix page that connects to WebSockets
Close the lid of your laptop
Wait a minute or so (WiFi to disconnect/OS goes to sleep)
The H15 error will occur.
I've set a 45s timeout on my UserChannel already; I'm running Elixir 1.4.2, Phoenix 1.2.1 and my phoenix.js was just upgraded to the one used in Phoenix 1.3.0-rc0 (link to rev)
I can see the error logged on the front-end, but after that it reconnects happily to the backend:
2017-03-04 23:28:55.351 receive: ok user:2 phx_reply (3)
2017-03-04 23:29:24.269 channel: error user:2
2017-03-04 23:29:28.713 push: user:2 phx_join (4)
2017-03-04 23:29:28.745 receive: ok user:2 phx_reply (4)
Any help or ideas how to solve this is much appreciated.
Edit: I'm familiar with Heroku's H15 error code definition so I'm looking for a fix or a workaround for getting rid of the errors (other than migrating away from Heroku, of course).
Phoenix docs recommend setting the timeout for the websocket transport to < 55 seconds:
defmodule HelloWeb.UserSocket do
use Phoenix.Socket
...
## Transports
transport :websocket, Phoenix.Transports.WebSocket,
timeout: 45_000
...
end
Their explanation is:
This ensures that any idle connections are closed by Phoenix before they reach Heroku’s 55-second timeout window.
I have big issues with deploying my application on heroku server. I've done it with help of tutorial from phoenix webpage. But it doesn't work. Below you can see the logs and configuration i used.
heroku logs
2016-02-07T04:28:55.623435+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web
process failed to bind to $PORT within 60 seconds of launch
2016-02-07T04:28:55.623435+00:00 heroku[web.1]: Stopping process with SIGKILL
2016-02-07T04:28:56.344649+00:00 heroku[web.1]: State changed from starting to c
rashed
2016-02-07T04:28:56.348324+00:00 heroku[web.1]: Process exited with status 137
2016-02-07T09:54:29.124036+00:00 heroku[web.1]: State changed from crashed to st
arting
2016-02-07T09:54:34.703208+00:00 heroku[web.1]: Starting process with command `m
ix run --no-halt`
2016-02-07T09:55:35.386922+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web
process failed to bind to $PORT within 60 seconds of launch
2016-02-07T09:55:35.386922+00:00 heroku[web.1]: Stopping process with SIGKILL
2016-02-07T09:55:36.330258+00:00 heroku[web.1]: State changed from starting to c
rashed
2016-02-07T09:55:36.323082+00:00 heroku[web.1]: Process exited with status 137
2016-02-07T12:14:35.182306+00:00 heroku[router]: at=error code=H10 desc="App cra
shed" method=GET path="/" host=kpsz.herokuapp.com request_id=7de0ee97-14cf-46f9-
a9a3-cbffa09ac379 fwd="83.7.11.212" dyno= connect= service= status=503 bytes=
2016-02-07T12:14:35.850018+00:00 heroku[router]: at=error code=H10 desc="App cra
shed" method=GET path="/favicon.ico" host=kpsz.herokuapp.com request_id=4e9b1979
-deb6-497b-80b2-655615f43d01 fwd="83.7.11.212" dyno= connect= service= status=50
3 bytes=
Configuration prod.exs
config :kpsz, Kpsz.Endpoint,
http: [port: System.get_env("PORT")],
url: [host: "example.com", port: 80],
cache_static_manifest: "priv/static/manifest.json",
secret_key_base: System.get_env("SECRET_KEY_BASE")
config :kpsz, Kpsz.Repo,
adapter: Ecto.Adapters.Postgres,
url: System.get_env("DATABASE_URL"),
pool_size: 20
I'm not sure if that's the problem but you could try to replace
http: [port: System.get_env("PORT")]
with
http: [port: {:system, "PORT"}]
At least that's what I'm using in my apps. You can also read the docs for more information http://www.phoenixframework.org/docs/heroku
I had a similar problem.
Adding a Procfile with a single line web: mix phoenix.server solved it.
heroku destroy project name
heroku create
and for some weird reason it works now...
Thanks very much for help.
I ran into this issue and, for me, the fix was as follow (copied from the answer on this question).
The fix is twofold:
to use the phoenix buildpack config and export the PORT as well, like so: config_vars_to_export=(DATABASE_URL PORT)
Avoid changing the MIX_ENV without recompiling the app, as that behavior is unsupported, as explained here.
As Recreation solves the problem, I suspect the problem is with loading environment variables.
Once I had a similar problem in my local. My observations
mix phoenix.server (you might see the message compiled so and so files)
Stopped the server
Export/Change the env variable in zshrc/bashrc file and source it
Start the server (you might not see that compilation message, as there are no changes in your codebase and so your new/edited env variables won't get reflected)
If you just add a space in that file will make it compile and take that env variable.
Reason for your problem might be: Setting env variables after the server is running/deployed, as env variables are converted to value during the compilation time itself.
I had a similar problem when deploying the phoenix umbrella project with docker.
I've taken these steps:
Heroku https://hexdocs.pm/phoenix/heroku.html#deploying-to-heroku-using-the-container-stack
Dockerfile https://hexdocs.pm/phoenix/releases.html#containers
I've also looked at other links, but these two was the best.
The only problem with that Dockerfile is that it needs $PORT (and $SECRET_KEY_BASE) on the build step which is not present. So we have to "stub" it somehow in prod.exs. BUT we have to provide real $PORT on a run step (runtime configuration). So you have to be sure that you renamed config/prod.secret.exs to config/releases.exs (the 1 step at https://hexdocs.pm/phoenix/releases.html#runtime-configuration) or just create config/releases.exs which would rewrite $PORT (and maybe some other application environment variables - for example, $SECRET_KEY_BASE).
UPD: as #Daniel mentions - releases is an Elixir 1.9 feature.
If someone could give me some help on heroku I would appreciate.
If I run this command on my local console, it works on -> localhost:8080
java -jar tools/jetty-runner-7.6.3.jar --port 8080 target/scala-2.9.1/myapp.war
So then I create the Procfile with this:
web: java -jar tools/jetty-runner-7.6.3.jar --port 8080 target/scala-2.9.1/myapp.war
Then I test by running:
foreman start
and it also works in -> localhost:8080
But after I deployed the app and make heroku open, it does not work...
[success] Total time: 1 s, completed Jun 22, 2012 11:16:59 AM
-----> Discovering process types
Procfile declares types -> web
-----> Compiled slug size is 70.0MB
-----> Launching... done, v5
http://myapp.herokuapp.com deployed to Heroku
To git#heroku.com:myapp.git
XXXXX..XXXXX master -> master
$heroku logs
2012-06-22T11:14:54+00:00 heroku[slugc]: Slug compilation started
2012-06-22T11:17:57+00:00 heroku[api]: Release v5 created by my_email#my_email.com
2012-06-22T11:17:57+00:00 heroku[api]: Deploy dcdd123 by my_email#my_email.com
2012-06-22T11:18:00+00:00 heroku[slugc]: Slug compilation finished
2012-06-22T11:18:21+00:00 heroku[router]: Error H14 (No web processes running) -> GET myapp.herokuapp.com/ dyno= queue= wait= service= status=503 bytes=
2012-06-22T11:18:30+00:00 heroku[router]: Error H14 (No web processes running) -> GET myapp.herokuapp.com/ dyno= queue= wait= service= status=503 bytes=
Any ideas, suggestions? Thanks,
Regards,
Andre
Andre - We have a Heroku Labs feature that allows you to deploy WAR files directly to Heroku. Check out https://gist.github.com/a8c62301591a11b4c6e9 for exact details on how to go about doing it.
If you are using jetty runner to run your war you do not need to hard code the port number.
Heroku sets the port internally.
So your Procfile should be like:
java -jar tools/jetty-runner-7.6.3.jar --port $PORT target/scala-2.9.1/myapp.war