I had a working deployment of Hasura on Heroku. It did run for about 2 years. However, recently, when I startup the application, there's a timeout error, resulting in the app just not starting anymore (see logs below). The credentials, urls etc are correct, since the application has been working before. There were no changes in settings.
I already had Heroku increase the timeout from 60 seconds to 180 seconds, but that doesn't help, but it gives me the reason for the failure: the connection to the postgres db fails.
However, the postgres database is running fine: I can connect to it and read the schemas and tables using psql (using the DATABASE_URL env variable from Heroku as URL). When I setup Hasura Cloud and connect to the postgres db on Heroku, it works fine (with the URL provided in the DATABASE_URL env variable in Heroku). So, the db runs, and the connection string is the right one.
This is my Dockerfile (updated Hasura to version 2.9.0):
$ cat Dockerfile
FROM hasura/graphql-engine:v2.9.0
# Enable the console
ENV HASURA_GRAPHQL_ENABLE_CONSOLE=true
# Enable debugging mode. It should be disabled in production.
ENV HASURA_GRAPHQL_DEV_MODE=true
# Heroku hobby tier PG has few limitations including 20 max connections
# https://devcenter.heroku.com/articles/heroku-postgres-plans#hobby-tier
ENV HASURA_GRAPHQL_PG_CONNECTIONS=15
# Change $DATABASE_URL to your heroku postgres URL if you're not using
# the primary postgres instance in your app
CMD graphql-engine \
--database-url $DATABASE_URL \
serve \
--server-port $PORT
## Comment the command above and use the command below to
## enable an access-key and an auth-hook
## Recommended that you set the access-key as a environment variable in heroku
#CMD graphql-engine \
# --database-url $DATABASE_URL \
# serve \
# --server-port $PORT \
# --access-key XXXXX \
# --auth-hook https://myapp.com/hasura-webhook
#
# Console can be enable/disabled by the env var HASURA_GRAPHQL_ENABLE_CONSOLE
Any ideas how I can find out the exact issue/how to debug it/how to solve it?
Thanks.
Output logs:
2022-08-08T07:26:47.538883+00:00 heroku[web.1]: Starting process with command `/bin/sh -c graphql-engine\ \ \ \ \ --database-url\ \postgres://edxvzxubxqcanh:68eaaa545e7742497f656a43c3d2946f039fc90806a07d691e02202a2541263c#ec2-108-128-201-166.eu-west-1.compute.amazonaws.com:5432/d25u4bqll5rg1a\ \ \ \ \ serve\ \ \ \ \ --server-port\ \52293`
2022-08-08T07:26:49.672834+00:00 app[web.1]: {"detail":{"info":{"admin_secret_set":true,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":true,"enabled_apis":["config","pgdump","graphql","metadata"],"enabled_log_types":["startup","webhook-log","websocket-log","http-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[{"audience":null,"claims_format":"json","claims_namespace":"https://hasura.io/jwt/claims","header":null,"issuer":null,"key":"<JWK REDACTED>","type":"<TYPE REDACTED>"}],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"info","port":52293,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":false,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":"public","use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"WSConnectionInitTimeout {unWSConnectionInitTimeout = Seconds {seconds = 3s}}","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Seconds {seconds = 5s}}"},"kind":"server_configuration"},"level":"info","timestamp":"2022-08-08T07:26:49.667+0000","type":"startup"}
2022-08-08T07:26:50.680661+00:00 app[web.1]: {"detail":{"info":{"database_url":"postgres://edxvzxubxqcanh:...#ec2-108-128-201-166.eu-west-1.compute.amazonaws.com:5432/d25u4bqll5rg1a","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2022-08-08T07:26:49.667+0000","type":"startup"}
2022-08-08T07:28:56.917452+00:00 app[web.1]: {"detail":{"message":"postgres connection failed, retrying(0)."},"level":"warn","timestamp":"2022-08-08T07:28:56.917+0000","type":"pg-client"}
2022-08-08T07:29:48.153244+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 180 seconds of launch
2022-08-08T07:29:48.246481+00:00 heroku[web.1]: Stopping process with SIGKILL
2022-08-08T07:29:48.392887+00:00 heroku[web.1]: Process exited with status 137
2022-08-08T07:29:48.786085+00:00 heroku[web.1]: State changed from starting to crashed
== EDIT 15/08/2022
After updating to hasura to version:
FROM hasura/graphql-engine:v2.11.0-beta.1
# Heroku only allows to install extensions in the heroku_ext schema
ENV HASURA_GRAPHQL_DEFAULT_EXTENSIONS_SCHEMA=heroku_ext
# https://github.com/hasura/graphql-engine/issues/8734
ENV HASURA_GRAPHQL_METADATA_DATABASE_EXTENSIONS_SCHEMA=heroku_ext
ENV HASURA_GRAPHQL_ENABLE_CONSOLE=true
ENV HASURA_GRAPHQL_DEV_MODE=true
ENV HASURA_GRAPHQL_PG_CONNECTIONS=15
CMD graphql-engine \
--database-url $DATABASE_URL \
serve \
--server-port $PORT
I still have the same issue, the connection to postgres doesn't work:
Starting process with command `/bin/sh -c graphql-engine\ \ \ \ \ --database-url\ \postgres://edxvzxubxqcanh:...#ec2-108-128-201-166.eu-west-1.compute.amazonaws.com:5432/d25u4bqll5rg1a\ \ \ \ \ serve\ \ \ \ \ --server-port\ \7315`
2022-08-15T14:09:24.454749+00:00 app[web.1]: {"detail":{"info":{"admin_secret_set":true,"auth_hook":null,"auth_hook_mode":null,"console_assets_dir":null,"cors_config":{"allowed_origins":"*","disabled":false,"ws_read_cookie":null},"enable_allowlist":false,"enable_console":true,"enable_maintenance_mode":false,"enable_metadata_query_logging":false,"enable_telemetry":true,"enabled_apis":["config","pgdump","metadata","graphql"],"enabled_log_types":["startup","webhook-log","http-log","websocket-log"],"events_fetch_batch_size":100,"experimental_features":[],"graceful_shutdown_timeout":60,"infer_function_permissions":true,"jwt_secret":[{"audience":null,"claims_format":"json","claims_namespace":"https://hasura.io/jwt/claims","header":null,"issuer":null,"key":"<JWK REDACTED>","type":"<TYPE REDACTED>"}],"live_query_options":{"batch_size":100,"refetch_delay":1},"log_level":"info","port":7315,"remote_schema_permissions":false,"server_host":"HostAny","stringify_numeric_types":false,"transaction_isolation":"ISOLATION LEVEL READ COMMITTED","unauth_role":"public","use_prepared_statements":true,"v1-boolean-null-collapse":false,"websocket_compression_options":"NoCompression","websocket_connection_init_timeout":"WSConnectionInitTimeout {unWSConnectionInitTimeout = Seconds {seconds = 3s}}","websocket_keep_alive":"KeepAliveDelay {unKeepAliveDelay = Seconds {seconds = 5s}}"},"kind":"server_configuration"},"level":"info","timestamp":"2022-08-15T14:09:24.404+0000","type":"startup"}
2022-08-15T14:09:25.468623+00:00 app[web.1]: {"detail":{"info":{"database_url":"postgres://edxvzxubxqcanh:...#ec2-108-128-201-166.eu-west-1.compute.amazonaws.com:5432/d25u4bqll5rg1a","retries":1},"kind":"postgres_connection"},"level":"info","timestamp":"2022-08-15T14:09:24.404+0000","type":"startup"}
2022-08-15T14:11:29.990364+00:00 heroku[router]: at=error code=H20 desc="App boot timeout" method=OPTIONS path="/v1/graphql" host=systemic-nrr-store.herokuapp.com request_id=d4620807-2a79-468c-b658-be14fd31ec55 fwd="178.51.9.123" dyno= connect= service= status=503 bytes= protocol=https
2022-08-15T14:11:30.005688+00:00 heroku[router]: at=error code=H20 desc="App boot timeout" method=OPTIONS path="/v1/graphql" host=systemic-nrr-store.herokuapp.com request_id=2370dc41-749e-44f8-81c4-73052eec90fe fwd="178.51.9.123" dyno= connect= service= status=503 bytes= protocol=https
2022-08-15T14:11:30.109890+00:00 heroku[router]: at=error code=H20 desc="App boot timeout" method=OPTIONS path="/v1/graphql" host=systemic-nrr-store.herokuapp.com request_id=796a2090-79f4-4a3b-89a7-c00b55870af9 fwd="178.51.9.123" dyno= connect= service= status=503 bytes= protocol=https
2022-08-15T14:11:31.738877+00:00 app[web.1]: {"detail":{"message":"postgres connection failed, retrying(0)."},"level":"warn","timestamp":"2022-08-15T14:11:31.738+0000","type":"pg-client"}
I'm trying to spin up an experiment on a Heroku app with psiturk. I have previously gotten this to work (~2019), but recently (since 2021) I have not been able to run psiturk on Heroku without the app crashing, even using exactly the same code.
Here are some of the steps I have taken so far:
The experiment runs locally, and I have confirmed that I can connect to my postgres database from the Heroku app.
I have also confirmed the Heroku is dynamically binding the port, and not using the one set in config.txt.
In the Heroku bash environment, I am able to start psiturk and turn on the server without the app crashing, but the app will crash when loading a debug of sandbox HIT link.
The Heroku app uses python and node buildpacks, and the Procfile points to a python script. I am using psiturk 2.3.11 and python 3.8.8, and I have followed the steps in the psiturk documentation for setting config variables.
Here is the Heroku log:
2021-03-29T16:44:16.409840+00:00 app[api]: Starting process with command `bash` by user danibassettlab#gmail.com
2021-03-29T16:44:39.951733+00:00 heroku[run.7975]: State changed from starting to up
2021-03-29T16:44:40.507250+00:00 heroku[run.7975]: Awaiting client
2021-03-29T16:44:40.583037+00:00 heroku[run.7975]: Starting process with command `bash`
2021-03-29T17:04:33.678083+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/pub/ad?assignmentId=debug98GDW3&hitId=debug6JU43S&workerId=debugG38KJ9&mode=debug" host=jeni-docker-test.herokuapp.com request_id=2bdae5da-5f59-475e-8ca6-9ea9a380d5c0 fwd="70.16.141.206" dyno= connect= service= status=503 bytes= protocol=http
2021-03-29T17:04:33.870529+00:00 heroku[router]: at=error code=H10 desc="App crashed" method=GET path="/favicon.ico" host=jeni-docker-test.herokuapp.com request_id=828565a0-923b-4283-b9de-17b4cf890459 fwd="70.16.141.206" dyno= connect= service= status=503 bytes= protocol=http
And the config.txt I'm using (with account information removed)
[HIT Configuration]
title = Key Sequence Learning Study
description = Psychology Study in which you will learn to quickly press keys to respond to squares on the screen.
lifetime = 96
us_only = true
approve_requirement = 95
ad_group = default_network
psiturk_keywords = network
organization_name = University of Pennsylvania
browser_exclude_rule = mobile, tablet
allow_repeats = false
server_timeout = 240
[Database Parameters]
database_url = postgres:<my_database_url>
table_name = participants_tmp
[Server Parameters]
host = 0.0.0.0
port = 22362
cutoff_time = 30
logfile = server.log
loglevel = 2
debug = false
login_username = examplename
login_pw = examplepassword
threads = 1
secret_key = 'this is my secret key which is hard to guess, i should change this'
[Task Parameters]
experiment_code_version = 'behavioral_pilot_v6_delay'
num_conds = 1
num_counters = 1
[Shell Parameters]
launch_in_sandbox_mode = true
bonus_message = "Bonus for Bassett Lab Experiment. Thanks for participating!"
use_psiturk_ad_server = false
ad_location = https://jeni-docker-test.herokuapp.com/pub
The crash happens before anything is logged to the psiturk server.log file in the experiment folder.
Any advice on next steps?
I found the answer on the psiturk google group. Psiturk 2 needs the parameter ON_HEROKU to be set (I had set ON_CLOUD insteadt. Running
heroku config:set ON_HEROKU=1
fixed the issue.
Alternatively you could upgrade to psiturk 3
The newer version of SQLAlchemy removed the deprecated dialect name postgres. You have to use postgresql. Among your database parameters, you have:
database_url = postgres:<my_database_url>
To fix this, rename postgres:// in the URL to postgresql://.
Alternatively consider to add sslmode='require' to your database URL.
My first time deploying streamlit on Heroku, and the scripts are on this GitHub. I'm actually deploying from local, so this is just to share my scripts to debug. The following is the error message.
2020-12-31T23:47:18.055589+00:00 heroku[router]:
at=error code=H20 desc="App boot timeout"
method=GET path="/" host=stormy-dawn-49490.herokuapp.com
request_id=b3df4074-d363-41e5-8ecd-d101070679f7
fwd="99.230.168.115"
dyno= connect= service= status=503 bytes= protocol=https
2020-12-31T23:48:01.327576+00:00 heroku[web.1]: Error R10 (Boot timeout) ->
Web process failed to bind to $PORT within 60 seconds of launch
I have four scripts, which is also linked to my GitHub repository the beginning of the question.
setup.sh
mkdir -p ~/.streamlit/
echo “\
[general]\n\
email = \”<dont#spam.us>\”\n\
“ > ~/.streamlit/credentials.toml
echo “\
[server]\n\
headless = true\n\
enableCORS=false\n\
port = $PORT\n\
“ > ~/.streamlit/config.toml
Procfile:
web: sh setup.sh && streamlit run app.py
requirements.txt:
fredapi==0.4.2
numpy==1.18.5
pandas==1.1.5
plotly==4.14.1
python-dateutil==2.8.1
python-slugify==4.0.1
Quandl==3.5.3
scikit-learn==0.23.2
streamlit==0.72.0
yfinance==0.1.55
gunicorn== 19.9.0
app.py: I'd suggest read here
I didn't put much in the app.py as I'm just trying how to deploy on Heroku
Thank you in advance!
Once I started using Channels with Phoenix, I started getting a lot of these errors on Heroku:
heroku[router]: at=error code=H15 desc="Idle connection" method=GET
path="/socket/websocket?guardian_token=eyJhbGciOiJIUz[...]&vsn=1.0.0" host=[...]
dyno=web.1 connect=0ms service=87803ms status=503 bytes=
A reliable way to reproduce this is on a laptop:
Open a Phoenix page that connects to WebSockets
Close the lid of your laptop
Wait a minute or so (WiFi to disconnect/OS goes to sleep)
The H15 error will occur.
I've set a 45s timeout on my UserChannel already; I'm running Elixir 1.4.2, Phoenix 1.2.1 and my phoenix.js was just upgraded to the one used in Phoenix 1.3.0-rc0 (link to rev)
I can see the error logged on the front-end, but after that it reconnects happily to the backend:
2017-03-04 23:28:55.351 receive: ok user:2 phx_reply (3)
2017-03-04 23:29:24.269 channel: error user:2
2017-03-04 23:29:28.713 push: user:2 phx_join (4)
2017-03-04 23:29:28.745 receive: ok user:2 phx_reply (4)
Any help or ideas how to solve this is much appreciated.
Edit: I'm familiar with Heroku's H15 error code definition so I'm looking for a fix or a workaround for getting rid of the errors (other than migrating away from Heroku, of course).
Phoenix docs recommend setting the timeout for the websocket transport to < 55 seconds:
defmodule HelloWeb.UserSocket do
use Phoenix.Socket
...
## Transports
transport :websocket, Phoenix.Transports.WebSocket,
timeout: 45_000
...
end
Their explanation is:
This ensures that any idle connections are closed by Phoenix before they reach Heroku’s 55-second timeout window.
I have big issues with deploying my application on heroku server. I've done it with help of tutorial from phoenix webpage. But it doesn't work. Below you can see the logs and configuration i used.
heroku logs
2016-02-07T04:28:55.623435+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web
process failed to bind to $PORT within 60 seconds of launch
2016-02-07T04:28:55.623435+00:00 heroku[web.1]: Stopping process with SIGKILL
2016-02-07T04:28:56.344649+00:00 heroku[web.1]: State changed from starting to c
rashed
2016-02-07T04:28:56.348324+00:00 heroku[web.1]: Process exited with status 137
2016-02-07T09:54:29.124036+00:00 heroku[web.1]: State changed from crashed to st
arting
2016-02-07T09:54:34.703208+00:00 heroku[web.1]: Starting process with command `m
ix run --no-halt`
2016-02-07T09:55:35.386922+00:00 heroku[web.1]: Error R10 (Boot timeout) -> Web
process failed to bind to $PORT within 60 seconds of launch
2016-02-07T09:55:35.386922+00:00 heroku[web.1]: Stopping process with SIGKILL
2016-02-07T09:55:36.330258+00:00 heroku[web.1]: State changed from starting to c
rashed
2016-02-07T09:55:36.323082+00:00 heroku[web.1]: Process exited with status 137
2016-02-07T12:14:35.182306+00:00 heroku[router]: at=error code=H10 desc="App cra
shed" method=GET path="/" host=kpsz.herokuapp.com request_id=7de0ee97-14cf-46f9-
a9a3-cbffa09ac379 fwd="83.7.11.212" dyno= connect= service= status=503 bytes=
2016-02-07T12:14:35.850018+00:00 heroku[router]: at=error code=H10 desc="App cra
shed" method=GET path="/favicon.ico" host=kpsz.herokuapp.com request_id=4e9b1979
-deb6-497b-80b2-655615f43d01 fwd="83.7.11.212" dyno= connect= service= status=50
3 bytes=
Configuration prod.exs
config :kpsz, Kpsz.Endpoint,
http: [port: System.get_env("PORT")],
url: [host: "example.com", port: 80],
cache_static_manifest: "priv/static/manifest.json",
secret_key_base: System.get_env("SECRET_KEY_BASE")
config :kpsz, Kpsz.Repo,
adapter: Ecto.Adapters.Postgres,
url: System.get_env("DATABASE_URL"),
pool_size: 20
I'm not sure if that's the problem but you could try to replace
http: [port: System.get_env("PORT")]
with
http: [port: {:system, "PORT"}]
At least that's what I'm using in my apps. You can also read the docs for more information http://www.phoenixframework.org/docs/heroku
I had a similar problem.
Adding a Procfile with a single line web: mix phoenix.server solved it.
heroku destroy project name
heroku create
and for some weird reason it works now...
Thanks very much for help.
I ran into this issue and, for me, the fix was as follow (copied from the answer on this question).
The fix is twofold:
to use the phoenix buildpack config and export the PORT as well, like so: config_vars_to_export=(DATABASE_URL PORT)
Avoid changing the MIX_ENV without recompiling the app, as that behavior is unsupported, as explained here.
As Recreation solves the problem, I suspect the problem is with loading environment variables.
Once I had a similar problem in my local. My observations
mix phoenix.server (you might see the message compiled so and so files)
Stopped the server
Export/Change the env variable in zshrc/bashrc file and source it
Start the server (you might not see that compilation message, as there are no changes in your codebase and so your new/edited env variables won't get reflected)
If you just add a space in that file will make it compile and take that env variable.
Reason for your problem might be: Setting env variables after the server is running/deployed, as env variables are converted to value during the compilation time itself.
I had a similar problem when deploying the phoenix umbrella project with docker.
I've taken these steps:
Heroku https://hexdocs.pm/phoenix/heroku.html#deploying-to-heroku-using-the-container-stack
Dockerfile https://hexdocs.pm/phoenix/releases.html#containers
I've also looked at other links, but these two was the best.
The only problem with that Dockerfile is that it needs $PORT (and $SECRET_KEY_BASE) on the build step which is not present. So we have to "stub" it somehow in prod.exs. BUT we have to provide real $PORT on a run step (runtime configuration). So you have to be sure that you renamed config/prod.secret.exs to config/releases.exs (the 1 step at https://hexdocs.pm/phoenix/releases.html#runtime-configuration) or just create config/releases.exs which would rewrite $PORT (and maybe some other application environment variables - for example, $SECRET_KEY_BASE).
UPD: as #Daniel mentions - releases is an Elixir 1.9 feature.