I enter the following code to connect to my heroku database using Python:
urlparse.uses_netloc.append("postgres")
url = urlparse.urlparse(os.environ["DATABASE_URL"])
conn = psycopg2.connect(
database=rul.pat[1:],
user=url.username,
password=url.password,
host=url.hostname,
port=url.port
)
I get a key error on the second line. Can someone describe what is happening? Do I need to add the DATABASE_URL to the environment somehow? I am running the Linux bash on Windows 10.
I found the answer here:
KeyError: 'DATABASE_URL' in django 1.4
in the terminal I typed something like this:
export DATABASE_URL=<url to database>
Related
I have followed the documentation and in my Python code I've added:
import os
import psycopg2
DATABASE_URL = os.environ['postgres://sficoXXX:842YYY#ec2-54-165-184-219.compute-1.amazonaws.com:5432/database-name']
conn = psycopg2.connect(DATABASE_URL, sslmode='require')
The URL I took from the session Config Vars (I used: heroku pg:credentials:url).
When I check the connection to the database using heroku pg:psql everything seems to be working fine.
But after deploying it shows the following error:
Failed to create session:
'postgres://sficoXXX:842YYY#ec2-54-165-184-219.compute-1.amazonaws.com:5432/database-name'
Traceback (most recent call last): File
"/Users/spoleto/.pyenv/versions/3.8.13/lib/python3.8/site-packages/otree/session.py",
line 447, in create_session_traceback_wrapper
return create_session(**kwargs) File "/Users/spoleto/.pyenv/versions/3.8.13/lib/python3.8/site-packages/otree/session.py",
line 418, in create_session
func(subsession) File "/Users/spoleto/PycharmProjects/upstream-reciprocity/prototypes/Consent/init.py",
line 35, in creating_session
DATABASE_URL = os.environ[postgres://sficoXXX:842YYY#ec2-54-165-184-219.compute-1.amazonaws.com:5432/database-name']
File "/Users/spoleto/.pyenv/versions/3.8.13/lib/python3.8/os.py", line
675, in getitem
raise KeyError(key) from None
Am I doing this right? Where does the error come from? How is the connection supposed to be established?
This is almost a copy/paste from the documentation.
The whole point of putting a connection string into an environment variable is so it doesn't need to be in your source code.
Instead of looking up the value of DATABASE_URL manually and pasting it into your source code, use the name of the environment variable:
DATABASE_URL = os.environ['DATABASE_URL']
Now your code will look for an environment variable with that name dynamically every time it runs, and set your DATABASE_URL variable to the value of the DATABASE_URL environment variable.
(The code you show in your question looks for an environment variable named postgres://..., which is very unlikely to exist.)
Note that this will fail with an IndexError if an environment variable named DATABASE_URL cannot be found. A safer way of doing this might be to use the .get() method (make sure to use round parentheses instead of square brackets):
DATABASE_URL = os.environ.get('DATABASE_URL')
Now you can even provide a fallback, e.g. for local development:
DATABASE_URL = os.environ.get('DATABASE_URL') or "postgres://postgres:postgres#localhost:5432/postgres"
Note: You didn't leak your whole connection string in your question, but I suggest you rotate your credentials anyway:
heroku pg:credentials:rotate
This will invalidate the old connection string and generate a new one.
The good news is that your DATABASE_URL environment variable will automatically be updated, and since your code now reads that value at runtime, it will continue to work!
I was following this guide on deploying to Heroku and this one for sending email.
Everything works fine in development. My variables are set in Heroku:
heroku config
...
MAILGUN_DOMAIN: https://api.mailgun.net/v3/xxxxxx.mailgun.org
MAILGUN_KEY: key-3-xxxxxx
...
And loaded from the config files like so:
config :take_two, Mailer,
domain: System.get_env("MAILGUN_DOMAIN"),
key: System.get_env("MAILGUN_KEY")
However when I try to send email on Heroku when the Mailgun config is set from environment variables I get this error:
** (FunctionClauseError) no function clause matching in IO.chardata_to_string/1
(elixir) lib/io.ex:346: IO.chardata_to_string(nil)
(elixir) lib/path.ex:467: Path.join/2
(elixir) lib/path.ex:449: Path.join/1
lib/client.ex:44: Mailgun.Client.send_without_attachments/2
This happens when the domain is not set for the Mailgun Client. But it is supposed to be set from the environment variable. I made a simple module to test:
defmodule TakeTwo.Mailer do
require Logger
use Mailgun.Client,
Application.get_env(:take_two, Mailer)
def blank_shot do
Logger.info Application.get_env(:take_two, Mailer)[:domain]
Logger.info Application.get_env(:take_two, Mailer)[:key]
send_email from: "steve#xxx.com", to: "speggy#xxx.com", subject: "Hello", text: "This is a blank shot"
end
When I run TakeTwo.Mailer.blank_shot I see the correct domain/key variables logged followed by the error. I am not sure how to debug the Mailgun client remotely.
Finally, if I recreate the above module in the shell (after running heroku run iex -S mix) it works just fine!?
I feel like when the original module is being loaded perhaps the environment variables have yet to be loaded??
The answer was a little buried in a comment so I wanted to make it easier to find. As the other answer mentions, the environment variables aren't available, but the buildpack lets you configure them to be:
I created a elixir_buildpack.config file and added the following:
config_vars_to_export=(DATABASE_URL MAILGUN_DOMAIN MAILGUN_KEY SECRET_KEY_BASE)
The environment variables aren't available at build time. I had the same issue and decided to get rid of the macro carrying the configuration. You can use this patch to move on.
I am unable to install SASS through command prompt.
I tried below steps
c:\gem install sass
I am getting below error:
ERROR: Could not find a valid gem 'sass' (>= 0), here is why:
Unable to download data from https://rubygems.org/ - Errno::ECONNREFUSED: No connection could be made because the target machine actively refused it. - connect(2) (https://rubygems.org/latest_specs.4.8.gz)
Please help me,
For http/https proxy with or without authentication:
Run one of the following commands in cmd.exe
set http_proxy=http://your_proxy:your_port
set http_proxy=http://username:password#your_proxy:your_port
set https_proxy=https://your_proxy:your_port
set https_proxy=https://username:password#your_proxy:your_port
You need to write this in the command prompt:
set HTTP_PROXY=http://your_proxy:your_port
You need to add http_proxy and https_proxy environment variables as described here.
When setting http_proxy and https_proxy, you are also probably going to need no_proxy for URLs on the same side of the proxy. https://msdn.microsoft.com/en-us/library/hh272656(v=vs.120).aspx
In Addition to #Yifei answer. If you have special character like #, &, $
You have to go with percent-encode | encode the special characters. E.g. instead of this:
http://foo:B#r#http-gateway.domain.org:80
you write this:
http://foo:B%40r#http-gateway.domain.org:80
So # gets replaced with %40.
You can try export http_proxy=http://your_proxy:your_port
None of the answers here actually helped my case (proxy + password), instead I found a solution on a Github issue:
https://github.com/rubygems/rubygems/issues/1068
Basically I had to set three variables:
set http_proxy=proxy_ip:port
set http_proxy_user=user
set http_proxy_pass=password
To set config vars for a Heroku app, you do this:
$ heroku config:set GITHUB_USERNAME=joesmith
How would I set a config var with the contents of a file?
Take a look at the heroku-config plugin, which adds a heroku config:push command to push key-value pairs in a file named .env to the app.
It also has a heroku config:pull command to do the opposite and works very well with foreman for running the app locally with the config in .env.
https://github.com/xavdid/heroku-config
Example
heroku config:push --file=.env.production
I know this is too late but still it will be helpful for future users who land here.
I also wanted a quick solution to add variable to heroku app but copy-paste is so boring.. So wrote a script to read values from the .env file and set it at the requested app - all things passed as an option:
https://gist.github.com/md-farhan-memon/e90e30cc0d67d0a0cd7779d6adfe62d1
Usage
./bulk_add_heroku_config_variables.sh -f='/path/to/your/.environment/file' -s='bsc-server-name' -k='YOUR_CONFIG_KEY1 YOUR_CONFIG_KEY2'
A simple pure-python solution using honcho and invoke:
from honcho.environ import parse
from invoke import run
def push_env(file='.env'):
"""Push .env key/value pairs to heroku"""
with open(file, 'r') as f:
env = parse(f.read())
cmd = 'heroku config:set ' + ' '.join(
f'{key}={value}' for key, value in env.items())
run(cmd)
The idea here is that you will get the same configuration as if you ran the project locally using honcho. Then I use invoke to run this task easily from the command line (using #task and c.run) but I've adapted it here to stand alone.
Normally I would set the pool size as
development:
adapter: postgresql
encoding: unicode
database: openkitchen_development
username: rails
host: localhost
pool: 10
password:
in database.yml. However heroku replaces the config file. I'm using girl_friday to
do background db work and need to increase the thread pool size.
Simply add a pool query parameter to the DATABASE_URL in your heroku config. To set the pool size to 15 in your heroku app use something like:
heroku config -s | awk '/^DATABASE_URL=/{print $0 "?pool=15"}' | xargs heroku config:add
For what it's worth, using the URL params method as described in other answers here is not recommended by Heroku. They reserve the right to reset or change this URL at any time, and long term this behavior will likely be removed for the Rails build behavior, anyway.
Setting additional parameters via an after-initialize application callback is the recommended way to modify the configuration of your heroku-postgresql databases per this dev center article.
In config/initializers/database_connection.rb:
Rails.application.config.after_initialize do
ActiveRecord::Base.connection_pool.disconnect!
ActiveSupport.on_load(:active_record) do
config = Rails.application.config.database_configuration[Rails.env]
config['pool'] = 10
ActiveRecord::Base.establish_connection(config)
end
end
Heroku now has a nice article on managing pool sizes - https://devcenter.heroku.com/articles/concurrency-and-database-connections#connection-pool
remvee's answer gets to the heart of what is needed but since his command caused my console to hang I thought I would write up how to do this manually.
heroku config
Look for the DATABASE_URL key. For this example lets say it is:
DATABASE_URL: mysql2://something.example.com/stuff?reconnect=true
Add "&pool=10" to the end of the URL (use & instead of ? because the url already has a parameter)
heroku config:add DATABASE_URL=mysql2://something.example.com/stuff?reconnect=true&pool=10
It's not very straight forward but you could try creating your own buildpack.
You'll nee to fork:
https://github.com/heroku/heroku-buildpack-ruby
Then modify the following:
https://github.com/heroku/heroku-buildpack-ruby/blob/master/lib/language_pack/ruby.rb#L325-387
Just add the pool size you require.
Then you can create a new Heroku app with your custom buildpack:
heroku create --stack cedar --buildpack https://github.com/yourgithubusername/heroku-buildpack-ruby.git
That should be it!