When an integrated service, like CodeShip CI, runs tests, an instance is spun-up on Heroku to run the CI suite.
How can I get the name of that branch/build-specific Heroku app, in a programmatic manner?
My use case: I want to give a developer heroku-cli access to the staging instance that was spun-up for their branch. Also, I want the instance URL so that QA can check it for accuracy.
I don't know if this is principally a CodeShip question, or a Heroku question. I can solve the rest of the integration, if I simply can get the name/info for this new instance.
Codeship defines certain environment variables with each run.
Thus I can build a URL based on all of the known pieces, per env:
app_name = 'foo'
env = 'staging'
name_parts = [
app_name,
env,
'pr'
ENV['CI_BUILD_NUMBER']
]
testing_url = 'http://' + name_parts.join('-') + '.herokuapp.com'
#=> foo-staging-pr-2729.herokuapp.com
send_to_chatroom(app_name + '_dev_notification')
Related
I'm trying to run a kubeflow pipeline setup and I have several environements (dev, staging, prod).
In my pipeline I'm using kfp.components.func_to_container_op to get a pipeline task instance (ContainerOp), and then execute it with the appropriate arguments that allows it to integrate with my s3 bucket:
from utils.test import test
test_op = comp.func_to_container_op(test, base_image='my_image')
read_data_task = read_data_op(
bucket,
aws_key,
aws_pass,
)
arguments = {
'bucket': 's3',
'aws_key': 'key',
'aws_pass': 'pass',
}
kfp.Client().create_run_from_pipeline_func(pipeline, arguments=arguments)
Each one of the environments is using different credentials to connect to it and those credentials are being passed in the function:
def test(s3_bucket: str, aws_key: str, aws_pass: str):
....
s3_client = boto3.client('s3', aws_access_key_id=aws_key, aws_secret_access_key=aws_pass)
s3_client.upload_file(from_filename, bucket_name, to_filename)
so for each environment I need to update the arguments to contain the correct credentials and it makes it very hard to maintain since each time that I want to update from dev to stg to prod I can't simply copy the code.
My question is what is the best approach to pass those credentials?
Ideally you should push any env-specific configurations as close to the cluster as possible (as far away from components).
You can create Kubernetes secret in each environemnt with different creadentials. Then use that AWS secret in each task:
from kfp import aws
def my_pipeline():
...
conf = kfp.dsl.get_pipeline_conf()
conf.add_op_transformer(aws.use_aws_secret('aws-secret', 'AWS_ACCESS_KEY_ID', 'AWS_SECRET_ACCESS_KEY'))
Maybe boto3 can auto-load the credentials using the secret files and the environment variables.
At least all GCP libraries and utilities do that with GCP credentials.
P.S. It's better to create issues in the official repo: https://github.com/kubeflow/pipelines/issues
I'm using the Review Apps feature integrated with Github on Heroku. In one of my apps, I set an environment variable called HOST_NAME . For example, if the site is http://www.purplebinder.com, then HOST_NAME would be set to www.purplebinder.com. It's used in a couple of places where we work with cookies and in our transactional emails.
When I open up a new pull request and spin up a review app, HOST_NAME should be something like purplebinder-pr-27.herokuapp.com.
Is there a way to set this value automatically? The Heroku documentation on review apps says an env var can inherit a value from the parent app or be hardcoded in app.json. Neither of those approaches work here, because the value needs to be different each time, and also different from the parent app.
Heroku also says an env var can be set "through a generator", but doesn't go into detail about what that is.
This question might be a duplicate of Setting ROOT_URL for Review Apps, but nobody answered that one. It's also similar to How to get Heroku app name from inside the app, but the answers there involved running a script after the app was created - here I'd like to set this value as part of the initial build.
From https://devcenter.heroku.com/articles/github-integration-review-apps#heroku_app_name-and-heroku_parent_app_name:
To help with scripting, two special config vars are available to
review apps. If you specify HEROKU_APP_NAME or HEROKU_PARENT_APP_NAME
as required or optional config vars in your app.json file, Heroku will
set those config vars to the new application name and the parent
application name respectively. They will then be available for use in
the postdeploy script so that you can do more advanced bootstrapping
and configuration.
Here is an example app.json file that uses
HEROKU_APP_NAME and HEROKU_PARENT_APP_NAME:
{
"name":"Advanced App",
"scripts": {
"postdeploy": "rake db:setup && bin/bootstrap"
},
"env": {
"HEROKU_APP_NAME": {
"required": true
},
"HEROKU_PARENT_APP_NAME": {
"required": true
}
}
}
If you add the heroku-buildpack-cli to your parent app, then it enables you to set environment variables from your post-deploy script. The command should look something like the following:
heroku config:set HOST_NAME=${HEROKU_APP_NAME}.herokuapp.com --app ${HEROKU_APP_NAME}
Here's an approach ignoring app.json for Rails installations:
in the relative config/<environment>.rb. I personally use production.rb and staging just references it.
if ENV.fetch("HEROKU_APP_NAME", "").include?("staging-pr-")
ENV["APPLICATION_HOST"] = ENV["HEROKU_APP_NAME"] + ".herokuapp.com"
ENV["ASSET_HOST"] = "https://" + ENV["APPLICATION_HOST"]
config.action_mailer.default_url_options = { host: ENV.fetch("APPLICATION_HOST") }
end
...
It's a bit misleading as the heroku environment variables will still have the old variables, but it works.
You can also create review environment for you application copying staging.rb or production.rb from config/environments. This would be useful.
After adding HEROKU_APP_NAME and HEROKU_PARENT_APP_NAME to your app.json, you can easily set;
config.action_mailer.default_url_options = { host: "#{ENV['HEROKU_APP_NAME']}.herokuapp.com" }
config.action_mailer.asset_host = "http://#{ENV['HEROKU_APP_NAME']}.herokuapp.com"
config.action_controller.asset_host = "#{ENV['HEROKU_APP_NAME']}.herokuapp.com"
config.action_cable.url = "wss://#{ENV['HEROKU_APP_NAME']}.herokuapp.com/cable"
I was following this guide on deploying to Heroku and this one for sending email.
Everything works fine in development. My variables are set in Heroku:
heroku config
...
MAILGUN_DOMAIN: https://api.mailgun.net/v3/xxxxxx.mailgun.org
MAILGUN_KEY: key-3-xxxxxx
...
And loaded from the config files like so:
config :take_two, Mailer,
domain: System.get_env("MAILGUN_DOMAIN"),
key: System.get_env("MAILGUN_KEY")
However when I try to send email on Heroku when the Mailgun config is set from environment variables I get this error:
** (FunctionClauseError) no function clause matching in IO.chardata_to_string/1
(elixir) lib/io.ex:346: IO.chardata_to_string(nil)
(elixir) lib/path.ex:467: Path.join/2
(elixir) lib/path.ex:449: Path.join/1
lib/client.ex:44: Mailgun.Client.send_without_attachments/2
This happens when the domain is not set for the Mailgun Client. But it is supposed to be set from the environment variable. I made a simple module to test:
defmodule TakeTwo.Mailer do
require Logger
use Mailgun.Client,
Application.get_env(:take_two, Mailer)
def blank_shot do
Logger.info Application.get_env(:take_two, Mailer)[:domain]
Logger.info Application.get_env(:take_two, Mailer)[:key]
send_email from: "steve#xxx.com", to: "speggy#xxx.com", subject: "Hello", text: "This is a blank shot"
end
When I run TakeTwo.Mailer.blank_shot I see the correct domain/key variables logged followed by the error. I am not sure how to debug the Mailgun client remotely.
Finally, if I recreate the above module in the shell (after running heroku run iex -S mix) it works just fine!?
I feel like when the original module is being loaded perhaps the environment variables have yet to be loaded??
The answer was a little buried in a comment so I wanted to make it easier to find. As the other answer mentions, the environment variables aren't available, but the buildpack lets you configure them to be:
I created a elixir_buildpack.config file and added the following:
config_vars_to_export=(DATABASE_URL MAILGUN_DOMAIN MAILGUN_KEY SECRET_KEY_BASE)
The environment variables aren't available at build time. I had the same issue and decided to get rid of the macro carrying the configuration. You can use this patch to move on.
I'm using Cucumber and Capybara for my automated front end tests.
I have two environments that I would like to run my tests on. One is a staging environment, and the other is the production environment.
Currently, I have my tests written to access staging directly.
visit('https://staging.somewhere.com')
I would like to re-use the tests in production (https://production.somewhere.com).
Would it be possible to store the URL in a variable in my step definitions
visit(domain)
and define domain using an environment variable called form the command line? Like
$> bundle exec cucumber features DOMAIN=staging
if I want to point the tests to my staging environment, or
$> bundle exec cucumber features DOMAIN=production
if I want it to run in production?
How do I go about setting this up? I'm fairly new to Ruby and I've been searching the forums for a straight forward information but could not find any. Let me know if I can provide more information. Thanks for your help!
In the project's config file, create a config.yml file
---
staging:
:url: https://staging.somewhere.com
production:
:url: https://production.somewhere.com
Then extra colon in the yml file allows the hash key to be called as a symbol.
In your support/env.rb file, add the following
require 'yaml'
ENV['TEST_ENV'] ||= 'staging'
project_root = File.expand_path('../..', __FILE__)
$BASE_URL = YAML.load_file(project_root + "/config/config.yml")[ENV['TEST_ENV']][:url]
This will default to the staging environment unless you override the TEST_ENV. Then, from your step or hook, you can call:
visit($BASE_URL)
or you might need :/
visit "#{$BASE_URL}"
This will allow you to use
bundle exec cucumber features TEST_ENV=production
I don't use cucumber much but you should be able to do
bundle exec cucumber features DOMAIN=staging
then in your tests use ENV['DOMAIN'] || YOUR_DEFAULT_DOMAIN to utilize this variable. YOUR_DEFAULT_DOMAIN should probably be your test environment.
See Here
I know the slug compiler removes the .git directory when creating a heroku slug, but is there any way to configure Heroku so that I can access the currently running git commit number from within my scripts?
I'd like to be able to have a small link on my sinatra app (run within Heroku) which says "running version e72fb274a0" (or something similar). How can I retrieve this, or force the slug compiler to add it to an environment variable?
PROGRESS:
I reckon the best way to do this is to make a custom buildpack which writes the git commit version number to the heroku slug before the .git directory is deleted.
I've tried to do this (see my fork of the ruby buildpack) but the line I've added – line 23 – doesn't seem to be doing the job. Heroku sees & uses the new buildpack, but doesn't seem to write the file to the slug.
Anyone have any idea why my custom buildpack isn't working as expected?
Thanks,
JP
A couple of options...
SOURCE_VERSION environment variable (build-time)
Since 1st April 2015, there's a SOURCE_VERSION environment variable available to builds running on Heroku. For git-pushed builds, this is the git commit SHA-1 of the source being built:
https://devcenter.heroku.com/changelog-items/630
(thanks to #srtech for pointing that out!)
An example of me using that variable in a build - if you look at the HTML served by the deployed app, you'll see the commit id is coming though in an HTML comment near the very bottom: https://gu-who.herokuapp.com/
/etc/heroku/dyno metadata file (run-time)
Heroku have beta functionality to write out a /etc/heroku/dyno metadata file onto your running dyno. If you email support you can probably get added to the beta. Here's a place where Heroku themselves are using it:
https://github.com/heroku/fix/blob/6c8ab7a/lib/heroku_dyno_metadata.rb
The contents look like this:
{
"dyno":{
"physical_id":"161bfad9-9e83-40b7-b385-78305db2f168",
"size":1,
"name":"run.7145"
},
"app":{
"id":null
},
"release":{
"id":50,
"commit":"2c3a0b24069af49b3de35b8e8c26765c1dba9ff0",
"description":null
}
}
..so release.commit is the field you're after. I used to use this method until the SOURCE_VERSION variable became available.
In 2018 this is what you want:
https://devcenter.heroku.com/articles/dyno-metadata
heroku labs:enable runtime-dyno-metadata -a <app name>
You can run a script before deploy that store this information (maybe on a YAML)
using these a = `ls` (note that is not ' "apostrophe" sign is ` "inverse accute" sign)
the a variable will have the result of this bash command,so you can do
git = `git log`
and then find the information you want it and store it.
So you will be able to retrieve it later.
Did this helped ?