Are there any official (or unofficial) ways of storing entire heroku configurations as code?
I think most heroku configurations can be stored in a bash script. But things like scheduled jobs and the like still have to be performed manually.
Is there any way to store entire app configurations as code so that the setup can be used directly or as a starting point for other applications?
I since learned that you can indeed use terraform with heroku.
Source:
Using Terraform with Heroku
Related
I have a single repository that hosts my lambda functions on github. I would like to be able to deploy the new versions whenever new logic is pushed to master.
I did a lot of reasearch and found a few different approaches, but nothing really clear. Would like to know what others feel would be the best way to go about this, and maybe some detail (if possible) into how that pipeline is setup.
Thanks
Welcome to StackOverflow. You can improve your question by reading this page.
You can setup a CI/CD pipeline using CircleCI with its GitHub integration (which is an online Service, so you don't need to maintain anything, like a Jenkins server, for example)
Upon every commit to your repository, a CircleCI build will be triggered. Once the build process is over, you can declare sls deploy, sam deploy, use Terraform or even create a script to upload the .zip file from your GitHub repo to an S3 Bucket and then, within your script, invoke the create-function command. There's an example how to deploy Serverless applications using CircleCI along with the Serverless Framework here
Other options include TravisCI, AWS Code Deploy or even maintain your own CI/CD Server. The same logic applies to all of these tools though: commit -> build -> deploy (using one of the tools you've chosen).
EDIT: After #Matt's answer, it clicked that the OP never mentioned the Serverless Framework (I, somehow, thought he was already using it, so I pointed the OP to tutorials using the Serverless Framework already). I then decided to update my answer with a few other options for serverless deployment
I know that this isn't exactly what you asked for but I use Serverless Framework (https://serverless.com) for deployment and I love it. I don't do my deployments when I push to my repo. Instead I push to my repo after I've deployed. I like this flow because a deployment can fail due to so many things and pushing to GitHub is much less likely to fail. I this way, I prevent pushing code that failed to deploy to my master branch.
I don't know if you're familiar with the framework but it is super simple. The website describes the simple steps to creating and deploy a function like this.
1 # Step 1. Install serverless globally
2 $ npm install serverless -g
3
4 # Step 2. Create a serverless function
5 $ serverless create --template hello-world
6
7 # Step 3. deploy to cloud provider
8 $ serverless deploy
9
10 # Your function is deployed!
11 $ http://xyz.amazonaws.com/hello-world
There are also a number of plugins you can use to integrate easily with custom domains on APIGateway, prune older versions of lambda functions that might be filling up your limits, etc...
Overall, I've found it to be the easiest way to manage and deploy my lambdas. Hope it helps!
Given that you're using AWS Lambda, you may want to consider CodePipeline to automate your release process. [SAM(https://docs.aws.amazon.com/lambda/latest/dg/serverless_app.html) may also be interesting.
I too had the same problem. I wanted to manage 12 lambdas with 1 git repository. I solved it by introducing travis-ci. travis-ci saved the time and really useful in many ways. We can check the logs whenever we want and you can share the logs to anyone by sharing the URL. The sample documentation of all steps can be found here. You can go through it. 👍
I am using gitlab as repository and want to push my code on ec2 whenever any commit is done on gitlab. The gitlab CD/CI documentation states that I have to add a file .gitlab-ci.yml at the root directory of my repo. This is actually a problem for me because, I want project repo to have only code and not any configuration related info like build and deploy etc. Also when anybody clones the repo, they would have access to location where my code is pushed/deployed on ec2. Is there any work around for this problem ?
You'll need to use a gitlab-ci.yml filke to deploy your application. The file provides instructions and a pipeline "infrastructure" which, if properly configured, will build, test and automatically deploy your code.
If you are worried about leaking credentials, you should use the built-in instance variables to mask your important bits, like a "$SERVERNAME" or "$DB_PASSWORD" for instance.
Lastly, you can use the power of gitignore, in order to not publish all of your credentials or sensitive bits to your projects' servers or instances.
I have been following this guide:
https://deliciousbrains.com/scaling-laravel-using-aws-elastic-beanstalk-part-3-setting-elastic-beanstalk/
However I am stuck at this point.
Not in terms of something not working, but in how it should be done properly. Which app I should deploy?
Is is the development app that is tested and deployed? Do I create another instance in AWS that will be only used to deploy ready apps? What is the pattern to follow?
At the moment I have local development server which runs on my PC, and also 1 Development instance EC2 on AWS. Do I need more than that on top of Elastic beanstalk?
Please advice me! Thanks!
The following pattern is the one that best fits your need. You're not just looking for a pattern, but an architecture. I'll try to help you with the information you provided.
First it is important that you really understand what Beanstalk is and how it works. See: http://docs.aws.amazon.com/en/elasticbeanstalk/latest/dg/Welcome.html
Answering your question, applications are typically placed in the beanstalk for scalable production, but nothing prevents you from setting up development environments for testing, too.
You do not need to create an instance to deploy, you can deploy from your own local machine, using the console, cli, or api. Look:
Console: https://sa-east-1.console.aws.amazon.com/elasticbeanstalk/home
EB Cli: http://docs.aws.amazon.com/en/elasticbeanstalk/latest/dg/eb-cli3.html
API: http://docs.aws.amazon.com/en/elasticbeanstalk/latest/api/Welcome.html
Having said that, I will cite a very useful scenario in several cases:
You create a beanstalk application from the console or cli and configure the integration with AWS CodeCommit. CodeCommit will prevent you from having to send the whole project to each deploy.
You create an instance of amazon to perform the implantation. This instance has a git repository of your project, it gets committed to the beanstalk environment settings (environment variables for example), and deploy to beanstalk using CodeCommit.
This scenario is very useful for a team project for beanstalk because you can use the deployment instance to hide sensitive details and configure deploy patterns.
I have a github project that uses travis for continuous integration. I would like to deploy my project on amazon ec2. In order to simplify deployment, I would like the deployed system to have the same configuration as the test system. Is this possible?
AFAICT, this requires two things: First, an ec2 preconfigured instance that matches the settings used by travis. Does one exist? And second, a way to execute travis.yml scripts from the command line. How can I do that?
As for executing .travis.yml scripts from the command line, if I were you I would instead take it the other way around and replace your .travis.yml script with something like this:
language: bleh
etc etc...
install:
- ./travis-scripts/install.sh
before_script:
- ./travis-scripts/before_script.sh
script:
- ./travis-scripts/script.sh
Of course, you will still have to write a script for installing whatever language versions, Travis plugins etc you need on your Amazon EC2 instance.
As for an Amazon EC2 instance that matches Travis VMs, I don't know about that because I'm not so familiar with Amazon AWS, but I can tell you that Travis VMs are based on Ubuntu 12.04, and there is a lot more specific information in the page about The Build Environment.
So you want something on a EC2 instance that can read your .travis.yml file and configure it in the same way that travis does when it tests?
I think that's a pretty long shot for a relatively simple problem like this. Travis is an integration and testing platform that uses a lot of other systems (like chef and docker) to do what it does with the .yml files. To use this system to run a single app sounds a bit overkill.
I would recommend using chef (or similar like puppet) to configure your production environment and deploy your app.
You could have one chef recipe that configures the production environment (DB's, configuration files, install stuff, etc...) and another that deploys, configures and starts your app. When you want to make changes to the production environment, you make changes to these files. They can easily be bundled with the project.
Well, my head is spinning a bit here. I started with what i thought would be a simple task, to take regular db dumps on heroku and push them to a personal S3 account for backup.
I am not sure the best a approach to do this. Accessing S3 within Java is crystal clear, getting the db dump from heroku is clear as mud right now...
Disclaimer: i don't know Ruby, and i don't really want to learn Ruby if i don't have to, i really want to use Java (that is why i chose play) and i want to have it hosted, that is why i chose Heroku :-)
So, I could use the heroku Scheduler, but i am not understanding what scripts are being executed here - is it all scripts in /bin? What kind of scripts are these, are they ruby scripts? How do i add them as 'tasks' when they aren't rake tasks?
Can I use the pgbackups via URL somehow? It looks like the rake examples do pg_dump instead, write to a tmp file and then move it around from there. I'm pretty unclear how to access the heroku databased stuff from a script, the examples i have seen so far are in rake, so any insight there would be helpful...
Or coming at it from inside my java app, what is the status of the Heroku java API? If there is a way to get to the heroku runtime from my java, or somehow use the heroku.jar?
It would great to get some overall guidance and best practices in this area - thanks!!!
From the google group i found this tidbit:
http://groups.google.com/group/heroku/browse_thread/thread/7fe984c3d2d01f21/9474f31138636332?lnk=gst&q=scheduler+#9474f31138636332
"Sorry for the delayed response. We updated the docs to mention running Procfile entries via heroku run:
http://devcenter.heroku.com/articles/oneoff-admin-ps
Anything that works via heroku run works via Heroku Scheduler. Just put the name of the process type as the 'task" in Scheduler. No special syntax required. And you can even pass it arguments. "
From this and James Ward's last example above i am considering this answered.