We are starting to use Terraform to build our aws ec2 infrastructure but would like to do this as securely as possible.
Ideally we would like to do is to create a key pair for each Windows ec2 instance dynamically and store the private key in Vault. This is possible, but I cannot think a way of implementing it without having the private key written to the tfstate file. Yes I know I can store the tfstate file in an encrypted s3 bucket but this does not seem like an optimal secure solution.
I am happy to write custom code if needs be to have the key pair generated via another mechanism and the name passed as a variable to Terraform but dont want to if there are other more robust and tested methods out there. I was hoping we could use Vault to do this but on researching it does not look possible.
Has anyone got any experience of doing something similar? Failing that, any suggestions?
The most secure option is to have an arbitrary keypair you destroy the private key for and user_data that joins the instances to a AWS Managed Microsoft AD domain controller. After that you can use conventional AD users, and groups to control access to the instances (but not group policy in any depth, regrettably). You'll need a domain member server to administrate AD at that level of detail.
If you really need to be able to use local admin on these Windows EC2 instances, then you'll need to create the keypair for decrypting the password once manually and then share it securely through a secret or password manager with other admins using something like Vault or 1Password.
I don't see any security advantage to creating a keypair per instance, just considerable added complexity. If you're concerned about exposure, change the administrator passwords after obtaining them and store those in your secret or password manager.
I still advise going with AD if you are using Windows. Windows with AD enables world-leading unified endpoint management and Microsoft has held the lead on that for a very long time.
we found that it is possible to take a snapshot of the root volume of an AMI with a Marketplace product code. From this snapshot it is easy to create a new AMI (this one has the product code), or mount it on a new instance and copy the content to another volume and create an AMI from it without the product code.
I like to be able to protect any installed proprietary software on the AMI against reverse engineering (reading) and prevent impersonating an instance-id with a product code from one that has been tampered with. We've read many different articles on the subject and have not found a way to prevent this without getting 'identity view' permission of the owner of the instance. Any suggestions are most appreciated.
Unfortunately, as a seller, you cannot prevent buyers in AWS Marketplace from accessing your AMI contents.
The AWS Marketplace policy requires the following:
AMIs must allow operating system (OS)-level administration capabilities to allow for compliance requirements, vulnerability updates, and log file access. Linux-based AMIs use SSH, and Windows-based AMIs use RDP.
https://docs.aws.amazon.com/marketplace/latest/userguide/product-and-ami-policies.html#accessibility
I cannot create a Data Collection in Watson Discovery as it tells me:
400 only 1 free environment is allowed per organization
which would suggest that I already have one setup, but I used the API to list environments and it shows the Watson News Environment and no other.
How do I clean up my system to start again?
I have tried deleting the service and starting again, but no joy.
The limit is at the Bluemix Organization level so it may be that there is an environment under an instance created by another user in your organization. If you are not the owner of the org, you may need to check with that person to see if there are other active environments.
I am using Enteprise Chef. There is just one validation key per organisation. That means that once I download it for my workstation, other devops people in team can't have it => they can't bootstrap nodes from workstation. If I automate process of bootstraping, for instance if I put the key on S3, then I also will have to think about keeping my workstation validation key and S3 key in sync (and all other devops people in team).
So question is:
What are the best practices for distributing this single validation key across nodes and workstations?
My best bet on this:
Use chef on your workstations to manage the distribution of the validation.pem file.
Another way is to set it on a shared place (cifs or nfs share) for all your team.
According to this blog post this will become unneeded with chef 12.2.
For the record, the validation key is only necessary for the node to register itself and create it's own client.pem file at first run, it should (must if you're security aware) be deleted from the node after this first run.
The chef_client cookbook can take care of cleaning the validator key and will help manage nodes configuration.
I have just entered into the world of e-commerce, and I am trying to get my Magento website up and running.
I am using AWS cloud for hosting my website. I am trying to use an architecture, where I can run multiple servers connected to a single DB server. Specifically, I want to use an AWS Auto scaling group along with ELB to start multiple EC2 instances, during high load. There is only one Mutli AZ RDS Database instance.
As initial trials, I tried creating 2 ec2 instances, and installed magento on both of them. I have used same RDS DB for both of them. But as is turns out, magento stores the base url of webserver in the database itself. Which means, I can only store one base url of magento website running one particular server.
To be precise magento stores base url in table core_config_data in column 'path' where row values ares "web/unsecure/base_url" and "web/secure/base_url", and the column 'value' for corresponding row specifies the url address of magento installed web server.
My question is how can I use multiple server using EC2 auto scaling, if magento permits only one server address in the base url.
Here's a partial view of the table with 2 rows -
config_id scope scope_id path value
5 default 0 web/unsecure/base_url http://server1.com/magento/
6 default 0 web/secure/base_url http://server1.com/magento/
Are there any other known methods to somehow use horizontal scaling during heavy load conditions in Magento.
I don't think load balancing works like that.
You need a load balancer that receives the requested URL and then passes it off to one of the servers running Magento - so I think you would pass the same url to both servers anyway, no?. I do not know how to do it.
You are trying to set up a very complicated system.
You could look to override some functions if you want to have different values for secure and non-secure urls. Try reading this code to get you started:
//file app/code/core/Mage/Core/Model/Store.php
//class Mage_Core_Model_Store
//function getBaseUrl()
//function getDistroServerVars()
//file app/code/core/Mage/Core/Model/Url.php
//class Mage_Core_Model_Url
//function getBaseUrl()
//file app/code/core/Mage/Core/Controller/Request/Http.php
//class Mage_Core_Model_Http
//function - I don't know, any of them, none of them
and look at any files with the string 'substDistroServerVars' in them or isDirectAccessFrontendName might expose something. getDistroServerVars is discussed at the end of this great article by the almighty Alan Storm.
But I don't think that is the real answer - for the real answer skip to the links at the end of this tedious monologue.
If this is your first foray into Magento and you really think you are going to get the volume of traffic into your shop that requires load balancing over two servers then you can afford, *must afford**, third party hosting and get professionals with many many many man years of experience running Magento on heavy loads across multiple servers. You will also want to hand off (at least) the images to a CDN.
*I mean, if your shop has that high a volume then it has a high revenue and you should invest that revenue in keeping your shop running: professional hosting with 24/7 support. Otherwise downtime will be expensive and a long implementation will mean lost revenue.
*If you are just trying this out for fun and to learn something about setting up Magento on multiple servers then I recommend two things:
1) Practice getting Magento running on one server first - and optimsing for volume there (caching, compilers, DB tuning, log file analysis, flat tables, cron jobs, CDNs, possibly combined JS and CSS, web server tuning and getting the headers right, possibly a full page cache and a sprinkling of Redis) - because that isn't a trivial list on one server never mind two + DB server and ELB.
And 2) practice getting Apache or nginx to serve load balanced content with your ecommerce SSL certificate in place. Only then should you try to join the two systems. Be prepared to spend many months on this - including figuring out Seige, AB or jmeter for simulated load testing.
But if you really want to get the AWS ELB set up here are a few excellent resources to get you started - particularly the detailed tutorial by Adrian Duke (first link) - pay great attention to the details in the last section of that article subtitled 'Magento', that may be the answer to your question.
Getting and scaling Magento in the cloud by Adrian Duke
Using AWS Auto Scaling with an Elastic Load Balancer cluster on EC2 (actually a WordPress install, not Magento, but Mr Shroder knows his Magento)
Running Magento in an AWS Environment (All hail Alan Storm)
I've had a rather large amount of success modifying the Magneto to be a beanstalk package. The steps (loosely) were:
Install GIT locally
Install AWS Command line tools
Install AWS Beanstlalk Command Line
Build module to upload image to s3 everytime it's uploaded in magento
Utilize OnePica's Magneto Extension
Use Amazon's REDIS Cache for caching data
Use RDS for database
Use Route53 for routing &
Use cloudfront for image, js & CSS Distro
Couple of drawbacks to AWS
Customizing magneto to look for things is a pain in the ass. As we speak I'm trying to get it to keep sessions persistent between EC2 instances as the loadbalancer chops it up.
Everytime you need to modify Magento in any way it's a git commit, (then we test locally, via a seperate beanstalk instance) and then push to production.
Short of that it's been fairly stable. We're not hitting high numbers yet, though.
Normally you put a load balancer in front of the nodes to distribute the load and each node is configured to use the same base_url. MySQL replication can be used if you want multiple db servers but I have never found the need to do this. Not used amazon ec2 with magento but have similar setup in a dedicated server environment with two nodes, one db server, load balancer, and shared media.
Diagram here is useful, especially with the shared storage for media, your going to need to do something like this. http://www.severalnines.com/blog/how-cluster-magento-nginx-and-mysql-multiple-servers-high-availability
Also, I found amazon seems to provide Elastic Load Balancing which is what your after I think. http://aws.amazon.com/documentation/elasticloadbalancing/