THE MISSION:
I have a development environment running on an Amazon AWS EC2 virtual server which i want to have tested by third parties.
THE PROBLEM:
I do NOT trust the companies who will test it not to sabotage environment and / or steal code. Therefore, i don't want them to know URL's, permanent IP's or even to access the web pages, which they could eventually use a crawler to find.
My environment includes web applications and socket servers. I do NOT want to expose the web applications, while giving access only to socket servers.
THE CONCEPT:
I have opted to use a secondary, impermanent Elastic IP pointing to the environment. this IP will be destroyed after 1 or 2 days, after basic tests have run. Subject to change (depending on suggestions from this thread).
THE QUESTION:
Can i create a secondary Elastic IP instance that allows access only to ports 5000-5100? If so, how?
THE ALTERNATIVE: In case this is not the most efficient procedure, what alternative would you propose?
MY SOLUTIONS: followed FAQ Launching Instance From Backup
create snapshot
create image from snapshot (snapshot menu - create image tag)
instances - launch instance
choose image created from snapshot as your root volume
edit security groups (opened port range for sockets only, no web)
deleted all web code from this instance
after 2 days, will delete instance
followed Create Image From, Instance
select (exclusively) running instance you wish to mirror
right click on selected instance
choose create image from dropdown
to 7. same as above
this second solution seems to be more stable (especially re: status check and connectivity issues).
any better solutions? thanx!
Related
I have just entered into the world of e-commerce, and I am trying to get my Magento website up and running.
I am using AWS cloud for hosting my website. I am trying to use an architecture, where I can run multiple servers connected to a single DB server. Specifically, I want to use an AWS Auto scaling group along with ELB to start multiple EC2 instances, during high load. There is only one Mutli AZ RDS Database instance.
As initial trials, I tried creating 2 ec2 instances, and installed magento on both of them. I have used same RDS DB for both of them. But as is turns out, magento stores the base url of webserver in the database itself. Which means, I can only store one base url of magento website running one particular server.
To be precise magento stores base url in table core_config_data in column 'path' where row values ares "web/unsecure/base_url" and "web/secure/base_url", and the column 'value' for corresponding row specifies the url address of magento installed web server.
My question is how can I use multiple server using EC2 auto scaling, if magento permits only one server address in the base url.
Here's a partial view of the table with 2 rows -
config_id scope scope_id path value
5 default 0 web/unsecure/base_url http://server1.com/magento/
6 default 0 web/secure/base_url http://server1.com/magento/
Are there any other known methods to somehow use horizontal scaling during heavy load conditions in Magento.
I don't think load balancing works like that.
You need a load balancer that receives the requested URL and then passes it off to one of the servers running Magento - so I think you would pass the same url to both servers anyway, no?. I do not know how to do it.
You are trying to set up a very complicated system.
You could look to override some functions if you want to have different values for secure and non-secure urls. Try reading this code to get you started:
//file app/code/core/Mage/Core/Model/Store.php
//class Mage_Core_Model_Store
//function getBaseUrl()
//function getDistroServerVars()
//file app/code/core/Mage/Core/Model/Url.php
//class Mage_Core_Model_Url
//function getBaseUrl()
//file app/code/core/Mage/Core/Controller/Request/Http.php
//class Mage_Core_Model_Http
//function - I don't know, any of them, none of them
and look at any files with the string 'substDistroServerVars' in them or isDirectAccessFrontendName might expose something. getDistroServerVars is discussed at the end of this great article by the almighty Alan Storm.
But I don't think that is the real answer - for the real answer skip to the links at the end of this tedious monologue.
If this is your first foray into Magento and you really think you are going to get the volume of traffic into your shop that requires load balancing over two servers then you can afford, *must afford**, third party hosting and get professionals with many many many man years of experience running Magento on heavy loads across multiple servers. You will also want to hand off (at least) the images to a CDN.
*I mean, if your shop has that high a volume then it has a high revenue and you should invest that revenue in keeping your shop running: professional hosting with 24/7 support. Otherwise downtime will be expensive and a long implementation will mean lost revenue.
*If you are just trying this out for fun and to learn something about setting up Magento on multiple servers then I recommend two things:
1) Practice getting Magento running on one server first - and optimsing for volume there (caching, compilers, DB tuning, log file analysis, flat tables, cron jobs, CDNs, possibly combined JS and CSS, web server tuning and getting the headers right, possibly a full page cache and a sprinkling of Redis) - because that isn't a trivial list on one server never mind two + DB server and ELB.
And 2) practice getting Apache or nginx to serve load balanced content with your ecommerce SSL certificate in place. Only then should you try to join the two systems. Be prepared to spend many months on this - including figuring out Seige, AB or jmeter for simulated load testing.
But if you really want to get the AWS ELB set up here are a few excellent resources to get you started - particularly the detailed tutorial by Adrian Duke (first link) - pay great attention to the details in the last section of that article subtitled 'Magento', that may be the answer to your question.
Getting and scaling Magento in the cloud by Adrian Duke
Using AWS Auto Scaling with an Elastic Load Balancer cluster on EC2 (actually a WordPress install, not Magento, but Mr Shroder knows his Magento)
Running Magento in an AWS Environment (All hail Alan Storm)
I've had a rather large amount of success modifying the Magneto to be a beanstalk package. The steps (loosely) were:
Install GIT locally
Install AWS Command line tools
Install AWS Beanstlalk Command Line
Build module to upload image to s3 everytime it's uploaded in magento
Utilize OnePica's Magneto Extension
Use Amazon's REDIS Cache for caching data
Use RDS for database
Use Route53 for routing &
Use cloudfront for image, js & CSS Distro
Couple of drawbacks to AWS
Customizing magneto to look for things is a pain in the ass. As we speak I'm trying to get it to keep sessions persistent between EC2 instances as the loadbalancer chops it up.
Everytime you need to modify Magento in any way it's a git commit, (then we test locally, via a seperate beanstalk instance) and then push to production.
Short of that it's been fairly stable. We're not hitting high numbers yet, though.
Normally you put a load balancer in front of the nodes to distribute the load and each node is configured to use the same base_url. MySQL replication can be used if you want multiple db servers but I have never found the need to do this. Not used amazon ec2 with magento but have similar setup in a dedicated server environment with two nodes, one db server, load balancer, and shared media.
Diagram here is useful, especially with the shared storage for media, your going to need to do something like this. http://www.severalnines.com/blog/how-cluster-magento-nginx-and-mysql-multiple-servers-high-availability
Also, I found amazon seems to provide Elastic Load Balancing which is what your after I think. http://aws.amazon.com/documentation/elasticloadbalancing/
I have an AWS account with 14 instances and using scalr. I added the Api reference details and it showed up, at that time instances were pretty low. As and when I keep adding new instances it accepted few and reject the rest. Now I have an instance newly made on AWS which is not getting loaded in scalr.
Any ideas?
Instances that you create using AWS will not show up in Scalr.
Instead, you create Farms (in Scalr) through the use of custom and/or pre-configured Scalr Roles. When you launch those farms/roles, it will launch the required instances in AWS. It's like a wrapper around AWS that provides extra features, but it will only ever know about instances that have been launched from a Scalr role.
It is possible to import an existing server into Scalr although it involves installing the scalarizr software onto that server and opening some ports. Full details can be found here. Once complete, you'll have a new role that you can add to a farm and then launch.
I've been watching some videos from the build conference re: Inside Windows Azure etc.
My take away from one of them was that unless I loaded in a preconfigured VHD into a virtual machine role, I would lose any system settings that I might have made should the instance be brought down or recycled.
So for instance, I have a single account with 2 Web Roles running multiple (small) websites. To make that happen I had to adjust the settings in the Hosts file. I know my websites will be carried over in the event of failure because they are defined in the ServiceConfiguration.csfg but will my hosts file settings also carry over to a fresh instance in the event of a failure?
i.e. how deep/comprehensive is my "template" with a web role?
The hosts file will be reconstructed on any full redeployment or reimage.
In general, you should avoid relying on changes to any file that is created by the operating system. If your application is migrated to another server it will be running on a new virtual machine with its own new copy of Windows, and so the changes will suddenly appear to have vanished.
The same will happen if you perform a deployment to the Azure "staging" environment and then perform a "swap VIP": the "staging" environment will not have the changes made to the operating system file.
Microsoft intentionally don't publish inner details of what Azure images look like as they will most likely change in future, but currently
drive C: holds the boot partition, logs, temporary data and is small
drive D: holds a Windows image
drive E: or F: holds your application
On a full deployment, or a re-image, you receive a new virtual machine so all three drives are re-created. On an upgrade, the virtual machine continues to run but the load balancer migrates traffic away while the new version of the application is deployed to drive F:. Drive E: is then removed.
So, answering your question directly, the "template" is for drive E: -- anything else is subject to change without your knowledge, and can't be relied on.
Azure provides Startup Scripts so that you can make configuration changes on instance startup. Often these are used to install additional OS components or make IIS-configuration changes (like disabling idle timeouts).
See http://blogs.msdn.com/b/lucascan/archive/2011/09/30/using-a-windows-azure-startup-script-to-prevent-your-site-from-being-shutdown.aspx for an example.
The existing answers are technically correct and answer the question, but hosting multiple web sites in a single web role doesn't require editing the hosts file at all. Just define multiple web sites (with different host headers) in your ServiceDefinition.csdef. See http://msdn.microsoft.com/en-us/library/gg433110.aspx
I'm connected to one of my WebRole instances. But what can I do now? I startet the Server Manager and navigated to Roles -> WebServer -> RD00123..., but what I see seems not to be the active instance. When I click on "Browse WebSite" I don't see the deployed site even I can't see or edit for instance the configured rewrite rules (defined in the web.config).
It's also funny that the inetpub is on systemdrive D (d:\inetpub\wwwroot) but the deployed (editable) files are on E (e:\approot). I'm wondering how this comes togehter.
Ok the things I want to know are:
How can I launch the unique instance version of the web role (I can't find an IP or port)?
How can I assume temporary debug changes in the web.config?
How can I restore the deployed version of the instance?
1) I don't believe you can. This is managed by the Azure Fabric controller. The most you can do is force an individual instance (the one you've RDP'd into) to reboot/restart.
2) Yes. But they won't be reverted unless the VM is recycled, so be prepared to back them out yourself. This is a good use of RDP
3) Not directly. Again, this is controlled by the Azure Fabric controller. But you could put the original bits you had back into the VM manually.
I'm not sure I understand the questions regarding server manager, but I can answer the question about where things live. The "base" Windows Azure VM image is the root drive where you see INetPub at. The application is deployed to a seperate VHD (thus its presence on another drive). If you request local storage, that will come from yet another virtual drive. This is just how the Azure Fabric manages the application deployments. Properly constructed, the roles of course won't care where things live.
Is it possible to clone a EC2 instance data and all?
You can make an AMI of an existing instance, and then launch other instances using that AMI.
The easier way is through the web management console:
go to the instance
select the instance and click on instance action
create image
Once you have an image you can launch another cloned instance, data and all. :)
There is no explicit Clone button. Basically what you do is create an image, or snapshot of an existing EC2 instance, and then spin up a new instance using that snapshot.
First create an image from an existing EC2 instance.
Check your snapshots list to see if the process is completed. This usually takes around 20 minutes depending on how large your instance drive is.
Then, you need to create a new instance and use that image as the AMI.
Nowadays it is even easier to clone the machine with EBS-backed instances released a while ago. This is how we do it in BitNami Cloud Hosting.
Basically you just take a snapshot of the instance which can be used
later to launch a new server. You can do it either using AWS console
(saving the EBS-backed instance as AWS AMI) or using the EC2 API
tools:
create a snapshot with ec2-create-snapshot
and then launch an instance from a snapshot
Cloning the instance is nothing else but creating the backup and then
launching a new server based on that. You can find bunch of articles
out there describing this problem, try to find the info about "how to
..." backup or resize the whole EC2 instance, for example this blog is
a really good place to start: alestic.com
To Answer your question: now AWS make cloning real easy see Launch instance from your Existing Instance
On the EC2 Instances page, select the instance you want to use
Choose Actions --> Image and Templates, and then Launch More Like This.
Review & Launch
This will take the existing instance as a Template for the new once.
or you can also take a snapshot of the existing volume and use the snapshot with the AMI (existing one) which you ping during your instance launch
You can use AWS API or console UI to create an AMI(Amazon Machine Image) of your running instance. You can specify to reboot the instance when create your AMI. Then you can use AWS API or console UI to launch more instances with the AMI you created.
You can do it very easily with a Cloud Management software -like enStratus, RightScale or Scalr (disclaimer: I work there). With the cloned farm you can:
Create a snapshot or a pre-made image to launch another day
Duplicate your configuration to test it before production