I am trying to build a cloud VM brokering service which can borrow computer power as VM's on-demand, from the private/public cloud computer infrastructure. I have following goals for my service.
Abstract out vendor specific API in to a library which will give flexibility to choose any of the vendors (eg. EC2, rackspace) VM's with out affecting my service built on top of the library.
Also I should have flexibility to borrow VM's from a pure private cloud infrastructure built using stacks like OpenStack/Eucalyptus. Due to huge upfront Capex we will be using public clouds but we plan to move to private cloud infrastructure. So from design perspective we want to hide those details transparent to brokering service.
My question is if there are any open-source/commercial libraries or cloud development platforms, which can give me this functionality over which I can just build my service without really bothering about vendor specific details.
I came across rightscale & scalr but I am not clear if they are tools or platform. I need a platform over which I can develop not just to tools to monitor and auto provision cloud deployments.
TIA.
For python there's boto and libcloud.
For Java there's jclouds and also a port of libcloud (scroll a bit further down the page).
These are all open source libraries.
Yes, there is! It's a ruby library called fog. It's the only library I have found which gives you a vendor agnostic interface to various cloud providers.
For an Openstack cloud (RackSpace and may be some other in future) you should consider using the following python libraries:
novaclient - client library for OpenStack Compute API
nova-adminclient - client for administering Openstack Nova
You will be able to write recipes to provision control and play with your VMs in an Openstack cloud.
Hope it helps. Let me know if you need any more help in this regard.
Related
I am currently working heavily in Azure. I am actually quite fond of ARM (Azure Resource Manager) right now and would love to keep using it. Right now in the old portal, We have a lot of resources tied up as Cloud Services. Now, I know cloud services are available in the new portal, but it seems that Microsoft is moving away from the classic cloud service model. Can someone explain if this is true? If so, what will the new model look like? I already use resources groups to manage Websites (WebApps), so I assume this is where the azure future lies. Will we see the "deprecation" of cloud services on down the line?
I am trying to understand if I need to begin re-structuring my Azure Infrastructure.
Any insight, explanation, or documentation is greatly appreciated.
So there are two things here - Cloud Services and managemenet of Cloud Services.
When you manage Cloud Services in current portal the underlying mechanism used is Azure Service Management (ASM) where as it is Azure Resource Manager (ARM) in the preview portal. To me, ARM is the new way of managing your Cloud resources in Azure (including Cloud Services).
I don't work for Microsoft so I would not know if Cloud Services themselves will be deprecated down the road or not but one thing I think will happen is that ASM will be deprecated in favor of ARM. At some point of time, the only option you will be left with managing your cloud resources will be through Azure Resource Manager. One example that makes me believe this thing is the presence of Classic resource providers (e.g. Classic Storage Resource Provider which enables you to manage storage accounts created in current portal via ASM in the preview portal which works exclusively on ARM).
Personally I can't see a place for cloud services in the new ARM world of Azure. I have always found them a convoluted concept that simply added complexity to a deployment.
In the ARM view of deployments servers are collected together in a VNet, and each server is attached to a Nic which in turn can be connected to the internet. A security group then takes care of ingress / egress rules.
This is a much cleaner deployment method, as it puts connectivity configuration at the server layer instead of mapping them all through a higher layer of abstraction.
I don't see the place of cloud services in ARM, however after a quick search it seems that there is a plan to implement it
Still no direction from the Azure Advisers group other than officially they will not drop support for Cloud Services. I think they are nearing giving us some kind of direction but I can't say anymore than that.
I asked a question about the future of Cloud Services on the recent Azure Compute AMA.
You can read the answers directly on Reddit for all details, below are a few interesting quotes (emphasis mine).
On ARM Integration for Cloud Services:
We are looking at ways to make the transition to ARM easier for Cloud Service customers- one of those options includes CS integration in ARM. This investigation is in the very early stages though, so if you are looking for a solution soon, check out VMSS/ACS/SF/Web Apps (meagan-msft)
And:
I think it's safe to say that if we make any significant investment in CS in the near future, it would be ARM integration, and as Meagan suggests, that's still in planning. Beyond that, there are no major feature improvements on the horizon. We believe the platform is pretty mature at this point. (seanmichaelmckenna)
So it doesn't look like any major innovations will hit Cloud Services soon, however:
Cloud Services are not going anywhere. In fact, many Microsoft services run on Cloud Services, so we heavily rely on them as well. They are fully supported, so feel free to continue to use them.
(meagan-msft)
For those who want to switch to a different Compute service, these recommendations were made:
However, if you would like to check out other services that are integrated with ARM today, we recommend checking out the following:
Web Apps for customers who want a fully managed platform and are building traditional web applications
Service Fabric for customers who want an opinionated application platform and managed infrastructure, but still need some control over the IAAS layer
VM Scale Sets for customers who need IaaS-level control with easy scaling, autoscale and load balancer integration
Azure Container service was also listed as a potential alternative.
Some things to consider (my understanding):
Service Fabric currently (2017) requires at least 5 VM instances, except for dev/test purposes. So probably only an option for larger services
VM Scale Sets is an IaaS offering, i.e. you have to manage OS updates etc. yourself. However, support for automatic OS updates is being worked on.
I have part of my team using komodo, and so I have looked into Stackato. At first Stackato looked nice, but I can not figure out how to use it. It is available for download as a VM image. You can also deploy it to an existing infrastructure. But what is the point of this. Most cloud platforms I've used(Bluemix, PWS, Openshift, etc...) are in a cloud instance, but Stackato does not seem to do this. Do I have to use Stackato with some infrastructure, or can I get a in-cloud version?
If by "in a cloud instance" you mean offered as a hosted service (like Heroku, Engine Yard, and Google App Engine) then you could try the Stackato Sandbox which is a hosted trial environment.
That said, the real benefit of Stackato is that it's PaaS software that you can run yourself (i.e. "Private PaaS"), so to really get a feel for it from an Admin/Provider's perspective, you should deploy a small cluster in a public cloud like HP Helion Public Cloud or Amazon EC2.
Yes, you can run it on your own servers, but I think the cloud hosted option what you're looking for. You can run a cluster of around 5 Stackato nodes with a free license from ActiveState, but the cloud hosting costs still apply.
My team is developing an application which will enable end users to easily create, configure and destroy amazon instances without having to use Amazon SDKs themselves. The process at our end comprises of 3 steps.
1. Create / Destroy VMs in the amazon cloud using Amazon SDK (Done)
2. Configure/Install new software in the newly created instance.
3. Track usage/command and control.
We are currently in the second step. I just realized that Amazon SDK does not provide APIs for installing new software in the remote machine. I am not talking about AmazonCloudFormation APIs because those APIs are used to create and manage AWS resources rather a software like, say, a browser.
Has anyone installed new software in an amazon instance? If yes, did you use one of a)Amazon SDK, b) Any third party APIs and c) custom solution?
Also, is it even possible to install new software in an amazon instance through java code?
The Amazon API primarily controls infrastructure. It does not have any control as to what happens inside the instance.
There are a couple of ways you can bootstrap your instance and install software. You can use user-data to pass a script that will run on first launch. You could use a provisioning system like chef or puppet. You could roll your own if it works better for you.
What you are describing sounds a lot like a Platform-as-a-Service (PaaS).
A PaaS would allow you to submit an application to the PaaS and let it start the machines and set up your software on them. A PaaS would also give you additional features like monitoring, cross-cloud support and updating the application on the fly.
There are a several PaaS vendors mentioned here: Looking for Paas Recommendations
Disclaimer: I work for Cloudify, an open-source PaaS.
I am planning to migrate few products on Cloud which will be used as a platform for the developer community. In short I am trying to host PaaS vendor for my products which can be consumed by developers for build and development process.
The plan is as below:
I am trying to use Amazon IaaS ( S3, EC2) as the hardware.
I will require a cloud management software which can be installed somewhere on one of my local systems and can manage the Amazon cloud.
I will deploy all my products on the Amazon Cloud with the help of the Cloud Management Software.
I will develop and provide APIs to my end users(developer community) to use my service as a PaaS.
What I am trying to achieve is as follows:
Vendor independence in terms of IaaS. Lets say tomorrow I move to another IaaS provider.
Customer support for the cloud management software.
Ease of setup and use for the cloud management software.
Evaluation so far:
I tried looking at Eucalyptus and it sounds promising, but I am still not able to find out if this will be supporting the public cloud setup as my requirement is. I believe this is more like a private cloud setup.
If anyone can help me compare the other available options, that would help me solving my issue. For e.g. RightScale, OpenStack, CloudStack, Nimbula etc.
There are several PaaS providers out there. There is a comparison here: Looking for PaaS providers recommendations
Disclaimer: I work for GigaSpaces, developing the Cloudify open-source PaaS stack.
Cloudify answers most of your requirements, especially vendor independence - it supports a large number of IaaS providers, including: EC2, HP, Rackspace, Azure and others.
Cloudify does require its management server to run in the same cloud as the applications it runs so it can collect monitoring information using private communications rather then over the internet. Why do you want to run your management server on-premise?
Let's assume that I have a limited budget to make a small in-house private cloud. Now I want to be able to combine this private cloud with one of the public clouds (e.g. Amazon EC2). what options do I have?
More specifically I want to be able to do the following:
Use my private cloud primarily but if the request rate or size of datasets increased, transfer part of the load/data to EC2
Store my confidential data on the private cloud and move the more general data to EC2. Upon receiving a request, I want to be able to do some computation on the public data and then combine that with some computation on confidential data. But the confidentiality of the data must not be compromised.
I am looking into this for a project and would appreciate any idea/suggestions or related material.
It's a difficult question because the variety and amount of paths you can take in order to do this is great.
Anyway, what you're describing it's an hybrid cloud:
First you have to build your private cloud, there are plenty of options to do this, you have CloudStack, OpenStack, Eucalyptus, Open Nebula, etc. If you choose the open source way (I recommend it) you can see this analysis of the different open source cloud computing solutions:
http://bit.ly/QeGpqK
When you have your own infrastructure managed by your private cloud, you need a third party provider like Amazon for deploy the "public side" of your infrastructure.
And now comes the hard part:
You have tu build your own logic in order to scale your infrastructure to your "public side", and this will be actually the 80% of the work, you have to plan when, what and where you will scale, you have to identify what data you want to store in the public side, etc...
Tools like Rackspace can help you a little bit:
http://www.rackspace.com/cloud/hybrid/
#arcade_fire provides an excellent overview of competing private cloud offerings that are OpenSource. To this list, you could add Microsoft's SCVMM and VMWare's vCloud. Depending on budget and workload, these proprietary offerings may also be of interest.
After choosing a cloud O/S, consider the following problems:
How do you make the public and private elements of the hybrid cloud transparent to your network?
How do you control the resulting Hybrid cloud?
The first issue is addressed by using what is referred to as VPC (virtual private cloud). This term was introduced by AWS to describe a public cloud deployment that sits on a private network. These machines are joined to your private cloud via a VPN. One end of the VPN is in the cloud, and the other in your data center. A google of "aws vpc architecture" will net you a good explanation. I liked EC2 to VPC: A transition worth doing
The second issue involves you choosing admin tools to manage your work load that support the API used to communicate to each of your public and private accounts. The architype example is RightScale, which supports a number of APIs, but there are others. #arcade_fire provides a link to Rackspace. Alternatively, you can find consultancies that can tailor a tool depending on the intended workload e.g. ShapeBlue (CloudStack ecosystem).
If you are planning to have your own hybrid cloud you have to have you own code that look after the scaleUP and other provision task .
for private cloud you can go for eucalyptus or OpenQRM . if you are comfortable with Linux you can use Open source KVM that comes bundled with Linux OS hosted on apache server , you can call its exposed methods from your java or PHP code and carry out the provision and de-provision task . or otherwise you can simply use the management console provided by KVM.
citrix -Xen is also one of the option.
Now for public cloud EC2 is an option other than that you can use various other Iaas.
for high availability you can Open-source apache HAProxy that take care of your load balancing.
as you are dealing with some data you have the options to go for BIG data providers that includes , mapreduse ,Terradata , IBM-netzza ,cloudere for graph and other analysis you can use splunk and as apache hadoop with pig and hive is always an option .
the scaleUP code you have to write along with the integration of private and public cloud. Amazon exposes its web service you can leverage it ....
there are professional vendors who offer this combined service of both a private (primarily) and a public cloud. it's called a hybrid cloud, you build your own private one to serve your project, and you've got some drawn elements from public clouds to serve you even better, in cases of increased data base while your own cloud is limited. i personally like stratoscale, their symphony product is good and serves all of a customer's needs (from my experience), but obviously you've got many out there (they work with openstack as well.)
The thing you're describing is undoubtedly a Hybrid cloud deployment model. Out of my experience with our team I can recommend you go ahead give a third party tool a chance. Third party services nowadays allow you to have all the relevant freedom of action within the cloud environment, which basically means you have the complete control over your cloud resources. Those services let you manage your on-premise private cloud along with using AWS as an-on premise, which is a very advanced function. You might want to check it out as well as for the costs, most of these tools offer a free trial.