Is there a way to make custom resource offers from a Mesos slave? Currently, the resource offers from the slave contain "cpus", "mem", "ports". I want to add custom resource capabilities like upload bandwidth limit, download bandwidth limit etc. There is an option of doing this via the --resources parameter while starting the slave. But what I am looking for is a way to achieve this via code, may be a pluggable module to Mesos, so that the person who is launching the slaves doesn't have to bother about specifying the custom resources.
Is this possible?
Thanks.
Resources reported by the Mesos agent (aka slave) should be specified by the --resources flag. If the flag is omitted, defaults are used. Note that custom resources advertised via --resources will not be isolated, i.e. Mesos agent will not ensure a task is using not more than allocated amount of such resource.
Why do you think a module is a nicer solution than a flag? Loading custom modules requires setting some command line flags as well.
Related
When I was trying to limit memory on mesos agent with --resource flag, it seems doesn't work at all. The slave can still use all the cpus and memory on virtual machine. Did I use it wrong?
By default Mesos Conterneizer uses PSOIX isolators (posix/cpu,posix/mem) since they are not a real isolator it's recomended to use cgroups isolators instead (cgroups/cpu,cgroups/mem).
--isolation=VALUE Isolation mechanisms to use, e.g., posix/cpu,posix/mem (or windows/cpu,windows/mem if you are on Windows), or cgroups/cpu,cgroups/mem, or network/port_mapping (configure with flag: --with-network-isolator to enable), or gpu/nvidia for nvidia specific gpu isolation, or load an alternate isolator module using the --modules flag. If cgroups/all is specified, any other cgroups related isolation options (e.g., cgroups/cpu) will be ignored, and all the local enabled cgroups subsystems on the agent host will be automatically loaded by the cgroups isolator. Note that this flag is only relevant for the Mesos Containerizer. (default: windows/cpu,windows/mem on Windows; posix/cpu,posix/mem on other platforms)
I'm trying to setup DC/OS on AWS with different resource roles (I don't want nodes reserved for Apache Cassandra used by other things). I know in theory how to do it in a plain Mesos that I set up manually (https://support.mesosphere.com/hc/en-us/articles/206474745-How-to-reserve-resources-for-certain-frameworks-in-Mesos-cluster-), but I don't know how to do it in a DC/OS cloud installation. Any advice is greatly appreciated.
DC/OS neither facilitates, nor prohibits services (aka frameworks) to register with a custom role. I'm not sure how to tell Cassandra to register under special role, Jenkins for example exposes it in its configurations (see screenshot below). In short, check with your framework, how to register in a custom role, DC/OS does not restrict this.
If you want to configure role-specific things, like weights or quota, just talk directly to Mesos, which is accessible via /mesos, DC/OS does not prohibit it either.
I'm working on configuring our CDH cluster for YARN (currently we use MR1). I've been trying to wrap my head around all the yarn memory parameters, to make sure we're fully utilizing our machines and ended up diagramming it:
We use Cloudera manager, so I've included relevant CDM settings and processes, which won't apply to every cluster.
My question: am I missing any configurable memory-allocation parameters, or am I interpreting any incorrectly? I've had trouble parsing the documentation, and want to avoid any trial-and-error possible.
I'm trying to get set up on the Amazon Cloud to run some hadoop MapReduce jobs but I'm struggling to successfully create a cluster. I have downloaded the ec2 files, have my certificates and keypair file, but I believe it's the AMIs that are causing me trouble. If I'm trying to run a cluster with a master node and n slave nodes, I start n+1 instances using standard compatible AMIs and then run the code "hadoop-ec2 launch-cluster name n" in the terminal. The master node is successful, but I get an error when the slave nodes start to launch, saying "missing parameter -h (AMI missing)" and I'm not entirely sure how to progress.
Also, some of my jobs will require an alteration in hadoops parameter settings (specifically the mapred-site.xml config file), is it possible to alter this file, and if so, how do I gain access to it? Is hadoop already installed on amazon machines, with this file accessible and alterable?
Thanks
Have you tried Amazon Elastic MapReduce? This is a simple API that brings up Hadoop clusters of a specified size on demand.
That's easier then to create own cluster manually.
But once the jobflow is finished by default it shuts the cluster down, leaving you with outputs on S3. If what you need is simply to do some crunching, this may be the way to go.
In case you need HDFS contents stored permanently (e.g. if you are running HBase on top of Hadoop) you may actually need own cluster on EC2. In this case you may find Cloudera's distribution of Hadoop for Amazon EC2 useful.
Altering Hadoop configuration on nodes it will start is possible using EC2 Bootstrap Actions:
Q: How do I configure Hadoop settings for my job flow?
The Elastic MapReduce default Hadoop configuration is appropriate for most workloads. However, based on your job flow’s specific memory and processing requirements, it may be appropriate to tune these settings. For example, if your job flow tasks are memory-intensive, you may choose to use fewer tasks per core and reduce your job tracker heap size. For this situation, a pre-defined Bootstrap Action is available to configure your job flow on startup. See the Configure Memory Intensive Bootstrap Action in the Developer’s Guide for configuration details and usage instructions. An additional predefined bootstrap action is available that allows you to customize your cluster settings to any value of your choice. See the Configure Hadoop Bootstrap Action in the Developer’s Guide for usage instructions.
About the way you are starting the cluster, please clarify:
If I'm trying to run a cluster with a master node and n slave nodes, I start n+1 instances using standard compatible AMIs and then run the code "hadoop-ec2 launch-cluster name n" in the terminal. The master node is successful, but I get an error when the slave nodes start to launch, saying "missing parameter -h (AMI missing)" and I'm not entirely sure how to progress.
How exactly you are trying start it? What exactly AMIs are you using?
I am trying to understand how Amazon implements the auto scaling feature. I can understand how it is triggered but I don't know what exactly happens during the auto scaling. How does it expand. For instance,
If I set the triggering condition as cpu>90. Once the vm's cpu usage increases above 90:
Does it have a template image which will be copied to the new machine and started?
How long will it take to start servicing the new requests ?
Will the old vm have any downtime ?
I understand that it has the capability to provide load balancing between the VMs. But, I cannot find any links/paper which explains how Amazon auto scaling works. It will be great if you can provide me some information regarding the same. Thank you.
Essentially, in the set up you register an AMI, and a set of EC2 start parameters - a launch configuration (Instance size, userdata, security group, region, availability zone etc) You also set up scaling policies.
Your scaling trigger fires
Policies are examined to determine which launch configuration pplies
ec2 run instance is called with the registered AMI and the launch configuration parameters.
At this point, an instance is started which is a combination of the AMI and the launch configuration. It registers itself with an IP address into the AWS environment.
As part of the initial startup (done by ec2config or ec2run - going from memory here) - the newly starting instance can connect to instance meta data and run the script stored in "userdata". This script can bootstrap software installation, operating system configuration, settings, anything really that you can do with a script.
Once it's completed, you've got a newly created instance.
Now - if this process was kicked off by auto-scale and elastic-load-balancing, at the point that the instance is "Windows is ready" (Check ec2config.log), the load balancer will add the instance to itself. Once it's responding to requests, it will be marked healthy, and the ELB will start routing traffic.
The gold standard is to have a generic AMI, and use your bootstrap script to install all the packages / msi's / gems or whatever you need onto the server. But what often happens is that people build a golden image, and register that AMI for scaling.
The downside to the latter method is that every release requires a new AMI to be created, and the launch configurations to be updated.
Hope that gives you a bit more info.
may be this can help you
http://www.cardinalpath.com/autoscaling-your-website-with-amazon-web-services-part-2/
http://www.cardinalpath.com/autoscaling-your-website-with-amazon-web-services-part-1/
this post helped me achiving this
Have a read of this chaps blog, it helped me when I doing some research on the subject.
http://www.codebelay.com/blog/2009/08/02/how-to-load-balance-and-auto-scale-with-amazons-ec2/