What is the significance of the decision in Mesos for frameworks to be offered resources by Mesos? This seems to be mentioned a lot, but ultimately all of the logic is in the Mesos allocation module, so whether it's Mesos making and revoking offers, or frameworks asking for resources, is this just a semantic difference?
Interesting question:
The original Mesos paper states the following rationale:
The master implements fine-grained sharing across
frameworks using resource offers. Each resource offer
is a list of free resources on multiple slaves. The master
decides how many resources to offer to each framework
according to an organizational policy
Frameworks requesting would have the following consequences:
Frameworks would have to be aware of resources in the cluster (e.g., does the cluster have GPUs)
The logic of choosing the request (by a framework) which should be granted given fairness and existing free cluster resources seems more complex and less scalable than the current allocation mechanisms (Not having hard evidence here, but just a feeling after having touched the Mesos allocator code)
Maybe most interestingly the Mesos Scheduler interface includes a requestResources(const std::vector& requests) call. The default Mesos DRF allocator does not implement this call, but nothing prevents you from implementing an allocator which does so.
If you are interested in more details about cluster scheduler I can recommend this blog post or the Omega paper.
Update:
This MesosCon talk discusses some future extensions to more optimistic offers: http://schd.ws/hosted_files/mesosconna2016/51/MesosCon_2016_OptimisticOffer.pdf
Related
What is the difference between Apache Mesos and Nomad?
Nomad seems to claim that it can do resource management so I wonder how that is different from Apache Mesos?
Nomad also claims the following on their website
Nomad is architecturally much simpler. Nomad is a single binary, both for clients and servers, and requires no external services for coordination or storage. Nomad combines features of both resource managers and schedulers into a single system. This makes Nomad operationally simpler and enables more sophisticated optimizations.
Well for anyone to bring up a cluster resource management alone is not enough. so for that Nomad obviously recommends to buy into rest of the HashiCorp products so I am not sure how it is architecturally simpler when one have to integrate with pretty much all of their products that are needed for a fully functional cluster?
Mesos does not support federation or multiple failure isolation regions. Nomad supports multi-datacenter and multi-region configurations for failure isolation and scalability.
Not sure if this is still true for Apache Mesos?
Nomad is currently advertised as an orchestrator for orchestrators.
Nomad only aims to provide cluster management and scheduling and is designed with the Unix philosophy of having a small scope while composing with tools like Consul for service discovery and Vault for secret management.
On the other hand, Mesos is more a framework for building distributed systems than just the container orchestrator. Of course, you can use it that way but it's only a minority of its features and not take full use of its two-level scheduling design.
Nomad is architecturally much simpler. Nomad is a single binary, both for clients and servers, and requires no external services for coordination or storage. Nomad combines a lightweight resource manager and a sophisticated scheduler into a single system. By default, Nomad is distributed, highly available, and operationally simple.
Mesos architecture is not that simple. It's multiple binaries project. Definietly not easy to set up and run. Multiple moving parts are always more complicated to setup than a monolith but enables customization.
Mesos does not support federation or multiple failure isolation regions. Nomad supports multi-datacenter and multi-region configurations for failure isolation and scalability.
That's true. Ther are some works to bring federations to Mesos but it's not done yet. https://youtu.be/kqyVQzwwD5E
Mesos and Nomad are created for sligthly different purpose. Although both of them are n-th level orchestrator and could be run one on another and probably could deliver similar features. Nomad is designed just to run simple stateless applications while Mesos alow pluging custom schedulers and make fine grained control of what/when/where is deployed.
Please explain me how containers are sharing hardware node resources between each other?
I feel my node is lack of CPU resources.
Even though I set maximum for cloudlets limit.
I have no real load on my node, but last month it very often stops responding for short periods and then restores with no actual reasons in logs.
I also feel my provider has poor experience with Jelastic administration.
Instead of looking for real reason why hardware node is overloaded, they just turn it off for a while and then turn it on.
See my screenshots. Zero CPU usage Zero CPU usage
Is it possible for hosting provider to "oversell" hardware node?
I am looking for support from Jelastic team here.
I know that you requested a reply from Jelastic in particular, but I suppose that it might help to get some insight from a hosting provider as well.
Is it possible for hosting provider to "oversell" hardware node?
The Jelastic platform itself does not have any limitations on this. The platform ensures that containers are distributed to the least loaded hardware nodes, but obviously if a hosting provider does not supply sufficient infrastructure / keep adding more, that distribution is worthless (i.e. all hardware is overloaded).
I feel my node is lack of CPU resources.
From those graphs it looks like you're hitting approx. 2GHz CPU, which for a LAMP application (right?) seems to be quite high. Are you sure that your bottleneck is CPU? If yes, how did you reach that conclusion / test that assertion?
I also feel my provider has poor experience with Jelastic administration.
Most of all, if you feel that your current Jelastic provider is not servicing your needs, did you consider moving to another one? The Jelastic ecosystem has over 30 different hosting providers. You can move your environment to another provider easily with the Export/Import feature and the ratings on the Jelastic Cloud Union site can help you to identify a good quality one in your preferred location.
I am checking out Apache Aurora with the scope of running scientific workflows (assuming a set of python scripts in a particular sequence). I've successfully managed to run a few of these aurora Jobs, and it looks great for my particular use-case.
I was wondering if there is a way to specify that a particular task (or job, in general) requires a number of GPU resources from my Apache Mesos cluster Of course Mesos needs to be aware of the GPU resources first, and it seems this is possible by defining these GPU resources as indicated here.
So the question is whether there is a way to communicate with Mesos via Aurora to accept offers with GPU resources available. As far as I can tell, the Resource object in Aurora is limited to CPU/Ram/Disk resources. Any hints are greatly appreciated.
Thanks!
I'm not familiar with Apache Aurora, but Mesosphere Marathon (a framework similar to Aurora in functionality) is limited to cpu, mem, and disk resources as well.
If you would like to use custom resources, you would probably need to write your own framework. Depending on your needs it may not be that difficult. For inspiration, check the RENDLER framework.
As mentioned in the thread you are referencing to, Mesos do not provide isolation for GPU (actually, for any custom) resources. Keep this is in mind when doing resource math.
When checking the Aurora tutorial I assume you can just specify this ressource as part of you job description:
resources = Resources(cpu = 2, ram = 4*GB, disk = 8*GB, gpu = 1),
Just keep in mind that this is in artificial resource for Mesos, so Mesos will not take care of resource isolation in this case. For example if you have several GPUs on one system, your code would have to manage the isolation/scheduling between the different GPUs.
Both Marathon and Aurora are built on Mesos and supposedly are engineered for running long running services. My questions are:
What are their differences? I have struggled in finding any good explanations regarding their key differences
Do these frameworks run anything that runs on Linux? For Marathon they state that it can run anything that "is executable in a shell" but this is sort of vague :)
Thanks!
Disclaimer: I am the VP of Apache Aurora, and have been the tech lead of the Aurora team at Twitter for ~5 years. My likely-biased opinions are my own and do not necessarily represent those of Twitter or the ASF.
Do these frameworks run anything that runs on Linux? For Marathon they
state that it can run anything that "is executable in a shell" but
this is sort of vague :)
Essentially, yes. Ultimately these systems are sophisticated machinery to execute shell code somewhere in a cluster :-)
What are their differences? I have struggled in finding any good
explanations regarding their key differences
Aurora and Marathon do indeed offer similar feature sets, both being classified as "service schedulers". In other words, you hand us instructions for how to run your application servers, and we do our best to keep them up.
I'll offer some differences in broad strokes. When it comes to shortcomings mentioned in each, I think it's safe to say that the communities are aware and intend to fix them.
Ease of use
Aurora is not easy to install. It will likely feel like you are trailblazing while setting it up. It exposes a thrift API, which means you'll need a thrift client to interact with it programmatically (a REST-like API is coming, but is vaporware at the moment), or use our command line client. Aurora has a DSL for configuration which can be daunting, but allows you to easily share templates and common patterns as you use the system more.
Marathon, on the other hand, helps you to run 'Hello World' as quickly as possible. It has great docs to do this in many environments and there's little overhead to get going. It has a REST API, making it easier to adapt to custom tools. It uses JSON for configuration, which is easy to start with but more prone to cargo culting.
Targeted use cases
Aurora has always been designed to handle a large engineering organization. The clusters at Twitter have tens of thousands of machines and hundreds of engineers using them. It is critical to Twitter's business. As a result, we take our requirements of scale, stability, and security very seriously. We make sure to only condone features that we believe are trustworthy at scale in production (for example, we have our Docker support labeled as beta because of known issues with Docker itself and the Mesos-Docker integration). We also have features like preemption that make our clusters suitable for mixing business-critical services with prototypes and experiments.
I can't make any claim for or against Marathon's scalability. On the feature front, Marathon has build out features quickly, but this can feel bleeding edge in practice (Docker support is a good example). This is not always due to Marathon itself, but also layers down the stack. Marathon does not provide preemption.
Ownership
To some, ownership and governance of a project is important. It feel that in practice it does not define the openness of a project, but for some people/companies the legal fine print can be a deal-breaker.
Marathon is owned by a company (Mesosphere)
To some, this is beneficial, to others is is not. It means that you can pay for support and features. It also means that there is something to be sold, and the project direction is ultimately decided by Mesosphere's interests.
Aurora is owned by the Apache Software Foundation
This means it is subject to the governance model of the ASF, driven by the community. Aurora does not have paying customers, and there is not currently a software shop that you can pay for development.
tl;dr If you are just getting your feet wet with running services on Mesos, I would suggest Marathon as your first port of call. It will be easier for you to get running and poke around the ecosystem. If you are forming the 'private cloud strategy' for a company, I suggest seriously considering Aurora, as it is proven and specifically designed for that.
So I've been evaluating both and this is my summary.
Aurora
[+] also handles recurring jobs
[+] finer grained, extensive file-based configuration
[+] has namespaces so multiple environments can co-exist
[-] read-only UI, no official API
[~] file based configuration and cli based execution brings overhead (which can be justified with more extensive feature set)
Marathon
[+] very easy to setup and use
[+] UI that provides control and extensive API (even with features missing from UI at the moment)
[+] event bus to listen in on api calls
[-] handles only long-running jobs
[-] does not have separate deployment-run-cleanup steps, these if necessary need to be combined in a script of one-liner
Even though Aurora has better capabilities, I prefer Marathon due to Auroras complexity/overhead and lack of UI (for control) & API
I have more experience with Marathon.
Ideological:
Marathon is a relatively tested product that is used in production at AirBnB. Aurora is an early Apache project (so YMMV).
Both are open source and active. Feel free to contribute pull requests or file issues!
Technical:
Marathon doesn't schedule batch tasks or cron jobs
Marathon has a friendly UI and better health indicators (in 0.8.x)
In regards to your second question, you can run any command or docker container, and Mesos will do the resource isolation for you. If you have 50% CentOS nodes and 50% Ubuntu nodes and you run a task that executes apt-get, the task will have a 50% chance of failure. Mesos and Marathon have no awareness of the actual machines.
Disclaimer: I don't have hands-on experience with Aurora, only with Marathon.
ad Q1: In a nutshell Apache Aurora is capable of doing what Marathon + Chronos can provide, that is, schedule both long-running services and recurring (batch) jobs; see also Aurora user guide.
ad Q2: Yes, anything. Currently based on cgroups and Docker but hey, you can roll your own.
We are implementing a system in the company I work for where by we will need to install the system in various sites of the same client (warehouses). The users in all sites should see the same information. The system should be able to work in each site when the network is down. What design architecture solution would be most suitable?
I suggest you consider CouchDB. Its robust replication feature is designed specifically for this sort of use case. It supports both continuous replication, which could keep the data in the various warehouses in sync in near-real-time during normal operation, and occasional replication, which could be used to sync data after a network outage.
There's a really good free O'Reilly book: CouchDB: The Definitive Guide, which has a chapter on replication.