I read the "Cluster from scratch" document on this website : http://clusterlabs.org/doc/
ans I didn't find the answer to my questions :
1) I'm wondering if the Linux Cluster with Pacemaker + Corosync can be used with C++ programs. All the examples are for Apache servers and mostly webservices. Is it just possible ?
2) Is there any document/website that explains the possible links between the cluster status graph (online, active,...) and a potential C++ application graph (application running, stopped, ...).
Short version: Resource agents can be written in any language including C++
Long version:
A resource agent is the glue between pacemaker and your daemon.
Something that knows how to start, stop and health check your daemon but doesn't hang around afterwards.
Its not completely clear whether you want the agent to be in C++ or want to write an agent for a C++ daemon. I suspect you're asking about the first but really need the second. Best thing to do is say hello upstream (irc or public mailing list) so we can continue the discussion.
As I understand you need to create another type of resources, namely your own C++ application. If so then you will need to implement your own resource agent.
I would propose to look into a Dummy resource agent https://github.com/ClusterLabs/pacemaker/blob/master/extra/resources/Dummy and refactor it for your own needs. Read more about resource agents in https://github.com/ClusterLabs/resource-agents/blob/master/doc/dev-guides/ra-dev-guide.asc
1) It is possible. I did some tests using c++ simple tcp code.
Related
I'm trying to get to grips with service fabric and I'm struggling a little bit. Some questions:
are all service fabric service instances single-threaded? I created a stateless web api, one instance, with a method that did a Task.Delay, then returned a string. Two requests to this service were served one after the other, not concurrently. So am I right in thinking then that the number of concurrent requests that can be served is purely a function of the service instance count in the application manifest? Edit Thinking about this, it is probably to do with the set up of OWIN Wep Api. Could it be it is blocking by session? I assumed there is no session by default?
I have long-running operations that I need to perform in service fabric (that can take several hours). Is there a recommended pattern that I can use for this in service fabric? These are currently handled using a storage queue that triggers a webjob. Maybe something with Reliable Queues and a RunAsync loop?
It seems you handled the first part so I will comment on the second part: "long-running operations".
We can see long running operations / workflows being handled far before service fabric came about. For this reason, we can build on the shoulders of giants by looking on the design patterns that software experts have been using for decades. For example, the famous and all inclusive Process Manager. Mind you that this pattern is sometimes an overkill. If it is in your case, just check out the rest of the related patterns in the Enterprise Integration Patterns book (by Gregor Hohpe).
As for the use of reliable collections, those are implementation details when choosing a data structure supporting the chosen design pattern.
I hope that helps
With regards to your second point - It really depends on the nature of your long running task.
Is your long running task the kind of workload that runs on an isolated thread that depends on local OS/VM level resources and eventually comes back with a result (A)? or is it the kind of long running task that goes through stages and builds up a model of the result through a series of persisted state changes (B)?
From what I understand of Service Fabric, it isn't really designed for running long running workloads (A), but more for writing horizontally-scalable, highly-available systems.
If you were absolutely keen on using service fabric (and your kind of workload tends to be more like B than A) I would definitely find a way to break down those long running tasks that could be processed in parallel across the cluster. But even then, there is probably more appropriate technologies designed for this such as Azure Batch?
P.s. If you are going to put a long running process in the RunAsync method, you should design the workload so it is interruptable and its state can be persisted in a way that can be resumed from another node in the cluster
In a stateful service, only the primary replica has write access to
state and thus is generally when the service is performing actual
work. The RunAsync method in a stateful service is executed only when
the stateful service replica is primary. The RunAsync method is
cancelled when a primary replica's role changes away from primary, as
well as during the close and abort events.
P.s.s Long running operations are the devil when trying to write scalable systems. Try and tackle that now and save yourself the future pain if possibe.
To the first point - this is purely a client issue. Chrome saw my requests as indentical and so delayed the 2nd request until the 1st got a response. Varying the parameter of the requests allowed them to be served concurrently.
I went through the video on introduction of DCOS. It was good but got me somewhat confused in terms of classification of component definitions in Mesosphere.
I get that DCOS is an ecosystem and Mesos is like a kernel. Please correct me if I am wrong. For eg. It's like Ubuntu and Linux kernel I presume.
What is marathon? Is it a service or framework or is it something else that falls in neither category? I am bit confused in terms of service vs framework vs application vs Task definition in Mesosphere's context.
Are the services(Cassandra, HDFS, Kubernetes, etc..) that he launches in the video can safely be also called as frameworks?
From 3, are these "services" running as executors in the slaves?
What should rails-app's type be here? Is it a task? So will it also have an executor?
Who makes the decision of autoscaling the rails-app to more nodes, when he increases the traffic using marathon.
1) I get that DCOS is an ecosystem and Mesos is like a kernel. Please
correct me if I am wrong. For eg. It's like Ubuntu and Linux kernel I
presume.
Correct!
2) What is marathon? Is it a service or framework or is it something
else that falls in neither category? I am bit confused in terms of
service vs framework vs application vs Task definition in Mesosphere's
context.
In Apache Mesos terminology, Marathon is a framework. Every framework consists of a framework scheduler and an executor. Many frameworks reuse the standard executor rather than providing their own. An app is a Marathon specific term, meaning the long-running task you launch through it. A task is the unit of execution, running on a Mesos agent (in an executor). In DC/OS (the product, Mesosphere is our company) we call frameworks in general services. Also, in the context of DC/OS, Marathon plays a special role: it acts as a sort of distributed initd, launching other services such as Spark or Kafka.
3) Are the services(Cassandra, HDFS, Kubernetes, etc..) that he
launches in the video can safely be also called as frameworks?
See above.
4) From 3), are these "services" running as executors in the slaves?
No. See above.
5) What should rails-app's type be here? Is it a task? So will it also
have an executor?
The Rails app may have one or more (Mesos) tasks running in executors on one or more agents.
6) Who makes the decision of autoscaling the rails-app to more nodes,
when he increases the traffic using marathon.
Not nodes but instances of the app. Also as #air suggested, with Marathon autoscaling is simple, see also this autoscaling example.
I am trying to understand how the various components of Mesos work together, and found this excellent tutorial that contains the following architectural overview:
I have a few concerns about this that aren't made clear (either in the article or in the official Mesos docs):
Where are the Schedulers running? Are there "Scheduler nodes" where only the Schedulers should be running?
If I was writing my own Mesos framework, what Scheduler functionality would I need to implement? Is it just a binary yes/no or accept/reject for Offers sent by the Master? Any concrete examples?
If I was writing my own Mesos framework, what Executor functionality would I need to implement? Any concrete examples?
What's a concrete example of a Task that would be sent to an Executor?
Are Executors "pinned" (permanently installed on) Slaves, or do they float around in an "on demand" type fashion, being installed and executed dynamically/on-the-fly?
Great questions!
I believe it would be really helpful to have a look at a sample framework such as Rendler. This will probably answer most of your question and give you feeling for the framework internal.
Let me now try to answer the question which might be still be open after this.
Scheduler Location
Schedulers are not on on any special nodes, but keep in mind that schedulers can failover as well (as any part in a distributed system).
Scheduler functionality
Have a look at Rendler or at the framework development guide.
Executor functionality/Task
I believe Rendler is a good example to understand the Task/Executor relationship. Just start reading the README/description on the main github page.
Executor pinning
Executors are started on each node when the first Task requiring such executor is send to this node. After this it will remain on that node.
Hope this helped!
To add to js84's excellent response,
Scheduler Location: Many users like to launch the schedulers via another framework like Marathon to ensure that if the scheduler or its node dies, then it can be restarted elsewhere.
Scheduler functionality: After registering with Mesos, your scheduler will start getting resource offers in the resourceOffers() callback, in which your scheduler should launch (at least) one task on a subset (or all) of the resources being offered. You'll probably also want to implement the statusUpdate() callback to handle task completion/failure.
Note that you may not even need to implement your own scheduler if an existing framework like Marathon/Chronos/Aurora/Kubernetes could suffice.
Executor functionality: You usually don't need to create a custom executor if you just want to launch a linux process or docker container and know when it completes. You could just use the default mesos-executor (by specifying a CommandInfo directly in TaskInfo, instead of embedded inside an ExecutorInfo). If, however you want to build a custom executor, at minimum you need to implement launchTask(), and ideally also killTask().
Example Task: An example task could be a simple linux command like sleep 1000 or echo "Hello World", or a docker container (via ContainerInfo) like image : 'mysql'. Or, if you use a custom executor, then the executor defines what a task is and how to run it, so a task could instead be run as another thread in the executor's process, or just become an item in a queue in a single-threaded executor.
Executor pinning: The executor is distributed via CommandInfo URIs, just like any task binaries, so they do not need to be preinstalled on the nodes. Mesos will fetch and run it for you.
Schedulers: are some strategy to accept or reject the offer. Schedulers we can write our own or we can use some existing one like chronos. In scheduler we should evaluate the resources available and then either accept or reject.
Scheduler functionality: Example could be like suppose say u have a task which needs 8 cpus to run, but the offer from mesos may be 6 cpus which won't serve the need in this case u can reject.
Executor functionality : Executor handles state related information of your task. Set of APIs you need to implement like what is the status of assigned task in mesos slave. What is the num of cpus currently available in mesos slave where executor is running.
concrete example for executor : chronos
being installed and executed dynamically/on-the-fly : These are not possible, you need to pre configure the executors. However you can replicate the executors using autoscaling.
I am trying to figure out how to use Mesos.
I have running mesos master and slave running (in single-node setup).
And I have understood that framework listens for resource offers and accepts theme if he can, and after that it goes to the executor to execute the task.
How I can send to mesos "Hi, I want to execute some task with 1 cpu and 256 mb", who's task is? the framework? or there is another api for doing this?
Yosi
I think I understood it finally after a debugging session :)
The framework gets a resource offer, when it get a resource offer he checks whether he have a task to launch the matches the resource offer - If so, he runs the task.
I thought there was an external service that I need to call and it will initiate a resource offer.
I came across God which seems good but I am wondering if anyone knows of other process monitoring and control frameworks that I can compare god with.
God has the following features:
Config file is written in Ruby
Easily write your own custom conditions in Ruby
Supports both poll and event based conditions
Different poll conditions can have different intervals
Integrated notification system (write your own too!)
Easily control non-daemonizing scripts
The last one is what i am having difficulty with.
Have a look at Ubic (CPAN page here but do read installation details on the github project page).
Ubic isn't a monitoring framework par se but a LSB compliant extensible Service Manager.
Its written and configurable all in Perl. A simple example would be:
# /etc/ubic/services/test
use Ubic::Service::SimpleDaemon;
return Ubic::Service::SimpleDaemon->new({ bin => "sleep 1000" });
To start above is: ubic start test. To check its running or not: ubic status test. To stop service (suprisingly!) is: ubic stop test.
Ubic keeps an eye on all its services so when test service stops after 1000 seconds then Ubic will automatically restart it again.
Some more links:
Mailing list
Ubic - how to implement your first service (blog post)
Ubic - code reuse and customizations (blog post)
/I3az/
I am a big fan of Monit. It's written in C, but does everything you want.
I particularly liked that I was able to compile a thin version that worked beautifully on an ARM based system with only 64 MB of RAM.
You might want to read God vs Monit on SO to get a comparison.
Bluepill is a great process monitoring/administration framework.
It's written in Ruby but it can monitor anything, I use it to monitor Unicorn processes.
It even runs on 1.9.2.
Doesn't leak memory.
Has support for demonizing processes that don't demonize themselves.
All around easy, even with RVM!