I am trying to figure out how to use Mesos.
I have running mesos master and slave running (in single-node setup).
And I have understood that framework listens for resource offers and accepts theme if he can, and after that it goes to the executor to execute the task.
How I can send to mesos "Hi, I want to execute some task with 1 cpu and 256 mb", who's task is? the framework? or there is another api for doing this?
Yosi
I think I understood it finally after a debugging session :)
The framework gets a resource offer, when it get a resource offer he checks whether he have a task to launch the matches the resource offer - If so, he runs the task.
I thought there was an external service that I need to call and it will initiate a resource offer.
Related
I have a Laravel application where the Application servers are behind a Load Balancer. On these Application servers, I have cron jobs running, some of which should only be run once (or run on one instance).
I did some research and found that people seem to favor a lock-system, where you keep all the cron jobs active on each application box, and when one goes to process a job, you create some sort of lock so the others know not to process the same job.
I was wondering if anyone had more details on this procedure in regards to AWS, or if there's a better solution for this problem?
You can build distributed locking mechanisms on AWS using DynamoDB with strongly consistent reads. You can also do something similar using Redis (ElastiCache).
Alternatively, you could use Lambda scheduled events to send a request to your load balancer on a cron schedule. Since only one back-end server would receive the request that server could execute the cron job.
These solutions tend to break when your autoscaling group experiences a scale-in event and the server processing the task gets deleted. I prefer to have a small server, like a t2.nano, that isn't part of the cluster and schedule cron jobs on that.
Check out this package for Laravel implementation of the lock system (DB implementation):
https://packagist.org/packages/jdavidbakr/multi-server-event
Also, this pull request solves this problem using the lock system (cache implementation):
https://github.com/laravel/framework/pull/10965
If you need to run stuff only once globally (so not once on every server) and 'lock' the thing that needs to be run, I highly recommend using AWS SQS because it offers exactly that: run a cron to fetch a ticket. If you get one, parse it. Otherwise, do nothing. So all crons are active on all machines, but tickets are 'in flight' when some machine requests a ticket and that specific ticket cannot be requested by another machine.
I went through the video on introduction of DCOS. It was good but got me somewhat confused in terms of classification of component definitions in Mesosphere.
I get that DCOS is an ecosystem and Mesos is like a kernel. Please correct me if I am wrong. For eg. It's like Ubuntu and Linux kernel I presume.
What is marathon? Is it a service or framework or is it something else that falls in neither category? I am bit confused in terms of service vs framework vs application vs Task definition in Mesosphere's context.
Are the services(Cassandra, HDFS, Kubernetes, etc..) that he launches in the video can safely be also called as frameworks?
From 3, are these "services" running as executors in the slaves?
What should rails-app's type be here? Is it a task? So will it also have an executor?
Who makes the decision of autoscaling the rails-app to more nodes, when he increases the traffic using marathon.
1) I get that DCOS is an ecosystem and Mesos is like a kernel. Please
correct me if I am wrong. For eg. It's like Ubuntu and Linux kernel I
presume.
Correct!
2) What is marathon? Is it a service or framework or is it something
else that falls in neither category? I am bit confused in terms of
service vs framework vs application vs Task definition in Mesosphere's
context.
In Apache Mesos terminology, Marathon is a framework. Every framework consists of a framework scheduler and an executor. Many frameworks reuse the standard executor rather than providing their own. An app is a Marathon specific term, meaning the long-running task you launch through it. A task is the unit of execution, running on a Mesos agent (in an executor). In DC/OS (the product, Mesosphere is our company) we call frameworks in general services. Also, in the context of DC/OS, Marathon plays a special role: it acts as a sort of distributed initd, launching other services such as Spark or Kafka.
3) Are the services(Cassandra, HDFS, Kubernetes, etc..) that he
launches in the video can safely be also called as frameworks?
See above.
4) From 3), are these "services" running as executors in the slaves?
No. See above.
5) What should rails-app's type be here? Is it a task? So will it also
have an executor?
The Rails app may have one or more (Mesos) tasks running in executors on one or more agents.
6) Who makes the decision of autoscaling the rails-app to more nodes,
when he increases the traffic using marathon.
Not nodes but instances of the app. Also as #air suggested, with Marathon autoscaling is simple, see also this autoscaling example.
I am trying to understand how the various components of Mesos work together, and found this excellent tutorial that contains the following architectural overview:
I have a few concerns about this that aren't made clear (either in the article or in the official Mesos docs):
Where are the Schedulers running? Are there "Scheduler nodes" where only the Schedulers should be running?
If I was writing my own Mesos framework, what Scheduler functionality would I need to implement? Is it just a binary yes/no or accept/reject for Offers sent by the Master? Any concrete examples?
If I was writing my own Mesos framework, what Executor functionality would I need to implement? Any concrete examples?
What's a concrete example of a Task that would be sent to an Executor?
Are Executors "pinned" (permanently installed on) Slaves, or do they float around in an "on demand" type fashion, being installed and executed dynamically/on-the-fly?
Great questions!
I believe it would be really helpful to have a look at a sample framework such as Rendler. This will probably answer most of your question and give you feeling for the framework internal.
Let me now try to answer the question which might be still be open after this.
Scheduler Location
Schedulers are not on on any special nodes, but keep in mind that schedulers can failover as well (as any part in a distributed system).
Scheduler functionality
Have a look at Rendler or at the framework development guide.
Executor functionality/Task
I believe Rendler is a good example to understand the Task/Executor relationship. Just start reading the README/description on the main github page.
Executor pinning
Executors are started on each node when the first Task requiring such executor is send to this node. After this it will remain on that node.
Hope this helped!
To add to js84's excellent response,
Scheduler Location: Many users like to launch the schedulers via another framework like Marathon to ensure that if the scheduler or its node dies, then it can be restarted elsewhere.
Scheduler functionality: After registering with Mesos, your scheduler will start getting resource offers in the resourceOffers() callback, in which your scheduler should launch (at least) one task on a subset (or all) of the resources being offered. You'll probably also want to implement the statusUpdate() callback to handle task completion/failure.
Note that you may not even need to implement your own scheduler if an existing framework like Marathon/Chronos/Aurora/Kubernetes could suffice.
Executor functionality: You usually don't need to create a custom executor if you just want to launch a linux process or docker container and know when it completes. You could just use the default mesos-executor (by specifying a CommandInfo directly in TaskInfo, instead of embedded inside an ExecutorInfo). If, however you want to build a custom executor, at minimum you need to implement launchTask(), and ideally also killTask().
Example Task: An example task could be a simple linux command like sleep 1000 or echo "Hello World", or a docker container (via ContainerInfo) like image : 'mysql'. Or, if you use a custom executor, then the executor defines what a task is and how to run it, so a task could instead be run as another thread in the executor's process, or just become an item in a queue in a single-threaded executor.
Executor pinning: The executor is distributed via CommandInfo URIs, just like any task binaries, so they do not need to be preinstalled on the nodes. Mesos will fetch and run it for you.
Schedulers: are some strategy to accept or reject the offer. Schedulers we can write our own or we can use some existing one like chronos. In scheduler we should evaluate the resources available and then either accept or reject.
Scheduler functionality: Example could be like suppose say u have a task which needs 8 cpus to run, but the offer from mesos may be 6 cpus which won't serve the need in this case u can reject.
Executor functionality : Executor handles state related information of your task. Set of APIs you need to implement like what is the status of assigned task in mesos slave. What is the num of cpus currently available in mesos slave where executor is running.
concrete example for executor : chronos
being installed and executed dynamically/on-the-fly : These are not possible, you need to pre configure the executors. However you can replicate the executors using autoscaling.
I read the "Cluster from scratch" document on this website : http://clusterlabs.org/doc/
ans I didn't find the answer to my questions :
1) I'm wondering if the Linux Cluster with Pacemaker + Corosync can be used with C++ programs. All the examples are for Apache servers and mostly webservices. Is it just possible ?
2) Is there any document/website that explains the possible links between the cluster status graph (online, active,...) and a potential C++ application graph (application running, stopped, ...).
Short version: Resource agents can be written in any language including C++
Long version:
A resource agent is the glue between pacemaker and your daemon.
Something that knows how to start, stop and health check your daemon but doesn't hang around afterwards.
Its not completely clear whether you want the agent to be in C++ or want to write an agent for a C++ daemon. I suspect you're asking about the first but really need the second. Best thing to do is say hello upstream (irc or public mailing list) so we can continue the discussion.
As I understand you need to create another type of resources, namely your own C++ application. If so then you will need to implement your own resource agent.
I would propose to look into a Dummy resource agent https://github.com/ClusterLabs/pacemaker/blob/master/extra/resources/Dummy and refactor it for your own needs. Read more about resource agents in https://github.com/ClusterLabs/resource-agents/blob/master/doc/dev-guides/ra-dev-guide.asc
1) It is possible. I did some tests using c++ simple tcp code.
I am wondering if there is a way to monitor these automatically. Right now, in our production/QA/Dev environments - we have bunch of services running that are critical to the application. We also have automatic ETLs running on windows task scheduler at a set time of the day. Currently, I have to log into each server and see if all the services are running fine or not, or check event logs for any errors, or check task scheduler to see if ETLs ran well etc etc... I have to do all the manually... I am wondering if there is a tool out there that will do the monitoring for me and send emails only in case something needs attention (like ETLs fail to run, or service get stopped for whatever reason or errors in event log etc). Thanks for the help.
Paessler PRTG Network Monitor can do all that. we have very good experience with it.
http://www.paessler.com/prtg/features
Nagios is the best tool for monitoring. It checks for the server status as well the defined services in it and if any service goes down or system goes down, sends the mail to specified mail id.
Refer the : http://nagios.org/
Thanks for the above information. I looked at the above options but they have a price.. what I did is an inexpensive way to address my concerns..
For my windows task scheduler jobs that run every night - I installed this tool/service from codeplex that is working great.
http://motash.codeplex.com/documentation#CommentsAnchor
For Windows services - I am just setting the "Recovery" Tab in each service "property" with actions to do when it fails. (like restart, reboot, or run a program which could be an email that will notify)
I built a simple tool (https://cronitor.io) for monitoring periodic/scheduled tasks. The name is a play on "cron" from the unix world, but it is system/task agnostic. All you have to do is make an http request to a unique tracking URL whenever your job runs. If your job doesn't check-in according to the rules you define then it will send you an email/sms message.
It also allows you to track the duration of your jobs by making calls at the beginning and end of your task. This can be really useful for long running jobs since you can be alerted if they start taking too long to run. For example, I once had a backup task that was scheduled every hour. About six months after I set it up it started taking longer than an hour to run!
There is https://eyewitness.io - which is for monitoring server cron tasks, queues and websites. It makes sure each of your cron jobs run when they are supposed to, and alerts you if they failed to be run.