I am currently exploring mesos. I have set up mesos cluster with one slave node added. The hardware added is 1 cpu-core, 2 GB RAM. but at mesos UI it is showing 1 cpu-core, and 1001 MB RAM. It is showing approximately 1GB less RAM. Can any one knows where remaining 1GB RAM is getting utilized ?
If you don't specify via resources how much RAM a Mesos Slave (now: Agent) is supposed to use the default kicks in, see the Mesos containerizer for details.
Related
We have three physical servers. Each server has 2 CPUs (32 cores), 96 TB HDD, and 768 GB RAM. We would like to use these servers in an Elasticsearch cluster.
Each server will be located in a different data center, connecting each server using a private connection.
How can be optimize our configuration for high performance? Also, how should we best run Elasticsearch on these machines. For example, should we use virtualization to create multiple nodes per machine, or not?
As you have huge RAM(768) available on each physical server and according to ES documentation on heap setting it shouldn't cross 32 GB, so you will have to use virtualization to create multiple nodes per physical server for better ultization of your infra.
Apart from these there are various cluster settings and node settings which you can optimize but as you have not provided them, its difficult to provide recommendation on them.
Another thing to note is that you have huge RAM and disk but CPU is not in proportion to it, so if you can increase them as well, it would be good.
So i have a Elasticsearch Cluster inside the Kubernetes.
The machine it is running on has 30 GB RAM and 8 cores.
Now according to the thumb rule 50% of the RAM is what we set as ES_JAVA_OPTS and remaining is used for file caching.
here it would be 15 GB
Also in the helm chart we have resource requirements mentioned like below:
resources:
limits:
cpu: 8
memory: 15Gi
requests:
cpu: 8
memory: 15Gi
My question is whether the 50% RAM is of the host machine (Which is 30 GB) or the limit specified in the helm chart 15 GB
Can someone explain how in kubernetes utilise the RAM
Because if it with respect to Host and file caching is not considered as the utilisation of Deployed Application we are OK. But if it within the Resources Limits i need to increase the to 30GB.
Edit:
The question here is that if one elasticsearch node used 50% of RAM as Heap and 50% as file caching and i mention the Heap as 15GB (50% of the RAM) in a 30GB machine. so should i mention the resoure limitations in the deployment template as somewhere around 15GB which Heap requires of need 30GB (Say 28GB) that from the rule Elasticsearch need to be able to cache files.
This comes as concern as if pod exceed the mentioned limit on the template at any given moment kubernetes restart the pod.
So in other words i want to know the RAM file caching is come into play in the overall memory usage of the pod or not.
Note: I am using instance storage as primary Storage of the ES Data as this is extremely fast as compare to EBS.
Conclusion:
Keep Heap half to the RAM in the system and Mentioned in the resources Limit(if any)
I am not a expert in k8s and docker but what I understand is that, docker container uses the host resources and using resource limit you can have a hard limit on the resources it can consume.
If you put a resource limit of 15GB, than overall your docker container can consume 15GB of host RAM.now whether it will share the file system cache with host or not depends on how you have configured your docker volume.
As docker container have the option to share the file system with host using the bind volume or have its own data volume(which is ephemeral and not suited for ES as its a stateful application). in first option it should share the file system cache with host and you should not increase the resource limit further(recommended as you have ES which is stateful) and in second option, as it will use its own file system you have to allocate RAM for its file system cache and have to increase RAM to 30 GB, but you have to give some space for Host OS as well.
Container will always see the node`s memory instead of the container one. In Kubernertes, even though you set a limit for the memory to a container, the container itself is not aware of this limit.
This has an effect on the applications that looks up for the memory available on the system and use that information to decide how memory it wants to reserve.
This is why you setup the JVM heap size. Without this specified the JVM will setup the maximum heap size based on the host/node total memory instead of the one available (that you`ve declared as limit) to the container.
Check out this article about how limits works in k8s.
I deployed Heron cluster using aurora-scheduler and Mesos. And when I ran the default WordCountTopology using this cluster, I found the ram demand of aurora task is 4G. However, the WordCountToplogy's configuration as follows:
componentRam: 1G
containerRamRequested: 1G
containerCpuRequested: 2 cores
containerDiskRequeted: 2G
Aurora task.json content is:
It shows that this task of aurora needs 4g ram resources. But I don't know why it requests 4G ram. And how to modified this ram requirement?
In addition, there are two slave hosts in my heron cluster and these host resources is:
In addition to the Ram requested by topology's components, there are some additional resources(cpu, memory) requested for heron's daemon processes,e.g stream-manager. Packing additional CPU in RR
A second cause for the larger resource request is due to Aurora only allows homogeneous containers. The packing algorithm will pick maximum container resources as the resource request for all the containers. For example, if a topology has two containers: one requests 2 cpus and the other requests 3 cpus. Then the eventually all containers will request 3 cpus.
Mesos is advertised as a system that lets you program against your datacenter like it's a single pool of resources (See the Mesos Website). But is this really true that you don't need to consider the configuration of the individual machines? Using Mesos, can you request more resources for a task than are available on a single machine?
For example, if you have 10 machines each with 2 cores and 2g of RAM and 20g HD, can you really request 10 cores, 15g of RAM and 100g of disk space for a single task?
If so, how does this work? Is Mesos able to address memory across machines for you, and use other CPUs as local threads and create a single filesystem from a number of distributed nodes?
How does it accomplish this without suffering from the Fallacies of distributed computing, especially those related to network latency and transport cost?
According to this Mesos architecture you can't aggregate resources from different slaves (agents / machines) to use them for one task.
As you can see there is strict "taks per agent" situation
Also their example says pretty much same
Let’s walk through the events in the figure.
Agent 1 reports to the master that it has 4 CPUs and 4 GB of memory
free. The master then invokes the allocation policy module, which
tells it that framework 1 should be offered all available resources.
The master sends a resource offer describing what is available on
agent 1 to framework 1. The framework’s scheduler replies to the
master with information about two tasks to run on the agent, using <2
CPUs, 1 GB RAM> for the first task, and <1 CPUs, 2 GB RAM> for the
second task. Finally, the master sends the tasks to the agent, which
allocates appropriate resources to the framework’s executor, which in
turn launches the two tasks (depicted with dotted-line borders in the
figure). Because 1 CPU and 1 GB of RAM are still unallocated, the
allocation module may now offer them to framework 2.
I understand Mesos architecture at a high level, but I'm not clear about the OS level techniques used to implement resources allocation. For example, Mesos offers a framework 1 CPU and 400MB memory, and another framework 2 CPUs and 1GB memory, how is this actually implemented at OS level?
tl;dr: Mesos itself doesn't "allocate" any resources at the OS-level. The resources are still allocated by the OS, although Mesos can use OS-level primitives like cgroups to ensure that a task doesn't use more resources than it should.
The Mesos agent at the node advertises that some resources are available at the host (e.g., 4 CPUs and 16GB of RAM) -- either by auto-detecting what is available at the host or because the available resources have been explicitly configured (recommended for production).
The master then offers those resources to a framework.
The framework can then launch a task, using some or all of the resources available at the agent: e.g., the framework might launch a task with 2 CPUs and 8GB of RAM.
The agent then launches an executor to run the task.
How strictly the "2 CPUs and 8GB of RAM" resource limit is enforced depends on how Mesos is configured. For example, if the agent host supports cgroups and the agent is started with --isolation='cgroups/cpu,cgroups/mem', cgroups will be used to throttle the CPU appropriately, and to kill the task if it tries to exceed its memory allocation.