When I was trying to limit memory on mesos agent with --resource flag, it seems doesn't work at all. The slave can still use all the cpus and memory on virtual machine. Did I use it wrong?
By default Mesos Conterneizer uses PSOIX isolators (posix/cpu,posix/mem) since they are not a real isolator it's recomended to use cgroups isolators instead (cgroups/cpu,cgroups/mem).
--isolation=VALUE Isolation mechanisms to use, e.g., posix/cpu,posix/mem (or windows/cpu,windows/mem if you are on Windows), or cgroups/cpu,cgroups/mem, or network/port_mapping (configure with flag: --with-network-isolator to enable), or gpu/nvidia for nvidia specific gpu isolation, or load an alternate isolator module using the --modules flag. If cgroups/all is specified, any other cgroups related isolation options (e.g., cgroups/cpu) will be ignored, and all the local enabled cgroups subsystems on the agent host will be automatically loaded by the cgroups isolator. Note that this flag is only relevant for the Mesos Containerizer. (default: windows/cpu,windows/mem on Windows; posix/cpu,posix/mem on other platforms)
Related
Using Elasticsearch sink connector to insert data to ES 7.2 instance hosted on VM.
Getting this : Elastic search max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
Is it possible to ignore bootstrap checks ?
How can I increase the virtual memory for Elasticsearch in the docker container ?
Bootstrap checks inspect a variety of Elasticsearch and system settings. If youre in development mode, any bootstrap checks that fail appear as warnings in the Elasticsearch log. If youre in production mode failed bootstrap will cause Elasticsearch to refuse to start.
Elasticsearch work mode is configured implicit. As soon as you configure a network setting like network.host,Elasticsearch assumes that you are moving to production and will upgrade the above warnings to exceptions.
Regarding your specific case you need to increase it on the host machine not the docker by running this command:
sudo sysctl -w vm.max_map_count=262144 . and then restart your docker-containers.
BTW not recommended but if you are running a single node you can skip the bootstrap checks by not binding transport to an external interface or by binding transport to an external interface and setting the discovery type to single-node.
How can stateful containers be run inside Mesos?
According to the Mesos documentation sandbox can be used to store state:
With the introduction of persistent volumes, executors and tasks
should never create files outside of the sandbox.
At the same time Sandbox files are scheduled for garbage collection when:
An executor is removed or terminated.
A framework is removed.
An executor is recovered unsuccessfully during agent recovery.
Is this the only way? Or can docker containers be used to maintain state (in a similar manner to a VM)?
So for example, can a container be created and run across 2 nodes? Can such a container contain state and not be disposed of after the task is completed?
The key statement in that quote from the Mesos documentation is
With the introduction of persistent volumes...
You're correct that sandboxes can be garbage collected. However, Mesos provides a primitive called persistent volumes which allows you to create volumes that will persist across task failures and agent restarts and will not be garbage collected.
Additionally, Mesos also now provides support for network storage via the Docker volume isolator. This allows you to mount network volumes using Docker volume drivers, which enables the use of a wide variety of storage back-ends.
Docker containers can store persistent state, but they must do so in either a Mesos persistent volume or a network-attached volume via the Docker volume isolator. These volumes live outside the Docker container and are mounted into the container, so they persist after the container has died.
Mesos tasks cannot be run across multiple nodes. Note that it would be possible for multiple tasks on different nodes to access the same network-attached volume via the Docker volume isolator, provided the back-end storage provider supports concurrent access.
Initially I had two machines to setup hadoop, spark, hbase, kafka, zookeeper, MR2. Each of those machines had 16GB of RAM. I used Apache Ambari to setup the two machines with the above mentioned services.
Now I have upgraded the RAM of each of those machines to 128GB.
How can I now tell Ambari to scale up all its services to make use of the additional memory?
Do I need to understand how the memory is configured for each of these services?
Is this part covered in Ambari documentation somewhere?
Ambari calculates recommended settings for memory usage of each service at install time. So a change in memory post install will not scale up. You would have to edit these settings manually for each service. In order to do that yes you would need an understanding of how memory should be configured for each service. I don't know of any Ambari documentation that recommends memory configuration values for each service. I would suggest one of the following routes:
1) Take a look at each services documentation (YARN, Oozie, Spark, etc.) and take a look at what they recommend for memory related parameter configurations.
2) Take a look at the Ambari code that calculates recommended values for these memory parameters and use those equations to come up with new values that account for your increased memory.
I used this https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.0/bk_installing_manually_book/content/determine-hdp-memory-config.html
Also, Smartsense is must http://docs.hortonworks.com/HDPDocuments/SS1/SmartSense-1.2.0/index.html
We need to define cores, memory, Disks and if we use Hbase or not then script will provide the memory settings for yarn and mapreduce.
root#ttsv-lab-vmdb-01 scripts]# python yarn-utils.py -c 8 -m 128 -d 3 -k True
Using cores=8 memory=128GB disks=3 hbase=True
Profile: cores=8 memory=81920MB reserved=48GB usableMem=80GB disks=3
Num Container=6
Container Ram=13312MB
Used Ram=78GB
Unused Ram=48GB
yarn.scheduler.minimum-allocation-mb=13312
yarn.scheduler.maximum-allocation-mb=79872
yarn.nodemanager.resource.memory-mb=79872
mapreduce.map.memory.mb=13312
mapreduce.map.java.opts=-Xmx10649m
mapreduce.reduce.memory.mb=13312
mapreduce.reduce.java.opts=-Xmx10649m
yarn.app.mapreduce.am.resource.mb=13312
yarn.app.mapreduce.am.command-opts=-Xmx10649m
mapreduce.task.io.sort.mb=5324
Apart from this, we have formulas there to do calculate it manually. I tried with this settings and it was working for me.
Is there a way to make custom resource offers from a Mesos slave? Currently, the resource offers from the slave contain "cpus", "mem", "ports". I want to add custom resource capabilities like upload bandwidth limit, download bandwidth limit etc. There is an option of doing this via the --resources parameter while starting the slave. But what I am looking for is a way to achieve this via code, may be a pluggable module to Mesos, so that the person who is launching the slaves doesn't have to bother about specifying the custom resources.
Is this possible?
Thanks.
Resources reported by the Mesos agent (aka slave) should be specified by the --resources flag. If the flag is omitted, defaults are used. Note that custom resources advertised via --resources will not be isolated, i.e. Mesos agent will not ensure a task is using not more than allocated amount of such resource.
Why do you think a module is a nicer solution than a flag? Loading custom modules requires setting some command line flags as well.
how do I change my MapReduce v1 configuration (eg number of slots) to yarn configuration (eg container virtual memory and virtual cpu)?
There is no longer the concept of a "slot". Instead there are containers, and each container is configured to have a certain amount of RAM and virtual CPU. Go to http://tiny.cloudera.com/yarn-tuning for a detailed writeup on the conversion process.