I have just downloaded and run Elasticsearch on my Windows 10 PC. I have not changed any settings, everything is default. The software is running and I can access it on port 9200. However, it appears to be consuming about 9 GB RAM.
I have nothing stored in it, nothing is indexed, it's a default setup.
Why does Elasticsearch need so much memory by default?
EDIT: It's the latest version, 8.1.2.
Elasticsearch latest version(8.1.2 in your case), comes with the bundled JDK and default settings, Elasticsearch default heap settings is 50% of RAM allocated to the machine, it looks like your machine RAM is ~20 Gig, if you want to change this settings, you can follow the steps given in the official jvm options document.
Note: You should allocate 50% of machine RAM to Elasticsearch heap if you running it in Production but it shouldn't cross the ~30 GB in any case.
Related
Windows default EBS size is 30 GB. We can take snapshot and create another volume but that is to increase size. Moreover my intention is to keep only one partition with 1 GB volume.
Any tricks here ?
As far as I'm aware there are no real tricks there.
I once tried reducing the Linux root partition size, however that VM wouldn't start up because the disk was too small and couldn't copy the snapshot to your disk.
You could check if Windows Nano suits your needs and this one comes with 8 GB.
The other option would be building your own image/AMI. However, looking at the Windows Server 2012/2016 system requirements they say you need a 60-GB system-partition. So 30 GB is already quite small and I'm sure you are unable to install any modern full Windows on a 1 GB drive. This isn't Windows 95 anymore...
question, how much resources I need to run apache usergrid?
I mean hardware resources, RAM CPU
I want to deploy apache usergrid to be used as backed in our apps, the apps have a low traffic now, are custom projects to be used in small users groups (<10k)
I want to know the minimum requirements to know if it is viable for us, thanks.
From what I see of usergrid, I can think that the most hungry for resources component will be Elasticsearch, so to have a production environment that's working well, I guess you should start following ES' requirements:
At least 8 GB of RAM
At least 4 cores (the more cores you give Elasticsearch, the more love you get as it tends to works with a lot of threading, i.e. give more cores rather than more CPU processing power)
Fast HDDs should perform fine
See this article on Elasticsearch.A last thing is that depending on your system, you can tune several settings on Elasticsearch to achieve a better throughput. (For instance see https://www.elastic.co/guide/en/elasticsearch/reference/master/tune-for-indexing-speed.html)
I have deployed latest version of Usergrid i.e. 2.1, which is working smoothly in apache-cassandra-2.1.20, apache-tomcat-7.0.85, elasticsearch-1.7.6 on single node of Cassandra on Ubuntu 16.04 with 8 GB Ram and 180 GB SSD. Hope this will help you.
I use this version on my dev environment : Redis-64 .
And I want to know if this version is suitable for the production environment?
If can use, then compared with under Linux, what need to be pay attention to?
Since version 3.0.3 the windows port developers abandoned the dlmalloc and began to use jemalloc as memory allocator. And the port was actually considered for production usage. The 3.0.500 build is approved for production by ms developers (see here).
And there is some kind of hell so how they bypassed the unix fork to save data to disk. Microsoft developers port call it point-in-time heap snapshot. And this is the most controversial part when used in production:
Redis under windows may need up to 3 times more memory than you need in linux version. This behavior is considered normal, because swap file in the windows can easily be up to 3 times larger than the actual amount of RAM.
I think this is acceptable only if the use Redis as LRU cache or not to save data to disk at all.
At least Redis under windows is absolutely susceptible if you Redis node use lot of memory. For example - we try to use Redis for windows (v2.8, v3.0.3, v3.0.5) on server with 512 gb of memory with 2 SSD drives (each 256 gb in raid 0) used as system disk. No any limits on windows swap file. Our test emulates our production - lots of writes and saves with RDB with utilization ~60-70% of memory. And here is was lots of hands up behaviours then this node try to save snapshots - memory consumption jumps, connection freeze during saving. Such behaviour never happens undex linux on same hardware.
I have a windows 7 laptop with 6GB RAM . What is the most RAM/resource efficient way to install pyspark & spark on this laptop just for learning purpose. I don't want to work on actual big data but small dataset is ideal since this is just for learning pyspark & spark in general. I would prefer the latest version of Spark.
FYI: I don't have hadoop installed.
Thanks
You've basically got three options:
Build everything from source
Install Virtualbox and use a pre-built VM like Cloudera Quickstart
Install Docker and find a suitable container
Getting everything up and running when you choose to build from source can be a pain. You've got to install the JDK, build hadoop and spark (both of which require you to install additional software to build them), set up a bunch of environment variables and then pray that didn't mess anything up.
VMs are nice, particularly the one from Cloudera, but you'll often be stuck with an older version of Spark and it might be tight with the resources you described.
I'd go with Docker.
Once you've got docker installed, it becomes very easy to try Spark (and lots of other technologies). My favorite containers for playing around use ipython or jupyter notebooks.
Install Docker:
https://docs.docker.com/installation/windows/
Jupyter Notebook Python, Spark, Mesos Stack
https://github.com/jupyter/docker-stacks/tree/master/pyspark-notebook
One thing to keep in mind is that you are going to have to allocate a certain amount of memory for the VM and the remaining memory still has to operate Windows. Windows 7 requires a minimum of 1 GB for a 32-bit OS or 2 GB for a 64-bit OS. So likely you are only going to wind up with around 4 GB of RAM for running the VM, which is not much.
Assuming you are 64-bit, note that Cloudera requires a minimum of 4 GB RAM to run CDH 5, but if you want to run Cloudera Express, you need 8 GB.
Running Docker from Windows will require you to use boot2docker, which keeps the entire VM in memory. It uses minimal memory (like around 27 MB) to run, so you should be fine there. A MUCH better solution than running VirtualBox!
Another option to consider would be to spin up a free machine on something like Amazon Web Services (http://aws.amazon.com) or Google Cloud (http://cloud.google.com). Particularly with the later, you can get a free trial amount of credits, which you could use to spin up a machine with more RAM than you would typically get with AWS.
Where can I change allocated memory of JDeveloper? Also Maximum and optimum memory for this IDE. My Machine has a 4GB RAM, I use SOA,BPEL frameworks in that.
Where can I change allocated memory of JDeveloper?
For JDeveloper 11.1.2.x, you can configure VM options in the JDeveloper configuration file at jdeveloper/jdev/bin/jdev.conf. For example, to set the maximum heap size to 2GB, add the following line to this file:
AddVMOption -Xmx2048M
Note that this only affects the IDE - to set memory limits for the internal Weblogic Server, you can set the USER_MEM_ARGS environment variable before launching the IDE, like
$ export USER_MEM_ARGS="-Xms256m -Xmx1024m"
Also Maximum and optimum memory for this IDE
The more, the better. There is no limit besides those imposed by the operating system or the Java virtual machine you are using (32 bit vs. 64 bit).
My Machine has a 4GB RAM, I use SOA,BPEL frameworks in that.
I really suggest that you add more memory. Use a 64 bit operating system and upgrade to at least 8 GB. You need to consider that it is not only the IDE which is consuming memory, but also the internal Weblogic Server, where you are running your application during development. The internal WLS is launched as a separate java process and also consumes a considerable amount of memory.