I am using janusgraph-0.5.3 (with Cassandra) and I want to know how to configure memory allocation to increase default memory allocated to 2GB for the gremlin server process.
I am trying to load bulk data on my gremlin-server, but it is failing with error. I would like to know if there is a way to check and increase the default memory allocation.
I need help in locating the .yaml configuration files as well as the values in these files that would need to change.
Thanks
I changed gremlin-server.sh file to take additional memory
# Set Java options
if [ "$JAVA_OPTIONS" = "" ] ; then
echo "Setting xmx and xss"
JAVA_OPTIONS="-Xms1024m -Xmx3074m -Xss2048k -javaagent:$JANUSGRAPH_LIB/jamm-0.3.0.jar -Dgremlin.io.kryoShimService=org.janusgraph.hadoop.serialize.JanusGraphKryoShimService"
fi
Related
I have deployed opentelemetry-collector as a container by pulling the image from https://hub.docker.com/r/otel/opentelemetry-collector/tags. I checked the container's memory usage using docker stats command and I got MEM USAGE / LIMIT -> 15.3MiB / 7.667GiB
Is there any possibility to reduce the memory usage for this default image to below 10MiB
I want to reduce the opentelemetry-collector container's memory usage to below 10MiB
You can add --memory=10m to your docker run command.
https://docs.docker.com/config/containers/resource_constraints/
Now - this isn't magic. If the process needs more than that to run then it will just crash.
If that is the case, then you will need to look at changing the configuration of the service and/or possibly its source code.
I am running systemd version 219.
root#EVOvPTX1_RE0-re0:/var/log# systemctl --version
systemd 219
+PAM -AUDIT -SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP -LIBCRYPTSETUP -GCRYPT +GNUTLS +ACL +XZ -LZ4 -SECCOMP +BLKID -ELFUTILS +KMOD -IDN
I have a service, let's call it foo.service which has the following.
[Service]
MemoryLimit=1G
I have deliberately added code to allocate 1M memory 4096 times which causes
4G memory alloc when a certain event is received. The idea is that after
the process consumes 1G of address space, memory alloc would start failing.
However, this does not seem to be the case. I am able to alloc 4G memory
without any issues. This tells me that the memory limit specified in the
service file is not enforced.
Can anyone let me know what am I missing ?
I looked at the proc file system - file named limits. This shows that the
Max address space is Unlimited, which also confirms that the memory limit
is not getting enforced.
This distinction is that you have allocated memory, but you haven't actually used it. In the output of top, this is the difference between the "VIRT" memory column (allocated) and the "RES" column (actually used).
Try modifying your experiment to assign values to elements of a large array instead of just allocating memory and see if you hit the memory limit that way.
Reference: Resident and Virtual memory on Linux: A short example
when I try and start redis service I keep getting this error:
"The Redis service on Local Computer started and then stopped. Some services stop automatically if they are not in use by other services or programs".
The only thing that works is restarting my computer, then the redis service is running on startup.
Is there any configuration I need to set up in order for it to work better?
I installed redis using the .msi, version 2.8.2104.
All help would be very appreciated! Thanks
Right click in the service in Window Services and go to Properties. Then go to Log On tab and select Local System account. Click on Ok button and start the service.
For those that may have a similar problem (like we did), I found another solution.
The machine we're running on (TEST) only had 7GB of free space on the drive. But we have 16GB of RAM. In our redis.windows.conf file, there is a setting called maxheap that was NOT set.
According to the documentation on maxheap:
# The maxheap flag controls the maximum size of this memory mapped file,
# as well as the total usable space for the Redis heap. Running Redis
# without either maxheap or maxmemory will result in a memory mapped file
# being created that is equal to the size of physical memory. During
# fork() operations the total page file commit will max out at around:
#
# (size of physical memory) + (2 * size of maxheap)
#
# For instance, on a machine with 8GB of physical RAM, the max page file
# commit with the default maxheap size will be (8)+(2*8) GB , or 24GB. The
# default page file sizing of Windows will allow for this without having
# to reconfigure the system. Larger heap sizes are possible, but the maximum
# page file size will have to be increased accordingly.
#
# The Redis heap must be larger than the value specified by the maxmemory
# flag, as the heap allocator has its own memory requirements and
# fragmentation of the heap is inevitable. If only the maxmemory flag is
# specified, maxheap will be set at 1.5*maxmemory. If the maxheap flag is
# specified along with maxmemory, the maxheap flag will be automatically
# increased if it is smaller than 1.5*maxmemory.
#
# maxheap <bytes>
So I set it to a reasonable value and the service started right up.
I found a read-write error on configuration (ini)
Please check out all the files and directory specified on the INI.
I an newbie in linux, I didnt get proper meaning of "mapping backed by a file" in process address space
can anybody explain?
It means that the real data is stored in a file, but it's being mapped to RAM, so that you can access it as if it were in RAM. It will be brought in from disk when required.
This would generally be used for something where the real data is huge, and so wouldn't all fit comfortably in RAM.
This can be understood as if the data that is stored in a file gets referred by some virtual space using which it can be read as if it was stored a part of the virtual memory or RAM.
" File-backed mapping maps an area of the process's virtual memory to
files; i.e. reading those areas of memory causes the file to be read.
It is the default mapping type."
Reference : http://en.wikipedia.org/wiki/Mmap
I'm testing using Apache's Jmeter, I'm simply accessing one page of my companies website and turning up the number of users until it reaches a threshold, the problem is that when I get to around 3000 threads JMeter doesn't run all of them. Looking at the Aggregate Graph
it only runs about 2,536 (this number varies but is always around here) of them.
The partial run comes with the following exception in the logs:
01:16 ERROR - jmeter.JMeter: Uncaught exception:
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Unknown Source)
at org.apache.jmeter.threads.ThreadGroup.start(ThreadGroup.java:293)
at org.apache.jmeter.engine.StandardJMeterEngine.startThreadGroup(StandardJMeterEngine.java:476)
at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:395)
at java.lang.Thread.run(Unknown Source)
This behavior is consistent. In addition one of the times JMeter crashed in the middle outputting a file that said:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (allocation.cpp:211), pid=10748, tid=11652
#
# JRE version: 6.0_31-b05
# Java VM: Java HotSpot(TM) Client VM (20.6-b01 mixed mode, sharing windows-x86 )
Any ideas?
I tried changing the heap size in jmeter.bat, but that didn't seem to help at all.
JVM is simply not capable of running so many threads. And even if it is, JMeter will consume a lot of CPU resources to purely switch contexts. In other words, above some point you are not benchmarking your web application but the client computer, hosting JMeter.
You have few choices:
experiment with JVM options, e.g. decrease default -Xss512K to something smaller
run JMeter in a cluster
use tools taking radically different approach like Gatling
I had a similar issue and increased the heap size in jmeter.bat to 1024M and that fixed the issue.
set HEAP=-Xms1024m -Xmx1024m
For the JVM, if you read hprof it gives you some solutions among which are:
switch to a 64 bits jvm ( > 6_u25)
with this you will be able to allocate more Heap (-Xmx) , ensure you have this RAM
reduce Xss with:
-Xss256k
Then for JMeter, follow best-practices:
http://jmeter.apache.org/usermanual/best-practices.html
http://www.ubik-ingenierie.com/blog/jmeter_performance_tuning_tips/
Finally ensure you use last JMeter version.
Use linux OS preferably
Tune the TCP stack, limits
Success will depend on your machine power (cpu and memory) and your test plan.
If this is not enough (for 3000 threads it should be OK), you may need to use distributed testing
Increasing the heap size in jmeter.bat works fine
set HEAP=-Xms1024m -Xmx1024m
OR
you can do something like below if you are using jmeter.sh:
JVM_ARGS="-Xms512m -Xmx1024m" jmeter.sh etc.
I ran into this same problem and the only solution that helped me is: https://stackoverflow.com/a/26190804/5796780
proper 100k threads on linux:
ulimit -s 256
ulimit -i 120000
echo 120000 > /proc/sys/kernel/threads-max
echo 600000 > /proc/sys/vm/max_map_count
echo 200000 > /proc/sys/kernel/pid_max
If you don't have root access:
echo 200000 | sudo dd of=/proc/sys/kernel/pid_max
After increasing Xms et Xmx heap size, I had to make my Java run in 64 bits mode. In jmeter.bat :
set JM_LAUNCH=java.exe -d64
Obviously, you need to run a 64 bits OS and have installed Java 64 bits (see https://www.java.com/en/download/manual.jsp)