I'm trying to start Redis on Windows 10. When I run redis-server.exe I get the following message though:
C:\Program Files\Redis>redis-server.exe
[4680] 01 Jun 19:57:30.844 #
The Windows version of Redis allocates a memory mapped heap for sharing with
the forked process used for persistence operations. In order to share this
memory, Windows allocates from the system paging file a portion equal to the
size of the Redis heap. At this time there is insufficient contiguous free
space available in the system paging file for this operation (Windows error
0x5AF). To work around this you may either increase the size of the system
paging file, or decrease the size of the Redis heap with the --maxheap flag.
Sometimes a reboot will defragment the system paging file sufficiently for
this operation to complete successfully.
Please see the documentation included with the binary distributions for more
details on the --maxheap flag.
Redis can not continue. Exiting.
I've tried adding a --maxheap value, but this causes the application to crash and print out a crash dump:
C:\Program Files\Redis>redis-server.exe --maxheap 150
[6728] 01 Jun 19:57:41.063 #
=== REDIS BUG REPORT START: Cut & paste starting from here ===
[6728] 01 Jun 19:57:41.063 # Redis version: 2.8.2400
[6728] 01 Jun 19:57:41.063 # Out Of Memory allocating 16 bytes.
[6728] 01 Jun 19:57:41.094 # --- ABORT
[6728] 01 Jun 19:57:41.094 # --- STACK TRACE
redis-server.exe!LogStackTrace(c:\release\redis\src\win32_interop\win32_stacktrace.cpp:95)(0x00000016, 0x00001C1D, 0x00000000, 0x00000001)
redis-server.exe!AbortHandler(c:\release\redis\src\win32_interop\win32_stacktrace.cpp:207)(0x00000001, 0xA222916F, 0x00000000, 0xDA31C9A7)
redis-server.exe!raise(f:\dd\vctools\crt\crtw32\misc\winsig.c:587)(0x00000001, 0x00000000, 0x00000010, 0x00000018)
redis-server.exe!abort(f:\dd\vctools\crt\crtw32\misc\abort.c:82)(0xFFE97BC0, 0x40141940, 0x00000010, 0x00000000)
redis-server.exe!redisOutOfMemoryHandler(c:\release\redis\src\redis.c:3404)(0x00000000, 0x4013FC28, 0x00001C1D, 0x0014FB80)
redis-server.exe!createSharedObjects(c:\release\redis\src\redis.c:1326)(0x4013BD44, 0x00000000, 0x00000001, 0x00000001)
redis-server.exe!initServer(c:\release\redis\src\redis.c:1733)(0x4013BD44, 0x4013BD44, 0x00000000, 0x4013BD44)
redis-server.exe!redis_main(c:\release\redis\src\redis.c:3504)(0x00490470, 0x00000000, 0x574F3E35, 0x00000000)
redis-server.exe!main(c:\release\redis\src\win32_interop\win32_qfork.cpp:1338)(0x00000000, 0x00000000, 0x00000000, 0x0034F000)
redis-server.exe!__tmainCRTStartup(f:\dd\vctools\crt\crtw32\startup\crt0.c:255)(0x00000000, 0x400344CC, 0x00000000, 0x00000000)
KERNEL32.DLL!BaseThreadInitThunk(f:\dd\vctools\crt\crtw32\startup\crt0.c:255)(0xD9E580E0, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(f:\dd\vctools\crt\crtw32\startup\crt0.c:255)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
ntdll.dll!RtlUserThreadStart(f:\dd\vctools\crt\crtw32\startup\crt0.c:255)(0x00000000, 0x00000000, 0x00000000, 0x00000000)
[6728] 01 Jun 19:57:41.172 #
=== REDIS BUG REPORT END. Make sure to include from START to END. ===
Please report this bug by following the instructions at:
http://github.com/MSOpenTech/redis/wiki/Submitting-an-Issue
Suspect RAM error? Use redis-server --test-memory to verify it.
I've tried running with the rest memory option suggested at the end of the crash dump, but this just gives me the same error as running the executable did without it:
C:\Program Files\Redis>redis-server.exe --test-memory
[756] 01 Jun 19:58:34.376 #
The Windows version of Redis allocates a memory mapped heap for sharing with
the forked process used for persistence operations. In order to share this
memory, Windows allocates from the system paging file a portion equal to the
size of the Redis heap. At this time there is insufficient contiguous free
space available in the system paging file for this operation (Windows error
0x5AF). To work around this you may either increase the size of the system
paging file, or decrease the size of the Redis heap with the --maxheap flag.
Sometimes a reboot will defragment the system paging file sufficiently for
this operation to complete successfully.
Please see the documentation included with the binary distributions for more
details on the --maxheap flag.
Redis can not continue. Exiting.
I have 14Gb of memory, so I'm finding it hard to believe it's a memory issue.
Found the answer after a wee bit of Google-ing: https://gist.github.com/anthavio/38d0ba4c916ab1911a47
Instead of using the --maxheap option (which seems to do nothing), add the value to the config:
maxheap 1000000000
Related
I want to allocate a specific 512MB page which is MEM_FREE, and I want
to change that page to MEM_RESERVE and PAGE_NOACCESS.
Hence, with Windbg, I found a page and I called to
NtAllocateVirtualMemory on that page address with PAGE_RESERVED and
PAGE_NOACCESS.
After this call - I didn't notice that the page got PAGE_NOACCESS
flag(with !address command), but I couldn't change a memory address
inside that page - (I got memory access error for eb command).
So the operation succeeded because I couldn't change the memory.
Do you have an idea why I don't see in windbg PAGE_NOACCESS for
that page after change it's permissions?
Next Step, I called to VirtualQuery on that free page, and the
function failed with error 998 (Invalid access to memory location)
Finally I want to identify free pages and reveal their size.
Do you have an idea how can I get this information it VirtualQuery fails?
Thanks in Advance!
MEM_RESERVE reserves a range of addresses in the VAD tree. It doesn't commit pages that can have protection applied to them. Read the description of the PAGE_* constants. The phrase "committed region of pages" is written repeatedly. The Protect parameter of NtAllocateVirtualMemory says it applies to a "committed region of pages". Similarly flProtect of VirtualAllocEx is valid when "the pages are being committed".
Here's a Python script the commits a page of memory that's protected with PAGE_NOACCESS. Then it attempts to read the first byte, which of course raises an access violation exception.
test.py:
from ctypes import *
MEM_COMMIT = 0x1000
PAGE_NOACCESS = 1
VirtualAlloc = WinDLL('kernel32').VirtualAlloc
VirtualAlloc.restype = c_void_p
addr = VirtualAlloc(None, 4096, MEM_COMMIT, PAGE_NOACCESS)
array = (c_char * 4096).from_address(addr)
array[0] # access violation
demo:
(test) C:\>cdb -xi ld python test.py
Microsoft (R) Windows Debugger Version 6.12.0002.633 AMD64
Copyright (c) Microsoft Corporation. All rights reserved.
CommandLine: python test.py
Symbol search path is: symsrv*symsrv.dll*C:\Symbols*
http://msdl.microsoft.com/download/symbols
Executable search path is:
(d70.d08): Break instruction exception - code 80000003 (first chance)
ntdll!LdrpDoDebuggerBreak+0x30:
00000000`77b78700 cc int 3
0:000> g
(d70.d08): Access violation - code c0000005 (first chance)
First chance exceptions are reported before any exception handling.
This exception may be expected and handled.
python34!PyBytes_FromStringAndSize+0x70:
00000000`64bd5cc0 0fb601 movzx eax,byte ptr [rcx]
ds:00000000`00190000=??
0:000> !address 190000
Usage: <unclassified>
Allocation Base: 00000000`00190000
Base Address: 00000000`00190000
End Address: 00000000`00191000
Region Size: 00000000`00001000
Type: 00020000 MEM_PRIVATE
State: 00001000 MEM_COMMIT
Protect: 00000001 PAGE_NOACCESS
I was trying to resize partitions of Intel Edison which runs on Yocto system. I destroyed partition 1 somehow. So when I reboot system it loops forever to boot the system correctly. So, interrupted boot sequence and checked for partitions to see what is wrong. As you can see down below partitions start from number two not one! According to error from bootloader it searches for part 1 but cannot find . So how can I add/ create/ repair manually partition 1 to recover system?
boot > printenv partitions
partitions=uuid_disk=${uuid_disk};name=u-boot0,start=1MiB,size=2MiB,uuid=${uuid_uboot0};name=u-boot-env0,size=1MiB,uuid=${uuid_uboot_env0};name=u-boot1,size=2MiB,uuid=${uuid_uboot1};name=u-boot-env1,size=1MiB,uuid=${uuid_uboot_env1};name=factory,size=1MiB,uuid=${uuid_factory};name=panic,size=24MiB,uuid=${uuid_panic};name=boot,size=32MiB,uuid=${uuid_boot};name=rootfs,size=512MiB,uuid=${uuid_rootfs};name=update,size=768MiB,uuid=${uuid_update};name=home,size=-,uuid=${uuid_home};
boot > mmc part
Partition Map for MMC device 0 -- Partition Type: DOS
Part Start Sector Num Sectors UUID Type
2 16 7634928 00000000-02 83
Boot Sequence
******************************
PSH KERNEL VERSION: b0182727
WR: 20104000
******************************
SCU IPC: 0x800000d0 0xfffce92c
PSH miaHOB version: TNG.B0.VVBD.0000000c
microkernel built 23:15:13 Apr 24 2014
******* PSH loader *******
PCM page cache size = 192 KB
Cache Constraint = 0 Pages
Arming IPC driver ..
Adding page store pool ..
PagestoreAddr(IMR Start Address) = 0x04899000
pageStoreSize(IMR Size) = 0x00080000
*** Ready to receive application ***
U-Boot 2014.04 (Oct 14 2014 - 15:19:04)
Watchdog enabled
DRAM: 980.6 MiB
MMC: tangier_sdhci: 0
In: serial
Out: serial
Err: serial
Hit any key to stop autoboot: 0
Target:blank
Partitioning already done...
Flashing already done...
**dfu_fill_entity_mmc: could not find partition #1 on mmc device #0!
ERROR: DFU entities configuration failed!**
at drivers/dfu/dfu.c:71/dfu_init_env_entities()
dfu - Device Firmware Upgrade
Usage:
dfu <USB_controller> <interface> <dev> [list|timeout]
- device firmware upgrade via <USB_controller>
on device <dev>, attached to interface
<interface>
[list] - list available alt settings
[timeout] - specify inactivity timeout in sec, doesn't work whit list
** Invalid partition 7 **
Error: Invalid Boot Flag (found 0xffef, expected 0xaa55)
## Kernel loading failed ...
zboot - Boot bzImage
Usage:
zboot [addr] [size] [initrd addr] [initrd size]
addr - The optional starting address of the bzimage.
If not set it defaults to the environment
variable "fileaddr".
size - The optional size of the bzimage. Defaults to
zero.
initrd addr - The address of the initrd image to use, if any.
initrd size - The size of the initrd image to use, if any.
Unknown boot mode: boot
Saving Environment to MMC...
Writing to MMC(0)... done
Resetting to default boot mode and reboot...
resetting ...
I solved that problem myself. After interrupting boot sequence use gpt command to add new partition to mmc. Then, system boots correctly.
My OS is Fedora 17. Recently, kernel tainted warning "kernel bug at kernel/auditsc.c:1772!-abrt" occurs:
This problem should not be reported (it is likely a known problem). A kernel problem occurred, but your kernel has been tainted (flags:GD). Kernel maintainers are unable to diagnose tainted reports.
Then, I get the following:
# cat /proc/sys/kernel/tainted
128
# dmesg | grep -i taint
[ 8306.955523] Pid: 4511, comm: chrome Tainted: G D 3.9.10-100.fc17.i686.PAE #1 Dell Inc.
[ 8307.366310] Pid: 4571, comm: chrome Tainted: G D 3.9.10-100.fc17.i686.PAE #1 Dell Inc.
It seems that the value "128" is much serious:
128 – The system has died.
How about this warning? Since chrome is flagged as the "Tainted" source, anybody also meet this matter?
To (over) simplify, 'tainted' means that the kernel is in a state other than what it would be in if it were built fresh from the open source origin and used in a way that it had been intended. It is a way of flagging a kernel to warn people (e.g., developers) that there may be unknown reasons for it to be unreliable, and that debugging it may be difficult or impossible.
In this case, 'GD' means that all modules are licensed as GPL or compatible (ie not proprietary), and that a crash or BUG() occurred.
The reasons are listed below:
See: oops-tracing.txt
---------------------------------------------------------------------------
Tainted kernels:
Some oops reports contain the string 'Tainted: ' after the program
counter. This indicates that the kernel has been tainted by some
mechanism. The string is followed by a series of position-sensitive
characters, each representing a particular tainted value.
1: 'G' if all modules loaded have a GPL or compatible license, 'P' if
any proprietary module has been loaded. Modules without a
MODULE_LICENSE or with a MODULE_LICENSE that is not recognised by
insmod as GPL compatible are assumed to be proprietary.
2: 'F' if any module was force loaded by "insmod -f", ' ' if all
modules were loaded normally.
3: 'S' if the oops occurred on an SMP kernel running on hardware that
hasn't been certified as safe to run multiprocessor.
Currently this occurs only on various Athlons that are not
SMP capable.
4: 'R' if a module was force unloaded by "rmmod -f", ' ' if all
modules were unloaded normally.
5: 'M' if any processor has reported a Machine Check Exception,
' ' if no Machine Check Exceptions have occurred.
6: 'B' if a page-release function has found a bad page reference or
some unexpected page flags.
7: 'U' if a user or user application specifically requested that the
Tainted flag be set, ' ' otherwise.
8: 'D' if the kernel has died recently, i.e. there was an OOPS or BUG.
9: 'A' if the ACPI table has been overridden.
10: 'W' if a warning has previously been issued by the kernel.
(Though some warnings may set more specific taint flags.)
11: 'C' if a staging driver has been loaded.
12: 'I' if the kernel is working around a severe bug in the platform
firmware (BIOS or similar).
13: 'O' if an externally-built ("out-of-tree") module has been loaded.
14: 'E' if an unsigned module has been loaded in a kernel supporting
module signature.
15: 'L' if a soft lockup has previously occurred on the system.
16: 'K' if the kernel has been live patched.
The primary reason for the 'Tainted: ' string is to tell kernel
debuggers if this is a clean kernel or if anything unusual has
occurred. Tainting is permanent: even if an offending module is
unloaded, the tainted value remains to indicate that the kernel is not
trustworthy.
Also showing numbers for the content of /proc/sys/kernel/tainted file:
Non-zero if the kernel has been tainted. Numeric values, which can be
ORed together. The letters are seen in "Tainted" line of Oops reports.
1 (P): A module with a non-GPL license has been loaded, this
includes modules with no license.
Set by modutils >= 2.4.9 and module-init-tools.
2 (F): A module was force loaded by insmod -f.
Set by modutils >= 2.4.9 and module-init-tools.
4 (S): Unsafe SMP processors: SMP with CPUs not designed for SMP.
8 (R): A module was forcibly unloaded from the system by rmmod -f.
16 (M): A hardware machine check error occurred on the system.
32 (B): A bad page was discovered on the system.
64 (U): The user has asked that the system be marked "tainted". This
could be because they are running software that directly modifies
the hardware, or for other reasons.
128 (D): The system has died.
256 (A): The ACPI DSDT has been overridden with one supplied by the user
instead of using the one provided by the hardware.
512 (W): A kernel warning has occurred.
1024 (C): A module from drivers/staging was loaded.
2048 (I): The system is working around a severe firmware bug.
4096 (O): An out-of-tree module has been loaded.
8192 (E): An unsigned module has been loaded in a kernel supporting module
signature.
16384 (L): A soft lockup has previously occurred on the system.
32768 (K): The kernel has been live patched.
65536 (X): Auxiliary taint, defined and used by for distros.
131072 (T): The kernel was built with the struct randomization plugin.
Source: https://www.kernel.org/doc/Documentation/sysctl/kernel.txt
Credit: https://askubuntu.com/questions/248470/what-does-the-kernel-taint-value-mean
On a 6 node Cassandra (datastax-community-2.1) cluster running on EC2 (i2.xlarge) instances, Jvm hosting the cassandra service randomly crashes with following error.
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 12288 bytes for committing reserved memory.
# Possible reasons:
# The system is out of physical RAM or swap space
# In 32 bit mode, the process size limit was hit
# Possible solutions:
# Reduce memory load on the system
# Increase physical memory or swap space
# Check if swap backing store is full
# Use 64 bit Java on a 64 bit OS
# Decrease Java heap size (-Xmx/-Xms)
# Decrease number of Java threads
# Decrease Java thread stack sizes (-Xss)
# Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
# Out of Memory Error (os_linux.cpp:2747), pid=26159, tid=140305605682944
#
# JRE version: Java(TM) SE Runtime Environment (7.0_60-b19) (build 1.7.0_60-b19)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.60-b09 mixed mode linux-amd64 compressed oops)
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
--------------- T H R E A D ---------------
Current thread (0x0000000008341000): JavaThread "MemtableFlushWriter:2055" daemon [_thread_new, id=23336, stack(0x00007f9b71c56000,0x00007f9b71c97000)]
Stack: [0x00007f9b71c56000,0x00007f9b71c97000], sp=0x00007f9b71c95820, free space=254k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x99e7ca] VMError::report_and_die()+0x2ea
V [libjvm.so+0x496fbb] report_vm_out_of_memory(char const*, int, unsigned long, char const*)+0x9b
V [libjvm.so+0x81d81e] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0xfe
V [libjvm.so+0x81d8dc] os::pd_commit_memory(char*, unsigned long, bool)+0xc
V [libjvm.so+0x81565a] os::commit_memory(char*, unsigned long, bool)+0x2a
V [libjvm.so+0x81bdcd] os::pd_create_stack_guard_pages(char*, unsigned long)+0x6d
V [libjvm.so+0x9522de] JavaThread::create_stack_guard_pages()+0x5e
V [libjvm.so+0x958c24] JavaThread::run()+0x34
V [libjvm.so+0x81f7f8] java_start(Thread*)+0x108
Modifications in cassandra-env.sh
MAX_HEAP_SIZE="8G"
HEAP_NEWSIZE="800M"
JVM_OPTS="$JVM_OPTS -XX:TargetSurvivorRatio=50"
JVM_OPTS="$JVM_OPTS -XX:+AggressiveOpts"
JVM_OPTS="$JVM_OPTS -XX:+UseLargePages"
Here are the full jvm crash logs from two machines
db_06
db_05
Writes are about 10K-15K/sec and there are very few reads. Cassandra 2.0.9 with same settings never crashed. Any clues to what might be causing the to crashes?
Thanks
This question is related to:
Riak node not working, but using 100% cpu
but since the poster seems to have left I'm posting my case here.
Last night I installed erlang(R15B01) from source, using the config options from the Riak website:
http://docs.basho.com/riak/1.2.1/tutorials/installation/Installing-Erlang/#Installing-on-Mac-OS-X
and Riak(1.4.1) on my 2013 MacBook Pro (2.8GHz i7, 16GB ram, OSX 10.8.3). I did not change the ulimit, as I assumed it would be fine for a vanilla run.
Installation went fine; warnings but no errors, and I was able to run the toy examples no problem.
However the empty instance quickly ate through all 4 cores and my machine started whining and overheating.
Looking in the logs I see the following error repeated a jillion times:
2013-10-11 09:04:04.266 [error] CRASH REPORT ¥
Process with 0 neighbours exited with reason: ¥
call to undefined function eleveldb:o
also tons of crash reports:
2013-10-11 09:14:47 =CRASH REPORT====
crasher:
initial call: riak_kv_index_hashtree:init/1
pid:
registered_name: []
exception exit: {{undef,[{eleveldb,open,
["./data/anti_entropy/479555224749202520035584085735030365824602865664",
[{create_if_missing,true},{max_open_files,20},{write_buffer_size,12886952}]],[]},
{hashtree,new_segment_store,2,[{file,"src/hashtree.erl"},{line,499}]},{hashtree,new,2,
[{file,"src/hashtree.erl"},{line,215}]},{riak_kv_index_hashtree,do_new_tree,2,
[{file,"src/riak_kv_index_hashtree.erl"},{line,421}]},{lists,foldl,3,[{file,"lists.erl"},
{line,1197}]},{riak_kv_index_hashtree,init_trees,2,[{file,"src/riak_kv_index_hashtree.erl"},
{line,366}]},{riak_kv_index_hashtree,init,1,[{file,"src/riak_kv_index_hashtree.erl"},
{line,226}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,304}]}]},
[{gen_server,init_it,6,[{file,"gen_server.erl"},{line,328}]},{proc_lib,init_p_do_apply,3,
[{file,"proc_lib.erl"},{line,227}]}]}
ancestors: [,riak_core_vnode_sup,riak_core_sup,]
messages: []
links: []
dictionary: []
trap_exit: false
status: running
heap_size: 987
stack_size: 24
reductions: 492
neighbours:
erlang.log says
=====
===== LOGGING STARTED Fri Oct 11 09:04:01 CEST 2013
=====
Node 'riak#127.0.0.1' not responding to pings.
config is OK
!!!!
!!!! WARNING: ulimit -n is 2560; 4096 is the recommended minimum.
!!!!
Exec: /tmp/riak-1.4.1/rel/riak/bin/../erts-5.9.1/bin/erlexec
-boot /tmp/riak-1.4.1/rel/riak/bin/../releases/1.4.1/riak
-config /tmp/riak-1.4.1/rel/riak/bin/../etc/app.config
-pa /tmp/riak-1.4.1/rel/riak/bin/../lib/basho-patches
-args_file /tmp/riak-1.4.1/rel/riak/bin/../etc/vm.args -- console
Root: /tmp/riak-1.4.1/rel/riak/bin/..
Erlang R15B01 (erts-5.9.1) [source] [64-bit] [smp:8:8] [async-threads:64]
[kernel-poll:true]
Eshell V5.9.1 (abort with ^G)
(riak#127.0.0.1)1>
After less than 10m there are already 144MB of logging files with variations of the above.
I had the same problem by building riak 1.4.6 from source.
I changed in the file etc/app.config the line to
{anti_entropy, {off, []}},
Leveldb is used by AAE. See the config parameter anti_entropy_leveldb_opts.
Use process of elimination:
It's hard to say without more information. Is the 200% being used by
beam.smp? Do you see anything in console.log, error.log or crash.log
that would indicate something odd is happening? Are there clients
communicating with the cluster at the time? If so what client/protocol are
they using and what types of operations are being performed (e.g.
get/put/map reduce/etc)?
References
Riak consuming too much CPU
Interesting sawtooth increasing CPU usage on lightly-used Riak
Inspecting a Node
Riak Performance Tuning
Open Files Limit
Configuration Files