I have an autoscaling cloudformation that I think I have set up to replace failed instances based on StatusCheckFailed_Instance. I want to test this. Can I test this by terminating one of the EC2 instances? Thanks!
Instance Status Check might fail for one of the following reasons:
Memory Errors
Out of memory: kill process
ERROR: mmu_update failed (Memory management update failed)
Device Errors
I/O error (Block device failure)
IO ERROR: neither local nor remote disk (Broken distributed block device)
Kernel Errors
request_module: runaway loop modprobe (Looping legacy kernel modprobe on older Linux versions)
"FATAL: kernel too old" and "fsck: No such file or directory while trying to open /dev" (Kernel and AMI mismatch)
"FATAL: Could not load /lib/modules" or "BusyBox" (Missing kernel modules)
ERROR Invalid kernel (EC2 incompatible kernel)
File System Errors
request_module: runaway loop modprobe (Looping legacy kernel modprobe on older Linux versions)
fsck: No such file or directory while trying to open... (File system not found)
General error mounting filesystems (Failed mount)
VFS: Unable to mount root fs on unknown-block (Root filesystem mismatch)
Error: Unable to determine major/minor number of root device... (Root file system/device mismatch)
XENBUS: Device with no driver...
... days without being checked, check forced (File system check required)
fsck died with exit status... (Missing device)
Operating System Errors
GRUB prompt (grubdom>)
Bringing up interface eth0: Device eth0 has different MAC address than expected, ignoring. (Hard-coded MAC address)
Unable to load SELinux Policy. Machine is in enforcing mode. Halting now. (SELinux misconfiguration)
XENBUS: Timeout connecting to devices (Xenbus timeout)
It seems to me that #1 is the easiest to implement to fail on demand. You can add web hook or launch a shell script with the delay to start some process which will result in OutOfMemory failure, to confirm that your autoscaling configuration works as configured.
Terminating the instance will not help to test your configuration, as when you gracefully terminating the instance, it is removed from the pool of available instances, and the check will not be performed.
More details on status checks can be found here: Troubleshooting Instances with Failed Status Checks
Related
i resently upgraded a system. after reboot i was not able to login again. all users have been rejected with Login incorrect. systemd with journaling was running and writing error messages to file in /var/log/journal as usual.
i so booted a system from a revovery usb stick (same distribution) mounted the root device of the failed system /mnt and tried to analyze the logs with journalctl --root=/mnt/var/log/journal -xe. journalctl did not find journal files.
question: how can i read systemd journal content of a dead system using a recovery system?
have fun
I may be a bit late, but I stumbled upon this question and here is what I found:
journalctl logs are located in /var/log/journal/*
journalctl app can read foreign journal files with the following switches:
--file= followed by the *.journal file of your choice. This option may be used multiples times
--root= followed by the root directory of you choice, probably a mounted partition
--image= followed by a disk image,
files as argument, with the option --file
I was in the progress of upgrading my ubuntu system from 14.04 to 16.06 using the GUI software updater.My battery died and machine went off. When I tried to restart,I get this error below.
run-init: /sbin/init: No such file or directory
Target file system does not have requested /sbin/init
run-init: /sbin/init: No such file or directory
run-init: /etc/init: Permission denied
run-init: /bin/init: No such file or directory
/bin/sh O: Can'topen splash
[5.210492] Kernel panic, not syncing: Attempted to kill init! exitcode=0x00007f00
kernel offset 0x0 from 0xffffffff8100000000
drm_kms_helper: panic occured, switching back to text console
At this point it stays in this black screen till I restart, and it happens like this again.
I searched on other forums and I was advised to download and run the bootinfoscript. I run it and had results in a text file but I am unable to attach it here (more than 30000 characters)
Kindly help me resolve this. I don't want to reinstall as I have too much to lose.
Thanks.
I'm on the home stretch of this App I'm working on and I can't seem to get it to port to iOS. When trying to build my application on iOS, I get an OutOfMemory exception over and over again.
Then I tried to build a basic Gluon Mobile application and port it to iOS and I get this:
:compileJava UP-TO-DATE
:processResources UP-TO-DATE
:classes UP-TO-DATE
:createDefaultIOSLauncher UP-TO-DATE
:compileIosJava UP-TO-DATE
:processIosResources UP-TO-DATE
:iosClasses UP-TO-DATE
:iosExtractNativeLibs UP-TO-DATE
:createIpa
RoboVM has detected that you are running on a slow HDD. Please consider mounting a RAM disk.
To create a 2GB RAM disk, run this in your terminal:
SIZE=2048 ; diskutil erasevolume HFS+ 'RoboVM RAM Disk' `hdiutil attach -nomount ram://$((SIZE * 2048))`
See http://docs.robovm.com/ for more info
RoboVM has detected that you are running on a slow HDD. Please consider mounting a RAM disk.
To create a 2GB RAM disk, run this in your terminal:
SIZE=2048 ; diskutil erasevolume HFS+ 'RoboVM RAM Disk' `hdiutil attach -nomount ram://$((SIZE * 2048))`
See http://docs.robovm.com/ for more info
Root pattern javax.annotations.**.* matches no classes
Root pattern javax.inject.**.* matches no classes
RoboVM has detected that you are running on a slow HDD. Please consider mounting a RAM disk.
To create a 2GB RAM disk, run this in your terminal:
SIZE=2048 ; diskutil erasevolume HFS+ 'RoboVM RAM Disk' `hdiutil attach -nomount ram://$((SIZE * 2048))`
See http://docs.robovm.com/ for more info
Root pattern javax.annotations.**.* matches no classes
Root pattern javax.inject.**.* matches no classes
Warning: javax.xml.bind.annotation.XmlRootElement is a phantom class!
Warning: java.nio.file.StandardOpenOption is a phantom class!
Warning: java.nio.file.FileSystem is a phantom class!
Warning: java.nio.file.OpenOption is a phantom class!
Warning: java.nio.file.FileSystems is a phantom class!
Warning: com.oracle.jrockit.jfr.TimedEvent is a phantom class!
Warning: com.oracle.jrockit.jfr.EventToken is a phantom class!
Warning: com.oracle.jrockit.jfr.ValueDefinition is a phantom class!
Warning: com.oracle.jrockit.jfr.EventDefinition is a phantom class!
Warning: com.oracle.jrockit.jfr.Producer is a phantom class!
Warning: com.oracle.jrockit.jfr.FlightRecorder is a phantom class!
Daemon stopping because JVM tenured space is exhausted
Daemon stopping because JVM tenured space is exhausted
My IMac is running 8GB of RAM on a 2.7GHz I5.
I have also attempted mounting a RoboVM RAM Disk to no success. Please help!
Usually, the iOS deployment requires a lot of memory, and it is a good practice to increase by default the maximum allocation memory pool of the JVM heap, up until 2 GB.
Running from your IDE, you can set this default value in the Gradle preferences.
For instance, on NetBeans, go to Preferences->Miscellaneous->Gradle, Scripts & Tasks, and as Gradle JVM arguments add -Xmx2048m:
Another option is to set a gradle property in your gradle.properties file (the one with the ANDROID_HOME property, under <user>/.gradle):
org.gradle.jvmargs=-Xmx2048m
This property file will be applied either running from IDE or from command line, so it is more appropriate.
A typical situation where an out of memory error is found is the first time the RoboVM compiler is launched. Luckily, all the compiled classes are cached, so restarting the task just resumes the process.
Also, if the process fails, sometimes can be convenient to stop all the deamon threads with gradle --stop and ./gradlew --stop, and start the task again.
If the process ends successfully, even if there are warning messages, just check on your iOS device that the app was installed and runs fine. Note that you could find memory issues as well, but this is a different issue.
Edit
when running long tasks, it's always convenient using --info to find out more about the process, with a more verbose output.
Also, the process can be run from the console (from NetBeans, right click on the build.gradle file and select Tools->Open in Terminal).
I'm running Docker for Windows (similar effect described below is observable on MacOSX)
I have a Docker Container in which a program tries to access a squashfs image. To access squashfs the kernel has to be either complied with loopdevice support statically or load the relevant kernel module.
When I try to mount the image or setup the loop device the kernel that's shared between docker containers cannot find the loopdevice module.
I could possibly use unsquashfs tool but the squashfs image is used for a reason: squashfs has a very decent property: it allows unlimited number of files and inodes - if I try to to unpack the image I quickly hit the inode limit of the images.
Is Moby Linux kernel which is shipped with docker a statically compiled kernel? What volume to mount to have access to its /lib/modules? lsmod run in a privileged container lists no loaded modules. Trying to modprobe loop yields the following error message:
root#6e1b23cc65e5:/# modprobe loop
modprobe: ERROR: ../libkmod/libkmod.c:556 kmod_search_moddep() could not open moddep file '/lib/modules/4.9.8-moby/modules.dep.bin'
i've installed krusader via macports but i'm unable to run krusader.
w-4jesc:web_cms_dms w-4jesc$ /Applications/MacPorts/KDE4/krusader.app/Contents/MacOS/krusader
KGlobal::locale::Warning your global KLocale is being recreated with a valid main component instead of a fake component, this usually means you tried to call i18n related functions before your main component was created. You should not do that since it most likely will not work
krusader(97879)/KSharedDataCache ensureFileAllocated: This system misses support for posix_fallocate() -- ensure this partition has room for at least 10547296 bytes.
krusader(97879)/KSharedDataCache: Unable to find an appropriate lock to guard the shared cache. This *should* be essentially impossible. :(
krusader(97879)/KSharedDataCache: Unable to perform initial setup, this system probably does not really support process-shared pthreads or semaphores, even though it claims otherwise.
krusader(97879)/KSharedDataCache: Unable to unmap shared memory segment 0x1142d2000
Dynamic session lookup supported but failed: launchd did not provide a socket path, verify that org.freedesktop.dbus-session.plist is loaded!
krusader(97879)/kdeui (kdelibs): Session bus not found
To circumvent this problem try the following command (with Linux and bash)
export $(dbus-launch)
KCrash: Application 'krusader' crashing...
KCrash: Attempting to start /opt/local/lib/kde4/libexec/drkonqi.app/Contents/MacOS/drkonqi from kdeinit
sock_file=/Users/w-4jesc/Library/Preferences/KDE/socket-w-4jesc.local/kdeinit4__tmp_launch-URcTjs_org.x_0
Warning: connect() failed: : No such file or directory
KCrash: Attempting to start /opt/local/lib/kde4/libexec/drkonqi.app/Contents/MacOS/drkonqi directly
QProcess: Destroyed while process is still running.
drkonqi(97882)/KSharedDataCache ensureFileAllocated: This system misses support for posix_fallocate() -- ensure this partition has room for at least 10547296 bytes.
drkonqi(97882)/KSharedDataCache: Unable to find an appropriate lock to guard the shared cache. This *should* be essentially impossible. :(
drkonqi(97882)/KSharedDataCache: Unable to perform initial setup, this system probably does not really support process-shared pthreads or semaphores, even though it claims otherwise.
drkonqi(97882)/KSharedDataCache: Unable to unmap shared memory segment 0x1145ea000
Dynamic session lookup supported but failed: launchd did not provide a socket path, verify that org.freedesktop.dbus-session.plist is loaded!
drkonqi(97882)/kdeui (kdelibs): Session bus not found
To circumvent this problem try the following command (with Linux and bash)
export $(dbus-launch)
dbus is already loaded:
sudo launchctl load -w /Library/LaunchAgents/org.freedesktop.dbus-session.plist
org.freedesktop.dbus-session: Already loaded
has every one a idea?
sudo launchctl load -w /Library/LaunchAgents/org.freedesktop.dbus-session.plist
MacPorts installer specifically says to start dbus without sudo:
# Don't forget that dbus needs to be started as the local
# user (not with sudo) before any KDE programs will launch
# To start it run the following command:
# launchctl load /Library/LaunchAgents/org.freedesktop.dbus-session.plist