I've found out the process [sync_supers] running twice, using 100% of cpu each of those.
It was triggered by the user share which is a user to access a share folder used by Samba users. The user share has access only to /home/share.
lucas#arturito:~$ cat /etc/passwd | grep share
share:x:1002:1002:Share,,,:/home/share:/bin/bash
tomcat7:x:115:125::/usr/share/tomcat7:/bin/false
I've never seen that process before and as per the stats I got from Munin, it's been running for an hour or so.
I've found out the process [sync_supers] running twice, using 100% of cpu each of those.
capture of htop
It was triggered by the user share which is a user to access a share folder used by Samba users. The user share has access only to /home/share.
lucas#arturito:~$ cat /etc/passwd | grep share
share:x:1002:1002:Share,,,:/home/share:/bin/bash
tomcat7:x:115:125::/usr/share/tomcat7:/bin/false
I've never seen that process before and as per the stats I got from Munin, it's been running for an hour or so.
munin stats
What's the sync_supers process? Is my box compromised?
I've ran chkrootkit, rkhunter and debsums and everything seems to be ok ...
I'm running:
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.5 LTS
Release: 14.04
Codename: trusty
Linux arturito 3.13.0-100-generic #147-Ubuntu SMP Tue Oct 18 16:48:51 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
I killed both the processes and they went away.
I'm kind of concerned about this. Is there anything else that I should do/check?
Thanks!
Lucas
Related
$ time sudo dbus-daemon --system
real 1m30.111s
user 0m0.017s
sys 0m0.003s
Barebone ArchLinux inside docker on ArchLinux.
D-Bus Message Bus Daemon 1.12.16
Tried dbus-x11 from AUR, same. Every time.
Edit/Details: the sudo invocation above takes 1:30 to execute, but the actual dbus-daemon process is spawned right away, and continues to run during and after the 1:30, successfully (i.e. it works). Reason I need dbus-daemon? for avahi-daemon (more specifically, to be able to run avahi-browse --all and discover stuff on my network).
Edit2: seems even though 'everything works' despite this slowness (avahi, network service discovery etc), the container becomes dead slow. Barely running sudo echo 'something' takes 25 seconds (a figure perhaps related to a timeout of 25000 inside /usr/share/dbus-1/system.conf). Just like an infection. For what it's worth, after reading more, seems the frustration of needing dbus is not restricted to the world of containerization - plenty of articles/communities like this and this.
I hit this issue with various Docker images, but not always. I digged deeper in this issue and found an interessting comment on the systemd repo.
The images I'm currently working on had systemd configured on passwd and group:
$ cat /etc/nsswitch.conf
passwd: sss files systemd
group: sss files systemd
Then I removed the systemd provider ($ sed -i 's/ systemd//g' /etc/nsswitch.conf) and the 90s hang was gone when starting with dbus-daemon --system --nofork. This was really a PITA to find out.
I could also verify that exactly this was the issue/difference for another Docker image I'm maintaining.
I am following the CoreOS in Action book (and also CoreOS online instruction) to bring up a 3-node cluster using Vagrant and VirtualBox on MacOS.
It all goes fine, machines come up & running and I can ssh into one of them, but it looks like the boxes brought up are missing fleetctl (which makes no sense, as it's such a core component of CoreOS):
$ vagrant ssh core-01 -- -A
Last login: Thu Mar 1 21:28:58 UTC 2018 from 10.0.2.2 on pts/0
Container Linux by CoreOS alpha (1702.0.0)
core#core-01 ~ $ fleetctl list-machines
-bash: fleetctl: command not found
core#core-01 ~ $ which fleetctl
which: no fleetctl in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/opt/bin)
What am I doing wrong?
I have changed the number of instances to 3, created a new "discovery token URL" and updated the user.data file; Googling around I seem to be the one and only person having this problem.
Thanks in advance for any suggestions you may have!
PS -- yes, I have tried (several times!) to vagrant destroy and rebuild the cluster: even nuked the repo and re-cloned it. Same issue every time.
The answer is going to make you a bit sad, here it is:
CoreOS no longer support fleet. It's gone. Ciao :(
https://coreos.com/blog/migrating-from-fleet-to-kubernetes.html
To this end, CoreOS will remove fleet from Container Linux on February 1, 2018, and support for fleet will end at that time. fleet has already been in maintenance mode for some time, receiving only security and bugfix updates, and this move reflects our focus on Kubernetes and Tectonic for cluster orchestration and management.
You are using Coreos 1702.0.0, fleet has been removed since Coreos 1675.0.1 https://coreos.com/releases/
I really don't understand here when I entered vagrant ssh for the first time in the terminal. I'm using laravel Homestead.
output was:
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.13.0-30-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information disabled due to load higher than 1.0
Get cloud support with Ubuntu Advantage Cloud Guest:
http://www.ubuntu.com/business/services/cloud
Last login: Fri Oct 3 01:24:38 2014 from 10.0.2.2
I really don't understand:
Last login: Fri Oct 3 01:24:38 2014 from 10.0.2.2 //because its first time I entered vagrant ssh
and this:
System information disabled due to load higher than 1.0
The last login you see may come from when the Vagrant Box was originally created. For example I just launched a new Vagrant VM based on hashicorp/precise64 and the "last login" showed as September 2012. Subsequent logins will show the last time that you logged in.
Regarding the system information being disabled, see this ServerFault question: System information disabled due to load higher than 1.0 amazon ec2.
Both of the above can safely be considered normal behavior.
Using a free 'micro' instance from Amazon to fire up a quick demo of MarkLogic. The rpm installs fine with no errors.
Some information that may be helpful:
[user#aws ~]$ rpm -qa | grep release
redhat-release-server-6Server-6.4.0.4.el6.x86_64
[user#aws ~]$ rpm -qa | grep MarkLogic
MarkLogic-7.0-1.x86_64
Starting the MarkLogic server for the very first time shows this:
[user#aws ~]$ sudo /etc/init.d/MarkLogic start
Initialize Configuration
Region: us-west-2 ML_NAME:
Set configuration: MARKLOGIC_ZONE="us-west-2c"
Instance is not managed
Waiting for device mounted to come online : /dev/xvdj
And here it sits with no other messages anywhere including /var/opt/MarkLogic/Logs which doesn't exist yet.
Even though Micro instances aren't officially supported, you can usually start one up. But, reports are that you will be quickly wishing you didn't.
That said, see the precise instructions at http://developer.marklogic.com/products/aws and, in particular, a disk at mounting /dev/sdf ; the server init script will wait forever to come up if you don't do that.
If the above didn't help, I've dug into the RPM enough to discover some issues on AWS.
For one, they use some sysconfig scripts to detect if they're on AWS. If you're running MarkLogic 6, these sysconfigs have a hardcoded drive and will wait indefinitely since it probably won't exist. Yours is 7, and still has some issues on AWS. To bypass this, you can make a /usr/bin/is-ec2.sh that contains:
#!/bin/bash
exit 1
That will prevent it from doing any ec2 detection. More details can be found on my write-up at this github post
Have a few applications where EC2 small instances are, well, too large. So the announcement of micro instances is just what the doctor ordered.
I'd like to take a small instance's EBS volume, detach it, and pair it up with a micro instance. At some point it might be great to go the other way and upsize a micro instance to a small or beyond.
For this failed experiment I tried:
Creating a new small instance with the Alestic Ubuntu 10.04 32 bit AMI (ami-1234de7b). Boots like a charm.
Power down my freshly minted micro instance, detach the volume that was created for me in the previous step.
Attach the small instance's volume to the micro instance.
Power up.
Nada.
What's odd is there is no console log output until I power down. Then I see it all.
[ 0.000000] Reserving virtual address space above 0xf5800000
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
...
[ 1.221261] VFS: Mounted root (ext3 filesystem) readonly on device 8:1.
[ 1.221261] VFS: Mounted root (ext3 filesystem) readonly on device 8:1.
[ 1.222164] devtmpfs: mounted
[ 1.222202] Freeing unused kernel memory: 216k freed
[ 1.223409] Write protecting the kernel text: 4328k
[ 1.223760] Write protecting the kernel read-only data: 1336k
init: console-setup main process (63) terminated with status 1
%Ginit: plymouth main process (45) killed by SEGV signal
init: plymouth-splash main process (196) terminated with status 2
cloud-init running: Thu, 09 Sep 2010 17:37:54 +0000. up 2.61 seconds
mountall: Disconnected from Plymouth
init: hwclock-save main process (291) terminated with status 1
Checking for running unattended-upgrades: * Asking all remaining processes to terminate...
[80G
[74G[ OK ]
* All processes ended within 1 seconds....
[80G
[74G[ OK ]
* Deconfiguring network interfaces...
[80G
[74G[ OK ]
* Deactivating swap...
[80G
[74G[ OK ]
* Unmounting local filesystems...
[80G
[74G[ OK ]
* Will now halt
[ 185.599636] System halted.
This method of swapping has worked well between same sized instanced in the past and it's my first attempt at doing this between different sizes. Is this just not possible or am I missing something fundamental in my EC2 knowledge?
Even though this will probably be migrated to Server Fault, I ran into the exact same problem with this instance earlier today.
It appears that this image assumes that there will be ephemeral storage present, when there is none on the micro instances. To work around this, comment out the following line in /etc/fstab:
/dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0
This should prevent the instance from hanging on startup, or at least it did for me with ami-1234de7b.
I created a new micro instance using alestic ami's (ami-2c354b7e). I was able to login to the system normally the first time. But once I reboot the system, I was not able to login again.
commenting the line indicated above worked for me. "/dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0"
Commenting the line out doesn't fix it fully. If you reboot, it will write the same line back in. You need to:
$ l="deb http://archive.ubuntu.com/ubuntu lucid-proposed main"
$ echo "$l" | sudo tee -a /etc/apt/sources.list
$ sudo apt-get update && sudo apt-get install cloud-init
$ dpkg-query --show cloud-init
I'm assuming this will be fixed in the official Ubuntu release soon and you won't have to do this, but for now...
Source: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/634102
Also, we have a couple images based off the official Ubuntu AMI's that work on Micro's: http://blog.simpledeployr.com/2010/09/new-ruby-amis-with-latest-ubuntu-lucid.html
I don't see a problem on your side. This could be a problem in Amazon's infrastructure.