I use Vagrant on a MacOS with an ubuntu64 16.04. Running htop, I can see vagrant ssh process can use virtually 530G (in VIRT Column).
Is it the normal behavior of Vagrant? Should I panic? Is it "normal" to have virtually 530G on a mac with 120G of disk and 16G of RAM? Or maybe did I not understand the meaning of VIRT?
The vagrant box runs on virtual box and has only 1G of RAM allocated.
Answer by chrisroberts on github:
Hi! I was able to reproduce this behavior, but with any vagrant command executed. The vagrant ssh command is the easiest to see this behavior simply because the process is left running for as long as the ssh session is alive.
The tl;dr version of below is simply: Don't worry about it. VIRT isn't allocated memory. If it were, you would either need massive swap space, or nothing would be working.
So, what's going on here? The vagrant installer includes a small go executable (vagrant) whose job is to setup the current environment with the proper locations of everything it needs. The installers bin directory, the lib directory for ruby and all the other friends, all the gems, and the vagrant gem itself. Once it has all this configured, it spawns off a new process, the actual Ruby vagrant process.
Because your example was referencing vagrant ssh, and as was previously pointed out (#7296 (comment)) a Kernel.exec happens meaning the Ruby process does not persist, I figured it must be the wrapper that was the culprit. After a bit of searching (mostly to find stackoverflow items saying "don't worry about VIRT") I stumbled upon:
keybase/keybase-issues#1908
They refer to the golang FAQ that talks about a bunch of VIRT being claimed up front and it not being a big deal, but never any absolutes about how much was actually being claimed. A link to lwn was dropped in there (keybase/keybase-issues#1908 (comment)) regarding golang's behavior on startup of claiming a huge chunk of VIRT, but still everything referenced a much lower amount than I was seeing locally. So I decided to go dig into the golang runtime code, and within malloc.go we find the answer:
golang src/runtime/malloc.go
The why it's happening is because of the go wrapper used to start vagrant. Because the VIRT you see is simply a reservation and not actually allocated, it's not a problem and not something that should be worried about.
(There are some interesting conversations on the golang ML around the pros and cons of this approach, all pretty great reads).
It's just a copy/paste (and bolded the TLDR), hope it could help someone else.
Related
I'm trying to reduce the amount of forking our Ruby/Rails app does. We shell out a lot with backticks, and each of these forks the entire process, which can cause a huge memory bloat.
I'm going through, identifying the ones that get called the most, and trying to replace them with code which achieves the same thing without making a shell call. However, in some cases I suspect it might still be forking under the hood anyway.
Is there a way to detect or log whenever a process forks? I'm using ubuntu 14.04. A log would be ideal as I can then keep an eye on it when I run the amended code.
I am working on Laravel with docker.
If I run php unit test in mac os, it takes few seconds.
However on windows 10, it takes few mins.
Is there anyway to fix this problem?
Thanks.
If you're running on a non-Linux OS, Docker has to virtualise your file system, and this requires a certain amount of time per file. For programs that are compiled into one executable, this is less of a problem at runtime (but clearly with its own compilation-time implications), but for scripting languages like PHP this can mean that every request runs super slowly, since every file that is used has to be 'translated' every time it is read. This is also a problem on Docker for Mac (so you're actually experiencing problems there, too, but less so, since at least it's a Linux system under the hood.) Linux is, I believe, completely virtualised on Windows which is going to add even more time.
This Reddit discusses the problem to an extent:
https://www.reddit.com/r/docker/comments/7xvlye/docker_for_macwindows_performances_vs_linux/
With this being particularly interesting (I have not tried it myself):
https://nickjanetakis.com/blog/setting-up-docker-for-windows-and-wsl-to-work-flawlessly
There is also a good community-created solution which we have used to solve our Docker for Mac problem. I don't see why their Windows options wouldn't work similarly well in your case. You can find it here:
https://github.com/EugenMayer/docker-sync/wiki/docker-sync-on-Windows
It basically sets up an intermediate service that copies all the files over into an intermediate volume (that uses the 'correct' filesystem) only when the file is updated, therefore speeding up run speed immensely.
I know it looks like quite an intimidating process, but this problem is fundamental, so you're going to have to do a certain amount of work to fix things!
FWIW I had that working on Docker 4 Mac, but it added a layer of complexity to our dev process that I found annoying, so in the end I've got myself a Linux box for work. To be honest, installing Linux as dual boot on my Windows machine (which has been my at-home solution) was probably easier than tweaking Docker 4 Mac to my satisfaction, so you might want to consider that. I have used this page twice:
https://itsfoss.com/install-ubuntu-1404-dual-boot-mode-windows-8-81-uefi/
And it's worked fine each time. One caveat - it suggests a low amount of disk for your root (/) volume, but Docker gets mounted on root so give it around 100G (not the 10-20G that page recommends.)
Recently My ICp 2.1.0.1 environment (built on openStack VM) master node become very slow. Actually it is the Linux VM that very slow, not only for ICP product, even some general simple Linux command (ls, pwd, cd, etc.).
The thing here is, I didn't even use this environment, no workload, system completely idled.
I used top to monitor the CPU usage but didn't find long running processes taking long time.
How is that happen?
Note, this same issue occur at least twice. I just setup the env and leave it there.
Although the root cause is not yet identified, but simply restart the cluster solved the issue:
https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0/manage_cluster/restart_cluster.html
The restarting of docker takes several hours to finish. But finally it becomes fast.
I'm not sure if the same issue might happen again in the future. Monitoring..
Note that I didn't reboot the Linux VM as I found after rebooting Linux last time, some docker containers cannot startup successful, which lead me to check below part:
https://www.ibm.com/support/knowledgecenter/SSBS6K_2.1.0/troubleshoot/restart_master_console.html
From ICP 2.1.0 release, there are some new features added, you need to ensure your VM hardware configuration matches the hardware requirements listed in below link, thanks.
https://www.ibm.com/support/knowledgecenter/en/SSBS6K_2.1.0/supported_system_config/hardware_reqs.html
I'm trying to use docker beta on OSX, mainly for Symfony development but the mounted volumes are incredible slow. Even for a vanilla Symfony project I get 6s page load time. That's unbearable! Has anyone found a solution to this issue? Trying to move away from vagrant but I just can't find any reasonable way to work with docker instead.
Okay the user Spiil gave a solution but I wanted to elaborate on the exact steps to take since I went through 12 hours trying to figure it out, but once you know how its super easy and fixes all the slow down issues!
The key here is to understand this solution creates NFS (Network File System) drives as the means of communication from the Docker Containers to your Mac instead of the standard OSX File System which is very slow currently either due to bugs or the way it works*
Follow these steps exactly.
1.) Clone this repo here (https://github.com/IFSight/d4m-nfs) in your home directory. To do this open up terminal and type cd ~
Then type git clone https://github.com/IFSight/d4m-nfs
Alternatively you can also do this in a one liner git clone https://github.com/IFSight/d4m-nfs ~/d4m-nfs
2.) Next go into the d4m-nfs folder and create a new file in the /etc folder and title it d4m-nfs-mounts.txt
3.) Add the following lines of code to this.
/Users/yourusername:/Users/yourusername:0:0
What the above does is allows you to still use relative folders with docker-compose and allows all ports to connect on it hence the 0:0.
EDIT
Do not put /Volumes here!!
4.) Go to your docker preferences and do the following
Make sure only /tmp is showing and NOTHING ELSE. I mean nothing else it won't work if there is anything else since it will create conflicts with the NFS systems that the script will make for you later. Restart docker and docker-compose down any containers as well.
5.) Finally navigate to the d4m-nfs directory we created in step 1 and type the following command, /bin/bash d4m-nfs.sh
edit The correct way to type the command above is this as another user from the github (if-kenn) pointed out, ./d4m-nfs.sh which uses the Shebang for what shell should run it.
If done correctly there should be no errors and this should work. Please note DO NOT RUN as sh d4m-nfs.sh this will create errors and you will have to delete your exports file to start over. In fact anytime you make any changes you will have to clear your exports file.
This is what mine looks like.
EDIT:: IMPORTANT -- Remove the /private and volumes! This should only be users/username now!
If you see anything other than this you were not running with bash. You can quickly get to the exports file like this in Mac if you make any errors and just clear it out to start over.
Just select go to folder
and then type /etc/exports
This is a nice shortcut to quickly get to it and clear it out in your favorite text editor.
Also make sure no containers are running or you will get the ........ loop of death. If this loop of death continues make sure you upgrade docker and then restart your computer. Yes restart... it seemed to be the only way to get it to work on my friends computer. Refer to this (https://github.com/IFSight/d4m-nfs/issues/3)
Note to .... loop. I recently found another solution. Make sure you are NOT logged as root, and make sure you pulled the git repo into your users ~ folder not the root ~ folder. In otherwords, it should be in Users/username.
Also, make sure /tmp folder has full write permissions since the script needs to write here or this won't work either. chmod 777 -R /tmp
6.) If you did it right when running the script it will look like this.
Then simply run your docker-compose up -d as usual in your symfony project folder (or whatever project you are using with docker) and everything should work... except NO MORE slow downs!
You will need to run this anytime you restart your computer or docker.
Also note if you get mounting errors showing up, you probably don't have your project stored in your Users/username directory. Remember that is where we mounted it. If your project is somewhere other than there you will need to modify the d4m-nfs-mounts.txt file accordingly.
Other Info:
For people reading this now, maybe it's better to wait for Docker to fix this issue. A pull request has already been accepted to improve performance(https://github.com/docker/docker/pull/31047).
This will be release somewhere in April 2017 and should be a big improvement.
I've tried some workarounds for Docker for Mac, but all of them had some pretty big disadvantages, mostly in usability. A good source for alternatives of the OSXFS can be found at: https://github.com/EugenMayer/docker-sync/wiki/Alternatives-to-docker-sync. Credits for Eugen Mayer for setting this up.
EDIT:
First improvement is implemented in the edge release. https://github.com/docker/for-mac/issues/77 has more info on this.
There's a long thread with explanation from Docker Team and various workarounds.
Currently, the issue is being tracked on GitHub.
While some workarounds may be better than others, I'm afraid the ideal option for now is to switch to Linux.
I spent a lot of my time in searching viable solution. And I found.
d4m-nfs
allow you use docker volumes via nfs.
In my case it gave increase performance 16 times! (1.8sec vs ~30sec)
Also d4m-nfs has quite a intricate manual, so here is another link with detailed example: https://github.com/laradock/laradock/issues/353#issuecomment-262897619
I just leave this here for other googlers.
Normaly volumes should be fast.
But you can not change anything to make them faster if you dont want to change the format of your disk.
But maybe the bottleneck is the CPU or RAM.
You can check that with the command docker stats. These are by default set to 2 cores and 2 GB RAM. You can change this in the Docker for Mac GUI.
I had exactly the same thing. For me using docker-bg-sync (see on GitHub) made a dramatic improvement in speed and CPU usage.
Not as nice as just mounting the volume as you have to start a new container for every sync but it does the job.
In latest docker 17.06.0-ce-mac18 volumes mounted with :cached seems to run quite decent.
I've found that creating a CoreOS VM under Parallels, then using the Docker that is inside CoreOS is far faster than Docker for Mac (currently running Version 17.12.0-ce-mac49 (21995)).
I'm doing Linux code builds using CMAKE/Ninja/GCC and it's almost twice as fast as the exact same build from Docker for Mac.
In my case, I have a ton of library sources that are part of the container (e.g. Boost, OpenSSL), and a decent amount of C++ code that I keep local to my Mac.
This seems to be a recent development. Docker/Mac has become much slower than I remember it being a month or two ago. Maybe it's just me...
We overcame this issue by synchronizing the local and the docker for mac filesystem using syncthing. We built an open source tool that follows this approach, in case it helps:
https://github.com/okteto/cnd
Here is my situation, when I download Gentoo and start to run it and downloaded the stage III Tarball from links and then tried to extract it a stream of white sentences flows down my screen really fast for about a minute just like in the YouTube tutorial I was viewing. However, after that instead of going to the correct stage it says cannot right not enough space on device and I tried repartitioning it but I'm not sure what device it is talkingaboutHowever, after that instead of going to the correct stage it says cannot right not enough space on device and I tried repartitioning it but I'm not sure what device it is talking about. Please help
Sorry you're having this issue, though in general, I truly believe Gentoo Handbook is quite well written and even a newbie can follow it... Here are some advices that I hope I can give you (most important is, digest the handbook and follow it carefully, not that I'm saying "RTFM", it's just that for Gentoo, handbook is essential and without it, we can get lost if you're just starting).
From my experiences, the "stream of white sentences" I'd presume would be from verbose un-tar'ing your stage3. Usually, I only want to see the errors so my suggestions is to remove the "v" (i.e. from "tar xjvpf" to "tar xjpf") so that only errors would appear when un-tar'ing. The caveat to this is that you'll be wondering if it hung or is busy un-tar'ing. Use Alt-F1 and Alt-F2 (if on console/tty mode back-and-forth) to log in on another TTY and do 'ps -auxf' to see if it's still tar'ing. If you're using GUI Terminal, just open another tab and 'ps auxf', you get the picture...
Also, learn the commands "df", it'll come in handy. If you're running out of disk space, perhaps you're trying to install/untar stage3 to your ramdisk (grin) rather than your mounted (i.e. "/mnt/gentoo"). Mount your root '/' device to '/mnt/gentoo' and cd to that mounted path then try it (don't forget to mount your '/boot' as well as your proc, dev, sys, etc before you chroot - again, follow the handbook as carefully as you can - oh also, distro such as Debian hybrids including Ubuntu uses symlink to shm, so read that part about 'rm /dev/shm' and follow it carefully; if you're using Gentoo LiveCD, you can ignore that part).
Other useful commands if you're confused (or new to) mounting devices would be to learn to experiment with commands such as 'lsblk' and 'mount' (by itself) to inspect the sizes of your partition (again, use of 'df' comes in handy as well) as well as what is your device (i.e. /dev/sda1 versus /dev/sdb1). Hint: when you do 'mkfs', use "-L" (or for some file system, it's "-N") to label/name your devices, so that when you use commands such as 'mount' or 'lsblk', you can spot them easier. If you're using GUI/desktop versions of some distro, hopefully there are tools such as "gparted" which can give you visual information in GUI of your devices which can be helpful. One think I'd advise you to stay away from if you're just starting, is to avoid RAID (i.e. mdadm) until you're comfortable with how grub/lilo works. Get your kernel (Gentoo-sources) compiled and MBR written (i.e. grub-install), try booting and have fun first (oh also, if you can avoid GUI like installing Gnome/KDE from the get-go, avoid it as well - you'll get into issues such as "should I use SystemD or OpenRC" and then get hit by the obstacle of some gnome parts needs you to use systemd but you've chosen openrc, and so on).
If I may add my opinions, in my opinion, Gentoo (also Arch and FreeBSD) is an excellent place to start if you want to learn the inside of Linux application workings (library dependencies, why packages are important rather than downloading each libs manually and compile them one by one, etc). I hope this won't discourage you from switching to another distro, but if it does frustrate you on installation and all you want to do is test-drive Linux, there are much easier distro that you'd not have to understand USE and other compilation mechanisms (if you have an old i586, it makes sense to build it with pick-and-chose libraries so that leaner can be faster, but if you have fast machine, why compile binaries when somebody who is expert at it already have done it for you?). SUSE and Fedora/RedHat/CentOS used to be the least frustrating for it was able to find/detect hardwares (legacy and new) but these days, I usually tell people, "if you know how to install Windows, you can install Ubuntu" so that too may be a good way to wet your feet. Good luck!
0_o wow, well.. how about some 411 like size of your hdd and exactly how you partioned it? Linux will look for specific directorys and if missing will instead start to install into the root dir. How you partion is an importent first step. Once you got a generally good partion setup most linux installs will go fine. Most basic tables include /, /home,/var and a swap.