While doing a docker pull I get this error message
Handler for POST /images/create returned error: ApplyLayer exit status
1 stdout: stderr: write XXXX: no space left on device
I checked the disk space and the Inodes and I have enough of them. I tried anyway to free some additional space and inodes. Despite this the error falls always on the same file (which rules out a potential space problem).
The image to pull is ~12Gb
Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda4
3,0M 295K 2,7M 10% /
Filesystem Size Used Avail Use% Mounted on /dev/sda4
47G 15G 30G 34% /
I tried exporting the image from another computer and importing it on the computer causing problems. However I get the same error of 'no space left on device'
I tried removing all images, even deleting completely /var/lib/docker, restarting the docker daemon, without any luck.
Running on MacBook 12.11 with ubuntu 14.02 installed.
kernel:
3.18.0-031800-generic #201412071935 SMP Mon Dec 8 00:36:34 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
docker version
Docker version 1.7.1, build 786b29d
docker info
Containers: 2
Images: 85
Storage Driver: devicemapper
Pool Name: docker-8:4-920131-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 6.185 GB
Data Space Total: 107.4 GB
Data Space Available: 33.83 GB
Metadata Space Used: 8.061 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.139 GB
Udev Sync Supported: false
Deferred Removal Enabled: false
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.18.0-031800-generic
Operating System: Ubuntu 14.04.2 LTS
CPUs: 8
Total Memory: 15.58 GiB
Name: geb
ID: TTQN:73Y5:IU35:CK53:UELX:HPRQ:BKMM:ZNNM:O56G:6WL3:RZXE:NV4H
WARNING: No swap limit support
devicemapper has caused me innumerable problems across various different kernels and OSs. Issues here.
Jerome Petazzo has some broader recommendations - although BTRFS also had/has ongoing serious issues, that has got it (and DeviceMapper, and AUFS on Ubuntu 12.04) pulled from production in previous deployments.
My preference for storage drivers is overlay, which is available since kernel 3.18 (further opinions expressed here):
Storage Driver: overlay
Backing Filesystem: extfs
Update /etc/default/docker with (more here):
DOCKER_OPTS="-s overlay"
(ensure you only have one DOCKER_OPTS - that file doesn't support BASH syntax as one may expect)
edit: I seem to recall the devicemapper fix is to delete everything as you have done, and reboot.
Related
Using the cos-stable container optimized OS on GCE. Micro instance so ram is pretty sparse. Tried to enable swap to prevent locking due to OOM during docker pulls, but can't get it to work.
I realize most folders are stateless, so I put the swapfile under home:
sudo fallocate -l 1G /home/user/swapfile
sudo chmod 600 /home/user/swapfile
sudo mkswap /home/user/swapfile
results in:
Setting up swapspace version 1, size = 1024 MiB (1073737728 bytes)
no label, UUID=6e965805-2ab9-450f-aed6-577e74089dbf
But sudo swapon /home/user/swapfile gives the error:
swapon: /home/user/swapfile: swapon failed: Invalid argument
Any ideas how to enable swap on COS?
Disk based swap is disabled in the COS image.
You can enable disk based swap with
sysctl vm.disk_based_swap=1
I have the following in my cloud-init:
bootcmd:
- sysctl vm.disk_based_swap=1
- fallocate -l 1G /var/swapfile
- chmod 600 /var/swapfile
- mkswap /var/swapfile
- swapon /var/swapfile
Swap is not supported in container optimized OS
Swap would effectively destroy much of the behavioral isolation Google offers between containers.
Guaranteed pods should never require swap. Burstable pods should have their requests met without requiring swap. BestEffort pods have no guarantee.
I highly suggest you use a bigger instance as a f1-micro only has 600MB of RAM and you still need to run the OS on the instance it addition with your containers
I have a production server running CentOS 6.9 on Alicloud in China. Instance is of ecs.sn1.3xlarge type. Recently one of my data disk became filled-up. So I decided to resize the volume and followed the step by step instructions available on this page: https://www.alibabacloud.com/help/doc-detail/25452.html.
Here are steps that I followed:
Resized disk form console
Rebooted system (Rebooting system didn't resized/populated disk on system)
umount disk
Run fdisk on desired disk
e2fsck -f /dev/vdb1 # check the file system
resize2fs /dev/vdb1 # resize the file system
Thank you in Advance
I followed these instructions to resize my VirtualBox Ubuntu VM on Mac:
http://osxdaily.com/2015/04/07/how-to-resize-a-virtualbox-vdi-or-vhd-file-on-mac-os-x/
This is after the change:
*****-M-D2KA:$ VBoxManage showhdinfo ~/VirtualBox\ VMs/P4_Runtime/P4_Runtime.vdi
UUID: ce0ccd77-f265-46cd-9679-e25e64f1c992
Parent UUID: base
State: locked read
Type: normal (base)
Location: /Users/*****/VirtualBox VMs/P4_Runtime/P4_Runtime.vdi
Storage format: VDI
Format variant: dynamic default
Capacity: 25000 MBytes
Size on disk: 9967 MBytes
Encryption: disabled
In use by VMs: P4_Runtime (UUID: 5ea52b11-997f-45d8-b7d6-effa37a3b649) [Snapshot 1 (UUID: 409c1035-2134-4532-a931-a29018d33dc6)]
Child UUIDs: 540ae750-5307-44ef-a313-95134ae353b7
165fe99e-490d-4dd9-9602-00e3aaa8f82c
But for some reason, it does not seem to take effect:
This is the "df -k" output in the VM, and I get "No space left on device" error:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda 10253588 9713020 0 100% /
What am I missing?
I found out what I missed. I used gparted to resize the partition.
Resizing the VHD doesn't change the size of the partition /dev/sda. You can run lsblk inside the guest to see the additional space. To get the extra space that is available in the guest OS, you may
Use something like gparted as mentioned here. Instructions to do that on VHD can be found here. Note that this might not be easiest, but you may be forced to if you plan to not move some of your mount points (Example if you're not ready to move /home/ to a new partition).
Or, create a new partition, again instructions on how to do is present here. I would prefer this option over the first.
I am trying to add more swap space in docker in order to avoid this error installing oracle database:
This system does not meet the minimum requirements for swap space.
Based on the amount of physical memory available on the system, Oracle
Database 11g Express Edition requires 2048 MB of swap space. This
system has 1023 MB of swap space. Configure more swap space on the
system and retry the installation.
I am following the instructions commented here:
https://forums.docker.com/t/docker-for-mac-configure-swap-space/20656/2
but when I execute mkswap I get "command not found":
mkswap /var/swap.file
Any idea?
Docker for Mac runs an Alpine Linux VM to host containers.
This is a prebuilt boot image that is designed for ease of use, and also updates over time so it can be hard to customise some times as most config is reset when you reboot it.
In this case you can persist a swap file change, but config like this has the possibility of changing between versions without notice. You might be better off running a custom VM for this so your swap configuration hangs around.
Docker for Mac 17.06.0
Swap is controlled by the do_swapfile function in the /etc/init.d/automount init script in the VM. If the swap file exists, it will be used as is. As the swap file is stored in /var it is persisted across reboots and can be manually customised.
Attach to the VM's tty from your mac with screen (brew install screen if you don't have it)
screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
Then in the VM, replace the existing swap file with a new one of the required size and reboot the box. The size of the file is the block size bs * count.
swapoff -a
dd if=/dev/zero of=/var/spool/swap bs=1k count=2097152
chmod 600 /var/spool/swap
mkswap /var/spool/swap
reboot
When the VM has rebooted, you should be able to connect again and see the new size of the VM's Swap space with free.
$ screen ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
/ # free
total used free shared buffers cached
Mem: 3526164 389952 3136212 165956 20968 208160
-/+ buffers/cache: 160824 3365340
Swap: 2097148 0 2097148
Have a few applications where EC2 small instances are, well, too large. So the announcement of micro instances is just what the doctor ordered.
I'd like to take a small instance's EBS volume, detach it, and pair it up with a micro instance. At some point it might be great to go the other way and upsize a micro instance to a small or beyond.
For this failed experiment I tried:
Creating a new small instance with the Alestic Ubuntu 10.04 32 bit AMI (ami-1234de7b). Boots like a charm.
Power down my freshly minted micro instance, detach the volume that was created for me in the previous step.
Attach the small instance's volume to the micro instance.
Power up.
Nada.
What's odd is there is no console log output until I power down. Then I see it all.
[ 0.000000] Reserving virtual address space above 0xf5800000
[ 0.000000] Initializing cgroup subsys cpuset
[ 0.000000] Initializing cgroup subsys cpu
...
[ 1.221261] VFS: Mounted root (ext3 filesystem) readonly on device 8:1.
[ 1.221261] VFS: Mounted root (ext3 filesystem) readonly on device 8:1.
[ 1.222164] devtmpfs: mounted
[ 1.222202] Freeing unused kernel memory: 216k freed
[ 1.223409] Write protecting the kernel text: 4328k
[ 1.223760] Write protecting the kernel read-only data: 1336k
init: console-setup main process (63) terminated with status 1
%Ginit: plymouth main process (45) killed by SEGV signal
init: plymouth-splash main process (196) terminated with status 2
cloud-init running: Thu, 09 Sep 2010 17:37:54 +0000. up 2.61 seconds
mountall: Disconnected from Plymouth
init: hwclock-save main process (291) terminated with status 1
Checking for running unattended-upgrades: * Asking all remaining processes to terminate...
[80G
[74G[ OK ]
* All processes ended within 1 seconds....
[80G
[74G[ OK ]
* Deconfiguring network interfaces...
[80G
[74G[ OK ]
* Deactivating swap...
[80G
[74G[ OK ]
* Unmounting local filesystems...
[80G
[74G[ OK ]
* Will now halt
[ 185.599636] System halted.
This method of swapping has worked well between same sized instanced in the past and it's my first attempt at doing this between different sizes. Is this just not possible or am I missing something fundamental in my EC2 knowledge?
Even though this will probably be migrated to Server Fault, I ran into the exact same problem with this instance earlier today.
It appears that this image assumes that there will be ephemeral storage present, when there is none on the micro instances. To work around this, comment out the following line in /etc/fstab:
/dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0
This should prevent the instance from hanging on startup, or at least it did for me with ami-1234de7b.
I created a new micro instance using alestic ami's (ami-2c354b7e). I was able to login to the system normally the first time. But once I reboot the system, I was not able to login again.
commenting the line indicated above worked for me. "/dev/sda2 /mnt auto defaults,comment=cloudconfig 0 0"
Commenting the line out doesn't fix it fully. If you reboot, it will write the same line back in. You need to:
$ l="deb http://archive.ubuntu.com/ubuntu lucid-proposed main"
$ echo "$l" | sudo tee -a /etc/apt/sources.list
$ sudo apt-get update && sudo apt-get install cloud-init
$ dpkg-query --show cloud-init
I'm assuming this will be fixed in the official Ubuntu release soon and you won't have to do this, but for now...
Source: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/634102
Also, we have a couple images based off the official Ubuntu AMI's that work on Micro's: http://blog.simpledeployr.com/2010/09/new-ruby-amis-with-latest-ubuntu-lucid.html
I don't see a problem on your side. This could be a problem in Amazon's infrastructure.