Following up on
docker no space left on device macOS
I found that docker needs huge space on macOS
This is for creating the following docker images (with packer):
CREATED SIZE
6 hours ago 81.7GB
8 hours ago 80.5GB
14 hours ago 230MB
14 hours ago 153MB
I.e., it is taking nearly 1/3 of my total disk spaces, and I need to allocate nearly half of my total disk spaces. Why it is taking so much space?
I've been building the same images under Linux and Windows and never had I been running out of allocated space and need to allocate more disk space again and again before.
% docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 4 0 162.5GB 162.5GB (100%)
Containers 0 0 0B 0B
Local Volumes 2 0 350.5MB 350.5MB (100%)
Build Cache 5 0 0B 0B
% df -h
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk1s1 932Gi 10Gi 555Gi 2% 488001 9767490159 0% /
devfs 194Ki 194Ki 0Bi 100% 673 0 100% /dev
/dev/disk1s2 932Gi 363Gi 555Gi 40% 712613 9767265547 0% /System/Volumes/Data
/dev/disk1s5 932Gi 2.0Gi 555Gi 1% 2 9767978158 0% /private/var/vm
map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /System/Volumes/Data/home
map -fstab 0Bi 0Bi 0Bi 100% 0 0 100% /System/Volumes/Data/Network/Servers
/dev/disk1s4 932Gi 507Mi 555Gi 1% 54 9767978106 0% /Volumes/Recovery
/dev/disk2s1 1.8Gi 1.5Gi 323Mi 83% 17770 4294949509 0% /Volumes/Docker
The above reports look reasonable to me, where have the remaining ~150G space gone to?
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 1 year ago.
Improve this question
I'm using Windows 10 with Docker on WSL2 (With Ubuntu20 Linux Subsystem).
I originally got all this installed on my C: disk, which was a 256Go Drive.
I've change my drive for a 1To drive and I extended my system partition. In windows all is fine but I still encounter "no space left on device" issues inside WSL2.
How can I resize my Linux disk so I can use more space?
Filesystem Type Size Used Avail Use% Mounted on
/dev/sdb ext4 251G 18G 221G 8% /
tmpfs tmpfs 4.9G 400M 4.5G 9% /mnt/wsl
tools 9p 931G 269G 663G 29% /init
none devtmpfs 4.9G 0 4.9G 0% /dev
none tmpfs 4.9G 8.0K 4.9G 1% /run
none tmpfs 4.9G 0 4.9G 0% /run/lock
none tmpfs 4.9G 0 4.9G 0% /run/shm
none tmpfs 4.9G 0 4.9G 0% /run/user
tmpfs tmpfs 4.9G 0 4.9G 0% /sys/fs/cgroup
C:\ 9p 931G 269G 663G 29% /mnt/c
D:\ 9p 466G 6.6G 460G 2% /mnt/d
E:\ 9p 1.9T 85G 1.8T 5% /mnt/e
/dev/sdd ext4 251G 164G 76G 69% /mnt/wsl/docker-desktop-data/isocache
none tmpfs 4.9G 12K 4.9G 1% /mnt/wsl/docker-desktop/shared-sockets/host-services
/dev/sdc ext4 251G 130M 239G 1% /mnt/wsl/docker-desktop/docker-desktop-proxy
/dev/loop0 iso9660 396M 396M 0 100% /mnt/wsl/docker-desktop/cli-tools
I have created an c3.2xlarge EC2 instance with the store volume specified as 2 x 80 GB (160 GB). But when I use df -H command, this is what i see, and there is not enough storage as specified.
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 62k 7.9G 1% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
/dev/xvda1 8.4G 1.1G 7.2G 14% /
I need an EC2 instance to have at least 80 gigs of storage, which instance should I choose?
Thanks for the points in the comments.
The problem was; I used EC2 Management Console and didn't add the volumes when I created the cluster, I terminated that cluster, created a new one, on Storage page Added New Volume, chose the volume type as Instance Store 0.
[]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdb 202:16 0 75G 0 disk /media/ephemeral0
Now the 80 Gig volume is there.
I'm using Hadoop YARN on EC2 over r3.xlarge instances, I launched the instances from an AMI using spark-ec2 scripts.
On https://aws.amazon.com/ec2/instance-types/, the specifications of r3.xlarge are the following:
vCPU: 4
Mem: 30.5 GiB
Storage: 1 x 80 GB
The Memory is good, free command gives me this result:
root#ip-xxx-xx-xx-xxx ~]$ free -g
total used free shared buffers cached
Mem: 29 2 27 0 0 1
But the storage not correspond to the indicated one.
root#ip-xxx-xx-xx-xxx ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 6.9G 783M 91% /
devtmpfs 15G 64K 15G 1% /dev
tmpfs 15G 0 15G 0% /dev/shm
Is it normal to have only ~40GB and not 80GB like this was specified in the documentation ? Or this is because I launched the instance from an AMI ?
The two tmpfs directories aren't where your missing 80gb is. This is looks like an Debian/Ubuntu distro. I can reproduce something similar to your df:
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 15G 12K 15G 1% /dev
tmpfs 3.0G 328K 3.0G 1% /run
/dev/xvda1 7.8G 790M 6.6G 11% /
none 15G 0 15G 0% /run/shm
Note /dev/xvda1. That's your boot partition that is on ELB. Your 80gb SSD is actually at /dev/xvdb. You need to make use of it:
mkdir -p /mnt/ssd && mkfs.ext4 /dev/xvdb \
&& echo '/dev/xvdb /mnt/ssd auto defaults,nobootwait 0 0' >> /etc/fstab \
&& mount /mnt/ssd
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 15G 12K 15G 1% /dev
tmpfs 3.0G 328K 3.0G 1% /run
/dev/xvda1 7.8G 790M 6.6G 11% /
/dev/xvdb 74G 52M 70G 1% /mnt/ssd
Congrats! You are now the proud owner of a 80gb mount. Okay, not quite 80gb. Lets get 80gb:
$ df -H
Filesystem Size Used Avail Use% Mounted on
udev 17G 13k 17G 1% /dev
tmpfs 3.3G 336k 3.3G 1% /run
/dev/xvda1 8.4G 828M 7.1G 11% /
/dev/xvdb 80G 55M 76G 1% /mnt/ssd
Your filesystem is probably on EBS, not the instance storage that comes with r3.xlarge. This is the default for most AMIs. Note the size of the EBS volume is not part of the image. You can choose it when you create the instance.
Instance store is available on the larger instance types as shown here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
AMI images have two options for the root storage device. The most common are EBS images, which use EBS for the root device. Since EBS isn't locked to specific hardware, these instances are much more flexible.
The other option is an AMI with an instance store root storage device. However, you lose the ability to stop the instance without terminating, change the instance type, resize the storage device, and manage the storage separately from the instance itself.
Instance store AMIs are often tagged with S3. For example: amzn-ami-hvm-2016.03.0.x86_64-s3 (ami-152bc275).
I have the following docker-compos.yml file:
web:
build: .
ports:
- "4200:4200"
- "35729:35729"
volumes:
- ..:/code
- ../home:/home/dev
which maps the 2 volumes above. When I login into my VM and run df -i i see
Filesystem Inodes IUsed IFree IUse% Mounted on
none 1218224 509534 708690 42% /
tmpfs 256337 18 256319 1% /dev
shm 256337 1 256336 1% /dev/shm
tmpfs 256337 11 256326 1% /sys/fs/cgroup
none 1000 0 1000 0% /code
none 1000 0 1000 0% /home/dev
/dev/sda1 1218224 509534 708690 42% /etc/resolv.conf
/dev/sda1 1218224 509534 708690 42% /etc/hostname
/dev/sda1 1218224 509534 708690 42% /etc/hosts
tmpfs 256337 18 256319 1% /proc/kcore
tmpfs 256337 18 256319 1% /proc/timer_stats
As you can /code and /home/dev my 2 volumes only have 1000 inodes, so when I run my build process and it ends up creating a ton of files, I get an error that I don't have enough inodes.
Host = OSX
Guest = CentOs 6.5
Using Virtualbox
My question is: how do I assign more inodes to my data volumes /code and /home/dev above?
I'm looking for a similar solution and so far, I've found this post to have some useful information in it:
how to free inode usage
Also, according to this post there doesn't seem to be a dynamic way of allocating inodes:
how can I increase the number of inodes ...
And finally, there is this line in the documentation:
it is not possible to expand the number of inodes on a filesystem after it is created
I'm attempting to decompress a csv file on my EC2 instance. The instance should definitely be large enough so I guess it has to do with partitioning, but I am new to that stuff and don't really understand the posts I've found here and here, or whether they apply to me. (I'm not using Hadoop nor do I have a full "/tmp" folder). The .csv.gz file is 1.6 GB and it should be 14 GB decompressed. Executing gzip -d data.csv.gz, I get the error gzip: data.csv: No space left on device, and df -h shows:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 2.8G 5.0G 36% /
devtmpfs 15G 56K 15G 1% /dev
tmpfs 15G 0 15G 0% /dev/shm
Thanks for your help!