I created a public VPC and then added a bunch of nodes to it so that I can use it for a spark cluster. Unfortunately, all of them have a partition setup that looks like the following:
ec2-user#sparkslave1: lsblk
/dev/xvda 100G
/dev/xvda1 5.7G /
I setup a cloud manager on top of these machines and all of the nodes only have 1G left for HDFS. How do I extend the partition so that it takes up all of the 100G?
I tried created /dev/xvda2, then created a volume group, added all of /dev/xvda* to it but /dev/xvda1 doesn't get added as it's mounted. I cannot boot from a live CD in this case, it's on AWS. I also tried resize2fs but it says that the root partition already takes up all of the available blocks, so it cannot be resized. How do I solve this problem and how do I avoid this problem in the future?
Thanks!
I don't think you can just resize the running root volume. This is how you'd go about increasing the root size:
create a snapshot of your current root volume
create a new volume from this snapshot of the size that you want (100G?)
stop the instance
detach the old small volume
attach the new bigger volume
start instance
I had a same problem before, but can't remember the solution. Did you try to run
e2resize /dev/xvda1
*This is when your using ext3, which is usually the default. The e2resize command will "grow" the ext3 filesystem to use the remaining free space.
Related
I want to use datadog for monitoring my EC2 Instance Disk utilization and create alerts for it. I am using system.disk.in_use metric but I am not getting my root mount point in from sectionavg:system.disk.in_use{device:/dev/loop0} by {host} and my root mount point is /dev/root. I can see every loop mount point in the list but can't see the root. due to this, the data I am getting in the monitor is different than the actual server, for example, df -hT is showing 99% root in the server but on datadog monitoring it is showing 60%.
I am not too familiar with how to use datadog, can someone please help?
Try to research about it but not able to resolve the issue.
You can also try to use the device label to read in only the root volume such as:
avg:system.disk.in_use{device_label:/} by {host}
I personally found the metric system.disk.in_use to equal the total and instead added a formula that calculated the utilization using system.disk.total and system.disk.free to be more accurate.
I have an EMR cluster running on AWS. I look in YARN and I see that 4 of my workers have this "unhealthy status" due to
1/2 local-dirs are bad: /mnt/yarn; 1/1 log-dirs are bad: /var/log/hadoop-yarn/containers
So, I ssh into the worker nodes -> run df and sure enough /mnt/yarn is at 99% disk spaced used. Also, a lot of the stderror and stdout files are taking up a lot of space in /var/log/hadoop-yarn/containers directory. My question is: what is safe to delete and what is not? I feel like I have been going down a rabbit hole and am still nowhere figuring out how to free up disk space in my worker nodes after reading for hours. I have been reading about /mnt/yarn/usercache directory, and it seems like the contents in that directory are "local resources" used to run my spark application. But /mnt/yarn/usercache/hadoop/filecache and /mnt/yarn/usercache/hadoop/appcache are taking up 3% and 96% of disk space, respectively, in /mnt/yarn
Probably you need to clear this folder - /var/log/hadoop-yarn/apps/hadoop/logs/
on HDFS. Try hdfs dfs -ls /var/log/hadoop-yarn/apps/hadoop/logs to view it.
Another option to check - /mnt/var/log/hadoop-yarn/containers on executors.
There should be another folders with name like "application_someId". These folders contains logs for finished and running spark job.
yes, you can delete the container files in /mnt/var/log/hadoop-yarn/containers (as well as the log files in there).. I had a very similar problem.
I deleted the files, stopped and restarted spark on EMR and my unhealthy nodes came back.
I already created space in the virtualbox so you can see this free 55gb.
But when i want to delete partition 2 and partition 5, i get an error message: rror deleting partition /dev/sda5: warning partition dev/sda5 is being used are you sure you want to continue and i cant do nothing with it.
I tried to delete these partition with sudo fdisk /dev/sda and after that I deleted these partition, but nothing changed, they are stayed there. How can incrase my SDA1 dark size then?
I know, it isn't a programming question, but i tried many links and i haven't got any idea how can I increase my Partition1..
The easiest way is to use a gparted live cd that you'll find here https://gparted.org/livecd.php.
Then boot your VM on it (since it's a virtual machine you won't have to burn a real cd) and follow the steps (there are plenty of tutos with screen captures on the net, won't try to make one more here).
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 8 years ago.
Improve this question
I would like to reduce EBS volume capacity without losing data.
I would like to change the capacity from 1 TB to 200 GB.
Please provide detailed steps on how to do it.
The approach I take to decreasing an EBS root volume is as follows:
Stop (not terminate) the target instance, and detach the root EBS volume. Alternatively, you can snapshot the root volume (or use an existing snapshot) and create a new EBS volume from that. (e.g. /dev/xvda1)
Note: the volume you use from the step above will be altered - so you may want to take a snapshot if you didn't already.
Create a new EBS volume that is the desired size (e.g. /dev/xvdg)
Launch an instance, and attach both EBS volumes to it
Check the file system (of the original root volume): (e.g.) e2fsck -f /dev/xvda1
Maximally shrink the original root volume: (e.g. ext2/3/4) resize2fs -M -p /dev/xvda1
Copy the data over with dd:
Choose a chunk size (I like 16MB)
Calculate the number of chunks (using the number of blocks from the resize2fs output): blocks*4/(chunk_size_in_mb*1024) - round up a bit for safety
Copy the data: (e.g.) dd if=/dev/xvda1 ibs=16M of=/dev/xvdg obs=16M count=80
Resize the filesystem on the new (smaller) EBS volume: (e.g.) resize2fs -p /dev/xvdg
Check the file system (of the new volume): (e.g.) e2fsck -f /dev/xvdg
Detach your new EBS root volume, and attach it to your original instance
Answer from ezhilrean is OK, but there is an easier way.
Let's say you have an instance with your /var partition on /dev/sdf1 and you want to reduce this from 300GB to 200GB (presuming that there is < 200GB of data on /var)
Create a new volume in the same AZ as the original volume
Attach it to the instance as /dev/sdg
Login to instance with root permissions
fdisk /dev/sdg
n (for New)
p (for Primary)
Accept defaults for other fdisk options
w (for Write)
fdisk will then exit. You now need to make a file system on the new partition
mkfs.ext4 /dev/sdg1 (presuming that ext4 was used on existing partition)
Next, mount you new partition at a temporary mount point
mkdir /new
mount /dev/sdg1 /new
Now, copy your data
cd /var
cp -ax * /new/
Update your /etc/fstab to use the new partition for /var
/dev/sdg1 /var ext4 defaults 0 0
Reboot
init 6
If you need your /var partition to have identifier /dev/sdf1, you can stop the instance, detach both EBS volumes, and re-attach the new smaller one as /dev/sdf Remember to change /etc/fstab before you do this
I have a S3 bucket mount as a volume in one EC2 instance with S3FS. I've created with PHP a directory structure there with more than 20GB now.
By the moment S3FS is exhausting instance memory and uploading is really slow so I want to move all the files to an EBS attached to the same instance.
I've tried S3CMD but there are some incompatibilities since S3FS creates zero sized objects in the bucket with the same name as directories.
Also tried writing a script to recursively copy the structure skipping those zero sized objects.
None worked.
Have anyone tried to do this? Thanks in advance for your help.
#hernangarcia Dont make things complicated, use recursive wget that wget -r followed by the url of the bucket end point. You can download all the contents to a EBS volume. Also my suggestion is not to store all those files which is like 20 GB on the root volume of the instance, instead attach another volume to it and then store all those file in that extra volume and if you have a High IOPS volume for it so that operations will be faster.