Spark application throws this error:
> Cannot create directory
> /home/user/hdp/.sparkStaging/application_00000000_000. Name node is in
> safe mode. Resources are low on NN. Please add or free up more
> resources then turn off safe mode manually. NOTE: If you turn off
> safe mode before adding resources, the NN will immediately return to
> safe mode. Use "hdfs dfsadmin -safemode leave" to turn safe mode off.
So I tried this: hdfs dfs -df -h
Filesystem Size Used Available Use%
hdfs://ip-xx-xx-xxx-xxx.xxxxxxxx.xx-xxxx-x.compute.internal:xxxx 58.5 G 5.0 G 46.5 G 9%
since it has enough free space, i tried doing df -h, and found out that
${dfs.name.dir} = /xx2/namenode consumed all of free space (27G) in /xxx/xxxx3
Filesystem Size Used Avail Use% Mounted on
xxxxxxxx 3.9G 72K 3.9G 1% /xxx
xxxxx 3.9G 0 3.9G 0% /xxx/shm
/xxx/xxxx1 9.8G 3.8G 5.9G 40% /
/xxx/xxxx2 5.0G 65M 5.0G 2% /xx1
/xxx/xxxx3 27G 27G 94M 100% /xx2
I dont know how to properly free some space for that directory.
Could you check your safemode threshold value in hdfs-default.xml
dfs.safemode.threshold.pct
dfs.namenode.safemode.threshold-pct
Try increasing the threshold,then you use below to leave safemode. After that you can revert back the changes once you have found your black sheep file causing the issue
hdfs dfsadmin -safemode leave
You can check for local files(/var/log/..) as hadoop works on top of disk space of system.Also check for the directories mentioned in hadoop.tmp.dir.Use hadoop fs -rmr on all data nodes and name node to flush
Related
We are trying to index large datasets to elastic search and indexing is stopped due to watermark reached and nodes are set to read-only.
We ran the command
GET /_cat/allocation?v
and from the output, we came to know that the disk space allocated for elastic is 10Gb and 95% is occupied.
We have some more free space on our machine that can be allocated to elastic.
We are trying to figure out how to increase the space allocation to elastic search.
Any pointers would be helpful.
Increase disc capacity to 100GB(based on data need) from 10GB(In AWS just upsized EBS volume) and follow below steps
connect to your instance
[ec2-user ~]$ df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/nvme0n1p1 xfs 8.0G 1.6G 6.5G 20% /
/dev/nvme1n1 xfs 8.0G 33M 8.0G 1% /data
[ec2-user ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1 259:0 0 30G 0 disk /data
nvme0n1 259:1 0 16G 0 disk
└─nvme0n1p1 259:2 0 8G 0 part /
└─nvme0n1p128 259:3 0 1M 0 part
[ec2-user ~]$ sudo growpart /dev/nvme0n1 1
[ec2-user ~]$ sudo resize2fs /dev/xvda1
Reference : We followed the recomendation form here. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html
I have a server on aws-ec2 with defaulit free tier. How can I increase the size of volume without using an AMI?
Here are the Steps which will help you to resize ec2 volume without AMI (Snapshots).
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
Choose Volumes, select the volume to modify, and then choose Actions, Modify Volume.
The Modify Volume window displays the volume ID and the volume's current configuration, including type, size, and IOPS. You can change any or all of these settings in a single action. Set new configuration values as follows:
To modify the type, choose a value for Volume Type.
To modify the size, enter an allowed integer value for Size.
If you chose Provisioned IOPS (IO1) as your volume type, enter an allowed integer value for IOPS.
After you have specified all of the modifications to apply, choose Modify, Yes.
Modifying volume size has no practical effect until you also extend the volume's file system to make use of the new storage capacity.
After then you have to run these command on ec2 terminal
ubuntu#ip-192-168-1-26:~$ sudo su
root#ip-192-168-1-26:/home/ubuntu# df -h
Filesystem Size Used Avail Use% Mounted on
udev 487M 0 487M 0% /dev
tmpfs 100M 12M 88M 12% /run
/dev/xvda1 7.8G 5.5G 2.0G 74% /
tmpfs 496M 0 496M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 496M 0 496M 0% /sys/fs/cgroup
tmpfs 100M 0 100M 0% /run/user/999
tmpfs 100M 0 100M 0% /run/user/1000
root#ip-192-168-1-26:/home/ubuntu# sudo file -s /dev/xvd*
/dev/xvda: DOS/MBR boot sector
/dev/xvda1: Linux rev 1.0 ext4 filesystem data, UUID=e6d1a865-817b-456f-99e7-118135343487, volume name "cloudimg-rootfs" (needs journal recovery) (extents) (large files) (huge files)
root#ip-192-168-1-26:/home/ubuntu# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 16G 0 disk
└─xvda1 202:1 0 8G 0 part /
root#ip-192-168-1-26:/home/ubuntu# sudo growpart /dev/xvda 1
CHANGED: partition=1 start=16065 old: size=16761118 end=16777183 new: size=33538334,end=33554399
root#ip-192-168-1-26:/home/ubuntu# sudo resize2fs /dev/xvda1
resize2fs 1.42.13 (17-May-2015)
Filesystem at /dev/xvda1 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/xvda1 is now 4192291 (4k) blocks long.
that's command will help you to resize ec2 volume
I'm using Hadoop YARN on EC2 over r3.xlarge instances, I launched the instances from an AMI using spark-ec2 scripts.
On https://aws.amazon.com/ec2/instance-types/, the specifications of r3.xlarge are the following:
vCPU: 4
Mem: 30.5 GiB
Storage: 1 x 80 GB
The Memory is good, free command gives me this result:
root#ip-xxx-xx-xx-xxx ~]$ free -g
total used free shared buffers cached
Mem: 29 2 27 0 0 1
But the storage not correspond to the indicated one.
root#ip-xxx-xx-xx-xxx ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 6.9G 783M 91% /
devtmpfs 15G 64K 15G 1% /dev
tmpfs 15G 0 15G 0% /dev/shm
Is it normal to have only ~40GB and not 80GB like this was specified in the documentation ? Or this is because I launched the instance from an AMI ?
The two tmpfs directories aren't where your missing 80gb is. This is looks like an Debian/Ubuntu distro. I can reproduce something similar to your df:
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 15G 12K 15G 1% /dev
tmpfs 3.0G 328K 3.0G 1% /run
/dev/xvda1 7.8G 790M 6.6G 11% /
none 15G 0 15G 0% /run/shm
Note /dev/xvda1. That's your boot partition that is on ELB. Your 80gb SSD is actually at /dev/xvdb. You need to make use of it:
mkdir -p /mnt/ssd && mkfs.ext4 /dev/xvdb \
&& echo '/dev/xvdb /mnt/ssd auto defaults,nobootwait 0 0' >> /etc/fstab \
&& mount /mnt/ssd
$ df -h
Filesystem Size Used Avail Use% Mounted on
udev 15G 12K 15G 1% /dev
tmpfs 3.0G 328K 3.0G 1% /run
/dev/xvda1 7.8G 790M 6.6G 11% /
/dev/xvdb 74G 52M 70G 1% /mnt/ssd
Congrats! You are now the proud owner of a 80gb mount. Okay, not quite 80gb. Lets get 80gb:
$ df -H
Filesystem Size Used Avail Use% Mounted on
udev 17G 13k 17G 1% /dev
tmpfs 3.3G 336k 3.3G 1% /run
/dev/xvda1 8.4G 828M 7.1G 11% /
/dev/xvdb 80G 55M 76G 1% /mnt/ssd
Your filesystem is probably on EBS, not the instance storage that comes with r3.xlarge. This is the default for most AMIs. Note the size of the EBS volume is not part of the image. You can choose it when you create the instance.
Instance store is available on the larger instance types as shown here: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html
AMI images have two options for the root storage device. The most common are EBS images, which use EBS for the root device. Since EBS isn't locked to specific hardware, these instances are much more flexible.
The other option is an AMI with an instance store root storage device. However, you lose the ability to stop the instance without terminating, change the instance type, resize the storage device, and manage the storage separately from the instance itself.
Instance store AMIs are often tagged with S3. For example: amzn-ami-hvm-2016.03.0.x86_64-s3 (ami-152bc275).
Today I started getting errors on simple operations, like creating small files in vim, the bash completion started to complain as well.
Here is the result of df -h :
vagrant#machine:/vagrant$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 40G 38G 249M 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
udev 2.0G 12K 2.0G 1% /dev
tmpfs 396M 396K 395M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 2.0G 0 2.0G 0% /run/shm
none 100M 0 100M 0% /run/user
overflow 1.0M 148K 876K 15% /tmp
192.168.50.1:/Users/nha/repo/assets 233G 141G 93G 61% /var/www/assets
vagrant 233G 141G 93G 61% /vagrant
So apparently / doesn`t have space anymore ? Isn't it weird since I have space in the other filesystems (or am I misreading something) ?
How do I get more space on my vm ?
Even though you have space on your Guest OS, the VM is limited.There are couple of steps required in order to increase the size of your disk:
first, vagrant haltto close your VM
resize disk
VBoxManage clonehd box-disk1.vmdk box-disk1.vdi --format vdi
VBoxManage modifyhd box-disk1.vdi --resize 50000
start Virtual box and change configuration of the VM to associate the new disk
use fdisk to resize disk
you need to create a new partition with the new space and allocate it, so first start the VM and logged on as super user
vagrant up && vagrant ssh
su -
the command (as illustrated from my instance) are
[root#oracle ~]# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): p
Disk /dev/sda: 52.4 GB, 52428800000 bytes
255 heads, 63 sectors/track, 6374 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00041a53
Device Boot Start End Blocks Id System
/dev/sda1 * 1 39 307200 83 Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 39 2611 20663296 8e Linux LVM
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 3
First cylinder (2611-6374, default 2611):
Using default value 2611
Last cylinder, +cylinders or +size{K,M,G} (2611-6374, default 6374):
Using default value 6374
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
[root#oracle ~]#
note you might need to change /dev/sda compare to your configuration
create a new partition (again logged on as super user su -)
su -
[root#oracle ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 linux lvm2 a-- 19.70g 0
[root#oracle ~]# pvcreate /dev/sda3
Physical volume "/dev/sda3" successfully created
[root#oracle ~]# pvs
PV VG Fmt Attr PSize PFree
/dev/sda2 linux lvm2 a-- 19.70g 0
/dev/sda3 lvm2 a-- 28.83g 28.83g
[root#oracle ~]# vgextend linux /dev/sda3
Volume group "linux" successfully extended
[root#oracle ~]# lvextend -l +100%FREE /dev/linux/root
[root#oracle ~]# resize2fs /dev/linux/home
resize2fs 1.41.12 (17-May-2010)
Filesystem at /dev/linux/home is mounted on /home; on-line resizing required
old desc_blocks = 1, new_desc_blocks = 2
Performing an on-line resize of /dev/linux/home to 7347200 (4k) blocks.
The filesystem on /dev/linux/home is now 7347200 blocks long.
You can increase space in your box, without losing data or creating new partitions.
Halt your VM;
Go to /home_dir/VirtualBox VMs
Change file format from .vmdk to .vdi. Then use command from the answer above to increase space.
Change the file extension back and change the file name.
Attach an extended disk to your VM.
VBoxManage storageattach <your_box_name> --storagectl "IDE Controller" --
port 0 --device 0 --type hdd --medium new_extended_file.vmdk
In your VirtualBox application go to Your_VM -> Settings -> Storage. Click on the controller and choose 'add new disk' below. Choose from existing disks the one you have just expanded.
Here's a step by step instruction how to expand the space in your vagrant box or virtual machine.
The easiest way to increase the size of the vagrant box is with the vagrant-disksize plugin.
In your vagrant root folder, run vagrant plugin install vagrant-disksize
Then add the new size to the Vagrantfile:
Vagrant.configure('2') do |config|
...
config.disksize.size = '60GB'
end
Then vagrant halt and vagrant up.
vagrant reload will not work.
I have read that the plugin has issues shrinking disk size if you overshoot.
EDIT:
On Mac, this plugin also resized the partition within the Guest OS (Ubuntu in my case).
On Windows, Vagrant reserves the space on the host OS (it enlarges the disk), but you can't use the space until resizing the partition from within the Guest OS.
I used GParted, but other solutions look simpler, such as: https://nguyenhoa93.github.io/Increase-VM-Partition
I sometimes have to destroy the machine and build it up again which in my case frees up quite a lot of space, you can do that by running
vagrant destroy
vagrant up
Please note this will result in database data being lost.
I'm attempting to decompress a csv file on my EC2 instance. The instance should definitely be large enough so I guess it has to do with partitioning, but I am new to that stuff and don't really understand the posts I've found here and here, or whether they apply to me. (I'm not using Hadoop nor do I have a full "/tmp" folder). The .csv.gz file is 1.6 GB and it should be 14 GB decompressed. Executing gzip -d data.csv.gz, I get the error gzip: data.csv: No space left on device, and df -h shows:
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.8G 2.8G 5.0G 36% /
devtmpfs 15G 56K 15G 1% /dev
tmpfs 15G 0 15G 0% /dev/shm
Thanks for your help!