I got a 6 Node Hadoop cluster with HDP 2.3 installed.
Node consist 2 drive one is of 50 TB and other is of 2 TB, now the disk with 2 TB is full. But i want to remove this disk from all node and only wanna use 50 TB disk for Hadoop.
Question
Can i safely remove data disk without causing missing/corrupt blocks on Cluster.
Update
output of mount command is below
/dev/sda6 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda3 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/tmp on /tmp type none (rw,bind)
/var/tmp on /var/tmp type none (rw,bind)
/home on /home type none (rw,bind)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/sdc1 on /data1 type xfs (rw)
/dev/sdb1 on /data type xfs (rw)
And value for property dfs.datanode.data.dir is /data/hadoop/hdfs/data,/data1/hadoop/hdfs/data
You cannot take the 2 TB drive as the NameNode metadata may get corrupt. Namenode will be waiting for the block report from this node and will near leave safemode if the data blocks in the form of block reports are not reported.
One workaround can be you can copy all the data into 50 TB drive and format the Namenode. You cannot perform this operation on the cases where data is sensitive.
Related
I am trying to implement A+B dual copy scheme partition on my Avenger96 board. I am using Yocto build system and also .wks file to create partitions. My .wks file:
part fsbl1 --source rawcopy --sourceparams="file=u-boot-spl.stm32" --part-name "fsbl1" --ondisk mmcblk --align 1 --size 256k
part fsbl2 --source rawcopy --sourceparams="file=u-boot-spl.stm32" --part-name "fsbl2" --ondisk mmcblk --align 1 --size 256k
part ssbl --source rawcopy --sourceparams="file=u-boot.itb" --part-name "ssbl" --ondisk mmcblk --align 1 --size 2M
part / --source rootfs --ondisk mmcblk0 --fstype=ext4 --label root_A --part-name "rootfs_A" --align 4096 --use-uuid --active --size 3G
part /rootfsB --source rootfs --ondisk mmcblk0 --fstype=ext4 --label root_B --part-name "rootfs_B" --align 4096 --use-uuid --size 3G
bootloader --ptable gpt
And I am able to build .wic.xz image and copied image to SD. It created 2 rootfs partitions.
But when I boot using this SD card, I could see both partitions are mounted. For example, there is also /dev/root which refers to current active partition /dev/mmcblk0p4 and also /dev/mmcblk0p5 (/rootfsB) when I do df -h
Filesystem Size Used Available Use% Mounted on
/dev/root 5.0G 1.1G 3.6G 23% /
devtmpfs 469.7M 0 469.7M 0% /dev
tmpfs 502.2M 0 502.2M 0% /dev/shm
tmpfs 502.2M 9.6M 492.5M 2% /run
tmpfs 502.2M 0 502.2M 0% /sys/fs/cgroup
tmpfs 502.2M 0 502.2M 0% /tmp
tmpfs 502.2M 16.0K 502.1M 0% /var/volatile
/dev/mmcblk0p5 5.0G 1.1G 3.6G 23% /rootfsB
tmpfs 100.4M 0 100.4M 0% /run/user/0
And boot messages also show:
Mounting /rootfsB...
Starting Start psplash boot splash screen...
[ OK ] Mounted /rootfsB.
Also doing mount from linux user-space results in:
/dev/mmcblk0p4 on / type ext4 (rw,relatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=480932k,nr_inodes=120233,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,relatime,gid=5,mode=620,ptmxmode=666)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev)
configfs on /sys/kernel/config type configfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /var/volatile type tmpfs (rw,relatime)
/dev/mmcblk0p5 on /rootfsB type ext4 (rw,relatime)
tmpfs on /run/user/0 type tmpfs (rw,nosuid,nodev,relatime,size=102840k,mode=700)
Is this the expected behavior for A+B type partitioning?
Can anyone please let me know what could be the issue and how to resolve it?
Your help will be much appreciated.
Thanks in advance.
P.S: Please let me know if any info is missing here.
This is actually what I would expect. You want to boot from the "active" (A) partition, but you also want to be able to update the "passive" (B) partition. After you updated the "passive" (B) partition you typically tell the bootloader to try to boot from it. If that works (B) becomes the "active" partition and (A) the "passive" one.
We are trying to index large datasets to elastic search and indexing is stopped due to watermark reached and nodes are set to read-only.
We ran the command
GET /_cat/allocation?v
and from the output, we came to know that the disk space allocated for elastic is 10Gb and 95% is occupied.
We have some more free space on our machine that can be allocated to elastic.
We are trying to figure out how to increase the space allocation to elastic search.
Any pointers would be helpful.
Increase disc capacity to 100GB(based on data need) from 10GB(In AWS just upsized EBS volume) and follow below steps
connect to your instance
[ec2-user ~]$ df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/nvme0n1p1 xfs 8.0G 1.6G 6.5G 20% /
/dev/nvme1n1 xfs 8.0G 33M 8.0G 1% /data
[ec2-user ~]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1 259:0 0 30G 0 disk /data
nvme0n1 259:1 0 16G 0 disk
└─nvme0n1p1 259:2 0 8G 0 part /
└─nvme0n1p128 259:3 0 1M 0 part
[ec2-user ~]$ sudo growpart /dev/nvme0n1 1
[ec2-user ~]$ sudo resize2fs /dev/xvda1
Reference : We followed the recomendation form here. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html
I have a server on aws-ec2 with defaulit free tier. How can I increase the size of volume without using an AMI?
Here are the Steps which will help you to resize ec2 volume without AMI (Snapshots).
Open the Amazon EC2 console at https://console.aws.amazon.com/ec2/.
Choose Volumes, select the volume to modify, and then choose Actions, Modify Volume.
The Modify Volume window displays the volume ID and the volume's current configuration, including type, size, and IOPS. You can change any or all of these settings in a single action. Set new configuration values as follows:
To modify the type, choose a value for Volume Type.
To modify the size, enter an allowed integer value for Size.
If you chose Provisioned IOPS (IO1) as your volume type, enter an allowed integer value for IOPS.
After you have specified all of the modifications to apply, choose Modify, Yes.
Modifying volume size has no practical effect until you also extend the volume's file system to make use of the new storage capacity.
After then you have to run these command on ec2 terminal
ubuntu#ip-192-168-1-26:~$ sudo su
root#ip-192-168-1-26:/home/ubuntu# df -h
Filesystem Size Used Avail Use% Mounted on
udev 487M 0 487M 0% /dev
tmpfs 100M 12M 88M 12% /run
/dev/xvda1 7.8G 5.5G 2.0G 74% /
tmpfs 496M 0 496M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 496M 0 496M 0% /sys/fs/cgroup
tmpfs 100M 0 100M 0% /run/user/999
tmpfs 100M 0 100M 0% /run/user/1000
root#ip-192-168-1-26:/home/ubuntu# sudo file -s /dev/xvd*
/dev/xvda: DOS/MBR boot sector
/dev/xvda1: Linux rev 1.0 ext4 filesystem data, UUID=e6d1a865-817b-456f-99e7-118135343487, volume name "cloudimg-rootfs" (needs journal recovery) (extents) (large files) (huge files)
root#ip-192-168-1-26:/home/ubuntu# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 16G 0 disk
└─xvda1 202:1 0 8G 0 part /
root#ip-192-168-1-26:/home/ubuntu# sudo growpart /dev/xvda 1
CHANGED: partition=1 start=16065 old: size=16761118 end=16777183 new: size=33538334,end=33554399
root#ip-192-168-1-26:/home/ubuntu# sudo resize2fs /dev/xvda1
resize2fs 1.42.13 (17-May-2015)
Filesystem at /dev/xvda1 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/xvda1 is now 4192291 (4k) blocks long.
that's command will help you to resize ec2 volume
I have created an c3.2xlarge EC2 instance with the store volume specified as 2 x 80 GB (160 GB). But when I use df -H command, this is what i see, and there is not enough storage as specified.
Filesystem Size Used Avail Use% Mounted on
devtmpfs 7.9G 62k 7.9G 1% /dev
tmpfs 7.9G 0 7.9G 0% /dev/shm
/dev/xvda1 8.4G 1.1G 7.2G 14% /
I need an EC2 instance to have at least 80 gigs of storage, which instance should I choose?
Thanks for the points in the comments.
The problem was; I used EC2 Management Console and didn't add the volumes when I created the cluster, I terminated that cluster, created a new one, on Storage page Added New Volume, chose the volume type as Instance Store 0.
[]$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 8G 0 disk
└─xvda1 202:1 0 8G 0 part /
xvdb 202:16 0 75G 0 disk /media/ephemeral0
Now the 80 Gig volume is there.
My rootfs is squashfs and I am trying to mount three JFFS2 partition on my flash device. But I want to set permissions for those partitions/directories.
The requirement is /readerconfig is only accessible by user with id
1001. /platform is only accessible by user with id 1001 /apps is only accessible (read,write and execute) by user with id 1000.
Below is my fstab entry.
rootfs / squashfs defaults 0 0
proc /proc proc defaults 0 0
/dev/mtdblock12 /readerconfig jffs2 rw,umask=0077,uid=1000,gid=1000,exec,noatime 0 0
/dev/mtdblock13 /platform jffs2 rw,umask=0077,uid=1000,gid=1000,exec,noatime 0 0
/dev/mtdblock15 /apps jffs2 rw,umask=0077,uid=1001,gid=1001,exec,noatime 0 0
tmpfs /var tmpfs mode=0777 0 0
tmpfs /tmp tmpfs mode=0777 0 0
sysfs /sys sysfs nosuid,noexec,nodev 0 0
/dev/mtdblock14 /etc jffs2 rw,mode=0777,uid=0,gid=0,noatime 0 0
Upon implementing, I found that the mount options like uid and gid are not set for JFFS2 file system mount points.
Here is the output of mount command on the target board.
> root#ti-omap3-am3517-evm:~# mount
>rootfs on / type rootfs (rw)
> /dev/root on / type squashfs (ro,relatime)
> devtmpfs on /dev type devtmpfs (rw,relatime,size=127428k,nr_inodes=31857,mode=755)
> proc on > /proc type proc (rw,relatime)
> /dev/mtdblock15 on /apps type jffs2 (rw,noatime)
> tmpfs on /var type tmpfs (rw,relatime,mode=777)
> tmpfs on /tmp type tmpfs (rw,relatime,mode=777)
> sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
> /dev/mtdblock12 on /readerconfig type jffs2 (rw,noatime)
> /dev/mtdblock13 on /platform type jffs2 (rw,noatime)
How do I set permissions for the directories /readerconfig, /platform and /apps directory?
Is it limitation of JFFS2 / mount command? or I can do it from rcS script? All alternatives are invited.
Jffs2 based mount point cannot be set with permissions. One has to change ownership using 'chown' command and/or set permissions using 'chmod' command.