CopyToLocal command is creating a hidden checksum file. Is there a way to avoid this being created?
Unless you're running a very old version, Apache Hadoop only creates the local checksum file if you opt in to creating it by passing the -crc option as showing in the Apache Hadoop documentation for the copyToLocal command.
> hdfs dfs -copyToLocal /LICENSE.txt
> ls -lrta
drwxr-xr-x+ 131 naurc001 SWNA\Domain Users 4.3K Jan 27 09:18 ../
-rw-r--r-- 1 naurc001 SWNA\Domain Users 140K Jan 27 10:11 LICENSE.txt
drwxr-xr-x 7 naurc001 SWNA\Domain Users 238B Jan 27 10:11 ./
> hdfs dfs -copyToLocal -crc /LICENSE.txt
> ls -lrta
drwxr-xr-x+ 131 naurc001 SWNA\Domain Users 4.3K Jan 27 09:18 ../
-rw-r--r-- 1 naurc001 SWNA\Domain Users 140K Jan 27 10:07 LICENSE.txt
-rw-r--r-- 1 naurc001 SWNA\Domain Users 1.1K Jan 27 10:07 .LICENSE.txt.crc
drwxr-xr-x 8 naurc001 SWNA\Domain Users 272B Jan 27 10:07 ./
If you have the option, then I recommend running a current version and testing to make sure the -crc option is not being passed anywhere.
Related
When trying to enter into Docker VM in mac, I am getting screen is terminating error whereby I cannot go into Docker Desktop. In Mac, Docker host is not the mac as it runs as a VM. Any solutions?
Thank you for the help.
Workarounds
I think it's a bug since version 2.3.0.4 of Docker Desktop for Mac, as I have used screen successfully to access the Docker VM in the past on Mac without this issue. (edit: this issue appears to be still present at version 2.4.0.0)
There are some alternative methods to access the Docker VM. Here is one that worked for me:
workaround method 1
run this command to enter shell of the Docker VM:
docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
Then you can do what you need to do at the usual location for Docker volumes:
/ # ls -lah /var/lib/docker/volumes/
total 72
drwx------ 10 root root 4.0K Aug 27 11:34 .
drwx--x--x 15 root root 4.0K Oct 31 00:51 ..
drwxr-xr-x 3 root root 4.0K Aug 26 10:44 14ce94e174839f1947efa6fcbf5ac1fb2ea3b0f0b3f25311fee333ee374576b6
drwxr-xr-x 3 root root 4.0K Aug 25 12:34 34eb2bd80931ff954e1da80c5383beb4def61129d4005432b77080531cd10a5a
drwxr-xr-x 3 root root 4.0K Aug 27 11:34 830de2ce31519c921b50c448964b54517ca4461d337b56a9fd6e5b354ace3247
drwxr-xr-x 3 root root 4.0K Aug 25 12:34 data-layer_mongo_data
drwxr-xr-x 3 root root 4.0K Aug 26 11:01 ebb440184703bfad17184bd5ff74b677b50d74b8f0fbdd116506a1fcacfb00cd
drwxr-xr-x 3 root root 4.0K Aug 27 11:27 hawakening-services_back-end_log
drwxr-xr-x 3 root root 4.0K Aug 27 11:27 hawakening-services_front-end_log
drwxr-xr-x 3 root root 4.0K Aug 27 11:27 hawakening-services_mongo_data
-rw------- 1 root root 64.0K Oct 31 00:51 metadata.db
workaround method 2
Here's another one that also worked for me.
Run this command to enter shell of the Docker VM:
docker run --rm -it --privileged --pid=host walkerlee/nsenter -t 1 -m -u -i -n sh
As before, here is where the Docker volumes' directories are:
/ # ls -lah /var/lib/docker/volumes/
total 72
drwx------ 10 root root 4.0K Aug 27 11:34 .
drwx--x--x 15 root root 4.0K Oct 31 00:51 ..
drwxr-xr-x 3 root root 4.0K Aug 26 10:44 14ce94e174839f1947efa6fcbf5ac1fb2ea3b0f0b3f25311fee333ee374576b6
drwxr-xr-x 3 root root 4.0K Aug 25 12:34 34eb2bd80931ff954e1da80c5383beb4def61129d4005432b77080531cd10a5a
drwxr-xr-x 3 root root 4.0K Aug 27 11:34 830de2ce31519c921b50c448964b54517ca4461d337b56a9fd6e5b354ace3247
drwxr-xr-x 3 root root 4.0K Aug 25 12:34 data-layer_mongo_data
drwxr-xr-x 3 root root 4.0K Aug 26 11:01 ebb440184703bfad17184bd5ff74b677b50d74b8f0fbdd116506a1fcacfb00cd
drwxr-xr-x 3 root root 4.0K Aug 27 11:27 hawakening-services_back-end_log
drwxr-xr-x 3 root root 4.0K Aug 27 11:27 hawakening-services_front-end_log
drwxr-xr-x 3 root root 4.0K Aug 27 11:27 hawakening-services_mongo_data
-rw------- 1 root root 64.0K Oct 31 00:51 metadata.db
I'm writing a shell script that uses the shasum to check if the contents of a directory have changed.
On Linux and FreeBSD, the shasum have the same behavior when I do shasum <directory> however, on MacOS the shasum give me hashes for files only.
FreeBSD
$ shasum CONTENTS/
7f986e5e5289c59db1bba48df92ffe4707830aaa CONTENTS/
Linux
$ shasum CONTENTS/
7f986e5e5289c59db1bba48df92ffe4707830aaa CONTENTS/
MacOS
$ shasum CONTENTS/
shasum: CONTENTS/:
How could I calculate the hash of a directory in MacOS?
TRY 1: Using TAR with pipes
Tried to use but seems that this tar option doesn't work on MacOS.
tar cO CONTENTS/ | shasum
tar: Option -O is not permitted in mode -c
da39a3ee5e6b4b0d3255bfef95601890afd80709 -
TRY 2: Using FIND/EXEC
It was consistent between MacOS and FreeBSD, but Linux returned a weird hash
find CONTENTS -type f -exec shasum {} \; | sort -k 2 | shasum
Linux
c2ddb9bc5f543e956f5cdcc76750cb78cc5f26f3
FreeBSD
3ac2a9d4e2fc5d2d2ec3c7f612e680990cc35824
MacOS
3ac2a9d4e2fc5d2d2ec3c7f612e680990cc35824
OTHER FINDINGS ON TAR
tar would be excellent as it "archives" a folder and then I could shasum it, however the order of how tar "walk" the folder structure is not consistent across operating systems. As some helpers mentioned in the comments that I should use the same version of tar in all systems.
Just an example, on system 1 I have this order:
drwxr-xr-x 0 root wheel 0 27 Jul 07:23 usr/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f1/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f1/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f1/f0/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f1/f0/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/f1/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/f1/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/f1/f0/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/f1/f0/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/f2/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/f2/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/f2/f1/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/f2/f1/aaa
and on system 2 I have the following order:
drwxr-xr-x 0 root wheel 0 27 Jul 07:23 usr/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f1/
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/f2/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/f2/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/f2/f1/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/f2/f1/aaa
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/f1/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/f1/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/f1/f0/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/f1/f0/aaa
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f1/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f1/f0/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f1/f0/aaa
From a tar standpoint it if all good, but due to the order, the shasum produces a different hash.
CONCLUSION
shasum is consistent among Linux and BSDs to check an individual file hash, but, when it comes to directories the consistency happens only on MacOS and FreeBSD, perhaps due to how files are sorted.
If sorting is enforced using the find command, consistency is only obtained in FreeBSD and MacOS, however this method is time prohibitive as it takes a significant amount of time to calculate the hashes for every single file and then the whole structure hash.
Using tar to create a temporary file and then doing a shasum also found to be inconsistent between Linux and BSDs, perhaps because of difference in the archiving method.
I think the only way forward is to redesign my solution.
I had this problem some time ago and I ended up tarring the directory and generating a hash from the tar.
$ mkdir -p test
$ echo 1 > test/tmp
$ tar cO test/ | md5sum
7b18a99a8ccfef1ebbfd1e7a8b2852ee
$ echo 2 > test/tmp
$ tar cO test/ | md5sum
644042dd530157e604641ea89b4e9152
Note that if you write the same content to a file, the modified stat is updated and a new hash will be generated for that directory.
$ echo 2 > test/tmp
$ tar cO test/ | md5sum
da25819594f123563a837d5786e51950 -
$ echo 2 > test/tmp
$ tar cO test/ | md5sum
9407b64d43b809a5828a9fc2297b4e9c -
p.s. you should change md5sum for shasum :-)
EDIT:
Discussed it shortly with a friend who uses MacOSX and he came up with:
$ mkdir tmp; echo 1 > test/a;
$ tar -cf - tmp/ | md5sum
26c43adc9eca9f63279d08a0d145dd7d -
$ echo 2 > test/a
$ tar -cf - tmp/ | md5sum
dc3a68cac0b0224be9b202d86e69c5bd -
Note that MACosx uses md5 instead of md5sum.
I am running below commands on console:
[root#master ~]# pwd
/root
[root#master ~]# vi test.txt
#editting file
[root#master ~]# su - hdfs -c "hdfs dfs -put /root/test.txt /user/ambari-qa"
put: `/root/test.txt': No such file or directory
the file is there
[root#master ~]# pwd
/root
[root#master ~]# ls -l
total 9636984
-rw-r--r--. 1 root root 1823777034 Jun 12 08:34 ambari-2.6.2.2-centos6.tar.gz
-rw-------. 1 root root 716 Mar 19 2012 anaconda-ks.cfg
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Desktop
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Documents
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Downloads
-rw-r--r-- 1 root root 15 Oct 18 20:16 enabled~
-rw-r--r-- 1 root root 15 Oct 18 20:38 enablez~
-rw-r--r--. 1 root root 511720220 Oct 15 19:33 HDP-2.6.5.0-centos6-rpm.tar.gz
-rw-r--r--. 1 root root 7303095942 May 15 03:50 HDP-2.6.5.0-centos6-rpm.tar.gz.1
-rw-r--r--. 1 root root 43941068 Aug 13 20:28 HDP-UTILS-1.1.0.22-centos6.tar.gz
-rw-r--r--. 1 root root 185646832 Oct 15 17:04 jdk-8u181-linux-x64.tar.gz
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Music
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Pictures
-rw-r--r--. 1 root root 12148 Dec 16 2011 post-install
-rw-r--r--. 1 root root 552 Dec 16 2011 post-install.log
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Public
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Templates
-rw-r--r-- 1 root root 2 Oct 23 05:39 test
-rwxr-xr-x 1 root root 3 Oct 23 06:37 test.txt
However,I am getting no such file or directory, I am creating the file in the path but I get that error?
Seems the problem was due to privilege at /root directory
dr-xr-x---. 25 root root 4096 Oct 23 06:37 root
using chmod 755 worked then
I am not able to create .bashrc file in the home directory. When I try to use touch I get:
touch: .bashrc: No such file or directory
When I type vim .bashrc in the home directory, and try to save using :w, it says
E166: Can't open linked file for writing
Edit:
Output of ls -l .bash*
lrwxr-xr-x 1 ankitsultana staff 34 Oct 30 2014 .bash -> /Users/ankitsultana/.dotfiles/bash
-rw------- 1 ankitsultana staff 7069 Nov 26 01:04 .bash_history
-rw-r--r-- 1 ankitsultana staff 2633 Nov 26 13:51 .bash_profile
-rw-r--r-- 1 ankitsultana staff 167 Sep 21 2014 .bash_profile-anaconda.bak
-rw-r--r-- 1 ankitsultana staff 1791 Jun 30 2015 .bash_profile.pysave
lrwxr-xr-x 1 ankitsultana staff 36 Oct 30 2014 .bashrc -> /Users/ankitsultana/.dotfiles/bashrc
I'm following "How to Write Go Code" and tries to delete stringutil.a under $GOPATH/pkg/darwin_amd64/github.com/user. The delete is successful but the file comes back on its own. I'm confused. What is happening?
zps-MacBook-Air:haibin haibin$ rm stringutil.a
zps-MacBook-Air:haibin haibin$ ls -lah
total 0
drwxr-xr-x 2 haibin staff 68B Feb 15 00:57 .
drwxr-xr-x 17 haibin staff 578B Feb 15 00:39 ..
zps-MacBook-Air:haibin haibin$ ls -lah
total 8
drwxr-xr-x 3 haibin staff 102B Feb 15 00:57 .
drwxr-xr-x 17 haibin staff 578B Feb 15 00:39 ..
-rw-r--r-- 1 haibin wheel 2.4K Feb 15 00:57 stringutil.a
You need to delete the stringutil.go source file that is under the src tree. The *.a file is a binary file which results from the compilation (and possibly linking) of source files.