I'm writing a shell script that uses the shasum to check if the contents of a directory have changed.
On Linux and FreeBSD, the shasum have the same behavior when I do shasum <directory> however, on MacOS the shasum give me hashes for files only.
FreeBSD
$ shasum CONTENTS/
7f986e5e5289c59db1bba48df92ffe4707830aaa CONTENTS/
Linux
$ shasum CONTENTS/
7f986e5e5289c59db1bba48df92ffe4707830aaa CONTENTS/
MacOS
$ shasum CONTENTS/
shasum: CONTENTS/:
How could I calculate the hash of a directory in MacOS?
TRY 1: Using TAR with pipes
Tried to use but seems that this tar option doesn't work on MacOS.
tar cO CONTENTS/ | shasum
tar: Option -O is not permitted in mode -c
da39a3ee5e6b4b0d3255bfef95601890afd80709 -
TRY 2: Using FIND/EXEC
It was consistent between MacOS and FreeBSD, but Linux returned a weird hash
find CONTENTS -type f -exec shasum {} \; | sort -k 2 | shasum
Linux
c2ddb9bc5f543e956f5cdcc76750cb78cc5f26f3
FreeBSD
3ac2a9d4e2fc5d2d2ec3c7f612e680990cc35824
MacOS
3ac2a9d4e2fc5d2d2ec3c7f612e680990cc35824
OTHER FINDINGS ON TAR
tar would be excellent as it "archives" a folder and then I could shasum it, however the order of how tar "walk" the folder structure is not consistent across operating systems. As some helpers mentioned in the comments that I should use the same version of tar in all systems.
Just an example, on system 1 I have this order:
drwxr-xr-x 0 root wheel 0 27 Jul 07:23 usr/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f1/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f1/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f1/f0/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f1/f0/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/f1/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/f1/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/f1/f0/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/f1/f0/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/f2/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/f2/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/f2/f1/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/f2/f1/aaa
and on system 2 I have the following order:
drwxr-xr-x 0 root wheel 0 27 Jul 07:23 usr/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f1/
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/f2/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/f2/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f3/f2/f1/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f3/f2/f1/aaa
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/f1/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/f1/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f2/f1/f0/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f2/f1/f0/aaa
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f1/aaa
drwxr-xr-x 0 root wheel 0 27 Jul 07:25 usr/f1/f0/
-rw-r--r-- 0 root wheel 0 27 Jul 07:25 usr/f1/f0/aaa
From a tar standpoint it if all good, but due to the order, the shasum produces a different hash.
CONCLUSION
shasum is consistent among Linux and BSDs to check an individual file hash, but, when it comes to directories the consistency happens only on MacOS and FreeBSD, perhaps due to how files are sorted.
If sorting is enforced using the find command, consistency is only obtained in FreeBSD and MacOS, however this method is time prohibitive as it takes a significant amount of time to calculate the hashes for every single file and then the whole structure hash.
Using tar to create a temporary file and then doing a shasum also found to be inconsistent between Linux and BSDs, perhaps because of difference in the archiving method.
I think the only way forward is to redesign my solution.
I had this problem some time ago and I ended up tarring the directory and generating a hash from the tar.
$ mkdir -p test
$ echo 1 > test/tmp
$ tar cO test/ | md5sum
7b18a99a8ccfef1ebbfd1e7a8b2852ee
$ echo 2 > test/tmp
$ tar cO test/ | md5sum
644042dd530157e604641ea89b4e9152
Note that if you write the same content to a file, the modified stat is updated and a new hash will be generated for that directory.
$ echo 2 > test/tmp
$ tar cO test/ | md5sum
da25819594f123563a837d5786e51950 -
$ echo 2 > test/tmp
$ tar cO test/ | md5sum
9407b64d43b809a5828a9fc2297b4e9c -
p.s. you should change md5sum for shasum :-)
EDIT:
Discussed it shortly with a friend who uses MacOSX and he came up with:
$ mkdir tmp; echo 1 > test/a;
$ tar -cf - tmp/ | md5sum
26c43adc9eca9f63279d08a0d145dd7d -
$ echo 2 > test/a
$ tar -cf - tmp/ | md5sum
dc3a68cac0b0224be9b202d86e69c5bd -
Note that MACosx uses md5 instead of md5sum.
Related
I am running below commands on console:
[root#master ~]# pwd
/root
[root#master ~]# vi test.txt
#editting file
[root#master ~]# su - hdfs -c "hdfs dfs -put /root/test.txt /user/ambari-qa"
put: `/root/test.txt': No such file or directory
the file is there
[root#master ~]# pwd
/root
[root#master ~]# ls -l
total 9636984
-rw-r--r--. 1 root root 1823777034 Jun 12 08:34 ambari-2.6.2.2-centos6.tar.gz
-rw-------. 1 root root 716 Mar 19 2012 anaconda-ks.cfg
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Desktop
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Documents
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Downloads
-rw-r--r-- 1 root root 15 Oct 18 20:16 enabled~
-rw-r--r-- 1 root root 15 Oct 18 20:38 enablez~
-rw-r--r--. 1 root root 511720220 Oct 15 19:33 HDP-2.6.5.0-centos6-rpm.tar.gz
-rw-r--r--. 1 root root 7303095942 May 15 03:50 HDP-2.6.5.0-centos6-rpm.tar.gz.1
-rw-r--r--. 1 root root 43941068 Aug 13 20:28 HDP-UTILS-1.1.0.22-centos6.tar.gz
-rw-r--r--. 1 root root 185646832 Oct 15 17:04 jdk-8u181-linux-x64.tar.gz
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Music
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Pictures
-rw-r--r--. 1 root root 12148 Dec 16 2011 post-install
-rw-r--r--. 1 root root 552 Dec 16 2011 post-install.log
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Public
drwxr-xr-x. 2 root root 4096 Oct 15 11:30 Templates
-rw-r--r-- 1 root root 2 Oct 23 05:39 test
-rwxr-xr-x 1 root root 3 Oct 23 06:37 test.txt
However,I am getting no such file or directory, I am creating the file in the path but I get that error?
Seems the problem was due to privilege at /root directory
dr-xr-x---. 25 root root 4096 Oct 23 06:37 root
using chmod 755 worked then
I install lua success in my mac, but i got "-bash: lua: command not found" message when i run lua cmd, but i run /usr/loca/bin/lua success, i confirm the path that lua installed is included in PATH. I record some information as follow:
lxr:bin wang$ pwd
/usr/local/bin
lxr:bin wang$ echo $PATH
.;/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:JAVA_HOME/bin
lxr:bin wang$ lua
-bash: lua: command not found
lxr:bin wang$ pwd
/usr/local/bin
lxr:bin wang$ ls -l
total 872
-rwxr-xr-x 1 root wheel 80 11 23 00:40 VBoxAutostart
-rwxr-xr-x 1 root wheel 82 11 23 00:40 VBoxBalloonCtrl
-rwxr-xr-x 1 root wheel 80 11 23 00:40 VBoxBugReport
-rwxr-xr-x 1 root wheel 77 11 23 00:40 VBoxDTrace
-rwxr-xr-x 1 root wheel 79 11 23 00:40 VBoxHeadless
-rwxr-xr-x 1 root wheel 77 11 23 00:40 VBoxManage
-rwxr-xr-x 1 root wheel 79 11 23 00:40 VBoxVRDP
-rwxr-xr-x 1 root wheel 77 11 23 00:40 VirtualBox
lrwxr-xr-x 1 root wheel 14 3 28 2016 git -> ../git/bin/git
lrwxr-xr-x 1 root wheel 37 3 28 2016 git-credential-osxkeychain -> ../git/bin/git-credential-osxkeychain
lrwxr-xr-x 1 root wheel 24 3 28 2016 git-cvsserver -> ../git/bin/git-cvsserver
lrwxr-xr-x 1 root wheel 20 3 28 2016 git-shell -> ../git/bin/git-shell
lrwxr-xr-x 1 root wheel 26 3 28 2016 git-upload-pack -> ../git/bin/git-upload-pack
lrwxr-xr-x 1 root wheel 15 3 28 2016 gitk -> ../git/bin/gitk
-rwxr-xr-x 1 root wheel 221128 12 1 15:53 lua
-rwxr-xr-x 1 root wheel 150520 12 1 15:53 luac
-rwxr-xr-x 1 root wheel 2670 11 28 2014 pstorm
-rwxr-xr-x 1 root wheel 1394 3 8 2017 ssh-copy-id
-rwxr-xr-x 1 root wheel 75 11 23 00:40 vbox-img
-rwxr-xr-x 1 root wheel 77 11 23 00:40 vboxwebsrv
lxr:bin wang$ ./lua
Lua 5.3.0 Copyright (C) 1994-2015 Lua.org, PUC-Rio
>
lxr:bin wang$
The first two characters in $PATH should probably be .:, not .;.
I'm following "How to Write Go Code" and tries to delete stringutil.a under $GOPATH/pkg/darwin_amd64/github.com/user. The delete is successful but the file comes back on its own. I'm confused. What is happening?
zps-MacBook-Air:haibin haibin$ rm stringutil.a
zps-MacBook-Air:haibin haibin$ ls -lah
total 0
drwxr-xr-x 2 haibin staff 68B Feb 15 00:57 .
drwxr-xr-x 17 haibin staff 578B Feb 15 00:39 ..
zps-MacBook-Air:haibin haibin$ ls -lah
total 8
drwxr-xr-x 3 haibin staff 102B Feb 15 00:57 .
drwxr-xr-x 17 haibin staff 578B Feb 15 00:39 ..
-rw-r--r-- 1 haibin wheel 2.4K Feb 15 00:57 stringutil.a
You need to delete the stringutil.go source file that is under the src tree. The *.a file is a binary file which results from the compilation (and possibly linking) of source files.
Below are the files in my local server
-rw-r----- 1 root root 0 Sep 25 15:03 one.xml
-rw-r----- 1 root root 0 Sep 25 15:03 two.xml
-rw-r----- 1 root root 0 Sep 25 15:03 data.csv
-rw-r----- 1 root root 0 Sep 25 15:03 free.png
-rw-r----- 1 root root 0 Sep 25 15:04 loaded.jpeg
I know transfering files of same extension as below
/usr/bin/sftp ${user}#${HostName} <<EOF
cd $InputPath
lcd $OutputPath
put *.csv
exit
EOF
scp ${InputPath}/*.csv ${user}#${HostName}:$OutputPath
But every time when i run the script i need to transfer only files with xml and jpeg extensions. ssh,scp,SFTP can be used. Any hep please??
You can try with something like this:
*.{jpeg,xml}
All together:
scp ${InputPath}/*.{jpeg,xml} ${user}#${HostName}:$OutputPath
Test
$ ls
a.jpeg a.png a.xml
$ ls *{jpeg,xml}
a.jpeg a.xml
I have installed the global npm package jslint and it lives here
$ ls -la /usr/local/bin/jslint
lrwxr-xr-x 1 lust admin 40 Feb 12 15:31 /usr/local/bin/jslint -> ../lib/node_modules/jslint/bin/jslint.js
$ ls -la /usr/local/lib/node_modules/jslint/bin
total 8
drwxr-xr-x 3 lust staff 102 Apr 16 2012 .
drwxr-xr-x 10 lust staff 340 Feb 12 15:31 ..
-rwxr-xr-x 1 lust staff 2330 Apr 16 2012 jslint.js
$ which jslint
/usr/local/bin/jslint
$ head -3 /usr/local/bin/jslint
#!/usr/bin/env node
var linter = require("../lib/linter");
So it is without any doubt whatsoever at this point that jslint is in fact being run from this dir and not here:
$ ls -la .npm/jslint/0.1.9/package/bin/
total 8
drwxr-xr-x 3 lust staff 102 Apr 16 2012 .
drwxr-xr-x 9 lust staff 306 Feb 12 15:31 ..
-rwxr-xr-x 1 lust staff 2330 Apr 16 2012 jslint.js
There appear to be two copies of the package, one in /usr/local/ and one in ~/.npm. Why is there one in .npm and is it safe for me to remove it?
~/.npm is a cache that npm uses to avoid re-downloading the same package multiple times. There's no harm in removing it. You can empty it with the command:
npm cache clean