Is there a short one-liner to get a file checksum, which works on both macos and ubuntu? It doesn't matter what algorithm or program, as long as I don't have to install or setup anything.
You could use OpenSSL, and the commands should be the same:
openssl sha256 filename | awk -F'= ' '{print $2}' # optional
Use whatever hashing algorithm you want, sha256, sha1, md5, etc.
Just try both of them:
md5 file 2>/dev/null; md5sum file 2>/dev/null;
That line will work on both OSs, running both commands and discarding the one that gives an error, it will print only the valid result.
With a quick OS check you can use either md5 (mac) or md5sum (ubuntu), alternatively you could alias one of them so you'd be using the same command on either OS.
On Linux, you can use md5sum file; on macOS, just md5 file. Both are default at a clean install, AFAIK. If you require that the command be the same, you can create an alias.
May I be so impertinent as to suggest writing your own?
python -c 'import sys, hashlib;
m = hashlib.sha256();
m.update(open(sys.argv[1]).read());
print("\t".join([m.hexdigest(), sys.argv[1]]))' file
The semicolons are gratuitous here, but necessary if you really want to force the issue and make this a literal one-liner.
Related
Note: I am particularly looking for a coding hack, not for an alternative solution. I am aware that awk, sed etc. can do this inline edit just fine.
$ echo '1' > test
$ cat test > test
$ cat test
$
Is there a way, to somehow make the second command output the original contents (1 in this case)? Again, I am looking for a hack which will work without visibly having to use a secondary file (using a secondary file in the background is fine). Another question on this forum solely focused on alternative solutions which is not what I am looking for.
You can store the content in a shell variable rather than a file.
var=$(<test)
printf "%s\n" "$var" > test
Note that this might only work for text files, not binary files. If you need to deal with them you can use encoding/decoding commands in the pipeline.
You can't do it without storing the data somewhere. When you redirect output to a file, the shell truncates the file immediately. If you use a pipeline, the commands in the pipeline run concurrently with the shell, and it's unpredictable which will run first -- the shell truncating the file or the command that tries to read from it.
With thanks to the comment made by #Cyrus to the original question
$ sudo apt install moreutils
$ echo '1' > test
$ cat test | sponge test
$ cat test
1
It does require installing an extra package and pre-checking for the binary using something like where sponge to check if it is installed.
if you happen to use macos, if the file isn't too gargantuan, you can always follow these steps :
perform the edits
pipe it to the clipboard (or "pasteboard" in mac lingo)
paste it back to original file name
|
{... edits to file1 ...} | pbcopy; pbpaste > file1
I am working on a battery of automatic tests which executes on 2 Unix virtual machines running with KSH. Those VMs are independant and they have practically the same .profile file. I would like to study their differences by launching:
tkdiff /usr/system/.profile system#{external_IP}:/usr/system/.profile
on the first VM but it doesn't work.
I suppose that directly accessing a hidden file is not possible. Is there a solution to my problem, or maybe an alternative?
If you want to compare different files on two remote machines, I suggest the following procedure:
1. Compare checksums:
First compare the checksums. Use sum, md5sum or sha256sum to compute a hash of the file. If the hash is the same, the probability of having the same file is extremely high! You can even increase that probability by check the total amount of characters, lines and words, in the file using wc.
$ file="/usr/system/.profile"
$ md5sum "$file" && wc "$file"
$ ssh user#host "md5sum '$file' && wc '$file'"
2. run a simple diff
Run a simple diff using the classic command line tools. They understand the POSIX standard to use - as /dev/stdin. This way you can do:
$ ssh user#host "cat -- '$file'" | diff "$file" -
note: with old versions of tkdiff or new versions of svn/git, it can be tricky here due to bugs in tkdiff. It will quickly throw errors of the form svn [XXXX] file .... is not a working copy or file xxxx is not part of a revision control system if one of the files might be under version control or you end up in a directory under version control. Stick to diff!
You are using the filename convention "user#host:/path/to/file" for the second argument to tkdiff.
That convention for naming is not native to Ksh, but instead is understood by some programs like scp and others (which can be interactive, e.g. to ask for a password for the remote system or other authentication related questions).
But from the tkdiff man page, it does not mention having built-in support for that filenaming convention userid#host:/path/to/file, and neither is such support built into ksh.
So you may need to use two steps, first to use scp or similar to copy the remote file locally then then use tkdiff with one argument the local file and the other the file-just-copied, or arrange to mount part of the other VM filesystem locally, and then use tkdiff with appropriate arguments.
Obviously, both files need to be readable by your userid or the user specified on the userid#host:/path/to/file for this to work.
You can directly made a remote ssh compare , run a remote display with help of cat command line, with this :
tkdiff <(ssh system#{external_IP}1 'cat /usr/system/.profile') <(ssh system#{external_IP}2 'cat /usr/system/.profile')
In your case to be able to compare with the local .profile file this :
tkdiff /usr/system/.profile <(ssh system#{external_IP} 'cat /usr/system/.profile')
Do you have just try with the simple diff command line (with -b -B option to remove blank line and space comparaison):
diff -b -B /usr/system/.profile <(ssh system#{external_IP} 'cat /usr/system/.profile')
I know that this topic has answers here, but I've got some problems and I would like to start from scratch.
First step is to create a key file:
openssl rand -base64 512 | tr -d '\r\n' > encrypted_data_bag_secret
but how to run this command on Windows? The tr command is not recognised.
I generated an openssl key and copied it to a txt file, then I was doing step by step like in Chef Docs but it doesn't work - a data bag isn't encrypted. I think I have to run this command above, but I don't know why to do this on windows
An equivalent in pure Ruby would be:
C:\chef\embedded\bin\ruby -e 'require "securerandom"; STDOUT.write(SecureRandom.base64(512))' > C:\chef\encrypted_data_bag_secret
Tweak the C:\chef path as appropriate, I don't have a Windows box handy to check the current default paths.
We may notice many download sites provide md5 string. For example, when I download ABC.zip, along with an md5 string like: “2743a6a9fe6f873df1c7ed8ac91df5d7 *ABC.zip”. I know the idea behind it, it’s Digest algorithm to prevent file forge.
My question is how a user calculates md5 string for the ABC.zip, and compare it with value site provides? Any existing tool to generate md5 string?
It depends a bit on your operating system ofcourse. Under most Linux/Unix distributions you have an md5 or md5sum program available.
Example:
# md5sum eclipse-SDK-3.6RC3-linux-gtk.tar.gz
8eca528d2c0b33dae10ba8750b2e4b94 eclipse-SDK-3.6RC3-linux-gtk.tar.gz
It also has a check mode which does exactly what you're looking for:
# md5sum -c test.md5
eclipse-SDK-3.6RC3-linux-gtk.tar.gz: OK
(test.md5 has the output of the previous command)
On Linux systems, the program is usually named md5sum.
On BSD systems, the program is usually named md5.
On Windows systems, aim users to http://en.wikipedia.org/wiki/Md5sum
Note that the md5sum and md5 utilities have a command-line option that can verify all the MD5 hashes listed in an MD5SUM file automatically:
sarnold#haig:~/bin$ md5sum * > /tmp/MD5SUM
sarnold#haig:~/bin$ md5sum -c /tmp/MD5SUM
aa-change: OK
aa-change.c: OK
briss: OK
mkvtom2ts: OK
muxer: OK
muxer_orig: OK
Depends. Programming languages usually have a library to do this.
OS X has a command line utility 'md5'. Linux has something similar. For windows, no idea, but you can probably find something easily.
For windows, if you install Cygwin, md5sum becomes available.
I need to join two binary files with a *.bat script on Windows.
How can I achieve that?
Windows type command works similarly to UNIX cat.
Example 1:
type file1 file2 > file3
is equivalent of:
cat file1 file2 > file3
Example 2:
type *.vcf > all_in_one.vcf
This command will merge all the vcards into one.
You can use copy /b like this:
copy /b file1+file2 destfile
If you have control over the machine where you're doing your work, I highly recommend installing GnuWin32. Just "Download All" and let the wget program retrieve all the packages. You will then have access to cat, grep, find, gzip, tar, less, and hundreds of others.
GnuWin32 is one of the first things I install on a new Windows box.
Shameless PowerShell plug (because I think the learning curve is a pain, so teaching something at any opportunity can help)
Get-Content file1,file2
Note that type is an alias for Get-Content, so if you like it better, you can write:
type file1,file2
Just use the dos copy command with multiple source files and one destination file.
copy file1+file2 appendedfile
You might need the /B option for binary files
In Windows 10's Redstone 1 release, the Windows added a real Linux subsystem for the NTOS kernel. I think originally it was intended to support Android apps, and maybe docker type scenarios. Microsoft partnered with Canonical and added an actual native bash shell. Also, you can use the apt package manager to get many Ubuntu packages. For example, you can do apt-get gcc to install the GCC tool chain as you would on a Linux box.
If such a thing existed while I was in university, I think I could have done most of my Unix programming assignments in the native Windows bash shell.
If you simply want to append text to the end of existing file, you can use the >> pipe. ex:
echo new text >>existingFile.txt
So i was looking for a similar solution with the abillity to preserve EOL chars and found out there was no way, so i do what i do best and made my own utillity
This is a native cat executable for windows - https://mega.nz/#!6AVgwQhL!qJ1sxx-tLtpBkPIUx__iQDGKAIfmb21GHLFerhNoaWk
Usage: cat file1 file2 file3 file4 -o output.txt
-o | Specifies the next arg is the output, we must use this rather than ">>" to preserve the line endings
I call it sharp-cat as its built with C#, feel free to scan with an antivirus and source code will be made available at request
I try to rejoin tar archive which has been splitted in a Linux server.
And I found if I use type in Windows's cmd.exe, it will causes the file being joined in wrong order.(i.e. type sometimes will puts XXXX.ad at first and then XXXX.ac , XXXX.aa etc ...)
So, I found a tool named bat in GitHub https://github.com/sharkdp/bat which has a Windows build, and has better code highlight and the important thing is, it works fine on Windows to rejoin tar archive!
Windows type command has problems, for example with Unicode characters on 512 bytes boundary. Try Cygwin's cat.
If you have to use a batch script and have python installed here is a polyglot answer in batch and python:
1>2# : ^
'''
#echo off
python "%~nx0" " %~nx1" "%~nx2" "%~nx3"
exit /b
rem ^
'''
import sys
import os
sys.argv = [argv.strip() for argv in sys.argv]
if len(sys.argv) != 4:
sys.exit(1)
_, file_one, file_two, out_file = sys.argv
for file_name in [file_one, file_two]:
if not os.path.isfile(file_name):
print "Can't find: {0}".format(file_name)
sys.exit(1)
if os.path.isfile(out_file):
print "Output file exists and will be overwritten"
with open(out_file, "wb") as out:
with open(file_one, "rb") as f1:
out.write(f1.read())
with open(file_two, "rb") as f2:
out.write(f2.read())
If saved as join.bat usage would be:
join.bat file_one.bin file_two.bin out_file.bin
Thanks too this answer for the inspiration.