Since macOS 11.3 broke my Perl script which I have been using to generate Sparkle appcasts for the last 12 years, I decided to instead start using the generate_appcast tool which has since been provided with Sparkle. Invoking generate_appcast with no arguments, I get some brief documentation which I interpret to mean that I should provide two arguments:
a -f followed by the path to my Sparkle private key file
the path to a directory of several recent versions of my app, all zipped
So I created a new directory and copied zip archives of the three most recent versions of my app into it. Those are the .zip archives, notarized by Apple, which I upload to my site for users to download.
Then I ran this command:
Air2:~ jk$ generate_appcast -f /path/to/My_Sparkle_priv.pem /path/to/directory/of/zips
The result:
Warning: Private key not found in the Keychain (-25300). Please run the generate_keys tool
Error generating appcast from directory /path/to/My_Sparkle_priv.pem
Error Domain=NSCocoaErrorDomain Code=256 "The file “My_Sparkle_priv.pem” couldn’t be opened." UserInfo={NSUserStringVariant=(
Folder
), NSURL=file:///path/to/My_Sparkle_priv.pem/, NSFilePath=/path/to/My_Sparkle_priv.pem, NSUnderlyingError=0x13a637e10 {Error Domain=NSPOSIXErrorDomain Code=20 "Not a directory"}}
Apparently it is not recognizing the key file I provided, and also oddly implies that it expects a directory instead of a regular file. In the brief documentation, there is an example marked [DEPRECATED] which omits the -f before the path to the key file, so I tried that but got the same result. I also tried putting the path to the zips first, but that result was even worse.
My key file is, I think, a pretty standard .pem text file that begins with the line -----BEGIN DSA PRIVATE KEY----- followed by 1133 ASCII characters, etc.
Where did I miss the boat?
Astonishingly, this seems to be due to an obvious programming error in the Sparkle generate_appcast Swift source code. In attempting to remove elements indexed N and N+1 from an array of command-line arguments, the code removes element N, and then removes element N+1, which of course removes elements N and N+2 instead. After I fixed this programming error, the problem is solved.
After I do some more head-scratching and maybe consulting with others smarter than me, I shall submit a pull request or whatever to the Sparkle project next week.
I have written a bit of code in python 2.7 to pull a list of files from a given directory and output to a csv file. Fairly simple and for the most part it works great. I am using this code to get the file names and add them to a list. Then I will print the list once gathered. There are command line options to determine which extensions to use. This is working fine.
for root, dirnames, filenames in os.walk(lines):
if searchtype == 'all':
for filename in fnmatch.filter(filenames, '*'):
matches.append(os.path.join(root, filename))
elif searchtype == 'audio':
for extensions in audio_ext:
for filename in fnmatch.filter(filenames, extensions):
matches.append(os.path.join(root, filename))
elif searchtype == 'video':
for extensions in video_ext:
for filename in fnmatch.filter(filenames, extensions):
matches.append(os.path.join(root, filename))
The issue occurs when I attempt to get the modified date before printing using this.
for entries in matches:
mod_date = datetime.datetime.fromtimestamp(os.path.getmtime(entries)).\
strftime ('%Y-%m-%d %H:%M:%S')
This works for some files and then errors out with error code 2.
[Error 2] The system cannot find the file specified: 'E:\\Mp3s\\Artists_A-D\\Beatles, The\\Anthology 1\\60 - Kansas City Hey-Hey-Hey-Hey!.mp3'
The file as printed from the list (matches) is as:
E:\Mp3s\Artists_A-D\Beatles, The\Anthology 1\60 - Kansas City Hey-Hey-Hey-Hey!.mp3
Now the file is there for sure and if I skip doing the modified date and just print out the file names there is no issues. It rolls through 50k files without issues. so I am bit stumped. At first I thought it was the ! messing up the path but it does not seem to be the case since it prints fine without the mod date. I even updated the mod date to see if that was it, still no joy. I am still fairly new to Python so any thoughts?
How can I use diff in terminal but ignore changes in file names?
Currently this is what i'm doing:
diff -wrN folder1 folder2 | grep '^>' | wc -l
How can I do git diff between two commit ids whilst:
ignoring file rename
only look at java files
ignore specific folder names e.g. folder 'a' and 'b'
perform the grep '^>' | wc -l
You seem unaware of the hardness of this problem, so I'd like to point out why this is so difficult.
Given two directories which are equal in the beginning and both contain, say, 1000 files. Now you rename, say, 500 files in one of the directories. Renamings can vary greatly. A file called foobar.txt originally can be named DSC-3457.orig.jpg afterwards. The diff command cannot really find it again without having any idea about what has been renamed into what.
Additionally, a file called x could be renamed to y, while a file called y could be renamed to x. In this case it even is questionable whether this should be regarded a mere renaming or if simply both files' contents have been exchanged.
This all means that in general you will have large problems to accomplish this. Standard tools will not do this out-of-the-box.
This said, I have two aspects I want to point out which might help you.
File Sizes
You can sort all files by their file sizes and then diff each pair of the two directories. This can work perfectly well if all changes you have are only renamings and if all files are of different size. If you have several files of the same size (maybe by pure chance or because they are all of the same format which has a fixed size), you are in trouble again and will have to compare each possible pair of the same-size group.
Git-Diff
You mentioned git-diff in the tags. git actually keeps a record in case a file is renamed. So if you intend to use git diff, you can rely to some degree on git's ability to detect renamings. This typically works if a file is removed and added with a new name in one single commit. If it gets added with a new name in one commit and then the older version is removed in another commit, this won't work properly. There is a lot more to learn about renames in git diff; see man git diff and search for rename in this case, there are about a dozen places this gets mentioned, so I won't try to summarize this here myself.
EDIT: You can use a command like git diff --find-renames --diff-filter=ACDMTUX (i. e. you let all kinds of changes pass the filter with the exception of renamings).
I am using Mac OS x 10.10 terminal & bash 3.2.57(1)-release.
I want to move all, but 3 most recent files.
Also, all the moved files should have modified date at the end of file name (not the current time, but the last modified date of every file).
For example :
before: aaaa12345.mp4
after: aaaa_20150405_200834.mp4
Following statement moves all, bust latest 3 files
mv -f $(ls -1t /Users/home/test/mv/* | tail -n +4) /Users/home/test/*
Questions is:
How to add modified date and remove 12345 at the end of all file names (except for latest 3 files), before moving?
I have a computer A with two directory trees. The first directory contains the original mod dates that span back several years. The second directory is a copy of the first with a few additional files. There is a second computer be which contains a directory tree which is the same as the second directory on computer A (new mod times and additional files). How update the files in the two newer directories on both machines so that the mod times on the files are the same as the original? Note that these directory trees are in the order of 10s of gigabytes so the solution would have to include some method of sending only the date information to the second computer.
The answer by Paul is partly correct, rsync is able to do this, however with different parameters. The correct command is
rsync -Prt --size-only original_dir copy_dir
where -P enables partial transfers and displays a progress indicator, -r recurses through subdirectories, -t preserves time stamps and --size-only doesn't transfer files that match in size.
The following command will make sure that TEST2 gets the same date assigned that TEST1 has
touch -t `stat -t '%Y%m%d%H%M.%S' -f '%Sa' TEST1` TEST2
Now instead of using hard-coded values here, you could find the files using "find" utility and then run touch via SSH on the remote machine. However, that means you may have to enter the password for each file, unless you switch SSH to cert authentication. I'd rather not do it all in a super fancy one-liner. Instead let's work with temp files. First go to the directory in question and run a find (you can filter by file type, size, extension, whatever pleases you, see "man find" for details. I'm just filtering by type file here to exclude any directories):
find . -type f -print -exec stat -t '%Y%m%d%H%M.%S' -f '%Sm' "{}" \; > /tmp/original_dates.txt
Now we have a file that looks like this (in my example there are only two entries there):
# cat /tmp/original_dates.txt
./test1
200809241840.55
./test2
200809241849.56
Now just copy the file over to the other machine and place it in the directory (so the relative file paths match) and apply the dates:
cat original_dates.txt | (while read FILE && read DATE; do touch -t $DATE "$FILE"; done)
Will also work with file names containing spaces.
One note: I used the last "modification" date at stat, as that's what you wrote in the question. However, it rather sounds as if you want to use the "creation" date (every file has a creation date, last modification date and last access date), you need to alter the stat call a bit.
'%Sm' - last modification date
'%Sc' - creation date
'%Sa' - last access date
However, touch can only change the modification time and access time, I think it can't change the creation time of a file ... so if that was your real intention, my solution might be sub-optimal... but in that case your question was as well ;-)
I would go through all the files in the source directory tree and gather the modification times from them into a script that I could run on the other directory trees. You will need to be careful about a few 'gotchas'. First, make sure that your output script has relative paths, and make sure you run it from the proper target directory, which should be the root directory of the target tree. Also, when changing machines make sure you are using the same timezone as you were on the machine where you generated the script.
Here's a Perl script I put together that will output the touch commands needed to update the times on the other directory trees. Depending on the target machines, you may need to tweak the date formats or command options, but this should give you a place to start.
#!/usr/bin/perl
my $STARTDIR="$HOME/test";
chdir $STARTDIR;
my #files = `find . -type f`;
chomp #files;
foreach my $file (#files) {
my $mtime = localtime((stat($file))[9]);
print qq(touch -m -d "$mtime" "$file"\n);
}
The other approach you could try is to attach the remote directory using NFS and then copy the times using find and touch -r.
I think rsync (with the right options)
will do this - it claims to only send
file differences, so presumably will
work out that there are no differences
to be transferred.
--times preserves the modification times, which is what you want.
See (for instance)
http://linux.die.net/man/1/rsync
Also add -I, --ignore-times don't skip files that match size and time
so that all files are "transferred' and trust to rsync's file differences optimisation to make it "fairly efficient" - see excerpt from the man page below.
-t, --times
This tells rsync to transfer modification times along with the files and update them on the remote system. Note that if this option is not used, the optimization that excludes files that have not been modified cannot be effective; in other words, a missing -t or -a will cause the next transfer to behave as if it used -I, causing all files to be updated (though the rsync algorithm will make the update fairly efficient if the files haven't actually changed, you're much better off using -t).
I used the following Python scripts instead.
Python scripts run much faster than an approach creating new processes for each file (like using find and stat). The solution below also works in case of timezone differences between systems, as it uses UTC times. It also works with paths containing spaces (but not paths containing newline!). It doesn't set times for symlinks, because the operating system provides no mechanism to modify the timestamp of a symlink, but in a file manager the time of the file the symlink points at is shown instead anyway. It uses a maxTime parameter to avoid resetting dates for files that are actually modified after copying from the original directory.
listMTimes.py:
import os
from datetime import datetime
from pytz import utc
for dirpath, dirnames, filenames in os.walk('./'):
for name in filenames+dirnames:
path = os.path.join(dirpath, name)
# Avoid symlinks because os.path.getmtime and os.utime get and
# set the time of the pointed file, and in the new directory,
# the link may have been redirected.
if not os.path.islink(path):
mtime = datetime.fromtimestamp(os.path.getmtime(path), utc)
print(mtime.isoformat()+" "+path)
setMTimes.py:
import datetime, fileinput, os, sys, time
import dateutil.parser
from pytz import utc
# Based on
# http://stackoverflow.com/questions/6999726/python-getting-millis-since-epoch-from-datetime
def unix_time(dt):
epoch = datetime.datetime.fromtimestamp(0, utc)
delta = dt - epoch
return delta.total_seconds()
if len(sys.argv) != 2:
print('Syntax: '+sys.argv[0]+' <maxTime>')
print(' where <maxTime> an ISO time, e. g. "2013-12-02T23:00+02:00".')
exit(1)
# A file with modification time newer than maxTime is not reset to
# its original modification time.
maxTime = unix_time(dateutil.parser.parse(sys.argv[1]))
for line in fileinput.input([]):
(datetimeString, path) = line.rstrip('\r\n').split(' ', 1)
mtime = dateutil.parser.parse(datetimeString)
if os.path.exists(path) and not os.path.islink(path):
if os.path.getmtime(path) <= maxTime:
os.utime(path, (time.time(), unix_time(mtime)))
Usage: in the first directory (the original) run
python listMTimes.py >/tmp/original_dates.txt
Then in the second directory (a copy of the original, possibly with some files modified/added/deleted) run something like this:
python setMTimes.py 2013-12-02T23:00+02:00 </tmp/original_dates.txt