So I am working on server files and have been successful in using Robocopy to make either copies of files or new ones in the designated directories.
robocopy "\\DIRECTORY\FOLDER\FOLDER1\FOLDER2" F:\DESTINATION\ /MIR /V /ETA /R:0 /W:0 /COPY:DATSO /LOG:"%logpath%%filename%" /NP
The thing I want to do now and what I am trying to focus on is being able to separate or at least group together all "NEW" files and all "SAME" files together because I get nearly 200k lines in the log and I don't want to be digging around when it can be just separated between the two categories and then grouped under which category the files fall under.
SAME
SAME
SAME
SAME
NEW
NEW
NEW
NEW
^That is the desired output but currently getting:
SAME
sAME
NEW
NEW
NEW
SAME
SAME
I am just reading through the manual as I am trying to make this so my explanation is hopefully clear. I am just focusing on getting the two separate after using Robocopy but the end goal is to pipeline all the new files somewhere else (that comes later though).
Related
I have a directory of images, currently at ~117k files for about 200 gig in size. My backup solution vomits on directories of that size, so I wish to split them into subdirectories of 1000. Name sorting or type discrimination is not required. I just want my backups to not go nuts.
From another answer, someone provided a way to move files into the split up configuration. However, that was a move, not a copy. Since this is a backup, I need a copy.
I have three thoughts:
1. Files are added to the large directory with random filenames, so alpha sorts aren't a practical way to figure out deltas. Even using a tool like rsync, adding a couple hundred files at the beginning of the list could cause a significant reshuffle and lots of file movement on the backup side.
2. The solution to this problem is to reverse the process: Do an initial file split, add new files to the backup at the newest directory, manually create a new subdir at the 1000 file mark, and then use rsync to pull files from the backup directories to the work area, eg rsync -trvh <backupdir>/<subdir>/ <masterdir>.
3. While some answers to similar questions indicate that rsync is a poor choice for this, I may need to do multiple passes, one of which would be via a slower link to an offsite location. The performance hit of using rsync and its startup parsing is far superior to the length of time reuploading the backup on a daily basis would take.
My question is:
How do I create a script that will recurse into all 117+ subdirectories and dump the contained files into my large working directory, without a lot of unnecessary copying?
My initial research produces something like this:
#!/bin/bash
cd /path/to/backup/tree/root
find . -type d -exec rsync -trvh * /path/to/work/dir/
Am I on the right track here?
It's safe to assume modern versions of bash, find, and rsync.
Thanks!
I have a windows script here that takes all files in a source folder, whether they're nested or not, and moves them to one single destination folder.
However I have some duplicates within the source folders and every time it comes across one it prompts to copy or skip it. It is quite cumbersome as some of the source folders have a lot of duplicates.
Can someone advise on how to edit this script to make it auto skip any duplicates without prompting me every time or maybe even copy both and just append the name of the duplicate? Either one would work. I am still new to the whole batch script scene and it is driving me nuts.
The script I have is:
FOR /R "C:\source folder" %i IN (*) DO MOVE "%i" "C:\destination folder"
Thank you!
So I am looking for a tool that can compare files in folders based on checksums (this is common, not hard to find); however, my use-case is that the files can exist in pretty deep folder paths that can change, I am expected to compare them every few months and ONLY create a package of the different files. I don't care what folders the files are in, the same file can move between folders regularly and files wouldn't change names much, only content (so checksums are a must).
My issue is that almost all of the tools I can find do care about the folder paths when they compare folders, I don't and I actually want it to ignore the folder paths. I rather not develop anything or at least only have to develop a small part of the process to save time.
To be clear the order I am looking for things to happen are:
Program scans directory from 1/1/2020 (A).
Program scans directory from 4/1/2020 (B)
Finds all files where checksum in B don't exist in A and make a new folder with differences (C).
Any ideas?
The scenario:
I have thousands of pictures (Both in .jpg and RAW), where each pair has the same name. I have looked through the JPGs and deleted hundreds of undesirable photos. Now I want the corresponding RAWs (The ones without "partner") deleted.
Everything is allowed: cmd, Windows Powershell, Scripts, hidden functions in Windows Explorer, ...
Speed is absolutely unimportant, you could have opened Windows Explorer, sorted by name highlighted all instances and deleted them in the time it took you to post this message and wait for an answer to your off-topic question.
Open a Command prompt window, (cmd.exe), CD to your directory holding these files; then enter this:
For %A In (*.RAW) Do #If Not Exist "%~nA.jpg" Del "%A"
I'd like to change folder contents without having Date modified change.
I sometimes do cleanup runs on old folders, trying to make space or clean up temp files and dead-ends, or adding relevant links or tags for findability. When I do this, I don't want to change the folder to Date modified: today 2015, because my folders are sorted by Date modified. A project from 2010 should remain timestamped with its last modified date from 2010, because I've only made meta-changes, not actual changes.
Currently I use SK Timestamp or Attribute Changer. I right click each folder before I want to make changes to it, I keep the Properties window open, make my modifications, then hit Apply to overwrite the original timestamps.
I'd like to do it more automated, like set a Cleanup Mode on the root folder D:\Workspace and until I pop that state no timestamps ever get changed in the subdirectories Project2010, Project2013..., Project2015
Either that or at least be able to copy the timestamp between 2 files. Like in this answer that mentions #COPY /B %1+,, %1, but not with current date.
Usage like: touch D:\Temp\Project2010\Source.txt Destination.ext
I had commented above with a couple of suggestions -- one involving modifying the system's date / time, and another involving robocopy. The system date / time one is a bit of trouble and requires Internet access, and the robocopy /dcopy:t switch didn't work at all in my tests.
But I found a better solution anyway. Use a for loop to capture the folder's date time to a variable. Then after you've made whatever changes you wish, use powershell to put the folder's date / time back the way it was.
#echo off
setlocal
set "dir=path\to\directory"
for %%I in (%dir%) do set "datetime=%%~tI"
:: Do whatever deletions and other maintenance you want here.
:: Then after all changes have completed, reset the dir timestamp.
powershell -command "(Get-Item '%dir%').LastWriteTime=(Get-Date '%datetime%')"