Iterate through files from UNC path to Local Directory - windows

Given a UNC and a local Directory how would I go about copying files from remove to local in a simple, recursive fashion.
eg: - This works currently and copies all files within MyDIR to MyDEST:
SET MyDIR="\\123.10.10.10\Logs\Files\Server12\Folder1"
SET MyDEST="C:\Users\Desktop\Logs\Files"
robocopy %MyDIR% %MyDEST% /MIR
The issue I'm running into is that within MyDIR, Server12 has multiple (usually 3) folders stored and within them are the files I need so some level of iteration would help.
So I could try a bit of manual work and
SET MyDIR1=.....Folder1"
SET MyDIR2=.....Folder2"
SET MyDIR3=.....Folder3"
...as needed
but I'd also have to
robocopy %MyDIR1% %MyDEST% /MIR
robocopy %MyDIR2% %MyDEST% /MIR
robocopy %MyDIR3% %MyDEST% /MIR
which is easy enough but is there a way to loop through a list or even better, a way to recurse through all of Server12's folders and copy over all files?
I was thinking of using Powershell but I'm just trying to throw something together for quick, occasional use. I'm willing to but not sure if I should go in that direction as I have a bit of experience in it but not much.
I was able to get some for loops working but had errors when trying to get this to work with UNC the paths ( as well as using the %CD% variable to list the a UNC directory ).

In the end copy worked for my needs but a problem with my local paths containing a dash (yes a '-' because my company uses OneDrive it's in the path) was making it difficult to get this working. I passed FORFILES my variable, which contained the dash (demonstrated below).
In the end I had to use a temp folder....
SET MyRootDIR="\\123.10.10.10\Logs\Files\Server12"
SET MyDEST="C:\Users\Desktop\Logs - files\Files"
SET MyTempDEST="C:\LOGSTemp"
REM Delete these folders to empty them
DEL /Q %MyDEST%\*.*
DEL /Q %MyTempDEST%\*.*
PushD %MyRootDIR% &&(
FORFILES /M *.* /S /C "cmd /c copy #file %MyTempDEST%"
) & PopD
MOVE %MyTempDEST%\*.* %MyRootDIR%
I need the files only, because my next steps are to decompress the files and perform a keyword search within them outputting the results to a results.txt file.
Variables are at top to make it easier to change to difference server instances.
It works well but it has to be done in two steps. In PushD if I change it to to my variable %MyRootDIR% it gives me error :
ERROR: Invalid argument/option - '-'.
Tried escaping etc with no luck but in the end the above works

Related

Bash robocopy from text file containing path to folders and files to copy with special characters

I had a bash xcopy script, but got a lot of issues. After a lot of searching and reading I guess robocopy was a better tool to do the copy.
(The script must run on windows 10 computers without installing anything else and without internet access)
I'm trying to make a bash script that copy (with robocopy) some local network folders and files to a local custom directory. The aim is to be able to access to the files off from the local network.
The path to folders and files are stored inside a txt file (each line = a path)
I want to keep the structure of folder I save locally.
For example the folder X:\Path\to some\local\network\folder\with\some & characters\ will result in C:\PathTolocalFolder\Path\to some\local\network\folder\with\some & characters\ (without the X:\ letter)
Based on many similar questions (but not all at the same time) I have done this :
#echo off
SETLOCAL EnableDelayedExpansion
cls
cd C:
chcp 28591 > nul
for /f "delims=*" %%a in ('type "X:\path with spaces & specials characters\List.txt"') do (
REM echo %%a
REM echo !%%a!
echo %%a to C:\PathTolocalFolder!%%a!
ROBOCOPY "%%~a" "C:\PathTolocalFolder!%%a!" /S /A+:RA /R:1 /W:5
)
It is partially a success, but :
As there are special characters everywhere in paths and files names, I got some issues. Specially with & characters. My double quotes doesn't solve the problem. How could I go better?
For some cases, I want to save some files but not the whole directory where they are. The full path to the file is inside the text file. But as robocopy needs to add a space between folder path and file filter I have do some manipulation. How can I detect and extract the file name when there is one to adapt the robocopy command?
I want to use an exclusion list like I was doing before with xcopy. But robocopy doesn't accept a file in input for exclusions. I tried this to extract the exclusion file:
for /f "usebackq tokens*" %%D in ("C:\path to exclusion file\exclusions.txt") do (
if NOT "!dirs!"=="" (
Set dirs=!dirs! "%%D"
else (
Set dirs ="%%D"
)
)
But doesn't really know what I am doing and how to combine with the first part.
Bonus questions I'm using the robocopy log file functionality (removed from below) is there a way to archive (by adding the date in the name for example) previous log file before creating the new one? Is it possible to remove the progress percents in the log file but to display it in the terminal instead? How to use the "/np" option for log file but not for terminal display?
It's hard to me to understand how the delayed variables are working in batch files and how the different methods to read a file or variable are working.
Any help is welcome :)
Sorry for my bad English skills
thank for having read

robocopy copy files to a new folder in the current folder without the new created folder

i want to copy the WHOLE folder in a new subfolder. currently i am using:
(robocopy "C:/Test" "C:/Test/a" /E> output.log) ^& IF %ERRORLEVEL% LSS 8 SET ERRORLEVEL = 0
after running this, there is in "C:/Test/a" also the "/a" folder. i dont want that new folder.
The robocopy command seems to get confused when a full source directory tree is to be copied and the destination directory is located somewhere in the source directory tree. Therefore your command line results in a directory C:\Test\a\a that partially contains stuff from the source.
Try with the /L option (do not copy but list items that would be copied), so the output log shows exactly what you would expect. As soon as you actually do copy (so with /L removed), there are some more items listed in the log unexpectedly. This behaviour indicates that robocopy does not evaluate the whole source directory tree in advance, but during the actual copying operation.
The easiest way to avoid that is to exclude the destination directory from being (re-)copied:
robocopy "C:\Test" "C:\Test\a" /E /XD "C:\Test\a"
By the way, the correct Windows path separator is the \. Although robocopy also accepts /, you should stick to \, because many commands do misinterpret the /.

Need a BAT file to copy & rename all files in specific tree

I need to create a .bat that runs through a multilayered directory... copying certain files that contain the following suffix: '.full.jpg' to save as '.jpg'
What I've tried:
copy /y "C:\Users\myname\Desktop\maindir\*.full.jpg" "C:\Users\myname\Desktop\maindir\*.jpg"
However, I cannot get it to work.
The .bat is located in the 'maindir' directory and ran from the terminal (cmd).
Here's an example scenario that maps closely to mine:
Existing Files:
C:\Users\myname\Desktop\maindir\a\a\picture1.full.jpg
C:\Users\myname\Desktop\maindir\a\a\picture3.full.jpg
C:\Users\myname\Desktop\maindir\a\b\picturea.full.jpg
C:\Users\myname\Desktop\maindir\a\b\pic1.full.jpg
C:\Users\myname\Desktop\maindir\b\a\foto.full.jpg
C:\Users\myname\Desktop\maindir\b\a\photo.full.jpg
C:\Users\myname\Desktop\maindir\b\b\pic1.full.jpg
C:\Users\myname\Desktop\maindir\b\c\pi2.full.jpg
Example Output Wanted:
C:\Users\myname\Desktop\maindir\a\a\picture1.full.jpg
C:\Users\myname\Desktop\maindir\a\a\picture1.jpg
C:\Users\myname\Desktop\maindir\a\a\picture3.full.jpg
C:\Users\myname\Desktop\maindir\a\a\picture3.jpg
C:\Users\myname\Desktop\maindir\a\b\picturea.full.jpg
C:\Users\myname\Desktop\maindir\a\b\picturea.jpg
C:\Users\myname\Desktop\maindir\a\b\pic1.full.jpg
C:\Users\myname\Desktop\maindir\a\b\pic1.jpg
C:\Users\myname\Desktop\maindir\b\a\foto.full.jpg
C:\Users\myname\Desktop\maindir\b\a\foto.jpg
C:\Users\myname\Desktop\maindir\b\a\photo.full.jpg
C:\Users\myname\Desktop\maindir\b\a\photo.jpg
C:\Users\myname\Desktop\maindir\b\b\pic1.full.jpg
C:\Users\myname\Desktop\maindir\b\b\pic1.jpg
C:\Users\myname\Desktop\maindir\b\c\pi2.full.jpg
C:\Users\myname\Desktop\maindir\b\c\pi2.jpg
I'd appreciate any help towards this as I haven't been able to do it yet. I will run across a directory structure whereby the top level directory will contain 15+ directories and each containing 20+ directories with 100+ files in each lowest directory.
Thanks.
#ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
SET "sourcedir=u:\Users\myname\Desktop\maindir"
FOR /r "%sourcedir%" %%a IN (*.full.jpg) DO (
FOR %%b IN ("%%~dpna") DO ECHO(COPY "%%a" "%%~dpnb.jpg"
)
GOTO :EOF
The inner for examines the drive-path-name only of the complete filename in %%a (ie. it drops the .jpg) and delivers the drive-path-name of that name (ie. drops the .full) to which you add .jpg and job done.
You would need to change the setting of sourcedir to suit your circumstances.
The required COPY commands are merely ECHOed for testing purposes. After you've verified that the commands are correct, change ECHO(COPY to COPY to actually copy the files.

Batch or VB Script to copy logs from multi directories to one directory and rename to avoid conflicts

Hope you can help. I have been trying to resolve this for a week but not getting anywhere and can't quite piece together what I need! - My scripting skills are far from great so please forgive my naivety!
Ok, The Problem......
I have an IIS server that has multiple sites that all save their logs in a separate directory, I need to copy the logs from the last 24 hours to a local directory on my machine so I can analyse these in Log Parser Lizard (GUI Version) on a daily basis.
I can map a drive from the remote server to my local machine via a hardware VPN, so this makes things a bit easier. Using the forfiles command I can re-curse the directories to find the logs that are only a day old, and using either copy/xcopy/ or Robocopy I can set a command to copy. My problem is that the IIS logs all have the same name so my copy command just keeps overwriting the previous file, rather than creating a new file. I have tried using the %random% parameter for the file name, but this again creates one random file that is overwritten with the next file, keeping the same name instead of creating lots of randomly named files in one directory.
I know that Log Parser commands include recurse, which I have used successfully, however the format of the log is changed slightly and the GUI Lizard cannot read the data within, so this is not a solution.
My code as it stands at this time is shown below, with IP's changed for obvious reasons. Any help would be greatly appreciated!
#echo off
NET USE Q: /Delete /yes
NET USE Q: \255.255.255.255\D$\Logs
cd C:
RD /S /Q C:\Weblogs\Production
MD C:\Weblogs\Production
forfiles.exe /p Q:\ /s /m *.log /d 0 /c "cmd /c robocopy /S /XC /XN /XO #file C:\Weblogs\Production\%random%.log"
NET USE Q: /delete
exit
%RANDOM% does not work for you in this case because it does not get solved per each iteration but only once at the forfiles invocation.
You'll need either to use in FORFILES some unique identifier, maybe concatenating #RELPATH and #FNAME may work for you in case you have only one level deep recursion.
Or either replace FORFILES with a FOR loop. Inside the loop you may have more freedom to calculate a unique ID, maybe a simple counter might work for you.
Edit: see this simple code sample, to get you started
#ECHO off
SETLOCAL enabledelayedexpansion
SET destdir=C:\LOGS
SET count=%RANDOM%
FOR /R %%A IN (*.log) DO (
SET /A count += 1
#ECHO COPY %%A !destdir!\%%~nA.!count!.log
)

How can I move the contents of one directory tree into another?

I have a directory which contains files and a number of levels of subdirectories:
C:\Source
I would like to move the contents of C:\Source into:
C:\Destination
Requirements:
All files and all subdirectories
within C:\SourceData must be moved
I will be running the command in a
batch file
I can't use Powershell or
any other scripting languages
Attempt 0
XCOPY /E "C:\Source" "C:\Destination"
This works perfectly, but it copies instead of moves. I can't copy then delete the source as I'm moving a very large set of files and there isn't enough disk space to have two copies of them at a time.
Attempt 1
MOVE "C:\Source" "C:\Destination"
This moves the entire C:\Source directory into C:\Destination so I end up with:
C:\Destination\Source
Attempt 2
With some help from this question and accepted answer I came up with:
for /r "C:\Source" %%x in (*) do move "%%x" "C:\Destination"
This moves the files within C:\Source but not the subdirectories or their contents. Note that I used %%x instead of %x as I'm using it in a batch file.
Using FOR seems promising but I'm not sure I've used the right syntax? Have I missed something?
Attempt 3
As suggested by Nick D, I tried rename:
RENAME "C:\Source" Destination
For the example scenario I gave this works fine. Unfortunately my real Destination directory is at a different level to the Source directory and this doesn't seem to be supported:
C:\>REN /?
Renames a file or files.
RENAME [drive:][path]filename1 filename2.
REN [drive:][path]filename1 filename2.
Note that you cannot specify a new drive or path for your destination file.
I get "The syntax of the command is incorrect." errors if I try to specify a more complex destination path, for example:
RENAME "C:\Source" "C:\MyOtherDirectory\Destination"
RENAME "C:\Source" "MyOtherDirectory\Destination"
Undoubtedly use robocopy. It is a simple but brilliantly useful tool.
robocopy /move /e sourcedir destdir
This will move all the files and folders, including empty ones, deleting each original file after it has moved it.
If you don't have robocopy installed you can download it by itself or as part of a Microsoft resource kit.
Update:
Adjusted code to a. check whether folders already exist at the destination, in which case move files in that folder over (and continue traversing the source directory structure), otherwise move the folder wholesale.
At the end of the script the source folder is removed altogether to eliminate these folders which have had their files moved over to an already existent folder at the destination (meaning these folders have been emptied but not deleted at the source).
Additionally we check whether a folder is both empty and already exists at the destination in which case we do nothing (and leave the source folder to be deleted to the last line of the script). Not doing this results in "The filename, directory name, or volume label syntax is incorrect." errors.
Phew! Please let me know how you get on with this! I have tested this and it seems to be working well.
for /d /r "c:\source" %%i in (*) do if exist "c:\destination\%%~ni" (dir "%%i" | find "0 File(s)" > NUL & if errorlevel 1 move /y "%%i\*.*" "c:\destination\%%~ni") else (move /y "%%i" "c:\destination")
move /y c:\source\*.* c:\destination
rd /s /q c:\source
In response to ufukgun's answer:
xcopy C:\Source\* C:\Destination\* /s /e /i /Y
/s - Copies directories and subdirectories except empty ones.
/e - Copies directories and subdirectories, including empty ones. Same as /S /E. May be used to modify /T.
/i - If destination does not exist and copying more than one file, assumes that destination must be a directory.
/y - Suppresses prompting to confirm you want to overwrite an existing destination file.
As stated in another answer, using xcopy may be not as efficient as it would be a native move command that only changes pointers in the filesystem.
Since XCOPY works, you could use XCOPY and DELETE, it's a workaround, but should work?
on Vista use
robocopy source destination /MIR
/MIR .. mirror a complete directory tree (also deletes files in the destination)
else
xcopy
Of course you have to delete the source afterwards :)
I have a directory which contains
files and a number of levels of
subdirectories:
C:\Source
I would like to move the contents of
C:\Source into:
C:\Destination
Maybe I'm missing something, but can't you just rename the folder?
As sent on twitter:
Try combining attempt 3 with attempt 1. Rename to get the destination folder correct, then move "Destination" to the correct place.
#echo on
set SOURCE=C:\Source
set DESTINATION=C:\Destination
xcopy %SOURCE%\* %DESTINATION%\* /s /e /i /Y
PAUSE
i use batch file like this...
or simply call:
xcopy C:\Source\* C:\Destination\* /s /e /i /Y
I use a utility called xxcopy. It can move files and have many useful options, you can use it like this :
xxcopy C:\Source C:\Destination /E /RC
the options :
/E to copy everything even empty folders
/RC to remove every source file after each successful copy
you can download free copy of xxcopy for personal use from :
http://www.xxcopy.com/xcpydnld.htm
Maybe a little off-topic, but still usefull:
Some people suggest to just copy + delete the source files, but
moving files is not the same as copying + deleting!
When using the (x)copy function, you allocate new space on the same volume on a harddisk. After the copying the files, the old files (the old allocated space which was required for the files) are beign marked as deleted. While you will achieve the same result as an end-user, moving does something diffrent and more efficient.
Moving files actually only changes some records in the MFT (master file table). These changes only consist of a different path for the user to locate its files. Physically the files still remain in the same sectors on the harddisk.
If all your trying to do is move a directory and the content up one level:
MOVE folder_you_wan_to_move ..
Note that .. refers to the next directory up.
I leave this code I have written based on ljs's attempt,
It moves directory trees as Windows does, overwriting the files that already exists in destination with those in source, and avoiding the slow copy and erase method except if the destination is in a different drive.
If your S.O is not in english must change line 6 with the text used by the Dir command when it finds 0 files.
movedir.bat source_dir destination_dir
#echo off
SETLOCAL ENABLEDELAYEDEXPANSION
if %1.==. echo movedir dir1[\*] dir2 & echo move dir1 inside dir2 overwriting coincident files & echo option \* moves only dir1 content & goto end
set S=%~f1
set D=%~f2
set "noFiles= 0 File"
set "R=0" rem R is option [\*] flag
if %S:~-2%.==\.. set "S=%S:~,-2%" & set "R=1"
if not exist "%S%" goto paramERR
if not exist "%D%" goto paramERR
set "Trim=0" & set "Sc=%S%"
:LP
set /A Trim+=1 & (set "Sc=!Sc:~1!") & if NOT !Sc!.==. goto LP
if %R%==0 (if exist "%D%\%~n1%~x1" (set "D=%D%\%~n1%~x1")) else move /y "%S%" "%D%" & goto end
CALL:movefiles "%S%" "%D%"
for /D /R "%S%" %%I in (*) do (set "Way=%%~fI") & (set "Way=!Way:~%Trim%!") & if exist "%D%!Way!" (CALL:movefiles "%%I" "%D%!Way!") else (move /y "%%I" "%D%!Way!\.." > NUL)
rd /s/q "%S%" & if %R%==1 md "%S%"
goto end
:movefiles
dir %1 | find "%noFiles%" > NUL & if ERRORLEVEL 1 move /y "%~1\*.*" %2 > NUL
goto :eof
:paramERR
echo Source or Destination does not exist
:end
I think the easiest thing you could do is to modify your Attempt 2
from
MOVE "C:\Source" "C:\Destination"
to
MOVE "C:\Source\*" "C:\Destination"
KISS ;-)
Edit: this seem not to work, so my advice is to trash away the crappy DOS command line and use Cygwin with BASH as a shell! (or just add the cygwin binaries to the path so you can use mv within DOS, thus not having to change the shell as your requirements state).

Resources