I want to copy the newest file that is on a mapped network directory. I already have the network directory mapped to ${drive.letter} using NET USE.
<copy todir="${dest.dir}" overwrite="true">
<first>
<sort>
<fileset dir="${drive.letter}\">
<filename name="FileIWant-*.tar.gz" />
</fileset>
<date xmlns="antlib:org.apache.tools.ant.types.resources.comparators"/>
</sort>
</first>
</copy>
It takes an incredibly long time for this task to complete and am wondering why it takes so long and if I can speed it up. The network directory has 20K+ files in it. Does the sort task run first, then the fileset selector? Or is it just going to take awhile since it is going over the network?
The only other solution I can think of it to copy all FileIWant-*.tar.gz files locally then perform the sort but I am unsure if the copy will change the timestamp.
If you were using UNIX, I'd say to use to do this in a one line command line. Using ls/find, sort and head, this can be a one liner. Which has the benefits of not having a lot of extra work (Ant is busy building the fileset and doing lots of things one at a time over a network) and letting the OS optimize it.
I then searched how to do that in Windows and found this one liner. Same idea. I suspect it will be faster to do on the OS level. (although not as fast as having the files locally.)
Another alternative is to run dir and parse/sort that result. I can't imagine copying the files locally is going to be faster if it has to be done at runtime.
I ended up writing a bat script to do this for me as it is much faster and I just call it from my Ant script.
echo Logging on to Server
net use %NetworkDir%
if not errorlevel 0 goto error
pushd %NetworkDir%
for /f "tokens=*" %%A in ('dir %FileIWant% /b /o:D') do (set sourceFile=%%A)
echo Copying %%A...
xcopy /V/F/Z/Y "%sourcefile%" "%copyTo%"
popd
Related
I know this question might have been asked in other ways, but all the things I have read and tried had not yet fixed my problem, so I am hoping to get some help here with context to my issue.
The problem:
I need to move files from a local drive to a network drive (the network drive is a SharePoint mapped library) on my server in Windows Azure (don't think the azure part matters, but it provides context).
My thought was to schedule a task that will run a bat file to move the files I need moved and do so frequently (every 5 to 10 minutes). The batch file I have created does what I need it to when I manually run the batch file but not when the task runs it.
Here is the batch file:
echo Write log file > LogStart.txt
C:\Windows\System32\robocopy.exe "\\PCICSWKS001\D$\ToBeMoved" "V:" /s /e /MOV /r:0 /W:0
echo Write log file > LogEnd.txt
You can see that it writes a log file before and after running robocopy.exe.
When the task runs it does write both of these log files so I know that the batch file is at least running.
You can also see that I have tried using the UNC path for the drive in the source, that is because I was reading that the task scheduler might not be able to pick up properly on the drive letters. When I put the machine name in for the destination and run the batch file myself or with the scheduler it errors.
Here is the action taken by the task:
Here is the task general tab
Any assistance would be much appreciated.
I had that same error with Robocopy after a few times of running it:
ERROR 3 (0x00000003) Getting File System Type of Destination
I think it has to do with how robocopy scans the destination when it has a fair amount of files on it. It worked fine when I started the copy job and the Azure destination was empty.
Anyway, I think you should try to use the AzCopy command as that should be less error prone and faster since it was designed for this kind of thing. It's command line switches are similar to robocopy so it should feel pretty familiar.
I am using a batch file to view the contents (xml files) of a certain folder.
I need to use a batch file for it.
Found this command on internet and it is working perfectly when I type in the command prompt, but when I type it in and save as a batch file, it doesn't give any out put at all. (Basically not running the content)
FOR /R D:\Myfolder %F in (*.*) do rename %~nF.xml %~nf1.xml
There are no restrictions in the folder either.
Shouldn't have worked from the prompt.
In a batch file, you need to change any % for the metavariable (%F in this case) to %%.
Having said that however, (or, from the prompt, %F and %f) are two different animals. It's virtually the only situation where batch is case-sensitive. Your command attempts to rename files using their NAME only (~nF) so if it was to encounter fred.txt it would attempt to rename Fred.xml to -er, 1.xml (I think, maybe %~nf.xml)
Best to say what you're trying to do. We're reasonably slick at crufting up solutions...
Lets say I want to copy a file or run a program, but the file or the location is not found. How can I let my computer pause the command window when these kind of problems occure? I made a batch file to copy files to another location, but also to run programs.
Any help will be appreciated.
In your batch script, use xcopy to do the actual copying, rather than the copy command -
xcopy, unlike copy, returns error codes depending on the result of the copy, which are documented here in the remarks section.
As Aleksandr mentioned, you can use the error codes as part of your script to pause on error.
Assuming you would like to pause on all errors, you could do script this as below:
xcopy /HECY <source> <destination>
if %ERRORLEVEL% NEQ 0 pause
Obviously you'll need to replace and in the above with the locations you are copying from and to respectively.
The /HECY switches are just an example, but in this case could be used to instruct xcopy to copy hidden files, recurse directories, and automatically overwrite files in the destination if they exist. You can tweak these to your specific needs.
Check the error code and pause if it is non zero
In the Windows command shell, cmd, is there a way to C:> dir /s but only to n sub directories?
Seems to me that I have made this work in the past, though I may have corrupted that sector in my brain with certain cell damaging liquids... :P
I guess I could redirect the output into a log and then skim it with findstr or qgerp but it seems like there should be a cleaner way.
"Cleaner way"? With DOS commands like "dir"? You've got a dry sense of humor, there ;)
You can do a "dos /?" to query available options. "limit recursion to 'n' directores with /s" is definitely not one of them.
If you want to do that, you'd need to write your own program ... or redirect to a text file and parse the file.
I am having a problem copying large DB files (~100GB) in an automated script I am trying to write for a Windows Server. I have tried using "copy", "robocopy", and even "eseutil".
My script is running on a Windows 2008 Server (destination of the file) and is pulling from a Windows 2003 Server (source of the file).
I have already tried changing the IRPStackSize registry setting, as well as both of the ones in the HKLM/SYSTEM/CurrentControlSet/Control/SessionManager/MemoryManagement hive. This was all done on the 2008 server and rebooted with no effect. Does anyone have a good workaround?
Copy and Robocopy both give me this:
Not enough server storage is available to process this command.
Eseutil.exe gives me this:
H:\TempSQLBackups>eseutil /y \\SRC_SERVER\SQL_BACKUPS\BIG_DB.BAK /d H:\TempSQLBackups\BIG_DB.bak
Extensible Storage Engine Utilities for Microsoft(R) Exchange Server
Version 08.01
Copyright (C) Microsoft Corporation. All Rights Reserved.
Initiating COPY FILE mode...
Source File: \\SRC_SERVER\SQL_BACKUPS\BIG_DB.BAK
Destination File: H:\TempSQLBackups\BIG_DB.bak
Copy Progress (% complete)
0 10 20 30 40 50 60 70 80 90 100
|----|----|----|----|----|----|----|----|----|----|
........FAILURE: ReadFile: The specified network name is no longer available.
Operation terminated unsuccessfully after 11336.16 seconds.
H:\TempSQLBackups>
I figured out how to fix this!
Use a LINUX machine to samba mount the source and destination directories/drives and copy them via the network. Personally, I use rsync since it will recreate the directory structure and only copy files that aren't there or are different. Thus, you can stop and/or restart at anytime without losing your progress.
I can't believe we're still paying Microsoft for this trash of an OS. I had similar problems and there seems to be no fix other than this one. Its a little slow but not nearly as slow as doing on natively since it will fail EVERY TIME.
At one point I thought robocopy would surely do it using the /IPG:xx option (InterPacketGap in milliseconds). Nope. It just PROLONGS the stack overflow and remote console lockout. I thought, maybe, Microsoft got it right with this OS. So much for Win2K8 being solid. Ugh! Windoze is for workstations. For servers you need a server OS not tinkertoy code.
Use the XCOPY with the /J option to avoid network failures of large files. This will ONLY works in 2008 R2 and Windows 7 though. This solved my timeout issue.
Please check.
Have you tried to copy the files with the old fasion way of drag and drop?
I would do this once, to make sure its not your network failing. Make sure that works, and then try look at other solutions.
1) Make sure your destination drive, is NTFS and NOT Fat32.
2) Check when its failing to copy, is it always at the same point? ( IE if it always failing after 2gb )
Have you tried xcopy? It works better for large files and recursive copy. doc
Also, from my own experience working with network drives and command line is a pain and buggy. It is also a good idea to map the network drive and use drive letter such as z:\
xcopy /K /R /E /I /S /C /H /G /X /Y s:\*.* t:\
/K Copies attributes. Normal Xcopy will reset read-only attributes.
/R Overwrites read-only files.
/E Copies directories and subdirectories, including empty ones.
/I If destination does not exist and copying more than one file, assumes that destination must be a directory.
/S Copies directories and subdirectories except empty ones.
/C Continues copying even if errors occur.
/H Copies hidden and system files also.
/Y Suppresses prompting to confirm you want to overwrite an existing destination file.
/G Allows the copying of encrypted files to destination that does not support encryption.
/X Copies file audit settings (implies /O).
The format for the second part of eseutil should be:
/d\\server\folder\filename
Notice there is no space after the /d