I'm working in a company that has far too many (millions) of record forms that all need organising.
Each file uses the following naming structure:
xxxxx-xx-xx-xxxxx e.g. 43144-02-40-21324.<ext>
I've used in the past a batch script that puts files into a folder of the same name, but I'm looking for something slightly different.
I'd eventually like to end up with the following folder structure:
C:\[root directory]\43144\02\40\21324.PDF
Is something like this possible?
My knowledge of batch scripts is non existant, does anyone have the ability to quickly throw this together?
Thanks in advance, David
#ECHO OFF
SETLOCAL
SET "sourcedir=."
SET "destdir=u:\temp"
FOR /f "tokens=1,2,3,*delims=-" %%a IN ('dir /b/a-d "%sourcedir%\*-*-*-*"') DO (
MD "%destdir%\%%a\%%b\%%c" 2>NUL
IF EXIST "%destdir%\%%a\%%b\%%c\%%d" (ECHO "%destdir%\%%a\%%b\%%c\%%d" already exists
) ELSE (
MOVE "%sourcedir%\%%a-%%b-%%c-%%d" "%destdir%\%%a\%%b\%%c\%%d" >nul
)
)
GOTO :EOF
This should set you on the right track - just need to set your source and destination directories...
Yes, this isn't very hard to do:
We need to loop over all the files
for %%F in (*.ext) ...
For simplicity reasons we look in the current folder, so set it appropriately beforehand. But you can also supply a folder to look in.
For every file found, just call a subroutine that does the work
... do call :process "%%~F"
Exit the main method
goto :eof
We need a subroutine now
rem :process <filename>
:process
Dissect the file name into its parts. Since those are all fixed-length we can just use substrings here.
set "Filename=%~1"
set "Part1=%Filename:~0,5%"
set "Part2=%Filename:~6,2%"
set "Part3=%Filename:~9,2%"
set "Rest=%Filename:~12%"
Now we should probably check whether the folder we need to copy this into already exists or not
if not exist C:\root\%Part1%\%Part2%\%Part3%\NUL mkdir C:\root\%Part1%\%Part2%\%Part3%
This will create every folder along the way if necessary. Very handy.
Rename and move the file now
move %1 C:\root\%Part1%\%Part2%\%Part3%\%Rest%
Exit the subroutine
goto :eof
This should be it, more or less (bugs in my implementation notwithstanding). So here again in full:
for %%F in (*.ext) do call :process "%%~F"
goto :eof
rem :process <filename>
:process
set "Filename=%~1"
set "Part1=%Filename:~0,5%"
set "Part2=%Filename:~6,2%"
set "Part3=%Filename:~9,2%"
set "Rest=%Filename:~12%"
if not exist C:\root\%Part1%\%Part2%\%Part3%\NUL mkdir C:\root\%Part1%\%Part2%\%Part3%
move %1 C:\root\%Part1%\%Part2%\%Part3%\%Rest%
goto :eof
You will have to iterate through files and for each file split name using FOR.
Then, you must ensure directories are created and copy/move your file to new path.
For %%f In (*.*) Do For /F "Tokens=1,2,3,4,5,6 Delims=-" %%i In ("%%f") Do Call :PutInPath %%f %%i %%j %%k %%k %%l
GoTo :EOF
:PutInPath
MD "C:\your dir\%2"
MD "C:\your dir\%2\%3"
MD "C:\your dir\%2\%3\%4"
REM You can use COPY, MOVE, REN, ...
COPY %1 "C:\your dir\%2\%3\%4\%5"
GoTo :EOF
#Echo off
SET extension=%1
setlocal enabledelayedexpansion
for %%f in ("*.%extension%") do (
SET substr=%%f
echo !substr!
copy !substr! !substr:~0,5!\!substr:~6,2!\!substr:~9,2!\!substr:~12,5!.%extension%
del !substr!
)
if the file format are fixed, we can do it like this!
Related
I'm writing batch script which I'll use to copy files from location A to location B with rename of a source file from location A if the same file exists already in location B.
Currently Im using snippet from another topic here on stack but it doesnt work on files from subfolders, could anyone help me with code below so it work on all files and subdirectiories from both locations? Many thanks!
#ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
SET sourcedir="D:\TEST\FROM"
SET destdir="D:\TEST\TO"
SET /a count=0
for %%c in (%sourcedir%\*.*) do (
CALL :select
ECHO copy "%%c" "%destdir%\%%~nc_!count!%%~xc" /s
)
GOTO :EOF
:select
SET /a count+=1
IF EXIST "%destdir%\%%c" GOTO select
GOTO :eof
Replace your for loop with the following for loop:
for /R "%sourcedir%" %%c in (*.*) do (what you like)
Also, why do you want the following piece of code?
copy "%%c" "%destdir%\%%~nc_!count!%%~xc" /s
Just copy "%%c" %destdir%
More generally you can write:
#ECHO OFF
SET sourcedir="D:\TEST\FROM"
SET destdir="D:\TEST\TO"
:: SET /a count=0
for /R "%sourcedir%" %%c in (*.*) do (
:: SET /a count+=1
IF NOT EXIST "%destdir%\%%c" (
echo copy "%%c" %destdir%
)
)
Hope you are fine with this, possible dublicate of Windows batch file with loop through subfolders
Sharing with what I managed to get so far, works for what I needed, however still not doing well for subfolders:
#ECHO OFF
SET "sourcedir= "
SET "destdir= "
SET "HH=%TIME:~0,2%"
SET "MM=%TIME:~3,2%"
SET "SS=%TIME:~6,2%"
SET "_Time=%HH%%MM%%SS%"
FOR /R "%sourcedir%" %%G IN (*.*) DO (
IF EXIST "%destdir%\%%~nG%%~xG" (
COPY /V /Z "%%G" "%destdir%\%%~nG_duplicate_%_Time%%%~xG"
) ELSE (
COPY /V /Z "%%G" "%destdir%\%%~nG%%~xG")
)
I need a big help from the community, please if somebody can give me some hints. I have the following windows batch script which is supposed to read more than 10 million records as different CSV files and merge them all together. I am running the script on the server. So it's not very slow. But the problem is that the code doesn't handle duplicated records. I am not sure how to change the script in order to handle the duplication records and only passed unique records. I would be very very appreciated for your help.
rem Set current working directory to Task folder
set FilePath=%~dp0
set FolderPath=%FilePath:~0,-1%
rem Set Space environment variables
call "%FolderPath%"\..\SpaceEnv.bat
rem Set Task specific environment variables
set TaskName=MergeCSVfiles
set fileName=result.csv
set LogFile=%TaskName%_%LogDateTime%.log
:begin
cd ..
cd "Source Files\DCM_Source\Inbox"
echo Staring merge %fileName% at: %time%
setlocal enabledelayedexpansion
set "first=1"
>%fileName% (
for %%F in (msource*.csv) do (
if not "%%F"=="%fileName%" (
set /p "header="<"%%F"
if defined first (
type "%%F"
set "first="
) else (
type "%%F" |find /V "!header!"
)
)
)
)
endlocal
echo Finish merging %fileName% at: %time%
******UPDATED******
Example of CSV file
Sites|Level 2 sites|Date-time (visit start)|Visit ID|Unique visitor ID|Date-time (event)|Sources|Visitor categories|Visitor ID|Visits
SE Romania|PRM|2018-01-01T00:30:04|1|-6427177464|2018-01-01T00:30:04|Portal sites|-|0|2
SE Romania|PRM|2018-01-01T00:30:04|1|-6427177464|2018-01-01T00:30:04|Portal sites|-|0|2
This code will dedupe a file. In order to do that it must be sorted. This means any header record at the top of the file will be sorted into the file. This is code I received from dbenham. I can't remember if he originally posted it on StackOverflow or DosTips.com. If the file is very large it will more than likely crash with an out of memory error.
#echo off
:: Call function to dedupe file
CALL :DEDUPE "filename.txt"
goto :eof
:DEDUPE
:: DEDUPE file
setlocal disableDelayedExpansion
set "file=%~1"
set "sorted=%file%.sorted"
set "deduped=%file%.deduped"
::Define a variable containing a linefeed character
set LF=^
::The 2 blank lines above are critical, do not remove
sort "%file%" >"%sorted%"
>"%deduped%" (
set "prev="
for /f usebackq^ eol^=^%LF%%LF%^ delims^= %%A in ("%sorted%") do (
set "ln=%%A"
setlocal enableDelayedExpansion
if /i "!ln!" neq "!prev!" (
endlocal
(echo %%A)
set "prev=%%A"
) else endlocal
)
)
>nul move /y "%deduped%" "%file%"
del "%sorted%"
GOTO :EOF
#ECHO OFF
SETLOCAL
SETLOCAL ENABLEDELAYEDEXPANSION
SET "sourcedir=U:\sourcedir"
SET "filenamecommon=q49264647*.csv"
:: switch to required source directory
PUSHD "%sourcedir%"
:: get header line
FOR %%f IN (%filenamecommon%) DO FOR /f "delims=" %%h IN (%%f) DO SET "header=%%h"&goto gotheader
:gotheader
COPY %filenamecommon% atempfilename
SET "lastline="
>resultfilename (
ECHO %header%
SETLOCAL enabledelayedexpansion
FOR /f "delims=" %%d IN ('sort atempfilename' ) DO (
IF "%%d" neq "!lastline!" IF "%%d" neq "%header%" ECHO %%d
SET "lastline=%%d"
)
endlocal
)
DEL atempfilename
popd
GOTO :EOF
You would need to change the setting of sourcedir to suit your circumstances.
I used file/dirctorynames that suit my system for testing.
Note : datafiles containing the characters ! or ^ or unbalanced " will not be processed correctly.
First, find the header line by setting header from any matching filename. Once header is set, forcibly abort the for loops.
copy and concatenate all of the required files to a tempfile.
output the header line, then sort the tempfile to group identical lines. Read the result and output only those lines that differed from the previous and were not header lines.
Applying /i to the if statements will make the entire routine disregard character-case.
Sort the tempfile
Ok. Give this code a try. I think this code would generate the result file with not duplicated records not matters its size. However, the time the program will take depends on several factors, although IMHO it should not be excessive because the core part of the process is based on findstr.exe command.
#echo off
setlocal
del result.csv 2>NUL
rem Process all input files
for /F "delims=" %%f in ('dir /B /O:-S msource*.csv') do (
echo Merging file: %%f
if not exist result.csv (
rem Initialize output file with first input file
copy "%%f" result.csv > NUL
) else (
rem Get records in this file that are not in result file
findstr /V /G:result.csv "%%f" > newRecords.csv
rem and add they to the result file
type newRecords.csv >> result.csv
)
)
del newRecords.csv
You may also try to eliminate the dash in /O:-S switch of dir command; perhaps this change will speed up the process a little...
I have a bunch of files say,
xxx111.txt
xxx112.txt
xxx113.txt
I want to remove the last 3 characters of all the file names and I'm using this script
#echo off
setlocal enabledelayedexpansion
set X=3
for %%f in (*) do if %%f neq %~nx0 (
set "filename=%%~nf"
set "filename=!filename:~,-%X%!"
ren "%%f" "!filename!%%~xf"
)
popd
pause
This runs perfectly when the output filenames are different. However, in the above case all file will be output as xxx.txt and the script throws me the error
"A duplicate file name exists, or the file cannot be found".
Is there any way to tweak this so that duplicate files will be renamed and maybe numbered 1,2,3...?
Unfortunately I cannot install any other software.
#echo off
setlocal EnableDelayedExpansion
set X=3
for /F "delims=" %%f in ('dir /A:-D /B') do if "%%f" neq "%~NX0" (
set "filename=%%~Nf"
set "filename=!filename:~,-%X%!"
if exist "!filename!%%~Xf" call :getNewName "%%~Xf"
ren "%%f" "!filename!%%~Xf"
)
popd
pause
goto :EOF
:getNewName ext
set i=0
:nextNum
set /A i+=1
if exist "%filename%%i%%~1" goto nextNum
set "filename=%filename%%i%"
exit /B
You should not use plain for %%f command when renaming files. Depending on where the new names are placed in the list of original names, they may be processed a second time by the for %%f. Always use for /F for renaming.
I have a hard drive with files and folders ordered in this similar manner:
F:\folder1\folder\folder\file.rar
F:\folder1\folder\folder\file1.rar
F:\folder1\folder\folder\file2.rar
F:\folder2\folder\file.rar
F:\folder2\folder\file1.rar
F:\folder3\folder\file.rar
F:\folder3\folder\folder\folder\file.rar
I'd like to move all files in this drive to F:\\*\ , Rename if a duplicate filename is found, and recursively delete empty folders afterwards. There's just too many of these folders to find out how deep each parent directory is. After executing the batch script the folders should look like:
F:\folder1\file.rar
F:\folder1\file1.rar
F:\folder1\file2.rar
F:\folder2\file.rar
F:\folder2\file1.rar
F:\folder3\file.rar
F:\folder3\file (1).rar
There might be folders with files already inside the F:\\*\ level. I want them to stay where they are.
try this:
#ECHO OFF &SETLOCAL
FOR /r "F:\" %%a IN (*.rar) DO (
SET "fname=%%~nxa"
SET "fpath=%%~fa"
FOR /f "tokens=1,2 delims=\" %%b IN ("%%~fa") DO SET "targetfolder=%%~b\%%~c"
SETLOCAL ENABLEDELAYEDEXPANSION
CALL :moveit "!fpath!" "!targetfolder!" "!fname!"
ENDLOCAL
)
GOTO :eof
:moveit
SETLOCAL
SET "nname=%~3"
:loop
SET /a fcount+=1
IF EXIST "%~2\%nname%" (
SET "nname=%~n3 (%fcount%)%~x3"
GOTO :loop
)
ECHO MOVE "%~1" "%~2\%nname%"
MOVE "%~1" "%~2\%nname%"
ENDLOCAL
EXIT /b
Is it posible to copy and make directories automatically from file name substrings using Robocopy?
I mean i have files like these. LAJ00306130201004626.rc the first 8 chararacters are control number (LAJ00306=control number) this would be the name of the folder and the rest are the date and time (Date=130201) (time=004626).
LAJ00306130201004626.rc
LAJ00306130202004626.rc
LAJ00306130203004626.rc
LAJ00307130201004626.rc
LAJ00307130202004626.rc
and i would like to copy and create folders from the file name like under and copy the files mentioned before in the new folders.
LAJ00306
LAJ00307
I hope to be clear if necessary ask me for more information
try this, look at the output and remove the echos before MD and ROBOCOPY, if it looks good:
#ECHO OFF &SETLOCAL ENABLEDELAYEDEXPANSION
SET "sourcefolder=."
SET "targetfolder=X:\data"
CD /d "%sourcefolder%"
FOR %%a IN (*.rc) DO (
SET "fname=%%a"
SET "folder=!fname:~0,8!"
SET "$!folder!=1"
)
FOR /f "delims=$=" %%a IN ('set "$"') DO (
ECHO MD "%targetfolder%\%%a" 2>nul
ECHO ROBOCOPY "%sourcefolder%" "%targetfolder%\%%a" "%%a*.rc"
)
Set sourcefolder and targetfolder for your folder tree.
Try this:
#echo off
pushd "c:\source folder"
setlocal enabledelayedexpansion
for %%a in (*.rc) do (
set "name=%%a"
robocopy "%cd%" "%%a" "D:\target directory\!name:~0,8!"
)
popd
Answers to your questions are:
pushd "drive:\path" makes the location the current working directory.
popd restores the last working directory
setlocal enabledelayedexpansion allows you to change and use variables within a loop, using the !variable! syntax.
If your 2000 files are in a single folder then it should work - but test it on some sample files first so that you can see how it will work.
#ECHO OFF
SETLOCAL
SET "sourcedir=."
SET "destdir=c:\destdir"
FOR /f "tokens=1*delims=_" %%i IN (
'dir /b /a-d "%sourcedir%\*_*."'
) DO XCOPY /b "%sourcedir%\%%i_%%j" "%destdir%\%%i\"
GOTO :EOF
This should accomplish the task described. You'd need to set up the source and destination directories to suit, of course. Add >nul to the end of the XCOPY line to suppress 'copied' messages.