I'm running a process that produces n number of outputs at different timestamps for a given input CSV. The output files are in CSV form and labelled as such:
"Output" _ RouteName _ Direction _ YYMMDD _ HHMMSS
I have a macro that reports on missing data in the files, I just need a list of the number of rows in each CSV.
I have been doing this by using the command:
#Echo Off
:_Loop
If "%~1"=="" Pause&Goto EOF
Find /C /V "Wont#findthisin#anyfile" %1 >> LineCount.txt
Shift
Goto _Loop
The command is called counter.cmd and I just drag the output CSV's into it and it creates this output in a text file for each output:
---------- R:\10_TECHNICAL\10_TESTRUN\RUN\AM\ITN\A6_1N\OUTPUT_A6_1N_180313_070112.CSV: 5
The problem is that, I can only use this command to process a maximum of ~ 100 files, and I have ~ 1000 output files. When I try to make it do all 1000 files I get this error:
I have relatively basic windows command scripting skills and so don't know how to overcome this problem. Any help would be appreciated!
Before I get to the source of your problem, and the fix, I want to point out a couple things that could improve your current script.
1) Your FIND command can be simplified to find /n /v "" filePath - it seems nonsensical, but it works.
2) GOTO is relatively slow. You can get rid of the GOTO loop by using the FOR command to process all of the arguments. You can use %* to retrieve all of the arguments.
3) Every time you redirect the file must be opened and the file pointer positioned to the end of file. This takes time. It is much faster to redirect only once.
Incorporating all of the above, your script could be as simple as
#echo off
> LineCount.txt (for %%F in (%*) do find /n /v "" "%%F")
pause
When you drag files onto a batch script, it creates a single command line containing the path to each of the files. A Windows command line is limited to 8191 bytes long. So there is no way your strategy will work if you have ~1000 files.
I'm assuming all of your files are within a single folder, in which case you could change your script to process a single folder path instead of a list of file paths.
#echo off
>LineCount.txt (for %%F in ("%~1\*.csv") do find /n /v "" "%%F")
pause
If the files are spread across a few folders, then you can add an extra loop to iterate each of the folders
#echo off
>LineCount.txt ( for %%A in (%*) do for %%F in ("%%~A\*.csv") do find /n /v "" %%F")
pause
I have to assume the restriction is with your %1 variable and the maximum allowed command line length.
This can probably be easily remedied with a single command line
#Find /C /V "" R:\10_Technical\10_TestRun\Run\AM\ITN\A6_1N\*.csv > LineCount.txt
Edit
You can drag and drop the folder containing your csv files onto the batch file too.
If you want the outputfile in the same directory as the csv's then use:
#Find /C /V "" "%~1\*.csv" > "%~1\LineCount.txt"
Or in the same directory as the batch file:
#Find /C /V "" "%~1\*.csv" > "%~dp0LineCount.txt"
You could even have it output to the directory holding that folder:
#Find /C /V "" "%~1\*.csv" > "LineCount.txt"
Related
I am trying to writing script by using shell scripting in pentaho.
First,file name "test_viewpayment4.txt".
I have to count the line, the correct output is 100(which not include the header), and I have to get the file name also. After that,creates the txt file which include count line and file name as picture below
https://i.stack.imgur.com/afrGg.png
Here is my code in Pentaho
https://i.stack.imgur.com/mYuWw.png
CMD.EXE /C
call cd /d d:\test
call set file=test_viewpayment4.txt
call Type d:\test\test_viewpayment4.txt | more +2 | find /V /C "~~~" > d:\test\Result5.txt
call echo %file% >> d:\test\Result5.txt
Here is the result in Result5.txt
100
test_viewpayment4.txt
However,the data is not in the same line.
Any help or pointing the right direction will be really appreciate it.
use a for loop to get a command's output to a variable:
for /f %%a in ('more +2 "d:\test\test_viewpayment4.txt" ^|find /v /c ""') do set lines=%%a
then just echo all in one command:
>> d:\test\Result5.txt echo %file% %lines%
(Note: none of the call commands is necessary in batch; also cmd /c is useless. Dismiss them, except Pentaho (which I don't know) needs them)
I have written the following .bat file, and it runs perfectly on my Windows 2000 machine, but will not run on my Windows 7 or Windows XP machines. Basically it just loops through the current directory and runs a checksum program which returns the checksum. The output of the program is saved to a text file and then formatted to remove the checksum of the output file.
#Echo Off
for /r %%f in (*.txt) do crc32sum.exe %%f >> all_checksums.txt
ren all_checksums.txt old.txt
findstr /v /e /c:"all_checksums.txt" old.txt > all_checksums.txt
del old.txt
When I run this file on my Win2k PC with a bunch of text files and the crc32sum.exe in a folder, it outputs the file. On other machines it outputs a blank file. I turned Echo on and kept only the for loop line and found that the output from executing the crc32sum.exe is nothing. If you manually run the crc32sum.exe file it outputs the checksum no problem.
Any ideas as to how to fix this?
EDIT: Here is a link to the software: http://www.di-mgt.com.au/src/digsum-1.0.1.zip
EDIT2: New development, it seems that the file works if the path of the folder has no spaces in it i.e. C:\temp or C:\inetpub\ftproot or C:\users\admin\Desktop\temp. Does anyone know how I can make this work with paths that have spaces? %%~f doesnt work it says unexpected.
Try this modified batch code which worked on Windows XP SP3 x86:
#echo off
goto CheckOutput
rem Command DEL does not terminate with an exit code greater 0
rem if the deletion of a file failed. Therefore the output to
rem stderr must be evaluated to find out if deletion was
rem successful or (for a single file) the file existence is
rem checked once again. For details read on Stack Overflow
rem the answer http://stackoverflow.com/a/33403497/3074564
rem The deletion of the file was successful if file created
rem from output message has size 0 and therefore the temp
rem file can be deleted and calculation of the CRC32 sums
rem can be started.
:DeleteOutput
del /F "all_checksums.txt" >nul 2>"%TEMP%\DelErrorMessage.tmp"
for %%E in ("%TEMP%\DelErrorMessage.tmp") do set "FileSize=%%~zE"
if "%FileSize%" == "0" (
set "FileSize="
del "%TEMP%\DelErrorMessage.tmp"
goto CalcCRC32
)
set "FileSize="
echo %~nx0: Failed to delete file %CD%\all_checksums.txt
echo.
type "%TEMP%\DelErrorMessage.tmp"
del "%TEMP%\DelErrorMessage.tmp"
echo.
echo Is this file opened in an application?
echo.
set "Retry=N"
set /P "Retry=Retry (N/Y)? "
if /I "%Retry%" == "Y" (
set "Retry="
cls
goto CheckOutput
)
set "Retry="
goto :EOF
:CheckOutput
if exist "all_checksums.txt" goto DeleteOutput
:CalcCRC32
for /R %%F in (*.txt) do (
if /I not "%%F" == "%CD%\all_checksums.txt" (
crc32sum.exe "%%F" >>"all_checksums.txt"
)
)
The output file in current directory is deleted if already existing from a previous run. Extra code is added to verify if deletion was successful and informing the user about a failed deletion with giving the user the possibility to retry after closing the file in an application if that is the reason why deletion failed.
The FOR command searches because of option /R recursive in current directory and all its subdirectories for files with extension txt. The name of each found file with full path always without double quotes is hold in loop variable F for any text file found in current directory or any subdirectory.
The CRC32 sum is calculated by 32-bit console application crc32sum in current directory for all text files found with the exception of the output file all_checksums.txt in current directory. The output of this small application is redirected into file all_checksums.txt with appending the single output line to this file.
It is necessary to enclose the file name with path in double quotes because even with no *.txt file containing a space character or one of the special characters &()[]{}^=;!'+,`~ in its name, the path of the file could contain a space or one of those characters.
For the files
C:\Temp\test 1.txt
C:\Temp\test 2.txt
C:\Temp\test_3.txt
C:\Temp\TEST\123-9.txt
C:\Temp\TEST\abc.txt
C:\Temp\TEST\hello.txt
C:\Temp\TEST\hellon.txt
C:\Temp\Test x\test4.txt
C:\Temp\Test x\test5.txt
the file C:\Temp\all_checksums.txt contains after batch execution:
f44271ac *test 1.txt
624cbdf9 *test 2.txt
7ce469eb *test_3.txt
cbf43926 *123-9.txt
352441c2 *abc.txt
0d4a1185 *hello.txt
38e6c41a *hellon.txt
1b4289fa *test4.txt
f44271ac *test5.txt
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
cls /?
del /?
echo /?
for /?
goto /?
if /?
rem /?
set /?
type /?
One of the help pages output on running for /? informs about %~I, %~fI, %~dI, %~pI, %~nI, %~xI, %~sI, %~aI, %~tI, %~zI.
Using in a batch file f (in lower case) as loop variable and referencing it with %%~f is a syntax error as command processor expects next the loop variable. %%~ff would be right, but could be different to %%~fI (name of a file/folder with full path and extension without quotes) in comparison to %%~I (string without surrounding quotes).
It is not advisable to use (those) small letters as loop variable. It is better to use upper case letters or character # as loop variable. The loop variable and also those modifiers are case sensitive while nearly everything else in a batch file is case insensitive.
I have a batch file to count the number of specific files in a folder. The contents are given below:
set xx = %DATE:~10,4%%DATE:~4,2%%DATE:~7,2%
set count=dir C:\Archive\*%xx%.csv | find "File(s)"
echo %count%
But the output of the last command displays as
echo
ECHO is on
What am I doing wrong here?? Can anyone help please?
To execute a command an retrieve its output you need the for /f command (see for /? help)
set "xx=%DATE:~10,4%%DATE:~4,2%%DATE:~7,2%"
for /f %%a in ('dir /a-d /b "c:\Archive\*%xx%.csv" ^| find /c /v ""') do set "count=%%a"
echo %count%
This will execute a dir command for the required files without folders included in the list, in bare format (no header, summary and no aditional file info) and instead of search for the File(s) (in a different windows locale the text is different), it uses find to count (/c) the number of non empty lines (/v ""). The output is a number that is stored in the for replaceable parameter %%a, and then copied to the required variable
I am attempting to create a script that will search through a directory for files whose extension are larger than a specific numerical value. The process I am monitoring creates files with numeric extensions when it runs. My end goal is to delete all files of a certain file name one the extension exceeds 800.
The process runs and creates a file as a result, mainly a log files. When it creates its 800th file the file name would be like "filename.800". The file name can vary but the extension will be anything from 000 - 999. I need to delete "filename.*" for the collection of files that reaches 800 (or other). There are other files of different names but have numeric extensions as well in the same directory. I'm trying to make this part of an automated cleanup process.
I could use some suggestions on how to get started detecting the files exist. My initial attempt below does not seem to heed the 'GTR', 'LSS' options.
forfiles /M *.* /C "%comspec% /c IF #EXT GTR 800 (ECHO #PATH)"
I don't mind entertaining other methods to address my task. Thanks in advance.
The #ext value includes enclosing quotes, so you need to include them in your IF comparison. But adding quotes will throw off the command line parser since the entire command is already enclosed in quotes. Use 0x22 to represent each internal quote.
forfiles /M * /C "cmd /c if #ext gtr 0x228000x22 echo #path"
However, FORFILES is very slow. It is much faster to use a simple FOR loop. The ~x modifier expands to the file extension (including the dot).
The following works on the command line.
for %f in (*) do #if %~xf gtr .800 echo %f
Double the percents if you want to use the command in a batch script.
If you want to delete the files precisely in the 800-999 range, then the easiest and fastest way is this:
for %%a in (8 9) do del filename.%%a??
Previous method could even be adjusted for other different ranges.
#ECHO OFF
SETLOCAL
::method1
FOR %%f IN (*.8* *.9*) DO IF %%~xf gtr .800 IF %%~xf leq .999 ECHO %%f
:: method2
ECHO ======= method 2 ============
FOR %%f IN (*.8* *.9*) DO (
ECHO %%~xf|FINDSTR /r ".[89][0-9][0-9]$" >NUL
IF NOT ERRORLEVEL 1 ECHO %%f
)
The first method may suit, but it also detects filenames such as xxx.8004 xxx.8a9.
The second method is slower, but I believe will reliably detect only filenames that end ".[8 or 9][2 digits]"
The filenames are only ECHOed to the console. Replace the ECHO with DEL to delete the files - after rigorous testing...
So I need a Windows Script that I can tell it a directory to go through and it will parse all sub-directories and while in each subdir, will archive all files with a certain file extension and keep it in the same subdir, then move onto the next one.
What's the best way to go about this? Perl Automation Scripting, AutoIt?
Any sample code you guys can give me?
Perl is more powerful than batch scripts but since Perl is not included with windows it seems overkill for tasks such as this one. This should for example work:
FOR /R C:\hello\ %%G IN (*.txt) DO "c:\Program Files\7-Zip\7z.exe" a %%G.zip %%G && del %%G
Note that you cannot do this directly in the prompt, you must save it as a .bat file. It is of course also possible to allow the user to specify the paths and extensions with command line like this:
FOR /R %1 %%G IN (%2) DO "c:\Program Files\7-Zip\7z.exe" a %%G.zip %%G && del %%G
More information about FOR and other windows command line commands can be found here: http://ss64.com/nt/
This would then be run with:
test.bat C:\Hello\ *.txt
EDIT: This obviously requires you to have 7-Zip installed but it's pretty obvious where to change the code if you want to use some other zipper. Also keep in mind to always be Extremely careful when experimenting with scripts such as this. One small mistake could have it delete a lot of files, so you should always test it on a copy of the file system until you are absolutely sure it works.
FORFILES is included with Windows and may be more applicable than FOR to what you're trying to do:
FORFILES [/P pathname] [/M searchmask]
[/S]
[/C command] [/D [+ | -] {MM/dd/yyyy | dd}]
Description:
Selects a file (or set of files) and executes a
command on that file. This is helpful for batch jobs.
Parameter List:
/P pathname Indicates the path to start searching.
The default folder is the current working
directory (.).
/M searchmask Searches files according to a searchmask.
The default searchmask is '*' .
/S Instructs forfiles to recurse into
subdirectories. Like "DIR /S".
/C command Indicates the command to execute for each file.
Command strings should be wrapped in double
quotes.
The default command is "cmd /c echo #file".
The following variables can be used in the
command string:
#file - returns the name of the file.
#fname - returns the file name without
extension.
#ext - returns only the extension of the
file.
#path - returns the full path of the file.
#relpath - returns the relative path of the
file.
#isdir - returns "TRUE" if a file type is
a directory, and "FALSE" for files.
#fsize - returns the size of the file in
bytes.
#fdate - returns the last modified date of the
file.
#ftime - returns the last modified time of the
file.
To include special characters in the command
line, use the hexadecimal code for the character
in 0xHH format (ex. 0x09 for tab). Internal
CMD.exe commands should be preceded with
"cmd /c".
/D date Selects files with a last modified date greater
than or equal to (+), or less than or equal to
(-), the specified date using the
"MM/dd/yyyy" format; or selects files with a
last modified date greater than or equal to (+)
the current date plus "dd" days, or less than or
equal to (-) the current date minus "dd" days. A
valid "dd" number of days can be any number in
the range of 0 - 32768.
"+" is taken as default sign if not specified.
Below one way I would do it in AutoIt since you asked. Replace the MsgBox line with whatever code you need to do whatever it is your wanting to do. AutoIt is fun stuff!
#include <File.au3>
archiveDir(InputBox("Path","Enter your start path."))
Func archiveDir($rootDirectory)
$aFiles = _FileListToArray($rootDirectory)
For $i = 1 To UBound($aFiles) - 1
If StringInStr(FileGetAttrib($aFiles[$i]),"D") Then archiveDir($rootDirectory & $aFiles[$i] & "\")
MsgBox(0,"This would be your archive step!",'"Archiving" ' & $rootDirectory & $aFiles[$i])
Next
EndFunc
One solution could be:
my $dirCnt = 0;
traverse_directory('C:\Test');
sub traverse_directory{
my $directory = shift(#_);
$dirCnt++;
my $dirHandle = "DIR".$dirCnt;
opendir($dirHandle, $directory);
while (defined(my $file = readdir($dirHandle))){
next if $file =~ /^\.\.?$/; # skip . and .. ...
if (-d "$directory\\$file"){ traverse_directory("$directory\\$file"); }
if ($file =~ /\.txt/){ #find txt files, for example
print "$file\n"; #do something with the text file here
}
}
closedir($dirHandle);
}