Kindly excuse me if it seems a bit amateur. I'm not so good with windows batch files.
I am using Instaloader to download single posts. Instaloader can take an input with a content ID. I am trying to write a batch file so that it can take content IDs as an input from a csv file and download it in a loop.
Here is my sample which works but I have to input an ID every time. I would like it to take the ID from a csv file until it finishes all the rows.
#echo off
:prompt
set /p post= "Enter ID: "
instaloader --no-metadata-json --no-compress-json --no-captions --dirname-pattern={profile} --filename-pattern {profile}_{date_utc} -- -%post%
echo.
goto prompt
I should use FOR I believe but I just couldn't get it work. Thanks for any suggestions.
I tried a few samples of using FOR but it just fails. I am not sure if there is a better approach to this. I don't mind using powershell if it can be done in a more efficient way like using Import-CSV first.
The PS attempt:
foreach ($post in Get-Content '.\IG Posts.txt')
{
do { instaloader --no-metadata-json --no-compress-json --no-captions --dirname-pattern={profile} --filename-pattern {profile}_{date_utc} -- -$post } while (eol)
}
Sample txt/csv (they are actual content IDs that instaloader would download:
Cm4AoMVy60x
Cm0UMTAyLJe
ClViwoRStoa
[Theoretical and untried]
#echo off
setlocal
if "%~1"=="" echo Syntax %0 "filename containing list"&goto :eof
for /f "usebackq delims=" %%b in ("%~1") do instaloader --no-metadata-json --no-compress-json --no-captions --dirname-pattern={profile} --filename-pattern {profile}_{date_utc} -- -%%b
goto :eof
where this batch should be invoked as thisbatch "filename"
%1 is the first parameter on the command line. %~1 is the same, minus any enclosing quotes. "%~1" is the (first paramater - enclosing quotes) enclosed in quotes.
In the for /f line, the usebackq means "the thing in quotes is a filename" and delims= means "there are no delimiters on the lines of the file".
So each line of the files is assigned to %%b in turn and instaloader is run using the contents of the line.
Note that if instaloader is a batch file, you should use call instaloader...
I'm assuming a textfile contains the required data, one to a line. If you want otherwise, please post a sample.
Related
I'm trying to parse a file that contains data, it has some numbers which I want to parse, store and use them as variables in the future. The problem is, these numbers are updated often so that's why I'm parsing them whenever I run my script.
The problem, is that when I collect the data as a variable in a for loop, it's not getting passed to my text file properly.
#echo off
setlocal EnableDelayedExpansion
for /f "skip=104 eol=/ tokens=4 delims=," %%A in (file location) do (
echo %%A
echo %%A > UICNumbers.txt
)
This is the output of the echo %%A command:
UIC00000701991
UIC00000710996
UIC00001701890
UIC00002701890
UIC00001701898
UIC00002701898
When I open the UICNumbers.txt file all I see listed is:
UIC00002701898
which is the last entry in the for loop variable. I tried changing the output to >> but this means every time I run the command it repeats every entry.
I want to create a batch file able to apply some processing on each JPG file in a folder hierarchy. The following script file works very well for that case (here I only echo the name of each file, but this should be replaced by some more complex statements in the real application):
:VERSION 1
#echo off
set "basefolder=C:\Base"
for /r %basefolder% %%f in (*.jpg) do echo %%f
Actually, I don't want to explore all the folder hierarchy under %basefolder%, but only a given list of subfolders. This modified script is able to deal with that case :
:VERSION 2
#echo off
set "basefolder=C:\Base"
set "subfolders=A B C"
for %%s in (%subfolders%) do (
pushd %basefolder%\%%~s"
for /r %%f in (*.jpg) do echo %%f
popd
)
Is there a solution to remove the pushd/popd pair of statements, to get something closer to the initial script. I thought that one of the following scripts would do the job:
:VERSION 3
#echo off
set "basefolder=C:\Base"
set "subfolders=A B C"
for %%s in (%subfolders%) do (
for /r %basefolder%\%%~s" %%f in (*.jpg) do echo %%f
)
or, using delayed expansion:
:VERSION 4
#echo off
setlocal enabledelayedexpansion
set "basefolder=C:\Base"
set "subfolders=A B C"
for %%s in (%subfolders%) do (
set "folder=%basefolder%\%%~s"
echo !folder!
for /r !folder! %%f in (*.jpg) do echo %%f
)
but none of them is working. When running the second one, the echo !folder! command in the external loop shows C:\Base\A, C:\Base\B and C:\Base\C as expected, but the inner loop doesn't echo any JPG file, so I guess that the recursive for /r command does not run correctly.
What am I doing wrong ?
Final edit after answers :
Thanks to #aschipfl who provided a link to the answer posted by #jeb on another question, quoted below:
The options of FOR, IF and REM are only parsed up to the special character phase. Or better the commands are detected in the special character phase and a different parser is activated then. Therefore it's neither possible to use delayed expansion nor FOR meta-variables in these options.
In other words, my versions 3 and 4 do not work because when defining the root folder of the FOR /R command, neither the %%~s nor the !folder! are correctly expanded by the expression parser. There is no way to change that, as this is a parser limitation. As I said in a comment below: the root folder option in the FOR /R command is basically only syntactic sugar to avoid the use of pushd/popd before and after the command. As this syntactic sugar is incomplete, we have to stick to the original syntax for some specific use cases, as the one presented here. The alternatives proposed by #Gerhard (using a subroutine CALL) or by #Mofi (parsing the result of a DIR command) are working, but they are neither more readable nor more efficient than the simple pushd/popd version I proposed initially.
My Approach for this would be really straight forward:
#echo off
set "basedir=C:\Base"
set "subfolders="A","B","C""
for %%i in (%subfolders%) do for /R "%basedir%" %%a in ("%%~i\*.jpg") do echo %%~fa
The double quotes inside of the subfolders variable is important here, it will ensure that folder names with whitespace are not seen as separators for the folder names. For instance:
set "subfolders="Folder A","Folder B","Folder C""
Edit
#echo off
set "basedir=C:\Base"
set "subfolders="A","B","C""
for %%i in (%subfolders%) do call :work "%%~i"
goto :eof
:work
for /R "%basedir%\%~1" %%a in (*.jpg) do echo %%~fa
It is in general not advisable to assign the value of a loop variable to an environment variable and next use the environment variable unmodified without or with concatenation with other strings being coded in batch file or defined already above the FOR loop within body of a FOR loop. That causes just problems as it requires the usage of delayed expansion which results in files and folders with one or more ! are not correct processed anymore inside body of the FOR loop caused by double parsing of the command line before execution, or command call is used on some command lines, or a subroutine is used called with call which makes the processing of the batch file much slower.
I recommend to use this batch file for the task:
#echo off
setlocal EnableExtensions DisableDelayedExpansion
set "basefolder=C:\Base"
set "subfolders=A B C "Subfolder D" SubfolderE"
for %%I in (%subfolders%) do for /F "delims=" %%J in ('dir "%basefolder%\%%~I\*.jpg" /A-D /B /S 2^>nul') do echo %%J
endlocal
The inner FOR loop starts for each subfolder defined in subfolders in background one more command process with %ComSpec% /c and the DIR command line appended as additional arguments. So executed is with Windows installed to C:\Windows for example for the first subfolder:
C:\Windows\System32\cmd.exe /c dir "C:\Base\A\*.jpg" /A-D /B /S 2>nul
The command DIR searches
in specified directory C:\Base\A and all it subdirectories because of option /S
for files because of option /A-D (attribute not directory) including those with hidden attribute set
matching the pattern *.jpg in long or short file name
and outputs to handle STDOUT of background command process just the matching file names because of option /B (bare format)
with full path because of option /S.
The error message output by DIR on nothing found matching these criteria is redirecting from handle STDERR to device NUL to suppress it.
Read the Microsoft documentation about Using command redirection operators for an explanation of 2>nul. The redirection operator > must be escaped with caret character ^ on FOR command line to be interpreted as literal character when Windows command interpreter processes this command line before executing command FOR which executes the embedded dir command line with using a separate command process started in background.
The output to handle STDOUT of background command process is captured by FOR respectively the command process which is processing the batch file. FOR processes the captured output line by line after started cmd.exe terminated itself. This is very often very important. The list of files to process is already in memory of command process before processing the first file name. This is not the case on using for /R as this results in accessing file system, getting first file name of a non-hidden file matching the wildcard pattern, run all commands in body of FOR and accessing the file system once again to get next file name. The for /R approach is problematic if the commands in body of FOR change a file to process like deleting, moving, modifying, copying it in same folder, or renaming a found file because of the entries in file system changes while for /R is iterating over these entries. That can easily result in some files are skipped or some files are processed more than once and it could result also an endless running loop, especially on FAT file system like FAT32 or exFAT. It is never good to iterate over a list of files on which the list changes on each iteration.
Command FOR on usage of /F ignores empty lines which do not occur here. A non-empty line is split up into substrings using a normal space and a horizontal tab as string delimiters by default. This line splitting behavior is not wanted here as there could be full qualified file names containing anywhere inside full name one or more spaces. For that reason delims= is used to define an empty list of delimiters which disables the line splitting behavior.
FOR with option /F would also ignore lines on which first substring starts with ; which is the default end of line character. This is no problem here because of command DIR was used with option /S and so each file name is output with full path which makes it impossible that any file name starts with ;. So the default eol=; can be kept.
FOR with option /F assigns by default just first substring to specified loop variable as tokens=1 is the default. This default can be kept here as splitting the lines (full file names) into substrings is disabled already with delims= and so there is always the full file name assigned to the loop variable.
This example uses just echo %%I to output the file names with full path. But it is now safe to replace this single command by a command block which does more with the JPEG files because of the list of JPEG files for each specified subfolder tree in base folder is always already completely in memory of command process processing the batch file.
I have a batch file which copies some local files up to a google storage area using the gsutil tool. The gsutil tool produces a nice log file showing the details of the files that were uploaded and if it was OK or not.
Source,Destination,Start,End,Md5,UploadId,Source Size,Bytes Transferred,Result,Description
file://C:\TEMP\file_1.xlsx,gs://app1/backups/file_1.xlsx,2018-12-04T15:25:48.428000Z,2018-12-04T15:25:48.804000Z,CPHHZfdlt6AePAPz6JO2KQ==,,18753,18753,OK,
file://C:\TEMP\file_2.xlsx,gs://app1/backups/file_2.xlsx,2018-12-04T15:25:48.428000Z,2018-12-04T15:25:48.813000Z,aTKCOQSPVwDycM9+NGO28Q==,,18753,18753,OK,
What I would like to do is to
check the status result in column 8 (OK or FAIL)
If the status is OK then move the source file to another folder (so that it is not uploaded again).
The problem is that the source filename is appended with "file://" which I can't seem to remove, example
file://C:\TEMP\file_1.xlsx
needs to be changed into this
C:\TEMP\file_1.xlsx
I am using a for /f loop and I am not sure if the manipulation of the variables %%A is different within a for /f loop.
#echo off
rem copy the gsutil log file into a temp file and remove the header row using the 'more' command.
more +1 raw_results.log > .\upload_results.log
rem get the source file name (column 1) and the upload result (OK) from column 8
for /f "tokens=1,8 delims=," %%A in (.\upload_results.log) do (
echo The source file is %%A , the upload status was %%B
set line=%%A
set line=!line:file://:=! >> output2.txt echo !line!
echo !line!
)
The output is like this.
The source file is file://C:\TEMP\file_1.xlsx , the upload status was OK
The source file is file://C:\TEMP\file_2.xlsx , the upload status was OK
I'm expecting it to dump the altered values out into a new file but it is not producing anything at the moment.
Normally I would extract from a specific character to the end of the string with something like this but it doesn't work with my For/f loop.
%var:~7%
Any pointers or a different way of doing it greatly appreciated.
Since the part to remove seems fixed it is easier to use substrings.
Also using for /f "skip=1" evades he neccessity of the external command more +1 and another intermediate file.
#echo off & setlocal EnableDelayedExpansion
type NUL>output2.txt
for /f "skip=1 eol=| tokens=1,8 delims=," %%A in (.\upload_results.log) do (
echo The source file is %%A , the upload status was %%B
set "line=%%A"
set "line=!line:~7!"
echo(!line!>>output2.txt
echo(!line!
)
File names and paths can contain also one or more exclamation marks. The line set line=%%A is parsed by Windows command processor a second time before execution with enabled delayed expansion. See How does the Windows Command Interpreter (CMD.EXE) parse scripts? Every ! inside the string assigned to loop variable A is on this line interpreted as begin or end of a delayed expanded environment variable reference. So the string of loop variable A is assigned to environment variable line with an unwanted modification if file path/name contains one or more exclamation marks.
For that reason it is best to avoid usage of delayed expansion. The fastest solution is for this task using a second FOR to get file:// removed from string assigned to loop variable A.
#echo off
del output2.txt 2>nul
for /F "skip=1 tokens=1,8 delims=," %%A in (upload_results.log) do (
echo The source file is %%A , the upload status was %%B.
for /F "tokens=1* delims=/" %%C in ("%%~A") do echo %%D>>output2.txt
)
Even faster would be without the first echo command line inside the loop:
#echo off
(for /F "skip=1 delims=," %%A in (upload_results.log) do (
for /F "tokens=1* delims=/" %%B in ("%%~A") do echo %%C
))>output2.txt
The second solution can be written also as single command line:
#(for /F "skip=1 delims=," %%A in (upload_results.log) do #for /F "tokens=1* delims=/" %%B in ("%%~A") do #echo %%C)>output2.txt
All solutions do following:
The outer FOR processes ANSI (fixed one byte per character) or UTF-8 (one to four bytes per character) encoded text file upload_results.log line by line with skipping the first line and ignoring always empty lines and lines starting with a semicolon which do not occur here.
The line is split up on every occurrence of one or more commas into substrings (tokens) with assigning first comma delimited string to specified loop variable A. The first solution additionally assigns eighth comma delimited string to next loop variable B according to ASCII table.
The inner FOR processes the string assigned to loop variable A with using / as string delimiter to get assigned to specified loop variable file: and to next loop variable according to ASCII table the rest of the string after first sequence of forward slashes which is the full qualified file name.
The full qualified file name is output with command echo and appended either directly to file output2.txt (first solution) or first to a memory buffer which is finally at once written into file output2.txt overwriting a perhaps already existing file with that file name in current directory.
For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.
del /?
echo /?
for /?
See also the Microsoft article about Using command redirection operators for an explanation of the redirections >, >> and 2>nul
Assuming I have a bunch of sqlcmd commands in .cmd files, order alphabetically e.g.:
01.setup.cmd
02.version1.cmd
03.version2.cmd
04.version3.cmd
how could one sequentially execute these in correct order with another .cmd file?
On windows:
for /F "tokens=*" %a in ('dir /b *.cmd') do call "%a"
This just loops over the result of dir /b *.cmd calling each in turn.
explanation from the docs:
FOR /F processing of a text file consists of reading the file, one
line of text at a time and then breaking the line up into individual
items of data called 'tokens'. The DO command is then executed with
the parameter(s) set to the token(s) found.
So my command says:
"tokens=*" don't give me individual tokens, give me the whole line as one hit
%a - name the line variable %a (note: it'll needs to be escaped as %%a if you're putting it in a batch file
('dir /b *.cmd') This is the input that it'll loop over. A bare directory listing for all .cmd files
then what I want it to do. Call the command %a.
If I didn't add the tokens bit it would work fine until you find a space in the file names.
I am currently trying to delete a variable from a batch file which is in a text file
e.g. delete %variable%
where %variable% = "test"
So the batch script would delete the instance of "test" in the specified text file
Can this be done?
#echo off
setlocal enabledelayedexpansion
set variable=test
for /f "delims=" %%l in (foo.txt) do (
set "line=%%f"
echo !line:%variable%=!
)
To avoid potential special character issues with the string you are searching for, it may be better to just call findstr directly like this:
type input.txt | findstr /b /e /v "remove_line" > input_without_remove_line.txt
/b and /e together will only match if the "remove_line" is the only text on a line. /v will switch the output to only print all lines that do not match.
If you are sure you're not passing special characters, it is pretty easy to wrap that in a small batch and replace remove line with %1 or %* to use your passed parameters.
Unfortunately, you will need to use a temporary file - replacing the file in place doesn't work with the DOS output redirection.
If you were wanting to delete all instances of a specified word within any line, batch and findstr are probably not the way to do it. Look at sed, which can do much more and is freely available. If you can't install another utility, even vbscript using cscript would be a better way to do this than batch.