I am trying to run a batch of Lighthouse audits from the command line and output the results to JSON files. To ensure they don't overwrite one another, each file name needs to be unique. I would like to use the timestamp as the filename ideally. I have been through many other Stack questions on a similar topic, but I cannot get this to work. I am currently using the below code, written within a batch file and called from Windows command line:
SETLOCAL ENABLEDELAYEDEXPANSION
for /f "delims=" %%a in (C:\Users\User\Desktop\Folder\Lighthouse_Project\urls.txt) DO (
set dte=!DATE:/=-%!
set tme=!TIME::=-%!
set tme=!%tme:.=-%!
ECHO Line is: %%a
lighthouse %%a --quiet --chrome-flags=" --headless" --output=json --output-path=C:\Users\User\Desktop\Folder\Lighthouse_Project\json_logs\!dte!!tme!.json
)
The first file produced has the correct name all subsequent JSONs have the literal name: '!dte!!tme!', and do overwrite one another. If you run the Lighthouse command outside of a for loop it executes perfectly, I'm aware that variables don't update within a for loop but I thought !var! could be used to counter this. The for loop iterates through a list of URLs stored in urls.txt, auditing each of them in turn and returning a JSON file.
Current outcome: !dte!!tme!.json
Desired oucome: 08-10-201913-55-12-07.json, 08-10-201913-56-14-00.json etc.
If anybody knows how to amend this to achieve the correct outcome I would be very grateful.
The urls.txt contains a list of URLs, each on a new line:
https://www.bbc.co.uk/weather/2643743
https://www.bbc.com/mundo
https://regex101.com/
https://www.bbc.co.uk/news
Example:
#Echo Off
SETLOCAL ENABLEDELAYEDEXPANSION
for /f "delims=" %%a in (C:\Users\Owner\Desktop\Folder\Lighthouse_Project\urls.txt) DO (
set dte=!DATE:/=-!
set tme=!TIME::=-!
set tme=!tme:.=-!
ECHO Line is: %%a
echo lighthouse "%%a" --quiet --chrome-flags="--headless" --output=json --output-path="C:\Users\Owner\Desktop\Folder\Lighthouse_Project\json_logs\!dte!!tme!.json"
Timeout 2 >NUL
)
Timeout -1 >NUL
This uses the exact content from your urls.txt example. I have added #Echo Off at the top to reduce screen clutter, and an echo in front of your lighthouse command so that it displays instead of runs. Additionally I included a couple of timeouts to both simulate a short timespan, (as the command is being echo'ed not ran), and to give you an opportunity to read the output at the end.
Here's the output from the above to prove my case:
Line is: https://www.bbc.co.uk/weather/2643743
lighthouse "https://www.bbc.co.uk/weather/2643743" --quiet --chrome-flags="--hea
dless" --output=json --output-path="C:\Users\Owner\Desktop\Folder\Lighthouse_Pro
ject\json_logs\08-10-201916-12-42-93.json"
Line is: https://www.bbc.com/mundo
lighthouse "https://www.bbc.com/mundo" --quiet --chrome-flags="--headless" --out
put=json --output-path="C:\Users\Owner\Desktop\Folder\Lighthouse_Project\json_lo
gs\08-10-201916-12-44-14.json"
Line is: https://regex101.com/
lighthouse "https://regex101.com/" --quiet --chrome-flags="--headless" --output=
json --output-path="C:\Users\Owner\Desktop\Folder\Lighthouse_Project\json_logs\0
8-10-201916-12-46-12.json"
Line is: https://www.bbc.co.uk/news
lighthouse "https://www.bbc.co.uk/news" --quiet --chrome-flags="--headless" --ou
tput=json --output-path="C:\Users\Owner\Desktop\Folder\Lighthouse_Project\json_l
ogs\08-10-201916-12-48-21.json"
Press any key to continue . . .
Your issue, as originally stated, was the inclusion of % characters in your initial set syntax, as you can see from the provided output.
By keeping the expansion without exiting the loop as you intended.. Note I have added a timeout of millisecond here to ensure we have a different time each time. ping 127.0.0.1 -n 1 -w 500> nul:
#echo off
setlocal enabledelayedexpansion
for /f "delims=" %%a in (%userprofile%\Desktop\Croud_DS_1\Lighthouse_Project\urls.txt) DO (
set dte=!date:/=-!
set tme=!time::=-!
set tme=!tme:.=-!
set tme=!tme:,=-!
echo Line is: %%a
lighthouse "%%a" --quiet --chrome-flags=" --headless" --output=json --output-path=%userprofile%\Desktop\Folder\Lighthouse_Project\json_logs\!dte!_!tme!.json
ping 127.0.0.1 -n 1 -w 500> nul
)
Please take note, I took the liberty of adding an underscore to make sure the date and time format is readable, so you'll have YYYY-MM-dd_HH-mm-ss-ms
Related
I'm trying to parse a file that contains data, it has some numbers which I want to parse, store and use them as variables in the future. The problem is, these numbers are updated often so that's why I'm parsing them whenever I run my script.
The problem, is that when I collect the data as a variable in a for loop, it's not getting passed to my text file properly.
#echo off
setlocal EnableDelayedExpansion
for /f "skip=104 eol=/ tokens=4 delims=," %%A in (file location) do (
echo %%A
echo %%A > UICNumbers.txt
)
This is the output of the echo %%A command:
UIC00000701991
UIC00000710996
UIC00001701890
UIC00002701890
UIC00001701898
UIC00002701898
When I open the UICNumbers.txt file all I see listed is:
UIC00002701898
which is the last entry in the for loop variable. I tried changing the output to >> but this means every time I run the command it repeats every entry.
I have this situation: I want to download a bunch of files named like this:
683482, 684483, 685484, 686485, 687486, 688487, 689488, 690489, 691490, 692491, ...
As you can see, the files are numbered with an increment of 1001. So, what's the easiest way to do a batch download?
Please try this:
#!/bin/bash
for i in {683482..1000000..1001}
do
wget $i
done
ECHO OFF
CLS
SETLOCAL ENABLEDELAYEDEXPANSION
FOR /L %%A IN (683482,1001, 692491) DO (
SET stringWget=wget %%A
!stringWget!
)
I'll go step by step:
ECHO OFF prevents windows command line from displaying every command on the command-prompt (this is optional; But, looks clean).
CLS clear screen; This clears command prompt's console display. It does not clear the temporary environment variables or command history.
SETLOCAL ENABLEDELAYEDEXPANSION is used for non-blocking statements and is Windows specific; When we use %ENVIRONMENTVARIABLE% , we are implying blocking statement and when we use !ENVIRONMENTVARIABLE! we are implying non-blocking statement (meaning, in a loop, we see updated values of %A instead of repeating what %A had when entering loop). We use %% instead of %in batch files.
FOR loop's syntax can be found in reference.
Reference: https://stackoverflow.com/a/3541415
I am new to writing batch scripts and was hoping for some help.
I have a folder of images. I want to cURL each of these images, one by one, to an image classifier I made on the web. This will then send me a result.
I then want to save the response of the server into a file - let's keep it simple a say a CSV, in this format:
File, Response
So I know to run the Batch on each folder, I can do the following -
for /f %%f in ('dir /b c:\') do echo curl -X POST -F "images_file=#%%f" "https://gateway-a.watsonplatform.net/visual-recognition/api/v3/classify?api_key=abcdefgh&version=2016-05-20"*
I know, I can somehow pipe the results into an output file using ">>".
I don't know how to put this together, and how to create the CSV as I am running the batch script. Can anyone help? I think I am 90% there just new to batch scripting.
You can do something like this in powershell :
Change the placeholders as per your requirement.
$dir_files=Get-ChildItem "C:\"
foreach($file in $dir_files)
{
$result=Invoke-WebRequest 'https://gateway-a.watsonplatform.net/visual-recognition/api/v3/classify?api_key=abcdefgh&version=2016-05-20' -Method Post -Headers 'images_file=#%%f'
$result | ConvertTo-Csv | Out-File "D:\outputfile.csv" -Append -Force
}
I just gave you the way to set off. I didn't go through the complete batch code.
Hope it helps you.
If you want to use batch, you can do:
#echo
SetLocal EnableDelayedExpansion
set imagedir=C:\
set csvfile="C:\results.csv"
FOR %%G IN ("%imagedir%*") DO (
set resultfile=%%G.out
echo curl -X POST -F "images_file=#%%G" "https://gateway-a.watsonplatform.net/visual-recognition/api/v3/classify?api_key=abcdefgh&version=2016-05-20"* -o "!resultfile!"
echo %%G, !resultfile! >> %csvfile%
)
EndLocal
exit /b 0
I am not that experienced with curl so I'm not able to verify the correctness of your curl command and let the echo before it. But I do know that you can actually specify a file to write the output to in Windows with the -o flag.
The FOR command (without any flag/option) is another way to loop through files in a directory. If you still want to use the for /f %%f in ('dir /b ...') do-style (perhaps because of the order in which you want to treat your images), you can simply replace the line FOR %%G IN ("%imagedir%*") DO ( with for /f %%f in ('dir /b "%imagedir%"') DO (. I saved the directory in the variable imagedir, it's then easier to change. But if you change it, don't forget the \ at the end (if you use my for-command) and don't add double quotes (I add them in the FOR command). Be aware that this will treat all FILES in the directory. You can specify extensions if you want by using "%imagedir%\*.jpg" "%imagedir%\*.gif" for example for only GIF and JPEG images.
As last I'll say that I added SetLocal EnableDelayedExpansion to use delayed expansion. It's actually only needed because I saved the name of the file where the result of the curl operation will be saved, in the variable resultfile. In batch, a FOR -block (FOR ... ( ... )), and other blocks delimited with ( ) (as IF-blocks), are parsed as one single command (as if they were written on line). It is not really possible to change the value of a variable and read that new value in the same command (or same line) without using some special "tricks". Delayed expansion is one of them. When you use delayed expansion on a variable, you surround it with ! instead of %. As resultfile will change in each iteration, I use !resultfile! instead of %resultfile%.
If you don't want to save the name of the outputfile in a variable, you can remove the use of the variable and the delayed expansion (remove the EndLocal at the end too).
Good luck!
I'd like to get a changelist description from perforce, which involves calling a p4 describe -s , so the ouput would be as below. Is there a way to get (trimmed characters from the third line) from the output just using windows batch syntax?
Change 6582 by username on 2016/12/06 00:35:41
MyChangeDescription
Affected files ...
... //depot/foo.txt#7 edit
... //depot/foo2.txt#6 edit
Give this a shot:
p4 -Ztag -F %Description% change -o 6582
#ECHO OFF
SETLOCAL
SET "sourcedir=U:\sourcedir"
SET "filename1=%sourcedir%\q40986156.txt"
FOR /f "usebackqskip=2tokens=*" %%a IN ("%filename1%") DO (
SET "desc=%%a"
GOTO show
)
:show
ECHO "%desc%"
GOTO :EOF
You would need to change the setting of sourcedir to suit your circumstances.
I used a file named q40986156.txt containing your data for my testing.
This uses a file as input. Since I don't have access to perforce, I can't test it but
#ECHO OFF
SETLOCAL
FOR /f "skip=2tokens=*" %%a IN ('p4 describe -s') DO (
SET "desc=%%a"
GOTO show
)
:show
ECHO "%desc%"
GOTO :EOF
should be equivalent.
Simply, read the output of the command, skip the first 2 lines, tokenise the entire line, skipping leading spaces. Assign the string found to a variable and immediately exit the loop.
I am working on a batch script and am trying to save part of the output of a command as a variable. My current script requires the user to enter a site code (e.g. 853), and it takes that code and uses it to query a sql database. The command I'm using is as follows:
sqlcmd -S"localhost\SQLONESOURCE" -Uusername -Pmypassword -dOneSource -Q"select AdmSiteID from [10.33.144.114].onesource.dbo.admsites where siteid='%sitecode%'"
When this command runs, I get the following output:
AdmSiteID------------------------------------
AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE
(1 rows affected)
I just want to take the AdmSiteID (AAAAAAAA-BBBB-CCCC-DDDD-EEEEEEEEEEEE) and save that as a variable. It will always be in that format with the same number of characters and dashes.
This batch code get the third line from your sqlcmd output:
for /F "skip=2" %%a in ('sqlcmd -S"localhost\SQLONESOURCE" -Uusername -Pmypassword -dOneSource -Q"select AdmSiteID from [10.33.144.114].onesource.dbo.admsites where siteid='%sitecode%'"') do (
set thirdLine=%%a
goto continue
)
:continue
echo %thirdLine%
The first thing to do is to ensure your SQLCMD output includes some sort of unique text that you can search for. For example, you could modify your query as follows:
"select 'FOO~' + AdmSiteID + '~' from [10.33.144.114].onesource.dbo.admsites where siteid='%sitecode%'"
Then write your SQLCMD output to a file by adding -o %OUTPUT_FILE% to your SQLCMD.EXE line, where %OUTPUT_FILE% has already been defined.
Then use a combination of for and findstr.exe to pull your file apart:
for /f "tokens=2 delims=~" %%D in ('%SystemRoot%\System32\findstr.exe /l /c:"FOO~" %OUTPUT_FILE%') do set YOUR_VARIABLE=%%D
This hasn't been tested, but I know the theory is sound. If it doesn't work, and you're struggling to fix it, let me know, and I'll spend more time on it.