BIG Query command using BAT file - windows

Question

echo Give yearmonth "yyyyMM"
setlocal enabledelayedexpansion
SET /p yearmonth=
SET ClientName[0]=abc
SET ClientName[1]=def
SET i = 0
:myLoop
if defined ClientName[%i%] (
call bq query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.!ClientName[%%i]!.%yearmonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/!ClientName[%%i]!/AWS/%yearmonth%/Metrics/data/*.csv'])"
set /a "i+=1"
GOTO :myLoop
)
Hi, I am trying to create a batch so that i can run Multiple BIG QUERY at once.
Above i tried to write a batch script putting command in a loop .
I am trying to create a table by using yearmonth as user input and then create array to create a table with different client name .
But I am unable to print if i =0 ClientName[i] = abc in a call query i am using !ClientName[%%i]! to print but its not working.
Call query inside loop is not running in GCP console, when i executed the bat file .
Can you please help me resolve this

Answer 1

It is bad practice to set variables as standalone alphabetical characters like i. One reason is exactly as you have experienced, you have confused for metavariable %%i with a set variable %i%.
You are expanding in the loop, but have not enabledelayedexpansion so there are 2 ways, which we will get to in a second.
setting variables should not have spaces before or after = excluding the likes of set /a
So, Method 1, without delayedexpansion (note how the variables are used with double %% in the loop with the call command).
#echo off
echo Give yearmonth "yyyyMM"
SET /p yearmonth=
SET ClientName[0]=abc
SET ClientName[1]=def
SET num=0
:myLoop
if defined ClientName[%num%] (
call bq query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.%%ClientName[%num%]%%.%yearmonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/%%ClientName[%num%]%%/AWS/%yearmonth%/Metrics/data/*.csv'])"
set /a num+=1
GOTO :myLoop
)
Method 2: (better method using delayedexpansion)
#echo off
setlocal enabledelayedexpansion
echo Give yearmonth "yyyyMM"
SET /p yearmonth=
SET ClientName[0]=abc
SET ClientName[1]=def
SET num=0
:myLoop
if defined ClientName[%num%] (
call bq query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.!ClientName[%num%]!.%yearmonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/!ClientName[%num%]!/AWS/%yearmonth%/Metrics/data/*.csv'])"
set /a num+=1
GOTO :myLoop
)

Answer 2

I would go for a more intuitive looping mechanism:
For /F "Tokens=1,* Delims==" %%G In ('"(Set ClientName[) 2>NUL"') Do Call "%ProgramFiles%\Google\Cloud SDK\google-cloud-sdk\bin\bq.cmd" query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.%%G.%yearmonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/%%G/AWS/%yearmonth%/Metrics/data/*.csv'])"
It does not require delayed expansion, or incrementing of variables, and does nothing if no variable named ClientName[n] is defined.
It would therefore mean you could have your variables listed but at some point comment one or more out:
Set "ClientName[0]=abc"
Set "ClientName[1]=def"
Rem Set "ClientName[2]=ghi"
Set "ClientName[3]=jkl"
When the loop ran, unlike in the incrementing mechanism, there would be no unnecessary parsing of a parenthesized If Defined %ClientName[2]% … code block.
However, there are several things in your question which open up the possibility of potential issues. The biggest being that you are seemingly expecting user input, but at the same time trusting that the input is provided, and is formatted according to your expectations. You must always, when writing scripts requiring interaction, assume that the end user cannot or will not always follow the instructions.
Here therefore is a more robustly designed script to perform your intended query:
#Echo Off
SetLocal EnableExtensions DisableDelayedExpansion
Rem Define a variable named bq pointing to your bigquery batch file.
Set "bq=%ProgramFiles%\Google\Cloud SDK\google-cloud-sdk\bin\bq.cmd"
Rem If the bigquery batch file does not exist, terminate the script.
If Not Exist "%bq%" Exit /B
Rem Ensure that there are not existing ClientName[n] variables defined.
For /F "Delims=" %%G In ('"(Set ClientName[) 2>NUL"') Do Set "%%G="
Rem Define your ClientName[n] variable list.
Set "ClientName[0]=abc"
Set "ClientName[1]=def"
:GetYearMonth
Rem Ensure that no existing variable named YearMonth is defined.
Set "YearMonth="
Rem Request user input of date saved to the variable named YearMonth.
Set /P "YearMonth=Please provide the date in the format yyyyMM>"
Rem If no input was received, ask again.
If Not Defined YearMonth GoTo GetYearMonth
Rem Remove any doublequotes from the YearMonth variable value.
Set "YearMonth=%YearMonth:"=%"
Rem If YearMonth variable value is not within allowed range, ask again.
Rem This example uses 197001 - 202212
(Set YearMonth) 2>NUL | %SystemRoot%\System32\findstr.exe /R^
/C:"^YearMonth=19[789][0123456789]0[123456789]$"^
/C:"^YearMonth=19[789][0123456789]1[012]$"^
/C:"^YearMonth=20[01][0123456789]0[123456789]$"^
/C:"^YearMonth=20[01][0123456789]1[012]$"^
/C:"^YearMonth=202[012]0[123456789]$"^
/C:"^YearMonth=202[012]1[012]$"^
1>NUL || GoTo GetYearMonth
Rem As input information is valid loop your command.
For /F "Tokens=1,* Delims==" %%G In ('"(Set ClientName[) 2>NUL"'
) Do Call "%bq%" query --use_legacy_sql=false "CREATE EXTERNAL TABLE `test.%%G.%YearMonth%` OPTIONS (format = 'CSV',skip_leading_rows = 1 uris = ['gs://test/%%G/AWS/%YearMonth%/Metrics/data/*.csv'])"
EndLocal