I have two Windows batch files, the first one contains a FOR looping command:
echo off
for /f %%1 in (somefile.txt) do something %%1
echo then more commands...
Then the second file something.bat contains a GCP BigQuery command, such as below. Notice the replaceable %%1 parameter supplied to it by the first batch file.
call bq query --use_legacy_sql=false "WITH intermediate AS (SELECT service.description AS SERVICE,
service.id as ID, invoice.month as MONTH, (SUM(CAST(cost * 1000000 AS int64)) +
SUM(IFNULL((SELECT SUM(CAST(c.amount * 1000000 as int64)) FROM UNNEST(credits) c), 0))) / 1000000
AS COST FROM myproject.billingexport.gcp_billing_export_v1_022EAF_F204503_AR2346 WHERE service.id =
\"%1\" group by 1, 2, 3) SELECT distinct SERVICE, (SELECT round(cast(COST as float64),2)
FROM intermediate WHERE MONTH = \"202001\") as Jan, FROM intermediate"
The commands work fine when executed one-by-one. But when the FOR command in file #1 completes, the rest of the batch file is skipped, and execution stops. I have tried various combinations of CALL (or absent that), or moving the >> output redirect from the FOR command versus in the something.bat. Nothing obvious seems amiss--must be something simple. Ideas?? Thanks.
Related
Currently, I have a sequence job in DataStage.
Here is the flow:
StartLoop Activity --> UserVariables Activity --> Job Activity --> Execute Command --> Endloop Activity
The job will run every 30 minutes (8 AM - 8 PM) to get real data. The first loop iteration will load data from 8 PM the previous day to 8 AM the current day, and the others will load data that happens in the last 30 minutes.
The UserVariables Activity is to pass variables (SQL statement) to filter data getting in the Job Activity. The first iteration the UserVariables pass variable A (SQL statement 1) to the Job Activity, from the second iteration, it will pass variable B (SQL statement 2) to the Job Activity.
The Execute Command I currently set the 'Sleep 1800' command for the job to sleep 30 minutes to end the iteration of the loop. But I realized now that it is affected by the running time of each iteration. So with my knowing-nothing about shell script, I have searched for solutions and have this file to sleep until a specific time when minute like 30 or 00 (delay 0-1 minute but it's fine).
The shell script is below, I ran it fine on my system but no success on making it as part of the job.
#!/bin/bash
minute=$(date +%M)
num_1=30
num_2=60
if [ $minute -le 30 ];
then
wait=$((($num_1 - $minute)*$num_2))
sleep $wait
fi
if [ $minute -gt 30 ];
then
wait=$((($num_2 - $minute)*$num_2))
sleep $wait
fi
I am now facing 2 problems right now that I need your help with.
The job runs the first iteration fine with the variable A below:
select * from my_table where created_date between trunc(sysdate-1) + 20/24 and trunc(sysdate) + 8/24;
But from the second iteration it failed with the Job Activity with the variable B below:
select * from my_table where created_date between trunc(sysdate-1/48, 'hh') + 30*trunc(to_number(to_char(sysdate-1/48,'MI'))/30)/1440 and trunc(sysdate, 'hh') + 30*trunc(to_number(to_char(sysdate,'MI'))/30)/1440;
In the parallel job, the log said:
INPUT,0: The following SQL statement failed: select * from my_table where created_date between trunc(sysdate-1/48, hh) + 30*trunc(to_number(to_char(sysdate-1/48,MI))/30)/1440 and trunc(sysdate, hh) + 30*trunc(to_number(to_char(sysdate,MI))/30)/1440.
I realized that maybe it failed to run the parallel job because it removed the single quote in hh and MI.
Is it because when passing variables from UserVariables Activity to Job Activity the variable will remove all the quotes? And how can I fix this?
2. How can I make the shell script above as part of the job like Execute Command or some other stage. I have searched for solutions and I think it's about the ExecSH Before/ After Routine Activity. But after reading from IBM pages, I still don't know where to start with it.
Sorry for adding 2 questions at 1 post that makes it so long but it's very relative to each other so it will take lots of time to answer if I separate it into 2 posts and you guys need more information about it.
Thank you!
Try escaping the single quote characters (precede each with a backslash).
Execute the shell script from an Execute Command activity ahead of the Job activity.
What have I tried
I wanted to create a way to make a faster pinging method for cmd than the normal '-t' command. So I came up with an idea to make two files and link them together:
The first st1.bat file contains
ping 10.10.10.10 -n 1
st2.bat
And the second st2.bat file contains
ping 10.10.10.10 -n 1
st1.bat
This creates a loop so it goes on forever.
The '-n' command is the number of packets you send. So by sending just one packet and then opening the second .bat file makes it a lot faster (30 milliseconds).
The command for extracting just the IP addresses is:
(#for /f "tokens=8" %a in ('tracert -4 -d wikipedia.org^|find "ms"') do #echo start ping %a)>ips.txt
What do I need
I want to combine these two codes and make 2 files for each IP to create a loop just like I showed you above. Every IP loop should have its own cmd window because if everything is in one window every packet should wait for the previous one to be done:
And finally link all the files together in one .bat file that I think will look like that:
Start st1.bat
Start st3.bat
Start st5.bat
Try to use: ping -n 1 -l 1
if you doesn't want
Firing with 1 bytes of data: try this command:
#echo off
:1
ping -n 1 -l 1 | find "TLL="
goto 1
I'm running several independent programs on a single machine in parallel.
The processes (say 100) are all relatively short (<5 minutes) and their output is limited to a few hundred lines (~kilobytes).
Usually the output in a terminal then becomes mangled because the processes write directly to the same buffer. I would like these outputs to be un-mangled so that it's easier to debug certain processes. I could write these outputs to temporary files but I would like to limit disk IO and would prefer another method if possible. It would require cleaning up and probably won't really improve code readability.
Is there any shell native method that allows buffers to be PID separated which then flushes to stdout/stderr when the process terminates ? Do you see any other way to do this ?
Update
I ended up using the tail -n 1000000 trick from the comment of #Gem. Since the commands I'm using are long and (covering multiple lines) and I was already using subshells ( ... ) & that was a quite minimal change from ( ... ) & to ( ... ) 2>&1 | tail -n 1000000 &.
You can do that with GNU Parallel. Use -k to keep the output in order and ::: to separate the arguments you want passed to your program.
Here we run 4 instances of echo in parallel:
parallel -k echo {} ::: {0..4}
0
1
2
3
4
Now add in --tag to tag your output lines with the filenames or parameters you are using:
parallel --tag -k 'echo "Line 1, param {}"; echo "Line 2, param {}"' ::: {1..4}
1 Line 1, param 1
1 Line 2, param 1
2 Line 1, param 2
2 Line 2, param 2
3 Line 1, param 3
3 Line 2, param 3
4 Line 1, param 4
4 Line 2, param 4
You should notice that each line is tagged on the left side with the parameters and that the two lines from each job are kept together.
You can now specify how your output is organised.
Use --group to group output by job
Use --line-buffer to buffer a line at a time
Use --ungroup if you want output all mixed up, but as soon as available
Sounds like you just want syslog, or rather logger its Bash interface. Example:
echo "Something happened!" | logger -i -p local0.notice
If you insist on getting output to stderr too use --stderr. rsyslog will handle buffering, atomic writes, etc, and is presumably pretty good at optimizing disk I/O. However you could also easily configure rsyslog to route the log facility (i.e. local0 or what ever you choose to use) where ever you want, such as on a tmpfs or dedicated disk, or even over TCP. See /etc/rsyslog.conf.
I have the following command in a batch file.
set tableName=%1
select count(1) from %tableName% where to_char(DATEVALUE,'yyyy-mm-dd hh24:mi:ss')^>(select to_char(max(DATEVALUE),'yyyy-mm-dd hh24:mi:ss') from FOO_TABLE); | sqlplus !connectionString!
This statement doesn't work. I can see that it connects to the database and then disconnects. But the following works:
select count(1) from %tableName% where to_char(DATEVALUE,'yyyy-mm-dd hh24:mi:ss')=(select to_char(max(DATEVALUE),'yyyy-mm-dd hh24:mi:ss') from FOO_TABLE); | sqlplus !connectionString!
I am guessing the problem could be with the greater than > symbol. I tried ^>,> and \>. None of them works. How can I get this sql statement to work.
(I have connectionString already set in my batch file in earlier lines).
The output in the command line is
Connected to:
Oracle Database ... (more db info)
SQL> Disconnected from Oracle Database ... (more db info)
It looks like you need to escape the ^ escape character as well; depending on exactly how you're running this, either:
... where to_char(DATEVALUE,'yyyy-mm-dd hh24:mi:ss')^^>(select ...
or
... where to_char(DATEVALUE,'yyyy-mm-dd hh24:mi:ss')^^^>(select ...
In a batch file where the query is echoed and piped the triple-escape works:
#setlocal EnableDelayedExpansion
#set connectionString=x/y#z
#set tableName=bar
#echo select count(1) from %tableName% where to_char(DATEVALUE,'yyyy-mm-dd hh24:mi:ss')^^^>(select to_char(max(DATEVALUE),'yyyy-mm-dd hh24:mi:ss') from FOO_TABLE); | sqlplus !connectionString!
Running that batch script shows the statement being run (and erroring in my case with ORA-00942, which is expected). With a single or double ^ it has nothing to run at the SQL prompt and a file is created instead, which seems to be what you're seeing.
I would like to test for the success/failure of a copy in a batch file, but I can't find any documentation on what if any errorlevel codes are returned. For example
copy x y
if %errorlevel%. equ 1. (
echo Copy x y failed due to ...
exit /B
) else (
if %errorlevel% equ 2. (
echo Copy x y failed due to ...
exit /B
)
... etc ...
)
I'd opt for xcopy in this case since the error levels are documented (see xcopy documentation, paraphrased below):
Exit code Description
========= ===========
0 Files were copied without error.
1 No files were found to copy.
2 The user pressed CTRL+C to terminate xcopy.
4 Initialization error occurred. There is not
enough memory or disk space, or you entered
an invalid drive name or invalid syntax on
the command line.
5 Disk write error occurred.
In any case, xcopy is a far more powerful solution. The equivalent documentation for copy does not document the error levels.
As an aside, you may want to rethink your use of the %errorlevel% variable. It has unexpected results, at least in some versions of Windows, if someone has explicitly done something silly like:
set errorlevel=22
In those cases, the actual variable will be used rather than grabbing the actual error level. The "normal" way of doing this is (in decreasing order since errorlevel is a "greater than or equal to" check):
if errorlevel 2 (
echo Copy x y failed due to reason 2
exit /B
)
if errorlevel 1 (
echo Copy x y failed due to reason 1
exit /B
)
In addition, if you are running Win7 or Win Server 2008 or later, you should look into Robocopy, which is now the preferred mass-copy solution.
It might also be worth pointing out that xcopy doesn't always return the error code you expect.
For example when trying to copy multiple files with a wildcard but there are no files to copy you expect a return error code of 1 ("No files were found to copy"), but it actually returns 0 ("Files were copied without error")
C:\Users\wilson>mkdir bla
C:\Users\wilson>mkdir blert
C:\Users\wilson>xcopy bla\* blert\
0 File(s) copied
C:\Users\wilson>echo %ERRORLEVEL%
0
I believe Copy only returns 0 for success or 1 for failure.
XCopy has documented return codes:
0 = Files were copied without error.
1 = No files were found to copy.
2 = The user pressed CTRL+C to terminate xcopy.
4 = Initialization error occurred. There is not enough memory or disk space, or you entered an invalid drive name or invalid syntax on the command line.
5 = Disk write error occurred.
There is also one point I would like to emphasize: xcopy as well as robocopy can only copy files, but they can't rename them.
While looking at the original situation (copy x y, which looks like a rename to me), I have the impression that the copy command still is the only one suitable for this purpose.
Error# Description
0 No error
1 Not owner
2 No such file or directory
3 Interrupted system call
4 I/O error
5 Bad file number
6 No more processes
7 Not enough core memory
8 Permission denied
9 Bad address
10 File exists
11 No such device
12 Not a directory
13 Is a directory
14 File table overflow
15 Too many open files
16 File too large
17 No space left on device
18 Directory not empty
999 Unmapped error (ABL default)