Parameter passing to a shell script using BashOperator in Airflow 2.0 - shell

I am using airflow 2.0 and recently came up with a problem. So in my dag, I am calling a shell script by passing arguments like date and hour and in the script it should ideally fetch the value and perform some operation. But unfortunately, the values are not getting substituted.
Date='{{ (next_execution_date + macros.timedelta(hours=9,days=-3)).strftime("%Y%m%d") }}' //global
BashOperator=(task_id="",
dag=dag,
bash_command='''
echo "hi"
'''
+
'''
/path /file.sh Date Hr
'''
)
In the shell script file.sh:
echo "Date is $1"
echo "Hour is $2"
Checked the logs and it printed- Date is Date, Hour is Hr. Can anyone please help me with correct way to pass these arguments?

You might have an issue with a quote being off at one point, hard to tell from your question.
This works for me (Airflow 2.5.1):
DAG
from airflow import DAG
from pendulum import datetime
from airflow.operators.bash import BashOperator
with DAG(
dag_id="scripting_dag",
start_date=datetime(2022,12,10),
schedule="#daily",
catchup=False,
) as dag:
t1 = BashOperator(
task_id="t1",
bash_command="my_script.sh",
env={
"my_date": "{{ (next_execution_date + macros.timedelta(hours=9,days=-3)).strftime('%Y%m%d') }}",
"my_hour": "{{ (next_execution_date + macros.timedelta(hours=9,days=-3)).strftime('%H') }}"
}
)
script my_script.sh:
echo $my_date
echo $my_hour
Logs
[2023-02-03, 16:35:28 UTC] {subprocess.py:75} INFO - Running command: ['/bin/bash', '-c', 'echo $my_date\necho $my_hour']
[2023-02-03, 16:35:28 UTC] {subprocess.py:86} INFO - Output:
[2023-02-03, 16:35:28 UTC] {subprocess.py:93} INFO - 20230201
[2023-02-03, 16:35:28 UTC] {subprocess.py:93} INFO - 01

Related

Airflow - How to get an Airflow variable inside the bash command in Bash Operator?

I have an Airflow variable
And I would like to get it inside a bash command on Bash Operator.
I tried:
t2 = BashOperator(
task_id= 'try_bash',
bash_command="echo {{var.aa}}",
dag=dag
and
t2 = BashOperator(
task_id= 'try_bash',
bash_command=" echo {{var.value.aa}}",
dag=dag
I know that to get a variable I can do
Variable.get("aa")
But I would like to get it directly inside the bash command.
Is it possible?
Thanks
This should work. Just a slight difference with what you originally had:
t2 = BashOperator(
task_id= 'try_bash',
bash_command='echo "{{ var.value.aa }}"',
dag=dag,
)

Sort command in bash script does not work when called from Jenkins

I am trying to execute a shell script on a windows node using Jenkins.
The bash script uses sort -u flag in one of the steps to filter out unique elements from an existing array
list_unique=($(echo "${list[#]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))
Note - shebang used in the script is #!/bin/bash
On calling the script from command prompt as - bash test.sh $arg1
I got the following error -
-uThe system cannot find the file specified.
I understand the issue was that with the above call, sort.exe was being used from command prompt and not the Unix sort command. To get around this I changed the path variable in Windows System variables and moved \cygwin\bin ahead of \Windows\System32
This fixed the issue and the above call gave me the expected results.
However, When the same script is called on this node using Jenkins, I get the same error again
-uThe system cannot find the file specified.
Jenkins stage calling the script
stage("Run Test") {
options {
timeout(time: 5, unit: 'MINUTES')
}
steps {
script {
if(fileExists("${Test_dir}")){
dir("${Test_dir}"){
if(fileExists("test.sh")){
def command = 'bash test.sh ${env.arg1}'
env.output = sh(returnStdout: true , script : "${command}").trim()
if (env.output == "Invalid"){
def err_msg = "Error Found."
sh "echo -n '" + err_msg + " ' > ${ERR_MSG_FILE}"
error(err_msg)
}
sh "echo Running tests for ${env.output}"
}
}
}
}
}
}
Kindly Help

Shell 'echo' error with ENV VAR been parsed from 'redis-cli INFO' output

I try to monitor Redis health with Zabbix agent and want to parse output for redis-cli INFO command:
$ redis-cli INFO
# Server
redis_version:4.0.9
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:9435c3c2879311f3
...
with shell script and compose JSON file with subset of info values:
{
"redis_version": "4.0.9",
"used_memory": 100,
"used_memory_rss": 200
}
But I have an error when I try to echo parsed values to console. This is my demo script:
#!/bin/bash
FILE_ECHO="/tmp/test_out.txt"
FILE_REDIS="/tmp/test_redis.txt"
echo "redis_version:4.0.9" >${FILE_ECHO}
/usr/bin/redis-cli INFO >${FILE_REDIS}
ECHO_VERSION=$(grep "redis_version" "${FILE_ECHO}")
REDIS_VERSION=$(grep "redis_version" "${FILE_REDIS}")
echo "{\"node\": \"${ECHO_VERSION}\"}"
echo "{\"node\": \"${REDIS_VERSION}\"}"
I expect that output will be:
{"node": "redis_version:4.0.9"}
{"node": "redis_version:4.0.9"}
in both cases, but actually I have:
{"node": "redis_version:4.0.9"}
"}node": "redis_version:4.0.9
This is my test_redis.txt.

execution of shell command from jenkinsfile

I am trying to execute set of commands from jenkinsfile.
The problem is, when I try to assign the value of stdout to a variable it is not working.
I tried different combinations of double quotes and single quotes, but so far no luck.
Here I executed the script with latest version of jenkinsfile as well as old version. Putting shell commands inside """ """ is not allowing to create new variable and giving error like client_name command does not exist.
String nodeLabel = env.PrimaryNode ? env.PrimaryNode : "slave1"
echo "Running on node [${nodeLabel}]"
node("${nodeLabel}"){
sh "p4 print -q -o config.yml //c/test/gradle/hk/config.yml"
def config = readYaml file: 'devops-config.yml'
def out = sh (script:"client_name=${config.BasicVars.p4_client}; " +
'echo "client name: $client_name"' +
" cmd_output = p4 clients -e $client_name" +
' echo "out variable: $cmd_output"',returnStdout: true)
}
I want to assign the stdout from the command p4 clients -e $client_name to variable cmd_output.
But when I execute the code the error that is thrown is:
NoSuchPropertyException: client_name is not defined at line cmd_output = p4 clients -e $client_name
What am I missing here?
Your problem here is that all the $ are interpreted by jenkins when the string is in double quotes. So the first 2 times there's no problem since the first variable comes from jenkins and the second time it's a single quote string.
The the third variable is in a double quote string, therefore jenkins tries to replace the variable with its value but it can't find it since it's generated only when the shell script is executed.
The solution is to escape the $ in $client_name (or define client_name in an environment block).
I rewrote the block:
String nodeLabel = env.PrimaryNode ? env.PrimaryNode : "slave1"
echo "Running on node [${nodeLabel}]"
node("${nodeLabel}"){
sh "p4 print -q -o config.yml //c/test/gradle/hk/config.yml"
def config = readYaml file: 'devops-config.yml'
def out = sh (script: """
client_name=${config.BasicVars.p4_client}
echo "client name: \$client_name"
cmd_output = p4 clients -e \$client_name
echo "out variable: \$cmd_output"
""", returnStdout: true)
}

Executing bash strings using scala.sys.process

I recently discovered sys.process package in Scala, and was amused by its power.
But when I try to combine it with bash pipes and backticks, I get stuck.
This obviously doesn't work:
scala> "echo `date`" !!
res0: String = "
"`date`
"
I tried to use the bash executable to get the desired behavior:
scala> "bash -e echo `date`" !!
/bin/echo: /bin/echo: cannot execute binary file
java.lang.RuntimeException: Nonzero exit value: 126
What am I doing wrong?
Edit:
scala> "bash -ic 'echo `date`'" !!
`date`': unexpected EOF while looking for matching `''
`date`': syntax error: unexpected end of file
java.lang.RuntimeException: Nonzero exit value: 1
You're doing multiple things wrong actually. You should be using the -c option of bash and you should be using a Seq[String] with each parameter to bash in its own String, or the scala library will just split the String at every space character. (This is why Rex Kerr's solution doesn't work.)
scala> import sys.process.stringSeqToProcess
import sys.process.stringSeqToProcess
scala> Seq("bash", "-c", "echo `date`")!!
res20: String =
"Sun Dec 4 16:40:04 CET 2011
"

Resources