Bash command in variable with herestring as input - bash

Using bash, I have a need to echo a series of commands before I run them. Take this example, which is nice and easy and works as expected. Note that all code examples have been run through ShellCheck which reports no issues detected.
OUTPUTS_CMD=(aws cloudformation describe-stacks --stack-name "#{StackName}" --query 'Stacks[0].Outputs')
echo "${OUTPUTS_CMD[*]}"
OUTPUTS=$("${OUTPUTS_CMD[#]}")
However, other commands require input, and while I can successfully echo these commands, I can't actually get them to run.
GET_FUNCTION_NAME_CMD=(jq --raw-output "'map(select(.OutputKey == \"ProxyFunctionName\")) | .[].OutputValue'")
echo "${GET_FUNCTION_NAME_CMD[*]} <<< \"\$OUTPUTS\""
FUNCTION_NAME=$("${GET_FUNCTION_NAME_CMD[#]}" <<< "$OUTPUTS")
The above example outputs the following, which if copied and pasted returns the correct value.
jq --raw-output 'map(select(.OutputKey == "ProxyFunctionName")) | .[].OutputValue' <<< "$OUTPUTS"
However, the command "${GET_FUNCTION_NAME_CMD[#]}" <<< "$OUTPUTS" returns an error. The same error occurs if I echo "$OUTPUTS" and pipe that in to the command saved in the variable instead of using a herestring, so I believe the error is with how the command is defined in the array.
$ FUNCTION_NAME=$("${GET_FUNCTION_NAME_CMD[#]}" <<< "$OUTPUTS")
jq: error: syntax error, unexpected INVALID_CHARACTER, expecting $end (Unix shell quoting issues?) at <top-level>, line 1:
'map(select(.OutputKey == "ProxyFunctionName")) | .[].OutputValue'
jq: 1 compile error
How can I get the command in the variable to run with a herestring?
Example value for $OUTPUTS
[
{
"OutputKey": "ProxyFunctionName",
"OutputValue": "MyFunctionName",
"Description": "Proxy Lambda Function ARN"
},
{
"OutputKey": "ProxyFunctionUrl",
"OutputValue": "https://my.function.url",
"Description": "Proxy Lambda Function invocation URL"
}
]

OUTPUTS=$(cat <<JSON
[
{
"OutputKey": "ProxyFunctionName",
"OutputValue": "MyFunctionName",
"Description": "Proxy Lambda Function ARN"
},
{
"OutputKey": "ProxyFunctionUrl",
"OutputValue": "https://my.function.url",
"Description": "Proxy Lambda Function invocation URL"
}
]
JSON
)
OutputKey=ProxyFunctionName
GET_FUNCTION_NAME_CMD="jq -r '.[] | objects | select(.OutputKey == \"$OutputKey\") | .OutputValue'"
echo "$GET_FUNCTION_NAME_CMD <<<\"\$OUTPUTS\""
# jq -r '.[] | objects | select(.OutputKey == "ProxyFunctionName") | .OutputValue' <<<"$OUTPUTS"
FUNCTION_NAME=$(eval $GET_FUNCTION_NAME_CMD <<<"$OUTPUTS")
echo $FUNCTION_NAME
# MyFunctionName

Related

Passing json to aws glue create-job after replacement done using jq

I have the following bash script that I execute in order to create new Glue Job via CLI:
#!/usr/bin/env bash
set -e
NAME=$1
PROFILE=$2
SCRIPT_LOCATION='s3://bucket/scripts/'$1'.py'
echo [*]--- Creating new job on AWS
aws glue create-job --profile $PROFILE --name $NAME --cli-input-json | jq '.Command.ScriptLocation = '\"$SCRIPT_LOCATION\"'' ./resources/config.json
I'm using jq as i need one of the values to be replaced on runtime before i pass the .json as --cli-input-json argument. How can i pass json with replaced value to this command? As of now, it prints out the json content (although with value already replaced).
Running the command above causes the following error:
[*]--- Creating new job on AWS
{
"Description": "Template for Glue Job",
"LogUri": "",
"Role": "arn:aws:iam::11111111111:role/role",
"ExecutionProperty": {
"MaxConcurrentRuns": 1
},
"Command": {
"Name": "glueetl",
"ScriptLocation": "s3://bucket/scripts/script.py",
"PythonVersion": "3"
},
"DefaultArguments": {
"--TempDir": "s3://temp/admin/",
"--job-bookmark-option": "job-bookmark-disable",
"--enable-metrics": "",
"--enable-glue-datacatalog": "",
"--enable-continuous-cloudwatch-log": "",
"--enable-spark-ui": "true",
"--spark-event-logs-path": "s3://assets/sparkHistoryLogs/"
},
"NonOverridableArguments": {
"KeyName": ""
},
"MaxRetries": 0,
"AllocatedCapacity": 0,
"Timeout": 2880,
"MaxCapacity": 0,
"Tags": {
"KeyName": ""
},
"NotificationProperty": {
"NotifyDelayAfter": 60
},
"GlueVersion": "3.0",
"NumberOfWorkers": 2,
"WorkerType": "G.1X"
}
usage: aws [options] <command> <subcommand> [<subcommand> ...] [parameters]
To see help text, you can run:
aws help
aws <command> help
aws <command> <subcommand> help
aws.exe: error: argument --cli-input-json: expected one argument
The command line
aws glue create-job --profile $PROFILE --name $NAME --cli-input-json | jq '.Command.ScriptLocation = '\"$SCRIPT_LOCATION\"'' ./resources/config.json
executes the command
aws glue create-job --profile $PROFILE --name $NAME --cli-input-json,
takes its standard output and uses it as input to
jq '.Command.ScriptLocation = '\"$SCRIPT_LOCATION\"'' ./resources/config.json
(which will ignore the input and read from the file given as argument). Please also note that blanks or spaces in $SCRIPT_LOCATION will break your script, because it is not quoted (your quotes are off).
To use the output of one command in the argument list of another command, you must use Command Substitution: outer_command --some-arg "$(inner_command)".
So your command should become:
aws glue create-job --profile $PROFILE --name $NAME --cli-input-json "$(jq '.Command.ScriptLocation = "'"$SCRIPT_LOCATION"'"' ./resources/config.json)"
# or simplified with only double quotes:
aws glue create-job --profile $PROFILE --name $NAME --cli-input-json "$(jq ".Command.ScriptLocation = \"$SCRIPT_LOCATION\"" ./resources/config.json)"
See https://superuser.com/questions/1306071/aws-cli-using-cli-input-json-in-a-pipeline for additional examples.
Although, I have to admit I am not 100% certain that the JSON content can be passed directly on the command line. From looking at the docs and some official examples, it looks like this parameter expects a file name, not a JSON document's content. So it could be possible that your command in fact needs to be:
# if "-" filename is specially handled:
jq ".Command.ScriptLocation = \"$SCRIPT_LOCATION\"" ./resources/config.json | aws glue create-job --profile $PROFILE --name $NAME --cli-input-json -
# "-" filename not recognized:
jq ".Command.ScriptLocation = \"$SCRIPT_LOCATION\"" ./resources/config.json > ./resources/config.replaced.json && aws glue create-job --profile $PROFILE --name $NAME --cli-input-json file://./resources/config.replaced.json
Let us know which one worked.

Trying to verify jq command output equal string or string has more than one occurrence (AWS ELB instances state query)

I'm trying to check that all instances attached to an AWS ELB are in a state of "InService",
For that, I created an AWS CLI command to check the status of the instances.
problem is that the JSON output returns the status of both instances.
So it is not that trivial to examine the output as I wish.
When I run the command:
aws elb describe-instance-health --load-balancer-name ELB-NAME | jq -r '.[] | .[] | .State'
The output is:
InService
InService
The complete JSON is:
{
"InstanceStates": [
{
"InstanceId": "i-0cc1e6d50ccbXXXXX",
"State": "InService",
"ReasonCode": "N/A",
"Description": "N/A"
},
{
"InstanceId": "i-0fc21ddf457eXXXXX",
"State": "InService",
"ReasonCode": "N/A",
"Description": "N/A"
}
]
}
What I've done so far is creating that one liner shell command:
export STR=$'InService\nInService'
if aws elb describe-instance-health --load-balancer-name ELB-NAME | jq -r '.[] | .[] | .State' | grep -q "$STR"; then echo 'yes'; fi
But I get "yes" as long as there is "InService" at the first command output
Is there a way I can get TRUE/YES only if I get twice "InService" as an output?
or any other way to determine that this is indeed what I got in return?
Without seeing an informative sample of the JSON it's not clear what the best solution would be, but the following meets the functional requirements as I understand them, without requiring any further post-processing:
jq -r '
def count(stream): reduce stream as $s (0; .+1);
if count(.[][] | select(.State == "InService")) > 1 then "yes" else empty end
'

Why jq does not see environment variables when run in script?

I have the following JSON file:
{
"1":
{
"media_content":"test3.xspf"
},
"2":
{
"media_content":"test3.xspf"
}
}
In the terminal, using bash as shell, I can execute the following commands:
export schedules="1"
echo $(jq '.[env.schedules]["media_content"]' json_file.json)
Which results in outputing this:
test3.xspf
So it works as expected, but when I place that jq command in a script and run it, it just returns null.
I did echo the values of schedules to make sure the value is non-null inside the script, and it is ok:
echo $schedules
But I did not manage to find the reason, why this command works when run directly in shell and does not work when run in script.
I run the script in the following ways:
bash script.sh
./script.sh
PS: yes, I did offer execute permission: chmod +x script.sh
HINT: env.schedules represents the environment variable 'schedules', and I did make sure that it is assigned in the script before calling jq.
EDIT: I am posting now a whole script, specifying the files tree.
There is one directory containing:
script.sh
json_file.json
static.json
script.sh:
export zone=$(cat static.json | jq '.["1"]');
echo "json block: "$zone
export schedules="$(echo $zone | jq '.schedules')"
echo "environment variable: "$schedules
export media_content=$(jq '.[env.schedules]["media_content"]' json_file.json)
echo "What I want to get: \"test3.xspf\""
echo "What I get: "$media_content
json_file.json:
{
"1":
{
"media_content":"test3.xspf"
},
"2":
{
"media_content":"test3.xspf"
}
}
static.json:
{
"1":
{
"x": "0",
"y": "0",
"width": "960",
"height": "540",
"schedules":"1"
}
}
If I run the script, it displays:
json block: { "x": "0", "y": "0", "width": "960", "height": "540", "schedules": "1" }
environment variable: "1"
What I want to get: "test3.xspf"
What I get: null
If I hardcode the variable:
export schedules="1"
The problem no longer occurs
The problem is simple.
It's not jq's fault.
It the unproper way the schedule's value is piped to the next command.
You have to remove the "s that surround the variable's value, add the second command that uses sed to do that:
export schedules="$(echo $zone | jq '.schedules')"
schedules=$( echo $schedules | sed s/\"//g )
Long answer
Let's see:
here schedules is a string and echo shows its value as being 1:
export schedules="1" ; echo $schedules
here even though double quotes are not mentioned:
export schedules=1 ; echo $schedules
But the result from this also generates additional "s:
export schedules=$(echo $zone | jq '.schedules')
If you print it now you will see additional "s:
echo $schedules # "1"
So just remove the "s from the value:
schedules=$( echo $schedules | sed s/\"//g )

jq: create array of object in json and insert the new object each time bash scripts executes [duplicate]

This question already has answers here:
Add new element to existing JSON array with jq
(3 answers)
Closed 3 years ago.
I want to create valid json using jq in bash.
each time when bash script will execute "Add new element to existing JSON array" and if file is empty create new file.
I am using following jq command to create my json (which is incomplete, please help me to complete it)
$jq -n -s '{service: $ARGS.named}' \
--arg transcationId $TRANSACTION_ID_METRIC '{"transcationId":"\($transcationId)"}' \
--arg name $REALPBPODDEFNAME '{"name ":"\($name )"}'\
--arg lintruntime $Cloudlintruntime '{"lintruntime":"\($lintruntime)"}' \
--arg status $EXITCODE '{"status":"\($status)"}' \
--arg buildtime $totaltime '{"buildtime":"\($buildtime)"}' >> Test.json
which is producing output like
{
"service": {
"transcationId": "12345",
"name": "sdsjkdjsk",
"lintruntime": "09",
"status": "0",
"buildtime": "9876"
}
}
{
"service": {
"transcationId": "123457",
"servicename": "sdsjkdjsk",
"lintruntime": "09",
"status": "0",
"buildtime": "9877"
}
}
but I don't want output in this format
json should be created first time like
what should be jq command for creating below jason
{
"ServiceData":{
"date":"30/1/2020",
"ServiceInfo":[
{
"transcationId":"20200129T130718Z",
"name":"MyService",
"lintruntime":"178",
"status":"0",
"buildtime":"3298"
}
]
}
}
and when I next time execute the bash script element should be added into the array like
what is the jq command for getting json in this format
{
"ServiceData":{
"date":"30/1/2020",
"ServiceInfo":[
{
"transcationId":"20200129T130718Z",
"name":"MyService",
"lintruntime":"16",
"status":"0",
"buildtime":"3256"
},
{
"transcationId":"20200129T130717Z",
"name":"MyService",
"lintruntime":"16",
"status":"0",
"buildtime":"3256"
}
]
}
}
also I want "date " , "service data" , "service info"
fields in my json which are missing in my current one
You don't give a separate filter to each --arg option; it just defines a variable which can be used in the single filter argument. You just want to add new object to your input. jq doesn't do in-place file editing, so you'll have to write to a temporary file and replace your original after the fact.
jq --arg transactionId "$TRANSACTION_ID_METRIC" \
--arg name "$REALPBPODDEFNAME" \
--arg lintruntime "$Cloudlintruntime" \
--arg status "$EXITCODE" \
--arg buildtime "$totaltime" \
'.ServiceData.ServiceInfo += [ {transactionID: $transactionId,
name: $name,
lintruntime: $lintruntime,
status: $status,
buildtime: $buildtime
}]' \
Test.json > tmp.json &&
mv tmp.json Test.json
Here's the same command, but using an array to store all the --arg options and a variable to store the filter so the command line is a little simpler. (You also don't need explicit line continuations inside an array definition.)
args=(
--arg transactionId "$TRANSACTION_ID_METRIC"
--arg name "$REALPBPODDEFNAME"
--arg lintruntime "$Cloudlintruntime"
--arg status "$EXITCODE"
--arg buildtime "$totaltime"
)
filter='.ServiceData.ServiceInfo += [
{
transactionID: $transactionId,
name: $name,
lintruntime: $lintruntime,
status: $status,
buildtime: $buildtime
}
]'
jq "${args[#]}" "$filter" Test.json > tmp.json && mv tmp.json Test.json

Bash with while, for and if works, until I add a second value

I have a script where I extract values from curl statements using jq. This script works well whenever the if-statement receives one true value, however, it breaks as soon as a second media package is found and has to be processed.
The code:
#!/bin/bash
source mh_auth.sh
rm -f times.txt
touch times.txt
#Read lines from file with mediapackage-id's that are on schedule.
while read LINE; do
curl --silent --digest -u $mh_username:$mh_password -H "X-Requested-Auth: Digest" -H "X-Opencast-Matterhorn-Authorization: true" "$mh_server/workflow/instances.json?mp=$LINE" > $LINE-curl.txt
#Format the file to make it more readable if you need to troubleshoot
/usr/bin/python -m json.tool $LINE-curl.txt > $LINE-curl-final.txt
#Test to see if this mediapackage has been published yet, and if it has, extract the necessary values for calculation
if grep -q -e 'Cleaning up"' $LINE-curl-final.txt; then
echo "Media Package found"
workflows=$( jq '.workflows.workflow[]' < $LINE-curl-final.txt )
for i in "${workflows}"
do
echo "Getting the end_time variable"
end_time=`echo ${i} | jq '.operations.operation[] | select(.description == "Cleaning up") | .completed'`
echo "Done getting end time variable"
echo "Getting ingest time variable"
ingest_time=`echo ${i} | jq '.operations.operation[] | select(.description == "Ingest") | .completed'`
echo "Done getting ingest time variable"
echo $ingest_time $end_time >> times.txt
echo last >> times.txt
done
else
echo "Media Package not published yet"
fi
rm -f $LINE-curl.txt
rm -f $LINE-curl-final.txt
done < scheduled-mediapackages.txt
A successful run yields the following:
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package not published yet
Media Package found
Getting the end_time variable
Done getting end time variable
Getting ingest time variable
Done getting ingest time variable
Whenever I add a second published media package containing exactly the same json as the first to my list, I get the following:
Media Package not published yet
Media Package not published yet
Media Package found
Getting the end_time variable
Done getting end time variable
Getting ingest time variable
Done getting ingest time variable
Media Package found
Getting the end_time variable
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
Done getting end time variable
Getting ingest time variable
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot index string with string
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
jq: error: Cannot iterate over null
Done getting ingest time variable
Any ideas how to fix this? I have been going around in circles and cannot get it to work?
I know it might seem arrogant to answer my own question, but for all those out there who has struggled like me, I wanted to add a fix. Before I get any slams for the code maybe being clumsy and "too much", just remember, I just started coding in bash, and this is working. Others might find this and with their experience be able to clean it up. For me though, this is what worked, so I stuck with it.
Thanks to Mark Setchell, I went back to the drawing board to figure out why the while loop was giving me a hard time. I figured out that some of the json return I got had more objects in it than others, so the jq command had to be adjusted to fit each command. I am only posting the relevant code for extracting the data I was looking for in my OP.
if grep -q -e 'Cleaning up"' $LINE-curl-final.txt; then
Because some media packages fail on first processing, then gets "picked up" to be re-processed, they have more than one workflow object. This if-statement searches for the succeeded one between them,and extracts the data with the adjusted jq query
get_count=$( jq -r '.[] | .totalCount' < $LINE-curl-final.txt )
if [[ "$get_count" -gt 1 ]]; then
state=$( jq -r '.workflows.workflow[] | select(.state == "SUCCEEDED" ) | .operations.operation[] | select(.description == "Cleaning up") | .state' < $LINE-curl-final.txt )
if [ "$state" = "SUCCEEDED" ];then
end_time=$( jq '.workflows.workflow[] | select(.state == "SUCCEEDED" ) | .operations.operation[] | select(.description == "Cleaning up") | .completed' < $LINE-curl-final.txt )
ingest_time=$( jq '.workflows.workflow [] | .operations.operation[] | select(.description == "Ingest") | .completed' < $LINE-curl-final.txt )
fi
else
state=$( jq -r '.workflows.workflow.operations.operation[] | select(.description == "Cleaning up") | .state' < $LINE-curl-final.txt )
if [ "$state" = "SUCCEEDED" ];then
end_time=$( jq '.workflows.workflow.operations.operation[] | select(.description == "Cleaning up") | .completed' < $LINE-curl-final.txt )
ingest_time=$( jq '.workflows.workflow.operations.operation[] | select(.description == "Ingest") | .completed' < $LINE-curl-final.txt )
fi
fi
Note the difference between
jq '.workflows.workflow.operations.operation[] | select(.description == "Cleaning up") | .completed'
and
jq '.workflows.workflow[] | select(.state == "SUCCEEDED" ) | .operations.operation[] | select(.description == "Cleaning up") | .completed'
I hope that this idea might help anybody else struggling with the same problem.

Resources