I have a program that triggers a script. The script works when I run it manually, but for some reason I get an error when the program runs it. It is supposed to trigger another command if the curl command returns true after parsing some JSON. Here is the relevant portion:
if
$(curl -s -b cookie "https://my.site.com/api" | /config/jq --arg var "test" '.Items[] | .item1[] | select( .Name== $var ) | .Valid') = "true"
then
It returns =: command not found. I tried removing the whitespaces from the = and it returns =true: command not found. I tried putting brackets around it like this:
if
[ $(curl -s -b cookie "https://my.site.com/api" | /config/jq --arg var "test" '.Items[] | .item1[] | select( .Name== $var ) | .Valid') = "true" ]
then
But that gives me an error related to the brackets. I have #!/bin/bash set at the beginning of the script and made sure there are no carriage return issues. What am I doing wrong?
Related
When I'm trying to execute script below and getting error: "curl: (3) URL using bad/illegal format or missing URL"
#!/bin/bash
stage="develop"
branch="branch_name"
getDefinition=$(curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions?api-version=5.1")
for def in $(echo "$getDefinition" | jq '.value[] | select (.path=="\\Some_path\\'$stage'") | .id'); do
getBuildInfo=$(curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1")
# echo $def
body=$(echo "${getBuildInfo}" | jq '.repository.defaultBranch = "refs/heads/release/'"${branch}"'"' | jq '.options[].inputs.branchFilters = "[\"+refs/heads/release/'"${branch}"'\"]"' | jq '.triggers[].branchFilters[] = "+refs/heads/release/'"${branch}"'"')
echo ${body} > data.json
done
It happens when I'm trying to pass variable ${def} into a line:
curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1"
But when I declare an array, curl works as expected.
Example:
declare -a def
def=(1 2 3 4)
curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1"
Could you please suggest how can I pass variable into URL properly?
Do you need to call curl for 4 times? If so.
for def in 1 2 3 4; do curl -u user#example.com:password -X GET "https://dev.azure.com/organization/project/_apis/build/definitions/${def}\?api-version=5.1"; done
Set IFS=$' \t\r\n' at the top of your script.
IFS is the Interactive Field Separator.
In UNIX, IFS is space, tab, newline, or $' \t\n', but on Windows this needs to be $' \t\r\n'.
The ^M character is \r.
I want to send a Message into a Mattermost channel with the help of a ShellScript/WebHook/cURL. The following code is the function to send the Message.
function matterSend() {
ENDPOINT=https://url.to.Mattermost/WebhookID
USERNAME="${USER}"
PAYLOAD=$(cat <<'EOF'
'payload={
"username" : "${USERNAME}",
"channel" : "TestChannel",
"text" : "#### Test to \n
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
}'
EOF
)
echo "CURL: curl -i -X POST -d ${PAYLOAD} ${ENDPOINT}"
curl -i -X POST -d "${PAYLOAD}" "${ENDPOINT}"
}
As you can see, when I ECHO the command I get:
curl -i -X POST -d 'payload={
"username" : "TestUser",
"channel" : "TestChannel",
"text" : "#### Test to \n
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
}' https://url.to.Mattermost/WebhookID
If I paste that code directly into the terminal and execute it, it works. But when I run the script with the help of a Jenkins-Job I get the Error:
Unable to parse incoming data","message":"Unable to parse incoming
data".
Why is it not working?
Without knowledge of the API you are connecting to, I would guess that you need
# Drop function keyword, indent body
matterSend() {
# Lowercase variable names; declare them local
local endpoint=https://url.to.Mattermost/WebhookID
local username=$USER
# Pro tip: don't use a variable for the payload if it's effectively static
payload=$(cat <<-__EOF
payload={
"username" : "$username",
"channel" : "TestChannel",
"text" : "#### Test to \\n| TestR | TestS | New Mode |\\n|:-----------|:-----------:|-----------------------------------------------:|\\n| ${2} | ${3} | ${1} :white_check_mark: |\\n"
}
__EOF
)
echo "CURL: curl -i -X POST -d $payload $endpoint"
curl -i -X POST -d "$payload" "$endpoint"
}
Replacing the newlines inside the "text" element with \n (and doubling the backslash because the here document is now being interpreted by the shell when it's assigned) is mildly speculative; perhaps remove the remaining newlines, too. The real beef is removing the misplaced literal single quotes around the payload.
Maybe also explore printf for formatting fixed-width tabular text.
The here document's <<-__EOF uses the unquoted separator __EOF and the dash before it says to remove any tabs from the beginning of each line. Needless to say, the indentation on those lines consists of a literal tab character.
Generating JSON (or XML, or other structured formats) via string concatenation leads to pain and suffering. Instead, use a tool that actually understands the format.
Using a compliant generator such as jq means you no longer need to be responsible for putting \ns in the data (as multi-character strings), changing "s in text to \", or any of the other otherwise-necessary munging.
matterSend() {
# Lowercase variable names; declare them local
local endpoint=https://url.to.Mattermost/WebhookID
local username=$USER
local text="#### Test to
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
payload=$(jq --arg username "$username" \
--arg channel "TestChannel" \
--arg text "$text" \
'{"username": $username, "channel": $channel, "text": $text}')
# Advice: use "set -x" if you want to trace the commands your script would run.
# ...or at least printf %q, as below; avoids misleading output from echo.
# printf '%q ' curl -i -X POST -d "$payload" "$endpoint" >&2; echo >&2
curl -i -X POST -d "$payload" "$endpoint"
}
I'm trying to create a ohmyzsh function for Salesforce's DX CLI based on Wade Wegner's guide here. In order to get the value I want I need to expand how he is using JQ which I've never heard of before. I get the premise for this use case but I'm struggling with one abstraction point (within the aliasConfig json). Here's my script so far
get_sfdx_defaultusername() {
config="$(cat .sfdx/sfdx-config.json 2> /dev/null)";
globalConfig="$(cat ~/.sfdx/sfdx-config.json)";
aliasConfig="$(cat ~/.sfdx/alias.json)";
defaultusername="$(echo ${config} | jq -r .defaultusername)"
defaultusernamealias="NEED HELP HERE"
globaldefaultusername="$(echo ${globalConfig} | jq -r .defaultusername)"
if [ ! $defaultusernamealias = "null" ]
then
echoString=$echoString$defaultusernamealias"$txtylw (alias)"
elif [ ! $defaultusername = "null" ]
then
echoString=$echoString$defaultusername"$txtylw (local)"
else
echoString=$echoString$globaldefaultusername"$txtylw (global)"
fi
echo $echoString"\n"
}
The alias.json looks like this:
{
"orgs": {
"HubOrg": "myemail#domain.com",
"my-scrath-org": "test-jdj1iflkor4k#mydomain.net"
}
}
Using the ${defaultusername} I know the value in this case to be "test-jdj1iflkor4k#mydomain.net", therefore I need it to set the value of defaultusernamealias to "my-scrath-org"
NOTE: The closest answer I found was this, but unfortunately I still couldn't get what I needed with it.
Congratulations on figuring out how to use to_entries.
One small suggestion is to avoid using shell interpolation to "construct" the jq program. A much better way to achieve the desired goal is to pass in the relevant values on the command-line. In your case, the following would be appropriate:
$ jq --arg username "$defaultusername" '
.orgs | to_entries[] | select(.value == $username ).key'
Another small point is to avoid using echo to send JSON to STDIN. There are several possibilities, including these patterns:
if you are using bash: jq .... <<< "$JSON"
use printf "%s" "$JSON" | jq ...
jq -n --argjson JSON "$JSON" '$JSON | ...'
In your case, the last of these alternatives would look like this:
$ jq --arg username "$defaultusername" --argjson JSON "$aliasConfig" '
$JSON
| .orgs | to_entries[] | select(.value == $username ).key'
I think I got it figured out here:
get_sfdx_defaultusername() {
config="$(cat .sfdx/sfdx-config.json 2> /dev/null)";
globalConfig="$(cat ~/.sfdx/sfdx-config.json)";
aliasConfig="$(cat ~/.sfdx/alias.json)";
defaultusername="$(echo ${config} | jq -r .defaultusername)"
defaultusernamealias="$(echo ${aliasConfig} | jq -r '.orgs | to_entries[] | select(.value =="'$defaultusername'").key' )"
globaldefaultusername="$(echo ${globalConfig} | jq -r .defaultusername)"
if [ ! $defaultusernamealias = "null" ]
then
echoString=$echoString$defaultusernamealias"$txtylw (alias)"
elif [ ! $defaultusername = "null" ]
then
echoString=$echoString$defaultusername"$txtylw (local)"
else
echoString=$echoString$globaldefaultusername"$txtylw (global)"
fi
echo $echoString"\n"
}
This allows me to show my current defaultusername org like so:
In case anyone is interested in using this or contributing to it, I published a github repo here
Given the following input:
J='{"a":1,"b":10,"c":100}
{"a":2,"b":20,"c":200}
{"a":3,"b":30,"c":300}'
The command
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{a,b}'
produces
{"a":1,"b":10}
{"a":2,"b":20}
{"a":3,"b":30}
but this command produces unexpected results:
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{$P1}'
{"P1":"a,b"}
{"P1":"a,b"}
{"P1":"a,b"}
How does one get jq to treat an arg string literally?
Using tostring gives an error
SELECT='a,b'; echo $J | jq -c -s --arg P1 $SELECT '.[]|{$P1|tostring}'
jq: error: syntax error, unexpected '|', expecting '}' (Unix shell quoting
issues?) at <top-level>, line 1:
.[]|{$SELECT|tostring}
jq: 1 compile error
SELECT needs to be a variable and not hardcoded in the script.
SELECT needs to be a variable and not hardcoded in the script.
Assuming you want to avoid the risks of "code injection" and that you want the shell variable SELECT to be a simple string such as "a,b", then consider this reduce-free solution along the lines you were attempting:
J='{"a":1,"b":10,"c":100}'
SELECT='a,b'
echo "$J" |
jq -c --arg P1 "$SELECT" '
. as $in | $P1 | split(",") | map( {(.): $in[.]} ) | add'
Output:
{"a":1,"b":10}
If you really want your data to be parsed as syntax...
This is not an appropriate use case for --arg. Instead, substitute into the code:
select='a,b'; jq -c -s '.[]|{'"$select"'}' <<<"$j"
Note that this has all the usual caveats of code injection: If the input is uncontrolled, the output (or other behavior of the script, particularly if jq gains more capable I/O features in the future) should be considered likewise.
If you want to split the literal string into a list of keys...
Here, we take your select_str (of the form a,b), and generate a map: {'a': 'a', 'b': 'b'}; then, we can break each data item into entries, select only the items in the map, and there's our output.
jq --arg select_str "$select" '
($select_str
| split(",")
| reduce .[] as $item ({}; .[$item]=$item)) as $select_map
| with_entries(select($select_map[.key]))' <<<"$j"
I have problem with assigning curl as variable and assign curl's output to variable:
#get results url, format json
URL=$(curl https://api.apifier.com/xy)
#jq is a cli json interpreter
#resultUrl contains the final URL which we want download
OK= "$URL" | jq '.resultsUrl'
#api probably is running
sleep 5
curl "$OK"
Maybe it is trivial, but I don't know where is the problem.
My guess is:
jq '.resultsUrl'
outputs the field resultsUrl with quotes, so curl does not process it correctly. Furthermore, $URL | ... does not work, you would have to use echo or curl directly.
Try
OK=$(curl -s https://api.apifier.com/v1/xHbBnrZ9rxF4CdKjo/crawlers/Example_Alcatraz_Cruises/execute?token=nJ9ohCHZPaJRFEb7nFqtzm76u | jq -r '.resultsUrl')
curl -s "$OK"
which results for me in
[{ "id": 2, "url": "https://www.alcatrazcruises.com/SearchEventDaySpan.aspx?date=02-25-2016&selected=", "loadedUrl": "https://www.alcatrazcruises.com/SearchEventDaySpan.aspx?date=02-25-2016&selected=", "requestedAt": "2016-02-25T23:24:52.611Z", "loadingStartedAt": "2016-02-25T23:24:54.663Z", "loadingFinishedAt": "2016-02-25T23:24:55.642Z", "loadErrorCode": null, "pageFunctionStartedAt": "2016-02-25T23:24:55.839Z", "pageFunctionFinishedAt": "2016-02-25T23:24:55.841Z", "uniqueKey": "https://www.alcatrazcruises.com/SearchEventDaySpan.aspx?date=02-25-2016&selected=", "type": "UserEnqueued", ...
This should be what you expect.
However, sometimes the first API call yields an error:
{
"type": "ALREADY_RUNNING",
"message": "The act is already running and concurrent execution is not allowed"
}
so resultsURL will be null, you will have to handle this error case.
Your line
OK= "$URL" | jq '.resultsURL'
sets the environment variable OK to an empty string, then tries to execute "$URL" as a command and pipe its output to jq. If you want to setOK to the result of a command, you have to use $OK=(...), just like you did when setting URL. The correct syntax is:
OK=$(echo "$URL" | jq '.resultsURL')
And to remove the quotes from the output of .jq, you can do:
OK=$(echo "$URL" | jq '.resultsURL' | tr -d '"')