I have problem with assigning curl as variable and assign curl's output to variable:
#get results url, format json
URL=$(curl https://api.apifier.com/xy)
#jq is a cli json interpreter
#resultUrl contains the final URL which we want download
OK= "$URL" | jq '.resultsUrl'
#api probably is running
sleep 5
curl "$OK"
Maybe it is trivial, but I don't know where is the problem.
My guess is:
jq '.resultsUrl'
outputs the field resultsUrl with quotes, so curl does not process it correctly. Furthermore, $URL | ... does not work, you would have to use echo or curl directly.
Try
OK=$(curl -s https://api.apifier.com/v1/xHbBnrZ9rxF4CdKjo/crawlers/Example_Alcatraz_Cruises/execute?token=nJ9ohCHZPaJRFEb7nFqtzm76u | jq -r '.resultsUrl')
curl -s "$OK"
which results for me in
[{ "id": 2, "url": "https://www.alcatrazcruises.com/SearchEventDaySpan.aspx?date=02-25-2016&selected=", "loadedUrl": "https://www.alcatrazcruises.com/SearchEventDaySpan.aspx?date=02-25-2016&selected=", "requestedAt": "2016-02-25T23:24:52.611Z", "loadingStartedAt": "2016-02-25T23:24:54.663Z", "loadingFinishedAt": "2016-02-25T23:24:55.642Z", "loadErrorCode": null, "pageFunctionStartedAt": "2016-02-25T23:24:55.839Z", "pageFunctionFinishedAt": "2016-02-25T23:24:55.841Z", "uniqueKey": "https://www.alcatrazcruises.com/SearchEventDaySpan.aspx?date=02-25-2016&selected=", "type": "UserEnqueued", ...
This should be what you expect.
However, sometimes the first API call yields an error:
{
"type": "ALREADY_RUNNING",
"message": "The act is already running and concurrent execution is not allowed"
}
so resultsURL will be null, you will have to handle this error case.
Your line
OK= "$URL" | jq '.resultsURL'
sets the environment variable OK to an empty string, then tries to execute "$URL" as a command and pipe its output to jq. If you want to setOK to the result of a command, you have to use $OK=(...), just like you did when setting URL. The correct syntax is:
OK=$(echo "$URL" | jq '.resultsURL')
And to remove the quotes from the output of .jq, you can do:
OK=$(echo "$URL" | jq '.resultsURL' | tr -d '"')
Related
In my bash script, when I run the following jq against my curl result:
curl -u someKey:someSecret someURL 2>/dev/null | jq -r '.schema' | jq -r -c '.fields'
I get back a JSON array as follows:
[{"name":"id","type":"int","doc":"Documentation for the id field."},{"name":"test_string","type":"string","doc":"Documentation for the test_string field"}]
My goal is to do a call with jq applied to return the following (given the example above):
{"id":1234567890,"test_string":"xxxxxxxxxx"}
NB: I am trying to automatically generate templated values that match the "schema" JSON shown above.
So just to clarify, that is:
all array objects (there could be more than 2 shown above) returned in a single comma-delimited row
doc fields are ignored
the values for "name" (including their surrounding double-quotes) are concatenated with either:
:1234567890 ...when the "type" for that object is "int"
":xxxxxxxxxx" ...when the "type" for that object is "string"
NB: these will be the only types we ever get for now
Can someone show me how I can expand upon my initial jq to return this?
NB: I tried working down the following path but am failing beyond this...
curl -u someKey:someSecret someURL 2>/dev/null | jq -r '.schema' | jq -r -c '.fields' | "\(.name):xxxxxxxxxxx"'
If it's not possible in pure JQ (my preference) I'm also happy for a solution that mixes in a bit of sed/awk magic :)
Cheers,
Stan
Given the JSON shown, you could add the following to your pipeline:
jq -c 'map({(.name): (if .type == "int" then 1234567890 else "xxxxxxxxxx" end)})|add'
With that JSON, the output would be:
{"id":1234567890,"test_string":"xxxxxxxxxx"}
However, it would be far better if you combined the three calls to jq into one.
I have created a simple script to store response from a third party API
The request is like this..
https://externalservice.io/orders?key=password&records=50&offset=0
The response is as follows:
{
"results": [
{
"engagement": {
"id": 29090716,
"portalId": 62515,
"active": true,
"createdAt": 1444223400781,
"lastUpdated": 1444223400781,
"createdBy": 215482,
"modifiedBy": 215482,
"ownerId": 70,
"type": "NOTE",
"timestamp": 1444223400781
},
},
],
"hasMore": true,
"offset": 4623406
}
If there is a hasMore attribute, I need to read the offset value to get the next set of records.
Right now I've created a script that simply loops over the estimated number of records (I believed there is) and thought incrementing the offset would work but this is not the case as the offset is not incremental.
#!/bin/bash
for i in {1..100}; do
curl -s "https://externalservice.io/orders?key=password&records=50&offset=$i" >>outfile.txt 2>&1
done
Can someone explain how I can read continue the script reading the offset value until hasMore=false?
You can read a value from a json using the jq utility:
$ jq -r ".hasMore" outfile
true
Here is what you could use:
more="true"
offset=0
while [ $more = "true" ]; do
echo $offset
response=$(curl -s "https://example.com/orders?offset=$offset")
more=$(echo $response | tr '\r\n' ' ' | jq -r ".hasMore")
offset=$(echo $response | tr '\r\n' ' ' | jq -r ".offset")
done
You can use jq utility to extract specific attribute from you json response:
$ jq -r ".hasMore" outfile
true
jq expects perfectly valid json input, otherwise it will print error.
Most common mistake is to echo json stored in variable. A mistake, because echo will interpret all escaped newlines in your json and will send unescaped newlines within values causing jq to throw parse error: Invalid string: control characters from U+0000 through U+001F must be escaped at line message.
To avoid modification to json (and avoiding an error) we need to use printf instead of echo.
Here is the solution without breaking json content:
more="true"
offset=0
while [ $more = "true" ]; do
echo $offset
response=$(curl -s "https://example.com/orders?offset=$offset")
more=$(printf "%s\n" "$response" | jq -r ".hasMore")
offset=$(printf "%s\n" "$response" | jq -r ".offset")
done
I want to send a Message into a Mattermost channel with the help of a ShellScript/WebHook/cURL. The following code is the function to send the Message.
function matterSend() {
ENDPOINT=https://url.to.Mattermost/WebhookID
USERNAME="${USER}"
PAYLOAD=$(cat <<'EOF'
'payload={
"username" : "${USERNAME}",
"channel" : "TestChannel",
"text" : "#### Test to \n
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
}'
EOF
)
echo "CURL: curl -i -X POST -d ${PAYLOAD} ${ENDPOINT}"
curl -i -X POST -d "${PAYLOAD}" "${ENDPOINT}"
}
As you can see, when I ECHO the command I get:
curl -i -X POST -d 'payload={
"username" : "TestUser",
"channel" : "TestChannel",
"text" : "#### Test to \n
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
}' https://url.to.Mattermost/WebhookID
If I paste that code directly into the terminal and execute it, it works. But when I run the script with the help of a Jenkins-Job I get the Error:
Unable to parse incoming data","message":"Unable to parse incoming
data".
Why is it not working?
Without knowledge of the API you are connecting to, I would guess that you need
# Drop function keyword, indent body
matterSend() {
# Lowercase variable names; declare them local
local endpoint=https://url.to.Mattermost/WebhookID
local username=$USER
# Pro tip: don't use a variable for the payload if it's effectively static
payload=$(cat <<-__EOF
payload={
"username" : "$username",
"channel" : "TestChannel",
"text" : "#### Test to \\n| TestR | TestS | New Mode |\\n|:-----------|:-----------:|-----------------------------------------------:|\\n| ${2} | ${3} | ${1} :white_check_mark: |\\n"
}
__EOF
)
echo "CURL: curl -i -X POST -d $payload $endpoint"
curl -i -X POST -d "$payload" "$endpoint"
}
Replacing the newlines inside the "text" element with \n (and doubling the backslash because the here document is now being interpreted by the shell when it's assigned) is mildly speculative; perhaps remove the remaining newlines, too. The real beef is removing the misplaced literal single quotes around the payload.
Maybe also explore printf for formatting fixed-width tabular text.
The here document's <<-__EOF uses the unquoted separator __EOF and the dash before it says to remove any tabs from the beginning of each line. Needless to say, the indentation on those lines consists of a literal tab character.
Generating JSON (or XML, or other structured formats) via string concatenation leads to pain and suffering. Instead, use a tool that actually understands the format.
Using a compliant generator such as jq means you no longer need to be responsible for putting \ns in the data (as multi-character strings), changing "s in text to \", or any of the other otherwise-necessary munging.
matterSend() {
# Lowercase variable names; declare them local
local endpoint=https://url.to.Mattermost/WebhookID
local username=$USER
local text="#### Test to
| TestR | TestS | New Mode |
|:-----------|:-----------:|-----------------------------------------------:|
| ${2} | ${3} | ${1} :white_check_mark: |
"
payload=$(jq --arg username "$username" \
--arg channel "TestChannel" \
--arg text "$text" \
'{"username": $username, "channel": $channel, "text": $text}')
# Advice: use "set -x" if you want to trace the commands your script would run.
# ...or at least printf %q, as below; avoids misleading output from echo.
# printf '%q ' curl -i -X POST -d "$payload" "$endpoint" >&2; echo >&2
curl -i -X POST -d "$payload" "$endpoint"
}
I'm having what I suspect is a quoting error in a bash script.
SECRET_VALUE_CMD="curl -s -L -H \"X-Vault-Token: $VAULT_TOKEN\" -X GET \"https://$VAULT_ADDR/v1/secret/$secret_path\""
SECRET_VALUE_RESPONSE=$(eval $SECRET_VALUE_CMD)
SECRET_VALUE=$(echo "$SECRET_VALUE_RESPONSE" | jq --raw-output '.data.value')
When I execute this in my script, I get the following to stderr:
parse error: Expected separator between values at line 1, column 63
and $SECRET_VALUE is blank.
An example of $SECRET_VALUE_RESPONSE is:
{"request_id":"XXXX-YYYY..,"lease_id":"","renewable":false,"lease_duration":nnnnnn,"data":{"value":"secret-value"},"wrap_info":null,"warnings":null,"auth":null}
I've tried adding escaped quotes around the parameters to eval and echo, but can't seem to find a working combination. Any help would be greatly appreciated!
Don't use eval. You could create a function to execute curl, for example:
get_secret_value() {
curl -s -L -H "X-Vault-Token: $VAULT_TOKEN" -X GET "https://$VAULT_ADDR/v1/secret/$secret_path"
}
secret_value=$(get_secret_value | jq --raw-output '.data.value')
I have a program that triggers a script. The script works when I run it manually, but for some reason I get an error when the program runs it. It is supposed to trigger another command if the curl command returns true after parsing some JSON. Here is the relevant portion:
if
$(curl -s -b cookie "https://my.site.com/api" | /config/jq --arg var "test" '.Items[] | .item1[] | select( .Name== $var ) | .Valid') = "true"
then
It returns =: command not found. I tried removing the whitespaces from the = and it returns =true: command not found. I tried putting brackets around it like this:
if
[ $(curl -s -b cookie "https://my.site.com/api" | /config/jq --arg var "test" '.Items[] | .item1[] | select( .Name== $var ) | .Valid') = "true" ]
then
But that gives me an error related to the brackets. I have #!/bin/bash set at the beginning of the script and made sure there are no carriage return issues. What am I doing wrong?