echo $(curl -s -u user:pwd "http://site/file.json" | jq -e -r '.data[]? | select(.state == "Active") | if . == null then "Installing" elif . == "Active" then "Active" else "Installing" end')
Following is the error:
parse error: Invalid numeric literal at line 2, column 0
In a scenario when file.json doesn't exist 404 page is returned and jq is throwing a parsing error. In such case I want to return string "Installing". Tried many things but nothing is working out, please help.
Write the response into a variable:
# -f makes curl return an error in case of HTTP error.
# Check "man curl" on how reliable this is.
response="$(curl -f ...)"
if [ $? -ne 0 ] ; then
echo "Installing"
else
jq FILTER <<< "${response}"
fi
Related
I have a function that checks for duplicate values held within objects inside a json file. When duplicates are found the function returns something like this:
{
"Basket1": [
Apple,
Orange
],
"Basket2": [
Apple,
Orange
]
}
If no duplicates are found then it returns am empty list:
{}
Currently I am using -s in bash like such to check if the there are dups found within the output:
<"$baskets" jq -L $HOME 'check_dups' > "$dups"
if [[ ! -s "$dups" ]];
then
echo -e "${RED}[Error]${NC} Duplicates found! Please review duplicates below" >&2
echo "$(cat "$dups" | jq '.')"
else
echo -e "${GREEN}[SUCCESS]${NC} No duplicates found" >&2
fi
However empty object returned if no dups are found will cause the -s file check in bash to succeed regardless. What would be the best way using jq or bash to check whether the output of this function is an empty object or not?
You can use error() to cause a failure if your input is identical to {}, and proceed otherwise.
jq '
if . == {} then error("empty document found") else . end
| ...rest of your processing here...
'
As a quick example:
<<<"{}" jq 'if . == {} then error("empty document found") else . end | {"output": (.)}'
...emits a nonzero exit status even without jq -e.
(Addressing a concern #Thomas brought up, error() has a different exit status than an input parsing error; the former is 4, the latter is 5, while compile errors are 3; so should there be a need to distinguish them it's entirely possible).
You can compare to the empty object {}, so . == {} or . != {} produce a boolean, which should do what you want.
Furthermore, you could use jq's -e option which sets the exit code based on the result, for integration into the shell:
<"$baskets" jq -L $HOME 'check_dups' > "$dups"
if jq -e '. == {}' <"$dups" >/dev/null
then ...
else ...
fi
You could stick with your approach using the shell's -s test by arranging for your jq command to emit nothing instead of {}. This could be done using the following at the end of your jq program:
if . == {} then empty else . end
I have a function that checks for duplicate values held within objects inside a json file. When duplicates are found the function returns something like this:
{
"Basket1": [
Apple,
Orange
],
"Basket2": [
Apple,
Orange
]
}
If no duplicates are found then it returns am empty list:
{}
Currently I am using -s in bash like such to check if the there are dups found within the output:
<"$baskets" jq -L $HOME 'check_dups' > "$dups"
if [[ ! -s "$dups" ]];
then
echo -e "${RED}[Error]${NC} Duplicates found! Please review duplicates below" >&2
echo "$(cat "$dups" | jq '.')"
else
echo -e "${GREEN}[SUCCESS]${NC} No duplicates found" >&2
fi
However empty object returned if no dups are found will cause the -s file check in bash to succeed regardless. What would be the best way using jq or bash to check whether the output of this function is an empty object or not?
You can use error() to cause a failure if your input is identical to {}, and proceed otherwise.
jq '
if . == {} then error("empty document found") else . end
| ...rest of your processing here...
'
As a quick example:
<<<"{}" jq 'if . == {} then error("empty document found") else . end | {"output": (.)}'
...emits a nonzero exit status even without jq -e.
(Addressing a concern #Thomas brought up, error() has a different exit status than an input parsing error; the former is 4, the latter is 5, while compile errors are 3; so should there be a need to distinguish them it's entirely possible).
You can compare to the empty object {}, so . == {} or . != {} produce a boolean, which should do what you want.
Furthermore, you could use jq's -e option which sets the exit code based on the result, for integration into the shell:
<"$baskets" jq -L $HOME 'check_dups' > "$dups"
if jq -e '. == {}' <"$dups" >/dev/null
then ...
else ...
fi
You could stick with your approach using the shell's -s test by arranging for your jq command to emit nothing instead of {}. This could be done using the following at the end of your jq program:
if . == {} then empty else . end
I have the following BASH code:
response=$( curl -Ls $endpoint )
if [ -n "$response" ]; then # nonempty
echo "$response" | jq .
fi
The problem is that sometimes the response can be non-empty but not JSON (if it's not a 200).
Is it possible to pipe the output through jq ONLY if it is valid JSON?
The following works:
echo $x | jq . 2>/dev/null || echo $x
Test:
> x='{"foo":123}'; echo $x | jq . 2>/dev/null || echo "Invalid: $x"
{
"foo": 123
}
> x='}'; echo $x | jq . 2>/dev/null || echo "Invalid: $x"
Invalid: }
However, I don't feel comfortable with it.
If you want to test the response type before submitting it to jq, it is possible if you test the Content-Type header from the server's response.
So you want curl to send you the full response headers and body with curl -i.
Here is an implementation of it:
#!/usr/bin/env sh
endpoint='https://worldtimeapi.org/api/timezone/Europe/Paris.json'
# Headers and body are delimited by an empty line, with CRLF as the line ending.
# See: RFC7230 HTTP/1.1 Message Syntax and Routing / section 3: Message Format
# https://tools.ietf.org/html/rfc7230#section-3
crlf="$(printf '\r\n_')" # add trailing _ to prevent trailing newline trim
crlf="${crlf%_}" # remove trailing _
http_delim="$crlf$crlf" # RFC7230 section 3
full_http_response="$(curl --silent --include --url "$endpoint")"
http_headers="${full_http_response%$http_delim*}"
http_body="${full_http_response#*$http_delim}"
case $http_headers in
'HTTP/1.1 200 OK'*'Content-Type: application/json'*)
# Yes, response body is JSON, so process it with jq.
jq -n "$http_body"
;;
esac
The following works:
echo $x | jq . 2>/dev/null || echo $x
Except for the use of echo here, this is actually a good approach - it has the advantages of both simplicity and efficiency. It is better than using the -e option naively as the return codes produced by -e are more complex.
In other words, there is much to be said for:
printf "%s" "$x" | jq . 2> /dev/null || printf "%s\n" "$x"
Efficiency
The argument for efficiency is as follows:
If $x holds valid JSON, then there is no overhead.
If $x is invalid as JSON, jq will quickly fail; in this case also, the overhead of calling jq will almost surely be no worse or not much worse than checking the Content-Type.
Warning
The official documentation for the return codes produced by jq when invoked without the -e option is not strictly correct, as illustrated by:
$ jq empty <<< 'foo bat' 2> /dev/null ; echo $?
4
This works:
response=$( curl -Ls -H 'Cache-Control: max-age=0' $endpoint )
if [ -n "$response" ]; then # nonempty
echo "Got server response"
# https://stackoverflow.com/questions/46954692/check-if-string-is-a-valid-json-with-jq
if jq --exit-status type >/dev/null 2>&1 <<<"$response"; then
# Parsed JSON successfully and got something other than false/null
echo "$response" | jq .
echo "... after $i seconds"
return 0
else
echo "Response is not valid JSON"
echo "$response"
return 1
fi
fi
I have a simple script that accepts 2 arguments, a URL and a log file location. Theoretically, it should capture the header status code from the curl command and if it is a 404, then append the URL to the log file. Any idea where it is failing?
#!/bin/bash
CMP='HTTP/1.1 404 Not Found' # This is the 404 Pattern
OPT=`curl --config /var/www/html/curl.cnf -s -D - "$1" -o /dev/null | grep 404` # Status Response
if [ $OPT = $CMP ]
then
echo "$1" >> "$2" # Append URL to File
fi
Your test is assigning the value of $CMP to $OPT, not comparing for equality. Try the following simpler method, which checks the return code of the grep command rather than looking for the comparison string in its output:
#!/bin/bash
CMP='HTTP/1.1 404 Not Found'
if $(curl -s -I "$1" | grep "$CMP" >/dev/null 2>&1); then
echo "$1" >> "$2"
fi
I am getting a 400 Bad Request response header when I use bash to fetch a web page but only when I call my function inside another function?
get_download_page() {
[ $# -eq 1 ] || die "get_location: 1 argument expected, recieved $#"
page="$(get_page $1)/download"
echo "http://www.curse.com$(curl --silent -I $page | awk 'NR==4 { print; exit }' | cut -d\ -f2)"
}
get_page() {
[ $# -eq 1 ] || die "get_page: 1 argument expected, recieved $#"
echo "www.curse.com/addons/wow/$1"
}
get_file() {
[ $# -eq 1 ] || die "get_file: 1 argument expected, recieved $#"
echo $(curl -s $1 | sed -rn 's/.*data-href="([^"]+)".*/\1/p')
}
fetch_addon() {
[ $# -eq 1 ] || die "fetch_addon: 1 argument expected, recieved $#"
download=$(get_download_page $1)
file=$(get_file $download)
echo $file
}
I.e.
Calling fetch_addon "bagnon" I get a bad request header.
But if I do: get_file "http://www.curse.com/addons/wow/bagnon/704176"
http://addons.curse.cursecdn.com/files/704/176/Bagnon_5.3.zip
Which is what I expect, even though get_download_page "bagnon" returns the same link that I am passing to get_file?
As shown by doing:
get_download_page "bagnon"
echo "http://www.curse.com/addons/wow/bagnon/704176"
Output:
http://www.curse.com/addons/wow/bagnon/704176
http://www.curse.com/addons/wow/bagnon/704176
After further investigation it seems that the two strings aren't exactly equal.
When I do
echo $(cmp -bl <(echo "$download") <(echo "http://www.curse.com/addons/wow/bagnon/704176"))
I get this output:
46 15 ^M 12 ^J
Ie the last character of the generated download link is ^M and the last character of the manually found download link is ^J
For some reason the header curl was returning had mac file endings so that caused the link it returned to end in ^M, when passing that back to curl it freaked out got the bad request header.
I fixed it by piping the link into
sed -r 's/^M/^J/g'
NB:
I couldn't just type "^M" I had to press Ctrl+V and Ctrl+M/J to get the correct character.