I have the following BASH code:
response=$( curl -Ls $endpoint )
if [ -n "$response" ]; then # nonempty
echo "$response" | jq .
fi
The problem is that sometimes the response can be non-empty but not JSON (if it's not a 200).
Is it possible to pipe the output through jq ONLY if it is valid JSON?
The following works:
echo $x | jq . 2>/dev/null || echo $x
Test:
> x='{"foo":123}'; echo $x | jq . 2>/dev/null || echo "Invalid: $x"
{
"foo": 123
}
> x='}'; echo $x | jq . 2>/dev/null || echo "Invalid: $x"
Invalid: }
However, I don't feel comfortable with it.
If you want to test the response type before submitting it to jq, it is possible if you test the Content-Type header from the server's response.
So you want curl to send you the full response headers and body with curl -i.
Here is an implementation of it:
#!/usr/bin/env sh
endpoint='https://worldtimeapi.org/api/timezone/Europe/Paris.json'
# Headers and body are delimited by an empty line, with CRLF as the line ending.
# See: RFC7230 HTTP/1.1 Message Syntax and Routing / section 3: Message Format
# https://tools.ietf.org/html/rfc7230#section-3
crlf="$(printf '\r\n_')" # add trailing _ to prevent trailing newline trim
crlf="${crlf%_}" # remove trailing _
http_delim="$crlf$crlf" # RFC7230 section 3
full_http_response="$(curl --silent --include --url "$endpoint")"
http_headers="${full_http_response%$http_delim*}"
http_body="${full_http_response#*$http_delim}"
case $http_headers in
'HTTP/1.1 200 OK'*'Content-Type: application/json'*)
# Yes, response body is JSON, so process it with jq.
jq -n "$http_body"
;;
esac
The following works:
echo $x | jq . 2>/dev/null || echo $x
Except for the use of echo here, this is actually a good approach - it has the advantages of both simplicity and efficiency. It is better than using the -e option naively as the return codes produced by -e are more complex.
In other words, there is much to be said for:
printf "%s" "$x" | jq . 2> /dev/null || printf "%s\n" "$x"
Efficiency
The argument for efficiency is as follows:
If $x holds valid JSON, then there is no overhead.
If $x is invalid as JSON, jq will quickly fail; in this case also, the overhead of calling jq will almost surely be no worse or not much worse than checking the Content-Type.
Warning
The official documentation for the return codes produced by jq when invoked without the -e option is not strictly correct, as illustrated by:
$ jq empty <<< 'foo bat' 2> /dev/null ; echo $?
4
This works:
response=$( curl -Ls -H 'Cache-Control: max-age=0' $endpoint )
if [ -n "$response" ]; then # nonempty
echo "Got server response"
# https://stackoverflow.com/questions/46954692/check-if-string-is-a-valid-json-with-jq
if jq --exit-status type >/dev/null 2>&1 <<<"$response"; then
# Parsed JSON successfully and got something other than false/null
echo "$response" | jq .
echo "... after $i seconds"
return 0
else
echo "Response is not valid JSON"
echo "$response"
return 1
fi
fi
Related
I'm creating a tool to parse an input file to re-construct a mailx command (server that creates the data for mailx cannot send emails, so I need to store data into a file so another server can rebuild the command and send it). I could output the whole command to a file and execute the file on the other server, but that's hardly secure / safe.... anyone could intercept the file and insert malicious stuff that would be run as root - this parsing tool is checking every minute for any files to parse and email using a systemd timer and service.
I have created the file, using 'markers / separators' with this format:
-------MESSAGE START-------
Email Body Text
Goes Here
-------MESSAGE END-------
-------SUBJECT START-------
Email Subject
-------SUBJECT END-------
-------ATTACHEMENT START-------
path to file to attach if supplied
-------ATTACHEMENT END-------
-------S OPTS START-------
list of mailx '-S' options eg from=EMAILNAME <email#b.c> or sendwait etc each one on a new line
-------S OPTS END-------
-------EMAIL LIST START-------
string of recipient emails comma separated eg. email1,email2,email3 etc..
-------EMAIL LIST END-------
And I have a program to parse this file and rebuild, and run the mailx command:
#!/bin/bash
## Using systemd logging to journal for this as its now being called as part of a service
## See: https://serverfault.com/questions/573946/how-can-i-send-a-message-to-the-systemd-journal-froma-the-command-line (kkm answer)
start_time="$(date +[%c])"
exec 4>&2 2> >(while read -r REPLY; do printf >&4 '<3>%s\n' "$REPLY"; done)
echo >&4 "<5>$start_time -- Started gpfs_flag_email.sh"
trap_exit(){
exec >2&
}
trap 'trap_exit' EXIT
email_flag_path="<PATH TO LOCATION>/email_flags/"
mailx_message_start="-------MESSAGE START-------"
mailx_message_end="-------MESSAGE END-------"
mailx_subject_start="-------SUBJECT START-------"
mailx_subject_end="-------SUBJECT END-------"
mailx_attachement_start="-------ATTACHEMENT START-------"
mailx_attachement_end="-------ATTACHEMENT END-------"
mailx_s_opts_start="-------S OPTS START-------"
mailx_s_opts_end="-------S OPTS END-------"
mailx_to_email_start="-------EMAIL LIST START-------"
mailx_to_email_end="-------EMAIL LIST END-------"
no_attachment=false
no_additional_opts=false
additional_args_switch="-S "
num_files_in_flag_path="$(find $email_flag_path -type f | wc -l)"
if [[ $num_files_in_flag_path -gt 0 ]]; then
for file in $email_flag_path*; do
email_message="$(awk "/$mailx_message_start/,/$mailx_message_end/" $file | egrep -v -- "$mailx_message_start|$mailx_message_end")"
email_subject="$(awk "/$mailx_subject_start/,/$mailx_subject_end/" $file | egrep -v -- "$mailx_subject_start|$mailx_subject_end")"
email_attachment="$(awk "/$mailx_attachement_start/,/$mailx_attachement_end/" $file | egrep -v -- "$mailx_attachement_start|$mailx_attachement_end")"
email_additional_opts="$(awk "/$mailx_s_opts_start/,/$mailx_s_opts_end/" $file | egrep -v -- "$mailx_s_opts_start|$mailx_s_opts_end")"
email_addresses="$(awk "/$mailx_to_email_start/,/$mailx_to_email_end/" $file | egrep -v -- "$mailx_to_email_start|$mailx_to_email_end" | tr -d '\n')"
if [[ -z "$email_message" || -z "$email_subject" || -z "$email_addresses" ]]; then
echo >&2 "MISSING DETAILS IN INPUT FILE $file.... Exiting With Error"
exit 1
fi
if [[ -z "$email_attachment" ]]; then
no_attachment=true
fi
if [[ -z "$email_additional_opts" ]]; then
no_additional_opts=true
else
additional_opts_string=""
while read -r line; do
if [[ ! $line =~ [^[:space:]] ]]; then
continue
else
additional_opts_string="$additional_opts_string \"${additional_args_switch} '$line'\""
fi
done <<<"$(echo "$email_additional_opts")"
additional_opts_string="$(echo ${additional_opts_string:1} | tr -d '\n')"
fi
if [[ $no_attachment = true ]]; then
if [[ $no_additional_opts = true ]]; then
echo "$email_message" | mailx -s "$email_subject" $email_addresses
else
echo "$email_message" | mailx -s "$email_subject" $additional_opts_string $email_addresses
fi
else
if [[ $no_additional_opts = true ]]; then
echo "$email_message" | mailx -s "$email_subject" -a $email_attachment $email_addresses
else
echo "$email_message" | mailx -s "$email_subject" -a $email_attachment $additional_opts_string $email_addresses
fi
fi
done
fi
find $email_flag_path -type f -delete
exit 0
There is however an issue with the above that I just can work out..... the -S opts completely screw up the email headers and I end up with emails being sent to the wrong people (I have set a reply-to and from options, but the email header is jumbled and the reply-to email ends up in the to: field) like this:
To: Me <a#b.com>, sendwait#a.lan , -S#a.lan <-s#a.lan>, another-email <another#b.com>
All I'm trying to do is rebuild the command as if I'd typed it in the CLI:
echo "EMAIL BODY MESSAGE" | mailx -s "EMAIL SUBJECT" -S "from=EMAILNAME <email#b.c>" -S "replyto=EMAILNAME <email#b.c>" -S sendwait my.email#b.com
I've tried quoting in ' ' and " " quoting the other mailx parameters around it etc etc... I have written other tools that pass variables as input arguments so I just cannot understand how I'm screwing this up.....
Any help would be appreciated...
EDIT
Thanks to Gordon Davisson's really helpful comments I was able to not only fix it but understand the fix as well using an array and appropriately quoting the variables... the tip about using printf was really really helpful in helping me understand what I was doing wrong and how to correct it :P
declare -a mailx_args_array
...
num_files_in_flag_path="$(find $email_flag_path -type f | wc -l)"
if [[ $num_files_in_flag_path -gt 0 ]]; then
for file in $email_flag_path*; do
....
mailx_args_array+=( -s "$email_subject" )
if [[ ! -z "$email_attachment" ]]; then
mailx_args_array+=( -a "$email_attachment" )
fi
if [[ ! -z "$email_additional_s_opts" ]]; then
while read -r s_opt_line; do
mailx_args_array+=( -S "$s_opt_line" )
done < <(echo "$email_additional_s_opts")
fi
mailx_args_array+=( "$email_addresses" )
echo "$email_message" | mailx "${mailx_args_array[#]}"
done
fi
I am using this bash script to post a new message to my rocket.chat instance.
#!/usr/bin/env bash
function usage {
programName=$0
echo "description: use this program to post messages to Rocket.chat channel"
echo "usage: $programName [-b \"message body\"] [-u \"rocket.chat url\"]"
echo " -b The message body"
echo " -u The rocket.chat hook url to post to"
exit 1
}
while getopts ":b:u:h" opt; do
case ${opt} in
u) rocketUrl="$OPTARG"
;;
b) msgBody="$OPTARG"
;;
h) usage
;;
\?) echo "Invalid option -$OPTARG" >&2
;;
esac
done
if [[ ! "${rocketUrl}" || ! "${msgBody}" ]]; then
echo "all arguments are required"
usage
fi
read -d '' payLoad << EOF
{"text": "${msgBody}"}
EOF
echo $payLoad
statusCode=$(curl \
--write-out %{http_code} \
--silent \
--output /dev/null \
-X POST \
-H 'Content-type: application/json' \
--data "${payLoad}" ${rocketUrl})
echo ${statusCode}
Everthings works fine, so i can send a new message like this
./postToRocket.sh -b "Hello from here" -u $RocketURL
But when i try to add a message with multiple lines like this
./postToRocket.sh -b "Hello from here\nThis is a new line" -u $RocketURL
it doesn't work. I get the following output:
{"text": "Hello from heren New Line"}
200
So what do i need to change, to use break line with these bash script. Any ideas?
First, the thing making the backslash in your \n disappear was the lack of the -r argument to read. Making it read -r -d '' payLoad will fix that. However, that's not a good solution: It requires your callers to pass strings already escaped for inclusion in JSON, instead of letting them pass any/every possible string.
To make valid JSON with an arbitrary string -- including one that can contain newline literals, quotes, backslashes, or other content that has to be escaped -- use jq:
payLoad=$(jq -n --arg msgBody "$msgBody" '{"text": $msgBody}')
...and then, after doing that, amend your calling convention:
./postToRocket.sh -b $'Hello from here\nThis is a new line' -u "$RocketURL"
I believe this has already been answered in SO here
Should work by adding the $ sign and using single quotes:
./postToRocket.sh -b $'Hello from here\nThis is a new line' -u $RocketURL
Hi i am testing web services using shell script by having multiple if condition, with the shell script coding i am getting success count, failure count and failure reason
success=0
failure=0
if curl -s --head --request DELETE http://localhost/bimws/delete/deleteUser?email=pradeepkumarhe1989#gmail.com | grep "200 OK" > /dev/null; then
success=$((success+1))
else
echo "DeleteUser is not working"$'\r' >> serverLog.txt
failure=$((failure+1))
fi
if curl -s --head --request GET http://localhost/bimws/get/getUserDetails?email=anusha4saju#gmail.com | grep "200 OK" > /dev/null; then
success=$((success+1))
else
curl -s --head --request GET http://localhost/bimws/get/getUserDetails?email=anusha4saju#gmail.com > f1.txt
echo "getUserDetails is not working"$'\r' >> serverLog.txt
failure=$((failure+1))
fi
if curl -s -i -X POST -H "Content-Type:application/json" http://localhost/bimws/post/addProjectLocationAddress -d '{"companyid":"10","projectid":"200","addresstypeid":"5","address":"1234 main st","city":"san jose","state":"CA","zip":"989898","country":"United States"}' | grep "200 OK" > /dev/null; then
success=$((success+1))
else
echo "addProjectLocationAddress is not working"$'\r' >> serverLog.txt
failure=$((failure+1))
fi
echo $success Success
echo $failure failure
but i am looking forward to test the web services from a file like i have file called web_services.txt which contains all my web services using shell script how do i execute and success count, failure count and failure reason
web_services.txt
All are different calls delete,get and post
http://localhost/bimws/delete/deleteUser?email=pradeepkumarhe1989#gmail.com
http://localhost/bimws/get/getUserDetails?email=anusha4saju#gmail.com
http://localhost/bimws/post/addProjectLocationAddress -d '{"companyid":"10","projectid":"200","addresstypeid":"5","address":"1234 main st"
,"city":"san jose","state":"CA","zip":"989898","country":"United States"}'
First of all, your current code does not correctly deal with empty lines. You need to skip those.
Your lines already contain shell commands. Running curl on them makes no sense. Instead, you should evaluate these commands.
Then, you need to modify curl so that it reports whether the request was successful by adding -f:
FILE=D:/WS.txt
success=0
failure=0
while read LINE; do
if test -z "$LINE"; then
continue
fi
if eval $(echo "$LINE" | sed 's/^curl/curl -f -s/') > /dev/null; then
success=$((success+1))
else
echo $LINE >> aNewFile.txt
failure=$((failure+1))
fi
done < $FILE
echo $success Success
echo $failure failure
I have a simple script that accepts 2 arguments, a URL and a log file location. Theoretically, it should capture the header status code from the curl command and if it is a 404, then append the URL to the log file. Any idea where it is failing?
#!/bin/bash
CMP='HTTP/1.1 404 Not Found' # This is the 404 Pattern
OPT=`curl --config /var/www/html/curl.cnf -s -D - "$1" -o /dev/null | grep 404` # Status Response
if [ $OPT = $CMP ]
then
echo "$1" >> "$2" # Append URL to File
fi
Your test is assigning the value of $CMP to $OPT, not comparing for equality. Try the following simpler method, which checks the return code of the grep command rather than looking for the comparison string in its output:
#!/bin/bash
CMP='HTTP/1.1 404 Not Found'
if $(curl -s -I "$1" | grep "$CMP" >/dev/null 2>&1); then
echo "$1" >> "$2"
fi
I am getting a 400 Bad Request response header when I use bash to fetch a web page but only when I call my function inside another function?
get_download_page() {
[ $# -eq 1 ] || die "get_location: 1 argument expected, recieved $#"
page="$(get_page $1)/download"
echo "http://www.curse.com$(curl --silent -I $page | awk 'NR==4 { print; exit }' | cut -d\ -f2)"
}
get_page() {
[ $# -eq 1 ] || die "get_page: 1 argument expected, recieved $#"
echo "www.curse.com/addons/wow/$1"
}
get_file() {
[ $# -eq 1 ] || die "get_file: 1 argument expected, recieved $#"
echo $(curl -s $1 | sed -rn 's/.*data-href="([^"]+)".*/\1/p')
}
fetch_addon() {
[ $# -eq 1 ] || die "fetch_addon: 1 argument expected, recieved $#"
download=$(get_download_page $1)
file=$(get_file $download)
echo $file
}
I.e.
Calling fetch_addon "bagnon" I get a bad request header.
But if I do: get_file "http://www.curse.com/addons/wow/bagnon/704176"
http://addons.curse.cursecdn.com/files/704/176/Bagnon_5.3.zip
Which is what I expect, even though get_download_page "bagnon" returns the same link that I am passing to get_file?
As shown by doing:
get_download_page "bagnon"
echo "http://www.curse.com/addons/wow/bagnon/704176"
Output:
http://www.curse.com/addons/wow/bagnon/704176
http://www.curse.com/addons/wow/bagnon/704176
After further investigation it seems that the two strings aren't exactly equal.
When I do
echo $(cmp -bl <(echo "$download") <(echo "http://www.curse.com/addons/wow/bagnon/704176"))
I get this output:
46 15 ^M 12 ^J
Ie the last character of the generated download link is ^M and the last character of the manually found download link is ^J
For some reason the header curl was returning had mac file endings so that caused the link it returned to end in ^M, when passing that back to curl it freaked out got the bad request header.
I fixed it by piping the link into
sed -r 's/^M/^J/g'
NB:
I couldn't just type "^M" I had to press Ctrl+V and Ctrl+M/J to get the correct character.