I am getting a 400 Bad Request response header when I use bash to fetch a web page but only when I call my function inside another function?
get_download_page() {
[ $# -eq 1 ] || die "get_location: 1 argument expected, recieved $#"
page="$(get_page $1)/download"
echo "http://www.curse.com$(curl --silent -I $page | awk 'NR==4 { print; exit }' | cut -d\ -f2)"
}
get_page() {
[ $# -eq 1 ] || die "get_page: 1 argument expected, recieved $#"
echo "www.curse.com/addons/wow/$1"
}
get_file() {
[ $# -eq 1 ] || die "get_file: 1 argument expected, recieved $#"
echo $(curl -s $1 | sed -rn 's/.*data-href="([^"]+)".*/\1/p')
}
fetch_addon() {
[ $# -eq 1 ] || die "fetch_addon: 1 argument expected, recieved $#"
download=$(get_download_page $1)
file=$(get_file $download)
echo $file
}
I.e.
Calling fetch_addon "bagnon" I get a bad request header.
But if I do: get_file "http://www.curse.com/addons/wow/bagnon/704176"
http://addons.curse.cursecdn.com/files/704/176/Bagnon_5.3.zip
Which is what I expect, even though get_download_page "bagnon" returns the same link that I am passing to get_file?
As shown by doing:
get_download_page "bagnon"
echo "http://www.curse.com/addons/wow/bagnon/704176"
Output:
http://www.curse.com/addons/wow/bagnon/704176
http://www.curse.com/addons/wow/bagnon/704176
After further investigation it seems that the two strings aren't exactly equal.
When I do
echo $(cmp -bl <(echo "$download") <(echo "http://www.curse.com/addons/wow/bagnon/704176"))
I get this output:
46 15 ^M 12 ^J
Ie the last character of the generated download link is ^M and the last character of the manually found download link is ^J
For some reason the header curl was returning had mac file endings so that caused the link it returned to end in ^M, when passing that back to curl it freaked out got the bad request header.
I fixed it by piping the link into
sed -r 's/^M/^J/g'
NB:
I couldn't just type "^M" I had to press Ctrl+V and Ctrl+M/J to get the correct character.
Related
I have the following BASH code:
response=$( curl -Ls $endpoint )
if [ -n "$response" ]; then # nonempty
echo "$response" | jq .
fi
The problem is that sometimes the response can be non-empty but not JSON (if it's not a 200).
Is it possible to pipe the output through jq ONLY if it is valid JSON?
The following works:
echo $x | jq . 2>/dev/null || echo $x
Test:
> x='{"foo":123}'; echo $x | jq . 2>/dev/null || echo "Invalid: $x"
{
"foo": 123
}
> x='}'; echo $x | jq . 2>/dev/null || echo "Invalid: $x"
Invalid: }
However, I don't feel comfortable with it.
If you want to test the response type before submitting it to jq, it is possible if you test the Content-Type header from the server's response.
So you want curl to send you the full response headers and body with curl -i.
Here is an implementation of it:
#!/usr/bin/env sh
endpoint='https://worldtimeapi.org/api/timezone/Europe/Paris.json'
# Headers and body are delimited by an empty line, with CRLF as the line ending.
# See: RFC7230 HTTP/1.1 Message Syntax and Routing / section 3: Message Format
# https://tools.ietf.org/html/rfc7230#section-3
crlf="$(printf '\r\n_')" # add trailing _ to prevent trailing newline trim
crlf="${crlf%_}" # remove trailing _
http_delim="$crlf$crlf" # RFC7230 section 3
full_http_response="$(curl --silent --include --url "$endpoint")"
http_headers="${full_http_response%$http_delim*}"
http_body="${full_http_response#*$http_delim}"
case $http_headers in
'HTTP/1.1 200 OK'*'Content-Type: application/json'*)
# Yes, response body is JSON, so process it with jq.
jq -n "$http_body"
;;
esac
The following works:
echo $x | jq . 2>/dev/null || echo $x
Except for the use of echo here, this is actually a good approach - it has the advantages of both simplicity and efficiency. It is better than using the -e option naively as the return codes produced by -e are more complex.
In other words, there is much to be said for:
printf "%s" "$x" | jq . 2> /dev/null || printf "%s\n" "$x"
Efficiency
The argument for efficiency is as follows:
If $x holds valid JSON, then there is no overhead.
If $x is invalid as JSON, jq will quickly fail; in this case also, the overhead of calling jq will almost surely be no worse or not much worse than checking the Content-Type.
Warning
The official documentation for the return codes produced by jq when invoked without the -e option is not strictly correct, as illustrated by:
$ jq empty <<< 'foo bat' 2> /dev/null ; echo $?
4
This works:
response=$( curl -Ls -H 'Cache-Control: max-age=0' $endpoint )
if [ -n "$response" ]; then # nonempty
echo "Got server response"
# https://stackoverflow.com/questions/46954692/check-if-string-is-a-valid-json-with-jq
if jq --exit-status type >/dev/null 2>&1 <<<"$response"; then
# Parsed JSON successfully and got something other than false/null
echo "$response" | jq .
echo "... after $i seconds"
return 0
else
echo "Response is not valid JSON"
echo "$response"
return 1
fi
fi
It seems whenever I test this on the shell it works but when I'm running it from a script it won't.
Shell:
fips=55555
echo https://www2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_$fips'_'roads.zip
>>https://www2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_55555_roads.zip
I'm trying to loop through some data in a text file like this:
And parse it/input into a curl url:
But the output has the beginning of the string overwritten like this:
./readFipsData.txt
>>_roads.zipw2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_48001
Note: I am using Git Bash for Windows if that makes any difference.
cat ./testFipsData.txt |
while read line
do
counter=0
for col in $line
do
if [ $counter == 0 ]
then
state=$col
elif [ $counter == 1 ]
then
county=$col
elif [ $counter == 2 ]
then
fips=$col
fi
((counter++))
done
#curl -k -o ./$county.zip "https://www2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_$fips_roads.zip"
echo https://www2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_$fips'_'roads.zip
done
Thanks to all of your suggestions here's what worked:
cat ./testFipsData.txt |
while read state county fips
do
fips=${fips%$'\r'}
curl -k -o ./$county.zip https://www2.census.gov/geo/tiger/TIGER2016/ROADS/tl_2016_$fips'_'roads.zip
done
So I have a shell script that contains a big if/fi block, and it was working fine until I decided to place an else case for this big if/fi block. Now I am getting this error:
/root/VVPN/Scripts/scriptPrincipal.sh: line 201: syntax error near unexpected token `else'
/root/VVPN/Scripts/scriptPrincipal.sh: line 201: `else'
I went through 8~10 stackoverflow posts where people had exactly the same error, except that all of them were simple syntax errors like a missing space after the [ of the if statement, or a : instead of a ; before the then keyword, or an else intended for an if that was already closed with a fi, etc... (you get the idea :p).
However I've checked my code for all these errors over and over and everything seems to be correct when it comes to if/else/fi syntax. I even showed the code to some colleagues and they too couldn't find the reason for this error.
Here's the code:
if [ ${CP} != "continue" ]
then
echo 'Downloading the necessary files from the server...'
# If folder F*** doesn't exist in /root/VVPN/Numbers
if [ ! -d ${N} ]
then
# Create folders F***, and F***/Results
mkdir ${N}
mkdir ${N}/Results
cd ${N}
# Get real number to factorize from server (e.g: wget server.com:8000/route/to/F1067/F1067)
ROUTE="VVPN/Numbers/${N}/${N}"
REAL_NUM_FILENAME=${N}
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${REAL_NUM_FILENAME} --continue
if [ $? -eq 0 ]; then
break
fi
sleep 1
done
# Get ECM-Program tuned for this number from server (e.g: wget server.com:8000/route/to/F1067/ecm)
ROUTE="VVPN/Numbers/${N}/ecm"
PROGRAM_NAME='ecm'
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${PROGRAM_NAME} --continue
if [ $? = 0 ]; then
break
fi
sleep 1
done
else
# The folder already exists, now we have to check whether the number has ",c" label or not
cd ${N}
fi
# Give the permission to execute program
chmod +x ecm
# Make 6 directories, one for each SPE
for i in {1..6}
do
mkdir "SPE${i}"
cp ecm "SPE${i}/"
done
# if there is no checkpoints:
if [ $CP = $N ]
then
# Get currentSigma for this number from server (e.g : wget server.com:8000/rout/to/F1067/currentSigma.txt)
ROUTE="VVPN/Numbers/${N}/currentSigma"
REAL_FILE_NAME='sigma'
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${REAL_FILE_NAME} --continue
if [ $? = 0 ]; then
break
fi
sleep 1
done
else
#The number has a ",c" label (= w/ checkpoint)
# Get i (server sigma) and C (current job counter) from the server (e.g: wget server.com:8000/route/to/F1067/icy)
ROUTE="icy?number=${N}"
SIGMA_C_Y='icy'
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE} -O ${SIGMA_C_Y} --continue
if [ $? = 0 ]; then
break
fi
sleep 1
done
i=$(cat ${SIGMA_C_Y} | cut -d "," -f1)
C=$(cat ${SIGMA_C_Y} | cut -d "," -f2)
Y=$(cat ${SIGMA_C_Y} | cut -d "," -f3)
echo $i > sigma
echo $C > /root/VVPN/Scripts/C
# Get the checkpoints from the server
ROUTE="VVPN/Numbers/${N}/CP/${i},${C},${Y}"
for speNum in {1..6}
do
SPE="SPE${speNum}"
touch $SPE/again
for bigX in {0..3}
do
CHECKPOINT="pointsCurve${bigX}.${Y}"
while true
do
wget --retry-connrefused --tries=inf -q ${SERVER}/${ROUTE}/${SPE}/${CHECKPOINT} -O ${CHECKPOINT} --continue
if [ $? = 0 ]; then
mv $CHECKPOINT $SPE
break
fi
sleep 1
done
done
done
fi
cd ..
else
echo "Found ${N}'s folder on this PS3"
fi
So the else mentioned in the error (at line 201) is actually the last else in the code. The script works fine without this else and the echo that comes right after.
Any help would be much appreciated :)
Could you change if [ $CP = $N ] as below;
if [ "$CP" == "$N" ]
This is not if else problem; For example; if you run the following code, output is same. So You should focus other commands inside if statement.
CP="continue1"
if [ ${CP} != "continue" ]
then
while #this is wrong
echo ok
echo ok
echo ok
else
fi
./test.sh: line 8: syntax error near unexpected token `else'
./test.sh: line 8: `else'
I have a simple script that accepts 2 arguments, a URL and a log file location. Theoretically, it should capture the header status code from the curl command and if it is a 404, then append the URL to the log file. Any idea where it is failing?
#!/bin/bash
CMP='HTTP/1.1 404 Not Found' # This is the 404 Pattern
OPT=`curl --config /var/www/html/curl.cnf -s -D - "$1" -o /dev/null | grep 404` # Status Response
if [ $OPT = $CMP ]
then
echo "$1" >> "$2" # Append URL to File
fi
Your test is assigning the value of $CMP to $OPT, not comparing for equality. Try the following simpler method, which checks the return code of the grep command rather than looking for the comparison string in its output:
#!/bin/bash
CMP='HTTP/1.1 404 Not Found'
if $(curl -s -I "$1" | grep "$CMP" >/dev/null 2>&1); then
echo "$1" >> "$2"
fi
When I run this by its self in the command line it seems to work fine, but when I have another script execute this, it doesn't work. Any ideas? I'm guessing it has to do with quotes, but not sure.
#!/bin/sh
#Required csvquote from https://github.com/dbro/csvquote
#TODO: Clean CSV File using CSVFix
#Version 3
echo "File Name: $1"
function quit {
echo "Quitting Script"
exit 1
}
function fileExists {
if [ ! -f "$1" ]
then
echo "File $1 does not exists"
quit
fi
}
function getInfo {
#Returns website url like: "http://www.website.com/info"
#Reads last line of a csv file, and gets the 2nd item.
RETURN=$(tail -n 1 $1 | csvquote | cut -d ',' -f 2 | csvquote -u)
echo $RETURN
}
function work {
CURLURL="http://127.0.0.1:9200/cj/_query"
URL=$(getInfo)
echo "URL: $URL"
CURLDATA='{ "query" : { "match" : { "PROGRAMURL" : '$URL' } } }'
#URL shows up as blank...???
echo "Curl Data: $CURLDATA"
RESPONSE=$(curl -XDELETE "$CURLURL" -d "$CURLDATA" -vn)
echo $RESPONSE
echo "Sleeping Allowing Time To Delete"
sleep 5s
}
fileExists $1
work $1
I cant see why a simpler version wont work: functions are useful, but I think there are too many, overcomplicating things, if what you are posting is the entirety of your script (in my opinion)
Your script is doing things using a broken lucky pattern: $1 variables are also arguments to shell functions as well as the main script. Think of them as local variables to a function. So when you are calling $(getInfo) it is calling that function with no argument, so actually runs tail -n 1 which falls back to stdin, which you are specifying to work as < $1. You could see this for yourself by putting echo getInfo_arg_1="$1" >&2 inside the function...
Note also you are not quoting $1 anywhere, this script is not whitespace in file safe, although this is only more likely to be a problem if you are having to deal with files sent to you from a Windows computer.
In the absence of other information, the following 'should' work:
#!/bin/bash
test -z "$1" && { echo "Please specify a file." ; exit 1; }
test -f "$1" || { echo "Cant see file '$1'." ; exit 1; }
FILE="$1"
function getInfo() {
#Returns website url like: "http://www.website.com/info"
#Reads last line of a csv file, and gets the 2nd item.
tail -n 1 "$1" | csvquote | cut -d ',' -f 2 | csvquote -u
}
CURLURL="http://127.0.0.1:9200/cj/_query"
URL=$(getInfo "$FILE")
echo "URL: $URL"
CURLDATA='{ "query" : { "match" : { "PROGRAMURL" : '$URL' } } }'
curl -XDELETE "$CURLURL" -d "$CURLDATA" -vn
echo "Sleeping Allowing Time To Delete"
sleep 5s
If it still fails you really need to post your error messages.
One other thing, especially if you are calling this from another script, chmod +x the script so you can run it without having to invoke it with bash directly. If you want to turn on debugging then put set -x near the start somewhere.