executing a curl request through bash script - bash

I have to insert many data in my application and through the graphical interface it takes many time. For this reason I want to create a bash script and make the requests through curl using the REST API (I have to manually specify the id).
The problem is that i get the error: The server refused this request because the request entity is in a format not supported by the requested resource for the requested method.
Here is the code
#!/bin/bash
for i in {1..1}
do
CURL='/usr/bin/curl -X POST'
RVMHTTP="http://192.168.1.101:8080/sitewhere/api/devices
-H 'accept:application/json'
-H 'content-type:application/json'
-H 'x-sitewhere-tenant:sitewhere1234567890'
--user admin:password"
DATA=" -d '{\"hardwareId":\"$i",\"siteToken\":\"4e6913db-c8d3-4e45-9436-f0a99b502d3c\",\"specificationToken\":\"82043707-9e3d-441f-bdcc-33cf0f4f7260\"}'"
# or you can redirect it into a file:
$CURL $RVMHTTP $DATA >> /home/bluedragon/Desktop/tokens
done
The format of my request has to be json

#!/usr/bin/env bash
rvmcurl() {
local url
url="http://192.168.1.101:8080/sitewhere/${1#/}"
shift || return # function should fail if we weren't passed at least one argument
curl -XPOST "${rvm_curl_args[#]}" "$url" "$#"
}
i=1 # for testing purposes
rvm_curl_args=(
-H 'accept:application/json'
-H 'content-type:application/json'
-H 'x-sitewhere-tenant:sitewhere1234567890'
--user admin:password
)
data=$(jq -n --arg hardwareId "$i" '
{
"hardwareId": $hardwareId,
"siteToken": "4e6913db-c8d3-4e45-9436-f0a99b502d3c",
"specializationToken": "82043707-9e3d-441f-bdcc-33cf0f4f7260"
}')
rvmcurl /api/devices -d "$data"
Note:
Commands, or command fragments intended to be parsed into multiple words, should never be stored in strings. Use an array or a function instead. Quotes inside such strings are not parsed as syntax, and instead (when parsed without eval, which carries its own serious risks and caveats) become literal values. See BashFAQ #50 for a full explanation.
Use a JSON-aware tool, such as jq, to ensure that generated data is legit JSON.
Fully-qualifying paths to binaries is, in general, an antipattern. It doesn't result in a significant performance gain (the shell caches PATH lookups), but it does reduce your scripts' portability and flexibility (preventing you from installing a wrapper for curl in your PATH, in an exported shell function, or otherwise).
All-caps variable names are in a namespace used for variables with meaning to the shell and operating system. Use names with at least one lowercase character for your own variables to prevent any chance of conflict.

Related

curl: (3) Port number ended with ' ' [Powershell]

I am trying to make a post request in powershell using curl. But I seem to be unfortunately getting this error.
I have tried removing spaces here and there, and googling the problem but have not found a solution.
curl.exe -X 'POST' -H #{'Content-Type'='application/json'; 'accept'='application/json'} -d \"{\"name\":\"test3\", \"auto_init\":true, \"default_branch\": \"master\", \"description\": \"My Test\", \"gitignores\": \"Vim\", \"issue_labels\":\"Default\", \"license\": \"DOC\", \"name\":\"test2\", \"private\":false, \"readme\":\"Default\",\"template\":false,\"trust_model\":\"default\"}\" http://localhost:3000/api/v1/user/repos?access_token=c11ceb97fa594a7e6c4b5519e4327908be3274b9
Re -H:
curl.exe is an external program, which means that you cannot meaningfully pass a PowerShell hashtable (#{ ... }) as an argument, because it will (uselessly) be passed as literal string System.Collections.Hashtable.
Instead, pass strings, as multiple -H options, each in the form '<name>: <value>'
Re -d:
PowerShell's escape character is ` (the so-called backtick), not \.
Since your argument is to be passed verbatim (contains no variable references to be interpolated), use a verbatim (single-quoted) string ('...').
However: The sad reality as of PowerShell 7.2 is that an extra, manual layer of \-escaping of embedded " characters is required in arguments passed to external programs. This may get fixed in a future version, which may require opt-in. See this answer to the linked duplicate for details.
To put it all together:
curl.exe -H 'Content-Type: application/json' -H 'accept: application/json' -d '{\"name\":\"test3\", \"auto_init\":true, \"default_branch\": \"master\", \"description\": \"My Test\", \"gitignores\": \"Vim\", \"issue_labels\":\"Default\", \"license\": \"DOC\", \"name\":\"test2\", \"private\":false, \"readme\":\"Default\",\"template\":false,\"trust_model\":\"default\"}' 'http://localhost:3000/api/v1/user/repos?access_token=c11ceb97fa594a7e6c4b5519e4327908be3274b9'
Note: I've omitted -X 'POST', because, as Daniel Stenberg notes, a POST request is implied when you use the -d option.

Is it sensible to automatically URL encode query parameters in a string?

Say I want to make the following request using curl:
https://api.foobar.com/widgets?begin=2018-09-10T01:00:00+01:00&object={"name":"barry"}
The URL encoded version of that string looks like this:
https://api.foobar.com/widgets?begin=2018-09-10T01%3A00%3A00%2B01%3A00&object=%7B%22name%22%3A%22barry%22%7D
Of course, when I'm making requests at the command line I would much rather look at the nicer looking (but not URL-valid) first version. I'm considering using a bash script to split out the different parts of the nice version, encode the relevant ones, and then glue it back together so I don't have to worry about it.
For example, after a couple of rounds of simple splitting on ?, &, and = I can easily get to:
https://api.foobar.com/widgets
begin
2018-09-10T01:00:00+01:00
object
{"name":"barry"}
And after that, URL encode the query string's two values and glue it all back together. I accept that any occurences of & and = in the query string will break this approach.
Is there anything else I should worry about that might make this a particularly stupid idea?
Use --data-urlencode with --get
curl --data-urlencode 'begin=2018-09-10T01:00:00+01:00' --data-urlencode 'object={"name":"barry"}' --get 'http://api.foobar.com/widgets'
-G, --get When used, this option will make all data specified with -d, --data, --data-binary or --data-urlencode to be used in an HTTP GET request instead of the POST request that otherwise would be used. The data will be appended to the URL with a '?' separator.
This is the script I came up with in the end, curl-encoded.sh:
#!/usr/bin/env bash
#
# Make HTTP request for given URL with query string URL encoded.
#
set -e
# Name args.
URL=$1
if [[ $URL = *"?"* ]]; then
DOMAIN_PATH="${URL%%\?*}";
QUERY_STRING="${URL#*\?}"
else
DOMAIN_PATH=$URL
QUERY_STRING=''
fi
# Split query string into key/value pairs.
IFS='&' read -ra PARAMETERS <<< "$QUERY_STRING"
for PARAMETER in "${PARAMETERS[#]}"; do
URLENCODED_PARAMETERS=("${URLENCODED_PARAMETERS[#]}" "--data-urlencode" "$PARAMETER")
done
# Make request.
curl --silent "${URLENCODED_PARAMETERS[#]/#/}" "${#:2}" --get "$DOMAIN_PATH"
You call:
./curl-encoded.sh https://api.foobar.com/widgets?foo=bar&object={"name":"barry"}
And the URL that's fetched is:
https://api.foobar.com/widgets?foo=bar&object=%7B%22name%22%3A%22barry%22%7D

Curl Post with File

I have data phone in phone.txt
+6285712341234
+6285712341235
+6285712341236
+6285712341237
+6285712341238
but I don't know how to use this data to curl, here's what I tried:
curl -X POST "https://rest-api.moceansms.com/rest/1/sms" -d "mocean-api-key={api_key}&mocean-api-secret={api_secret}&mocean-from={name}&mocean-to={phone.txt}&mocean-text=Hello"
I should use phone data to send SMS to everyone; I googled for a solution, but with no luck (I don't even know whether the keywords I used to look for a solution where correct or not).
You should use --data flag:
Check:
https://curl.haxx.se/mail/archive-2007-03/0097.html
https://curl.haxx.se/docs/manpage.html#-d
Here the entire explanation from man:
-d, --data
(HTTP) Sends the specified data in a POST request to the HTTP server,
in the same way that a browser does when a user has filled in an HTML
form and presses the submit button. This will cause curl to pass the
data to the server using the content-type
application/x-www-form-urlencoded. Compare to -F, --form.
--data-raw is almost the same but does not have a special interpretation of the # character. To post data purely binary, you
should instead use the --data-binary option. To URL-encode the value
of a form field you may use --data-urlencode.
If any of these options is used more than once on the same command
line, the data pieces specified will be merged together with a
separating &-symbol. Thus, using '-d name=daniel -d skill=lousy' would
generate a post chunk that looks like 'name=daniel&skill=lousy'.
If you start the data with the letter #, the rest should be a file
name to read the data from, or - if you want curl to read the data
from stdin. Multiple files can also be specified. Posting data from a
file named 'foobar' would thus be done with -d, --data #foobar. When
--data is told to read from a file like that, carriage returns and newlines will be stripped out. If you don't want the # character to
have a special interpretation use --data-raw instead.
See also --data-binary and --data-urlencode and --data-raw. This
option overrides -F, --form and -I, --head and --upload.

How to urlencode data into a URL, with bash or curl

How can a string be urlencoded and embedded into the URL? Please note that I am not trying to GET or POST data, so the -G and --data and --data-urlencode options of curl don't seem to do the job.
For example, if you used
curl -G http://example.com/foo --data-urlencode "bar=spaced data"
that would be functionally equivalent to
curl http://example.com/foo?bar=spaced%20data"
which is not desired.
I have a string foo/bar which must be urlencoded foo%2fbar and embedded into the URL.
curl http://example.com/api/projects/foo%2fbar/events
One hypothetical solution (if I could find something like this) would be to preprocess the data in bash, if there exists some kind of urlencode function.
DATA=foo/bar
ENCODED=`urlencode $DATA`
curl http://example.com/api/projects/${ENCODED}/events
Another hypothetical solution (if I could find something like this) would be some switch in curl, similar to this:
curl http://example.com/api/projects/{0}/events --string-urlencode "0=foo/bar"
The specific reason I'm looking for an answer to this question is the Gitlab API. For example, gitlab get single project NAMESPACE/PROJECT_NAME is URL-encoded, eg. /api/v3/projects/diaspora%2Fdiaspora (where / is represented by %2F). Further to this, you can request individual properties in the project, so you end up with a URL such as http://example.com/projects/diaspora%2Fdiaspora/events
Although this question is gitlab-specific, I imagine it's generally applicable to REST API's in general, and I'm surprised I can't find a pre-existing answer on stackoverflow or internet search.
The urlencode function you propose is easy enough to implement:
urlencode() {
python -c 'import urllib, sys; print urllib.quote(sys.argv[1], sys.argv[2])' \
"$1" "$urlencode_safe"
}
...used as:
data=foo/bar
encoded=$(urlencode "$data")
curl "http://example.com/api/projects/${encoded}/events"
If you want to have some characters which are passed through literally -- in many use cases, this is desired for /s -- instead use:
encoded=$(urlencode_safe='/' urlencode "$data")

variables inside CURL POST command using shell script and while loop

Hi I have to call curl POST method for a number of records, using bash script. For that i am using while loop.
Actual posting with hardcoded values:
curl -X POST -u user:pass --data '<automation><operation action="assemble" package="service Item"/></automation>' http://localhost:8080/form
When i try to use while loop for a set of different actions and different packages, the $packagename field is not working. When i use $packagename as a variable to package option of data section, it is not taking, since we must pass data in double quotes. (" ") It is always taking the same value which is in double quotes.
Could someone suggest a solution for this?
End the single quoted string and switch to double quotes around the variable.
curl -X POST -u user:pass --data '<automation><operation action="assemble" package="'"$packagename"'"/></automation>' http://localhost:8080/form

Resources