I have data phone in phone.txt
+6285712341234
+6285712341235
+6285712341236
+6285712341237
+6285712341238
but I don't know how to use this data to curl, here's what I tried:
curl -X POST "https://rest-api.moceansms.com/rest/1/sms" -d "mocean-api-key={api_key}&mocean-api-secret={api_secret}&mocean-from={name}&mocean-to={phone.txt}&mocean-text=Hello"
I should use phone data to send SMS to everyone; I googled for a solution, but with no luck (I don't even know whether the keywords I used to look for a solution where correct or not).
You should use --data flag:
Check:
https://curl.haxx.se/mail/archive-2007-03/0097.html
https://curl.haxx.se/docs/manpage.html#-d
Here the entire explanation from man:
-d, --data
(HTTP) Sends the specified data in a POST request to the HTTP server,
in the same way that a browser does when a user has filled in an HTML
form and presses the submit button. This will cause curl to pass the
data to the server using the content-type
application/x-www-form-urlencoded. Compare to -F, --form.
--data-raw is almost the same but does not have a special interpretation of the # character. To post data purely binary, you
should instead use the --data-binary option. To URL-encode the value
of a form field you may use --data-urlencode.
If any of these options is used more than once on the same command
line, the data pieces specified will be merged together with a
separating &-symbol. Thus, using '-d name=daniel -d skill=lousy' would
generate a post chunk that looks like 'name=daniel&skill=lousy'.
If you start the data with the letter #, the rest should be a file
name to read the data from, or - if you want curl to read the data
from stdin. Multiple files can also be specified. Posting data from a
file named 'foobar' would thus be done with -d, --data #foobar. When
--data is told to read from a file like that, carriage returns and newlines will be stripped out. If you don't want the # character to
have a special interpretation use --data-raw instead.
See also --data-binary and --data-urlencode and --data-raw. This
option overrides -F, --form and -I, --head and --upload.
Related
So, I've been beating my head around this for a few days and could use some help.
Initially, I found the gitlab blog post from 2017: https://about.gitlab.com/blog/2017/09/05/how-to-automatically-create-a-new-mr-on-gitlab-with-gitlab-ci/
However it seems like a lot of it is outdated(?) but I was able to use POSTMAN to carve down a working model (using token as a parameter instead of a header, etc.)
Here's the thing though, I cannot get the Merge Request to trigger and I believe it's due to --data (-d) not interpolating the variables.
I've hardcoded the payload with the values exactly, and pushed to my branch and it generated the MR just fine.
But any method I use to try and incorporate variables will result in the cURL going through, but no MR being generated.
What's more, the GL Runner does not output the response - so it's difficult for me to figure out exactly what the issue is.
I've tried escaping with back slashes, I've tried single quote setups with escaping the single quotes and doing inner double quotes with the ${} variables, but nothing.
I've validated that the variables being sent in from the yml exists (echoing them out before the curl call and the body setup), but I'm just lost at this point.
I've still yet to try JQ, but I'm not sure if the runner has it installed, but I'll find out when I return back to test.
Any help would be lovely.
General format of the scripts code:
#!/usr/bin/env bash
TARGET_BRANCH="test"
echo "Host: ${HOST}"
echo "Project Id: ${CI_PROJECT_ID}
echo "SOURCE BRANCH: ${CI_COMMIT_REF_NAME}
echo "Target Branch: ${TARGET_BRANCH}
BODY="{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}";
echo "Body: ${BODY}"
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data "${BODY}"
Now I want to note, that after testing this and seeing the curl go through, but no MR generated, I modified the --data value to go from ${BODY} to:
(assume to be valid value, just obscuring for reasons)
--data '{
"id": <x>,
"source_branch": "dev",
"target_branch": "test",
"remove_source_branch": false,
"title":"WIP: dev"
}'
And this worked! I the MR generated just fine.
So I I then tested by replacing the single quote with double quotes, and escaping the inner double quotes but still keeping the hardcoded values. Back to no MR being generated.
I'm quote stumped - I even went back to the BODY=... block, and tried to reformat that.
Then I went to the outer single, inner double, escaping
'{
"id": '"${CI_PROJECT_ID}"',
...
'}
and still nothing.
So I'm taking a break after all the attempts and speaking with some teammates to no avail, and throwing a hail marry Stack Overflow for some advice.
Added note:
This is not a typical application project repo, so any discussion on "this isn't a good standard practice, use issues as an anchor point for generating MRs" doesn't apply when the use of the repo will not be using issues, have minimum user hands on, etc.
I value those kinds of insights, but again, the usecase for this repo is not an actual "project", so to speak.
Thank you
I'm a bit confused about what it is you're asking. It sounds like you got it to work just fine by passing the json string directly to curl's data flag, and you're just wondering why the other attempts didn't work. If that's the case, here's what I see which would make your other attempts fail:
# ...
BODY="{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}";
echo "Body: ${BODY}"
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data "${BODY}"
In this first option, the variable looks just fine, but when you're using it in your curl command, you're quoting the variable (ie, "$BODY"). There's nothing wrong with this when you're doing concatenation, but here it's likely causing a syntax issue since when the variable is extracted, the result is a json string that's doubly quoted:
""{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}""
I would try passing the $BODY variable without any quotes:
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data $BODY
For the third attempt, I assume there's a typo in your question since you have what should be the final single-quote in front of the closing brace. The other issue is that you're terminating the single-quoted string in the middle of the json:
'{
"id": '"${CI_PROJECT_ID}"',
...
}'
Since this starts with a single-quote, the first single quote it sees will terminate the string. The first single quote is around "${CI_PROJECT_ID}"', so the string being passed to curl is:
'{
"id": '
To clarify why this curl command isn't throwing an exit code, this is still a legal string, and is perfectly fine to curl, so it sends it on to GitLab API. When parsing the data however, GitLab can't parse it, so nothing happens over there. It would result in a 4xx HTTP code however, but as you mentioned, you can't see some of that information in the job output.
To fix this, I'd remove the single quotes around $CI_PROJECT_ID:
'{
"id": "${CI_PROJECT_ID}",
...
}'
or, since $CI_PROJECT_ID is and always will be an integer, you can remove all the quotes around it:
'{
"id": $CI_PROJECT_ID,
...
}'
I am trying to make a post request in powershell using curl. But I seem to be unfortunately getting this error.
I have tried removing spaces here and there, and googling the problem but have not found a solution.
curl.exe -X 'POST' -H #{'Content-Type'='application/json'; 'accept'='application/json'} -d \"{\"name\":\"test3\", \"auto_init\":true, \"default_branch\": \"master\", \"description\": \"My Test\", \"gitignores\": \"Vim\", \"issue_labels\":\"Default\", \"license\": \"DOC\", \"name\":\"test2\", \"private\":false, \"readme\":\"Default\",\"template\":false,\"trust_model\":\"default\"}\" http://localhost:3000/api/v1/user/repos?access_token=c11ceb97fa594a7e6c4b5519e4327908be3274b9
Re -H:
curl.exe is an external program, which means that you cannot meaningfully pass a PowerShell hashtable (#{ ... }) as an argument, because it will (uselessly) be passed as literal string System.Collections.Hashtable.
Instead, pass strings, as multiple -H options, each in the form '<name>: <value>'
Re -d:
PowerShell's escape character is ` (the so-called backtick), not \.
Since your argument is to be passed verbatim (contains no variable references to be interpolated), use a verbatim (single-quoted) string ('...').
However: The sad reality as of PowerShell 7.2 is that an extra, manual layer of \-escaping of embedded " characters is required in arguments passed to external programs. This may get fixed in a future version, which may require opt-in. See this answer to the linked duplicate for details.
To put it all together:
curl.exe -H 'Content-Type: application/json' -H 'accept: application/json' -d '{\"name\":\"test3\", \"auto_init\":true, \"default_branch\": \"master\", \"description\": \"My Test\", \"gitignores\": \"Vim\", \"issue_labels\":\"Default\", \"license\": \"DOC\", \"name\":\"test2\", \"private\":false, \"readme\":\"Default\",\"template\":false,\"trust_model\":\"default\"}' 'http://localhost:3000/api/v1/user/repos?access_token=c11ceb97fa594a7e6c4b5519e4327908be3274b9'
Note: I've omitted -X 'POST', because, as Daniel Stenberg notes, a POST request is implied when you use the -d option.
Say I want to make the following request using curl:
https://api.foobar.com/widgets?begin=2018-09-10T01:00:00+01:00&object={"name":"barry"}
The URL encoded version of that string looks like this:
https://api.foobar.com/widgets?begin=2018-09-10T01%3A00%3A00%2B01%3A00&object=%7B%22name%22%3A%22barry%22%7D
Of course, when I'm making requests at the command line I would much rather look at the nicer looking (but not URL-valid) first version. I'm considering using a bash script to split out the different parts of the nice version, encode the relevant ones, and then glue it back together so I don't have to worry about it.
For example, after a couple of rounds of simple splitting on ?, &, and = I can easily get to:
https://api.foobar.com/widgets
begin
2018-09-10T01:00:00+01:00
object
{"name":"barry"}
And after that, URL encode the query string's two values and glue it all back together. I accept that any occurences of & and = in the query string will break this approach.
Is there anything else I should worry about that might make this a particularly stupid idea?
Use --data-urlencode with --get
curl --data-urlencode 'begin=2018-09-10T01:00:00+01:00' --data-urlencode 'object={"name":"barry"}' --get 'http://api.foobar.com/widgets'
-G, --get When used, this option will make all data specified with -d, --data, --data-binary or --data-urlencode to be used in an HTTP GET request instead of the POST request that otherwise would be used. The data will be appended to the URL with a '?' separator.
This is the script I came up with in the end, curl-encoded.sh:
#!/usr/bin/env bash
#
# Make HTTP request for given URL with query string URL encoded.
#
set -e
# Name args.
URL=$1
if [[ $URL = *"?"* ]]; then
DOMAIN_PATH="${URL%%\?*}";
QUERY_STRING="${URL#*\?}"
else
DOMAIN_PATH=$URL
QUERY_STRING=''
fi
# Split query string into key/value pairs.
IFS='&' read -ra PARAMETERS <<< "$QUERY_STRING"
for PARAMETER in "${PARAMETERS[#]}"; do
URLENCODED_PARAMETERS=("${URLENCODED_PARAMETERS[#]}" "--data-urlencode" "$PARAMETER")
done
# Make request.
curl --silent "${URLENCODED_PARAMETERS[#]/#/}" "${#:2}" --get "$DOMAIN_PATH"
You call:
./curl-encoded.sh https://api.foobar.com/widgets?foo=bar&object={"name":"barry"}
And the URL that's fetched is:
https://api.foobar.com/widgets?foo=bar&object=%7B%22name%22%3A%22barry%22%7D
I have to insert many data in my application and through the graphical interface it takes many time. For this reason I want to create a bash script and make the requests through curl using the REST API (I have to manually specify the id).
The problem is that i get the error: The server refused this request because the request entity is in a format not supported by the requested resource for the requested method.
Here is the code
#!/bin/bash
for i in {1..1}
do
CURL='/usr/bin/curl -X POST'
RVMHTTP="http://192.168.1.101:8080/sitewhere/api/devices
-H 'accept:application/json'
-H 'content-type:application/json'
-H 'x-sitewhere-tenant:sitewhere1234567890'
--user admin:password"
DATA=" -d '{\"hardwareId":\"$i",\"siteToken\":\"4e6913db-c8d3-4e45-9436-f0a99b502d3c\",\"specificationToken\":\"82043707-9e3d-441f-bdcc-33cf0f4f7260\"}'"
# or you can redirect it into a file:
$CURL $RVMHTTP $DATA >> /home/bluedragon/Desktop/tokens
done
The format of my request has to be json
#!/usr/bin/env bash
rvmcurl() {
local url
url="http://192.168.1.101:8080/sitewhere/${1#/}"
shift || return # function should fail if we weren't passed at least one argument
curl -XPOST "${rvm_curl_args[#]}" "$url" "$#"
}
i=1 # for testing purposes
rvm_curl_args=(
-H 'accept:application/json'
-H 'content-type:application/json'
-H 'x-sitewhere-tenant:sitewhere1234567890'
--user admin:password
)
data=$(jq -n --arg hardwareId "$i" '
{
"hardwareId": $hardwareId,
"siteToken": "4e6913db-c8d3-4e45-9436-f0a99b502d3c",
"specializationToken": "82043707-9e3d-441f-bdcc-33cf0f4f7260"
}')
rvmcurl /api/devices -d "$data"
Note:
Commands, or command fragments intended to be parsed into multiple words, should never be stored in strings. Use an array or a function instead. Quotes inside such strings are not parsed as syntax, and instead (when parsed without eval, which carries its own serious risks and caveats) become literal values. See BashFAQ #50 for a full explanation.
Use a JSON-aware tool, such as jq, to ensure that generated data is legit JSON.
Fully-qualifying paths to binaries is, in general, an antipattern. It doesn't result in a significant performance gain (the shell caches PATH lookups), but it does reduce your scripts' portability and flexibility (preventing you from installing a wrapper for curl in your PATH, in an exported shell function, or otherwise).
All-caps variable names are in a namespace used for variables with meaning to the shell and operating system. Use names with at least one lowercase character for your own variables to prevent any chance of conflict.
How can a string be urlencoded and embedded into the URL? Please note that I am not trying to GET or POST data, so the -G and --data and --data-urlencode options of curl don't seem to do the job.
For example, if you used
curl -G http://example.com/foo --data-urlencode "bar=spaced data"
that would be functionally equivalent to
curl http://example.com/foo?bar=spaced%20data"
which is not desired.
I have a string foo/bar which must be urlencoded foo%2fbar and embedded into the URL.
curl http://example.com/api/projects/foo%2fbar/events
One hypothetical solution (if I could find something like this) would be to preprocess the data in bash, if there exists some kind of urlencode function.
DATA=foo/bar
ENCODED=`urlencode $DATA`
curl http://example.com/api/projects/${ENCODED}/events
Another hypothetical solution (if I could find something like this) would be some switch in curl, similar to this:
curl http://example.com/api/projects/{0}/events --string-urlencode "0=foo/bar"
The specific reason I'm looking for an answer to this question is the Gitlab API. For example, gitlab get single project NAMESPACE/PROJECT_NAME is URL-encoded, eg. /api/v3/projects/diaspora%2Fdiaspora (where / is represented by %2F). Further to this, you can request individual properties in the project, so you end up with a URL such as http://example.com/projects/diaspora%2Fdiaspora/events
Although this question is gitlab-specific, I imagine it's generally applicable to REST API's in general, and I'm surprised I can't find a pre-existing answer on stackoverflow or internet search.
The urlencode function you propose is easy enough to implement:
urlencode() {
python -c 'import urllib, sys; print urllib.quote(sys.argv[1], sys.argv[2])' \
"$1" "$urlencode_safe"
}
...used as:
data=foo/bar
encoded=$(urlencode "$data")
curl "http://example.com/api/projects/${encoded}/events"
If you want to have some characters which are passed through literally -- in many use cases, this is desired for /s -- instead use:
encoded=$(urlencode_safe='/' urlencode "$data")