I am trying to pass parameters to cURL through the command line, this way:
curl -s -X POST -H "Content-Type: text/xml" -H "Cache-Control: no-cache" -d '<Data Token="someToken" Name='"$appName"' ID='"$someVar"' ParseAppID='"$someVar"' ParseRESTKey='"$someVar"' AndroidPackage='"$someVar"' Version="1"></Data>' 'https://prefix.something.com/somePath?InputType=Xml'
(This line is actually extracted from the Postman app).
I Googled this issue and found whole lot of solutions that did not work for me (links are to SO past questions...):
I tried isolating the variables by ending the single quotes, this way: 'before...'"${someVar}"'...after...'. Could not complete the request.
I tried passing the variables using a file (-d #fileName). Failed to post.
I tried replacing the single quotes surrounding the <Data> tokens with double quotes - but the command apparently cannot accept such substitution.
The errors I get are either <Error></Error> or The server encountered an error and could not complete your request.
Is there any chance that there exists some other solution?
Has anyone encoutered such problem before?
I would be greatful for any help.
You aren't supplying quotes around the value of ID like you are for Name. That is, you need
'<Data Token="someToken" Name="'"$appName"'" ...>'
^^^
|||
||+- shell quote to protect $appName
|+- shell quote enclosing the XML
+- literal quote embedded in the XML
which results in the string (assuming appName=foo)
<Data Token="someToken" Name="foo" ...>
Related
So, I've been beating my head around this for a few days and could use some help.
Initially, I found the gitlab blog post from 2017: https://about.gitlab.com/blog/2017/09/05/how-to-automatically-create-a-new-mr-on-gitlab-with-gitlab-ci/
However it seems like a lot of it is outdated(?) but I was able to use POSTMAN to carve down a working model (using token as a parameter instead of a header, etc.)
Here's the thing though, I cannot get the Merge Request to trigger and I believe it's due to --data (-d) not interpolating the variables.
I've hardcoded the payload with the values exactly, and pushed to my branch and it generated the MR just fine.
But any method I use to try and incorporate variables will result in the cURL going through, but no MR being generated.
What's more, the GL Runner does not output the response - so it's difficult for me to figure out exactly what the issue is.
I've tried escaping with back slashes, I've tried single quote setups with escaping the single quotes and doing inner double quotes with the ${} variables, but nothing.
I've validated that the variables being sent in from the yml exists (echoing them out before the curl call and the body setup), but I'm just lost at this point.
I've still yet to try JQ, but I'm not sure if the runner has it installed, but I'll find out when I return back to test.
Any help would be lovely.
General format of the scripts code:
#!/usr/bin/env bash
TARGET_BRANCH="test"
echo "Host: ${HOST}"
echo "Project Id: ${CI_PROJECT_ID}
echo "SOURCE BRANCH: ${CI_COMMIT_REF_NAME}
echo "Target Branch: ${TARGET_BRANCH}
BODY="{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}";
echo "Body: ${BODY}"
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data "${BODY}"
Now I want to note, that after testing this and seeing the curl go through, but no MR generated, I modified the --data value to go from ${BODY} to:
(assume to be valid value, just obscuring for reasons)
--data '{
"id": <x>,
"source_branch": "dev",
"target_branch": "test",
"remove_source_branch": false,
"title":"WIP: dev"
}'
And this worked! I the MR generated just fine.
So I I then tested by replacing the single quote with double quotes, and escaping the inner double quotes but still keeping the hardcoded values. Back to no MR being generated.
I'm quote stumped - I even went back to the BODY=... block, and tried to reformat that.
Then I went to the outer single, inner double, escaping
'{
"id": '"${CI_PROJECT_ID}"',
...
'}
and still nothing.
So I'm taking a break after all the attempts and speaking with some teammates to no avail, and throwing a hail marry Stack Overflow for some advice.
Added note:
This is not a typical application project repo, so any discussion on "this isn't a good standard practice, use issues as an anchor point for generating MRs" doesn't apply when the use of the repo will not be using issues, have minimum user hands on, etc.
I value those kinds of insights, but again, the usecase for this repo is not an actual "project", so to speak.
Thank you
I'm a bit confused about what it is you're asking. It sounds like you got it to work just fine by passing the json string directly to curl's data flag, and you're just wondering why the other attempts didn't work. If that's the case, here's what I see which would make your other attempts fail:
# ...
BODY="{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}";
echo "Body: ${BODY}"
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data "${BODY}"
In this first option, the variable looks just fine, but when you're using it in your curl command, you're quoting the variable (ie, "$BODY"). There's nothing wrong with this when you're doing concatenation, but here it's likely causing a syntax issue since when the variable is extracted, the result is a json string that's doubly quoted:
""{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}""
I would try passing the $BODY variable without any quotes:
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data $BODY
For the third attempt, I assume there's a typo in your question since you have what should be the final single-quote in front of the closing brace. The other issue is that you're terminating the single-quoted string in the middle of the json:
'{
"id": '"${CI_PROJECT_ID}"',
...
}'
Since this starts with a single-quote, the first single quote it sees will terminate the string. The first single quote is around "${CI_PROJECT_ID}"', so the string being passed to curl is:
'{
"id": '
To clarify why this curl command isn't throwing an exit code, this is still a legal string, and is perfectly fine to curl, so it sends it on to GitLab API. When parsing the data however, GitLab can't parse it, so nothing happens over there. It would result in a 4xx HTTP code however, but as you mentioned, you can't see some of that information in the job output.
To fix this, I'd remove the single quotes around $CI_PROJECT_ID:
'{
"id": "${CI_PROJECT_ID}",
...
}'
or, since $CI_PROJECT_ID is and always will be an integer, you can remove all the quotes around it:
'{
"id": $CI_PROJECT_ID,
...
}'
lifecycle:
preStop:
exec:
command: ["sh", "-c", "curl -v -X PUT -d '\"SHUTTING_DOWN\"' http://localhost:8080/v1/info/state"]
I am expecting this will produce a curl url like
curl -v -X PUT -d '"SHUTTING_DOWN"' http://localhost:8080/v1/info/state
How ever I am getting with extra single quotes surrounded ''"SHUTTING_DOWN"''
curl -v -X PUT -d ''"SHUTTING_DOWN"'' http://localhost:8080/v1/info/state
Any pointers, where am I going wrong?
I'd suggest getting rid of as many layers of quotes as you can. In the original example you have a layer of quotes from YAML, plus a layer of quotes from the sh -c wrapper. Since you need the HTTP PUT body itself to have both single and double quotes – you need to send the string '"SHUTTING_DOWN"' with both kinds of quotes over the wire – getting rid of as much quoting as you can is helpful.
In both the shell and YAML, the two kinds of quotes behave differently. Backslash escaping only works in double-quoted strings and so you probably need that at the outer layer; then you need single quotes inside the double quotes; and then you need backslash-escaped double quotes inside that.
In YAML specifically the quotes around strings are usually optional, unless they're required to disambiguate things (forcing 'true' or '12345') to be strings. This lets you get rid of one layer of quoting. You also may find this slightly clearer if you use YAML block style with one list item on a line.
command:
- /bin/sh
- -c
- curl -v -X PUT -d "'\"SHUTTING_DOWN\"'" http://localhost:8080/v1/info/state
I might even go one step further here, though. You're not using environment variable expansion, multiple commands, or anything else that requires a shell. That means you don't need the sh -c wrapper. If you remove this, then the only layer of quoting you need is YAML quoting; you don't need to worry about embedding a shell-escaped string inside a YAML-quoted string.
You do need to make sure the quotes are handled correctly. If the string begins with a ' or " then YAML will parse it as a quoted string, and if not then there are no escaping options in an unquoted string. So again you probably need to put the whole thing in a double-quoted string and backslash-escape the double quotes that are part of the value.
Remember that each word needs to go into a separate YAML list item. curl like many commands will let you combine options and arguments, so you can have -XPUT as a single argument or -X and PUT as two separate arguments, but -X PUT as a single word will include the space as part of that word and confuse things.
command:
- curl
- -v
- -X
- PUT
- -d
- "'\"SHUTTING_DOWN\"'"
- http://localhost:8080/v1/info/state
I am trying to make a post request in powershell using curl. But I seem to be unfortunately getting this error.
I have tried removing spaces here and there, and googling the problem but have not found a solution.
curl.exe -X 'POST' -H #{'Content-Type'='application/json'; 'accept'='application/json'} -d \"{\"name\":\"test3\", \"auto_init\":true, \"default_branch\": \"master\", \"description\": \"My Test\", \"gitignores\": \"Vim\", \"issue_labels\":\"Default\", \"license\": \"DOC\", \"name\":\"test2\", \"private\":false, \"readme\":\"Default\",\"template\":false,\"trust_model\":\"default\"}\" http://localhost:3000/api/v1/user/repos?access_token=c11ceb97fa594a7e6c4b5519e4327908be3274b9
Re -H:
curl.exe is an external program, which means that you cannot meaningfully pass a PowerShell hashtable (#{ ... }) as an argument, because it will (uselessly) be passed as literal string System.Collections.Hashtable.
Instead, pass strings, as multiple -H options, each in the form '<name>: <value>'
Re -d:
PowerShell's escape character is ` (the so-called backtick), not \.
Since your argument is to be passed verbatim (contains no variable references to be interpolated), use a verbatim (single-quoted) string ('...').
However: The sad reality as of PowerShell 7.2 is that an extra, manual layer of \-escaping of embedded " characters is required in arguments passed to external programs. This may get fixed in a future version, which may require opt-in. See this answer to the linked duplicate for details.
To put it all together:
curl.exe -H 'Content-Type: application/json' -H 'accept: application/json' -d '{\"name\":\"test3\", \"auto_init\":true, \"default_branch\": \"master\", \"description\": \"My Test\", \"gitignores\": \"Vim\", \"issue_labels\":\"Default\", \"license\": \"DOC\", \"name\":\"test2\", \"private\":false, \"readme\":\"Default\",\"template\":false,\"trust_model\":\"default\"}' 'http://localhost:3000/api/v1/user/repos?access_token=c11ceb97fa594a7e6c4b5519e4327908be3274b9'
Note: I've omitted -X 'POST', because, as Daniel Stenberg notes, a POST request is implied when you use the -d option.
I have a URL that I curl in a bash function, and it is appended by a string parameter. The URL has backslashes in it, and they work fine. However, if there is a backslash in the parameter, it fails.
The important part is:
curl "http://latex.codecogs.com/png.latex?\bg_white&space;\huge&space;$1" > image.png
Now, if the parameter is "f(x)=x^{2}" it'll work fine. But if I try "f(x)=\sqrt{x}" then I only get back the part before the backslash. I've tried single and double quotes on both parts, but nothing is working.
How do I pass a parameter with a backslash and then concatenate it with another string and still maintain the backslash?
Try replacing your backslash into their url encoded format %5C
f(x)=%5Csqrt{x}
The comment by #benjamin-w in the question worked:
curl -G "http://latex.codecogs.com/png.latex" --data-urlencode '\bg_white&space;\huge&space;'"$1" > image.png
Hi I have to call curl POST method for a number of records, using bash script. For that i am using while loop.
Actual posting with hardcoded values:
curl -X POST -u user:pass --data '<automation><operation action="assemble" package="service Item"/></automation>' http://localhost:8080/form
When i try to use while loop for a set of different actions and different packages, the $packagename field is not working. When i use $packagename as a variable to package option of data section, it is not taking, since we must pass data in double quotes. (" ") It is always taking the same value which is in double quotes.
Could someone suggest a solution for this?
End the single quoted string and switch to double quotes around the variable.
curl -X POST -u user:pass --data '<automation><operation action="assemble" package="'"$packagename"'"/></automation>' http://localhost:8080/form