So, I've been beating my head around this for a few days and could use some help.
Initially, I found the gitlab blog post from 2017: https://about.gitlab.com/blog/2017/09/05/how-to-automatically-create-a-new-mr-on-gitlab-with-gitlab-ci/
However it seems like a lot of it is outdated(?) but I was able to use POSTMAN to carve down a working model (using token as a parameter instead of a header, etc.)
Here's the thing though, I cannot get the Merge Request to trigger and I believe it's due to --data (-d) not interpolating the variables.
I've hardcoded the payload with the values exactly, and pushed to my branch and it generated the MR just fine.
But any method I use to try and incorporate variables will result in the cURL going through, but no MR being generated.
What's more, the GL Runner does not output the response - so it's difficult for me to figure out exactly what the issue is.
I've tried escaping with back slashes, I've tried single quote setups with escaping the single quotes and doing inner double quotes with the ${} variables, but nothing.
I've validated that the variables being sent in from the yml exists (echoing them out before the curl call and the body setup), but I'm just lost at this point.
I've still yet to try JQ, but I'm not sure if the runner has it installed, but I'll find out when I return back to test.
Any help would be lovely.
General format of the scripts code:
#!/usr/bin/env bash
TARGET_BRANCH="test"
echo "Host: ${HOST}"
echo "Project Id: ${CI_PROJECT_ID}
echo "SOURCE BRANCH: ${CI_COMMIT_REF_NAME}
echo "Target Branch: ${TARGET_BRANCH}
BODY="{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}";
echo "Body: ${BODY}"
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data "${BODY}"
Now I want to note, that after testing this and seeing the curl go through, but no MR generated, I modified the --data value to go from ${BODY} to:
(assume to be valid value, just obscuring for reasons)
--data '{
"id": <x>,
"source_branch": "dev",
"target_branch": "test",
"remove_source_branch": false,
"title":"WIP: dev"
}'
And this worked! I the MR generated just fine.
So I I then tested by replacing the single quote with double quotes, and escaping the inner double quotes but still keeping the hardcoded values. Back to no MR being generated.
I'm quote stumped - I even went back to the BODY=... block, and tried to reformat that.
Then I went to the outer single, inner double, escaping
'{
"id": '"${CI_PROJECT_ID}"',
...
'}
and still nothing.
So I'm taking a break after all the attempts and speaking with some teammates to no avail, and throwing a hail marry Stack Overflow for some advice.
Added note:
This is not a typical application project repo, so any discussion on "this isn't a good standard practice, use issues as an anchor point for generating MRs" doesn't apply when the use of the repo will not be using issues, have minimum user hands on, etc.
I value those kinds of insights, but again, the usecase for this repo is not an actual "project", so to speak.
Thank you
I'm a bit confused about what it is you're asking. It sounds like you got it to work just fine by passing the json string directly to curl's data flag, and you're just wondering why the other attempts didn't work. If that's the case, here's what I see which would make your other attempts fail:
# ...
BODY="{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}";
echo "Body: ${BODY}"
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data "${BODY}"
In this first option, the variable looks just fine, but when you're using it in your curl command, you're quoting the variable (ie, "$BODY"). There's nothing wrong with this when you're doing concatenation, but here it's likely causing a syntax issue since when the variable is extracted, the result is a json string that's doubly quoted:
""{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}""
I would try passing the $BODY variable without any quotes:
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data $BODY
For the third attempt, I assume there's a typo in your question since you have what should be the final single-quote in front of the closing brace. The other issue is that you're terminating the single-quoted string in the middle of the json:
'{
"id": '"${CI_PROJECT_ID}"',
...
}'
Since this starts with a single-quote, the first single quote it sees will terminate the string. The first single quote is around "${CI_PROJECT_ID}"', so the string being passed to curl is:
'{
"id": '
To clarify why this curl command isn't throwing an exit code, this is still a legal string, and is perfectly fine to curl, so it sends it on to GitLab API. When parsing the data however, GitLab can't parse it, so nothing happens over there. It would result in a 4xx HTTP code however, but as you mentioned, you can't see some of that information in the job output.
To fix this, I'd remove the single quotes around $CI_PROJECT_ID:
'{
"id": "${CI_PROJECT_ID}",
...
}'
or, since $CI_PROJECT_ID is and always will be an integer, you can remove all the quotes around it:
'{
"id": $CI_PROJECT_ID,
...
}'
This question already has answers here:
How to pass a variable in a curl command in shell scripting
(5 answers)
Closed 1 year ago.
My script deals with browser automation using cURL. I need to POST user input username. I can do this by hard-coding the username like this:
# sid -> session Id
# eid -> element Id (input box here)
curl -d '{"value":["username"]}' http://localhost:9515/session/$sid/element/$eid/value
This successfully posts username but I want to read username from user and pass this to value.
I tried:
read userName
curl -d '{"value":[$userName]}' http://localhost:9515/session/$sid/element/$eid/value
This gives me "missing command parameter" error. Passing only $userName instead of [$userName] gives "invalid argument: 'value' must be a list"
How do I pass a variable (userName) to the curl POST request in this case?
It looks like you've tried to use the variable within single quotes. Try to use double quotes instead:
read userName
curl -d "{\"value\":\"${userName}\"}" http://localhost:9515/session/$sid/element/$eid/value
Explanation:
Single quotes (' ') operate similarly to double quotes, but do not permit referencing variables, since the special meaning of $ is turned off. Within single quotes, every special character except ' gets interpreted literally. Consider single quotes ("full quoting") to be a stricter method of quoting than double quotes ("partial quoting").
Taken from: https://tldp.org/LDP/abs/html/quotingvar.html
I have to insert many data in my application and through the graphical interface it takes many time. For this reason I want to create a bash script and make the requests through curl using the REST API (I have to manually specify the id).
The problem is that i get the error: The server refused this request because the request entity is in a format not supported by the requested resource for the requested method.
Here is the code
#!/bin/bash
for i in {1..1}
do
CURL='/usr/bin/curl -X POST'
RVMHTTP="http://192.168.1.101:8080/sitewhere/api/devices
-H 'accept:application/json'
-H 'content-type:application/json'
-H 'x-sitewhere-tenant:sitewhere1234567890'
--user admin:password"
DATA=" -d '{\"hardwareId":\"$i",\"siteToken\":\"4e6913db-c8d3-4e45-9436-f0a99b502d3c\",\"specificationToken\":\"82043707-9e3d-441f-bdcc-33cf0f4f7260\"}'"
# or you can redirect it into a file:
$CURL $RVMHTTP $DATA >> /home/bluedragon/Desktop/tokens
done
The format of my request has to be json
#!/usr/bin/env bash
rvmcurl() {
local url
url="http://192.168.1.101:8080/sitewhere/${1#/}"
shift || return # function should fail if we weren't passed at least one argument
curl -XPOST "${rvm_curl_args[#]}" "$url" "$#"
}
i=1 # for testing purposes
rvm_curl_args=(
-H 'accept:application/json'
-H 'content-type:application/json'
-H 'x-sitewhere-tenant:sitewhere1234567890'
--user admin:password
)
data=$(jq -n --arg hardwareId "$i" '
{
"hardwareId": $hardwareId,
"siteToken": "4e6913db-c8d3-4e45-9436-f0a99b502d3c",
"specializationToken": "82043707-9e3d-441f-bdcc-33cf0f4f7260"
}')
rvmcurl /api/devices -d "$data"
Note:
Commands, or command fragments intended to be parsed into multiple words, should never be stored in strings. Use an array or a function instead. Quotes inside such strings are not parsed as syntax, and instead (when parsed without eval, which carries its own serious risks and caveats) become literal values. See BashFAQ #50 for a full explanation.
Use a JSON-aware tool, such as jq, to ensure that generated data is legit JSON.
Fully-qualifying paths to binaries is, in general, an antipattern. It doesn't result in a significant performance gain (the shell caches PATH lookups), but it does reduce your scripts' portability and flexibility (preventing you from installing a wrapper for curl in your PATH, in an exported shell function, or otherwise).
All-caps variable names are in a namespace used for variables with meaning to the shell and operating system. Use names with at least one lowercase character for your own variables to prevent any chance of conflict.
I am trying to pass parameters to cURL through the command line, this way:
curl -s -X POST -H "Content-Type: text/xml" -H "Cache-Control: no-cache" -d '<Data Token="someToken" Name='"$appName"' ID='"$someVar"' ParseAppID='"$someVar"' ParseRESTKey='"$someVar"' AndroidPackage='"$someVar"' Version="1"></Data>' 'https://prefix.something.com/somePath?InputType=Xml'
(This line is actually extracted from the Postman app).
I Googled this issue and found whole lot of solutions that did not work for me (links are to SO past questions...):
I tried isolating the variables by ending the single quotes, this way: 'before...'"${someVar}"'...after...'. Could not complete the request.
I tried passing the variables using a file (-d #fileName). Failed to post.
I tried replacing the single quotes surrounding the <Data> tokens with double quotes - but the command apparently cannot accept such substitution.
The errors I get are either <Error></Error> or The server encountered an error and could not complete your request.
Is there any chance that there exists some other solution?
Has anyone encoutered such problem before?
I would be greatful for any help.
You aren't supplying quotes around the value of ID like you are for Name. That is, you need
'<Data Token="someToken" Name="'"$appName"'" ...>'
^^^
|||
||+- shell quote to protect $appName
|+- shell quote enclosing the XML
+- literal quote embedded in the XML
which results in the string (assuming appName=foo)
<Data Token="someToken" Name="foo" ...>
I am trying to fetch data from this page using wget and curl in PHP. As you can see by using your browser, the default result is 20 items but by setting the GET parameter iip to number x, I can fetch x items, i.e. http:// www.example.com/ foo ?a=26033&b=100
The problem is that the iip parameter only works in browsers. If I try to fetch the last link using wget or curl, only 20 items are returned. Why? Try this at the command-line:
curl -O http://www.example.com/foo?a=26033&iip=b
wget http://www.example.com/foo?a=26033&iip=b
Why can't I use the GET parameter iip?
Try adding quotes:
curl -O 'http://www.objektvision.se/annonsorer?ai=26033&iip=100'
wget 'http://www.objektvision.se/annonsorer?ai=26033&iip=100'
The & has special functionality on the command line which is likely causing the issues.
Try quoting the argument. At least in cmd, & is used to delimit two commands that are run individually.
You'll have to enclose your URL in either " or ', since the & has a special meaning in shellscript... That'll give you:
curl -O "http://www.objektvision.se/annonsorer?ai=26033&iip=100"
wget "http://www.objektvision.se/annonsorer?ai=26033&iip=100"
& is a reserved word in shell. Just escape it like this \&