var with quotes inside var with quotes [duplicate] - bash

This question already has answers here:
Building command strings using variables with various quote levels and spaces
(3 answers)
Closed 4 years ago.
I want to set my user agent from a string I've got in a variable, however there are some other options as well that I want to pass in the same string, so this is what I ended up with:
cookie="-b cookie -c cookie"
agent="My Bot"
opt="-A \"$agent\" $cookie"
curl http://example.com $opt
When I run my script it fetches the site but it doesn't set the whole user agent but just the part before the first space and then goes
curl: (6) Could not resolve host: Bot
I'm guessing there is something messing up with the quotes, but if I replace curl with echo this is what I see, which seems pretty accurate to me
http://example.com -A "My Bot" -b cookie -c cookie
What am I doing wrong?

To handle whitespace correctly store arguments in arrays, not strings.
cookie=(-b cookie -c cookie)
agent="My Bot"
opt=(-A "$agent" "${cookie[#]}")
curl http://example.com "${opt[#]}"
I'm guessing there is something messing up with the quotes, but if I replace curl with echo this is what I see, which seems pretty accurate to me.
echo is misleading. If you want to be absolutely sure, use set -x.
$ set -x
$ curl http://example.com $opt
+ curl http://example.com -A '"My' 'Bot"' -b cookie -c cookie
See how it turned "My Bot" into two arguments, '"My' and 'Bot"'?

Related

Generate GitLab Merge Request when a push comes in via CI/CD Pipeline using Bash and cURL

So, I've been beating my head around this for a few days and could use some help.
Initially, I found the gitlab blog post from 2017: https://about.gitlab.com/blog/2017/09/05/how-to-automatically-create-a-new-mr-on-gitlab-with-gitlab-ci/
However it seems like a lot of it is outdated(?) but I was able to use POSTMAN to carve down a working model (using token as a parameter instead of a header, etc.)
Here's the thing though, I cannot get the Merge Request to trigger and I believe it's due to --data (-d) not interpolating the variables.
I've hardcoded the payload with the values exactly, and pushed to my branch and it generated the MR just fine.
But any method I use to try and incorporate variables will result in the cURL going through, but no MR being generated.
What's more, the GL Runner does not output the response - so it's difficult for me to figure out exactly what the issue is.
I've tried escaping with back slashes, I've tried single quote setups with escaping the single quotes and doing inner double quotes with the ${} variables, but nothing.
I've validated that the variables being sent in from the yml exists (echoing them out before the curl call and the body setup), but I'm just lost at this point.
I've still yet to try JQ, but I'm not sure if the runner has it installed, but I'll find out when I return back to test.
Any help would be lovely.
General format of the scripts code:
#!/usr/bin/env bash
TARGET_BRANCH="test"
echo "Host: ${HOST}"
echo "Project Id: ${CI_PROJECT_ID}
echo "SOURCE BRANCH: ${CI_COMMIT_REF_NAME}
echo "Target Branch: ${TARGET_BRANCH}
BODY="{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}";
echo "Body: ${BODY}"
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data "${BODY}"
Now I want to note, that after testing this and seeing the curl go through, but no MR generated, I modified the --data value to go from ${BODY} to:
(assume to be valid value, just obscuring for reasons)
--data '{
"id": <x>,
"source_branch": "dev",
"target_branch": "test",
"remove_source_branch": false,
"title":"WIP: dev"
}'
And this worked! I the MR generated just fine.
So I I then tested by replacing the single quote with double quotes, and escaping the inner double quotes but still keeping the hardcoded values. Back to no MR being generated.
I'm quote stumped - I even went back to the BODY=... block, and tried to reformat that.
Then I went to the outer single, inner double, escaping
'{
"id": '"${CI_PROJECT_ID}"',
...
'}
and still nothing.
So I'm taking a break after all the attempts and speaking with some teammates to no avail, and throwing a hail marry Stack Overflow for some advice.
Added note:
This is not a typical application project repo, so any discussion on "this isn't a good standard practice, use issues as an anchor point for generating MRs" doesn't apply when the use of the repo will not be using issues, have minimum user hands on, etc.
I value those kinds of insights, but again, the usecase for this repo is not an actual "project", so to speak.
Thank you
I'm a bit confused about what it is you're asking. It sounds like you got it to work just fine by passing the json string directly to curl's data flag, and you're just wondering why the other attempts didn't work. If that's the case, here's what I see which would make your other attempts fail:
# ...
BODY="{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}";
echo "Body: ${BODY}"
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data "${BODY}"
In this first option, the variable looks just fine, but when you're using it in your curl command, you're quoting the variable (ie, "$BODY"). There's nothing wrong with this when you're doing concatenation, but here it's likely causing a syntax issue since when the variable is extracted, the result is a json string that's doubly quoted:
""{
\"id\": ${CI_PROJECT_ID},
\"source_branch\": \"${CI_COMMIT_REF_NAME}\",
\"target_branch\": \"${TARGET_BRANCH}\",
\"remove_source_branch\": false,
\"title\": \"WIP: ${CI_COMMIT_REF_NAME}\",
}""
I would try passing the $BODY variable without any quotes:
# Create MR
curl -X POST https://${HOST}/api/v4/projects/${CI_PROJECT_ID}/merge_requests?private_token${PRIVATE_TOKEN}"\
--header "PRIVATE_TOKEN: ${PRIVATE_TOKEN}"\
--header "Content-Type: application/json"\
--data $BODY
For the third attempt, I assume there's a typo in your question since you have what should be the final single-quote in front of the closing brace. The other issue is that you're terminating the single-quoted string in the middle of the json:
'{
"id": '"${CI_PROJECT_ID}"',
...
}'
Since this starts with a single-quote, the first single quote it sees will terminate the string. The first single quote is around "${CI_PROJECT_ID}"', so the string being passed to curl is:
'{
"id": '
To clarify why this curl command isn't throwing an exit code, this is still a legal string, and is perfectly fine to curl, so it sends it on to GitLab API. When parsing the data however, GitLab can't parse it, so nothing happens over there. It would result in a 4xx HTTP code however, but as you mentioned, you can't see some of that information in the job output.
To fix this, I'd remove the single quotes around $CI_PROJECT_ID:
'{
"id": "${CI_PROJECT_ID}",
...
}'
or, since $CI_PROJECT_ID is and always will be an integer, you can remove all the quotes around it:
'{
"id": $CI_PROJECT_ID,
...
}'

Can variables be passed inside arguments in Bash [duplicate]

This question already has answers here:
How to pass a variable in a curl command in shell scripting
(5 answers)
Closed 1 year ago.
My script deals with browser automation using cURL. I need to POST user input username. I can do this by hard-coding the username like this:
# sid -> session Id
# eid -> element Id (input box here)
curl -d '{"value":["username"]}' http://localhost:9515/session/$sid/element/$eid/value
This successfully posts username but I want to read username from user and pass this to value.
I tried:
read userName
curl -d '{"value":[$userName]}' http://localhost:9515/session/$sid/element/$eid/value
This gives me "missing command parameter" error. Passing only $userName instead of [$userName] gives "invalid argument: 'value' must be a list"
How do I pass a variable (userName) to the curl POST request in this case?
It looks like you've tried to use the variable within single quotes. Try to use double quotes instead:
read userName
curl -d "{\"value\":\"${userName}\"}" http://localhost:9515/session/$sid/element/$eid/value
Explanation:
Single quotes (' ') operate similarly to double quotes, but do not permit referencing variables, since the special meaning of $ is turned off. Within single quotes, every special character except ' gets interpreted literally. Consider single quotes ("full quoting") to be a stricter method of quoting than double quotes ("partial quoting").
Taken from: https://tldp.org/LDP/abs/html/quotingvar.html

Strange characters appearing in bash variable expansion

Trying to do the following on contos7 works as I expect:
pod_in_question=$(curl -u uname:password -k very.cluster.com/api/v1/namespaces/default/pods/ | grep -i '"name": "myapp-' | cut -d '"' -f 4)
echo "$pod_in_question"
curl -u uname:password -k -X DELETE "very.cluster.com/api/v1/namespaces/default/pods/${pod_in_question}"
However, trying the same thing on MacOS (10.12.1) yields:
curl: (3) [globbing] bad range in column 92
When I try to curl the last line with a -g option it substitutes with a malformed name such as: myapp-\\x1b[m\\x1b[Kl1eti\
The echo statement would always execute just fine and show something like myapp-v7454 which I later want to put into the last curl statement. So where are these other characters coming from?
A robust solution - Basic cURL CLI debugging.
This answer is revised after it's been identified that the issue for the OP relates to curl applying color output.
There's a proposed answer which explains clearly what the embedded special characters meant, and instructions to override the grep behaviour to not output color. Certainly this is a good practise for grep use in piping. There are however a number of best practises that can help diagnose this or a similar issue with cURL and ultimately lead to the most robust solution.
Re-creating the problem
Assuming it's a JSON Content-Type, we use echo {'"name": "myapp-7414"'} to simulate the output from cURL
We filter the text and set a variable with it that we use in a cURL command
We force grep to output color, since it doesn't normally by default when outputting to a tty.
Recreation:
myvar=$(echo {'"name": "myapp-7414"'} | grep --color=always -i '"name": "myapp-' | cut -d '"' -f 4)
curl "https://www.google.com/${myvar}"
Output:
curl: (3) [globbing] bad range in column 32
First up:
'{}' are special characters to cURL, period.
The best practise for URL syntax in cURL:
If Variable Expansion is required:
Apply the -g switch to disable potential globbing done by cURL
Otherwise:
Use $variable as part of a "quoted" url string, instead of ${variable}
Second: In addition to -g, we add --libcurl /tmp/libcurl so we can get some insight into what cURL is seeing.
   Recreation with -g and --libcurl:
curl -g --libcurl /tmp/libcurl "https://www.google.com/${myvar}"
Output:
<p>Your client has issued a malformed or illegal request <ins>That’s all we know.
Perfect, at least now everything is getting to the server and back! Let's see what cURL sent out to the server:
cat /tmp/libcurl
Surely enough we find this line: (note the bold part).
curl_easy_setopt(hnd, CURLOPT_URL, "https://www.google.com/myapp-\033[m7414");
So we know that:
The shell is doing something strange with our variable.
cURL knows not to try glob once we send the -g switch. That way - If there is an error with the shell variable, we can actually see what it is. We shouldn't be debugging a globbing error if we're not trying to use URL Ranges.
The special characters are colors. They represent the --color=always that we added to simulate the OPs environment.
At this point. Since it looks like we're working with JSON data, why not just use a widely available, high performance JSON parsing tool. That has a number of benefits, including:
Not relying on any environment that could affect string filtering
Can request the data we want (aka. "name")
The app name "myapp" can change and we won't have to re-write the code to retrieve it.
It's cleaner and accounts for things I haven't considered yet.
If we used jq for example (while we're at it, we don't need the -g switch because we don't need '{}' for the variable because we're already double " the URL):
myvar=$(echo {'"name": "myapp-7414"'} | jq -r .name)
curl --libcurl /tmp/libcurl "https://www.google.com/$myvar"
Now we get:
<p>The requested URL /myapp-7414 was not found on this server. That’s all we know.
Great. It's all working now. It should be obvious that the test URL here being www.google.com is obviously not going to know was myapp-7414 was.
So we've gone from :
Globbing bad range, to:
Malformed URL, to:
URL not found on server.
We could also as suggested elsewhere change the grep output and override it to --color=never (As I have noted: If grep has to be used, the --color=never is a great way to use it as a best practise when piping strings, period.). However, given the portability issues already experienced because of string filtering, and the fact that we are already handed structured data on a plate that can be parsed reliably, the more robust solution would be to do just that, if possible.
The substitution you showed at the last part looks like one of your calls injected ANSI escape sequences. It's possible that grep isn't detecting non-TTY output and is colorizing.
On a terminal that supports ANSI escape sequences, your particular codes might not be visible. The codes ^E[m^E[K set the screen mode and clear the current line. That's why you thought the echo command proved your data was correct.
You can examine the raw data with:
echo "$pod_in_question" | hexdump -C
And you should see there are other characters in there which did not appear in your terminal before. When you put these "invisible" codes into the URL, curl tries to encode them and then fails when it encounters a control character (ESC).
The solution is to add the argument --color=never to your grep call, which will disable colorization.

Pass variable number of arguments to command [duplicate]

This question already has answers here:
Why does shell ignore quoting characters in arguments passed to it through variables? [duplicate]
(3 answers)
Closed 6 years ago.
I'm using FPM to create Debian packages, and I've run into a little problem.
My bash script takes 5 arguments.
TARGET=$1
VERSION=$2
DESCRIPTION=$3
DEPENDENCIES=$4
REVISION=$5
The troublesome one is $4, where I pass it the following string
-d "apt-transport-https > 0.8.16~exp12ubuntu10.15" -d "mongodb > 1:2.0.4-1ubuntu2" -d "ntp > 1:4.2.6.p3+dfsg-1ubuntu3.1"
The double quotes are escaped in my string, so when I echo $DEPENDENCIES, the quotes show up correctly.
FPM uses the -d flag can be used multiple times, and I need to be able to pass a list of parameters from my script to fpm.
I would like to do something like:
fpm ...blah blah details... $DEPENDENCIES path
$DEPENDENCIES should pass the multiple flags to fpm, but it only seems to recognize the first one. If I write the arguments out manually, it works fine, but trying to use a string to pass the parameters doesn't work.
I'm not sure what's up. Help?
This will be helpful reading: http://mywiki.wooledge.org/BashFAQ/050
If I were you, I would rearrange your parameters
#!/bin/bash
TARGET=$1
VERSION=$2
DESCRIPTION=$3
REVISION=$4
shift 4
DEPENDENCIES=( "$#" )
Dependencies is "all the rest" of the arguments, properly separated. You will invoke your script like this
./script tgt ver "this is the description" rev-1.1.1 -d "apt-transport-https > 0.8.16~exp12ubuntu10.15" -d "mongodb > 1:2.0.4-1ubuntu2" -d "ntp > 1:4.2.6.p3+dfsg-1ubuntu3.1"
In your script, the DEPENDENCIES array will contain these elements
DEPENDENCIES[0]=-d
DEPENDENCIES[1]="apt-transport-https > 0.8.16~exp12ubuntu10.15"
DEPENDENCIES[2]=-d
DEPENDENCIES[3]="mongodb > 1:2.0.4-1ubuntu2"
DEPENDENCIES[4]=-d
DEPENDENCIES[5]="ntp > 1:4.2.6.p3+dfsg-1ubuntu3.1"
In your script, call fpm like:
fpm ...blah blah details... "${DEPENDENCIES[#]}" path

curl to compile a list of redirected pages

Suppose I have a bash script that goes through a file that contains a list of old URLs that have all been redirected.
curl --location http://destination.com will process a page by following a redirect. However, I'm interested not in the content, but on where the redirect points so that I can update my records.
What is the command-line option for curl to output what that new location for the URL is?
You wou want to leave out the --location/-L flag, and use -w, checking the redirect_url variable. curl -w "%{redirect_url}" http://someurl.com should do it.
Used in a script:
REDIRECT=`curl -w "%{redirect_url}" http://someurl.com`
echo "http://someurl.com redirects to: ${REDIRECT}"
From the curl man page:
-w, --write-out <format>
Make curl display information on stdout after a completed transfer. The
format is a string that may contain plain text mixed with any number
of variables. The format can be specified as a literal "string", or
you can have curl read the format from a file with "#filename" and to
tell curl to read the format from stdin you write "#-".
The variables present in the output format will be substituted by the
value or text that curl thinks fit, as described below. All variables
are specified as %{variable_name} and to output a normal % you just
write them as %%. You can output a newline by using \n, a carriage
return with \r and a tab space with \t.
NOTE: The %-symbol is a special symbol in the win32-environment, where
all occurrences of % must be doubled when using this option.
The variables available are:
...
redirect_url When an HTTP request was made without -L to follow
redirects, this variable will show the actual URL a redirect would
take you to. (Added in 7.18.2)
...
This might work (as a starting point)
curl -sI google.com | head -1 | grep 301 | wc -l
man curl
then
search redirect_url
redirect_url When a HTTP request was made without -L to follow
redirects, this variable will show the actual URL a redirect would
take you to. (Added
in 7.18.2)
the variable above is for -w/--write-out <format>

Resources