I am trying to conditionally add arguments to a docker call in a bash script but docker says the flag is unknown, but I can run the command verbatim by hand and it works.
I have tried a few strategies to add the command, including using a string instead of an array, and I have tried using a substitution like the solution here ( using ${array[#]/#/'--pull '} ): https://stackoverflow.com/a/68675860/10542275
docker run --name application --pull "always" -p 3000:3000 -d private.docker.repository/group/application:version
This bash script
run() {
getDockerImageName "/group" "$PROJECT_NAME:$VERSION" "latest";
local imageName=${imageName};
local additionalRunParameters=${additionalRunParameters};
cd "$BASE_PATH/$PROJECT_NAME" || exit 1;
stopAnyRunning "$PROJECT_NAME";
echo docker run --name "$PROJECT_NAME" \
"${additionalRunParameters[#]}" \
-p 3000:3000 \
-d "$imageName";
// docker run --name application --pull "always" -p 3000:3000 -d private.docker.repository/group/application:version
docker run --name "$PROJECT_NAME" \
"${additionalRunParameters[#]}" \
-p 3000:3000 \
-d "$imageName";
//unknown flag: --pull "always"
}
The helper 'getDockerImageName'
# Gets the name of the docker image to use for deploy.
# $1 - The path to the image in the container registry
# $2 - The name of the image and the tag
# $3 - 'latest' if the deploy should use the container built by CI
export imageName="";
export additionalRunParameters=();
getDockerImageName() {
imageName="group/$2";
if [[ $3 == "latest" ]]; then
echo "Using docker image from CI...";
docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "https://$DOCKER_BASE_URL";
imageName="${DOCKER_BASE_URL}${1}/$2";
additionalRunParameters=('--pull "always"');
fi
}
Don't put code (such as arguments) in a variable. Basically, use an array is good, and you are almost doing that. This line -
local additionalRunParameters=${additionalRunParameters};
is probably what's causing you trouble, along with
additionalRunParameters=('--pull "always"');
which is embedding the spaces between what you seem to have meant to be two operands (the option and its argument), turning them into a single string that is an unrecognized garble. //unknown flag: --pull "always" is telling you the flag it's parsing is --pull "always", which is NOT the --pull flag docker DOES know, followed by an argument.
Also,
export additionalRunParameters=(); # nope
arrays don't really export. Take that out, it will only confuse someone.
A much simplified set of examples:
$: declare -f a b c
a ()
{
foo=('--pull "always"') # single value array
}
b ()
{
echo "1: \${foo}='${foo}' <= scalar, returns first element of the array";
echo "2: \"\${foo[#]}\"='${foo[#]}' <= returns entire array (be sure to put in quotes)";
echo "3: \"\${foo[1]}\"='${foo[1]}' <= indexed, returns only second element of array"
}
c ()
{
foo=(--pull "always") # two values in this array
}
$: a # sets ${foo[0]} to '--pull "always"'
$: b
1: ${foo}='--pull "always"' <= scalar, returns first element of the array
2: "${foo[#]}"='--pull "always"' <= returns entire array (be sure to put in quotes)
3: "${foo[1]}"='' <= indexed, returns only second element of array
$: c # setd ${foo[0]} to '--pull' and ${foo[1]} to "always"
$: b
1: ${foo}='--pull' <= scalar, returns first element of the array
2: "${foo[#]}"='--pull always' <= returns entire array (be sure to put in quotes)
3: "${foo[1]}"='always' <= indexed, returns only second element of array
So what you need is:
getDockerImageName(){
. . . # stuff
additionalRunParameters=( --pull "always" ); # no single quotes
}
and just take OUT
local additionalRunParameters=${additionalRunParameters}; # it's global, no need
You have one more issue though - "${additionalRunParameters[#]}" \ is good, as long as the array isn't empty. In your example it will apparently always be loaded with the same values, so I don't see why you are adding all this extra complication of putting it into a global array that gets loaded incidentally in another function... seems like an antipattern. Just put the arguments you are universally enforcing anyway on the command itself.
However, on the possibility that you simplified some details out, then if this array is ever empty it's going to pass a literal quoted empty string as an argument on the command line, and you're likely to get something like the following error:
$: docker run --name foo "" -p 3000:3000 -d bar # docker doesn't understand the empty string
docker: invalid reference format.
Maybe, rather than
additionalRunParameters=( --pull "always" ); # no single quotes
and
"${additionalRunParameters[#]}" \
what you really want is
pull_always=true
and
${pull_always:+ --pull always } \
...with no quotes, so that if the var has been set (with anything) it evaluates to the desired result, but if it's unset and unquoted it evaluates to nothing and gets ignored, as nothing actually gets passed in - not even a null string.
Good luck. Let us know if you need more help.
Related
sample code below
psql -h $host -U postgres -d postgres -At -c "select partner_country_id as country , case when (threshold is null) then global_threshold else threshold end as threshold from ra_country_day_threshold " \
| while read -a Record
do
arrIN=(${Record[0]//|/ })
col1=${arrIN[0]}
col2=${arrIN[1]}
country_array["$col1"]="$col2"
echo "Col1:$col1 Col2:$col2"
done
echo "Elements:${country_array[#]}"
echo "length: ${#country_array[#]}"
Result
empty elements and length 0
The answer is simple, while command create a subprocess with its own context, if you create a new variable in that context, it will not be accessible outside of it.
Meaning the variable will not be accessible when you are outside the loop.
My suggestion is that you store the result inside a temporary file that will be available within all your script, then outside your loop, read that file.
I have a bash script where i'm trying to call a curl which is having a variable value as input. When trying to execute the bash script the variable value is not getting expanded in double quotes.
Expected curl in script after variable expansion should be as following:
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id="fgsd34sff334" '
Getting executed as follows when observed in debug mode:
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=\""$idp_sub"\" '
My bash script is as follows:
#!/bin/bash
idp_sub=""
for idp_sub in $(cat /opt/SP/jboss/home/mayur/es_idp_sub.txt)
do
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=\""$idp_sub"\" ' -o /opt/SP/jboss/home/mayur/es_delete_response.txt
done
How does do i expand the variable value within double quotes as shown above in expected output ?
Your double-quoted string is inside single quotes, where it won't be expanded.
Compare:
foo=bar
echo 'foo=\""$foo\"'
echo 'foo="'"$foo"'"'
In the second example, we end the single quotes, and double-quote $foo, then start new single quotes for the final '.
It's probably easier to read if we expand using printf instead:
printf 'foo=%s\n' "$foo"
That's something you might want to run as a process substitution.
BUT...
This is a wrong and dangerous way to construct an SQL query (and the web server is also poor, if it forwards arbitrary queries - I hope it has no write permissions to the data). Read about "SQL command injection" and come back to this code when you understand the issues.
Nothing inside single quotes will be expanded by bash, including any double-quotes, and variable names. The good news is you can end your single-quoted section and immediately start a double-quoted section to introduce the variable, and it will all be concatenated into a single argument for the application (curl). Try:
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=\"'"$idp_sub"'\" ' -o /opt/SP/jboss/home/mayur/es_delete_response.txt
You can make your code strongly injection-proof by rejecting any string containing a double-quote, but you might reject some strings that have been legitimately escaped.
If you can use the q syntax to quote the string, you can make it more injection-proof, but I guess the attacker just has to inject ]":
/usr/bin/curl -s -vvvv http://hmvddrsvr:8044/query/service -u iamusr:pssd -d 'statement=DELETE FROM `test_bucket` WHERE type = "Metadata" AND market = "ES" AND status = "active" AND meta(test_bucket).id=q\"['"$idp_sub"]'\" ' -o /opt/SP/jboss/home/mayur/es_delete_response.txt
You could then search for and reject the pattern string ]" as your anti-injection, which will allow a much wider class of legitimate strings. You would have to tell the users that you have applied q[] quoting to their input, so they don't have to.
I am trying to access an env variable in Jenkins pipeline and want to use it in a Shell Script executing in the same pipeline but a differnt step,
pipeline {
agent any
tools {
maven 'M2'
}
environment {
stable_revision = sh(script: 'curl -H "Authorization: Basic $base64encoded" "https://xyz.sad" | jq -r "name"', returnStdout: true)
}
stages {
stage('Initial-Checks') {
steps {
echo "Stable Revision: ${env.stable_revision}" //displays the value
bat "sh && sh undeploy.sh"
}}
...
}}
This is a sample Shell script, it has many lines, but I have an issue in only accessing the above stable_revision variable,
#!/bin/bash
echo xyz = ${stable_revision} #### this gives the right value xyz = 22
echo xyz2 = ${stable_revision}/d ### here after the value the /d is written in new line
For example, let's say the stable_revision value is 22, then in the SH script echo I am getting the value as,
xyz2 = 22
/d
I want the value to be xyz2 = 22/d
You can use .trim() to strip off a trailing newline.
environment {
stable_revision = sh(script: 'curl -H "Authorization: Basic $base64encoded" "https://xyz.sad" | jq -r "name"', returnStdout: true).trim()
}
returnStdout (optional):
If checked, standard output from the task is returned as the step value as a String, rather than being printed
to the build log. (Standard error, if any, will still be printed to
the log.) You will often want to call .trim() on the result to strip
off a trailing newline.
https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#sh-shell-script
If you use bash instead of sh for your commands, you can benefit from Bash's built-in string transformations
Here it trims all trailing characters from the [:space:] class witch includes actual spaces and newlines.
echo "xyz2 = ${stable_revision%%[[:space:]]}/d"
If $stable_revision is always an integer, you can force the shell to use it like an integer with:
echo "xyz2 = $((stable_revision))/d"
If you are sure that $stable_revision contains no space, you can force the shell to trim all spaces by using it like a table element:
sr=($stable_revision); echo "xyz2 = ${sr[0]}/d"
You can also use the automatic trimming of a sub-shell returned value, that would trim any leading, trailing and duplicate spaces in-between:
echo "xyz2 = $(echo ${stable_revision})/d"`
Say I have two bash functions:
dock() { sudo docker $# ;}
and
dock-ip() { sudo docker inspect --format '{{ .NetworkSettings.IPAddress }}' $# ;}
How to get bash auto-completion working with the second function?
With the first one, it is as easy as adding:
_completion_loader docker; complete -F _docker dock
This will not work for the second one. The autocomplete source for Docker is in /usr/share/bash-completion/completions/docker on Debian Stretch. I have more functions like dock-run, dock-exec, etc. so I don't want to write a custom completion function for each of them.
Also, complete -F _docker_container_inspect dock-ip only partially works; tab only lists containers, not completes partial strings.
Research:
How do I autocomplete nested, multi-level subcommands? <-- needs custom functions
https://superuser.com/questions/436314/how-can-i-get-bash-to-perform-tab-completion-for-my-aliases <-- automated for top commands only
With a combined hour of bash completion experience, I took apart the docker completion script (/usr/share/bash-completion/completions/docker) and the bash_completion.sh script to come up with a wrapper function:
# Usage:
# docker_alias_completion_wrapper <completion function> <alias/function name>
#
# Example:
# dock-ip() { docker inspect --format '{{ .NetworkSettings.IPAddress }}' $# ;}
# docker_alias_completion_wrapper __docker_complete_containers_running dock-ip
function docker_alias_completion_wrapper {
local completion_function="$1";
local alias_name="$2";
local func=$(cat <<EOT
# Generate a new completion function name
function _$alias_name() {
# Start off like _docker()
local previous_extglob_setting=\$(shopt -p extglob);
shopt -s extglob;
# Populate \$cur, \$prev, \$words, \$cword
_get_comp_words_by_ref -n : cur prev words cword;
# Declare and execute
declare -F $completion_function >/dev/null && $completion_function;
eval "\$previous_extglob_setting";
return 0;
};
EOT
);
eval "$func";
# Register the alias completion function
complete -F _$alias_name $alias_name
}
export -f docker_alias_completion_wrapper
I then created my alias/functions like this:
# Get container IP
dock-ip() { docker inspect --format '{{ .NetworkSettings.IPAddress }}' $# ;}
docker_alias_completion_wrapper __docker_complete_containers_running dock-ip
# Execute interactive container
dock-exec() { docker exec -i -t --privileged $# ;}
docker_alias_completion_wrapper __docker_complete_containers_all dock-exec
...
Be sure to call _completion_loader docker; at the top of your profile aliases script to load the main Docker completion scripts. I invite more skilled bash programmers to improve this answer, please.
Consider this PS1
PS1='\n${_:+$? }$ '
Here is the result of a few commands
$ [ 2 = 2 ]
0 $ [ 2 = 3 ]
1 $
1 $
The first line shows no status as expected, and the next two lines show the
correct exit code. However on line 3 only Enter was pressed, so I would like the
status to go away, like line 1. How can I do this?
Here's a funny, very simple possibility: it uses the \# escape sequence of PS1 together with parameter expansions (and the way Bash expands its prompt).
The escape sequence \# expands to the command number of the command to be executed. This is incremented each time a command has actually been executed. Try it:
$ PS1='\# $ '
2 $ echo hello
hello
3 $ # this is a comment
3 $
3 $ echo hello
hello
4 $
Now, each time a prompt is to be displayed, Bash first expands the escape sequences found in PS1, then (provided the shell option promptvars is set, which is the default), this string is expanded via parameter expansion, command substitution, arithmetic expansion, and quote removal.
The trick is then to have an array that will have the k-th field set (to the empty string) whenever the (k-1)-th command is executed. Then, using appropriate parameter expansions, we'll be able to detect when these fields are set and to display the return code of the previous command if the field isn't set. If you want to call this array __cmdnbary, just do:
PS1='\n${__cmdnbary[\#]-$? }${__cmdnbary[\#]=}\$ '
Look:
$ PS1='\n${__cmdnbary[\#]-$? }${__cmdnbary[\#]=}\$ '
0 $ [ 2 = 3 ]
1 $
$ # it seems that it works
$ echo "it works"
it works
0 $
To qualify for the shortest answer challenge:
PS1='\n${a[\#]-$? }${a[\#]=}$ '
that's 31 characters.
Don't use this, of course, as a is a too trivial name; also, \$ might be better than $.
Seems you don't like that the initial prompt is 0 $; you can very easily modify this by initializing the array __cmdnbary appropriately: you'll put this somewhere in your configuration file:
__cmdnbary=( '' '' ) # Initialize the field 1!
PS1='\n${__cmdnbary[\#]-$? }${__cmdnbary[\#]=}\$ '
Got some time to play around this weekend. Looking at my earlier answer (not-good) and other answers I think this may be probably the smallest answer.
Place these lines at the end of your ~/.bash_profile:
PS1='$_ret$ '
trapDbg() {
local c="$BASH_COMMAND"
[[ "$c" != "pc" ]] && export _cmd="$c"
}
pc() {
local r=$?
trap "" DEBUG
[[ -n "$_cmd" ]] && _ret="$r " || _ret=""
export _ret
export _cmd=
trap 'trapDbg' DEBUG
}
export PROMPT_COMMAND=pc
trap 'trapDbg' DEBUG
Then open a new terminal and note this desired behavior on BASH prompt:
$ uname
Darwin
0 $
$
$
$ date
Sun Dec 14 05:59:03 EST 2014
0 $
$
$ [ 1 = 2 ]
1 $
$
$ ls 123
ls: cannot access 123: No such file or directory
2 $
$
Explanation:
This is based on trap 'handler' DEBUG and PROMPT_COMMAND hooks.
PS1 is using a variable _ret i.e. PS1='$_ret$ '.
trap command runs only when a command is executed but PROMPT_COMMAND is run even when an empty enter is pressed.
trap command sets a variable _cmd to the actually executed command using BASH internal var BASH_COMMAND.
PROMPT_COMMAND hook sets _ret to "$? " if _cmd is non-empty otherwise sets _ret to "". Finally it resets _cmd var to empty state.
The variable HISTCMD is updated every time a new command is executed. Unfortunately, the value is masked during the execution of PROMPT_COMMAND (I suppose for reasons related to not having history messed up with things which happen in the prompt command). The workaround I came up with is kind of messy, but it seems to work in my limited testing.
# This only works if the prompt has a prefix
# which is displayed before the status code field.
# Fortunately, in this case, there is one.
# Maybe use a no-op prefix in the worst case (!)
PS1_base=$'\n'
# Functions for PROMPT_COMMAND
PS1_update_HISTCMD () {
# If HISTCONTROL contains "ignoredups" or "ignoreboth", this breaks.
# We should not change it programmatically
# (think principle of least astonishment etc)
# but we can always gripe.
case :$HISTCONTROL: in
*:ignoredups:* | *:ignoreboth:* )
echo "PS1_update_HISTCMD(): HISTCONTROL contains 'ignoredups' or 'ignoreboth'" >&2
echo "PS1_update_HISTCMD(): Warning: Please remove this setting." >&2 ;;
esac
# PS1_HISTCMD needs to contain the old value of PS1_HISTCMD2 (a copy of HISTCMD)
PS1_HISTCMD=${PS1_HISTCMD2:-$PS1_HISTCMD}
# PS1_HISTCMD2 needs to be unset for the next prompt to trigger properly
unset PS1_HISTCMD2
}
PROMPT_COMMAND=PS1_update_HISTCMD
# Finally, the actual prompt:
PS1='${PS1_base#foo${PS1_HISTCMD2:=${HISTCMD%$PS1_HISTCMD}}}${_:+${PS1_HISTCMD2:+$? }}$ '
The logic in the prompt is roughly as follows:
${PS1_base#foo...}
This displays the prefix. The stuff in #... is useful only for its side effects. We want to do some variable manipulation without having the values of the variables display, so we hide them in a string substitution. (This will display odd and possibly spectacular things if the value of PS1_base ever happens to begin with foo followed by the current command history index.)
${PS1_HISTCMD2:=...}
This assigns a value to PS1_HISTCMD2 (if it is unset, which we have made sure it is). The substitution would nominally also expand to the new value, but we have hidden it in a ${var#subst} as explained above.
${HISTCMD%$PS1_HISTCMD}
We assign either the value of HISTCMD (when a new entry in the command history is being made, i.e. we are executing a new command) or an empty string (when the command is empty) to PS1_HISTCMD2. This works by trimming off the value HISTCMD any match on PS1_HISTCMD (using the ${var%subst} suffix replacement syntax).
${_:+...}
This is from the question. It will expand to ... something if the value of $_ is set and nonempty (which it is when a command is being executed, but not e.g. if we are performing a variable assignment). The "something" should be the status code (and a space, for legibility) if PS1_HISTCMD2 is nonempty.
${PS1_HISTCMD2:+$? }
There.
'$ '
This is just the actual prompt suffix, as in the original question.
So the key parts are the variables PS1_HISTCMD which remembers the previous value of HISTCMD, and the variable PS1_HISTCMD2 which captures the value of HISTCMD so it can be accessed from within PROMPT_COMMAND, but needs to be unset in the PROMPT_COMMAND so that the ${PS1_HISTCMD2:=...} assignment will fire again the next time the prompt is displayed.
I fiddled for a bit with trying to hide the output from ${PS1_HISTCMD2:=...} but then realized that there is in fact something we want to display anyhow, so just piggyback on that. You can't have a completely empty PS1_base because the shell apparently notices, and does not even attempt to perform a substitution when there is no value; but perhaps you can come up with a dummy value (a no-op escape sequence, perhaps?) if you have nothing else you want to display. Or maybe this could be refactored to run with a suffix instead; but that is probably going to be trickier still.
In response to Anubhava's "smallest answer" challenge, here is the code without comments or error checking.
PS1_base=$'\n'
PS1_update_HISTCMD () { PS1_HISTCMD=${PS1_HISTCMD2:-$PS1_HISTCMD}; unset PS1_HISTCMD2; }
PROMPT_COMMAND=PS1_update_HISTCMD
PS1='${PS1_base#foo${PS1_HISTCMD2:=${HISTCMD%$PS1_HISTCMD}}}${_:+${PS1_HISTCMD2:+$? }}$ '
This is probably not the best way to do this, but it seems to be working
function pc {
foo=$_
fc -l > /tmp/new
if cmp -s /tmp/{new,old} || test -z "$foo"
then
PS1='\n$ '
else
PS1='\n$? $ '
fi
cp /tmp/{new,old}
}
PROMPT_COMMAND=pc
Result
$ [ 2 = 2 ]
0 $ [ 2 = 3 ]
1 $
$
I need to use great script bash-preexec.sh.
Although I don't like external dependencies, this was the only thing to help me avoid to have 1 in $? after just pressing enter without running any command.
This goes to your ~/.bashrc:
__prompt_command() {
local exit="$?"
PS1='\u#\h: \w \$ '
[ -n "$LASTCMD" -a "$exit" != "0" ] && PS1='['${red}$exit$clear"] $PS1"
}
PROMPT_COMMAND=__prompt_command
[-f ~/.bash-preexec.sh ] && . ~/.bash-preexec.sh
preexec() { LASTCMD="$1"; }
UPDATE: later I was able to find a solution without dependency on .bash-preexec.sh.