I am trying to run a RTC 4.x command to add components to a workspace. The list of components have spaces in the names so they need to be surrounded by quotes. I am storing this list in a simple string variable:
COMPONENTS="\"TestComp\" \"Common Component\""
When I just echo out COMPONENTS it displays correctly, but when I use it in a scm command odd things happen to the quotes. I am running this in Jenkins so I can get some additional output, but the same thing happens when I run it on the command line so this is not a Jenkins issue.
From the console log:
+ COMPONENTS='"TestComp" "Common Component"'
+ echo '"TestComp"' '"Common' 'Component"'
"TestComp" "Common Component"
The command is trying to run the following:
+ scm workspace add-components TEST_Workspace -s Test_Stream '"TestComp"' '"Common' 'Component"'
Which produces:
Problem running 'workspace add-components':
Unmatched component ""Common".
Typically, you need to use an array to store items that may themselves contain whitespace:
components=("TestComp" "Common Component")
scm workspace add-components TEST_Workspace -s Test_Stream "${components[#]}"
Quoting an array expansion indexed with # produces a sequence of words, one per element of the array, rather than a single word.
Related
I have a Bash script in which I call rsync in order to perform a backup to a remote server. To specify that my Downloads folder be backed up, I'm passing "'${HOME}/Downloads'" as an argument to rsync which produces the output:
rsync -avu '/Volumes/Norman Data/Downloads' me#example.com:backup/
Running the command with the variable expanded as above (through the terminal or in the script) works fine, but because of the space in the expanded variable and the fact that the quotes (single ticks) are ignored when included in the variable being passed as part of an argument (see here), the only way I can get it not to choke on the space is to do:
stmt="rsync -avu '${HOME}/Downloads' me#examle.com:backup/"
eval ${stmt}
It seems like there would be some vulnerabilities presented by running eval on anything not 100% private to that script. Am I correct in thinking I should be doing it a different way? If so, any hints for a bash-script-beginner would be greatly appreciated.
** EDIT ** - I actually have a bit more involved use case than. the example above. For the paths passed, I have an array of them, each containing spaces, that I'm then combining into 1 string kind of like
include_paths=(
"'${HOME}/dir_a'"
"'${HOME}/dir_b' --exclude=video"
)
for item in "${include_paths[#]}"
do
inc_args="${inc_args}" ${item}
done
inc_args evaluates to '/Volumes/Norman Data/me/dir_a' '/Volumes/Norman Data/me/dir_b' --exclude=video
which I then try to pass as an argument to rsync but the single ticks are read as literals and it breaks after the 1st /Volumes/Norman because of the space.
rsync -avu "${inc_args}" me#example.com:backup/
Using eval seems to read the single ticks as quotes and executes:
rsync -avu '/Volumes/Norman Data/me/dir_a' '/Volumes/Norman Data/me/dir_b' --exclude=video me#example.com:backup/
like I need it to. I can't seem to get any other way to work.
** EDIT 2 - SOLUTION **
So the 1st thing I needed to do was modify the include_paths array to:
remove single ticks from within double quoted items
move any path-specific flags (ex. --exclude) to their own items directly after the path it should apply to
I then built up an array containing the rsync command and its options, added the expanded include_paths and exclude_paths arrays and the connection string to the remote host.
And finally expanded that array, which ran my entire, properly quoted rsync command. In the end the modified array include_paths is:
include_paths=(
"${HOME}/dir_a"
"${HOME}/dir_b"
"--exclude=video"
"${HOME}/dir_c"
)
and I put everything together with:
cmd=(rsync -auvzP)
for item in "${exclude_paths[#]}"
do
cmd+=("--exclude=${item}")
done
for item in "${include_paths[#]}"
do
cmd+=("${item}")
done
cmd+=("me#example.com:backup/")
set -x
"${cmd[#]}"
Use an array for the commands/option instead of a plain variable.
stmt=(rsync -avu "${HOME}/Dowloads" me#example.com:backup/)
Execute it using the builtin command
command "${stmt[#]}"
...Or I personally just put the options/arguments in an array.
options=(-avu "${HOME}/Download" me#example.com:backup/)
The execute it using rsync
rsync "${options[#]}"
If you have newer version of bash which that supports the additional P.E. parameter expansion, then you could probably quote the array.
options=(-avu "${HOME}/Download" me#example.com:backup/)
Check the output by applying the P.E.
echo "${options[#]#Q}"
Should print
'-avu' '/Volumes/Norman Data/Downloads' 'me#examle.com:backup/'
Then you can just
rsync "${options[#]#Q}"
I am trying to run 'make' command from my bash script to build the code.
I can see that all parameters got assigned and able to display the command that i am trying to run. I could not see any issue with the command. But the issue is when it tries to run the command via bash script it fails.
My command is :- ./build_script LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm"
Script Snippet of parsing all the arguments and constructing make command:-
makeargs=""
for build_flag do
if [[ "$build_flag " = "PROJECTS="* ]]; then
apps =`echo $build_flag |sed "s/PROJECTS=//" `
makeargs="$makeargs PROJECTS=\"$apps \""
else
makeargs="$makeargs $build_flag"
fi
done
echo "make DCOV=1 $makeargs $maketest"
make DCOV=1 $makeargs $maketest
When i run the script, I can see the build command has constructed properly.
Output :-
make DCOV=1 LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm" run_all
GNUmakefile:16: warning: overriding commands for target `/'
GNUmakefile:19: warning: ignoring old commands for target `/'
make: *** No rule to make target `bfm"'. Stop.
I try to print PROJECTS variable in my 'GNUmakefile' present in build_main folder. I can see the output : PROJECTS is "bfm . It is not taking whole "cuda bfm" as a whole string.
Note:- When i try to run the same build command :- make DCOV=1 LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm" run_all explicitly it works fine.
Seems like issue with Interpreting variables with makefile.
Any solution for this ? Please help.
Thanks!
Change makeargs string to array before passing it as an arguments group.
eval makeargs_array=( $makeargs )
make UVC=1 "${makeargs_array[#]}" $maketest
Without converting to array, if you enable debug, it shows last line interpretation as
make DCOV=1 LIC=1 DOCUMNETS=1 'PROJECTS="cuda' bfm '"'
Which is clearly ignoring double-quote and considering space as separator.
Even double-quote is getting passed as a separate argument in this case.
Explanation:
Word-splitting
It says,
The shell scans the results of parameter expansion, command
substitution, and arithmetic expansion that did not occur within
double quotes for word splitting.
If we use "$makeargs" i.e. surrounded by double-quote, it is not considered by word-splitting and results in LIC=1 DOCUMNETS=1 "PROJECTS=cuda bfm"
But again its a complete string, while requirement is to split the string to use as arguments.
So now using $makeargs.
Word-splitting gets in action as per the default IFS (space, tab, newline), we get result as LIC=1 DOCUMNETS=1 PROJECTS="cuda bfm "
Double-quoted part of string didn't affect the word-splitting since, subject to splitting is complete string here.
Why array worked here?
Array itself expands each element as separate word when using # and here no further word-splitting requires after expansion.
Arrays
I'm using a bash script to make changes to an SQL database. One of the values i'm updating uses dollar signs. The current value being something like "$$$$$" and i need to change it to "$$$$$$$$$$". However, a a $ in a bash script is used for variables.
How can i allow this small section of my bash script to used a $ as a normal character?
function prep() {
DATE_STAMP=$(date +%m%d%Y)
log "Changing mask to 10 characters"
log "$(/opt/CPU/bin/connx-query -q "update TYPE set TYPE.MASK = '$$$$$$$$$$'")"
}
As it stands right now, its just replacing each dollar sign with some random number found earlier in my script.
Bash provides different types of quoting, each with different rules about substitution (single quote ', double quote ", here document/string <<<"string" and and $'.
The double quote (used in the log ... update) will enable variable substitution, replacing each pair of $$ with the current shell PID (looks like random number).
Few options:
Consider quoting each '$' to prevent expansion
log "$(/opt/CPU/bin/connx-query -q "update TYPE set TYPE.MASK = '\$\$\$\$\$\$\$\$\$\$'")"
Over thought my own question. I can just escape the $. '\$\$\$\$\$\$\$\$\$\$'
I'm using jamplus to build a vendor's cross-platform project. On osx, the C tool's command line (fed via clang to ld) is too long.
Response files are the classic answer to command lines that are too long: jamplus states in the manual that one can generate them on the fly.
The example in the manual looks like this:
actions response C++
{
$(C++) ##(-filelist #($(2)))
}
Almost there! If I specifically blow out the C.Link command, like this:
actions response C.Link
{
"$(C.LINK)" $(LINKFLAGS) -o $(<[1]:C) -Wl,-filelist,#($(2:TC)) $(NEEDLIBS:TC) $(LINKLIBS:TC))
}
in my jamfile, I get the command line I need that passes through to the linker, but the response file isn't newline terminated, so link fails (osx ld requires newline-separated entries).
Is there a way to expand a jamplus list joined with newlines? I've tried using the join expansion $(LIST:TCJ=\n) without luck. $(LIST:TCJ=#(\n)) doesn't work either. If I can do this, the generated file would hopefully be correct.
If not, what jamplus code can I use to override the link command for clang, and generate the contents on the fly from a list? I'm looking for the least invasive way of handling this - ideally, modifying/overriding the tool directly, instead of adding new indirect targets wherever a link is required - since it's our vendor's codebase, as little edit as possible is desired.
The syntax you are looking for is:
newLine = "
" ;
actions response C.Link
{
"$(C.LINK)" $(LINKFLAGS) -o $(<[1]:C) -Wl,-filelist,#($(2:TCJ=$(newLine))) $(NEEDLIBS:TC) $(LINKLIBS:TC))
}
To be clear (I'm not sure how StackOverflow will format the above), the newLine variable should be defined by typing:
newLine = "" ;
And then placing the carat between the two quotes and hitting enter. You can use this same technique for certain other characters, i.e.
tab = " " ;
Again, start with newLine = "" and then place carat between the quotes and hit tab. In the above it is actually 4 spaces which is wrong, but hopefully you get the idea. Another useful one to have is:
dollar = "$" ;
The last one is useful as $ is used to specify variables typically, so having a dollar variable is useful when you actually want to specify a dollar literal. For what it is worth, the Jambase I am using (the one that ships with the JamPlus I am using), has this:
SPACE = " " ;
TAB = " " ;
NEWLINE = "
" ;
Around line 28...
I gave up on trying to use escaped newlines and other language-specific characters within string joins. Maybe there's an awesome way to do that, that was too thorny to discover.
Use a multi-step shell command with multiple temp files.
For jamplus (and maybe other jam variants), the section of the actions response {} between the curly braces becomes an inline shell script. And the response file syntax #(<value>) returns a filename that can be assigned within the shell script, with the contents set to <value>.
Thus, code like:
actions response C.Link
{
_RESP1=#($(2:TCJ=#)#$(NEEDLIBS:TCJ=#)#$(LINKLIBS:TCJ=#))
_RESP2=#()
perl -pe "s/[#]/\n/g" < $_RESP1 > $_RESP2
"$(C.LINK)" $(LINKFLAGS) -o $(<[1]:C) -Wl,-filelist,$_RESP2
}
creates a pair of temp files, assigned to shell variable names _RESP1 and _RESP2. File at path _RESP1 is assigned the contents of the expanded sequence joined with a # character. Search and replace is done with a perl one liner into _RESP2. And link proceeds as planned, and jamplus cleans up the intermediate files.
I wasn't able to do this with characters like :;\n, but # worked as long as it had no adjacent whitespace. Not completely satisfied, but moving on.
I read the following command from the batch file to run Maven on Windows mvn.bat:
if not "_%M2_HOME:~-1%"=="_\" goto checkMBat
And
if "%#eval[2+2]" == "4" goto 4NTArgs
What does this batch script mean?
ADD 1
As I tried, it seems _%M2_HOME:~-1% returns the _ plus the last 1 letter of the environment variable "_%M2_HOME%. But what's the name of this syntax?
%VAR:~-1% gets the last character in the envvar. The first snippet verifies that the envvar M2_HOME doesn't end with \. Note: Maven's docs say,
Note: For Maven 2.0.9, also be sure that the M2_HOME doesn't have a '\' as last character.
This might be related. They probably want to prepend M2_HOME to subdir names and always include a dirsep. The variable substitution in "_%...%" is unaffected by the initial underscore. Experessing it that way just ensures that the underscore is at the beginning of the output. I can't say for certain, but it may have been expressed that way to avoid a backslashed quote, e.g. "\".
The second is not any CMD/batch that I'm familiar with. The comment (assuming this comes from mvn.bat) says "4NT shell", which I take to mean that this batch file could be run in the Take Command Console which probably has extensions to MS CMD features. For example, %#eval[...] probably does numeric evaluation in 4NT. This would effectively be a check to see if the script were running in a 4NT shell.
The first one takes the last character of %M2_HOME%, adds an underscore to the front, and checks to see if the resulting string is _\ - in short, it checks that the last character of %M2_HOME% is a backslash by using substrings.
The second one is how you determine if 4NT is installed on your computer; if it is, there will be a variable function called #eval.
I found the explanation to "_%M2_HOME:~-1%" below link. It's a variable substring operation.
http://ss64.com/nt/syntax-substring.html