From the windows command line, I can successfully call my script as follows:
python spot_check.py "stop|CHST SQ_ARRIVAL|2.3" "stop|14 ST_ARRIVAL|2.6" "19:06:28" "19:15:00"
However, if I want to use the VS Code debugger, and I pass the same arguments using the args attribute in launch.json
"args": [
"stop|CHST SQ_ARRIVAL|2.3",
"stop|14 ST_ARRIVAL|2.6" ,
"19:06:28",
"19:15:00",
]
Then I get the following error:
(base) c:\Users\1266143\Desktop\stringlines_ml>cd c:\Users\1266143\Desktop\stringlines_ml && cmd /C "set "PYTHONIOENCODING=UTF-8" && set "PYTHONUNBUFFERED=1" && C:\Users\1266143\AppData\Local\Continuum\anaconda3\python.exe c:\Users\1266143\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\ptvsd_launcher.py --default --client --host localhost --port 61850 c:\Users\1266143\Desktop\stringlines_ml\spot_check.py "stop|CHST SQ_ARRIVAL|2.3" "stop|14 ST_ARRIVAL|2.6" 19:06:28 19:15:00"
'CHST' is not recognized as an internal or external command,
operable program or batch file.
The part that reads 'CHST' is not recognized as an internal or external command, operable program or batch file. leads me to believe that the | is being interpreted as a redirect, rather than as a character in a string literal argument, and the space following CHST means CHST is being interpreted as a command. But why would these arguments evaluate differently on the command line than in Visual Studio? How can I ensure that these arguments are passed correctly to my command line application when in debug mode?
These aren't the quotes you're looking for
You require quotes around your arguments, as shown when running the script/program directly on the command-line (i.e. "stop|CHST SQ_ARRIVAL|2.3")
But in the JSON, the first set of quotes will get stripped off when the JSON is interpreted, so the string "stop|CHST SQ_ARRIVAL|2.3" in the JSON becomes just stop|CHST SQ_ARRIVAL|2.3 before it's fed to later processes.
Then all the arguments get fed to the Command Line or Python interpreter, which will look something like this (although it will likely be a huge line with a bunch of debugging flags and such):
c:/mypath/myfile stop|CHST SQ_ARRIVAL|2.3 stop|14 ST_ARRIVAL|2.6 19:06:28 19:15:00
The quotes you thought you had around the arguments no longer exist. This means that the parser interprets the vertical bar symbol as the "Pipe" command, which tell it that the first command is done, and it should take the output of that command and "pipe" it to the command that follows.
So, the parser thinks you told it to:
Run the command c:/mypath/myfile stop
Take the output of that command and pipe it to the command CHST SQ_ARRIVAL
Pipe the output of that command to the command 2.3 stop
etc.
Since it can't find the command CHST with the argument SQ_ARRIVAL, it gives you the error message you are seeing.
The fix is in
If you want the quotes to end up being passed along as a part of the argument you'll need to layer them. How to do this depends on how the JSON interpreter will handle multiple sets of quotes (I'm not sure how it does).
A few things to try:
Use triple quotes: """stop|CHST SQ_ARRIVAL|2.3""" - in some parsers, when it sees the first quote it starts a string, but if it sees 2 quotes in a row after that, it makes them into a quote inside the string, rather than ending it. So the first and last quote start and end the string, while the other two pairs of quotes will be condensed into a quote on the outside of your argument
Use a backslash in front of the quotes inside the JSON string: "\"stop|CHST SQ_ARRIVAL|2.3\"" - in many parsers the backslash character is an "escape" character and any character immediately after it is considered a string literal that will be put directly into the string, even if it is normally a special character.
Use single quotes inside the string: "'stop|CHST SQ_ARRIVAL|2.3'" - Since Python can use either single or double quotes as a string, normally any arguments going to a python interpreter with single quotes will also be considered a string. However, I'm not sure the arguments will get that far in this case, they will probably be interpreted by the shell first, which likely will not consider single quotes to be the start of a string (but you never know for sure..).
Which method works may depend on what shell you are using (i.e. Windows command prompt, Powershell, Git Bash, sh, c-sh, etc.). Each of them could handle command line interpretation of strings differently.
If none of these works, knowing the root cause, a further search should turn up the answer. Good luck!
Related
I am trying to give an argument to my python program through the terminal.
For this I am using the lines:
import sys
something = sys.argv[1]
I now try to put in a string like this through the bash terminal:
python my_script.py 2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
This returns a bash error because some of the characters in the string are bash special characters.
How can I use the string exactly as it is?
You can put the raw string into a file, for example like this, with cat and a here document.
cat <<'EOF' > file.txt
2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
EOF
and then run
python my_script.py "$(< file.txt)"
You can also use the text editor of your choice for the first step if you prefer that.
If this is a reoccurring task, which you have to perform from time to time, you can make your life easier with a little alias in your shell:
alias escape='read -r string ; printf "Copy this:\n%q\n" "${string}"'
It is using printf "%q" to escape your input string.
Run it like this:
escape
2m+{N7HiwH3[>!"4y?t9*y#;/$Ar3wF9+k$[3hK/WA=aMzF°L0PaZTM]t*P|I_AKAqIb0O4# cm=sl)WWYwEg10DDv%k/"c{LrS)oVd§4>8bs:;9u$ *W_SGk3CXe7hZMm$nXyhAuHDi-q+ug5+%ioou.,IhC]-_O§V]^,2q:VBVyTTD6'aNw9:oan(s2SzV
Copy this:
2m+\{N7HiwH3\[\>\!\"4y\?t9\*y#\;/\$Ar3wF9+k\$\[3hK/WA=aMzF°L0PaZTM\]t\*P\|I_AKAqIb0O4#\ cm=sl\)WWYwEg10DDv%k/\"c\{LrS\)oVd§4\>8bs:\;9u\$\ \*W_SGk3CXe7hZMm\$nXyhAuHDi-q+ug5+%ioou.\,IhC\]-_O§V\]\^\,2q:VBVyTTD6\'aNw9:oan\(s2SzV
You can use the escaped string directly in your shell, without additional quotes, like this:
python my_script.py 2m+\{N7HiwH3\[\>\!\"4y\?t9\*y#\;/\$Ar3wF9+k\$\[3hK/WA=aMzF°L0PaZTM\]t\*P\|I_AKAqIb0O4#\ cm=sl\)WWYwEg10DDv%k/\"c\{LrS\)oVd§4\>8bs:\;9u\$\ \*W_SGk3CXe7hZMm\$nXyhAuHDi-q+ug5+%ioou.\,IhC\]-_O§V\]\^\,2q:VBVyTTD6\'aNw9:oan\(s2SzV
In order to make life easier, shells like bash do a little bit of extra work to help users pass the correct arguments to the programs they instruct it to execute. This extra work usually results in predictable argument arrays getting passed to programs.
Oftentimes, though, this extra help results in unexpected arguments getting passed to programs; and sometimes results in the execution of undesired additional commands. In this case, though, it ended up causing Bash to emit an error.
In order to turn off this extra work, Bash allows users to indicate where arguments should begin and end by surrounding them by quotation marks. Bash supports both single quotes (') and double quotes (") to delimit arguments. As a last resort, if a string may contain single and double quotes (or double quotes are required but aren't aggressive enough), Bash allows you to indicate that a special- or whitespace-character should be part of the adjacent argument by preceding it with a backslash (\\).
If this method of escaping arguments is too cumbersome, it may be worth simplifying your program's interface by having it consume this data from a file instead of a command line argument. Another option is to create a program that loads the arguments from a more controlled location (like a file) and directly execs the target program with the desired argument array.
I am trying to study both bash and ruby and I am trying to pass a variable that contains a JSON object from ruby to be printed using a bash script.
I tried storing the JSON object in a variable then used that variable as an argument for my bash script but I am not getting the result that I wanted.
so in my ruby code I have this:
param = request.body
cmd = "ruby_to_bash #{param}"
exec cmd
and in my bash script, I am simply outputting the value of the argument:
echo $1
For example I have this JSON:
{"first":"one","second":"two","third":"three"}
my code only gives me this output:
{first:
I want to display the whole JSON object. How can I do this? Any kind of help will be much appreciated.
Both bash and ruby treat double quotes in a kinda special way: double-quoted strings are interpolated before passed to the receiver. I strongly encourage you to start with learning somewhat one in the first place instead of doing zero progress in both due to induced errors.
Answering the question stated, one must escape double quotes when passing back and forth since both ruby and bash treat them as content delimiters and discard when the content is already parsed in the resulting string.
This would work:
exec %|ruby_to_bash #{param.gsub('"', '\"')}|
I use %| and | as string delimiters to avoid clashes against inner quotes.
Sidenote: the above won’t work for the input containing spaces, but I purposely avoid to show how to deal with spaces since it leads to the dead end; once handled spaces with surrounding the interpolated param with single quotes, we are prone to be screwed up with inner single quotes in JSON object and so forth.
The code above is not someone should ever produce in production.
I think your best bet is, as usual in such cases, not to involve a shell at all.
When you say this:
cmd = "ruby_to_bash #{param}"
exec cmd
Your exec call is invoking a shell which parses cmd and then runs ruby_to_bash so you're doing this:
Build a command line.
Replace the current process with /bin/sh.
/bin/sh parses the command line that you should mashed together. This is where you need to worry about quoting and escaping to get past the shell.
/bin/sh exec's ruby_to_bash.
You could bypass the whole problem by using the multi-argument form of Kernel#exec:
exec(cmdname, arg1, ...)
command name and one or more arguments (no shell)
which doesn't bother with a shell at all. If you say:
exec 'ruby_to_bash', param
You won't involve a shell and the quotes or spacing in param won't matter to anyone other than ruby_to_bash.
I ham having difficulty understanding how to pass a variable to a ./configure command that includes variable expansion and quotes.
myvars.cfg
myFolderA="/home/myPrefix"
myFolderB="/home/stuffB"
myFolderC="/home/stuffC"
optsA="--prefix=${myFolderA}"
optsB="CPPFLAGS=\"-I${myFolderB} -I${myFolderC}\""
cmd="/home/prog/"
myScript.sh
#!/bin/bash
. /home/myvars.cfg
doCmd=("$cmd/configure" "${optsA}" "${optsB}")
${doCmd[#]}
The doCmd should look like this
/home/prog/configure --prefix=/home/myPrefix CPPFLAGS="-I/home/stuffB -I/home/stuffC"
however it seems when running bash it is adding single quotes
/home/prog/configure --prefix=/home/myPrefix 'CPPFLAGS="-I/home/stuffB' '-I/home/stuffC"'
causing an error of
configure: error: unrecognized option: `-I/home/stuffC"'
Is there a way to pass a variable that needs top be expanded and contains double quotes?
As your script is written, there is no point to using the doCmd array. You could simply write the command:
"$cmd/configure" "${optsA}" "${optsB}"
Or, more simply:
"$cmd/configure" "$optsA" "$optsB"
However, it is possible that you've simplified the script in a way which hides the need for the array. In any case, if you use the array, you need to ensure that its elements are not word-split and filepath expanded, so you must quote its expansion:
"${doCmd[#]}"
Also, you need to get rid of the quotes in optsB. You don't want to pass
CPPFLAGS="-I/home/stuffB -I/home/stuffC"
to the configure script. You want to pass what the shell would pass if you typed the above string. And what the shell would pass would be a single command-line argument with a space in it, looking like this:
CPPFLAGS=-I/home/stuffB -I/home/stuffC
In order to get that into optsB, you just write:
optsB="CPPFLAGS=-I${myFolderB} -I${myFolderC}"
Finally, the shell is not "adding single quotes" into the command line. It is showing you a form of the command whch you could type at the command-line. Since the argument (incorrectly) contains a quote symbol, the shell shows you the command with its arguments skingle-quoted, so that you can see that the optB has been (incorrectly) split into two arguments, each of which contains (incorrectly) one double quote.
You could have found much of the above and more by pasting your script into https://shellcheck.net. As the bash tag summary suggests, you should always try that before asking a shell question here because a lot of the time, it will solve your problem instantly.
I run with the file with command line arguments:
samplebash.bsh fakeusername fakepassword&123
.bsh file:
echo "Beginning script..."
argUsername='$1'
argPassword='$2'
protractor indv.js --params.login.username=$argUsername --params.login.password=$argPassword
Output:
Beginning script...
123: command not found
The Issue: For some reason, it interprets what follows the & symbol from the password as a command, how do I avoid this?
The problem isn't happening in your script, it's happening in your original command line. & is a command terminator, which specifies that the command before it should be executed in the background. So your command was equivalent to:
samplebash.bsh fakeusername fakepassword &
123
You need to quote the argument to prevent special characters from being interpreted by the shell.
samplebash.bsh fakeusername 'fakepassword&123'
Also, you shouldn't put single quotes around a variable like you do in your assignments, that prevents the variable from being expanded. So it should be:
argUsername=$1
argPassword=$2
And you should put double quotes around the variables when you use them in the command, to prevent wildcards and whitespace from being interpreted.
protractor indv.js --params.login.username="$argUsername" --params.login.password="$argPassword"
As a general rule, you should always put double quotes around variables unless you know they're not needed.
I run the following shell command:
nvim +CheckHealth +'w ~/Desktop/file.txt' +qall
This calls nvim (Neovim) and tells it to run three commands in succession:
CheckHealth to verify common errors. It runs in a buffer.
w ~/Desktop/file.txt to save that same buffer to a file.
qall to close all buffers.
I’m trying to run this from ruby, using system. If I run it as a single argument, it works fine:
system("nvim +CheckHealth +'w ~/Desktop/file.txt' +qall")
However, it fails (it runs but does not output a file) if ran as multiple arguments:
system("nvim", "+CheckHealth", "+'w ~/Desktop/file.txt'", "+qall")
What am I doing wrong? Note I am not asking for workarounds. I have a workaround, which is to run it as a single argument. My question is why doesn’t it work when ran as multiple arguments? What am I misunderstanding about system?
When you use the single argument version of system:
system("nvim +CheckHealth +'w ~/Desktop/file.txt' +qall")
You're launching a shell and handing it that whole string:
nvim +CheckHealth +'w ~/Desktop/file.txt' +qall
to execute. That means that everything in that string will be interpreted by the shell; in particular, the shell will be handling the single quotes in +'w ~/Desktop/file.txt' and by the time vim gets to parse its argument list, it sees three arguments that look like this:
+CheckHealth
+w ~/Desktop/file.txt
+qall
In the multi-argument version of system:
system("nvim", "+CheckHealth", "+'w ~/Desktop/file.txt'", "+qall")
no shell will be launched (a good thing as you don't have to worry about shell command injection and escaping) so the single quotes in the +w argument won't be removed by the shell. That means that vim sees these arguments:
+CheckHealth
+'w ~/Desktop/file.txt'
+qall
Presumably vim isn't happy with the single quotes in the second argument.
Executive summary:
The single argument version of system uses a shell to parse the command line, the multi-argument version doesn't use a shell at all.
The single quotes in +'w ~/Desktop/file.txt' are there to keep the shell from treating that as two arguments, they're not there for vim.
If you're using the multi-argument version of system (which you should be doing), then you'd say:
system("nvim", "+CheckHealth", "+w ~/Desktop/file.txt", "+qall")
and not have to worry about quoting and escaping things to get past the shell.