I'm using Paramiko's exec_command() to run wp-cli commands on a remote server. Any command argument with spaces in it (no matter how I quote it) gives me a "Too many positional arguments" error. Here's sample code:
sshClient.exec_command("wp option update blogname \"Text with spaces\" 2>&1")
I've ssh'd into the remote machine from my local terminal and the command (wp option update blogname "Text with spaces") works fine there. I've also tried using combinations of double and single quotes.
Basically it's like the inner quotes are being ignored entirely and "Text with spaces" is being seen as 3 additional arguments. Is this about how Python parses strings with quotes, or about paramiko's exec_command()? What am I missing?
So this ended up being an issue with how the shell handles quotes in a command. Thanks to Martin for his comment pointing me in the right direction. Here is what finally worked for me:
sshClient.exec_command("wp option update blogname '\"Text with spaces\"' 2>&1")
After some Googling, I found this guide very helpful in understanding what was going on:
Shell Command Line Quoting Mechanisms
My understanding of the issue is that the shell on the remote machine where the command is being run strips out quotes, and always uses spaces as separators for arguments. So in order for "Text with spaces" to be interpreted by the remote shell as a single argument, you have to wrap it in single quotes, and escaped double quotes.
With some deeper understanding of how the shell interprets commands it makes more sense, but at face value it's not intuitive at all.
Related
We have an application that keeps some info in an encrypted file. To edit the file we have to put the text editor name in an environment variable in bash, for example, EDITOR=vi. Then we run the application and it opens the decrypted file in vi. I am trying to come up with a bash script that updates the encrypted file. The only solution that I can think of is passing sed command instead of vi to the EDITOR variable. It works perfectly for something like EDITOR='sed -i s#aaaa#bbbb#'.
Problem starts when I need space and regular expression. For example: EDITOR='sed -i -r "s#^(\s*masterkey: )(.*)#\1xxxxx#"' which return error. I tried running the EDITOR in bash with $EDITOR test.txt and I can see the problem. It doesn't like double quotes and space between them so I added a backslash before the double quotes and \s instead of space. Now it says unterminated address regex. For several hours I googled and couldn't find any solution. I tried replacing single quotes with double quotes and vice versa and everything that I could find on the internet but no luck.
How can I escape and which characters should I escape here?
Update:
Maybe if I explain the whole situation somebody could suggest an alternative solution. There is an application written by Ruby and it is inside a container. Ruby application has a secret_key_base for production and we supposed to change the key with EDITOR=vi rails credentials:edit --environment=production. I don't know Ruby and google did not return any ruby solution for automation so I could only think about sending sed instead of vi to Ruby.
How can I escape and which characters should I escape here?
That is not possible. Word splitting on the result of expansion cannot be escaped from inside the result of that expansion, it will always run. Note that filename expansion is also running over the result of the expansion.
Create an executable file with the script content and set EDITOR to it.
You could export a bash shell function, after some tries I got to:
myeditor() {
sed -i -E 's#^(\s*masterkey: )(.*)#\1xxxxx#' "$#"
}
export -f myeditor
EDITOR='bash -c "$#" _ myeditor'
From the windows command line, I can successfully call my script as follows:
python spot_check.py "stop|CHST SQ_ARRIVAL|2.3" "stop|14 ST_ARRIVAL|2.6" "19:06:28" "19:15:00"
However, if I want to use the VS Code debugger, and I pass the same arguments using the args attribute in launch.json
"args": [
"stop|CHST SQ_ARRIVAL|2.3",
"stop|14 ST_ARRIVAL|2.6" ,
"19:06:28",
"19:15:00",
]
Then I get the following error:
(base) c:\Users\1266143\Desktop\stringlines_ml>cd c:\Users\1266143\Desktop\stringlines_ml && cmd /C "set "PYTHONIOENCODING=UTF-8" && set "PYTHONUNBUFFERED=1" && C:\Users\1266143\AppData\Local\Continuum\anaconda3\python.exe c:\Users\1266143\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\ptvsd_launcher.py --default --client --host localhost --port 61850 c:\Users\1266143\Desktop\stringlines_ml\spot_check.py "stop|CHST SQ_ARRIVAL|2.3" "stop|14 ST_ARRIVAL|2.6" 19:06:28 19:15:00"
'CHST' is not recognized as an internal or external command,
operable program or batch file.
The part that reads 'CHST' is not recognized as an internal or external command, operable program or batch file. leads me to believe that the | is being interpreted as a redirect, rather than as a character in a string literal argument, and the space following CHST means CHST is being interpreted as a command. But why would these arguments evaluate differently on the command line than in Visual Studio? How can I ensure that these arguments are passed correctly to my command line application when in debug mode?
These aren't the quotes you're looking for
You require quotes around your arguments, as shown when running the script/program directly on the command-line (i.e. "stop|CHST SQ_ARRIVAL|2.3")
But in the JSON, the first set of quotes will get stripped off when the JSON is interpreted, so the string "stop|CHST SQ_ARRIVAL|2.3" in the JSON becomes just stop|CHST SQ_ARRIVAL|2.3 before it's fed to later processes.
Then all the arguments get fed to the Command Line or Python interpreter, which will look something like this (although it will likely be a huge line with a bunch of debugging flags and such):
c:/mypath/myfile stop|CHST SQ_ARRIVAL|2.3 stop|14 ST_ARRIVAL|2.6 19:06:28 19:15:00
The quotes you thought you had around the arguments no longer exist. This means that the parser interprets the vertical bar symbol as the "Pipe" command, which tell it that the first command is done, and it should take the output of that command and "pipe" it to the command that follows.
So, the parser thinks you told it to:
Run the command c:/mypath/myfile stop
Take the output of that command and pipe it to the command CHST SQ_ARRIVAL
Pipe the output of that command to the command 2.3 stop
etc.
Since it can't find the command CHST with the argument SQ_ARRIVAL, it gives you the error message you are seeing.
The fix is in
If you want the quotes to end up being passed along as a part of the argument you'll need to layer them. How to do this depends on how the JSON interpreter will handle multiple sets of quotes (I'm not sure how it does).
A few things to try:
Use triple quotes: """stop|CHST SQ_ARRIVAL|2.3""" - in some parsers, when it sees the first quote it starts a string, but if it sees 2 quotes in a row after that, it makes them into a quote inside the string, rather than ending it. So the first and last quote start and end the string, while the other two pairs of quotes will be condensed into a quote on the outside of your argument
Use a backslash in front of the quotes inside the JSON string: "\"stop|CHST SQ_ARRIVAL|2.3\"" - in many parsers the backslash character is an "escape" character and any character immediately after it is considered a string literal that will be put directly into the string, even if it is normally a special character.
Use single quotes inside the string: "'stop|CHST SQ_ARRIVAL|2.3'" - Since Python can use either single or double quotes as a string, normally any arguments going to a python interpreter with single quotes will also be considered a string. However, I'm not sure the arguments will get that far in this case, they will probably be interpreted by the shell first, which likely will not consider single quotes to be the start of a string (but you never know for sure..).
Which method works may depend on what shell you are using (i.e. Windows command prompt, Powershell, Git Bash, sh, c-sh, etc.). Each of them could handle command line interpretation of strings differently.
If none of these works, knowing the root cause, a further search should turn up the answer. Good luck!
I run with the file with command line arguments:
samplebash.bsh fakeusername fakepassword&123
.bsh file:
echo "Beginning script..."
argUsername='$1'
argPassword='$2'
protractor indv.js --params.login.username=$argUsername --params.login.password=$argPassword
Output:
Beginning script...
123: command not found
The Issue: For some reason, it interprets what follows the & symbol from the password as a command, how do I avoid this?
The problem isn't happening in your script, it's happening in your original command line. & is a command terminator, which specifies that the command before it should be executed in the background. So your command was equivalent to:
samplebash.bsh fakeusername fakepassword &
123
You need to quote the argument to prevent special characters from being interpreted by the shell.
samplebash.bsh fakeusername 'fakepassword&123'
Also, you shouldn't put single quotes around a variable like you do in your assignments, that prevents the variable from being expanded. So it should be:
argUsername=$1
argPassword=$2
And you should put double quotes around the variables when you use them in the command, to prevent wildcards and whitespace from being interpreted.
protractor indv.js --params.login.username="$argUsername" --params.login.password="$argPassword"
As a general rule, you should always put double quotes around variables unless you know they're not needed.
Brand new to KornShell (ksh). I have an absolutely painful and hopefully simple problem.
I need to run a script with a string argument that includes single quotations mark and a variable. In the shell, it looks like this:
run_script -x '*082512*'
And that works fine.
But once I try to run it in a script with a variable for the date, I can not produce those single quotes. I have made the "today's date" variable without problem:
today=$(date "+%m%d%y")
But the problem occurs when I try to run the script. I have tried every possible combination of quotes within quotes and escaping the single quotes etc. etc. to no avail.
Any insight regarding this issue would be appreciated.
commenting here because the formatting is better.
Did you try
today=$(date "+%m%d%y")
run_script -x '*'"$today"'*'
As Glenn Jackman points out, quote are only 'seen' at the first level of shell invocation, where you typed in your command.
If the above doesn't help you, consider editing your question to show exact use case, with exact error messages or other non--expected behavior.
Good luck.
Alright, I have a script that takes a few arguments, runs data, and then rsyncs the data out to a different server. The problem is that to run the data, I have to take one of the arguments and then run a report using it, which is very bash unfriendly (Ex. [3023.2<>1], [5111.3$]="5", etc).
So if I'm going to run the command, I need to put the argument in single quotes, which would then make the argument not be pulled into it.
Thus if I were to run the script...
arg1 = [5111.3$]="5"
runjob specfile.spx '$arg1'
This wouldn't work, but if I were to run it with double quotes, then there is a good chance that the argument that gets passed in will have double quotes. Any ideas on how to get around this?
Use single quotes around the value when you set it, then use double-quotes around the variable when you expand it:
$ arg1='[5111.3$]="5"'
$ echo "$arg1"
[5111.3$]="5"
Quote escapes. Try
[5111.3$]=\"5\"
The Advanced Scripting Guide has a good section on quoting.