I need to launch a system command with two specials characters inside
system "my command "#{$value command here}" other ARGS here"
i can't use directly
system "my command #{$value command here} other ARGS here"
I need to insert " character at the begining and the end of my value command.How i can do that using system ?
Whenever you're composing system commands you must do one of two things. Either escape the values using shellescape to be sure the shell parses them correctly:
require 'shellwords'
system "my command #{"$value command here".shellescape} other ARGS here"
Or you can split out the arguments separately so the shell doesn't need to interpret them:
system("my", "command", "$value command here", "other", "ARGS", "here")
The second form is usually better, everything's much more explicit and there's no chance of you forgetting to escape something.
Remember when running shell commands that contain user data can expose you to severe risk. You absolutely must escape anything and everything that's not fixed by you in advance to avoid a shell-level exploit. On top of that remember filenames can and will contain spaces, especially on files uploaded from Windows where Untitled (1).rtf is a very common name.
Related
I'm using Paramiko's exec_command() to run wp-cli commands on a remote server. Any command argument with spaces in it (no matter how I quote it) gives me a "Too many positional arguments" error. Here's sample code:
sshClient.exec_command("wp option update blogname \"Text with spaces\" 2>&1")
I've ssh'd into the remote machine from my local terminal and the command (wp option update blogname "Text with spaces") works fine there. I've also tried using combinations of double and single quotes.
Basically it's like the inner quotes are being ignored entirely and "Text with spaces" is being seen as 3 additional arguments. Is this about how Python parses strings with quotes, or about paramiko's exec_command()? What am I missing?
So this ended up being an issue with how the shell handles quotes in a command. Thanks to Martin for his comment pointing me in the right direction. Here is what finally worked for me:
sshClient.exec_command("wp option update blogname '\"Text with spaces\"' 2>&1")
After some Googling, I found this guide very helpful in understanding what was going on:
Shell Command Line Quoting Mechanisms
My understanding of the issue is that the shell on the remote machine where the command is being run strips out quotes, and always uses spaces as separators for arguments. So in order for "Text with spaces" to be interpreted by the remote shell as a single argument, you have to wrap it in single quotes, and escaped double quotes.
With some deeper understanding of how the shell interprets commands it makes more sense, but at face value it's not intuitive at all.
From the windows command line, I can successfully call my script as follows:
python spot_check.py "stop|CHST SQ_ARRIVAL|2.3" "stop|14 ST_ARRIVAL|2.6" "19:06:28" "19:15:00"
However, if I want to use the VS Code debugger, and I pass the same arguments using the args attribute in launch.json
"args": [
"stop|CHST SQ_ARRIVAL|2.3",
"stop|14 ST_ARRIVAL|2.6" ,
"19:06:28",
"19:15:00",
]
Then I get the following error:
(base) c:\Users\1266143\Desktop\stringlines_ml>cd c:\Users\1266143\Desktop\stringlines_ml && cmd /C "set "PYTHONIOENCODING=UTF-8" && set "PYTHONUNBUFFERED=1" && C:\Users\1266143\AppData\Local\Continuum\anaconda3\python.exe c:\Users\1266143\.vscode\extensions\ms-python.python-2019.11.50794\pythonFiles\ptvsd_launcher.py --default --client --host localhost --port 61850 c:\Users\1266143\Desktop\stringlines_ml\spot_check.py "stop|CHST SQ_ARRIVAL|2.3" "stop|14 ST_ARRIVAL|2.6" 19:06:28 19:15:00"
'CHST' is not recognized as an internal or external command,
operable program or batch file.
The part that reads 'CHST' is not recognized as an internal or external command, operable program or batch file. leads me to believe that the | is being interpreted as a redirect, rather than as a character in a string literal argument, and the space following CHST means CHST is being interpreted as a command. But why would these arguments evaluate differently on the command line than in Visual Studio? How can I ensure that these arguments are passed correctly to my command line application when in debug mode?
These aren't the quotes you're looking for
You require quotes around your arguments, as shown when running the script/program directly on the command-line (i.e. "stop|CHST SQ_ARRIVAL|2.3")
But in the JSON, the first set of quotes will get stripped off when the JSON is interpreted, so the string "stop|CHST SQ_ARRIVAL|2.3" in the JSON becomes just stop|CHST SQ_ARRIVAL|2.3 before it's fed to later processes.
Then all the arguments get fed to the Command Line or Python interpreter, which will look something like this (although it will likely be a huge line with a bunch of debugging flags and such):
c:/mypath/myfile stop|CHST SQ_ARRIVAL|2.3 stop|14 ST_ARRIVAL|2.6 19:06:28 19:15:00
The quotes you thought you had around the arguments no longer exist. This means that the parser interprets the vertical bar symbol as the "Pipe" command, which tell it that the first command is done, and it should take the output of that command and "pipe" it to the command that follows.
So, the parser thinks you told it to:
Run the command c:/mypath/myfile stop
Take the output of that command and pipe it to the command CHST SQ_ARRIVAL
Pipe the output of that command to the command 2.3 stop
etc.
Since it can't find the command CHST with the argument SQ_ARRIVAL, it gives you the error message you are seeing.
The fix is in
If you want the quotes to end up being passed along as a part of the argument you'll need to layer them. How to do this depends on how the JSON interpreter will handle multiple sets of quotes (I'm not sure how it does).
A few things to try:
Use triple quotes: """stop|CHST SQ_ARRIVAL|2.3""" - in some parsers, when it sees the first quote it starts a string, but if it sees 2 quotes in a row after that, it makes them into a quote inside the string, rather than ending it. So the first and last quote start and end the string, while the other two pairs of quotes will be condensed into a quote on the outside of your argument
Use a backslash in front of the quotes inside the JSON string: "\"stop|CHST SQ_ARRIVAL|2.3\"" - in many parsers the backslash character is an "escape" character and any character immediately after it is considered a string literal that will be put directly into the string, even if it is normally a special character.
Use single quotes inside the string: "'stop|CHST SQ_ARRIVAL|2.3'" - Since Python can use either single or double quotes as a string, normally any arguments going to a python interpreter with single quotes will also be considered a string. However, I'm not sure the arguments will get that far in this case, they will probably be interpreted by the shell first, which likely will not consider single quotes to be the start of a string (but you never know for sure..).
Which method works may depend on what shell you are using (i.e. Windows command prompt, Powershell, Git Bash, sh, c-sh, etc.). Each of them could handle command line interpretation of strings differently.
If none of these works, knowing the root cause, a further search should turn up the answer. Good luck!
Yesterday I asked a similar question about escaping double quotes in env variables, although It didn't solve my problem (Probably because I didn't explain good enough) so I would like to specify more.
I'm trying to run a script (Which I know is written in Perl), although I have to use it as a black box because of permissions issue (so I don't know how the script works). Lets call this script script_A.
I'm trying to run a basic command in Shell: script_A -arg "date time".
If I run from the command line, it's works fine, but If I try to use it from a bash script or perl scrip (for example using the system operator), it will take only the first part of the string which was sent as an argument. In other words, it will fail with the following error: '"date' is not valid..
Example to specify a little bit more:
If I run from the command line (works fine):
> script_A -arg "date time"
If I run from (for example) a Perl script (fails):
my $args = $ENV{SOME_ENV}; # Assume that SOME_ENV has '-arg "date time"'
my $cmd = "script_A $args";
system($cmd");
I think that the problem comes from the environment variable, but I can't use the one quote while defining the env variable. For example, I can't use the following method:
setenv SOME_ENV '-arg "date time"'
Because it fails with the following error: '"date' is not valid.".
Also, I tried to use the following method:
setenv SOME_ENV "-arg '"'date time'"'"
Although now the env variable will containe:
echo $SOME_ENV
> -arg 'date time' # should be -arg "date time"
Another note, using \" fails on Shell (tried it).
Any suggestions on how to locate the reason for the error and how to solve it?
The $args, obtained from %ENV as you show, is a string.
The problem is in what happens to that string as it is manipulated before arguments are passed to the program, which needs to receive strings -arg and date time
If the program is executed in a way that bypasses the shell, as your example is, then the whole -arg "date time" is passed to it as its first argument. This is clearly wrong as the program expects -arg and then another string for its value (date time)
If the program were executed via the shell, what happens when there are shell metacharacters in the command line (not in your example), then the shell would break the string into words, except for the quoted part; this is how it works from the command line. That can be enforced with
system('/bin/tcsh', '-c', $cmd);
This is the most straightforward fix but I can't honestly recommend to involve the shell just for arguments parsing. Also, you are then in the game of layered quoting and escaping, what can get rather involved and tricky. For one, if things aren't right the shell may end up breaking the command into words -arg, "date, time"
How you set the environment variable works
> setenv SOME_ENV '-arg "date time"'
> perl -wE'say $ENV{SOME_ENV}' #--> -arg "date time" (so it works)
what I believe has always worked this way in [t]csh.
Then, in a Perl script: parse this string into -arg and date time strings, and have the program is executed in a way that bypasses the shell (if shell isn't used by the command)
my #args = $ENV{SOME_ENV} =~ /(\S+)\s+"([^"]+)"/; #"
my #cmd = ('script_A', #args);
system(#cmd) == 0 or die "Error with system(#cmd): $?";
This assumes that SOME_ENV's first word is always the option's name (-arg) and that all the rest is always the option's value, under quotes. The regex extracts the first word, as consecutive non-space characters, and after spaces everything in quotes.† These are program's arguments.
In the system LIST form the program that is the first element of the list is executed without using a shell, and the remaining elements are passed to it as arguments. Please see system for more on this, and also for basics of how to investigate failure by looking into $? variable.
It is in principle advisable to run external commands without the shell. However, if your command needs the shell then make sure that the string is escaped just right to to preserve quotes.
Note that there are modules that make it easier to use external commands. A few, from simple to complex: IPC::System::Simple, Capture::Tiny, IPC::Run3, and IPC::Run.
I must note that that's an awkward environment variable; any way to ogranize things otherwise?
† To make this work for non-quoted arguments as well (-arg date) make the quote optional
my #args = $ENV{SOME_ENV} =~ /(\S+)\s+"?([^"]+)/;
where I now left out the closing (unnecessary) quote for simplicity
Whenever I try to add a file with whitespace in its name to git, it shows an Error as
"fatal: pathspec 'Tic' did not match any files"
Since I was New to git and Linux based terminal I have no idea how to do it.
Screen-shot or my error:
You should be able to quote the filename (eg. "file name"), or use an escape sequence (eg. file\< space >name).
To expand on #ergonaut's correct answer, just for the sake of clarity, this is actually neither a git nor a Linux issue. This is just a general requirement for command lines across the board.
On any command line, each word (or in this case, string of words) is evaluated separately as either a command or a parameter to a command. So, for example, git's add command is expecting a single parameter to come immediately after it (a filename). In this case, the next word it sees is just "Tic". Since it's only looking for a single parameter, it stops evaluating anything else at that point and that's why it's complaining about not being able to find the "Tic" file. When the words are enclosed in quotes, the entire string is evaluated as a single parameter, therefore fixing the issue.
Always wrap filenames that contain spaces with quotes when using them on the command line. Or even better, avoid using spaces in filenames. :-)
Try:
git add Tic\ tac\ toe.c
\ is used to escape special characters, though this is more bash related rather than git specific.
Alternatively, you could put the name of the file in quotes.
git add "Tic tac toe.c"
Start writing the first word and then press "Tab" button to use autocomplete
and you will be happy ))
I have a file located in the directory "C:\Documents and Settings\test.exe" but when I write the command `C:\Documents and Settings\test.exe in single qoutes(which I am not able to display in this box), used for executing the commands in Ruby, I am not able to do so and the error that I recieve is No file or Directory found. I have tried replacing "\" with "//" and "\" but nothing seems to work. I have also used system, IO.popen and exec commands but all efforts are in vain. Also exec commands makes the program to make an exit which I don't want to happen.
Thanks in advance.
The backtick environment is like double-quotes, so backslash are used for escaping. Further, Ruby will interpret the spaces as separating command-line arguments, so you need to quote the entire thing:
`"C:\\Documents and Settings\\test.exe"`
Another option is to use system and force a second argument. If system gets more than one argument, it treats the first argument as the path to the command to execute and you don't need to quote the command:
system('C:\Documents and Settings\test.exe','')
Note the use of single quotes, so we don't have escape the backslashes.
Of course, this won't get you the standard out/error, so if you are on Ruby 1.9.2, you can use the awesomely handy Open3 library, which works like system, but gives you more information about the process you just ran:
require 'open3'
stdout,stderr,status = Open3.capture3('C:\Documents and Settings\test.exe','')
puts stdout # => string containing standard output of your command
puts stderr # => string containing standard ERROR of your command
if status.success?
puts "It worked!"
else
puts "OH NOES! Got exit code #{status.exitstatus}"
end
`"C:\Documents and Settings\test.exe"`
or
`exec "C:\Documents and Settings\test.exe"`
or whatever in qoutes