As "is known", a script my-script-file which starts with
#!/path/to/interpreter -arg1 val1 -arg2 val2
is executed by exec calling /path/to/interpreter with 2(!) arguments:
-arg1 val1 -arg2 val2
my-script-file
(and not, as one might naively expect, with 5 arguments
-arg1
val1
-arg2
val2
my-script-file
as has been explained in many previous questions, e.g.,
https://stackoverflow.com/a/4304187/850781).
My problem is from the POV of an interpreter developer, not script writer.
How do I detect from inside the interpreter executable that I was called from shebang as opposed to the command line?
Then I will be able to decide whether I need to split my first argument
by space to go from "-arg1 val1 -arg2 val2" to ["-arg1", "val1", "-arg2", "val2"] or not.
The main issue here is script files named with spaces in them.
If I always split the 1st argument, I will fail like this:
$ my-interpreter "weird file name with spaces"
my-interpreter: "weird": No such file or directory
On Linux, with GNU libc or musl libc, you can use the aux-vector to distinguish the two cases.
Here is some sample code:
#define _GNU_SOURCE 1
#include <stdio.h>
#include <errno.h>
#include <sys/auxv.h>
#include <sys/stat.h>
int
main (int argc, char* argv[])
{
printf ("argv[0] = %s\n", argv[0]);
/* https://www.gnu.org/software/libc/manual/html_node/Error-Messages.html */
printf ("program_invocation_name = %s\n", program_invocation_name);
/* http://man7.org/linux/man-pages/man3/getauxval.3.html */
printf ("auxv[AT_EXECFN] = %s\n", (const char *) getauxval (AT_EXECFN));
/* Determine whether the last two are the same. */
struct stat statbuf1, statbuf2;
if (stat (program_invocation_name, &statbuf1) >= 0
&& stat ((const char *) getauxval (AT_EXECFN), &statbuf2) >= 0)
printf ("same? %d\n", statbuf1.st_dev == statbuf2.st_dev && statbuf1.st_ino == statbuf2.st_ino);
}
Result for a direct invocation:
$ ./a.out
argv[0] = ./a.out
program_invocation_name = ./a.out
auxv[AT_EXECFN] = ./a.out
same? 1
Result for an invocation through a script that starts with #!/home/bruno/a.out:
$ ./a.script
argv[0] = /home/bruno/a.out
program_invocation_name = /home/bruno/a.out
auxv[AT_EXECFN] = ./a.script
same? 0
This approach is, of course, highly unportable: Only Linux has the getauxv function. And there are surely cases where it does not work well.
Question
I am willing to pass arguments to a jshell script. For instance, I would have liked something like this:
jshell myscript.jsh "some text"
and then to have the string "some text" available in some variable inside the script.
However, jshell only expects a list of files, therefore the answer is:
File 'some text' for 'jshell' is not found.
Is there any way to properly pass arguments to a jshell script?
Workaround so far
My only solution so far is to use an environment variable when calling the script:
ARG="some test" jshell myscript.jsh
And then I can access it in the script with:
System.getenv().get("ARG")
And what about option -R
> jshell -v -R-Da=b ./file.jsh
for script
{
String value = System.getProperty("a");
System.out.println("a="+value);
}
/exit
will give you
> jshell -v -R-Da=b ./file.jsh
a=b
Another way, would be following:
{
class A {
public void main(String args[])
{
for(String arg : args) {
System.out.println(arg);
}
}
}
new A().main(System.getProperty("args").split(" "));
}
and execution
> jshell -R-Dargs="aaa bbb ccc" ./file_2.jsh
Update
Previous solution will fail with more complex args. E.g. 'This is my arg'.
But we can benefit from ant and it's CommandLine class
import org.apache.tools.ant.types.Commandline;
{
class A {
public void main(String args[])
{
for(String arg : args) {
System.out.println(arg);
}
}
}
new A().main(Commandline.translateCommandline(System.getProperty("args")));
}
and then, we can call it like this:
jshell --class-path ./ant.jar -R-Dargs="aaa 'Some args with spaces' bbb ccc" ./file_2.jsh
aaa
Some args with spaces
bbb
ccc
Of course, ant.jar must be in the path that is passed via --class-path
Oracle really screwed this up, there is no good way to do this. In addition to #mko's answer and if you use Linux(probably will work on Mac too) you can use process substitution.
jshell <(echo 'String arg="some text"') myscript.jsh
And then you can just use arg in myscript.jsh for example:
System.out.println(arg) // will print "some text"
You can simplify it with some bash function and probably write a batch file that will write to a temp file and do the same on windows.
It's completely beyond me how Oracle could ignore this. 8-() But anyway: if your system uses bash as shell, you can combine this approach replacing the shebang with the idea to (ab-)use system properties to transport the whole command line into a variable:
//usr/bin/env jshell --execution local "-J-Da=$*" "$0"; exit $?
String commandline = System.getProperty("a");
System.out.println(commandline);
/exit
This way, you can call the script on the commandline simply adding the arguments: thisscript.jsh arg1 arg2 would print arg1 arg2.
Please note that this joins all parameters into one String, separated by one space. You can split it again with commandline.split("\s"), but please be aware that this isn't exact: there is no difference between two parameters a b and one parameter "a b".
If you have a fixed number of arguments, you can also pass all of these into separate system properties with "-J-Darg1=$1" "-J-Darg2=$1" "-J-Darg3=$1" etc. Please observe that you have to use -R-D... if you are not using --execution local
Another variant is generating the script on the fly with bash's process substitution. You can use such a script also simply as thisscript.jsh arg1 arg2 also on Unix-like systems having a bash.
#!/usr/bin/env bash
jshell <(
cat <<EOF
System.out.println("$1");
System.out.println("$2");
/exit
EOF
)
This allows to access individual parameters, though it will break when there are double quotes or other special characters in a parameter. Expanding on that idea: here's a way to put all parameters into an Java String array, quoting some of those characters:
#!/usr/bin/env bash
set -- "${#//\\/\\\\}"
set -- "${#//\"/\\\"}"
set -- "${#/#/\"}"
set -- "${#/%/\",}"
jshell <(
cat <<EOF
String[] args = new String[]{$#};
System.out.println(Arrays.asList(args));
/exit
EOF
)
The set -- statements double backslashes, quote double quotes and prefix a " and append a ", to transform the arguments into a valid Java array.
Recently, I was inspired by answers from Oleg and Hans-Peter Störr enough to try to combine them so that a) I could use normal shell arguments b) write regular Java code expecting a String[] args input:
//usr/bin/env jshell <(ARGS=; for var in "$#"; do ARGS+="\"$var\","; done; echo "String[] args = {$ARGS}") "$0"; exit $?
System.out.println("RESULT: " + Arrays.asList(args));
/exit
Using Hans' header line and then inlining as demonstrated by Oleg which builds the $# args into a String[] args var.
With that you can chmod +x your script and execute it with regular old arguments:
]$ ./script.jsh foo bar
RESULT: [test, bar]
Lua's io.popen is useful for capturing stdout of a command, but it relies on /bin/sh for word-splitting, glob expansion, etc.
So I can do the following and it works great
-- this unfortunately uses /bin/sh
local command = "ls -1";
local ph = io.popen(command, "r")
while true do
local line = ph:read()
if line == nil then
break
end
io.write("I read a line! " .. line .. "\n")
end
io.write("Done with line reading!\n")
Is there some function that's a thin wrapper around spawn* or fork/exec* that I can pass {"ls", "-1"} to in order to avoid the shell?
I can do something like this in bash:
myCommand arg1 arg2 2>&1 >> myLogFolder/myLogFile.log | tee -a myLogFolder/myLogFile.log
I would like to be able to say this instead:
log.rb myLogFolder/myLogFile.log myCommand arg1 arg2
Using the log.rb script would accomplish two things:
Result in a simpler statement with less redirection tricks and only a single specification of the log file.
Create the log folder, if necessary.
I was looking at Ruby's popen and spawn options, but I don't see a way to split the STDERR stream to two destinations.
This Ruby script satisfies my needs, but maybe there is a better way.
logPath = ARGV[0]
logFolder = File.dirname(logPath)
command = ARGV.slice(1..-1).join(" ")
`mkdir -p #{logFolder}`
exec "#{command} 2>&1 >> #{logPath} | tee -a #{logPath}"
Try this article about implementing functionality similar to tee using Ruby. You should be able to use that as a starting point for a pure-ruby (or at least exec-free) implementation of your shell code.
You can use Open3 module (manual) It returns 3 objects: stdin, stdout, stderr
However, you are not able to preserve the order between stdout and stderr.
Example:
#include <stdio.h>
int main(int argc, char *argv[]) {
fprintf(stdout, "Normal output\n");
fprintf(stderr, "Error output\n");
fprintf(stdout, "Normal output\n");
}
It is captured as:
Normal output
Normal output
Error output
The only way how to preserve order is to run program twice. :-(
Sample ruby code:
#!/usr/bin/env ruby
require 'open3'
require 'pathname'
logfile = ARGV.first()
cmd = ARGV.drop(1).join(" ")
dir = Pathname(logfile).dirname
if not File.directory?(dir)
Dir.mkdir(dir)
end
file = File.open(logfile, "w")
stdin, stdout, stderr = Open3.popen3(cmd)
stdin.close()
file.puts(stdout.read())
error = stderr.read()
file.puts(error)
puts error
file.close
Here's a barebones Python app that simply prints the command-line arguments passed in:
import sys
if __name__ == "__main__":
print "Arguments:"
for i in range(len(sys.argv)):
print "[%s] = %s" % (i, sys.argv[i])
And here's some sample runs:
python args.py hello world
Arguments:
[0] = args.py
[1] = hello
[2] = world
python args.py "hello world"
Arguments:
[0] = args.py
[1] = hello world
python args.py "hello\world"
Arguments:
[0] = args.py
[1] = hello\world
So far so good. But now when I end any argument with a backslash, Python chokes on it:
python args.py "hello\world\"
Arguments:
[0] = args.py
[1] = hello\world"
python args.py "hello\" world "any cpu"
Arguments:
[0] = args.py
[1] = hello" world any
[2] = cpu
I'm aware of Python's less-than-ideal raw string behavior via the "r" prefix (link), and it seems clear that it's applying the same behavior here.
But in this case, I don't have control of what arguments are passed to me, and I can't enforce that the arguments don't end in a backslash. How can I work around this frustrating limitation?
--
Edit: Thanks to those who pointed out that this behavior isn't specific to Python. It seems to be standard shell behavior (at least on Windows, I don't have a Mac at the moment).
Updated question: How can I accept args ending in a backslash? For example, one of the arguments to my app is a file path. I can't enforce that the client sends it to me without a trailing backslash, or with the backslash escaped. Is this possible in any way?
That's likely the shell treating \ as an escape character, and thus escaping the character. So the shell sends \" as " (because it thinks you are trying to escape the double quote). The solution is to escape the escape character, like so: $ python args.py "hello\world\\".
The Microsoft Parameter Parsing Rules
These are the rules for parsing a command line passed by CreateProcess() to a program written in C/C++:
Parameters are always separated by a
space or tab (multiple spaces/tabs
OK)
If the parameter does not contain
any spaces, tabs, or double quotes,
then all the characters in the
parameter are accepted as is (there
is no need to enclose the parameter
in double quotes).
Enclose spaces and tabs in a double
quoted part
A double quoted part can be anywhere
within a parameter
2n backslashes followed by a "
produce n backslashes + start/end
double quoted part
2n+1 backslashes followed by a "
produce n backslashes + a literal
quotation mark
n backslashes not followed by a
quotation mark produce n backslashes
If a closing " is followed
immediately by another ", the 2nd "
is accepted literally and added to
the parameter (This is the
undocumented rule.)
For a detailed and clear description see http://www.daviddeley.com/autohotkey/parameters/parameters.htm#WINCRULESDOC
The backslash at the end is interpreted as the start of an escape sequence, in this case a literal double quote character. I had a similar problem with handling environment parameters containing a path that sometimes ended with a \ and sometimes didn't.
The solution I came up with was to always insert a space at the end of the path string when calling the executable. My executable then uses the directory path with the slash and a space at the end, which gets ignored. You could possibly trim the path within the program if it causes you issues.
If %SlashPath% = "hello\"
python args.py "%SlashPath% " world "any cpu"
Arguments:
[0] = args.py
[1] = hello\
[2] = world
[3] = any cpu
If %SlashPath% = "hello"
python args.py "%SlashPath% " world "any cpu"
Arguments:
[0] = args.py
[1] = hello
[2] = world
[3] = any cpu
Hopefully this will give you some ideas of how to get around your problem.
The backslash 'escapes' the character following it. This means that the closing quotation marks become a part of the argument, and don't actually terminate the string.
This is the behaviour of the shell you're using (presumably bash or similar), not Python (although you can escape characters within Python strings, too).
The solution is to escape the backslashes:
python args.py "hello\world\\"
Your Python script should then function as you expect it to.
The backslash (\) is escaping the ". That's all. That is how it is supposed to work.
If this is on Windows, then you are not using a standard Windows command prompt (or shell). This must be bash doing this. The Windows command prompt doesn't treat backslash as an escape character (since it's the file path separator).
Extra trivia point: the quoting character in Windows command prompts is caret: ^
When the user passes your function a string "hello\", regardless of what their intention was, they sent the actual string hello", just like if a user passed a filepath like "temp\table" what they have really typed, intentionally or not, is "temp able" (tab in the middle).
This being said, a solution to this problem means that if a user inputs "temp\table" and honestly means "temp able", you are going to process this into "temp\table" and now you've programmatically destroyed the users input.
With this warning in mind, if you still want to do this, you can look for the string representation of these escaped-characters and replace them. As a really easy example, something like this:
def allow_tabs(str_w_tab):
str_w_tab.replace('\t','\\t')
print str_w_tab
Now if you want to handle all the other escape characters, you'll have to do something similar for each one. As for being able to do this for the example: "hello\", the user passed you the string hello", and whether they intended to or not, they never closed the double-quote, so this is what your program sees.
On 'nix based systems, this is a fundamental shell limitation, as others have said here. So, just suck it up. That said, it's really not that important because you don't often need backslashes in arguments on those platforms.
On Windows, however, backslashes are of critical value! A path ending in one would explicitly denote a directory vs a file. I have seen the documentation for MS C (see: https://learn.microsoft.com/en-us/previous-versions/17w5ykft(v=vs.85) ), and within the Python source (e.g. in subprocess.list2cmd https://github.com/python/cpython/blob/master/Lib/subprocess.py), explaining this problem with quoting a process argument and have it not able to end with a backslash. So, I forgive the Python developers for keeping the logic the same - but not the MS C ones! This is not a cmd.exe shell issue or a universal limitation for arguments in Windows! (The caret ^ is the equivalent escape character in that natural shell.)
Batch Example (test.bat):
#echo off
echo 0: %0
echo 1: %1
echo 2: %2
echo 3: %3
Now execute it (via cmd.exe):
test.bat -t "C:\test\this path\" -v
Yields:
0: test.bat
1: -t
2: "C:\test\this path\"
3: -v
As you can see - a simple batch file implicitly understands what we want!
But... let's see what happens in Python, when using the standard argparse module (https://docs.python.org/3/library/argparse.html), which is intertwined with sys.argv initial parsing by default:
broken_args.py
import os
import argparse # pip install argparse
parser = argparse.ArgumentParser( epilog="DEMO HELP EPILOG" )
parser.add_argument( '-v', '--verbose', default=False, action='store_true',
help='enable verbose output' )
parser.add_argument( '-t', '--target', default=None,
help='target directory' )
args = parser.parse_args()
print( "verbose: %s" % (args.verbose,) )
print( "target: %s" % (os.path.normpath( args.target ),) )
Test that:
python broken_args.py -t "C:\test\this path\" -v
Yields these bad results:
verbose: False
target: C:\test\this path" -v
And so, here's how I solved this. The key "trick" is first fetching the full, raw command line for the process via the Windows api:
fixed_args.py
import sys, os, shlex
import argparse # pip install argparse
IS_WINDOWS = sys.platform.startswith( 'win' )
IS_FROZEN = getattr( sys, 'frozen', False )
class CustomArgumentParser( argparse.ArgumentParser ):
if IS_WINDOWS:
# override
def parse_args( self ):
def rawCommandLine():
from ctypes.wintypes import LPWSTR
from ctypes import windll
Kernel32 = windll.Kernel32
GetCommandLineW = Kernel32.GetCommandLineW
GetCommandLineW.argtypes = ()
GetCommandLineW.restype = LPWSTR
return GetCommandLineW()
NIX_PATH_SEP = '/'
commandLine = rawCommandLine().replace( os.sep, NIX_PATH_SEP )
skipArgCount = 1 if IS_FROZEN else 2
args = shlex.split( commandLine )[skipArgCount:]
return argparse.ArgumentParser.parse_args( self, args )
parser = CustomArgumentParser( epilog="DEMO HELP EPILOG" )
parser.add_argument( '-v', '--verbose', default=False, action='store_true',
help='enable verbose output' )
parser.add_argument( '-t', '--target', default=None,
help='target directory' )
args = parser.parse_args()
print( "verbose: %s" % (args.verbose,) )
print( "target: %s" % (os.path.normpath( args.target ),) )
Confirm the fix:
python fixed_args.py -t "C:\test\this path\" -v
Yields these good results:
verbose: True
target: C:\test\this path