I am having trouble using run-program with arguments. The documentation at http://www.clisp.org/impnotes/shell.html is very incomplete for a newbie like me, and I would need to see some examples.
In particular, how can I get the directory list of files with "txt" extension?
This works:
(ext:run-program "ls" ) ; I am running on Mac OS X
but if I add arguments, it doesn't work. I have tried:
(ext:run-program "ls" :arguments "*.txt")
(ext:run-program "ls" :arguments '(*.txt))
(ext:run-program "ls *.txt)
Can anyone tell me the right syntax and, hopefully, provide some more examples of run-program?
Note that the above is just an example. What I want is to be able to use run-command, not to list the directory, which I can do with list-directory.
Thanks for any help.
List Files
First of all, if all you want is to list files, you can use the standard
directory
or CLISP-specific ext:dir.
Globbing
Second, the "*" globbing is expanded by shell, not individual commands, so you need to use
ext:run-shell-command
instead of ext:run-program:
> (ext:run-shell-command "ls *.txt")
foo.txt
> (ext:run-program "ls" :arguments '("foo.txt"))
foo.txt
> (ext:run-program "ls")
foo.txt
bar.c
Summary
The manual assumes a certain level of familiarity with the way modern OSes work in this respect. The other excellent answer addresses those issues in detail; briefly:
run-program executes another program using the standard fork/exec paradigm
run-shell-command goes through shell, so you can use globbing and things like loops
shell is mostly for interactive use
More examples
You can find more example of run-program in the CLISP test suite.
If you want to read from a compressed file, you can do this:
;; return lines as a list
(with-input-from-file (in (ext:make-pipe-input-stream "gunzip -c myfile.gz"))
(loop for l = (read-line in nil nil)
while l
collect l))
Actually, you can use CLISP as your shell!
Try something like
(ext:run-program "stat" :arguments '("/tmp" "/sys"))
I am using stat, not ls in this example.
What you might want is globbing (read glob(7)). You should do it yourself in Lisp, or else run /bin/sh -c; in other words it is up to you to build the expanded list passed after :arguments. (You can't pass *.txt as a single argument, it would have the same effect as a quoted shell argument like in ls '*.txt' shell command)
This is not CLISP specific. ext:run-program is probably forking then calling (in the child) execve(2) so you have to do the globbing yourself.
Maybe you want the CLISP equivalent of glob(3) or fnmatch(3). I don't know enough CLISP if it has them or not.
Read Advanced Linux Programming to get a more clear picture of this. On POSIX systems the invoking process (often your shell when typing a command, or your Common Lisp program if using ext:run-program ....) has to expand and build the argument list. The point is that when you run ls *.txt it is your shell (not the ls process) which is expanding *.txt, so ls is getting an expanded argument list like e.g. a.txt foo.txt bar.txt; so you need to do the same in your Lisp code.
Related
How is shell expanding the '*' as in
ls -l *
Is there any system calls happening behind the scene to read all the file in the directory ? I tried strace but not showing any system calls specific to globbing
The glob is expanded by bash, not by ls. When you run strace ls * only the ls is traced, but the glob expands before ls or even strace runs so you won't see the system calls of it in the trace.
To trace the expansion of a glob use strace bash -c 'echo *'. This will also include system calls from the shell startup. To show only the system calls done for echo * you can filter the results with ...
strace bash -c '[ -e startglob ]; echo *; [ -e endglob ]' 2>&1 |
sed -n '/"startglob"/,/"endglob"/p'
Read glob(7). Globbing would use several syscalls(2): access(2), stat(2) opendir(3), readdir(3) (which uses getdents(2)...) closedir(3). See also nftw(3).
And GNU bash is (like GNU libc and the Linux kernel) free software, you can download its source code and study it, compile it and improve it.
You can use strace(1) or gdb(1) to understand how GNU bash (or any other Linux application) works.
Of course, globbing is done by the shell (before fork(2) and execve(2)...), not by the /bin/ls process
How is shell expanding the * as in ls -l *
When meeting a globing pattern like *, the shell reads all the directory entries and compare them one by one with the pattern.
A quick glance at the source code tells me this is probably done in glob.c, especially in the glob_vector() function.
Is there any system calls happening behind the scene to read all the file in the directory ?
Of course, you cannot read the content of a directory without the help of the kernel.
That said, system calls are low-level, so don't expect to find any call to opendir() and readdir() which are libc calls. Rather, you will find calls to open() and getdents()/getdents64() instead.
I tried strace but not showing any system calls specific to globbing
strace is NOT the right tool for that. This is not your kernel that is doing globing, this is your shell, possibly with the help of libc functions, therefore there is no system call involved (except from the above-mentioned low-level calls to read the directory content).
If you want to trace calls to library functions like opendir(), readdir(), glob(), strcmp(),... you MUST use ltrace:
ltrace bash -c 'ls *'
That said, Bash has its own globing system, more advanced than the basic one specified by the POSIX standard, so it won't rely on library functions like glob() or fnmatch(). Don't expect to see calls to them in ltrace output.
If your aim is to use globbing in your own application and study how it is done, you may want to have a look at glob() which is simpler than Bash's globbing (FYI, I recently posted an example of use here).
I'm trying to call pdflatex from a guile scheme file. This is the Guile command I'm using:
(system*
"cat" "foo.txt" "|" "pdflatex" "-jobname" "\"bar\"")
This is the error I get back after running the file:
cat: invalid option -- 'j'
Try 'cat --help' for more information.
If I run the command from bash shell it runs normally.
cat foo.txt | pdflatex -jobname "bar"
-jobname is the correct command for pdflatex, but system* seems to have a problem with it.
I'm using (GNU Guile) 2.2.4 and pdfTeX 3.14159265-2.6-1.40.20.
Use system, not system*. It takes a single string as the argument, and executes it using the shell, which will perform the desired piping.
(system "cat foo.txt | pdflatex -jobname 'bar'")
system* doesn't use the shell. As the manual explains:
system* is similar to system, but accepts only one string per-argument, and performs no shell interpretation. The command is executed using fork and execlp. Accordingly this function may be safer than system in situations where shell interpretation is not required.
Note that your command is a Useless use of cat since pdflatex takes the filename as an argument. You could use system* to execute it directly.
(system* "pdflatex" "-jobname" "bar" "foo.txt")
Also, you don't need to add extra quotes around bar when you use system*; since it doesn't use the shell, it doesn't parse special characters.
CMake's execute_process command seems to only let you, well, execute a process - not an arbitrary line you could feed a command shell. The thing is, I want to use pipes, file descriptor redirection, etc. - and that does not seem to be possible. The alternative would be very painful for me (I think)...
What should I do?
PS - CMake 2.8 and 3.x answer(s) are interesting.
You can execute any shell script, using your shell's support for taking in a script within a string argument.
Example:
execute_process(
COMMAND bash "-c" "echo -n hello | sed 's/hello/world/;'"
OUTPUT_VARIABLE FOO
)
will result in FOO containing world.
Of course, you would need to escape quotes and backslashes with care. Also remember that running bash would only work on platforms which have bash - i.e. it won't work on Windows.
execute_process command seems to only let you, well, execute a process - not an arbitrary line you could feed a command shell.
Yes, exactly this is written in documentation for that command:
All arguments are passed VERBATIM to the child process. No intermediate shell is used, so shell operators such as > are treated as normal arguments.
I want to use pipes
Different COMMAND within same execute_process invocation are actually piped:
Runs the given sequence of one or more commands with the standard output of each process piped to the standard input of the next.
file descriptor redirection, etc. - and that does not seem to be possible.
For complex things just prepare separate shell script and run it using execute_process. You can pass variables from CMake to this script using its parameters, or with prelimiary configure_file.
I needed to pipe two commands one after the other and actually learned that each COMMAND of the execute_process is piped already. So at least that much is resolved by simply adding commands one after the other:
execute_process(
COMMAND echo "Hello"
COMMAND sed -e 's/H/h/'
OUTPUT_VARIABLE GREETINGS
OUTPUT_STRIP_TRAILING_WHITESPACE)
Now the variable GREETINGS is set to hello.
If you indeed need a lot of file redirection (as you stated), you probably want to write an external script and then execute that script from CMakeLists.txt. It's really difficult to get all the escaping right in CMake.
If you can simplify your scripts to one command generating a file, then another handling that file, etc. then you can always use the INPUT_FILE and OUTPUT_FILE options. Or pass a filename to your command for the input.
It's often much cleaner to handle one file at a time. Although I understand that some commands may need multiple sources and destinations.
How can I use the "nice" command with an alias?
As an example:
alias list=ls
list # works
nice -10 list # doesn't work
How could I make that last line work?
Alias is a shell feature, and nice is an external program:
$ type nice
nice is hashed (/usr/bin/nice)
It's the program nice that runs the command passed as an argument, calling the C function execve, so all the arguments for it need to be evaluated BEFORE the call.
So, it would probably better not to use an alias and simply put the whole command needed there, but if you really want to, you could try something like this:
$ nice -10 `alias list | sed "s/^\(alias \)\?[^=]\+='//; s/'$//;"`
alias list prints the alias definition in the format alias list='ls' (or list='ls', if it's /bin/sh), so I did some sed substitutions there to get only the command it expands to.
If you're sure to use only bash you can use ${BASH_ALIASES[list]} instead, as pointed out in the comments:
$ nice -10 ${BASH_ALIASES[list]}
Though perhaps not as exotically exciting as nice -10 $UserVar1; or nice -10 ${BASH_ALIASES[list]}, you may also have the nice -10 list you asked for, though via wrapper script instead of alias:
# one-time setup
mkdir -p ~/.local/aliases
echo 'PATH=$HOME/.local/aliases:$PATH' >> ~/.bashrc
# open new terminal window, or
source ~/.bashrc
# create the wrapper. $# to passthrough args.
echo 'ls $#' > ~/.local/aliases/list
chmod +x ~/.local/aliases/list
nice -10 list # works :)
nice -10 list --color=always -lathr # args passthrough also works :)
For Zsh BASH_ALIASES will not work. So you may use it like this:
nice -10 `list`
This is a bad idea. You are defining a command alias but are not even using it as an alias expansion. Poor coding practice here. This is what you want.
declare -x UserVar1='ls';
nice -10 $UserVar1;
And if you will not change the definition of UserVar1 later on in your code. There are zero reasons you can justify to use a variable instead of the actual command name.
You are headed for disaster. Plain and simple. Use a variable or the command name itself it is far safer and marginally more efficient and easier to read.
I switched from bash to zsh and I was wondering if there was a way to put arguments after file name like in bash
Example:
cp dir1 dir2 -r
Thank you
This depends only on the command, not on the shell. The shell passes the arguments in the order they're given, and makes no special treatment for arguments beginning with a -.
zsh has some expanded/different features in the area of globbing and tab completion (two of the primary reasons folks may switch to zsh). Both of these provide you interesting way to add command line parameters. Is that what you are asking about?
Note also that most commands are not impacted by the shell you choose: ls, awk, grep, vim, etc. Obviously things like alias and function that are shell commands are potentially different.