I tried to make an alias for committing several different git projects. I tried something like
cat projectPaths | \
xargs -I project git --git-dir=project/.git --work-tree=project commit -a
where projectPaths is a file containing the paths to all the projects I want to commit. This seems to work for the most part, firing up vi in sequence for each project so that I can write a commit msg for it. I do, however, get a msg:
"Vim: Warning: Input is not from a terminal"
and afterward my terminal is weird: it doesn't show the text I type and doesn't seem to output any newlines. When I enter "reset" things pretty much back to normal, but clearly I'm doing something wrong.
Is there some way to get the same behavior without messing up my shell?
Thanks!
Using the simpler example of
ls *.h | xargs vim
here are a few ways to fix the problem:
xargs -a <( ls *.h ) vim
or
vim $( ls *.h | xargs )
or
ls *.h | xargs -o vim
The first example uses the xargs -a (--arg-file) flag which tells xargs to take its input from a file rather than standard input. The file we give it in this case is a bash process substitution rather than a regular file.
Process substitution takes the output of the command contained in <( ) places it in a filedescriptor and then substitutes the filedescriptor, in this case the substituted command would be something like xargs -a /dev/fd/63 vim.
The second command uses command substitution, the commands are executed in a subshell, and their stdout data is substituted.
The third command uses the xargs --open-tty (-o) flag, which the man page describes thusly:
Reopen stdin as /dev/tty in the child process before executing the
command. This is useful if you want xargs to run an interactive
application.
If you do use it the old way and want to get your terminal to behave again you can use the reset command.
The problem is that since you're running xargs (and hence git and hence vim) in a pipeline, its stdin is taken from the output of cat projectPaths rather than the terminal; this is confusing vim. Fortunately, the solution is simple: add the -o flag to xargs, and it'll start git (and hence vim) with input from /dev/tty, instead of its own stdin.
The man page for GNU xargs shows a similar command for emacs:
xargs sh -c 'emacs "$#" < /dev/tty' emacs
(in this command, the second "emacs" is the "dummy string" that wisbucky refers to in a comment to this answer)
and says this:
Launches the minimum number of copies of Emacs needed, one after the
other, to edit the files listed on xargs' standard input. This example
achieves the same effect as BSD's -o option, but in a more flexible and
portable way.
Another thing to try is using -a instead of cat:
xargs -a projectPaths -I project git --git-dir=project/.git --work-tree=project commit -a
or some combination of the two.
If you have GNU Parallel http://www.gnu.org/software/parallel/ installed you should be able to do this:
cat projectPaths |
parallel -uj1 git --git-dir={}/.git --work-tree={} commit -a
In general this works too:
cat filelist | parallel -Xuj1 $EDITOR
in case you want to edit more than one file at a time (and you have set $EDITOR to your favorite editor).
-o for xargs (as mentioned elsewhere) only works for some versions of xargs (notably it does not work for GNU xargs).
Watch the intro video to learn more about GNU Parallel http://www.youtube.com/watch?v=OpaiGYxkSuQ
Interesting! I see the exact same behaviour on Mac as well, doing something as simple as:
ls *.h | xargs vim
Apparently, it is a problem with vim:
http://talideon.com/weblog/2007/03/xargs-vim.cfm
Related
I have a one-liner that spits out all of the files modified in my current feature branch, which is branched off of a shared, upstream development branch. I then hope to feed the files that exist to the phpcs linter via xargs -- something like this:
git diff --name-only shared-upstream-development-branch | grep "\.php$" | xargs test -f {} && echo {} | xargs vendor/bin/phpcs
However, when I run this, I get something like the following:
test: extra argument
‘path/to/my/file.php’
I feel like I'm close to having a working solution.
How can I modify the one-liner above to correctly see if each PHP file still exists, then feed it onward to phpcs?
I know that everything up through the output of the grep command works well, as removing the two parts of the one-liner that refer to xargs gives me a nice list of file names.
(I also tried using --diff-filter=d to filter out deleted files, but this does not seem to work with my version of git, as I still get a complaint from phpcs about how a file "does not exist.")
&& separates commands, and is not an argument to xargs; you need to execute an explicit shell to use &&.
xargs sh -c 'test -f "$1" && echo "$1"' _ {}
I suppose the answer here might be trivial, but I it might require some intricate bash knowledge. I have been browsing bash docs for a few hours now and can't seem to find the answer.
I'm working on a python repository, and came up with a simple script to lint only the files that differ between the current branch and master. Here's the minimal working example, extracted from said script (lint.sh):
#!/bin/bash
paths=$(git diff --name-only -r origin/master...HEAD | grep \.py$)
flake8 $paths
For testing purposes, let's say I only committed one file, bad.py, with the following contents:
hello
there
The expected output of bash lint.sh is:
bad.py:1:1: F821 undefined name 'hello'
bad.py:2:1: F821 undefined name 'there'
However, the output is empty. When run in debug mode, bash shows the following commands:
++ git diff --name-only -r origin/master...HEAD
++ grep '.py$'
+ paths='bad.py'
+ flake8 'bad.py'
Which is what I expect. Also, when I simply run flake8 bad.py, the output is as expected.
I expect this might have something to do with parameter passing which varies between different bash versions. The output of bash --version:
GNU bash, version 4.4.23(1)-release (x86_64-apple-darwin17.5.0)
I will appreciate all insights
Very sorry this isn't exactly an answer, but it surely didn't fit in a comment!
The hint here to me is the following:
+ paths='bad.py'
+ flake8 'bad.py'
In my execution of the same script, I get the following:
$ bash -x lint.sh
++ git diff --name-only -r origin/master...HEAD
++ grep '.py$'
+ paths=bar.py
+ flake8 bar.py
bar.py:1:1: F821 undefined name 'hello'
bar.py:2:1: F821 undefined name 'world'
Notice here how my output does not contain quotes around the filename or the assignment. bash won't usually add quotes unless they are necessary. What this tells me is there's probably some sort of control character in that string (my best guess is either colors or \b + some other characters (this might be one of the few cases where a screenshot is actually helpful!)).
Here's one way that I was able to reproduce your findings:
mkdir -p bin
cat > bin/grep << EOF
#!/usr/bin/env bash
exec /bin/grep --color=always "\$#"
EOF
chmod +x bin/grep
# simulate having this `grep` on your path
PATH=$PWD/bin:$PATH bash -x lint.sh
(and while this seems like an odd thing to do, in the past I've put my own grep in ~/bin so I could add --line-buffered --color=auto now that GREP_OPTIONS is deprecated -- one might erroneously add --color=always and have it work... for the most part). Today I use an alias instead since I ran into sharp edges even with that.
The output in that case matches yours above:
$ PATH=$PWD/bin:$PATH bash -x lint.sh
++ git diff --name-only -r origin/master...HEAD
++ grep '.py$'
+ paths='bar.py'
+ flake8 'bar.py'
But the tricky hint is in the highlighting
addendum
While unrelated to your problem, here's probably a better way to accomplish what you want:
# if you have GNU xargs
git diff -z --name-only origin/master...HEAD -- '*.py' | xargs --null --no-run-if-empty flake8
# if you need to be more portable (I see you're probably on macos)
git diff -z --name-only origin/master...HEAD -- '*.py' | xargs -0 flake8 /dev/null
Explanation of the different parts:
git diff -z: output filenames with null bytes delimiting. This prevents splicing if filenames contain spaces or other special characters
xargs --null: split the input by null bytes when splatting arguments
xargs --no-run-if-empty: don't run the executable at all if there's no arguments (this is a GNU extension)
xargs -0: same as xargs --null, however if you're stuck with non-GNU xargs you won't have access to the long options
flake8 /dev/null: this is a sneaky trick, since there's no "no run if empty" option to bsd xargs, it's always going to invoke flake8. If flake8 gets invoked with zero arguments, it defaults to recursing your current working directory (and linting all your files). By putting /dev/null at the beginning this prevents this behaviour and instead lints an empty file!
Addendum 2, you probably might want to consider using a git hooks framework to handle all of this for you, I maintain pre-commit which aims to smooth out a lot of the rough edges around git (such as this one!).
I used to use fswatch v0.0.2 like so (in this instance to run django test suit when a file changed)
$>fswatch . 'python manage.py test'
this works fine.
I wanted to exclude some files that were causing the test to run more than once per save (Sublime text was saving a .tmp file, and I suspect .pyc files were also causing this)
So I upgraded fswatch to enable the -e mode.
However the way fswatch has changed which is causing me troubles - it now accepts a pipe argument like so:
$>fswatch . | xargs -n1 program
I can't figure out how to pass in arguments to the program here. e.g. this does not work:
$>fswatch . | xargs -n1 python manage.py test
nor does this:
$>fswatch . | xargs -n1 'python manage.py test'
how can I do this without packaging up my command in a bash script?
fswatch documentation (either the Texinfo manual, or the wiki, or README) have examples of how this is done:
$ fswatch [opts] -0 -o path ... | xargs -0 -n1 -I{} your full command goes here
Pitfalls:
xargs -0, fswatch -0: use it to make sure paths with newlines are interpreted correctly.
fswatch -o: use it to have fswatch "bubble" all the events in the set into a single one printing only the number of records in the set.
-I{}: specifying a placeholder is the trick you missed for xargs interpreting correctly your command arguments in those cases where you do not want the record (in this case, since -o was used, the number of records in the set) to be passed down to the command being executed.
Alternative answer not fighting xargs' default reason for being - passing on the output as arguments to the command to be run.
fswatch . | (while read; do python manage.py test; done)
Which is still a bit wordy/syntaxy, so I have created a super simple bash script fswatch-do that simplifies things for me:
#!/bin/bash
(while read; do "$#"; done)
usage:
fswatch -r -o -e 'pyc' somepath | fswatch-do python manage.py test someapp.SomeAppTestCase
Suppose I have a command git-local (it could be a Bash function or a binary in /usr/local/bin) and suppose I would like git-local to have the same tab completion as the command git has. Finally, suppose that I'm efficient (read lazy) and I don't want to go find the code that the git commmand uses to manually copy over and bloat my .bashrc (or whatever external file I paste it in and the source). Is there a simple way I can have git-local use the same autocompletion as git?
8.7 Programmable Completion Builtins:
If the -p option is supplied, or if no options are supplied, existing completion specifications are printed in a way that allows them to be reused as input.
Something like
$(complete -p git | awk '$NF="git-local"')
maybe?
E.g.:
$ complete -p foobar
-bash: complete: foobar: no completion specification
$ complete -p traceroute
complete -F _known_hosts traceroute
$ $(complete -p traceroute | awk '$NF="foobar"')
$ complete -p foobar
complete -F _known_hosts foobar
I have four text files A.txt, B.txt, C.txt and D.txt
I have to perform a series of vim editing in all these files.
Currently how I am doing is open each files and do the same vim commands one by one.
Is it possible to make a script file which I can run from the command prompt, means without open the actual file for vim editing.
for example, if I have to perform the below vim commands after opening the A.txt file in vim editor:
:g/^\s*$/d
:%s/^/[/
:%s/\(.*\)\(\s\+\)\(.*\)/\3\2\1
:%s/$/]/
:%s/design/test/
Is it possible to make a script file and put all these commands including gvim A.txt (first command in the file).
and edit run the script file from command prompt.
If it is possible, please let me know how to do it and how it can be done with single or multiple files at a time?
vim -c <command> Execute <command> after loading the first file
Does what you describe, but you'll have to do it one file at a time.
So, in a windows shell...
for %a in (A,B,C,D) do vim -c ":g/^\s*$/d" -c "<another command>" %a.txt
POSIX shells are similar, but I don't have a machine in front of me at the moment.
I imagine you could load all the files at once and do it, but it would require repeating the commands on the vim command line for each file, similar to
vim -c "<command>" -c "<command>" -c ":n" (repeat the previous -c commands for each file.) <filenames go here>
EDIT: June 08 2014:
Just an FYI, I discovered this a few minutes ago.
vim has the command bufdo to do things to each buffer (file) loaded in the editor. Look at the docs for the bufdo command. In vim, :help bufdo
The amount of -c commands directly passed to Vim on the command-line is limited to 10, and this is not very readable. Alternatively, you can put the commands into a separate script and pass that to Vim. Here's how:
Silent Batch Mode
For very simple text processing (i.e. using Vim like an enhanced 'sed' or 'awk', maybe just benefitting from the enhanced regular expressions in a :substitute command), use Ex-mode.
REM Windows
call vim -N -u NONE -n -es -S "commands.ex" "filespec"
Note: silent batch mode (:help -s-ex) messes up the Windows console, so you may have to do a cls to clean up after the Vim run.
# Unix
vim -T dumb --noplugin -n -es -S "commands.ex" "filespec"
Attention: Vim will hang waiting for input if the "commands.ex" file doesn't exist; better check beforehand for its existence! Alternatively, Vim can read the commands from stdin. You can also fill a new buffer with text read from stdin, and read commands from stderr if you use the - argument.
Full Automation
For more advanced processing involving multiple windows, and real automation of Vim (where you might interact with the user or leave Vim running to let the user take over), use:
vim -N -u NONE -n -c "set nomore" -S "commands.vim" "filespec"
Here's a summary of the used arguments:
-T dumb Avoids errors in case the terminal detection goes wrong.
-N -u NONE Do not load vimrc and plugins, alternatively:
--noplugin Do not load plugins.
-n No swapfile.
-es Ex mode + silent batch mode -s-ex
Attention: Must be given in that order!
-S ... Source script.
-c 'set nomore' Suppress the more-prompt when the screen is filled
with messages or output to avoid blocking.
With all the commands you want to run on each file saved in a script, say "script.vim", you can execute that script on one file like this (as others have mentioned):
vim -c "source script.vim" A.txt
Taking this one step further, you can save your file at the end of the script, either by putting a :w command inside the script itself, or passing it from the command-line:
vim -c "source script.vim | w" A.txt
Now, you can run any command in Vim on multiple files, by using the argdo command. So your command turns into:
vim -c "argdo source script.vim | w" A.txt B.txt C.txt D.txt
Finally, if you want to quit Vim after running your script on every file, just add another command to quit:
vim -c "argdo source script.vim | w" -c "qa" A.txt B.txt C.txt D.txt
Try the following syntax:
ex foo.txt <<-EOF
g/^\s*$/d
%s/^/[/
%s/\(.*\)\(\s\+\)\(.*\)/\3\2\1
%s/$/]/
%s/design/test/
wq " Update changes and quit.
EOF
The ex command is equivalent to vim -E. Add -V1 for verbose output.
Alternative one-liner syntax is for example:
ex +"g/^\s*$/d" +"%s/^/[/" +"%s/design/test/" -cwq foo.txt
To load commands from the file, use -s cmds.vim.
You can also use shebang for Vim to parse the file from the argument.
For more examples, see:
How to edit files non-interactively (e.g. in pipeline)?
BashFAQ/021
JimR and Ingo have provided excellent answers for your use case.
Just to add one more way to do it, however, you could use my vimrunner plugin to script the interaction in ruby: https://github.com/AndrewRadev/vimrunner.
Example:
vim = Vimrunner.start
vim.edit "file.txt"
vim.insert "Hello"
vim.write
vim.kill
This can be useful for more complicated interactions, since you get the full power of a programming language.
Use vim -s ... to script not only colon commands, but also normal-mode commands such as = for formatting:
Create a file with all keystrokes that you want vim to execute.
Run vim -s SCRIPT-FILE FILE-TO-EDIT
For example: Let's say you want to use vim's = command to re-indent all the lines of myfile.html. First, using vim itself, make a file named myscript that has this:
gg=G:wq
(gg moves to the top of the file; =G re-indents from the current location to the end of the file; :wq<Enter> saves the file and exits.)
Then, run this to have vim launch, edit myfile.html, and save it:
vim -s myscript myfile.html