Here's a Makefile
roman#debian ~/D/O/devops> cat Makefile
install:
-cat projects.txt | xargs -n 2 bash -c 'git clone $0 $1'
Here's projects.txt
roman#debian ~/D/O/devops> cat projects.txt
git#github.com:xxx/xxx1.git app-xxx1
git#github.com:xxx/xxx2.git app-xxx2
Here's what happens when I just copy this command to bash - it works:
roman#debian ~/D/O/devops> cat projects.txt | xargs -n 2 bash -c 'git clone $0 $1'
fatal: destination path 'app-xxx1' already exists and is not an empty directory.
It's using git clone properly it's just repo exists.
Now when you do make install this fails, all variables are blank:
roman#debian ~/D/O/devops> make install
cat projects.txt | xargs -n 2 bash -c 'git clone '
You must specify a repository to clone.
I'd like to use only xargs method in here, else it becomes too wordy, also there's even more problems when using loops. I've also tried to use $(1) but no luck
make is interpreting $0 and $1 as make variables and trying to expand them. Try replacing $ with $$...
install:
-cat projects.txt | xargs -n 2 bash -c 'git clone $$0 $$1'
Use -I {} to read lines and use that for the input inside Makefile
install:
-cat projects.txt | xargs -I '{}' git clone '{}'
make expands dollar sign references to be its own variables. While it is usual to write $(name), it is possible to write $c where c is a single character (as long as it is not a $), so when you use $0 and $1 in your command line, they will be substituted with whatever values those variables have inside make. (make does not know anything about bash quoting, so $-references will be expanded anywhere in the line.)
Since make does not automatically define $0 and $1 and it is unlikely that you have defined them in your makefile, the most likely is that they will be substituted with the empty string, resulting in the error you observe. To avoid this problem, escape the dollar signs by doubling them:
-cat projects.txt | xargs -n 2 bash -c 'git clone $$0 $$1'
Having said that, it is not at all clear to me why you want to use bash -c here. What's wrong with
xargs -n2 git clone < projects.txt
Worked for me when I used two $.
$ cat Makefile
install:
cat projects.txt | xargs -n 2 bash -c 'echo git clone $$0 $$1'
$ make install
cat projects.txt | xargs -n 2 bash -c 'echo git clone $0 $1'
git clone git#github.com:xxx/xxx1.git app-xxx1
git clone git#github.com:xxx/xxx2.git app-xxx2
Related
I'm trying to split the following command to use it in a script:
git diff HEAD --name-only && git submodule foreach git diff HEAD --name-only | grep -v '^Entering')
So I've tried to do this:
GIT_CMD=$(git diff HEAD --name-only)
GIT_SUBMODULE_CMD=$(git submodule foreach git diff HEAD --name-only)
CMD="$GIT_CMD && $GIT_SUBMODULE_CMD | grep -v '^Entering'"
echo $CMD
and the output I get is:
<file names> | grep -v '^Entering'
$GIT_CMD && $GIT_SUBMODULE_CMD are executed but I don't know what to do with the "| grep" part.
Putting the command inside $() will execute the command and place the output in the variable.
Try:
GIT_CMD="git diff HEAD --name-only"
GIT_SUBMODULE_CMD="git submodule foreach git diff HEAD --name-only"
CMD="$GIT_CMD && $GIT_SUBMODULE_CMD | grep -v '^Entering'"
echo $CMD
Now you are just assigning the command to the variable and not the output of the command.
In my Makefile I want to execute multiple steps as a part of a single target. Those steps should be done sequentially, because they depend on one another. This is the simplified case:
target:
git archive --remote=some-remote master --format zip -o ./zipfile.zip
echo "$(VARIABLE_IN_MAKE):$(shell unzip -z ./zipfilezip | tail -1)" > ./textfile
$(shell cat ./textfile)
The problem here is that the shell command - $(shell unzip -z ./zipfilezip | tail -1) is executed as soon as the rule is "loaded", i.e. before the zipfile even exists. That returns errors. The cat command is also expanded too early.
What is the correct way to execute a subshell not before, but only after all the steps above have finished? Do I have to wrap all the commands in a bash -c call? Or chain them via &&?
Get rid of these $(shell...). Each line of a make recipe is already a shell script:
target:
git archive --remote=some-remote master --format zip -o ./zipfile.zip
echo "$(VARIABLE_IN_MAKE):$$(unzip -z ./zipfilezip | tail -1)" > ./textfile
cat ./textfile
Note that, in the second line, $$(unzip -z ./zipfilezip | tail -1) is expanded twice: a first time by make before passing the recipe to the shell, leading to $(unzip -z ./zipfilezip | tail -1), and a second time by the shell that treats it as a command substitution. This is why the $$ is needed: to escape the first expansion by make. If you were using $(unzip -z ./zipfilezip | tail -1) directly, make would expand it as the empty string (unless you have a make variable which name is unzip -z ./zipfilezip | tail -1, but this is very unlikely).
I suppose the answer here might be trivial, but I it might require some intricate bash knowledge. I have been browsing bash docs for a few hours now and can't seem to find the answer.
I'm working on a python repository, and came up with a simple script to lint only the files that differ between the current branch and master. Here's the minimal working example, extracted from said script (lint.sh):
#!/bin/bash
paths=$(git diff --name-only -r origin/master...HEAD | grep \.py$)
flake8 $paths
For testing purposes, let's say I only committed one file, bad.py, with the following contents:
hello
there
The expected output of bash lint.sh is:
bad.py:1:1: F821 undefined name 'hello'
bad.py:2:1: F821 undefined name 'there'
However, the output is empty. When run in debug mode, bash shows the following commands:
++ git diff --name-only -r origin/master...HEAD
++ grep '.py$'
+ paths='bad.py'
+ flake8 'bad.py'
Which is what I expect. Also, when I simply run flake8 bad.py, the output is as expected.
I expect this might have something to do with parameter passing which varies between different bash versions. The output of bash --version:
GNU bash, version 4.4.23(1)-release (x86_64-apple-darwin17.5.0)
I will appreciate all insights
Very sorry this isn't exactly an answer, but it surely didn't fit in a comment!
The hint here to me is the following:
+ paths='bad.py'
+ flake8 'bad.py'
In my execution of the same script, I get the following:
$ bash -x lint.sh
++ git diff --name-only -r origin/master...HEAD
++ grep '.py$'
+ paths=bar.py
+ flake8 bar.py
bar.py:1:1: F821 undefined name 'hello'
bar.py:2:1: F821 undefined name 'world'
Notice here how my output does not contain quotes around the filename or the assignment. bash won't usually add quotes unless they are necessary. What this tells me is there's probably some sort of control character in that string (my best guess is either colors or \b + some other characters (this might be one of the few cases where a screenshot is actually helpful!)).
Here's one way that I was able to reproduce your findings:
mkdir -p bin
cat > bin/grep << EOF
#!/usr/bin/env bash
exec /bin/grep --color=always "\$#"
EOF
chmod +x bin/grep
# simulate having this `grep` on your path
PATH=$PWD/bin:$PATH bash -x lint.sh
(and while this seems like an odd thing to do, in the past I've put my own grep in ~/bin so I could add --line-buffered --color=auto now that GREP_OPTIONS is deprecated -- one might erroneously add --color=always and have it work... for the most part). Today I use an alias instead since I ran into sharp edges even with that.
The output in that case matches yours above:
$ PATH=$PWD/bin:$PATH bash -x lint.sh
++ git diff --name-only -r origin/master...HEAD
++ grep '.py$'
+ paths='bar.py'
+ flake8 'bar.py'
But the tricky hint is in the highlighting
addendum
While unrelated to your problem, here's probably a better way to accomplish what you want:
# if you have GNU xargs
git diff -z --name-only origin/master...HEAD -- '*.py' | xargs --null --no-run-if-empty flake8
# if you need to be more portable (I see you're probably on macos)
git diff -z --name-only origin/master...HEAD -- '*.py' | xargs -0 flake8 /dev/null
Explanation of the different parts:
git diff -z: output filenames with null bytes delimiting. This prevents splicing if filenames contain spaces or other special characters
xargs --null: split the input by null bytes when splatting arguments
xargs --no-run-if-empty: don't run the executable at all if there's no arguments (this is a GNU extension)
xargs -0: same as xargs --null, however if you're stuck with non-GNU xargs you won't have access to the long options
flake8 /dev/null: this is a sneaky trick, since there's no "no run if empty" option to bsd xargs, it's always going to invoke flake8. If flake8 gets invoked with zero arguments, it defaults to recursing your current working directory (and linting all your files). By putting /dev/null at the beginning this prevents this behaviour and instead lints an empty file!
Addendum 2, you probably might want to consider using a git hooks framework to handle all of this for you, I maintain pre-commit which aims to smooth out a lot of the rough edges around git (such as this one!).
Reason: I want to compare two arbitrary different commits using a difftool. I know the hashes from a search and I don't want to copy these hashes, thus I am looking for a command that does something like
$ log_str=$(git log --all -S"new_tour <-" --pretty=format:"%h")
$ git difftool -t kdiff3 log_str[1] log_str[2] myfile.txt
I would like to be able to address arbitrary indices - not always 1 and 2
It would be great if the answer also gives a hint, how to figure out, what the structure of log_str is. Is it a character? An array of characters? A list? ... using the Bash.
I found some related help here and here, but I can't make it work.
Now I do:
$ git log --pretty=format:"%h"
3f69dc7
b8242c6
01aa74f
903c5aa
069cfc5
and
$ git difftool -t kdiff3 3f69dc7 b8242c6 myfile.txt
I would take a two step approach using a temporary file:
git log --all -S'SEARCH' --pretty=format:"%h" > tmp_out
git diff "$(sed -n '1p' tmp_out)" "$(sed -n '2p' tmp_out)" myfile.txt
rm tmp_out
sed is used to display line 1 and line 2 of the file.
With variables:
search="foo"
index_a="1"
index_b="2"
file="myfile.txt"
git log --all -S"${search}" --pretty=format:"%h" > tmp_out
git diff "$(sed -n "${index_a}p" tmp_out)" "$(sed -n "${index_b}p" tmp_out)" "${file}"
rm tmp_out
in a bash function:
search_diff() {
search="${1}"
index_a="${2}"
index_b="${3}"
file="${4}"
git log --all -S"${search}" --pretty=format:"%h" > tmp_out
git diff "$(sed -n "${index_a}p" tmp_out)" "$(sed -n "${index_b}p" tmp_out)" "${file}"
rm tmp_out
}
search_diff "foo" 2 3 myfile.txt
I'd like to be able to comment out a single flag in a one-line command. Bash only seems to have from # till end-of-line comments. I'm looking at tricks like:
ls -l $([ ] && -F is turned off) -a /etc
It's ugly, but better than nothing. Is there a better way?
The following seems to work, but I'm not sure whether it is portable:
ls -l `# -F is turned off` -a /etc
My preferred is:
Commenting in a Bash script
This will have some overhead, but technically it does answer your question
echo abc `#put your comment here` \
def `#another chance for a comment` \
xyz etc
And for pipelines specifically, there is a cleaner solution with no overhead
echo abc | # normal comment OK here
tr a-z A-Z | # another normal comment OK here
sort | # the pipelines are automatically continued
uniq # final comment
How to put a line comment for a multi-line command
I find it easiest (and most readable) to just copy the line and comment out the original version:
#Old version of ls:
#ls -l $([ ] && -F is turned off) -a /etc
ls -l -a /etc
$(: ...) is a little less ugly, but still not good.
Here's my solution for inline comments in between multiple piped commands.
Example uncommented code:
#!/bin/sh
cat input.txt \
| grep something \
| sort -r
Solution for a pipe comment (using a helper function):
#!/bin/sh
pipe_comment() {
cat -
}
cat input.txt \
| pipe_comment "filter down to lines that contain the word: something" \
| grep something \
| pipe_comment "reverse sort what is left" \
| sort -r
Or if you prefer, here's the same solution without the helper function, but it's a little messier:
#!/bin/sh
cat input.txt \
| cat - `: filter down to lines that contain the word: something` \
| grep something \
| cat - `: reverse sort what is left` \
| sort -r
Most commands allow args to come in any order. Just move the commented flags to the end of the line:
ls -l -a /etc # -F is turned off
Then to turn it back on, just uncomment and remove the text:
ls -l -a /etc -F
How about storing it in a variable?
#extraargs=-F
ls -l $extraargs -a /etc
If you know a variable is empty, you could use it as a comment. Of course if it is not empty it will mess up your command.
ls -l ${1# -F is turned off} -a /etc
ยง 10.2. Parameter Substitution
For disabling a part of a command like a && b, I simply created an empty script x which is on path, so I can do things like:
mvn install && runProject
when I need to build, and
x mvn install && runProject
when not (using Ctrl + A and Ctrl + E to move to the beginning and end).
As noted in comments, another way to do that is Bash built-in : instead of x:
$ : Hello world, how are you? && echo "Fine."
Fine.
It seems that $(...) doesn't survive from ps -ef.
My scenario is that I want to have a dummy param that can be used to identify the very process. Mostly I use this method, but the method is not workable everywhere. For example, python program.py would be like
mkdir -p MyProgramTag;python MyProgramTag/../program.py
The MyProgramTag would be the tag for identifying the process started.