I have a pipeline of bash commands that looks like this:
cat report.json | \
./jq '.outdated.dependencies[] | {group, name, "version": .available.milestone} | .group + ":" + .name + ":" + .version' | \
xargs -0 -d'\n' -I DEPENDENCY echo ./gradlew clean test -PdependencyOverrides=DEPENDENCY
The output of this command is:
./gradlew clean test -PdependencyOverrides="org.flywaydb:flyway-core:4.0.3"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-c3p0:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-core:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-entitymanager:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-java8:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-validator:5.3.3.Final"
./gradlew clean test -PdependencyOverrides="com.fasterxml.jackson.core:jackson-databind:2.8.5"
./gradlew clean test -PdependencyOverrides="com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.8.5"
./gradlew clean test -PdependencyOverrides="javax.el:javax.el-api:3.0.1-b04"
./gradlew clean test -PdependencyOverrides="org.apache.logging.log4j:log4j-api:2.7"
./gradlew clean test -PdependencyOverrides="org.apache.logging.log4j:log4j-core:2.7"
./gradlew clean test -PdependencyOverrides="org.apache.logging.log4j:log4j-slf4j-impl:2.7"
./gradlew clean test -PdependencyOverrides="org.mockito:mockito-core:2.2.28"
./gradlew clean test -PdependencyOverrides="org.postgresql:postgresql:9.4.1212"
./gradlew clean test -PdependencyOverrides="com.getsentry.raven:raven-log4j2:7.8.1"
./gradlew clean test -PdependencyOverrides="com.sparkjava:spark-core:2.5.4"
./gradlew clean test -PdependencyOverrides="org.springframework:spring-jdbc:4.3.4.RELEASE"
./gradlew clean test -PdependencyOverrides="org.springframework.restdocs:spring-restdocs-restassured:1.1.2.RELEASE"
This is exactly what I am expecting. If I copy and paste one of these commands, they do exactly what I want them to do. Now I don't want to print these commands but directly execute them. However, as soon as I remove the echo from the xargs command, it doesn't work. "It doesn't work" means that the property is not correctly passed to gradle.
In order to debug this issue, I tried to append | bash -x to my pipeline which revealed the issue but I don't know how to solve it.
Appending this yields the following output as the first line:
+ ./gradlew clean test $'-PdependencyOverrides=org.flywaydb:flyway-core:4.0.3\r'
I assume, the problem is that bash for some reason I don't know puts the arguments in quotes and prepends them with $. If I execute the command exactly as it is printed here, the same error occurs as if I let xargs directly run the command.
What do I have to change so the commands are executed exactly the way they are printed when I add the echo to the xargs command?
Here is the report.json file if anyone wants to reproduce this locally and jq can be downloaded here.
Gradle is not needed to reproduce this problem, as executing the command in a folder without a gradlew file just yields:
+ ./gradlew clean test $'-PdependencyOverrides=org.flywaydb:flyway-core:4.0.3\r'
bash: line 1: ./gradlew: No such file or directory
which still demonstrates the problem.
I am not sure what the exact problem is, so if there is a better title feel free to change it.
$'' is a valid way of quoting strings in Bash. It's being used because xtrace escapes strings if necessary to print the exact command which was run. You get the same kind of result if you try to printf '%q\n' "$something", where something contains special characters like newline, tab, backspace, etc.
\r won't be visible when you look at output in some editor/terminal which does not escape special characters. For example, if you open a file with \r at the end of lines it will simply show [dos] at the bottom of the window to indicate that the line separator is \r\n (DOS/Windows standard) rather than \n (*nix standard).
Related
I am running the following command, from a sh file.
echo "This is job" $SLURM_ARRAY_TASK_ID
#! Command line that we want to run:
SLURM_ARRAY_TASK_ID=0
varData='mvn -e clean compile exec:java -Dexec.mainClass=App.Main -Dexec.args="wikiLineSPlits/wiki-004/ wiki-004-'$SLURM_ARRAY_TASK_ID'.txt wiki-004-'$SLURM_ARRAY_TASK_ID' wikiOuts/ wiki-004"'
echo $varData
$varData
Here, running the code results in the following output:
mvn -e clean compile exec:java -Dexec.mainClass=App.Main -Dexec.args="wikiLineSPlits/wiki-004/ wiki-004-0.txt wiki-004-0 wikiOuts/ wiki-004"
Unknown lifecycle phase "wiki-004-0.txt"
But if I copy-paste the output from echo $varData to my terminal screen, then the program executes scuesfully. I am not sure what is the source of error. Any help would be appreciated.
Output from echo $varData:
mvn -e clean compile exec:java -Dexec.mainClass=App.Main -Dexec.args="/home/ak2329/rds/hpc-work/feverDataset/wikiLineSPlits/wiki-004/ wiki-004-0.txt wiki-004-0 /home/ak2329/rds/hpc-work/feverDataset/wikiOuts/ wiki-004"
I would try:
$(echo $varData)
As the last line in your script
The problem is that each stage of your maven command line is being passed the -Dexec arguments, and neither clean nor compile knows what to do with them. If you rewrite your command as to stages
mvn clean compile
and
mvn exec
and only pass the -Dexec args to the second stage, then you should not get those errors.
I finally decided to directly use the shell command, instead of storing it to a variable.
mvn -e clean compile exec:java -Dexec.mainClass=App.Main -Dexec.args="wikiLineSPlits/wiki-004/ wiki-004-'$SLURM_ARRAY_TASK_ID'.txt wiki-004-'$SLURM_ARRAY_TASK_ID' wikiOuts/ wiki-004"
Previously the command used was:
varData='mvn -e clean compile exec:java -Dexec.mainClass=App.Main -Dexec.args="wikiLineSPlits/wiki-004/ wiki-004-'$SLURM_ARRAY_TASK_ID'.txt wiki-004-'$SLURM_ARRAY_TASK_ID' wikiOuts/ wiki-004"'
echo $varData
In a TeamCity job I'm trying to read Maven project version from the root pom.xml e.g. 1.0.0-SNAPSHOT and change the build number to 1.0.0-N where N is a value of %build.counter%. The goal is to replace SNAPSHOT with a build identifier.
So far I tried to combine build-helper:parse-version together with exec:exec to call echo to write ##teamcity[buildNumber '1.0.0-1'] in the build log and change the build number.
On my workstation I can execute following Maven command to print the ##teamcity instruction in the command line:
mvn build-helper:parse-version exec:exec \
-Dexec.executable="echo" \
-Dexec.args="##teamcity[buildNumber '${parsedVersion.majorVersion}.${parsedVersion.minorVersion}.${parsedVersion.incrementalVersion}-%build.counter%']"
However when I define a Maven Step in my TeamCity build then TeamCity parses the ##teamcity instruction directly from the Maven command line invocation printed in the build logs resulting in following Maven call:
mvn build-helper:parse-version exec:exec \
-Dexec.executable="echo" \
-Dexec.args=""
and following being done by TeamCity:
##teamcity[buildNumber '${parsedVersion.majorVersion}.${parsedVersion.minorVersion}.${parsedVersion.incrementalVersion}-11']
Is there a way to escape the parameter containing ##teamcity in the Maven command line invocation so TeamCity ignores it during Maven invocation but passes it to exec:exec Maven goal so it can output:
##teamcity[buildNumber '1.0.0-11']
Please let me know if there is an easier way to achieve my goal using TeamCity Build Steps and without modifying project's pom.xml. Do I have to use PowerShell or Batch script for this?
Interesting use case. I tried experimenting with this and got the positive result.
The echo command was omitting the single quote in the final output so I never managed to get TeamCity to consume the service message.
Instead of calling the Maven build step, I defined Command Line build step that launched a script. The scripts task is to generate the desired output:
##teamcity[buildNumber '1.0.0-61']
The script itself is really ugly, but kinda works. Perhaps you can come up with more elegant solution:
#!/bin/bash
export ARGUMENTS="teamcity[buildNumber 'Q\${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.
mvn build-helper:parse-version exec:exec -Dexec.executable="echo" -Dexec.args="$ARGUMENTS" | grep teamcity | sed "s/Q/\'/g" | awk '{print "##"$0}'
For some reason, if I put ## into the initial string and didn't use awk at the end of the command sequence, then it didn't work and I ended up with a result like '1.0.0-???', hence the hack with awk at the end.
Hope this helps.
I am trying to setup Travis CI to build a latex report. When building the latex report some steps have to be repeated and so the first time they are called there is a non-zero return code.
My travis.yml so far is
language: R
before_install:
- tlmgr install index
script:
- latex report
- bibtex report
- latex report
- latex report
- dvipdf report.dvi report.pdf
However in Travis Docs it states
If script returns a non-zero exit code, the build is failed, but continues to run before being marked as failed.
So if my first latex report command has a non zero return code it will fail the build.
I would only like the build to fail if the last latex report or dvipdf report failed.
Does anyone have any idea or help?
Thanks in advance.
Just append || true to your command.
(complex) Example:
- (docker run --rm -v $(pwd)/example:/workdir stocker-alert || true) 2>&1 | tee >(cat) | grep 'Price change within 1 day'
The docker command returns < 0 (because it's a negative test), but we want to continue anyway
2>&1 - stderr is forwarded to stdin (to be picked up by grep later)
tee - the output is printed (for debugging) and forwarded to grep
Finally grep asserts if the output contains a required string. If not, grep returns > 0 failing the build.
If we wanted to ignore greps result we would need another ||true after grep.
Taken from schnatterer/stock-alert.
Not directly related with your original question but I had quite the same problem.
I found a solution using latexmk. This runs latex and bibtex as many times as needed.
If you look at my Travis configuration file :
https://github.com/73VW/TechnicalReport/blob/master/.travis.yml
You will see that you simply have to add it in apt dependencies.
Then you can run it like this: latexmk -pdf -xelatex [Your_latex_file]
I have created a new gradle task in build.gradle:
task callCL(type: Exec) {
println "hello"
commandLine './rerun.sh'
}
Which suppose to run rerun.sh:
#!/bin/bash
cucumber -f rerun --out rerun.txt
file="rerun.txt"
if [ -f "$file" ] then
cucumber #rerun.txt
rm $file
fi
I'm using IntelliJ as an IDE. How can I run this task?
I have tried to run in the zshell console and got this error:
gradle callCL
zsh: command not found: gradle
But in the IDE I use gradle all the time so it must be installed.
How can I fix this? And is my writing ok?
Try this:
1. Make sure GRADLE_HOME, GRADLE_OPTS are set.
2. Make sure $PATH has GRADLE_HOME/bin in it.
3. which gradle should return you a valid output.
4. then, see below, if this works on command prompt, then your IDE setting just need to know where's is GRADLE_HOME aka its installed / executable (either gradle or gradle.bat)
NOTE: I have used my own dummy rerun.sh file, you can you use build.gradle (as shown below).
$ cat rerun.sh
#!/bin/bash
echo Im re-running a command echo
echo something
echo ...
echo
$ cat build.gradle
task callCL(type: Exec) {
println "-----"
println "hello"
println "-----"
executable "bash"
args "-c", "bash ./rerun.sh"
//The following will do as well as magic number in bash is already set to #!/bin/bash
//args "-c", "./rerun.sh"
}
$ /cygdrive/c/gradle-2.0/bin/gradle callCL
-----
hello
-----
:callCL
Im re-running a command echo
something
...
BUILD SUCCESSFUL
Total time: 2.006 secs
This looks like problem with gradle not being found on path (in your shell).
You may use GVM to easily install gradle so that its available on your PATH.
I'd like to know whether there's a way to kind of 'query' the state of an Maven execution from within a Shell script (used for build process).
The point is that I would like to let the whole build script fail as soon as a single error appears within one of the Shell scripts Maven executions.
e.g.
(0) mvn -f .../someDir clean
(1) mvn -f .../1/pom.xml install
(2) mvn -f .../2/pom.xml -PgenerateWadl
So if e.g. there occurs an error within (0), then (1) and (2) must no more be executed, but instead the build script should quit with an error message directly after (0).
I'm not THAT much into Shell scripting, but I know of the $? variable to get the return value of an earlier execution. But as Maven simply seems to write errors out to the console, this might not work, does it?
I would have liked to research more information concerning the "$?", but it's quite hard to google for it.
A simple way is to use the -e option.
set -e
mvn -f .../somedir clean
mvn -f .../otherdir install
mvn -f .../thirddir -PgenerateWadl package
This will automatically have the shell exit with error if any command in sequence exits with a non-zero status (unless it is part of an explicit test such as an if or while or a chain like a || b).
You can watch this happen by inserting a call to false between any of the Maven calls.
See the set POSIX spec for details on set -e.
mvn ... && mvn ... && mvn ...
The execution will only proceed if the previous one was successful.