In a TeamCity job I'm trying to read Maven project version from the root pom.xml e.g. 1.0.0-SNAPSHOT and change the build number to 1.0.0-N where N is a value of %build.counter%. The goal is to replace SNAPSHOT with a build identifier.
So far I tried to combine build-helper:parse-version together with exec:exec to call echo to write ##teamcity[buildNumber '1.0.0-1'] in the build log and change the build number.
On my workstation I can execute following Maven command to print the ##teamcity instruction in the command line:
mvn build-helper:parse-version exec:exec \
-Dexec.executable="echo" \
-Dexec.args="##teamcity[buildNumber '${parsedVersion.majorVersion}.${parsedVersion.minorVersion}.${parsedVersion.incrementalVersion}-%build.counter%']"
However when I define a Maven Step in my TeamCity build then TeamCity parses the ##teamcity instruction directly from the Maven command line invocation printed in the build logs resulting in following Maven call:
mvn build-helper:parse-version exec:exec \
-Dexec.executable="echo" \
-Dexec.args=""
and following being done by TeamCity:
##teamcity[buildNumber '${parsedVersion.majorVersion}.${parsedVersion.minorVersion}.${parsedVersion.incrementalVersion}-11']
Is there a way to escape the parameter containing ##teamcity in the Maven command line invocation so TeamCity ignores it during Maven invocation but passes it to exec:exec Maven goal so it can output:
##teamcity[buildNumber '1.0.0-11']
Please let me know if there is an easier way to achieve my goal using TeamCity Build Steps and without modifying project's pom.xml. Do I have to use PowerShell or Batch script for this?
Interesting use case. I tried experimenting with this and got the positive result.
The echo command was omitting the single quote in the final output so I never managed to get TeamCity to consume the service message.
Instead of calling the Maven build step, I defined Command Line build step that launched a script. The scripts task is to generate the desired output:
##teamcity[buildNumber '1.0.0-61']
The script itself is really ugly, but kinda works. Perhaps you can come up with more elegant solution:
#!/bin/bash
export ARGUMENTS="teamcity[buildNumber 'Q\${parsedVersion.majorVersion}.\${parsedVersion.minorVersion}.\${parsedVersion.
mvn build-helper:parse-version exec:exec -Dexec.executable="echo" -Dexec.args="$ARGUMENTS" | grep teamcity | sed "s/Q/\'/g" | awk '{print "##"$0}'
For some reason, if I put ## into the initial string and didn't use awk at the end of the command sequence, then it didn't work and I ended up with a result like '1.0.0-???', hence the hack with awk at the end.
Hope this helps.
Related
I am running the following command, from a sh file.
echo "This is job" $SLURM_ARRAY_TASK_ID
#! Command line that we want to run:
SLURM_ARRAY_TASK_ID=0
varData='mvn -e clean compile exec:java -Dexec.mainClass=App.Main -Dexec.args="wikiLineSPlits/wiki-004/ wiki-004-'$SLURM_ARRAY_TASK_ID'.txt wiki-004-'$SLURM_ARRAY_TASK_ID' wikiOuts/ wiki-004"'
echo $varData
$varData
Here, running the code results in the following output:
mvn -e clean compile exec:java -Dexec.mainClass=App.Main -Dexec.args="wikiLineSPlits/wiki-004/ wiki-004-0.txt wiki-004-0 wikiOuts/ wiki-004"
Unknown lifecycle phase "wiki-004-0.txt"
But if I copy-paste the output from echo $varData to my terminal screen, then the program executes scuesfully. I am not sure what is the source of error. Any help would be appreciated.
Output from echo $varData:
mvn -e clean compile exec:java -Dexec.mainClass=App.Main -Dexec.args="/home/ak2329/rds/hpc-work/feverDataset/wikiLineSPlits/wiki-004/ wiki-004-0.txt wiki-004-0 /home/ak2329/rds/hpc-work/feverDataset/wikiOuts/ wiki-004"
I would try:
$(echo $varData)
As the last line in your script
The problem is that each stage of your maven command line is being passed the -Dexec arguments, and neither clean nor compile knows what to do with them. If you rewrite your command as to stages
mvn clean compile
and
mvn exec
and only pass the -Dexec args to the second stage, then you should not get those errors.
I finally decided to directly use the shell command, instead of storing it to a variable.
mvn -e clean compile exec:java -Dexec.mainClass=App.Main -Dexec.args="wikiLineSPlits/wiki-004/ wiki-004-'$SLURM_ARRAY_TASK_ID'.txt wiki-004-'$SLURM_ARRAY_TASK_ID' wikiOuts/ wiki-004"
Previously the command used was:
varData='mvn -e clean compile exec:java -Dexec.mainClass=App.Main -Dexec.args="wikiLineSPlits/wiki-004/ wiki-004-'$SLURM_ARRAY_TASK_ID'.txt wiki-004-'$SLURM_ARRAY_TASK_ID' wikiOuts/ wiki-004"'
echo $varData
I have a pipeline of bash commands that looks like this:
cat report.json | \
./jq '.outdated.dependencies[] | {group, name, "version": .available.milestone} | .group + ":" + .name + ":" + .version' | \
xargs -0 -d'\n' -I DEPENDENCY echo ./gradlew clean test -PdependencyOverrides=DEPENDENCY
The output of this command is:
./gradlew clean test -PdependencyOverrides="org.flywaydb:flyway-core:4.0.3"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-c3p0:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-core:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-entitymanager:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-java8:5.2.5.Final"
./gradlew clean test -PdependencyOverrides="org.hibernate:hibernate-validator:5.3.3.Final"
./gradlew clean test -PdependencyOverrides="com.fasterxml.jackson.core:jackson-databind:2.8.5"
./gradlew clean test -PdependencyOverrides="com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.8.5"
./gradlew clean test -PdependencyOverrides="javax.el:javax.el-api:3.0.1-b04"
./gradlew clean test -PdependencyOverrides="org.apache.logging.log4j:log4j-api:2.7"
./gradlew clean test -PdependencyOverrides="org.apache.logging.log4j:log4j-core:2.7"
./gradlew clean test -PdependencyOverrides="org.apache.logging.log4j:log4j-slf4j-impl:2.7"
./gradlew clean test -PdependencyOverrides="org.mockito:mockito-core:2.2.28"
./gradlew clean test -PdependencyOverrides="org.postgresql:postgresql:9.4.1212"
./gradlew clean test -PdependencyOverrides="com.getsentry.raven:raven-log4j2:7.8.1"
./gradlew clean test -PdependencyOverrides="com.sparkjava:spark-core:2.5.4"
./gradlew clean test -PdependencyOverrides="org.springframework:spring-jdbc:4.3.4.RELEASE"
./gradlew clean test -PdependencyOverrides="org.springframework.restdocs:spring-restdocs-restassured:1.1.2.RELEASE"
This is exactly what I am expecting. If I copy and paste one of these commands, they do exactly what I want them to do. Now I don't want to print these commands but directly execute them. However, as soon as I remove the echo from the xargs command, it doesn't work. "It doesn't work" means that the property is not correctly passed to gradle.
In order to debug this issue, I tried to append | bash -x to my pipeline which revealed the issue but I don't know how to solve it.
Appending this yields the following output as the first line:
+ ./gradlew clean test $'-PdependencyOverrides=org.flywaydb:flyway-core:4.0.3\r'
I assume, the problem is that bash for some reason I don't know puts the arguments in quotes and prepends them with $. If I execute the command exactly as it is printed here, the same error occurs as if I let xargs directly run the command.
What do I have to change so the commands are executed exactly the way they are printed when I add the echo to the xargs command?
Here is the report.json file if anyone wants to reproduce this locally and jq can be downloaded here.
Gradle is not needed to reproduce this problem, as executing the command in a folder without a gradlew file just yields:
+ ./gradlew clean test $'-PdependencyOverrides=org.flywaydb:flyway-core:4.0.3\r'
bash: line 1: ./gradlew: No such file or directory
which still demonstrates the problem.
I am not sure what the exact problem is, so if there is a better title feel free to change it.
$'' is a valid way of quoting strings in Bash. It's being used because xtrace escapes strings if necessary to print the exact command which was run. You get the same kind of result if you try to printf '%q\n' "$something", where something contains special characters like newline, tab, backspace, etc.
\r won't be visible when you look at output in some editor/terminal which does not escape special characters. For example, if you open a file with \r at the end of lines it will simply show [dos] at the bottom of the window to indicate that the line separator is \r\n (DOS/Windows standard) rather than \n (*nix standard).
I've been trying to find ways to cut my Jenkins build time as much as possible, and thanks to this helpful SO post, I found pbzip2: Utilizing multi core for tar+gzip/bzip compression/decompression
Works great! A 6 min compression time was brought down to 2 mins on my machine with the following:
tar -v -c --use-compress-program=pbzip2 -f parallel.tar.bzip2 myapplication.app
But Jenkins just barfs with a Execute Shell task where I put in the above command:
+ tar -v -c --use-compress-program=pbzip2 -f parallel.tar.bz2 myapplication.app
a myapplication.appBuild step 'Execute shell' marked build as failure
The fact that the "Build step" line is getting mashed together with the output from the tar tells me it might be a background process issue that tar/pbzip2 is introducing.
I've tried introducing a #!/bin/bash -xe and get the same results. I've tried wrapping the tar command in an if statement. I've also tried putting tar in a background thread itself with & and waiting for it. Same result.
Is there any techniques I could implement to help the Jenkins process out?
Found out that even though I can run this command as the jenkins user through command line, pbzip2 wasn't defined in the PATH for the Jenkins job. Pretty misleading since there wasn't useful output.
I'd like to know whether there's a way to kind of 'query' the state of an Maven execution from within a Shell script (used for build process).
The point is that I would like to let the whole build script fail as soon as a single error appears within one of the Shell scripts Maven executions.
e.g.
(0) mvn -f .../someDir clean
(1) mvn -f .../1/pom.xml install
(2) mvn -f .../2/pom.xml -PgenerateWadl
So if e.g. there occurs an error within (0), then (1) and (2) must no more be executed, but instead the build script should quit with an error message directly after (0).
I'm not THAT much into Shell scripting, but I know of the $? variable to get the return value of an earlier execution. But as Maven simply seems to write errors out to the console, this might not work, does it?
I would have liked to research more information concerning the "$?", but it's quite hard to google for it.
A simple way is to use the -e option.
set -e
mvn -f .../somedir clean
mvn -f .../otherdir install
mvn -f .../thirddir -PgenerateWadl package
This will automatically have the shell exit with error if any command in sequence exits with a non-zero status (unless it is part of an explicit test such as an if or while or a chain like a || b).
You can watch this happen by inserting a call to false between any of the Maven calls.
See the set POSIX spec for details on set -e.
mvn ... && mvn ... && mvn ...
The execution will only proceed if the previous one was successful.
I have a batch file that executes three Maven commands, one after the other. Each command can be successfully executed in the script - by itself!. But when I add all three commands to the same file, only the first one executes before the script exits. Any idea why?
mvn install:install-file -DgroupId=gdata -DartifactId=base -Dversion=1.0 -Dfile=gdata-base-1.0.jar -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=blogger -Dversion=2.0 -Dfile=gdata-blogger-2.0.jar -Dpackaging=jar -DgeneratePom=true
mvn install:install-file -DgroupId=gdata -DartifactId=blogger-meta -Dversion=2.0 -Dfile=gdata-blogger-meta-2.0.jar -Dpackaging=jar -DgeneratePom=true
Also, if I copy all three commands and paste them into a command shell (cmd.exe), they execute one after the other with no problem. So this is apparently some issue with the dos batch file.
Maven uses batch files to do its business. With any batch script, you must call another script using the call command so it knows to return back to your script after the called script completes. Try prepending call to all commands.
Another thing you could try is using the start command which should work similarly.
Having call helps. However today it didn't.
This is how I solved it:
Bat file contents (if you want to stop batch when one of cmds errors)
cmd1 && ^
cmd2 && ^
cmd3 && ^
cmd4
Bat file contents (if you want to continue batch when one of cmds errors)
cmd1 & ^
cmd2 & ^
cmd3 & ^
cmd4
To execute more Maven builds from one script you shall use the Windows call function in the following way:
call mvn install:install-file -DgroupId=gdata -DartifactId=base -Dversion=1.0 -Dfile=gdata-base-1.0.jar -Dpackaging=jar -DgeneratePom=true
call mvn install:install-file -DgroupId=gdata -DartifactId=blogger -Dversion=2.0 -Dfile=gdata-blogger-2.0.jar -Dpackaging=jar -DgeneratePom=true
call mvn install:install-file -DgroupId=gdata -DartifactId=blogger-meta -Dversion=2.0 -Dfile=gdata-blogger-meta-2.0.jar -Dpackaging=jar -DgeneratePom=true
It should be that the particular mvn command execs and does not return, thereby not executing the rest of the commands.
Try writing the following batch file and executing it:
Echo one
cmd
Echo two
cmd
Echo three
cmd
Only the first two lines get executed. But if you type "exit" at the command prompt, the next two lines are processed. It's a shell loading another.
To be sure that this is not what is happening in your script, just type "exit" when the first command ends.
HTH!
Dos commands in my batch file were running only when I type EXIT in command/DOS window. This problem solved when I removed CMD from batch file. No need of it.