Jenkins parallel blocks are great but they do raise the bar for human comprehension as they interleave output.
def mysteps = [:]
mysteps['something'] = { sh "do-something.sh" }
if (wantOtherThing) {
mysteps['otherthing'] = { sh "do-otherthing.sh" }
}
parallel mysteps
This executes creating console output like so:
[something] ...
[something] ...
[otherthing] ...
[something] ...
...
The case above offers a simple option - redirect output to a log file and cat that to the log later. If I used a series of jenkins plugins & tasks (e.g. ansible-playbook task) then un-interleaving the output is more of a challenge. In that case the only option seems to be create specific log files and store them as build outputs.
Is there another approach to keeping the console spartan, comprehensible while still maintaining:
somewhat dynamic console so folks can watch the build
enough debug information so we can tell why a job failed?
If you look at the output in Blue Ocean, it separates the output for each parallel task
Related
I have a script which accepts multiple options as arguments. The list of valid options may be large (>20) which could result with conflicts. Is there a practical way to combine the conflicted ones together, and non-conflicted into another without creating multiple lists, groups, etc. Changes to one group will result in changes to other groups.
For example, the list of available options:-a, -b, -c, -d
The following options conflict: [-a, -c], [-a, -d]
The following options don't conflict: [-a, -b], [-c, -d]
EDIT, A more precise example:
For example, the script allows to start/stop a specific task and has additional options for creating/deleting logs.
A normal start would look like :
./script -start Task -logFile C:\out.tmp
And the script should notify the user in case something like:
./script -start Task -stop Task is executed, since start and stop are two opposite actions.
Another conflicting action:
./sript -start Task -logFile C:\out.tmp -deleteLog C:\out.tmp , which would create a log file and delete it at the same time
Now, if the options are start, stop, logFile, deleteLog,
The following would be conflicting: [start, stop], [logFile, deleteLog]
The following would not be conflicting: [start, logFile], [stop, deleteLog]
Let us assume that, as you are writing the source, you can decide how to organize your arguments for parsing time. For example (using JSON notation, you can easily adapt this to C structs, Java enums or what-have-you), you can annotate the available options to explicitly indicate which of them conflict:
const options = [
{
name: "start",
description: "starts foobaring the fizzbuzz",
parameters: [
{
name: "task",
type: "string",
optional: false,
description: "the type of task to foobar"
}
]
conflicts: ["stop"] // <-- explicit simple conflict detection
},
{ ... }
]
This would be used by a command-line parsing module to
generate a nice help screen, possibly including conflicting options
return a map of command-line options, say parsedArgs, so that parsedArgs['start'] would correspond to the parameters of the start parameter if it was specified.
detect simple conflicts, by complaining if conflicting options are detected at parse-time.
Note that there may be additional conflicts that it may not be worthwhile to detect at the parsing stage. For instance, if the value of option foo must be larger than bar + baz, it is better to code a check for this fact after parsing, rather than complicating the parser to handle arbitrary relationships between option values.
I've been struggling with this for a day and a half or so. I'm trying to replicate the following Ant concept in Gradle:
<target name="test">
...
<runexe name="<filename> params="<params>" />
...
</target>
where runexe is declared elsewhere as
<macrodef name="runexe" >
...
</macrodef>
and might also be a taskdef or a scriptdef i.e. I'd like to be able to call a reusable, pre-defined block of code and pass it the necessary parameters from within Gradle tasks. I've tried many things. I can create a task that runs the exe without any trouble:
task runexe(type: Exec){
commandLine 'cmd', '/c', 'dir', '/B'
}
task test(dependsOn: 'runexe') {
runexe {
commandLine 'cmd', '/c', 'dir', '/N', 'e:\\utilities\\'
}
}
test << {
println "Testing..."
// I want to call runexe here.
...
}
and use dependsOn to have it run. However this doesn't allow me to run runexe precisely when I need to. I've experimented extensively with executable, args and commandLine. I've played around with exec and tried several different variations found here and around the 'net. I've also been working with the free books available from the Gradle site.
What I need to do is read a list of files from a directory and pass each file to the application with some other arguments. The list of files won't be known until execution time i.e. until the script reads them, the list can vary and the call needs to be made repeatedly.
My best option currently appears to be what I found here, which may be fine, but it just seems that there should be a better way. I understand that tasks are meant to be called once and that you can't call a task from within another task or pass one parameters but I'd dearly like to know what the correct approach to this is in Gradle. I'm hoping that one of the Gradle designers might be kind enough to enlighten me as this is a question asked frequently all over the web and I'm yet to find a clear answer or a solution that I can make work.
If your task needs to read file names, then I suggest to use the provided API instead of executing commands. Also using exec will make it OS specific, therefore not necessarily portable on different OS.
Here's how to do it:
task hello {
doLast {
def tree = fileTree(dir: '/tmp/test/txt')
def array = []
tree.each {
array << it
print "${it.getName()} added to array!\n"
}
}
}
I ultimately went with this, mentioned above. I have exec {} working well in several places and it seems to be the best option for this use case.
To please an overzealous moderator, that means this:
def doMyThing(String target) {
exec {
executable "something.sh"
args "-t", target
}
}
as mentioned above. This provides the same ultimate functionality.
Main question: Would groovy's execute() method allow me to run a command that takes a file as an argument, any maybe run the command in background mode?
Here is my issue. I was able to use groovy's execute() for simple commands like ls for example. Suppose now I want to start a process like Kafka from a groovy script (end result is to replace bash files with groovy scripts). So I start with these lines:
def kafkaHome = "Users/mememe/kafka_2.11-0.9.0.1"
def zkStart = "$kafkaHome/bin/zookeeper-server-start.sh"
def zkPropsFile = "$kafkaHome/config/zookeeper.properties"
Now, executing the command below form my mac terminal:
/Users/mememe/kafka_2.11-0.9.0.1/bin/zookeeper-server-start.sh /Users/mememe/kafka_2.11-0.9.0.1/config/zookeeper.properties
starts up the the process just fine. And, executing this statement:
println "$zkStart $zkPropsFile"
prints the above command line as is. However, executing this command from within the groovy script:
println "$zkStart $zkPropsFile".execute().text
simply hangs! And trying this:
println "$zkStart $zkPropsFile &".execute().text
where I make it a background process goes further, but starts complaining about the input file and throws this exception:
java.lang.NumberFormatException: For input string: "/Users/mememe/kafka_2.11-0.9.0.1/config/zookeeper.properties"
Trying this gives the same exception as above:
def proc = ["$zkStart", "$zkPropsFile", "&"].execute()
println proc.text
What am I missing please? Thank you.
Yes, try using the consumeProcessOutpusStream() method:
def os = new File("/some/path/toyour/file.log").newOutputStream()
"$zkStart $zkPropsFile".execute().consumeProcessOutputStream(os)
You can find the the method in the Groovy docs for the Process class:
http://docs.groovy-lang.org/docs/groovy-1.7.2/html/groovy-jdk/java/lang/Process.html
Which states:
Gets the output and error streams from a process and reads them to keep the process from blocking due to a full output buffer. The stream data is thrown away but blocking due to a full output buffer is avoided. Use this method if you don't care about the standard or error output and just want the process to run silently - use carefully however, because since the stream data is thrown away, it might be difficult to track down when something goes wrong. For this, two Threads are started, so this method will return immediately.
In my build.gradle script, I have a lot of tasks, each depending on zero or more other tasks.
There are three 'main' tasks which can be called: moduleInstallation, backupFiles and restoreFiles.
Here's the question: I would like to be able to tell Gradle which tasks to execute and which don't need to execute. For example, when calling moduleInstallation, I want all depending tasks to execute (regardless of their UP-TO-DATE flag), but not the restore tasks. I've tried altering the phase in which the tasks get executed (e.g. config phase, execution phase,...) and a couple of other things, but all tasks just keep getting executed.
A solution I've thought of was just stating in the main tasks that, when this main task is called (f.e. moduleInstallation), we set the UP-TO-DATE flag of all non-related tasks to false, so they don't get executed. Is that possible?
EDIT: Here's an example:
When moduleInstallation is called (which depends on backupFiles), restoreFiles (which depends on restoreFromDate) is executed too.
First main action
task moduleInstallation << {
println "Hello from moduleInstallation"
}
task backupFiles {
doLast {
println "Hello from backupFiles"
}
}
Second main action
task restoreFiles {
println "Hello from restoreFiles"
}
task restoreFromDate {
println "Hello from restoreFromDate"
}
Dependencies:
moduleInstallation.dependsOn backupFiles
restoreFiles.dependsOn restoreFromDate
So when I type gradle moduleInstallation in the terminal, I get the following output:
Hello from restoreFromDate
Hello from restoreFiles
Hello from backupFiles
Hello from moduleInstallation
The second snippet has to use doLast (or its << shortcut) like the first snippet. Otherwise, the code is configuration code and will always be evaluated, no matter which tasks are eventually going to be executed. In other words, it's not the restoreFiles and restoreFromDate tasks that are being executed here (as one can tell from the bits of command line output that you don't show), but (only) their configuration code.
To better understand what's going on here (which is crucial for understanding Gradle), I recommend to study the Build Lifecycle chapter in the Gradle User Guide.
I have a Capistrano deploy file (Capfile) that is rather large, contains a few namespaces and generally has a lot of information already in it. My ultimate goal is, using the Tinder gem, paste the output of the entire deployment into Campfire. I have Tinder setup properly already.
I looked into using the Capistrano capture method, but that only works for the first host. Additionally that would be a lot of work to go through and add something like:
output << capture 'foocommand'
Specifically, I am looking to capture the output of any deployment from that file into a variable (in addition to putting it to STDOUT so I can see it), then pass that output in the variable into a function called notify_campfire. Since the notify_campfire function is getting called at the end of a task (every task regardless of the namespace), it should have the task name available to it and the output (which is stored in that output variable). Any thoughts on how to accomplish this would be greatly appreciated.
I recommend not messing with the Capistrano logger, Instead use what unix gives you and use pipes:
cap deploy | my_logger.rb
Where your logger reads STDIN and STDOUT and both records, and pipes it back to the appropriate stream.
For an alternative, the Engineyard cap recipies have a logger – this might be a useful reference if you do need to edit the code, but I recommend not doing.
It's sort of a hackish means of solving your problem, but you could try running the deploy task in a Rake task and capturing the output using %x.
# ...in your Rakefile...
task :deploy_and_notify do
output = %x[ cap deploy ] # Run your deploy task here.
notify_campfire(output)
puts output # Echo the output.
end