I'm using buildbot for few months now and I'm really happy with it. It's plugged with github.
I'm working on a image processing software, and processing time is really important. Until recently, I was doing automatic build + tests, and I'm now monitoring time spend with the tests.
Because monitoring of processing time has been implemented just recently, I'd like to build all the previous commit since few months, so I can see some potential processing time drawback.
I can trigger manual build on a particular commit with the ForceScheduler, but is there an easy way to do that on the last 500 commits for instance ?
Assuming you have PBSource configured as a changesource, you can simply write a script to call buildbot sendchange on the command line.
buildbot sendchange --master {MASTERHOST}:{PORT} --auth {USER}:{PASS}
--who {USER} {FILENAMES..}
see: http://buildbot.readthedocs.org/en/latest/manual/cmdline.html
Related
I’m trying to run a maintenance workflow on all agents.
It’s a cleaning job like “docker system prune”. I wish it runs on all self-host agents (about 12 agents) weekly on Sunday night.
I noticed that workflows can run with on schedule event. This is great.
But I didn’t find a way to make all self-hosted agents to run the workflow. Any suggestions?
I believe that this problem is more about your operating system(s), not about workflows on GitHub side. It might be possible to do as workflow, but then your agents are requiring either to be on host operating system, or have access for Docker socket. (I don't know how you are hosting them). Either way, might be insecure depending on if host is used on something else as well.
As GitHub docs are stating, you are responsible about your operating system.
In general, you can schedule maintenance jobs with cron, might be the most used one. How to install it, depends on your operating system.
To add scheduled work, run command crontab -e, select editor and add line to the end:
0 3 * * 0 /usr/bin/docker system prune -f to run 03:00 AM Sunday weekly.
However, if you really want to use workflows, you could read some docs on here. It states, that "Labels allow you to send workflow jobs to specific types of self-hosted runners, based on their shared characteristics." So you could create specific maintenance job for every runner with different label. This requires many scheduled jobs as runners are not intended to launch multiple times for same job.
I recently submitted a training job with a command that looked like:
gcloud ai-platform jobs submit training foo --region us-west2 --master-image-uri us.gcr.io/bar:latest -- baz qux
(more on how this command works here: https://cloud.google.com/ml-engine/docs/training-jobs)
There was a bug in my code which cause the job to just keep running, rather than terminate. Two weeks and $61 later, I discovered my error and cancelled the job. I want to make sure I don't make that kind of mistake again.
I'm considering using the timeout command within the training container to kill the process if it takes too long (typical runtime is about 2 or 3 hours), but rather than trust the container to kill itself, I would prefer to configure GCP to kill it externally.
Is there a way to achieve this?
As a workaround, you could write a small script that runs your command and then sleeps the time you want until running a cancel job command.
As a timeout definition is not available in AI Platform training service, I took the liberty to open a Public Issue with a Feature Request to record the lack of this command. You can track the PI progress here.
Except the script mentioned above, you can also try:
TimeOut Keras callback, or timeout= Optuna param (depending on which library you actually use)
Cron-triggered Lambda (Cloud Function)
I've just started using wercker and I'd like a job to run regularly (e.g. daily, hourly). I realize this may be an anti-pattern, but is it possible? My intent is not to keep the container running indefinitely, just that my workflow is executed on a particular interval.
You can use a call to the Wercker API to trigger a build for any project which is set up already in Wercker.
So maybe set up a cron job somewhere that uses curl to make the right API call?
TXTFIT = test execution time for individual test
Hello,
I'm using Sonar to analyze my Maven Java Project. I'm testing with JUnit and generating reports on the test execution time with the Maven Surefire plugin.
In my Sonar I can see the test execution time and drill down to see how long each individual test took. In the time machine I can only compare the overall test execution time between two releases.
What I want is to see how the TXTFIT changed from the last version.
For example:
In version 1.0 of my software the htmlParserTest() takes 1sec to complete. In version 1.1 I add a whole bunch of test (so the overall execution time is going the be way longer) but also the htmlParserTest() suddenly takes 2secs, I want to be notified "Hey mate, the htmlParserTest() takes twice as long as it used to. You should take a look at it".
What I'm currently struggling to find out:
How exactly do the TXTFIT get from the surefire xml report into sonar?
I'm currently looking at AbstractSurefireParser.java
but I'm not sure if that's actually the default surefire plugin.
I was looking at 5 year old stuff. I'm currently checking out this. Still have no idea, where Sonar is getting the TXTFIT from and how or where it is connecting it to the Source Files.
Can I find the TXTFIT in the Sonar DB?
I'm looking at the local DB from my test Sonar with DBVisualizer and I don't really know where to look. The SNAPSHOT_DATA doesn't seem like it's readable by humans.
Are the TXTFIT even saved in the DB?
Depending on this I have to write a Sensor that actually saves them or a widget that simply shows them on the dashboard
Any help is very much appreciated!
The web service api/tests/* introduced in version 5.2 allows to get this information. Example: http://nemo.sonarqube.org/api/tests/list?testFileUuid=8e3347d5-8b17-45ac-a6b0-45df2b54cd3c
We have CC.NET as continuous integration environment. CC gets every commits from Git, build and publish to server.
This is config:
<buildpublisher
<sourceDir>Path_to_dir_with_source</sourceDir>
<publishDir>path_to_deploy</publishDir>
<cleanPublishDirPriorToCopy>true</cleanPublishDirPriorToCopy>
<useLabelSubDirectory>false</useLabelSubDirectory>
<alwaysPublish>true</alwaysPublish>
</buildpublisher>
But our QA engeneer wants to get "fresh" build every morning, not 20 times in a day :)
Anybody know how to make this with CC.NET?
[UPDATE]
We still need to build every commit, but put this commit to web server only once in a day
Use the ScheduleTrigger block:
<scheduleTrigger time="23:30" buildCondition="ForceBuild" name="Scheduled">
<weekDays>
<weekDay>Monday</weekDay>
</weekDays>
</scheduleTrigger>
You can also use a cronTrigger, or an intervalTrigger, but for your case, the scheduleTrigger seems simplest.
http://www.cruisecontrolnet.org/projects/ccnet/wiki/Trigger_Blocks
Added to address the comment:
You can have multiple triggers in one Project block.
We have an Interval Trigger on our end as well as a Schedule Trigger. You can have as many triggers as you need.
If you want to keep the builds separate, you can also have a compeltely separate Project block - one that operates on a Schedule trigger, and one that operates on an interval, but there are risks, and the configuration needs to be planned. The following comes to mind as potential considerations:
If you do this, be sure you check out the source code to different directories, or you can run into conflicts if both are running at the same time.
Even if the source code checkout is to different directories, you can still run into conflicts if both projects are publishing to the same output location.
You can add a schedule trigger that forces a build of your project at a specific time, i.e. the following would run a build every work-day at 5 am in the morning:
<triggers>
<scheduleTrigger time="05:00" buildCondition="ForceBuild">
<weekDays>
<weekDay>Monday</weekDay>
<weekDay>Tuesday</weekDay>
<weekDay>Wednesday</weekDay>
<weekDay>Thursday</weekDay>
<weekDay>Friday</weekDay>
</weekDays>
</scheduleTrigger>
</triggers>