I'm explicitly mentioning the file names under the custom artifacts path section to deploy the changed/ modified files to the artifactory. Is there any way Teamcity detects only the changed files and moves to the artifactory?
Is it possible to run a custom script for that?
If you are using TeamCity Artifactory plugin, then, the plugin is using checksum deploy, that means that, if I'll use your example, if you have 10 files that 7 were unchanged, the plugin will send the checksum of the file, if the checksum exist in Artifactory, it will not deploy it again but will only create a "pointer" in the database to point to that location. If a file checksum doesn't exist in Artifactory it will then deploy it to Artifactory.
I'm guessing here that this request is to save network traffic and build time, if I'm wrong, could you please share the need behind the question?
Related
I have maven-base ear file.
The ear contains lot of jars the built by same CI-CD process - monolith.
I have implementation of incremental build, that improve the CI-CD times by build the artifacts that changed or affected only. and take the other artifacts from artifactory.
The problem is - when I start new version, I have to build all the artifacts in order to create the artifacts in artifactory the first time. This takes long of time.
Is there a quick way to copy all the artifacts from version 0.1 to version 0.2 in artifactory?
In the beginning of version 0.2, The artifacts are the same.
Thanks
TL;DR
It's not that straight forward.
Personally I think it's not the right way to go. You should probably break your internal dependencies based on their lifecycles, and reuse existing versions instead of rebuilding them.
In more details:
Technically, you can use the JFrog CLI's copy command to copy files in Artifactory. You can also use the Copy Item API directly if you prefer. Copy is a cheap and quick operation in Artifactory because of its checksum based storage (the content is stored only once).
But, that's not enough, mainly because of Maven.
Maven uses a file path format which contains the version - both as a folder and in the file names. The Maven artifacts file path format is (simplified): /<group>/<artifact>/<version>/<artifact>-<version>.<ext>. For example, the jar file of the artifact org.acme:foo:0.1 will have a file like: /org/acme/foo/0.1/foo-0.1.jar.
It means that copying files from folder of 0.1 to 0.2 is not enough, you also need to rename them accordingly.
And that's still not enough - Maven also embeds the the version in files, mainly in the pom file. It means that copying it is also not enough, you need to modify the content of that xml file.
But, it also depends on how you use Artifactory. The best practice is to publish also a build-info when you upload to Artifactory. That usually means that Artifactory stores properties on the files, properties which are specific to the build as part of which they were created and uploaded. These properties are also copied as part of the copy operation mentioned above, and you should remove them from the new version.
To do all of that you can probably write a simple script which lists the files, copy and rename the files, adjust the content of the pom file, and remove the properties.
So my job has 2 parts. The first part downloads all the required maven artifacts from a central online repository. The second part uses the download folder as an online repository(by changing URL from central repo to local folder) to build our product.
But there are certain artifacts which do not have a maven-metadata.xml after the first part of the job. As a result, the second part fails. I don't want to hardcode the online repo and curl as it's not what we're looking for. Neither do I want to copy some other maven-metadata.xml and edit my changes. Also, I cant give the offline option during build, as I do need the artifacts to be picked up from my local repo(downloads).
Any solution to this? Thanks IA.
I am using Artifactory to support an enterprise multi-module project. Often, we change the names of modules and the associated dependencies in POM files are not updated to use the new module name. Because SNAPSHOT dependencies are not automatically cleaned up on a regular interval, these old module references can stay there for months. I discovered a few when I migrated Artifactory to another server and the old module dependencies resulted in build errors. I am building these SNAPSHOT artifacts nightly using Jenkins so I would like some way to automate cleaning up the SNAPSHOT artifacts.
Does Artifactory (or another artifact server such as Nexus) support a concept where if a SNAPSHOT artifact is older than X days, the artifact is deleted? Is there another way to automate artifact server cleanup to accomplish what I want to do? The only thing I can think of is to create a cron job to clear out libs-snapshot-local on a regular interval before the nightly build starts. Has someone already built this capability?
As far as I know, Artifactory doesn't have an automated way to delete modules that are older than a certain value. At my shop we've written a Groovy client that uses Artifactory's REST API to do exactly this.
Note that, if your artifacts are shared libraries, you need to be careful that nothing depends on them before you delete them. Our script takes this into account, too.
If you're interested in following up, post a comment and I'll see if it's OK to share our script with you.
Another solution might be a user plugin. You can write a simple Groovy script that will run in Artifactory itself (as opposite to remote invocation by REST Gareth proposed) on a scheduled basis, searching for artifacts not downloaded for a long time and deleting them.
I've made a Ruby script to delete artifacts which aren't download for X days. The way it works just like what JBaruch mentioned in his answer.
It isn't a plugin. It works with Artifactory Open Source. Plugin is only supported by Artifactory Pro.
The source code: https://gist.github.com/aleung/5203736
I'm building a project, which is made up from several (sometimes unrelated) modules and some more non standard java modules (built with ANT).
Each maven module is deployed to the releases repository on completion.
If the build fails in the middle, I might have some modules already deployed, so if I try to rebuild, the new attempt to deploy will fail since the artifacts are already deployed.
Is it possible to force a deploy or instead, remove the deployed artifact before I deploy again?
It sounds like the middleware admins have configured your remote repo instance (Nexus or Artifactory or whatever) to not allow artifact redeployment, and as #khmarbaise says there are good reasons for that. Nexus can be configured to allow artifact deletion by users in a particular role or with artifact deletion privileges. If your admins have it set up that way perhaps you can request the delete privilege and remove the offending artifacts. Or, perhaps the Nexus admin will agree to do it for you.
If neither of these is possible, here are some things to try which might keep this from happening in the future:
If you are using the release plugin, do a dry run (-DdryRun=true on the release:prepare command line) first. Maven should report any errors without committing to SCM.
Try running mvn install on your group of projects first. This will install the artifacts to your local repo, not the remote. If there's a problem you can whack the artifacts out of your local repo and start from scratch, repeating until you get a complete build.
If you are running a multi-module build, there are command line options that allow resuming a Maven build from a particular project forward.
Define -Dmaven.deploy.skip=true on the Maven command line. This is similar to suggestion #2, except Maven will actually load & configure the deploy plugin, it just won't do the actual deploy to the remote repo. Once everything works, remove the skip property.
I know it might be late, but in Nexus there's an option where allows the redeployment of artifacts.
Just select the repositories in the left, choose the repository you want to change the policy and then set it to Allow Redeploy.
The possible options have been increased ;)
Use the parameter deployAtEnd (More information: here). With this parameter the artifacts are deployed only, if all artifacts were built successfully.
I was wondering if there is a Hudson/Jenkins plugin that ties repository artifacts to the build that created them? I was looking at the question "remove artifacts from nexus repository" and thinking that deleting a build in Jenkins should also offer the option to remove the artifacts the build created.
We are currently running Jenkins 1.447 and the Nexus Open Source 1.9.2.3. Our Jenkins builds create artifacts in our Nexus repository using the maven deploy goal. I appears that once those artifacts have been deployed, there is no similarly automated mechanism to remove them. We would like to tie the Jenkins build to its Nexus artifacts. I figure if we have decided to remove the build from Jenkins, we have no use for the build and therefore, have no need to store the artifacts for that build either. We would like deleting the build to trigger deleting the Nexus artifacts.
If there's nothing available I figure I could start writing something, but I wanted to check and see how others handle this.
you can use the REST API from Nexus to build your own Jenkins Plugin that does that Job for you. You could store the Jenkins Build Job number using the Nexus custom-metadata plugin. Once the build is deleted you can have your custom Jenkins Plugin delete all artifacts in Nexus that have that build number in their metadate. I had a similar Problem and wrote a custom Jenkins Plugin. Have a look at the tutorial and the source code on github. It should be pretty straight forward.
Tutorial:
http://blog.codecentric.de/en/2012/08/tutorial-create-a-jenkins-plugin-to-integrate-jenkins-and-nexus-repository/
Sourcecode:
https://github.com/marcelbirkner/nexus-metadata-plugin/
You can purge artifacts from a local repository via the maven-dependency-plugin.
If you have a release it does not make sense to number it based on the numbers of the build server. The usual use case is to use SNAPSHOT's for exactly this purpose. Furthermore the usual use case is to delete SNAPSHOT's based on a scheduled task after a time from Nexus but releases will never be deleted from Nexus.
Since you know the name of the release, you could build a custom job or trigger to use a wget command to delete the artifact from the nexus repository.
As the proper user in Nexus you do have the ability to delete release artifacts, not just SNAPSHOTS.