We've recently implemented Team City and I've been tasked with making it work with our existing VSTS environment. We'd ideally like to have Team City build and VSTS release. We've got the Team City generated Artifacts sitting inside VSTS however you can't trigger CI off them as per this article here; https://www.visualstudio.com/en-us/docs/build/concepts/definitions/release/artifacts
What I'd like to do is have Team City trigger the release via the VSTS REST API. I have have followed the documentation here https://www.visualstudio.com/en-us/docs/integrate/api/rm/releases but I'm having issues getting VSTS to use the Team City artifacts. No matter what I try I get the following error: "VS402962: No artifact version id is specified corresponding to artifact source 'TeamCity Build'. Specify a valid value and try again."
This is what my JSON body for the REST call looks like:
{
"definitionId": 13,
"description": "Build from TeamCity",
"artifacts": [
{
"alias": "Build Dev (TeamCity)",
"instanceReference": {
"id": 160,
"name": "160"
}
}
]
}
I have tried this with and without the name as per this question: VS402881: No artifact version is specified corresponding to artifact source 'MyBuild.' Release Management vNext REST API but I'm not having any luck.
I have found the issue. You need to remove (Team City) from the alias as it's not part of the Artifact Alias.
Related
After export versionCatalogs as a library, and being used by other applications, the error state
> No matching variant of io.github.elye:plugin-dependencies:1.0.0 was found. The consumer was configured to find attribute 'org.gradle.category' with value 'platform', attribute 'org.gradle.usage' with value 'version-catalog' but:
- Variant 'apiElements' capability io.github.elye:plugin-dependencies:1.0.0:
- Incompatible because this component declares attribute 'org.gradle.category' with value 'library', attribute 'org.gradle.usage' with value 'java-api' and the consumer needed attribute 'org.gradle.category' with value 'platform', attribute 'org.gradle.usage' with value 'version-catalog'
- Variant 'runtimeElements' capability io.github.elye:plugin-dependencies:1.0.0:
- Incompatible because this component declares attribute 'org.gradle.category' with value 'library', attribute 'org.gradle.usage' with value 'java-runtime' and the consumer needed attribute 'org.gradle.category' with value 'platform', attribute 'org.gradle.usage' with value 'version-catalog'
My versionCatalogs library is writen as below
plugins {
id 'java-library'
id 'kotlin'
id 'version-catalog'
}
java {
sourceCompatibility = JavaVersion.VERSION_1_7
targetCompatibility = JavaVersion.VERSION_1_7
}
catalog {
// declare the aliases, bundles and versions in this block
versionCatalog {
alias('androidx-core').to('androidx.core:core-ktx:1.6.0')
alias('androidx-appcompat').to('androidx.appcompat:appcompat:1.3.1')
alias('androidx-constraintlayout').to('androidx.constraintlayout:constraintlayout:2.1.0')
alias('android-material').to('com.google.android.material:material:1.4.0')
bundle('androidx', ['androidx-core',
'androidx-appcompat',
'androidx-constraintlayout'])
}
}
ext {
PUBLISH_GROUP_ID = 'io.github.elye'
PUBLISH_VERSION = '1.0.0'
PUBLISH_ARTIFACT_ID = 'plugin-dependencies'
}
apply from: "./publish-module.gradle"
When I try to access the versionCatalog library, it is just as below
dependencyResolutionManagement {
// Some other codes
versionCatalogs {
xlibs {
from("io.github.elye:plugin-dependencies:1.0.0")
}
}
}
How can I fix the error?
Looks like I just need to remove the below from the library's build.gradle file.
plugins {
id 'java-library'
id 'kotlin'
}
Hence the code looks like below
plugins {
id 'version-catalog'
}
catalog {
// declare the aliases, bundles and versions in this block
versionCatalog {
alias('androidx-core').to('androidx.core:core-ktx:1.6.0')
alias('androidx-appcompat').to('androidx.appcompat:appcompat:1.3.1')
alias('androidx-constraintlayout').to('androidx.constraintlayout:constraintlayout:2.1.0')
alias('android-material').to('com.google.android.material:material:1.4.0')
bundle('androidx', ['androidx-core',
'androidx-appcompat',
'androidx-constraintlayout'])
}
}
ext {
PUBLISH_GROUP_ID = 'io.github.elye'
PUBLISH_VERSION = '1.0.0'
PUBLISH_ARTIFACT_ID = 'plugin-dependencies'
}
apply from: "./publish-module.gradle"
I didn't quite follow the Elye's answer, so I'll explain what I changed in my build.gradle.kts file to get around the problem.
I have a separate project that contains the toml version catalog. That project is simply responsible for publishing the version catalog to my nexus repository. In my case, I not only publish the toml version catalog, but I also publish 4 properties files. Those properties files correspond to the values of the versions, libraries, bundles and plugin sections of the toml version catalog. I do that so my runtime application can get access to the information contained in the toml file. Normally the toml version catalog is only available to the build components. This allows me to make that data available to either runtime components or externally built custom gradle plugins.
Here's what my settings.gradle.kts file contains:
dependencyResolutionManagement {
repositoriesMode.set(RepositoriesMode.FAIL_ON_PROJECT_REPOS)
repositories {
maven {
url = uri(System.getenv()["MyReleaseRepoUrl"].toString())
credentials {
username = System.getenv().get("MyRepoUser").toString()
password = System.getenv().get("MyRepoUserPW").toString()
}
}
google()
mavenCentral()
}
versionCatalogs {
create("libs") {
from(files("src/main/resources/libs.versions.toml"))
}
}
So the settings file creates a version catalog for the toml file which I store in the "src/main/resoures" directory
Here's what my build.gradle.kts file contains:
plugins {
id ("version-catalog")
id ("maven-publish")
kotlin("jvm")
id ("com.xyz.versions.plugin")
}
publishing {
publications {
create<MavenPublication>("VersionCatalog") {
groupId = "com.xyz"
artifactId = "com.xyz.version.catalog"
version = libs.versions.toml.get()
//artifact("src/main/resources/libs.versions.toml")
from(components["versionCatalog"])
}
create<MavenPublication>("VersionsProperties") {
groupId = "com.xyze"
artifactId = "com.xyz.version.versions"
version = libs.versions.toml.get()
artifact("src/main/resources/versions.properties")
}
...
The plugin "version-catalog" is what makes "components["versionCatalog"]" available. Maven-publish is the plugin I use to do the publishing. And "com.xyz.versions.plugin" is a buildSrc plugin that is written in Kotlin. That plugin iterates each section of the version catalog and produces a corresponding properties file.
And lastly you see my definition of what I publish. I show how I publish the toml version catalog, and I show how I publish the versions.properties file that I generated. You should notice that I have the "artifact" line commented out in the publication description for the toml file. If I published the file using the "artifact" line I get the "no matching variant..." issue when I try to use the published version catalog in some other project. If I use the "from components" line instead, I don't get the "no matching variant..." issue when I import the version catalog in another project.
As I explained in the comment to my previous solution, it actually doesn't work. The problem with the code I posted is that it ended up publishing an empty version catalog; and with an empty published version catalog that ends up avoiding the error. But of course, that's not what you want happening. The solution to getting the version catalog published was for me to publish the version catalog using the "artifact" line instead of the "components" line. I don't totally understand the circumstances that cause you to publish an empty version catalog but publishing it as an artifact instead of a component gets around that issue. There are lots of people who have observed this same issue and I think it's a Gradle bug.
With that said, the issue of "no matching variant..." would come up when I tried to use that published version catalog. In my case, it turns out that I was publishing the version catalog reusing the previous version number. I had set up my maven repository to let me do that. What I did to get around the problem was to run "gradlew build --refresh-dependencies" in the project that I was importing the version catalog artifact. When I watched the build, I got the clear sense that the version catalog was be reloaded by watching the build output screen. I let that task run for a while, but the task seemed to be hung. After 10 minutes of waiting, I just went ahead and control C'ed out of the task. I then ran the task "gradlew build" without the refresh dependency option. This time, it ran without producing the "no matching variant...". I also had the build script print out the value of one of the versions and libraries aliases. It printed out the values that was contained in the imported version catalog.
I'd posted two answers and both of these answers at first seemed to resolve this problem, and then days later the problem started reoccurring. I think this time I have a better understanding of what's wrong and I'll try to set the record straight in hopes that this info is useful to someone else experience this problem.
The first thing to understand is that the Gradle documentation on how to publish a version catalog and later on import that published version catalog is close to nonexistent or impossible to find. Like most Gradle documentation, the best you can hope for is some coding examples that shows you what to do; usually you get a coding example with no explanation as to what it doing. When it comes to publishing a version catalog, you'll see examples that show you how to publish using "from(components["versionCatalog']). So, the question is why do you need to publish it as a component? After much searching around, I discovered that beginning in Gradle version 6, the publish of Gradle Module Metadata in now the default when using the "maven-publish" or "ivy-publish" plugins. What's harder to find is that the importing a version catalog from a repository relies on this additional metadata in order to work. This additional metadata gets stored in the repository along with the other required maven files when you publish an artifact to the repository. For most of us, we don't really need to know about these other files that get automatically generated and when you publish an artifact to a repository. The publishing plugin hides that detail from us. When you publish an artifact, you not only add the artifact to the repository, but you also add a POM, maven-metadata, and a number of message digest calculation. There's also a define directory structure as to where these files get store in the repository. And now beginning with Gradle version 6, maven-publish adds the file "module" to the mix.
Here's an example of what that "module" file contains:
{
"formatVersion": "1.1",
"component": {
"group": "com.mycompany",
"module": "com.mycompany.version.catalog",
"version": "0.0.1-SNAPSHOT",
"attributes": {
"org.gradle.status": "integration"
}
},
"createdBy": {
"gradle": {
"version": "7.4"
}
},
"variants": [
{
"name": "versionCatalogElements",
"attributes": {
"org.gradle.category": "platform",
"org.gradle.usage": "version-catalog"
},
"files": [
{
"name": "com.mycompany.version.catalog-0.0.1-SNAPSHOT.toml",
"url": "com.mycompany.version.catalog-0.0.1-SNAPSHOT.toml",
"size": 12663,
"sha512": "0078f8cd00d4a49577bdee7917cafaac0a292761d197012529580c2a561c630360ced5689c07ff40373c32f5e3873b0910987acbc74a659b160c50fb35598b6b",
"sha256": "051977a0ecdccea2e0708fa34d12390f3b4505010e7f00f4390c0c0cc5459e1d",
"sha1": "8ce5ca0cda001dd67799c98a2605107674a3c92b",
"md5": "8f348b9eef2b0ccb7d77d0fe17989f31"
The important thing to notice is the presents of the attributes "org.gradle.catalog" with value "platform" and "org.gradle.usage" with the value "version-catalog".
If you look at Elye's original posted message, those are the attributes that can't be found. So, what the error message is saying that when it tried to import the version catalog artifact from the repository it first retrieved the module file and found that this artifact is not publish as the kind of artifact that it was expecting to find. Instead, the artifact that it did find had the attributes 'org.gradle.category' with value 'library' and the attribute 'org.gradle.usage' with value 'java-api'. The other possibility is that the repository, did not contain a "module" file, and if that's the case, it assumes those attributes and values as the default values. So, what the error message is telling us is that it found an artifact that match the request, but it's not been published as the type of artifact that we were expecting.
So that is what I believe is what's going on, and if that's all you're looking for you can stop here. The message means you've not properly published the version catalog. What I'll now attempt to explain is what happened to me and how I got confused. I'll also explain how I manifested the same problem but from a different angle.
In my situation, I have a project, whose only purpose is to define a version catalog that can be published. I have a number of other external projects that import that published version catalog. I've authored the version catalog so that it contains the version number of the version catalog that it publishes. When I publish the version catalog, I publish it using the version number contained in the version catalog itself. In order to do that, the version catalog needs to be created in the setting.gradle.kts script file. The build.gradle.kts script that that does the publishing get access to the version number using the accessor "libs.version.toml.get()".
The problem is that I can't publish that version catalog using the version catalog in my settings file as a component. Hence that got me publishing it as a file instead of a component. If I publish it as a file and not a component, the "module" file is not published, and that cause the error about mismatched attributes. The only way I could get it to be published as with the module file was to apply the "version-catalog" plugin. That resulted in publishing an empty version catalog. In other words, it didn't know that I had a version catalog created in the setting file. Later I discovered that I could add a "versionCatalog" section to my build.gradle.kts script. In that section I added a catalog that came from the same file that the version catalog in my settings file was using. In other words, I've got two instances of version catalogs both derived for the same file. That works, and when I now publish from component["version-catalog"], I no longer publish an empty version catalog In doing it this way, the module file gets added to the repository. And because I got the version catalog in the settings file, I can extract the version number I need in my build.gradle.kts file. Life is good!
One of things that I was doing that added to the problem was that I was republishing the catalog using the same version number. I would of course refresh my dependencies each time I did this. When I first started this, I was publishing using "components", when I later discovered that the catalog was empty, I started publishing using "files". The problem with that is I believe that a republish using files, didn't cause the "module" file to go away. So with the presents of the "module" file, that masked the problem that I wasn't properly publishing the version catalog. What's funny is that it worked for a couple of days and then the problem started reoccurring. I did look at the contents of my repository, and I saw that the module file was still there. And I recall that it had the right attributes, but after about 3 days, and refreshing my dependencies, the error came back. So they's a bit of a mystery as to why? I couldn't find anything in the maven repository from a data perspective that would signal that the module file shouldn't be associated with the given artifact. The only thing that comes to mind is perhaps the datetime of the module file is not timewise near enough to the datetime of the artifact. I think that may be what's happening, but that's pure speculation. It could also be some weird caching issue. I don't power my machine off every day, and the problem might have emerged after a reboot of my computer. The problem is that how Gradle interacts with the repository in retrieving a version catalog is not describe.
Since this problem, I took the time to learn a little more about maven snapshot artifacts. I had avoided it use only because I was unaware how snapshots are implemented. I've started using snapshots and have learned a little more on how snapshots get added to the repository. I've a good feeling that this is going to straighten out some of the weird things I've been observing.
Given a solution containing both a Web Api project, an Api Client Project, and an assembly project containing shared Dto objects as follows:
Solution
Web Api Project
Shared Dto Project
Api Client Project
Unit Tests
I have an Azure Devops build pipeline to build and test all of the projects.
Within that build pipeline, I have a step that creates 2 Nuget packages for the Api Client and the Shared Dto projects.
However, I don't need to create new packages all of the time, say I only added a unit test or fixed a bug in the Web Api project, I don't want to create new Nuget packages (or worry about bumping the Package version) if nothing in the Api Client Project (or its dependencies) has changed, but I do still want to regression test it.
I thought that perhaps I could do this by having multiple build pipelines, one for the Web Api project, and others for the Api Client & Shared projects, then use path filters to trigger the appropriate builds.
Are there any other ways to achieve this?
Whats the best practice?
I don't want to have to maintain more build definitions than I need.
I ended up using a slight modification of
this answer
# Checks which files have been updated to determine if new Nuget packages are required
$editedFiles = git diff HEAD HEAD~ --name-only
echo "$($editedFiles.Length) files modified:"
$editedFiles | ForEach-Object {
echo $_
Switch -Wildcard ($_ ) {
'Whds.Configuration.Models/*' {
# If the Models project is updated, we need to create both packages
Write-Output "##vso[task.setvariable variable=CreateModelsPackage]True"
Write-Output "##vso[task.setvariable variable=CreateClientPackage]True"
}
'Whds.Configuration.Standard/*' { Write-Output "##vso[task.setvariable
variable=CreateClientPackage]True" }
# The rest of your path filters
}
}
This script sets variables which are then referenced in custom conditions in the dotnet pack step in the build pipeline:
and(succeeded(), eq(variables['CreateModelsPackage'], 'True'))
If the Dto project is changed, both variables are set in order to create both packages.
If only the client (aka Standard) project is the only thing that has changed, the package for the Dto project will not be created.
Are there any other ways to achieve this? Whats the best practice? I don't want to have to maintain more build definitions than I need.
There are different ways to achieve it, but not sure which one is the best practice, it all depends on your requirements or tastes.
The simple method is similar to your thought. Also need to create a new build pipeline. The difference is that we do not need to maintain this build definition.
Details:
Add a new pipeline without any more task in this pipeline, and use
path filters to trigger the appropriate builds (Api Client and the
Shared Dto projects).
Add a build completion to your original Azure Devops build pipeline:
Add a custom condition for the step that creates 2 Nuget packages based on the Build.Reason, like:
and(succeeded(), eq(variables['Build.Reason'], 'BuildCompletion'))
Now, the steps to create 2 Nuget packages only executed when the file changes come from a specific project. Of course the limitation of this solution is that if you already have a build completion, it will not work.
If the above method is not what you want, we could invoke the REST API commits to get the commit info for each build:
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/commits/{commitId}?changeCount=100&api-version=5.1
We could find the changes/path in the returned body:
"changes": [
{
"item": {
"gitObjectType": "blob",
"path": "/.gitattributes",
"url": "https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/items/.gitattributes?versionType=Commit"
},
"changeType": "add"
},
{
"item": {
"gitObjectType": "blob",
"path": "/.gitignore",
"url": "https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/items/.gitignore?versionType=Commit"
},
"changeType": "add"
},
{
"item": {
"gitObjectType": "tree",
"path": "/MyWebSite",
"isFolder": true,
"url": "https://dev.azure.com/fabrikam/_apis/git/repositories/278d5cd2-584d-4b63-824a-2ba458937249/items/MyWebSite?versionType=Commit"
},
"changeType": "add"
},
Then we could use the powershell script to traverse these paths to see if they include Api Client and the Shared Dto projects, if yes, we set a variables with a value, add condition based on this value for the steps that creates 2 Nuget packages.
Note: Before use the REST API commits, we need use the Commits - Get Commits to get the latest commit Id:
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/commits?$top=1&api-version=5.1
Hope this helps.
Reference to this post that I had earlier: Possible to modify or delete rows from a table in BigQuery dataset with a Cloud Data Fusion pipeline? I am trying to do the suggested answer to compile the latest version of Google Cloud Platform plugin and upload to Data Fusion so I can use the latest features.
We have downloaded the code, compiled it and got 2 files:
google-cloud-0.13.0-SNAPSHOT.jar
google-cloud-0.13.0-SNAPSHOT.json
Inside the JSON file, the last lines for the parent artifacts were:
},
"parents": [
"system:cdap-data-pipeline[6.1.0-SNAPSHOT,7.0.0-SNAPSHOT)",
"system:cdap-data-streams[6.1.0-SNAPSHOT,7.0.0-SNAPSHOT)"
]
}
Initially I went to Data Fusion and choose to upload a new plugin, but I got an error about the parent artifacts not existing. So I did some digging and found out the version of used artifacts on Data Fusion currently to be 6.0.1:
So I modified the parent artifacts to the correct versions, and now the last few lines in the JSON file show:
},
"parents": [
"system:cdap-data-pipeline[6.0.1-SNAPSHOT,7.0.0-SNAPSHOT)",
"system:cdap-data-streams[6.0.1-SNAPSHOT,7.0.0-SNAPSHOT)"
]
}
When I try to upload the plugin again, it seems to pass the artifacts check step, but it fails on some sort of a class check and I see this in the upload screen:
Class could not be found while inspecting artifact for plugins. Please
check dependencies are available, and that the correct parent artifact
was specified. Error class: class java.lang.NoClassDefFoundError,
message: io/cdap/cdap/etl/api/validation/ValidationException.
So now I'm really lost about what's wrong here. I'm doubting that the artifacts version that is being used in Data Fusion does not have the class that is throwing the error? If so, how do I update the artifact itself?
Or if there is something else that I am missing in this whole process, then I would really appreciate any guidance or support on this!
Regards
You can try using the release/0.12 branch of the google-cloud plugins repo. That is compatible with the 6.0 version of Cloud Data Fusion.
My intent is to store SNAPSHOT and RELEASE firmware binary images in Artifactory.
I have setup a custom repository layout where the artifact path pattern is set to -
[org]/[module]/[baseRev]-[folderItegRev]/[baseRev]-[folderItegRev](-[fileItegRev]).[ext]
Here are screenshots of the configuration -
I created a new repository based on the generic package type and chose the custom repository layout that I created.
I have uploaded release and snapshot artifacts to this repo and this is what it looks like on the web UI -
myrepo
mygroup/myartifactid
1.0.0-RELEASE
1.0.0-RELEASE.bin
2.0.0-RELEASE
2.0.0-RELEASE.bin
3.0.0-SNAPSHOT
3.0.0-SNAPSHOT-20170630161531.bin
3.0.0-SNAPSHOT-20171202161531.bin
3.0.0-SNAPSHOT-20171231161531.bin
Now if I use the REST API to search for the latest artifact based on layout as per the API published here
GET http://artifactory-server:8082/artifactory/api/search/latestVersion?g=mygroup&a=myartifactid&v=3.0.0-SNAPSHOT
I get the below response -
{
"errors": [
{
"status": 404,
"message": "Unable to find artifact versions"
}
]
}
What am I doing wrong? I want to be able to search for the latest release and snapshot versions using the REST API.
Make sure that "Folder Integration Revision RegExp" and "File Integration Revision RegExp" are defined for your custom layout.
In the case of your layout, they should be:
Folder Integration Revision RegExp: SNAPSHOT
File Integration Revision RegExp:(?:[0-9]{14})
To make sure the layout is properly configured, test it against a sample path, for example: mygroup/myartifactid/3.0.0-SNAPSHOT/3.0.0-SNAPSHOT-20170630161531.bin
If everything is configured correctly the test result should show the various parts of the layout: organization, module etc.
In addition, it is better, in terms of performance, to specify which repositories you wish to query, for example:
GET http://localhost:8081/artifactory/api/search/latestVersion?g=mygroup&a=myartifactid&v=3.0.0-SNAPSHOT&repos=myrepo
I am trying to find a list of all branches for a given project. Is it possible to get this information through Teamcity REST API? I found a different answer showing how to get a list of branches for a given build configuration:
Can you use the team city rest api to fetch plan branch names?
But this way I would have to run the query for all build configurations under given project.
However TeamCity has a concept of "active" branches on a given project. I am wondering if it is possible to fetch exactly that.
Actually, it is possible now.
Implemented for 2017.1 as an experimental features:
listing of the branches for a project (merged list of all project's build configuration's branches) via .../app/rest/projects/XXX/branches?locator=policy:XXX
additional branch node fields: "active", "lastActivity" timestamp, "builds" (with locator), available via "fields" parameter of the request
added "branches" into buildType node, available only via "fields" parameter of the request
Source: https://youtrack.jetbrains.com/issue/TW-44148#comment=27-2018515
I'm using exactly this url: http://TCSERVERADDRESS/app/rest/projects/PROJECTNAME/branches and it works good for me.
TeamCtiy REST API does not support showing active branches now. You are welcome to drop a feature request in the tracker