I'm having a very busy build server. On good day we create almost 200gb of artifacts. Cleanup policy can be run once a day which is not enough for my case. I've searched the teamcity documentation and found zero API endpoints to support trigger cleanup manually.
Is it possible to trigger cleanup manually from script/program? How to achieve this?
In worst case I could fiddler up and trace what happens when i manually force cleanup, but its messy road and i don't want to go on it.
here's also C# version
void Main()
{
var cookieContainer = new CookieContainer();
var baseAddress = new Uri("http://teamcity");
var contentDictionary = new Dictionary<string,string>();
contentDictionary["cleanupPageAction"]= "startCleanup";
using (var handler = new HttpClientHandler() { CookieContainer = cookieContainer,
Credentials = new NetworkCredential("user","password","domain")})
using (var client = new HttpClient(handler) { BaseAddress = baseAddress })
{
var content = new FormUrlEncodedContent(contentDictionary);
var result = client.PostAsync("/admin/cleanupPolicies.html", content).Result;
result.EnsureSuccessStatusCode();
}
}
There is no endpoint in the API for this. If you do decide you are happy enough to use the same HTTP POST the dashboard uses for a manual cleanup here it is, simple enough in my opinion:
curl -d "cleanupPageAction=startCleanup" \
http://user:password#builds.company.com/admin/cleanupPolicies.html
Related
Writing a dotnet core app. I need to log in with network credentials as the service (which happens to be a TFS on-prem server) uses those to authenticate. From my (and another team members') windows machine, the following code works:
Console.WriteLine("Type in your DOMAIN password:");
var pass = GetPassword(); //command line secure string magic from SO
var networkCredential = new NetworkCredential("USERNAME", pass, "DOMAINNAME");
string tfsDefaultCollection = "https://TFSURL/DefaultCollection";
string testUrl = $"{tfsDefaultCollection}/_apis/tfvc/changesets/1234/changes?api-version=2.2";
var httpClientHandler = new HttpClientHandler
{
Credentials = networkCredential
};
var client = new HttpClient(httpClientHandler)
{
BaseAddress = new Uri(testUrl)
};
httpClientHandler.PreAuthenticate = true;
var test = client.GetAsync(testUrl).Result;
Console.WriteLine(test);
But it doesn't work from my mac. I get a 401 unauthorized. Both used the same, hardwired connection. AND this works on my mac:
curl --ntlm --user "DOMAINNAME\USERNAME" "https://TFSURL/DefaultCollection/_apis/tfvc/changesets/1234/changes?api-version=2.2"
So that rules out a connectivity question, I would think. Am I missing something I need to be doing on my mac? Can anybody point me to some documentation or way to troubleshoot what both of these requests are doing at the lowest level to see if there is a difference?
Well finally some google-foo got me there. There's a bug in dotnet core for linux/mac. This issue describes the fix:
https://github.com/dotnet/corefx/issues/25988#issuecomment-412534360
It has to do with the host machine you are connecting to uses both Kerberos and NTLM authentication methods.
Implemented below:
AppContext.SetSwitch("System.Net.Http.UseSocketsHttpHandler", false);
Console.WriteLine("Type in your DOMAIN password:");
var pass = GetPassword(); //command line secure string magic from SO
var networkCredential = new NetworkCredential("USERNAME", pass, "DOMAINNAME");
string tfsDefaultCollection = "https://TFSURL/DefaultCollection";
string testUrl = $"{tfsDefaultCollection}/_apis/tfvc/changesets/1234/changes?api-version=2.2";
var myCache = new CredentialCache
{
{
new Uri(testUrl), "NTLM",
networkCredential
}
};
var httpClientHandler = new HttpClientHandler
{
Credentials = myCache
};
var client = new HttpClient(httpClientHandler)
{
BaseAddress = new Uri(testUrl)
};
httpClientHandler.PreAuthenticate = true;
var test = client.GetAsync(testUrl).Result;
Console.WriteLine(test);
Thanks to #dmcgill50 for getting me on the right googling track.
I am trying to build a .NET small app to predict images using my model that was trianed on AutoML.
But I am getting this error:
Cloud AutoML API has not been used in project 618104708054 before or
it is disabled. Enable it by visiting
https://console.developers.google.com/apis/api/automl.googleapis.com/overview?project=618104708054
then retry. If you enabled this API recently, wait a few minutes for
the action to propagate to our systems and retry
First - this is not the project I am using.
Second - If I go to the link with my real project id - it says to me that the api is working well.
My code look like these:
public static string SendPOST(string url, string json)
{
var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
httpWebRequest.ContentType = "application/json";
httpWebRequest.Method = "POST";
httpWebRequest.Headers.Add("Authorization", "Bearer GOOGLE_CLOUD_TOKEN");
using (var streamWriter = new StreamWriter(httpWebRequest.GetRequestStream()))
{
streamWriter.Write(json);
streamWriter.Flush();
streamWriter.Close();
}
var httpResponse = (HttpWebResponse)httpWebRequest.GetResponse();
using (var streamReader = new StreamReader(httpResponse.GetResponseStream()))
{
var result = streamReader.ReadToEnd();
//var res = new JavaScriptSerializer().Deserialize<Response>(result);
//return res;
return result;
}
}
I will appriciate your help,
Thanks.
I finally succeded to make it, the only issue is that I needed to create a service account using the web console:
https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts?supportedpurview=project&project=&folder=&organizationId=
And then to download the json key and push it via the gcloud command from my PC -
gcloud auth activate-service-account --key-file="[/PATH/TO/KEY/FILE.json]
I found the solution in this post:
"(403) Forbidden" when trying to send an image to my custom AutoML model via the REST API
I am currently using the Event Store to handle my events. I currently need to replay a particular type of event as I have made changes in the way they are subscribed and written to DB.
Is this possible? If so, how can it be done? Thanks.
You cannot tell EventStore to replay a specific event onto a persistent subscription because the point of the persistent subscription is to keep state for the subscribers.
To achieve this kind of fix you would really need a catch up application to do the work.
And really if you think about, if you replayed ALL the events to a new database then you would have the correct data in there?
So I have a console application that reuses the same logic as the persistent connection but the only difference is:
I change the target database connection string - So this would be a new Database or Collection (not the broken one)
It connects to EventStore and replays all the events from the start
It rebuilds the entire database to the correct state
Switch the business over to the new database
This is the point of EventStore - You just replay all the events to build any database at any time and it will be correct
Your persistent connections deal with new, incoming events and apply updates.
If you enable $by_event_type projection than you can access that projection stream under
/streams/$et-{event-type}
https://eventstore.org/docs/projections/system-projections/index.html
Then you can read it using .net api if you wish.
Here is some code to get you started
private static T GetInstanceOfEvent<T>(ResolvedEvent resolvedEvent) where T : BaseEvent
{
var metadataString = Encoding.UTF8.GetString(resolvedEvent.Event.Metadata);
var eventClrTypeName = JObject.Parse(metadataString).Property(EventClrTypeHeader).Value;
var #event = JsonConvert.DeserializeObject(Encoding.UTF8.GetString(resolvedEvent.Event.Data), Type.GetType((string) eventClrTypeName));
if (!(#event is BaseEvent))
{
throw new MessageDeserializationException((string) eventClrTypeName, metadataString);
}
return #event as T;
}
private static IEventStoreConnection GetEventStoreConnection()
{
var connectionString = System.Configuration.ConfigurationManager.ConnectionStrings["EventStore"].ConnectionString;
var connection = EventStoreConnection.Create(connectionString);
connection.ConnectAsync().Wait();
return connection;
}
private static string GetStreamName<T>() where T : BaseEvent
{
return "$et-" + typeof(T).Name;
}
And to read events you can use this code snippet
StreamEventsSlice currentSlice;
long nextSliceStart = StreamPosition.Start;
const int sliceCount = 200;
do
{
currentSlice = await esConnection.ReadStreamEventsForwardAsync(streamName, nextSliceStart, sliceCount, true);
foreach (var #event in currentSlice.Events)
{
var myEvent = GetInstanceOfEvent<OrderMerchantFeesCalculatedEvent>(#event);
TransformEvent(myEvent);
}
nextSliceStart = currentSlice.NextEventNumber;
} while (currentSlice.IsEndOfStream == false);
I am trying to understand how the self host configuration based Integration Tests are running.
In the code below, Should I be registering my config with the WebApiConfig. Registering or not seems to make no difference.
Is the full pipeline really being tested or is this an illusion? Since, If I am not using the config and the routes defined in my API instead declaring my own as I have done here, I am probably just not testing the full pipleine.
Is there any othere way of testing the api completely. The code below is testing a lot of things besides just my pipeline(like the client, SelfHosting etc..). This seems like an overkill to me. Any ideas ?
var config = new HttpSelfHostConfiguration("http://localhost:9090/");
config.Routes.MapHttpRoute("Default", "{api}/{controller}/{id}", new { id = RouteParameter.Optional });
config.IncludeErrorDetailPolicy = IncludeErrorDetailPolicy.Always;
MyApiProject.WebApiConfig.Register(config);
using (var server = new HttpSelfHostServer(config))
{
server.OpenAsync().Wait();
using (var client = new HttpClient())
{
using (var response = client.PostAsync("http://localhost:9090/api/login",
new FormUrlEncodedContent(new List<KeyValuePair<string,string>> { new KeyValuePair<string, strin("Foo","Bar)}), CancellationToken.None).Result)
{
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
}
using (var response = client.GetAsync("http://localhost:9090/api/login").Result)
{
Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
}
}
server.CloseAsync().Wait();
}
If you just want to test your controllers, you can write more targeted unit tests to test them. If you want to test the full pipeline your code looks fine except that instead of using a selfhost, you can just use HttpServer saving the network overhead. Also, if you are testing the full pipeline it is better to use the routes that you have in your actual app rather than adding a new route as that would be testing routing as well.
Also, refer to this blog post by Youssef for some ideas on testing your web APIs.
I am trying to write an activity in Google+ using the dotnet-client. The issue is that I can't seem to get the configuration of my client app correctly. According to the Google+ Sign-In configuration and this SO question we need to add the requestvisibleactions parameter. I did that but it did not work. I am using the scope https://www.googleapis.com/auth/plus.login and I even added the scope https://www.googleapis.com/auth/plus.moments.write but the insert still did not work.
This is what my request url looks like:
https://accounts.google.com/ServiceLogin?service=lso&passive=1209600&continue=https://accounts.google.com/o/oauth2/auth?scope%3Dhttps://www.googleapis.com/auth/plus.login%2Bhttps://www.googleapis.com/auth/plus.moments.write%26response_type%3Dcode%26redirect_uri%3Dhttp://localhost/%26state%3D%26requestvisibleactions%3Dhttp://schemas.google.com/AddActivity%26client_id%3D000.apps.googleusercontent.com%26request_visible_actions%3Dhttp://schemas.google.com/AddActivity%26hl%3Den%26from_login%3D1%26as%3D-1fbe06f1c6120f4d<mpl=popup&shdf=Cm4LEhF0aGlyZFBhcnR5TG9nb1VybBoADAsSFXRoaXJkUGFydHlEaXNwbGF5TmFtZRoHQ2hpa3V0bwwLEgZkb21haW4aB0NoaWt1dG8MCxIVdGhpcmRQYXJ0eURpc3BsYXlUeXBlGgdERUZBVUxUDBIDbHNvIhTeWybcoJ9pXSeN2t-k8A4SUbfhsygBMhQivAmfNSs_LkjXXZ7bPxilXgjMsQ&scc=1
As you can see from there that there is a request_visible_actions and I even added one that has no underscore in case I got the parameter wrong (requestvisibleactions).
Let me say that my app is being authenticated successfully by the API. I can get the user's profile after being authenticated and it is on the "insert moment" part that my app fails. My insert code:
var body = new Moment();
var target = new ItemScope();
target.Id = referenceId;
target.Image = image;
target.Type = "http://schemas.google.com/AddActivity";
target.Description = description;
target.Name = caption;
body.Target = target;
body.Type = "http://schemas.google.com/AddActivity";
var insert =
new MomentsResource.InsertRequest(
// this is a valid service instance as I am using this to query the user's profile
_plusService,
body,
id,
MomentsResource.Collection.Vault);
Moment result = null;
try
{
result = insert.Fetch();
}
catch (ThreadAbortException)
{
// User was not yet authenticated and is being forwarded to the authorization page.
throw;
}
catch (Google.GoogleApiRequestException requestEx)
{
// here I get a 401 Unauthorized error
}
catch (Exception ex)
{
} `
For the OAuth flow, there are two issues with your request:
request_visible_actions is what is passed to the OAuth v2 server (don't pass requestvisibleactions)
plus.moments.write is a deprecated scope, you only need to pass in plus.login
Make sure your project references the latest version of the Google+ .NET client library from here:
https://developers.google.com/resources/api-libraries/download/stable/plus/v1/csharp
I have created a project on GitHub showing a full server-side flow here:
https://github.com/gguuss/gplus_csharp_ssflow
As Brettj said, you should be using the Google+ Sign-in Button as demonstrated in the latest Google+ samples from here:
https://github.com/googleplus/gplus-quickstart-csharp
First, ensure you are requesting all of the activity types you're writing. You will know this is working because the authorization dialog will show "Make your app activity available via Google, visible to you and: [...]" below the text that starts with "This app would like to". I know you checked this but I'm 90% sure this is why you are getting the 401 error code. The following markup shows how to render the Google+ Sign-In button requesting access to Add activities.
<div id="gConnect">
<button class="g-signin"
data-scope="https://www.googleapis.com/auth/plus.login"
data-requestvisibleactions="http://schemas.google.com/AddActivity"
data-clientId="YOUR_CLIENT_ID"
data-accesstype="offline"
data-callback="onSignInCallback"
data-theme="dark"
data-cookiepolicy="single_host_origin">
</button>
Assuming you have a PlusService object with the correct activity type set in data-requestvisibleactions, the following code, which you should be able to copy/paste to see it work, concisely demonstrates writing moments using the .NET client and has been tested to work:
Moment body = new Moment();
ItemScope target = new ItemScope();
target.Id = "replacewithuniqueforaddtarget";
target.Image = "http://www.google.com/s2/static/images/GoogleyEyes.png";
target.Type = "";
target.Description = "The description for the activity";
target.Name = "An example of add activity";
body.Target = target;
body.Type = "http://schemas.google.com/AddActivity";
MomentsResource.InsertRequest insert =
new MomentsResource.InsertRequest(
_plusService,
body,
"me",
MomentsResource.Collection.Vault);
Moment wrote = insert.Fetch();
Note, I'm including Google.Apis.Plus.v1.Data for convenience.
Ah it's that simple! Maybe not? I am answering my own question and consequently accept it as the answer (after a few days of course) so others having the same issue may be guided. But I will definitely up-vote Gus' answer for it led me to the fix for my code.
So according to #class answer written above and as explained on his blog the key to successfully creating a moment is adding the request_visible_actions parameter. I did that but my request still failed and it is because I was missing an important thing. You need to add one more parameter and that is the access_type and it should be set to offline. The OAuth request, at a minimum, should look like: https://accounts.google.com/o/oauth2/auth?scope=https://www.googleapis.com/auth/plus.login&response_type=code&redirect_uri=http://localhost/&request_visible_actions=http://schemas.google.com/AddActivity&access_type=offline.
For the complete and correct client code you can get Gus' example here or download the entire dotnet client library including the source and sample and add what I added below. The most important thing that you should remember is modifying your AuthorizationServerDescription for the Google API. Here's my version of the authenticator:
public static OAuth2Authenticator<WebServerClient> CreateAuthenticator(
string clientId, string clientSecret)
{
if (string.IsNullOrWhiteSpace(clientId))
throw new ArgumentException("clientId cannot be empty");
if (string.IsNullOrWhiteSpace(clientSecret))
throw new ArgumentException("clientSecret cannot be empty");
var description = GoogleAuthenticationServer.Description;
var uri = description.AuthorizationEndpoint.AbsoluteUri;
// This is the one that has been documented on Gus' blog site
// and over at Google's (https://developers.google.com/+/web/signin/)
// This is not in the dotnetclient sample by the way
// and you need to understand how OAuth and DNOA works.
// I had this already, see my original post,
// I thought it will make my day.
if (uri.IndexOf("request_visible_actions") < 1)
{
var param = (uri.IndexOf('?') > 0) ? "&" : "?";
description.AuthorizationEndpoint = new Uri(
uri + param +
"request_visible_actions=http://schemas.google.com/AddActivity");
}
// This is what I have been missing!
// They forgot to tell us about this or did I just miss this somewhere?
uri = description.AuthorizationEndpoint.AbsoluteUri;
if (uri.IndexOf("offline") < 1)
{
var param = (uri.IndexOf('?') > 0) ? "&" : "?";
description.AuthorizationEndpoint =
new Uri(uri + param + "access_type=offline");
}
// Register the authenticator.
var provider = new WebServerClient(description)
{
ClientIdentifier = clientId,
ClientSecret = clientSecret,
};
var authenticator =
new OAuth2Authenticator<WebServerClient>(provider, GetAuthorization)
{ NoCaching = true };
return authenticator;
}
Without the access_type=offline my code never worked and it will never work. Now I wonder why? It would be good to have some explanation.