Gradle Download Task with Basic Auth behaves differently on GitHub Actions - gradle

I am using gradle with the Download Task provided by Michael Kraemer (see: https://github.com/michel-kraemer/gradle-download-task). The remote site requires HTTPS and a Basic Auth. Although I configured everything and the task works perfectly on my local machine, execution by the GitHub Actions on a ubuntu-linux runner behaves differently.
On my local machine, the request sent contains Basic Auth headers and the server response with a 302 redirect providing the expected download file. However, on the build server it turns out, that the Download Tasks downloads a HTML file. This is the typical response of the server application if one did not specify Basic Auth. So, I now assume that the GitHub runners omit the Basic Auth headers when the above gradle plugin spawns a http client / connection with Basic Auth.
Is anyone experiencing something alike? Is there a workaround for this situation? I cannot change the authentification method of the server where I am required to download the software from, so I'd be happy to get some support here. :-)
Here is the definition of the download task from my build.gradle.kts:
tasks.register<Download>("download") {
src(https://software.vendor.com/path/to/file)
dest(file("${DOWNLOAD_FOLDER}/vendor-software-${VERSION}.zip"))
username("USER")
password("PASS")
authScheme("Basic")
overwrite(false)
tempAndMove(true)
onlyIfModified(true)
useETag(true)
}
One addition: It turns out that the fact that GitHub encodes all secrets values could be part of the issue. In fact, when I use the header configuration and configure the Basic Auth header directly, the process works:
tasks.register<Download>("download") {
src(https://software.vendor.com/path/to/file)
dest(file("${DOWNLOAD_FOLDER}/vendor-software-${VERSION}.zip"))
header("Authorization", "Basic VVNFUjpQQVNT")
overwrite(false)
tempAndMove(true)
onlyIfModified(true)
useETag(true)
}

Not a perfect answer to the issue, nor does it explain why it occurs, but as an workaround, I uploaded a new secret that contains the Base64 encoded basic authorization header and use the header notation as mentioned above. This way, the download works, still it would be great to understand, why this happens.

Related

look for assistance with the url rewrite module in IIS

I'm trying to set up a reverse proxy for single sign on to an application. the URL to reach my app and the defined authentication method is "http://servername.domain/app/main?authsource=sso_rp"
My current config just sends me to the application login prompt. I've tested my authentication source and confirmed its working. My question is, does my URL rewrite config look like it should? there are no conditions or variables defined. i know I'm missing something simple but this isn't documented on the vendor end so seeking your help.

How do I disable HTTPS in ASP.NET Core 2.1 + Kestrel?

So it appears with the advent of ASP.NET Core 2.1, Kestrel now automatically creates an HTTPS endpoint along side the HTTP one, and default project templates are setup to redirect from HTTP to HTTPS (which is easy enough to undo).
However my question is... how can I disable HTTPS entirely for my project. I've read through the docs and played with a variety of config settings for HTTPS but nothing I do seems to allow me to turn it off and just run an HTTP project.
Am I crazy or just missing something. I would expect this to be super easy to do.
In the Startup.cs, remove the middleware
app.UseHttpsRedirection();
If you are using Visual Studio 2017, then you can do the following:
Go to your project properties. (Right-click > Properties)
Click on the Debug tab.
Under Web Server Settings, deselect Enable SSL.
Save, build, and try again.
This will update the iisExpress settings in the launchSettings.json file.
In the file Properties/launchSettings.json of your project, look of the key applicationUrl. You will find something like:
...
"applicationUrl": "https://localhost:5001;http://localhost:5000",
...
Remove the https endpoint and it's done.
Edit
As noted by #Xorcist the file launchSettings.json is not published. So, the solution above will only work in a development environment. To disable https and, in general, to configure the urls you want to listen to, both in production and in development, you can also do one of the following:
Use --urls parameters of dotnet run, will have the same effect as the applicationUrl in launchSettings.json. For instance: dotnet run --urls=http://0.0.0.0:5000,https://0.0.0.0:5001. Again, remove the one you don't want to use.
The same can be achieved with the ASPNETCORE_URLS enviroment variable.
As mentioned in the answer by #Konstantin to this question, in ASP Net Core 2.1 you can also configure Kestrel endpoints in the appsettings.json (it seems this cannot be done in 2.0).
Finally, the same can also be achieved with the useUrls extension method WebHost.CreateDefaultBuilder(args).UseUrls("http://0.0.0.0:5000"). I prefer the other solution because this ones hardcodes you're application endpoints, and can't be changed without recompiling the application.
All the possible options are explained in detail in the Microsoft Docs on this.
Update (09 Dec 2020): these options are still valid for Net Core 3.1, as per Microsoft Docs, except for the appsettings one. Maybe it still works but I am not sure.
Update (19 May 2021): these options are still valid for Net 5, as per Microsoft Docs, except for the appsettings one. Maybe it still works but I am not sure.
Turns out the proper way to achieve what I wanted to do, was to specifically configure Kestrel with .UseKestrel() and simply specify a single address, like this:
WebHost.CreateDefaultBuilder(args)
.UseKestrel(options => {
if (context.Configuration[WebHostDefaults.EnvironmentKey] == Environments.Development) {
options.Listen(IPAddress.Loopback, 5080); //HTTP port
}
})
.UseStartup<Startup>();
in effect overriding the default setup, and displaying this warning when Kestel starts:
warn: Microsoft.AspNetCore.Server.Kestrel[0]
Overriding address(es) 'https://localhost:5001, http://localhost:5000'. Binding to endpoints defined in UseKestrel() instead.
Note the check for development environment; in production the default ports are different (80) and without HTTPS.
if a second address is specified it will assume that address is to be secured with the built-in developer cert, as such:
WebHost.CreateDefaultBuilder(args)
.UseKestrel(options => {
options.Listen(IPAddress.Loopback, 5080); //HTTP port
options.Listen(IPAddress.Loopback, 5443); //HTTPS port
})
.UseStartup<Startup>();
you may of course specifically secure your SSL address as described here:
https://learn.microsoft.com/en-us/aspnet/core/fundamentals/servers/kestrel?view=aspnetcore-2.1&tabs=aspnetcore2x
which is necessary for production setups.
In the Program.cs, Add UseUrls as following:
WebHost.CreateDefaultBuilder(args)
.UseUrls("http://localhost:5000")
.UseStartup<Startup>();
And In The Startup.cs remove/comment the following:
app.UseHttpsRedirection();
The dotnet CLI now has a template for this.
dotnet new webapi --no-https
With ASPNET CORE 2.2, I simply set the web server URL to http not https and it picks it up on its own. I run it as a self hosted process.
Go to your project properties.
Click on the Debug tab.
Under Web Server Settings, set the URL to http://xxx
Try again :)
Turning off https lies in these 3 changes...
Properties/launchSettings.json
set sslPort to 0
remove the https url from the applicationUrl
Startup.cs
Remove or comment-out app.UseHttpsRedirection()
For Development & not in production:
in project properties disable Enable SSL
One more way for disabling https epecially is handy when docker is used.
Set enviroment variable in Dockerfile with only one HTTP url in value.
#https + http
ENV ASPNETCORE_URLS=http://+:5001;http://+:5000
#http only
ENV ASPNETCORE_URLS=http://+:5000
#joanlofe answer is excellent one, but there is also "stupid" way how one can reintroduce HTTPS on 5001 port. If you call Clear on your config sources (for proper layering of config sources for example) it means that one implicit source is gone -- "launchSettings.json" template. So if you rely on this file instead of "appsettings.json" (and by default you probably are) your app will enable HTTPS on port 5001.
My local k8s deployment was failing due to the existence of
"Kestrel": {
"Certificates": {
"Default": {
...
}
}
}
in an appsettings.json override, even after following the other steps here. If you're trying to strip a server of SSL (for example, if SSL is now terminated upstream), make sure to get rid of this configuration as well.
This seems pretty obvious now that I found it, but it still tripped us up for a few hours.

How can I force a meteor app to make all HTTP calls through a proxy?

I'm trying to emulate curl through a proxy server. The meteor docs don't mention any proxy settings for HTTP.* methods.
Is there a meteor-specific solution? Right now I'm using ProxyChains.
Ideally I'd use a SOCKS proxy and only HTTP.* calls would go through it, but I'm open to all calls from the application going through any type of proxy.
Meteor 1.1 update
You can pass options directly to the npm request module via the npmRequestOptions parameter to HTTP.*. The functionality was enabled by this commit made after I filed an issue in 2013 (see below).
You no longer need to use the http-more package.
Old answer, pre-Meteor 1.1
One method would be pass a proxy parameter to HTTP.* calls, which use the request module, which supports proxies as an option.
proxy isn't a recognized option in the HTTP package, and I've filed a request to simply pass through unrecognized options. It was rejected by one of the Meteor core developers.
I'd rather people vote on that issue, asking for unknown options to be passed through instead of being ignored. In the meantime, I've created a package that does pass through options: http-more.
Here's a Meteor proxy package: https://npmjs.org/package/seafish-http-proxy-meteor
It's not available through atmosphere, but it is an npm package designed for meteor, which means it will be very easy to integrate.

Download build drop from hosted Team Foundation Service

Using the hosted Team Foundation Service at tfs.visualstudio.com, one has the option in a Build Definition to "Copy build output to the server" which creates a zip of the drop folder that can be downloaded over https via team web access. I really need to download this drop automatically, so I can chain input to the next stage in my build pipeline.
Unfortunately, the drop URL is not obvious, but can be created using the TfsDropDownloader.
TL;DR - I can't get the TfsDropDownloader to work, I'm hoping someone else has used this tool or a similar method to succesfully download a drop from https://tfs.visualstudio.com
Using the command line TfsDropDownloader.exe I can do this:
TfsDropDownloader.exe /c:"https://MYPROJECTNAME.visualstudio.com/DefaultCollection" /t:"ProjectName" /b:"BuildDefinitionName" /u:username /p:password
...and get an empty zip file with the correct build label name of the last successful build e.g. BuildDefinitionName_20130611.1.zip
Running the source code in the debugger, this is because the URL that is generated for downloading:
https://tflonline.visualstudio.com/DefaultCollection/_apis/resources/containers/804/drop/BuildDefinitionName_20130611.1.zip
..returns a content type of application/json, which is unsupported. This exception is swallowed by the application, but not before the empty zip file is created.
Is it possible the REST API on Team Foundation Service has changed in some way so the generated URL is no longer correct?
Note that I am using the "alternate credentials" defined on my Team Foundation Service account (i.e. not my live ID) - using anything else gets me TF30063: not authorized.
I got it working by using alternate credentials, but I also had to access the REST API via a different path.
The current TfsDropDownloader builds a URL that looks like this:
https://project.visualstudio.com/DefaultCollection/_apis/resources/containers/804/drop/BuildDefinitionName_20130611.1.zip
This returns empty JSON whenever I try to use it. I'm definitely authenticated, because if I tweak the URL to:
https://project.visualstudio.com/DefaultCollection/_apis/resources/containers/804/drop
I get a nice JSON listing of every single file in the drop, but no zip.
From spying on the SSL traffic to https://tfs.visualstudio.com with Fiddler I saw that clicking the "Download drop as zip" link I can see that there is another endpoint at:
https://project.visualstudio.com/DefaultCollection/ProjectName/_api/_build/ItemContent?buildUri=vstfs%3a%2f%2f%2fBuild%2fBuild%2f639&path=%2Fdrop
...which does give you a zip. The "vstfs%3a%2f%2f%2fBuild%2fBuild%2f639" portion is the URL encoded BuildUri.
So I've changed my version of GetServerPath in the TfsDropDownloader source to do this:
private static string GetServerPath(TfsConnection collection, IBuildDetail buildDetail)
{
var downloadPath = string.Format("{0}{1}/_api/_build/ItemContent?buildUri={2}&path=%2Fdrop",
collection.Uri,
HttpUtility.UrlPathEncode(buildDetail.TeamProject),
HttpUtility.UrlEncode(buildDetail.Uri.ToString()));
return downloadPath;
}
This works for me for the time being. Hopefully this helps someone else with the same problem!

How do I setup an external status page in TeamCity like their standard one that won't require me to login?

We have just moved from CCNET to TeamCity for Continuous Integration.
In TeamCity, I can use the ExternalStatus page/widget fine.
We run Cradiator (http://cradiator.codeplex.com) as a build radiator and notifier in our room and I have modified the Team Piazza TeamCity plug-in to return XML in the CCNET format so Cradiator can read it.
The problem I have is that the modified TeamPiazza page needs authentication, whereas the TeamCity externalstatus page doesn't.
So my question is: how can I make the custom page I have not require authentication?
TeamCity provides the AuthorizationInterceptor interface in their Open API that you can inject into your plugin code which allows you to control the authorisation requirement.
shamelessPlug This is what I used when writing the tcMonitor status page. /shamelessPlug
Here is example code on how to use it:
/* Add the objects into the constructor and spring will make them
available for you */
public StatusPageController(SBuildServer server,
AuthorizationInterceptor authorizationInterceptor,
UrlMapping urlMapper) {
// Tell teamcity that auth is not required for this page.
authorizationInterceptor.addPathNotRequiringAuth(myUrl);
you should post directly to the support forum http://www.jetbrains.net/devnet/community/teamcity/teamcity and post the answer back here if you want.
If a Guest User is enabled in TeamCity you can access the page with the HTTP access pattern for guest authentication.
For example if your page is
http://buildserver/teamcity/piazza.htm
you can access it by appending a /guestAuth into the URL:
http://buildserver/teamcity/guestAuth/piazza.htm

Resources