Does .Net Core support User Secrets per environment? - visual-studio

Let's say I have connection string for Development environment specified in appsettings.Development.json and connection string for the Staging environment specified in appsettings.Staging.json
All I need to do to switch between Development and Staging is to navigate to Visual Studio Debug tab in project properties and change the value for ASPNETCORE_ENVIRONMENT environment variable.
Now, of course I don't want to have connection string in appsettings.*.json for security reasons. So I move it to User Secrets.
Problem is - it seems there is just one secrets.json file that is used by all the environments. There are no secrets.Development.json or secrets.Staging.json. This means after I switch from Development to Staging environment via Visual Studio Debug tab I then also need to change connection strings manually in secrets.json which kind of defeats the purpose of having built-in support for the environments.
Is this correct that User Secrets are not supported on per-environment basis? If so - is there another approach that would avoid having to modify Secret connection string manually when switching environments?

If you check the tool's parameters with dotnet user-secrets --help you'll see you can specify different secrets per configuration (Debug, Release, any other you want) but not per environment. Which is not a bad decision if you think about it.
The ASPNETCORE_ENVIRONMENT environment variable is meant to tell your application whether the current machine or container is a Development, Production or other environment, so it can pick the appropriate settings file. This environment variable isn't expected to change from one application execution to the next. Even when using containers, the environment variables are passed from the host to the container and aren't expected to change during the container's lifetime.
The secrets files are supposed to be per machine, for development purposes, so there's no need to keep separate files per environment. It makes much more sense to use separate files for configuration, allowing developers to simply change from Dev to Release or Testing or any other custom configuration they may have.
Specifying secrets per configuration
The dotnet user-secrets tool works by reading the UserSecretsId value from the project file and storing the secrets in a JSON file with the same name, eg c952ecfc-344e-43e1-bb67-1ac05973d6c6.json. It's possible to store a UserSecretsId for each configuration.
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Release|AnyCPU'">
<UserSecretsId>c952ecfc-344e-43e1-bb67-1ac05973d6c6</UserSecretsId>
</PropertyGroup>
<PropertyGroup Condition="'$(Configuration)|$(Platform)'=='Debug|AnyCPU'">
<UserSecretsId>7D104000-2230-4EDE-8AE6-63BDDA0BD0C5</UserSecretsId>
</PropertyGroup>
When the -c parameter is used to specify a configuration, the user-secrets tool will read the UserSecretsId value from the corresponding section and use it to store or read secrets.
The dotnet user-secrets init command doesn't recognize the -c parameter, so the csproj file needs to be modified directly.
Once that's done, one can set and read secrets by specifying the configuration, eg :
❯ dotnet user-secrets set -c Debug Key1 Value1
Successfully saved Key1 = Value1 to the secret store.
❯ dotnet user-secrets set -c Release Key1 Value2
Successfully saved Key1 = Value2 to the secret store.
❯ dotnet user-secrets list -c Debug
Key1 = Value1
❯ dotnet user-secrets list -c Release
Key1 = Value2

I also needed this and I think I've come up with an elegant solution.
secrets.json file is shared among all environments you are using, what you can do is add the environment parent to each node in the file and then do the little trick (last 2 code fragments).
Suppose you have a configuration, e.g. in appsettings.json or appsettings.{environment}.json:
{
"Key1": "value1",
"Secret1": "<set yourself>"
}
and then you have the secret part in secrets.json:
{
"Secret1": "my secret value"
}
you can get and bind the whole section easily:
IConfiguration configuration = new ConfigurationBuilder()
.AddJsonFile("appsettings.json")
.AddJsonFile($"appsettings.{environment}.json")
.AddUserSecrets<Program>()
.Build();
var myConfiguration = configuration.Get<MyConfiguration>();
Now comes the problem, I want to have multiple environments in the secrets.json. I personally used to have secrets for all environments I needed and just comment/uncomment what I wanted, however, it's manual work. So I will prefix them with environment name instead.
{
"Development:Secret1": "my secret development value",
"Staging:Secret1": "my secret staging value"
}
You have to load the environment-specific configuration(s) from the IConfiguration instance and override the existing myConfiguration values using:
configuration.GetSection(environment).Bind(myConfiguration);
\\ or
configurationRoot.Bind(environment, configuration);
And that's it.
If you run it using environment="Development", you will have "my secret development value" loaded. If you run it using environment="Staging", you will have "my secret staging value" loaded.
Additional details
The double dot character (:) acts as a new section, so if you write
{
"Development:Secret1": "my secret development value",
"Staging:Secret1": "my secret staging value"
}
it's the same as
{
"Development":
{
"Secret1": "my secret development value"
},
"Staging":
{
"Secret1": "my secret staging value"
}
}
the trick is about loading just the environment-specific part and binding it to the myConfiguration instance.
I must mention that no matter what environment you are using, all the secrets are actually loaded in the memory.

The Secret Manager (https://learn.microsoft.com/en-us/aspnet/core/security/app-secrets?view=aspnetcore-3.1) is designed strictly for development, not any other stage (environment), since it is inherently insecure (local dev secrets are not encrypted). See the warning on the page linked. So there is no need to have per environment secrets storage vis-a-vis that tool. For other environments (staging, prod, etc), Microsoft would likely steer you toward their secure secrets storage service -- Key Vault. You can use the Secret Manager for dev secrets and then store the other environments in Key Vault. I have done this in many Asp.Net Core apps and it works well. For Key Vault info, see this:
https://learn.microsoft.com/en-us/aspnet/core/security/key-vault-configuration?view=aspnetcore-3.1

There is no "out of the box" way to use different secrets per environment. However you can use the options pattern to bind your configs depending on environment:
Create a secrets.json with your environments at the top level:
{
"Local": {
"Secret1": "local secret",
"ConnectionStrings": {
"DB": "LocalDBConnectionstring"
}
},
"Development": {
"Secret1": "dev secret",
"ConnectionStrings": {
"DB": "DevDBConnectionstring"
}
}
}
Create classes to map your secret configurations:
public class SecretConfigurationEnvironment
{
public string secret1 { get; set; }
public ConnectionStringsConfig ConnectionStrings { get; set; }
}
public class ConnectionStringsConfig
{
public string DB { get; set; }
}
Map the secrets to your classes in your program.cs:
var builder = WebApplication.CreateBuilder(args);
builder.Services.Configure<SecretConfigurationEnvironment>(
builder.Configuration.GetSection(builder.Environment.EnvironmentName));
Use your configs with dependencyInjection whereever you need them:
public class MyClass {
private readonly IOptions<SecretConfigurationEnvironment> _secretConfig;
public MyClass(IOptions<SecretConfigurationEnvironment> secretConfig) {
_secretConfig = secretConfig
}
public void MyMethod() {
var envDBConnectionString = _secretConfig.value.Connectionstrings.DB
}
}
Now your configs will change depending on your environment and you can still use all your configs stored in the appsettings.environment.json files (also with options if you like)
That being said, here is the usual disclaimer: Don't use secrets.json for staging or production secrets.
The code i provided is for .net 6, wep-api. Older versions or different project-types are very similar though and can be found at the microsoft documentation.

Related

How do environment variables and appSettings file(s) get used during publish?

If I have two settings files
appSettings.json and appSettings.Development.json
When I use publish from Visual Studio, are both supposed to be copied to the target folder? I'm not sure, because they both show up in the target folder (on a dev server) when I publish. I was under the impression that they combined at build time and ONLY the appSettings.json file was published. If not, then do I need to consider manually coding for these differences as Ive seen in a few examples ?
eg. This example is loading the settings via code (NOT how Im doing it)
Note - they are using the environment name, ASPNETCORE_ENVIRONMENT setting
public Startup(IHostingEnvironment env)
{
var builder = new ConfigurationBuilder()
.SetBasePath(env.ContentRootPath)
.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
.AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true, reloadOnChange: true)
.AddEnvironmentVariables();
Configuration = builder.Build();
}
Some of my Startup class is shown below.
Note: I am not referencing the environment setting.
public Startup(IConfiguration configuration)
{
Configuration = configuration;
}
public IConfiguration Configuration { get; }
// This method gets called by the runtime. Use this method to add services to the container.
public void ConfigureServices(IServiceCollection services)
{
services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);
// In production, the Angular files will be served from this directory
services.AddSpaStaticFiles(configuration =>
{
configuration.RootPath = "ClientApp/dist";
});
}
// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
if (env.IsDevelopment())
{
app.UseDeveloperExceptionPage();
}
else
{
app.UseExceptionHandler("/Error");
// The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
app.UseHsts();
}
app.UseHttpsRedirection();
app.UseStaticFiles();
app.UseSpaStaticFiles();
app.UseMvc(routes =>
{
routes.MapRoute(
name: "default",
template: "{controller}/{action=Index}/{id?}");
});
app.UseSpa(spa =>
{
// To learn more about options for serving an Angular SPA from ASP.NET Core,
// see https://go.microsoft.com/fwlink/?linkid=864501
spa.Options.SourcePath = "ClientApp";
if (env.IsDevelopment())
{
spa.UseAngularCliServer(npmScript: "start");
}
});
}
[ Update ]
I found my answer here - the key I was missing was updating the csproj file for the publish settings related to environment.
https://learn.microsoft.com/en-us/aspnet/core/host-and-deploy/aspnet-core-module?view=aspnetcore-2.2#configuration-with-webconfig
So I assume that if I have several different environments, each with its own settings file, that a publish would result in putting ALL of them out to the target dir?
It's a bit confusing with ASP.NET Core, especially if you're coming from having worked with ASP.NET previously. The build configurations (Debug, Release) really have no bearing on anything that happens with ASP.NET Core. An ASP.NET Core app is technically environment-agnostic. Whereas with an older ASP.NET app, you'd have to publish for a specific environment, you can theoretically take the same ASP.NET Core publish and run it in any of your environments. This is of course aided by the fact that Web.config is not utilized by ASP.NET Core.
This, then, is the reason why all the environment-specific JSON files come along for the ride. Which is ultimately used is based on the value of the ASPNETCORE_ENVIRONMENT environment variable set at runtime, not which build configuration you chose when publishing. Which is actually really nice when you think about it. You can take the same published app, run it in your "staging" environment to ensure everything is working and then deploy it to your "production" environment, simply by ensuring that each environment has the appropriate value for ASPNETCORE_ENVIRONMENT set. This makes release pipelines trivial.
That said, it's still possible to use things like the #if DEBUG compiler directives, and if you do that then there will be differences in your ASP.NET Core app depending on the build configuration chosen, but you should really avoid doing that in the first place. In general, you should rely only on the IHostingEnvironment abstraction in an ASP.NET Core app to determine what happens in what environment.

How to store a private key in Github and VSTS Continuous Integration?

I have a private API key that I am using in my public Github project. I would like to keep this key private. Currently, I am doing this using an Environment Variable.
The problem I'm running into is that this project is also integrated into VSTS Continuous Integration environment. So when my code runs there, the key is not available since it's only on my local machine.
I've tried using Azure Key Vault as recommended by Microsoft here. But that code didn't work for me. I downloaded the Nuget package that they recomended. However, when I tried to use this code the compiler had no idea what IWebHost or Startup is:
public static IWebHost BuildWebHost(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.ConfigureAppConfiguration((ctx, builder) =>
{
var keyVaultEndpoint = GetKeyVaultEndpoint();
if (!string.IsNullOrEmpty(keyVaultEndpoint))
{
var azureServiceTokenProvider = new AzureServiceTokenProvider();
var keyVaultClient = new KeyVaultClient(
new KeyVaultClient.AuthenticationCallback(
azureServiceTokenProvider.KeyVaultTokenCallback));
builder.AddAzureKeyVault(
keyVaultEndpoint, keyVaultClient, new DefaultKeyVaultSecretManager());
}
})
.UseStartup<Startup>()
.Build();
Does anyone have an idea of how to do this?
You can use Azure Key Vault task to download the private keys to the agent machine firstly, and then use key by related variable in VSTS build.
Assume there has an priavte key mykey in Azure key vault, after Azure Key Vault task, you can use the variable $(mykey) to get the key value.

How to use a windows environment variable to hide an API key?

I have a Visual Studio 2015 ASP.NET Core 1.0 project that uses the Google Maps API. Currently, I could hard code the API key in two places:
1) the script tag of a view e.g.
<script src="https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY_HERE"></script>
2) the config.json file of the project e.g.
{
"Keys": { "GoogleMapsAPIKey": "YOUR_API_KEY_HERE" }
}
Either way the key is exposed when using source control so why not use a Windows Environment Variable?
Within Windows System Properties I added a new environment variable called GoogleMapsAPIKey and pasted in the actual key.
Now how do I use this environment variable in either the script tag or in config.json?
The purpose of the question is to get a general answer on how to use environment variables to hide API keys in such situations.
Add the environment variables configuration provider. Example from here:
public Startup(IHostingEnvironment hostingEnvironment)
{
var builder = new ConfigurationBuilder()
.SetBasePath(hostingEnvironment.ContentRootPath)
.AddJsonFile("config.json")
.AddEnvironmentVariables();
}
If you just want to hide the key during development, you can use user secrets.

Customize appsetings.json for user in ASP Core 1 app

My goal is to have appsettings.json file with production configurations and have possibility to costomize it for every developer, e.g. use local connection strings. So it does not similar to transform web.config mechanism, i don't want depends on bulid configuration. Can anyone provide solution for this goal?
In one of my past project we do so: we store all configure information in custom config.xml and parsed it into the custom structure. Web.config contains only server configaration. every developer has own copy of config files with his own data. Solution is that application use configuration files from path, that specified in environment path in windows via Environment.GetEnvironmentVariable("key").
Does anyone have idea better than my one?
This is how I manage configuration: see comments in the code
public Startup(IHostingEnvironment env, IApplicationEnvironment appEnv)
{
var builder = new ConfigurationBuilder()
.AddJsonFile("appsettings.json"); // this one has default configuration
// this file name is added to my gitignore so it won't get committed,
// I keep local dev configuration there
builder.AddJsonFile("appsettings.local.overrides.json", optional: true);
if (env.IsDevelopment())
{
// This reads the configuration keys from the secret store.
// if you need a more secure place for dev configuration use usersecrets
// For more details on using the user secret store see http://go.microsoft.com/fwlink/?LinkID=532709
builder.AddUserSecrets();
}
// the order in which config sources is added is important, a source added later
// will override the same settings from a source added before
// environment variables is usually for production and therefore added last to give it higher priority
builder.AddEnvironmentVariables();
Configuration = builder.Build();
}

How to speed up Azure deployment from Visual Studio 2010

I have Visual Studio 2010 solution with an Azure Service and an ASP.NET MVC 3 solution that serves as a Web Role for the Azure service. No other roles attached to the service other than that.
Every deployment to the Azure staging (or production, for that matter) environment takes up to 20 minutes to complete, form the moment I click publish on Visual Studio until all instances (2) are started.
As you can imagine this makes it a PITA to publish often, or to quick-fix some bugs. Is there a way to speed the process up? Would it be faster to upload the package to de Blob storage and upgrade from there? How would I go about achieving that?
I feel on-line docs on Azure leave a lot to be desired. Particularly when it comes to troubleshooting by the way.
Thanks.
One idea for reducing the need (and frequency) for redeploying is to move static content into blob storage, external to the package. For instance, move your css and javascript to blob storage, along with images. Once this is done, you'd only have to recompile / redeploy for .NET code changes. You can upload updated css, at any time, to blob storage. If you want to test this in staging first, you could always have a staging vs. production container name for your static content and store that container name in a config setting.
This doesn't change the deployment time when you do need to redeploy, but at least you can reduce how often you go through that process...
You should enable Web Deploy in your Azure project. It works this way :
1/ Create a RDP account (don't forget, you need to upload a certificate with its private key so that Azure can decipher the password). That is hidden in the Deploy Dialog Box for your Azure deployment project.
2/ Enable Web Deployment - same place
Once you've published the app that way, right-click in the web application (not the azure deployment project) and select Publish. The pop-up has everything defined except the password, enter that as well and you'll upload your changes to Azure in a matter of seconds.
CAVEAT : this is meant for single-instance web apps, definitely not the way to go for a production upgrade strategy, and the Blob storage answer already mentioned is the best option in that case.
Pierre
My solution to this problem is only to push a new package when I am changing code in the RoleEntryPoint or with the Service Definition. In Azure 1.3 you now have the ability to use Remote Desktop Connection. Using RDC, I will compile my code locally and use copy/paste to place it on the Azure server in the appropriate directory. Once the production code is running correctly, I can then push the fully tested version to staging and then do a VIP swap. This limits the number of times I actually have to deploy a package.
You actually have quite a long window in which you can keep modifying your code in Azure before you have to publish a new package. The new package is only really needed for those cases where Azure has to shutdown/restart your role instance.
It's a nice idea to try uploading your project to blob storage first, but unfortunately this is what Visual Studio is doing for you behind the scene anyway. As has been pointed out elsewhere, most of the time in doing the deploy is not the upload itself, but the stopping and starting of all of your update domains.
If you're just running this site in a development environment, then the only way I know to speed it up is to run just one instance. If this is the live environment, then... sorry, I think you're out of luck.
So that I don't have to deploy to the cloud to test minor changes, what I've found works quite well is to engineer the site so that it works when running in local IIS just like any other MVC site.
The biggest barrier to this working are settings that you have in the cloud config. The way we get around this is to make a copy of all of the settings in your cloud config and put them in your web.config in the appSettings. Then rather than using RoleEnvironment.GetConfigurationSettingValue() create a wrapper class that you call instead. This wrapper class checks RoleEnvironment.IsAvailable to see if it is running in the Azure fabric, if it is, it calls the usual config function above, if not, it calls WebConfigurationManager.AppSettings[].
There are a few other things that you'll want to do around getting the config setting change events which hopefully you can figure out from the code below:
public class SmartConfigurationManager
{
private static bool _addConfigChangeEvents;
private static string _configName;
private static Func<string, bool> _configSetter;
public static bool AddConfigChangeEvents
{
get { return _addConfigChangeEvents; }
set
{
_addConfigChangeEvents = value;
if (value)
{
RoleEnvironment.Changing += RoleEnvironmentChanging;
}
else
{
RoleEnvironment.Changing -= RoleEnvironmentChanging;
}
}
}
public static string Setting(string configName)
{
if (RoleEnvironment.IsAvailable)
{
return RoleEnvironment.GetConfigurationSettingValue(configName);
}
return WebConfigurationManager.AppSettings[configName];
}
public static Action<string, Func<string, bool>> GetConfigurationSettingPublisher()
{
if (RoleEnvironment.IsAvailable)
{
return AzureSettingsGet;
}
return WebAppSettingsGet;
}
public static void WebAppSettingsGet(string configName, Func<string, bool> configSetter)
{
configSetter(WebConfigurationManager.AppSettings[configName]);
}
public static void AzureSettingsGet(string configName, Func<string, bool> configSetter)
{
// We have to store these to be used in the RoleEnvironment Changed handler
_configName = configName;
_configSetter = configSetter;
// Provide the configSetter with the initial value
configSetter(RoleEnvironment.GetConfigurationSettingValue(configName));
if (AddConfigChangeEvents)
{
RoleEnvironment.Changed += RoleEnvironmentChanged;
}
}
private static void RoleEnvironmentChanged(object anotherSender, RoleEnvironmentChangedEventArgs arg)
{
if ((arg.Changes.OfType<RoleEnvironmentConfigurationSettingChange>().Any(change => change.ConfigurationSettingName == _configName)))
{
if ((_configSetter(RoleEnvironment.GetConfigurationSettingValue(_configName))))
{
RoleEnvironment.RequestRecycle();
}
}
}
private static void RoleEnvironmentChanging(object sender, RoleEnvironmentChangingEventArgs e)
{
// If a configuration setting is changing
if ((e.Changes.Any(change => change is RoleEnvironmentConfigurationSettingChange)))
{
// Set e.Cancel to true to restart this role instance
e.Cancel = true;
}
}
}
The uploading itself takes a bit more than a minute most of the time. It's the starting up of the instances that take up most of the time.
What you can do is to deploy your fixes to staging first (note that it costs money so don't let it be there for too long). Swapping from staging to production only takes a couple of seconds. So while your application's still running you can upload the patched version, let your testers test it on staging and when they give the go then simply swap it to production.
I haven't tested your possible alternative approach by first uploading to blob storage first. But I think that's overhead as it doesn't speed up starting up the instances.

Resources