Nifi custom properties (per-environment property files) looks to be a perfect way to define an environment specific paths and credentials. The only issue is how to keep sensitive information there? There is Nifi Encrypt-Config Tool described in more details here.
Is Nifi Encrypt-Config Tool capable of encrypting variable files (defined with nifi.variable.registry.properties) besides nifi.properties?
As far as I understood, it encrypts only nifi.properties. It's important, because with Nifi Docker Image I can define only nifi.variable.registry.properties (NIFI_VARIABLE_REGISTRY_PROPERTIES env var) without ability to modify nifi.properties.
The NiFi encrypt-config tool interacts with the following configuration files:
nifi.properties
login-identity-providers.xml
authorizers.xml
bootstrap.conf
flow.xml.gz
It does not handle any linked custom variable definition files, and there is no mechanism for sensitive variables to be properly secured and stored. Variables do not support any sensitive values at all for this reason.
Variables are treated as deprecated in modern versions of NiFi -- still supported but their use is discouraged -- and parameters were introduced in version 1.10.0 as a modern solution. Parameters do support sensitive values and are accessible from every property descriptor at the framework level rather than on a per-field basis depending on the developer's explicit decision to support them. You should prioritize parameters for the storage of sensitive values needed in your flow definitions.
Depending on your threat model, you may have less robust but acceptable alternatives:
If you accept the security level of environment variables, you can populate these directly and they will be referenced in any properties which support Expression Language, the same as "NiFi variables"
You can edit the nifi.properties file through a custom Docker image, startup scripts, etc. Any modified or added properties in that file can be encrypted by adding their key (property key descriptor, not cryptographic key) as a comma-delimited list to nifi.sensitive.props.additional.keys in that file. These properties will also be protected by the toolkit and decrypted in memory during NiFi application startup. However, nifi.properties is meant to hold framework-level configuration values, not component-level properties.
Related
We are currently set path of properties file which contains secret/access key for Credentials File for AWSCredentialsProviderControlerService . Issue, is we are changing properties path for prod and non prod each time we run nifi workflow. trying to come up no change on Configuration on Credential File path, so that access/secret key would be read regardless of prod and non prod. Since credential file wont support Nifi Expresion language, trying to make use of ACCESS KEY/SECRET properties ${ENV:equalsIgnoreCase("prod"):ifElse(${ACESS_PROD},${ACESS_NONPROD})} Issue we are facing, we are not able to store these access key/secret keys to the registry. Hence unable to implement this change. Is there any way to read access/secret key regardless of environment in Nifi. Curently we are using 1 property file for non prod nifi and 2nd property file for prod properties. In this set up, manually need to changed to credential file path when switching from prod to non prod. Trying to seamlessly work without changing path of credential file. Is there any way to make this happen?
enter image description here
The process that uses the AWSCredentialsProviderControlerService does not support param or variables, but the AWSCredentialsProviderControlerService "credential file" property supports "Parameters Context" as entries, make use of this for your solution.
Example:
Trigger something --> RouteOnAttribute --> If Prod (run executestreamcmd and change the Parameter Context Value to point to prod credfile) else if DEV(run executestreamcmd and change the Parameter Context Value to point to prod credfile) --> then run you AWS Processor.
You can use the toolkit client to edit the parameter context, or event nipyapi python module. It will not be fast tohu.
We have a number of (developer) existDb database servers, and some staging/production servers.
Each have their own configuration, that are slightly different.
We need to select which configuration to load and use in queries.
The configuration is to be stored in an XML file within the repository.
However, when syncing the content of the servers, a single burnt-in XML file is not sufficient, since it is overwritten during copying from the other server.
For this, we need the physical name of the actual database server.
The only function found, request:get-server-name that is not quite stable since a single eXist server can be accessed through a number of various (localhost, intranet or external) URLs. However, that leads to unnecessary duplication of the configuration, one for each external URL...
(Accessing some local files in the file system is not secure and not fast.)
How to get the physical name of the existDb server from XQuery?
I m sorry but I don't fully understand your question, are you talking about exist's default conf.xml or your own configuration file that you need to store in a VCS repo? Should the xquery be executed on one instance and trigger an event in all others, or just some, or...? Without some code it is difficult to see why and when something gets overwritten.
you could try console:jmx-token which does not vary depending on URL (at least it shouldn't)
Also you might find it much easier to use a docker based approach. Either with multiple instances coordinated via docker-compose or to keep the individual configs from not interfering with each other when moving from dev to staging to production https://github.com/duncdrum/exist-docker
If I understand correctly, you basically want to be able to get the hostname or the IP address of a server from XQuery. If the functions in the XQuery Request module are not doing as you wish, then another option would be to set a Java System Property when starting eXist-db. This system property could be the internal DNS name or IP of your server, for example: -Dour-server-name=server1.mydomain.com
From XQuery you could then read that Java System property using util:system-property("our-server-name").
Here is the link 12 factor
I am confused weather if I should store values inside my app.properties file vs environment variable.
App.properties
Memory_Folder_Test = Test
Memory_Folder_Prod = Production
Memory_Folder_Dev = Development
Strong_threshold = 10
Low_Threshold = 2
Username = FirstUser
Password = PasswordSecret
So theoretically where should I put these values in? application.properties or as environment variables? If I did not read wrong the purpose of 12 factor is to remove putting values in properties file and externalize it.
You can store the values in application.properties file, however, spring allows you to override those values using environment variables. Hence, it is a 12 factor compliant.
You store the properties externally using something like spring cloud config. You then use the environment properties to define the configurations (like the url) needed to access cloud config from your applications
I prefer to store environment variables in files, encrypt the files and check the encrypted files into git, via blackbox: https://github.com/StackExchange/blackbox
Blackbox will handle encryption/decryption so that it makes it rather difficult to check the unencrypted creds into your repo. Also, the way openpgp works, you can enable teams of devs to encrypt/decrypt the files.
That project is maintained by StackExchange (aka the guys who run this site). It takes some time figuring out openpgp/gpg (which blackbox depends on), but it has been well worth it for me. I've been using in linux and also in windows (via the windows linux subsystem).
I have NiFi template exported as xml. I am using rest-api to upload template to a NiFi instance. Now, I want to update/add some properties (say, password) of the template from rest-api (or any other option available, programatically).
I read the docs and various community threads without success. Referred links:
How to set props of processor
Update nifi flow on the fly
Open for any approach,
Thanks
I think there is a bit of confusion in your wording. Correct me if I'm wrong but I believe what you want to do is:
Create a template in one location
Export it
Upload it to another NiFi instance
Add the template to the canvas (so now it's just components on your NiFi canvas)
Edit the properties of the components that were added
There are generally two different reasons you would want to edit the properties after importing a template: the properties are specific to the instance you're running on; they were sensitive properties.
With the addition of the "variable registry" in NiFi-0.7.0 you can have multiple files that at NiFi's start-up are read in to give custom variables to use. Here is a section about it in the NiFi docs. This allows you to have custom variables to reference via Expression Language (EL) specific to each environment you run on.
The "variable registry" doesn't help for the sensitive properties though, because the EL used to reference them doesn't get exported with the template (since the property is sensitive). You will need to use the rest-api to update the processor properties explicitly. The NiFi docs give the exact call to use to update a processor (under Processors -> Put). Upgrading the variable registry to work securely is on the NiFi roadmap.
If I was completely off and you simply want to modify a template after importing it into a NiFi instance. You would have to add the template to your graph, delete the template from the listing, re-create it using the components on your graph. After templates are imported/created they are immutable.
We deploy services as docker containers using Marathon. The containers include a base config file, Marathon pulls an environment config file (which has a subset of the base keys) at deployment time so when the app starts it has;
environment.toml
config.toml
when reading the config we need to conflate the values in both files to a single set, effectively masking/shadowing the values present in both files with those in the environment file.
I didn't find this functionality in the Viper docs. Unless I have missed something it seems my options are;
Write a package that uses Viper to read both files and perform the conflation.
Extend Viper
Before I start writing code, is there already a mechanism for doing this?