Where\How to store distributed configuration data - oracle

Question

A single installation of our product stores it's configuration in a set of database tables.
None of the installations 'know' about any other installation.
It's always been common for customers to install multiple copies of our product in different datacentres, which are geographically far apart. This means that the configuration information needs to be created once, then exported to other systems. Some of the config is then modified to suit local conditions. e.g. changing IP addresses, etc. This is a clunky, error prone approach.
We're now getting requests for the ability to have a more seamless strategy for sharing of global data, but still allowing local modifications.
If it weren't for the local modifications bit then we could use Oracle's data replication features.
Due to HA requirements having all the configuration in the one database isn't an option.
Has anyone else encountered this problem and have you ever figured out a good programmatic solution for this? Know of any good papers that might describe a partial or full solution?
We're *nix based, and use Oracle. Changes should be replicated to all nodes pretty quickly (a second or 2).

Answer 1

I'm not sure how possible it is for you to change the way you handle your configuration, but we implemented something similar to this by using the idea of local overrides. Specifically, you have two configuration tables that are identical (call them CentralConfig and LocalConfig). CentralConfig is maintained at the central location, and is replicated out to your satellite locations, where it is read-only. LocalConfig can be set up at the local site. Your procedure which queries configuration data first looks for the data in the LocalConfig table, and if not found, retrieves it from the CentralConfig table.
For example, if you were trying to do this with the values in the v$parameter table, you could query your configuration using the FIRST_VALUE function in SQL analytics:
SELECT DISTINCT
NAME
, FIRST_VALUE(VALUE) OVER(PARTITION BY NAME
ORDER BY localsort
) VALUE
FROM (SELECT t.*
, 0 localsort
FROM local_parameter t
UNION
SELECT t.*
, 1 localsort
FROM v$parameter t
)
ORDER BY NAME;
The localsort column in the unions is there just to make sure that the local_parameter values take precedence over the v$parameter values.
In our system, it's actually much more sophisticated than this. In addition to the "name" for the parameter you're looking up, we also have a "context" column that describes the context we are looking for. For example, we might have a parameter "timeout" that is set centrally, but even locally, we have multiple components that use this value. They may all be the same, but we may also want to configure them differently. So when the tool looks up the "timeout" value, it also constrains by scope. In the configuration itself, we may use wildcards when we define what we want for scope, such as:
CONTEXT NAME VALUE
------------- ------- -----
Comp Engine A timeout 15
Comp Engine B timeout 10
Comp Engine % timeout 5
% timeout 30
The configuration above says, for all components, use a timeout of 30, but for Comp Engines of any type, use a timeout of 5, however for Comp Engines A & B, use 15 & 10 respectively. The last two configurations may be maintained in CentralConfig, but the other two may be maintained in LocalConfig, and you would resolve the settings this way:
SELECT DISTINCT
NAME
, FIRST_VALUE(VALUE) OVER(PARTITION BY NAME
ORDER BY (TRANSLATE(Context
, '%_'
, CHR(1) || CHR(2)
) DESC
, localsort
) VALUE
FROM (SELECT t.*
, 0 localsort
FROM LocalConfig t
WHERE 'Comp Engine A' LIKE Context
UNION
SELECT t.*
, 1 localsort
FROM CentralConfig t
WHERE 'Comp Engine A' LIKE Context
)
ORDER BY NAME;
It's basically the same query, except that I'm inserting that TRANSLATE expression before my localsort and I'm constraining on Context. What it's doing is converting the % and _ characters to chr(1) & chr(2), which will make them sort after alphanumeric characters in the descending sort. In this way, the explicitly defined "Comp Engine A" will come before "Comp Engine %", which in turn will come before "%". In cases where the contexts are defined identically, local config takes precedence over central ones; if you wanted local to always trump central, even in cases when central was scoped more tightly, you'd just reverse the positions of the two sort terms.

Answer 2

The way we're doing this is similar with Steve's.
First you need a Central Configure Service to save all the configure you want to apply to the distributed environment. Every time you want to modify the config, modify it in the Central Configure Service. In each production host you can write a loop script to update the configure.
For a more sophisticated solution, you need to set up some strategy to avoid a wrong configure batch into all servers, that would be a disaster. Maybe you need a simple lock or a grey release process.