In my spring application , I need to dynamically initialize multiple data sources based on some values set up in the application configuration.
I am aware of the AbstractRoutingDataSource class provided by spring jdbc library but it helps only when you need to initialize a single data source based on a single look up key value at a time.
Is it possible to extend the AbstractRoutingDataSource and change its behavior to support multiple key look up and data source resolution? Is there any other alternative approach ? Reference
Basically I am trying to achieve something like this through AbstractDataSourceRouter class:
public class DataSourceRouter extends AbstractRoutingDataSource {
#Value("${com.listdb.datasource.switch}")
private short listDBSwitch;
#Value("${com.scoringdb.datasource.switch}")
private short scoringDbSwitch;
#Value("${com.configmaster.datasource.switch}")
private short configDbSwitch;
private List<String> configuredDataSources;
/**
* Determine the current lookup key. This will typically be
* implemented to check a thread-bound transaction context.
* <p>Allows for arbitrary keys. The returned key needs
* to match the stored lookup key type, as resolved by the
* {#link #resolveSpecifiedLookupKey} method.
*/
#Override
protected Object determineCurrentLookupKey() {
if(ListUtil.isListNotEmpty(configuredDataSources)) {
configuredDataSources =new ArrayList<String>();
String listDBString = (listDBSwitch == 1)?DataSources.LIST.toString() : null;
String configDBString = (configDbSwitch == 1) ? DataSources.CONFIGMASTER.toString() :null;
String scoringDBString = (scoringDbSwitch == 1) ? DataSources.SCORING.toString() : null;
/**
* Add all configured data source keys for look up
*/
configuredDataSources.add(listDBString);
configuredDataSources.add(configDBString);
configuredDataSources.add(scoringDBString);
}
return configuredDataSources;
}
}
Any help/suggestions?
This is not really possible with current spring/hibernate versions even if it would be neat to have it that way. If you need multiple data sources and use AbstractRoutingDataSource to achieve this, then one possible solution is to let spring initialize one DB (the default/configuration DB) and add e.g. init.sql script (or flyway/liquibase such if you are more into that) that initializes all other under the same AbstractRoutingDataSource.
This approach works nicely and gives you more control over your (hopefully test!) environment. Personally I like to have more control over DB schema then any auto-initializers can provide, however that's only a taste/style issue.
Related
We are using java 8 parallel stream to process a task, and we are submitting the task through ForkJoinPool#submit. We are not using jvm wide ForkJoinPool.commonPool, instead we are creating our own custom pool to specify the parallelism and storing it as static variable.
We have validation framework, where we subject a list of tables to a List of Validators, and we submit this job through the custom ForkJoinPool as follows:
static ForkJoinPool forkJoinPool = new ForkJoinPool(4);
List<Table> tables = tableDAO.findAll();
ModelValidator<Table, ValidationResult> validator = ValidatorFactory
.getInstance().getTableValidator();
List<ValidationResult> result = forkJoinPool.submit(
() -> tables.stream()
.parallel()
.map(validator)
.filter(result -> result.getValidationMessages().size() > 0)
.collect(Collectors.toList())).get();
The problem we are having is, in the downstream components, the individual validators which run on separate threads from our static ForkJoinPool rely on tenant_id, which is different for every request and is stored in an InheritableThreadLocal variable. Since we are creating a static ForkJoinPool, the threads pooled by the ForkJoinPool will only inherit the value of the parent thread, when it is created first time. But these pooled threads will not know the new tenant_id for the current request. So for subsequent execution these pooled threads are using old tenant_id.
I tried creating a custom ForkJoinPool and specifying ForkJoinWorkerThreadFactory in the constructor and overriding the onStart method to feed the new tenant_id. But that doesnt work, since the onStart method is called only once at creation time and not during individual execution time.
Seems like we need something like the ThreadPoolExecutor#beforeExecute which is not available in case of ForkJoinPool. So what alternative do we have if we want to pass the current thread local value to the statically pooled threads?
One workaround would be to create the ForkJoinPool for each request, rather than make it static but we wouldn't want to do it, to avoid the expensive nature of thread creation.
What alternatives do we have?
I found the following solution that works without changing any underlying code. Basically, the map method takes a functional interface which I am representing as a lambda expression. This expression adds a preExecution hook to set the new tenantId in the current ThreadLocal and cleaning it up in postExecution.
forkJoinPool.submit(tables.stream()
.parallel()
.map((item) -> {
preExecution(tenantId);
try {
return validator.apply(item);
} finally {
postExecution();
}
}
)
.filter(validationResult ->
validationResult.getValidationMessages()
.size() > 0)
.collect(Collectors.toList())).get();
The best option in my view would be to get rid of the thread local and pass it as an argument instead. I understand that this could be a massive undertaking though. Another option would be to use a wrapper.
Assuming that your validator has a validate method you could do something like:
public class WrappingModelValidator implements ModelValidator<Table. ValidationResult> {
private final ModelValidator<Table. ValidationResult> v;
private final String tenantId;
public WrappingModelValidator(ModelValidator<Table. ValidationResult> v, String tenantId) {
this.v = v;
this.tenantId = tenantId;
}
public ValidationResult validate(Table t) {
String oldValue = YourThreadLocal.get();
YourThreadLocal.set(tenantId);
try {
return v.validate(t);
} finally {
YourThreadLocal.set(oldValue);
}
}
}
Then you simply wrap your old validator and it will set the thread local on entry and restore it when done.
Is there a way to pass data to dependencies registered with either Execution Context Scope or Lifetime Scope in Simple Injector?
One of my dependencies requires a piece of data in order to be constructed in the dependency chain. During HTTP and WCF requests, this data is easy to get to. For HTTP requests, the data is always present in either the query string or as a Request.Form parameter (and thus is available from HttpContext.Current). For WCF requests, the data is always present in the OperationContext.Current.RequestContext.RequestMessage XML, and can be parsed out. I have many command handler implementations that depend on an interface implementation that needs this piece of data, and they work great during HTTP and WCF scoped lifestyles.
Now I would like to be able to execute one or more of these commands using the Task Parallel Library so that it will execute in a separate thread. It is not feasible to move the piece of data out into a configuration file, class, or any other static artifact. It must initially be passed to the application either via HTTP or WCF.
I know how to create a hybrid lifestyle using Simple Injector, and already have one set up as hybrid HTTP / WCF / Execution Context Scope (command interfaces are async, and return Task instead of void). I also know how to create a command handler decorator that will start a new Execution Context Scope when needed. The problem is, I don't know how or where (or if I can) "save" this piece of data so that is is available when the dependency chain needs it to construct one of the dependencies.
Is it possible? If so, how?
Update
Currently, I have an interface called IProvideHostWebUri with two implementations: HttpHostWebUriProvider and WcfHostWebUriProvider. The interface and registration look like this:
public interface IProvideHostWebUri
{
Uri HostWebUri { get; }
}
container.Register<IProvideHostWebUri>(() =>
{
if (HttpContext.Current != null)
return container.GetInstance<HttpHostWebUriProvider>();
if (OperationContext.Current != null)
return container.GetInstance<WcfHostWebUriProvider>();
throw new NotSupportedException(
"The IProvideHostWebUri service is currently only supported for HTTP and WCF requests.");
}, scopedLifestyle); // scopedLifestyle is the hybrid mentioned previously
So ultimately unless I gut this approach, my goal would be to create a third implementation of this interface which would then depend on some kind of context to obtain the Uri (which is just constructed from a string in the other 2 implementations).
#Steven's answer seems to be what I am looking for, but I am not sure how to make the ITenantContext implementation immutable and thread-safe. I don't think it will need to be made disposable, since it just contains a Uri value.
So what you are basically saying is that:
You have an initial request that contains some contextual information captured in the request 'header'.
During this request you want to kick off a background operation (on a different thread).
The contextual information from the initial request should stay available when running in the background thread.
The short answer is that Simple Injector does not contain anything that allows you to do so. The solution is in creating a piece of infrastructure that allows moving this contextual information along.
Say for instance you are processing command handlers (wild guess here ;-)), you can specify a decorator as follows:
public class BackgroundProcessingCommandHandlerDecorator<T> : ICommandHandler<T>
{
private readonly ITenantContext tenantContext;
private readonly Container container;
private readonly Func<ICommandHandler<T>> decorateeFactory;
public BackgroundProcessingCommandHandlerDecorator(ITenantContext tenantContext,
Container container, Func<ICommandHandler<T>> decorateeFactory) {
this.tenantContext = tenantContext;
this.container = container;
this.decorateeFactory = decorateeFactory;
}
public void Handle(T command) {
// Capture the contextual info in a local variable
// NOTE: This object must be immutable and thread-safe.
var tenant = this.tenantContext.CurrentTenant;
// Kick off a new background operation
Task.Factory.StartNew(() => {
using (container.BeginExecutionContextScope()) {
// Load a service that allows setting contextual information
var context = this.container.GetInstance<ITenantContextApplier>();
// Set the context for this thread, before resolving the handler
context.SetCurrentTenant(tenant);
// Resolve the handler
var decoratee = this.decorateeFactory.Invoke();
// And execute it.
decoratee.Handle(command);
}
});
}
}
Note that in the example I make use of an imaginary ITenantContext abstraction, assuming that you need to supply the commands with information about the current tenant, but any other sort of contextual information will obviously do as well.
The decorator is a small piece of infrastructure that allows you to process commands in the background and it is its responsibility to make sure all the required contextual information is moved to the background thread as well.
To be able to do this, the contextual information is captured and used as a closure in the background thread. I created an extra abstraction for this, namely ITenantContextApplier. Do note that the tenant context implementation can implement both the ITenantContext and the ITenantContextApplier interface. If however you define the ITenantContextApplier in your composition root, it will be impossible for the application to change the context, since it does not have a dependency on ITenantContextApplier.
Here's an example:
// Base library
public interface ITenantContext { }
// Business Layer
public class SomeCommandHandler : ICommandHandler<Some> {
public SomeCommandHandler(ITenantContext context) { ... }
}
// Composition Root
public static class CompositionRoot {
// Make the ITenantContextApplier private so nobody can see it.
// Do note that this is optional; there's no harm in making it public.
private interface ITenantContextApplier {
void SetCurrentTenant(Tenant tenant);
}
private class AspNetTenantContext : ITenantContextApplier, ITenantContext {
// Implement both interfaces
}
private class BackgroundProcessingCommandHandlerDecorator<T> { ... }
public static Container Bootstrap(Container container) {
container.RegisterPerWebRequest<ITenantContext, AspNetTenantContext>();
container.Register<ITenantContextApplier>(() =>
container.GetInstance<ITenantContext>() as ITenantContextApplier);
container.RegisterDecorator(typeof(ICommandHandler<>),
typeof(BackgroundProcessingCommandHandlerDecorator<>));
}
}
A different approach would be to just make the complete ITenantContext available to the background thread, but to be able to pull this off, you need to make sure that:
The implementation is immutable and thus thread-safe.
The implementation doesn't require disposing, because it will typically be disposed when the original request ends.
Use case:
A one-time read of data set X (from database) into a Collection C. [Collection size could be say 5000]
Use Collection C to process/enrich items in a Spring Batch Step (say enrichStep)
If C is much greater than what can be passed via ExecutionContext, how can we make it available in the ItemProcessor of the enrichStep?
In your enrichStep add a StepExecutionListener.beforeStep and load your huge collection in a HugeCollectionBeanHolder bean.
In this way you will load collection only once (when step start or re-start) and without persist it into execution context.
In your enrich processor wire the HugeCollectionBeanHolder to access huge collection.
class HugeCollectionBeanHolder {
Collection<Item> hudeCollection;
void setHugeCollection(Collection<Item> c) { this.hugeCollection = c;}
Collection<Item> getHugeCollection() { return this.hugeCollection;}
}
class MyProcessor implements ItemProcessor<Input,Output> {
HugeCollectionBeanHolder hcbh;
void setHugeCollectionBeanHolder(HugeCollectionBeanHolder bean) { this.hcbh = bean;}
// other methods...
}
You can also look at Spring Batch: what is the best way to use, the data retrieved in one TaskletStep, in the processing of another step
I'd like to change the representation of C# Doubles to rounded Int64 with a four decimal place shift in the serialization C# Driver's stack for MongoDB. In other words, store (Double)29.99 as (Int64)299900
I'd like this to be transparent to my app. I've had a look at custom serializers but I don't want to override everything and then switch on the Type with fallback to the default, as that's a bit messy.
I can see that RegisterSerializer() won't let me add one for an existing type, and that BsonDefaultSerializationProvider has a static list of primitive serializers and it's marked as internal with private members so I can't easily subclass.
I can also see that it's possible to RepresentAs Int64 for Doubles, but this is a cast not a conversion. I need essentially a cast AND a conversion in both serialization directions.
I wish I could just give the default serializer a custom serializer to override one of it's own, but that would mean a dirty hack.
Am I missing a really easy way?
You can definitely do this, you just have to get the timing right. When the driver starts up there are no serializers registered. When it needs a serializer, it looks it up in the dictionary where it keeps track of the serializers it knows about (i.e. the ones that have been registered). Only it it can't find one in the dictionary does it start figuring out where to get one (including calling the serialization providers) and if it finds one it registers it.
The limitation in RegisterSerializer is there so that you can't replace an existing serializer that has already been used. But that doesn't mean you can't register your own if you do it early enough.
However, keep in mind that registering a serializer is a global operation, so if you register a custom serializer for double it will be used for all doubles, which could lead to unexpected results!
Anyway, you could write the custom serializer something like this:
public class CustomDoubleSerializer : BsonBaseSerializer
{
public override object Deserialize(BsonReader bsonReader, Type nominalType, Type actualType, IBsonSerializationOptions options)
{
var rep = bsonReader.ReadInt64();
return rep / 100.0;
}
public override void Serialize(BsonWriter bsonWriter, Type nominalType, object value, IBsonSerializationOptions options)
{
var rep = (long)((double)value * 100);
bsonWriter.WriteInt64(rep);
}
}
And register it like this:
BsonSerializer.RegisterSerializer(typeof(double), new CustomDoubleSerializer());
You could test it using the following class:
public class C
{
public int Id;
public double X;
}
and this code:
BsonSerializer.RegisterSerializer(typeof(double), new CustomDoubleSerializer());
var c = new C { Id = 1, X = 29.99 };
var json = c.ToJson();
Console.WriteLine(json);
var r = BsonSerializer.Deserialize<C>(json);
Console.WriteLine(r.X);
You can also use your own serialization provider to tell Mongo which serializer to use for certain types, which I ended up doing to mitigate some of the timing issues mentioned when trying to override existing serializers. Here's an example of a serialisation provider that overrides how to serialize decimals:
public class CustomSerializationProvider : IBsonSerializationProvider
{
public IBsonSerializer GetSerializer(Type type)
{
if (type == typeof(decimal)) return new DecimalSerializer(BsonType.Decimal128);
return null; // falls back to Mongo defaults
}
}
If you return null from your custom serialization provider, it will fall back to using Mongo's default serialization provider.
Once you've written your provider, you just need to register it:
BsonSerializer.RegisterSerializationProvider(new CustomSerializationProvider());
I looked through the latest iteration of the driver's code and checked if there's some sort of backdoor to set custom serializers. I am afraid there's none; you should open an issue in the project's bug tracker if you think this needs to be looked at for future iterations of the driver (https://jira.mongodb.org/).
Personally, I'd open a ticket -- and if a quick workaround is necessary or required, I'd subclass DoubleSerializer, implement the new behavior, and then use Reflection to inject it into either MongoDB.Bson.Serialization.Serializers.DoubleSerializer.__instance or MongoDB.Bson.Serialization.BsonDefaultSerializationProvider.__serializers.
I'm new to Google's Guava library and am interested in Guava's Caching package. Currently I have version 10.0.1 downloaded. After reviewing the documentation, the JUnit tests source code and even after searching google extensively, I still can't figure out how to use the Caching package. The documentation is very short, as if it was written for someone who has been using Guava's library not for a newbie like me. I just wish there are more real world examples on how to use Caching package propertly.
Let say I want to build a cache of 10 non expiring items with Least Recently Used (LRU) eviction method. So from the example found in the api, I build my code like the following:
Cache<String, String> mycache = CacheBuilder.newBuilder()
.maximumSize(10)
.build(
new CacheLoader<String, String>() {
public String load(String key) throws Exception {
return something; // ?????
}
});
Since the CacheLoader is required, I have to include it in the build method of CacheBuilder. But I don't know how to return the proper value from mycache.
To add item to mycache, I use the following code:
mycache.asMap().put("key123", "value123");
To get item from mycache, I use this method:
mycache.get("key123")
The get method will always return whatever value I returned from CacheLoader's load method instead of getting the value from mycache. Could someone kindly tell me what I missed?
Guava's Cache type is generally intended to be used as a computing cache. You don't usually add values to it manually. Rather, you tell it how to load the expensive to calculate value for a key by giving it a CacheLoader that contains the necessary code.
A typical example is loading a value from a database or doing an expensive calculation.
private final FooDatabase fooDatabase = ...;
private final LoadingCache<Long, Foo> cache = CacheBuilder.newBuilder()
.maximumSize(10)
.build(new CacheLoader<Long, Foo>() {
public Foo load(Long id) {
return fooDatabase.getFoo(id);
}
});
public Foo getFoo(long id) {
// never need to manually put a Foo in... will be loaded from DB if needed
return cache.getUnchecked(id);
}
Also, I tried the example you gave and mycache.get("key123") returned "value123" as expected.