So I was searching the web looking for best practices when implementing the repository pattern with multiple data stores when I found my entire way of looking at the problem turned upside down. Here's what I have...
My application is a BI tool pulling data from (as of now) four different databases. Due to internal constraints, I am currently using LINQ-to-SQL for data access but require a design that will allow me to change to Entity Framework or NHibernate or the next data access du jour. I also hold steadfast to decoupled layers in my apps using an IoC framework (Castle Windsor in this case).
As such, I've used the Repository pattern to abstract the actual data access code from my business layer. As a result, my business object is coded against some I<Entity>Repository interface and the IoC Container is used to manage the actual implementation. In this case, I would expect to have a concrete Linq<Entity>Repository that implements the interface using LINQ-to-SQL to do the work. Later I could replace this with an EF<Entity>Repository with no changes required to my business layer.
Also, because I'm coding against the interface, I can easily mock the repository for unit testing purposes.
So the first question that I have as I begin coding the application is whether I should have one repository per DataContext or per entity (as I've typically done)? Let's say one database contains Customers and Sales with the expected relationship. Should I have a single OrderTrackingRepository with methods that work with both entities or have a separate CustomerRepository and a different SalesRepository?
Next, as a BI tool, the primary interface is for reporting, charting, etc and often will require a "mashup" of data across multiple sources. For instance, the reality is that one database contains customer information while another handles sales information and a third holds other financial information but one of my requirements is to display aggregated information that spans all three. Plus, I have to support dynamic filtering in the UI. Obviously working directly against the LINQ-to-SQL or EF DataContext objects (Table<Entity>, for instance) will allow me to pretty much do anything. What's the best approach to expose that same functionality to my business logic when abstracting the DAL with a repository interface?
This article: link text indicates that EF4 has turned this approach around and that the repository is nothing more than an IQueryable returned from the EF DataContext which brings up a whole other set of questions.
But, I think I've rambled on enough...
UPDATE (Thanks, Steven!)
Okay, let me put a more tangible (for me, at least) example on the table and clarify a few points that will hopefully lead to an approach I can better wrap my head around.
While I understand what Steven has proposed, I have a team of developers I have to consider when implementing such things and I'm afraid they will get lost in the complexity (yes, a real problem here!).
So, let's remove any direct tie-in with Linq-to-Sql because I don't want a solution that is dependant upon the way L2S works - or even EF, for that matter. My intent has been to abstract away the data access technology being used so that I can change it as needed without requiring collateral changes to the consuming code in my business layer. I've accomplished this in the past by presenting the business layer with IRepository interfaces to work against. Perhaps these should have been named IUnitOfWork or, more to my liking, IDataService, but the goal is the same. These interfaces typically exposed methods such as Add, Remove, Contains and GetByKey, for example.
Here's my situation. I have three databases to work with. One is DB2 and contains all of the business information for a customer (franchise) such as their info and their Products, Orders, etc. Another, SQL Server database contains their financial history while a third SQL Server database contains application-specific information. The first two databases are shared by multiple applications.
Through my application, the customer may enter/upload their financial information for a given time period. When entered, I have to perform the following steps:
1.Validate the entered data against a set of static rules. For example, the data must contain a legitimate customer ID value (in the case of an upload). This requires a lookup in the DB2 database to verify that the supplied customer ID exists and is current.
2.Next I have to validate the data against a set of dynamic rules which are contained in the third (SQL Server) database. An example may be that a given value cannot exceed a certain percentage of another value.
3.Once validated, I persist the data to the second SQL Server database containing the financial data.
All the while, my code must have loosely-coupled dependencies so I may mock them in my unit tests.
As part of the analysis, I know that I have three distinct data stores to work with and about a half-dozen or so entities (at this time) that I am working with. In generic terms, I presume that I would have three DataContexts in my application, one per data store, with the entities exposed by the appropriate data context.
I could then create a separate I{repository|unit of work|service} for each entity that would be consumed by my business logic with a concrete implementation that knows which data context to use. But this seems to be a risky proposition as the number of entities increases, so does the number of individual repository|UoW|service types.
Then, take the case of my validation logic which works with multiple entities and, thereby, multiple data contexts. I'm not sure this is the most efficient way to do this.
The other requirement that I have yet to mention is on the reporting side where I will need to execute some complex queries on the data stores. As of right now, these queries will be limited to a single data store at a time, but the possibility is there that I might need to have the ability to mash data together from multiple sources.
Finally, I am considering the idea of pulling out all of the data access stuff for the first two (shared) databases into their own project and have been looking at WCF Data Services as a possible approach. This would give me the basis for a consistent approach for any application making use of this data.
How does this change your thinking?
In your case I would recommend returning IEnummerables's for your data queries for the repo. I usually aggregate calls from multiple repo's through a service class that represents the domain problem and encapsulates my business logic. To keep it clean I try keep my repros focused on the domain problem. I liken my Datacontext to a repo, and extract an interface using a T4 template to make life easier for mocking. But there is nothing stopping you using a traditional repo that encapsulates your calls. Doing it this way will allow you to switch ORM's at any stage.
EDIT: IQueryable IS NOT THE ANSWER! :-)
I have also done a lot of work in this area, and INITIALLY came to the same conclusion, however it is NOT a good solution. The point of the Repo is to abstract queries into discrete chunks of work. Exposing IQueryable is too adhoc and raises some issues later down the line. You loose your ability to scale. You loose your ability to optimize queries (Lets say I want to move to a highly optimized stored proc). You loose your ability to use IoC for the repo to switch out data access layers (switch the project from SQL to Mongo). You loose your ability to provide effective data caching in the Repo (Which is a major strength in the Repo pattern). I would recommend taking a CLOSE look as to WHY we have a Repo pattern. It isn't simply an "ORM" mapping layer. What made this really clear to me was the CQRS pattern.
Further to this allowing the ad-hoc nature of IQueryable opens you to misfitting reuse of queries. It is GENERALLY not a good idea to reuse queries, since query to query you see slight deviations, which ends up with 2 byproducts: Queries become too broad and inefficient. Queries become riddled with unmaintainable IF THEN statements to cater for the deviations.
IQueryable is easy, but opens you up to an unmaintainable mess.
Look at this SO answer. I think it shows a simplified model of what you want. IQueryable<T> is indeed our new Repository :-). DataContext and ObjectContext are our Unit of Work.
UPDATE 2:
Here is a blog post that describes the model you might be looking for.
UPDATE 3
It would be wise to hide the shared databases behind a service. This will solve several problems:
This will make the database private to the service, which makes it much easier to change the implementation when needed.
You can put the needed validation logic (for database 1) in that service and can create tests for that validation logic in that project.
Clients accessing that service can assume correctness of the service, and its validation logic.
The result of this is that your application will send data to the service to validate it. Call the service to fetch data. Query its own private database (database 3) and join the data of the three data source locally together. I've never been a fan of using cross-database or even cross-server (in your situation) database calls and letting the database join everything together. Transactions will be promoted to distributed-transactions and it's hard to predict how many data the servers will exchange.
When you abstract the shared databases behind the service, things get easier (at least from your application's point of view). Your application calls services it trusts which limits the amount of code in that application and the amount of tests. You still want to mock the calls to such a service, but that would be pretty easy. It should also solve the problem of validating over multiple data sources.
Validation is always a hard part. I'm very familiar with Validation Application block, and love it for it's flexibility. It isn't however an easy framework, but you might take a peek at what you can do with it. For instance, I've written several articles about integration with O/RM tools and how to 'embed' a context (context as in DataContext/Unit of Work) in Validation Application Block.
Please have a look at my IRepository pattern implementation using EF 4.0.
My solution has the following features:
supports connections to multiple dbs
One repository per entity
Support for execution of queries
Unit of work pattern implementation
Support for validating entities using VAB guidance
Common operations are kept at base class level. High use of OOPS techniques for code re-usability and ease of maintenance.
Related
I guess traditionally, one would for a RESTful web service use one type of DTO objects for POJO/JSON conversion, and a separate DTO object for database entity/POJO conversion?
Spring Boot should be more opinionated and easier to use, but would you still use different DTO object types for JSON and database entity representation, or do you convert entity objects directly to JSON?
Let me share my opinion.
At first, I think your question has nothing to do with spring boot. Spring boot just provides a fancy and lightweight way to start the application and allows to build the app in an easier manner.
But still you have your rest controller there and from that point it doesn't differ much for any other type of application.
So what you're actually asking is whether it makes sense to maintain an abstraction of JSON objects and converting them to the Business Logic Entity objects and later on converting them once again to Database objects or its enough to maintain only 2 levels and ditch the Json level.
I think the answer is "it depends".
First of all, In general currently the trend is a simplification. So maybe its enough to maintain only 1 level of objects.
There are a lot of advantages of such an approach:
Obviously less code to maintain
Speed of development and testing (POJOs should be checked, converters should be tested and so forth)
Speed of execution - you don't need to waste the CPU time on conversion. A kind of obvious implication.
Less obvious: Memory consumption. Lets say you work with a big bulk of data returned by your DAO. Let's say it occupies 10MB of memory (just for the sake of example). Now if you start to convert, to Business Entities, you'll spend yet another 10MB and now if its A JSon objects, well its again 10MB. The point is that all these objects may co-exist in memory simultaneously. Of course GC will probably take care of them if you implemented everything right, but this is a different story.
However there is one drawback of such a simplification.
In one word I would call it a Commitment
There are three Types of APIs in the application.
The API you're committed to at the level of Web Service - The JSon structure.
The chances are that various clients (not necessary using the JVM at all) are running against your Web service and consume the data. So they really expect you to provide a JSon objects of the given structure.
The API of your business. If your Business logic layer is pretty complicated, you probably have an entire team that develops that logic. So you usually work at the level of APIs between the teams.
The level of DAO - the same story as Business Logic actually.
So now, what happens if you, say, change that API at one level. Does it mean that all the levels will be broken?
Example
Lets say, we don't maintain "JSon" level. In this case, if we change the API at the level of Business Logic, the JSON will also change automatically. All the rest frameworks will happily convert the object for us, and the chances are that the user will get another data.
Another example
Lets say, your BL layer provides a Person entity that looks like this:
class Person {
String firstName;
String lastName;
List<Language> languages;
}
class Language {
...
}
Now, let's say you have a UI that consumes your REST service that provides a list of Persons upon request. What if there are 2 different pages in UI. One that shows only the Persons (in this case it doesn't make sense to provide a list of language, spoken by a person).
In the second page however you want to get the full information.
So, you'll end up exposing 2 web services or complicating the existing one by some parameters (the more params like this you have, the less it resembles the rest :) )
Maybe separation would help a little here? I don't know.
Bottom line.
I would say that as long as you can live without such a separation - do it. It can work even for quite big projects. And of course it can work for small or middle-sized projects.
If you find yourself struggling around fixes and you feel like such a separation would solve the issues - do the separation.
Hope this helps to understand the implications and chose what works for you
This is basically asking the same question as in How to handle views in a multilayer-application. However, that post didn't receive much feedback.
Here's the problem: we have built a 3-tiered web application with the following tiers:
-Data Access (using repositories)
-Service
-UI (MVC 3)
DTO's are passed between the UI (Controller) Layer and Service Layer. Heavier Domain Models, containing a lot of domain-level logic, are passed between the Service and Data Access Layers. Everything is decoupled using IOC and the app follows SOLID principals (or tries too) --a big happy decoupled family!
Currently the DTO->Domain Model and Domain Model->DTO conversion happens all in the service layer.
So, finally to my question:
We are going to need to start displaying more complex read-only subsets of information, (i.e. summary views joining multiple entities doing rollup totals, etc). So what is the best practice for representing this type of read-only data in the n-tiered system? Having to map read-only Domain Model types to DTO types in this case doesn't make sense to me. In most cases, there would be no difference between the 2 types anyway. My thought would be to "break" the layering boundaries for these read-only types, having the Data Access Layer serve up the DTO's directly and pass those through to the Service Layer and on to the UI.
Can anyone point me in the right direction?
Much Thanks!
Your thought on breaking the layering for reading and then displaying values make sense completely. After all, the architecture/design of the system should help you and not the other way around.
Displaying report-like data to the user should be queried simply from the database and pushed to the view; no domain/dto conversion, especially if you're in a web app. You will save yourself a lot of trouble by doing this.
Personally, I had some attempts to go through these mappings just to display some read only data and it worked poorly; the performance, the unnecessary mappings, the odd things I had to do just to display some kind of report-like views. In this case, you'll likely have your domain model and a read model. You can look up CQRS pattern, it might guide you away from thinking that you want to use the same data model for both writes and reads.
So, to answer you question, I believe that in this case the best way would be to skip layering and read DTOs directly from the database through a thin layer.
I'm trying to introduce caching into an existing server application because the database is starting to become overloaded.
Like many server applications we have the concept of a data layer. This data layer has many different methods that return domain model objects. For example, we have an employee data access object with methods like:
findEmployeesForAccount(long accountId)
findEmployeesWorkingInDepartment(long accountId, long departmentId)
findEmployeesBySearch(long accountId, String search)
Each method queries the database and returns a list of Employee domain objects.
Obviously, we want to try and cache as much as possible to limit the number of queries hitting the database, but how would we go about doing that?
I see a couple possible solutions:
1) We create a cache for each method call. E.g. for findEmployeesForAccount we would add an entry with a key account-employees-accountId. For findEmployeesWorkingInDepartment we could add an entry with a key department-employees-accountId-departmentId and so on. The problem I see with this is when we add a new employee into the system, we need to ensure that we add it to every list where appropriate, which seems hard to maintain and bug-prone.
2) We create a more generic query for findEmployeesForAccount (with more joins and/or queries because more information will be required). For other methods, we use findEmployeesForAccount and remove entries from the list that don't fit the specified criteria.
I'm new to caching so I'm wondering what strategies people use to handle situations like this? Any advice and/or resources on this type of stuff would be greatly appreciated.
I've been struggling with the same question myself for a few weeks now... so consider this a half-answer at best. One bit of advice that has been working out well for me is to use the Decorator Pattern to implement the cache layer. For example, here is an article detailing this in C#:
http://stevesmithblog.com/blog/building-a-cachedrepository-via-strategy-pattern/
This allows you to literally "wrap" your existing data access methods without touching them. It also makes it very easy to swap out the cached version of your DAL for the direct access version at runtime quite easily (which can be useful for unit testing).
I'm still struggling to manage my cache keys, which seem to spiral out of control when there are numerous parameters involved. Inevitably, something ends up not being properly cleared from the cache and I have to resort to heavy-handed ClearAll() approaches that just wipe out everything. If you find a solution for cache key management, I would be interested, but I hope the decorator pattern layer approach is helpful.
question:
i'm building (or trying to build) a front end for an enterprise level web-app. the existing structure is driven by stored procedures (sql 2008 db). A stored proc is implemented as a class that ultimately handles the execution and the results are returned as an object.
i'm new to this game and would welcome an explanation about how my time would best be served...i read a repository pattern is a best practice, but all of the examples i read implement Entity Framework or Linq, etc..do i need an ORM? why or why not? i'd like to be able to have a maximum performance environment so that users can play with those result sets. thanks in advance
Well, I would suggest deciding on your use cases.
Some of the things that nHibernate / ORM's generally are not good for are:
Batch jobs.
Reporting
So if your work primarily involves either of those then you're best off not wasting your time, that being said there's nothing wrong with having multiple strategies... Building out a domain model is great for simplifying complex business rules, performance is generally very good too... Reporting and batch jobs can be built out separately, there's no reason why the different strategies can't co-exist... I woul however do my best to keep them decoupled...
So if you've got a big hairy business logic layer and it's littered with datasets / data access code and business logic IN your stored procedures then you will likely find it worth your while to invest in an ORM, but consider it a re-factoring step... IE you're improving existing code and making it testable before extending it...
In any case there's no one 'best' answer, the smartest thing I've done at previous companies has been to build new functionality (Test driven of course) in whichever data access pattern that seems to make sense to the functionality... Keep interfaces clean and decoupled... After doing that for awhile it usually becomes obvious which strategy / pattern is best suited for the application overall...
Good luck
Your question is somewhat unclear. Stored Procedures are SQL queries stored on the database which are used to interact with the data. However, it sounds like you already have an existing data access layer (DAL) which uses stored procedures and returns objects to you to play with. If this is the case, I would not throw away the entire data access layer and replace it with EF or any other ORM. Unless the existing DAL isn't working for you for either design or performance reasons, there's no reason to reinvent the wheel.
Currently I'm doing a project whose specifications are unclear - well who doesn't. I wonder what's the best development strategy to design a DB, that's going to be extended sooner or later with additional tables and relations. I want to include "changeability".
My main concern is that I want to apply design patterns (it's a university project) and I want to separate the constant factors from those, that change by choosing appropriate design patterns - in my case MVC and a set of sub-patterns at model level.
When it comes to the DB however, I may have to resdesign my model in my MVC approach, because my domain model at a later stage my require a different set of classes representing the DB tables. I use Hibernate as an abstraction layer between DB and application.
Would you start with a very minimal DB, just a few tables and relations? And what if I want an efficient DB, too? I wonder what strategies are applied in the real world. Stakeholder analysis for example isn't a sufficient planing solution when it comes to changing requirements. I think - at a DB level - my design pattern ends. So there's breach whose impact I'd like to minimize with a smart strategy.
In unclear situations I prefer a minimalistic DB design, supporting the needs known right now. My experience is that any effort to be clever, to model for future needs makes the model more complex. When the new needs arise, they are often in unforseen areas. The extra modeling for future needs doesn't fit the new needs, but rather makes the needed refactoring even harder.
As you already have chosen Hibernate to be able to decouple the DB design and the OO model, I think that sticking with an as simple DB as possible is a good choice.
What you describe is typical for almost every project. There are a few things you can do however.
Try to isolate the concepts (not their realizations) of your problem domain. Remember: Extending a data model is almost always easy (add a new table, a new column etc.) but changing your data model is hard and requires data migration.
I advocate using an Agile development process: Implement only what you need right now, but make sure you understand the complete problem before modeling it.
Another thing you should check before starting to hack away your code is wether your chosen infrastructure is appropriate. Using a relational database when you want to change your schema's very often is usually a bad match. Document databases are schema-less and hence more flexible. I think you should evaluate wether using a relational database is really appropriate for you application.
"Currently I'm doing a project whose specifications are unclear"
Given the 'database' tag, I assume you are asking this question in a database context.
Remember that a database is a set of ASSERTIONS OF FACT (capitalization intended).
If it is unclear what kind of "assertions of fact" your user wants to be registered by the database, then you simply cannot define (the structure of) your database.
And you will be helping both yourself and your user by first trying to clear out everything that is unclear.
In a simple answer: BE MINIMALISTIC.
Try to figure out the main entities. Don´t worry about the properties, you will fill them later. Then, create the relations between the entities. Create a test application using wour favorite ORM (Hibernate?), build some unit tests, and voilà, you have your minimal DB operational. :)
No project begins with requirements entirely known and fixed for all time. Use an agile, iterative approach to the database design so you that you can accommodate change during development.
All database designs are extensible and subject to change during their lifetime. Don't try to avoid change. Just make sure you have the right people and processes in place to manage change effectively when it happens.