I would like to write an application based on core data, but I don't know if it's worth, there will be several tables of a few or several thousand data.
What is the situation at the front?
A few thousand records isn't that many in the grand scheme of things, and so is likely to be fine. Though without knowing exactly what you want to do with the data or what platform you're running it on, it's difficult to be sure.
The important thing to remember about Core Data is that it isn't primarily a persistence API i.e. one primarily concerned with getting data onto and off disk like SQL. It is primarily an API for creating the entire model layer for a Model-View-Controller (MVC) design app. As such, it provides a complete data management solution from persistence to object-graph management to integration with the UI.
Core Data is such a comprehensive solution that in Cocoa using bindings, it is possible to create entire apps without writing any custom code.
Any performance you might hypothetically lose in persistence operations with Core Data is almost always overshadowed by the performance gains of the object-graph management and UI integration.
Related
For example,
You have an IT estate where a mix of batch and real-time data sources exists from multiple systems, e.g. ERP, Project management, asset, website, monitoring etc.
The aim is to integrate the datasources into a cloud environment (agnostic).
There is a need for reporting and analytics on combinations of all data sources.
Inevitably, some source systems are not capable of streaming, hence batch loading is required.
Potential use-cases for performing functionality/changes/updates based on the ingested data.
Given a steer for creating a future-proofed platform, architecturally, how would you look to design it?
It's a very open-end question, but there are some good principles you can adopt to help direct you in the right direction:
Avoid point-to-point integration, and get everything going through a few common points - ideally one. Using an API Gateway can be a good place to start, the big players (Azure, AWS, GCP) all have their own options, plus there's lots of decent independent ones like Tyk or Kong.
Batches and event-streams are totally different, but even then you can still potentially route them all through the gateway so that you get the centralised observability (reporting, analytics, alerting, etc).
Use standards-based API specifications where possible. A good REST based API, based off a proper resource model is a non-trivial undertaking, not sure if it fits with what you are doing if you are dealing with lots of disparate legacy integration. If you are going to adopt REST, use OpenAPI to specify the API's. Using this standard not only makes it easier for consumers, but also helps you with better tooling as many design, build and test tools support OpenAPI. There's also AsyncAPI for event/async API's
Do some architecture. Moving sh*t to cloud doesn't remove the sh*t - it just moves it to the cloud. Don't recreate old problems in a new place.
Work out the logical components in your new solution: what does each of them do (what's it's reason to exist)? Don't forget ancillary components like API catalogues, etc.
Think about layering the integration (usually depending on how they will be consumed and what role they need to play, e.g. system interface, orchestration, experience APIs, etc).
Want to handle data in a consistent way regardless of source (your 'agnostic' comment)? You'll need to think through how data is ingested and processed. This might lead you into more data / ETL centric considerations rather than integration ones.
Co-design. Is the integration mainly data coming in or going out? Is the integration with 3rd parties or strictly internal?
If you are designing for external / 3rd party consumers then a co-design process is advised, since you're essentially designing the API for them.
If the API's are for internal use, consider designing them for external use so that when/if you decide to do that later it's not so hard.
Taker a step back:
Continually ask yourselves "what problem are we trying to solve?". Usually, a technology initiate is successful if there's a well understood reason for doing it, which has solid buy-in from the business (non-IT).
Who wants the reporting, and why - what problem are they trying to solve?
As you mentioned its an IT estate aka enterprise level solution mix of batch and real time so first you have to identify what is end goal of this migration. You can think of refactoring applications. If you are trying to make it event driven then assess the refactoring efforts and cost. Separation of responsibility is the key factor for refactoring and migration.
If you are thinking about future proofing your solution then consider Cloud for storing and processing your data. Not necessary it will be cheap but mix of Cloud and on-prem could be a way. There are services available by cloud providers to move your data in minimal cost. Cloud native solutions are there for performing analysis on your data. Database migration service in AWS or Azure can move data and then capture on-going changes. So you can keep using on-prem db & apps and perform analysis for reporting on cloud. It will ease out load on your transactional DB. Most data sync from on-prem to cloud is near real time.
MVC sets up clear distinction between Model, View and Controller.
For the model, now adays, web frameworks provides ability to map the model directly to database entities (ORM), which, IMHO, end up causing performance issues at runtime due to direct database I/O.
The thing is, if that's really the case, why model ORM is so pupular and every web frameworks want to support it either organically or not.
To a web site has huge amount of traffic, it definitely won't work. But what's the work around? Connect directly to database is definitely not a wise solution here.
What's your question?
Is it a good idea to use direct db access from webpages?
A: No.
Is it a good idea to use ORM's?
A: Debatable : See How can I design a Java web application without an ORM and without embedded SQL
Is it a good idea to use MVC model?
A: Yes - it has nothing to do with "Direct" database access - it's about separating your application logic from your model and your display. (Put simply).
And the rationale for not putting database logic inside webpages has nothing to do with performance - it's about security/maintainability etc etc. Calling a usp from a webpage is likely to be MORE performant than using an ORM, but it's bad because the performance gain is negligible, and the cons are significant.
As to workaround: if you mean how do you hook up a database to a web application...?
The simplest way is to use something like Entity Frameworks or Linq-Sql with your Model - there are plenty of examples of this in tutorials on the web.
A better method IMO, is to have a separate Services layer (which may be WCF based), and have all the database access inside that, with DTO's transferring the data to your Web Application which has it's own ViewModel.
Mvc is not about orm but about separation of display logics and business logics. There is no reason your exposed model needs to be identical to you database model and many reasons to ensure that the exposed model closely matches what is to be displayed.
The other part of the solution to scale well would be to implement caching in the control and be able to distribute load on sevaral instances.
I think #BonyT has given a good answer, (and I've voted for it :) ), I'd just add that:
"web frameworks provide the ability to map the model directly to database entities (ORM), which, IMHO, ends up causing performance issues at runtime due to direct database I/O"
Even if this is true, using an ORM can solve a lot of problems with a model being easy to update and translate back and forth between a database. Solving a performance hit by buying extra web servers or cloud instances is much cheaper than having to buy extra developers or extra hours in development to solve things other people have already written ORMs to do for you.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I want to know your opinion about using data-aware components in projects. Which are the 'strength' and 'weak' points of developing applications(win32 and web), by using Delphi and data-aware components(from Delphi's standard suite or third-party)?
Using FireBird I've worked a lot with IBObjects, which are a mature suite of components and worked very well.
But there are also a lot of other RDBMS (MySQL, MSSQL, DB2, Oracle, SQLite, Nexus, Paradox, Interbase, FireBird etc). If you have developed big projects, on which you've used a lot data-aware components please answer with the database type and data-aware components suite name.
I'm also interested on DB2 (AS400). What components have you used with success, or which components are really a pain to work with?
I've found that using the data-aware components results in an application with no clear distinction between business and UI logic.
This is fine for small projects but as they grow larger the code becomes less and less maintainable.
All the various bits of event code (and their interactions) can become a real nightmare to understand!
Invariably in such cases I've ditched data-aware components and have switched to a (hand-coded) MVC design.
This does require a lot of up-front coding effort but results (IMHO) in a project that is maintainable, extensible and debuggable.
Having tried both data-aware and non data-aware style of Delphi applications I'm back in the data-aware component camp these days. It takes a bit of work and discipline to correctly layer the code but it's still quicker than doing everything by hand using non data-aware controls.
A few of my tips for data-aware component usage are
Don't just rewrite FishFact on a larger scale. Put some thought into your design.
Don't use a TDataModule, use many TDataModules each responsible for just a small aspect of your applications data.
TDatasets belong on TDataModules, while TDataSources belong on TForms (unless used for master/detail relationships).
Use in-memory datasets such as the DataSnap TClientDataSet.
Your ClientDataSets don't have to mirror your database tables exactly. DataSnap allows you massage your data structures in memory so you can produce datasets tailored for specific purposes. Specifically you can do things like
Joining two or more tables into the one editable dataset
Denormalizing master detail table structures, can simplify your UI code sometimes.
Create in-memory only fields (like calculated fields but you can write to them also)
TClientDataSet nested tables are useful but not the only way to express master detail relationships. Sometimes it's better to do it the old way with two independent TClientDataSets joined through a TDataSource.
Take a look at ORM solutions.
It's a nice approach with multi-tier architecture. See ORM for DELPHI win32
Data aware controls are great, but you have to make sure you get your business code in a separate layer.
That is not difficult, but you need to be aware on how you can do that.
One approach is to have your DataSet components in a DataModule (or other non visual container).
Another handy trick is to use a TClientDataSet to do the UI entry, and use that as an intermediate buffer between the UI and the business layer. The business layer then uses TDataSet components specific to your data layer.
--jeroen
Delphi data-aware components are not depended on the back-end database engine you are using, so using Firebird or MS SQL Server or Oracle or others doesn't matter to your data-aware components. They only know the datasource component assigned to them and do all their DB related stuff via that.
For me, if something can be done with data-aware components in a nice way, I will use them. These are usually small projects which should be done in a short-time. In bigger projects, I might totally rule out data-aware components or use them in forms that are merely used for data presentation and do not receive user input. When it comes to receiving user input, I might use non-data-aware components because I have more flexibility in controlling them and validate the input. Of course data-ware components can be still useful in such scenarios too. You still can validate user input in dataset events like OnBeforePost. Also if you are using a multi-tier design, and your client app represents data presenter layer, your input validation is done in the middle-tier so you can receive input using data-aware components in the client app, and send them to the middle-tier for validation and further processing.
Data-aware components are usful from a RAD and prototyping perspective, especially when you're designing reports or grids that are based on data. i.e. you can tinker at design time.
So I use them like that. But when it comes time to transform it into shipping code, I almost always sever the connections, remove the SQL from the queries, and do everything in code. It's much more predictable and maintainable that way, especially in a multi-developer environment with version control. When the SQL is embedded in the form somewhere, it's a big pain to try to figure out where the SQL actually resides. And it's especially bad to have SQL in two places, and then have to figure out which is in effect.
You can use Unidac which supports many database servers, including Firebird (that i use) and has very nice features.
Coupled with Remobject SDK you will have a nice combination of n-tier architecture and database abstraction.
I've found no clear answer so far, but maybe I've searched the wrong way.
My Question is, can Core Data to be used as a Persitence Storage for a Server Project? Where are Core Data's Limits, how much Data can be handled with Core Data and SQLite? SQLite should handle a lot of Data very well according to their website. I know of a properitary Java Persitence Manager with an Oracle DB as Storage that handles Millions of Entries and 3000 Clients without Problems. For my own Project I wonder if I can use Core Data on the Server Side for User Mangament and intern microblogging, texting with up to 5000 clients. Will it handle such big amounts of Data or do I have to manage something like that myself? Does anyone happend to have experience with huge amounts if Data and Core Data?
Thank you
twickl
I wouldn't advise using Core Data for a server side project. Core Data was designed to handle the data of individual, object-oriented applications therefore it lacks many of the common features of dedicated server software such as easily handling multiple simultaneous accesses.
Really, the only circumstance where I would advise using it is when the server side logic is very complex and the number of users small. For example, if you wanted to write an in house web app and have almost all the logic on the server, then Core Data might serve well.
Apple used to have WebObjects which was a package to manage servers using an object-oriented DB much like Core Data. (Core Data was inspired by a component of WebObjects called Enterprise Objects.) However, IIRC Apple no longer supports WebObjects for external use.
Your better off using one of the many dedicated server packages out there than trying to roll your own.
I have no experience using Core Data in the manner you describe, but my understanding of the architecture leads me to believe that it could be used, depending on how you plan to query and manipulate the data.
Core Data is very good at maintaining an object graph and using faults to bring parts into memory as needed. In that manner, it could be good on a server for reducing memory requirements even with a large data set.
Core Data is not very good at manipulating collections of objects without loading them into memory, making a change, and writing them back out to disk. Brent Simmons wrote a blog post about this, where he decide to stop using Core Data for some of his RSS reader's model objects because an operation like "mark all as read" didn't scale. While you would like to be able to say something like UPDATE articles SET status = 'read', Core Data must load each article, set its status property, then write it back to disk.
This isn't because Apple engineers are stupid, but because the query layer can't make assumptions about the storage layer (you could be using XML instead of SQLite) and it also must take into account cascading changes and the fact that some article objects may already be loaded into memory and will need to be updated there.
Note that you can also write your own storage providers for Core Data, see Aaron Hillegass's BNRPersistence project. So if Core Data was "mostly good" you might be able to improve on it for your application.
So, a possible answer to your question is that Core Data may be appropriate to your application, as long as you do not need to rely on batch updates to large number of objects. In general, no algorithm or data structure is appropriate for every scenario. Engineering is about wisely choosing between trade-offs. You won't find anything that works well for many clients in every case. It always matters what you are doing.
Recently I had a project in which I had to get some data from particular software system to a portlet. The software used a database, and I spent a fair bit of time modeling the data I wanted and then creating a web service so that my portlet could grab the information.
Then it suddenly struck me that I was wasting my time. I grabbed BIRT, tossed it into a portlet, and then just wrote some reports that directly grabbed the necessary data from the database. I was done in an afternoon.
I understand that reporting is a one way street, but this got me thinking. Reporting tools can be very effective for creating reports (duh) from your actual data, but when you're doing this you're bypassing your model which except in simple cases is not a direct representation of your data as it exists in your database.
If you're writing a data-intensive application and require the ability to perform non-trivial reporting, do you bypass your application and use something like BIRT or Crystal Reports? How do you manage these tools as part of your overall process? Do you consider the reports you write as being part of your application and treat them as such? A report is a view and a model and a controller (if you will) all in one big mess, how do you deal with and interpret and plan for that?
Revised question: it's possible and even common that a report will perform some business calculations that in a perfect world you would like to have contained in your application. This can lead to a mismatch of information given back to the user. On the other hand, reporting tools make it so easy to gather and display information that it's hard to take a purist's approach and do everything from within the application. Are there any good techniques for ensuring that the data in your reports matches the data that you might be showing in the regular GUI?
I see reporting as simply another view on the data, not a view/model/controller in one (well, maybe a view and controller in one).
We have our reports (built in sql 2008 reporting services) consume a service in our application layer to get data (keeping with our standard, that data access is in a repository). These functions could do a simple query or handle very complex processing that would be a nightmare in your reporting evironment or a stored procedure. In practice, we find this takes no longer than coding up some one-off stored procedure that will, as your system grows and grows, become a nightmare to maintain.
Treating reporting as simply a one-off or not integrating into your application design is a huge mistake.
Reporting is crucial. Reporting is mostly crucial to share values collected in one system to external users, e.g. users not directly using the system (eg management for sales figures). So reporting is a lot more than just displaying facts and figures and is something central to almost every system that drives a commercial.
At least the more advanced systems allow you to enhance them: with your own reusable "controls". Even a way back can be implemented - if you just use the correct plugins. Once I wrote a system to send emails out of a report, because the system did not allow for change. It worked - though it was not meant to be used that way ;)
Reports make a good part of the application, and you gain a lot freedom if you make reports changeable for your customers. Sometimes you come up with more possibilities than you thought of when you built the system in the first place.
So yes, for me reporting is part of the system.
Reports are part of your app but because they are generally something a user will have strong ideas about than, say, your data capture UI, I'd sacrifice purity for convenience/speed of delivery and get back to "real" coding... :-)
As soon as you've done a report, users want another one or change the colour or optional grouping or more filtering or... something that takes you away from whizzier stuff... so I don't bust a gut maintaining purity.
This is a fine line indeed. You don't want to spend too much time building reports (that users want you to change all the time anyway) but you don't want to duplicate logic by putting business logic into your reports! With our reporting products at Data Dynamimcs I think we have reached a happy medium between these two tradeoffs.
By using the ObjectDataProvider (see links below for more info) you can bind the report directly to business objects (plain old objects) so you don't have to bypass your business layer for getting data. At the same time we provide a way to reference and use functions from other libraries in your report. This way if you have some code configured already to do some business logic calculations you can reuse those functions directly within your report. You can see an example of this in the links below too.
Binding to Objects for your Data (see "Object Provider" section): http://www.datadynamics.com/Help/ddReports/ddrconDataSetAndObjectDataSource.html
Adding Custom Code to your reports Walkthrough: http://www.datadynamics.com/Help/ddReports/ddrwlkCustomCode.html
Using Custom Assemblies (referencing shared libraries/dlls from your report): http://www.datadynamics.com/Help/ddReports/ddrconCustomCode.html, and http://www.datadynamics.com/Help/ddReports/ddrtskCreatingAnInstanceMethod.html
Scott Willeke
Data Dynamics / GrapeCity
The way I've always worked with reports is to consider part reports as part of the code-base, and stored in the source along with the application. In some contexts, reports are more important than the application, in that management makes business decisions off of report data, having the wrong information can cause them to cancel a product line, cancel a campaign, or fire a sales person. Obviously, this depends highly on your management and your application.
Regarding keeping your model consistent, this is a bit trickier question. One way to ensure consistent model between reports and your application is to use stored procedures (or views) to retrieve data, depending on your application's architecture.