I wrote my first Playground Backend with Quarkus, unfortunetly the persist perfomance of many objects is in my code very bad.
What I do is
List<CustomerDto> customerData; // Size 100.000
customerData.parallel().forEach..
* map to Entity
* CustomerEntity.persistFlush
This takes a lot of time. And looks like very bad code.
What is a good way to store a lot of List<CustomerDto> customerData;
Related
I wanna ask about if we write query in controller instead model. does it have any effect? such as load data become slower or others?
the data that i want to use is over 1000.
if not, what makes the load of data slow in web.
like the ajax request is needed 4-5 sec, and some is until 1 minutes
There is virtually no difference between running a query in a controller or a model except for the small (negligible) amount of overhead a model adds (if it were me I wouldn't worry about it). If you want to adhere to the MVC design pattern, database queries are typically done in the model, not the controller. But this is more of a stylistic thing in CodeIgniter as it is not strictly enforced as it is in other frameworks that use ORM.
With that being said, if you are experiencing long execution times I would recommend not getting all the data at once, and instead using load more, pagination, datatables, or a similar system to reduce the amount of records selected at once to something more manageable depending on your acceptable execution/response times.
You can test what works in your situation best by setting benchmark points: https://www.codeigniter.com/user_guide/libraries/benchmark.html
I have been working with CodeIgniter for some few years and what I have realized is to properly do your queries in models.
The idea of MVC is about code separation where the M(Model) handles or does the heavy liftings of the application which mostly relates to databases, the V(View) does the presentation that is where users of the system get to interact with the application and the C(Controller) acts as a messenger or intermediary between the Model and the View.
This releases the controller from lots of processes so that it doesn't have to query from the database, try to upload lots of data before then showing it on a view.
I always use the (I call it the little magic) {elapsed_time} and {memory_usage}to check how my application work per whatever logic I implement, you can give it a try.
I think this helps.
I'm working on a SPA utilising BreezeJS and AngularJS, handling lots of entities (one of the types has ~60k entities). This is not an ordinary website, it's made for a specific purpose.
Most of the time the entities are shown in sortable, paged lists.
The above mentioned mass of entities gets cached and queried in a worker thread, so that the UI doesn't get blocked. We want to keep client-server communication to a minimum after application initialisation, hence the need for caching lots of data.
The results from the entityManager in the worker thread get imported to the entityManager in the UI thread and further processing follows. This all works fine, my only problem is that performing an orderBy on such a huge dataset takes too long for Breeze to complete (2.5-3.5 secs) without indexes.
This means that showing the next page is unacceptably slow if ordering is in place.
Is there a way to equip the Breeze cache with indexes and get Breeze to use them somehow?
If not, is this feature planned to be implemented?
I could of course craft indexes for this particular model and amend the query to be run against the cache, but it wouldn't be easy to maintain, considering the dataset is allowed to change.
Breeze cache does not have indexes (indices?) today ... not even for the primary key. It would require a substantial increase in code base size and complexity to support them and we haven't felt that would be worthwhile for the workloads we usually see.
60K items clearly changes that equation.
I think indexing would be a cool optional module, a plugin of some sort. I don't think it would be that hard to maintain, given that the EntityManager raises events when anything changes in the cache. If you feel like taking it on as a community contribution ... perhaps to Breeze Labs, I'd be happy to advise and help a little.
For some time I have been working with ZK and now I have an application hosted in the cloud but this does not meet my performance expectations. My application uses ZK for the view, Hibernate for the interaction with the MySQL database and Spring for the management of beans related to database interaction.
Example scenario:
At initialization a list is retrieved from the database using hibernate (we are talking about max 200 objects, max 6 standard attributes). This takes some time, but is acceptable.
I add or delete objects from the list I use #NotifyChange("list") to tell the view that the list has changed.
The list is updated but this takes long, in my opinion (~2 seconds)
Step 3 takes about as long as step 1 which I don't understand, the Spring managed objects for database interaction have yet been initialized. Another thing to note is that I don't reload the entire list from the database when adding or deleting an object from the list; A local copy is kept of the objects in the spring managed bean #Autowired to my ViewModels. (Yes, potentially out of sync with the database, but in my case not much of an issue) But this eliminates hibernate/spring/database as the bottleneck, right?.
Also I ran this on the localhost, and of course that is faster but still not instant as I would expect/want.
Now I thought that it might have to do something with the creation and deletion of objects in java related to lists. I did research on how the ZK binder works and how objects are updated but I can't find anything about how the binder handles updates on lists.
Does anyone know how the ZK binder updates a list in the view when in the viewmodel the method is annotated with #NotifyChange("listA") ? Does it actually throw away the old list and sets the whole new list, or does it somehow have a merge on differences?
Also comments on what else could be the bottleneck are very welcome!
Thanks in advance,
Robert
I assume you use Listbox to represent your list.
First you need to know, that zk uses the same data
structure for MVVM that it uses for MVC. This means
that it uses ListModel and the performance tips
are usefull for MVVM as well.
Of course zk can't implement the standart model
in a way, that it fits everybodys needs.
So what you have to do is implement your own
model. Usually the best is to extend from ListModelList.
This way you can smart update the list.
i work alot with ZK but i have never use #NotifyChange, when i want to reload the data of a grid or listbox I always render or load it all again. For me it works fine and fast in less than 1 second. Here is an example of my binding:
>
And yes, when the data changes i recall my method, load it again and works really fine.
#Listen("onCreate = #listboxSolicitudes")
public void loadList() {
List<ConsultasSolicitudes> listaConsulta = null;
listaConsulta = ConsultasSeguimientoDeSolicitudesDAO.obtenerSolicitudesAsignadas((Long) cbResponsable.getSelectedItem().getValue());
ListModelList<ConsultasSolicitudes> solModel = new ListModelList<ConsultasSolicitudes>(listaConsulta);
listboxSolicitudesAsignadas.setModel(solModel);
}
Maybe this can help you, but something must be wrong if it´s taking that long.
I have a question regarding the performance of RequestFactory and GWT. I have a Domain Entity with 8 fields that returns around 1000 EntityProxies. The time between the request fires and it responds is around 20 seconds. I do the same but returning 10 EntityProxies and the time is 17 seconds, almost the same.
Is this because I'm working in development mode, or when I release the code to the web the time will be the same?
Is there any way to improve the performance? , I'm only reading data so perhaps something that only read and doesn't writes may be the solution?
I read this post with something similar to my problem:
GWT Requestfactory performance suggestions
Thanks a lot.
PD: I read somewhere that one solution could be to create an xml in the server, send it to the client and recreate the object there, I don't want to do this because it would really change the design of my app.
Thank you all for the help, I realize now that perhaps using Request Factory to retrieve thousands of records was a mistake.
I initially used a Locator to override isLive() and Find() methods according to this post:
gwt-requestfactory-performance-suggestions
The response time was reduced to about 13 seconds, but it is still too high.
But I solved it easily. Instead of returning 1000+ Entities , I created a new database table which each field has all the same field records (1000+) concatenated by a separator (each db field has a length of about 10000 ) and I only have one record in the table with around 8 fields.
Something like this:
Field1 | Field2 | Field3
Field1val;Field1val;Field1val;....... | Field2val;Field2val;Field2val;...... | Field3val;Field3val;Field3val;......
I return that One record through RequestFactory to my client and it reduced the speed a lot!, around 1 second. I parse this large String in the client and the duration of that is about 500ms. So instead of wasting around 20 seconds now it takes around 1-2 seconds to accomplish the same.
By the way I am only displaying information, it is not necessary to Insert, Delete or Update records so this solution works for me.
Thought I could share this solution.
Performance Profiling and Fixing issues in GWT is tricky. Avoid all profiling in GWT Hosted mode. They do not mean anything useful.
You should profile only in WEB mode.
GWT RequestFactory by design is slower than GWT RPC and GWT JSON etc. This is a trade off w.r.t GWT RF ability to calculate delta and send only small amount information to server on save.
You should recheck you application design to avoid loading 1000's of proxies. RF is mean for "Form" like applications. The only reason you might need 1000's of proxies is for a Grid display. You probably can use paginated async grid in that scenario.
You should profile your app in order to find out how much time is spent on following steps:
Entities retrieved from the database (server): This can be improved using second level cache and optimized queries
Entities serialized to JSON (server): There is a overhead here because RequestFactory and AutoBean respectively rely on reflections. You can try to only transmit the Entities that you are also going to display on the client. Another optimization which greatly reduces latency is to override the isLive method of your EntitiyLocator and to return true
HTTP request from server to client to tranmit the data (wire): You can think about using gzip compression to reduce the amount of data that has to be transferred (important if you send a lof of objects over the wire).
De-serialization on the client (client): This should be quite fast. There was a benchmark that showed that AutoBean serialization was one of the fastest ways to serialize JSON. Again this will benefit from not sending the whole object graph over the wire.
One way to improve performance is to use caching. You can use HTML5 localstorage to cache data on the client. This applies specifically to data that doesn't change often.
My C# 3.5 application uses SQL Server 2008 R2, NHibernate and CastleProject ActiveRecord. The application imports emails to database along with their attachments. Saving of emails and attachments is performed by 50 emails in new session and transaction scope to make sure they are not stored in memory (there can be 100K of emails in some mailbox).
Initially emails are saved very quickly. However, closer to 20K emails performance degrades dramatically. Using dotTrace I got the following picture:
Obviously, when I save an attachment, NHibernate tries to see if it really should save it and probably compares with another attachments in the session. To do so, it compares them byte by byte what takes almost 500 seconds (for the snapshot on the picture) and 600M enumerator operations.
All this looks crazy, especially when I know for sure that SaveAndFlush indeed should save the attachment without any checks: I know for sure that it is new and should be saved.
However, I cannot figure out, how to instruct NHibernate to avoid this check (IsUpdateNecessary). Please advise.
P.S. I am not sure but it might appear that degradation of performance closer to 20K is not connected with having some older mails in memory: I noticed that in mailbox I am working with, larger emails are stored later than smaller so the problem may be only in attachments comparison.
Update:
Looks like I need something like StatelessSessionScope, but there is no documentation on it even at CastleProject site! If I do something like
using (TransactionScope txScope = new TransactionScope())
using (StatelessSessionScope scope = new StatelessSessionScope())
{
mail.Save();
}
it fails with exception that Save is not supported by stateless session. I am supposed to insert objects into session, but I do not have any Session (only SessionScope, which adds up to SessionScope only single OpenSession method which accepts strange paramenters).
May be I missed it in that long text, but are you using stateless session for importing data? Using that prevents a lot of checks and also bypasses first level cache, thus using minimal resources.
Looks like I've found an easy solution: for my class Attachment, causing biggest performance penalty, I overridden the following method:
protected override int[] FindDirty(
object id,
System.Collections.IDictionary previousState,
System.Collections.IDictionary currentState, NHibernate.Type.IType[] types)
{
return new int[0];
}
Thus, dirty check always consider it dirty and does not do that crazy per-byte comparison.