I am currently using data provider for passing json data loaded externally. this is a large data with olhcv data per minute for 1 month. I am experiencing slowness while running it on localhost. It is taking 8 seconds to render full data.
Recently I saw the dataLoader feature of amStockChart. But I am experiencing issues with replacing my dataprovider with dataLoader as I have customized too many things. Can I please know if its worth the effort of moving to dataLoader from dataProvider?
Related
The problem is that we have encountered a scenario where we want to directly read/load all data of an es index file into the memory, rather than using the scroll polling query method of the es api.
The reason we do this is that， by this way, it will not slowdown the performance of our es instance which using by other applications. This is very important for us.
After loading the index data into the memory, we can parse and transfer them to other big data applications, such as the kafka, flink directly.
If you have more than 10000 records to read, the best way is by using the Scroll API. You can all the data using this API and store everything in memory if it fits.
I'm curious though about
it will not slowdown the performance of our es instance which using by other applications.
Why do you think that using the standard API will be slow?
I am to generate a report from the database with 1000's of records. This report is to be generated on monthly basis and at times the user might want to get a report spanning like 3 months. Already as per the current records, a month's data set can reach to like 5000.
I am currently using vue-excel to which makes an api call to laravel api and there api returns the resource which is now exported by vue-excel. The resource does not only return the model data but there are related data sets I also need to fetch.
This for smaller data sets works fine that is when I am fetching like 3000 records but for anything larger than this, the server times out.
I have also tried to use laravel excel with the query concern actually timed them and both take same amount of time because laravel excel was also mapping to get me the relations.
So basically, my question is: is there some better way to do this so as get this data faster and avoid the timeouts
just put this on start of the function
ini_set(max_execution_time, 84000); //84000 is in seconds
this will override the laravel inbuild script runtime max value.
I have a web app that displays the analysis data in browser with elasticsearch as backend data store.
Everything was cool as elasticsearch was handling about 1TB data and search queries were blazing fast.
Then came the decision to add data from all services into the app, close to a peta byte, and we switched to bigquery.[yes, we abandoned the elasticsearch and started querying bigquery directly ].
Now users of my app are complaining that their queries are slow, they are taking seconds (4~10~15), which used to display under a second before.
Naturally the huge amount of data here is to be blamed but I am wondering if there is a way to bring back elasticsearch into the game and make elasticsearch and bigquery play together nicely so that I can get the petaytes of storage from bigquery but still retain the lightspeed search of elasticsearch.
I am sure I am not the first one to face this issue rather I believe I am bit late to the bigquery party so I should be able to reap the benefits of delayed entry by getting all the problems already solved.
Thanks in advance if you can point me to the right direction.
This is a common pattern I see deployed by customers:
Use Elasticsearch to display results from the latest day/week - whatever fits within Elasticsearch's RAM.
Use BigQuery for everything else.
In this way your users will get sub-second results for 90% of their queries, and they will also be able to go wherever they want to go if Elasticsearch can't find an answer within its resources.
I'm not sure what are your users interfaces for getting data - but that's where this logic would need to be deployed.
(of course, expect improvements in the connections and speed as tech progresses)
I'm calling ExecuteMultipleRequest to insert 25 records of a custom entity at a time. Each batch is taking roughly 20 seconds.
Some info about the custom entity:
I did not create its schema and can't have it changed;
It has 124 attributes (colums);
On each CreateRequest the entity has 6 attribute values filled: 2 Lookup and 4 Money. ExecuteMultipleRequest is being called from a middleware component in a corporate network, which connects to the CRM in the cloud. The CRM instance used is a sandbox, so it may have some restrictions (CPU/bandwidth/IO/etc), that I'm not aware of.
I can issue concurrent requests, but considering I can only have 2 concurrent requests per organization (https://msdn.microsoft.com/en-au/library/jj863631.aspx#limitations), it would only cut the time in half. That is still not a viable time.
For each new custom CRM process created I need to load at most 5000 entity records, in less than 10 minutes.
What can I do to improve the performance of this load? Where should I be looking at?
Would a DataImport (https://msdn.microsoft.com/en-us/library/hh547396.aspx) be faster than ExecuteMultipleRequest?
Only really got suggestions for this, you would probably have to experiment and investigate to see what works for you.
Can you run your middleware application in a physical location closer to your CRM Online site?
ExecuteMultipleRequest supports much larger batch sizes, up to 1000.
Have you compared to just using a single execute request.
Do you have lots of processes (workflows, plugins) that occur in CRM when the data import is running? This can have a big performance impact. Perhaps these can be disabled during data import. E.g. you could pre-process the data before import so a plugin wouldnt need to be executed.
The concurrent requests limitation only applied to ExecuteMultipleRequest, have you tried running lots of parallel single execute requests?
I have a data set of 11 MB. It's slow to load it every time the document is loaded.
d3.csv("https://s3.amazonaws.com/vidaio/QHP_Individual_Medical_Landscape.csv", function(data) {
// drawing code...
});
I know that crossfilter can be used to slice-and-dice the data once it's loaded in browser. But before that, dataset is big. I only use an aggregation of the data. It seems like I should pre-process the data on server before sending it to client. Maybe, use crossfilter on server side. Any suggestion on how to handle/process large dataset for d3?
Is your data dynamic? If it's not, then you can certainly aggregate it and store the result on your server. The aggregation would only be required once. Even if the data is dynamic, if the changes are infrequent then you could benefit from aggregating only when the data changes and caching that result. If you have highly dynamic data such that you'll have to aggregate it fresh with every page load, then doing it on the server vs. the client could depend on how many simultaneous users you expect. A lot of simultaneous users might bring your server to its knees. OTOH, if you have a small number of users, then your server probably (possibly?) has more horsepower than your users' browsers, in which case it will be able to perform the aggregation faster than the browser. Also keep in mind the bandwidth cost of sending 11 MB to your users. Might not be a big deal ... unless they're loading the page a lot and doing it on mobile devices.
Try simplifying the data (also suggested in the comment from Stephen Thomas)
Try pre-parsing the data into json. This will likely result in a larger file (more network time) but have less parsing overhead (lower client cpu). If your problem is the parsing this could save time
Break the data up by some kind of sharding key, such as year. Limit the to that shard and then load up the other data files on demand as needed
Break up the data by time, but show everything in the UI. load the charts on the default view (such as most recent timeframe) but then asynchronously add the additional files as they arrive (or when they all arrive)
How about server side (gZip) compression. should be a few kb after compressing and browser will de-compress on the background.