I have a large dataset to query and display in website on an array.
I made a pagination system with a scroll but i can only display a maximum of 100 items at a time so i'm facing issue when i want to display data of page 200 and more because i have to scroll until them and it take too long.
I have check other parts of my code and i didn't find other perf issue, is just the scroll queries which make my api call too long. I tried setting the request size from 100 to 10000 but it doesn't change anything.
I don't think sliced scroll can be a solution or then I didn't understand the functionality.
I'm desperately searching a way to skip the scroll queries before datas that i'm searching even it's not a precise method.
Hoping someone has a solution or at least a clue.
Edit:
More details about what i'm trying to achieve.
I log some actions of my users like calls in Elasticsearch indexes. They do millions of actions per month so Elasticsearch seems like a good option to store them knowing that i don't have to update them after they are stored .
I'm creating a page where my users can search for actions they've performed, but they're doing the "query" themselves. I mean they can select the period and many other parameters, order them by many parameters, etc. The number of result can be 1 or 100,000 items, but I can't show 100,000 items on my page for UI reasons, so I have to manage a pagination and send only part of the result to the page.
I made a scroll query to do it for now with a size of 1000, and i scroll until i'm in the current page of my pagination. I tried to vary the size but it's not really concluent because I can't know the number of result before the query is made.
And the deeper my user go in the pagination, the longer the query take.
I could increase the index.max_result_window with an unreachable number (but I don't know what that implies) make a simple query with a from and a second scroll query for export case but I wonder if they are a way to skip some step in a scroll when i know i'm going to take 100 items after the 1 000 000th item ?
Edit: I watched how google design its pagination and i notice that if you want to go deep in search results you can't unless you go step by step. You can't go directly to the 500th page.
This is how I done mine
So I just redesign my pagination to do the same as Google and force my users to use more precise filters to get less result. Thank you #Val for getting me to ask the right questions :)
I've configured Grafana to use Elasticsearch as a data source and prepared a few panels.
My document in ES index contains only a few fields. It describes some system actions, respectively there are such fields as userId, action date, and others.
Now I faced with the issue that I can't calculate the amount of time left when the action happened. I mean if the action happened 2 days ago, I need to have the number 2 in my table. Pretty simple I supposed.
I couldn't find any metric or transformation that can allow me to do it.
Any suggestion, help, please.
Resolved my issue with scripted field.
In table visualization, I just picked any numeric field, selected Min metric, and added script like next:
Instant Currentdate = Instant.ofEpochMilli(new Date().getTime());
Instant Startdate = Instant.ofEpochMilli(doc['activity'].value.getMillis());
ChronoUnit.DAYS.between(Startdate, Currentdate);
As a result, it shows me the number I needed.
I don't find this solution the best, so in case anybody know some better way to resolve this issue, please add a comment.
I have an analysis that uses a filter in order to limit the results.
I put this analysis into a Dashboard together with a Dashboard prompt.
The Dashboard prompt consists of various columns and all of them are of type "Choice list" and all have the option "Limit values by" = "All prompts".
However, this is not working as I expected. I thought that the values appearing in he Choice list should be limited by the filters applied in the analysis but it seems that the dashboard prompt first shows all the possible values before having applied the analysis filter.
Is this correct?
If this is the way it works, the problem I am facing is that some of the values appearing in the Choice list of the prompt columns will result in NO DATA in the analysis.
Thanks for your help!
"I thought that the values appearing in he Choice list should be limited by the filters applied in the analysis"
Exactly the other way around. Prompts send their selected values to the filter that's in the analysis and hence cut off the data stream.
It is normal that a prompt retrieves selection values for which a certain point of view retrieves no data. Other way round would make no sense. Imagine you sell 5 products and one didn't sell at all in August. Do you want to remove August from your month prompt?
The interdependence of prompts that you mention is for example - limit regions to only regions of a selected country. Limit customers to only customers of a selected business unit etc.
What you write and expect is that the prompt should go over the effective data (facts) and only retrieve values for which data (facts) exists. As said above that's not the most logical thing to do in an analytical environment since one change of point of view can mean that data "exists" or "doesn't exist" - then you change the point of view and the situation changes. And you want that. You don't want to hardcode points of view which over time or when somebody else looks at the data (personalized data security) they will get different results.
Don't hardcode too much. Don't restrict the system artificially.
Update: https://imgur.com/BxGnbbB
Here's a screenshot of where you can restrict the prompt
I am wondering if there is a way to limit the number of rows generated from grafana templating.
I can have a drop down variable "$x"on my grafana page and I can select the row editor and say repeat row for every value under $x.
Based on different criteria, I can have x anywhere between 1 and like 160 rows. I need to only be looking at about 10 at a time. I am wondering if I can control the number of rows shown/change the rows shown somewhere in grafana.
I can manually select items from the $x drop down to show only a few items of course, but its a matter of selecting only say.. 10 items right when the page loads.
Please let me know if I am not describing the problem correctly or if I need to clarify more.
Thanks,
Karan
As far as I know there is no direct option for this but there are some ways you may be able to achieve what you want.
You could select your ~10 default entries and then save the dashboard this will save the selected ones in the dashboards JSON. (or modify the JSON of the dashboard directly)
You could use the regex field in the template settings to filter some of your values and split them in groups this way. (one variable per regex group)
You could change your data in elasticsearch in order to use multiple fields where you can split on.
see PR #5616 as #Daniel Lee mentioned
In general I think you get a faster response to this in grafana directly at github.
I am trying to remove all references to a table from a Crystal XI report. Crystal is telling me that a column from that table is currently being used, because there is a little green check mark over the field in the field viewer. Also, if I try to remove the entire table, I get a warning. The warning is almost useless though because it doesn't tell me where the field is used. Now, back when programmers were real programmers, and mice were things cats chased, I could just grep a directory or file and find all references to a variable I was interested in. But how do I do this in Crystal? I have already tried exporting the report to a Report Definition, which helped find some instances of the troublesome field. Unfortunately, that format does not include all formulas, just some. Please tell me I don't have to buy a third party app (or write my own COM thingy) just to do this seemingly simple thing.
EDIT to add details about tangential point:
In case anyone is wondering, I am not crazy - I have duplicated the issue where a formula's definition does not show up in the exported Report Definition. I created a new blank report, created one formula named stealth that returns 1234. I then used that formula in the Section Expert for the details section, in the "suppress" formula, setting it to {#stealth} == 0. the use of the formula shows up, but not the definition. So when my unwanted column was used in the formula, I was not be able to find it! Here's what the rpt def looks like (after deleting some blank lines):
Crystal Report Professional v11.0 (32-bit) - Report Definition
1.0 File Information
Report File:
Version: 11.0
2.0 Record Sort Fields
3.0 Group Sort Fields
4.0 Formulas
4.1 Record Selection Formula
4.2 Group Selection Formula
4.3 Other Formulas
5.0 Sectional Information
5.1 Page Header Section
Visible, Keep Together
5.2 Page Footer Section
Visible, New Page After, Keep Together, Print At Bottom of Page
5.3 Report Header Section
Visible, New Page Before
5.4 Report Footer Section
Visible, New Page After
5.5 Details Section
Visible
Subsection.1
Visible, Keep Together
Format Formulas
Visible: {#stealth}= 0
If all else fails ...
File -> Export -> Export Report, then choose the Report Definition (TXT) option.
That will give you a plain-text representation of every element of the report. You can grep or CTRL-F or (insert search tool of your choice) through that. "Find in Formulas" usually works, but I've had to go the export route a couple of times, for no apparent reason.
Edit: Of course, if I'd bothered to completely read your post, I'd see that you've already done this.
Very curious.
If you right click on the field in Field Explorer and select Find in Formulas, it should bring up a dialog listing all of the places it is being used in formulas. On the left hand side of the dialog is a tree of all the possible places it could be, including oddball places like record selector and page formatting functions. Unfortunately, it does not seem to list running total fields.
EDIT: Oops, all the places it exists is listed at the bottom of the dialog; the tree view is the entire "DOM" of the report.
I know this is an old post, but...
Not knocking the Find in Formulas, it's been saving me today, but i was having trouble finding the last instance of the field. Even after all of the formulas and the droppings on the report were taken care of, I still had one lone use hiding somewhere.
I found it hiding as a Subreport Link. Right click on the Subreport -> "Change Subreport Links..." and there was the culprit. Dropping in this post because I figured someone else might have this problem too.
Fields can also sometimes be hiding within "Record Sort Expert"
Responding to an old post, but ran into a similar issue. I had a group based on the formula I wanted to delete that had a specified order. When I changed the grouping to a different field, the specified order remained. When I removed the specified order, my formula could be deleted.
This was tested on XIr2...
You change the tables datasource through the "set datasource location" dialog. Now, when it goes into the column mapping mode, uncheck match-type and pick a new column that would cause an error in a formula. (i.e if the column you're looking for is a string replace it with a datetime column). Go to the preview and you should get an error box like "A string is required here.", close that error and up pops the offending formula!
One more suggestion. After following a lot of the suggestions here, my report was still telling me the formula was in use. I had to close the report. When I opened it again, the check mark was gone and it let me delete it. This was on Crystal v 11.0.0.1282
In my case the Formula Field happened to be part of an old Running Total Field, which itself was not included in the report. Once I deleted this old Running Total Field I could delete the unused Formula Field.
Very late, but i use CR 2008 (12.3.0.601) and just today (6/16/2015) i am trying to document only the formulae of my report. I knew about exporting the Report Definition, and Finding a Formula in all Formulae. But there are about 50 Formulae. I discovered that the exported Report Definition didn't document all of my Formulae, but I didn't bother to uncover the logic behind that; instead, i plopped all Formulae into a section, then exported the Report Definition. Voila. Of course, i still need to cull all the unnecessary definition elements. But at least i have all Formulae.
So with all the great selections.. I still had one instance hiding from me. I found out where it was by creating a clone of the data table and renaming\deleting the field.
I then used the "Set Database Location" as suggested above to point to my new table. It did error out when it could not find that field but still didn't tell me where it really was (it just said report field).
I did NOT map it and clicked continue which deleted the field from the report. I then mapped it back to the real table and I was good.
In my case, there was a Chart, and the field was being used as one of the "on Change" fields.
Although an old post, this functional gap still exists within Crystal Reports itself. We have a fully functional 14 day trial of our third party software that uses the latest Crystal.net API to search for plain text within a library of Crystal RPT files in one fell swoop. Also searches the data saved within reports, and text within labels ... as well as datasource behind all your reports ( stored procedures, views, and table data ) with support for SQL Server, SQL Azure, MySQL, Oracle, Amazon RDS, DB2 and Access.
More info and trial downloads at http://www.finditez.com
Note, you will need to download and install the compatible SAP Crystal.net runtime connector for searching your RPT file library.