Say I have a collection with 100 million records/documents in it.
I want to create a series of reports that involve summing of values in certain columns and grouping by various columns.
What references for XQuery and/or MarkLogic can anyone point me to that will allow me to do this quickly?
I saw cts:avg-aggregate which looks fine. But then I need to group as well..
Also, since I am dealing with a large amount of data and it will take some time to go through it all, I am thinking about setting this up as a job that runs at night to update the report.
I thought of using corb to run through the records and then do something with the output from that. Is this the right approach with MarkLogic and reporting?
Perhaps this guide would help:
http://developer.marklogic.com/blog/group-by-the-marklogic-way
You have several options which are discussed above:
cts:estimate
cts:element-value-co-occurrences
cts:value-tuples + cts:frequency
I have to change the data source in quite a few reports. Its easy when the original data source uses table, but its more complicated when instead has a SQL command (well, practically its a view but made in the report, not in the original database).
Lets say that report has originally such command:
SELECT nbr FROM equipment WHERE equipment.owner='ABC'
I know that in the new database Equipment.nbr is called now Items.ID, so I can easily map this. But what about the rest of command, the "WHERE" part? In the new database there is obviously no Equipment.owner and possibly might not even be Items.owner. Does crystal simply drop this part? I know how to remake it, by simply adding selection formula to the report, but first I have to be know what happened to the "WHERE" condition, and after such mapping I can't anymore preview the SQL command in the data source.
Confrim me where you are using this query.
If it is in Data soruce then no issue this report will work.
If it is in formula then I have doubt crystal report won't accept this format even in basic syntax mode.
I have some reports written in Crystal 2008 using business views. These reports have a date parameter set up and I have a selection on the date defined in the select expert. However, when I run the report it appears to retrieve all the data from the database and only then filter out based on the date. As you can imagine this slows down the report quite a bit. I also clicked on Database-Show SQL Query and confirmed that the date parameter did not appear in the SQL Query. This behavior seems very strange to me. This did not use to happen to me when I used Crystal 8.5 with dictionaries. Is this a limitation using business views?
I did some searching and found that I can create a report using a database command. This helped improve performance on one of my reports but when I tried to do something similar on a different report, even though I was using the database command, it still did not appear in the appear to be doing the selection on the database before retrieving the data and the report took forever to run. I also didn't see the selection in the SQL Query.
Do I need to add the parameter to the database command? Will I be able to prompt the user to enter the value when they run the report?
I hope there is a way to do this properly using business views because otherwise I'll have to rewrite all my reports to use another method.
Any ideas or advice are welcome. Thank you very much!
I had a similar problem. I used the command, but my report was still taking longer than i had hoped to run. so i added a where statement into the command to start checking dates starting from 2009. that sped up my report a little.
you may want to consider creating a stored procedure if you think you are pushing CR to the limit. that may also help sped up the report.
I figured out what the problem is. My business view had fields in it that were formulas. If you try to use selection criteria using a formula, it does not add the criteria to the WHERE clause in the SQL Query. Luckily, I was able to find other fields besides the formula in the business view to do the selection.
This is a problem that my friend asked over the phone. The C# 3.5 program he has written is filling a Dataset from a Patient Master table which has 350,000 records. It uses the Microsoft ADO.NET driver for Oracle. The ExecuteQuery method takes over 30 seconds to fill the dataset. However, the same query (fetching about 20K records) takes less than 3 second in Toad . He is not using any Transactions within the program. It has an index on the column (Name) which is being used to search.
These are some alternatives i suggested :-
1) Try to use a Data Reader and then populate a Data table and pass it to the form to bind it to the Combo box (which is not a good idea since it is likely to take same time)
2) Try Oracles' ADO.NET Driver
3) Use Ants profiler to see if you can identify any particular ADO.NET line.
Has anyone faced similar problems and what are some ways of resolving this.
Thanks,
Chak.
You really need to do an extended SQL trace to see where the slowness is coming from. Here is a paper from Cary Millsap (of Method R and formerly of Hotsos) that details doing this:
http://method-r.com/downloads/doc_details/10-for-developers-making-friends-with-the-oracle-database-cary-millsap
Toad would typically only fetch the first x rows (500 in my setup). So double check if the comparison is valid.
Then you should try to seperate the db stuff from the form stuff if possible to see if the db is taking up the time.
If that's the case, try the Oracle libraries if that is any faster, we've seen 50% improvements between the latest Oracle driver and the standard Microsoft driver.
Without knowing the actual code he uses to accomplish his tasks and not knowing the number of rows he's actually fetching (I'm hoping he doesn't read all 350K of them?) it's impossible to say anything that's gonna help him.
Have him add a code snippet to the question for clarity.
I have an SSRS report that calls out to a stored procedure. If I run the stored procedure directly from a query window, it will return in under 2 seconds. However, the same query run from an 2005 SSRS report takes up to 5 minutes to complete. This is not just happening on the first run, it happens every time. Additionally, I don't see this same problem in other environments.
Any ideas on why the SSRS report would run so slow in this particular environment?
Thanks for the suggestions provided here. We have found a solution and it did turn out to be related to the parameters. SQL Server was producing a convoluted execution plan when executed from the SSRS report due to 'parameter sniffing'. The workaround was to declare variables inside of the stored procedure and assign the incoming parameters to the variables. Then the query used the variables rather than the parameters. This caused the query to perform consistently whether called from SQL Server Manager or through the SSRS report.
I will add that I had the same problem with a non-stored procedure query - just a plain select statement. To fix it, I declared a variable within the dataset SQL statement and set it equal to the SSRS parameter.
What an annoying workaround! Still, thank you all for getting me close to the answer!
Add this to the end of your proc: option(recompile)
This will make the report run almost as fast as the stored procedure
I had the same problem, here is my description of the problem
"I created a store procedure which would generate 2200 Rows and would get executed in almost 2 seconds however after calling the store procedure from SSRS 2008 and run the report it actually never ran and ultimately I have to kill the BIDS (Business Intelligence development Studio) from task manager".
What I Tried: I tried running the SP from reportuser Login but SP was running normal for that user as well, I checked Profiler but nothing worked out.
Solution:
Actually the problem is that even though SP is generating the result but SSRS engine is taking time to read these many rows and render it back.
So I added WITH RECOMPILE option in SP and ran the report .. this is when miracle happened and my problem got resolve.
I had the same scenario occuring..Very basic report, the SP (which only takes in 1 param) was taking 5 seconds to bring back 10K records, yet the report would take 6 minutes to run. According to profiler and the RS ExecutionLogStorage table, the report was spending all it's time on the query. Brian S.'s comment led me to the solution..I simply added WITH RECOMPILE before the AS statement in the SP, and now the report time pretty much matches the SP execution time.
I simply deselected 'Repeat header columns on each page' within the Tablix Properties.
If your stored procedure uses linked servers or openquery, they may run quickly by themselves but take a long time to render in SSRS. Some general suggestions:
Retrieve the data directly from the server where the data is stored by using a different data source instead of using the linked server to retrieve the data.
Load the data from the remote server to a local table prior to executing the report, keeping the report query simple.
Use a table variable to first retrieve the data from the remote server and then join with your local tables instead of directly returning a join with a linked server.
I see that the question has been answered, I'm just adding this in case someone has this same issue.
I had the report html output trouble on report retrieving 32000 lines. The query ran fast but the output into web browser was very slow. In my case I had to activate “Interactive Paging” to allow user to see first page and be able to generate Excel file. The pros of this solution is that first page appears fast and user can generate export to Excel or PDF, the cons is that user can scroll only current page. If user wants to see more content he\she must use navigation buttons above the grid. In my case user accepted this behavior because the export to Excel was more important.
To activate “Interactive Paging” you must click on the free area in the report pane and change property “InteractiveSize”\ “Height” on the report level in Properties pane. Set this property to different from 0. I set to 8.5 inches in my case. Also ensure that you unchecked “Keep together on one page if possible” property on the Tablix level (right click on the Tablix, then “Tablix Properties”, then “General”\ “Page Break Options”).
I came across a similar issue of my stored procedure executing quickly from Management Studio but executing very slow from SSRS. After a long struggle I solved this issue by deleting the stored procedure physically and recreating it. I am not sure of the logic behind it, but I assume it is because of the change in table structure used in the stored procedure.
I Faced the same issue. For me it was just to unckeck the option :
Tablix Properties=> Page Break Option => Keep together on one page if possible
Of SSRS Report. It was trying to put all records on the same page instead of creating many pages.
Aside from the parameter-sniffing issue, I've found that SSRS is generally slower at client side processing than (in my case) Crystal reports. The SSRS engine just doesn't seem as capable when it has a lot of rows to locally filter or aggregate. Granted, these are result set design problems which can frequently be addressed (though not always if the details are required for drilldown) but the more um...mature...reporting engine is more forgiving.
In my case, I just had to disconnect and connect the SSMS. I profiled the query and the duration of execution was showing 1 minute even though the query itself runs under 2 seconds. Restarted the connection and ran again, this time the duration showed the correct execution time.
I was able to solve this by removing the [&TotalPages] builtin field from the bottom. The time when down from minutes to less than a second.
Something odd that I could not determined was having impact on the calculation of total pages.
I was using SSRS 2012.
Couple of things you can do, without executing the actual report just run the sproc from within the data tab of reporting services. Does it still take time?
Another option is to use SQL Profiler and determine what is coming in and out of the database system.
Another thing you can do to test it, so to recreate a simple report without any parameters. Run the report and see if it makes a difference. It could be that your RS report is corrupted or badly formed that may cause the rendering to be really slow.
Had the same problem, and fixed it by giving the shared dataset a default parameter and updating that dataset in the reporting server.
DO you use "group by" in the SSRS table?
I had a report with 3 grouped by fields and I noticed that the report runed very slowly despite having a light query, to the point where I can't even dial values in the search field.
Than I removed the groupings and now the report goes up in seconds and everything works in an instant.
In our case, no code was required.
Note from our Help Desk: "Clearing out your Internet Setting will fix this problem."
Maybe that means "clear cache."