I have a internal webpage that makes data from excel searchable and readable from a 3rd party excel export file. The webpage allows for the uploading of multiple excel files in which the data gets read and stored in a MySQL database.
We want to update the application to keep a history of the uploaded data (it's data that has monthly values) so we can easily search, filter and generate graphs from the uploaded data.
So I am using Laravel 5.4 and have maatwebsite\excel to import and parse the excel file.
The Excel file always consists of the following columns (Dummy File)
| Item group | item # | item name | Item Currency | <month> <year> |
After Item Currency there is always 36 columns for the past 3 years of data from the current month so a column would be named like dec 2017
Now in Laravel I have created a Model for the item named Item and a model for the monthly values named ItemMonthly
Now I am able to read the file and create columns dynamically in the database but I feel like this is very ugly and not efficient at all:
(Gist) Code for Models and Excel Function
Biggest problem
Because I need to read all the monthly data and since I need them in order of month I can't really rename all the columns as far as I know. I need to be able to get all the columns to render in a Highchart graph and in a Datatable. and some items don't have the same monthly data (some only go till 2015 for example.
Needed advice
I've read a couple of solutions here some of them saying instead of creating columns in MySQL just store the monthly data as a json object in a single column.
Some answers just completely advice on changing from MySQL to MongoDB
I am kind of at a loss to find the best approach for this, and am sincerely wondering if MySQL is the right way to go. The solutions I have been trying so far all seem to involve really hacky ways of doing this.
If there is more info needed please let me know. I don't want to write an immense wall of text but I also want to provide the correct amount of information.
Many thanks!
Scenario, in error a new option set label of "Update" was added with value 100,000,000 to a field in the default managed solution in Production.
Identical label with value 866,100,002 was added to same field in the unmanaged Development Solution, when latest round of export and import occurred a duplicate option set label for "Update" was added to the managed solution in Production.
I now have data in the tables with both values and duplicate labels in the managed solution.
Question: how to unwind this mess - can I delete the label relating to value 100,000,000 seeing how it will just get duplicated upon the next solution import?
What happens to the data in the Database - is there a way to update the recorded values of 100,000,000 to a correct 866,100,002?
Do an Advanced Find for records having optionset value 100,000,000, export to excel (with selected option to reimport). Bulk update the exported records to the correct 866,100,002 optionset value & reimport it. This is first thing, it will correct the data.
Then you can delete the dupe label in picklist & monitor for future imports.
Test it in lower environments. Take solution/database backup as a precaution.
I am trying to import a dataset into a custom entity in Dynamics CRM 2015 on-prem, using the import template for the entity, saved as CSV.
My dataset is quite small - only 10 rows. (Yes, I know it would probably take less time to just enter them manually).
When I import the data, CRM reads it as 3,001 records. The extra records show up totally blank. I am sure I don't have any extraneous data in other rows and columns.
Has anyone seen this, or have any idea what could be going on?
I have faced the same issue once. Sometimes the csv file containing some empty rows.
I have copied only the rows which containing data to a new csv file and imported it again. This approach solved my issue.
You can try once.
Another option is to open your CSV in Notepad or some other text editor. If there are additional rows, you will see them (rows that look like ",,,,,,,". You can delete them in the text editor and save, thereby being sure that Excel won't add the blank values back.
So there are a couple of similar questions, but all are using javascript, which isn't ideal as it requires the record to be opened / saved.
So, how can you calculate age based off birthdate. There are 200,000 records this would need to be done on and it's using CRM 2015, so can involve calculated fields as well.
It's going to be reported on in the background, so we can't use Javascript.
Workflows are a possibility, but running them on 200,000 records daily isn't exactly elegant!
Any other suggestions?
I've come across this requirement a number of times.
I've solved it by writing a Scribe or SSIS job which runs nightly and updates the Contact.Age field.
In order to not update every Contact record with the calculated age (as most ages won't have changed), I've used one of the following:
For on-premise CRM (where I have SQL access to the database), I wrote a query to return:
contactid
contact age
contact DoB
calculated age (calculated column from DoB and getdate)
The Scribe or SSIS job would only update records where Contact.Age != CalculatedAge
For hosted CRM (where I don't have SQL access to the DB):
Add a field called 'Next birthday'
The Scribe/SSIS job would search for records where NextBirthday is null or prior to today. It would update the Age and NextBirthday field.
Both of these methods mean that if the nightly job doesn't run for whatever reason, then when it's next run it will catch up on any records that are now out of date.
http://blogs.msdn.com/b/crm/archive/2009/02/27/creating-a-birthday-contact-list.aspx has an example using a pre-plugin to populate the birth month, year, and day fields. This could be adapted to instead perform the calculation to populate an age field. That being said, this would only work for new records or records that are changed.
If you wanted to do this via workflow, you'd have to have a workflow assembly to perform the calculation to populate an age field. As an alternative that doesn't require any code, you could create an Advanced Find query for All birthdays in a certain time frame, i.e. "Birthday on or before 2/17/1975" (this should limit the amount of records returned and reduce it from the total of the 200,000). Include the birthday and a new Age field created in the columns shown. I simply created an age field as a text field with a size of 5 characters since I'm intending only to store the years old someone is in it. Export the contacts to Excel marking the options, "Static worksheet with records from all pages in the current view" and "Make this data available for re-importing by including required column headings". Make sure to include the Owner column in order to prevent reassigning all these records to yourself when reimporting the records.
Then in Excel, create a formula like the following in the Age column assuming the birthdate field is in Column L, "=DATEDIF($L2,NOW(),"y")", which will update the age field with how many years old someone is as of the current date. Note that you might have to perform this calculation on a separate column and copy in just the values in order to ensure that Excel does not change the data type for the Age column or you will not be able to import that data back into Microsoft CRM. Fill that formula down so all records are updated, and save the file. Then in Microsoft CRM, import these records by pointing to the updated XML file (Excel 2003 XML format). Here your only restrictions are going to be on the size of the import file (CRM Limits this to 8 MB per file) and will be restricted to 10,000 records for the export, so this is another reason to break up the records you are exporting for reimport.
If you do update these via a workflow, you can update more than 250 at a time using a solution like the one in the CRM 2013 Bulk Workflow tool for XRM Toolbox http://www.zero2ten.com/blog/crm-2013-bulk-workflow-tool-for-xrmtoolbox/ , which allows you to select a group of records using FetchXML as the criteria for the records to apply the workflow to, noting that this may take some time to process if you are running this at the same time for all 200,000 records.
Ideally, my preference would be to have a plugin or JavaScript, but can see with your requirements that you would need to have this run either daily or on a monthly basis (although I would not run it for all 200,000 since everyone's age does not change each day). Just choose the records that have birthdays in a particular month or on a particular date to run the workflows on or to export and reimport for since that's going to be much less intensive for server processing and will be able to complete much faster than having to update all 200,000 at a time.
I am trying to remove all references to a table from a Crystal XI report. Crystal is telling me that a column from that table is currently being used, because there is a little green check mark over the field in the field viewer. Also, if I try to remove the entire table, I get a warning. The warning is almost useless though because it doesn't tell me where the field is used. Now, back when programmers were real programmers, and mice were things cats chased, I could just grep a directory or file and find all references to a variable I was interested in. But how do I do this in Crystal? I have already tried exporting the report to a Report Definition, which helped find some instances of the troublesome field. Unfortunately, that format does not include all formulas, just some. Please tell me I don't have to buy a third party app (or write my own COM thingy) just to do this seemingly simple thing.
EDIT to add details about tangential point:
In case anyone is wondering, I am not crazy - I have duplicated the issue where a formula's definition does not show up in the exported Report Definition. I created a new blank report, created one formula named stealth that returns 1234. I then used that formula in the Section Expert for the details section, in the "suppress" formula, setting it to {#stealth} == 0. the use of the formula shows up, but not the definition. So when my unwanted column was used in the formula, I was not be able to find it! Here's what the rpt def looks like (after deleting some blank lines):
Crystal Report Professional v11.0 (32-bit) - Report Definition
1.0 File Information
Report File:
Version: 11.0
2.0 Record Sort Fields
3.0 Group Sort Fields
4.0 Formulas
4.1 Record Selection Formula
4.2 Group Selection Formula
4.3 Other Formulas
5.0 Sectional Information
5.1 Page Header Section
Visible, Keep Together
5.2 Page Footer Section
Visible, New Page After, Keep Together, Print At Bottom of Page
5.3 Report Header Section
Visible, New Page Before
5.4 Report Footer Section
Visible, New Page After
5.5 Details Section
Visible
Subsection.1
Visible, Keep Together
Format Formulas
Visible: {#stealth}= 0
If all else fails ...
File -> Export -> Export Report, then choose the Report Definition (TXT) option.
That will give you a plain-text representation of every element of the report. You can grep or CTRL-F or (insert search tool of your choice) through that. "Find in Formulas" usually works, but I've had to go the export route a couple of times, for no apparent reason.
Edit: Of course, if I'd bothered to completely read your post, I'd see that you've already done this.
Very curious.
If you right click on the field in Field Explorer and select Find in Formulas, it should bring up a dialog listing all of the places it is being used in formulas. On the left hand side of the dialog is a tree of all the possible places it could be, including oddball places like record selector and page formatting functions. Unfortunately, it does not seem to list running total fields.
EDIT: Oops, all the places it exists is listed at the bottom of the dialog; the tree view is the entire "DOM" of the report.
I know this is an old post, but...
Not knocking the Find in Formulas, it's been saving me today, but i was having trouble finding the last instance of the field. Even after all of the formulas and the droppings on the report were taken care of, I still had one lone use hiding somewhere.
I found it hiding as a Subreport Link. Right click on the Subreport -> "Change Subreport Links..." and there was the culprit. Dropping in this post because I figured someone else might have this problem too.
Fields can also sometimes be hiding within "Record Sort Expert"
Responding to an old post, but ran into a similar issue. I had a group based on the formula I wanted to delete that had a specified order. When I changed the grouping to a different field, the specified order remained. When I removed the specified order, my formula could be deleted.
This was tested on XIr2...
You change the tables datasource through the "set datasource location" dialog. Now, when it goes into the column mapping mode, uncheck match-type and pick a new column that would cause an error in a formula. (i.e if the column you're looking for is a string replace it with a datetime column). Go to the preview and you should get an error box like "A string is required here.", close that error and up pops the offending formula!
One more suggestion. After following a lot of the suggestions here, my report was still telling me the formula was in use. I had to close the report. When I opened it again, the check mark was gone and it let me delete it. This was on Crystal v 11.0.0.1282
In my case the Formula Field happened to be part of an old Running Total Field, which itself was not included in the report. Once I deleted this old Running Total Field I could delete the unused Formula Field.
Very late, but i use CR 2008 (12.3.0.601) and just today (6/16/2015) i am trying to document only the formulae of my report. I knew about exporting the Report Definition, and Finding a Formula in all Formulae. But there are about 50 Formulae. I discovered that the exported Report Definition didn't document all of my Formulae, but I didn't bother to uncover the logic behind that; instead, i plopped all Formulae into a section, then exported the Report Definition. Voila. Of course, i still need to cull all the unnecessary definition elements. But at least i have all Formulae.
So with all the great selections.. I still had one instance hiding from me. I found out where it was by creating a clone of the data table and renaming\deleting the field.
I then used the "Set Database Location" as suggested above to point to my new table. It did error out when it could not find that field but still didn't tell me where it really was (it just said report field).
I did NOT map it and clicked continue which deleted the field from the report. I then mapped it back to the real table and I was good.
In my case, there was a Chart, and the field was being used as one of the "on Change" fields.
Although an old post, this functional gap still exists within Crystal Reports itself. We have a fully functional 14 day trial of our third party software that uses the latest Crystal.net API to search for plain text within a library of Crystal RPT files in one fell swoop. Also searches the data saved within reports, and text within labels ... as well as datasource behind all your reports ( stored procedures, views, and table data ) with support for SQL Server, SQL Azure, MySQL, Oracle, Amazon RDS, DB2 and Access.
More info and trial downloads at http://www.finditez.com
Note, you will need to download and install the compatible SAP Crystal.net runtime connector for searching your RPT file library.