I have a internal webpage that makes data from excel searchable and readable from a 3rd party excel export file. The webpage allows for the uploading of multiple excel files in which the data gets read and stored in a MySQL database.
We want to update the application to keep a history of the uploaded data (it's data that has monthly values) so we can easily search, filter and generate graphs from the uploaded data.
So I am using Laravel 5.4 and have maatwebsite\excel to import and parse the excel file.
The Excel file always consists of the following columns (Dummy File)
| Item group | item # | item name | Item Currency | <month> <year> |
After Item Currency there is always 36 columns for the past 3 years of data from the current month so a column would be named like dec 2017
Now in Laravel I have created a Model for the item named Item and a model for the monthly values named ItemMonthly
Now I am able to read the file and create columns dynamically in the database but I feel like this is very ugly and not efficient at all:
(Gist) Code for Models and Excel Function
Biggest problem
Because I need to read all the monthly data and since I need them in order of month I can't really rename all the columns as far as I know. I need to be able to get all the columns to render in a Highchart graph and in a Datatable. and some items don't have the same monthly data (some only go till 2015 for example.
Needed advice
I've read a couple of solutions here some of them saying instead of creating columns in MySQL just store the monthly data as a json object in a single column.
Some answers just completely advice on changing from MySQL to MongoDB
I am kind of at a loss to find the best approach for this, and am sincerely wondering if MySQL is the right way to go. The solutions I have been trying so far all seem to involve really hacky ways of doing this.
If there is more info needed please let me know. I don't want to write an immense wall of text but I also want to provide the correct amount of information.
Many thanks!
Related
I am trying to extract tables from pdf files. I am using UiPath's Document Understanding for this purpose. I have to create a template for this purpose and then use that template for other similar invoices. The issue I am facing is that the number of items in the table is varying. For example some pdf's have table which contain 4 items whereas other pdf files have table which contain only 1 item. So if I create a template using the pdf which has a table containing 4 items then it works. But then the same template when used for files which have table containing 1 item then it does not properly extract the data in the table. Is there any solution for this?
The solution should be able to extract tables from similar invoices containing varying number of items in the table. The format and layout of the invoice and the table is similar. The only thing varying is the number of items in each table.
Thanks for your time and help!
I'm having a strange issue with exporting/updating/importing data in our on-premises Dynamics 365 (8.2). I was doing a bulk update of over 3000 records by exporting the records to an Excel workbook, updating the data in a specific column, then importing the workbook back into CRM. It worked for all of the records except 14 of them, which according to the import log was for the reason that "You cannot import data to this record because the record was updated in Microsoft Dynamics 365 after it was exported." I looked at the Audit History of those 14 records, and find that they have not been modified in any way for a good two months. Strangely, the modified date of the most recent Audit History entry for ALL 14 records is the exact same date/time.
We have a custom workflow that runs once every 24 hours on a schedule that automatically updates the Age field of our Contact records based on the value in the respective Birthday field. For these 14 records, ALL of them have a birthday of November 3rd, but in different years. What that means though is that the last modification that was done to them was on 11/3/2019 via the workflow. However, I cannot understand why the system "thinks" that this should prevent a data update/import.
I am happy to provide any additional information that I may have forgotten to mention here. Can anyone help me, please?
While I was not able to discover why the records would not update, I was able to resolve the issue. Before I share what I did to update the records, I will try and list as many things as I can remember that I tried that did not work:
I reworked my Advanced Find query that I was using to export the records that needed updated to return ONLY those records that had actual updates. Previously, I used a more forgiving query that returned about 30 or so records, even though I knew that only 14 of them had new data to import. I did so because the query was easier to construct, and it was no big deal to remove the "extra" records from the workbook before uploading it for import. I would write a VLOOKUP for the 30-something records, and remove the columns for which the VLOOKUP didn't find a value in my dataset, leaving me with the 14 that did have new data. After getting the error a few times, I started to ensure that I only exported the 14 records that needed to be updated. However, I still got the error when trying to import.
I tried formatting the (Do Not Modify) Modified On column in the exported workbook to match the date format in the import window. On export of the records, Excel was formatting this column as m/d/yyyy h:mm while the import window with the details on each successful and failed import showed this column in mm/dd/yyyy hh:mm:ss format. I thought maybe if I matched the format in Excel to the import window format it might allow the records to import. It did not.
I tried using some Checksum verification tool to ensure that the value in the (Do Not Modify) Checksum column in the workbook wasn't being written incorrectly or in an invalid format. While the tool I used didn't actually give me much useful information, it did recognize that the values were checksum hashes, so I supposed that was helpful enough for my purposes.
I tried switching my browser from the new Edge browser (the one that uses Chromium) to just IE as suggested on the thread provided by Arun. However, it did not resolve the issue.
What ended up working in the end was Arun's suggestion to just do some arbitrary edit to all the records and exporting them afterward. This was okay to do for just 14 records, but I'm still slightly vexed as this wouldn't really be a feasible solution of it were, say, a thousand records that were not importing. There was no field that ALL 14 Contact records had in common that I could just bulk edit, and bulk edit back again. What I ended up doing was finding a text field on the Contact Form that did not have any value in it for any of the records, putting something in that field, then going to each record in turn and removing the value (since I don't know of a way to "blank out" or clear a text field while bulk editing. Again, this was okay for such a small number of records, but if it were to happen on a larger number, I would have to come up with an easier way to bulk edit and then bulk "restore" the records. Thanks to Arun for the helpful insights, and for taking the time to answer. It is highly appreciated!
When you first do an import of an entity (contacts for example) you see that your imported excel contains 3 hidden columns (Do Not Modify) Contact, (Do Not Modify) Row Checksum, (Do Not Modify) Modified On.
When you want to create new instances of the entity, just edit the records and clear the content of the 3 hidden colums.
This error will happen when there is a checksum difference or rowversion differs from the exported record vs the record in database.
Try to do some dummy edit for those affected records & try to export/reimport again.
I could think of two reasons - either the datetime format confusing the system :( or the the community thread explains a weird scenario.
Apparently when importing the file, amending and then saving as a different file type alters the spreadsheet's parameters.
I hence used Internet Explorer since when importing the file, the system asks the user to save as a different format. I added .xlsx at the end to save it as the required format. I amended the file and imported it back to CRM..It worked
For me it turned out to be a different CRM time zone setting for the exporter and importer. Unfortunately this setting doesn't seem to be able to be changed by an administrator via the user interface.
The setting is available for each user under File->Options->Time Zone.
Currently, am trying to retrieve data from this page: https://www.hdb.gov.sg/cs/infoweb/residential/renting-a-flat/renting-from-the-open-market/rental-statistics , as you can see, there are 4 quarters in a year, and for each quarter, there is a different table. I wish to extract the table but currently, i am unable to automate the process, only able to take one. On top of that, i wish to add two columns to the retrieved data table which is "Quarter" and "Year". Any suggestions? Attached photos are my workflow and my excel.
Get the number of years/ loop through the years (or start with the 1st year up to the last year).
For each year try to get the data via data scraping (the elements exist, just hidden/not expanded ; do one table datascraping for data modelling and reuse it within the loop). For the datascraping you need to change the selector, to make it usable for all tables by using the year and the quarter (just a generic example, like * year * quarter *). Columns are the same for all tables.
I haven't seen details within the website menu or within the page, is good to check if robots are allowed to scrape for data
Above would be the quickest way. More complex with FindChidren activity.
I'm stuck with Google Sheets.
Situation:
I have a data table with projects. Each project as a few attributes, most importantly including which team member has worked on the project this month.
Goal:
I need to convert the data to a new table that is built up differently. I need one row per project per active team member.
Sample data and goal:
https://docs.google.com/spreadsheets/d/1QcNPsvHX8hBNUpCJiutof8yD8ukFYcCXM_pLNrQmDUs/edit?usp=sharing (can edit)
As you can see, SEO and Island now have two rows instead of one, as Jan AND Chris have worked on the projects this month.
Approach:
I tried FILTER, QUERY (with pivot) and thought about Scripting (basically its an iteration over the Matrix B3:E8...). However, I am not particularly skilled at Sheets and am very thankful for your help. Thanks a billion, guys!!!
You can do this a fairly standard way by using Textjoin to join together the corresponding column headers and other data for the non-blank cells, then separating it into rows then rows and columns with the Transpose and Split functions:
=ArrayFormula(split(transpose(split(textjoin("¶",true,if(B3:E8="","",A3:A8&"|"&F3:F8&"|"&G3:G8&"|"&H3:H8&"|"&I3:I8&"|"&B2:E2)),"¶")),"|"))
I'm trying to build a star schema in Oracle 12c. In my case my data source is not a relational database but a single excel/csv file which is populated via a google form, which means I don't have any sort of reference from a source system such as auto incremental keys/ids. Now what would be the best approach to build a star schema given this condition?
File row sample:
<submitted timestamp>,<submitted by user>,<region>,<country>,<branch>,<branch location>,<branch area>,<branch type>,<branch name>,<branch private? yes/no value>,<the following would be all "fact" values (measurements),...,...,...
In case i wanted to build a "branch" dimension, how would I handle updates/inserts after the first load into the dimension table?
Thought solution so far:
I had thought of making a concatenated string "key" with the branch values, which would make it unique (underscore would be the "glue" to concatenate the values), eg:
<region>_<country>_<branch>_<branch location> as branch_key
I would insert all the distinct branches into a staging table, including they branch_key column for each one of them, then when trying to load into the dimension I could compare which key does not exists yet in my dimension table and then insert it. As for updates, I'm a bit stuck on how to handle that, I had thought of having another file mapping which branches are active having a expiration date column. Basically trying to simulate what I could do having the data in a database instead of CSV files.
This is all I can think of so far, do you have any other recommendations/ideas on how to implement this? Take on consideration that the data source cannot as in I have to read these csv files, since data is not stored anywhere else.
Thank you.