I noticed that whenever I import the same set of CSV my records are duplicated even if I have skip_unchanged=True. Ideally, I wanted is if the CSV gets imported again it should not import the same records and prevent duplicate but it should update if there are changes to it.
I have this configuration in my resource file
bill_date = fields.Field(
attribute="bill_date", column_name="date", widget=widgets.DateWidget()
)
then import_id_fields = ("account_number",)
I also tried printing the original and instance from the skip_row method but I get this in the logs,
print(f"{getattr(original, "bill_date")} - {getattr(instance, "bill_date")}")
RESULT: None - 2021-06-07
UPDATE
Fixed my issue, I mistakenly added get_instance = False during one of the test.
This should work fine. What you need to do is to ensure that account_number is included in the csv feed, and that it can uniquely identify a record in the table you are importing into.
Then, when the import occurs, the logic tries to load the existing record using account_number and will update the row if it is present, otherwise it creates a new row.
This is documented here, and you can debug the get_or_init_instance() method if it is not working.
If skip_unchanged is true, then the logic will compare each field declared in your fields list, and will not update if there are no changes between the stored data and the imported data.
Related
Currently working with Excel and Laravel to import a sheet to my database. The uploading and showing works (yay!) However, I noticed it seems to leave out some columns. The next to last two. The last one gets filled and the one before that does too, so I am curious as to why this does not work as it should.
Database rows:
Table structure:
Excel sheet I am importing (there are more rows but I am limiting it to the troublesome area):
Data import class (this is where I assign the values to the variables):
importeren.blade.php (this is where the importing happens). Reading out the data for each entry using a foreach loop.
As you can see, they are all numeric values, and all of them load fine except for "ADVJaarwerk" and "PeriodiekeADV"
Sidenote: I put char64 because my laravel application demanded me to do so. The tables all run fine except for the specified two. I am honestly wondering why.
Make sure the ADVJaarwerk and PeriodiekeADV attributes are mass assignable in your model:
protected $fillable = [
...
'ADVJaarwerk',
'PeriodiekeADV',
...
];
it because of your $row['ADVJaarwerk'] or $row['PeriodiekeADV'] write wrong naming conversion is wrong
change to excel file field name adj jaarwerk and variable name $row['adj_jaarwerk'];
change to excel file field name periodiekeadv and variable name $row['periodiekeadv'];
and check fillable [] property in model..
i hope so it works
I'm having a strange issue with exporting/updating/importing data in our on-premises Dynamics 365 (8.2). I was doing a bulk update of over 3000 records by exporting the records to an Excel workbook, updating the data in a specific column, then importing the workbook back into CRM. It worked for all of the records except 14 of them, which according to the import log was for the reason that "You cannot import data to this record because the record was updated in Microsoft Dynamics 365 after it was exported." I looked at the Audit History of those 14 records, and find that they have not been modified in any way for a good two months. Strangely, the modified date of the most recent Audit History entry for ALL 14 records is the exact same date/time.
We have a custom workflow that runs once every 24 hours on a schedule that automatically updates the Age field of our Contact records based on the value in the respective Birthday field. For these 14 records, ALL of them have a birthday of November 3rd, but in different years. What that means though is that the last modification that was done to them was on 11/3/2019 via the workflow. However, I cannot understand why the system "thinks" that this should prevent a data update/import.
I am happy to provide any additional information that I may have forgotten to mention here. Can anyone help me, please?
While I was not able to discover why the records would not update, I was able to resolve the issue. Before I share what I did to update the records, I will try and list as many things as I can remember that I tried that did not work:
I reworked my Advanced Find query that I was using to export the records that needed updated to return ONLY those records that had actual updates. Previously, I used a more forgiving query that returned about 30 or so records, even though I knew that only 14 of them had new data to import. I did so because the query was easier to construct, and it was no big deal to remove the "extra" records from the workbook before uploading it for import. I would write a VLOOKUP for the 30-something records, and remove the columns for which the VLOOKUP didn't find a value in my dataset, leaving me with the 14 that did have new data. After getting the error a few times, I started to ensure that I only exported the 14 records that needed to be updated. However, I still got the error when trying to import.
I tried formatting the (Do Not Modify) Modified On column in the exported workbook to match the date format in the import window. On export of the records, Excel was formatting this column as m/d/yyyy h:mm while the import window with the details on each successful and failed import showed this column in mm/dd/yyyy hh:mm:ss format. I thought maybe if I matched the format in Excel to the import window format it might allow the records to import. It did not.
I tried using some Checksum verification tool to ensure that the value in the (Do Not Modify) Checksum column in the workbook wasn't being written incorrectly or in an invalid format. While the tool I used didn't actually give me much useful information, it did recognize that the values were checksum hashes, so I supposed that was helpful enough for my purposes.
I tried switching my browser from the new Edge browser (the one that uses Chromium) to just IE as suggested on the thread provided by Arun. However, it did not resolve the issue.
What ended up working in the end was Arun's suggestion to just do some arbitrary edit to all the records and exporting them afterward. This was okay to do for just 14 records, but I'm still slightly vexed as this wouldn't really be a feasible solution of it were, say, a thousand records that were not importing. There was no field that ALL 14 Contact records had in common that I could just bulk edit, and bulk edit back again. What I ended up doing was finding a text field on the Contact Form that did not have any value in it for any of the records, putting something in that field, then going to each record in turn and removing the value (since I don't know of a way to "blank out" or clear a text field while bulk editing. Again, this was okay for such a small number of records, but if it were to happen on a larger number, I would have to come up with an easier way to bulk edit and then bulk "restore" the records. Thanks to Arun for the helpful insights, and for taking the time to answer. It is highly appreciated!
When you first do an import of an entity (contacts for example) you see that your imported excel contains 3 hidden columns (Do Not Modify) Contact, (Do Not Modify) Row Checksum, (Do Not Modify) Modified On.
When you want to create new instances of the entity, just edit the records and clear the content of the 3 hidden colums.
This error will happen when there is a checksum difference or rowversion differs from the exported record vs the record in database.
Try to do some dummy edit for those affected records & try to export/reimport again.
I could think of two reasons - either the datetime format confusing the system :( or the the community thread explains a weird scenario.
Apparently when importing the file, amending and then saving as a different file type alters the spreadsheet's parameters.
I hence used Internet Explorer since when importing the file, the system asks the user to save as a different format. I added .xlsx at the end to save it as the required format. I amended the file and imported it back to CRM..It worked
For me it turned out to be a different CRM time zone setting for the exporter and importer. Unfortunately this setting doesn't seem to be able to be changed by an administrator via the user interface.
The setting is available for each user under File->Options->Time Zone.
I have a metadata Name as CONTACTS(SOURCE.CSV|TAGET.CSV). Now I read this file using reader and populate the value in table that I created as CONTACT_TABLE(PK NUMBER, Source_name varchar2(500),target_name varchar2(500)) after that I want to read these source.csv and target.csv file stored in my table CONTACT_TABLE AND populate the value in other table called SOURCE_COLUMN_TARGET_COLUMN_TABLE(PK,FK as pk of contact_table,source_column,target_column) this table should contain all the column of source and target and should have one to one relationship with that, for example, source.csv(fn)-----target.csv(firstName)
My objective is whenever we add some other attribute in source or target I should not change the entire mapping for eg if we add source.csv(email) and target.csv(email) it should directly map
Thanks!
please help!
I have this task completed before Friday and I searched every source I found dynamic mapping thing and parameter thing but it was not very helpful I want to do this way itself
Not clear what you are asking actually. The source analyser uses source files(.csv) on import itself and thereby contains the same format in source qualifier.
So, if any of the values gets added into your existing files (source.csv, target.csv) then it becomes a new file for your existing mapping. hence, you dont need to change the whole mapping just that you need to import it again.
Scenario, in error a new option set label of "Update" was added with value 100,000,000 to a field in the default managed solution in Production.
Identical label with value 866,100,002 was added to same field in the unmanaged Development Solution, when latest round of export and import occurred a duplicate option set label for "Update" was added to the managed solution in Production.
I now have data in the tables with both values and duplicate labels in the managed solution.
Question: how to unwind this mess - can I delete the label relating to value 100,000,000 seeing how it will just get duplicated upon the next solution import?
What happens to the data in the Database - is there a way to update the recorded values of 100,000,000 to a correct 866,100,002?
Do an Advanced Find for records having optionset value 100,000,000, export to excel (with selected option to reimport). Bulk update the exported records to the correct 866,100,002 optionset value & reimport it. This is first thing, it will correct the data.
Then you can delete the dupe label in picklist & monitor for future imports.
Test it in lower environments. Take solution/database backup as a precaution.
I am getting an "Violation of UNIQUE KEY constraint 'AK_User'. Cannot insert duplicate key in object 'dbo.tblUsers when trying to copy data from an excel file to sql db using SSIS.
Is there any way of ingnoring this error, and let the package continue to the next record without stopping?
What I need is if it inserts three records but the first record is a duplicate, instead of failing, it should continue with the other records and insert them.
There is a System variable called propagate which can be used to continue or stop the execution of package .
1.Create an ON-Error event handler for the task which is failing .Generally it is created for the entire Data Flow Task.
2.Press F4 to get the list of all variables and click on the Icon at the top
to show System Variable.By default Propagate variable will be True ,you need to change it to false ,which basically means that SSIS wont propagate the Error to other component and let the execution continue
Update 1:
To skip the bad rows there are basically 2 ways to do so :-
1.Use Lookup
Try to match the primary key column values in source and destination and then use Lookup No Match Output to your destination.If the value doesn't match with the destination then insert the rows else just skip the rows or redirect to some table or flat file using Lookup Match Output
Example
For more details on Lookup refer this article
2.Or you can redirect the error rows to a flat file or a table .Every SSIS Data Flow components has a Error Output .
For example for Derived component ,the error output dialogue box is
But this condition may not helpful to u in your case as redirect error rows in destination doesn't work properly .If an error occurs it redirects the entire data without inserting any row in the destination .I think this happens because OLEDB destination does a bulk insert or inserts data using transactions.So try to use lookup to achieve your functionality .