So there are a couple of similar questions, but all are using javascript, which isn't ideal as it requires the record to be opened / saved.
So, how can you calculate age based off birthdate. There are 200,000 records this would need to be done on and it's using CRM 2015, so can involve calculated fields as well.
It's going to be reported on in the background, so we can't use Javascript.
Workflows are a possibility, but running them on 200,000 records daily isn't exactly elegant!
Any other suggestions?
I've come across this requirement a number of times.
I've solved it by writing a Scribe or SSIS job which runs nightly and updates the Contact.Age field.
In order to not update every Contact record with the calculated age (as most ages won't have changed), I've used one of the following:
For on-premise CRM (where I have SQL access to the database), I wrote a query to return:
contactid
contact age
contact DoB
calculated age (calculated column from DoB and getdate)
The Scribe or SSIS job would only update records where Contact.Age != CalculatedAge
For hosted CRM (where I don't have SQL access to the DB):
Add a field called 'Next birthday'
The Scribe/SSIS job would search for records where NextBirthday is null or prior to today. It would update the Age and NextBirthday field.
Both of these methods mean that if the nightly job doesn't run for whatever reason, then when it's next run it will catch up on any records that are now out of date.
http://blogs.msdn.com/b/crm/archive/2009/02/27/creating-a-birthday-contact-list.aspx has an example using a pre-plugin to populate the birth month, year, and day fields. This could be adapted to instead perform the calculation to populate an age field. That being said, this would only work for new records or records that are changed.
If you wanted to do this via workflow, you'd have to have a workflow assembly to perform the calculation to populate an age field. As an alternative that doesn't require any code, you could create an Advanced Find query for All birthdays in a certain time frame, i.e. "Birthday on or before 2/17/1975" (this should limit the amount of records returned and reduce it from the total of the 200,000). Include the birthday and a new Age field created in the columns shown. I simply created an age field as a text field with a size of 5 characters since I'm intending only to store the years old someone is in it. Export the contacts to Excel marking the options, "Static worksheet with records from all pages in the current view" and "Make this data available for re-importing by including required column headings". Make sure to include the Owner column in order to prevent reassigning all these records to yourself when reimporting the records.
Then in Excel, create a formula like the following in the Age column assuming the birthdate field is in Column L, "=DATEDIF($L2,NOW(),"y")", which will update the age field with how many years old someone is as of the current date. Note that you might have to perform this calculation on a separate column and copy in just the values in order to ensure that Excel does not change the data type for the Age column or you will not be able to import that data back into Microsoft CRM. Fill that formula down so all records are updated, and save the file. Then in Microsoft CRM, import these records by pointing to the updated XML file (Excel 2003 XML format). Here your only restrictions are going to be on the size of the import file (CRM Limits this to 8 MB per file) and will be restricted to 10,000 records for the export, so this is another reason to break up the records you are exporting for reimport.
If you do update these via a workflow, you can update more than 250 at a time using a solution like the one in the CRM 2013 Bulk Workflow tool for XRM Toolbox http://www.zero2ten.com/blog/crm-2013-bulk-workflow-tool-for-xrmtoolbox/ , which allows you to select a group of records using FetchXML as the criteria for the records to apply the workflow to, noting that this may take some time to process if you are running this at the same time for all 200,000 records.
Ideally, my preference would be to have a plugin or JavaScript, but can see with your requirements that you would need to have this run either daily or on a monthly basis (although I would not run it for all 200,000 since everyone's age does not change each day). Just choose the records that have birthdays in a particular month or on a particular date to run the workflows on or to export and reimport for since that's going to be much less intensive for server processing and will be able to complete much faster than having to update all 200,000 at a time.
Related
I'm having a strange issue with exporting/updating/importing data in our on-premises Dynamics 365 (8.2). I was doing a bulk update of over 3000 records by exporting the records to an Excel workbook, updating the data in a specific column, then importing the workbook back into CRM. It worked for all of the records except 14 of them, which according to the import log was for the reason that "You cannot import data to this record because the record was updated in Microsoft Dynamics 365 after it was exported." I looked at the Audit History of those 14 records, and find that they have not been modified in any way for a good two months. Strangely, the modified date of the most recent Audit History entry for ALL 14 records is the exact same date/time.
We have a custom workflow that runs once every 24 hours on a schedule that automatically updates the Age field of our Contact records based on the value in the respective Birthday field. For these 14 records, ALL of them have a birthday of November 3rd, but in different years. What that means though is that the last modification that was done to them was on 11/3/2019 via the workflow. However, I cannot understand why the system "thinks" that this should prevent a data update/import.
I am happy to provide any additional information that I may have forgotten to mention here. Can anyone help me, please?
While I was not able to discover why the records would not update, I was able to resolve the issue. Before I share what I did to update the records, I will try and list as many things as I can remember that I tried that did not work:
I reworked my Advanced Find query that I was using to export the records that needed updated to return ONLY those records that had actual updates. Previously, I used a more forgiving query that returned about 30 or so records, even though I knew that only 14 of them had new data to import. I did so because the query was easier to construct, and it was no big deal to remove the "extra" records from the workbook before uploading it for import. I would write a VLOOKUP for the 30-something records, and remove the columns for which the VLOOKUP didn't find a value in my dataset, leaving me with the 14 that did have new data. After getting the error a few times, I started to ensure that I only exported the 14 records that needed to be updated. However, I still got the error when trying to import.
I tried formatting the (Do Not Modify) Modified On column in the exported workbook to match the date format in the import window. On export of the records, Excel was formatting this column as m/d/yyyy h:mm while the import window with the details on each successful and failed import showed this column in mm/dd/yyyy hh:mm:ss format. I thought maybe if I matched the format in Excel to the import window format it might allow the records to import. It did not.
I tried using some Checksum verification tool to ensure that the value in the (Do Not Modify) Checksum column in the workbook wasn't being written incorrectly or in an invalid format. While the tool I used didn't actually give me much useful information, it did recognize that the values were checksum hashes, so I supposed that was helpful enough for my purposes.
I tried switching my browser from the new Edge browser (the one that uses Chromium) to just IE as suggested on the thread provided by Arun. However, it did not resolve the issue.
What ended up working in the end was Arun's suggestion to just do some arbitrary edit to all the records and exporting them afterward. This was okay to do for just 14 records, but I'm still slightly vexed as this wouldn't really be a feasible solution of it were, say, a thousand records that were not importing. There was no field that ALL 14 Contact records had in common that I could just bulk edit, and bulk edit back again. What I ended up doing was finding a text field on the Contact Form that did not have any value in it for any of the records, putting something in that field, then going to each record in turn and removing the value (since I don't know of a way to "blank out" or clear a text field while bulk editing. Again, this was okay for such a small number of records, but if it were to happen on a larger number, I would have to come up with an easier way to bulk edit and then bulk "restore" the records. Thanks to Arun for the helpful insights, and for taking the time to answer. It is highly appreciated!
When you first do an import of an entity (contacts for example) you see that your imported excel contains 3 hidden columns (Do Not Modify) Contact, (Do Not Modify) Row Checksum, (Do Not Modify) Modified On.
When you want to create new instances of the entity, just edit the records and clear the content of the 3 hidden colums.
This error will happen when there is a checksum difference or rowversion differs from the exported record vs the record in database.
Try to do some dummy edit for those affected records & try to export/reimport again.
I could think of two reasons - either the datetime format confusing the system :( or the the community thread explains a weird scenario.
Apparently when importing the file, amending and then saving as a different file type alters the spreadsheet's parameters.
I hence used Internet Explorer since when importing the file, the system asks the user to save as a different format. I added .xlsx at the end to save it as the required format. I amended the file and imported it back to CRM..It worked
For me it turned out to be a different CRM time zone setting for the exporter and importer. Unfortunately this setting doesn't seem to be able to be changed by an administrator via the user interface.
The setting is available for each user under File->Options->Time Zone.
Currently, am trying to retrieve data from this page: https://www.hdb.gov.sg/cs/infoweb/residential/renting-a-flat/renting-from-the-open-market/rental-statistics , as you can see, there are 4 quarters in a year, and for each quarter, there is a different table. I wish to extract the table but currently, i am unable to automate the process, only able to take one. On top of that, i wish to add two columns to the retrieved data table which is "Quarter" and "Year". Any suggestions? Attached photos are my workflow and my excel.
Get the number of years/ loop through the years (or start with the 1st year up to the last year).
For each year try to get the data via data scraping (the elements exist, just hidden/not expanded ; do one table datascraping for data modelling and reuse it within the loop). For the datascraping you need to change the selector, to make it usable for all tables by using the year and the quarter (just a generic example, like * year * quarter *). Columns are the same for all tables.
I haven't seen details within the website menu or within the page, is good to check if robots are allowed to scrape for data
Above would be the quickest way. More complex with FindChidren activity.
I want to scrub(or encrypt) the email information from a few tables which are older than a few years.
This I am planning to do as part of a job, next time when I run the job how can I omit the rows which are already scrubbed or encrypted.
I am looking for an approach which will be having good performance.
"I want to scrub(or encrypt) the email information from a few tables which are older than a few years"
I hope this means you have a date column on these tables which you can use to determine which ones need to be scrubbed. The most efficient way of tackling the job is to track that date in an operational table, recording the most recent date scrubbed.
For example you have ten years' worth of data, and you need to scrub records which are more than four years old. Now this would work:
update t23
set email = null
where date_created < add_months(sysdate, -48);
But it seems like you want to batch things up. So build a tracking table, which at its simplest would be
create table tracker (
last_date_scrubbed);
Populate the last_date_scrubbed with a really old date say date '2010-01-01'
Now you can write a query like this
update t23
set email = null
where date_created
< (select last_date_scrubbed + interval '1' year from tracker);
That will clean all records older than 2011. Increment the date in the tracker table by one year. Run the query again to clean stuff from 2011. Repeat until you get to your target state of cleanliness. At which point you can switch to running the query monthly , with an interval of one month , or whatever.
Obviously you should proceduralize this. A procedure is the best way to encapsulate the steps and make sure everything is kept in step. Also you can use the database scheduler to run the procedure.
"there is one downside to this approach. I thought that you want to be free upon choosing which rows to be updated."
I don't see any requirement to track which individual rows have been scrubbed. After all, the end state is that every record older than a certain date has been scrubbed. When I have done jobs like this previously all anybody wanted to know was, "how many rows have we done so far and how many have we still got to do?" Which can be answered by tracking the sql%rowcount for each run.
For The best performance, you can add a Flag Column to your main table. a Column like IsEncrypted. then every time you try to run any query for the "not Encrypted rows" you easily use WHERE when IsEncrypted Column is false to condition on those rows only. there are other ways though.
EDIT
another way is to create a logger table. basically what this table does, is that it records any more information you want about a certain ID in another table. have another table called EncryptionLogger, in it you would have at least two columns: EmailTableId, IsEncrypted. then in any query you can simply get any rows WHERE their Ids are NOT IN this table.
I'm trying to put together an attendance report for a school that tracks student attendance codes for that student for every day on the calendar month in a DynamicsCRM system being used as a managed service (that is to say, I build queries using FetchXML and cannot use SQL). The format for the report requires that a column for every day in the month be listed for the report. My student table that tracks this attendance however only contains records for days where an attendance value is recorded, and I do not have an object available that can return every day in a month for me.
I am looking for a solution other than hardcoding 31 columns and using conditionals to control the display of the last three day columns. Ideally, I'd like a conditional in my matrix column grouping that would look at the date value for the previously generated column and determine if the next date record from my resultset is sequentially the next day of that month, and if not, create the next sequential date, move to the next column and perform the check again until it is true. Is there a way I can do this, or another means to accomplish my goal that does not involve hard-coding day columns into a table or matrix? Right now, I have nothing; I can barely imagine how I think this should look.
What I did to solve the same issue has been to create a scheduled process each day to create a record and deactivate it.
I have then been able to distinguish actual records (the active ones) from these 'placeholders' (inactive ones) in my querying.
My app has a class that saves picture that users upload. Each object in the class has a city property that holds the name of the city that the picture was taken at, and a like property that tracks the number of likes.
I want to be able to send a query that returns one picture per city and each picture should have the highest ranking of likes in the city it belongs to. How can I do that?
One way which I first thought about is doing multiple queries by fetching the most liked picture of a city and save it in an array, and then do the same to other cities.
However, each country has more than one city, thus it's not that efficient.
Parse doesn't support the ordinary operations used in databases. Besides, I tried to use a compound query. Unfortunately, I can't set limit or ordering on the subqueries. Any good solution for this?
It would be easy using group by. Unfortunately, Parse does not support "select distinct" or "group by" features.
As you've suggested you need to fetch for each country all the cities, and for each one get the top most rated photo.
BUT, since Parse has strict restrictions on the duration time execution of a request ( 3 sec for an event listener, 7 sec for a custom function ), I suggest you to do this in a background job, saving in a new table the top rated photo for each city. In this way you can easily query the db from client. The Background jobs can be executed up to 15 minuted before parse drop them, so you could make that kind of queries without timeouts.
Hope it helps