WMI CommandLineEventConsumer Not Functioning - windows

I am attempting to learn more about WMI, and I am currently stuck wondering why my CommandLineEventConsumer does not launch. Here is a simple MOF script I adapted from microsoft documentations.
Sample MOF File that I ran with mofcomp.
// Set the namespace as root\subscription.
// The CommandLineEventConsumer is already compiled
// in the root\subscription namespace.
#pragma namespace ("\\\\.\\Root\\subscription")
class MyCmdLineConsumer
{
[key]string Name;
};
// Create an instance of the command line consumer
// and give it the alias $CMDLINECONSUMER
instance of CommandLineEventConsumer as $CMDLINECONSUMER
{
Name = "TestConsume";
CommandLineTemplate = "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe -ExecutionPolicy Bypass -File C:\\cmdline_test.ps1";
RunInteractively = True;
};
// Create an instance of the event filter
// and give it the alias $CMDLINEFILTER
// The filter queries for instance creation event
// for instances of the MyCmdLineConsumer class
instance of __EventFilter as $CMDLINEFILTER
{
Name = "TestFilter";
Query = "SELECT * FROM __InstanceCreationEvent"
" WHERE TargetInstance.__class = \"MyCmdLineConsumer\"";
QueryLanguage = "WQL";
};
// Create an instance of the binding
// between filter and consumer instances.
instance of __FilterToConsumerBinding
{
Consumer = $CMDLINECONSUMER;
Filter = $CMDLINEFILTER;
};
instance of LogFileEventConsumer as $CONSUMER
{
// If the file does not already exist, it is created.
Filename = "c:\\Logfile.log";
IsUnicode = FALSE;
// Name of this instance of LogFileEventConsumer
Name = "LogfileEventConsumer_Example";
// See "Using Standard String Templates";
Text = "%TargetInstance.Name%";
// TargetInstance is the instance of LogFileEvent
// requested in the filter
};
instance of __FilterToConsumerBinding
{
Consumer=$CONSUMER;
Filter=$CMDLINEFILTER;
DeliverSynchronously=FALSE;
};
instance of MyCmdLineConsumer
{
Name = "CmdLineEventConsumer Test";
};
Cmdline_test.ps1 just contains calc.exe, which I expected to show up. From my findings here, I am able see that LogFileEventConsumer is working because of the log that was created in C://Logfile.log. This proves to me that the EventFilter and Binding works, hence I am unsure why CommandLineEventConsumer does not work. I have also sieved through the WMI-activity/operational logs in Event Viewer but am unable to see it being triggered. Seeking some advice here!

Related

Replay a particular type of event from eventstore

I am currently using the Event Store to handle my events. I currently need to replay a particular type of event as I have made changes in the way they are subscribed and written to DB.
Is this possible? If so, how can it be done? Thanks.
You cannot tell EventStore to replay a specific event onto a persistent subscription because the point of the persistent subscription is to keep state for the subscribers.
To achieve this kind of fix you would really need a catch up application to do the work.
And really if you think about, if you replayed ALL the events to a new database then you would have the correct data in there?
So I have a console application that reuses the same logic as the persistent connection but the only difference is:
I change the target database connection string - So this would be a new Database or Collection (not the broken one)
It connects to EventStore and replays all the events from the start
It rebuilds the entire database to the correct state
Switch the business over to the new database
This is the point of EventStore - You just replay all the events to build any database at any time and it will be correct
Your persistent connections deal with new, incoming events and apply updates.
If you enable $by_event_type projection than you can access that projection stream under
/streams/$et-{event-type}
https://eventstore.org/docs/projections/system-projections/index.html
Then you can read it using .net api if you wish.
Here is some code to get you started
private static T GetInstanceOfEvent<T>(ResolvedEvent resolvedEvent) where T : BaseEvent
{
var metadataString = Encoding.UTF8.GetString(resolvedEvent.Event.Metadata);
var eventClrTypeName = JObject.Parse(metadataString).Property(EventClrTypeHeader).Value;
var #event = JsonConvert.DeserializeObject(Encoding.UTF8.GetString(resolvedEvent.Event.Data), Type.GetType((string) eventClrTypeName));
if (!(#event is BaseEvent))
{
throw new MessageDeserializationException((string) eventClrTypeName, metadataString);
}
return #event as T;
}
private static IEventStoreConnection GetEventStoreConnection()
{
var connectionString = System.Configuration.ConfigurationManager.ConnectionStrings["EventStore"].ConnectionString;
var connection = EventStoreConnection.Create(connectionString);
connection.ConnectAsync().Wait();
return connection;
}
private static string GetStreamName<T>() where T : BaseEvent
{
return "$et-" + typeof(T).Name;
}
And to read events you can use this code snippet
StreamEventsSlice currentSlice;
long nextSliceStart = StreamPosition.Start;
const int sliceCount = 200;
do
{
currentSlice = await esConnection.ReadStreamEventsForwardAsync(streamName, nextSliceStart, sliceCount, true);
foreach (var #event in currentSlice.Events)
{
var myEvent = GetInstanceOfEvent<OrderMerchantFeesCalculatedEvent>(#event);
TransformEvent(myEvent);
}
nextSliceStart = currentSlice.NextEventNumber;
} while (currentSlice.IsEndOfStream == false);

Firefox file downloading process structure figure

For my project, I need to study some info like "FireFox/Gecko file downloading structure overview"(if any), or somewhat "file downloading process flow chart of FireFox/Gecko". I couldn't find something like that in the Internet so far. Is there any info about it? Thanks a lot.
PS: It must include the paths about all file downloading through FireFox browser, which are via the network connection info APIs and file handling APIs, just like "httpOpenRequest" or "DoFileDownload" API(if any).
What would be the Firefox downloading process API paths?? Is there any figure or chart?
Please help me...
You are probably going to need to look at the code to get the information you desire. You will need to build the flowchart yourself.
There are a couple of different ways downloading is done in the code.
If you are talking about a Firefox add-on performing a download, then it is probably being done using Downloads.jsm (although there is an older method for doing so). The source code for that JavaScript module is at resource://gre/modules/Downloads.jsm (This URL is only valid in Firefox). There appear to be several files all located in the jsloader\resource\gre\modules directory within the zip format file called omni.ja in the root of the Firefox distribution. You can just copy that file and change the name to omni.zip and access it as a normal .zip file.
If you are wanting to know how Firefox saves a page when it is requested by the user: It is defined in the context menu with the oncommand value being gContextMenu.saveLink();. saveLink() is defined in: chrome://browser/content/nsContextMenu.js. It does some housekeeping and then calls saveHelper() which is in the same file.
The saveHelper() code is the following:
// Helper function to wait for appropriate MIME-type headers and
// then prompt the user with a file picker
saveHelper: function(linkURL, linkText, dialogTitle, bypassCache, doc) {
// canonical def in nsURILoader.h
const NS_ERROR_SAVE_LINK_AS_TIMEOUT = 0x805d0020;
// an object to proxy the data through to
// nsIExternalHelperAppService.doContent, which will wait for the
// appropriate MIME-type headers and then prompt the user with a
// file picker
function saveAsListener() {}
saveAsListener.prototype = {
extListener: null,
onStartRequest: function saveLinkAs_onStartRequest(aRequest, aContext) {
// if the timer fired, the error status will have been caused by that,
// and we'll be restarting in onStopRequest, so no reason to notify
// the user
if (aRequest.status == NS_ERROR_SAVE_LINK_AS_TIMEOUT)
return;
timer.cancel();
// some other error occured; notify the user...
if (!Components.isSuccessCode(aRequest.status)) {
try {
const sbs = Cc["#mozilla.org/intl/stringbundle;1"].
getService(Ci.nsIStringBundleService);
const bundle = sbs.createBundle(
"chrome://mozapps/locale/downloads/downloads.properties");
const title = bundle.GetStringFromName("downloadErrorAlertTitle");
const msg = bundle.GetStringFromName("downloadErrorGeneric");
const promptSvc = Cc["#mozilla.org/embedcomp/prompt-service;1"].
getService(Ci.nsIPromptService);
promptSvc.alert(doc.defaultView, title, msg);
} catch (ex) {}
return;
}
var extHelperAppSvc =
Cc["#mozilla.org/uriloader/external-helper-app-service;1"].
getService(Ci.nsIExternalHelperAppService);
var channel = aRequest.QueryInterface(Ci.nsIChannel);
this.extListener =
extHelperAppSvc.doContent(channel.contentType, aRequest,
doc.defaultView, true);
this.extListener.onStartRequest(aRequest, aContext);
},
onStopRequest: function saveLinkAs_onStopRequest(aRequest, aContext,
aStatusCode) {
if (aStatusCode == NS_ERROR_SAVE_LINK_AS_TIMEOUT) {
// do it the old fashioned way, which will pick the best filename
// it can without waiting.
saveURL(linkURL, linkText, dialogTitle, bypassCache, false,
doc.documentURIObject, doc);
}
if (this.extListener)
this.extListener.onStopRequest(aRequest, aContext, aStatusCode);
},
onDataAvailable: function saveLinkAs_onDataAvailable(aRequest, aContext,
aInputStream,
aOffset, aCount) {
this.extListener.onDataAvailable(aRequest, aContext, aInputStream,
aOffset, aCount);
}
}
function callbacks() {}
callbacks.prototype = {
getInterface: function sLA_callbacks_getInterface(aIID) {
if (aIID.equals(Ci.nsIAuthPrompt) || aIID.equals(Ci.nsIAuthPrompt2)) {
// If the channel demands authentication prompt, we must cancel it
// because the save-as-timer would expire and cancel the channel
// before we get credentials from user. Both authentication dialog
// and save as dialog would appear on the screen as we fall back to
// the old fashioned way after the timeout.
timer.cancel();
channel.cancel(NS_ERROR_SAVE_LINK_AS_TIMEOUT);
}
throw Cr.NS_ERROR_NO_INTERFACE;
}
}
// if it we don't have the headers after a short time, the user
// won't have received any feedback from their click. that's bad. so
// we give up waiting for the filename.
function timerCallback() {}
timerCallback.prototype = {
notify: function sLA_timer_notify(aTimer) {
channel.cancel(NS_ERROR_SAVE_LINK_AS_TIMEOUT);
return;
}
}
// set up a channel to do the saving
var ioService = Cc["#mozilla.org/network/io-service;1"].
getService(Ci.nsIIOService);
var channel = ioService.newChannelFromURI(makeURI(linkURL));
if (channel instanceof Ci.nsIPrivateBrowsingChannel) {
let docIsPrivate = PrivateBrowsingUtils.isWindowPrivate(doc.defaultView);
channel.setPrivate(docIsPrivate);
}
channel.notificationCallbacks = new callbacks();
let flags = Ci.nsIChannel.LOAD_CALL_CONTENT_SNIFFERS;
if (bypassCache)
flags |= Ci.nsIRequest.LOAD_BYPASS_CACHE;
if (channel instanceof Ci.nsICachingChannel)
flags |= Ci.nsICachingChannel.LOAD_BYPASS_LOCAL_CACHE_IF_BUSY;
channel.loadFlags |= flags;
if (channel instanceof Ci.nsIHttpChannel) {
channel.referrer = doc.documentURIObject;
if (channel instanceof Ci.nsIHttpChannelInternal)
channel.forceAllowThirdPartyCookie = true;
}
// fallback to the old way if we don't see the headers quickly
var timeToWait =
gPrefService.getIntPref("browser.download.saveLinkAsFilenameTimeout");
var timer = Cc["#mozilla.org/timer;1"].createInstance(Ci.nsITimer);
timer.initWithCallback(new timerCallback(), timeToWait,
timer.TYPE_ONE_SHOT);
// kick off the channel with our proxy object as the listener
channel.asyncOpen(new saveAsListener(), null);
}

Xamarin Social post without interface prompt - ShareItemAsync

I'm trying to post directly to facebook/twitter without prompting the user with a UIViewController using the following code:
// 1. Create the service
var facebook = new FacebookService {
ClientId = "<App ID from developers.facebook.com/apps>",
RedirectUrl = new System.Uri ("<Redirect URL from developers.facebook.com/apps>")
};
// 2. Create an item to share
var item = new Item { Text = "Xamarin.Social is the bomb.com." };
item.Links.Add (new Uri ("http://github.com/xamarin/xamarin.social"));
// 3. Present the UI on iOS
var shareController = facebook.GetShareUI (item, result => {
// result lets you know if the user shared the item or canceled
DismissViewController (true, null);
});
PresentViewController (shareController, true, null);
BUT the Xamarin.Social instructions say:
As an alternative to presenting the share UI, you can share items directly using the **ShareItemAsync** method of the service.
https://github.com/xamarin/Xamarin.Social
I can't find any examples or explicit tutorials on how to use this. Can anyone help me on this please?
If you look at the source of Xamarin.Social, internally ShareItemAsync is used anyways to carry out the requests. GetShareUI is just a wrapper around ShareItemAsync.
From the source of ShareViewController (which gets the UI), you can see how they are using ShareItemAsync to carry out the requests. Here's the snippet for you.
try {
service.ShareItemAsync (item, account).ContinueWith (shareTask => {
StopSharing ();
.
.
.
.
.
}, TaskScheduler.FromCurrentSynchronizationContext ());
}
So all you need to do is create the item, get hold of the account and call the method on the service, something like this
var item = new Item { Text = "Xamarin.Social is the bomb.com." };
item.Links.Add (new Uri ("http://github.com/xamarin/xamarin.social"));
var account = facebook.GetAccountsAsync().FirstOrDefault();
facebook.ShareItemAsync(item, account);

Dynamics CRM 2011 Bulk Update

Running Dynamics CRM 2011 rollout 3. Need to update millions of customer records periodically (delta updates). Using standard update (one by one) takes a few weeks. Also we don't want to touch the DB directly as it may break stuff in the future.
Is there a bulk update method in the Dynamics CRM 2011 webservice/REST API we can use? (WhatWhereHow)
I realize this is post is over 2 years old, but I can add to it in case someone else reads it and has a similar need.
Peter Majeed's answer is on target in that CRM processes requests one record at a time. There is no bulk edit that works the way you are looking for. I encourage you not to touch the DB directly if you need/want Microsoft support.
If you are looking at periodic updates of millions of records, you have a few options. Consider using Scribe or develop your own custom import utility or script using the CRM SDK.
Scribe is probably going to be your best option since it is cost effective for data imports and will allow you to easily update and insert from the same file.
If you write your own .Net/SDK based utility, I'd suggest making it multithreaded and either programmatically break up your input file in memory or on disk and have each thread work with its own subset of the data - that is, of course, if the order of execution does not have to be chronological according to the contents of the input file. If you can divide and conquer the input file over multiple threads, you can reduce the overall execution time considerably.
Also, if your corporate policy allows you to have access to one of the CRM Servers and you can place your code directly on the server and execute it from there - you can eliminate the network latency between a workstation running the code and the CRM web services.
Last but not least, if this large volume of import data is coming from another system, you can write a CRM plug-in to run on the Retrieve and RetrieveMultiple messages (events) in CRM for your specific entity, programmatically retrieve the desired data from the other system (and if the other system is unavailable - just use the cached copy in CRM), and keep CRM up to date in real-time or on a 'last-cached-on' basis. This is certainly more coding effort, but it potentially eliminates the need for a large synchronization job to be run every few weeks.
Yes and no, mostly no. Someone can correct me if I'm mistaken, in which case I'll gladly edit/delete my answer, but everything that's done in Dynamics CRM is done one at a time. It doesn't even try to handle set-based inserts/updates/deletes. So unless you go straight to direct DB operations, it will take you weeks.
The webservice does allow for "bulk" inserts/deletes/updates, but I put "bulk" in quotes because all it does is set up an asynchronous process where it does all the relevant data operations - yep - one at a time. There's a section of the SDK that addresses this sort of data management (linked). And to update the records this way, you'd have to first suffer the overhead of selecting all the data you want to update, then creating an xml file that contains the data, and finally updating the data (remember: one row at a time). So it would actually be more efficient to just loop through your data and issue an Update request for each yourself.
(I will note that our org hasn't experienced any memorable issues regarding direct DB access to handle what the SDK doesn't, nor have I seen anything in my personal internet readings that suggest others have.)
Edit:
See iFirefly's answer below for some other excellent ways to address this issue.
I realize this is an old question but it comes up high on "CRM Bulk Update" so the Update Rollup 12 feature ExecuteMultiple needs to be mentioned here -- its not going to work around your issue (massive volume) because as iFirefly and Peter point out CRM does everything one at a time. What it does do is package all your requests into a single envelope letting CRM handle the execution of each update and reduce the number of round trips between your app and the server if you do end up issuing an Update request for every record.
This is quite an old question, but nobody mentioned the fasted way (but also the most challenging) of updating/creating huge amounts of records in CRM 201X - using built-in import feature, which is totally doable using CRM SDK. There is a perfect MSDN article about that:
https://msdn.microsoft.com/en-us/library/gg328321(v=crm.5).aspx. In short you have to:
1) Build Excel file containing the data you want to import (simply export some data from CRM 201X and check how the structure looks like, remember that the first 3 columns are hidden)
2) create Import Map entity (specify the file you created)
3) Create column mappings if necessary
4) Create Import and ImportFile entity, providing proper mappings
5) Parse data using ParseImportRequest
6) Tranform data using TransformImportRequest
7) Import data using ImportRecordsImportRequest
This were the steps for CRM 2011, now in 2017 we have more versions available and there are slight differences between them. Check the sample that is available on MSDN and in SDK:
https://msdn.microsoft.com/en-us/library/hh547396(v=crm.5).aspx
Of course point 1, will be the most difficult part, because you have to build XML or docx file perfectly corresponding to what CRM expects, but I'm assuming you are doing it from external app, so you can use some great .NET libraries that will make things much simpler.
I never saw anything faster than standard CRM import when it comes to updating/creating records, even if you go for parallelism and Batch Update requests.
If something goes wrong with the MSDN sites, I'm posting here an example from the link above that is showing how to import data to CRM programatically:
using System;
using System.ServiceModel;
using System.Collections.Generic;
using System.Linq;
// These namespaces are found in the Microsoft.Xrm.Sdk.dll assembly
// located in the SDK\bin folder of the SDK download.
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Query;
using Microsoft.Xrm.Sdk.Client;
using Microsoft.Xrm.Sdk.Messages;
using Microsoft.Xrm.Sdk.Metadata;
// These namespaces are found in the Microsoft.Crm.Sdk.Proxy.dll assembly
// located in the SDK\bin folder of the SDK download.
using Microsoft.Crm.Sdk.Messages;
namespace Microsoft.Crm.Sdk.Samples
{
/// <summary>
/// This sample shows how to define a complex mapping for importing and then use the
/// Microsoft Dynamics CRM 2011 API to bulk import records with that mapping.
/// </summary>
public class ImportWithCreate
{
#region Class Level Members
private OrganizationServiceProxy _serviceProxy;
private DateTime _executionDate;
#endregion
/// <summary>
/// This method first connects to the organization service. Afterwards,
/// auditing is enabled on the organization, account entity, and a couple
/// of attributes.
/// </summary>
/// <param name="serverConfig">Contains server connection information.</param>
/// <param name="promptforDelete">When True, the user will be prompted to delete all
/// created entities.</param>
public void Run(ServerConnection.Configuration serverConfig, bool promptforDelete)
{
using (_serviceProxy = ServerConnection.GetOrganizationProxy(serverConfig))
{
// This statement is required to enable early bound type support.
_serviceProxy.EnableProxyTypes();
// Log the start time to ensure deletion of records created during execution.
_executionDate = DateTime.Today;
ImportRecords();
DeleteRequiredRecords(promptforDelete);
}
}
/// <summary>
/// Imports records to Microsoft Dynamics CRM from the specified .csv file.
/// </summary>
public void ImportRecords()
{
// Create an import map.
ImportMap importMap = new ImportMap()
{
Name = "Import Map " + DateTime.Now.Ticks.ToString(),
Source = "Import Accounts.csv",
Description = "Description of data being imported",
EntitiesPerFile =
new OptionSetValue((int)ImportMapEntitiesPerFile.SingleEntityPerFile),
EntityState = EntityState.Created
};
Guid importMapId = _serviceProxy.Create(importMap);
// Create column mappings.
#region Column One Mappings
// Create a column mapping for a 'text' type field.
ColumnMapping colMapping1 = new ColumnMapping()
{
// Set source properties.
SourceAttributeName = "src_name",
SourceEntityName = "Account_1",
// Set target properties.
TargetAttributeName = "name",
TargetEntityName = Account.EntityLogicalName,
// Relate this column mapping with the data map.
ImportMapId =
new EntityReference(ImportMap.EntityLogicalName, importMapId),
// Force this column to be processed.
ProcessCode =
new OptionSetValue((int)ColumnMappingProcessCode.Process)
};
// Create the mapping.
Guid colMappingId1 = _serviceProxy.Create(colMapping1);
#endregion
#region Column Two Mappings
// Create a column mapping for a 'lookup' type field.
ColumnMapping colMapping2 = new ColumnMapping()
{
// Set source properties.
SourceAttributeName = "src_parent",
SourceEntityName = "Account_1",
// Set target properties.
TargetAttributeName = "parentaccountid",
TargetEntityName = Account.EntityLogicalName,
// Relate this column mapping with the data map.
ImportMapId =
new EntityReference(ImportMap.EntityLogicalName, importMapId),
// Force this column to be processed.
ProcessCode =
new OptionSetValue((int)ColumnMappingProcessCode.Process),
};
// Create the mapping.
Guid colMappingId2 = _serviceProxy.Create(colMapping2);
// Because we created a column mapping of type lookup, we need to specify lookup details in a lookupmapping.
// One lookupmapping will be for the parent account, and the other for the current record.
// This lookupmapping is important because without it the current record
// cannot be used as the parent of another record.
// Create a lookup mapping to the parent account.
LookUpMapping parentLookupMapping = new LookUpMapping()
{
// Relate this mapping with its parent column mapping.
ColumnMappingId =
new EntityReference(ColumnMapping.EntityLogicalName, colMappingId2),
// Force this column to be processed.
ProcessCode =
new OptionSetValue((int)LookUpMappingProcessCode.Process),
// Set the lookup for an account entity by its name attribute.
LookUpEntityName = Account.EntityLogicalName,
LookUpAttributeName = "name",
LookUpSourceCode =
new OptionSetValue((int)LookUpMappingLookUpSourceCode.System)
};
// Create the lookup mapping.
Guid parentLookupMappingId = _serviceProxy.Create(parentLookupMapping);
// Create a lookup on the current record's "src_name" so that this record can
// be used as the parent account for another record being imported.
// Without this lookup, no record using this account as its parent will be imported.
LookUpMapping currentLookUpMapping = new LookUpMapping()
{
// Relate this lookup with its parent column mapping.
ColumnMappingId =
new EntityReference(ColumnMapping.EntityLogicalName, colMappingId2),
// Force this column to be processed.
ProcessCode =
new OptionSetValue((int)LookUpMappingProcessCode.Process),
// Set the lookup for the current record by its src_name attribute.
LookUpAttributeName = "src_name",
LookUpEntityName = "Account_1",
LookUpSourceCode =
new OptionSetValue((int)LookUpMappingLookUpSourceCode.Source)
};
// Create the lookup mapping
Guid currentLookupMappingId = _serviceProxy.Create(currentLookUpMapping);
#endregion
#region Column Three Mappings
// Create a column mapping for a 'picklist' type field
ColumnMapping colMapping3 = new ColumnMapping()
{
// Set source properties
SourceAttributeName = "src_addresstype",
SourceEntityName = "Account_1",
// Set target properties
TargetAttributeName = "address1_addresstypecode",
TargetEntityName = Account.EntityLogicalName,
// Relate this column mapping with its parent data map
ImportMapId =
new EntityReference(ImportMap.EntityLogicalName, importMapId),
// Force this column to be processed
ProcessCode =
new OptionSetValue((int)ColumnMappingProcessCode.Process)
};
// Create the mapping
Guid colMappingId3 = _serviceProxy.Create(colMapping3);
// Because we created a column mapping of type picklist, we need to specify picklist details in a picklistMapping
PickListMapping pickListMapping1 = new PickListMapping()
{
SourceValue = "bill",
TargetValue = 1,
// Relate this column mapping with its column mapping data map
ColumnMappingId =
new EntityReference(ColumnMapping.EntityLogicalName, colMappingId3),
// Force this column to be processed
ProcessCode =
new OptionSetValue((int)PickListMappingProcessCode.Process)
};
// Create the mapping
Guid picklistMappingId1 = _serviceProxy.Create(pickListMapping1);
// Need a picklist mapping for every address type code expected
PickListMapping pickListMapping2 = new PickListMapping()
{
SourceValue = "ship",
TargetValue = 2,
// Relate this column mapping with its column mapping data map
ColumnMappingId =
new EntityReference(ColumnMapping.EntityLogicalName, colMappingId3),
// Force this column to be processed
ProcessCode =
new OptionSetValue((int)PickListMappingProcessCode.Process)
};
// Create the mapping
Guid picklistMappingId2 = _serviceProxy.Create(pickListMapping2);
#endregion
// Create Import
Import import = new Import()
{
// IsImport is obsolete; use ModeCode to declare Create or Update.
ModeCode = new OptionSetValue((int)ImportModeCode.Create),
Name = "Importing data"
};
Guid importId = _serviceProxy.Create(import);
// Create Import File.
ImportFile importFile = new ImportFile()
{
Content = BulkImportHelper.ReadCsvFile("Import Accounts.csv"), // Read contents from disk.
Name = "Account record import",
IsFirstRowHeader = true,
ImportMapId = new EntityReference(ImportMap.EntityLogicalName, importMapId),
UseSystemMap = false,
Source = "Import Accounts.csv",
SourceEntityName = "Account_1",
TargetEntityName = Account.EntityLogicalName,
ImportId = new EntityReference(Import.EntityLogicalName, importId),
EnableDuplicateDetection = false,
FieldDelimiterCode =
new OptionSetValue((int)ImportFileFieldDelimiterCode.Comma),
DataDelimiterCode =
new OptionSetValue((int)ImportFileDataDelimiterCode.DoubleQuote),
ProcessCode =
new OptionSetValue((int)ImportFileProcessCode.Process)
};
// Get the current user to set as record owner.
WhoAmIRequest systemUserRequest = new WhoAmIRequest();
WhoAmIResponse systemUserResponse =
(WhoAmIResponse)_serviceProxy.Execute(systemUserRequest);
// Set the owner ID.
importFile.RecordsOwnerId =
new EntityReference(SystemUser.EntityLogicalName, systemUserResponse.UserId);
Guid importFileId = _serviceProxy.Create(importFile);
// Retrieve the header columns used in the import file.
GetHeaderColumnsImportFileRequest headerColumnsRequest = new GetHeaderColumnsImportFileRequest()
{
ImportFileId = importFileId
};
GetHeaderColumnsImportFileResponse headerColumnsResponse =
(GetHeaderColumnsImportFileResponse)_serviceProxy.Execute(headerColumnsRequest);
// Output the header columns.
int columnNum = 1;
foreach (string headerName in headerColumnsResponse.Columns)
{
Console.WriteLine("Column[" + columnNum.ToString() + "] = " + headerName);
columnNum++;
}
// Parse the import file.
ParseImportRequest parseImportRequest = new ParseImportRequest()
{
ImportId = importId
};
ParseImportResponse parseImportResponse =
(ParseImportResponse)_serviceProxy.Execute(parseImportRequest);
Console.WriteLine("Waiting for Parse async job to complete");
BulkImportHelper.WaitForAsyncJobCompletion(_serviceProxy, parseImportResponse.AsyncOperationId);
BulkImportHelper.ReportErrors(_serviceProxy, importFileId);
// Retrieve the first two distinct values for column 1 from the parse table.
// NOTE: You must create the parse table first using the ParseImport message.
// The parse table is not accessible after ImportRecordsImportResponse is called.
GetDistinctValuesImportFileRequest distinctValuesRequest = new GetDistinctValuesImportFileRequest()
{
columnNumber = 1,
ImportFileId = importFileId,
pageNumber = 1,
recordsPerPage = 2,
};
GetDistinctValuesImportFileResponse distinctValuesResponse =
(GetDistinctValuesImportFileResponse)_serviceProxy.Execute(distinctValuesRequest);
// Output the distinct values. In this case: (column 1, row 1) and (column 1, row 2).
int cellNum = 1;
foreach (string cellValue in distinctValuesResponse.Values)
{
Console.WriteLine("(1, " + cellNum.ToString() + "): " + cellValue);
Console.WriteLine(cellValue);
cellNum++;
}
// Retrieve data from the parse table.
// NOTE: You must create the parse table first using the ParseImport message.
// The parse table is not accessible after ImportRecordsImportResponse is called.
RetrieveParsedDataImportFileRequest parsedDataRequest = new RetrieveParsedDataImportFileRequest()
{
ImportFileId = importFileId,
PagingInfo = new PagingInfo()
{
// Specify the number of entity instances returned per page.
Count = 2,
// Specify the number of pages returned from the query.
PageNumber = 1,
// Specify a total number of entity instances returned.
PagingCookie = "1"
}
};
RetrieveParsedDataImportFileResponse parsedDataResponse =
(RetrieveParsedDataImportFileResponse)_serviceProxy.Execute(parsedDataRequest);
// Output the first two rows retrieved.
int rowCount = 1;
foreach (string[] rows in parsedDataResponse.Values)
{
int colCount = 1;
foreach (string column in rows)
{
Console.WriteLine("(" + rowCount.ToString() + "," + colCount.ToString() + ") = " + column);
colCount++;
}
rowCount++;
}
// Transform the import
TransformImportRequest transformImportRequest = new TransformImportRequest()
{
ImportId = importId
};
TransformImportResponse transformImportResponse =
(TransformImportResponse)_serviceProxy.Execute(transformImportRequest);
Console.WriteLine("Waiting for Transform async job to complete");
BulkImportHelper.WaitForAsyncJobCompletion(_serviceProxy, transformImportResponse.AsyncOperationId);
BulkImportHelper.ReportErrors(_serviceProxy, importFileId);
// Upload the records.
ImportRecordsImportRequest importRequest = new ImportRecordsImportRequest()
{
ImportId = importId
};
ImportRecordsImportResponse importResponse =
(ImportRecordsImportResponse)_serviceProxy.Execute(importRequest);
Console.WriteLine("Waiting for ImportRecords async job to complete");
BulkImportHelper.WaitForAsyncJobCompletion(_serviceProxy, importResponse.AsyncOperationId);
BulkImportHelper.ReportErrors(_serviceProxy, importFileId);
}
/// <summary>
/// Deletes any entity records that were created for this sample.
/// <param name="prompt">Indicates whether to prompt the user
/// to delete the records created in this sample.</param>
/// </summary>
public void DeleteRequiredRecords(bool prompt)
{
bool toBeDeleted = true;
if (prompt)
{
// Ask the user if the created entities should be deleted.
Console.Write("\nDo you want these entity records deleted? (y/n) [y]: ");
String answer = Console.ReadLine();
if (answer.StartsWith("y") ||
answer.StartsWith("Y") ||
answer == String.Empty)
{
toBeDeleted = true;
}
else
{
toBeDeleted = false;
}
}
if (toBeDeleted)
{
// Retrieve all account records created in this sample.
QueryExpression query = new QueryExpression()
{
EntityName = Account.EntityLogicalName,
Criteria = new FilterExpression()
{
Conditions =
{
new ConditionExpression("createdon", ConditionOperator.OnOrAfter, _executionDate),
}
},
ColumnSet = new ColumnSet(false)
};
var accountsCreated = _serviceProxy.RetrieveMultiple(query).Entities;
// Delete all records created in this sample.
foreach (var account in accountsCreated)
{
_serviceProxy.Delete(Account.EntityLogicalName, account.Id);
}
Console.WriteLine("Entity record(s) have been deleted.");
}
}
#region Main method
/// <summary>
/// Standard Main() method used by most SDK samples.
/// </summary>
/// <param name="args"></param>
static public void Main(string[] args)
{
try
{
// Obtain the target organization's web address and client logon
// credentials from the user.
ServerConnection serverConnect = new ServerConnection();
ServerConnection.Configuration config = serverConnect.GetServerConfiguration();
var app = new ImportWithCreate();
app.Run(config, true);
}
catch (FaultException<Microsoft.Xrm.Sdk.OrganizationServiceFault> ex)
{
Console.WriteLine("The application terminated with an error.");
Console.WriteLine("Timestamp: {0}", ex.Detail.Timestamp);
Console.WriteLine("Code: {0}", ex.Detail.ErrorCode);
Console.WriteLine("Message: {0}", ex.Detail.Message);
Console.WriteLine("Trace: {0}", ex.Detail.TraceText);
Console.WriteLine("Inner Fault: {0}",
null == ex.Detail.InnerFault ? "No Inner Fault" : "Has Inner Fault");
}
catch (System.TimeoutException ex)
{
Console.WriteLine("The application terminated with an error.");
Console.WriteLine("Message: {0}", ex.Message);
Console.WriteLine("Stack Trace: {0}", ex.StackTrace);
Console.WriteLine("Inner Fault: {0}",
null == ex.InnerException.Message ? "No Inner Fault" : ex.InnerException.Message);
}
catch (System.Exception ex)
{
Console.WriteLine("The application terminated with an error.");
Console.WriteLine(ex.Message);
// Display the details of the inner exception.
if (ex.InnerException != null)
{
Console.WriteLine(ex.InnerException.Message);
FaultException<Microsoft.Xrm.Sdk.OrganizationServiceFault> fe = ex.InnerException
as FaultException<Microsoft.Xrm.Sdk.OrganizationServiceFault>;
if (fe != null)
{
Console.WriteLine("Timestamp: {0}", fe.Detail.Timestamp);
Console.WriteLine("Code: {0}", fe.Detail.ErrorCode);
Console.WriteLine("Message: {0}", fe.Detail.Message);
Console.WriteLine("Trace: {0}", fe.Detail.TraceText);
Console.WriteLine("Inner Fault: {0}",
null == fe.Detail.InnerFault ? "No Inner Fault" : "Has Inner Fault");
}
}
}
// Additional exceptions to catch: SecurityTokenValidationException, ExpiredSecurityTokenException,
// SecurityAccessDeniedException, MessageSecurityException, and SecurityNegotiationException.
finally
{
Console.WriteLine("Press <Enter> to exit.");
Console.ReadLine();
}
}
#endregion Main method
}
}
Not sure how this would go with millions of records, but you can select your records, then click the Edit button in the ribbon. This will bring up the "Edit Multiple Records" dialog. Any changes you make will be applied to all your records.
The BulkUpdate API works well for me; it is 10 times faster than updating records one at a time. Following is a snippet that performs a bulk update:
public override ExecuteMultipleResponse BulkUpdate(List<Entity> entities)
{
ExecuteMultipleRequest request = new ExecuteMultipleRequest()
{
Settings = new ExecuteMultipleSettings()
{
ContinueOnError = true,
ReturnResponses = true
},
Requests = new OrganizationRequestCollection()
};
for (int i = 0; i < entities.Count; i++)
{
request.Requests.Add(new UpdateRequest() { Target = entities[i] });
}
return (ExecuteMultipleResponse) ServiceContext.Execute(request);
}
I worked on a very large data migration project for Dynamics CRM 2011. We needed to load about 3 million records over a weekend. I ended up building a console application (single thread) and ran multiple instances on multiple machines. Each console application had an id (1, 2, etc.) and was responsible for loading segments of the data based on a unique SQL WHERE clause that matched the application's id.
You could do the same thing with updates. Each instance could query a subset of the records to update and can perform the updates via the SDK. Since we loaded millions of records over a weekend I think you could perform millions of updates (if relatively small) in just a few hours.
Microsoft PFE team for dynamics CRM wrote
new Another CRM SDK library that make use of parallelization
to bulk execute requests ensuring thread safety.
You may try : Parallel Execute Requests
I would be interested to know if it works and scales to millions of records.
CRM doesn't implement a way to update bulk data; there are 3 ways to improve the bulk update operation performance but internally they cannot change the fact that CRM updates record one by one.
Basically the ideas are:
reduce the time wasted on communicating to CRM server
use parallelism to do multiple operations at the same time
make sure the update process does NOT trigger any workflows/plugins. Otherwise you might never see the end of the process...
3 ways to improve bulk operation performance:
After RollUp 12 there is a ExecuteMultipleRequest feature, which allows you to send up to 1000 requests at once. This means you may save some time from sending 1000 requests to CRM web service, however, these requests are processed one after another. So if your CRM server is well configured, most likely this method won't help too much.
You may use OrganizationServiceContext instance to do bulk update. OrganizationServiceContext implements unit of work pattern so you can do multiple updates and transmit these operations to the server in one call. Comparing to ExecuteMultipleRequest, it doesn't have a limit on request amount, but if it encounters a failure during the update, it will rollback all the changes.
Use multithreading or multitask. Either way would improve the speed, but they are likely to generate some connection failures or SQL errors, so you would need to add some retry logic in the code.
One of my client had exactly the same problem. He solved it by creating a custom ETL and doing parallelism attacking two front-end. The whole thing was made in C#. Nowaday, it could be possible with KingswaySoft or Scribe.

FindItems() and BindToItems() give inconsistent results for EmailMessage.Sender.Address

After quite a lot of debugging, I've refined a complicated Managed EWS problem down to the following two simple-ish test cases. The first one works, the second one fails:
var view = new ItemView(100) { PropertySet = new PropertySet { EmailMessageSchema.Id } };
var findResults = ews.FindItems(WellKnownFolderName.Inbox, view)
var bindResults = ews.BindToItems(findResults.Select(r => r.Id), new PropertySet { EmailMessageSchema.Sender });
// Sanity check
Assert.AreEqual(1, bindResults.Count());
// The results I care about
Assert.AreEqual("David Seiler", bindResults[0].Sender.Name);
Assert.AreEqual("david.seiler#yahoo.com", bindResults[0].Sender.Address);
One might try to cut out the BindToItems() call, and use FindItems() directly:
var view = new ItemView(100) { PropertySet = new PropertySet { EmailMessageSchema.Sender } };
var findResults = ews.FindItems(WellKnownFolderName.Inbox, view)
// This part still works fine
Assert.AreEqual(1, findResults.Count());
// So does this
Assert.AreEqual("David Seiler", findResults[0].Sender.Name);
// ...but this fails! Sender.Address is null
Assert.AreEqual("david.seiler#yahoo.com", findResults[0].Sender.Address);
Can anyone tell me where I've gone wrong? It really seems, from the documentation, as though this should work. Not all properties can be read through FindItems(), it's true, but those properties usually throw when I try to access them, and anyway there's a list of those properties on MSDN and Sender isn't on it. What's going on?
Actually I don't know why, but in the second option, it only load basic information of the sender like the name, but not the Address.
If you want to load all the sender properties but do not want to bind the full message you can add the following line before the first assert
service.LoadPropertiesForItems(findResults.Items, new PropertySet(EmailMessageSchema.Sender));

Resources