collection processing in BackgroundWorker - windows-phone-7

I try to make my ListBox connected to ObservaleCollection be more efficient so for the DB query I implemented a BackgroundWorker to do the job. Then whithin this backgroundworker I want to add every lets say 70 ms 3 entries to the UI, so the UI on larger number of entries (lets say 100) does not get blocked.
Here is the code:
void updateTMWorker_DoWork(object sender, DoWorkEventArgs e)
{
var MessagesInDB = from MessageViewModel tm in MessagesDB.Messages
where tm.Type.Equals(_type)
orderby tm.Distance
select tm;
// Execute the query and place the results into a collection.
Dispatcher.BeginInvoke(() => { MessagesClass.Instance.Messages = new ObservableCollection<MessageViewModel>(); });
Collection<MessageViewModel> tempM = new Collection<MessageViewModel>();
int tempCounter = 0;
foreach (MessageViewModel mToAdd in MessagesInDB)
{
if (MessagesClass.Instance.Messages.IndexOf(mToAdd) == -1)
{
tempM.Add(mToAdd);
tempCounter = tempCounter + 1;
}
if (tempCounter % 3 == 0)
{
tempCounter = 0;
Debug.WriteLine("SIZE OF TEMP:" + tempM.Count());
Dispatcher.BeginInvoke(() =>
{
// add 3 messages at once
MessagesClass.Instance.Messages.Add(tempM[0]);
MessagesClass.Instance.Messages.Add(tempM[1]);
MessagesClass.Instance.Messages.Add(tempM[2]);
});
tempM = new Collection<MessageViewModel>();
Thread.Sleep(70);
}
}
// finish off the rest
Dispatcher.BeginInvoke(() =>
{
for (int i = 0; i < tempM.Count(); i++)
{
MessagesClass.Instance.Messages.Add(tempM[i]);
}
});
}
The output is:
SIZE OF TEMP:3
A first chance exception of type 'System.ArgumentOutOfRangeException' occurred in mscorlib.dll
in the line: MessagesClass.Instance.Messages.Add(tempM[0]); where the code tries to access the first element of tempM
Any hints whats wrong? Why can't I access the tempM elements, although the collection size is > 0?

You forget about thread synchronization. Look at your code:
1: Debug.WriteLine("SIZE OF TEMP:" + tempM.Count());
Dispatcher.BeginInvoke(() =>
{
// add 3 messages at once
3: MessagesClass.Instance.Messages.Add(tempM[0]);
MessagesClass.Instance.Messages.Add(tempM[1]);
MessagesClass.Instance.Messages.Add(tempM[2]);
});
2: tempM = new Collection<MessageViewModel>();
tempM already will be null when MessagesClass.Instance.Messages.Add(tempM[0]); is executed. So, use some sort or synchronization objects, for example:
EventWaitHandle Wait = new AutoResetEvent(false);
Debug.WriteLine("SIZE OF TEMP:" + tempM.Count());
Dispatcher.BeginInvoke(() =>
{
// add 3 messages at once
MessagesClass.Instance.Messages.Add(tempM[0]);
MessagesClass.Instance.Messages.Add(tempM[1]);
MessagesClass.Instance.Messages.Add(tempM[2]);
Wait.Set();
});
// wait while tempM is not in use anymore
Wait.WaitOne();
tempM = new Collection<MessageViewModel>();

Related

BroadcastBlock missing items

I have a list of project numbers that I need to process. A project could have about 8000 items and I need to get the data for each item in the project and then push this data into a list of servers. Can anybody please tell me the following..
1) I have 1000 items in iR but only 998 were written to the servers. Did I loose items by using broadCastBlock?
2) Am I doing the await on all actionBlocks correctly?
3) How do I make the database call async?
Here is the database code
public MemcachedDTO GetIR(MemcachedDTO dtoItem)
{
string[] Tables = new string[] { "iowa", "la" };
using (SqlConnection connection = new SqlConnection(ConfigurationManager.ConnectionStrings["test"].ConnectionString))
{
using (SqlCommand command = new SqlCommand("test", connection))
{
DataSet Result = new DataSet();
command.CommandType = CommandType.StoredProcedure;
command.Parameters.Add("#ProjectId", SqlDbType.VarChar);
command.Parameters["#ProjectId"].Value = dtoItem.ProjectId;
connection.Open();
Result.EnforceConstraints = false;
Result.Load(command.ExecuteReader(CommandBehavior.CloseConnection), LoadOption.OverwriteChanges, Tables);
dtoItem.test = Result;
}
}
return dtoItem;
}
Update:
I have updated the code to the below. It just hangs when I run it and only writes 1/4 of the data to the server? Can you please let me know what I am doing wrong?
public static ITargetBlock<T> CreateGuaranteedBroadcastBlock<T>(IEnumerable<ITargetBlock<T>> targets, DataflowBlockOptions options)
{
var targetsList = targets.ToList();
var block = new ActionBlock<T>(
async item =>
{
foreach (var target in targetsList)
{
await target.SendAsync(item);
}
}, new ExecutionDataflowBlockOptions
{
CancellationToken = options.CancellationToken
});
block.Completion.ContinueWith(task =>
{
foreach (var target in targetsList)
{
if (task.Exception != null)
target.Fault(task.Exception);
else
target.Complete();
}
});
return block;
}
[HttpGet]
public async Task< HttpResponseMessage> ReloadItem(string projectQuery)
{
try
{
var linkCompletion = new ExecutionDataflowBlockOptions
{
MaxDegreeOfParallelism = 2
};
var cts = new CancellationTokenSource();
var dbOptions = new DataflowBlockOptions { CancellationToken = cts.Token };
IList<string> projectIds = projectQuery.Split(',').ToList();
IEnumerable<string> serverList = ConfigurationManager.AppSettings["ServerList"].Split(',').Cast<string>();
var iR = new TransformBlock<MemcachedDTO, MemcachedDTO>(
dto => dto.GetIR(dto), new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3 });
List<ActionBlock<MemcachedDTO>> actionList = new List<ActionBlock<MemcachedDTO>>();
List<MemcachedDTO> dtoList = new List<MemcachedDTO>();
foreach (string pid in projectIds)
{
IList<MemcachedDTO> dtoTemp = new List<MemcachedDTO>();
dtoTemp = MemcachedDTO.GetItemIdsByProject(pid);
dtoList.AddRange(dtoTemp);
}
foreach (string s in serverList)
{
var action = new ActionBlock<MemcachedDTO>(
async dto => await PostEachServerAsync(dto, s, "setitemcache"));
actionList.Add(action);
}
var bBlock = CreateGuaranteedBroadcastBlock(actionList, dbOptions);
foreach (MemcachedDTO d in dtoList)
{
await iR.SendAsync(d);
}
iR.Complete();
iR.LinkTo(bBlock);
await Task.WhenAll(actionList.Select(action => action.Completion).ToList());
return Request.CreateResponse(HttpStatusCode.OK, new { message = projectIds.ToString() + " reload success" });
}
catch (Exception ex)
{
return Request.CreateResponse(HttpStatusCode.InternalServerError, new { message = ex.Message.ToString() });
}
}
1) I have 1000 items in iR but only 998 were written to the servers. Did I loose items by using broadCastBlock?
Yes in the code below you set BoundedCapacity to one, if at anytime your BroadcastBlock cannot pass along an item it will drop it. Additionally a BroadcastBlock will only propagate Completion to one TargetBlock, do not use PropagateCompletion=true here. If you want all blocks to complete you need to handle Completion manually. This can be done by setting the ContinueWith on the BroadcastBlock to pass Completion to all of the connected targets.
var action = new ActionBlock<MemcachedDTO>(dto => PostEachServerAsync(dto, s, "set"), new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3, BoundedCapacity = 1 });
broadcast.LinkTo(action, linkCompletion);
actionList.Add(action);
Option: Instead of the BroadcastBlock use a properly bounded BufferBlock. When your downstream blocks are bound to one item they cannot receive additional items until they finish processing what they have. That will allow the BufferBlock to offer its items to another, possibly idle, ActionBlock.
When you add items into a throttled flow, i.e. a flow with a BoundedCapacity less than Unbounded. You need to be using the SendAsync method or at least handling the return of Post. I'd recommend simply using SendAsync:
foreach (MemcachedDTO d in dtoList)
{
await iR.SendAsync(d);
}
That will force your method signature to become:
public async Task<HttpResponseMessage> ReloadItem(string projectQuery)
2) Am I doing the await on all actionBlocks correctly?
The previous change will permit you to loose the blocking Wait call in favor of a await Task.WhenAlll
iR.Complete();
actionList.ForEach(x => x.Completion.Wait());
To:
iR.Complete();
await bufferBlock.Completion.ContinueWith(tsk => actionList.ForEach(x => x.Complete());
await Task.WhenAll(actionList.Select(action => action.Completion).ToList());
3) How do I make the database call async?
I'm going to leave this open because it should be a separate question unrelated to TPL-Dataflow, but in short use an async Api to access your Db and async will naturally grow through your code base. This should get you started.
BufferBlock vs BroadcastBlock
After re-reading your previous question and the answer from #VMAtm. It seems you want each item sent to All five servers, in that case you will need a BroadcastBlock. You would use a BufferBlock to distribute the messages relatively evenly to a flexible pool of servers that each could handle a message. None the less, you will still need to take control of propagating completion and faults to all the connected ActionBlocks by awaiting the completion of the BroadcastBlock.
To Prevent BroadcastBlock Dropped Messages
In general you two options, set your ActionBlocks to be unbound, which is their default value:
new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 3, BoundedCapacity = Unbounded });
Or broadcast messages your self from any variety of your own construction. Here is an example implementation from #i3arnon. And another from #svick

How to route, group, or otherwise split up messages into consistent sets using TPL Dataflow

I'm new to TPL Dataflow and I'm looking for a construct which will allow splitting up a list of source messages for evenly distributed parallel processing while maintaining order of the messages message through individual pipelines. Is there a specific Block or concept within the DataFlow API that can be used to accomplish this or is it more of a matter providing glue code or custom Blocks between existing Blocks?
For those familiar with Akka.NET I'm looking for functionality similar to the ConsistentHashing router which allow sending messages to a single router which then forwards these messages on to individual routees to be handled.
Synchronous example:
var count = 100000;
var processingGroups = 5;
var source = Enumerable.Range(1, count);
// Distribute source elements consistently and evenly into a specified set of groups (ex. 5) so that.
var distributed = source.GroupBy(s => s % processingGroups);
// Within each of the 5 processing groups go through each item and add 1 to it
var transformed = distributed.Select(d => d.Select(i => i + 3).ToArray());
List<int[]> result = transformed.ToList();
Check.That(result.Count).IsEqualTo(processingGroups);
for (int i = 0; i < result.Count; i++)
{
var outputGroup = result[i];
var expectedRange = Enumerable.Range(i + 1, count/processingGroups).Select((e, index) => e + (index * (processingGroups - 1)) + 3);
Check.That(outputGroup).ContainsExactly(expectedRange);
}
In general I don't think what you're looking for is pre-made in Dataflow as it may be with a ConsistentHashing router. However, by adding an id to the pieces of data you wish to flow you can process them in any order, in parallel and reorder them when the processing finishes.
public class Message {
public int MessageId { get; set; }
public int GroupId { get; set; }
public int Value { get; set; }
}
public class MessageProcessing
{
public void abc() {
var count = 10000;
var groups = 5;
var source = Enumerable.Range(0, count);
//buffer all input
var buffer = new BufferBlock<IEnumerable<int>>();
//split each input enumerable into processing groups
var messsageProducer = new TransformManyBlock<IEnumerable<int>, Message>(ints =>
ints.Select((i, index) => new Message() { MessageId = index, GroupId = index % groups, Value = i }).ToList());
//process each message, one action block may process any group id in any order
var processMessage = new TransformBlock<Message, Message>(msg =>
{
msg.Value++;
return msg;
}, new ExecutionDataflowBlockOptions() {
MaxDegreeOfParallelism = groups
});
//output of processed message values
int[] output = new int[count];
//insert messages into array in the order the started in
var regroup = new ActionBlock<Message>(msg => output[msg.MessageId] = msg.Value,
new ExecutionDataflowBlockOptions() {
MaxDegreeOfParallelism = 1
});
}
}
In the example the GroupId of a message isn't used but it could be used in a more complete example for coordinating groups of messages. Also, handling follow up posts to the bufferblock could be done by changing the output array to a List and setting up a corresponding list element each time an enumerable of integers is posted to the buffer block. Depending on your exact use, you may need to support multiple users of the output, and this can be folded back into the flow.
You can dynamically create a pipeline with linking the blocks between each other based on predicate:
var count = 100;
var processingGroups = 5;
var source = Enumerable.Range(1, count);
var buffer = new BufferBlock<int>();
var consumer1 = new ActionBlock<int>(i => { });
var consumer2 = new ActionBlock<int>(i => { });
var consumer3 = new ActionBlock<int>(i => { });
var consumer4 = new ActionBlock<int>(i => { Console.WriteLine(i); });
var consumer5 = new ActionBlock<int>(i => { });
buffer.LinkTo(consumer1, i => i % 5 == 1);
buffer.LinkTo(consumer2, i => i % 5 == 2);
buffer.LinkTo(consumer3, i => i % 5 == 3);
buffer.LinkTo(consumer4, i => i % 5 == 4);
buffer.LinkTo(consumer5);
foreach (var i in source)
{
buffer.Post(i);
// consider async option if you able to do it
// await buffer.SendAsync(i);
}
buffer.Complete();
Console.ReadLine();
The code above will write only numbers from 4th group, processing other groups silently, but I hope you got the idea. There is a general practice to link a block for at least one consumer without filtering for messages not being dropped if they aren't accepted by any consumers, and you can do this if you don't have a default handler (NullTarget<int> simply ignores all the messages it got):
buffer.LinkTo(DataflowBlock.NullTarget<int>());
The downside of this is a continuation of it's advantages: you have to provide predicates, as there is no built-in structures for this. However, it still could be done.

EWS The server cannot service this request right now

I am seeing errors while exporting email in office 365 account using ews managed api, "The server cannot service this request right now. Try again later." Why is that error occurring and what can be done about it?
I am using the following code for that work:-
_GetEmail = (EmailMessage)item;
bool isread = _GetEmail.IsRead;
sub = _GetEmail.Subject;
fold = folder.DisplayName;
historicalDate = _GetEmail.DateTimeSent.Subtract(folder.Service.TimeZone.GetUtcOffset(_GetEmail.DateTimeSent));
props = new PropertySet(EmailMessageSchema.MimeContent);
var email = EmailMessage.Bind(_source, item.Id, props);
bytes = new byte[email.MimeContent.Content.Length];
fs = new MemoryStream(bytes, 0, email.MimeContent.Content.Length, true);
fs.Write(email.MimeContent.Content, 0, email.MimeContent.Content.Length);
Demail = new EmailMessage(_destination);
Demail.MimeContent = new MimeContent("UTF-8", bytes);
// 'SetExtendedProperty' used to maintain historical date of items
Demail.SetExtendedProperty(new ExtendedPropertyDefinition(57, MapiPropertyType.SystemTime), historicalDate);
// PR_MESSAGE_DELIVERY_TIME
Demail.SetExtendedProperty(new ExtendedPropertyDefinition(3590, MapiPropertyType.SystemTime), historicalDate);
if (isread == false)
{
Demail.IsRead = isread;
}
if (_source.RequestedServerVersion == flagVersion && _destination.RequestedServerVersion == flagVersion)
{
Demail.Flag = _GetEmail.Flag;
}
_lstdestmail.Add(Demail);
_objtask = new TaskStatu();
_objtask.TaskId = _taskid;
_objtask.SubTaskId = subtaskid;
_objtask.FolderId = Convert.ToInt64(folderId);
_objtask.SourceItemId = Convert.ToString(_GetEmail.InternetMessageId.ToString());
_objtask.DestinationEmail = Convert.ToString(_fromEmail);
_objtask.CreatedOn = DateTime.UtcNow;
_objtask.IsSubFolder = false;
_objtask.FolderName = fold;
_objdbcontext.TaskStatus.Add(_objtask);
try
{
if (counter == countGroup)
{
Demails = new EmailMessage(_destination);
Demails.Service.CreateItems(_lstdestmail, _destinationFolder.Id, MessageDisposition.SaveOnly, SendInvitationsMode.SendToNone);
_objdbcontext.SaveChanges();
counter = 0;
_lstdestmail.Clear();
}
}
catch (Exception ex)
{
ClouldErrorLog.CreateError(_taskid, subtaskid, ex.Message + GetLineNumber(ex, _taskid, subtaskid), CreateInnerException(sub, fold, historicalDate));
counter = 0;
_lstdestmail.Clear();
continue;
}
This error occurs only if try to export in office 365 accounts and works fine in case of outlook 2010, 2013, 2016 etc..
Usually this is the case when exceed the EWS throttling in Exchange. It is explain in here.
Make sure you already knew throttling policies and your code comply with them.
You can find throttling policies using Get-ThrottlingPolicy if you have the server.
One way to solve the throttling issue you are experiencing is to implement paging instead of requesting all items in one go. You can refer to this link.
For instance:
using Microsoft.Exchange.WebServices.Data;
static void PageSearchItems(ExchangeService service, WellKnownFolderName folder)
{
int pageSize = 5;
int offset = 0;
// Request one more item than your actual pageSize.
// This will be used to detect a change to the result
// set while paging.
ItemView view = new ItemView(pageSize + 1, offset);
view.PropertySet = new PropertySet(ItemSchema.Subject);
view.OrderBy.Add(ItemSchema.DateTimeReceived, SortDirection.Descending);
view.Traversal = ItemTraversal.Shallow;
bool moreItems = true;
ItemId anchorId = null;
while (moreItems)
{
try
{
FindItemsResults<Item> results = service.FindItems(folder, view);
moreItems = results.MoreAvailable;
if (moreItems && anchorId != null)
{
// Check the first result to make sure it matches
// the last result (anchor) from the previous page.
// If it doesn't, that means that something was added
// or deleted since you started the search.
if (results.Items.First<Item>().Id != anchorId)
{
Console.WriteLine("The collection has changed while paging. Some results may be missed.");
}
}
if (moreItems)
view.Offset += pageSize;
anchorId = results.Items.Last<Item>().Id;
// Because you’re including an additional item on the end of your results
// as an anchor, you don't want to display it.
// Set the number to loop as the smaller value between
// the number of items in the collection and the page size.
int displayCount = results.Items.Count > pageSize ? pageSize : results.Items.Count;
for (int i = 0; i < displayCount; i++)
{
Item item = results.Items[i];
Console.WriteLine("Subject: {0}", item.Subject);
Console.WriteLine("Id: {0}\n", item.Id.ToString());
}
}
catch (Exception ex)
{
Console.WriteLine("Exception while paging results: {0}", ex.Message);
}
}
}

how to improve the performance of function which takes almost 1 second in loading time (asp.net mvc 3)

I did a profile tracing to check what functions are taking long times , One of the method takes near 1 second and is called 10+ times and i guess it should be a candidate for review. I have included the method, Can anyone tell me how can it possible be improved.
[NonAction]
private ProductModel.ProductMiscModel PrepareProductMiscModel(Product product)
{
if (product == null)
throw new ArgumentNullException("product");
var model = new ProductModel.ProductMiscModel();
var productVariants = _productService.GetProductVariantsByProductId(product.Id);
var getManufactureImage = _manufacturerService.GetProductManufacturersByProductId(product.Id)
.Select(x =>
{
var m = x.Manufacturer.ToModel();
m.PictureModel.ImageUrl = _pictureService.GetPictureUrl(x.Manufacturer.PictureId, _mediaSetting.ManufacturerThumbPictureSize, true);
m.PictureModel.Title = string.Format(_localizationService.GetResource("Media.Manufacturer.ImageLinkTitleFormat"), m.Name);
m.PictureModel.AlternateText = string.Format(_localizationService.GetResource("Media.Manufacturer.ImageAlternateTextFormat"), m.Name);
return m;
})
.ToList();
model.manufactureName = getManufactureImage;
switch (productVariants.Count)
{
case 0:
{
//var productVariant = productVariants[0];
model.Sku = null;
model.ShowSku = false;
// model.attributeName = 0;
} break;
case 1:
//only one variant
{ var productVariant = productVariants[0];
model.Sku = productVariant.Sku; //null;
model.ShowSku = true;
// model.attributeName = _productAttributeService.GetProductVariantAttributesByProductVariantId(productVariant.Id);
model.productSpecification = _specificationAttributeService.GetProductSpecificationAttributesByProductId(productVariant.Product.Id);
}
break;
}
return model;
}
_manufactureService.GetProductManufactureByProductId
public virtual IList<ProductManufacturer> GetProductManufacturersByProductId(int productId, bool showHidden = false)
{
if (productId == 0)
return new List<ProductManufacturer>();
string key = string.Format(PRODUCTMANUFACTURERS_ALLBYPRODUCTID_KEY, showHidden, productId);
return _cacheManager.Get(key, () =>
{
var query = from pm in _productManufacturerRepository.Table
join m in _manufacturerRepository.Table on
pm.ManufacturerId equals m.Id
where pm.ProductId == productId &&
!m.Deleted &&
(showHidden || m.Published)
orderby pm.DisplayOrder
select pm;
var productManufacturers = query.ToList();
return productManufacturers;
});
}
Use StopWatch in the method to determine which part it is that takes long time.
you might want to include the picture url in the original list instead of traversing each item and call _pictureService.GetPictureUrl.

Pivot control dynamically adding pivot items throws ArgumentException was unhandled when select a index of pivot

i have a problem to add pivot item in pivot control ..i want to add pivot item in pivot(Mycontrol) when i add next button .code for next button
static int selectitem;
private void nextbuttonOnclick(object sender, RoutedEventArgs e)
{
if (selectitem != list.Count() - 1)
select(selectitem + 1);
else
{
select(0);
}
}
now it call the select method to add pivot item to pivot
void select(int i)
{
MyControl.Items.Clear();
pivotItem = new PivotItem();
Grid sta = new Grid();
WebBrowser wb = new WebBrowser();
sta.Background = new SolidColorBrush(Colors.White);
var address = "<h3>" + list.ElementAt(i).header + "</h3>" + "<br>" + "<img width=\"949\" height=\"449\" src=" + list.ElementAt(i).Imagee + " >" + "<br>" + list.ElementAt(i).Detail.ToString();
var ByteData = Encoding.UTF8.GetBytes(address);
System.Text.Encoding enc = System.Text.Encoding.UTF8;
string myString = enc.GetString(ByteData, 0, ByteData.Length);
try
{
wb.Loaded += (sendr, ev) =>
{
wb.NavigateToString(myString);
};
}
catch (Exception ex)
{
}
wb.Margin = new Thickness(0, 0, 0, 0);
sta.Children.Add(wb);
pivotItem.Content = sta;
MyControl.Items.Add(pivotItem);
try
{
if (i == -1)
MyControl.SelectedIndex = 0;
else
{
selectitem = i;
MyControl.SelectedIndex = i;
}
}
catch (IndexOutOfRangeException v)
{
}
}
but after adding one it shows a exception ie ArgumentException was unhandled
...thanx in advance
The ArgumentException is happening because of this line:
MyControl.Items.Clear();
That line removes all the items in the pivot control, but your internal selectitem variable doesn't account for that. In your code, there will always be only one pivot item (which sort of defeats the purpose of the pivot control). When you go to add item #2, the selectitem variable is set to 1 and the line MyControl.SelectedIndex = 1; cases the exception since there is only 1 pivot item (0 based arrays, and all that stuff).
Try removing the line that clears the pivot items above and see if it works for you. Otherwise, you're going to have to change the MyControl.SelectedIndex = i; line to be MyControl.SelectedIndex = 0;.
Just FYI - when I ran your sample, I needed to initialize selecitem to -1, not 0, or I got the same error message. I also added code to set the PivotItem.Header property to something so it was easier to flip between pivot items.
Just out of curiosity, why are you not doing this through DataBinding instead of all this code to load up items manually?

Resources