We have an Outlook addin which does the following
User clicks on a button to save the email onto the server.
During saving, we upload the PR_INTERNET_MESSAGE_ID_W_TAG of the email to the server.
The addin also has a System.Timer timer object, which fires every 15 mins. Within this timer, we call an async function.
The async function calls the API, retrieves a list of unique ids and the addin sets / resets the icon for the email.
We are facing a problem, when the user downloads 300 000 emails from Exchange to Outlook, it takes a lot of time to process them. 20 mins for 2994 emails and Outlook becomes very slow, and at times unresponsive.
Timer Code
private async void OnProcessEmails(object sender, EventArgs e)
{
await mailItemHer.SetIconAync();
}
Part of SetIconAync() code
// lstUniqueValues are a list of PR_INTERNET_MESSAGE_ID_W_TAG returned by the server
// Count of lstUniqueValues = 2994
foreach (string uniqueValue in lstUniqueValues)
{
string filter = "#SQL=" + "http://schemas.microsoft.com/mapi/proptag/0x1035001F" + " = '" + uniqueValue + "'"; var itemToProcess = itmsInInbox.Find(filter);
// Count of itmsInInbox = 300000
var itemToProcess = itmsInInbox.Find(filter);
if (itemToProcess != null && (itemToProcess is MailItem))
{
MailItem mailItm = itemToProcess as MailItem;
var iconProperty = mailItm.PropertyAccessor.GetProperty("http://schemas.microsoft.com/mapi/proptag/0x10800003");
if ((int)iconProperty != iconValue)
{
mailItm.PropertyAccessor.SetProperty("http://schemas.microsoft.com/mapi/proptag/0x10800003", iconValue);
mailItm.Save();
}
if(mailItm !=null)
{
Marshal.ReleaseComObject(mailItm);
mailItm = null;
}
}
}
Please suggest an approach which can solve this issue. I am also open to any other suggestions.
Update
There is no user intervention ( ie clicking on the email, or any button) , as this takes place periodically within the timer Elapsed event.
The code can be made a little better, but still it shall take time
MailItem mailItm = itemToProcess as MailItem;
// filter out only the actual emails
if(mailItm.MessageClass == "IPM.Note)
{
// code
A few suggestions:
You can combine several queries on message id into a single "OR" query and process multiple matches that it returns.
When you remove the icon, you don't have to remote the message from a cache - you can store a dictionary of message ids vs entry ids locally (file system or as a hidden message in an Outlook folder accessible using MAPIFolder.GetStorage in Outlook Object Model). Then when access is restored, you can open messages by their entry ids using Namespace.GetItemFromID instead of searching.
You can try to run search queries on a secondary thread. Outlook Object Model can only be used on the primary thread when running inside the outlook.exe process. You can use Extended MAPI on a secondary thread, but it is only accessible in unmanaged C++ or or Delphi. If using Redemption is an option (I am its author), you can perform searches on a secondary thread - save the value of Namespace.MAPIOBJECT on the primary thread (it is the IMAPISession MAPI interface), create an instance of the RDOSession object on a secondary thread (that will initialize the MAPI system on that thread), and assign the value saved on the primary thread to RDOSession.MAPIOBJECT property (that will ensure that both Outlook and Redemption share the same MAPI session). You can then open the target folder using RDOSession.GetFolderFromID and search using RDOFolder.Items.Find/FindNext/Restrict (do not use the "#SQL=" prefix)
Related
I'm writing a small, internal web application that reads in form data and creates an excel file which then gets emailed to the user.
However, I'm struggling to understand how I can implement real-time updates for the user as the process is being completed. Sometimes the process takes 10 seconds, and sometimes the process takes 5 minutes.
Currently the user waits until the process is complete before they see any results - They do not see any updates as the process is being completed. The front-end waits for a 201 response from the server before displaying the report information and the user is "blocked" until the RC is complete.
I'm having difficulty understanding how I can asynchronously start the Report Creation (RC) process and at the same time allow the user to navigate to other pages of the site. or see updates happening in the background. I should clarify here that the some of the steps in the RC process use Promises.
I'd like to poll the server every second to get an update on the report being generated.
Here's some simple code to clarify my understanding:
Endpoints
// CREATE REPORT
router.route('/report')
.post(function(req, res, next) {
// Generate unique ID to keep track of report later on.
const uid = generateRandomID();
// Start report process ... this should keep executing even after a response (201) is returned.
CustomReportLibrary.createNewReport(req.formData, uid);
// Respond with a successful creation.
res.status(201);
}
}
// GET REPORT
router.route('/report/:id')
.get(function(req, res, next){
// Get our report from ID.
let report = CustomReportLibrary.getReport(req.params.id);
// Respond with report data
if(report) { res.status(200).json(report); }
else { res.status(404); }
}
CustomReportLibrary
// Initialize array to hold reports
let _dataStorage = [];
function createNewReport(data, id) {
// Create an object to store our report information
let reportObject = {
id: id,
status: 'Report has started the process',
data: data
}
// Add new report to global array.
_dataStorage.push(reportObject);
// ... continue with report generation. Assume this takes 5 minutes.
// ...
// ... update _dataStorage[length-1].status after each step
// ...
// ... finish generation.
}
function getReport(id) {
// Iterate through array until report with matching ID is found.
// Return report if match is found.
// Return null if no match is found.
}
From my understanding, CustomerReportLibrary.createNewReport() will execute in the background even after a 201 response is returned. In the front-end, I'd make an AJAX call to /report/:id on an interval basis to get updates on my report. Is this the right way to do this? Is there a better way to do this?
I think you are on the right way. HTTP 202 (The request has been accepted for processing, but the processing has not been completed) is a proper way to handle your case.
It can be done like this:
client sends POST /reports, server starts creating new report and returns:
202 Accepted
Location: http://api.domain.com/reports/1
client issues GET /reports/1 to get status of the report
All the above flow is async, so users are not blocked.
I am using a simple form to allow people to sign up for an event. Their details are saved to a Sharepoint list. I have a quota of people who can sign up for an event (say 100 people).
How can I prevent the 100th and the 101st person from signing up concurrently, causing the quota check to allow the 101st person to sign up (because the 100th person isn't in the list yet)?
Place the ItemAdding code inside a lock statement to make sure that only one thread at a time can enter the critical section of code:
private Object _lock = new Object();
public override void ItemAdding(SPItemEventProperties properties)
{
lock(_lock)
{
// check number of the list items and cancel the event if necessary
}
}
I came up with this idea of a solution for a farm with multiple WFEs - a shared resource (a row in a table in pseudo-code above) gets locked during the time the item is added to the list:
private Object _lock = new Object();
public override void ItemAdding(SPItemEventProperties properties)
{
try
{
// 1. begin a SQL Server transaction
// 2. UPDATE dbo.SEMAPHORE
// SET STATUS = 'Busy'
// WHERE PROCESS = 'EventSignup'
lock(_lock)
{
// 3. check number of the list items and cancel the event if necessary
}
}
finally
{
// 4. UPDATE dbo.SEMAPHORE
// SET STATUS = ''
// WHERE PROCESS = 'EventSignup'
// 5. commit a SQL Server transaction
}
}
I left the lock statement because I'm not sure what will happen if the same front-end server tries to add the item #100 and #101 - will the transaction lock the row or will it not because the same connection to SQL Server will be used?
So then you can use event receivers item adding method. at item adding, your item is not created, you can calculate the current count of signed up people. if it is bigger then 100 you can cancel item adding.
but sure, more than one item adding method can be fired, to prevent that you can calculate the current count of people and increase the count +1, and keep that value somewhere else (on a field on event item perhaps) and all item adding methods can check that value before adding the item.
item added method is too late for these operations.
this would be the solution i would use.
I guess if you are updating a column, lets say - "SignUp Count", then one of the users will get the Save Conflict issue. Whoever updated the value for the first time wins and the second one will fail.
Regards,
Nitin Rastogi
I have a bit of linq to entities code in a web app. It basically keeps a count of how many times an app was downloaded. I'm worried that this might happen:
Session 1 reads the download count (eg. 50)
Session 2 reads the download count (again, 50)
Session 1 increments it and writes it to the db (database stores 51)
Session 2 increments it and writes it to the db (database stores 51)
This is my code:
private void IncreaseHitCountDB()
{
JTF.JTFContainer jtfdb = new JTF.JTFContainer();
var app =
(from a in jtfdb.Apps
where a.Name.Equals(this.Title)
select a).FirstOrDefault();
if (app == null)
{
app = new JTF.App();
app.Name = this.Title;
app.DownloadCount = 1;
jtfdb.AddToApps(app);
}
else
{
app.DownloadCount = app.DownloadCount + 1;
}
jtfdb.SaveChanges();
}
Is it possible that this could happen? How could I prevent it?
Thank you,
Fidel
Entity Framework, by default, uses an optimistic concurrency model. Google says optimistic means "Hopeful and confident about the future", and that's exactly how Entity Framework acts. That is, when you call SaveChanges() it is "hopeful and confident" that no concurrency issue will occur, so it just tries to save your changes.
The other model Entity Framework can use should be called a pessimistic concurrency model ("expecting the worst possible outcome"). You can enable this mode on an entity-by-entity basis. In your case, you would enable it on the App entity. This is what I do:
Step 1. Enabling concurrency checking on an Entity
Right-click the .edmx file and choose Open With...
Choose XML (Text) Editor in the popup dialog, and click OK.
Locate the App entity in the ConceptualModels. I suggest toggling outlining and just expanding tags as necessary. You're looking for something like this:
<edmx:Edmx Version="2.0" xmlns:edmx="http://schemas.microsoft.com/ado/2008/10/edmx">
<!-- EF Runtime content -->
<edmx:Runtime>
<!-- SSDL content -->
...
<!-- CSDL content -->
<edmx:ConceptualModels>
<Schema Namespace="YourModel" Alias="Self" xmlns:annotation="http://schemas.microsoft.com/ado/2009/02/edm/annotation" xmlns="http://schemas.microsoft.com/ado/2008/09/edm">
<EntityType Name="App">
Under the EntityType you should see a bunch of <Property> tags. If one exists with Name="Status" modify it by adding ConcurrencyMode="Fixed". If the property doesn't exist, copy this one in:
<Property Name="Status" Type="Byte" Nullable="false" ConcurrencyMode="Fixed" />
Save the file and double click the .edmx file to go back to the designer view.
Step 2. Handling concurrency when calling SaveChanges()
SaveChanges() will throw one of two exceptions. The familiar UpdateException or an OptimisticConcurrencyException.
if you have made changes to an Entity which has ConcurrencyMode="Fixed" set, Entity Framework will first check the data store for any changes made to it. If there are changes, a OptimisticConcurrencyException will be thrown. If no changes have been made, it will continue normally.
When you catch the OptimisticConcurrencyException you need to call the Refresh() method of your ObjectContext and redo your calculation before trying again. The call to Refresh() updates the Entity(s) and RefreshMode.StoreWins means conflicts will be resolved using the data in the data store. The DownloadCount being changed concurrently is a conflict.
Here's what I'd make your code look like. Note that this is more useful when you have a lot of operations between getting your Entity and calling SaveChanges().
private void IncreaseHitCountDB()
{
JTF.JTFContainer jtfdb = new JTF.JTFContainer();
var app =
(from a in jtfdb.Apps
where a.Name.Equals(this.Title)
select a).FirstOrDefault();
if (app == null)
{
app = new JTF.App();
app.Name = this.Title;
app.DownloadCount = 1;
jtfdb.AddToApps(app);
}
else
{
app.DownloadCount = app.DownloadCount + 1;
}
try
{
try
{
jtfdb.SaveChanges();
}
catch (OptimisticConcurrencyException)
{
jtfdb.Refresh(RefreshMode.StoreWins, app);
app.DownloadCount = app.DownloadCount + 1;
jtfdb.SaveChanges();
}
}
catch (UpdateException uex)
{
// Something else went wrong...
}
}
You can prevent this from happenning if you only query the download count column right before you are about to increment it, the longer the time spent between reading and incrementing the longer the time another session has to read it (and later rewriting - wrongly - incremented number ) and thus messing up the count.
with a single SQL query :
UPDATE Data SET Counter = (Counter+1)
since its Linq To Entities,it means delayed execution,for another session to screw up the Count (increment the same base,losing 1 count there) it would have to try to increment the app.Download count i beleive between the two lines:
else
{
app.DownloadCount += 1; //First line
}
jtfdb.SaveChanges(); //Second line
}
thats means that the window for the change to occur, thus making the previous count old, is so small that for an application like this is virtually impossible.
Since Im no LINQ pro, i dont know whether LINQ actually gets app.DownLoadCount before adding one or just adds one through some SQL command, but in either case you shouldnt have to worry about that imho
You could easily test what would happen in this scenario - start a thread, sleep it, and then start another.
else
{
app.DownloadCount = app.DownloadCount + 1;
}
System.Threading.Thread.Sleep(10000);
jtfdb.SaveChanges();
But the simple answer is that no, Entity Framework does not perform any concurrency checking by default (MSDN - Saving Changes and Managing Concurrency).
That site will provide some background for you.
Your options are
to enable concurrency checking, which will mean that if two users download at the same time and the first updates after the second has read but before the second has updated, you'll get an exception.
create a stored procedure that will increment the value in the table directly, and call the stored procedure from code in a single operation - e.g. IncrementDownloadCounter. This will ensure that there is no 'read' and therefore no possibility of a 'dirty read'.
My main question here is dealing with the pramas map when having a one-to-many relationship managed within one dynamic form, as well as best practices for dealing with one-to-many when editing/updating a domain object through the dynamic form. The inputs for my questions are as follows.
I have managed to hack away a form that allows me to create the domain objects shown below in one Dynamic form, since there is no point in having a separate form for creating phone numbers and then assigning them to a contact, it makes sense to just create everything in one form in my application. I managed to implement something similar to what I have asked in my Previous Question (thanks for the people who helped out)
class Contact{
String firstName
String lastName
// ....
// some other properties
// ...
static hasMany = [phones:Phone]
static mapping = {
phones sort:"index", cascade: "all-delete-orphan"
}
}
class Phone{
int index
String number
String type
Contact contact
static belongsTo = [contact:Contact]
}
I basically managed to get the values from the 'params' map and parse them on my own and create the domain object and association manually. I.e. i did not use the same logic that is used in the default scaffolding, i.e.
Contact c = new Contact(params)
etc...., i just looped through all the params and hand crafted my domain objects and saved them and everything works out fine.
My controller has code blocks that look like this (this is stripped down, just to show a point)
//create the contact by handpicking params values
def cntct = new Contact()
cntct.firstName = params.firstName
cntct.lastName = params.lastName
//etc...
//get array of values for number,type
def numbers = params['phone.number']
def types = params['phone.type']
//loop through one of the arrays and create the phones
numbers.eachWithIndex(){ num, i ->
//create the phone domain object from
def phone = new Phone()
phone.number = num
phone.type = types[i]
phone.index = i
cntct.addToPhones(phone)
}
//save
My questions are as follows:
What is the best practice of handeling such a situation, would using Command objects work in this case, if yes where can i found more info about this, all the examples I have found during my search deal with one-to-one relationships, I couldn't find an example for one-to-many?
What is the best way to deal with the relatiohsips of the phones in this case, in terms of add/removing phones when editing the contact object. I mean the creation logic is simple since I have to always create new phones on save, but when dealing with updating a contact, the user might have removed a phone and/or editing an exiting one and/or added some new phones. Right now what I do is just delete all the phones a contact has and re-create them according to what was posted by the form, but I feel that's not the best way to do it, I also don't think looping over the existing ones and comparing with the posted values and doing a manual diff is the best way to do it either, is there a best practice on how to deal with this?
Thanks, hopefully the questions are clear.
[edit] Just for more information, phone information can be added and deleted dynamically using javascript (jquery) within the form [/edit]
disclaimer: i do not know if the following approach works when using grails. Let me know later.
See better way for dynamic forms. The author says:
To add LineItems I have some js that calculates the new index and adds that to the DOM. When deleting a LineItem i have to renumber all the indexes and it is what i would like to avoid
So what i do
I have a variable which stores the next index
var nextIndex = 0;
When the page is loaded, i perform a JavaScript function which calculates how many child The collection has and configure nextIndex variable. You can use JQuery or YUI, feel free.
Adding a child statically
I create a variable which store the template (Notice {index})
var child = "<div>"
+= "<div>"
+= "<label>Name</label>"
+= "<input type="text" name=\"childList[{index}].name\"/>"
+= "</div>"
+= "</div>"
When the user click on the Add child button, i replace {index} - by using regex - by the value stored in the nextIndex variable and increment by one. Then i add to the DOM
See also Add and Remove HTML elements dynamically with Javascript
Adding a child dinamically
Here you can see The Paolo Bergantino solution
By removing
But i think it is the issue grow up when deleting. No matter how many child you remove, does not touch on the nextIndex variable. See here
/**
* var nextIndex = 3;
*/
<input type="text" name="childList[0].name"/>
<input type="text" name="childList[1].name"/> // It will be removed
<input type="text" name="childList[2].name"/>
Suppose i remove childList1 What i do ??? Should i renumber all the indexes ???
On the server side i use AutoPopulatingList. Because childList1 has been removed, AutoPopulatingList handles it as null. So on the initialization i do
List<Child> childList = new AutoPopulatingList(new ElementFactory() {
public Object createElement(int index) throws ElementInstantiationException {
/**
* remove any null value added
*/
childList.removeAll(Collections.singletonList(null));
return new Child();
}
});
This way, my collection just contains two child (without any null value) and i do not need to renumber all the indexes on the client side
About adding/removing you can see this link where i show a scenario wich can gives you some insight.
See also Grails UI plugin
Thanks,
Your answer brought some insight for me to do a wider search and I actually found a great post that covers all the inputs in my question. This is just a reference for anyone reading this. I will write a blog entry on how I implemented my case soon, but this link should provide a good source of ino with a working exmaple.
http://www.2paths.com/2009/10/01/one-to-many-relationships-in-grails-forms/
Most of the time I use ajax to manage such problem.
So when the user clicks add new phone I get the template UI from the server for manageability purpose ( the UI just same GSP template that I use to edit, update the phone), so this way you are not mixing your UI with your js code, whenever you want to change the UI you have to deal only with our GSP code.
Then after getting the UI I add it to the page using jquery DOM manipulation. Then after filling the form when they hit add(save) the request is sent to the server via ajax and is persisted immediately.
When the user clicks edit phone the same UI template is loaded from the server filled with existing phone data, then clicking update will update the corresponding phone immediately via ajax, and same thing applies to delete operation.
But one day I got an additional scenario for the use case that says, "until I say save contact no phone shall be saved on the backend, also after adding phones to the contact on the ui if navigate away to another page and come back later to the contact page the phones I added before must be still there." ugh..
To do this I started using the Session, so the above operations I explained will act on the phone list object I stored on the session instead of the DB. This is simple perform all the operation on the phonesInSession but finally dont forget to do this(delete update):
phonesToBeDeleted = phonesInDB - phonesInSession
phonesToBeDeleted.each{
contact.removeFromPhones(it)
it.delete()
}
I know I dont have to put a lot of data in session but this is the only solution I got for my scenario.
If someone has got similar problem/solution please leave a comment.
First, in all your input fields names you add an #:
<input type="text" name="references[#].name"/>
Second, add call a function before submitting:
<g:form action="save" onsubmit="replaceAllWildCardsWithConsecutiveNumbers();">
Third, this is the code for the function that you call before submitting the form:
function replaceAllWildCardsWithConsecutiveNumbers(){
var inputs = $('form').find("[name*='#']");
var names = $.map(inputs, function(el) { return el.name });
var uniqueNames = unique(names);
for (index in uniqueNames) {
var uniqueName = uniqueNames[index];
replaceWildCardsWithConsecutiveNumbers("input", uniqueName);
replaceWildCardsWithConsecutiveNumbers("select", uniqueName);
}
}
function unique(array){
return array.filter(function(el, index, arr) {
return index === arr.indexOf(el);
});
}
function replaceWildCardsWithConsecutiveNumbers(inputName, name){
counter = 0;
$(inputName + "[name='" + name + "']").each(function (i, el) {
var curName = $(this).attr('name');
var newName = curName.replace("#", counter);
$(this).attr('name', newName);
counter += 1;
});
}
Basically, what the code for replaceAllWildCardsWithConsecutiveNumbers() does, is to create a list for all input (or select) elements whose name contains an #. Removes the duplicates. And then iterates over them replacing the # with a number.
This works great if you have a table and you are submitting the values to a command object's list when creating a domain class for the first time. If you are updating I guess you'll have to change the value of counter to something higher.
I hope this helps someone else since I was stuck on this issue for a while myself.
I am using the Exchange Web Services Managed API to work with Tasks (Exchange 2007 SP1). I can create them fine. However, when I try to do updates, it works for all of the fields except for the .Body field. Whenever I try to access (read/update) that field, it gives the following error:
"You must load or assign this property before you can read its value."
The code I am using looks like this:
//impersonate the person whose tasks you want to read
Me.Impersonate(userName); //home-made function to handle impersonation
//build the search filter
Exchange.SearchFilter.SearchFilterCollection filter = New Exchange.SearchFilter.SearchFilterCollection();
filter.Add(New Exchange.SearchFilter.IsEqualTo(Exchange.TaskSchema.Categories, "Sales"));
//do the search
EWS.Task exTask = esb.FindItems(Exchange.WellKnownFolderName.Tasks, filter, New Exchange.ItemView(Integer.MaxValue));
exTask.Subject = txtSubject.Text; //this works fine
exTask.Body = txtBody.Text; //This one gives the error implying that the object isn't loaded
The strange thing is that, inspecting the property bag shows that the object contains 33 properties, but {Body} is not one of them. That property seems to be inherited from the base class .Item, or something.
So, do I need to re-load the object as type Item? Or reload it via .Bind or something? Keep in mind that I need to do this with thousands of items, so efficiency does matter to me.
Calling the Load method solved my problem :)
foreach (Item item in findResults.Items)
{
item.Load();
string subject = item.Subject;
string mailMessage = item.Body;
}
I had the same problem when using the EWS. My Code is requesting the events(Appointments) from the
Outlook calendar, at the end I couldn't reach to the body of the Event itself.
The missing point in my situation was the following "forgive me if there is any typo errors":
After gathering the Appointments, which are also derived from EWS Item Class, I did the following:
1- Create a List with the type Item:
List<Item> items = new List<Item>();
2- Added all appointments to items list:
if(oAppointmentList.Items.Count > 0) // Prevent the exception
{
foreach( Appointment app in oAppointmentList)
{
items.Add(app);
}
}
3- Used the exchanged service "I have already created and used":
oExchangeService.LoadPropertiesForItems(items, PropertySet.FirstClassProperties);
now if you try to use app.Body.Text, it will return it successfully.
Enjoy Coding and Best Luck
I forgot to mention the resource:
http://social.technet.microsoft.com/Forums/en-US/exchangesvrdevelopment/thread/ce1e0527-e2db-490d-817e-83f586fb1b44
He mentioned the use of Linq to save the intermediate step, it will help you avoid using the List items and save some memory!
RockmanX
You can load properties using a custom property set. Some properties are Extended properties instead of FirstClassProperties.
Little example:
_customPropertySet = new PropertySet(BasePropertySet.FirstClassProperties, AppointmentSchema.MyResponseType, AppointmentSchema.IsMeeting, AppointmentSchema.ICalUid);
_customPropertySet.RequestedBodyType = BodyType.Text;
appointment.Load(_customPropertySet);