Normally, billings should execute in the background on a scheduled date (I haven't figured out how to do that yet, but that's another topic).
But occasionally, the user may wish to execute a billing manually. Once clicked, I would like to be sure the operation runs to completion regardless of what happens on the user side (e.g. closes browser, machine dies, network goes down, whatever).
I'm pretty sure db.SaveChanges() wraps its DB operations in a transaction, so from a server perspective I believe the whole thing will either finish or fail, with no partial effect.
But what about all the work between the POST and the db.SaveChanges()? Is there a way to be sure the user can't inadvertently or intentionally stop that from completing?
I guess a corollary to this question is what happens to a running Asynchronous Controller or a running Task or Thread if the user disconnects?
My previous project was actually doing a billing system in MVC. I distinctly remember testing out what would happen if I used Task and then quickly exited the site. It did all of the calculations just fine, ran a stored procedure in SQL Server, and sent me an e-mail when it was done.
So, to answer your question: If you wrap the operations in a Task it should finish anyways with no problems.
Related
Use a StackOverflow Q&A thread as an example - when you vote up, vote down, or favorite a question, you can see the UI quickly respond to that action with changes in the # of up-votes on the side.
How can we achieve that effect? If send every of such action to back-end for processing and use the returned response to update UI, you will see a slow update and feel the glitches. But if put some of the logic on the front-end, you will also need to take care of the fraud/abuse etc before reflecting the action on UI, i.e - before changing the # of up-votes, don't you need to make sure that's a valid click by an valid user first?
You make sure that a valid user is using the app before a user clicks on anything. This is done through authentication, and it must include various protection mechanisms against malicious users.
When a user clicks, a call is made to a server. In a properly architected app this call is lightweight, and the server responds very quickly. I don't know why you believe that "you will see a slow update and feel the glitches". Adding an upvote to the database should take a few hundred milliseconds at most (including the roundtrip from the client), especially if the commit is asynchronous or a memcache is used.
If a database update results in a need to do some complex operations, typically these operations are not done right away. For example, a cron job may run periodically to compute new rankings, etc., precisely because you do not want every user to wait. Alternatively, a task is created and put in a task queue to be executed when resources are available - again to make sure that a user does not wait.
In some apps a UI is updated immediately after the call to the server is made, before any response from a server arrives. You can do it when the consequences of a failed call are negligible. For example, if an upvote fails to be saved in the database, it's not a disaster, especially if it happens once in a million tries. Again, in a properly architected app calls fail extremely rarely.
This is a decision that an app developer needs to make. I would not update a UI before a server response if such an update may lead a user to believe that some other action is now possible. For example, if a user uploads a new photo, I would not show icons to edit or share this photo until I know that the photo is safely saved.
I have a workflow that runs when an entity is created and it creates two other entities and puts them on a queue. It then waits until each entity's status reason is set to done. After which is continues.
Basically two teams will work an order and then it will continue processing after both teams are done.
Most of the time it works. However sometimes it waits forever. I'll re-active and re-resolve the other tasks, but it just never wakes up.
What can I do? The workflows aren't really powerful enough for me to have it poll with a timeout (there are no loops). I'd like to avoid on-change plugins for these other entities to get workflow behavior all scattered about.
Edit:
Restarting the CRM services (not sure which did it, I restarted them all) allowed the workflow to resume. However, I'd still like to know how to make this more reliable.
I had the same problem (and a lot more) with workflows in CRM 2011 and decided not to use them (except for very special purposes).
The main reason is because of their very limited error handling. Another reason is that it is inconvenient to put them under source control. Another reasons are: Worflows cannot run offline and user impersonation is also not supported. For a comparison look here: http://goo.gl/9ht1QJ
Use plugins instead of workflows, then you have full control.
But keep in mind that plugins (unlike workflows) are not designed for long running tasks.
So they have a default max execution time of 120 sec and are not stateful/persisted. But in most cases (and i think also in your case) that is not a problem.
Just change your eventing a little bit:
Implement and register a plugin step for: entity is created and it creates two other entities and puts them on a queue
Implement and register another step: entity's status reason is set to done, query for other entity and check status, if done continue processing
If you really do not want use plugins for you business logic you can consider implementing a plugin which restarts/resumes faulted workflows.
But thats not a very nice solution.
I have a C# console app that processes a queue (in db) and processes user uploaded contents.
It is running as a scheduled task.
If some user drops some content, the process won't pick it up until it is time for it to run.
If the job runs once every 5 minutes then it has to wait for max 5 minutes to run.
I want to process user content right away.
If user1 drops content and then user2 drops content after 30 seconds, I want 2 instances of my job running.
Is it possible to trigger the job\task to run from the C# code (MVC controller)?
Essentially it sounds like you're just looking to perform an asynchronous operation. Depending on the version of .NET, there are a number of options. For example, the Task Parallel Library is a simple way to invoke an asynchronous operation. So if the task is encapsulated into some object (for the same of example, let's say a method called Process on an object called content) then it might look like this:
var processContent = new Task(() => content.Process());
processContent.Start();
The task would then continue asynchronously and flow control would return to the application (which is particularly ideal in a web application where you don't want a user looking at an unresponsive browser, naturally).
If you're using .NET 4.5, you can also make use of the async and await keywords to perhaps make it a little more clear. At its heart, you're talking about running a process in a separate thread.
This all assumes that the web application and the console application share the same back-end business logic code. Which, of course, is probably what they should be doing. If for some reason that's not an option (and I recommend looking into making it an option), you can start a process from code. It could be something as simple as:
Process.Start("C:\Path\To\ConsoleApplication.exe");
Though forking all of these processes can look pretty messy and might be very difficult to manage in terms of error handling/logging/etc. Another reason why it's best to keep the logic in the same process as the application, just in a separate thread.
A web application I am developing needs to perform tasks that are too long to be executed during the http request/response cycle. Typically, the user will perform the request, the server will take this request and, among other things, run some scripts to generate data (for example, render images with povray).
Of course, these tasks can take a long time, so the server should not hang for the scripts to complete execution before sending the response to the client. I therefore need to perform the execution of the scripts async, and give the client a "the resource is here, but not ready" and probably tell it a ajax endpoint to poll, so it can retrieve and display the resource when ready.
Now, my question is not relative to the design (although I would very much enjoy any hints on this regard as well). My question is: does a system to solve this issue already exists, so I do not reinvent the square wheel ? If I had to, I would use a process queue manager to submit the task and put a HTTP endpoint to shoot out the status, something like "pending", "aborted", "completed" to the ajax client, but if something similar already exists specifically for this task, I would mostly enjoy it.
I am working in python+django.
Edit: Please note that the main issue here is not how the server and the client must negotiate and exchange information about the status of the task.
The issue is how the server handles the submission and enqueue of very long tasks. In other words, I need a better system than having my server submit scripts on LSF. Not that it would not work, but I think it's a bit too much...
Edit 2: I added a bounty to see if I can get some other answer. I checked pyprocessing, but I cannot perform submission of a job and reconnect to the queue at a later stage.
You should avoid re-inventing the wheel here.
Check out gearman. It has libraries in a lot of languages (including python) and is fairly popular. Not sure if anyone has any out of the box ways to easily connect up django to gearman and ajax calls, but it shouldn't be do complicated to do that part yourself.
The basic idea is that you run the gearman job server (or multiple job servers), have your web request queue up a job (like 'resize_photo') with some arguments (like '{photo_id: 1234}'). You queue this as a background task. You get a handle back. Your ajax request is then going to poll on that handle value until it's marked as complete.
Then you have a worker (or probably many) that is a separate python process connect up to this job server and registers itself for 'resize_photo' jobs, does the work and then marks it as complete.
I also found this blog post that does a pretty good job summarizing it's usage.
You can try two approachs:
To call webserver every n interval and inform a job id; server processes and return some information about current execution of that task
To implement a long running page, sending data every n interval; for client, that HTTP request will "always" be "loading" and it needs to collect new information every time a new data piece is received.
About second option, you can to learn more by reading about Comet; Using ASP.NET, you can do something similiar by implementing System.Web.IHttpAsyncHandler interface.
I don't know of a system that does it, but it would be fairly easy to implement one's own system:
create a database table with jobid, jobparameters, jobresult
jobresult is a string that will hold a pickle of the result
jobparameters is a pickled list of input arguments
when the server starts working on a job, it creates a new row in the table, and spwans a new process to handle that, passing that process the jobid
the task handler process updates the jobresult in the table when it has finished
a webpage (xmlrpc or whatever you are using) contains a method 'getResult(jobid)' that will check the table for a jobresult
if it finds a result, it returns the result, and deletes the row from the table
otherwise it returns an empty list, or None, or your preferred return value to signal that the job is not finished yet
There are a few edge-cases to take care of so an existing framework would clearly be better as you say.
At first You need some separate "worker" service, which will be started separately at powerup and communicated with http-request handlers via some local IPC like UNIX-socket(fast) or database(simple).
During handling request cgi ask from worker state or other data and replay to client.
You can signal that a resource is being "worked on" by replying with a 202 HTTP code: the Client side will have to retry later to get the completed resource. Depending on the case, you might have to issue a "request id" in order to match a request with a response.
Alternatively, you could have a look at existing COMET libraries which might fill your needs more "out of the box". I am not sure if there are any that match your current Django design though.
Probably not a great answer for the python/django solution you are working with, but we use Microsoft Message Queue for things just like this. It basically runs like this
Website updates a database row somewhere with a "Processing" status
Website sends a message to the MSMQ (this is a non blocking call so it returns control back to the website right away)
Windows service (could be any program really) is "watching" the MSMQ and gets the message
Windows service updates the database row with a "Finished" status.
That's the gist of it anyways. It's been quite reliable for us and really straight forward to scale and manage.
-al
Another good option for python and django is Celery.
And if you think that Celery is too heavy for your needs then you might want to look at simple distributed taskqueue.
The workflow is being published as a wcf service, and I need to guarantee that workflows execute sequentially. Is there a way--in code or in the config--to guarantee the runtime doesn't launch two workflows concurrently?
There is no way to configure the runtime to limit the number of workflows in progress.
Consider though that its the responsibility of the workflow itself to control flow. Hence the workflow itself should have means to determine if another instance of itself is currently in progress.
I would consider creating an Activity that would transactionally attempt to update a DB record to the effect that an instance of this workflow is in progress. If it finds that another is currently in progress it could take the appropriate action. It could fail or it could queue itself using an EventActivity to be alerted when the previous workflow has completed.
You probably will need to check at workflow start for another running instance.
If found, cancel it.
I don't agree that this needs to be handled at the WorkflowRuntime level. I like the idea of a custom Activity, sort of a MutexActivity that would be a CompositeActivity that has a DB backend. The first execution would log to the database it has a hold of the mutex. Subsequent calls would queue up their workflow IDs and then go idle. When the MutexActivity completes, it would release the Mutex, load up the next workflow in the queue and invoke the contained child activities.