I have a C# console app that processes a queue (in db) and processes user uploaded contents.
It is running as a scheduled task.
If some user drops some content, the process won't pick it up until it is time for it to run.
If the job runs once every 5 minutes then it has to wait for max 5 minutes to run.
I want to process user content right away.
If user1 drops content and then user2 drops content after 30 seconds, I want 2 instances of my job running.
Is it possible to trigger the job\task to run from the C# code (MVC controller)?
Essentially it sounds like you're just looking to perform an asynchronous operation. Depending on the version of .NET, there are a number of options. For example, the Task Parallel Library is a simple way to invoke an asynchronous operation. So if the task is encapsulated into some object (for the same of example, let's say a method called Process on an object called content) then it might look like this:
var processContent = new Task(() => content.Process());
processContent.Start();
The task would then continue asynchronously and flow control would return to the application (which is particularly ideal in a web application where you don't want a user looking at an unresponsive browser, naturally).
If you're using .NET 4.5, you can also make use of the async and await keywords to perhaps make it a little more clear. At its heart, you're talking about running a process in a separate thread.
This all assumes that the web application and the console application share the same back-end business logic code. Which, of course, is probably what they should be doing. If for some reason that's not an option (and I recommend looking into making it an option), you can start a process from code. It could be something as simple as:
Process.Start("C:\Path\To\ConsoleApplication.exe");
Though forking all of these processes can look pretty messy and might be very difficult to manage in terms of error handling/logging/etc. Another reason why it's best to keep the logic in the same process as the application, just in a separate thread.
Related
I am in need to call an external *.exe compiled in C++
from ASP.NET WEB API 2 using Process (System.Diagnostics)
This executable does some image processing stuff and use lot of memory.
SO my question is if change my API calls to Async. or implement threads will it help, Or it doesn't matter?
Note: All i have is executable so i can not go for a CLI Wrapper.
You can separate the two. Your api is one thing, it needs to be fast, responsive to be able to serve the clients. Your image processing thing is different.
You could implement a queuing system. The api is responsible for adding a new item to this queue and nothing more. You could keep track of what tasks are being run in a separate sql table let's say. Imagine you have a sql table called Tasks. Your api chucks data in there and the status is "Not Running".
Some other app which lives on another machine entirely keeps an eye on this table and takes care of running that executable for each item. When it starts, it changes the status to Running, when it completes it's Done. You do whatever else you need. You could have an api endpoint which takes the ID of the task so your client can keep calling this endpoint to see what the status is. Or you could raise an event when it's done, depending on your application needs.
Bottom line, keep things separate, you gain nothing for blocking the api while a resources heavy task is running. Think what happens if you start that process 5 times, at the same time. You've just killed your api basically.
The app that does the heavy work, could even be located on a separate machine, so it doesn't affect the api at all.
I have a workflow that runs when an entity is created and it creates two other entities and puts them on a queue. It then waits until each entity's status reason is set to done. After which is continues.
Basically two teams will work an order and then it will continue processing after both teams are done.
Most of the time it works. However sometimes it waits forever. I'll re-active and re-resolve the other tasks, but it just never wakes up.
What can I do? The workflows aren't really powerful enough for me to have it poll with a timeout (there are no loops). I'd like to avoid on-change plugins for these other entities to get workflow behavior all scattered about.
Edit:
Restarting the CRM services (not sure which did it, I restarted them all) allowed the workflow to resume. However, I'd still like to know how to make this more reliable.
I had the same problem (and a lot more) with workflows in CRM 2011 and decided not to use them (except for very special purposes).
The main reason is because of their very limited error handling. Another reason is that it is inconvenient to put them under source control. Another reasons are: Worflows cannot run offline and user impersonation is also not supported. For a comparison look here: http://goo.gl/9ht1QJ
Use plugins instead of workflows, then you have full control.
But keep in mind that plugins (unlike workflows) are not designed for long running tasks.
So they have a default max execution time of 120 sec and are not stateful/persisted. But in most cases (and i think also in your case) that is not a problem.
Just change your eventing a little bit:
Implement and register a plugin step for: entity is created and it creates two other entities and puts them on a queue
Implement and register another step: entity's status reason is set to done, query for other entity and check status, if done continue processing
If you really do not want use plugins for you business logic you can consider implementing a plugin which restarts/resumes faulted workflows.
But thats not a very nice solution.
I have an ASP.WEB Web Api controller that needs to fire and forget some slow code. What would be a good way to do that? That is I want the controller to return an HTML response to the browser, while the slow code keeps running somewhere.
Is it a good idea to grab a worker thread from the tread pool and pass in a complex object created by the controller? Or do I need to write a separate windows service to do the work?
Your solution depends on the specifics or your situation and your workload.
You can certainly start of a new task Factory.StartNew when you receive a request.
There is nothing wrong with this technically.
Things you should think about though:
Do I have to return data back to the customer?
This task will use up web server resources so if those tasks take very long time and you get a lot of traffic you may run into situation where your customers are waiting in line to just start being processed. In this situation I think backend server with Windows Service be a much better idea.
All tasks above are subject to IIS Resets. They may be killed during processing your background task.
Why not use the Windows scheduler?
I have several applications that have to run at certain times according to business rules not the typical every weekday at 1pm.
I also need a way for the applications to provide feedback of their progress so that I can have rules that notify me when the applications are running slow or aren't even running anymore.
What Windows API should I be looking into? (like, a time version of the FileWatcher apis)
What's the best way to have the application notify the scheduler of its progress (files, sockets, windows messages, ???)?
For Vista/Win2k8, there's the nice Task Scheduler 2.0 API: http://msdn.microsoft.com/en-us/library/aa384138(VS.85).aspx. Previous version have the Task Scheduler 1.0 API, but I've never used it.
AppControls has a CronJob component that you can use to create scheduled events. This saves your program from having to wake up every minute and check the schedule itself. Instead, just schedule the job and indicate a callback method.
I have used this component for scheduling jobs myself and have been very happy with the way that it works.
I think what you really want is a common framework for your applications that report to something (you or the system messages or tracing or perfmon, event log, whatever) and also to receive via some inter process protocol a way to receive messages and respond.
based on the reporting you can change the scheduling or make changes, etc.
So, there is some monitor app, and then each of your other apps does common reporting.
events I can think of:
- started
- stopped
- error
- normal log messages
- and of course specific things your apps do.
I think there are probably existing classes/framework that do this - you'll have to check around.
If it were me, I would make a service that could talk to all the other apps and perhaps was even an http server. It would be able to route messages to particular apps and start stop those processes and query them.
There are lots of ways to do what you want though. those were just off the top of my head.
Alternatively you might just be able to get these to be services and they handle messages sent to them. Their normal processing does nothing until they are "woken up" with some task command.
You have more questions in one. Normally you should split them. But let's overlook this and try to answer.
To schedule certain events (including running an application): Use TJvScheduledEvents from JVCL. IMHO JVCL is the best Delphi open source library around with extensive number of components, developers & support. TJvScheduledEvents is quite neat, uses threads for event scheduling and also you have in JVCL a detailed editor for your events (it needs a small hack to use it though).
To provide 'feedback' from your applications to a (remote) central point: A very very very good solution (if your requirements permit) is to log the progress of your applications in a table (let's call it LOG) on a Firebird server. In LOG you can have the following fields: COMPUTER, USERNAME, APPNAME, MSG, LOGDATE (etc. etc.). In the After Insert trigger of the LOG table you can fire an event (let's call it NEW_LOG). In your console app you can register the interest for this event and so, your application will be automatically updated with everything which happens in any of your applications, so you can do log analysis, graphs etc. Of course you can do it with IB, but IB costs.
...going on Windows API route you need headers (which probably aren't translated), you'll encounter our dearest Pointers/PChars etc. etc. Of course, building from scratch everything isn't worthwhile but when this is already done in a Delphi way, why don't use it?
Use service with a timer that is fired regulary (for example each minute). It reads the schedule and looks if some are due before the next iteration. If so, you can execute them.
You can add an interface that shows all running apps. For the feedback and query that using a desktop application.
I'm working on a consumer web app that needs to do a long running background process that is tied to each customer request. By long running, I mean anywhere between 1 and 3 minutes.
Here is an example flow. The object/widget doesn't really matter.
Customer comes to the site and specifies object/widget they are looking for.
We search/clean/filter for widgets matching some initial criteria. <-- long running process
Customer further configures more detail about the widget they are looking for.
When the long running process is complete the customer is able to complete the last few steps before conversion.
Steps 3 and 4 aren't really important. I just mention them because we can buy some time while we are doing the long running process.
The environment we are working in is a LAMP stack-- currently using PHP. It doesn't seem like a good design to have the long running process take up an apache thread in mod_php (or fastcgi process). The apache layer of our app should be focused on serving up content and not data processing IMO.
A few questions:
Is our thinking right in that we should separate this "long running" part out of the apache/web app layer?
Is there a standard/typical way to break this out under Linux/Apache/MySQL/PHP (we're open to using a different language for the processing if appropriate)?
Any suggestions on how to go about breaking it out? E.g. do we create a deamon that churns through a FIFO queue?
Edit: Just to clarify, only about 1/4 of the long running process is database centric. We're working on optimizing that part. There is some work that we could potentially do, but we are limited in the amount we can do right now.
Thanks!
Consider providing the search results via AJAX from a web service instead of your application. Presumably you could offload this to another server and let you web application deal with the content as you desire.
Just curious: 1-3 minutes seems like a long time for a lookup query. Have you looked at indexes on the columns you are querying to improve the speed? Or do you need to do some algorithmic process -- perhaps you could perform some of this offline and prepopulate some common searches with hints?
As Jonnii suggested, you can start a child process to carry out background processing. However, this needs to be done with some care:
Make sure that any parameters passed through are escaped correctly
Ensure that more than one copy of the process does not run at once
If several copies of the process run, there's nothing stopping a (not even malicious, just impatient) user from hitting reload on the page which kicks it off, eventually starting so many copies that the machine runs out of ram and grinds to a halt.
So you can use a subprocess, but do it carefully, in a controlled manner, and test it properly.
Another option is to have a daemon permanently running waiting for requests, which processes them and then records the results somewhere (perhaps in a database)
This is the poor man's solution:
exec ("/usr/bin/php long_running_process.php > /dev/null &");
Alternatively you could:
Insert a row into your database with details of the background request, which a daemon can then read and process.
Write a message to a message queue which a daemon then read and processed.
Here's some discussion on the Java version of this problem.
See java: what are the best techniques for communicating with a batch server
Two important things you might do:
Switch to Java and use JMS.
Read up on JMS but use another queue manager. Unix named pipes, for instance, might be an acceptable implementation.
Java servlets can do background processing. You could do something similar to this technology in a web technology with threading support. I don't know about PHP though.
Not a complete answer but I would think using AJAX and passing the 2nd step to something thats faster then PHP (C, C++, C#) then a PHP function pick the results off of some stack most likely just a database.