My application currently polls database every n seconds to see if there are any new records.
To reduce network round trips, and CPU cycles of this polling i was thinking to replace it with CQN based approach where database will itself update subscribed application if there is any Commit to database.
The only problem is what if Oracle was NOT able to notify application due to any connection issue between oracle and subscribed application or if the application was crashed or killed due to any reason? ... Is there a way to know if application have missed any CQN notification?
Is polling database via application code itself the only way for mission critical applications?
You didn't say whether every 'n' seconds means you're expecting data every few seconds, or you just need your "staleness" to as low as that. That has an impact on the choice of CQN, because as per docs, https://docs.oracle.com/en/database/oracle/oracle-database/12.2/adfns/cqn.html#GUID-98FB4276-0827-4A50-9506-E5C1CA0B7778
"Good candidates for CQN are applications that cache the result sets of queries on infrequently changed objects in the middle tier, to avoid network round trips to the database. These applications can use CQN to register the queries to be cached. When such an application receives a notification, it can refresh its cache by rerunning the registered queries"
However, you have control over how persistent you want the notifcations to be:
"Reliable Option:
By default, a CQN registration is stored in shared memory. To store it in a persistent database queue instead—that is, to generate reliable notifications—specify QOS_RELIABLE in the QOSFLAGS attribute of the CQ_NOTIFICATION$_REG_INFO object.
The advantage of reliable notifications is that if the database fails after generating them, it can still deliver them after it restarts. In an Oracle RAC environment, a surviving database instance can deliver them.
The disadvantage of reliable notifications is that they have higher CPU and I/O costs than default notifications do."
The documentation for SAPGUISession.Sync says:
Instructs UFT to wait until the SAP GUI for Windows session is available.
Is this 1:1 comparable to Web add-on´s Page.Sync? If so, when should I call it? Do I have to call Sync...
after each input sent to the SAP GUI?
after each input sent to the SAP GUI if a server roundtrip takes place after this input is received? (How can I identify that one takes place?)
after each context-changing input sent to the SAP GUI?
only once after launching the SAP session?
I haven´t had a chance to use the Windows-SAP Support in UFT yet, that´s why I find the documentation to be rather sparse.
Thanks...
AFAIK it's the same as web's Sync, there's usually no need to use it. Synchronisation in UFT usually comes from the need to identify an object before acting upon it. Sync is useful in cases where an object in the old state of the application may match the expected object in the new state of the application. Usually Sync is added to tests ad-hoc when it fails due to synchronisation issues.
SAP´s Sync works just as Web´s Sync, but one important difference is:
While Page.Sync is often no guarantee that the app really is idle when Page.Sync Returns, for SAP applications, SAPSession.Sync´s returning does indeed guarantee this.
So whenever the SAP Client is doing Server roundtrips, SAPSession.Sync is a very save way of obtaining synchronization (i.e. awaiting SAP client´s idle state).
I'm trying to create a real-time webbrowser game in ASP.Net MVC3, but I'm unsure about what's the best approach to processing 'real-time' events on the server side.
Imagine that the client wants to upgrade a building, upgrading a building takes time. A record gets inserted in the database that holds the end-time and on the client a ajax timer start's running. I was thinking about having a windows service running all the time. Every second the service checks the table and does the real processing when the time passed the end-time. However I could imagine that when you have a huge amount of data to process this can get problematic.
What would be the best way of doing this?
Cheers.
I've seen games like that before. You do not need to count down yourself, you just need to store the datetime for whenever the building will be ready.
When a page is requested you send a timestamp of when the building will be completed, and use javascript to count down. For example, like this.
I am considering to use the Oracle Advanced Queueing technology for asynchronous communication. My aim is to use it for concurrent process execution (asynchronous PL/SQL procedure calls).
The current legacy implementation for the concurrent process execution is made of Unix KornShell (ksh) scripts which we are starting from front end via SSH connection in background mode. It works fine for us, but I am unhappy with that kind of solution because of:
Security (front end starts a SSH connection and executes ksh scripts in background mode. From our colleagues I noticed that this kind of login will be restricted in our company.)
Maintenance (Not everyone of our team is familiar with ksh scripts)
Diversity in technology (I try to decrease the diversity in technology because of know how and migration efforts)
Logging (Our back end system logs into database log tables, the concurrent execution logs partially into a log file)
By moving from ksh to the database I will be able to increase overall quality of my system:
Security (No SSH connections anymore, the front end will send messages to the database and the database message listener will react to the messages and execute procedures asynchronously)
Maintenance (We use PL/SQL, where we are familiar in)
Diversity in technology (By next OS migration we will need to migrate only the database objects and the data)
Logging (We will fully use our back end logging solution)
What do you think about my considerations and what are your experiences with Oracle Advanced Queueing? Especially in stability, performance and maintenance? Are there better alternatives?
I obviously don't know the details of your project, but if asynchronous PL/SQL procedure calls is your only goal, it may be easier to use DBMS_SCHEDULER. Your program could submit jobs to "run now" through the scheduler that call your PL/SQL. In my opinion, the scheduler is a much easier thing to work with than AQs.
The management of flows with Asynchronous queues Oracle brings with it advantages and disadvantages:
ADVANTAGES
Ability to manage flows by type creating ad hoc code on which to
create Handler (JOB EVENT or APPLY PROCESS) to manage the various
Sub Flows
Easy to put out a whole type of flows closing DEQUEUE Queue.
Managing Priorities (Parameter in the creation of Coda) of MSG for
INSERT TIME or PRIORITY (msg parameter in the Payload) Managing
message with a deadline or an Elapsed TIME.
Align the paradigm to a solution to EVENT no POLLING
DISADVANTAGES
The load of Business Logic will all be on the DB.
When Installing New PKG you will need to stop the queues (queuing and DEQUEUEING) to restart the HANDLER that point to the PKG.
Having to implement a recovery system msg Incorrectly Processed.
I think a good solution would be to use the CODE JMS (JMS provider) on the tails ORACLE so as to move the BL on JAVA and to use the various potentials of the language including the Logging.
A web application I am developing needs to perform tasks that are too long to be executed during the http request/response cycle. Typically, the user will perform the request, the server will take this request and, among other things, run some scripts to generate data (for example, render images with povray).
Of course, these tasks can take a long time, so the server should not hang for the scripts to complete execution before sending the response to the client. I therefore need to perform the execution of the scripts async, and give the client a "the resource is here, but not ready" and probably tell it a ajax endpoint to poll, so it can retrieve and display the resource when ready.
Now, my question is not relative to the design (although I would very much enjoy any hints on this regard as well). My question is: does a system to solve this issue already exists, so I do not reinvent the square wheel ? If I had to, I would use a process queue manager to submit the task and put a HTTP endpoint to shoot out the status, something like "pending", "aborted", "completed" to the ajax client, but if something similar already exists specifically for this task, I would mostly enjoy it.
I am working in python+django.
Edit: Please note that the main issue here is not how the server and the client must negotiate and exchange information about the status of the task.
The issue is how the server handles the submission and enqueue of very long tasks. In other words, I need a better system than having my server submit scripts on LSF. Not that it would not work, but I think it's a bit too much...
Edit 2: I added a bounty to see if I can get some other answer. I checked pyprocessing, but I cannot perform submission of a job and reconnect to the queue at a later stage.
You should avoid re-inventing the wheel here.
Check out gearman. It has libraries in a lot of languages (including python) and is fairly popular. Not sure if anyone has any out of the box ways to easily connect up django to gearman and ajax calls, but it shouldn't be do complicated to do that part yourself.
The basic idea is that you run the gearman job server (or multiple job servers), have your web request queue up a job (like 'resize_photo') with some arguments (like '{photo_id: 1234}'). You queue this as a background task. You get a handle back. Your ajax request is then going to poll on that handle value until it's marked as complete.
Then you have a worker (or probably many) that is a separate python process connect up to this job server and registers itself for 'resize_photo' jobs, does the work and then marks it as complete.
I also found this blog post that does a pretty good job summarizing it's usage.
You can try two approachs:
To call webserver every n interval and inform a job id; server processes and return some information about current execution of that task
To implement a long running page, sending data every n interval; for client, that HTTP request will "always" be "loading" and it needs to collect new information every time a new data piece is received.
About second option, you can to learn more by reading about Comet; Using ASP.NET, you can do something similiar by implementing System.Web.IHttpAsyncHandler interface.
I don't know of a system that does it, but it would be fairly easy to implement one's own system:
create a database table with jobid, jobparameters, jobresult
jobresult is a string that will hold a pickle of the result
jobparameters is a pickled list of input arguments
when the server starts working on a job, it creates a new row in the table, and spwans a new process to handle that, passing that process the jobid
the task handler process updates the jobresult in the table when it has finished
a webpage (xmlrpc or whatever you are using) contains a method 'getResult(jobid)' that will check the table for a jobresult
if it finds a result, it returns the result, and deletes the row from the table
otherwise it returns an empty list, or None, or your preferred return value to signal that the job is not finished yet
There are a few edge-cases to take care of so an existing framework would clearly be better as you say.
At first You need some separate "worker" service, which will be started separately at powerup and communicated with http-request handlers via some local IPC like UNIX-socket(fast) or database(simple).
During handling request cgi ask from worker state or other data and replay to client.
You can signal that a resource is being "worked on" by replying with a 202 HTTP code: the Client side will have to retry later to get the completed resource. Depending on the case, you might have to issue a "request id" in order to match a request with a response.
Alternatively, you could have a look at existing COMET libraries which might fill your needs more "out of the box". I am not sure if there are any that match your current Django design though.
Probably not a great answer for the python/django solution you are working with, but we use Microsoft Message Queue for things just like this. It basically runs like this
Website updates a database row somewhere with a "Processing" status
Website sends a message to the MSMQ (this is a non blocking call so it returns control back to the website right away)
Windows service (could be any program really) is "watching" the MSMQ and gets the message
Windows service updates the database row with a "Finished" status.
That's the gist of it anyways. It's been quite reliable for us and really straight forward to scale and manage.
-al
Another good option for python and django is Celery.
And if you think that Celery is too heavy for your needs then you might want to look at simple distributed taskqueue.