My application currently polls database every n seconds to see if there are any new records.
To reduce network round trips, and CPU cycles of this polling i was thinking to replace it with CQN based approach where database will itself update subscribed application if there is any Commit to database.
The only problem is what if Oracle was NOT able to notify application due to any connection issue between oracle and subscribed application or if the application was crashed or killed due to any reason? ... Is there a way to know if application have missed any CQN notification?
Is polling database via application code itself the only way for mission critical applications?
You didn't say whether every 'n' seconds means you're expecting data every few seconds, or you just need your "staleness" to as low as that. That has an impact on the choice of CQN, because as per docs, https://docs.oracle.com/en/database/oracle/oracle-database/12.2/adfns/cqn.html#GUID-98FB4276-0827-4A50-9506-E5C1CA0B7778
"Good candidates for CQN are applications that cache the result sets of queries on infrequently changed objects in the middle tier, to avoid network round trips to the database. These applications can use CQN to register the queries to be cached. When such an application receives a notification, it can refresh its cache by rerunning the registered queries"
However, you have control over how persistent you want the notifcations to be:
"Reliable Option:
By default, a CQN registration is stored in shared memory. To store it in a persistent database queue instead—that is, to generate reliable notifications—specify QOS_RELIABLE in the QOSFLAGS attribute of the CQ_NOTIFICATION$_REG_INFO object.
The advantage of reliable notifications is that if the database fails after generating them, it can still deliver them after it restarts. In an Oracle RAC environment, a surviving database instance can deliver them.
The disadvantage of reliable notifications is that they have higher CPU and I/O costs than default notifications do."
Related
I have a database in Azure SQL Server, which stores messages from IoT devices. Those devices send periodic messages to a listener which is set up (along with a lot of logic before it ends up in SQL DB) and runs in node-red. Everything works well for a couple of weeks now.
Enter Logic Apps. I have a simple trigger that executes two stored procedures on a schedule (one issues major update on one table that is also used in trigger (on a table in SQL) firing when node-red inserts data, second inserts data into a table feeding PowerBI dashboard. Total runtime for two executions is less than 20 seconds.
The moment this trigger is enabled, node-red starts experiencing time-outs when connecting to the server. And I am literal: for 3 days no issues, enabled Logic Apps trigger and I can see incomplete data saved in DB. It doesn't even have to run, just enabled causes issues.
It was disabled last week due to same issues, though we didn't know then this was the cause. We assumed neighbors were crowding us and tier was low on that db. But we forked out some cash and upgraded. Yet problem is back. And I can definitely state this is the ONLY change that was made on Azure in regards to that setup in 3 days.
I am kinda stumped - not sure what is happening, so not sure how to fix it except disabling trigger again. What is going on?
I have a simple rest endpoint that executes Postgres procedure.
This procedure returns the current state of device.
For example:
20 devices.
Client app connect to API and make 20 responses to that endpoint every second.
For x clients there are x*20 requests.
For 2 clients 40 requests.
It causes a big cpu load on Postgres server only if there are many clients and/or many devices.
I didn’t create it but I need to redesign it.
How to limit concurrent queries to db only for it? It would be a hot fix.
My second idea is to create background worker that executes queries only one in the same time. Then the endpoint fetches data from memory.
I would try the simple way first. Try to reduce
the amount of database connections in the pool OR
the amount of working threads in the build-in Tomcat.
More flexible option would be to put the logic behind a thread pool limiting the amount of working threads. It is not trivial, if the Spring context and database is used inside a worker. Take a look on a Spring annotation #Async.
Offtopic: The solution we are discussing here looks like a workaround. The discussed solution alone will most probably increase the throughput only by factor 2 maybe 3. It is not JEE conform and it will be most probably not very stable. It is better to refactor the application avoiding such a problem. Another option would be to buy a new database server.
Update: JEE compliant solution would be to implement some sort of bulkhead pattern. It will limit the amount of concurrent running requests and reject it, if the some critical number is reached. The server application answers with "503 Service Unavailable". The client application catches this status and retries a second later (see "exponential backoff").
We are building a reporting app on Laravel that need to fetch users data from a third-party server that allow 1 request per seconds.
We need to fetch 100K to 1000K rows based on user and we can fetch max 250 rows per request.
So the restriction is:
1. We can send 1 request per seconds
2. 250 rows per request
So, it requires 400-4000 request/jobs to fetch a user data, So, loading data for multiple users is very time-consuming and the server gets slow.
So, now, we are planning to load the data using multiple servers, like 4-10 servers to fetch users data, so we can send 10 requests per second from 10 servers.
How can we design the system and process jobs from multiple servers?
Is it possible to use a dedicated server for hosting Redis and connect to that Redis server from multiple servers and execute jobs? Can any conflict/race-condition happen?
Any hint or prior experience related to this would be really helpful.
The short answer is yes, this is absolutely possible and is something I've implemented in production apps many times before.
Redis is just like any other service and can run anywhere, with clients from anywhere, connecting to it. It's all up to your configuration of the server to dictate how exactly that happens (and adding passwords, configuring spiped, limiting access via the firewall, etc.). I'd reccommend reading up on the documentation they have in the Administration section here: https://redis.io/documentation
Also, when you do make the move to a dedicated Redis host, with multiple clients accessing it, you'll likely want to look into having more than just one Redis server running for reliability, high availability, etc. Redis has efficient and easy replication available with a few simple configuration commands, which you can read more about here: https://redis.io/topics/replication
Last thing on Redis, if you do end up implementing a master-slave set up, you may want to look into high availability and auto-failover if your Master instance were to go down. Redis has a really great utility built into the application that can monitor your Master and Slaves, detect when the Master is down, and automatically re-configure your servers to promote one of the slaves to the new master. The utility is called Redis Sentinel, and you can read about that here: https://redis.io/topics/sentinel
For your question about race conditions, it depends on how exactly you write your jobs that are pushed onto the queue. For your use case though, it doesn't sound like this would be too much of an issue, but it really depends on the constraints of the third-party system. Either way, if you are subject to a race condition, you can still implement a solution for it, but would likely need to use something like a Redis Lock (https://redis.io/topics/distlock). Taylor recently added a new feature to the upcoming Laravel version 5.6 that I believe implements a version of the Redis Lock in the scheduler (https://medium.com/#taylorotwell/laravel-5-6-preview-single-server-scheduling-54df8e0e139b). You can look into how that was implemented, and adapt for your use case if you end up needing it.
I am in process of developing an MFC based Windows based application, using PostgreSQL which would perform
Fetches information from the UI
Performs some logic and store related information to the database
The stored information has to be send immediately OR at schedule interval (ex. at 5:00 on xyz date) over the network
Currently, we have developed a dispacher mechanism (thread ), which constantly polls the database for new information inserted in the database. The thread fetches the information and send to the network module.
But, I feel this is not the correct approach as
Polling every time is a overhead. There can be times when there is nothing to execute
It is not real time , because we poll after every 5 seconds
So
Is there a way to send a trigger to my network module as soon as information is updated in database?
Or any better way to achieve this task?
Thanks in advance.
You can use the listen/notify feature of PostgreSQL for this.
http://www.postgresql.org/docs/current/static/sql-listen.html
http://www.postgresql.org/docs/current/static/sql-notify.html
The clients interested in the messages would execute a listen statement and the trigger would then notify them.
I don't use C# so, but according to the manual you can retrieve the messages in an asynchronous manner - which still involves some "lightweight" polling as the notification message is only sent as part of the answer of the server. The manual claims that running an "empty" statement (such as ;) will be enough. Using Java/JDBC I used a simple select 42 which doesn't impose a big workload on the server as no tables are touched.
This polling is defintely faster and more scalable than actually retrieving the table's data.
Yes you are right #RDX, you shouldnt poll it every time rather you could write a trigger in Postgres and from that trigger try calling a java program which could be seen in the below thread.
Calling java pgm from Postgres trigger
Do I really need Session Replication?
I am working on a number of web projects for a firm. Most of the projects are about one or two pages of input and then doing a save to a mysql database. Very Basic projects. My SA's are pushing to try to get session replication working in JBoss but I don't really see any need for it and all of its overhead.
We need load balancing and clustering so if the server does go down we can move the new requests to the backup service but I am not to big in session replication.
This is very low volume projects. I my eyes what is the odds of a user being in the project as the server goes down on the one or two pages.
I need to convince the SAs that session replication is an un-necessary complication in this instance. I am looking for pros and cons of session replication so that I can better structure my argument.
Well, the "pro" is that you have session failover, either in deliberate cluster member restarting or in inadvertent cluster-member failure. That's it.
Some of the "cons" are:
Session objects and their included objects have to be Serializable
You have to choose Session persistence or replication and manage their configurations and/or datastore
You have to think about Session persistence/replication policies (e.g. every write, request end, time scheduled) and still risk losing the session or losing the most current state of it if a failure occurs before recent changes have been stored/replicated
Non-zero performance impact of replicating or or persisting, inversely related to how robust the replication policy is. (That is, the more likely that you'll get every session change replicated promptly, the worse the performance.)
We do session replication because we considered failover to be an absolute requirement years ago when we started this, but I think if I had it to do over again I'd suggest we don't bother for the majority of our applications.