I have just done an update to a GO application using pipeline, I noticed in my redis monitoring app that the number of clients connected just skyrocketed from 1/2 to 300+
I can't seem to find information on if this is normal for pipelines, they are connecting as a new client OR if this is unusual behaviour.
Has anyone else seen this ?
Redis connection is a global.
Redis setup is default.
used in some go routines.
Example usage..
pipe := rdb.Pipeline()
pipe.set(ctx, "hi", "hi")
_, err := pipe.Execute()
What tends to happen is everytime pipeline is ran like the above it will create a new client - even though the pipeline is using the global redis connection.
Only happens with pipeline, other redis calls are fine.
Related
I’m trying to test concurrently request (brute force test), with 1000 request/sec (same request id). I’m using Rest API.
The service running on distributed system, I’m using 3 pods.
The problem is I got duplication when inserting data, because we use distributed 3 pods consume concurrently (the request).
Then I tried some method to handle duplication:
Using sync.Mutex still got duplicate because each pods has process different var sync.Mutex . So it’s not lock each pods.
Using singleflight it’s work on 1 pods, but when run on distributed pods, it still duplicate, because different var singleflight also.
Using Redis Lock DLM (Distributed Lock Management). It’s work on 1 pods, but when distributed, it’s still duplicate also.
The one that I’ve not yet implemented is using Message Queue.
Before I try that, is there any simple method to handle duplication from distributed system ?
We are building a reporting app on Laravel that need to fetch users data from a third-party server that allow 1 request per seconds.
We need to fetch 100K to 1000K rows based on user and we can fetch max 250 rows per request.
So the restriction is:
1. We can send 1 request per seconds
2. 250 rows per request
So, it requires 400-4000 request/jobs to fetch a user data, So, loading data for multiple users is very time-consuming and the server gets slow.
So, now, we are planning to load the data using multiple servers, like 4-10 servers to fetch users data, so we can send 10 requests per second from 10 servers.
How can we design the system and process jobs from multiple servers?
Is it possible to use a dedicated server for hosting Redis and connect to that Redis server from multiple servers and execute jobs? Can any conflict/race-condition happen?
Any hint or prior experience related to this would be really helpful.
The short answer is yes, this is absolutely possible and is something I've implemented in production apps many times before.
Redis is just like any other service and can run anywhere, with clients from anywhere, connecting to it. It's all up to your configuration of the server to dictate how exactly that happens (and adding passwords, configuring spiped, limiting access via the firewall, etc.). I'd reccommend reading up on the documentation they have in the Administration section here: https://redis.io/documentation
Also, when you do make the move to a dedicated Redis host, with multiple clients accessing it, you'll likely want to look into having more than just one Redis server running for reliability, high availability, etc. Redis has efficient and easy replication available with a few simple configuration commands, which you can read more about here: https://redis.io/topics/replication
Last thing on Redis, if you do end up implementing a master-slave set up, you may want to look into high availability and auto-failover if your Master instance were to go down. Redis has a really great utility built into the application that can monitor your Master and Slaves, detect when the Master is down, and automatically re-configure your servers to promote one of the slaves to the new master. The utility is called Redis Sentinel, and you can read about that here: https://redis.io/topics/sentinel
For your question about race conditions, it depends on how exactly you write your jobs that are pushed onto the queue. For your use case though, it doesn't sound like this would be too much of an issue, but it really depends on the constraints of the third-party system. Either way, if you are subject to a race condition, you can still implement a solution for it, but would likely need to use something like a Redis Lock (https://redis.io/topics/distlock). Taylor recently added a new feature to the upcoming Laravel version 5.6 that I believe implements a version of the Redis Lock in the scheduler (https://medium.com/#taylorotwell/laravel-5-6-preview-single-server-scheduling-54df8e0e139b). You can look into how that was implemented, and adapt for your use case if you end up needing it.
I'm trying to handle Couchbase bootstrap failure gracefully and not fail the application startup. The idea is to use "Couchbase as a service", so that if I can't connect to it, I should still be able to return a degraded response. I've been able to somewhat achieve this by using the Couchbase async API; RxJava FTW.
Problem is, when the server is down, the Couchbase Java client goes crazy and keeps trying to connect to the server; from what I see, the class that does this is ConfigEndpoint and there's no limit to how many times it tries before giving up. This is flooding the logs with java.net.ConnectException: Connection refused errors. What I'd like, is for it to try a few times, and then stop.
Got any ideas that can help?
Edit:
Here's a sample app.
Steps to reproduce the problem:
svn export https://github.com/asarkar/spring/trunk/beer-demo.
From the beer-demo directory, run ./gradlew bootRun. Wait for the application to start up.
From another console, run curl -H "Accept: application/json" "http://localhost:8080/beers". The client request is going to timeout due to the failure to connect to Couchbase, but Couchbase client is going to flood the console continuously.
The reason we choose to have the client continue connecting is that Couchbase is typically deployed in high-availability clustered situations. Most people who run our SDK want it to keep trying to work. We do it pretty intelligently, I think, in that we do an exponential backoff and have tuneables so it's reasonable out of the box and can be adjusted to your environment.
As to what you're trying to do, one of the tuneables is related to retry. With adjustment of the timeout value and the retry, you can have the client referenceable by the application and simply fast fail if it can't service the request.
The other option is that we do have a way to let your application know what node would handle the request (or null if the bootstrap hasn't been done) and you can use this to implement circuit breaker like functionality. For a future release, we're looking to add circuit breakers directly to the SDK.
All of that said, these are not the normal path as the intent is that your Couchbase Cluster is up, running and accessible most of the time. Failures trigger failovers through auto-failover, which brings things back to availability. By design, Couchbase trades off some availability for consistency of data being accessed, with replica reads from exception handlers and other intentionally stale reads for you to buy into if you need them.
Hope that helps and glad to get any feedback on what you think we should do differently.
Solved this issue myself. The client I designed handles the following use cases:
The client startup must be resilient of CB failure/availability.
The client must not fail the request, but return a degraded response instead, if CB is not available.
The client must reconnect should a CB failover happens.
I've created a blog post here. I understand it's preferable to copy-paste rather than linking to an external URL, but the content is too big for an SO answer.
Start a separate thread and keep calling ping on it every 10 or 20 seconds, one CB is down ping will start failing, have a check like "if ping fails 5-6 times continuous then close all the CB connections/resources"
Recently I've been running worker machines that are brought up & down to perform tasks against Parse collections. But a small percentage of those machines are never torn down. I understand that NodeJS itself doesn't have a timeout set on connections so for a non-responsive server, my nodejs code may end up sitting there indefinitely, doing nothing and thus my worker machine wouldn't be torn down.
My question is: Does the parse js client have a timeout set?
Update # 1
Looking through Parse's nodejs client and I saw that they use https://github.com/driverdan/node-XMLHttpRequest which doesn't have the concept of timeout:
https://github.com/driverdan/node-XMLHttpRequest/pull/67
Could this be the culprit when a connection gets lost in limbo while talking to Parse and the code just sits there until a worker timesout?
So as I discovered, the answer is NO there isn't a default timeout.
I updated the underlying library for Parse JS SDK to provide a timeout:
https://github.com/ShoppinPal/node-XMLHttpRequest.git
And patched the most current version of parse sdk to use it:
https://github.com/ShoppinPal/parse-js
we have the system like the following:
Client: A regular client
P Server: An apache server which basically runs like a proxy (forwards queries to M Server)
M Server: Main server. All the work is done here.
and
Query Q1 is not equal to query Q1' because P Server adds some more variables (like server time, client IP) Q1 before sending it to M Server.
Response R1 doesn't change at all. It is forwarded to client without changing.
Same applies for Q2.
The problem is, if ajax requests with Q1 and Q2 are sent at the same time, Q2 waits in P Server for R1 to finish.
So this is basically a PHP/Apache problem. But, I don't know how to dig this problem. Is it multiple php script problem or multiple request problem?
I tried session_write_close() but that didn't change anything.
PHP's default session handler is file based and uses exclusive locks. Which means the request related to the same session will be serialised. Actually this is a positive thing, it can avoid lots of strange parallel programming issues, race conditions/etc.
You can change the session handler to something less paranoid if you prefer. Or you can write an own file based one, without exclusive file locking to avoid this behaviour. Anyway be careful when accessing critical resources... (semaphores, locking, etc can be your friends.)
There are a few related comments here:
php blocking when calling the same file concurrently
http://00f.net/2011/01/19/thoughts-on-php-sessions/
You can verify this behaviour with inotifywatch on your session's directory. (Formerly inotify things could cause kernel panics and other strange behaviours... test it only on a test server...)