We have a microservice with methods /foo and /bar (it's the very same microservices). It happened that we need to call bar from foo:
req --> /foo --> /bar
We bumped with the following issue, with big traffic: when N requests come in the same time, server register all N threads for calling the foo code. Then this code calls bar, and obviously, everything freezes, as there is no thread available to handle it.
Is there any pattern on how to deal with this situation?
(btw, I personally would not allow calling nested calls on the same context, but the decision is not mine).
Related
In one of my projects, my microservice XYZ talks to ABC. ABC has a elasticsearch REST high level client that queries a particular ES index. Let's call the endpoint defined in XYZ where XYZ talks to ABC as "/getData"
I have noticed that when /getData is not called often, the first invocation of it times out. For example, if the endpoint has not been called in 24 hours and then it gets called, it will timeout but subsequent calls don't timeout. And the pattern I have observed is that subsequent calls are also faster in returning the response.
I am using Spring, so is this some kind of lazy loading happening? Where the connections to ES become idle if they haven't been used in a while and if so, can I configure it somehow to avoid this problem.
I would like to understand how to detect the failed service ( in a fast / reliably manner ), ie the service what is a root of all 5xx responses?
Let me try to elaborate. Lets assume we have 300+ microservices and they have only synchroneous http interaction via GET request without any data modifications ( we assume it for simplicity ). Each customer request may transform in calling ~10 different microservices, moreover it could be a 'calling chain' of requests, ie API Gateway calls 3 different microservices, each of them calls 1-5 more, each of these 1-5 calls 1-5 more etc.
We closely monitor 5xx errors on each of microservice and react on these errors.
Now one of the microservices fails. It appears to be somewhere in the end of a 'calling chain', which means that other microservices which depend on it will start to return 5xx as well.
Yes, there are circuit breakers, yes they become 'triggered / opened' and instead of calling the downstream service, they right away return error as well ( in most cases we cannot return a good fallback like empty response ).
So we see that relatively big amount of microservices return 5xx. Like 30-40 microservices return 5xx, we see 30-40 triggered / opened circuit breakers.
How to detect a failed microservice, a root of all evil, in a fast manner?
Did anybody encounter this issue?
Regards
You will need to implement a distributed tracing solution that tracks the origin transaction with a global ID. The name of this global identifier is typically called Correlation ID and it is generated by the very first service which creates the request and propagated to all the other microservices that work together to fulfill the request.
Take a look at OpenTracing for your implementation needs. It provides libraries for you to add the instrumentation required for identifying faulty microservices in a distributed environment.
However, if you really do have 300 microservices all using synchronous calls...maybe it is time to consider using asynchronous communications to eliminate the temporal coupling inherent in synchronous communications.
Sample Code to explain my problem.
Sender
The Sender, that sends the request to the Server(over ZMQ) in format
["sender-1", "sender-1-bdc1c14624076b691b8d9e15fbd754f1"]
..
["sender-99","sender-99-a2123d02c2989ef57da370bb13ba60e4"]
Server
The Server, upon receiving the data from sender relays it back to receiver in the same format.
["sender-1", "sender-1-bdc1c14624076b691b8d9e15fbd754f1"]
...
["sender-99","sender-99-a2123d02c2989ef57da370bb13ba60e4"]
Receiver
The Receiver, upon receiving the request from the Server just prints the message.
Description:
If I don't use a mutex defined in this line(inside the server). I see some data appearing at the receiver end which does not adhere to the above format/standard.
Example the Server would print (at this line)
"Sending sender-97 -- sender-97-9a284488454c8e8fd22bbbcf678895e9"
"Sending sender-98 -- sender-98-447eb5be94e7f6d949e764d7c88239ad"
But on the receiver end, I see messages that look like this.
sender-97 -- sender-98
sender-98-22d9f01a8f801f559a9ff4e388813251 --
Question:
To me, it seems like a possible Thread issue(I may be wrong). Wherein the data that is passed to (inside Server) Celluloid (send) is getting changed by other thread.
I was under the impression that Celluloid handles almost all of your threads issue.
Is my understanding correct about all this?
You need an asynchronous logger.
Whenever you use puts you are outputting to a buffer which is actually very slow, compared to other things. What you ought to do, is use an actor used purely in the place of puts everywhere.
The best thing would be to instantiate a supervised actor, say named :logger and then interact with it inside your other actors by Celluloid[:logger].async.ouput() or else forward output to Celluloid[:logger].async so that every time you use output or console or something like that, it fires off an asynchronous method call. That way, even though your actor doing work is moving on to something else, the console output will still be in perfect sequence.
Your console output is being mangled because of the above issue with asynchrony in programming itself, not Celluloid ... and this is an extremely common problem with and without ZeroMQ.
Yes defer is disrupting the sequence of your Receiver output, but no this is not a Thread error in Celluloid. This is the natural behavior of asynchronous operations. In this case, you need to remove defer {} and keep async.received_data() like you have it.
Otherwise, as you see, Server will bombard Receiver with messages out of sequence. It doesn't help either that you're directly managing threads on top of all this, rather than letting Celluloid::ZMQ do it for you.
Also: I would remove all "external" thread management, and remove the defer {} wrapper... and let Celluloid::ZMQ smooth everything out, and keep sequence for you.
Let's say I'm writing a Spring web-service that gets called by an external application. That application requests data that I need to load from an external resource. Furthermore, the design has it that it calls my service more than once with different parameters. In other words, the user sitting in front of the application presses one button, which generates a bunch of requests to my web-service in a very short time frame.
My web-service parses the parameters and comes up with necessary requests to the external resource. The logic has it that it may cause calling the external resource with the same parameters over and over again, which makes this the ideal candidate for caching.
Example:
The user presses that one button in the application
Application initiates ten requests to my web-service
My web-service receives them in parallel
After analysing the parameters of all requests, overall I'd need to call the external resources 15 times, but the parameters are mostly equal and only show that three calls would be enough to serve the 15 intended calls.
However, one call to the external resource may take some time.
As far as I understand how Spring does caching it writes the result of a #Cachable method into the cache. Apparently this means that before it treats another invocation of that method with the same parameters as cache hit, it must have a result of a previous invocation. This means that it doesn't provide support for pending method calls.
I need something like "Hey, I just saw a method invocation with the same parameters a second ago, but I'm still waiting for the result of that invocation. While I can't provide a result yet, I will hold that new invocation and reuse the result for it."
What are my options? Can I make Spring do that?
You can't make Spring do that out-of-the-box for very good reasons. The bottom line is that locking and synchronizing is very hard using a specific cache implementation so trying to do that in an abstraction is a bit insane. You can find some rationale and some discussion here
There is a discussion of using ehcache's BlockingCache SPR-11540
Guava also has such feature but the cache needs to be accessed in a very specific way (using a callback) that the CacheInterceptor does not really fit. It's still our plan to try to make that work at some point.
Do not forget that caching must be transparent (i.e. putting it on and off only leads to a performance change). Trying to parse arguments and compute what call should be made to your web service has high chances to lead to side effects. Maybe you should cache things at a different place?
if there is a synchronized block in the service() method of say for example in ActionServlet of struts, how will multiple requests/threads work if it is a busy site having large number of hits.
Will each thread wait for the next other one to release lock from the synchronized block? Will that create delay in response
Don't synchronize the service method of a servlet.
If you synchronize the service method of a servlet, you are in fact making a "reservation on access" for a thread at a time for that instance of the Servlet.
The Struts ActionServlet class is a HttpServlet and basically the doGet and doPost methods are of interest here. If we are to speak about Struts, the process method is the main entry point, but the same principle applies to all methods as it does for the general service method.
The idea is this.
When you declare a servlet in your web.app, the servlet container (e.g. Tomcat) will create only one instance of that servlet. This means there is only one instance to serve all requests.
If more requests arrive at the same time, each request thread gets it chance at the service method because there is no synchronization enforced.
If you have 10 request threads, each will execute concurrently in the service method. This is normally safe because processing done in the service method does not involve any state related to the current request it is handling. You go into issues if you add state to your servlets. Here is an article with more details on the subject.
Now back to Struts.
Struts uses a pattern called a Front Controller with the ActionServlet being that controller. This will in turn delegate specific requests to specific Action classes as specified in its configuration (a.k.a struts-config.xml).
All incoming request pass though here. If you place synchronization at this point (the Struts process method or the servlet service method higher up) you are reserving the servlet for a thread at a time. In case of struts, you are reserving all processing of a request to a single thread at a time.
That means that if 10 request arrive simultaneousely, in the case without synchronization all can execute side by side, while in the case with synchronization request 2 will have to wait until request 1 is done, 3 waits for 2 and so on (i.e. requests are sequentially processed). And this means poor performance.
Maybe for the lucky user that made request 1 there will be no performance impact but number 10 will have to wait. Then how about number 100? 200?
There is no need to synchronize the entry point if you program your application with thread safety in mind. If your application is of the sort that you just can't avoid synchronization, then synchronizing the entry point will reduce performance.
P.S. Just one other thing. If you are thinking into moving the synchronization lower in the process, namely the Action classes, note that they are not thread safe either and there is only one instance of it inside the Struts framework.