Let's assume we have an HTTP/HTTPS traffic hitting an endpoint via GET request. The endpoint then returns a response. Is there a mechanism to delay the response for X seconds(assuming X < request timeout)? I have tried using BREAKPOINT already intuitively and was wondering if there is a direct solution for this.
There are two solutions:
Using the Network Condition.
Right-Click on the request you'd like to delay -> Tools -> Network Condition -> Create a rule with the 3G profile.
The Network Condition will be applied to the entire domain. If you'd like to add the delay for a particular request (path + query), you might check out the sleep() func from the Scripting tool.
function onResponse(context, url, request, response) {
console.log("Start sleep");
// Sleep 5 seconds
sleep(5000);
// Done
return response;
}
Related
Project Reactor has a variety of timeout() operators.
The very basic implementation raises TimeoutException in case no item arrives within the given Duration. The exception is propagated downstream , and to upstream it sends cancel signal.
Basically my question is: is it possible to somehow react (and do something) specifically to timeout that happened downstream, not just to cancelation that sent after timeout happened?
My question is based on the requirements of my real business case and also I'm wondering if there is a straight solution.
I'll simplify my code for better understanding what I want to achieve.
Let's say I have the following reactive pipeline:
Flux.fromIterable(List.of(firstClient, secondClient))
.concatMap(Client::callApi) // making API calls sequentially
.collectList() // collecting results of API calls for further processing
.timeout(Duration.ofMillis(3000)) // the entire process should not take more than duration specified
.subscribe();
I have multiple clients for making API calls. The business requirement is to call them sequantilly, so I call them with concatMap(). Then I should collect all the results and the entire process should not take more than some Duration
The Client interface:
interface Client {
Mono<Result> callApi();
}
And the implementations:
Client firstClient = () ->
Mono.delay(Duration.ofMillis(2000L)) // simulating delay of first api call
.map(__ -> new Result())
// !!! Pseudo-operator just to demonstrate what I want to achieve
.doOnTimeoutDownstream(() ->
log.info("First API call canceled due to downstream timeout!")
);
Client secondClient = () ->
Mono.delay(Duration.ofMillis(1500L)) // simulating delay of second api call
.map(__ -> new Result())
// !!! Pseudo-operator just to demonstrate what I want to achieve
.doOnTimeoutDownstream(() ->
log.info("Second API call canceled due to downstream timeout!")
);
So, if I have not received and collected all the results during the amount of time specified, I need to know which API call was actually canceled due to downstream timeout and have some callback for this "event".
I know I could put doOnCancel() callback to every client call (instead of pseudo-operator I demonstrated) and it would work, but this callback reacts to cancelation, which may happen due to any error.
Of course, with proper exception handling (onErrorResume(), for example) it would work as I expect, however, I'm interesting if there is some straight way to somehow react specifically to timeout in this case.
I am using cypress to test our web application.
In certain pages there are different endpoint requests that are executed multiple times. [ e.g. GET /A GET /B GET /A].
What would be the best practise in cypress in order to wait for all requests to finish and guarantee that page has been fully loaded.
I don't want to use a ton cy.wait() commands to wait for all request to be processed. (there are a lot of different sets of requests in each page)
You can use the cy.route() feature from cypress. Using this you can intercept all your Get requests and wait till all of them are executed:
cy.server()
cy.route('GET', '**/users').as('getusers')
cy.visit('/')
cy.wait('#getusers')
I'm sure this is not recommended practice but here's what I came up with. It effectively waits until there's no response for a certain amount of time:
function debouncedWait({ debounceTimeout = 3000, waitTimeout = 4000 } = {}) {
cy.intercept('/api/*').as('ignoreMe');
let done = false;
const recursiveWait = () => {
if (!done) {
// set a timeout so if no response within debounceTimeout
// send a dummy request to satisfy the current wait
const x = setTimeout(() => {
done = true; // end recursion
fetch('/api/blah');
}, debounceTimeout);
// wait for a response
cy.wait('#ignoreMe', { timeout: waitTimeout }).then(() => {
clearTimeout(x); // cancel this wait's timeout
recursiveWait(); // wait for the next response
});
}
};
recursiveWait();
}
According to Cypress FAQ there is no definite way. But I will share some solutions I use:
Use the JQuery sintax supported by cypress
$('document').ready(function() {
//Code to run after it is ready
});
The problem is that after the initial load - some action on the page can initiate a second load.
Select an element like an image or select and wait for it to load. The problem with this method is that some other element might need more time.
Decide on a maindatory time you will wait for the api requests (I personaly use 4000 for my app) and place a cy.wait(mandatoryWaitTime) where you need your page to be loaded.
I faced the same issue with our large Angular application doing tens of requests as you navigate through it.
At first I tried what you are asking: to automatically wait for all requests to complete. I used https://github.com/bahmutov/cypress-network-idle as suggested by #Xiao Wang in this post. This worked and did the job, but I eventually realized I was over-optimizing my tests. Tests became slow. Test was waiting for all kinds of calls to finish, even those that weren't needed at that point in time to finish (like 3rd party analytics etc).
So I'd suggest not trying to wait for everything at a step, but instead finding the key API calls (you don't need to know the full path, even api/customers is enough) in your test step, use cy.intercept() and create an alias for it. Then use cy.wait() with your alias. The result is that you are waiting only when needed and only for the calls that really matter.
// At this point, there are lots of GET requests that need to finish in order to continue the test
// Intercept calls that contain a GET request with a request path containing /api/customer/
cy.intercept({ method: 'GET', url: '**/api/customer/**' }).as("customerData");
// Wait for all the GET requests with path containing /api/customer/ to complete
cy.wait("#customerData");
// Continue my test knowing all requested data is available..
cy.get(".continueMyTest").click()
I have a question about Spring Reactive WebClient...
Few days ago I decided to play with the new reactive stuff in Spring Framework and I made one small project for scraping data only for personal purposes. (making multiple requests to one webpage and combining the results).
I started using the new reactive WebClient for making requests but the problem I found is that the client not emitting response for every request. Sounds strange. Here is what I did for fetching data:
private Mono<String> fetchData(String uri) {
return this.client
.get()
.uri(uri)
.header("X-Fsign","SW9D1eZo")
.retrieve()
.bodyToMono(String.class)
.timeout(Duration.ofSeconds(35))
.log("category", Level.ALL, SignalType.ON_ERROR, SignalType.ON_COMPLETE, SignalType.CANCEL, SignalType.REQUEST);
}
And the function that calls fetchData:
public Mono<List<Stat>> fetch() {
return fetchData(URL)
.map(this::extractUrls)
.doOnNext(System.out::println)
.doOnNext(s-> System.out.println("all ids are "+s.size()))
.flatMapIterable(q->q)
.map(s -> s.substring(7, 15))
.map(s -> "http://d.flashscore.com/x/feed/d_hh_" + s + "_en_1") // list of N-length urls
.flatMap(this::fetchData)
.map(this::extractHeadToHead)
.collectList();
}
and the subscriber:
FlashScoreService bean = ctx.getBean(FlashScoreService.class);
bean.fetch().subscribe(s->{
System.out.println("finished !!! " + s.size()); //expecting same N-length list size
},Throwable::printStackTrace);
The problem is if I made a little bit more requests > 100.
I didn't get responses for all of them, no error is thrown or error response code is returned and subscribe method is invoked with size different from the number of requests.
The requests I made are based on List of Strings (urls) and after all responses are emitted I should receive all of them as list because I'm using collectList(). When I execute 100 requests, I expect to receive list of 100 responses but actually I'm receiving sometimes 100, sometimes 96 etc ... May be something fails silently.
This is easy reproducible here is my github project link.
Sample output:
all ids are 176
finished !!! 171
Please give me suggestions how I can debug or what I'm doing wrong. Help is appreciated.
Update:
The log shows if I pass 126 urls for example:
onNext(ReactorClientHttpResponse{request=[GET/some_url],status=200}) is called 121 times. May be here is the problem.
onComplete() is called 126 times which is the exact same length of the passed list of urls
but how it's possible some of the requests to be completed without calling onNext() or onError( ) ? (success and error in Mono)
I think the problem is not in the WebClient but somewhere else. Environment or server blocking the request, but may be I should receive some error log.
ps. Thanks for the help !
This is a tricky one. Debugging the actual HTTP frames received, it seems we're really not getting responses for some requests. Debugging a little more with Wireshark, it looks like the remote server is requesting the end of the connection with a FIN, ACK TCP packet and that the client acknowledges it. The problem is this connection is still taken from the pool to send another GET request after the first FIN, ACK TCP packet.
Maybe the remote server is closing connections after they've served a number of requests; in any case it's perfectly legal behavior. Note that I'm not reproducing this consistently.
Workaround
You can disable connection pooling on the client; this will be slower and apparently doesn't trigger this issue. For that, use the following:
this.client = WebClient.builder()
.clientConnector(new ReactorClientHttpConnector(new Consumer<HttpClientOptions.Builder>() {
#Override
public void accept(HttpClientOptions.Builder builder) {
builder.disablePool();
}
}))
.build();
Underlying issue
The root problem is that the HTTP client should not onComplete when the TCP connection is closed without sending a response. Or better, the HTTP client should not reuse a connection while it's being closed. I'll report back here when I'll know more.
How long can the browser wait before an error is shown before server answers for request? Can this time be unlimited?
If you are using a jQuery $.ajax call you can set the timeout property to control the amount of time before a request returns with a timeout status. The timeout is set in milliseconds, so just set it to a very high value. You can also set it to 0 for "unlimited" but in my opinion you should just set a high value instead.
Note: unlimited is actually the default but most browsers have default timeouts that will be hit.
When an ajax call is returned due to timeout it will return with an error status of "timeout" that you can handle with a separate case if needed.
So if you want to set a timeout of 3 seconds, and handle the timeout here is an example:
$.ajax({
url: "/your_ajax_method/",
type: "GET",
dataType: "json",
timeout: 3000, //Set your timeout value in milliseconds or 0 for unlimited
success: function(response) { alert(response); },
error: function(jqXHR, textStatus, errorThrown) {
if(textStatus==="timeout") {
alert("Call has timed out"); //Handle the timeout
} else {
alert("Another error was returned"); //Handle other error type
}
}
});
Yes and no. Yes the server can do it or be configured to do so, no the browsers (i dont know about version/distributor specifics) may have timeouts enabled.
There are 2 solutions though for achieving/emulating this over HTTP:
If this is simple a long running script and you're waiting for results this isnt the way to go, you should instead do as previous poster mentioned and use async processing with server polling for the results, this would be a much more sure fire solution. For example: a thumbnail script from an image processor server side: the user uploads an image, the server immediately returns a 200 and a "Job ID". The client (javascript^^) can then use the JobID to request the job status/result.
If your goal is to have something like a realtime connection between browser and server (1 way connection, once the request is made by the browser no further info can be sent without using new requests (ajax^^)), this is called long polling/reverse ajax and can be used for real-time communication over http. There are several techniques using 2 long polled requests in parallel so that once one of them timeout the second one becomes the active and the first one attempts to reconnect.
Can you explain a bit more about what you're trying to achieve - do you have a long running process on a server, do you want to change the settings on just a local machine or are you after a way to manage it for large numbers of users?
How long the browser will wait depends on a number of factors e.g. where the timeout occurs - is it at the TCP level, the server or the local browser?
If you've got a long running process on a server and you want to update a webpage afterwards the typical way to handle it is to run the long process asynchronously and notify the client when it's complete e.g. have an ajax call that polls the server, or use HTTP 1.1 and serve out a notification stream to the client.
In either case it's still possible for the connection to be closed so the client will still need the ability to re-open it.
I found, that in case of a normal (HTML page) request, browsers run to timeout after cca. 30 secs. It's important, because other participiants probably follows it: proxies, routers (do routers play in this game? I'm not sure). I am using 4 sec long server-side delay (if there's nothing to send to the client), and my AJAX client performs another HTTP request immediatelly (I am on local network, there's no internet lag). 4 sec is long enough to not to overload the server and network with frequented polls, and is short enough for the case, when somehow one poll falls out of the row which the client can't detect and handle.
Also, there're other issues with comet (long HTTP request): browser's limit on number of simultaneous HTTP request, handling of client-side events (must sent to the server immediatelly), server/network down detection and recovery, multi user handling etc.
I occasionally have some long running AJAX requests in my Wicket application. When this occurs the application is largely unusable as subsequent AJAX requests are queued up to process synchronously after the current request. I would like the request to terminate after a period of time regardless of whether or not a response has been returned (I have a user requirement that if this occurs we should present the user an error message and continue). This presents two questions:
Is there any way to specify a
timeout that's specific to an AJAX
or all AJAX request(s)?
If not, is there any way to kill the current request?
I've looked through the wicket-ajax.js file and I don't see any mention of a request timeout whatsoever.
I've even gone so far as to try re-loading the page after some timeout on the client side, but unfortunately the server is still busy processing the original AJAX request and does not return until the AJAX request has finished processing.
Thanks!
I think it won't help you to let the client 'cancel' the request. (However this could work.)
The point is that the server is busy processing a request that is not required anymore. If you want to timeout such operations you had to implement the timeout on the server side. If the operation takes too long, then the server aborts it and returns some error value as the result of the Ajax request.
Regarding your queuing problem: You may consider to use asynchronous requests in spite of synchronous ones. This means that the client first sends a request for starting the long running process. This request immediately returns. Then the client periodically polls the server and asks if the process has finished. Those poll requests also return immediately saying either that the process is still running or that it has finished with a certain result.
Failed solution: After a given setTimeout I kill the active transports and restart the channel, which handles everything on the client side. I avoided request conflicts by tying each to an ID and checking that against a global reference that increments each time a request is made and each time a request completes.
function longRunningCallCheck(refId) {
// make sure the reference id matches the global id.
// this indicates that we are still processing the
// long running ajax call.
if(refId == id){
// perform client processing here
// kill all active transport layers
var t = Wicket.Ajax.transports;
for (var i = 0; i < t.length; ++i) {
if (t[i].readyState != 0) {
t[i].onreadystatechange = Wicket.emptyFunction;
t[i].abort();
}
}
// process the default channel
Wicket.channelManager.done('0|s');
}
}
Unfortunately, this still left the PageMap blocked and any subsequent calls wait for the request to complete on the server side.
My solution at this point is to instead provide the user an option to logout using a BookmarkablePageLink (which instantiates a new page, thus not having contention on the PageMap). Definitely not optimal.
Any better solutions are more than welcome, but this is the best one I could come up with.