I want to hit a service which has a paginated response using Web Client. ie. I hit a service, check if it returns hasMoreElements as TRUE, then call the service again with updated request parameters like START_ROW, END_ROW, PAGE_NUMBER. What is the best approach to achieve this? Currently am just looping through the results and hitting the service again. But their should be a better approach to this. PFB my pseudocode. Any libraries I can use?
boolean hasMoreElements=true;
while(!hasMoreElements==false)
{
response=webClient.post().header(HEADERS).bodyValue(REQUEST).block();
Get the NEW START ROW, END ROW, AND PAGE NUMBER and SET in the REQUEST
Get the hasMoreElements value
}
Use JPARepository with paging for this.
You can return a list of objects and check if its length is less then the limit passed, if yes then you can stop fetching.
You could also return a Page or a Slice instead which gives you a little bit more information about the current and next fetch cycle
I am currently working on performance improvements for a React-based SPA. Most of the more basic stuff is already done so I started looking into more advanced stuff such as service workers.
The app makes quite a lot of requests on each page (most of the calls are not to REST endpoints but to an endpoint that basically makes different SQL queries to the database, hence the amount of calls). The data in the DB is not updated too often so we have a local cache for the responses, but it's obviously getting lost when a user refreshes a page. This is where I wanted to use the service worker - to keep the responses either in cache store or in IndexedDB (I went with the second option). And, of course, the cache-first approach does not fit here too well as there is still a chance that the data may become stale. So I tried to implement the stale-while-revalidate strategy: fetch the data once, then if the response for a given request is already in cache, return it, but make a real request and update the cache just in case.
I tried the approach from Jake Archibald's offline cookbook but it seems like the app is still waiting for real requests to resolve even when there is a cache entry to return from (I see those responses in Network tab).
Basically the sequence seems to be the following: request > cache entry found! > need to update the cache > only then show the data. Doing the update immediately is unnecessary in my case so I was wondering if there is any way to delay that? Or, alternatively, not to wait for the "real" response to be resolved?
Here's the code that I currently have (serializeRequest, cachePut and cacheMatch are helper functions that I have to communicate with IndexedDB):
self.addEventListener('fetch', (event) => {
// some checks to get out of the event handler if certain conditions don't match...
event.respondWith(
serializeRequest(request).then((serializedRequest) => {
return cacheMatch(serializedRequest, db.post_cache).then((response) => {
const fetchPromise = fetch(request).then((networkResponse) => {
cachePut(serializedRequest, response.clone(), db.post_cache);
return networkResponse;
});
return response || fetchPromise;
});
})
);
})
Thanks in advance!
EDIT: Can this be due to the fact that I put stuff into IndexedDB instead of cache? I am sort of forced to use IndexedDB instead of the cache because those "magic endpoints" are POST instead of GET (because of the fact they require the body) and POST cannot be inserted into the cache...
I have a HTML form, which I submit via http post.
There are two cases:
Case 1: The data is valid and data on the server will be updated accordingly
Case 2: The data is invalid and the http response contains an error message for the user.
Which http status codes should be used for each case?
I use htmx to submit the form. This means I don't need to use the POST/Redirect/GET pattern.
This question is not about JSON-APIs.
The complete list of HTTP response codes published by the Mozilla Foundation is pretty comprehensive and easy-to-read, so I'd recommend always consulting it as a guide. For the generic use-cases mentioned by you, there are a couple of different codes you can return - depending on what happens with the data on the server, and what you want to happen in the user's browser.
CASE 1: data is valid, data on server is updated
Base on your short description, the different status codes that might be applicable are:
200 (OK): you are updating an existing record on your own server - eg., the user is submitting a form which updates their existing contact information on your website - and information was received, and the record updated successfully. The response would usually contain a copy of the updated record.
201 (Created): you are not updating an existing record, but rather, creating a new record on your server - eg., your user is adding a new phone number to their contact details, which is saved in your database as a separate 'phone' record. The response should contain a copy of the newly created record.
205 (Reset Content): the same as 200, but implies that the browser view needs to be refreshed. This is useful when the record that is being updated has values that are dynamically calculated by the server, and which might change automatically depending on the values you're submitting. For example, if you have a user adding extra information to their online profile, which might grant them special status, badges and privileges on the website. This means, that if the user is viewing their profile information, that information will need to be updated with the new 'status' automatically granted by the server. The 205 response body will normally be empty, which means that to update the browser view your response-handling code will need to:
do further ajax requests and update the relevant part(s) of your
interface with new information from the server, or
redirect the user to a new URL, or
reload the entire page.
If working with HTMX, a 200 or 201 response would include the actual html snippet of the record that you want updated on the page - and HTMX will replace it automatically for you when it receives the response. With a 205 response, you could send an HX-Trigger response header that would call a custom event on the interface elements that need to update themselves - see the examples in the docs.
CASE 2: data is invalid, data on server is not updated
The status code that needs to be returned in case of an error varies depending on what caused the error. Errors that the server believes are the responsibility of the client - such as 'sending invalid data' - have a status code in the 4XX range. Some of the common errors in that range include 404 ('Not Found'), 403 ('Forbidden'), and 'Unauthorised' (401).
In the case of a client sending data that the server cannot process because it is 'not valid' - either because the request itself is malformed, or because the data doesn't pass some business validation logic - the current advice is to return status 400 (Bad Request).
Many years ago, some people believed that the status code 400 should only be used to indicate a malformed request (syntactical error) - not to indicate a failure in business validation logic (semantic error). There was a lot of debate, and temporarily a new status code (422) was created, that was supposed to cover semantic errors, exclusively. In 2014, however, the official definition of the status 400 code was changed to allow for the inclusion of both syntactical and semantical errors - which rendered status 422 essentially unnecessary.
You can find lots of discussions and explanations online about the differences between 400 and 422, and some people still argue passionately about this to this day. In practice, however, the 400 code is all you'll need - and you can include a response body in it that explains in detail, if needed, the cause of the error.
Note that when working with HTMX, a response with a 400 code should trigger an htmx:responseError event automatically. You can trap that event, for example, to update your form interface elements, in case of data validation errors caught by the server.
Well, 200 OK and 201 Created are the best for successful result.
For invalid data I would return 422 Unprocessable Entity, because the headers are correct, but body is not (though parseable by the server). The caveat is some HTTP clients won't handle 422 properly and in this case you have to use 400 Bad Request, however, the most of the modern clients will be fine.
You have said it is not about JSON APIs, but how will you meet this type of requirement - it is not clear whether this is relevant for your scenario???
SERVER DRIVEN BEHAVIOUR
I cannot see how a client could ever decide an HTTP status code based on input data. How would the client deal with these examples?
The call is not authenticated (cookie or token) - an API would return 401 - this tells the UI to perform a retry action.
The call is not authorized - an API would return 403 or 404 and the UI would present an error display.
The data is malformed or invalid according to domain specific checks - an API would return 400 and tell the UI what is wrong so that it can perform actions.
Something went wrong in server processing, eg data cannot be saved because database is down.
MY THOUGHTS
htmx looks interesting but a key requirement before using it would be ensuring that htmx can read server side error responses and use values returned. Maybe there is an elegant way to do this ...
Maybe I am just paranoid :). But it is worth being careful when choosing technologies that there are no blocking issues. Lack of error handlng control would be a blocking issue in most systems.
I'm using htmx 1.8 with asp.net core 6.0.
This works for me.
controller:
//server side validation failed
Response.StatusCode = 422;
return PartialView("Core", product);
client side javascript:
document.body.addEventListener('htmx:beforeOnLoad', function (evt) {
if (evt.detail.xhr.status === 422) {
//
// allow 422 responses to swap as we are using this as a signal that
// a form was submitted with bad data and want to rerender with the
// error messages
//
evt.detail.shouldSwap = true;
evt.detail.isError = false;
}
});
200 OK or 201 Created are the best choice for a successful POST request.
However, for invald data, you can pass 415 Unsupported Media Type
I have a publisher that executes a long-running and large query on MongoDB and returns the data in a Flux. Entities that are marked in the database as "processed" will be filtered out and the entities are then buffered and passed to a concatMap operator (so that all buffered ≤elements are processed before elements in the next buffer are processed). It looks something like this:
Flux<Entity> entitiesFromMongoDb = myMongoRepository.executeLargeQuery();
entitiesFromMongoDb.filter(entity -> !entity.isProcessed())
.buffer(10)
.concatMap(bufferedEntityList ->
Flux.fromIterable(bufferedEntityList)
.flatMap(makeExternalCall)
.then()));
Where makeExternalCall calls a third-party remote server and sets the entity to processed after the call has been made. In most cases this works fine, but when the remote server is really slow or has an error then makeExternalCall will retry (with exponential backoff) the operation to the remote server. In some cases it can take quite a while before all 10 external calls have been processed. In fact it can take so long that the myMongoRepository.executeLargeQuery() publisher is restarted and the query is executed again. Now we run into a problem that I'll try to describe here:
Entity A is read from the database (i.e. it's returned in the flux produced by myMongoRepository.executeLargeQuery()). It's not yet marked as "processed" which means that entity.isProcessed() will return false and it'll be retained in the stream.
The external server is really slow or down so that makeExternalCall is forced to retry the operation before entity A has been marked as "processed" in the DB.
myMongoRepository.executeLargeQuery() is restarted and the query is executed again.
Entity A is read from the database once more. But the problem is that there's already another instance of entity A in-flight since it has not yet been marked as "processed" by the previous call to myMongoRepository.executeLargeQuery().
This means that makeExternalCall will be called twice for entity A, which is not optimal!
I could make an additional request to the DB and check the status of processed for each entity in the makeExternalCall method, but this will cause additional load (since an extra request is necessary for each entity) to the DB which is not optimal.
So my question is:
Is there a way to somehow "restart" the entire stream, and thus clear intermediary buffers (i.e. remove entity A that is in-flight from the ongoing stream) when the MongoDB query triggered by myMongoRepository.executeLargeQuery() is restarted/re-executed? Or is there a better way to handle this?
I'm using Spring Boot 2.2.4.RELEASE, project reactor 3.3.2.RELEASE and spring-boot-starter-data-mongodb-reactive 2.2.4.RELEASE.
Not sure If I understood the problem completely. But trying to answer as it sounds interesting.
As you need to be aware of the requests which are already being processed by the makeExternalCall, can you maintain a set / local cache which contains the entities which are being processed?
Set<Entity> inProgress = new HashSet<>();
Flux<Entity> entitiesFromMongoDb = myMongoRepository.executeLargeQuery();
entitiesFromMongoDb.filter(entity -> !entity.isProcessed())
.buffer(10)
.map(bufferedEntityList -> { // remove the inprogress requests to avoid redundant processing
bufferedEntityList.removeIf(inProgress::contains);
return bufferedEntityList;
})
.concatMap(bufferedEntityList ->
inProgress.addAll(bufferedEntityList);
Flux.fromIterable(bufferedEntityList)
.flatMap(makeExternalCall) //assuming once processed, it emits the entity object
.map(entity -> { //clear growing set
inProgress.remove(entity);
return entity;
})
.then()));
This approach is not a good solution when you need to scale your application horizontally. In that case instead of maintaining a local cache, you could go for an external cache server like redis.
We're trying to figure out what effect enabling AppCache in the frontend controller has on caching without calling any cache directives on the response object.
I had presumed that simply adding the following line and setting default_ttl to 1:
$kernel = new AppCache($kernel);
would not change the behaviour of the application without calling a cache directive on the response. But as soon as we add this line (and cache:clear) our server is able to handle far more requests per second, which suggests that there is some caching going on.
Turning on debug and setting default_ttl to an hour all we see in the http headers is
X-Symfony-Cache: GET /: miss
Does this mean that there is no reverse proxy caching going on? If so what explains the performance increase?
Any clarification on what happens in this situation would be awesome.
This line
$kernel = new AppCache($kernel);
enables the Symfony2 Reverse Proxy. For further explanation follow this link: http://symfony.com/doc/current/book/http_cache.html#symfony2-reverse-proxy. The performance increase should be clear now.
The header means that the "Symfony-Cache" got a "GET" request and found no cached data ("miss"). If you call the same page multiple times in a row the header should change to something like:
X-Symfony-Cache: GET /: HIT 42