I am developing a component that will provide a GET REST endpoint that will return a large (up to 2MB) JSON array of data. I will be using REDIS to cache the JSON array and the REST endpoint is implemented using a Web API 2 project.
I assumed that I could just return the data in the Response Stream so that I don't have to have very large strings in memory, but when I took a look at StackExchange.Redis I couldn't find any methods that return a Stream.
It appears that https://github.com/ctstone/csredis does, but this project looks pretty static now.
Am I missing something, or is there a workaround for this?
Related
we have created some API endpoints that return GeoJson responses to specific calls (Featurecollection). We have stored spatial data in our DB (SQL-Server), we are using EF6 and GeoJson.NET to convert the spatial data into proper GeoJson. This is working fine, most of the time. Sometimes, though, the responses contains over 3000 polygons (postcode areas) and their properties (name etc.), that the response grows to nearly 250Mb. For this we already use Gzip to compress those responses. After compression the response is +/- 75Mb. That is still taking too long to download for some of our clients.
Is it somehow possible to divide the GeoJson response into smaller parts and send that to the client? I know something similar can be done with binary data (like images), there we use chunked transfer. Can we also use that for Json data? If so, does anyone know how?
Firefox console with large API response
We have spent a lot of hours on the inet and on stackoverflow, but none of the findings satisfied us in the way we planned a file upload in Spring context.
A few words towards our architecture. We have a node.js client which uploads files into a Spring Boot app. Let us call this REST endpoint our "client endpoint". Our Spring Boot application acts as middleware and calls endpoints of a "foreign system", so we call this endpoint a "foreign" one, due to distinction. The main purpose is the file handling between these two endpoints and some business logic in between.
Actually, the interface to our client looks like this:
public class FileDO {
private String id;
private byte[] file;
...
}
Here we are very flexible because it is our client and our interface defintion.
Due to the issue that under load our system has run out of memory sometimes, we plan to reorganize our code into a more stream-based, reactive approach. When i write "under load", i mean heavily under load, e.g. hundreds of file uploads at the same time with big files from at least some MB to at most 1GB. We know, that this tests don't represent real applications use cases, but we want to be prepared.
We spent some research into our challenge and we ended up with profiler tools showing us that according to our REST endpoints we store the files as byte arrays completely in our memory. Thats all, but not efficient.
Currently we are facing this requirement to deliver a REST endpoint for file upload and push these files into another REST endpoint of some foreign system. Doing so, our main applications intention is to be some middle tier for file upload. According to this initial situation we are looking forward to not have those files as a whole in our memory. Best would be a stream, maybe reactive. We are partially reactive with some business functions already, but at the very beginning of being familiar with all that stuff.
So, what are our steps so far? We introduced a new Client (node.js --> Spring Boot) interface as the following one. This works so far. But is it really a stream based approach? First metrics have shown, that this doesn't reduce memory utilization.
#PostMapping(value="/uploadFile", consumes = MediaType.MULTIPART_FORM_DATA_VALUE)
#ResponseStatus(HttpStatus.CREATED)
public Mono<Void> upload(#RequestPart(name = "id") String id, #RequestPart(name = "file") Mono<FilePart> file) {
fileService.save(id, file);
...
}
First question: is this type Mono<> right here? Or should we better have Flux of DataBuffer or something? And, if so, how the client shoud behave and deliver data in such a format that it is really a streaming approach?
The FileService class then should post this file(s) into the foreign system, perhaps do something else with given data, at least log the id and the file name. :-)
Our code in this FileService.save(..) actually looks like the following in between:
...
MultipartBodyBuilder bodyBuilder = new MultipartBodyBuilder();
bodyBuilder.asyncPart(...take mono somehow...);
bodyBuilder.part("id", id);
return webClient.create("url-of-foreign-system")
.uri("/uploadFile")
.syncBody(bodyBuilder.build())
.retrieve()
.bodyToMono(Result.class);
...
Unfortunately, the second REST endpoint, the one of our foreign system, looks little different to our first one. It will be enriched by data from another system. It takes some FileDO2 with an id and a byte array and some other meta data specific to the second foreign system.
As said, our approach should be to minimize the memory footprint of the actions in between client and foreign system. Sometimes we have not only to deliver data to that system, but also do some business logic that maybe slows down the whole streaming process.
Any ideas to do that in a whole? Currently we have not clue to do that all...
We appreciate any help or ideas.
I wanted to test the response times of a GraphQL endpoint, and a RESTful endpoint as I haven't ever used GraphQL before, and I am about to use it in my next Laravel project.
So I am using Lighthouse PHP package to serve a GraphQL endpoint from my Laravel app, and also I have created a RESTful endpoint.
Both endpoints(GraphQL and RESTful) are intended to get all Users(250 users) from my local Database.
So based on the test what I have noticed here is that, when I tested this both endpoints on Postman, the RESTful endpoint response is faster than GraphQL endpoint.
Can I know why GraphQL endpoint's response takes more time than RESTful while both endpoints are getting same data?
GraphQL endpoint result for GET request (response time: 88ms)
GraphQL endpoint result for POST request (response time: 88ms)
RESTful endpoint result (response time: 44ms)
There's no such thing as a free lunch.
GraphQL offers a lot of useful features, but those same features invariably incur some overhead. While a REST endpoint can effectively pull data from some source and regurgitate it back to the client, even for a relatively small dataset, GraphQL will have to do some additional processing to resolve and validate each individual field in the response. Not to mention the processing required to parse and validate the request itself. And this overhead only gets bigger with the size of the data returned.
If you were to introduce additional features to your REST endpoint (request and response validation, support for partial responses, ability to alias individual response fields, etc.) that mirrored GraphQL, you would see the performance gap between the two shrink. Even then, though, it's still somewhat of an apples and oranges comparison, since a GraphQL service will go through certain motions simply because that's what the spec says to do.
TLDR: Your REST example is simple and less complicated
In Lighthouse it is creating a AST for parsing the graphql request and your schema. It then afterwards passes all the directives and so on to figure out what you are trying to do. It also has to validate your query, to see if you can actually run it on the schema.
Depending on how you defined it in your application, there is a lot of steps it is passing through. However this can be reduced by multiple different ways, the parsing of your graphql schema can be cached, you could cache the result, use deferred fields (prob. wont speed up this example). You can read more about this in the performance section of the docs.
You are not specifying how your REST is setup, if you are using some kind of REST standard where it has to parse the data also. If you add more features, there is more code to run through, hence higher load speed.
As of Lighthouse v4, we have made significant performance increases by lazy-loading the minimally required fields and types from the schema. That turns out to bring about a 3x to 10x performance increase, depending on the size of your schema.
You probably still won't beat a single REST endpoint for such a simple query. Lighthouse will begin to shine on more heavily nested queries that join across multiple relationships.
Try enabling opcache on the server. This decreased my gql response time from 200ms to 20ms
I have an ASP.NET Core server application that uses Entity Framework Core to provide data of its SQL server.
There are clients that can consume the data via REST API calls.
At the beginning of the communication, it is required to have all the data on the clients but using the existing REST calls it takes minutes as the context contains thousands of entities.
So I looked for ideas and tried the following.
The best looking idea was some kind of serializing, so I created the following method:
public byte[] GetData()
{
string data = Newtonsoft.Json.JsonConvert.SerializeObject(this.ChangeTracker,
new Newtonsoft.Json.JsonSerializerSettings {
ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore });
return CompressAsGZip(data);
}
The results:
Serializing (and then compressing) the ChangeTracker of the context
Initially the ChangeTracker is empty so I can't do it unless I query all the data
Serializing (and then compressing) the entire DbContext
It has so many objects that after 20%, I got an OutOfMemoryException
Should I create a database backup and send the compressed bak file? I guess I couldn't restore it to anywhere as the client database provider is different (SQLite).
What would be the best way to send all the data to the client as fast as it can be?
I am new into designing a RESTFul webservice and I have to design a webservice that is going to respond one get method.
Below are the points that I have keep in mind.
Webservice will accept start and end date.
We have to hit the database and have to return the data in form of json/xml. Json object that needs to be returned is having three fields.
Data is containing large amount of json object and the webservice is going to be called every minute.
I need to understand the best practice to design such RESTFul webservice. Because the major challenge that I feel is webservice is going to be called every minute and we cannot afford of creating new object that finally returned as JSON or XML objects. We cannot cache the results because data is going to change every minute.
Please help.
Thanks in advance.