In an HTTP service implementation, I'm allowing multipart http file upload with the PUT method.
However, before writing everything into the hard drive, I'd like to check if the size of the file is OK. E.g. has to be under 50MB.
How to do this in go?
You might want to check out http.MaxBytesReader. It limitates the maximum amount of data which can be received from/sent by a single request.
MaxBytesReader prevents clients from accidentally or maliciously sending a large request and wasting server resources.
Related
I have an empty API in laravel code with nginx & apache server. Now the problem is that the API takes a lot of time if I try with different files and the API responds quickly if I try with blank data.
Case 1 : I called the API with a blank request, that time response time will be only 228ms.
Case 2 : I called the API with a 5MB file request, then file transfer taking too much time. that's why response time will be too long that is 15.58s.
So how can we reduce transfer start time in apache or nginx server, Is there any server configuration or any other things that i missed up ?
When I searched on google it said keep all your versions up-to-date and use php-fpm, but when I configure php-fpm and http2 protocol on my server I noticed that it takes more time than above. All server versions are up-to-date with the current version.
This has more to do with the fact one request has nothing to process so the response will be prompt, whereas, the other request requires actual processing and so a response will take as long as the server requires to process the content of your request.
Depending on the size of the file and your server configuration, you might hit a limit which will result in a timeout response.
A solution to the issue you're encountering is to chunk your file upload. There are a few packages available so that you don't have to write that functionality yourself, an example of such a package is the Pionl Laravel Chunk Upload.
An alternative solution would be to offload the file processing to a Queue.
Update
When I searched on google about chunking it's not best solution for
small file like 5-10 MB. It's a best solution for big files like
50-100 MB. So is there any server side chunking configuration or any
other things or can i use this library to chunking a small files ?
According to the library document this is a web library. What should I
use if my API is calling from Android and iOS apps?
True, chunking might not be the best solution for smaller files but it is worthwhile knowing about. My recommendation would be to use some client-side logic to determine if sending the file in chunks is required. On the server use a Queue to process the file upload in the background allowing the request to continue processing without waiting on the upload and a response to be sent back to the client (iOS/Android app) in a timely manner.
I created an API recently that allows video and audio files to be uploaded and it was working fine until one customer uploaded a large video and I got this error:
Failed to read the request form. Multipart body length limit 134217728 exceeded.
From many posts it's evident that the philosophy behind request size limitation is to reduce the chances of DDoS attack. (Please correct me if I'm wrong).
What security holes do I open up if I do not limit the max request length for file upload to my webserver?
https://manage.accuwebhosting.com/knowledgebase/2997/ASPNet-Error--Maximum-request-length-exceeded.html
However, I uploaded a large video myself, and watched the bandwidth both on my local PC and on the server. I have a VPS for this app and I can watch bandwidth using Task Manager's Performance tab, by selecting Ethernet and seeing the graph of send and receive.
The point is, I saw that a high traffic going on for some minutes when I was uploading the file, and then I saw that error. This means that the resources of the server are being consumed even when there ASP.NET Core rejects the request.
So, I don't understand this part. How ASP.NET Core diminishes the chance of DoS, or DDoS attacks by limiting request size, while bandwidth is taken in reality? A logical way would to drop the request from the beginning for multipart/form-data content types if the request payload/body was too huge and even do not consume the bandwidth.
Dotnet can only know the size of the file by uploading it (consuming bandwidth in your example).
The solution to this problem is to set a smaller limit for the size of uploaded files.
The way dotnet works, it will stream the file (process it little by little). Once it hits the max upload size, it will return an error.
You are right that this doesn't prevent an attacker from uploading many large files - dotnet will still have to upload/stream them until the max size is reached. The way to mitigate attacks like these is through other means - rate limiting, IP blocking, DDoS protection at the CDN etc
Found that the issue is at the load balancer. Basically the api is behind a load balancer proxy. I need to configure the nginx there. I will ask a fresh question for that
I have created an http Server in Golang using stock net/http package. Once in a while I do get some http call with very huge data in the url(It is an API serevr and is expected). For such requests server is responding with HTTP 414 code.
Now I need to know the current length supported by Golang standard http package. From the truncated requests my guess is 10,000 bytes. Is there a way to raise it to something bigger, like 20,000 bytes. I understand that this might affect the server performance, but I need this as a hotfix until we move all the API's to POST.
Post is the way to go, but I need a hotfix. Our clients need huge time to move to POST, I need to support GET for now. I am owner of server, I guess there should be a way to raise the url length limit
Edit:-
In the doc:- https://golang.org/src/net/http/server.go, there is MaxHeaderBytes field. The default value is 1MB which is way more than the maximum data I will ever receive(20KB), other header data should not be that big I guess. Then why is it failing with over 8KB of request data?
I am working on an application where the user has the potential to download thousands of files in one request into a zip file. Obviously, this will not be practical for our server. What would be the best way to go about serving up thousands of files to users?
Right now, what I have been working on is just have the jquery fileDownload library make a request for 100 files, then in the success handler call the fileDownload again for another 100 files offset by 100. The problem with this is that the fileDownload library (or the server) waits about 20 seconds until the fileDownload fail callback is called.
The other problem with this method is it isn't practical for the client to receive hundreds of pop windows asking them if they want to download 100 files.
We also won't be able to send back thousands of files in the response because our server doesn't and won't have that much memory.
This is purely opinion based on my experience but two options i have seen in use:
Option 1:
Batch process files, compress, then advise user of download location. This should be limited number of files and size tho as it can burn out the server resources. I don't recommend this if you have large number of users.
Option 2 (Best):
Batch process files into compressed file, then either enable uses to FTP into the location to obtain the files, or if your users have FTP location, have the file transfered over to the FTP location. I can tell you definitely this is most effective and is used by number of corporations i have been invovled with.
If you configure Apache to enable gzip compression for your static HTML/CSS/JS/etc. files, it automatically outputs a gzipped version to any client that sends an appropriate Accept-Encoding request header. (And for other clients, it just sends the raw uncompressed file.)
My question is: does Apache recompress the raw file every time it is requested by a gzip-accepting client? Or does it cache the gzipped copy, and only recompress it if it notices the last-modified time on the file has changed?
And if it does cache a gzipped copy of your files, where is this cache stored?
No it doesn't cache the gzipped file.
However the cost of compressing the file is less than the cost of squirting the extra packets across the network, hence even without caching you will see lower overall CPU usage (and lower memory usage, and fewer context switches) on your server - and a faster response at the client.
Note that the compressed file is NOT stored in the temp folder - mod_deflate reads input into a fixed size buffer in memory - and when the buffer is full (or the stream ends) the content is compressed and handed back to the webserver.
It will use even less CPU (although speed won't improve noticably) if the content is pre-compressed or cached serverside - there's multiple ways of doing this - mod_rewrite can test for the presence of filename.gz and serve it up in place of filename or you can use a reverse proxy (assuming the content is also served up with caching instructions).
No, it does not. This is described in the mod_deflate documentation now:
Since mod_deflate re-compresses content each time a request is made, some performance benefit can be derived by pre-compressing the content and telling mod_deflate to serve them without re-compressing them.
Apache does not keep any cached files. It only keeps the files you tell it to keep. Here is how compression works:
Browser requests page and states it accepts compression
Server finds page and reads the header of the request.
Server sends page to the browser (compresses if stated it accepts in header request - compressed file is stored in memory/temp folder)
Browser receives the information and displays (after decompression if compressed). The browser then caches the page and images.
Server removes any hint of the compressed file from memory/temp folder to free up space for the next request. It does log in the access_log the transaction.
The browser when it requests the same file or page again. It sends a request to the server stating that it accepts compression, and the current files and Modified Date. From here the server would respond that they are the same and send no additional information, or send only the changed files based on the Modified Date.