We're writing a Spring service that makes an HTTP endpoint available through which a video (or audio) file from an Amazon S3 store can be streamed. The basic idea is that you can type in an url in the Google Chrome address bar, and the service will fetch the file from S3 and stream it, in such a way that the user can start watching immediately without having to wait for a download to complete, and that the user can click on a random spot in the video's progress bar and immediately start watching the video from that spot.
The way I understand this should work in theory, is that Chrome starts downloading the file. The service responds with HTTP 200 and includes an Accept-Ranges: bytes and a Content-Length: filesize header. The filesize is known, because we can query that as metadata from S3 without fetching the entire file. Including these headers causes the browser to cancel the download, and request the file again with a Range: bytes=0-whatever header (where whatever is some chunk size that Chrome decides). The service then responds with HTTP 206 (Partial content) and the requested byte range, which we can determine easily because S3 supports the same range protocol. Chrome then requests successive chunks from the service, until the stream ends.
On the Spring side, we're sending the data out in a ResponseEntity<InputStreamResource> (as per this SO answer).
However, we observe in practice that while Chrome's cancels its first request after a few hundred bytes. However, it sends a second request with a Range: bytes=0- header, effectively asking for the entire file. The server responds with an HTTP 206. As a result, is has only downloaded a few hundred bytes of video, and the video obviously doesn't start playing.
Interestingly, in Firefox it all works properly. Unfortunately, our app needs to support Chrome. Are we missing some part of the protocol?
It turns out we had an off-by-one error in the Content-Range response header.
The syntax is Content-Range: bytes start-end/total. With a total of 10, if you want to get the entire range, you need to specify bytes 0-9/10, not 0-10/10, which was what we were doing.
Of course with the larger sizes of real files, and the actual ranges of chunks in the middle of such files, this error was a lot harder to notice than in the contrived example in the previous paragraph... ಠ_ಠ
Related
I'm building an application that builds a pdf file and returns it to the client whenever it receives a request.
Since some of these pdf files might take some time to generate, I would like to periodically send some sort of status update back to client while it is running.
When it's finished building the pdf file, it should be returned to the client as well.
Something akin to:
func buildReport(writer http.ResponseWriter, request *http.Request){
//build pdf build pdf file
for { //for example purposes only
writer.Write([]byte("building. Please wait."))
}
pdf.OutputFileAndClose("report.pdf")
//set header to pdf so that the client knows it's a PDF
writer.Header().Set("Content-Type", "application/pdf")
http.ServeFile(writer, request, "report.pdf")
}
func main() {
http.HandleFunc("/", buildReport)
http.ListenAndServe(":8081", nil)
}
Setting the header might not work, as the writer can only have one header.
TL;DR is that it cannot be implemented that way. You need to
An API that requests the PDF creation. That queues PDF creation job in a task queue (so that too many PDF creation requests won't blow the HTTP server worker pool)
Provide an API that allows you to check where are you with the PDF rendering (I am assuming that the job can provide interim stats). This is going to be polled by the client on a regular basis.
An API to pull the PDF once it is ready.
Hope this helps and best of luck with your project.
This is by no means comprehensive, but a reasonable example of how you might construct your API (which needs to be asynchronous, as the previous respondent pointed out) can be found here: https://www.adayinthelifeof.nl/2011/06/02/asynchronous-operations-in-rest/
The job queue model is a pretty common one. I would recommend you also write a basic API binding library (you'd want this for your own testing purposes in any case) so that your users can understand how you intend them to use the API, and in writing it, you'll get a better sense of how asynchronous REST interactions feel from the end user side.
Contrary to what others have said, what you want is in fact
directly possible but requires fullfillment of the two preconditions:
HTTP/1.1 and above.
You'll be sending custom content to the clients — not PDF data
directly, — and they're prepared to accept and parse it.
You can then employ the so-called "chunked" payload encoding specifically
invented to handle "streamed" downloads where the server does not know how
many bytes it's about to send.
So you may invent some creative kind of payload where you first periodically
stream a "no op" / "progress" marker and then the actual payload.
Say, while the file is being prepared you periodically send a line of text
reading "PROCESSING" + LF then, when a result is ready you send
a line of text "READY" SIZE + LF where SIZE is the size, in bytes,
of the immediately following PDF document. After the document is streamed,
the server signals the end of data.
Hence the stream would look like
PROCESSING
PROCESSING
…
PROCESSING
READY 8388608
%PDF-1.3
…
%%EOF
The clients have to be able to parse this information from the stream
they're receiving and have a simple FSM in place to switch from state to
state as they fetch your stream.
The server has to make sure it flushes the stream after each "informational" line otherwise the whole thing would not be "interactive".
If you have a good idea about the overall state of the processing of the
document, each "status update" line could include the percentage of the work done, like in "PROCESSINGNN" + LF.
I hope you can help
I have a image server that generates images on the fly.
I'm using varnish to cache generated images.
I need to record how many requests (per image) varnish receives as well as if it was a hit or miss (pass gets marked as miss). Currently, I'm writing access logs with hit/miss to file, I then using crontab process this access-log file and write the data to my db...
What I would like to do instead is:
Have Varnish make a request to my backend notifying it of a cache hit (and if possible the response size (bytes)).
My backend could then save this data...
Is this at all possible and if so how?
In-case anybody is interested:
2 varnish instances each with 1 (java+tomcat) backend.
Service manipulates and generates each image specific to the requirements made in the request...
Below are per day:
Over 35 million page views where each page has at least 3 images in it.
Varnish gets around 3+ million requests for images (images are also cached by the browser).
Varnish has a 87% hit rate
Response times for a hit are a few micro seconds
Response times for a miss are 50ms to 1000ms depending on the size of the image (both source and output)
The best way of doing this is to have a helper process that tails varnishlog output and
does the HTTP calls when needed.
You can do this by logging the necessary data with std.log() in vcl_deliver, so the
helper process gets all the data it needs. Use obj.hits > 0 to check if this was a cache hit.
If you really really need to do it inline (and slowing down all your cache hits badly), you
can use libvmod-curl:
https://github.com/varnish/libvmod-curl
If you are going to send a request to a stats server from within your vcl I would try to incorporate some type of aggregate request, where you send it every 100 (or whatever) requests instead of every single incoming request.
Like the other answer, I would recommend using varnishncsa (or varnishlog) with a process that tails the log file. There could be some delay in that method but if that is acceptable then I would consider post processing the varnish log when logrotated runs. This way you have a full day's worth of data and you can churn through it, producing whatever report you need.
I have page, on which i am holding base64 representation of some images(around 1mb each ), now i am posting this data via ajax to the server(contentType is default - url-encoded). This works fine if i have one or two images to be sent, but if i have more than 2mb of request data, the server doesn't accept that, and request parameters doesn't have anything, so i increased the maxPostSize in my tomcat, and it started accepting more data as well, but i am a bit apprehensive if this would create memory issues, especially if i have lot of images ?
Also i tried changing the contentType to multipart/formdata, but it errors out, "saying the request was rejected because no multipart boundary was found".
EDIT
I think i should elaborate more, actually requirement is something like this - User clicks on an upload link, he should be able to upload the file and then he should be able to see a thumbnail of the image on the page(all this without refreshing the page). I tried following approaches for this.
Reading the file using file reader, showing the thumbnail and then explicitly triggering the upload, when user clicks on save, simple but not cross browser, doesn't work in IE
Allowed user to upload the file send the base64 version of the image from the server and when user clicks on save, send the base64 string back to the server and convert it back to byte array and save to the db.
Now, i have a screen where all the records are by default editable, so clicking on save means, sending the image strings for all the records to the server, which will ofcourse create memory issues.
"3". Not implemented yet, but thinking of first saving the other fields(the non image fields) and then explicitly saving the images one by one(looks okay, but number of requests will be high)
Waiting for someone to suggest a 4th approach, hope i have explained enough
Disclaimer ... Not done anything like this, but...
Why not send each image separately? :)
To test iPhone and iPad fetching and caching images from an external web server, I'd like to make my own server delay for 0.5, 1, or 3 seconds before an image is returned, using a URL that looks like:
http://www.mysite.com/getImage.cgi?pic=pic001.png&delayWanted=3
is there a simple way to to this?
Using Ruby, the two ways I was thinking of was to use CGI and change the HTTP header to return the type of image/png, the no-cache header and "expire time = 1 year ago", and provide the content size, and then open the image file and output the data, but this probably will need to best match how a standard web server returns the HTTP headers. Another way is to sleep first, and then simply send an HTTP code of redirect to the real picture's URL, so the web server should handle the rest. Or is there a simpler way?
I don't know about Ruby, but if you can insert a Linux box you are root on into the network path (or the server you run Ruby on qualifies) there's "netem" for emulating lifelike network conditions: delay, packet loss, jitter...
http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
I think the simpler way to do that is as you say, using sleep, for example
sleep(2.minutes)
it takes only one line of code and delays your response.
I'm using the VB6 Winsock control. When I do a POST to a server I get back the response as multiple Data arrival events.
How do you know when all the data has arrived?
(I'm guessing it's when the Winsock_Close event fires)
I have used VB6 Winsock controls in the past, and what I did was format my messages in a certain way to know when all the data has arrived.
Example: Each message starts with a "[" and ends with a "]".
"[Message Text]"
When data comes in from the DataArrival event check for the end of the message "]". If it is there you received at least one whole message, and possibly the start of a new one. If more of the message is waiting, store your message data in a form level variable and append to it when the DataArrival event fires the next time.
In HTTP, you have to parse and analyze the reply data that the server is sending back to you in order to know how to read it all.
First, the server sends back a list of CRLF-delimited header lines, which are terminated by a blank CRLF-delimited line by itself. You then have to look at the actual values of the 'Content-Length' and 'Transfer-Encoding' headers to know how to read the remaining data.
If there is no 'Transfer-Encoding' header, or if it does not contain a 'chunked' item in it, then the 'Content-Length' header specifies how many remaining bytes to read. But if the 'Transfer-Encoding' header contains a 'chunked' item, then you have to read and parse the remaining data in chunks, one at a time, in order to know when the data ends (each chunk reports its own size, and the last chunk reports a size of 0).
And no, you cannot rely on the connection being closed after the reply has been sent, unless the 'Connection' header explicitally says 'close'. For HTTP 1.1, that header is usually set to 'keep-alive' instead, which means the socket is left open so the client can send more requests on the same socket.
Read RFC 2616 for more details.
No, the Close event doesn't fire when all the data has arrived, it fires when you close the connection. It's not the Winsock control's job to know when all the data has been transmitted, it's yours. As part of your client/server communication protocol implementation, you have to tell the client what to expect.
Suppose your client wants the contents of a file from the server. The client doesn't know how much data is in the file. The exchange might go something like this:
client sends request for the data in the file
the server reads the file, determines the size, attaches the size to the beginning of the data (let's say it uses 4 bytes) that tells the client how much data to expect, and starts sending it
your client code knows to strip the first 4 bytes off any data that arrives after a file request and store it as the amount of data that is to follow, then accumulate the subsequent data, through any number of DataArrival events, until it has that amount
Ideally, the server would append a checksum to the data as well, and you'll have to implement some sort of timeout mechanism, figure out what to do if you don't get the expected amount of data, etc.