I'm using the VB6 Winsock control. When I do a POST to a server I get back the response as multiple Data arrival events.
How do you know when all the data has arrived?
(I'm guessing it's when the Winsock_Close event fires)
I have used VB6 Winsock controls in the past, and what I did was format my messages in a certain way to know when all the data has arrived.
Example: Each message starts with a "[" and ends with a "]".
"[Message Text]"
When data comes in from the DataArrival event check for the end of the message "]". If it is there you received at least one whole message, and possibly the start of a new one. If more of the message is waiting, store your message data in a form level variable and append to it when the DataArrival event fires the next time.
In HTTP, you have to parse and analyze the reply data that the server is sending back to you in order to know how to read it all.
First, the server sends back a list of CRLF-delimited header lines, which are terminated by a blank CRLF-delimited line by itself. You then have to look at the actual values of the 'Content-Length' and 'Transfer-Encoding' headers to know how to read the remaining data.
If there is no 'Transfer-Encoding' header, or if it does not contain a 'chunked' item in it, then the 'Content-Length' header specifies how many remaining bytes to read. But if the 'Transfer-Encoding' header contains a 'chunked' item, then you have to read and parse the remaining data in chunks, one at a time, in order to know when the data ends (each chunk reports its own size, and the last chunk reports a size of 0).
And no, you cannot rely on the connection being closed after the reply has been sent, unless the 'Connection' header explicitally says 'close'. For HTTP 1.1, that header is usually set to 'keep-alive' instead, which means the socket is left open so the client can send more requests on the same socket.
Read RFC 2616 for more details.
No, the Close event doesn't fire when all the data has arrived, it fires when you close the connection. It's not the Winsock control's job to know when all the data has been transmitted, it's yours. As part of your client/server communication protocol implementation, you have to tell the client what to expect.
Suppose your client wants the contents of a file from the server. The client doesn't know how much data is in the file. The exchange might go something like this:
client sends request for the data in the file
the server reads the file, determines the size, attaches the size to the beginning of the data (let's say it uses 4 bytes) that tells the client how much data to expect, and starts sending it
your client code knows to strip the first 4 bytes off any data that arrives after a file request and store it as the amount of data that is to follow, then accumulate the subsequent data, through any number of DataArrival events, until it has that amount
Ideally, the server would append a checksum to the data as well, and you'll have to implement some sort of timeout mechanism, figure out what to do if you don't get the expected amount of data, etc.
Related
I am creating an application that allows users to submit JSON or Base64 image data via socket.io
The goal I am trying to achieve is:
if JSON is submitted, the message can have a maximum size of 1MB
if a Base64 image is submitted, the message can have a maximum size of 5MB
From the socket.io docs I can see that:
you can specify a maxHttpBufferSize option value that allows you to limit the maximum message size
namespaces allow you to split logic over a single connection
However, I can't figure out the correct way to get the functionality to work the way I have described above.
Would I need to:
set up 2 separate io instances on the server, one for JSON data and the other for Base64 images (therefore allowing me to set separate maxHttpBufferSize values for each), and then the client can use the correct instance, depending on what they want to submit (if so, what is the correct way of doing this?)
set up 1 instance with a maxHttpBufferSize of 5MB, and then add in my own custom logic to determine message sizes and prevent further actions if the data is JSON and over 1MB in size
set this up in some totally different way that I haven't thought of
Many thanks
From what I can see in the API, maxHttpBufferSize is a parameter for the underlying Engine.IO server (of which there is one instance per Socket.IO Server Instance). Obviously you're free to set up two servers but I doubt it makes sense to separate the system into two entirely different applications.
Talk of using Namespaces to separate logic is more about handling different messages at different endpoints (for example you would register a removeUserFromChat message handler to a user connecting via an /admin namespace, but you wouldn't want to register this to a user connecting via the /user namespace).
In the most recent socket server I set up, I defined my own protocol where part of the response would contain a HTTP status code, as well as a description that could be displayed to the user. For example I would return 200 on success. If I was uploading a file via a REST HTTP Interface, I would expect a 400 (BAD REQUEST) response if my request couldn't be processed - and I believe that this makes sense for your use case. Alternatively you could define your own custom 4XX error code if the file is too large, and handle this in your UI purely based on the code returned. Obviously you don't need to follow the HTTP protocol, and the design decisions are ultimately up to you, but in my opinion it makes sense to return some kind of error response in your message handler.
I suspect that the maxHttpBufferSize has different use at lower levels than your use case. When sending content over network, content is split into 'n bytes' of packets and when a application writes 'n' bytes, the network sends a packet over network (the less the n, more overhead due to network headers. The more the n, high latency because of waiting involved in accumulating n bytes before sending). Documentation is not clear about maxHttpBufferSize but it could be the packet size (n) configuration, not limit on the max data on connection.
It seems, http request header Content-Length might serve your purpose. It gives the actual object size based on that you can make a decision.
I'm building an application that builds a pdf file and returns it to the client whenever it receives a request.
Since some of these pdf files might take some time to generate, I would like to periodically send some sort of status update back to client while it is running.
When it's finished building the pdf file, it should be returned to the client as well.
Something akin to:
func buildReport(writer http.ResponseWriter, request *http.Request){
//build pdf build pdf file
for { //for example purposes only
writer.Write([]byte("building. Please wait."))
}
pdf.OutputFileAndClose("report.pdf")
//set header to pdf so that the client knows it's a PDF
writer.Header().Set("Content-Type", "application/pdf")
http.ServeFile(writer, request, "report.pdf")
}
func main() {
http.HandleFunc("/", buildReport)
http.ListenAndServe(":8081", nil)
}
Setting the header might not work, as the writer can only have one header.
TL;DR is that it cannot be implemented that way. You need to
An API that requests the PDF creation. That queues PDF creation job in a task queue (so that too many PDF creation requests won't blow the HTTP server worker pool)
Provide an API that allows you to check where are you with the PDF rendering (I am assuming that the job can provide interim stats). This is going to be polled by the client on a regular basis.
An API to pull the PDF once it is ready.
Hope this helps and best of luck with your project.
This is by no means comprehensive, but a reasonable example of how you might construct your API (which needs to be asynchronous, as the previous respondent pointed out) can be found here: https://www.adayinthelifeof.nl/2011/06/02/asynchronous-operations-in-rest/
The job queue model is a pretty common one. I would recommend you also write a basic API binding library (you'd want this for your own testing purposes in any case) so that your users can understand how you intend them to use the API, and in writing it, you'll get a better sense of how asynchronous REST interactions feel from the end user side.
Contrary to what others have said, what you want is in fact
directly possible but requires fullfillment of the two preconditions:
HTTP/1.1 and above.
You'll be sending custom content to the clients — not PDF data
directly, — and they're prepared to accept and parse it.
You can then employ the so-called "chunked" payload encoding specifically
invented to handle "streamed" downloads where the server does not know how
many bytes it's about to send.
So you may invent some creative kind of payload where you first periodically
stream a "no op" / "progress" marker and then the actual payload.
Say, while the file is being prepared you periodically send a line of text
reading "PROCESSING" + LF then, when a result is ready you send
a line of text "READY" SIZE + LF where SIZE is the size, in bytes,
of the immediately following PDF document. After the document is streamed,
the server signals the end of data.
Hence the stream would look like
PROCESSING
PROCESSING
…
PROCESSING
READY 8388608
%PDF-1.3
…
%%EOF
The clients have to be able to parse this information from the stream
they're receiving and have a simple FSM in place to switch from state to
state as they fetch your stream.
The server has to make sure it flushes the stream after each "informational" line otherwise the whole thing would not be "interactive".
If you have a good idea about the overall state of the processing of the
document, each "status update" line could include the percentage of the work done, like in "PROCESSINGNN" + LF.
We've noticed that with concurrent chunking enabled, the first chunk isn't always sent first. Sometimes, the first chunk is sent last. How do we force the chunks to be sent in order while also being sent concurrently? At the very least, how do we force the first chunk to be sent first?
Your server should not necessarily care about the order of the chunks. Fine Uploader sends a POST when all chunks have completed, so your server knows when to combine them. The library does in fact send the requests in order, but it's possible that they are not arriving at your server in the order they were sent. This is not something Fine Uploader has any control over, and depending on the order of requests to arrive or complete in this case is a recipe for failure.
I am trying to check pop and smtp values entered by user.. I wish to validate that pop and smtp say for example(pop.gmail.com,smtp.gmail.com) which is entered by user is correct or wrong.
For that I am sending only one request to server by taking both pop and smtp values entered by user which will do two tasks
1. Checks user entered pop by making connection to that particular server ,
2. Checks user entered smtp by sending 1 mail to some dummy mail id..
I finished all these tasks..
But now what my requirement is, I have to show the user after validating each thing.. I mean in ui i have to show as
POP connection Checked.. ok
smtp connection Checked.. ok like that.
But i sent only one request to server for doing both these tasks..So i need to get intermediate status from server after finishing each tasks..So only i can update in client side UI.. But i don't know is it possible to get intermediate responses from server for a single request... Any idea friends? If so can you come up with a little bit of code...
Expecting the suggestions?
you should take a look in the long polling technique, it is possible to retrieve partial response but it doesn't work on all browsers.
You can use HEAD request instead of GET or POST which only return HTTP header
Slightly off topic - but sending a dummy mail can be "dangerous".
Many servers "note" if you try and send to a local address, which does not exist. For example - if the server's domain is "whatever.com" and you send to a random address, say aaa#whatever.com, and "aaa" is not a valid user, then the server notices this.
The server may then take an action like blocking you, as a sender, for a period of time. (This helps to reduce spam from dictionary attacks.) So your "test" ends up effectively blocking the real mail from being delivered.
The reverse is also true. Let's say you try to send to an external address, which you know is valid (your own email address for example) as the test. In this case the from address must be a valid internal address. If you use an invalid internal address, or worse an address which is not internal, it's likely the server will refuse to deliver the mail (at best) and at worst again institute a temporary block.
The key factor in both these situations is that although the SMTP protocol is very "loose", SMTP servers watch very closely for "bad behavior" because this is one way of distinguishing a spamming program. So any hide of "incorrect" behavior can lead to it arbitrarily refusing to accept your mails (usually for a limited period of time.)
Incidentally, back to your original question.
Both of your tests are pretty much instantaneous. Even if the email server is on the other side of the world you can do both checks inside a couple seconds. So chances are even if you send back 2 packets, to the user they'll appear as "arriving together". And since 1 request from the browser can only handle 1 response from the server you would need to send the response in 2 packets.
ie do first test - send first part of response - do second test - send second part of response.
For a normal HTTP packet this is no big deal. Do some sort of flush / send after the first response is ready, and then again after the second response. The browser is used to displaying partial pages as they arrive.
However for an AJAX request you'll need to get into your framework at quite a low level. Most frameworks, that I'm aware of, require the incoming Async packet to be "complete" before they start to parse it. This is especially true if the packet is formatted as say xml where partial parsing is useless in pretty much all cases.
I have two phones connected to a Wifi access point, both have ip in the private range.
One of the phones has a HTTP server running on it and the other phone acts like a client. The client sends GET requests data to the server as name/
value pairs in the URL query string. At the moment the server is only sending on HTTP.OK on receiving the query string.
What is happening is the client may not be stationary and maybe moving around so it may not be possible for it to be in range always of the Wifi access
point due to that I am not getting all the data sent from the client at the server end.
I want to ensure that all data sent is actually received by the server.
What kind of error correction should I implement? Can I check for some relevant HTTP error codes or the like?
If the HTTP server doesn't receive the entire query string in a GET request, then the HTTP request cannot possibly be valid as these parameters are on the first line of the request.
The server will be unable to handle the request and in this case will likely return status code 400 (Bad Request).
If your client receives this (which seems unlikely that it would fail to transmit the request, yet receive the response), then you'll know to retransmit. In general, the properties of TCP connections like automatic retransmissions, checksums and timeouts should be all you need for successful delivery, or to determine failure.
You need to check for timeouts on the client. That depends on the process/language used.
EDIT: http://wiki.forum.nokia.com/index.php/Using_Http_and_Https_in_Java_ME
Looks like you simply set a timeout and catch IO errors.
Premature optimization.
Connection integrity is already dealt with in the lower parts of the network stack. So if there were any dropouts in the middle of the request (assuming it spanned more than a single packet) the TCP stack would attempt to recover them before passing the data on to the server.
If you need to prove this to yourself, then just add a checksum as the last part of the query.
C.