I have two phones connected to a Wifi access point, both have ip in the private range.
One of the phones has a HTTP server running on it and the other phone acts like a client. The client sends GET requests data to the server as name/
value pairs in the URL query string. At the moment the server is only sending on HTTP.OK on receiving the query string.
What is happening is the client may not be stationary and maybe moving around so it may not be possible for it to be in range always of the Wifi access
point due to that I am not getting all the data sent from the client at the server end.
I want to ensure that all data sent is actually received by the server.
What kind of error correction should I implement? Can I check for some relevant HTTP error codes or the like?
If the HTTP server doesn't receive the entire query string in a GET request, then the HTTP request cannot possibly be valid as these parameters are on the first line of the request.
The server will be unable to handle the request and in this case will likely return status code 400 (Bad Request).
If your client receives this (which seems unlikely that it would fail to transmit the request, yet receive the response), then you'll know to retransmit. In general, the properties of TCP connections like automatic retransmissions, checksums and timeouts should be all you need for successful delivery, or to determine failure.
You need to check for timeouts on the client. That depends on the process/language used.
EDIT: http://wiki.forum.nokia.com/index.php/Using_Http_and_Https_in_Java_ME
Looks like you simply set a timeout and catch IO errors.
Premature optimization.
Connection integrity is already dealt with in the lower parts of the network stack. So if there were any dropouts in the middle of the request (assuming it spanned more than a single packet) the TCP stack would attempt to recover them before passing the data on to the server.
If you need to prove this to yourself, then just add a checksum as the last part of the query.
C.
Related
I'm looking for a proper way to have one goroutine sending out request packets to specific servers while a second goroutine receiving the responses and handling them, maybe even create a new goroutine for each response to handle.
The architecture of the game is that there are multiple masterservers, which can be asked for ip lists of registered servers.
After getting the ips and ports from the masterservers, each of the ips gets a request for its data, like server name, map, players, etc.
Also, are there better ways to handle this?
Currently I am creating a goroutine per request that also waits for a response afterwards.
The waiting for a response timeouts after 35ms and continues to send 1.2 times the previous amount of request packets to have a small burst of requests. Also the timeout is doubled on every retry.
I'd like to know if there are better strategies that have proven to be more robust and have a lower latency, that are not too complex.
Edit:
I only create the client side sockets, but would have, if there is no better approach, a client that sends UDP request packets that contain a different socket's address as sender value in order to receive the answers on a different socket that acts kind of like a server, where all the response packets are collected. In order to separate the sending socket from the receiving socket.
This question is tagged as client-server as one of the sockets is supposed to act like a server, even tho all it does is receive expected answers in response to request packets sent by the client socket.
I have a need to be able to validate TOS/DSCP marks on response data from a set of HTTP servers. Would it be possible, given a list of target URLs to test, if there is a way in go to generate the HTTP request, and then be able to examine the response's TCP packet details in order to obtain the TOS value?
My assumption at this point is that it may require creating a socket, and then dynamically generating a TCP packet that contains the HTTP request payload. I've been searching around to see if there were any libraries that would aid in this task, but haven't found anything specific yet.
Note: a simple TCP connection will not provide enough data - the target servers in question will alter TOS/DSCP marks dynamically based on the HTTP server name (so essentially, a single physical server will respond with different TOS marks depending on the vHost requested), so it is important to be able to verify the TOS on actual HTTP response packets, and not something simple like a ping. The TOS values in the TCP 3-way handshake cannot be trusted either - it must be a packet containing the HTTP data.
I did end up solving this problem using gopacket/pcap and net/http.
In a nutshell, what I ended up doing is writing a function that creates a channel, and then calls a goroutine that does the actual packet capture and parsing. The goroutine passes the captured TOS value back to the channel, and then the original function does the http request, and then reads the channel to get the TOS result. Still a bit of a work-in-progress, but so far, this solution seems to be working fairly well.
I want to know the performance gain from doing a HTTP batch request. is it only reducing the number of round trips to one instead of n times where n is the number of HTTP requests? if it's like that I guess you can keep http connection opened and send your http messages through and once finish you can close it to get performance gain.
The performance gain of doing batch requests depends on what you are doing with them. However just as an agnostic approach here you go:
If you can manage a keep-alive connection, yes this means you don't have to do the initial handshake for the connection. That reduces some overhead and certainly saves time spent handling subsequent packets along this connection. Because of this you can "pipeline" requests and decrease overall load latency (all else not considered). However, requests in HTTP1.1 are still bound to be FIFO so you can have hangups. This is where batching is useful. Since even with a keep-alive connection you can have this hangup (HTTP/2 will allow asynchronous handling) you can still have some significant latency between requests.
This can be mitigated further by batching. If possible you lump all the data needed for subsequent requests into one and this way everything is processed together and sent back as one response. Sure it may take a bit longer to handle a single packet as opposed to the sequential method, but your throughput is increased per time because roundtrip latency for request->response is not multiplied. Thus you get an even better performance gain in terms of requests handling speeds.
Naturally this approach depends on what you're doing with the requests for it to be effective. Sometimes batching can put too much stress on a server if you have a lot of users doing this with a lot of data so to increase overall concurrent throughput across all users you sometimes need to take the technically slower sequential approach to balance things out. However, the best approach will be known by you upon some simple monitoring and analysis.
And as always, don't optimize prematurely :)
Consider this typical scenario: the client has the identifier of a resource which resides in a database behind an HTTP server, of which resource they want to get an object representation.
The general flow to execute that goes like this:
The client code constructs an HTTP client.
The client builds an URI and sets the proper HTTP request fields.
Client issues the HTTP request.
Client OS initiates a TCP connection, which the server accepts.
Client sends the request to the server.
Server OS or webserver parses the request.
Server middleware parses the request components into a request for the server application.
Server application gets initialized, the relevant module is loaded and passed the request components.
The module obtains an SQL connection.
Module builds an SQL query.
The SQL server finds the record and returns that to the module.
Module parses the SQL response into an object.
Module selects the proper serializer through content negotiation, JSON in this case.
The JSON serializer serializes the object into a JSON string.
The response containing the JSON string is returned by the module.
Middleware returns this response to the HTTP server.
Server sends the response to the client.
Client fires up their version of the JSON serializer.
Client deserializes the JSON into an object.
And there you have it, one object obtained from a webserver.
Now each of those steps along the way is heavily optimized, because a typical server and client execute them so many times. However, even if one of those steps only take a millisecond, when you for example have fifty resources to obtain, those milliseconds add up fast.
So yes, HTTP keep-alive cuts away the time the TCP connection takes to build up and warm up, but each and every other step will still have to be executed fifty times. Yes, there's SQL connection pooling, but every query to the database adds overhead.
So instead of going through this flow fifty separate times, if you have an endpoint that can accept fifty identifiers at once, for example through a comma-separated query string or even a POST with a body, and return their JSON representation at once, that will always be way faster than individual requests.
Consider a service running on a server for a customer c1,but customer c1 times out after 'S' sec for what so ever be the reason so customer again fires the same request ,so server is running duplicate query hence it gets overloaded, resolve this glitch. Please help me !!!
I assume you are on the server side and hence cannot control multiple requests coming in from the same client.
Every client should be having an IP address associated with them. In your load balancer(if you have one) or in your server you need to keep an in-memory cache which keeps track of all requests, their IP addresses, timestamp when request originated and timestamp when request processing finished. Next you define and appropriate time measure - which should be near about 70-80% percentile of processing time for all your requests. Lets say X seconds.
Now, before you accept any request at your loadbalancer/ server you need to check in this in-memory cache whether the same IP has sent the same request and the time elapsed since the last request is less than X. If so do not accept this request and instead send a custom error stating something like "previous request still under processing. Please try after some time".
In case IP address is not enough for identifying a client, as the same client may be sending requests to different endpoints on your server for different services, then you need to store another identifier which maybe a kind of token/session identifier - such as c1 or customer id in your case. Ideally, a customer can send only 1 request from 1 IP Address to an endpoint at any 1 point of time. Just in case you have mobile and web interfaces then you can add the channel-type(web/mobile/tablet) as well to the list of identifying parameters .
So now, a combination of - customer id(c1), IP address, request URL,request time, channel-type will always be unique for a request coming in. Using a key of all these parameters in your cache to uniquely fetch information for a request and validating whether to start processing the request or send a custom error message to prevent overloading the server with re-requests - should solve the problem defined above.
Note - 'S' seconds i.e. client-side timeout - given that the client-side timeout is not in our control - should not concern the server-side and will have no bearing on the design I have detailed above.
I am trying to check pop and smtp values entered by user.. I wish to validate that pop and smtp say for example(pop.gmail.com,smtp.gmail.com) which is entered by user is correct or wrong.
For that I am sending only one request to server by taking both pop and smtp values entered by user which will do two tasks
1. Checks user entered pop by making connection to that particular server ,
2. Checks user entered smtp by sending 1 mail to some dummy mail id..
I finished all these tasks..
But now what my requirement is, I have to show the user after validating each thing.. I mean in ui i have to show as
POP connection Checked.. ok
smtp connection Checked.. ok like that.
But i sent only one request to server for doing both these tasks..So i need to get intermediate status from server after finishing each tasks..So only i can update in client side UI.. But i don't know is it possible to get intermediate responses from server for a single request... Any idea friends? If so can you come up with a little bit of code...
Expecting the suggestions?
you should take a look in the long polling technique, it is possible to retrieve partial response but it doesn't work on all browsers.
You can use HEAD request instead of GET or POST which only return HTTP header
Slightly off topic - but sending a dummy mail can be "dangerous".
Many servers "note" if you try and send to a local address, which does not exist. For example - if the server's domain is "whatever.com" and you send to a random address, say aaa#whatever.com, and "aaa" is not a valid user, then the server notices this.
The server may then take an action like blocking you, as a sender, for a period of time. (This helps to reduce spam from dictionary attacks.) So your "test" ends up effectively blocking the real mail from being delivered.
The reverse is also true. Let's say you try to send to an external address, which you know is valid (your own email address for example) as the test. In this case the from address must be a valid internal address. If you use an invalid internal address, or worse an address which is not internal, it's likely the server will refuse to deliver the mail (at best) and at worst again institute a temporary block.
The key factor in both these situations is that although the SMTP protocol is very "loose", SMTP servers watch very closely for "bad behavior" because this is one way of distinguishing a spamming program. So any hide of "incorrect" behavior can lead to it arbitrarily refusing to accept your mails (usually for a limited period of time.)
Incidentally, back to your original question.
Both of your tests are pretty much instantaneous. Even if the email server is on the other side of the world you can do both checks inside a couple seconds. So chances are even if you send back 2 packets, to the user they'll appear as "arriving together". And since 1 request from the browser can only handle 1 response from the server you would need to send the response in 2 packets.
ie do first test - send first part of response - do second test - send second part of response.
For a normal HTTP packet this is no big deal. Do some sort of flush / send after the first response is ready, and then again after the second response. The browser is used to displaying partial pages as they arrive.
However for an AJAX request you'll need to get into your framework at quite a low level. Most frameworks, that I'm aware of, require the incoming Async packet to be "complete" before they start to parse it. This is especially true if the packet is formatted as say xml where partial parsing is useless in pretty much all cases.