I've written a class in Ruby that acts as an HTTP client. The code is minimal but the reason I'm not using 'net/http' is because this method allowes me to have more control over the requests being made and documentation for the HTTP is not helpful at all.
Anyway, the problem is the socket will only work for one request and response. Sending a second or subsequent request gives me an empty response.
For example:
Open connection to google
GET "/"
Response is the google.ca html
GET "/"
Response is empty
I tried closing and opening the connection between the requests but that only slowed it down and didn't fix the problem. I still got empty responses.
So what is the problem here?
Is there a method that lets me check to see if the TCPSocket object has an open connection so I don't accidentally open a new one?
Try:
require "socket"
host = "google.com"
port = 80
socket = TCPSocket.new host,port
request = "GET / HTTP/1.0\r\nHost:#{host}\r\n\r\n"
socket.print request
response = socket.read
This will return Google's main page. If you want to send request after request then change it to "HTTP/1.1" and read a response, then send the next request.
Related
I am trying to make a web server in ruby using socket for experimentation, and I am unable to get the payload from a post request until the client closes. Then, I can not send any data back when the client does close.
Code:
require "socket"
server = TCPServer.new($HOST, $PORT)
client = server.accept
while true
puts client.gets
#Prints all headers and request but I don't see the actual payload (key=value) until the client disconnects.
end
By using if line.chomp.length == 0 to get the end of the headers and client.read() for the data, I can get the data.
I am trying to fetch from API server using Net::HTTP.
puts "#{uri}".green
response = Net::HTTP.new('glassdoor.com').start { |http|
# always proxy via your.proxy.addr:8080
response = http.get(uri, {'Accept' => 'application/json'})
puts "Res val: #{response.body}".blue
}
I got the uri from the console and pasted in the browser, and I received the JSON response.
But using the Ruby Net::HTTP get I receive some security message:
Why the difference? The browser and the Ruby script are behind the same public IP.
You were detected as a crawler (correctly, by the way). Note that those requests (from browser and the script) are not just the same. The browser sends some headers, such as accepted language, user agent etc. You can peek into it using web inspector tool in the browser. On the other side, in your script you only set Accept header (and to JSON, suspicious on its own, as browser would never do that). And you do not send any user agent. It's easy to see that this is an automates request, not natural traffic from the browser.
I have a WebSocket server written which only handles upgrade requests which are GET requests. If a POST or any other kind of request with the required headers comes it is handled by a HTTP server.
In the specification it is not stated explicitly that the WebSocket upgrade request should be a GET request.
If the upgrade request is not a GET request should the server handle it as a WebSocket upgrade request, should it pass it to be handled by the HTTP server or should it respond to it with a status code like 400 Bad Request ?
Could this be a design decision where the server decides not to handle methods which are not GET requests?
From section 4.1 (Client Requirements) of the webSocket specification, it says this:
The method of the request MUST be GET, and the HTTP version MUST
be at least 1.1
And, then later in section 4.2.1 (Reading the Client's Opening Handshake) of the webSocket specification, it says this:
The client's opening handshake consists of the following parts. If
the server, while reading the handshake, finds that the client did
not send a handshake that matches the description below (note that as
per [RFC2616], the order of the header fields is not important),
including but not limited to any violations of the ABNF grammar
specified for the components of the handshake, the server MUST stop
processing the client's handshake and return an HTTP response with an
appropriate error code (such as 400 Bad Request).
An HTTP/1.1 or higher GET request, including a "Request-URI"
[RFC2616] that should be interpreted as a /resource name/
defined in Section 3 (or an absolute HTTP/HTTPS URI containing
the /resource name/).
So, there are multiple places where it says the http request must be a GET.
As for your specific questions:
Should WebSocket server only handle GET requests?
Yes, a webSocket connection will always start with a GET request, not a POST or any other method.
If the upgrade request is not a GET request should the server handle it as a WebSocket upgrade request, should it pass it to be handled by the HTTP server or should it respond to it with a status code like 400 Bad Request ?
As described in the above reference portion of the specfication, the server should respond with a status code like 400 Bad Request.
Could this be a design decision where the server decides not to handle methods which are not GET requests?
Yes.
I am trying to create an HTTP client that uses persistent connections. My Code works when I send my first request and get my first response. However, when I send a second request, I am unable to get a second response. I am not sure why? I got the same error when I was coding in C.
Here is the code
require 'socket'
include Socket::Constants
socket = Socket.new( AF_INET, SOCK_STREAM, 0 )
sockaddr = Socket.pack_sockaddr_in( 80, 'www.google.com' )
socket.connect( sockaddr )
# This Works
socket.write( "GET / HTTP/1.0\r\n\r\n" )
results = socket.read
# This Works
socket.write( "GET / HTTP/1.0\r\n\r\n" )
# THIS DOESN'T WORK
results = socket.read
I do not want to use built libraries like Net::HTTP. What do I need to do to make this work?
You cannot make 2 HTTP requests on the same connection, unless you've told the server that you're expecting to do so. This is how HTTP persistent connection works. At a minimum, you have to make sure to add this to your request header:
Connection: keep-alive
Servers have differing support for persistent connections, although it's become common for servers to support basic persistent connections. Here's an SO question that asks What exactly does a “persistent connection” mean?
Start there, and you'll find what you need to know to make persistent connections work correctly. You may have to check the HTTP response headers for an indication that the server will honor your request, or you may have to check that the server didn't simply close the connection when it was finished writing the first response. On your final request through a persistent connection, you should also specify the header:
Connection: close
Also check out these resources:
IETF HTTP 1.1 specification
W3 HTTP 1.1 section 8: Persistent Connections
Safari Books Online HTTP: The Definitive Guide - Persistent Connections
I have a custom Http Handler which manipulates HTTP POST and GET. I got the project working on a seperate isolated server now need to put it in production...
using (var client = new WebClient())
{
client.Credentials = CredentialCache.DefaultCredentials;
client.UploadFile("serverlocation:port", fileToUpload);
}
For some reason now when using client.UploadFile("", file); i.e. forcing the HTTP POST
System.Net.WebException: The remote server returned an error: (417) Expectation failed.
at System.Net.WebClient.UploadFile(Uri address, String method, String fileName)
What could this be? I know the code works, so what else? Maybe the server blocks HTTP POST requests?
I have tried adding:
ServicePointManager.Expect100Continue = false;
But have had no success though i'm not 100% sure where this code should before, I assume before i'm using the WebClient
Edit 0 :
I have just read the following:
Because of the presence of older implementations, the protocol allows
ambiguous situations in which a client may send "Expect: 100-
continue" without receiving either a 417 (Expectation Failed) status
or a 100 (Continue) status. Therefore, when a client sends this
header field to an origin server (possibly via a proxy) from which it
has never seen a 100 (Continue) status, the client SHOULD NOT wait
for an indefinite period before sending the request body.
I believe this request is going through a proxy, which may have something to do with the issue.
Edit 1:
Believe this problem has to be with 100-continue because, using fiddler to see exactly what my application is sending with WebClient.UploadFile shows this:
POST http://XXX.XXX.XXX.XXX:8091/file.myhandledextension HTTP/1.1
Content-Type: multipart/form-data; boundary=---------------------8ccd1eb03f78bc2
Host: XXX.XXX.XXX.XXX:8091
Content-Length: 4492
Expect: 100-continue
Despite having put that line: ServicePointManager.Expect100Continue = false; before the using statement. I don't think this line actually works.
I ended up solving this by putting the ServicePointManager.Expect100Continue = false; in the constructor for the calling WebClient class.
Then I used Fiddler to examine the POST request to ensure Expect: 100-continue was not in the request anymore.